More than 3,000 people have fallen for the ruse since 2013, Timothy Camus, a Treasury deputy inspector general for tax administration, said Thursday.
The scam has claimed victims in almost every state, Camus said. One unidentified victim lost more than $500,000.
"The criminals do not discriminate. They are calling people everywhere, of all income levels and backgrounds," Camus told the Senate Finance Committee at a hearing. "The number of complaints we have received about this scam make it the largest, most pervasive impersonation scam in the history of our agency."
The scam is so widespread that investigators believe there is more than one group of perpetrators, including some overseas.
Camus said even he received a call from one of the scammers at his home on a Saturday. He said he had a stern message for the caller: "Your day will come."
So far, Camus said, two people in Florida have been arrested. They were accused of being part of a scam that involved people in call centers in India contacting U.S. taxpayers and pretending to be IRS agents.
"These criminal acts are perpetrated by thieves hiding behind telephone lines and computers, preying on honest taxpayers and robbing the Treasury of tens of billions of dollars every year," said Sen. Orrin
The inspector general's office started warning taxpayers about the scam a year ago, but it has since ballooned. Tax scams often increase during filing season.
As part of the scam, fake IRS agents call taxpayers, claim they owe taxes, and demand payment using a prepaid debit card or a wire transfer. Those who refuse are threatened with arrest, deportation or loss of a business or driver's license, Camus said.
The callers might even know the last four digits of the taxpayer's Social Security number, Camus said.
They request prepaid debit cards because they are harder to trace than bank cards. Prepaid debit cards are different from bank cards because they are not connected to a bank account. Instead, consumers buy the cards at stores, and use them just like a bank card, until the money runs out or they add more.
Real IRS agents usually contact people first by mail, Camus said. And they never demand payment by debit card, credit card or wire transfer.
"Our message is simple," Camus said. "If someone calls unexpectedly claiming to be from the IRS with aggressive threats if you do not pay immediately, it is a scam artist calling. The IRS does not initiate contact with taxpayers by telephone. If you do owe money to the IRS, chances are you have already received some form of a notice or correspondence from the IRS in your mailbox."