Few phone calls will send someone into a panic like one from the Internal Revenue Service warning you that you're about to be sued, offering no other details, and demanding you hand over the money right away.

As the IRS warns on its website, these calls are always scams. The agency will never ring you out of the blue, it doesn't immediately arrest or sue non-payers, and IRS agents will never ask for credit card information over the phone.

Even with the copious warnings, it's estimated that over 5,500 people are victims of phishing phone calls and emails each year, the scammers getting away with over $29 million in 2016 — and that's just counting the people who know they are victims and have come forward.

The IRS tries mightily to fight against scams, while also combating the filing of fraudulent returns. In fact, the IRS cut the number of identity theft affidavits filed in half during the 2016 tax season.

But this progress is imperiled by a chronic staffing shortage — one that will only be made worse by President Donald Trump's hiring freeze for federal employees.

According to a report from the Center on Budget and Policy Priorities, the IRS has had to cut 14 percent of its workforce, or 13,000 employees, since 2010.

The staffing shortage could have a devastating effect on the agency's ability to combat scams and identity theft. The CBPP estimates that the Taxpayer Protection Program, which helps victims of scams, has so few employees that as few as 10 percent of calls it receives are even being answered.

With the 2017 tax season in full swing, scammers are hitting the ground running — and the understaffed IRS is scrambling to keep up.

Several dozen companies, colleges, and universities have already reported being fooled into sending either social security numbers or 2016 W-2 forms to scammers because of phishing emails sent in what are known as "Business Email Compromise" (BEC) attacks.

According to security website Salted Hash, new BEC attacks come in the form of emails purportedly from company executives asking for employee data.

According to Salted Hash, these attacks increase as April 15 gets closer; in March 2016, the site found three dozen cases of major companies being fooled into sending employee data to scammers.

IRS Commissioner John Koskinen is worried about his agency's ability to tackle this and other duties.

"A hiring freeze would affect the filing season because it would interfere with our ongoing ability to hire temporary and seasonal employees,” Koskinen told reporters during a January conference call.  

"We had high hopes that we would be able to increase the amount of money we got for taxpayer service, identify theft, and cyber security," Koskinen said. "So we still have gaps in our ability to provide work in all those areas.”

Trump's pick to head the Treasury Department, Steven Mnuchin, also thinks the IRS is understaffed.

“I don't think there's any other government agency that's gone down 30 percent, and especially for an agency that collects revenues, this is something that I'm concerned about," he told Sen. Orrin Hatch (R-Ut.), as Vox reported.

While it may not have the staff to help all those who have been scammed, the best bet is not being victimized in the first place. The IRS recommends if you receive an unwanted call or email from a faux IRS agent that you hang up or delete the email, immediately, and never give information to anyone without first confirming their identity.