* Libertarian firm argues tax agency lacks authority
* IRS claims it can regulate tax return preparers
* Thousands of large, small businesses await decision
WASHINGTON, Sept 23 (Reuters) - The U.S. government's first effort to regulate tens of thousands of tax return preparers will be tested on Tuesday when the Internal Revenue Service squares off in court against a libertarian law firm.
The Institute for Justice, funded by leading conservative activists, will argue that the IRS lacks the legal authority to require return preparers to pass a competency test, and to pay for continuing education classes.
The IRS will say that it has ample congressional authority to impose such rules on the return prep industry, and that tougher oversight is needed to protect Americans from unscrupulous and incompetent preparers.
The outcome of the case, not expected for several months, will have a big impact on the tax return prep industry and on the government's ability to police it.
More than 78 million Americans in 2011 paid someone to prepare their tax returns, which are complicated and will get more so as the IRS helps implement President Barack Obama's new healthcare laws.
The return prep industry posted revenue this year of $9.4 billion, said research group IBISWorld Inc.
H&R Block Inc is by far the largest competitor, with about 32 percent of the market, while mid-tier rivals such as Jackson Hewitt Tax Service Inc and JTH Tax Inc have a combined 3 percent share, IBISWorld estimated.
The remainder is scattered among thousands of small, "dinner table" return preparers. The industry has never been regulated.
The IRS had no immediate comment on Monday.
Concerns about tax fraud and sloppy returns led the IRS in 2011 to begin moving toward greater oversight.
The agency spent $50 million to design a competency test and open more than 250 testing centers, while signing up about 100,000 preparers to take the test. The IRS collected more than $105 million in registration and test fees.
The Institute for Justice sued in March 2012 to try to block the program. Based in Arlington, Va., the group litigates over issues such as private school vouchers and eminent domain. It was launched in 1991 with funding from wealthy industrialists and conservative activists David and Charles Koch.
Sabina Loving, a Chicago tax preparer, is the lead plaintiff in the case. She came to the institute via its entrepreneurship clinic at the University of Chicago. She was put in touch with institute attorney Dan Alban.
"We're interested in protecting people's right to economic liberty," Alban said. "Congress hadn't passed new laws authorizing the agency to do this."
In January, the institute won a district court order suddenly halting the IRS's new initiative. The government is appealing that ruling.
"This was a major project" for the agency, said Floyd Williams, formerly IRS chief of legislative affairs. "Everyone was shell-shocked at the ruling."
On Tuesday, a three-judge panel of the U.S. Court of Appeals for the District of Columbia will hear the case. All three of the judges were appointed by Republican presidents.
Fees collected from return preparers are being held by the IRS and may have to be refunded if the IRS loses.
In court filings, the IRS is arguing that it has broad congressional authority to write tax code rules. The institute argues Congress never passed a law specifically allowing the IRS to impose return preparer rules on its own.
Despite broad, bipartisan political support for regulating return preparers, the IRS lost several opportunities to get that specific authority signed into law, prompting the agency to write its own rules, said Don Williamson, a tax accountant and executive director of American University's Kogod Tax Center.
"As a tax practitioner, I'm in favor of regulation," Williamson said. "As a citizen, this is classic over-reach."
Amid expectations that IRS regulation would squeeze smaller players out of the return prep market, large competitors such as Jackson Hewitt and H&R Block generally backed IRS oversight.
In Charlotte, North Carolina, Najam Usmani is watching the court case closely. Usmani owns 22 Charlotte-area Jackson Hewitt stores and employs about 100 people during tax season.
"More regulation is good for us," Usmani said. "People are doing taxes out of their basement. We've been seeing a decline in the business because of all these moms and pops who open up out of nowhere."
The case is Sabina Loving et al v. Internal Revenue Service No. 13-5061.