* Justice Department battles rules hindering cheaper cards
* American Express vows to fight antitrust lawsuit
* Shares of AmEx finish down 6.5 percent (Adds comments from AmEx, government, investor, analyst, lawyer, background on regulation, updates shares to close)
WASHINGTON/NEW YORK, Oct 4 (Reuters) - The U.S. Justice Department sued American Express Co (AXP.N), Visa Inc and MasterCard Inc on Monday, accusing them of violating antitrust laws and citing rules that prevented merchants from encouraging consumers to use cheaper credit cards.
Simultaneously, the Justice Department settled with Visa (V.N) and MasterCard (MA.N), which agreed to allow merchants to offer discounts to consumers who use less expensive types of credit or debit cards. The companies said the settlement, subject to court approval, did not involve any payment.
The lawsuit has the potential to cut into a significant source of profits for American Express and threatens to reshape the competitive landscape of the card processing business.
Shares of American Express, which charges merchants more on average for processing credit card transactions than its rivals, closed down 6.5 percent at $39.05.
Consultant Philip J. Philliou, a former executive for American Express and MasterCard, said via email that the lawsuit was "problematic for AmEx.
"If nothing else, it highlights to retailers and consumers the premium that retailers pay to accept AmEx as a form of payment," he wrote.
"This ruling may impact which card a consumer pulls out of their wallet," Philliou added.
American Express said the lawsuit would hurt consumers by limiting their ability to use their AmEx card.
"We have no intention of settling the case," Kenneth Chenault, chairman and chief executive officer of American Express, said in a statement.
American Express executives and lawyers declined repeatedly on a conference call with investors and reporters to quantify the potential impact of losing the lawsuit, or to say how much it would have cost the company in revenue to settle the case.
"I think, with reason, AmEx thinks the negative impact would be material," consultant Eric Grover, who previously worked at Visa, said in an email.
Attorney General Eric Holder said at a press conference on Monday that credit card companies "put merchants and consumers in a no-win situation. Accept our card, pay our fees and don't even think about trying to get a discount."
Merchants pay fees to banks and processing networks like Visa and MasterCard every time a customer pays for something by using a credit or debit card. These so-called interchange fees usually amount to between 1 to 3 percent of each total bill.
American Express, which lends directly to consumers and processes credit card transactions, charges merchants a higher percentage on average. AmEx said on the conference call that merchants get more business from American Express cardholders in exchange for the higher fees because its wealthy customers tend to spend more than the average credit card user.
Visa, MasterCard and American Express and their affiliated banks collected more than $35 billion in fees from U.S. merchants in 2009, according to the lawsuit.
American Express cardholders charged $419.8 billion worth of purchases in 2009, while MasterCard cardholders charged $476.9 billion and Visa card holders charged $764.2 billion.
Christine Varney, head of the Justice Department's antitrust division, said the government would try to resolve the case as quickly as possible.
But American Express Vice Chairman Edward Gilligan told the conference call, "We anticipate that this process will take several years before it is resolved.
Both merchants and consumer groups hailed the government's action against the credit card companies.
Ed Mierzwinski, consumer program director of the U.S. Public Interest Research Group, said in a statement that the lawsuit and settlement "will finally open competition in the card payment market, ultimately leading to lower prices to consumers."
But in a statement late Monday afternoon, the American Bankers Association said it looked "forward to seeing whether (merchants') new-found pricing authority will actually result in consumer benefit or merely be used to pad their own bottom lines."
Jan McDavid, an antitrust attorney who represents American Express in other matters, said the government would have trouble proving that American Express had market power, essential to this case, because it had said the opposite in 1998 when it sued Visa and MasterCard in an antitrust case.
"Visa and MasterCard have 70 percent of the credit card market. If you think about debit cards, American Express's market share goes down even more," said McDavid. "I think Christine's got a tussle on her hands."
Visa shares closed down 0.10 percent at $73.24, and MasterCard shares closed down 1 percent at $222.64. Investors attributed the declines to uncertainty about a law due to go into effect next year that will limit debit-card transaction fees merchants pay both companies and their partner banks.
"Any time you can get a headwind like the DoJ lawsuit behind you that's a positive ... but I think the overriding issue right now is (still) the Durbin amendment" restricting debit fees, said Michael Nix, a portfolio manager at Greenwood Capital Associates, who owns Visa shares.
Visa said that as part of the settlement, it would allow merchants to offer discounts to consumers who use certain types of cards, such as non-rewards credit cards, which carry lower interchange fees.
Until this year, these fees have been relatively unregulated in the United States.
"Credit cards cost phenomenally more in the U.S. than abroad and the DOJ action will help open the market," said David Balto, a former policy director at the Federal Trade Commission's competition office. (By Diane Bartz in Washington and Maria Aspan in New York; Editing by Tim Dobbyn)