* Judge says settlement "goes too far"
* Justice Department has also criticized the deal
* Google says ruling is disappointing (Adds authors, publishers, Justice reaction, share prices)
By Diane Bartz
WASHINGTON, March 22 (Reuters) - A U.S. judge on Tuesday rejected a $125 million settlement between Google Inc (GOOG.O) and authors that would have let the company publish millions of books online to create the world's largest digital library.
New York Judge Denny Chin said the deal gave Google a significant competitive advantage and "would simply go too far" in giving it the power to "exploit" digitized copyrighted works by selling subscriptions to them online without permission.
Google has scanned roughly 12 million books from some of the country's finest libraries, in what it has said was an effort to provide easier access to the world's knowledge.
It was sued in 2005 by the Authors Guild and Association of American Publishers for violating copyright laws, but reached a settlement by agreeing to pay $125 million to people whose copyrighted books have been scanned, and to locate and share revenue with the authors who have yet to come forward.
Still, some critics contended that the deal gave Google an unfair competitive advantage and broke antitrust law. Chin -- a judge on the Second Circuit Court of Appeals who heard the case while he sat in district court -- agreed on both counts.
"The ASA (amended settlement agreement) would give Google a significant advantage over competitors, rewarding it for engaging in wholesale copying of copyrighted works without permission," Chin wrote in rejecting the deal.
He urged Google and authors and publishers to amend the settlement to only include books whose copyright owners have agreed to the arrangement, rather than a blanket deal that would require authors to "opt out" should they not want their scanned books to be sold online.
Speaking for the publishers, John Sargent, chief executive of Macmillan publishing group, said he hoped the deal could be renegotiated.
"The publisher plaintiffs are prepared to enter into a narrower settlement along those lines to take advantage of its groundbreaking opportunities. We hope the other parties will do so as well," said Sargent in an email statement.
Chin rejected the settlement "without prejudice," meaning a revised pact could be submitted.
The Authors Guild's President Scott Turow also noted Chin's push for an "opt-in" deal and said it hoped to talk to Google and the publishers group to "arrive at a settlement within the court's parameters that makes sense for all parties."
Critics of the proposed settlement also include Amazon.com Inc (AMZN.O), which sells the Kindle digital reader that would not be compatible with Google's library, and Microsoft Corp (MSFT.O). Sony Corp (6758.T), which makes an e-reader compatible with Google's software, favors the pact.
The Justice Department has looked into the deal, including at a group of books described as "orphan works" because they are still in copyright but the rights holder cannot be found. The settlement gives Google the right to market these works.
The Justice Department was pleased with Chin's decision, said spokeswoman Gina Talamona, who said the settlement "created concerns regarding antitrust, class certification and copyright issues."
The French government had opposed the deal but reached an agreement with Google in early 2010 to allow French works to be scanned without surrendering control of copyright. The German government opposes the deal.
The ruling comes as Google Co-founder Larry Page is set to take the CEO reins in April from Eric Schmidt. Page is an early champion of digital books and personally scanned many of the works himself early on in the project.
Analysts said the setback would have little impact on Google's financials, since any revenue would have paled in comparison to Google's main Internet search business.
Google shares closed up 0.14 percent at $577.32 on Tuesday.
"This is clearly disappointing, but we'll review the court's decision and consider our options," Google's managing counsel, Hillary Ware said in an emailed statement.
Since the settlement, Google launched in December an electronic bookstore with three million books, with permission from the relevant publishers.
Under the agreement, authors and publishers would register works and be paid for books and other publications the search giant puts online. The settlement covered books that were out of print but still under copyright.
The case is The Authors Guild et al v. Google, Inc, U.S. District Court, Southern District of New York, No. 05-08136. (Reporting by Grant McCool and Diane Bartz; editing by Richard Chang, Andre Grenon and Carol Bishopric)