* Only new market entrants allowed to bid for 20-yr frequencies
* RAI, Mediaset can only bid for mobile phone TV frequencies
* Ruling fresh blow to Mediaset, former PM Berlusconi (Adds detail, background)
ROME, Nov 14 (Reuters) - Italy's telecoms authority ruled on Wednesday that only new entrants to the digital television market could compete for new full 20-year frequency licences, excluding state broadcaster RAI and former Prime Minister Silvio Berlusconi's Mediaset.
The authority ruled that no company could hold more than five so-called multiplex groups of channels transmitted on digital bandwidth at the end of the auction.
That effectively ruled out RAI and Mediaset, Italy's biggest commercial broadcaster, which already each own five multiplex.
The authority's ruling left the two groups free to bid for new five-year frequency options on 700 megahertz bands dedicated to mobile telephone television.
The auction, which Industry Minister Corrado Passera has said will be held before the end of the year, is estimated to fetch 1.1 billion to 1.2 billion euros.
The ruling is another blow to Berlusconi, who was convicted last month of tax fraud and is on trial for allegedly having sex with an under-aged prostitute. He has seen the popularity of his People of Freedom party plummet since he lost office a year ago.
On Tuesday Mediaset announced that in the third quarter it posted its first-ever quarterly loss and its share price has fallen more than 40 percent since the start of the year.
Earlier this year, Mario Monti's government scrapped plans by his predecessor, Berlusconi, to assign the frequencies through a process known as a "beauty contest" based on the quality of a broadcaster's projected use of the frequencies rather than auctioning them to the highest bidder. Critics said would have favoured Mediaset and RAI.
Besides Mediaset and RAI, Telecom Italia Media, Sky Italia, Hutchison Whampoa's local telecoms operator 3, Europa Way, Prima Tv and Canale Italia had all lined up for the beauty contest before it was cancelled. (Reporting by Alberto Sisto, writing by Gavin Jones; Editing by Dan Grebler)