* Rokos suing Brevan Howard to overturn non-compete agreement
* Plans to launch his own fund
* Brevan Howard says Rokos seeks to avoid "responsibilities"
* Rokos rejects hedge fund's counterclaim in new filing (Writes through, adds detail, industry comment, background)
LONDON, Aug 26 (Reuters) - Former Brevan Howard star trader Chris Rokos has stepped up his legal challenge to a partnership agreement imposing a five-year ban on him setting up in competition against the hedge fund firm he co-founded.
The contractual dispute offers a rare insight into one of Europe's largest hedge funds, with its response to Rokos's challenge and the potential loss of investors to a new rival highlighting the business risks such funds face when top traders strike out on their own.
Court documents seen by Reuters show that Rokos, who left Brevan Howard in 2012, issued a new filing last week after a counterclaim the firm filed this month in response to the lawsuit begun by Rokos in May.
The firm had urged the court to reject the Rokos lawsuit, which is seeking to overturn an agreement preventing him from starting his own fund and raising money from investors until 2018, but Rokos's latest filing denies that Brevan Howard is entitled to enforce the agreement and reiterated his original claims.
Rokos's surname is represented by the letter R in Brevan Howard and he was marketed by the firm as a "star trader" who was paid $900 million during his stint at the hedge fund.
Yet a fight between the two parties and an unusually long five-year employment restriction -- 12 months is more typical -- has surprised many in the $3 trillion industry.
"I am surprised that they (Rokos and Brevan Howard) are handling the issue in such a way, and publicly," said Michele Gesualdi, portfolio manager at hedge fund investor Kairos. "It's not in the interest of anybody."
Rokos, who made about $4 billion for Brevan's main fund between 2004 and 2012, accounting for about 17 percent of its profits, called the restraint on his ability to start a fund as "unreasonable", preventing him from working "for one quarter of his remaining career".
Such a curb would "atrophy" his skills and his professional reputation would be "irreparably damaged", he said in the lawsuit.
The restrictions are "contrary to public policy and in restraint of trade, and unenforceable", Rokos said. He also asked the court to rule that Brevan Howard should continue to pay him a share of the firm's profits until 2018.
The firm, meanwhile, has argued that Rokos is seeking to avoid "obligations and responsibilities" designed to protect the hedge fund's business, goodwill and reputation, because "they no longer fit with his current wishes".
"Yet he still seeks to retain all financial benefits conferred on him under those agreements and to exploit the high personal profile, reputation and track record that Brevan Howard has afforded him to damage its interest," the firm said.
Brevan Howard, which has lost money this year in a rare miss since its launch more than a decade ago, managed $36 billion at the end of July, it said in letter to investors that was seen Reuters.
Since leaving the hedge fund, 43-year-old Rokos has spent his time running a family office in London to manage his own money, but the first signs of his discontent with the non-compete clause emerged within two weeks of his receiving yearly payments due to him as a retired partner of the firm.
The money -- $72.97 million, court documents showed -- was paid only after Rokos confirmed that he was sticking to the terms of the agreement with Brevan. Within days, Rokos's lawyers sent a letter to Brevan Howard disclosing his intension to start a fund.
Alan Kilkenny, a spokesman for Rokos, declined to comment on Tuesday, as did a spokesman for Brevan Howard.
The $25.8 billion Brevan Howard Master Fund has never lost money in a calendar year since its launch in 2003, according to the letter to investors.
It returned 20.4 percent in 2008. That performance at a time when a global financial crisis wiped billions of dollars off financial markets served to attract new investors, more than doubling the firm's assets under management since then.
The fund has had a tough time this year, losing money every month in the first half before gaining 0.7 percent in July to cut 2014 losses to 3.7 percent. (Editing by David Goodman)