* EPS of 45 cents tops analysts' estimate of 42 cents
* Revenue rises 8 percent to $130.1 million
* Incoming CEO Tilly: VIX products to be a main focus
* Reserves $5 million for possible resolution of SEC probe
By Ann Saphir
Feb 8 (Reuters) - CBOE Group Holdings Inc reported a bigger-than-expected rise in fourth-quarter earnings on Friday on a jump in trading of contracts tied to Wall Street's favorite fear gauge, and the exchange operator's CEO-elect promised further focus this year on those high-margin contracts.
The operator of the oldest U.S. stock-options trading venue also said it was in talks with the U.S. Securities and Exchange Commission to settle a previously disclosed probe into its self-regulatory compliance and had reserved $5 million for a possible resolution.
CBOE has revealed few details of the probe, which is centered on the exchange's role as a front-line overseer of Chicago brokerage OptionsXpress. The investigation has cast a shadow on the mostly lustrous legacy of longtime chief executive William Brodsky, who plans to hand the reins to President and Chief Operating Officer Ed Tilly in May.
Brodsky, who will continue as CBOE's executive chairman, successfully turned the Chicago-based exchange operator from a private member-run club to a public company 2-1/2 years ago, and has since overseen growth that has kept it ahead of the competition in a crowded field. There are about a dozen U.S. stock-option markets.
Several high-level compliance officials left CBOE after it acknowledged the probe a year or so ago, and the CBOE recently removed board directors with ties to trading firms in a move it said responded to the SEC probe.
NYSE Euronext was fined $5 million last September to settle SEC allegations that the parent of the Big Board gave some customers an early look at data.
Nasdaq OMX Group Inc is also in talks with SEC for flubbing Facebook Inc's initial public offering, sources have told Reuters.
On a call with analysts on Friday to discuss the quarter's financial results, CBOE executives said they had boosted fees to pay for higher regulatory costs and would do more capital spending to enhance systems and software for regulatory purposes.
They did not discuss the probe or the possible settlement.
Instead, they focused on the surge of investor interest in options and futures on the CBOE Volatility Index. This is one of the exchange's most profitable contracts because an exclusive licensing agreement keeps all rivals from offering lookalike versions.
Trading in CBOE's exclusive index products, which also include options on the Standard & Poor's 500 Index, accounted for about 31 percent of overall buying and selling at the company's exchanges and generated 59 percent of revenue in the quarter.
Trading of the relatively new VIX fear gauge contracts is still on a "hockey-stick" trajectory of growth, Chief Financial Officer Alan Dean said.
That success is driving CBOE to try to shift even more of its business into such high-profit contracts, Tilly said, calling proprietary products "what we do best."
CBOE opened a London hub a few days ago to capture European business and plans to expand trading hours later this year. Later this month, it will move one of its S&P 500 index options from its all-electronic New York market, where it had a lackluster launch, to its main Chicago market, a move Tilly said should help boost volume.
In contrast to its thriving index business, the company's single-stock options trading suffers from relentless pressure from competitors that include NYSE Euronext and Nasdaq. CBOE's market share dropped in the quarter to just above 26 percent from slightly above 29 percent the prior quarter.
CBOE has cut pricing to lure back lost business, but the success of that move is not clear.
Shares of CBOE were up 0.1 percent at $34.34 on Friday afternoon on the Nasdaq.
In the fourth quarter, net income rose to $39.2 million, or 45 cents a share, from $31.3 million or 35 cents a share a year earlier. Analysts on average had expected 42 cents a share, according to Thomson Reuters I/B/E/S.
Revenue rose 8 percent to $130.1 million.
Trading fell to a daily average of 4.1 million contracts, but activity in CBOE's exclusive index options helped boost the average per-contract fee to 35.5 cents from 32.1 cents.
The company said it expects operating expenses of $189 million to $194 million in 2013.