RPT-Year of tough decisions for Europe's investment bankers

* Regulatory, euro zone pressure mount for investment banks

* All firms cutting back, some could even exit industry

* Slumping fees will force hand of banks reluctant to quit

* U.S. firms, Barclays, Deutsche seen among those to thrive

By Sarah White

LONDON, Jan 16 (Reuters) - Investment bankers in Europe will likely have to display more than their usual flair for outwitting rivals in 2012.

Against a backdrop of regulatory pressures, restricted availability of finance and strong economic headwinds, banks are battling for a position among the leaders in any given sector, outside which returns get increasingly hard to sustain.

And without that leading position, dramatic scalebacks and strategic overhauls are an increasingly likely course.

Some, like Britain's majority government-owned Royal Bank of Scotland, have already abandoned global ambitions for their investment banks. And new regulations and turmoil in the euro zone may accelerate this pullback in the rest of the industry in coming months.

"For the 20 firms that used to want to be top five in investment banking, you've now got about 10 to 12," said Peter Hahn of Cass Business School.

Saddled with high costs and hit by more than six months of slumping trading income, even those topping global industry rankings, like the major U.S. investment banks, are having to shed jobs and underperforming divisions.

From Switzerland's UBS, wounded by a rogue trading scandal, to Germany's Deutsche Bank, which topped advisory fee tables in Europe last year, all firms are combing through their businesses to try to ensure their models are viable.

New rules that make trading bonds for instance more capital-intensive mean those that struggled to match the revenues of their biggest rivals will be forced to rein in their ambitions for this division.

UBS and its Swiss peer Credit Suisse have been among those slashing risky assets in this area.

However, few -- bar RBS, which is pulling out entirely from most of its equities businesses and from mergers and acquisitions -- have so far opted for a full-blow retreat.

Many still harbour ambitions to keep a grip on as many markets as they can, with an eye the profits they fear they will miss out on in an upturn. Drastic decisions taken this year would dash years of expansion plans and be very hard to reverse.

"Pride runs deep," said Matthew Czepliewicz, banks analyst at Collins Stewart.

Even some of the least profitable business, like lending to big companies, usually done at low rates to lure clients into handing over more lucrative appointments in future, are hard to chop out as banks fear burning their bridges with customers.

"Unless you get a gun to your head, it's difficult to cut relationships," said one senior London debt banker. "It's easy to sell off assets, but when you're cutting into the ... heart of an institution, it feels very painful."

NO SHINE

But this reluctance is likely to change this year as pressures mount.

French banks were pushed into a retreat from certain lending markets after struggling to access dollar funding. Similar strains this year and the cost of cutting back will further hit investment bank revenue in 2012 at Societe Generale, Reuters reported last week.

Regulatory demands are also increasing, as banks will globally have to hit number capital demands by 2013.

And with euro zone debt woes still clouding the outlook for merger and acquisition activity, hitting stock trading volumes and dampening companies' enthusiasm for borrowing, revenues will limp along at best.

"Nothing will shine. The hope is that it's not as grim as last year -- that's not an unreasonable assumption," said Czepliewicz.

Investment banking fees fell 6 percent globally last year compared with 2011, Thomson Reuters data shows, and the slowdown deepened in the fourth quarter. Investment bank revenues could drop by between 5 and 15 percent in 2012, according to a Morgan Stanley forecast.

Regional debt woes also mean politicians will be anxious to avoid repeating the taxpayer bailouts of the 2008 financial crisis, but may intervene to avert that risk.

"If the firms don't do it, governments will," said Brian Scott-Quinn, director of banking programmes at the ICMA centre, part of Henley Business School at the University of Reading, referring to scaleback plans.

U.S. firms retain a firm hold on the top ranks of investment banks and their distance from the troubled euro zone would help this year, added Czepliewicz.

He and other analysts tip Barclays Capital (part of Barclays Plc ) as one of the best positioned in Europe, in part down to its position outside the euro zone and its franchise in debt markets. Deutsche Bank, a trading powerhouse, has the type of scale tipped to benefit it compared to rivals.

CUTTING KNOWHOW

For those not under immediate pressure from losses or funding problems, costs are still a problem. How they cut now will ultimately determine their fate in the investment bank hierarchy -- and it's tough to know where to start.

"You'll be looking at areas where you rank number three, four, five or six, and unless you're in the top bit (you would have to think of) withdrawing, as it's not likely your returns will cover your cost of capital," said Scott-Quinn.

But bankers will want to preserve the units most likely to still bring in fees -- such as merger advisory and bond issuance -- and some of the associated ones that help them win mandates and build on their returns, like loans businesses, trading platforms or derivative products.

Just advising on bond issues without a secondary trading operation to match can be difficult if issuers want their banks to support liquid trading in their bonds, while investment banks use trading operations to get a feel for market prices.

Stock trading, meanwhile, guzzles less capital under new rules, potentially making it an attractive unit to keep. But thin margins also make it a business that is only truly profitable on a huge scale.

"It's a good idea to have a model that's weighted towards equities. But if everyone thinks that at the same time, it makes it pretty tough," said one European head of equities trading.

And as increased competition squeezes commissions and margins, it is setting investment banks on a collision course that could create more than a few casualties this year.