* Operating expenses, capex in Q3 closely watched
* New Instant search to boost CPC, paid clicks - analyst
* Google to report 3Q results on Thursday
* Wall Street expects revenue excl TAC of $5.26 billion
SAN FRANCISCO, Oct 12 (Reuters) - Wall Street wants Google Inc's (GOOG.O) new products and initiatives to start paying off, as its accelerating spending spree nibbles away at margins and alarms investors.
The Internet giant's free-spending -- from more than 20 acquisitions this year alone to internal projects such asself-driven cars and big bets in wind energy -- has weighed on the company's stock, which has underperformed the market this year.
Headcount, capital expenditures and operating expenses will be key issues for investors when Google reports third-quarter results Thursday, particularly after a rare profit shortfall in the second quarter wiped 7 percent off its shares in a single day. [ID:nN15232993]
But with Google's shares up roughly 20 percent since mid-August, analysts are betting it will justify its spending with details of improving business prospects.
"People don't mind expenses if you're growing revenue," said BGC Partners' analyst Colin Gillis.
Google has two-thirds of the Internet search market, but is facing a renewed threat from Yahoo Inc (YHOO.O) and Microsoft Corp (MSFT.O) which have forged a search partnership.
At the same time, social networking companies such as Facebook are attracting increasing amounts of online advertising, posing a growing threat to Google's business.
Google is also shelling out cash to develop new technology and build a viable smartphone business based on its Android phone software to take on Apple's iPhone juggernaut.
This week, it also announced it was joining an estimated $5 billion undersea cable project to carry power from offshore windfarms to the east coast of the United States.
The spending spree is taking a toll. Operating margins slipped to 35 percent last quarter from 37 percent in the first, while headcount increased by roughly 2,000 employees in the first six months of the year alone.
Google has roughly $30 billion in cash and marketable securities.
Analysts expect Google to report revenue, excluding the fees that Google shares with website partners, of $5.26 billion in the third quarter, up 3.3 percent from the second but up roughly 20 percent from a year earlier, with adjusted earnings of $6.68 a share.
Some of Google's attempts to find the next big thing have failed. For example, it pulled the plug on its much-hyped Wave product that combined instant messaging and online collaboration this year. Nevertheless, analysts say some initiatives are beginning to help business.
Its new Instant technology, which speeds up searches by predicting queries, could improve revenue by prompting web surfers to click on ads more frequently, say analysts.
Paid clicks and cost-per-click, the two metrics that investors use to gauge the health of Google's search ad business, should both improve compared to the second quarter, said UBS analyst Brian Pitz, citing Instant search as one of several factors.
"There's some strong fundamental trends that look pretty positive, whether it's product search doing well, Instant search driving some growth, and just generally positive monetization trends," Pitz said of business in the third quarter.
Its fledgling Android software for smartphones came from nowhere two years ago and is now challenging Apple's position in the market.
A recent report by industry research firm Gartner predicted Android would overtake Apple's iOS this year to become the No. 2 operating system for cell phones, after Nokia's Symbian.
While it does not break out financial results for its mobile business, some analysts say Android is becoming an important part of the company's efforts to establish itself in mobile advertising.
Citigroup analyst Mark Mahaney estimates that revenue from mobile ads could reach a run rate of $450 million by the end of the year.
Google shares were up 1 percent at $543.85 in early Nasdaq trading on Tuesday. They have fallen 13 percent since the beginning of the year, while Nasdaq has risen close to 6 per cent. (Additional reporting by Doris Frankel; Editing by Kenneth Li and Derek Caney)