Wall Street banks and brokers are poring over their trading systems and rethinking the way they test software to make sure they don't become the next Knight Capital Group, the trading firm whose survival was imperiled by a software glitch on Wednesday.
Knight Capital's $440 million loss from errant trades, which has forced the company to consider selling all or part of itself, is the third time in five months that technical bugs have caused trouble for Wall Street players.
Executives at trading firms said they are debating among one another whether new regulations could prevent these snafus. But they also said glitches were a wake-up call for firms to improve their controls on their own, without being pushed into it. At a time when Wall Street is cutting costs, spending money on better systems to test software and manage risk could be an expensive proposition.
"We want to make sure that what happened to Knight doesn't happen to us," said the head of one investment bank. His company was looking carefully at how it tests new trading systems, to make sure traders know when new systems are being implemented and can be on the lookout for suspicious activity during those periods.
Their efforts face plenty of obstacles. As more trading has moved from exchange floors to computers over the last decade, the speed of execution jumped along with the potential for cascading problems.
Trading firms, market makers, brokers, investment banks, and exchanges and other trading venues are linked in a network of complex computer systems that compete to execute trades as fast as possible. That competition, combined with the never-ending array of new rules, forces market participants to constantly improve their systems.
But the intricate network of players and systems creates a much wider range of potential problems for trading systems, making testing costly and difficult. Good testing requires a firm to imagine everything that can possibly go wrong and how the system will interact with other systems. Predicting every plausible scenario is not easy, said one trading head at a major Wall Street firm.
Regulators have set up "circuit breakers" that require exchanges to suspend trading in stocks that move too much too quickly. But dealers and other market players usually have even more sophisticated circuit breakers for their own trading.
"You need an algorithm to monitor the trading algorithm," said John Bates, chief technology officer at Progress Software, which provides trading software.
Knight appeared to lack these sorts of circuit breakers, or at least did not implement them well enough, traders said.
Two other major trading glitches have beset Wall Street since March. BATS Global Markets, an exchange, was unable to complete its own initial public offering because of a technical problem. Nasdaq botched the market debut of Facebook due to systems bugs, costing it tens of millions of dollars, while UBS AG lost more than $350 million in trading Facebook shares and is blaming Nasdaq.
LOWER VOLUMES, HIGHER PRESSURE
Dealers don't always do a great job of testing, said Colin Clark, a developer with Cloud Event Processing, a firm that works with big Wall Street banks and exchanges on software and technology.
"They're not investing enough in testing now," Clark said, noting the pressure banks face to cut costs.
U.S. stock trading volume is 40 percent below its peak volume in 2009, according to Tabb Group, falling from an average of 12 billion trades a day to 6 million to 7 million currently. That decline has weighed on profits at firms, and strained the willingness of many to invest in unglamorous areas like testing systems, said Larry Tabb, founder of Tabb Group, a consulting firm that focuses on capital markets.
"There are fewer people minding the store, and the people in place are stretched too thin," Tabb said.
But others on Wall Street dismissed the notion that dealers are not investing enough in their systems, noting that dealers spend millions on technology in part to be more efficient and reduce staffing levels.
In addition to testing new systems, firms are checking their mechanisms to protect them from bad trades. Most dealers have systems that automatically stop trading, or at least generate warnings for traders, when volumes or price movements are suspicious.
Dealers can try to improve their own systems, but it is possible that new rules could prevent breakdowns as well. One idea that dealers have discussed in the wake of the Knight Capital mess is eliminating "market orders," or orders that are executed at whatever the prevailing market price is. Retail investors often place market orders.
Customers could instead be required to place "limit orders" that specify the maximum price at which they will buy a stock, for example. With limit orders, if the market price of a stock surges because of technical glitches, many orders to buy shares would not be executed.
Another idea dealers have discussed is to slow down trading, to ensure that they have a handle on what they are doing at all times.
"If so many trades are happening in a millisecond, it's hard to keep up with everything you're doing," said one of the senior Wall Street traders.