Nasdaq says software bug caused trading outage

NEW YORK Thu Aug 29, 2013 4:02pm EDT

A man walks past the Nasdaq MarketSite in New York's Times Square, August 23, 2013. REUTERS/Andrew Kelly

A man walks past the Nasdaq MarketSite in New York's Times Square, August 23, 2013.

Credit: Reuters/Andrew Kelly

NEW YORK (Reuters) - Nasdaq OMX Group's (NDAQ.O) massive trading halt last week was due to a software bug and other internal technology issues triggered by problems at NYSE Euronext's NYX.N Arca exchange that led a key backup system to fail, the exchange operator said on Thursday.

Nasdaq said it was "deeply disappointed" by the three-hour outage on August 22, and while it pointed to connection problems between rival NYSE's electronic exchange Arca and the Nasdaq-run system that receives all traffic on quotes and orders for Nasdaq stocks, it took ultimate responsibility for the glitch.

"Our backup system did not work," Bob Greifeld, Nasdaq's chief executive, said in an interview.

"There was a bug in the system, it didn't fail over properly, and we need to work hard to make sure it doesn't happen again," he said, referring to the inability of the system to fully revert to backup mode.

New York Stock Exchange parent NYSE declined to comment.

Nasdaq said it was in the process of identifying potential design changes to make the Securities Information Processor, or SIP, more resilient, "including architectural improvements, information security, disaster recovery plans and capacity parameters."

The exchange plans to present its initial recommendations for change to the SIP governing committee, made up of U.S. exchanges and the Financial Industry Regulatory Authority, within 30 days.

Nasdaq said that on the morning of August 22, Arca connected and disconnected to the SIP more than 20 times, eating up capacity. It said the SIP's capacity was further eroded as Arca sent a stream of inaccurate stock symbols to the SIP, generating numerous rejection messages.

Each data port connecting to the SIP can handle 10,000 messages per second, Nasdaq said. But in this case, the traffic from Arca was more than double that, the exchange said.

"The confluence of these events vastly exceeded the SIP's planned capacity, which caused its failure and then revealed a latent flaw in the SIP's software code," the Nasdaq report said.

This software flaw prevented the processor's built-in backup system from resetting properly, delaying the return of data. With the system degraded, the exchange said it decided to halt trading to ensure fair market conditions.

Nasdaq said it took 30 minutes to resolve the problem and then nearly three more hours to test and evaluate scenarios to reopen the market in a fair and orderly manner.

"Nasdaq OMX is deeply disappointed in the events of August 22 and our performance is unacceptable to our members, issuers and the investing public," the exchange said. "While getting to 100 percent performance in all of our activities, including our technology is difficult - it is our objective."

(Editing by Kenneth Barry and Matthew Lewis)

We welcome comments that advance the story through relevant opinion, anecdotes, links and data. If you see a comment that you believe is irrelevant or inappropriate, you can flag it to our editors by using the report abuse links. Views expressed in the comments do not represent those of Reuters. For more information on our comment policy, see http://blogs.reuters.com/fulldisclosure/2010/09/27/toward-a-more-thoughtful-conversation-on-stories/
Comments (4)
tmc wrote:
Middle-Managers…..LMFAO!!

Aug 29, 2013 4:46pm EDT  --  Report as abuse
Obsilutely wrote:
That’s why you don’t test in production!!

Aug 29, 2013 7:08pm EDT  --  Report as abuse
AlkalineState wrote:
There’s a woman to blame, somewhere.

Aug 29, 2013 7:43pm EDT  --  Report as abuse
This discussion is now closed. We welcome comments on our articles for a limited period after their publication.