Max Bowie: Filtering Out the Noise

The Dow Jones Industrial Average’s recent rollercoaster ride—which, as of August 11, had changed direction each day for the prior seven trading days and posted a net change of at least 400 points for four consecutive days for the first time in history—has left analysts baffled. At the same time, investors are worried about the worth of their savings and retirement funds. But the situation has also left the data industry scrutinizing the excessive volumes generated by the market volatility and no doubt wondering whether their existing infrastructures will be able to handle a repeat event.

Affordable bandwidth, high-performance processors and data-compression technologies have largely overcome capacity issues, but the huge volumes of data created by the volatile swings have brought the issue back to the fore. According to marketdatapeaks.com—the volume-monitoring joint venture between Exegy, Essex Radez and the Financial Information Forum—US data rates hit 5.25 million messages per second (MPS) on August 4 and 4.9 million twice on August 10, edging perilously close to the capacity of 5.257 million mps recommended by the Options Price Reporting Authority. (These figures are only for options data, while marketdatapeaks’ figures include equities markets.)

According to figures collected by low-latency ticker plant vendor SpryWare from the FIF Market Data Capacity Working Group, consolidated US equity messages increased by almost 90 percent to 1.03 billion messages per day (MPD) between June 2009 and June 2011, while OPRA messages rose by more than 140 percent to 7.14 billion MPD. In comparison, total equity messages on August 5 were almost 2.18 billion—an increase of 110 percent over the peaks set two months before, while OPRA was 77 percent higher with 12.6 billion messages.

Excess Volume 
This still leaves sufficient headroom to not hit OPRA’s bandwidth recommendations for July to support 17.3 billion messages per day, but certainly exceeds the rule of thumb that firms generally like to build in capacity of 200 percent over and above the latest peaks—an expensive proposition, but one that doubtless kept the markets moving during recent weeks.

Although data vendors seem to have handled the high volumes, the exchanges didn’t cope as well. On August 10, Nasdaq had to reset the Securities Information Processor for Channel 6 of its UTP Quotation Data Feed—which carries data on securities starting with the letters S through Z, and includes symbols that experienced heavy trading activity—after maxing out the volume of messages it could support in one day (100 million).

Andy Nybo, principal and head of derivatives at research firm Tabb Group, says firms must balance whether to build in additional capacity to support peaks that may or may not occur, even if it strains their budgets. “Some firms impacted will re-evaluate their technical infrastructures that manage their data activities, and will evaluate how much they need to invest and re-invest,” he says, adding that whatever the cost, not being able to handle peak volumes could prove even more costly.

Over the Limit
Whenever vendors have battled volumes with conflation techniques, consumers—insistent on receiving the full picture of the marketplace—have turned to vendors who can provide every tick. However, as that becomes less practical, we may see some firms balancing the needs of their traders against the limits of their infrastructure and demanding that data sources provide more sophisticated ways to filter out noise without diluting the value of data.

Ticker plants have allowed firms to do this on their own site, but that still requires the firm to have sufficient capacity to process the full incoming feeds. So I expect to see clients shift this burden on to the exchanges themselves, demanding custom-tailored feeds that supply only the information they need. Deutsche Börse has allowed clients to filter out the data they don’t want from its CEF Alpha feed for several years—look for others to do so too.