Max Bowie: Filtering Out the Noise

The Dow Jones Industrial Average’s recent rollercoaster ride—which, as of August 11, had changed direction each day for the prior seven trading days and posted a net change of at least 400 points for four consecutive days for the first time in history—has left analysts baffled. At the same time, investors are worried about the worth of their savings and retirement funds. But the situation has also left the data industry scrutinizing the excessive volumes generated by the market volatility and no doubt wondering whether their existing infrastructures will be able to handle a repeat event.
Affordable bandwidth, high-performance processors and data-compression technologies have largely overcome capacity issues, but the huge volumes of data created by the volatile swings have brought the issue back to the fore. According to marketdatapeaks.com—the volume-monitoring joint venture between Exegy, Essex Radez and the Financial Information Forum—US data rates hit 5.25 million messages per second (MPS) on August 4 and 4.9 million twice on August 10, edging perilously close to the capacity of 5.257 million mps recommended by the Options Price Reporting Authority. (These figures are only for options data, while marketdatapeaks’ figures include equities markets.)
According to figures collected by low-latency ticker plant vendor SpryWare from the FIF Market Data Capacity Working Group, consolidated US equity messages increased by almost 90 percent to 1.03 billion messages per day (MPD) between June 2009 and June 2011, while OPRA messages rose by more than 140 percent to 7.14 billion MPD. In comparison, total equity messages on August 5 were almost 2.18 billion—an increase of 110 percent over the peaks set two months before, while OPRA was 77 percent higher with 12.6 billion messages.
Excess Volume
This still leaves sufficient headroom to not hit OPRA’s bandwidth recommendations for July to support 17.3 billion messages per day, but certainly exceeds the rule of thumb that firms generally like to build in capacity of 200 percent over and above the latest peaks—an expensive proposition, but one that doubtless kept the markets moving during recent weeks.
Although data vendors seem to have handled the high volumes, the exchanges didn’t cope as well. On August 10, Nasdaq had to reset the Securities Information Processor for Channel 6 of its UTP Quotation Data Feed—which carries data on securities starting with the letters S through Z, and includes symbols that experienced heavy trading activity—after maxing out the volume of messages it could support in one day (100 million).
Andy Nybo, principal and head of derivatives at research firm Tabb Group, says firms must balance whether to build in additional capacity to support peaks that may or may not occur, even if it strains their budgets. “Some firms impacted will re-evaluate their technical infrastructures that manage their data activities, and will evaluate how much they need to invest and re-invest,” he says, adding that whatever the cost, not being able to handle peak volumes could prove even more costly.
Over the Limit
Whenever vendors have battled volumes with conflation techniques, consumers—insistent on receiving the full picture of the marketplace—have turned to vendors who can provide every tick. However, as that becomes less practical, we may see some firms balancing the needs of their traders against the limits of their infrastructure and demanding that data sources provide more sophisticated ways to filter out noise without diluting the value of data.
Ticker plants have allowed firms to do this on their own site, but that still requires the firm to have sufficient capacity to process the full incoming feeds. So I expect to see clients shift this burden on to the exchanges themselves, demanding custom-tailored feeds that supply only the information they need. Deutsche Börse has allowed clients to filter out the data they don’t want from its CEF Alpha feed for several years—look for others to do so too.
Only users who have a paid subscription or are part of a corporate subscription are able to print or copy content.
To access these options, along with all other subscription benefits, please contact info@waterstechnology.com or view our subscription options here: https://subscriptions.waterstechnology.com/subscribe
You are currently unable to print this content. Please contact info@waterstechnology.com to find out more.
You are currently unable to copy this content. Please contact info@waterstechnology.com to find out more.
Copyright Infopro Digital Limited. All rights reserved.
As outlined in our terms and conditions, https://www.infopro-digital.com/terms-and-conditions/subscriptions/ (point 2.4), printing is limited to a single copy.
If you would like to purchase additional rights please email info@waterstechnology.com
Copyright Infopro Digital Limited. All rights reserved.
You may share this content using our article tools. As outlined in our terms and conditions, https://www.infopro-digital.com/terms-and-conditions/subscriptions/ (clause 2.4), an Authorised User may only make one copy of the materials for their own personal use. You must also comply with the restrictions in clause 2.5.
If you would like to purchase additional rights please email info@waterstechnology.com
More on Emerging Technologies
GenAI too risky for collateral processes
The technology has been heralded as game-changing for other areas of finance, but its potential to hallucinate may disqualify it from sensitive settlement procedures.
Waters Wavelength Ep. 324: A philosophical conversation about AI
This week, Reb and Nyela discuss BNY’s digital workers, and what the use of AI in society signals for the future.
Cloud Wars: Are EU and APAC firms really pining for homegrown options?
Waters Wrap: In the wake of tariffs and regional instability, there’s chatter about non-US firms lessening their dependency on the major hyperscalers. Anthony is not buying it.
Google gifts Linux, capital raised for Canton, one less CTP bid, and more
The Waters Cooler: Banks team up for open-source AI controls, S&P injects GenAI into Capital IQ, and Goldman Sachs employees get their own AI assistant in this week’s news roundup.
Numerix strikes Hundsun deal as China pushes domestic tech
The homegrown tech initiative—‘Xinchuang’—is a new challenge for foreign vendors.
RBC’s partnership with GenAI vendor Cohere begins to bear fruit
The platform aims to help the Canadian bank achieve its lofty AI goals.
Deutsche Bank casts a cautious eye towards agentic AI
“An AI worker is something that is really buildable,” says innovation and AI head
TMX buys ETF biz, Iress reinvests in trading tools, UBS data exposed, and more
The Waters Cooler: Euroclear’s next-gen service, MarketAxess launches e-trading for IGBs, and new FX services are in this week’s news round-up.