Max Bowie: Filtering Out the Noise

The Dow Jones Industrial Average’s recent rollercoaster ride—which, as of August 11, had changed direction each day for the prior seven trading days and posted a net change of at least 400 points for four consecutive days for the first time in history—has left analysts baffled. At the same time, investors are worried about the worth of their savings and retirement funds. But the situation has also left the data industry scrutinizing the excessive volumes generated by the market volatility and no doubt wondering whether their existing infrastructures will be able to handle a repeat event.
Affordable bandwidth, high-performance processors and data-compression technologies have largely overcome capacity issues, but the huge volumes of data created by the volatile swings have brought the issue back to the fore. According to marketdatapeaks.com—the volume-monitoring joint venture between Exegy, Essex Radez and the Financial Information Forum—US data rates hit 5.25 million messages per second (MPS) on August 4 and 4.9 million twice on August 10, edging perilously close to the capacity of 5.257 million mps recommended by the Options Price Reporting Authority. (These figures are only for options data, while marketdatapeaks’ figures include equities markets.)
According to figures collected by low-latency ticker plant vendor SpryWare from the FIF Market Data Capacity Working Group, consolidated US equity messages increased by almost 90 percent to 1.03 billion messages per day (MPD) between June 2009 and June 2011, while OPRA messages rose by more than 140 percent to 7.14 billion MPD. In comparison, total equity messages on August 5 were almost 2.18 billion—an increase of 110 percent over the peaks set two months before, while OPRA was 77 percent higher with 12.6 billion messages.
Excess Volume
This still leaves sufficient headroom to not hit OPRA’s bandwidth recommendations for July to support 17.3 billion messages per day, but certainly exceeds the rule of thumb that firms generally like to build in capacity of 200 percent over and above the latest peaks—an expensive proposition, but one that doubtless kept the markets moving during recent weeks.
Although data vendors seem to have handled the high volumes, the exchanges didn’t cope as well. On August 10, Nasdaq had to reset the Securities Information Processor for Channel 6 of its UTP Quotation Data Feed—which carries data on securities starting with the letters S through Z, and includes symbols that experienced heavy trading activity—after maxing out the volume of messages it could support in one day (100 million).
Andy Nybo, principal and head of derivatives at research firm Tabb Group, says firms must balance whether to build in additional capacity to support peaks that may or may not occur, even if it strains their budgets. “Some firms impacted will re-evaluate their technical infrastructures that manage their data activities, and will evaluate how much they need to invest and re-invest,” he says, adding that whatever the cost, not being able to handle peak volumes could prove even more costly.
Over the Limit
Whenever vendors have battled volumes with conflation techniques, consumers—insistent on receiving the full picture of the marketplace—have turned to vendors who can provide every tick. However, as that becomes less practical, we may see some firms balancing the needs of their traders against the limits of their infrastructure and demanding that data sources provide more sophisticated ways to filter out noise without diluting the value of data.
Ticker plants have allowed firms to do this on their own site, but that still requires the firm to have sufficient capacity to process the full incoming feeds. So I expect to see clients shift this burden on to the exchanges themselves, demanding custom-tailored feeds that supply only the information they need. Deutsche Börse has allowed clients to filter out the data they don’t want from its CEF Alpha feed for several years—look for others to do so too.
Only users who have a paid subscription or are part of a corporate subscription are able to print or copy content.
To access these options, along with all other subscription benefits, please contact info@waterstechnology.com or view our subscription options here: http://subscriptions.waterstechnology.com/subscribe
You are currently unable to print this content. Please contact info@waterstechnology.com to find out more.
You are currently unable to copy this content. Please contact info@waterstechnology.com to find out more.
Copyright Infopro Digital Limited. All rights reserved.
As outlined in our terms and conditions, https://www.infopro-digital.com/terms-and-conditions/subscriptions/ (point 2.4), printing is limited to a single copy.
If you would like to purchase additional rights please email info@waterstechnology.com
Copyright Infopro Digital Limited. All rights reserved.
You may share this content using our article tools. As outlined in our terms and conditions, https://www.infopro-digital.com/terms-and-conditions/subscriptions/ (clause 2.4), an Authorised User may only make one copy of the materials for their own personal use. You must also comply with the restrictions in clause 2.5.
If you would like to purchase additional rights please email info@waterstechnology.com
More on Emerging Technologies
DeepSeek success spurs banks to consider do-it-yourself AI
Chinese LLM resets price tag for in-house systems—and could also nudge banks towards open-source models.
Standard Chartered goes from spectator to player in digital asset game
The bank’s digital assets custody offering is underpinned by an open API and modular infrastructure, allowing it to potentially add a secondary back-end system provider.
Saugata Saha pilots S&P’s way through data interoperability, AI
Saha, who was named president of S&P Global Market Intelligence last year, details how the company is looking at enterprise data and the success of its early investments in AI.
Data partnerships, outsourced trading, developer wins, Studio Ghibli, and more
The Waters Cooler: CME and Google Cloud reach second base, Visible Alpha settles in at S&P, and another overnight trading venue is approved in this week’s news round-up.
Are we really moving on from GenAI already?
Waters Wrap: Agentic AI is becoming an increasingly hot topic, but Anthony says that shouldn’t come at the expense of generative AI.
Cloud infrastructure’s role in agentic AI
The financial services industry’s AI-driven future will require even greater reliance on cloud. A well-architected framework is key, write IBM’s Gautam Kumar and Raja Basu.
Waters Wavelength Ep. 310: SigTech’s Bin Ren
This week, SigTech’s CEO Bin Ren joins Eliot to discuss GenAI’s progress since ChatGPT’s emergence in 2022, agentic AI, and challenges with regulating AI.
Microsoft exec: ‘Generative AI is completely passé. This is the year of agentic AI’
Microsoft’s Symon Garfield said that AI advancements are prompting financial services firms to change their approach to integrating AI-powered solutions.