Max Bowie: Filtering Out the Noise

The Dow Jones Industrial Average’s recent rollercoaster ride—which, as of August 11, had changed direction each day for the prior seven trading days and posted a net change of at least 400 points for four consecutive days for the first time in history—has left analysts baffled. At the same time, investors are worried about the worth of their savings and retirement funds. But the situation has also left the data industry scrutinizing the excessive volumes generated by the market volatility and no doubt wondering whether their existing infrastructures will be able to handle a repeat event.
Affordable bandwidth, high-performance processors and data-compression technologies have largely overcome capacity issues, but the huge volumes of data created by the volatile swings have brought the issue back to the fore. According to marketdatapeaks.com—the volume-monitoring joint venture between Exegy, Essex Radez and the Financial Information Forum—US data rates hit 5.25 million messages per second (MPS) on August 4 and 4.9 million twice on August 10, edging perilously close to the capacity of 5.257 million mps recommended by the Options Price Reporting Authority. (These figures are only for options data, while marketdatapeaks’ figures include equities markets.)
According to figures collected by low-latency ticker plant vendor SpryWare from the FIF Market Data Capacity Working Group, consolidated US equity messages increased by almost 90 percent to 1.03 billion messages per day (MPD) between June 2009 and June 2011, while OPRA messages rose by more than 140 percent to 7.14 billion MPD. In comparison, total equity messages on August 5 were almost 2.18 billion—an increase of 110 percent over the peaks set two months before, while OPRA was 77 percent higher with 12.6 billion messages.
Excess Volume
This still leaves sufficient headroom to not hit OPRA’s bandwidth recommendations for July to support 17.3 billion messages per day, but certainly exceeds the rule of thumb that firms generally like to build in capacity of 200 percent over and above the latest peaks—an expensive proposition, but one that doubtless kept the markets moving during recent weeks.
Although data vendors seem to have handled the high volumes, the exchanges didn’t cope as well. On August 10, Nasdaq had to reset the Securities Information Processor for Channel 6 of its UTP Quotation Data Feed—which carries data on securities starting with the letters S through Z, and includes symbols that experienced heavy trading activity—after maxing out the volume of messages it could support in one day (100 million).
Andy Nybo, principal and head of derivatives at research firm Tabb Group, says firms must balance whether to build in additional capacity to support peaks that may or may not occur, even if it strains their budgets. “Some firms impacted will re-evaluate their technical infrastructures that manage their data activities, and will evaluate how much they need to invest and re-invest,” he says, adding that whatever the cost, not being able to handle peak volumes could prove even more costly.
Over the Limit
Whenever vendors have battled volumes with conflation techniques, consumers—insistent on receiving the full picture of the marketplace—have turned to vendors who can provide every tick. However, as that becomes less practical, we may see some firms balancing the needs of their traders against the limits of their infrastructure and demanding that data sources provide more sophisticated ways to filter out noise without diluting the value of data.
Ticker plants have allowed firms to do this on their own site, but that still requires the firm to have sufficient capacity to process the full incoming feeds. So I expect to see clients shift this burden on to the exchanges themselves, demanding custom-tailored feeds that supply only the information they need. Deutsche Börse has allowed clients to filter out the data they don’t want from its CEF Alpha feed for several years—look for others to do so too.
Only users who have a paid subscription or are part of a corporate subscription are able to print or copy content.
To access these options, along with all other subscription benefits, please contact info@waterstechnology.com or view our subscription options here: https://subscriptions.waterstechnology.com/subscribe
You are currently unable to print this content. Please contact info@waterstechnology.com to find out more.
You are currently unable to copy this content. Please contact info@waterstechnology.com to find out more.
Copyright Infopro Digital Limited. All rights reserved.
As outlined in our terms and conditions, https://www.infopro-digital.com/terms-and-conditions/subscriptions/ (point 2.4), printing is limited to a single copy.
If you would like to purchase additional rights please email info@waterstechnology.com
Copyright Infopro Digital Limited. All rights reserved.
You may share this content using our article tools. As outlined in our terms and conditions, https://www.infopro-digital.com/terms-and-conditions/subscriptions/ (clause 2.4), an Authorised User may only make one copy of the materials for their own personal use. You must also comply with the restrictions in clause 2.5.
If you would like to purchase additional rights please email info@waterstechnology.com
More on Emerging Technologies
Academic warns of systemic risk from AI-powered trading
Strategies generated by LLMs exhibit “very strange, correlated trading behavior”, says Lopez Lira.
Anthropic partners with S&P, TT to build AI hub, Talos acquires Coin Metrics, and more
The Waters Cooler: Bloomberg adopts agentic AI, DSB report homes in on fairer data costs, and asset managers are coming for your utilities in this week’s news roundup.
The Model Context Protocol brings agents to life—along with risk
Waters Wrap: From chat to infrastructure modernization, Anthropic’s MCP offers a ‘bridge’ to agentic AI, but its early days may prove disillusioning.
BofA ramps up AI deployment, patents
The bank has 1,400 patents in AI and machine learning, either granted or pending, alongside a growing portfolio of 250 models.
BNY CEO updates on ‘platform’ operating model, AI rollout
In its Q2 earnings call, the bank outlined its progress on rolling out its new operating model and ‘Eliza’ internal AI assistant.
NZX outlines plans to bolster fast-growing dark pool
Since launching one year ago, NZX’s dark book has 5.5% of the exchange’s total turnover, and price improvement per trade on average is 11 basis points, but the exchange has more in store.
Waters Wavelength Ep. 325: Octaura’s Brian Bejile
The CEO joins the podcast to talk about the vendor’s modernization efforts in credit and CLOs.
Agentic AI comes to Bloomberg Terminal via Anthropic protocol
The data giant’s ubiquitous terminal has been slowly opening up for years, but its latest enhancement represents a forward leap in what CTO Shawn Edwards calls, “the way we should talk to the world.”