2012 Review: Big Data Gets Bigger Role in Financial Markets

With use of Big Data on the rise in the financial markets, end-users, vendors and exchanges sought ways to manage and extract value from it, including by leveraging new systems as well as enhancing and repurposing existing technologies.

While stagnant trading activity meant that data volumes did not reach the levels predicted by The Options Price Reporting Authority early this year (IMD, Feb. 6)—though they may yet reach the almost 14 million messages per second predicted for July 2013—firms nevertheless expanded their data management burden by seeking to extract additional alpha from as many new sources of information as possible.

To alleviate the pain—particularly the cost—for firms of building and maintaining an infrastructure to store and manage this data, the International Securities Exchange released its managed ISE Premium Hosted Database of full OPRA data, Level 1 US equities data and options analytics, developed in partnership with options analytics provider Hanweck Associates, to enable traders to subscribe to the data they need or pay for one-off queries to support back-testing and analysis requirements (IMD, April 30). In a similar move, NYSE Technologies, the data and trading technology arm of NYSE Euronext, rolled out its Market Data Analytics Lab (MDAL) managed database of its historical trade and quote (TAQ) data as well as hosted analytics and tools for querying the data (IMD, May 21).

Meanwhile, analytic database provider ParAccel expanded its Analytic Offload capabilities, which enable firms to shift complex analytic workloads from their standard data warehouses—which can handle static and simpler analytics but slow down when performing more dynamic analytics—to the vendor’s database, with the addition of On-Demand Integration modules for loading data from Oracle and Teradata databases, to speed analysis (IMD, June 25).

Complex event processing is also increasingly being applied to Big Data analysis, such as through StreamBase Systems’ Big Data “Acceleration Pack” that integrates with Hadoop Flume, social media aggregator Gnip, and unstructured sentiment and momentum data from Recorded Future. StreamBase also teamed with hardware ticker plant vendor Exegy on an adapter for integrating low-latency data from Exegy’s data appliance into StreamBase’s CEP platform to support algorithmic trading strategy development, testing and deployment, and enable firms to apply Big Data analytics created in StreamBase’s platform to data from Exegy (IMD, Oct. 15).

This focus on Big Data culminated in a Big Data special interest group set up by data and trading technology benchmarking organization the Securities Technology Analysis Center (IMD, Nov. 19), which will provide a forum for discussions between end-users and vendors about dealing with Big Data challenges and will define benchmark standards for Big Data analytical tasks, such as model simulation and back-testing.