Open Platform: The Challenges and Opportunities of Big Data in Finance

With data volumes tripling and trading speeds now faster than ever, many financial services companies are struggling to monitor their trading activity and react in a timely manner to warning signs. One of the biggest fears for the financial services industry is messy outages or performance degradation on critical trading systems. In this worst-case scenario, firms can go out of business, or experience major revenue and reputational damage (a single server cost Knight Capital more than $400 million in 45 minutes, while Goldman Sachs' options desk is rumored to have lost almost $100 million in 15 minutes).

In light of this massive risk, the industry is looking for solutions that will help firms do more with what they have and make better use of their resources. Now, the good news. Financial services firms actually already have the information they need in their proprietary Big Data to understand when they are at risk and understand the cause.

Many firms are so overwhelmed with real-time data that warning signs are often missed until after a major disruption occurs. Financial services firms need tools and techniques to guarantee smart trading through smart decision-making. By using historical data to create real-time intelligence, financial services firms can actually leverage predictive analytics as a tool to identify unusual activity through their own Big Data before abnormal events occur.

Even as the complexity of systems increases, the financial services industry is still cutting IT staff and budgets, increasing the need to do more with less. They still need to be able to support functions like automated trading systems monitoring, real-time risk management, complex technology infrastructure monitoring, and transactional monitoring and surveillance.

Real-time predictive analytics performed using case-based reasoning (CBR) is the key to transforming the onslaught of data overwhelming the financial services industry into actionable intelligence. CBR is a decision support system that helps human operators leverage real-time data to improve outcomes and reduce downtime, and is already proving successful in industries such as oil and gas exploration to prevent drilling outages and improve health and safety at drill sites. This type of data intelligence is essential for the financial services industry to leverage, as there is no room for failure in either industry.

The ease with which CBR can leverage existing data is a key part of its "do more with less" approach for the financial industry. CBR sits on top of IT and trading tools, analyzing data patterns in real time and identifying erroneous behavior with current resources, so legacy IT infrastructure is actually gaining value from existing data without needing costly improvements and manpower to manage it. And in today's busy market environment, checking patterns and potential risks can be as easy as checking email notifications or using a web-based dashboard to access case-based reasoning platforms.

Being reactive won't cut it in today's high-speed trading environment: firms must prevent outages rather than react to them. Regulatory compliance requirements are also constantly in flux, and managing deadlines and changing requirements is an overwhelming task. In 2014, firms will need tools and processes to help them use their data to foresee risks or breaches well before they happen, and to know the exact way to handle issues based on what worked and what didn't work in the past. Compliance and risk officers need to look at past events to measure the risk of something similar happening in the future to protect their own organizations and the firms and consumers they work with.

Regulatory discussions around trading technology are also dictating the need for "kill switches" that use real-time intelligence combined with human decision management. The Securities and Exchange Commission's review of the Knight Capital snafu highlighted a flood of emails that should have alerted Knight to software problems and a previous trading loss that offered chances for Knight to tighten its processes and learn from prior mistakes.

The truth is, many anomalous events share similarities with past events and can be anticipated when a similar chain reaction is set in motion. Other novel events can be anticipated by combining an understanding of past events with complex systems analysis and complexity theory. By leveraging resources that are already on-site, Big Data analytics can be used to identify firm- and market-impacting events. Throwing resources and solutions at problems after the fact─when it's too late to change the outcome─is not an efficient or cost-effective way of handling these issues.

Bringing data to life in trading is the key to making the most out of ever-increasing data volumes. Until now, the two main impediments to taking action based on predictive analytics have been the difficulty of processing these data volumes and the confidence to take action when statistical systems say so. By having accurate, evidence-based information at their fingertips, financial firms can be assured that they are staying on top of their data, traders and infrastructure. Maintaining a real-time view into the huge amount of data flowing through a firm will let systems operators, risk managers and compliance officers stay ahead of any problems, rather than frantically reacting to them. The benefits of predictive analytics in 2014 for the financial industry are endless: firms just need to learn to take the opportunity to leverage their Big Data and not be overwhelmed by it.