Shedding Light on Dark Data
Making connections among sources yields predictive information
The term "dark data" and what it represents is being raised more often lately by industry service providers and consultancies as an issue of concern. With this increased attention, some are uncertain about what exactly "dark data" means in the context of financial services industry data management.
Dark data shouldn't be understood literally, as though it were about using data for "nefarious purposes," says Charles Fiori, a consultant with long industry experience at firms and service providers, including Bear Stearns, BrokerTec, Goldman Sachs, JPMorgan, Lehman Brothers and Reuters.
Fiori defines dark data as data whose existence is either unknown to a firm, known but inaccessible, too costly to access or inaccessible because of compliance concerns.
Defining dark data also means accounting for issues other than accessibility, according to Tim Lind, global head of financial regulation solutions at Thomson Reuters. "It's about connection and drawing connections," he says. "It's data that we store anyway for compliance or business purposes, that's sitting there waiting for you to discover its true meaning-or it's looking for, as we do in the regular analytics world, the predictive value of data when it comes together."
Fragmentation is a major challenge for connecting data to collect and reconcile all dark data. "We see some institutions out there with 30 trading systems that grew up over time in different regions servicing different businesses, but still within the same family," says Lind. "So the data standards, the operational structure and, daresay, the governance of how data is managed and how data standards are propagated within an institution, are the core challenges for getting data together."
Risk Aggregation Aims
The true purpose of collecting dark data by aggregating data is to aggregate all available risk information, says Lind. "Any data not being utilized properly, is unable to be connected, or is outside the grid somehow because of structure, operations or governance perspectives, is dark data," he says. "That data tells an interesting story about the risk an enterprise has with a counterparty."
BCBS 239, the Basel Committee on Banking Supervision's risk data aggregation principles, are challenging firms to access and manage their dark data, according to Lind. These rules, which have a US equivalent in CCAR (Comprehensive Capital Analysis and Review), require firms to stress test the risk in their holdings. To meet the requirements of these principles, firms must collect and aggregate data in a similar way as they would to obtain dark data.
"A lot of these principles are still outside the grasp of most financial institutions," says Lind. "They're still not quite able to link transactions and positions across asset classes, lines of business and geographies, to get a bank-wide picture of their exposure to a counterparty, to an issuer or to a country. These core risk aggregation and management items are about the ability to connect data we store already, a.k.a. dark data."
In fact, BCBS 239 may be spurring firms to recognize dark data as an issue, according to Hubert Holmes, managing director, reference data at Interactive Data, a data services provider. "Dark data has not been much of an issue until big data, data management, data governance, regulatory and risk mandates have brought it more to the attention of firms," he says.
Potential Solutions
Solving the issues inherent in the nature of dark data would most likely require structuring, tagging or organizing the data in a database. If that has not been done, or is not possible, the solution may be doing an audit, according to Holmes. "There are many risks that arise for having such information but not really managing it," he says, adding that these include legal or regulatory risks, intelligence, reputation and operational risks. "Some dark data is worthless and some is highly valuable, and everything in between. You can't tell without an audit."
Devising a way to address dark data is necessary to have a foundation for advanced analytics, according to Lind. "Start a little closer to home," he says. "We have a ton of work to do in structure, data standards, aggregation and drawing those connections to the most common objects like instruments and entities."
Only users who have a paid subscription or are part of a corporate subscription are able to print or copy content.
To access these options, along with all other subscription benefits, please contact info@waterstechnology.com or view our subscription options here: http://subscriptions.waterstechnology.com/subscribe
You are currently unable to print this content. Please contact info@waterstechnology.com to find out more.
You are currently unable to copy this content. Please contact info@waterstechnology.com to find out more.
Copyright Infopro Digital Limited. All rights reserved.
You may share this content using our article tools. Printing this content is for the sole use of the Authorised User (named subscriber), as outlined in our terms and conditions - https://www.infopro-insight.com/terms-conditions/insight-subscriptions/
If you would like to purchase additional rights please email info@waterstechnology.com
Copyright Infopro Digital Limited. All rights reserved.
You may share this content using our article tools. Copying this content is for the sole use of the Authorised User (named subscriber), as outlined in our terms and conditions - https://www.infopro-insight.com/terms-conditions/insight-subscriptions/
If you would like to purchase additional rights please email info@waterstechnology.com
More on Data Management
Putting the ‘A’ in CDO: The rise of the chief data and analytics officer
As data and analytics become more intertwined, banks and vendors are creating a new role—the chief data and analytics officer—to help them take advantage of the opportunities it presents. It may sound easy, but rethinking data can be a gargantuan task.
The IMD Wrap: Talk about ‘live’ data, NAFIS 2024 is here
This year’s North American Financial Information Summit takes place this week, with an expanded agenda. Max highlights some of the must-attend sessions and new topics. But first, a history lesson...
Waters Wavelength Podcast: S&P’s CTO on AI, data, and the future of datacenters
Frank Tarsillo, CTO at S&P Global Market Intelligence, joins the podcast to discuss the firm’s approach to AI, the importance of data, and what might be in store for datacenters in the coming years.
Breaking out of the cells: banks’ long goodbye to spreadsheets
Dealers are cutting back on their use of Excel amid tighter regulation and risk concerns.
BMO’s cloud migration strategy eases AI adoption
The Canadian bank is embracing a more digital future as its cloud strategy makes gains and it looks to both traditional machine learning and generative AI for further augmentation.
Waters Wrap: GenAI and rising tides
As banks, asset managers, and vendors ratchet up generative AI experiments and rollouts, Anthony explains why collaboration between business and tech teams is crucial.
People Moves: NorQuant, Tradition, Duco, HKEx, SimCorp, Hazeltree, Xceptor, Broadridge, and more
A look at the past month’s people moves in the capital markets technology and data space.
Northern Trust building internal cloud data ‘marketplace’
Using a mix of in-house expertise and third-party technologies, the firm has constructed a cloud-based data mesh that gives internal staff access to proprietary datasets and analytical tools to deliver greater insights into client activity.
Most read
- Waters Wavelength Podcast: S&P’s CTO on AI, data, and the future of datacenters
- Chris Edmonds takes the reins at ICE Fixed Income and Data Services
- DTCC urges affirmation focus ahead of T+1 move