Big Data Vs. Dark Data

In a story last week, I had looked at the use of the term "dark data" and what it means. That is a newer buzzword than one that has been thrown around much more over the past several years, "big data."

Often when "big data" has been used in financial services operations discussions, particularly in data management operations discussions, it really pertains to what is being done with data, such as how firms mine larger amounts of data or how they organize it to get the most insight and value out of it.

So this raises the question of whether "big data" is really even an appropriate term. Increased activity concerning know-your-customer (KYC) data might lead one to think that type of data is rising to the level of "big data," but if compared to something like phone call records, even from just one major cell service provider, KYC data could look small in volume by comparison.

Is there a threshold that KYC data or financial services industry data as a whole needs to cross to truly be "big" and not just "medium," perhaps? Customer details, if cross-referenced with extensive transaction data, could be that "big" in this industry.

Failing that, firms may be better off not getting caught up in "big data" hysteria. It's more effective to work to know exactly what data you do have, how frequently you are getting it and what you are trying to achieve with it—namely, what insights you are trying to generate from it.

In effect, dark data and big data are really about the same thing—data management. Dark data describes knowing where all the meaningful data is and how to aggregate it, while big data infers understanding the complete picture of what data is coming in, especially if the volume of that data is getting larger.

To join a discussion on how "big data" should be defined, visit Inside Reference Data's LinkedIn discussion group.