Anthony Malakian: The Swimmers in the Lake

The arms race for programmer talent has been a long-discussed topic in Waters, and there’s no reason to think it won’t continue to be a challenge for the foreseeable future—at least until our robot overlords finally wrest control of civilization from our hands.

While at the Buy-Side Technology North American Summit, held in mid-October in Manhattan, one panel discussed the merits of deploying data lakes versus data warehouses. The debate was interesting and led me to write a feature on the subject. It also got me thinking about the talent challenge, but more on that in a minute.

Two of the panelists—one from MetLife, the other representing JPMorgan Asset Management—were fairly evangelical on the subject of data lakes. The funny thing is, however, that their definition of a data lake would, I imagine, differ from some hardcore programmers. JPMorgan’s Scott Burleigh, in fact, said that his firm governs the data before it enters the lake, which I think many would argue makes the lake more of a warehouse than a pure lake. 

But I also think that this is a natural progression of new technologies and strategies: The definition is morphed, stretched and molded to fit a particular industry. The capital markets space is likely to have a different view of data lakes than that adopted by music streaming services, in the same way that Wall Street had a different definition of the cloud compared with Silicon Valley where the technology first emerged.

Limited Skillset
I was speaking with Gartner’s Nick Heudecker on this topic, who has written a research paper on data lakes. He has also noticed that the term data lake is being thrown around to describe a lot of things. It’s even been used interchangeably with Hadoop. The key, he says, is to look at the underlying architecture if you really want to see what it is. 

You’ll need innovative programmers sitting alongside the data scientists and quants to help extrapolate value from these lakes.

“If I’m building a very detailed semantics layer and add security in the way that I would do in my data warehouse, and I’m optimizing data for storage or reuse, is it really a data lake, or is it a different implementation of a data warehouse architecture? People confuse the physical implementation of the data warehouse with the architectural considerations,” he said. “And that’s fine; call it what you want. But I want to know about the architecture and from there we can figure out what characteristics you’re going to need.”

The real question is, who is going to build that architecture? As Dan Graham of Teradata told me, a data lake is “limited to what a programmer can do. It’s up to a programmer to do a lot of do-it-yourself work in the data lake.” 

At some point, this raw, unstructured data is going to have to be cleaned and used, otherwise it’s just a data dumping ground without a purpose. And the cleaning part might not be as easy when it’s on such a massive scale. 

“All of these data cleansing techniques are a foreign idea and a new area of exploration for the Hadoop data lake people,” Graham noted. 

The problem for capital markets firms is they don’t want to have to hire separate cleansing specialists to go along with their Hadoop specialists.

Data Needs Programmers
I’m tired of the term “big data.” It’s too all-encompassing and vague. But as the volume of data grows at exponential rates—according to IBM, 2.5 quintillion bytes of data is created daily—a term needs to be created and accepted, so for the time being let’s just go with big data.

If firms want to find purpose and value in that sea of information, data lakes will prove to be a starting point. It’s like oil refining: you collect it, separate it, convert it, and treat it. You want to collect as much as possible, but the art is in the downstream stages.

You’ll need innovative programmers sitting alongside the data scientists and quants to help extrapolate value from these lakes. So, if you think the talent race was going to slow down anytime soon, you’re wrong.