Some Final Thoughts After the Bloomberg Outage

Ubiquity is a good thing; it means you've done both a fantastic job of building a reliable product and have marketed the product well. The problem with ubiquity is that everyone knows you and when something goes awry, everyone has a complaint and an opinion.

I would have to imagine that you all have heard by now about how Bloomberg suffered a global outage that essentially froze end-of-day trading in Asia and start-of-day dealing in Europe. We here at Waters spoke with the head of a Parisian asset manager, the CTO of a European futures market-making firm, the CTO of a US hedge fund which specializes in emerging markets, another very large US hedge fund and finally a trader at a European-based bank ... so I think that we received a fairly wide view of how this outage affected trading firms.

First, it sounds as though most everyone found out that this was not an internal issue but a Bloomberg issue from a news site or TV, and not from Bloomberg. I heard of people being woken up at 4 a.m. because of frantic texts coming from their offices in India and of people getting more information from speculation on Twitter than from the data-service behemoth.

One hedge fund CTO I spoke with was fairly pissed because his firm, the emerging markets specialist, couldn't finish its trading day in Asia because they depend on Bloomberg's OIMS AIM platform to enter orders and they hadn't heard from Bloomberg as to what the root cause of the problem was.

It would seem that the silence was almost as frustrating as the outage itself. After service had been fully restored by 11 a.m. ET, Bloomberg released this statement:

We experienced a combination of hardware and software failures in the network, which caused an excessive volume of network traffic. This led to customer disconnections as a result of the machines being overwhelmed. We discovered the root cause quickly, isolated the faulty hardware, and restarted the software. We are reviewing our multiple redundant systems, which failed to prevent this disruption.

Too Big to Fail

There's a lot of talk about Bloomberg being "too big to fail" in light of this event. As my colleague Max Bowie writes in his excellent rundown of the event, this kind of outage is rare ─ if not totally unheard of ─ when it comes to Bloomberg terminals. Because of the quality of the platform itself, and because Bloomberg's service has been of such high quality, the vendor has become synonymous for trading terminals the way that Q-tip has become synonymous for things that you shove into your ear.

To bash Bloomberg off this one incident is perhaps unfair. But, there are two big issues that will need to be improved upon. First, how did Bloomberg's robust network of backup systems manage to fail? That needs to be Priority Number One. Second on their to-do list, communication with clients will need to be improved in case of another similar outage.

There are other data terminal providers like Thomson Reuters or Money.com that are trying to pick away at Bloomberg's sizeable market share. It's a hard sell.

One of my colleagues told me that they read (sorry, I'm not sure which publication ... I'll add it in if I find out) that while some firms actually do have backup systems that they could switch over to (in this case, Thomson Reuters), the traders weren't familiar enough with the platform to use it or they just didn't trust the service. So to me, this will only prove a bump in the road, rather than a full-on changing of the tides.

Finally, look back at that comment from Bloomberg: excessive volume of network traffic...machines being overwhelmed...customer disconnections...multiple redundant systems, which failed to prevent this disruption. What does all this sound like?

I've heard from two contacts that say that this seems like a distributed denial of service (DDoS) cyber attack, though they have no proof and so far, Bloomberg denies that it was an attack.

But the more that they're quiet about what actually happened and what went wrong, the more that those rumors will persist. That's the nature of being ubiquitous.

Five Random Thoughts/Links

* Tomorrow is our annual North American Trading Architecture Summit in New York. I hope to see you there. Please feel free to come up and chat during one of the breaks. Also, that night we will announce the winners for all of this year's Sell-Side Technology Awards.

* Our special guest speaker at this year's Sell-Side Technology awards will be former heavyweight boxer Gerry Cooney. As someone who still serves as a freelance boxing writer, this is very exciting. Win or loss, a Cooney fight would always entertain. Here's a good mini-documentary on the Cooney-Larry Holmes fight, which is the last fight to really play on black-and-white racial divides.

* Speaking of boxing, if you haven't watched the Lucas Matthysse-Ruslan Provodnikov fight, get on your HBO GO account and check it out. Scintillating stuff.

* Just when we thought we were finally done hearing about them, the Philadelphia Eagles (and their sociopathic ─ I kid, I kid ─ head coach Chip Kelly) have gone and signed Tim Tebow. The next time you see Tim Bourgaize Murray, an Eagles fan, please make sure to give him flak for this.

* I was reading a fantastic profile of Boston Corbett ─ the man who shot and killed John Wilkes Booth ─ when I came across these two paragraphs...please enjoy the rest of your work week:

His rash tendencies exhibited themselves in strange ways. One day while he was ministering in the summer of 1858, Corbett was ogled by a pair of prostitutes, and the lower half of his body responded invitingly. He went home, took a pair of scissors, snipped an incision under his scrotum, and removed his testicles, then headed out to a prayer meeting.

In the Bible, Matthew 19:12 quotes Christ as saying "there are eunuchs who have made themselves eunuchs for the sake of the kingdom of heaven." Corbett made himself a eunuch and didn't check himself into Massachusetts General Hospital until he'd finished his prayers, had a full dinner, and taken a light stroll through the city that evening.