The removal of the technology company building an ambitious database to track all equity and options trades in the US sent shockwaves through Wall Street at the end of January, providing a dramatic end to the first act of an already-theatrical story.

Yet those familiar with the construction of the Consolidated Audit Trail (CAT) say that the decision to drop Thesys CAT—a subsidiary of technology vendor Thesys Technologies, which beat numerous established firms to win the contract in early 2017—is just the latest in a series of episodes that have characterized the troubled history of the CAT.

The dozens of insiders and experts who spoke with WatersTechnology for this report paint a picture of a combination of mismanagement on all sides, infighting, inflated expectations, regulatory inaction and overly ambitious projections, miring the CAT in a state of limbo, half-active and half-finished, even as it remains dangerously out of compliance with the rule that authorized it in the first place.

These sources, most of whom asked to remain anonymous, say problems first began to appear with the CAT long before January 31, when the self-regulatory organizations (SROs) tasked with running the project dismissed Thesys, going back even before the bidding process, and continuing through the subsequent delays that occurred in 2017 and the go-live in 2018. The blame for the disruption, they say, cannot be laid at the feet of any one of the entities involved, but is spread among the vendors, the SROs and the regulators alike.

Early Errors

Following the 2010 Flash Crash (see box: “Herding the CAT”), in 2012 the US Securities and Exchange Commission (SEC) approved Rule 613, mandating the National Market System (NMS) exchanges to begin work on a comprehensive audit trail of market activity. The Financial Industry Regulatory Authority (Finra) had operated its own Order Audit Trail System (Oats) since the late 1990s, but the CAT would be an altogether different beast: Whereas Oats only collected data from the NMS exchanges, the CAT would capture all trade data, from all market participants. Yet, despite the regulatory impetus, sources close to the regulators and the SROs at that time say there wasn’t universal support for the CAT.

“I think what the industry really wanted was Finra to continue doing Oats, because it’s crappy and opaque. The CAT is granular, time-stamped data that’s shared among the exchanges, so NYSE gets to see Nasdaq’s order handling, Nasdaq gets to see Bats, and so forth,” says a source familiar with the early days of the CAT.

Despite a lack of unanimity on its purpose, however, the SROs published a request-for-proposal in 2013, with responses due by March in the following year, in which firms would bid to build and operate the system on behalf of the NMS Plan. It would take years to reach that stage.

Throughout 2014, the SROs shortlisted 10 firms from 31 initial bidders and filed their initial plan with the SEC. It would be amended and restated multiple times before the SEC put it forward for public comment in April 2016, finally approving it in November—already four years after its mandate to create the CAT.

By this time, the number of competitors vying to build the CAT had been whittled down to a handful. Finra had long been considered the front runner, owing to its experience operating Oats, but sources say that a change of the guard at the very top of the regulator also changed its outlook. Richard Ketchum, then CEO of Finra, had been an ardent supporter of the CAT—and, crucially, of Finra being the one to build it. However, Ketchum announced his retirement in 2016, with Robert Cook taking over as the head of the regulator.

Sources say that Finra had been growing increasingly uncomfortable with the demands that the contract would place on it—some say there was an element of a conflict of interest between it processing this data and being the body responsible for regulating broker-dealers; others say Finra’s discomfort stemmed more from what it would be asked to perform as part of the project.

“The only thing, in my opinion, that tanked it for Finra was the fact that Ketchum had just left: He was the one who really wanted the CAT for Finra. Ketchum was a businessman—he saw the dollar signs. Cook didn’t want the CAT. Cook is a true-believer regulator, and he saw the conflict-of-interest issue,” says one person familiar with the bidding process, although sources close to the regulator’s thinking dispute that Cook put the kibosh on the CAT.

The design kept changing and there was just never going to be enough time for such a monumental project [to be delivered]

Source familiar with the early days of the CAT.

According to those involved at the time, Finra’s interest began to wane—although it remains a vocal proponent of the CAT’s benefits in terms of surveillance—leaving the field open for Thesys to step in. The vendor, at the time an arm of proprietary trading shop Tradeworx, had previously built a similar system for the SEC, known as the Market Information and Data Analysis System, or Midas.

This helped, of course, but it wasn’t the only reason why the SROs chose Tradeworx. Multiple sources confirm to WatersTechnology that, in comparison to some of the bids submitted by other shortlisted firms, Thesys proposed to build the CAT at a cost far below everyone else’s projections. The bid numbers remain confidential, and cannot be independently verified, but estimates of Thesys’ bid from sources place it around 50 percent cheaper or more than those at the top end of the scale. A 2017 Finra rule filing would subsequently reveal that the CAT would cost the industry just over $50 million for that fiscal year, with $37.5 million earmarked for development.

Multiple sources cite this as the start of problems with the CAT’s construction, leaving aside what some call prevarication and delay tactics on the part of the SROs up to this date. Thesys, they say, underestimated both the cost and the complexity of what it was bidding on. In effect, the company bit off more than it could chew.

“We all thought Finra was going to win,” says a person close to the selection process. “They were both bidders and on the selection committee. It was a ridiculous process. But then, Finra backed out at the last minute. So, at the end of the process, the only bidder left standing was Thesys, and that bid was accepted—a bid that, in my opinion, was way too low to build the CAT.”

But if the SROs thought that at the time, it didn’t show. Thesys/Tradeworx was confirmed as the plan processor for the CAT on January 17, 2017, although WatersTechnology understands that the contract wasn’t officially signed until April. This gave the vendor just seven months to deliver the platform for the first phase.

Things started to go wrong almost immediately.

Missteps

Thesys would later break away from Tradeworx and become its own business, Thesys Technologies, and establish Thesys CAT LLC as a subsidiary company designed to handle the build-out and operations of the CAT in early 2018. This effectively firewalled it from Thesys’ other operations, partly to nullify concerns that a third-party vendor would have an enormous advantage over its competitors owing to the information it had flowing through its pipes on a daily basis. For now, however, it had an enormous contract on its hands, but a shocking lack of guidance on how to accomplish it.

One of the first critical missteps in the CAT’s construction, multiple sources say, was the dissolution of working groups and management structure within the NMS Committee, following the appointment of Thesys as a plan processor. This triggered months where the vendor was operating in the dark, without coordination or direction from the SROs, outside of loosely organized conference calls and meetings.

Insiders say any chance of delivery was crippled by the inability of the SROs to agree on basic elements of the platform’s design and specifications. “The design kept changing and there was just never going to be enough time for such a monumental project [to be delivered],” says a person familiar with the early days of the CAT’s development.

The fact that there was no single guiding hand behind the project—rather, dozens of people with their own agendas trying to come to a common conclusion—meant decision-making was critically slow at the best of times.

“From a technical point of view, it’s very complicated, but also from a decision-making, organizational structure—all those kinds of things—it was very complicated as well, and it didn’t have enough time,” says David Campbell, vice president at Broadridge, which had also bid for the CAT contract.

There were several delays in the RFP process, and each time the SROs submitted a request for approval of delays, and each time the SEC agreed.

Source familiar with the RFP process.

The CAT, effectively, was in trouble from the word go. Leaving aside the fact that Thesys had been given the contract with little time to meet the first-phase deadline, the CAT had also started to attract an enormous amount of criticism and concern regarding cybersecurity. A number of major breaches had occurred already in 2017: credit agency Equifax had been compromised by criminals, exposing the personal data of tens of millions of Americans, and even the SEC’s company filings system was breached, resulting in fears that personally identifiable information had potentially been compromised.

With even the SEC seemingly not safe from hackers, the CAT became something of a cybersecurity whipping boy in the media and in Congress, resulting in SEC chair Jay Clayton being grilled in both House and Senate hearings about what information the CAT would be collecting, and how it would be protected. The SEC found itself in an awkward position—it was the SROs that were tasked with building the CAT, yet the SEC was increasingly being asked to shoulder responsibility for its defenses. In addition, it was getting dragged into an uncomfortable discussion about what types of information the CAT would collect, and whether that was appropriate.

As Clayton was being grilled by Congress, politicians were urging the SEC to the delay the rollout of the CAT. Republican Congressman Jeb Hensarling of Texas, then-chair of the House Financial Services Committee, added his concerns at a hearing on October 4, 2017. “With the [CAT] serving as a central repository for order and trading activity data, I urge the SEC, again, to delay its implementation date until the Commission can ensure that the appropriate safeguards and internal controls are in place to protect this data,” he said—something Clayton firmly ruled out, although he acknowledged that the SEC would reexamine what type of data it would collect.

The CAT would later come under additional political pressure when a bill that sought to prevent the CAT from collecting personally identifiable information (PII), cleared the House Financial Services Committee in mid-2018. That bill could have forced a re-engineering of the entire system at a delicate time, months away from the revised November 2018 implementation of phase one, but failed to clear the House floor.

Delays

At the time, this all proved too much for the SROs and for Thesys, all of whom were quickly realizing that they didn’t have a hope of sticking to the original timeline. As November approached, the NMS Plan put together an alternate proposal in which timelines for the various phases of implementation would be delayed by up to a year or more.

They had little reason to believe that the SEC (which declined to comment for this article) wouldn’t approve the changes. Up until that point, the agency had proved remarkably amenable to amendments and adjustments to the 2014 plan, to the point where some sources say that even extraordinarily dubious reasons for amendments were being passed through and rubber stamped.

“There were several delays in the RFP [request-for-proposal] process, and each time the SROs submitted a request for approval of delays, and each time the SEC agreed to these well-written—even though they were bullshit—submissions,” says the source involved in the original bidding process.

On November 13, 2017, the SROs hand-delivered their revised plan to the SEC, two days before they were due to begin submitting reports to the CAT, which at that point, simply did not exist.

Clayton refused the request on November 14.

“I recognize that recently the SROs have worked together to develop an action plan for bringing the CAT on-line, albeit on a delayed basis. Further, it is clear that the SROs’ increased engagement with the SEC in recent days has been constructive,” said Clayton in a statement. “However, I am not in a position to support the issuance of the requested relief on the terms currently proposed.”

As far as the SEC was concerned, he said, the original timeframes still applied. November 15 came and went without a single report being filed, and the CAT was officially out of compliance.

Despite this, an enforcement action from the SEC failed to materialize. In the absence of any other guidance, the SROs said they were continuing to work for their revised—and rejected—plan. The SEC would remain publicly silent on the matter for months to come.

I am not in a position to support the issuance of the requested relief on the terms currently proposed.

SEC chair Jay Clayton, in a statement from November 14, 2017.

The SROs decided that work couldn’t proceed without an appropriate chief information security officer (CISO) being appointed to oversee the protection of the CAT, thanks to the criticism it had come under in recent months. Here, too, the weaknesses of the NMS structure—and getting nearly two-dozen people to agree on something—became apparent. People familiar with the meetings to approve a CISO describe a series of candidates being presented to the SROs, all of whom were rejected for one reason or another in a series of meetings rife with disagreement and vacillation.

“It was the longest hiring of a CISO in history,” says one person familiar with the hiring process.

Discussions continued for months, until in early February, Thesys announced the appointment of Vas Rajan, the former CISO of CLS Bank, as its cyber chief. He hit the ground running, filing a security plan shortly after, which was approved in May. Indeed, this period marked something of a turning point for the project. It was around this time that the Operating Committee re-established its working groups to guide the development of the CAT, a process which stretched through to the summer. It also appointed a leadership team, led by Nasdaq’s Cindy Retterer and supported by NYSE’s Soniya Shrivastav, Finra’s Shelly Bohlin and Cboe’s Tom Busch in June to drive the project forward.

“The CAT NMS group and the SROs really started to figure this out last summer when they came up with the streams and the stream leads and they started to assign more individual responsibility to this person, or this group is in charge of figuring this out, versus everything being done at that all-the-SROs-together level,” says Broadridge’s Campbell.

It was also around this time that work began in earnest on building the platform itself, and, perhaps, when the scope of what was being asked began to become clear to its builders.

Thesys CAT had partnered with a number of firms from the beginning—law firm Latham & Watkins (for legal counsel), IBM (to host the platform), and broker Rosenblatt Securities were all named as partners in its initial bid—though it is now understood that Amazon Web Services would eventually host the platform. However, the real firepower came from the appointment of Sapient. Widely regarded as a specialist in the field of regulatory reporting, Sapient had been engaged in similar trade reporting projects in Europe through reforms introduced by the Markets in Financial Instruments Regulation, helping banks to interface with Approved Publication Arrangements that went live in January 2017, not entirely without incident.

Now, it was being contracted to build the systems on the other end—albeit secretly. Sources familiar with Sapient’s work say that its involvement in the project was kept very quiet, even internally, and that only a select group knew that it was working on the CAT.

The distribution of work between Thesys and Sapient is difficult to determine, although it’s clear that Sapient handled a range of projects in at least an augmentative capacity, from building the front-end user interfaces through to query engines, and major architectural concerns. With the new structure and coordination provided by the Operating Committee, elements of the CAT began to take shape, although sources say that work remained slow even then.

It still wasn’t enough. While the SEC acknowledged that the project was now working to its new proposed timeframes—even though it stopped short of explicitly approving them—and that progress was being made, delays were, at this point, inevitable.

Thesys informed the SROs that the full functionality for phase one of implementation would not be available for the November 15 launch, although it would begin ingesting data in a limited capacity. That day came and went largely without incident, sources say, and full functionality was intended to be put in place by January 31. That deadline, too, would end up being pushed back, according to subsequent testimony from the SEC’s Clayton, who appeared to be rapidly losing patience.

“Here’s the analogy that I would use: When the SEC wrote this rule, think of 13 crazy aunts and uncles and they are the people that the SROs put in charge of this,” says a source familiar with the initial process. “Think of the SEC putting them in a van, driving them out into the desert, handing them shovels and saying, ‘Dig your own graves, we’ll be back in a year!’ The SEC comes back, and these guys are still standing with shovels and they’re like, ‘But we told you to dig your own graves! This time, we’re serious. We’ll come back in a year and this time you’d better have dug them.’ They come back and they’re still standing there.”

For the SEC, it was fast becoming the last straw.

Repercussions

It was after the November go-live that the SEC seemed to show a pronounced change in attitude. On December 11, testifying before the Senate Committee on Banking, Housing and Urban Affairs, Clayton came out swinging on the subject of the CAT.

“While the CAT has now begun receiving equity and options data with limited functionality, the SROs remain out of compliance with the CAT NMS Plan today,” he said. “The SROs are making some progress, but the development and implementation process remains slow and cumbersome due largely to what I believe are project governance and project management issues experienced by the SROs.”

The SROs had, Clayton said, set out a more detailed roadmap after requests from SEC staff, but Thesys had subsequently informed them that it would not be able to deliver full functionality for the first phase in line with this, and the SROs had subsequently informed the regulator that the first phase would not be fully implemented until March 31—nearly 18 months after the original deadline.

“We remain frustrated with failure of the SROs to meet their obligations, and the delays in the development of the CAT,” he continued.

I worked with [Kimmel] and I’ve seen her get the ball across the goal line. Sometimes you have to compromise to get it done, but she knows how to do it.

Thomas Jordan, CEO, Jordan & Jordan.

The US government shutdown didn’t help matters. Starting on December 22, it continued for more than a month, with the government officially reopening on January 25, 2019. Because the SEC draws its funding from a mix of government appropriations and fees paid to it by regulated entities, and as such, could keep critical functions operating, it wasn’t as hobbled as some federal bodies. But its ability to move on issues with the CAT was still hindered, even as market oversight continued amid the chaos of other departments being reduced to skeleton staff.

However, when the government reopened, the SEC moved quickly and dramatically, signaling that Clayton’s remarks hadn’t just been hot air in the face of a Senate committee. On January 29, 2019, it announced the appointment of former Refinitiv compliance chief Manisha Kimmel as a senior policy advisor, reporting directly to Clayton, with a brief explicitly focused around the CAT.

Many see this as a long-awaited appointment of a “CAT tsar” by the SEC—a person who would finally represent the agency in meetings regarding the CAT, and who would be empowered to drive the project forward. This, some say, had been clearly lacking to date, leading to many of the governance and project management issues faced by the CAT, highlighted in Clayton’s testimony.

Crucially, people who know Kimmel say that she is precisely the right person from the job. She has been deeply entrenched in the CAT project already, having served on its advisory committee, as well as through her work as a managing director of data industry association the Financial Information Forum. More than that, they say, she can get the CAT over the line.

“You have to be able to execute. There are people who can talk the talk, who can teach, but they can’t get the ball across the goal line,” says Thomas Jordan, president and CEO of consultancy Jordan & Jordan. “I worked with her and I’ve seen her get the ball across the goal line. Sometimes you have to compromise to get it done, but she knows how to do it.”

Moreover, sources say, these moves betray a growing impatience not just among the SEC’s top brass, but among its staff as well, in the languid progress that has been made on the CAT to date. In any case, Kimmel’s appointment presaged the biggest controversy to face the CAT project yet, when just two days later, the SROs announced that Thesys would no longer be the plan processor.

Endgame

By all accounts, there had been problems brewing between Thesys CAT and the SROs for some time. Sources familiar with Thesys say that the vendor had been growing frustrated with the SROs over the timeliness of their payments to the vendor for work accomplished, while those familiar with the SROs say that the vendor had finally begun to appreciate the scale of what it had signed up for. It had encountered problems with technical aspects and was asking the Operating Committee for more money, they claim.

Then, on January 31, The Wall Street Journal reported that Thesys had been fired as the plan processor. The Operating Committee released a statement shortly after, on February 1, saying that Thesys would no longer build and operate the CAT, although it would assist in the transition to a new processor, and that while some testing phases would be affected, the core technical specifications would not change.

The full picture of exactly what happened around January 31, and the exact reasons behind Thesys’ ousting, remain unclear, with accounts differing about precisely what occurred. People on both sides accuse the other of handling the situation poorly—sources familiar with the situation say, alternately, that when Thesys asked for more money to accomplish its tasks, relations became acrimonious, while others say that the news of Thesys’ firing had already been leaked while the company was being told it would not be needed for the next phases of implementation. A number of alternate strategies were put on the table, including a possible acquisition of Thesys CAT LLC by a third party, but these were ultimately not pursued, and the Operating Committee issued its statement saying that it was parting ways with the vendor.

[The CAT] is more complicated than what it’s replacing, and there are a lot of open questions that people still have.

David Campbell, Broadridge.

News reports tipped Finra, which had been the front runner initially, to become the new plan processor. Sources confirm to WatersTechnology that Finra is in discussions to take over the role, although Finra itself declined to comment and referred queries to spokespeople for the SROs. At the time of publication, the identity of the new plan processor had not been formally announced.

Just how long it will take the CAT to transfer from Thesys to Finra is also unclear, although industry experts have been warning broker-dealers, who are due to report to the CAT in phase two, that they shouldn’t expect any significant delays to implementation timelines.

Indeed, reaction to the news has been measured, even if it took most of the industry by surprise. Most say that, despite the change in processor, enough experience has been gained for the project to shed many of the difficulties that have plagued it to date.

“I think there have been a lot of lessons learned and hopefully, while they’re negotiating and getting the new plan processor in place, they can utilize some of those lessons learned to make sure that the next step of the process works better,” says Broadridge’s Campbell.

The Long March

On February 20, the CAT NMS Operating Committee will host a CAT webcast to discuss where things stand with the plan.

Putting concerns over this being another delay tactic on the part of the SROs aside, there is still a multitude of issues that need to be addressed. Chief among them is the state of the technology underlying the CAT: While most sources assume that CAT NMS owns the intellectual property, the particulars of the contract are not public. Also important is that while Thesys will not be the plan processor moving forward, the CAT is currently operational in a limited form, and ingesting upwards of 100 billion records per day.

Since CAT NMS has said that core technical specifications won’t change, it’s a safe bet that the same technology will remain in place. And the fact that the platform is hosted in the cloud will make some aspects of a transition easier, but other details are yet to be fully addressed. For example, an even more disruptive element could be the parts that haven’t been pushed to production yet, or are still being developed. Taking over a technology project is a difficult task at the best of times, let alone one as nationally significant as the CAT, which already attracts extreme scrutiny from both regulators and the media, and which still isn’t finished.

“There are also a lot of other things still outstanding in terms of fields, linkages, a lot of the detail underneath specifically how the scenarios would need to work,” says Broadridge’s Campbell. “[The CAT] is more complicated than what it’s replacing, and there are a lot of open questions that people still have.”

As to the future of the CAT, the time has long passed when it would simply be theory, or a form of Oats-plus [see box: “Oats… Plus/Minus”]. Those actively working with firms on projects say that the broker-dealers, not the SROs, have already been sold on what the CAT can provide for them in terms of data management, and that firms have already budgeted plans and assigned resources to those efforts.

“While things are perceived to be up in the air given the change within the plan processor, the effort that has been done up to date is being looked at twofold,” says Michael Drews, managing principal at Capco. “First, [industry participants] are not overly concerned that this is going to go away; this is going to happen and they’re going to continue to prepare and ready themselves. The other aspect is you’re looking at data that has been siloed historically, and the ability to link that data… can be leveraged for more than just this reporting obligation, and it can benefit you from a business perspective and functional-area perspective.”

That said, an element of risk to the project still remains. SEC enforcement action is still a distinct possibility, particularly now that the regulator has been so vocal about its displeasure with the SROs—and at this point, say people familiar with the regulator, another delay may be inevitable.

Indeed, should delays occur—as is already the case with the testing schedule being pushed back—they will have to be short and sweet, lest they risk causing further disruption to an industry already beset with regulatory projects.

“The problem is that at these companies you have certain resources assigned to projects and there’s no shortage of projects,” says Jordan & Jordan’s Jordan. “There are still issues that have to get resolved in Europe on Mifid. What if Brexit happens? There’s going to be competition for those resources. So this delay has to be short so that people can keep their resources in place that they’ve been putting on getting ready for the CAT.”

Most expect the transition to a new processor to be smoother when compared to the CAT’s original journey, and that the problems of the past seem to be behind the project. Yet, if the chaos of the past few years—and, particularly, the past few weeks—has demonstrated, when it comes to the CAT, nothing is certain.