Goldman Sachs Hit By Triarch Parameter Problem

NEW YORK--Goldman Sachs recently suffered interruptions in its Reuters Triarch 2000 market data distribution platform because Reuters failed to fully document the parameters of a Triarch communications protocol and to provide notification that a fix is included in the data distribution software.

Sources say the shutdown in Triarch service could have been avoided had Reuters informed Goldman Sachs and other customers of the potential problem.

Goldman Sachs in New York became aware of a problem after it had upgraded to the Source Sink Library (SSL), version 4.0, of Triarch, confirm sources. The investment bank discovered that, at high volumes, its Triarch installation, in fact, "its entire backbone", needed to be shut down, sources say.

An industry source says that other major trading floors have experienced similar problems with their Triarch installations, and some floors have been down for hours. Industry observers point out that problems with Triarch communication protocols are rare; it's likely that the ever-increasing trading volumes of world markets will force the issue. While Reuters officials say they are unaware of shutdowns that have lasted for hours, they have had to respond to these problems.

While the exact number of Triarch positions affected at Goldman Sachs is unknown, Goldman has both Triarch and Tibco's The Information Bus (TIB) installations. In February 1995, the firm deployed Triarch for 370 equity traders working out of its 1 New York Plaza offices (Trading Systems Technology, May 13, 1996). In an earlier installation, Goldman Sachs implemented Triarch for 10 trading positions in its quantitative strategies group (TST, November 30, 1992). Officials at Goldman Sachs did not return phone calls by press time; the investment bank kicked off an IPO last week.

With Reuters as the sole owner of the two major market data distribution platforms, customers' options are limited especially if they're customers with large installations, says an industry observer.

The problem lies in the configuration of the Reuters Reliable Controlled Protocol (RRCP), one of the two protocols that make up the Reuters Reliable Data Protocol (RRDP), the other being the Reuters Reliable Monitor Protocol (RRMP). The RRCP software daemon serves as a check upon the sequence numbers on message packets sent from Triarch source servers so that packets are not missed or lost.

The problems start when the RRCP detects that a message is out of sequence. At that point, the RRCP protocol requests that the server re-send the correct sequence of messages. Until the server does this to the satisfaction of the RRCP, the protocol will not complete the distribution to the RRMP level. If this continues, it leads to serious problems. Trading rooms can suffer a network storm, lost messages or stale data.

High volumes can trigger problems with RRCP if the volumes exceed pre-set parameters. Once the RRCP begins to fail, the sink, source and service distributors of the SSL infrastructure cannot communicate properly. The only way to fix the problem is to halt network communications among the various parts of SSL and restart them. To say the least, this is not an easy process.

To get around this problem, Reuters has included in SSL 4 a variable that allows users to adjust the RRCP parameter governing volumes so that they will not overwhelm the protocol. However, although Reuters made certain that the RRCP could handle high volumes, nowhere is the existence of the variable documented, says an industry source familiar with the situation. In addition, Reuters has not provided any warnings about high-volume levels, says the source. So, Goldman Sachs, among others, was left in the dark. "It is not an example of elegant software support," says a source.

Goldman Sachs learned firsthand what the lack of documentation and support can mean. In an effort to fix the problem, Goldman Sachs flew in a Triarch consultant, an "ex-Reuters guy", who was able to discover a remedy for the situation, say sources.

While Reuters officials decline to comment upon the specifics of Goldman Sachs' experiences and the lack of documentation for the RRCP variable, Graham Albutt, president, Reuters Trading Systems for the Americas, acknowledges that "there have been a number of these conversations that have gone on" about RRCP. "Certainly, it is true to say that volumes are up," Albutt says.

In fact, Reuters has done more than talk, Albutt says. The vendor has dispatched Triarch engineers to user sites as well as put user firms in touch with its development groups in Oak Brook, Illinois and its Effix division in France. "The whole of Triarch is parameterized," Albutt says. "There is a need to alter those parameters" according to user-specific configurations.

Albutt confirms that problems with RRCP "could lead to some delays" in market data delivery. "It could lead to an application not responding quickly enough," he says. "The answer is tuning the system to meet the traffic profiles."