Non-fiction Book of the Month - Flash Boys
HIDDEN IN PLAIN SIGHT
By the summer of 2009 the line had a life of its own, and two thousand men were digging and boring the strange home it needed to survive. Two hundred and five crews of eight men each, plus assorted advisors and inspectors, were now rising early to figure out how to blast a hole through some innocent mountain, or tunnel under some riverbed, or dig a trench beside a country road that lacked a roadside—all without ever answering the obvious question: Why? The line was just a one-and-a-half- inch-wide hard black plastic tube designed to shelter four hundred hair-thin strands of glass, but it already had the feeling of a living creature, a subterranean reptile, with its peculiar needs and wants. It needed its burrow to be straight, maybe the most insistently straight path ever dug into the earth. It needed to connect a data center on the South Side of Chicago to a stock exchange in north- ern New Jersey. Above all, apparently, it needed to be a secret.
The workers were told only what they needed to know. They tunneled in small groups apart from each other, with only a local sense of where the line was coming from or where it was going to. They were specifically not told of the line’s purpose— to make sure they didn’t reveal that purpose to others. “All the time, people are asking us, ‘Is this top secret? Is it the govern- ment?’ I just said, ‘Yeah,’ ” said one worker. The workers might not have known what the line was for, but they knew that it had enemies: They all knew to be alert to potential threats. If they saw anyone digging near the line, for instance, or noticed anyone asking a lot of questions about it, they were to report what they’d seen immediately to the head office. Otherwise they were to say as little as possible. If people asked them what they were doing, they were to say, “Just laying fiber.” That usually ended the conversation, but if it didn’t, it didn’t really matter. The construction crews were as bewildered as anyone. They were used to digging tunnels that connected cities to other cities, and people to other people. This line didn’t connect anyone to anyone else. Its sole purpose, as far as they could see, was to be as straight as possible, even if that meant they had to rocksaw through a mountain rather than take the obvious way around it. Why?
Right up until the end, most workers didn’t even ask the question. The country was flirting with another depression and they were just happy for the work. As Dan Spivey said, “No one knew why. People began to make their reasons up.”
Spivey was the closest thing the workers had to an explanation for the line, or the bed they were digging for it. And Spivey was by nature tight-lipped, one of those circumspect southerners with more thoughts than he cared to share. He’d been born and raised in Jackson, Mississippi, and, on those rare occasions he spoke, he sounded as if he’d never left. He’d just turned forty but was still as lean as a teenager, with the face of a Walker Evans tenant farmer. After some unsatisfying years working as a stockbroker in Jackson he’d quit, as he put it, “to do something more sporting.” That turned out to be renting a seat on the Chicago Board Options Exchange and making markets for his own account. Like every other trader on the Chicago exchanges, he saw how much money could be made trading futures contracts in Chicago against the present prices of the individual stocks trading in New York and New Jersey. Every day there were thousands of moments when the prices were out of whack—when, for instance you could sell the futures contract for more than the price of the stocks that comprised it. To capture the profits, you had to be fast to both markets at once. What was meant by “fast” was changing rapidly. In the old days—before, say, 2007—the speed with which a trader could execute had human limits. Human beings worked on the floors of the exchanges, and if you wanted to buy or sell anything you had to pass through them. The exchanges, by 2007, were simply stacks of computers in data centers. The speed with which trades occurred on them was no longer constrained by people. The only constraint was how fast an electronic signal could travel between Chicago and New York—or, more precisely, between the data center in Chicago that housed the Chicago Mercantile Exchange and a data center beside the Nasdaq’s stock exchange in Carteret, New Jersey.
What Spivey had realized, by 2008, was that there was a big difference between the trading speed that was available between these exchanges and the trading speed that was theoretically possible. Given the speed of light in fiber, it should have been pos- sible for a trader who needed to trade in both places at once to send his order from Chicago to New York and back in roughly 12 milliseconds, or roughly a tenth of the time it takes you to blink your eyes, if you blink as fast as you can. (A millisecond is one thousandth of a second.) The routes offered by the vari- ous telecom carriers—Verizon, AT&T, Level 3, and so on—were slower than that, and inconsistent. One day it took them 17 milliseconds to send an order to both data centers; the next, it took them 16 milliseconds. By accident, some traders had stumbled across a route controlled by Verizon that took 14.65 milliseconds. “The Gold Route,” the traders called it, because on the occasions you happened to find yourself on it you were the first to exploit the discrepancies between prices in Chicago and prices in New York. Incredibly to Spivey, the telecom carriers were not set up to understand the new demand for speed. Not only did Verizon fail to see that it could sell its special route to traders for a fortune; Verizon didn’t even seem aware it owned anything of special value. “You would have to order up several lines and hope that you got it,” says Spivey. “They didn’t know what they had.” As late as 2008, major telecom carriers were unaware that the financial markets had changed, radically, the value of a millisecond.
Upon closer investigation, Spivey saw why. He went to Washington, DC, and got his hands on the maps of the existing fiber cable routes running from Chicago to New York. They mostly followed the railroads and traveled from big city to big city. Leaving New York and Chicago, they ran fairly straight toward each other, but when they reached Pennsylvania they began to wiggle and bend. Spivey studied a map of Pennsylvania and saw the main problem: the Allegheny Mountains. The only straight line running through the Alleghenies was the interstate highway, and there was a law against laying fiber along the interstate highway. The other roads and railroads zigzagged across the state as the landscape permitted. Spivey found a more detailed map of Pennsylvania and drew his own line across it. “The straightest path allowed by law,” he liked to call it. By using small paved roads and dirt roads and bridges and railroads, along with the occasional private parking lot or front yard or cornfield, he could cut more than a hundred miles off the distance traveled by the telecom carriers. What was to become Spivey’s plan, then his obsession, began with an innocent thought: I’d like to see how much faster someone would be if they did this.
In late 2008, with the global financial system in turmoil, Spivey traveled to Pennsylvania and found a construction guy to drive him the length of his idealized route. For two days they rose together at five in the morning and drove until seven at night. “What you see when you do this,” says Spivey, “is very small towns, and very tiny roads with cliffs on one side and a sheer rock wall on the other.” The railroads traveling east to west tended to tack north and south to avoid the mountains: They were of limited use. “Anything that wasn’t absolutely east- west that had any kind of curve in it I didn’t like,” Spivey said. Small country roads were better for his purposes, but so tightly squeezed into the rough terrain that there was no place to lay the fiber but under the road. “You’d have to close the road to dig up the road,” he said.
The construction guy with him clearly suspected he might be out of his mind. Yet when Spivey pressed him, even he couldn’t come up with a reason why the plan wasn’t at least theoretically possible. That’s what Spivey had been after: a reason not to do it. “I was just trying to find the reason no [telecom] carrier had done it,” he says. “I was thinking: Surely I’ll see some road-block.” Aside from the construction engineer’s opinion that no one in his right mind wanted to cut through the hard Allegheny rock, he couldn’t find one.
That’s when, as he puts it, “I decided to cross the line.” The line separated Wall Street guys who traded options on Chicago exchanges from people who worked in the county agencies and Department of Transportation offices that controlled public rights-of-way through which a private citizen might dig a secret tunnel. He sought answers to questions: What were the rules about laying fiber-optic cable? Whose permission did you need? The line also separated Wall Street people from people who knew how to dig holes and lay fiber. How long would it take? How many yards a day might a crew with the right equipment tunnel through rock? What kind of equipment was required? What might it cost?
Soon a construction engineer named Steve Williams, who lived in Austin, Texas, received an unexpected call. As Williams recalls, “It was from a friend of mine. He said, ‘I have an old friend whose cousin is in trouble, and he has some construction questions he needs answers to.’ ” Spivey himself then called. “This guy gets on the phone,” recalls Williams, “and is asking questions about case sizes, and what kind of fiber you use, and how would you dig in this ground and under this river.” A few months later Spivey called him again—to ask him if he would supervise the laying of a fifty-mile stretch of fiber, starting in Cleveland. “I didn’t know what I was getting into,” said Williams. Spivey told him nothing more about the project than what he needed to know to lay a single fifty-mile stretch of cable. In between, Spivey had persuaded Jim Barskdale, the former CEO of Netscape Communications and a fellow native of Jackson, to fund what Spivey estimated to be a $300 million tunnel. They named the company Spread Networks, though they disguised the construction behind shell companies with dull names like Northeastern ITS and Job 8. Jim Barksdale’s son, David Barksdale, came on board—to cut, as quietly as possible, the four hundred or so deals they needed to cut with townships and counties in order to be able to tunnel through them. Williams then proved so adept at getting the line into the ground that Spivey and Barksdale called and asked him to take over the entire project. “That’s when they said, ‘Hey, this is going all the way to New Jersey,’ ” Williams said.
Leaving Chicago, the crews had raced across Indiana and Ohio. On a good day they were able to lay two to three miles of the line in the ground. When they arrived in western Pennsylvania they hit the rock and the pace slowed, sometimes to a few hundred feet a day. “They call it blue rock,” says Williams. “It’s hard limestone. And it’s a challenge to get through.” He found himself having the same conversation, over and over again, with Pennsylvania construction crews. “I’d explain to them that we need to go through some mountain, and one after another they would say, ‘That’s crazy.’ And I would say, ‘I know that’s crazy, but that’s how we’re doing it.’ And they would ask, ‘Why?’ And I’d say, ‘It’s more of a customized route to the owner’s wishes.’ ” To which they really didn’t have much to say except, “Oh.” His other problem was Spivey, who was all over him about the slightest detours. For instance, every so often the right-of-way crossed over from one side of the road to the other, and the line needed to cross the road within its boundaries. These constant road crossings irritated Spivey—Williams was making sharp right and left turns. “Steve, you’re costing me a hundred nano- seconds,” he’d say. (A nanosecond is one billionth of one second.) And: “Can you at least cross it diagonally?”
Spivey was a worrier. He thought that when a person took risks, the thing that went wrong was usually a thing the person hadn’t thought about, and so he tried to think about the things he wouldn’t naturally think about. The Chicago Mercantile Exchange might close and move to New Jersey. The Calumet River might prove impassable. Some company with deep pockets—a big Wall Street bank, a telecom carrier—might discover what he was doing and do it themselves. That last fear—that someone else was already out there, digging his own straight tunnel—consumed him. Every construction person he talked to thought he was out of his mind, and yet he was sure the Alleghenies were crawling with people who shared his obsession. “When something becomes obvious to you,” he said, “you immediately think surely someone else is doing this.”
What never crossed his mind was that, once his line was fin- ished, Wall Street would not want to buy the line. Just the reverse: He assumed that the line would be the site of a gold rush. Maybe for that reason, he and his backers hadn’t thought much about how to sell the line until the time came to do it. It was compli- cated. What they were selling—speed—was only valuable to the extent that it was scarce. What they did not know was the degree of scarcity that would maximize the line’s market value. How much was it worth to a single player in the U.S. stock market to have an advantage in speed over everyone else? How much to twenty-five different players—to share the same advantage over the rest of the market? To answer these sorts of questions, it helps to know how much money traders can make purely from speed in the U.S. stock market, and how, exactly, they make it. “No one knew this market,” says Spivey. “It was opaque.”
They considered holding a Dutch auction—that is, start at some high reserve price and lower it until the line was bought by a single Wall Street firm, which would then enjoy a monopoly. They weren’t confident that any one bank or hedge fund would fork over the many billions of dollars they assumed the monopoly was worth, and they didn’t like the sound of the inevitable headlines in the newspapers: Barksdale Makes Billions Selling Out Ordinary American Investor. They hired an industry consultant named Larry Tabb, who had caught Jim Barksdale’s attention with a paper he’d written called “The Value of a Millisecond.” One way to price access to the line, Tabb thought, was to figure out how much money might be made from it, from the so-called spread trade between New York and Chicago—the simple arbitrage between cash and futures. Tabb estimated that if a single Wall Street bank were to exploit the countless minuscule discrepancies in price between Thing A in Chicago and Thing A in New York, they’d make profits of $20 billion a year. He further estimated that there were as many as four hundred firms then vying to capture the $20 billion. All of them would need to be on the fastest line between the two cities—and there were only places for two hundred of them on the line.
Both estimates happily coincided with Spivey’s sense of the market, and he took to saying, with obvious pleasure, “We have two hundred shovels for four hundred ditch diggers.” But what to charge for each shovel? “It was really a total wet finger in the air,” says Brennan Carley, who had worked closely with a lot of high-speed traders, and who had been hired by Spivey to sell his network to them. “All of us were just guessing.” The number they came up with was $300,000 a month, roughly ten times the price of the existing telecom lines. The first two hundred stock market players willing to pay in advance and sign a five-year lease would get a deal: $10.6 million for five years. The traders who leased Spread’s line would also need to buy and maintain their own signal amplifiers, housed in thirteen amp sites along Spread’s route. All-in, the up-front cost to each of the two hundred traders would come to about $14 million, or a grand total of $2.8 billion.
By early 2010 Spread Networks still hadn’t informed a single prospective customer of their existence. A year after the workers had started digging, the line was, incredibly, still a secret. To maximize the line’s shock value and minimize the chance that someone else would seek to replicate what they had done, or even announce their intention to do so, they decided to wait until March 2010, three months before the line was due to be completed, before they tried to sell it. How to approach the rich and powerful men whose businesses they were about to disrupt? “The general modus operandi was to find someone at one of these firms one of us knew,” says Brennan Carley. “We’d say, ‘You know me. You know of Jim Barksdale. We have something we want to come over and talk to you about. We can’t tell you what it is until we get there. And, by the way, we want you to sign an NDA [non-disclosure agreement] before we come in.’ ” That’s how they went to Wall Street—in stealth. “There were CEOs at every meeting,” says Spivey. The men with whom they met were among the most highly paid people in the financial markets. The first reaction of most of them was total disbelief. “People told me later that they thought, Surely not, but let’s talk to him anyway,” says Spivey. Anticipating their skepticism, he carried with him a map, four feet by eight feet. He finger- walked them through his cross-country tunnel. Even then people still demanded proof. You couldn’t actually see a fiber-optic line buried three feet under the ground, but the amp sites were highly visible thousand-square-foot concrete bunkers. Light fades as it travels; the fainter it becomes, the less capable it is of transmitting data. The signals transmitted from Chicago to New Jersey needed to be amplified every fifty to seventy-five miles, and for the amplifiers that did the work, Spread had built these maximum-security bunkers along the route. “I know you guys are straight shooters,” one trader said to them. “But I never heard of you before. I want to see a picture of this place.” Every day for the next three months, Spivey emailed this man a photograph of the most recent amp site under construction to show him that it was actually being built.
Once their disbelief faded, most of the Wall Street guys were just in awe. Of course they all still asked the usual questions. What do I get for my $14 million in assorted fees and expenses? (Two glass fibers, one for each direction.) What happens if the line’s cut by a backhoe? (We have people on the line who will have it up and running in eight hours.) Where is the backup if your line goes down? (Sorry, there isn’t one.) When can you supply us with the five years of audited financial statements that we require before we do business with any firm? (Um, in five years.) But even as they asked their questions and ticked their boxes, they failed to disguise their wonder. Spivey’s favorite meeting was with a trader who sat stone-faced listening to him for fifteen minutes on the other side of a long conference table, then leapt to his feet and shouted, “SHIT, THIS IS COOL!”
In these meetings what didn’t get said was often as interesting as what did. The financial markets were changing in ways even professionals did not fully understand. Their new ability to move at computer, rather than human, speed had given rise to a new class of Wall Street traders, engaged in new kinds of trading. People and firms no one had ever heard of were getting very rich very quickly without having to explain who they were or how they were making their money: These people were Spread Networks’ target audience. Spivey actually didn’t care to pry into their warring trading strategies. “We never wanted to come across as if we knew how they were making money on this,” he said. He didn’t ask, they didn’t say. But the response of many of them suggested that their entire commercial existence depended on being faster than the rest of the stock market— and that whatever they were doing wasn’t as simple as the age-old cash to futures arbitrage. Some of them, as Brennan Carley put it, “would sell their grandmothers for a microsecond.” (A microsecond is one millionth of a second.) Exactly why speed was so important to them was not clear; what was clear was that they felt threatened by this faster new line. “Somebody would say, ‘Wait a second,’ ” recalls Carley. “ ‘If we want to continue with the strategies we are currently running, we have to be on this line. We have no choice but to pay whatever you’re asking. And you’re going to go from my office to talk to all of my competitors.’ ”
“I’ll tell you my reaction to them,” says Darren Mulholland, a principal at a high-speed trading firm called Hudson River Trading. “It was, ‘Get out of my office.’ The thing I couldn’t believe was that when they came to my office they were going to go live in a month. And they didn’t even know who the clients were! They only discovered us from reading a letter we’d written to the SEC. . . .Who takes those kinds of business risks?” For $300,000 a month plus a few million more in up-front expenses, the people on Wall Street then making perhaps more money than people have ever made on Wall Street would enjoy the right to continue doing what they were already doing. “At that point they’d get kind of pissed off,” says Carley. After one sales meeting, David Barksdale turned to Spivey and said, Those people hate us. Oddly enough, Spivey loved these hostile encounters. “It was good to have twelve guys on the other side of the table, and they are all mad at you,” he said. “A dozen people told us only four guys would buy it, and they all bought it.” (Hudson River Trading bought the line.) Brennan Carley said, “We used to say, ‘We can’t take Dan to this meeting, because even if they have no choice, people do not want to do business with people they’re angry with.’ ”
When the salesmen from Spread Networks moved from the smaller, lesser-known Wall Street firms to the big banks, the view inside the post-crisis financial world became even more intriguing. Citigroup, weirdly, insisted that Spread reroute the line from the building next to the Nasdaq in Carteret to their offices in lower Manhattan, the twists and turns of which added several milliseconds and defeated the line’s entire purpose. The other banks all grasped the point of the line but were given pause by the contract Spread required them to sign. This contract prohibited anyone who leased the line from allowing others to use it. Any big bank that leased a place on the line could use it for its own proprietary trading but was forbidden from sharing it with its brokerage customers. To Spread this seemed an obvious restriction: The line was more valuable the fewer people that had access to it. The whole point of the line was to create inside the public markets a private space, accessible only to those willing to pay the tens of millions of dollars in entry fees. “Credit Suisse was outraged,” says a Spread employee who negotiated with the big Wall Street banks. “They said, ‘You’re enabling people to screw their customers.’ ” The employee tried to argue that this was not true—that it was more complicated than that— but in the end Credit Suisse refused to sign the contract. Morgan Stanley, on the other hand, came back to Spread and said, We need you to change the language. “We say, ‘But you’re okay with the restrictions?’ And they say, ‘Absolutely, this is totally about optics.’ We had to wordsmith it so they had plausible deniability.” Morgan Stanley wanted to be able to trade for itself in a way it could not trade for its customers; it just didn’t want to seem as if it wanted to. Of all the big Wall Street banks, Goldman Sachs was the easiest to deal with. “Goldman had no problem signing it,” the Spread employee said.
It was at just this moment—as the biggest Wall Street banks were leaping onto the line—that the line stopped in its tracks. There’d been challenges all along the route. After leaving Chicago they had tried and failed six times to tunnel 120 feet under the Calumet River. They were about to give up and find a slower way around when they stumbled upon a century-old tunnel that hadn’t been used in forty years. The first amp site after leaving Carteret was supposed to be near a mall in Alpha, New Jersey. The guy who owned the land said no. “He said he knew it was going to be some kind of terrorist target and he didn’t want it in the neighborhood,” said Spivey. “There’s always little gotchas out there that you have to be careful of.”
Pennsylvania had proved even more difficult than Spivey had imagined. Coming from the east, the line ran to a small forest in Sunbury, just off the east bank of the Susquehanna River, where it stopped and waited for its western twin. The line coming from the west needed to cross the Susquehanna. That stretch of river was breathtakingly wide. There was one drill in the world—it would cost them $2 million to rent—capable of boring a tunnel under the river. In June 2010, the drill was in Brazil. “We need a drill that is in Brazil,” says Spivey. “That idea is quite alarming. Obviously someone is using the drill. When do we get to use it?” At the last minute they overcame some objections from Pennsylvania bridge authorities and were permitted to cross the river on the bridge—by boring holes through its concrete pylons and running the cable on the underside of the bridge.
At which point the technical problems gave way to social problems. Leaving the bridge, the road split; one branch went north; the other, south. If you attempted to travel due east, you hit a dead end. The road just stopped, near a sign beside a levee that said, Welcome to Sunbury. Blocking the line’s path were two big parking lots. One belonged to a company that manufactured wire rope, the cable used on ski lifts; the other was owned by a century-old grocery store named Weis Markets. To reach its twin in the Sunbury forest, the line needed to pass through one of these parking lots or travel around the entire city. The owners of both Weis Markets and the Wirerope Works were hostile or suspicious, or both; they weren’t returning calls. “The whole state has been abused by coal companies,” Steve Williams explained. “When you say you want to dig, everyone gets suspicious.”
Going around rather than through the town, Spivey calculated, would cost several months and a lot of money and would add four microseconds to his route. It would also prevent Spread Networks from delivering the cable on time to the Wall Street banks and traders ready to write checks for $10.6 million for it. But the guy who ran the wire rope factory was for some reason so angry with Spread’s local contractor that he wouldn’t speak to them. The guy who ran the Weis Markets was even harder to reach. His secretary told Spread that he was at a golf tournament, and unavailable. He’d already decided—without informing Spread Networks—to reject the somewhat strange offer of low six figures plus free high-speed Internet access they had offered him in exchange for a ten-foot easement under his parking lot. The line passed too close to his ice cream–making plant. The chairman had no interest in signing over a permanent easement that would make it difficult to expand the ice cream plant.
In July 2010 the line dropped back underground beneath the bridge in Sunbury and just stopped. “We had all this fiber out there and we needed it to talk to each other and it couldn’t,” said Spivey. Then, for some reason he never fully understood, the wire rope people softened. They sold him the easement he needed. The day after Spread Networks acquired lifetime rights to a ten-foot-wide path under the wire rope factory’s parking lot, it sent out its first press release: “Round-trip travel time from Chicago to New Jersey has been cut to 13 milliseconds.” They’d set a goal of coming in at under 840 miles and beaten it; the line was 827 miles long. “It was the biggest what-the-fuck moment the industry had had in some time,” said Spivey.
Even then, none of the line’s creators knew for sure how the line would be used. The biggest question about the line— Why?—remained imperfectly explored. All its creators knew was that the Wall Street people who wanted it wanted it very badly—and also wanted to find ways for others not to have it. In one of his first meetings with a big Wall Street firm, Spivey had told the firm’s boss the price of his line: $10.6 million plus costs if he paid up front, $20 million or so if he paid in installments. The boss said he’d like to go away and think about it. He returned with a single question: “Can you double the price?”