With a spacecraft in trouble and the White House watching, SpaceX had to deliver

Within minutes of liftoff, it was clear the Dragon spacecraft was in trouble.

Inside mission control on the morning of March 1, 2013, the SpaceX team was desperately trying to figure out what went wrong and soon pinpointed the problem: A few valves were stuck.

Lori Garver, NASA’s deputy administrator, was beside herself. The Obama administration had placed a bold bet on Elon Musk’s SpaceX, awarding it hundreds of millions of dollars on contracts to fly cargo to the International Space Station, despite the critics who said it was foolish to trust a private outfit with such a complicated endeavor.

This was a fundamental shift for NASA, a move that some in the agency’s highest reaches were wary of, and a risky bet by the White House. Under President Obama, NASA had retired the space shuttle and hired contractors — SpaceX and Boeing — to fly missions to the International Space Station as if they were providing a taxi service to space. That, in turn, would allow NASA to focus on missions in deep space and recapture some of the glory that had faded in the decades since the Apollo era put 12 men on the moon.

In a 2010 speech at the Kennedy Space Center, Obama acknowledged the risks of the proposal. “Now, I recognize that some have said it is unfeasible or unwise to work with the private sector in this way,” he said. But the U.S. space program needed a kick-start, he said: “We will also accelerate the pace of innovations as companies — from young start-ups to established leaders — compete to design and build and launch new means of carrying people and materials out of our atmosphere.”

Musk had become the face of this new policy, the brash Silicon Valley tech billionaire who founded SpaceX in 2002 with the goal of colonizing Mars. At first, the company had struggled, and it nearly went out of business after three failed attempts to reach orbit. But after a successful launch in 2008, SpaceX won a $1.6 billion NASA cargo contract, prompting Musk to change his log-in password to “ILoveNASA.”

Now, he had to show that his Hawthorne, Calif., upstart could deliver. And with Dragon struggling in 2013, he was starting to sweat.

Obama’s shift

Musk wasn’t the only one with a lot to lose. If the spacecraft didn’t dock with the station, if the mission somehow failed, Garver feared the critics would again blast Obama’s decision.

Many were already upset with the White House for canceling the President George W. Bush-era Constellation program — a plan to return to the moon using big rockets and spacecraft built by the traditional industrial base led by Boeing and Lockheed Martin, a plan that SpaceX was fighting to disrupt.

Sen. Richard Shelby, R-Ala., a powerful member of the Appropriations Committee, said at the time that Obama’s plan “begins the death march for the future of U.S. human spaceflight,” and that Obama was turning NASA into the agency of “pipe dreams and fairy tales.”

Obama’s shift also drew criticism from Michael Griffin, a former NASA administrator.

“One day it will be like commercial airline travel, just not yet,” he said of space flight. “It’s like 1920. Lindbergh hasn’t flown the Atlantic, and they’re trying to sell 747s to Pan Am.”

To assuage concerns, the White House decided Obama would visit the United Launch Alliance, the joint venture between Lockheed Martin and Boeing. The message was clear: Although the president canceled one of their major programs, the contractors were still a vital part of the U.S. space program. His presence there would be an endorsement and a signal to Congress.

But there was a problem: The Alliance was about to launch a highly classified space plane known as the X-37B that would ultimately stay in orbit for months at a time. But doing what? The Pentagon wouldn’t say. The program was secret, which was why the president couldn’t just swing by for a photo op. The National Security Council wouldn’t hear of it.

So the White House scrambled. Instead, the president would visit SpaceX, a development that the company welcomed. A presidential visit would represent a public relations triumph over its archrival, even if it was, as Musk said later, “a sheer accident.”

An accident that had raised the stakes even further — for Musk and the White House. This was SpaceX’s second official cargo delivery flight to the space station. It had to work, thought Garver, the NASA deputy administrator. They had to find a way to rescue Dragon, and fast.

But as they tried to figure out what was wrong, Steve Davis, SpaceX’s director of advanced projects, had begun to prepare for the worst — aborting the mission.

“Is the vehicle even functioning enough that you can bring it back?” he wondered. “We weren’t sure. That was the only time we had ever planned for an emergency reentry, which is like a big thing because you have to whip it through airspace. You have to reroute planes in real time. It’s not awesome. And so we were in panic mode.”

‘What if we just cut the skirt?’

SpaceX had been in panic mode before. In late 2010, on the eve of the Falcon 9’s second launch and the first test flight of the Dragon spacecraft, a last-minute inspection of the rocket revealed a crack in the nozzle, or skirt, of the second-stage engine.

“You’re not going to fly with a crack,” Davis said. “We’re like, ‘What do we do?’”

The normal thing would be to take the rocket apart, replace the engine skirt, reinspect it. And then “you’re up and launching in a month,” he said. No one wanted to lose that much time.

Musk had a wild idea: “What if we just cut the skirt? Like, literally cut around it?” That is, what if they trimmed off the bottom as if it were a fingernail?

“He went person by person and said, ‘Would this have any adverse effect on you?’” Davis recalled.

Davis said that because the skirt would be shorter, they would get less performance from the engine. “But we had so much margin built into it, it didn’t matter,” he said. Everyone concurred, and “literally within 30 minutes, the decision was made.”

The company flew a technician from California to Cape Canaveral; armed with a pair of shears, like the kind used to trim hedges, he cut around the crack.

“And we flew the next day successfully,” Davis said. “That could have been the dumbest thing we ever did, but it was amazing.”

That was not how NASA would have handled it. But its officials agreed that there wasn’t any reason it wouldn’t work and approved the launch, astounded by how quickly SpaceX was addressing the problem.

That sort of go-for-it ethos had become a SpaceX trademark. Gwynne Shotwell, the company’s president, described the culture this way: “Head down. Plow through the line.”

“We had to be super scrappy,” Musk said. “If we did it the standard way, we would have run out of money. For many years, we were week to week on cash flow, within weeks of running out of money. It definitely creates a mind-set of smart spending. Be scrappy or die: Those were our two options. Buy scrap components, fix them up, make them work.”

So when the company was rebuilding launchpad 40 at the Cape Canaveral Air Force Station, one of SpaceX’s employees spotted a 125,000-gallon liquid nitrogen tank and thought, “Maybe we could use this?”

Despite sitting outside for years, the tank seemed in decent shape, and the SpaceX’s 10-member team on the Cape wanted it. They called the Air Force asking permission, but their calls went unreturned. They persisted and were put in touch with a company that had been hired to haul away the tank and destroy it.

The company was willing to part with the tank for $1 more than the cost of scrapping it — $86,000. So the members of the SpaceX team became the scavengers of Cape Canaveral, looking for leftover hardware as if they were on a treasure hunt. Old rail cars from the 1960s that had been used to ferry helium between New Orleans and Cape Canaveral became the new storage tanks. Instead of spending $75,000 on new air-conditioning chillers for the ground equipment building, someone found a deal on EBay for $10,000.

Cost drove lots of decisions, even how the company built its rockets. Once Musk got wind that the air-conditioning system used to keep the satellite cool in the rocket’s fairing, or nose cone, was going to cost more than $3 million, he confronted the designer about it.

“What’s the volume in the fairing?” he wanted to know. The answer was a couple thousand square feet, less than the size of a house.

He turned to Shotwell and asked her how much a new air-conditioning system for a house cost.

“We just changed our air conditioning,” she replied. “It was six thousand bucks.”

“Why is this $3 or $4 million when your air-conditioning system is $6,000?” he asked. “Go back and figure this out.”

The company did, buying six commercial A/C units with bigger pumps that could handle a larger airflow.

‘The Heimlich maneuver’

Now, as Dragon was in trouble with the stuck valves, SpaceX had to figure out on the fly how to make it work.

As the SpaceX team scrambled, Bill Gerstenmaier, NASA’s associate administrator for human exploration and operations, and Michael Suffredini, the space station’s program office manager, were in the room watching.

They were two of the agency’s most senior officials, with nearly 60 years at NASA between them. They had served through the Challenger and Columbia disasters, had seen all sorts of problems in space, and now, as NASA faced another potential crisis, they were talking softly between themselves.

Fearing the political fallout of a failed mission, Garver, the deputy NASA administrator, wanted them to take over, to swoop in and save SpaceX.

There were no better people to come fix this. But the two NASA elder statesmen just watched, offering a bit of advice — a whisper here, a suggestion there — to the SpaceX crew. Mostly, they stayed out of the way.

“They were like grandparents,” Garver recalled. “And it was almost like grandpa taking them fishing: ‘Try over there. There might be some fish over there.’” A soft touch designed to let the kids learn to fish on their own, rather than an impatient dad grabbing the pole and catching the fish for them.

“If there was something we saw that we could have interjected, we would have done it,” Gerstenmaier recalled. But it wasn’t NASA’s spacecraft.

“We really were in an advisory role,” Suffredini said.

As they watched, the people in the control room worked the problem. The valves were stuck, so they’d need something to make it unstuck. On a spacecraft circling the globe at 17,500 mph, that was no easy task. But the SpaceX team knew that if pressure could be built ahead of the valves and then suddenly released, it might just deliver the kick needed to jar them open.

“It’s like the spacecraft equivalent of the Heimlich maneuver,” Musk said later.

One of the engineers typed up a command, right then, on the fly, programming the spacecraft to build up the pressure. Then, they tried to beam the new command up to the Dragon, as if it were an iPhone update. At that moment, the NASA elders knew they were witnessing something special. It wasn’t that they had fixed a problem with the spacecraft; that happened all the time. It was how fast they did it.

“The SpaceX mind-set had always been about adapting quickly, and it really shined that day,” Suffredini said. “They had really an in-depth understanding of that system and the software, and that’s one of the secrets of their success. They probably had the kid in there who wrote the original code.”

But the SpaceX crew was having a hard time communicating with the spacecraft. The code wouldn’t transmit. So someone got the Air Force on the phone, which gave the company access to a more powerful satellite dish, which allowed, at last, the uplink.

The code worked. The valves opened. The mission was a success.

This article is adapted from the forthcoming book “The Space Barons: Elon Musk, Jeff Bezos, and the Quest to Colonize the Cosmos” by Christian Davenport.