String of Missteps Doomed Orbiter; JPL Found at Fault

Inattention, miscommunication and overconfidence at the Jet Propulsion Laboratory were among the management problems that added up to a fatal error in arithmetic and doomed the $125-million Mars Climate Orbiter earlier this year, a NASA accident review board concluded Wednesday.

The investigation also turned up potentially fatal problems with its sister spacecraft, the Mars Polar Lander, that JPL engineers in Pasadena are now scrambling to fix as the robotic craft approaches Mars for a landing attempt early next month.

“No one wants another mistake to go undetected,” said JPL Director Edward Stone. “We are doubling and redoubling our efforts.”

In its formal report made public Wednesday, the NASA panel confirmed that JPL and its subcontractors confused English and metric units in a series of miscalculations that led to the orbiter’s disappearance Sept. 23 as it began to circle Mars.

Whatever its narrow technical conclusion, however, the board called attention to a series of major management missteps at JPL and its leading industrial partner in space--Lockheed Martin Astronautics--that contributed to the loss of the orbiter.

Among a litany of problems, the board noted that the navigation team was not familiar with the orbiter. The spacecraft team was equally ignorant of the navigation problems. Crucial computer files were corrupted for months or never properly tested, the report said.

Reaction to the findings was harsh.

“These people could not manage their way out of a Glad bag,” said John Pike, space policy director for the Federation of American Scientists, a nonprofit group that monitors science and technology. “This is far worse than I thought, in terms of generally slipshod, inattentive management. It is sobering and sad.”

Those same management deficiencies are also the subject of a separate audit by the NASA inspector general. That report criticized JPL for not paying enough attention to the problems at Lockheed and its other contractors in the space agency’s ambitious Mars ’98 exploration program, which included both the orbiter and the Mars Lander probes.

Launching five spacecraft in six months and then attempting to manage several of them simultaneously, JPL managers failed to do enough to control costs or to ensure that the spacecraft were properly built by subcontractors, the inspector general’s audit concluded.

Indeed, the contractor’s cost of the Mars Orbiter and Lander project soared 44% to $121.3 million, the inspector general reported, even as Lockheed failed to properly staff the project or deliver properly functioning hardware. The overall mission stayed within the budget constraints of NASA’s “better, cheaper, faster” mandate, JPL officials said, but only by using up contingency funds that could have been devoted to other tasks.

Lockheed--the world’s largest and perhaps most troubled defense contractor--was also involved in several other major spaceflight failures this year that stemmed from an overemphasis on cost-cutting, mismanagement and poor quality control.

Ineffective Designs, Poor Workmanship

In a recent review of five JPL projects, the NASA auditors were especially troubled by a staffing pattern of too few people at the onset of a project when key engineering decisions are being made and too many later on to catch up with slipping schedules. This contributed to poor workmanship and ineffective engineering designs, they said.

To make up for its initial understaffing of the Mars ’98 project, Lockheed’s staffing for the last two-thirds of the project was 80% more than planned, the auditors found. To make up for lost time, engineers and technicians were required to work 70 hours a week or more.

“Rushing to meet milestones could . . . result in engineering decisions being made based on schedule need instead of the need to reduce risk of failure,” NASA auditors had warned JPL in their report, delivered just a few days before the orbiter vanished in space.

With many engineers working on several projects at once, such seesaw staffing also could have a ripple effect throughout the unmanned space exploration program, several space analysts said.

The project problems were so serious that even before the Mars Orbiter disappeared, JPL had moved to slash Lockheed’s performance award from $12.5 million to $3.5 million. After the Mars Lander portion of the mission is completed next year, JPL may move to cut that award even more in response to the orbiter loss.

Even so, the auditors concluded, JPL managers too often let contractors police themselves.

Noel Hinners, vice president for flight systems at Lockheed Martin’s space systems group, said Wednesday that the company was responsible for assuring that critical measures of the Mars Orbiter’s performance were delivered to JPL in metric units.

But Lockheed engineers prepared computer files in English units instead because they forgot to check the technical specifications, he said.

“To be very blunt, it was overlooked,” Hinners said.

In its investigation of the Mars Orbiter mishap, the accident review board also discovered that:

* There was a widespread perception among Mars Climate Orbiter team that “orbiting Mars is routine,” which caused the team to not pay enough attention to the risks of interplanetary spaceflight.

* There was so little communication between navigators and spacecraft designers that the navigation team did not understand “essential spacecraft characteristics” and the spacecraft team did not understand “the navigation challenge.” Key people often failed to attend mission reviews. At least one design decision eliminated a spacecraft function that could have saved the craft by eliminating the need for constant momentum adjustments that went awry when the wrong measurements were used in calculations.

* The operations navigation personnel were so unfamiliar with the Mars Orbiter spacecraft that they assumed it would act the same way as the Mars Global Surveyor craft, which was based on an entirely different design. The team was “isolated,” understaffed and inadequately trained.

* Crucial mission computer files were delivered to JPL with file formatting errors and corrupted data, which then went uncorrected for four months. Other computer software was not adequately tested or verified. Overall, the glitches made it harder to detect the metric miscalculation.

* A contingency for a last-minute engine firing, which might have saved the orbiter, was built into the flight plan but the engineering details were never worked out, so that when the navigators asked for the maneuver to be performed in a last-ditch effort to salvage the mission, they were turned down.

“There is still confusion,” the board said, about why the crucial contingency maneuver was not performed or whether it was necessary.

Although several senior JPL officials were permitted to discuss the board’s findings publicly, JPL would not allow any member of the Mars Climate Orbiter team to answer questions about their reactions to the board’s conclusions.

The board also recommended eight actions to ensure that the Polar Lander settles down safely at the Martian south pole in December, including independent navigation measurements to ensure accurate tracking and a compete review of all the landing procedures and computer software. With additional engineers and navigators pressed into service, JPL is attempting to solve the problems in the next three weeks.

The most serious concern involves the lander’s hydrazine-fueled descent engines.

In 35 years of planetary exploration, no one has ever used the kind of soft landing technique to be employed by the Mars Lander next month. All 12 thrusters must fire perfectly in unison to set the lander down properly.

But chilled for 11 months in space, the fuel could freeze and the thrusters could misfire, stall or explode when turned on.

In the second thoughts prompted by the orbiter loss, JPL engineers decided to heat the descent engines and the propellent system up to 46.4 degrees Fahrenheit several hours before igniting them.

The report released Wednesday was prepared by nine NASA experts led by Arthur G. Stephenson, who is director of NASA’s George C. Marshall Space Flight Center in Huntsville, Ala. JPL itself mounted two additional internal review teams.

Despite its unflinching assessment of the orbiter technical problems, the report steered clear of any broader issues of the space agency’s space exploration philosophy of “faster, better, cheaper,” and the stringent budget constraints.

Under any circumstances, spaceflight is the most unforgiving engineering environment. It is a matter of margins, experts said--of budget, schedule, and engineering tolerances. Misdirected sensors, broken antennas, engine misfirings and temporary computer glitches are the routine emergencies of spaceflight, often overcome just as routinely by a combination of engineering inspiration and experience.

Some experts worry those margins have been shaved too fine in the effort to meet all of the nation’s ambitious space exploration goals at minimal cost.

“As devastating as the report is, it raises as many questions as it answered,” said Pike. “All of these problems are symptomatic of too much work and not enough people. They are all symptomatic of an underfunded program.”

(BEGIN TEXT OF INFOBOX / INFOGRAPHIC)

In Need of a Safe Landing

*

Engineers at the Jet Propulsion Laboratory are scrambling to ensure that the Mars Polar Lander sets down safely next month on the Red Planet. A NASA accident review board detailed what went wrong with the Mars Climate Orbiter in September. Here are some of the board’s conclusions:

* JPL officials were overconfident, with a widespread perception that “orbiting Mars is routine.”

* Navigators did not understand “essential spacecraft characteristics.”

* The spacecraft team did not understand “the navigation challenge.”

* Crucial mission computer files were delivered to JPL with file formatting errors and corrupted data.

* Other computer software was not adequately tested or verified.

* A last-minute engine firing, which might have saved the orbiter, was built into the flight plan but the engineering details were never worked out.

* Training was inadequate. Crucial information never reached the navigation team.

Source: NASA, Caltech, JPL/Los Angeles Times