FAA Software Flaw Spotlights Malady of Digital Age
A lone air traffic controller, sitting in the darkened confines of the Palmdale Air Route Traffic Control Center, innocently shut down much of the air traffic across the Southwest last week by merely typing in a few too many characters on his computer.
The glitch occurred because of an odd combination of a major flaw in the Federal Aviation Administration’s newly upgraded computer software and an antiquated flight information system used by Mexico.
But the computer snafu spotlights a Digital Age malady familiar to many home and business computer users: A computer done in by faulty software code. Although electronic devices from telephone switches to desktop PCs are at the mercy of software written by engineers, in aviation it can be a life-and-death matter.
The shutdown ultimately grounded or delayed more than 2,000 flights at Los Angeles International Airport, plus hundreds more in Northern California, Nevada, Arizona and Utah, and the ripple effect delayed other flights across the country.
“Software glitches come with the territory as automation becomes very complex,” said Lee A. Hollaar, a professor in the School of Computing at the University of Utah, who has conducted research on hardware and software trade-offs in computer system designs.
Hollaar faulted the FAA for not testing its new software for a longer period before putting it online. He also said the agency should give more frequent training to controllers to refresh their skills both on the upgraded computer and an older system that was used for backup after the upgraded system failed.
“If you never have to go back to the backup system, you forget how to use it,” Hollaar said.
Although FAA officials were quick to absolve the air traffic controller of any fault, some aviation specialists say the agency could improve training and other procedures related to the software upgrades.
“The people [using] the software just didn’t seem to have the right training, and they are supposed to be the experts” guiding the flights through the skies, said Ron Rahrig, vice president of the Southwest region of the Professional Airways Systems Specialists. The Washington, D.C.-based aviation labor union represents computer systems specialists and safety inspectors.
The problem at Palmdale--one of the nation’s busiest traffic control centers--stemmed from a previously unknown bad instruction among the 250,000 lines of computer code in the multimillion-dollar FAA software upgrade program.
About 100 engineers, working in Atlantic City, N.J., created the program, which is part of a massive nationwide effort to upgrade the nation’s aviation facilities. The software--an update of the FAA’s air traffic control program--has been installed at 17 of the 21 air traffic control facilities and used without incident until last week.
Because the software is customized for each air hub, FAA officials suspect the bad instructions exist in the portion of the software customized for Palmdale.
The FAA installed its new software in an IBM G3 computer in Palmdale between midnight and 3 a.m. on Oct. 19.
About 6 a.m. that day, an unidentified air traffic controller took a routine call from Mexican authorities passing along routing data for flights en route from that country.
The air traffic controller rotated the track ball on his keyboard to align his cursor to type in nine characters of flight information, which turned out to be four more characters than the flight center’s upgraded software could handle. The software glitch made the entire system vulnerable to a crash simply because of the entry of the extra characters.
The routing codes represent the flight number, air speed and elevation of the plane--this data appears next to a blinking dot representing the flight on a controller’s radar screen. Normally, the flight number is electronically transmitted between computers handing off flights from one traffic control center to the next.
But Mexico does not use an electronic routing system. Instead, Mexican authorities phone in the data to U.S. flight controllers, who enter it manually.
Because the airspace in Southern California is so heavily congested, the Palmdale controller had to enter a few extra characters to distinguish the Mexican flights, whose U.S. destination was Oakland, officials say.
Later, when some Mexican flights entered U.S. airspace and were handed off from Palmdale to the Fremont air traffic facility, these computer entries triggered an error. That caused the shutdown of Palmdale’s computer system for about four hours, forcing air traffic controllers to direct traffic with an backup system with much less automation.
“The software couldn’t understand any message that was longer than five characters,” said Gregg Dvorak, director of operational support for the FAA unit that created the software. “The controller didn’t do anything wrong. The software coding was incorrect.”
The Palmdale facility has since reverted to its old flight-routing software. Meanwhile, the FAA has imposed a moratorium on the installation of the new software, which may have also been responsible for a computer failure at the Fremont air hub this week.
In that incident, scores of flights over Northern California and Nevada were delayed Monday after computer engineers reinstalled software in a computer at the FAA center in Fremont. FAA inspectors have not yet determined whether the problem was caused by a hardware problem, a software glitch or operator error.
The FAA’s Dvorak said the faulty Palmdale software had been tested for about 3,000 hours on other computers. But, Dvorak said, it “is impossible to test for every possible event.”
Dvorak also said air traffic controllers are supposed to get training on the backup software system at least once a week. But Rahrig and others said controllers working the night shift often didn’t get training because the review sessions were held during the day.
