Fail-Safe Fails: FAA Looks for Answers

When radar screens suddenly went dark at Palmdale’s regional air traffic center on Tuesday, controller Bruce Bates and his colleagues knew instinctively what to do: They grabbed their cellphones and started calling for help.

The simple solution to a high-tech problem played out as pilots flying at high altitudes over Southern California and much of Nevada and Arizona tried in vain to reach controllers in the Los Angeles Air Route Traffic Control Center, where even the radios were dead.

The trouble had begun an hour before, when a traffic accident downed power lines to the facility, triggering backup generators.

But those units were soon also knocked out, federal officials said Wednesday, by a system of surge protectors designed to avoid just such a crisis.

“The anxiety level definitely shot through the roof,” Bates said, describing the scene. “We have seen glitches like a pop in the system, but the system keeps functioning. When it didn’t come back on after about 10 to 15 seconds, all of us knew there was a major problem.”

The phone calls for help went to controllers at other centers across the West, and they ultimately guided pilots to their destinations.

Federal officials said there were no close calls between aircraft as a result of the two-hour power outage at the Palmdale facility Tuesday evening. But controllers said it could take weeks to analyze radar data to ensure that safety wasn’t compromised by planes flying too close to one another.

The center, which directs commercial jets flying at high altitudes, initially lost power after a truck hit a nearby utility pole around 4:19 p.m. Backup generators kicked on immediately. But about 75 minutes later, a system designed to protect sensitive equipment failed, causing radar to go dark.

The outage snarled air traffic across the country, causing hundreds of flights to be held on the ground in other cities while controllers in Oakland, Salt Lake City and Albuquerque diverted flights already in the air away from the Los Angeles Basin.

On Wednesday, local officials demanded to know why backup systems weren’t more robust. An aide to Los Angeles County Supervisor Mike Antonovich, who represents Palmdale, said the supervisor asked Southern California Edison to provide a dedicated power line to the regional facility to prevent similar incidents.

“We want to ensure that a traffic accident in the Antelope Valley can’t knock out air traffic control for the entire western United States,” said Antonovich spokesman Tony Bell.

Controllers lost radio and radar communication with pilots after the failure of several switches that are intended to keep power spikes from damaging air traffic control systems. Technicians attributed the problem to an order from the Federal Aviation Administration two years ago that the system be reduced to save money.

“They did that in order to save money on the electrical bill,” said Ray Baggett, Western region vice president for Professional Airways Systems Specialists, the union that represents FAA technicians. “They are doing everything they can to reduce redundancy and back up systems. They are decreasing their safety margin ... to achieve cost efficiency.”

FAA officials denied that cost-cutting played any role in the failure, saying that their initial conclusions are that the system went down because of the way it was designed.

“We are absolutely not reducing redundancies to improve efficiency, that’s not a policy,” said Laura Brown, an FAA spokeswoman.

Fifteen minutes after the backup generators went down, the radio systems at the Palmdale center came back on, allowing controllers to talk with pilots, but the radarscopes remained down until about 7:30 p.m.

This was the second time in less than two years that troubles at the Palmdale center, which directs high-altitude traffic over an area of about 178,000 square miles, disrupted air service nationwide.

In September 2004, the facility lost radio communications for nearly three hours after a technician failed to perform required maintenance. Backup generators had also been configured incorrectly, which caused them to fail.

The shutdown led to at least five instances in which planes flew too close to one another.

Tuesday’s breakdown may have implications for air traffic control systems throughout the country, because the backup power system that failed is also used in other control centers that handle both high-altitude flights and planes that are approaching and leaving major airports.

“We are going to have to do some research on the failure and how to prevent it” in other control centers,” said Richard Riggs, a systems expert with the FAA technicians’ union.

The system that failed, known by the acronym ACEPS, was installed by the FAA during the 1990s as part of a major modernization effort designed to make the air traffic control system more reliable.

It is also in operation at the Southern California Terminal Radar Approach Control center in San Diego, which handles thousands of flights approaching and departing from area airports every day, according to a Transportation Department document.

ACEPS, which stands for Air Route Traffic Control Center Critical and Essential Power Systems, is supposed to keep electricity flowing to critical equipment in the event of a power failure. It relies on several generators, backed up by batteries that provide transitional power immediately after a loss of commercial electricity.

The system includes several power conditioning units -- the ones that triggered the failure in Palmdale -- that act as surge protectors to prevent damage to equipment during a power switch-over.

Technicians said four of these switches were in place two years ago, when the FAA asked that one be removed. On Tuesday, one of the three remaining switches failed, overloading the surge protector and blacking out power to the entire facility. Had the original system been in place, technicians said, Tuesday’s outage would not have occurred.

The FAA disagreed.

“Whether we have three or four or 10 of these things is not the problem; the problem we’ve had with these things is that they take each other down like dominoes,” said Brown, adding the agency could not find any documentation showing that it asked technicians to alter the power surge system.

Los Angeles airport officials called on the FAA to provide details about its backup systems and contingency plans. The airlines, the FAA and airport officials plan to meet Tuesday to discuss the outage.

Southern California airports bore the brunt of the delays Tuesday, when airlines canceled 35 outbound flights at Los Angeles International Airport and controllers diverted 67 incoming aircraft. About 25,000 passengers at LAX were affected.

“We’re very distressed that tens of thousands of travelers were significantly inconvenienced by this event,” said Paul Haney, deputy executive director of airports and security for the city agency that operates LAX. “The airports and the airline industry incurred significant costs to deal with the outage.”

Lufthansa Airlines lost $150,000 on one flight alone when it had to pay for hundreds of passengers to stay overnight after they left LAX two hours late and missed connections in Frankfurt.

Nationwide, 348 flights were delayed and airlines canceled 49 others. Controllers diverted 102 aircraft.

Carriers were still scrambling to accommodate travelers on other flights Wednesday, largely because most planes are already full.

Passenger Kevin Rooney of Inyokern, Calif., said he had already endured a sweltering two-hour layover in Philadelphia -- in a terminal with broken air conditioning -- on his way home from a trip to Manchester, N.H., when his flight to Las Vegas was suddenly rerouted to Salt Lake City.

“Was I inconvenienced? Yes, to the extent that it made a long drive home seem even longer. But, I drove back through Death Valley and the later hour meant I saw exactly one other vehicle.”

Times staff writer Sharon Bernstein contributed to this report. Oldham reported from Los Angeles and Alonso-Zaldivar from Washington.

*

(BEGIN TEXT OF INFOBOX)

Out of control

A vehicle accident in the Antelope Valley on Tuesday created a ripple effect, disrupting air traffic across the country.

Q: Why did the air route traffic control center go dark?

A: The trouble started when a vehicle hit a power pole near the Palmdale center. That caused the main power to fail, and the backup generator system immediately fired up. Officials believe that a surge protection system, designed to ensure sensitive air traffic equipment isn’t harmed by power spikes, failed. That caused the backup system to go down.

Q: How widespread was the breakdown?

A: Nationwide, 348 flights were delayed and airlines canceled 49 flights. Controllers diverted 102 aircraft. About 25,000 passengers were affected at LAX alone.

Q: Did the breakdown create dangerous conditions for planes in the air?

Federal officials said that there were no close calls between aircraft during the blackout. But some controllers said more extensive studies must be done before concluding safety wasn’t compromised.

Q: What’s the next step?

The airlines, the Federal Aviation Administration and local airport officials will meet Tuesday at LAX to discuss what caused the outage. Airport officials have also asked the FAA to provide details about its backup systems and contingency plans.

Source: Times reporting

Los Angeles Times