A mistake on a single bundle of wires Monday cascaded into a major blackout in and around Los Angeles, inconveniencing millions of people and renewing questions about the vulnerability of the region’s power system.
Coming one day after a purported Al Qaeda threat of attack on the city, the midday outage pricked nerves and caused isolated incidents of panic. Plumes of flame and smoke heightened the drama as refineries, temporarily shut by the outage, flared off excess gases.
But backup generators, many newly installed since California’s 2001 energy crisis, kept many companies and most emergency services operating without major disruption, and there were no reports of deaths or serious injuries caused by the blackout.
The Los Angeles Department of Water and Power, the city-owned utility, said the outage occurred when workers cut through wires while installing a monitoring system at an electrical transmitting station in Toluca Lake.
The mistake rippled through the electrical grid, threatening to overload another transmission station and two electrical generating plants: the Scattergood generating station south of Los Angeles International Airport and the Haynes generating station near Long Beach.
The DWP shut down the generating facilities to avoid damage, sharply reducing the amount of power available to the city. That caused blackouts in neighborhoods across the city, with heavy concentrations in parts of the San Fernando Valley, South Los Angeles and the downtown area. All of Burbank’s 52,000 customers and half of Glendale’s 80,000 also lost power. Both cities’ electrical systems are linked to the DWP.
“This strikes me as something under the category of unbelievably bad luck, where you cut one line and have that kind of cascading effect,” said Bob Finkelstein, executive director of the Utility Reform Network in San Francisco, a consumer advocacy group.
“One DWP worker is going to feel really, really bad for a long time.”
The automated system workers were installing was meant to detect surges or drops in voltage, said Ed Miller, the DWP’s director of power systems, operations and maintenance.
“They cut a bundle of wires,” Miller said. “The supposition is that by cutting them together, they created a short that triggered the circuit breakers.” Miller said cutting wires one by one might have avoided an electrical short.
Ironically, the system the work crew was trying to install would have identified the power problem much more quickly, Miller said. It would be able to “decipher just what had happened.” As it happened, DWP engineers needed an hour to determine where in the system the outage began.
An official with the union that represents DWP workers charged that, regardless of whether lines were cut, the real cause of the outage was the DWP’s decision to test the new relay system in the middle of the day, rather than at night when power loads would have been lower -- but costs would have been higher because of overtime.
“They are too cheap to pay overtime, so they are testing on a full load at 1 in the afternoon,” said Brian D’Arcy, business manager for the International Brotherhood of Electrical Workers, Local 18.
“You clearly don’t test the relay at one in the afternoon. If you are cheap and stupid, this is what happens.”
The DWP said such work is routinely conducted during the day.
DWP officials insisted the system had worked as it is supposed to -- shutting down and causing a temporary inconvenience to avoid widespread permanent damage. Miller compared the shutdown to the way a circuit breaker turns off power to a house to prevent overheated lines from causing a deadly fire.
At the same time, however, Miller said he planned to investigate whether the ability of a single incident to black out half the power supply indicated a need for major changes in the way the city’s electrical system is configured. “If I decide I need a differently configured system, that could take a couple of years,” he said.
“I’m very proud of my people. There was no system damage, and nobody was hurt,” said Miller.
“The system is not fragile,” he added. “We have a very strong system.”
Others were less sanguine.
“As an engineer, it’s unnerving that one individual act can cut power to hundreds of thousands of customers,” said Los Angeles City Councilman Tony Cardenas, chairman of a panel overseeing the DWP.
Cardenas, who was stuck in traffic on Van Nuys Boulevard when traffic signals went out, said he would demand to know whether the outage betrayed a broader vulnerability for the city’s electrical system.
“I’ve been told over and over that this kind of vulnerability doesn’t exist,” Cardenas said.
Mary Nichols, who heads UCLA’s Institute of the Environment and was appointed last month as a DWP commissioner, said she, too, had “questions and concerns” about “why the system isn’t more resilient and why there isn’t more redundancy built into it.”
And a spokeswoman for Cal-ISO, the independent organization that oversees the statewide electrical grid, said the incident was “disconcerting.” Stephanie McCorkle criticized officials of the DWP, which is independent of Cal-ISO’s system, for not being more forthcoming with information about the outage, saying the DWP did not officially alert the rest of the statewide grid until 3:45 p.m., more than three hours after the outage began.
“There was no coordination, and that concerns us,” she said.
Although unseasonably cool weather makes the electrical grid less susceptible to widespread failures because of decreased energy demands, McCorkle said she was not sure whether hot weather could have caused the DWP outage to spread.
The outage occurred at 12:37 p.m., cutting power to 750,000 homes and businesses, or about 2 million people, DWP officials said. It struck about half of the DWP’s service area. Other parts of the region, most of which is served by Southern California Edison, were not affected.
Power was restored to 90% of the downed customers within 90 minutes, DWP officials said.
Although there apparently were no major problems caused by the outage, there was no shortage of anxiety and discomfort.
Most of the city’s 4,300 traffic signals were knocked out, either going dark or flashing red. In one instance, along Olympic Boulevard near Beverly Hills, motorist Beto Ramos said all the lights turned green.
“I had a green light, everyone had green lights,” Ramos said. “No one knew what to do. We’re so used to stop, go, stop, go. To see nothing but green lights was real weird, like all of society had paused.”
State officials have been pushing cities to install low-power stoplights that can be equipped with backup power to prevent outages. So far, San Francisco has adopted the lights, but Los Angeles has not because of the cost, said Rob Schlichting of the state energy commission.
Some commuter trains were delayed about 15 minutes during the lunch hour after power surges were detected by the Metropolitan Transportation Authority’s rail operations center. Transit buses also were delayed by traffic congestion.
Workers in high-rise office buildings in downtown Los Angeles faced twin inconveniences: Elevators stopped, forcing people to either stay in their offices or descend many flights of stairs. And air conditioning shut down, turning plush offices into carpeted saunas.
“It got hot real quick in here,” said Wells Fargo employee Sean Maddox, stuck in the 60-story Aon building.
Another tool of modern life -- high-speed Internet connections -- was knocked out in some parts of the city because of the outage. Patti Rockenwegner, a spokesperson for Comcast, which provides cable and broadband Internet service to about 500,000 customers in Southern California, said some customers in Hollywood and South Los Angeles lost service for about 40 minutes when the outage tripped fuses.
Hours after power was restored, however, some customers in other parts of the city, including the Fairfax and Mid-Wilshire areas, were reporting that Comcast lines remained out of service.
The outage occurred one day after the fourth anniversary of the Sept. 11 terrorist attacks, and a day after a Southern California man suspected of having ties to Al Qaeda warned of an impending terror attack on Los Angeles and Melbourne, Australia.
Although Mayor Antonio Villaraigosa and Police Chief William J. Bratton insisted that they knew of no “credible threat” to the city, many Angelenos wondered, if only momentarily, whether the outage was the result of a terrorist attack.
“It is the easiest way to trip up our routine, [to] cut the power,” said Matt Jarrette, who found himself in gridlock as he drove to work at the nonprofit California Assoc. of Physician Groups downtown. “The day after Sept. 11 -- it is disconcerting.”
The scene near harbor-area refineries was disconcerting in a different way. As of 5 p.m. Monday, regional air regulators had received more than 40 complaints about smoke and stack-top flames from residents in Wilmington, Harbor City, Long Beach and other areas in southern Los Angeles County, said Sam Atwood, spokesman for the South Coast Air Quality Management District. Six inspectors were dispatched to investigate the complaints and the mandatory shutdown reports from the refineries.
When refineries lose power, they release built-up pressure by sending oil products up their tall stacks, where it is burned off by stack-top flares, producing flames and heavy smoke. The burn-off is a safety mechanism that prevents flammable fuels from spreading, refinery officials said.
The 200-plus students at the Harbor Teacher Prep Academy in Wilmington were kept in their classrooms or moved to the cafeteria until the end of the school day as a precaution after school officials detected a noxious smell from a nearby oil refinery, said Los Angeles Unified School District spokeswoman Susan Cox. The school is on the Los Angeles Harbor Community College campus.
Refinery officials said the shutdowns were temporary and were not expected to affect Southern California gasoline supplies or prices.
The disruptions might well have been worse but for the proliferation of backup generators in the years since 2001, when the energy crisis threatened the state with frequent outages.
There has been an increase of several hundred generators in the region since then, according to the air quality district, which issues permits for generators.
Hospitals are required by state law to maintain backup generators, and many in Los Angeles were forced to switch to backup power Monday.
Generators also allowed operations to continue uninterrupted at Van Nuys Airport and Bob Hope Airport in Burbank, both of which were affected by the outage. Van Nuys serves private aircraft, while Burbank handles commercial flights as well as private planes. Los Angeles International Airport did not lose power.
Few Los Angeles schools have backup generators, but many have backup batteries to power emergency lights and alarms. Those systems were triggered Monday at several Los Angeles campuses.
New campuses that are part of the district’s ambitious building program are being equipped with backup generators, officials said.
Parents seeking word of the situation in the schools may have been taken aback by the message on the L.A. Unified emergency phone line, however. Callers who dialed Monday seeking word about the outage heard the following: “Topanga Elementary and Wonderland Elementary will be closed Tuesday Jan. 11 because of heavy rain. All other LAUSD campuses will be open as usual.... “
Contributing to the coverage of the blackout were Times staff writers Fred Alvarez, Sandy Banks, Patricia Ward Beiderman, Andrew Blankstein, Nancy Cleeland, Frank Clifford, Marla Cone, Jean Guccione, Carla Hall, Erica Hayasaki, Duke Helfand, Daniel Hernandez, Matt Lait, Marc Lifsher, Sara Lin, Caitlin Liu, Dave McKibben, Jean Merl, John O’Dell, Jennifer Oldham, Charles Ornstein, Lance Pugmire, Sam Quinones, Joel Rubin, H.G. Reza, Lisa Richardson, Louis Sahagun, Catherine Saillant, Jesus Sanchez, Beth Shuster, Larry B. Stammer, Rebecca Trounson, Dan Weikel and Richard Winton.