Utility officials seek to learn why safeguards failed in blackout

The failure of a single piece of equipment in Yuma, Ariz., ignited a massive blackout that left more than 4 million people without power, baffling utility officials and highlighting the vulnerability of the U.S. electrical grid.

Authorities in Arizona said Friday that safeguards built into the system should have prevented the breakdown at a Yuma substation from cascading across southern Arizona and into California and northern Mexico.

They didn’t, and the resulting instability led to the sudden shutdown of the San Onofre nuclear power plant, cutting off power to a large swath of Southern California. “We lost all connection to the outside world,” said James Avery, San Diego Gas & Electric’s senior vice president of power supply. “This happened in a matter of seconds.”

PHOTOS: Blackout leaves millions without power

Energy experts and utility officials agreed that the breakdown was troubling.

“We’re struggling,” said Daniel Froetscher, vice president of energy delivery for APS, the largest electricity provider in Arizona. “We have to take a hard look at the system design and figure out exactly what happened.… We don’t know the underlying causes.”

Among other concerns, some experts said the failure of safeguards suggests the potential for a saboteur to take down a regional power system.

Such incidents “remind us that on a day-to-day basis that we rely on a very complicated electrical system,” said Sarah Ladislaw, a senior fellow with the Center for Strategic and International Studies who specializes in energy security.

APS officials launched an investigation to determine the precise chain of events that began early Thursday at the North Gila substation outside of Yuma. Utility employees had noticed a problem with a series capacitor, a piece of equipment about the size of a small car that helps the utility manage voltage, Froetscher said. APS personnel were dispatched to take it offline.

Typically, the utility can shut down an individual capacitor and reroute power without any disruption of service, Froetscher said.

But this time, something went wrong. After the North Gila capacitor was taken offline, the 500-kilovolt transmission line that runs through the substation went down.

At that point — 3:27 p.m. — the grid should have compensated for the loss of the line, which runs from Yuma to the Imperial Valley and San Diego, Froetscher said. It is essentially one of two lines that carry power to San Diego. The other runs down from the north, along the coast through San Onofre.

“The intent is for the system to automatically open and close different breakers and relays and switches to reroute power from the line that was lost to other lines to continue to provide service,” Froetscher said. “Most times when a line goes out unexpectedly, the system performs exactly as it’s expected to and customers never know the difference.”

For about 10 minutes, the system seemed to be working properly. But by 3:38 p.m., residents in Yuma began to lose power.

From there, outages spread across the Southwest. It would be early Friday, about 12 hours later, before power was fully restored.

Now the question facing investigators is: “Why?”

Froetscher said it’s possible there were errors made by APS in the work done on the capacitor and the steps it took after the single line went down.

“The bigger issue that will be the focus of many people’s work is, what really was the contributing cause to the outage in Yuma and the subsequent outages in California?” he said. “It would be …. premature at this point for me to guess.”

Jon Wellinghoff, chairman of the Federal Energy Regulatory Commission, said that despite the blackout, federal officials have been making progress in recent years on improving the reliability of the energy grid.

“Any time people are out of power, it is a big issue,” Wellinghoff said. But he noted the incident was far less severe than the 2003 blackout that left large parts of the Northeast in the dark.

In the meantime, San Diego Gas & Electric Co. and other utilities are working to get power plants back up, a task that must be done carefully. Fifteen power plant units, including two at the San Onofre Nuclear Generating Station, automatically shut down as part of an automatic safety response.

The utility’s president, Michael Niggli, said that San Onofre “acted to protect itself by taking itself offline.”

Froetscher said he didn’t know why power plants in California went offline. He said no Arizona plants were shut down.

He noted that nuclear power plants normally trip offline automatically when there is a system disruption, ensuring that they have adequate time to start emergency power to cool their nuclear cores. It was unclear whether non-nuclear plants tripped off automatically or responded to requests for a voluntary shutdown.

A series of blackouts over the last 20 years, the worst of them in the Northeast, led the electrical industry and federal regulatory agencies to improve automatic protection and require far greater coordination among regional operators of the U.S. grid. That has boosted U.S. reliability to 99.999%, among the best in the world, but that has still left some chance for failures.

Almost every cascading blackout involves not a single breakdown, but multiple system breakdowns.

“It has never been one thing that goes wrong,” said Andrew Phillips, a high voltage transmission system expert at the Electric Power Research Institute. “Many things all go wrong on a single day.”

The largest recent single outage was the 2003 event in the Northeast, when virtually the entire region was blacked out and 50 million people were affected. A federal investigation identified a wide range of causes and recommended a series of improvements that were intended to preclude another such failure.

About five years later, the electrical power industry decreed that it had so vastly improved the system that a similar event was “much less likely to occur.”

President Obama has been an advocate for the development of a “smart grid,” a modernized, higher-tech system that would be more efficient, more reliable and would expand access for alternative energy. But Ladislaw, the energy security expert, said “smart grid” projects that were launched by federal stimulus funds are now running out of money.

She and others have noted that a grid that relies more heavily on computer technology could become more vulnerable to security attacks. The White House warned in June that an updated grid could be open to threats that include “malware, compromised devices, insider threats and hijacked systems.”

sam.allen@latimes.com

ralph.vartabedian@latimes.com

mitchell.landsberg@latimes.com

Los Angeles Times staff writers Nicole Santa Cruz in Yuma, Ariz., Marc Lifsher in Sacramento and Richard Marosi, Louis Sahagun and Tony Barboza in San Diego contributed to this report.