DWP Issues Blackout Study; Chief Apologizes

Last week’s massive power outage in Los Angeles could have been prevented if workers had double-checked faulty designs and whether lines were electrified, the Department of Water and Power said Thursday in a report recommending sweeping changes.

The 24-page evaluation of the Sept. 12 incident provides the most detailed account yet of an accident that cut power to 2 million people in Los Angeles, Glendale and Burbank.

In releasing the report, DWP General Manager Ron Deaton apologized to residents for any inconvenience caused by the 1 1/2 -hour outage.

“It was an unfortunate event,” Deaton said in a letter to the mayor in which he praised city employees for restoring power quickly. “It also gives us a humbling insight into the procedures that must be improved and helps us better prepare for a major disaster.”

Mayor Antonio Villaraigosa said he was “deeply concerned” by the findings and said the analysis does not go far enough. He ordered a more detailed study to be done in the next 30 days to find ways to limit the scope of future power outages and reduce public health problems caused when oil refineries lost power and burned off excess fuel.

“The preliminary conclusions about the cause of the outage are troubling because they confirm that the outage was the result of multiple errors and a lack of communication,” the mayor wrote to Deaton after the release of the report.

The outage occurred when a worker installing a new automated control system at the Toluca Lake receiving station cut a bundle of three wires that were electrified, triggering a short circuit that led to the shutdown of other transmission and generation stations to avoid damage.

All similar work has been suspended until a more detailed evaluation is completed, Deaton said.

The incident affected 897,922 homes and businesses, more than half of the DWP’s customers, the report said.

The report concluded that several people were in positions to prevent the incident, but does not identify them. A private engineering contractor that drafted the initial design for the automated control system did not account for the replacement of an old breaker relay “since it was outside their requested scope,” and DWP design engineers indicated they would design the replacement for the old relay but had not gotten to it in time, the report said.

A test engineer discovered the omission and tried to fix the problem, but a print with errors was given to the work crew, which cut the wires without testing them to see if they were electrified.

“This outage was a result of a chain of events,” the report concluded. “A design error resulted in a major rework by the field test engineer. If a design review of the contract engineering work had been conducted by LADWP engineers, the errors would have been caught prior to being released to the field for construction.”

The miscommunications continued with the field workers at the Toluca Lake facility.

Knowing the correct design called for the existing wiring to remain, the test engineer didn’t cut power to the wiring, the report said. “However, because the incorrect prints given to the wiremen showed the wires to be moved, the wiremen disconnected the energized wires.”

The report troubled city elected officials, including Councilman Tony Cardenas, who chairs the council panel overseeing the DWP. “It sounds like they got sloppy when it comes to making last-minute design changes and telling the field people to ad lib,” Cardenas said.

The incident cost the agency $660,000 in damage to facilities and losses from having to sell excess electricity for below-market rates, the report said.

The report included 17 recommendations, including a requirement that all design papers drafted by private contractors be double-checked by DWP engineers and that training and procedures be changed for wiremen, including a mandate that they cut one wire at a time when dealing with relay circuits, to avoid the chance that two live wires would cross.

The analysis also recommended that the department revise its plans for communicating with the public and elected officials, noting the outage began at 12:32 p.m., but a public relations specialist did not emerge from the DWP headquarters until 1:30 p.m. The Los Angeles Police Department was notified at 1:05 p.m.

Villaraigosa and Cardenas said faster public notice should have been given, especially in light of reports the day before the blackout that a suspected Al Qaeda videotape threatening an attack on Los Angeles had surfaced.

The delay in telling the public that the outage was an accident may have contributed to public anxiety over whether the event was caused by terrorism, Cardenas said.

“It took them almost an hour before they told the public they ruled terrorism out, that it was an accident,” Cardenas said.

An engineer, Cardenas also said the report released Thursday did not sufficiently consider the vulnerability of the electric system or provide enough details on avoiding similar incidents.

“The report is a little thin,” Cardenas said, adding the council still wants to question DWP officials. In his letter to the mayor, Deaton vowed to take steps to avoid another such incident. “On behalf of the LADWP, I apologize to the citizens of Los Angeles for any inconvenience this outage may have caused.”

Villaraigosa also said he wants more than just a list of procedural changes.

“I need your commitment that the Los Angeles Department of Water and Power will take the steps necessary to prevent any outages that are not the result of natural disaster and to quickly mitigate those that are the result of natural disaster,” Villaraigosa said.