Avoidable Errors Led to Big Outage, Consultant Finds

A Sept. 12 power outage that affected half of Los Angeles was caused by multiple, preventable mistakes -- from faulty design of an equipment replacement project to poor communication among workers, according to a review by a private engineering firm.

The review, ordered by Mayor Antonio Villaraigosa and released Tuesday, confirmed that the outage occurred when workers cut live electrical wires with insulated tools after consulting erroneous design drawings, but goes further in finding problems with the way the city upgrades and operates its electrical system.

“It was a comedy of errors,” DWP board member Nick Patsaouras said after reading the 56-page report.

The review found, for example, that four electrical generating units, including two at the Haynes Generating Station in the San Fernando Valley, shut down automatically even though their design should have allowed them to keep operating.

“It was not part of their design to trip off as they did,” said Jim Dyer, senior consultant for Electric Power Group, the firm that reviewed the incident.

In all, the review found 15 “causal factors” that contributed to the outage and its widespread nature. The blackout left 2 million people without power for 90 minutes.

“The report indicates several areas of concern, ranging from employee training to quality control to system design,” Villaraigosa said, adding that he expects the DWP to “implement whatever steps are needed to ensure the safety and reliability of our electric system.”

DWP General Manager Ron Deaton said he is evaluating the findings.

“We will take corrective actions as expediently as possible,” he said. “Our primary concern is to do everything possible to ensure that an incident such as the Sept. 12 outage is not repeated.”

The outage occurred while DWP workers were installing new protective relay equipment for a transformer at a receiving station in Toluca Lake.

Acting on a faulty design, a worker cut a bundle of three live wires that were supposed to be left intact, triggering a short circuit that led to the shutdown of other transmission and generation stations to avoid damage.

The project is part of a systemwide effort to upgrade relays at 179 substations.

Electric Power Group, hired by the city under a $75,000 contract, reviewed the events leading up to and after the outage and found five significant problems that require changes in the way the DWP operates.

The project gave overlapping responsibility for designing the replacement project to a private contractor, Orsa Engineering, and to city employees, and both failed to adequately review the design, allowing design errors to go unnoticed.

Orsa provided the DWP drawings with cable numbering errors, the review found.

In the future, one group should be assigned responsibility for design, and there should be incentives or penalties to improve quality control, the review said.

Patsaouras said he would like to determine whether any of the contractors can be held financially liable for the incident, which caused more than $600,000 in damage to agency equipment.

In addition, DWP engineers failed to follow through in reviewing designs. The consultant suggested the agency determine whether there are enough engineers on staff to keep track of project designs.

The review also cited “inadequate job site tailgates,” during which the complete scope of work is reviewed, safety measures are discussed and workers confer on which cables are to be cut and relocated and which ones are to be left in place.

There was also inadequate training of wiremen regarding safe work practices, such as treating all existing wires as energized and cutting them one at a time instead of in a bundle.

The private consultant said a fundamental flaw of the system is that multiple substations are connected in loops without enough breaks or bypass lines so that a short-circuit at one affects all others.

The consultant recommended that transmission lines bypass some substations to allow them to operate without being affected if a problem occurs at another station.

Additional circuit breakers and changes at generating stations to keep them operating instead of tripping offline also should be considered, the report said. Four power generating units tripped offline “unexpectedly” during the event, the report said.

The consultant recommended that dispatchers be trained to figure out what to do in simulated emergencies, and that a system be developed to allow dispatchers to determine which of the 1,500 alarms that went off required priority action.

The report also found that displays of the activity of the electric transmission systems are not state of the art and do not allow workers to get clear information about what is happening.