Hackers Victimize Cal-ISO

For at least 17 days at the height of the energy crisis, hackers mounted an attack on a computer system that is integral to the movement of electricity throughout California, a confidential report obtained by The Times shows.

The hackers’ success, though apparently limited, brought to light lapses in computer security at the target of the cyber-attack, the California Independent System Operator, which oversees most of the state’s massive electricity transmission grid.

Officials at Cal-ISO say that the lapses have been corrected and that there was no threat to the grid. But others familiar with the attack say hackers came close to gaining access to key parts of the system, and could have seriously disrupted the movement of electricity across the state.

Democratic and Republican lawmakers were angered by the security breach at an entity that is such a basic part of California’s power system, given its fragility during the state’s continuing energy crisis. One called the attack “ominous.”

An internal agency report, stamped “restricted,” shows that the attack began as early as April 25 and was not detected until May 11. The report says the main attack was routed through China Telecom from someone in Guangdong province in China.

In addition to using China Telecom, hackers entered the system by using Internet servers based in Santa Clara in Northern California and Tulsa, Okla., the report says. James Sample, the computer security specialist at Cal-ISO who wrote the report, said he could not tell for certain where the attackers were located.

“You don’t know where people are really from,” Sample said. “The only reason China stuck out is because of the recent political agenda China had with the U.S. . . . An ambitious U.S. hacker could have posed as a Chinese hacker.”

The breach occurred amid heightened Sino-American tensions after the collision between a Chinese military jet and a U.S. spy plane. In early May, there were hundreds of publicly reported computer attacks apparently originating from China. Most of those incidents involved mischief; anti-American slogans were scrawled on government Web sites.

The attack on the Cal-ISO computer system apparently had the potential for more serious consequences, given that the hackers managed to worm their way into the computers at the agency’s headquarters in Folsom, east of Sacramento, that were linked to a system that controls the flow of electricity across California. The state system is tied into the transmission grid for the Western United States.

“This was very close to being a catastrophic breach,” said a source familiar with the attack and Cal-ISO’s internal investigation of the incident.

On May 7 and 8, as the infiltration was occurring, California suffered widespread rolling blackouts, but Cal-ISO officials said Friday that there was no connection between the hacking and the outages, which affected more than 400,000 utility customers.

“It did not affect markets or reliability,” said Stephanie McCorkle, a spokeswoman for Cal-ISO.

Officials of the agency made no public acknowledgment of the attack until Friday when contacted by The Times. The agency did, however, call the FBI, which is investigating.

McCorkle said Cal-ISO did not make a public disclosure about the hacking “because it didn’t impact the reliability of any of our internal networks.”

“It didn’t have a negative consequence and would not have impacted the public or market participants,” McCorkle said.

After the attack was discovered, the report says, investigators found evidence that the hackers apparently were trying to “compile” or write software that might have allowed them to get past so-called firewalls protecting far more sensitive parts of the computer system.

The attackers focused on parts of the grid agency’s computer system that are under development. In what may have been the most significant lapse, the system being developed was not behind a firewall, a security element designed to keep out those who are not entitled to access.

Additionally, so-called tripwires that might have alerted agency security personnel to the unauthorized entry were nonexistent. Nor were there logs within the system that might have identified users entering the system as the infiltration was occurring, the report notes.

What’s more, dozens of ports into the computer system were open, when only a handful should have been available.

“All servers should be hardened regardless of their role or location in the network,” the report says. “Only ports that are required to be open should be opened; all others should be disabled.”

Complicating the investigation, workers at Cal-ISO rebooted their computers when the machines balked, apparently in response to the infiltration.

“This action limited our ability to discover all files and activity that may be related to this compromise,” the report says.

Sample, the security engineer who wrote the report, downplayed the potential threat and said the attack was “something that we’ve been anticipating.”

“It was a compromise, not really an attack,” he said.

State legislators were not comforted by such distinctions.

“That’s really amazing on two counts: that there were computers not behind a firewall and it took 17 days to discover,” said state Sen. Debra Bowen (D-Marina del Rey), who chairs her chamber’s Energy Committee.

Bowen, who was informed of the breach by The Times, called it a “serious matter” and said she was “very concerned to learn about this from the L.A. Times, rather than from the ISO itself.” The lack of official notification, she said, adds to her skepticism about whether the agency has been forthcoming.

“It is embarrassing, so I can understand they would not want to talk about it,” Bowen said. “We’re going to ask some questions.”

The Independent System Operator, established in 1998 when the state opened the newly deregulated electricity market to competition, is an essential component of the state’s electricity system.

The purpose of the nonprofit entity is to balance the flow of electricity across the state and make last-minute power purchases to match demand and avoid blackouts. The Legislature reconfigured the agency earlier this year, giving Gov. Gray Davis the power to appoint the five-member board that oversees it.

“It is troubling that it happened,” said Sen. Tom McClintock (R-Thousand Oaks). “It is disturbing that it took so long to be corrected. And it is galling that it was not reported to the Legislature.”

McClintock labeled as “ominous” the possibility that the attack came from China. He said he is preparing a request for all documents related to the breach and is considering requesting a formal legislative inquiry.

ISO board member Mike Florio, who represents consumers, said he had a vague recollection that the board was informed of the attack. But he also was surprised to learn some of the details.

“We hire people to deal with this stuff,” he said, “and they said they dealt with it.”

*

RELATED STORIES

Setting prices: Cal-ISO urges U.S. to intervene on pricing. B1

Budget impact: Demand for funds threatens youth program. B1

Natural gas: Prices fall and questions rise about a supplier. B10

New plant: Calpine plans a power facility in Riverside County. B10

Edison: Utility officials will pursue governor’s rescue plan. C1