Hello ... ? Glitch Kills Phones for Thousands

At least 150,000 customers and probably many more over a large swath of Southern California lost their phone and Internet service for up to 12 hours Tuesday because of a computer glitch at a Long Beach central switching plant.

The outage, which also cut off 911 service from many land lines, struck communities from Hermosa Beach to Newport Beach along the coast and as far inland as Whittier and Garden Grove. Some cellphone service was also lost, though on a more limited basis.

The problem, discovered at 2:23 a.m., was caused by a software error in a downtown Long Beach office of Verizon Communications Inc. The malfunction corrupted the main software that connects calls and operates the local network, Verizon said.

The backup system, which should have deployed, failed to activate, and Verizon officials said they were trying to figure out why.

The hard-drive crash at the Long Beach facility Tuesday morning had a ripple effect at 15 other Verizon central switching offices from Fountain Valley to Redondo Beach and inland to Downey, as well as at dozens of cell towers connected to the network.

The breakdown of telephone connections was the second time in about a month that the failure of a small part of a complex network caused households across large chunks of the region to lose a basic service.

In mid-September an electrical worker accidentally cut the wrong cable at a power substation, shutting down power to much of Los Angeles for hours. Both incidents underscore how basic utilities relied upon by millions of people are vulnerable to relatively simple disruptions.

Even though the phone shutdown left some residents unable to reach 911 for hours, authorities reported no known crime or medical problems resulting from the telephone blackout.

But it did make for a vexing day across the coastal region.

On Main Street in Seal Beach, merchants had trouble because credit card scanners didn’t work. Surgeons at Long Beach Memorial Medical Center and Miller Children’s Hospital were able to perform scheduled operations only because their cellphones would allow patients to reach them for post-op help. Several morning flights at Long Beach Airport were delayed up to 45 minutes at the facility because of phone-related problems with ground communications.

At Lowell Elementary School in Long Beach, it was strangely quiet in the front office. “Only two calls made it through this morning, and our computer programs aren’t working, and we can’t reach the district,” secretary Betty Wohlgezogen said at noon.

The state’s fifth-largest city immediately had moved into what it declared “communication failure protocol.” At an emergency communication center near Long Beach Airport from which Long Beach police and fire officials operated Tuesday, a team of volunteer ham radio operators fanned out to 17 of the largest nursing homes or assisted living centers to communicate with medics, hospitals and first responders. The radio operator plan, city officials said, was designed to prevent a repeat of the New Orleans hurricane tragedy in which several bedridden patients were marooned and died at a nursing home.

The outage was so complete that Long Beach officials had to drive to KJZZ-FM (88.1), the city’s designated emergency broadcast station, to let the radio station know what to tell listeners.

To cope with the failure of the 911 system, police departments in several affected cities placed more patrol cars on the streets. .

Phone industry experts were surprised by the scope of the outage but said it showed how interrelated the telecommunications network is -- both landline phones and cellphones.

“People don’t realize how interconnected telecom is,” said Kathleen A. Dunleavy, a spokeswoman for Sprint Nextel Corp. “People think wireless operates wholly wirelessly, but it doesn’t. The cell towers are dependent on [high-capacity] T-1 lines.”

Cellphone service is typically wireless only to the nearest cell tower. From there, high-capacity T-1 lines take the voice and data traffic, including ordinary calls, e-mail and Internet browsing.

The 15 central offices hit by the outage are local hubs for telecommunications, each one serving 10,000 to 15,000 customers, said Verizon spokesman Jonathan Davies. The office contains switches, routers and other electronic gear to connect calls and provide features such as call waiting and voicemail. It’s often where the long-distance lines and the Internet meet the local lines.

The company will analyze the cause and figure out ways to prevent another collapse.

Verizon Wireless, which is 55% owned by Verizon, has two backup networks connecting its cell towers. One is a high-capacity fiber-optic network, the other a wireless microwave system. Both of those cellular backup systems seemed to have fewer problems than the land-line backup system in Long Beach that failed.

Verizon and other carriers late Tuesday were still trying to figure out exactly how widespread the outage was to both land and cellular phones.

T-Mobile USA said its customers also were affected by network congestion because so many of them switched to cellphones to make calls. Customers as far south as Laguna Beach reported problems, said spokesman Graham Crow.

Carriers typically have backup systems already in place, said analyst Charles Golvin of Forrester Research Inc., a telecom research firm. Putting more in wouldn’t be economical, he said.

“Nothing is going to be foolproof,” Golvin said.

Phone service was restored around 2 p.m. But the breakdown left both residents and authorities with a new appreciation about how reliant they are on the phone lines.

“The good lesson here is just how much we depend on our communications systems. It’s huge,” said Laguna Beach Capt. Danell Adams. Under cloudy skies at the Seal Beach Municipal Pier, Kent Trollen was philosophical about the telecommunications breakdown.

“It has been very peaceful,” Trollen said. “But it accentuates the neurosis of today’s lifestyle.”

*

Times staff writers Susana Enriquez and Caitlin Hammer contributed to this report.