Calm Night on Y2K Watch at EarthLink ISP in Pasadena
There would be no champagne at midnight. Food and drink are never allowed in EarthLink Network’s Operations Center, even on the night the world bid farewell to the 1900s.
But for the dozen or so technical staffers on hand New Year’s Eve at EarthLink Network to monitor the Internet service provider’s critical computer systems as the Y2K bug threatened, the lack of alcohol was hardly a cause for dampened spirits.
“This is a part of Internet history, and everyone here wants to be a part of this,” said Arman Afsar, director of network operations for Pasadena-based EarthLink, one of the largest ISPs in the country, with about 1.6 million customers.
On a normal Friday night, the Network Operations Center--affectionately known as “the NOC”--would be monitored by a supervisor and two technicians. This night, the official staff was boosted to a supervisor and three technicians to watch for any glitches stemming from the Y2K bug, said John Ko, the 26-year-old supervisor on duty. Those four were joined by about 10 others as the clocks rolled toward 2000.
“I thought it would be cool to be with the tech people, where the news would be happening,” said Marcy Harbut, an operations documentation specialist who spent much of her time surfing the Net and chatting with co-workers. “It’s a big social thing. Plus, I get a couple of extra days off.”
For the most part, the atmosphere in NOC was joyful and relaxed. Two large televisions broadcast images of exploding fireworks in festive world capitals. Workers, mostly decked out in T-shirts and fleece jackets bearing EarthLink logos, reclined in ergonomic chairs.
Once or twice an hour, the mood turned serious as they watched more than 20 computer monitors arrayed in a horseshoe curve in the sealed room that resembles a NASA mission control room.
“We perk up as each time zone switches,” said Anthony Castillo, a NOC technician.
Network engineers at EarthLink held dress rehearsals over the previous two weeks to prepare for the coming of Y2K. The exercises allowed teams to practice their network monitoring and “make sure everyone is in place for whatever happens,” said Afsar, who was on duty from 10 p.m. Thursday until about 4 a.m. Saturday.
The actual stroke of midnight wasn’t the most critical hour for EarthLink. Its computer servers are set to Greenwich Mean Time, so as far as they were concerned the new year began at 4 p.m. Friday. That hour passed without a single computer confusing “2000” for “1900,” Afsar said.
The calm at EarthLink was mirrored around the country. IBM, Sun Microsystems, Dell and Compaq all reported virtually no problems for their customers as the new year dawned.
At Hewlett-Packard’s command center in Palo Alto, a team of 35 dwindled to 15 before the millennium rolled past the U.S. East Coast without incident. The remaining experts, who needed caffeine and sandwiches more than emergency power generators or their network of 70 Iridium satellite phones, mainly watched fireworks on television.
How mild were the calls? Some Chinese customers were puzzled that when the appointed hour came, their computers flashed 12:00 a.m.--correct, but not 24:00, the military-style time to which they were accustomed.
At EarthLink, most of the action was linked to the Internet Operators Group, a global coalition of Internet service providers and equipment vendors including America Online, AT&T;, Cisco Systems and Lucent Technologies. The group, known as IOPS, used telephone conference calls and a Web site to report any sign of trouble and seek assistance in fixing problems as they arose.
“EarthLink’s here. So far, so good. No problems,” said NOC technician Robert Tellez, reporting his company’s status during an IOPS roll call.
Tellez had stayed up until the wee hours of Friday morning to see how the Y2K bug affected New Zealand and Australia, the first two major countries to pass into 2000. “When they were fine, I knew it would be an easy day,” he said.
The technicians assembled in the Network Operations Center marked the new year’s arrival by watching Mayor Richard Riordan and comedian Jay Leno light up the Hollywood sign on TV. Then, calmly and methodically, they began checking the computers.
One problem emerged with a connection between Pasadena and an EarthLink call center in Sacramento. The computer screens indicated that four high-speed lines were down. Ken Hillman, NOC’s manager, reached for a phone and confirmed that all the links were good. A management program that automatically updates itself every night at midnight had prevented those links from responding to NOC’s Y2K checks.
“This is not Y2K-related,” Afsar said.
The only Y2K problem that did emerge happened a few minutes later. Ironically, it had to do with the IOPS Web site used for reporting Y2K-related troubles. Although the problem did not affect EarthLink customers, the assembled group worked on it diligently until it was solved about 40 minutes later.
Like most of those on duty, Afsar wasn’t disappointed by the relative lack of excitement.
“Thank God nothing happened,” Afsar said. “But later we’ll sit back and say, ‘Hey, we were there.’ ”
Times staff writer Joseph Menn contributed to this report.
*
THE UP SIDE OF Y2K EFFORTS
Y2K investment may boost economy. A1 . . . Federal officials lauded managers for making rollover a nonevent. A12
