Firms Find It Pays to Plan to Recover From a Disaster
When corporations’ disaster planners get together, the talk naturally turns to disasters.
They talk about the large East Coast bank that recently was forced to borrow--at a stiff premium--about $1 billion from the Federal Reserve Bank after a computer snafu froze the bank’s own money supply for 12 hours.
Or, they talk about the 1972 earthquake in Managua, Nicaragua, that reduced one soft-drink bottling plant to rubble but left another bottler standing.
Then there was the nearly 50,000 gallons of water that--along with the roof and an air-conditioning system--crashed down on Mazda Motors of America’s computer complex in Compton during a February, 1983, rainstorm. Amazingly, within five working days, Mazda had constructed a new computer building and installed a new computer to serve its 500 dealers in 31 states.
That rapid response was in contrast to what happened in 1959 when fire destroyed a Pentagon computer center. Even though officials used the full power of the Defense Department to obtain high priority with suppliers, they had no disaster plan, so it took 30 days to rebuild the facility.
That fire showed military data-processing managers--and their corporate counterparts--that “guns, guards, locks and dogs don’t work” when it comes to protecting computer records from destruction, according to Laurence B. Compton, who spent nearly 20 years devising computer security plans for the Strategic Air Command and the Air Force Military Personnel Center.
Despite the continuing threat of fires, earthquakes, floods and man-made disasters, only 15% of the nation’s companies have created emergency recovery plans, according to Diane C. Smith, manager of contingency planning for FCA Computer Services, a subsidiary of American Savings & Loan and co-founder of the Assn. of Contingency Planners, based in Long Beach.
However, because corporate dependence upon computers has grown so rapidly, “most companies can operate for roughly one week without their data center before they go under,” Compton said. Now a vice president of Total Assets Protection, an Arlington, Tex., company that develops disaster recovery programs for corporations, he added: “Some banks and insurance companies say they can’t operate after just 2 1/2 days.”
In addition to cash flow problems, when a center stops computing, “the backlog of work becomes so large that you can’t keep up,” Compton said.
Disaster planning is important because 70% of businesses that “sustain a significant interruption . . . go under,” according to Barbara Foster, a former Marin County disaster planner who now serves as a private consultant for the American Red Cross’ Marin County Private Sector Disaster Preparedness Project.
In California, corporate interest in disaster recovery programs has been heightened because “we’ve had more magnitude-six earthquakes during the past six years than in the prior two decades,” according to James Watkins, a state official with the Office of Emergency Services.
That growing interest has prompted an increase in the number of backup computer facilities operated by third-party companies that assist corporations with disaster preparedness.
‘Hot’ and ‘Cold’ Rooms
Sunguard Data Systems of Wayne, Pa., recently announced plans for a computer backup facility in the Rancho Bernardo area of San Diego, and Comdisco Disaster Recovery Services of Chicago said it planned a similar facility in Cypress. Those facilities--which include completely equipped “hot” computer rooms and empty “cold” rooms--appeal to corporations that do not want to create and maintain their own disaster recovery facilities.
For a fee (one disaster preparedness expert suggested that “hot sites” generally charge an initial subscription fee of between $5,000 and $18,000, a one-time pre-use fee of $25,000 and a $4,000 to $10,000 daily operating fee), a customer can move its staff in and use the center until a backup system is built.
Pennsylvania Life Insurance of Santa Monica, a member of American Can Co.’s financial services division, has constructed a recovery plan that “will let us recover (critical) data processing in five business days,” according to Bill Homan, a vice president and director of the company’s data center.
Pennsylvania Life’s recovery plan is based on the assumption that it will remain in business because its “critical” computer functions--including billing, claims processing and the writing of new policies--will be resumed within that five-day period at a “hot” computer site owned by Sunguard.
Surprisingly, since 1979, none of Sunguard’s 278 clients has ever experienced a full-fledged disaster. And only four of Comdisco’s 560 clients (including one disaster caused by a frozen water line and another generated by a fire related to a Santa Ana wind condition) have had to rebuild critical computer functions at a Comdisco site.
That does not mean that disasters do not happen, however.
“Two months ago, I had four calls in one week from (non-client) companies that desperately wanted to use our facilities,” Comdisco President Raymond R. Hipp said.
Also, recent heavy rains in Louisiana generated a flurry of calls from desperate data-processing managers searching for immediately available backup facilities. In keeping with Comdisco’s contractual agreements with current customers, though, Hipp turned those potential customers away, despite offers to pay “heavy premiums.”
Although many companies turn to third-party suppliers or reciprocal agreements with companies that have compatible equipment, some companies, such as Rockwell International, prefer to go it alone during a disaster.
Rockwell spends between $1 million and $2 million annually on disaster preparedness, including $200,000 for rent and maintenance of a “cold site” that could quickly be turned into an active computer center should a disaster wreck one of the company’s existing data-processing centers.
Integrated Planning
“I’m all for calling it alternate-site processing because the phrase ‘disaster recovery’ scares the hell out of me,” said Larry Manly, vice president of operations for Rockwell’s Information Systems Center, during a recent meeting in Los Angeles of the Assn. of Contingency Planners. Instead of creating a separate system to handle disasters, Rockwell has integrated contingency planning into normal work routines, giving the company a “large, knitted-together type of solution,” Manly said.
Rather than rely on “canned tests” of the system, Rockwell once each quarter orders a different computer center to process its complex, 90,000 employee payroll program. After the test, a management committee reviews the written, step-by-step plan to “fix problems that regularly occur.”
Some companies take pains to ensure that tests are realistic. An insurance company reportedly called data center employees at home and told them that the center had been destroyed in an airplane crash. Employees--who only later learned that the center and its managers were safe--subsequently rushed to recreate the company’s computing center at an alternate site.
That kind of realism can pay off, according to John Smith, superintendent of operations at Los Angeles International Airport, who each year oversees a simulated disaster that involves 34 separate government and military agencies.
“You don’t find too many heroes because even trained people can panic or turn into official sightseers,” said Smith, who also has directed emergency operations in real life. “(People) just don’t want to get involved, and when you are frightened, you do strange things.”
Proper Training Key
However, properly trained employees can avert disaster.
When a 1983 rainstorm cracked the roof on Johnson & Johnson’s dental products plant and office in East Windsor, N.J., “water started pouring into the building,” said Larry Bowker, director of management services. Although a quick-thinking employee managed to shroud the company’s computer hardware in plastic sheeting, “there was about a half-inch of water in the building, and there was dirt all over the place,” Bowker said. “IBM said we’d never get (the computer) back up in a week.”
However, Johnson & Johnson’s disaster recovery plan called for a quicker return to business because “after a five-day time period--it becomes critical if you’re not manufacturing product, shipping it and (tracking) accounts receivable--you don’t have a cash flow.”
Luckily, the rainstorm that began on Wednesday ended on Thursday, and work crews rebuilt the roof on Friday. Bowker said Data Clean Corp., a company that cleans and maintains computer rooms, had Johnson & Johnson’s computer room ready for operation by Sunday night. “It could have been a lot worse,” Bowker said. “All it takes is a (broken) sprinkler system and you can’t do accounts receivable, billings and orders, the basic stuff that a company needs to do.”
Instead of focusing on “cataclysmic events,” recovery plans should focus on “what you (can) protect against and what you can live with financially,” Manly suggested. “You obviously can’t have idle (computer) capacity sitting around waiting for a disaster to happen.” And, he said, “in the event of a cataclysmic event, (Rockwell has) decided to put the business behind us and concentrate on the personal aspects of recovering.”
At IBM--which has outlined its emergency program in a formal corporate document that charges senior managers at each of its facilities with emergency planning--protection of company assets is intertwined with avoiding and minimizing personal injuries and helping the local community return to normal, according to IBM spokeswoman Debra Gottheimer.
During last year’s Mexico City earthquake, for example, IBM used company planes to fly in food and other emergency supplies that were distributed by Mexican authorities, Gottheimer said.
Broadening Scope
Although corporate planning has been focused on the data processing needed to maintain cash flow, contingency planners have been broadening their scope to include the management of employees during a disaster.
“I think there are three parts of contingency planning--people, data and facilities, in that order,” Smith said. “This profession is evolving into something called crisis management, which encompasses a lot more than just data or buildings.”
Aware that earthquakes could rock its San Francisco headquarters building, Levi Strauss’ contingency planners have stocked the building with emergency supplies including food, water, shovels and blankets, according to spokeswoman Joyce Bustinduy.
Because key executives could be lost--or the headquarters building destroyed--Strauss’ plan includes “plans for where (executives) would meet and who would be in charge. That way the key executives can get together and figure out what’s going to happen next,” Bustinduy said.
“(Companies) began to spend time on scenarios like an 8.3 earthquake at a plant on the San Andreas Fault, and the people who had been concentrating on data processing are now being asked to create corporate plans that take care of people, too,” Smith said. “Data-processing people are gradually becoming the corporate managers of corporate disaster planning.”
Aware that transportation will be difficult after a major disaster such as an earthquake, one California bank has contracted with various companies that will transport the bank’s key executives after a disaster, according to Barbara Poland, the emergency preparedness coordinator for General Telephone of California in Thousand Oaks and a leader of the Los Angeles Business and Industry Council for Emergency Planning and Preparedness that was formed in 1983 by the City of Los Angeles.
The council sponsors training seminars ranging from “how to use a fire extinguisher” to “convincing top management that planning is necessary,” Poland said.
“It takes some organization and leadership to get those things done,” Poland said, “but you can see (an earthquake) coming down the road. It’s happened in Greece, Turkey and Mexico City, so you can’t assume that it won’t happen here.”
