Disaster Recovery Planning in the Calm Before the Storm
May 11, 2006
The devastating hurricane season of 2005 won't be forgotten any time soon. So how is the coming hurricane season, which will officially begin June 1, likely to measure up?
Forecasters expect another busy one, but fortunately, not as many intense storms are expected to develop as last year a total of nine hurricanes, five of them major (category 3-5), predict Colorado State University researchers, with an 81 percent chance of a major storm hitting the United States. The average from 1950-2000 was 5.9 hurricanes, 2.3 of them major. Last year saw a record 15 hurricanes, seven of them major, including a record-four category 5 storms.
With Gulf Coast devastation fresh in their minds, storage managers and CIOs are hopefully scurrying around taking ample precautions as another potentially difficult year looms.
Donna Scott, an analyst at Gartner, notes a far greater boardroom awareness of disaster recovery (DR) since 9/11.
"Every major disaster including the recent blackouts and Katrina raises awareness of how important IT availability is to the continuing health of any business," says Scott.
Yet according to Lenny Monsour, director of product management at SunGard Availability Services in Wayne, Penn., greater awareness does not necessarily translate into firm DR commitment. He reports that after a hurricane, or any other regional disaster, his company typically sees an upsurge in interest, which often doesn't result in new sales or contracts.
"For most companies, the notion seems to be that 'this won't happen to me,'" Monsour says. "It is critical that businesses in high-risk regions take the appropriate precautions before a hurricane hits."
Monsour recommends that businesses in high-risk regions first outline their business processes to understand how they rely on various technology systems. Sales processes, delivery process, finance process, human resources process and other elements of the business should all be reviewed in order to develop an appropriate plan to help a business recover should its primary office be significantly damaged by a hurricane.
"Documents such as a Business Impact Analysis and a tested Business Continuity Plan help a company ensure they can respond to a hurricane," he says. "They specify how critical applications can be recovered in minutes and how less critical systems will be brought back online at a slower rate. Additionally, companies should have a place for employees to go to recover and continue working."
Site Selection Looms Large
That location, though, had better be chosen wisely. Stephen Foskett, director of strategy services at GlassHouse Technologies, discovered that a lot of companies in New Orleans had their data off-site but still in the path of the storm. Thus site selection means being far enough away to be outside the disaster zone, yet close enough that staff can reach it.
"Also, make sure that there are systems (such as servers, networks and other facilities) there to use the data when it is needed," says Foskett. "I know of one bank that had their data replicated off-site but still couldn't bring their systems up because they lacked the right servers at their remote data center."
Scott believes it unlikely, during some types of disasters, that people with family within the affected area would be willing to travel away from their loved ones. She suggests, therefore, that large organizations operate two data centers for example, one in Houston and another in Dallas and split IT and storage resources between both sites. Perhaps half the production load is done at each location and backed up to the other data center. Alternatively, smaller companies can bring in remote consulting resources as part of the DR plan. People who are grooved in on the company's IT needs and are available to function during an emergency.
"You need these resources to be involved in DR testing and possibly sitting in on weekly meetings," says Scott. "It's too late to think about calling SunGard once disaster strikes."
Keeping Costs In Line
Being fully prepared can, of course, result in staggering expenditures. So is there a way to cut costs?
"It takes big money to become windproof/waterproof, have a sturdy stand-by power supply and be able to mirror applications to a remote site," says Clive Longbottom, an analyst at U.K.-based Quocirca. "As only the largest companies can afford it, hosting is frequently used to reduce the bill."
He notes that outside services almost always have better DR plans, more durable facilities, all the back up power they need, and multiple sites that can mirror each other. They also tend to be in old nuclear bunkers, bank vaults, or specially built units that can withstand most of what nature can throw at them.
Another way companies attempt to reduce costs is to forgo the expense of the latest technologies and make do with recovery from tape. Tape can be invaluable in cases of data corruption, and are certainly a lot cheaper than online copies (replication). But it can take days or even weeks to recover systems from tape.
"Don't rely on tape backup alone for DR protection," says Foskett. "There are some serious limitations to relying on tape. Some New Orleans companies found out too late that their 'off-site' tapes were stranded in the same city, surrounded by flood water for weeks."
Foskett suggests a DR plan designed around techniques like long-distance replication and clustering. These, however, are seriously expensive technologies, so DR planners must try to limit the amount of data that is afforded maximum protection.
Scott recommends organizations interested in replication take a closer look at the different ways replication can be achieved storage controller-based replication and host-based replication are just two possibilities, she says.
But replication and clustering are far from the only technologies required. Monsour adds vaulting technologies to the list of technologies needed to get businesses back up and running fast.
Longbottom includes back up power, closed system air conditioning (very expensive), water pumps, and air-lock entrance systems (to maintain standard pressure within the building).
Scott also touts the virtues of virtualization as a way to keep DR simple and the budget under control, particularly on Windows-based system.
"Virtualization helps you abstract the hardware and software so it doesn't matter if you are shifting data from Dell to HP boxes, for example," she says.
Those hoping a hurricane never pays them a visit won't be heartened by recent trends. Since 1995, the average number of major hurricanes has tripled and now averages four per year. 2006 should see five major ones when the season begins next month though the National Oceanic and Atmospheric Administration's National Hurricane Center and National Centers for Environmental Prediction may revise that figure when they release their official hurricane forecast on May 22.
"Apathy still exists, though every disaster raises the profile," says Scott. "Nobody should assume they won't be impacted. Executives need to budget for DR, since IT going down means the business may not survive."
This article was originally published on Enterprise Storage Forum.