Safe in the Eye of the Storm
Hurricane season officially begins June 1. The latest meteorological forecasts anticipate a busy hurricane season not as severe as last year's. Between four and six Category 3 or above hurricanes are expected. Bear in mind however, that 2005's prediction of major storms were way off the mark: Fifteen were predicted; 28 occurred. A few storage vendors came out of last year's devastating hurricane season unscathed. How did they do it?
Consequently, many organizations are scrambling to prepare. Spurred on by their Katrina experiences or those of some of their closest business partners, they are taking steps to safeguard their staff, hardware and data.
Ruden McClosky, for example, one of the largest law firms in Florida, with 10 offices across the state, suffered two direct hits in 2005 from hurricanes Katrina and Wilma. Its Fort Lauderdale office lost power for 12 hours during the former and was offline the entire day during the latter. Power remained spotty at best for several days, and a few windows were blown in.
"Compared to the office buildings next door, we fared relatively well," says Ben Weinberger, Ruden McClosky's IT director. "Two similar sized buildings next door lost over 80 percent of their windows, with furniture and files strewn about the streets outside."
XOsoft to the Rescue
The law firm minimized damage last year by failing over to its secondary data center. By using WANSyncHA software from XOsoft on the Sunday night prior to the storm, it experienced no downtime and no data loss for the offices and people who still had power. Almost all primary servers ran remotely and were fully accessible to attorneys in the firm's Tampa, St. Petersburg, Sarasota, Tallahassee, and Orlando offices via Citrix. The main exception was the voicemail system, which suffered some downtime. It ran on a single server in Fort Lauderdale without any failover options available.
What's the firm doing to ready itself for the coming season? A new voicemail system with failover capabilities is currently being evaluated. In the meantime, the company continues to revise its DR plan and hurricane planning. The goal is to improve communications with all firm personnel before, during and after an event.
"We do not have more formalized agreements for emergency replacement of workstations, phones and physical space, as traditionally we have relied upon additional capacity at each of our offices to be able to accommodate personnel from other offices, though we're looking into alternatives," says Weinberger. "In retrospect, given how well I feel we are protected now, at least with respect to our IT systems and data, it's hard to believe how unprotected we were before starting this project."
Another XOsoft user, Chaffe McCall, L.L.P., is the oldest law firm in Louisiana, with offices in New Orleans, Baton Rouge, and a smaller one in Houston. Last summer, it used a tape-based solution and wound up having to scramble to get the backup tapes to high ground after Katrina. This year, the firm has added a disk-based solution that automatically replicates mission essential data and e-mail to a backup server at a remote location. Further, the current disaster recovery plan allows the firm to remain up and running in the face of a natural disaster.
"We now replicate our SQL, Exchange and file servers to standby equipment in the Baton Rouge office," says James Zeller, senior network manager of Chaffe McCall. "We have also taken steps to try to ensure better communication between employees after evacuation by setting up an emergency Web site and alternate Web-based e-mail system for post-disaster communications, separate from our Exchange server."
A Long Recovery
Like Chaffe McCall, the University of New Orleans (UNO) also had to move fast to safeguard its backup tapes. One set was locked on the sixth floor of a building in downtown New Orleans. There was also a week-old backup on campus, locked in the computer room. The only problem the service provider had abandoned the downtown building and could not be contacted.
"Our campus was inaccessible for four weeks, and for that time we had no idea if any of our equipment had survived," says Jim Burgard, UNO's assistant vice chancellor for University Computing and Communications.
At the time of the disaster, UNO did not have a second site. It has built one as the result of its recovery efforts from Katrina. Louisiana State University (LSU) in Baton Rouge (80 miles away), provided eight servers and helped UNO set up its Web site. Today, the college has 25 servers running at LSU as its official secondary data center.
The site, however, is far from a mirror image of the main center. In New Orleans, UNO has an HP StorageWorks EVA 5000-based Fibre Channel SAN with 6 TB of data. It serves all mission-critical servers. Such a setup proved beyond the financial resources of the college. Instead, it opted for lower cost HP SCSI drives and cabinets and set them up in a RAID configuration to mirror key systems.
"We now have instant availability of our mission-critical applications such as Peoplesoft ERP, Internet, e-mail and SQL Server," says Burgard.
In addition, UNO is upgrading its UPS and adding a gas-fired generator during the summer so it has an alternate power supply on campus. Interestingly, he attributes the relatively unscathed state of his SAN and servers to the fact that he didn't attempt to make do with UPS during Katrina.
"We shut everything down, as we didn't want to rely on UPS when there was no one here to monitor it," says Burgard. "Other universities suffered significant damage after their UPS batteries went dead, and then the power came on and went off many times."
He sees the gas generator as a big asset during any future event. Last year, while power was down a couple of months after the hurricane, he noticed high humidity and worried about its impact on his hardware. Within a few weeks, UNO will be able to run its servers, SAN and AC despite city power being down.
While the secondary site at LSU is a start, the chancellor is concerned that the hot site is only 80 miles away. After all, it is conceivable that a bad storm could take out both facilities on the same day. Burgard, though, is relatively confident that there would be no flooding at Baton Rouge, but admits he could be left without power at both places.
The solution, therefore, will probably be to move the hot site further north to Shreveport, the northern edge of the state. Due to budgetary constraints, however, this may not be feasible. If so, UNO may well move a few essential components to Shreveport and leave the LSU site as is.
"At the very least we can put a domain name server, a domain controller and some kind of mirrored data storage in a more remote site," says Burgard. "That gives us the seed components to retain our e-mail, Web and data regardless of what happens."
A key lesson learned in the aftermath of Katrina, he notes, is the reliability of any tape backup supplier. He kept his tapes with a local firm, but during the storm, they disappeared and could not be contacted after the disaster so the backup tapes locked securely in the downtown facility proved useless in recovery. As a result, UNO has moved its offsite backup services to Iron Mountain due to its national presence and its ability to store tapes well away from the coast.
One big problem remains, however. UNO has yet to return to the number of IT staff it had prior to the hurricane. Burgard has struggled to return to the 45-man crew he once had.
"Recruitment is a challenge due to the uncertainty in the area concerning the levees and maintaining security," says Burgard. "In recent months we have lost two more people, added three and remain a few down."
This article was originally published on Enterprise Storage Forum.