When we discovered the source of the outage, we knew we were in trouble. Power outages are among the most common scenarios for disaster-recovery planning, but we hadn't planned for a trigger-happy CFO taking down our backup system. From the look on his face, I wasn't sure whether our CIO, Steve Fox, was just thinking about strangling Beane, or whether he was actually going to do it.
What Else Can Go Wrong?
We didn't know how long it might take to fix the transfer switch, so we swung into action, using our disaster-recovery plan as a guide. Following a meeting with my network manager, Dirk Packett, and my telecom manager, Sandra Hook, we were able to redirect incoming calls to one of our other call centers. This gave us a temporary workaround, but as the minutes ticked by, we noticed some glaring disconnects between our plan and the reality of the situation.
According to the plan, the downed call center was supposed to transfer operations to one of our warehouses several miles away. Unfortunately, most of the space at the warehouse had been converted into a repair lab since the last update of the recovery plan. There was some space left, so management decided we should move as many call-center employees into the backup location as possible.
The disaster-recovery plan didn't tell us where to find the computers for the backup location, so we began moving machines from the primary building to the backup site. This process took hours, and required us to scrounge up the necessary network hubs and switches. Luckily, the warehouse was cabled properly.
Discover the benefits of scalable backbone and network services for application and cloud providers.
Network automation is gaining momentum. Here's how you can drive this powerful technology to its full potential.
Subsea cable alternatives can provide a level of comfort for those concerned about disruptions. Realistically, they will continue to play a small, but critical role in the short term.
