Data Center Power Outage
Q: What happens what a rat gnaws through the wires leading to a live circuit breaker?
A: You get one very dead rat, and one very dark data center.
Six months ago this very scenario occurred, forcing us to tap into a “temporary” power feed. This past Saturday we finally began the process of moving back on to the main power supply. Working in conjunction with several groups, we managed to gracefully power down all 600+ servers, enabling a complete power outage of our building. General Electric was supposed to have their portion of this monumental undertaking done in 8 hours.
As I write this post it is now Sunday evening, and there’s no word on when GE will have the power back online. Needless to say, this does not bode well for anybody involved.
And so I sit here with phone in hand, waiting for the panicked “all hands on deck” call, ordering me into work to assist with bringing all of our systems back online.
…And this event was so well planned, unlike the initial unexpected rat BBQ that initiated the original emergency power cutover. This is shaping up to be one exciting week. *sigh*





