Post Mortem of Last Week's Fire

FYI, here's the official response which Cybercon has provided me regarding last week's fire in the building and the outages it caused:

Cybercon Incident Report: Power Outage on June 6, 2009

1. Incident Date/Time: June 6, 2009 between 10:40am to 11:30am

2. Type of Incident: Power Outage

3. Incident Description: Around 10:40am both the Electric Utility company's (Ameren UE) Transformer fuse and Building's Automatic Transfer Switch's motor blew up.
As a result we lost the utility power and the generator didn't automatically kick in due to ATS motor failure.
Fire trucks were dispatched.

The building owner Digital Realty Trust (http://www.digitalrealtytrust.com/, a public company traded on New York Stock Exchange) manages the building power including the generators and Automatic Transfer Switches. After the fire department inspection, the building management switched the power to the backup generator manually.
Around 11:30am power was restored for all servers and equipment in our data center.
The generators provided power to our data center until around 5pm when utility power was restored.

4. Affected Services: The power outage affects all customer's servers and equipment powered by our Powerline A. Servers and equipment powered by our Powerline B were not affected. After power is restored around 11:30am, servers that failed to auto restart will have longer downtime. Customers who use Cybercon DNS servics incurred longer downtime due to firewall issues with our DNS servers. Due to DNS issues Cybercon SMC web portal failed to work which affect some customers. Cybercon's phone system is on UPS but not on the generator. The Phone system's UPS battery lasted about 30 minutes and died, causing the phone system to go down which affected some customers.

5. Action lists:

* The building management has restored power.

* Cybercon name servers are now on dual power sources (on both powerline A and Powerline B)

* Cybercon phone system are now on both UPS and generator.

* We will discuss with the building management (Digital Realty Trust) about installing the best of class Automatic Transfer Switch gears to prevent future failures of automatic failover to backup generators.

Please accept our apology for any trouble it caused to your company and your customers.