You may have noticed that the WunderCounter was offline between about 1:43 pm and 10:46 pm today. This highly unusual stretch of downtime was the result of a fire in the building. As I understand it, the fire was on a different floor, but it did knock out the power at Cybercon. The backup power did kick in, but some glitch in the system prevented the backup power from properly coming online. The result was that all of my machines lost power at the same time. "Hard shutdowns" being what they are, not all of the machines came back online without some prompting. In a facility where hundreds of servers have been crashed at exactly the same time and initially having only weekend staff on hand, you can imagine what things were probably like.
Without giving you a complete blow by blow of how today has played out, let's just say that most (but not all) of my servers are now back online. Enough machines are online to run the WunderCounter, so that's good news indeed.
I'll be contacting the data centre on Monday to get a "post mortem" from them and to see what they intend to do to ensure that this doesn't happen again in future. I've been with them for about 9 years now and I don't recally this sort of downtime having occurred in the past, so this really is a rare occasion. They've actually been quite helpful with keeping the WunderCounter up and running with excellent uptime over the past years, so I think they deserve the benefit of the doubt.
I do sincerely apologize for any inconvenience this downtime may have caused you!