Saturday, June 6, 2009

June 6, 2009: Outages Caused by a Fire in the Building

You may have noticed that the WunderCounter was offline between about 1:43 pm and 10:46 pm today. This highly unusual stretch of downtime was the result of a fire in the building. As I understand it, the fire was on a different floor, but it did knock out the power at Cybercon. The backup power did kick in, but some glitch in the system prevented the backup power from properly coming online. The result was that all of my machines lost power at the same time. "Hard shutdowns" being what they are, not all of the machines came back online without some prompting. In a facility where hundreds of servers have been crashed at exactly the same time and initially having only weekend staff on hand, you can imagine what things were probably like.

Without giving you a complete blow by blow of how today has played out, let's just say that most (but not all) of my servers are now back online. Enough machines are online to run the WunderCounter, so that's good news indeed.

I'll be contacting the data centre on Monday to get a "post mortem" from them and to see what they intend to do to ensure that this doesn't happen again in future. I've been with them for about 9 years now and I don't recally this sort of downtime having occurred in the past, so this really is a rare occasion. They've actually been quite helpful with keeping the WunderCounter up and running with excellent uptime over the past years, so I think they deserve the benefit of the doubt.

I do sincerely apologize for any inconvenience this downtime may have caused you!

5 comments:

Anonymous said...

Hello, Can you help me?

I would like to know if there is any way of knowing who comments in my blog. For instance, what information can you collect from this comment that I'm making?

IP address? Country of origin? An actual address? Information on my ISP?

Thanks in advance

Brian

vilerichard said...

@Anonymous: there's no way to leave the tracking code on the add comment page, so I can really only go by the time the comment was made. Perhaps you are in Lisbon, PT? I won't post your entire IP, but it begins with 85.138.

Hayden56 said...

Hello! I have a wundercounter account that I pay for and have enjoyed using it a lot! I wanted to thank you for your quick posting of what happened the other day regarding the fire.

I am still having a issue with the counter since the fire, for some reason my hits are not showing up as highlighted and I use the counter on a lot of pages. I can't tell what pages they have hit because the new data hit is not highlighting the page where the hits were.

Could you check into this for me?

I also attempted to email you but it came back undeliverable three different times. I also called and left a message and am now posting in this blog!

Once again, I greatly appreciate your assistance as I love this program!!

Darla

vilerichard said...

@Hayden56 I did get your telephone message and thanks for reporting this. One other user has reported the same issue. The code has not changed since Saturday.

Let me just say that as far as the highlighting goes, the pages highlighted should be any page which has seen its count increased since the time of your previous login. If you log in twice close together, this may be why you don't see the highlighted pages.

It could also be a browser issue. Have you recently upgraded your browser? When you see the text on your dashboard which says pages with new data are highlighted, is this text highlighted in your browser?

vilerichard said...

@Hayden56 BTW, thanks for letting me know about the errors with the support form. That has now been fixed.