Tuesday, January 31, 2012

Network Issues (resolved)

It looks like things were resolved around 1:30 AM EST.  I'm told it was a bad network cable, which had actually been in place for years, that became unreliable and caused all of the issues.  That's certainly more convenient to fix than replacing expensive hardware, but it's maybe not the obvious first place to look.  In the 10+ years the WunderCounter has been online, I can't recall that a network cable was ever the root of a problem like this.

On a technical level, the ping times inside the data centre to the machine are now all under 5ms and there is no packet loss, so things should be as they normally were.  I apologize for any inconvenience this problem has caused.

Network Issues

There is currently an issue with the network which is causing the WunderCounter to load very slowly and occasionally not at all.  It's an intermittent problem and the folks in the data centre have been troubleshooting this for several hours now.  Sometimes the load times are lightning fast, so it's a bit of a head scratcher at this point.

The problem appears to be related to the load balancer which is in front of the rack, but this hasn't been established 100%.  The techs have ruled out the main switch as the cause of the problem and they're working to restore the network to the speed it should normally be at.

My apologies for the outages.  Since we can't say exactly what the issue is, it's hard to give a timeline for it being fixed.   My hope is, of course, that this is fixed very quickly.  I'll post an update here when I know that the problem has been resolved.