[Clients] Server failure

Steve Hanson shanson at cruiskeenconsulting.com
Mon Sep 28 11:02:03 CDT 2009


Hi -
In the last 24 hours we have been hit with what amounts to a perfect 
storm of server problems, for which I apologize.

Yesterday (Sunday) afternoon, during what we thought was some simple 
server maintenance, I accidentally broke server miserver1.  Because of 
this, we lost a number of web sites, and one customer's email was down.  
Many of the web sites were brought back up during the afternoon and 
evening by moving them to server miserver2.

However, in the middle of doing THAT, our colocation provider had some 
severe networking problems, which made BOTH of the servers 
intermittently unavailable for two hours last night (this  not only made 
ALL the web sites intermittently available and at times completely 
unavailable, but of course also made it very difficult to restore sites, 
etc.).

However - as of 10:45 Central time, all services are available and 
normal again.  We're working on some changes in the infrastructure to 
spread server resources into a second data center so that we can recover 
more gracefully from these sorts of issues in the future.

Again, I apologize.  Those of you who are on paid support contracts will 
receive an extra free month of hosting on your contract to make up for 
this.  It's unacceptable, and we realize that.

Steve


More information about the Clients mailing list