Server Hard Drive Gets toasted

On saturday, I noticed that I could no longer check mail on my account. My mail server also has a couple of websites and a mysql database running on it. Those servers apeared to work fine. I opened a remote shell and trying looking through the qmail logs and the syslogs for errors by couldn't find any. Then I decided to forget it for a while cause I had more urgent matters to attend to (mail usually leads to more work anyway). That turned out to be a very bad decision cause soon after that SSH also died.

In the end I had to ask the web hosting company to restart the server. When it came back up apache was running and the machine could be pinged but SSH, POP, IMAP and SMTP were all dead.  Without SSH I was dependent on the hosting company to put it right and fortunately their support is excellent. ( It's a small outfit in California called  the Webquarry ). It turned out that one of the two hard drives on the server had just kicked off and died. Because the /home folder was on this drive SSH and the mail servers all stopped responding.

If /home was the only thing on it, we could have chucked it out and ignored a few emails that have gone missing and be done with it. Unfortunately the drive had some other stuff also on it which meant the OS was barely usable. So a new OS had to be put in (good thing too - it had Fedora Core 6 on it all this while). Then the sites had to be restored from the local copies and last but not least the mail accounts had to be recreated.

This toast hard drive has tought me several lessons though. The first and most important is that it really is time to move all the static content to Amazon S3 - it not only removes a huge load from the server(s) but also helps to make the hardware last longer; specially the hard drives. Hard drives on servers never get a moments rest and it's no suprise that they burn out.