Some of you may have noticed the recent server outage. This has been due to faulty server hardware. It all started when I decided to migrate the server to a virtual machine (I was tired of the inevitable boot failures when compiling new kernels and having to move the server to a desk with a monitor in order to fix it).
Anyway, the first step in migration is to make a backup. Half way through the backup, the hard drive died. Not only did this mean that I lost a considerable amount of data (not much irreplaceable, mostly just configuration and stats files), but I also had to quickly drive to a local store and buy an (overpriced) hard drive so that I could resume operations as quickly as possible. 48 sleepless hours later, I had the server back up and running with minimal data loss (thanks to my scheduled, remote backups). So, just like any sane person, I decided to take a backup of the virtual machine. Half way through the backup, the new hard drive failed.
I refuse to believe this is a coincidence, my conclusion is that either my server's IDE controller is failing and destroying hard drives, or the power supply unit isn't regulating voltage correctly and is frying the drive... or maybe even the drive is overheating under the strain of backing up data. Either way, I can no longer trust my Mac Mini as a server.
As a stopgap solution, I am using a virtual private server from Linode.com. I've got to say, I'm very impressed with their service. I'm not being paid to type this, but their VPSes are fast, reliable and cheap. The administration website is also superb. Setting up the server at Linode took a further 24+ hours, since I had to start from scratch again.
While Linode seems like the ideal solution, I don't like trusting third parties with my data (I also don't like subscription charges), so I've ordered a new load of hardware to construct a new server consisting of:
It's been a while since I bought any computer equipment (aside from my Macbook Air), but I find it pretty surprising at how cheap all of this is. When I built my first Mini-ITX system, it cost more than everything in this current order and was terribly under powered.
Notice that the new system has two hard drives - yes, I intend to RAID1 mirror my server now (granted, this wont save me from dodgy IDE controllers or PSUs, but it still makes sense).
Another option would have been the T7 ATOM 330 from TranquilPC, but this could only store 2x 2.5" hard drives, which would be more expensive. Had I not recently already bought a 5-Bay NAS, the BBS2 would also have been a possibility. I do, however, prefer to keep my server and my NAS separate.
Once the new hardware arrives (hopefully tomorrow), I'll get the fun task of migrating about 4GB worth of data from my Linode back home to my new server.
Lessons learned? Be damned sure to backup your "/etc" directory, unless you get pleasure from having to reconfigure EVERYTHING in the event of a hard drive failure. Luckily, I had everything else of importance backed up already.
All in all, this has been a pretty expensive and tiresome fiasco, and it's not even over yet!