So we haven't had too much downtime for a while, so today thought it'd be a good time to start.
Thanks for the mail FD, mail is pushed straight to my phone so I actually saw it straight away.. but using an ssh application on an iphone makes it fucking hard to do proper fault analysis!
Anyway, I rebooted the server and it's back online. Since then I've restarted everything properly again, making sure everything is clean as we were getting some database errors. Hopefully that's the last of that.
I have this horrible feeling that a server failure is imminent, we're due for something catastrophic to happen, which is especially annoying because we own the hardware these days. Maybe one of our friends in Florida will be able to go and do hardware replacements for us if it happens
Fortunately, we have backups...