Okay, core server was down for around about 6 hours again today. It's just come back online.
At this stage I don't know why it's going down (I can't find anything nasty in the OS or the hardware event logs) but I've at least fixed why it's not coming back up (I think DK broke it, I never configured disks to mount via UUID and the UUID for the images RAID device was wrong).
So yeah, at least when it goes down it should come back up within a few minutes.
And hopefully it'll stop doing that, because it's really fucking hard to work on a server that's 16,000km away.
thumb generation is broken for videos but they should still upload.. i'll take another look later.. nice catch on the footer that would of been broken for like 2-3 years now lol
Especially when we have absolutely NO idea why the server is crashing. Nothing in the logs on the server, nothing in the logs on the service processor .. nothing of any real use aside from a theory or two which is very difficult to test.
Oh well. We'll get there.... just may take a bit more downtime first. The REALLY annoying thing is that the server used to reboot, and die upon boot.. then I fixed the issue why it wasn't booting, so that when it happened the site was only offline for a short amount of time.
Now when it dies, it just .. dies. So it's offline until we notice, then we need to contact the colo company and get them to run a network cable to the service processor so we can investigate. Usually they're very quick with that, but when you request it at 2am Sunday morning (their time), it took them 7 hours to do.
It quite possibly is a hardware issue, but there's nothing we can do about it if it is. To start with the server is about 12,000 miles away from us, so we have no physical access, and the server is Sun hardware, parts are extremely expensive, and we can't just "try" replacing parts to see if it helps. We can't get physical access to the machine, so replacing hardware will be disastrously expensive.
To put it in prices anybody can understand, Sun quotes $1600 for a CPU (we've got 2), $1100 for a motherboard, $800 for 2G of RAM (we've got 8G in this server), around $750 for a hard drive (we've got 4), $600 for a power supply. On top of that, we have no support contract or anything so we'll be paying $200 an hour for somebody to do the actual work, we simply can't afford to start replacing parts without a reason to do so. And that reason would come in the form of logs, but nothing is logged either via the operating system or the service processor (which controls all the hardware, console, diagnostics, power etc).. And because the box has rebooted before we ever get a chance to see the console, we can't even see if there were any messages before it failed.
Right now, *(ROCK)*US*(HARD PLACE)* and I'm running out of ideas.
Posted by: quasi [x] - (64.215.250.---)
Date: June 08, 2008 12:51PM
I wrote a novel a few years back and decided to start another one. How's this for a beginning?
It was a hot, humid afternoon in South Florida, so hot and the air filled with so much moisture it was like trying to breathe bath water; not unusual for half the year there, but never welcome. It had been a stress filled day and my head pounded as if it was stopped at a traffic light next to some punk kid’s car that was beating out it’s hip-hop rhythms. I thanked whatever Gods prevailed as I stepped into the coolness of my air conditioned digs, and sat down at my computer to check out the latest on plus613 only to find that those Gods had deserted me after all; plus had gone down like a cheap hooker over on the eastside. “Damn,” I muttered as I trudged to the fridge for a cold Pepsi to drown my sorrows in, “what a world, what a fucked up world.”
Just kidding, guys, but this technology is, as I've said before, a house of cards, a very large & expensive house of cards but still a house of cards. Thanks for hangin' in there.
