September 7th, 2002

FF Sparks (Casual)

RIP Noderunner...

The damage is more than I expected; the RAID array is completely shot. (Yes, I'm still stupefied as well as to how a RAID array can experience catestrophic failure.) I'm salvaging what I can...but it may be a wash. All the files are there, but we've lost all the names and all the directories of anything. It will take me weeks to put the server back together by hand, if I ever can.

Lest anyone blame the two who colocate the machine for me for free, it's my own damned fault; I should've found a way to make better offsite backup images, but twenty-six gigabytes of data is a lot to find any regular way of backing up. I should've noticed when one of the RAID plexes went stale, but it was warned about on the console and foobar is a headless machine in Maryland; I'm in Seattle. I never saw the console messages. I just...I should've paid closer attention rather than assuming since everything was working that everything was...well, /WORKING/.

In retrospect, my real mistake is that I was trying to run a machine with the amount of services of a small professional ISP, but never charged money for it. ISPs have catastrophic failures too, but they have money to pay to fix things or to get better equipment; having never taken in money for this, I have the money for neither, and so we end up with this.

This isn't to say that Noderunner's totally deceased. All the data /is/ there, just scattered in files now named for their disk location. I'll be looking at how to put things back together and fix up the RAID array later; I'm not going to let it go down completely. But I no longer have any idea when it will be back up.

I just...I feel physically ill. I may get offline for a while.
FF Sparks (Casual)

Saved.

dpassmor is my personal god. David managed to perform Scary RAID Voodoo and get back all the files.

This does not mean foobar is live again quite yet!

The drive is mounted read-only for the moment (though everyone can get to their files again). The RAID partition is being backed up offline to another machine. At that point, foobar will be taken down again and the RAID partition destroyed and reconfigured clean, and then all the data copied back. I am /also/ looking into how to set up an automated off-site backup for the ungodly amount of data we have. Everything will be live again before too long, however...and the machine /did/ come through intact.

Everyone owes David big, big thanks. If you're female and in the Silver Springs area of Maryland, I suggest a date or something as thank you to him! :)
  • Current Music
    Brent watching NASCAR in the other room