Rachel "Sparks" Blackman (seattlesparks) wrote,
Rachel "Sparks" Blackman

  • Mood:
  • Music:

Diary of a Tired Sysadmin

Keep in mind, all of this happens while still doing actual job-work.

August 22nd, 2003, mid-day: myth.legendary.org goes offline due to a mistake made (by me) while upgrading it. With no one there to reboot it, myth -- and Firan's ikonboard and backup system -- go down. Oh, well... we can wait until dpassmore gets home.

August 26th, 2003, afternoon: a huge power outage in Silver Spring takes down everything in the house. Worse still... no system except those with journaling filesystems come up properly. Peregrine (running Firan), Noderunner, Godlike and Iago.net all spend some time downed. Godlike comes back with the power, nothing else does.

August 26th, 2003, evening: Build a temporary refugee game and run it on pixel, my PowerBook, to have a place for people to gather for news and information. Put together glitch.noderunner.net -- a new FreeBSD server here on my home network.

August 27th, 2003: Build Refugee Station, a game for people to hang out on when other games are down. Holodeck rooms, game lounges, etc. dpassmor FedEx's a key to adamdray.

August 28th, 2003: Build an infrastructure on glitch.noderunner.net for backup services, to keep various things backed up. Set up Invision Power Board and MySQL, in preparation for setting up web forums for Twilight and to move Firan's forums to glitch and Invision Power Board. Key arrives via FedEx, in the evening. I catch the flu.

August 29th, 2003: adamdray goes to dpassmore's house with the key, and gets myth.legendary.org, foobar.noderunner.net both back online. I babysit services back into existence with foobar, and immediately try to do backups from myth. Peregrine, however, has a wonked filesystem. A backup from August 15th for Firan is available, but nothing more recent. I port Firan's mutant variant of TinyMUX to MacOS X again, get the August 15th database up and running on pixel, and write some tools to bring the database current without killing the server. Turn pixel (my PowerBook laptop) into a temporary server for Firan, to allow staff a place to meet.

August 30th, 2003: Set up redundancy for nameservices, e-mail, etc. More work on Refugee Station, etc.

August 31th, 2003: Foobar goes down. Again. It appears to be a file table overflow due to backed-up virus-generated e-mail hitting the mailserver all at once. Tweak the tools on Refugee Station to give everyone more tools to use to pass the time. Set up TwilightFalling.net for Twilight.

September 1st, 2003: adamdray goes back to dpassmor's house once more to try bringing Iago.net up now that he's been given the root password, and restarts foobar for me again. Talk him through a disk check. Copy ASiT and Twilight off of foobar and over to glitch, just in case. Get ASiT.Riverdark.net moved over for ASiT. Relax. dpassmor gets home.

September 2nd, 2003: Foobar goes down /again/, due to a catastrophic drive failure. dpassmor puts the drive in the freezer for me to get it to mount long enough to rescue. Luckily, I have a spare drive. dpassmor manages to recover the August 26th database for Firan, and rebuild Peregrine with a journaling filesystem, and I use the tools I had written to bring that database current and get Firan running again. Then we get foobar's drives mounted again, and I start a copy/backup of the root drive to the spare drive. I also back up the entire /data (user directories, websites, FTP sites, databases, etc.) directory, wipe the wonky VINUM partition we've been using for it, create a better partition, and copy everything back. Also attempt to re-do the boot configuration to boot off of the new drive.

And that's where we stand right now. I'm finishing the restore of /data and then I'll try rebooting foobar. If all goes well, we come back up tonight. If all does not go well, then I may have to impose on David to tweak the configuration a bit and get it up. Either way, everything's backed up.

We all owe both Adam and David major thanks, David especially.

So tired. Zzzzz...
  • Post a new comment


    Anonymous comments are disabled in this journal

    default userpic

    Your IP address will be recorded