kubicle

One fine day I get an email from an engineer. Apparently he removed glibc, no idea what he was doing but I was asked to somehow rebuild the box and preserve data. The problem was, we weren’t able to log into the system. Single user mode didn’t work either so now I’m stumped.

Here’s the message I kept getting on the console:

exec of init (/sbin/init) failed !!!: 2
umount /initrd/dev failed: 2
kernel panic – not syncing. Attempted to kill init!

Ok, no biggie, let’s slap another drive on the server, install linux and mount the other drives in order to copy the data off. Then it occurred to me, the server has 4 drive bays and has no room for an additional drive. Now what the hell do I do? Google search resulted in various “fixes,” but every one of them suggested that I copy the required folders/files from an identical box elsewhere, but I wouldn’t even know where to start. Ok, fair enough if he removed glibc, let’s try to reinstall glibc shall we? Before I reply and tell everyone the bad news, I attempted to boot off a rescue CD and see if I could figure out why it’s complaining. I attempted to chroot the /mnt/sysimage folder but nope I wasn’t able to – /bin/sh was missing. YOU HAVE TO BE KIDDING?!

Well, there goes my plan of reinstalling glibc. Then it hit me, all “rpm -ivh” does is extract the compress files into its appropriate directory. Let’s just extract them myself without having to run “rpm -ivh .” Luckily my RedHat DVDs are available via NFS/HTTP. I was able to download the packages one by one and extract the rpms as follows.

Note, I saved the RPMS under /mnt/sysimage (which is essentially your / folder under normal circumstances).

#shell>cd /mnt/sysimage
#shell>wget http://servername/packagename
#shell>pwd
/mnt/sysimage/
#shell>rpm2cpio PACKAGENAME | cpio -idmv

I had to run the last command for every dependencies for glibc. How did I figure that out? Once I downloaded glibc’s RPM, I ran “rpm -ivh,” which then gave me a list of the required dependencies. (Tip: run “rpm -ivh *.rpm –test” to make sure you satisfy all the dependecies) After I extracted all the RPMs, I rebooted and crossed my finger. Four minutes later I had a working system! I was able to SSH into the server and my engineer is happy once again.