I was having some serious problems when an external USB hard disk stopped working. Considering this was a critical Redhat Enterprise 5 system, I needed to find a way to get it back online quick.
Redhat has an apparent issue, if it cannot mount a drive, it will halt the boot process until it has been either fixed or brought back online.
I mounted an external USB 1TB device for use as a backup drive. Apparently, this device stopped working (looking into that problem now), and Redhat could no longer find it. Because of this, my system stopped booting to the OS and put me automatically into a recovery mode.
At the moment, I am not able to gain physical access to the machine and unplug the faulty USB drive. So, here’s how I got around the problem.
At first I tried to simply disable the USB port in BIOS. Sure, this will disable the drive detection. But, apparently the drive had failed so it was already not being detected by Redhat. My problem was not related to detection of the drive.
After a bit of digging, I determined the system has a problem MOUNT a non-existent drive. It was giving me the following error: “fsck.ext3: No such file or directory while trying to open /dev/bookback”
The system put me into a recovery console. So, I thought I could edit /etc/fstab and simply remove the mount point from the list. However, when I tried this, I found that recovery console is READ ONLY. It would not let me edit the file.
To get around the problem, I ran the following command:
mount -w -o remount /
I then attempted to edit /etc/fstab again, and it worked!
After rebooting, my system is no longer searching to mount the USB backup device. This is an excellent short-term fix, considering that I am not physically on site. Apparently, if I was on site, I probably would need to either replace the USB device or run this same procedure anyway. My longer term solution will be to mount a more reliable backup solution. I will not connect an off-the-shelf USB drive again.
I also want to say that I was only able to get into my console by using HP’s iLO Advanced external control. Without this (or any other remote console ability), I would not be able to see the boot console.
Thanks for reading!
Special thanks to Arne B, sysadmin, for his help and diagnostics on this problem. I would not have been able to get this to work without his help.