Startup and Shutdown

[Dec 1, 2007]  SUSE Linux Enterprise troubleshooting Fixing boot problems by repairing a broken initrd by Sander van Vugt

Troubleshooting is a near-science by itself on which I could spend many articles, but I'll try to keep it brief. During the system boot procedure, several phases occur, starting in GRUB, the Linux boot loader. Roughly, these are the following:

1. GRUB loads the kernel
2. GRUB loads the initrd
3. The root file system is accessed by the kernel
4. The /sbin/init process takes over.
5. The initial boot stage happens
6. The default runlevel is activated
7. A login prompt occurs.

When a problem occurs, try to pin-point it to any of these seven phases. In some cases it is possible to tell exactly what happens, more often you will see that you can only give a rough indication of what is happening. In the case of a kernel panic, you can be sure about one thing: GRUB has loaded successfully and you are not yet at phase 4 of the boot procedure where the init process takes over. If a kernel panic occurs immediately after a driver installation, this is often caused by an error in the initrd.

How can we be sure? Sometimes it is quite obvious that the error is in initrd, as GRUB tells you that it failed to load the file /boot/initrd, in other cases some forensic work is needed as only a vague driver error message is generated. In the latter case, you have to check if the driver that fails is included in the initrd, as this helper file is used by the kernel to include drivers that are needed immediately. On SUSE Linux Enterprise, the file /etc/sysconfig/kernel contains a list of all drivers that should be included in the initrd. When you run the mkinitrd command, these drivers are written to your new initrd. When this happens automatically, something could go wrong.

Step 2: Fixing it
If an error occurs in the initrd, you will not be able to boot your server anymore. So, to fix it, you need the rescue system that is available from the installation dvd. This rescue system loads a complete Linux system off of the installation media. The next step is to mount all your Linux file systems off of that disk. Next, you need to run mkinitrd. You can only do this once the local file systems are all mounted, because the initrd has to be written to the local file systems. However, there is a caveat.

The problem with this approach is in the disk devices access in combination with the necessary use of a chroot environment. To start, you need to mount your server's file systems on a temporary mount point like /mnt. Let's say that you have the /boot directory on /dev/sda1 and your / directory on /dev/sda2. To mount them, you need the following two commands:

1. mount /dev/sda2 /mnt
2. mount /dev/sda1 /mnt/boot

Since the mkinitrd command wants to write the new initrd in /boot and the /boot on your hard drive is now in /mnt/boot, you need to change the root directory to be set to /mnt. You can use chroot to do that:

chroot /mnt

The contents of /mnt now becomes /, so all path references are OK. But we still have a problem. If you look in the /proc and /dev directory on your new root environment, you'll see that /proc is empty and /dev is as good as empty. Both are dynamically created file systems and they are created at the moment that your server boots. This means that they were created in / when the server booted from the rescue cd. Now, since the new root is in /mnt, you cannot access them anymore. We need to fix this.

1. Type exit to exit from the chroot environment. You'll now get back to the original /mnt under which your servers local file systems where mounted.
2. Use mount -t proc none /mnt/proc to make the proc file system available from the /mnt environment.
3. Use mount -o bind /dev /mnt/dev which will make the original /dev which was filled by the udev process when booting available from /mnt/dev.

Now that you have the repair environment all in place, you need to check that the line in /etc/sysconfig/kernel that is used to generate a new initrd is as it should be. You are looking for the following line:

INITRD_MODULES="ata_piix processor thermal fan jbd ext3 dm_mod edd pciback"

This line will be different on every server, so check to make sure that all modules are included that are necessary to start your server (your server's documentation will help you with that.)

Now under /mnt you have the complete environment that is needed to repair your server, so take the following two steps to fix your server.

1. Activate /mnt using cd /mnt and make it your new root environment using chroot .
2. Issue the command mkinitrd to write the new initrd to /boot.

You have now fixed the initrd. Reboot your server and check that everything is working all right. 

Recommended Links

Copyright © 1996-2009 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. Submit comments This document is an industrial compilation designed and created exclusively for educational use and is placed under the copyright of the Open Content License(OPL). Site uses AdSense so you need to be aware of Google privacy policy. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

Disclaimer:

• The statements, views and opinions presented on this web page are those of the author and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with.
• We do not warrant the correctness of the information provided or its fitness for any purpose
• In no way this site is associated with or endorse  cybersquatters using the term "softpanorama" with other main or country domains (e.g. softpanorama.com) with bad faith intent to profit from the goodwill belonging to someone else.

Last modified: August 21, 2009