Unix Boot Process

The boot program is stored in a standard location on a bootable device. For a normal boot from disk, for example, the boot program might be located in block 0 of the root disk or, less commonly, in a special partition on the root disk. In the same way, the boot program may be the second file on a bootable tape or in a designated location on a remote file server in the case of a network boot of a diskless workstation.

There is usually more than one bootable device on a system. The firmware program may include logic for selecting the device to boot from, often in the form of a list of potential devices to examine. In the absence of other instructions, the first bootable device that is found is usually the one that is used.

The boot program is responsible for loading the Unix kernel into memory and passing control of the system to it. Some systems have two or more levels of intermediate boot programs between the firmware instructions and the independently-executing Unix kernel. Other systems use different boot programs depending on the type of boot.

PC systems follow this same basic procedure. When the power comes on or the system is reset, the BIOS starts the master boot program, located in the first 512 bytes of the system disk. This program then typically loads the boot program located in the first 512 bytes of the active partition on that disk, which then loads the kernel. Sometimes, the master boot program loads the kernel itself. The boot process from other media is similar.

The firmware program is basically just smart enough to figure out if the hardware devices it needs are accessible (e.g., can it find the system disk or the network) and to load and initiate the boot program. This first-stage boot program often performs additional hardware status verification, checking for the presence of expected system memory and major peripheral devices. Some systems do much more elaborate hardware checks, verifying the status of virtually every device and detecting new ones added since the last boot.

The kernel is the part of the Unix operating system that remains running at all times when the system is up. The kernel executable image itself, conventionally named unix (System V-based systems), vmunix (BSD-based system), or something similar. It is traditionally stored in or linked to the root directory. Here are typical kernel names and directory locations for the various operating systems we are considering:

Once control passes to the kernel, it prepares itself to run the system by initializing its internal tables, creating the in-memory data structures at sizes appropriate to current system resources and kernel parameter values. The kernel may also complete the hardware diagnostics that are part of the boot process, as well as installing loadable drivers for the various hardware devices present on the system.

When these preparatory activities have been completed, the kernel creates another process that will run the init program as the process with PID 1.

^[4] Process 0, if it exists, is really part of the kernel itself. Process 0 is often the scheduler (controls which processes execute at what time under BSD) or the swapper (moves process memory pages to and from swap space under System V). However, some systems assign PID 0 to a different process, and others do not have a process 0 at all.

Booting to Multiuser Mode

As we've seen, init is the ancestor of all subsequent Unix processes and the direct parent of user login shells. During the remainder of the boot process, init does the work needed to prepare the system for users.

One of init's first activities is to verify the integrity of the local filesystems, beginning with the root filesystem and other essential filesystems, such as /usr. Since the kernel and the init program itself reside in the root filesystem (or sometimes the /usr filesystem in the case of init), you might wonder how either one can be running before the corresponding filesystem has been checked. There are several ways around this chicken-and-egg problem. Sometimes, there is a copy of the kernel in the boot partition of the root disk as well as in the root filesystem. Alternatively, if the executable from the root filesystem successfully begins executing, it is probably safe to assume that the file is OK.

In the case of init, there are several possibilities. Under System V, the root filesystem is mounted read-only until after it has been checked, and init remounts it read-write. Alternatively, in the traditional BSD approach, the kernel handles checking and mounting the root filesystem itself.

Still another method, used when booting from tape or CD-ROM (for example, during an operating system installation or upgrade), and on some systems for normal boots, involves the use of an in-memory (RAM) filesystem containing just the limited set of commands needed to access the system and its disks, including a version of init. Once control passes from the RAM filesystem to the disk-based filesystem, the init process exits and restarts, this time from the "real" executable on disk, a result that somewhat resembles a magician's sleight-of-hand trick.

Other activities performed by init include thefollowing:

• Checking the integrity of the filesystems, traditionally using the fsck utility

• Mounting local disks

• Designating and initializing paging areas

• Performing filesystem cleanup activities: checking disk quotas, preserving editor recovery files, and deleting temporary files in /tmp and elsewhere

• Starting system server processes (daemons) for subsystems like printing, electronic mail, accounting, error logging, and cron

• Starting networking daemons and mounting remote disks

• Enabling user logins, usually by starting getty processes and/or the graphical login interface on the system console (e.g., xdm), and removing the file /etc/nologin, if present

These activities are specified and carried out by means of the system initialization scripts , shell programs traditionally stored in /etc or /sbin or their subdirectories and executed by init at boot time. These files are organized very differently under System V and BSD, but they accomplish the same purposes. They are described in detail later in this chapter.

Once these activities are complete, users may log in to the system. At this point, the boot process is complete, and the system is said to be in multiuser mode.

Booting to Single-User Mode

Once init takes control of the booting process, it can place the system in single-user mode instead of completing all the initialization tasks required for multiuser mode. Single-user mode is a system state designed for administrative and maintenance activities, which require complete and unshared control of the system. This system state is selected by a special boot command parameter or option; on some systems, the administrator may select it by pressing a designated key at a specific point in the boot process.

To initiate single-user mode, init forks to create a new process, which then executes the default shell (usually /bin/sh) as user root. The prompt in single-user mode is the number sign (#), the same as for the superuser account, reflecting the root privileges inherent in it. Single-user mode is occasionally called maintenance mode.

Another situation in which the system might enter single-user mode automatically occurs if there are any problems in the boot process that the system cannot handle on its own. Examples of such circumstances include filesystem problems that fsck cannot fix in its default mode and errors in one of the system initialization files. The system administrator must then take whatever steps are necessary to resolve the problem. Once this is done, booting may continue to multiuser mode by entering CTRL-D, terminating the single-user mode shell:

# ^D                             Continue boot process to multiuser mode. 
Tue Jul 14 14:47:14 EDT 1987     Boot messages from the initialization files. 
 . . . 

Alternatively, rather than picking up the boot process where it left off, the system may be rebooted from the beginning by entering a command such as reboot (AIX and FreeBSD) or telinit 6. HP-UX supports both commands.

Single-user mode represents a minimal system startup. Although you have root access to the system, many of the normal system services are not available at all or are not set up. On a mundane level, the search path and terminal type are often not set correctly. Less trivially, no daemons are running, so many Unix facilities are shut down (e.g., printing). In general, the system is not connected to the network. The available filesystems may be mounted read-only, so modifying files is initially disabled (we'll see how to overcome this in a bit). Finally, since only some of the filesystems are mounted, only commands that physically reside on these filesystems are available initially.

This limitation is especially noticeable if /usr was created on a separate disk partition from the root filesystem and is not mounted automatically under single-user mode. In this case, even commands stored in the root filesystem (in /bin, for example) will not work if they use shared libraries stored under /usr. Thus, if there is some problem with the /usr filesystem, you will have to make do with the tools that are available. For such situations, however rare and unlikely, you should know how to use the ed editor if vi is not available in single-user mode; you should know which tools are available to you in that situation before you have to use them.

On a few systems, vendors have exacerbated this problem by making /bin a symbolic link to /usr/bin, thereby rendering the system virtually unusable if there is a problem with a separate /usr filesystem.

Password protection for single-user mode

On older Unix systems, single-user mode does not require a password be entered to gain access. Obviously, this can be a significant security problem. If someone gained physical access to the system console, he could crash it (by hitting the reset button, for example) and then boot to single-user mode via the console and be automatically logged in as root without having to know the root password.

Modern systems provide various safeguards. Most systems now require that the root password be entered before granting system access in single-user mode. On some System V-based systems, this is accomplished via the sulogin program that is invoked automatically by init once the system reaches single-user mode. On these systems, if the correct root password is not entered within some specified time period, the system is automatically rebooted

Here is a summary of single-user mode password protection by operating system:

console none unknown off insecure

sp:S:respawn:/sbin/sulogin

Firmware passwords

Some systems also allow you to assign a separate password to the firmware initialization program, preventing unauthorized persons from booting the computer.

For example, on SPARC systems, the eeprom command may be used to require a password and set its value (via the security-mode and security-password parameters, respectively).

On PC-based systems, the BIOS monitor program must generally be used to set such a password. It is accessed by pressing a designated key (often F1 or F8) shortly after the system powers on or is reset.

On Linux systems, commonly used boot-loader programs have configuration settings that accomplish the same purpose. Here are some configuration file entries for lilo and grub:

password = something           /etc/lilo.conf
password -md5 xxxxxxxxxxxx     /boot/grub/grub.conf

The grub package provides the grub-md5-crypt utility for generating the MD5 encoding for a password. Linux boot loaders are discussed in detail in Chapter 16.

Starting a Manual Boot

Virtually all modern computers can be configured to boot automatically when power comes on or after a crash. When autobooting is not enabled, booting is initiated by entering a simple command in response to a prompt: sometimes just a carriage return, sometimes a b, sometimes the word boot. When a command is required, you often can tell the system to boot to single-user mode by adding a -s or similar option to the boot command, as in these examples from a Solaris and a Linux system:

ok boot -s             Solaris 
boot: linux single     Linux

In the remainder of this section, we will look briefly at the low-level boot commands for our supported operating systems.

AIX

AIX provides little in the way of administrator intervention options during the boot process.However, the administrator does have the ability to preconfigure the boot process in two ways.

Some AIX systems respond to a specific keystroke at a precise moment during the boot process and place you in the System Management Services facility, where the boot device list can also be specified.

The first is to use the bootlist command to specify the list and ordering of boot devices for either normal boot mode or service mode. For example, this command makes the CD-ROM drive the first boot device for the normal boot mode:

# bootlist -m normal cd1 hdisk0 hdisk1 rmt0

If there is no bootable CD in the drive, the system next checks the first two hard disks and finally the first tape drive.

The second configuration option is to use the diag utility to specify various boot process options, including whether or not the system should boot automatically in various circumstances. These items are accessed via the Task Selection submenu.

FreeBSD

FreeBSD (on Intel systems) presents a minimal boot menu:

F1  FreeBSD
F2  FreeBSD
F5  Drive 1     Appears if there is a second disk with a bootable partition.

This menu is produced by the FreeBSD boot loader (installed automatically if selected during the operating system installation, or installed manually later with the boot0cfg command). It simply identifies the partitions on the disk and lets you select the one from which to boot. Be aware, however, that it does not check whether each partition has a valid operating system on it (see Chapter 16 for ways of customizing what is listed).

The final option in the boot menu allows you to specify a different disk (the second IDE hard drive in this example). If you choose that option, you get a second, similar menu allowing you to select a partition on that disk:

F1  FreeBSD
F5  Drive 0

In this case, the second disk has only one partition.

Shortly after selecting a boot option, the following message appears:^[7]

^[7] We're ignoring the second-stage boot loader here.

Hit [Enter] to boot immediately, or any other key for the command prompt

If you strike a key, a command prompt appears, from which you can manually boot, as in these examples:

disk1s1a:> boot -s             Boot to single-user mode

disk1s1a:> unload              Boot an alternate kernel
disk1s1a:> load kernel-new
disk1s1a:> boot

If you do not specify a full pathname, the alternate kernel must be located in the root directory on the disk partition corresponding to your boot menu selection.

FreeBSD can also be booted by the grub open source boot loader, which is discussed—along with a few other boot loaders—in the Linux section below.

HP-UX

HP-UX boot commands vary by hardware type. These examples are from an HP 9000/800 system. When power comes on initially, the greater-than-sign prompt (>)^[8] is given when any key is pressed before the autoboot timeout period expires. You can enter a variety of commands here. For our present discussion, the most useful are search (to search for bootable devices) and co (to enter the configuration menu). The latter command takes you to a menu where you can specify the standard and alternate boot paths and options. When you have finished with configuration tasks, return to the main menu (ma) and give the reset command.

^[8] Preceded by various verbiage.

Alternatively, you can boot immediately by using the bo command, specifying one of the devices that search found by its two-character path number (given in the first column of the output). For example, the following command might be used to boot from CD-ROM:

> bo P1

The next boot phase involves loading and running the initial system loader (ISL). When it starts, it asks whether you want to enter commands with this prompt:

Interact with ISL? y

If you answer yes, you will receive the ISL> prompt, at which you can enter various commands to modify the usual boot process, as in these examples:

ISL> hpux -is                   Boot to single user mode
ISL> hpux /stand/vmunix-new     Boot an alternate kernel
ISL> hpux ll /stand             List available kernels

Linux

When using lilo, the traditionalLinux boot loader, the kernels available for booting are predefined. When you get lilo's prompt, you can press the TAB key to list the available choices. If you want to boot one of them into single-user mode, simply add the option single (or -s) to its name. For example:

boot: linux single

You can specify kernel parameters generally by appending them to the boot selection command.

If you are using the newer grub boot loader, you can enter boot commands manually instead of selecting one of the predefined menu choices, by pressing the c key. Here is an example sequence of commands:

grub> root (hd0,0)                              Location of /boot
grub> kernel /vmlinuz=new ro root=/dev/hda2
grub> initrd /initrd.img
grub> boot

The root option on the kernel command locates the partition where the root directory is located (we are using separate / and /boot partitions here).

If you wanted to boot to single-user mode, you would add single to the end of the kernel command.

In a similar way, you can boot one of the existing grub menu selections in single-user mode by doing the following:

1. Selecting it from the menu

2. Pressing the e key to edit it

3. Selecting and editing the kernel command, placing single at the end of the line

4. Moving the cursor to the first command and then pressing b for boot

On non-Intel hardware, the boot commands are very different.

Solaris

At power-on, Solaris systems may display the ok console prompt. If not, it is because the system is set to boot automatically, but you can generate one with the Stop-a or L1-a key sequence. From there, the boot command may be used to initiate a boot, as in this example:

ok boot -s        Boot to single user mode
ok boot cdrom     Boot from installation media

The second command boots an alternate kernel by giving its full drive and directory path. You can determine the available devices and how to refer to them by running the devalias command at the ok prompt.

Booting from alternate media

Booting from alternate media, such as CD-ROM or tape, is no different from booting any other non-default kernel. On systems where this is possible, you can specify the device and directory path to the kernel to select it. Otherwise, you must change the device boot order to place the desired alternate device before the standard disk location in the list.

Boot Activities in Detail

We now turn to a detailed consideration of theboot process from the point of kernel initialization onward.

Boot messages

The following example illustrates a generic Unix startup sequence. Themessages included here are a composite of those from several systems, although the output is labeled as for a mythical computer named the Urizen, a late-1990s system running a vaguely BSD-style operating system. While this message sequence does not correspond exactly to any existing system, it does illustrate the usual elements of booting on Unix systems, under both System V and BSD.

We've annotated the boot process output throughout:

> b                                             Initiate boot to multiuser mode. 
Urizen Ur-Unix boot in progress...    
testing memory                                  Output from boot program. 
checking devices                                Preliminary hardware tests. 
loading vmunix                                  Read in the kernel executable.

Urizen Ur-Unix Version 17.4.2: Fri Apr 24 23 20:32:54 GMT 1998 
Copyright (c) 1998 Blakewill Computer, Ltd.     Copyright for OS. 
Copyright (c) 1986 Sun Microsystems, Inc.       Subsystem copyrights.
Copyright (c) 1989-1998 Open Software Foundation, Inc. 
... 
Copyright (c) 1991 Massachusetts Institute of Technology 
All rights reserved.                            Unix kernel is running now.
    
physical memory = 2.00 GB                       Amount of real memory. 

Searching SCSI bus for devices:                 Peripherals are checked next. 
rdisk0 bus 0 target 0 lun 0 
rdisk1 bus 0 target 1 lun 0 
rdisk2 bus 0 target 2 lun 0 
rmt0 bus 0 target 4 lun 0 
cdrom0 bus0 target 6 lun 0 
Ethernet address=8:0:20:7:58:jk                 Ethernet address of network adapter.

Root on /dev/disk0a                             Indicates disk partitions used as /, . . . 
Activating all paging spaces                     . . . as paging spaces and . . . 
swapon: swap device /dev/disk0b activated. 
Using /dev/disk0b as dump device                 . . . as the crash dump location.
    
                                                Single-user mode could be entered here, . . . 
INIT: New run level: 3                           . . . but this system is booting to run level 3. 
                                                Messages produced by startup scripts follow.
The system is coming up. Please wait.           Means "Be patient."
Tue Jul 14 14:45:28 EDT 1998    
    
Checking TCB databases                          Verify integrity of the security databases. 
Checking file systems:                          Check and mount remaining local filesystems. 
fsstat: /dev/rdisk1c (/home) umounted cleanly;  Skipping check. 
fsstat: /dev/rdisk2c (/chem) dirty              This filesystem needs checking.
Running fsck: 
/dev/rdisk2c: 1764 files, 290620 used, 110315 free 
Mounting local file systems.

Checking disk quotas: done.                     Daemons for major subsystems start first, . . . 
cron subsystem started, pid = 3387 
System message logger started. 
Accounting services started.
                                                 . . . followed by network servers, . . . 
Network daemons started: portmap inetd routed named rhwod timed. 
NFS started: biod(4) nfsd(6) rpc.mountd rpc.statd rpc.lockd. 
Mounting remote file systems. 
Print subsystem started.                         . . . and network-dependent local daemons.
sendmail started.

Preserving editor files.                        Save interrupted editor sessions. 
Clearing /tmp.                                  Remove files from /tmp.
Enabling user logins.                           Remove the /etc/nologin file. 
Tue Jul 14 14:47:45 EDT 1998                    Display the date again.
    
Urizen Ur-Unix 9.1 on hamlet                    The hostname is hamlet.
    
login:                                          Unix is running in multiuser mode.

There are some things that are deliberately anachronistic about this example boot sequence—running fsck and clearing /tmp, for instance—but we've retained them for nostalgia's sake. We'll consider the scripts and commands that make all of these actions happen in the course of this section.

boot log files

Most Unix versions automatically save some or all of the bootmessages from the kernel initialization phase to a log file. The system message facility, controlled by the syslogd daemon, and the related System V dmesg utility are often used to capture messages from the kernel during a boot (syslog is discussed in detail Chapter 3). In the latter case, you must execute the dmesg command to view the messages from the most recent boot. On FreeBSD systems, you can also view them in the /var/run/dmesg.boot file.

It is common for syslogd to maintain only a single message log file, so boot messages may be interspersed with system messages of other sorts. The conventional message file is /var/log/messages.

The syslog facility under HP-UX may also be configured to produce a messages file, but it is not always set up at installation to do so automatically. HP-UX also provides the /etc/rc.log file, which stores boot output from the multiuser phase.

Under AIX, /var/adm/ras/bootlog is maintained by the alog facility. Like the kernel buffers that are its source, this file is a circular log that is maintained at a predefined fixed size; new information is written at the beginning of the file once the file is full, replacing the older data. You can use a command like this one to view the contents of this file:

# alog -f /var/adm/ras/bootlog -o

init

In general, init controls the multiuser mode boot process. init runs whatever initialization scripts it has been designed to run, and the structure of the init program determines the fundamental design of the set of initialization scripts for that Unix version: what the scripts are named, where they are located in the filesystem, the sequence in which they are run, the constraints placed upon the scripts' programmers, the assumptions under which they operate, and so on. Ultimately, it is the differences in the System V and BSD versions of init that determines the differences in the boot process for the two types of systems.

Although we'll consider those differences in detail later, in this section, we'll begin by looking at the activities that are part of every normal Unix boot process, regardless of the type of system. In the process, we'll examine sections of initialization scripts from a variety of different computer systems.

System initialization scripts usually perform a few preliminary actions before getting down to the work of booting the system. These include defining any functions and local variables that may be used in the script and setting up the script's execution environment, often beginning by defining HOME and PATH environment variables:

HOME=/; export HOME 
PATH=/bin:/usr/bin:/sbin:/usr/sbin; export PATH

The path is deliberately set to be as short as possible; generally, only system directories appear in it to ensure that only authorized, unmodified versions of commands get executed

Alternatively, other scripts are careful always to use full pathnames for every command that they use. However, since this may make commands excessively long and scripts correspondingly harder to read, some scripts take a third approach and define a local variable for each command that will be needed at the beginning of the script:

mount=/sbin/mount 
fsck=/sbin/fsck 
rm=/usr/bin/rm 
...

The commands would then be invoked in this way:

${rm} -f /tmp/*

This practice ensures that the proper version of the command is run while still leaving the individual command lines very readable.

Whenever full pathnames are not used, we will assume that the appropriate PATH has previously been set up in the script excerpts we'll consider.

Preparing filesystems

Preparing thefilesystem for use is the first and most important aspect of the multiuser boot process. It naturally separates into two phases: mounting the root filesystem and other vital system filesystems (such as /usr), and handling the remainder of the local filesystems.

Filesystem checking is one of the key parts of preparing the filesystem. This task is the responsibility of the fsck utility.

Variously pronounced as "fisk" (like the baseball player Carlton, rhyming with "disk"), "ef-es-see-kay," "ef-es-check," and in less genteel ways.

For traditional Unix filesystem types (such as ufs under FreeBSD and ext2 under Linux), fsck's job is to ensure that the data structures in the disk partition's superblock and inode tables are consistent with the filesystem's directory entries and actual disk block consumption. It is designed to detect and correct inconsistencies between them, such as disk blocks marked as in use that are not claimed by any file, and files existing on disk that are not contained in any directory. fsck deals with filesystem structure, but not with the internal structure or contents of any particular file. In this way, it ensures filesystem-level integrity, not data-level integrity.

In most cases, the inconsistencies that arise are minor and completely benign, and fsck can repair them automatically at boot time. Occasionally, however, fsck finds more serious problems, requiring administrator intervention.

If the system is rebooting after a crash, it is quite normal to see many messages indicating minor filesystem discrepancies that have been repaired. By default, fsck fixes problems only if the repair cannot possibly result in data loss. If fsck discovers a more serious problem with the filesystem, it prints a message describing the problem and leaves the system in single-user mode; you must then run fsck manually to repair the damaged filesystem. For example (from a BSD-style system):

/dev/disk2e: UNEXPECTED INCONSISTENCY; 
RUN fsck MANUALLY                        Message from fsck.
Automatic reboot failed . . . help!      Message from /etc/rc script.
Enter root password:                     Single-user mode.
# /sbin/fsck -p /dev/disk2e              Run fsck manually with -p.
...                                      Many messages from fsck.
BAD/DUP FILE=2216 OWNER=190 M=120777     Mode=> file is a symbolic link, so deleting it is safe.
S=16 MTIME=Sep 16 14:27 1997
CLEAR? y 
*** FILE SYSTEM WAS MODIFIED ***
# ^D                                     Resume booting.  
Mounting local file systems.             Normal boot messages
...

In this example, fsck found a file whose inode address list contained duplicate entries or addresses of known bad spots on the disk. In this case, the troublesome file was a symbolic link (indicated by the mode), so it could be safely removed (although the user who owned it will need to be informed). This example is intended merely to introduce you to fsck;

Checking and mounting the root filesystem

The root filesystem is the first filesystem that the boot process accesses as it prepares the system for use. On a System V system, commands like these might be used to check the root filesystem, if necessary:

/sbin/fsstat ${rootfs} >/dev/null 2>&1 
if [ $? -eq 1 ] ; then 
    echo "Running fsck on the root file system." 
    /sbin/fsck -p ${rootfs} 
fi

The shell variable rootfs has been defined previously as the appropriate special file for the root filesystem. The fsstat command determines whether a filesystem is clean (under HP-UX, fsclean does the same job). If it returns an exit value of 1, the filesystem needs checking, and fsck is run with its -p option, which says to correct automatically all benign errors that are found.

On many systems, the root filesystem is mounted read-only until after it is known to be in a viable state as a result of running fsstat and fsck as needed. At that point, it is remounted read-write by the following command:

# mount -o rw,remount /

On FreeBSD systems, the corresponding command is:

# mount -u -o rw /

Preparing other local filesystems

The traditional BSD approach to checking the filesystems is to check all of them via a single invocation of fsck (although the separate filesystems are not all checked simultaneously), and some System V systems have adopted this method as well. The initialization scripts on such systems include a fairly lengthy case statement, which handles the various possible outcomes of the fsck command:

/sbin/fsck -p 
retval=$? 
case $retval in                                  Check fsck exit code. 
0)                                               No remaining problems,
   ;;                                               so just continue the boot process 
4)                                               fsck fixed problems on root disk. 
   echo "Root file system was modified." 
   echo "Rebooting system automatically." 
   exec /sbin/reboot -n 
   ;; 
8)                                               fsck failed to fix filesystem. 
   echo "fsck -p could not fix file system."
   echo "Run fsck manually."    
   ${single}                                     Single-user mode. 
   ;; 
12)                                              fsck exited before finishing. 
   echo "fsck interrupted ... run manually." 
   ${single}
   ;; 
*)                                               All other fsck errors.
   echo "Unknown error in fsck." 
   ${single}
   ;; 
esac

This script executes the command fsck -p to check the filesystem's consistency. The -p option stands for preen and says that any needed repairs that will cause no loss of data should be made automatically. Since virtually all repairs are of this type, this is a very efficient way to invoke fsck. However, if a more serious error is found, fsck asks whether to fix it. Note that the options given to fsck may be different on your system.

Next, the case statement checks the status code returned by fsck (stored in the local variable retval) and performs the appropriate action based on its value.

If fsck cannot fix a disk on its own, you need to run it manually when it dumps you into single-user mode. Fortunately, this is rare. That's not just talk, either. I've had to run fsck manually only a handful of times over the many hundreds of times I've rebooted Unix systems, and those times occurred almost exclusively after crashes due to electrical storms or other power loss problems. Generally, the most vulnerable disks are those with continuous disk activity. For such systems, a UPS device is often a good protection strategy.

Once all the local filesystems have been checked (or it has been determined that they don't need to be), they can be mounted with the mount command, as in this example from a BSD system:

mount -a -t ufs

mount's -a option says to mount all filesystems listed in the system's filesystem configuration file, and the -t option restricts the command to filesystems of the type specified as its argument. In the preceding example, all ufs filesystems will be mounted. Some versions of mount also support a nonfs type, which specifies all filesystems other than those accessed over the network with NFS.

Saving a crash dump

When a system crashes due to an operating system-level problem, most Unix versions automatically write the current contents of kernel memory—known as a crash dump—to a designated location, usually the primary swap partition. AIX lets you specify the dump location with the sysdumpdev command, and FreeBSD sets it via the dumpdev parameter in /etc/rc.conf. Basically, a crash dump is just a core dump of the Unix kernel, and like any core dump, it can be analyzed to figure out what caused the kernel program—and therefore the system—to crash.

Since the swap partition will be overwritten when the system is booted and paging is restarted, some provision needs to be made to save its contents after a crash. The savecore command copies the contents of the crash dump location to a file within the filesystem. savecore exits without doing anything if there is no crash dump present. The HP-UX version of this command is called savecrash.

savecore is usually executed automatically as part of the boot process, prior to the point at which paging is initiated:

savecore /var/adm/crash

savecore's argument is the directory location to which the crash dump should be written; /var/adm/crash is a traditional location. On Solaris systems, you can specify the default directory location with the dumpadm command.

The crash dumps themselves are conventionally a pair of files named something like vmcore.n (the memory dump) and kernel.n, unix.n, or vmunix.n (the running kernel), where the extension is an integer that is increased each time a crash dump is made (so that multiple files may exist in the directory simultaneously). Sometimes, additional files holding other system status information are created as well.

HP-UX creates a separate subdirectory of /var/adm/crash for each successive crash dump, using names of the form crash.n. Each subdirectory holds the corresponding crash data and several related files.

The savecore command is often disabled in the delivered versions of system initialization files since crash dumps are not needed by most sites. You should check the files on your system if you decide to use savecore to save crash dumps.

Starting paging

Once the filesystem is ready and any crash dump has been saved,paging can be started. This normally happens before the major subsystems are initialized since they might need to page, but the ordering of the remaining multiuser mode boot activities varies tremendously.

Paging is started by the swapon -a command, which activates all the paging areas listed in the filesystem configuration file.

Security-related activities

Another important aspect of preparing the system for users is ensuring that available security measures are in place and operational. Systems offering enhanced security levels over the defaults provided by vanilla Unix generally include utilities to verify the integrity of system files and executables themselves. Like their filesystem-checking counterpart fsck, these utilities are run at boot time and must complete successfully before users are allowed access to the system.

In a related activity, initialization scripts on many systems often try to ensure that there is a valid password file (containing the system's user accounts). These Unix versions provide the vipw utility for editing the password file. vipw makes sure that only one person edits the password file at a time. It works by editing a copy of the password file; vipw installs it as the real file after editing is finished. If the system crashes while someone is running vipw, however, there is a slight possibility that the system will be left with an empty or nonexistent password file, which significantly compromises system security by allowing anyone access without a password.

Commands such as these are designed to detect and correct such situations:

if [ -s /etc/ptmp ]; then                             Someone was editing /etc/passwd. 
   if [ -s /etc/passwd ]; then                        If passwd is non-empty, use it . . . 
      ls -l /etc/passwd /etc/ptmp >/dev/console 
      rm -f /etc/ptmp                                  . . . and remove the temporary file. 
   else                                               Otherwise, install the temporary file. 
      echo 'passwd file recovered from /etc/ptmp' 
      mv /etc/ptmp /etc/passwd 
   fi 
elif [ -r /etc/ptmp ]; then                           Delete any empty temporary file. 
    echo 'removing passwd lock file' 
    rm -f /etc/ptmp 
fi

The password temporary editing file, /etc/ptmp in this example, also functions as a lock file. If it exists and is not empty (-s checks for a file of greater than zero length), someone was editing /etc/passwd when the system crashed or was shut down. If /etc/passwd exists and is not empty, the script assumes that it hasn't been damaged, prints a long directory listing of both files on the system console, and removes the password lock file. If /etc/passwd is empty or does not exist, the script restores /etc/ptmp as a backup version of /etc/passwd and prints the message "passwd file recovered from /etc/ptmp" on the console.

The elif clause handles the case where /etc/ptmp exists but is empty. The script deletes it (because its presence would otherwise prevent you from using vipw) and prints the message "removing passwd lock file" on the console. Note that if no /etc/ptmp exists at all, this entire block of commands is skipped.

Checking disk quotas

Most Unix systems offer an optional disk quota facility, which allows the available disk space to be apportioned among users as desired. It, too, depends on database files that need to be checked and possibly updated at boot time, via commands like these:

echo "Checking quotas: \c" 
quotacheck -a 
echo "done." 
quotaon -a

The script uses the quotacheck utility to check the internal structure of all disk quota databases, and then it enables disk quotas with quotaon. The script displays the string "Checking quotas:" on the console when the quotacheck utility begins (suppressing the customary carriage return at the end of the displayed line) and completes the line with "done." after it has finished (although many current systems use fancier, more aesthetically pleasing status messages).

Starting daemons and initializing local subsystems

Once all the prerequisite system devices are ready, important subsystems such as electronic mail, printing, and accounting can be started. Most of them rely on daemons (server processes). These processes are started automatically by one of the boot scripts. On most systems, purely local subsystems that do not depend on the network are usually started before networking is initialized, and subsystems that do need network facilities are started afterwards.

For example, a script like this one (from a Solaris system) could be used to initialize the cron subsystem, a facility to execute commands according to a preset schedule

if [ -p /etc/cron.d/FIFO ]; then
if /usr/bin/pgrep -x -u 0 -P 1 cron >/dev/null 2>&1; then
echo "$0: cron is already running"
exit 0
fi
elif [ -x /usr/sbin/cron ]; then
/usr/bin/rm -f /etc/cron.d/FIFO
/usr/sbin/cron &
fi

The script first checks for the existence of the cron lock file (a named pipe called FIFO whose location varies). If it is present, the script next checks for a current cron process (via the pgrep command). It the latter is found, the script exits because cron is already running. Otherwise, the script checks for the existence of the cron executable file. If it finds the file, the script removes the cron lock file and then starts the cron server.

The precautionary check to see whether cron is already running isn't made on all systems. Lots of system initialization files simply (foolishly) assume that they will be run only at boot time, when cron obviously won't already be running. Others use a different, more general mechanism to determine the conditions under which they were run. We'll examine that shortly.

Other local subsystems started in a similar manner include:

update

A process that periodically forces all filesystem buffers (accumulated changes to inodes and data blocks) to disk. It does so by running the sync command, ensuring that the disks are fairly up-to-date should the system crash. The name of this daemon varies somewhat: bdflush is a common variant, AIX calls its version syncd, the HP-UX version is syncer, and it is named fsflush on Solaris systems. Linux runs both update and bdflush. Whatever its name, don't disable this daemon or you will seriously compromise filesystem integrity.

syslogd

The system message handling facility that routes informational and error messages to log files, specific users, electronic mail, and other destinations according to the specifications in its configuration file (see Chapter 3).

Accounting

this subsystem is started using the accton command. If accounting is not enabled, the relevant commands may be commented out.

System status monitor daemons

some systems provide daemons that monitor the system's physical conditions (e.g., power level, temperature, and humidity) and trigger the appropriate action when a problem occurs. For example, the HP-UX ups_mond daemon watches for a power failure, switching to an uninterruptible power supply (UPS) to allow an orderly system shutdown, if necessary.

Subsystems that are typically started after networking (discussed in the next section) include:

• Electronic mail: the most popular electronic mail server is sendmail, which can route mail locally and via the network as needed. Postfix is a common alternative (its server process is also called sendmail).

• Printing: the spooling subsystem also may be entirely local or used for printing to remote systems in addition to (or instead of) locally connected ones. BSD-type printing subsystems rely on the lpd daemon, and System V systems use lpsched. The AIX printing server is qdaemon.

There may be other subsystems on your system with their own associated daemon processes; some may be vendor enhancements to standard Unix. We'll consider some of these when we look at the specific initialization files used by the various Unix versions later in this chapter.

The AIX System Resource Controller

On AIX systems, system daemons are controlled by the S ystem Resource Controller (SRC). This facility starts daemons associated with the various subsystems and monitors their status on an ongoing basis. If a system daemon dies, the SRC automatically restarts it.

The srcmstr command is the executable corresponding to the SRC. The lssrc and chssys commands may be used to list services controlled by the SRC and change their configuration settings, respectively. We'll see examples of these commands at various points in this book.

Connecting to the network

Network initialization begins by setting the system's network hostname, if necessary, and configuring the network interfaces (adapter devices), enabling it to communicate on the network. The script that starts networking at boot time contains commands like these:

ifconfig lo0 127.0.0.1 
ifconfig ent0 inet 192.168.29.22 netmask 255.255.255.0

The specific ifconfig commands vary quite a bit. The first parameter to ifconfig, which designates the network interface, may be different on your system. In this case, lo0 is the loopback interface, and ent0 is the Ethernet interface. Other common names for Ethernet interfaces include eri0, dnet0, and hme0 (Solaris); eth0 (Linux); tu0 (Tru64); xl0 (FreeBSD); lan0 (HP-UX); en0 (AIX); and ef0 and et0 (some System V). Interfaces for other network media will have different names altogether. Static routes may also be defined at this point using the route command. Networking is discussed in detail in Chapter 5.

Networking services also rely on a number ofdaemon processes. They are usually started with commands of this general form:

if [ -x server-pathname ]; then 
  preparatory commands 
  server-start-cmd 
  echo Starting server-name 
fi

When the server program file exists and is executable, the script performs any necessary preparatory activities and then starts the server process. Note that some servers go into background execution automatically, while others must be explicitly started in the background. The most important network daemons are listed below

Once basic networking is running, other services and subsystems that depend on it can be started. In particular, remote filesystems can be mounted with a command like this one, which mounts all remote filesystems listed in the system's filesystem configuration file:

mount -a -t nfs     On some systems, -F replaces -t.

Housekeeping activities

Traditionally, multiuser-mode boots also include a number of cleanup activities such as the following:

• Preserving editor files from vi and other ex-based editors, which enable users to recover some unsaved edits in the event of a crash. These editors automatically place checkpoint files in /tmp or /var/tmp during editing sessions. The expreserve utility is normally run at boot time to recover such files. On Linux systems, the elvis vi-clone is commonly available, and elvprsv performs the same function as expreserve for its files.

• Clearing the /tmp directory and possibly other temporary directories. The commands to accomplish this can be minimalist:

  rm -f /tmp/*

  
  

• utilitarian:

  cd /tmp; find . ! -name . ! -name .. ! -name lost+found \    
                         ! -name quota\* -exec rm -fr {} \;

  or rococo:

  # If no /tmp exists, create one (we assume /tmp is not
  # a separate file system).
  if [ ! -d /tmp -a ! -l /tmp ]; then    
     rm -f /tmp    
     mkdir /tmp 
  fi  
  for dir in /tmp /var/tmp /usr/local/tmp ; do 
     if [ -d $dir ] ; then       
        cd $dir    
        find . \( \( -type f \( -name a.out -o        \     
                 -name \*.bak -o -name core -o -name \*~ -o    \ 
                 -name .\*~ -o -name #\*# -o -name #.\*# -o    \    
                 -name \*.o -o \( -atime +1 -mtime +3 \) \) \)  \    
                 -exec rm -f {} \; -o -type d -name \*        \    
                 -prune -exec rm -fr {} \; \)     
     fi    
  cd / 
  done

  The first form simply removes from /tmp all files other than those whose names begin with a period. The second form might be used when /tmp is located on a separate filesystem from the root filesystem to avoid removing important files and subdirectories. The third script excerpt makes sure that the /tmp directory exists and then removes a variety of junk files and any subdirectory trees (with names not beginning with a period) from a series of temporary directories.

On some systems, these activities are not part of the boot process but are handled in other ways (see Chapter 15 for details).

Allowing users onto the system

The final boot-time activities complete the process of making the system available to users. Doing so involves both preparing resources users need to log in and removing barriers that prevent them from doing so. The former consists of creating the getty processes that handle each terminal line and starting a graphical login manager like xdm—or a vendor-customized equivalent facility—for X stations and the system console, if appropriate. On Solaris systems, it also includes initializing the Service Access Facility daemons sac and ttymon.

On most systems, the file /etc/nologin may be created automatically when the system is shut down normally. Removing it is often one of the very last tasks of the boot scripts. FreeBSD uses /var/run/nologin.

/etc/nologin may also be created as needed by the system administrator. If this file is not empty, its contents are displayed to users when they attempt to log in. Creating the file has no effect on users who are already logged in, and the root user can always log in. HP-UX versions prior to 11i do not use this file.

4.2. Initialization Files and Boot Scripts

This section discusses the Unix initialization files: command scripts that perform most of the work associated with taking the system to multiuser mode. Although similar activities take place under System V and BSD, the mechanisms by which they are initiated are quite different. Of the systems we are considering, FreeBSD follows the traditional BSD style, AIX is a hybrid of the two, and all the other versions use the System V scheme.

Understanding the initialization scripts on your system is a vital part of system administration. You should have a pretty good sense of where they are located and what they do. That way, you'll be able to recognize any problems at boot time right away, and you'll know what corrective action to take. Also, from time to time, you'll probably need to modify them to add new services (or to disable ones you've decided you don't need). We'll discuss customizing initialization scripts later in this chapter.

Aspects of the boot process are also controlled by configuration files that modify the operations of the boot scripts. Such files consist of a series of variable definitions that are read in at the beginning of a boot script and whose values determine which commands in the script are executed. These variables can specify things like whether a subsystem is started at all, the command-line options to use when starting a daemon, and the like. Generally, these files are edited manually, but some systems provide graphical tools for this purpose.

Initialization Files Under FreeBSD

The organization of system initialization scripts on traditionalBSD systems such as FreeBSD is the essence of simplicity. In the past, boot-time activities occurred via a series of only three or four shell scripts, usually residing in /etc, with names beginning with rc. Under FreeBSD, this number has risen to about 20 (although not all of them apply to every system).

Multiuser-mode system initialization under BSD-based operating systems is controlled by the file /etc/rc. During a boot to multiuser mode, init executes the rc script, which in turn calls other rc.* scripts. If the system is booted to single-user mode, rc begins executing when the single-user shell is exited.

The boot script configuration files /etc/default/rc.conf, /etc/rc.conf, and /etc/rc.conf.local control the functioning of the rc script. The first of these files is installed by the operating system and should not be modified. The other two files contain overrides to settings in the first file (although the latter is seldom used).

Here are some example entries from /etc/rc.conf:

accounting_enable="YES"
check_quotas="YES"
defaultrouter="192.168.29.204"
hostname="ada.ahania.com"
ifconfig_xl0="inet 192.168.29.216 netmask 255.255.255.0"
inetd_enable="YES"
nfs_client_enable="YES"
nfs_server_enable="YES"
portmap_enable="YES"
sendmail_enable="NO"
sshd_enable="YES"

This file enables the accounting, inetd, NFS, portmapper, and ssh subsystems and disables sendmail. It causes disk quotas to be checked at boot time, and specifies various network settings, including the Ethernet interface.

Initialization Files on System V Systems

The system initialization scripts on a System V-style system are much more numerous and complexly interrelated than those under BSD. They all revolve around the notion of the current system run level, a concept to which we now turn.

At any given time, a computer system can be in one of three conditions: off (not running, whether or not it has power), single-user mode, or multiuser mode (normal operating conditions). These three conditions may be thought of as three implicitly defined system states.

System V-based systems take this idea to its logical extreme and explicitly define a series of system states, called run levels, each of which is designated by a one-character name that is usually a number. At any given time, the system is at one of these states, and it can be sent to another one using various administrative commands.

In most implementations, states 1 and s/S are not distinguished in practice, and not all states are predefined by all implementations. State 3 is the defined normal operating mode for networked systems. In practice, some systems collapse run levels 2 and 3, supporting all networking functions at run level 2 and ignoring run level 3, or making them identical so that 2 and 3 become alternate names for the same system state. We will use separate run levels 2 and 3 in our examples, making run level 3 the system default level.

Note that the pseudo-run levels (a, b, c, and q/Q) do not represent distinct system states, but rather function as ways of getting init to perform certain tasks on demand.

The command who -r may be used to display the current run level and the time it was initiated:

$ who -r 
. run level 3  Mar 14 11:14  3  0  S     Previous run level was S.

The output indicates that this system was taken to run level 3 from run level S on March 14. The 0 value between the 3 and the S indicates the number of times the system had been at the current run level immediately prior to entering it this time. If the value is nonzero, it often indicates previous unsuccessful boots.

On Linux systems, the runlevel command lists the previous and current run levels.

Now for some concrete examples. Let's assume a system whose normal, everyday system state is state 3 (networked multiuser mode). When you boot this system after the power has been off, it moves from state 0 to state 3. If you shut the system down to single-user mode, it moves from state 3 through state 0 to state s. When you reboot the system, it moves from state 3 through state 6 and state 0, and then back to state 3.^[12]

^[12] In practice, booting to state 3 often involves implicitly moving through state 2, given the way that inittab configuration files employing both states are usually set up.

Using the telinit command to change run levels

The telinit utility may be used to change the current systemrun level. Its name comes from the fact that it tells the init process what to do next. It takes the new run level as its argument. The following command tells the system to reboot:

# telinit 6

You can also just use init itself: init 6.

AIX also omits the telinit command, since it does not implement run levels in the usual manner.

Initialization files overview

System V-style systems organize the initialization process in a much more complex way, using three levels of initialization files:

• /etc/inittab, which is init's configuration file.

• A series of primary scripts named rcn (where n is the run level), typically stored in /etc or /sbin.

• A collection of auxiliary, subsystem-specific scripts for each run level, typically located in subdirectories named rcn.d under /etc or /sbin.

• In addition, some systems also provide configuration files that define variables specifying or modifying the functioning of some of these scripts.

On a boot, when init takes control from the kernel, it scans its configuration file, /etc/inittab, to determine what to do next. This file defines init's actions whenever the system enters a new run level; it contains instructions to carry out when the system goes down (run level 0), when it boots to single-user mode (run level S), when booting to multiuser mode (run level 2 or 3), when rebooting (run level 6), and so on.

Each entry in the inittab configuration file implicitly defines a process to be run at one or more run levels. Sometimes, this process is an actual daemon that continues executing as long as the system remains in a given run level. More often, the process is a shell script that is executed when the system enters one of the run levels specified in its inittab entry.

When the system changes run levels, init consults the inittab file to determine the processes that should be running at the new run level. It then kills all currently running processes that should not be running at the new level and starts all processes specified for the new run level that are not already running.

Typically, the commands to execute at the start of each run level are contained in a script named rcn, where n is the run level number (these scripts are usually stored in the /etc directory). For example, when the system moves to run level 2, init reads the /etc/inittab file, which tells it to execute rc2. rc2 then executes the scripts stored in the directory /etc/rc2.d. Similarly, when a running system is rebooted, it moves first from run level 2 to run level 6, a special run level that tells the system to shut down and immediately reboot, where it usually executes rc0 and the scripts in /etc/rc0.d, and then changes to run level 2, again executing rc2 and the files in /etc/rc2.d. A few systems use a single rc script and pass the run level as its argument: rc 2.

The init configuration file

As we've seen, top-level control of changing system states is handled by the file /etc/inittab, read by init. This file contains entries that tell the system what to do when it enters the various defined system states.

Entries in the inittab have the following form:

cc:levels:action:process

where cc is a unique, case-sensitive label identifying each entry (subsequent entries with duplicate labels are ignored). levels is a list of run levels to which the entry applies; if it is blank, the entry applies to all of them. When the system enters a new state, init processes all entries specified for that run level in the inittab file, in the order they are listed in the file.

process is the command to execute, and action indicates how init is to treat the process started by the entry. The most important action keywords are the following:

wait

Start the process and wait for it to finish before going on to the next entry for this run state.

respawn

Start the process and automatically restart it when it dies (commonly used for getty terminal line server processes).

once

Start the process if it's not already running. Don't wait for it.

boot

Execute entry only at boot time; start the process but don't wait for it.

bootwait

Execute entry only at boot time and wait for it to finish.

initdefault

Specify the default run level (the one to reboot to).

sysinit

Used for activities that need to be performed before init tries to access the system console (for example, initializing the appropriate device).

off

If the process associated with this entry is running, kill it. Also used to comment out unused terminal lines.

Comments may be included on separate lines or at the end of any entry by preceding the comment with a number sign (#).

Here is a sample inittab file:

# set default init level -- multiuser mode with networking 
is:3:initdefault:

# initial boot scripts 
fs::bootwait:/etc/bcheckrc </dev/console >/dev/console 2>&1 
br::bootwait:/etc/brc </dev/console >/dev/console 2>&1

# shutdown script 
r0:06:wait:/etc/rc0  >/dev/console 2>&1 </dev/console

# run level changes 
r1:1:wait:/sbin/shutdown -y -iS -g0 >/dev/console 2>&1 
r2:23:wait:/etc/rc2 >/dev/console 2>&1 </dev/console 
r3:3:wait:/etc/rc3  >/dev/console 2>&1 </dev/console 
pkg:23:once:/usr/sfpkg/sfpkgd    # start daemon directly

# off and reboot states 
off:0:wait:/sbin/uadmin 2 0 >/dev/console 2>&1 </dev/console 
rb:6:wait:/sbin/uadmin 2 1 >/dev/console 2>&1 </dev/console

# terminal initiation 
co:12345:respawn:/sbin/getty console console 
t0:234:respawn:/sbin/getty tty0 9600 
t1:234:respawn:/sbin/getty tty1 9600 
t2:234:off:/sbin/getty tty2 9600

# special run level 
acct:a:once:/etc/start_acct      # start accounting

This file logically consists of seven major sections, which we've separated with blank lines. The first section, consisting of a single entry, sets the default run level, which in this case is networked multiuser mode (level 3).

The second section contains processes started when the system is booted. In the sample file, this consists of running the /etc/bcheckrc and /etc/brc preliminary boot scripts commonly used on System V systems in addition to the rcn structure. The bcheckrc script's main function is to prepare the root filesystem and other critical filesystems like /usr and /var. Both scripts are allowed to complete before init goes on to the next inittab entry.

The third section of the sample inittab file specifies the commands to execute whenever the system is brought down, either during a system shutdown and halt (to run level 0) or during a reboot (run level 6). In both cases, the script /etc/rc0 is executed, and init waits for it to finish before proceeding.

The fourth section, headed "run level changes," specifies the commands to run when system states 1, 2, and 3 begin. For state 1, the shutdown command listed in the sample file takes the system to single-user mode. Some systems execute the rc1 initialization file when the system enters state 1 instead of a shutdown command like the one above.

For state 2, init executes the rc2 initialization script; for state 3, init executes rc2 followed by rc3. In all three states, each process is allowed to finish before init goes on to the next entry. The final entry in this section starts a process directly instead of calling a script. The sfpkgd daemon is started only once per run level, when the system first enters run level 2 or 3. Of course, if the daemon is already running, it will not be restarted.

The fifth section specifies commands to run (after rc0) when the system enters run levels 0 and 6. In both cases, init runs the uadmin command, which initiates system shutdown. The arguments to uadmin specify how the shutdown is to be handled. Many modern systems have replaced this legacy command, folding its functionality into the shutdown command (as we'll see shortly). Of the System V systems we are considering, only Solaris still uses uadmin.

The sixth section initializes the system's terminal lines via getty processes (which are discussed in Chapter 12).

The final section of the inittab file illustrates the use of special run level a. This entry is used only when a telinit a command is executed by the system administrator, at which point the start_acct script is run. The run levels a, b, and c are available to be defined as needed.

The rcn initialization scripts

As we've seen, init typically executes a script named rcn when entering run level n (rc2 for state 2, for example). Although theboot (or shutdown) process to each system state is controlled by the associated rcn script, the actual commands to be executed are stored in a series of files in the subdirectory rcn.d. Thus, when the system enters state 0, init runs rc0 (as directed in the inittab file), which in turn runs the scripts in rc0.d.

The contents of an atypically small rc2.d directory (on a system that doesn't use a separate run level 3) are listed below:

$ ls -C /etc/rc2.d 
K30tcp           S15preserve    S30tcp     S50RMTMPFILES  
K40nfs           S20sysetup     S35bsd     S75cron 
S01MOUNTFSYS     S21perf        S40nfs     S85lp

All filenames begin with one of two initial filename characters (S and K), followed by a two-digit number, and they all end with a descriptive name. The rcn scripts execute the K-files (as I'll call them) in their associated directory in alphabetical order, followed by the S-files, also in alphabetical order (this scheme is easiest to understand if all numbers are the same length; hence the leading zeros on numbers under 10). Numbers do not need to be unique.

In this directory, files would be executed in the order K30tcp, K40nfs, S01MOUNTFSYS, S15preserve, and so on, ending with S75cron and S85lp. K-files are generally used to kill processes (and perform related functions) when transitioning to a different state; S-files are used to start processes and perform other initialization functions.

The files in the rc*.d subdirectories are usually links to those files in the subdirectory init.d, where the real files live. For example, the file rc2.d/S30tcp is actually a link to init.d/tcp. You see how the naming conventions work: the final portion of the name in the rcn.d directory is the same as the filename in the init.d directory.

The file K30tcp is also a link to init.d/tcp. The same file in init.d is used for both the kill and start scripts for each subsystem. The K and S links can be in the same rcn.d subdirectory, as is the case for the TCP/IP initialization file, or in different subdirectories. For example, in the case of the print spooling subsystem, the S-file might be in rc2.d while the K-file is in rc0.d.

The same file in init.d can be put to both uses because it is passed a parameter indicating whether it was run as a K-file or an S-file. Here is an example invocation, from an rc2 script:

# If the directory /etc/rc2.d exists, 
# run the K-files in it ... 
if [ -d /etc/rc2.d ]; then 
   for f in /etc/rc2.d/K* 
   { 
      if [ -s ${f} ]; then 
#        pass the parameter "stop" to the file
         /bin/sh ${f} stop 
      fi 
   }
# and then the S-files: 
   for f in /etc/rc2.d/S* 
   { 
      if [ -s ${f} ]; then 
#        pass the parameter "start" to the file
         /bin/sh ${f} start 
      fi 
   } 
fi

When a K-file is executed, it is passed the parameter stop; when an S-file is executed, it is passed start. The script file will use this parameter to figure out whether it is being run as a K-file or an S-file.

Here is a simple example of the script file, init.d/cron, which controls the cron facility. By examining it, you'll be able to see the basic structure of a System V initialization file:

#!/bin/sh 
case $1 in
   # commands to execute if run as "Snncron" 
   'start')
      # remove lock file from previous cron 
      rm -f /usr/lib/cron/FIFO
      # start cron if executable exists 
      if [ -x /sbin/cron ]; then
         /sbin/cron
         echo "starting cron." 
      fi 
   ;;

   # commands to execute if run as "Knncron" 
   'stop') 
         pid=`/bin/ps -e | grep ' cron$' | \
            sed -e 's/^  *//' -e 's/ .*//'`
         if [ "${pid}" != "" ]; then
            kill ${pid}
         fi 
    ;;

   # handle other arguments 
   *) 
      echo "Usage: /etc/init.d/cron {start|stop}" 
      exit 1 
   ;; 
esac

The first section in the case statement is executed when the script is passed start as its first argument (when it's an S-file); the second section is used when it is passed stop, as a K-file. The start commands remove any old lock file and then start the cron daemon if its executable is present on the system. The stop commands figure out the process ID of the cron process and kill it if it's running. Some scripts/operating systems define additional valid parameters, including restart (equivalent to stop then start) and status.

The file /etc/init.d/cron might be linked to both /etc/rc2.d/S75cron and /etc/rc0.d/K75cron. The cron facility is then started by rc2 during multiuser boots and stopped by rc0 during system shutdowns and reboots.

Sometimes scripts are even more general, explicitly testing for the conditions under which they were invoked:

set `who -r`                                 Determine previous run level. 
if [ $8 != "0" ]                             The return code of the previous state change. 
then 
   exit 
fi 
case $arg1 in 'start') 
   if [ $9 = "S" ]                           Check the previous run level. 
   then 
      echo "Starting process accounting" 
      /usr/lib/acct/startup 
   fi 
   ;;
...

This file uses various parts of the output from who -r:

$ who -r 
.  run level 2   Mar 14 11:14   2    0    S

The set command assigns successive words in the output from the who command to the shell script arguments $1 through $9. The script uses them to test whether the current system state was entered without errors, exiting if it wasn't. It also checks whether the immediately previous state was single-user mode, as would be the case on this system on a boot or reboot. These tests ensure that accounting is started only during a successful boot and not when single-user mode has been entered due to boot errors or when moving from one multiuser state to another.

Boot script configuration files

On many systems, the functioning of the various boot scripts can be controlled and modified by settings in one or more relatedconfiguration files. These settings may enable or disable subsystems, specify command-line arguments for starting daemons, and the like. Generally, such settings are stored in separate files named for the corresponding subsystem, but sometimes they are all stored in a single file (as on SuSE Linux systems, in /etc/rc.config).

Here are two configuration files from a Solaris system; the first is /etc/default/sendmail:

DAEMON=yes     Enable the daemon.
QUEUE=1h       Set the poll interval to 1 hour.

The next file is /etc/default/samba:

# Options to smbd
SMBDOPTIONS="-D"
# Options to nmbd
NMBDOPTIONS="-D"

The first example specifies whether the associated daemon should be started, as well as one of its arguments, and the second file specifies the arguments to be used when starting the two Samba daemons.

File location summary

Solaris initialization scripts

Solaris uses a standard System V boot script scheme. The script rcS (in /sbin) replaces bcheckrc, but it performs the same functions. Solaris uses separate rcn scripts for each run level from 0 through 6 (excluding rc4, which a site must create on its own), but the scripts for run levels 0, 5, and 6 are just links to a single script, called with different arguments for each run level. There are separate rcn.d directories for run levels 0 through 3 and S.

Unlike on some other systems, run level 5 is a "firmware" (maintenance) mode, defined as follows:

s5:5:wait:/sbin/rc5        >/dev/msglog 2>&1 </dev/console
of:5:wait:/sbin/uadmin 2 6 >/dev/msglog 2>&1 </dev/console

These entries illustrate the Solaris msglog device, which sends output to one or more console devices via a single redirection operation.

Solaris inittab files also usually contain entries for the Service Access Facility daemons, such as the following:

sc:234:respawn:/usr/lib/saf/sac -t 300 ...
co:234:respawn:/usr/lib/saf/ttymon ...

Run level 3 on Solaris systems is set up as the remote file-sharing state. When TCP/IP is the networking protocol in use, this means that general networking and NFS client activities—such as mounting remote disks—occur as part of run level 2, but NFS server activities do not occur until the system enters run level 3, when local filesystems become available to other systems. The rc2 script, and thus the scripts in rc2.d, are executed for both run levels by an inittab entry like this one:

s2:23:wait:/sbin/rc2 ...

Linux initialization scripts

Most Linux systems use a vanilla, System V-style boot script hierarchy. The Linux init package supports the special action keyword ctrlaltdel that allows you to trap CTRL-ALT-DELETE sequences (the standard method of rebooting a PC), as in this example, which calls the shutdown command and reboots the system:

ca::ctrlaltdel:/sbin/shutdown -r now

Linux distributions also provide custom initial boot scripts (run prior to rc). For example, Red Hat Linux uses /etc/rc.d/rc.sysinit for this purpose, and SuSE Linux systems use /etc/init.d/boot. These scripts focus on the earliest boot tasks such as checking and mounting filesystems, setting the time zone, and initializing and activating swap space.

AIX: Making System V work like BSD

It's possible to eliminate most of the layers of initialization scripts that are standard under System V. Consider this AIX inittab file:

init:2:initdefault: 
brc::sysinit:/sbin/rc.boot 3 >/dev/console 2>&1
rc:2:wait:/etc/rc 2>&1 | alog -tboot > /dev/console srcmstr:2:respawn:/usr/sbin/srcmstr
tcpip:2:wait:/etc/rc.tcpip > /dev/console 2>&1  
nfs:2:wait:/etc/rc.nfs > /dev/console 2>&1  
ihshttpd:2:wait:/usr/HTTPServer/bin/httpd > /dev/console 2>&1 
cron:2:respawn:/usr/sbin/cron
qdaemon:2:wait:/usr/bin/startsrc -sqdaemon
cons::respawn:/etc/getty /dev/console 
tty0:2:respawn:/etc/getty /dev/tty0

Other than starting a server process for the system console and executing the file /etc/bcheckrc at boot time, nothing is defined for any run level other than state 2 (multiuser mode).

This is the approach taken by AIX. When the system enters state 2, a series of initialization files are run in sequence: in this case, /etc/rc, /etc/rc.tcpip, and /etc/rc.nfs (with the System Resource Controller starting up in the midst of them). Then several daemons are started via their own inittab entries. After the scripts complete, getty processes are started. Since /etc/rcn.d subdirectories are not used at all, this setup is a little different from that used on BSD systems.

More recent AIX operating system revisions do include hooks for other run levels, modifying the preceding inittab entries in this way:

# Note that even run level 6 is included!
tcpip:23456789:wait:/etc/rc.tcpip > /dev/console 2>&1

The /etc/rc.d/rcn.d subdirectories are provided, but they are all empty.

Customizing the Boot Process

Sooner or later, you will want to make additions or modifications to the standard boot process. Making additions is less risky than changing existing scripts. We'll consider the two types of modifications separately.

NOTE

Before adding to or modifying system boot scripts, you should be very familiar with their contents and understand what every line within them does. You should also save a copy of the original script so you can easily restore the previous version should problems occur.

Adding to the boot scripts

When you want to add commands to the boot process, the first thing you need to determine is whether there is already support for what you want to do. See if there is an easy way to get what you want: changing a configuration file variable, for example, or adding a link to an existing file in init.d.

If the operating system has made no provisions for the tasks you want to accomplish, you must next figure out where in the process the new commands should be run. It is easiest to add items at the end of the standard boot process, but occasionally this is not possible.

It is best to isolate your changes from the standard system initialization files as much as possible. Doing so makes them easier to test and debug and also makes them less vulnerable to being lost when the operating system is upgraded and the previous boot scripts are replaced by new versions. Under the BSD scheme, the best way to accomplish this is to add a line to rc (or any other script that you need to change) that calls a separate script that you provide:

. /etc/rc.site_specific >/dev/console 2>&1

Ideally, you would place this at the end of rc, and the additional commands needed on that system would be placed into the new script. Note that the script is sourced with the dot command so that it inherits the current environment from the calling script. This does constrain it to being a Bourne shell script.

On System V systems, there are more options. One approach is to add one or more additional entries to the inittab file (placing them as late in the file as possible):

site:23:wait:/etc/rc.site_specific >/dev/console 2>&1 
h96:23:once:/usr/local/bin/h96d

The first entry runs the same shell script we added before, and the second entry starts a daemon process. Starting a daemon directly from inittab (rather than from some other initialization file) is useful in two circumstances: when you want the daemon started only at boot time and when you want it to be restarted automatically if it terminates. You would use the inittab actions once and respawn, respectively, to produce these two ways of handling the inittab entry.

Alternatively, if your additions need to take place at a very specific point in the boot process, you will need to add a file to the appropriate rcn.d subdirectories. Following the System V practice is best in this case: place the new file in the init.d directory, giving it a descriptive name, and then create links to other directories as needed. Choose the filenames for the links carefully, so that your new files are executed at the proper point in the sequence. If you are in doubt, executing the ls -1 command in the appropriate directory provides you with an unambiguous list of the current ordering of the scripts within it, and you will be able to determine what number to use for your new one.

Eliminating certain boot-time activities

Disabling parts of the boot process is also relatively easy. The method for doing so depends on the initialization scripts used by your operating system. The various possibilities are (in decreasing order of preference):

• Disable a subsystem by setting the corresponding control variable to no or 0 in one of the boot script configuration files. For example:

  sendmail_enable="no"

• Remove the link in the rcn.d directory to the init.d directory in the case of System V-style boot scripts. Alternatively, you can rename the link, for example, by adding another character to the beginning (I add an underscore: _K20nfs). That way, it is easy to reinstate the file later.

• In some cases, you will need to comment out an entry in /etc/inittab (when a daemon that you don't want is started directly).

• Comment out the relevant lines of initialization scripts that you don't want to use. This is the only option under FreeBSD when no rc.conf parameter has been defined for a command or subsystem.

Modifying standard scripts

While it is usually best to avoid it, sometimes you have no choice but to modify the commands in standard boot scripts. For example, certain networking functions stopped working on several systems I take care of immediately after an operating system upgrade. The reason was a bug in an initialization script, illustrated by the following:

# Check the mount of /. If remote, skip rest of setup. 
mount | grep ' / ' | grep ' nfs ' 2>&1 > /dev/null 
if [ "$?" -eq 0 ] 
then 
    exit 
fi

The second line of the script is trying to figure out whether the root filesystem is local or remote—in other words, whether the system is a diskless workstation or not. It assumes that if it finds a root filesystem that is mounted via NFS, it must be a diskless system. However, on my systems, lots of root filesystems from other hosts are mounted via NFS, and this condition produced a false positive for this script, causing it to exit prematurely. The only solution in a case like this is to fix the script so that your system works properly.

Whenever you change a system script, keep these recommendations in mind:

• As a precaution, before modifying them in any way, copy the files you intend to change, and write-protect the copies. Use the -p option of the cp command, if it is supported, to duplicate the modification times of the original files as well as their contents; this data can be invaluable should you need to roll back to a previous, working configuration. For example:

  # cp -p /etc/rc /etc/rc.orig
  # cp -p /etc/rc.local /etc/rc.local.orig
  # chmod a-w /etc/rc*.orig 

  
  

  If your version of cp doesn't have a -p option, use a process like this one:

  # cd /etc
  # mv rc rc.orig; cp rc.orig rc
  # mv rc.local rc.local.orig; cp rc.local.orig rc.local
  # chmod a-w rc.orig rc.local.orig 

  
  

  Similarly, when you make further modifications to an already customized script, save a copy before doing so, giving it a different extension, such as .save. This makes the modification process reversible; in the worst case, when the system won't boot because of bugs in your new versions—and this happens to everyone—you can just boot to single-user mode and copy the saved, working versions over the new ones.

• Make some provision for backing up modified scripts regularly so that they can be restored easily in an emergency. This topic is discussed in detail in Chapter 11.

• For security reasons, the system initialization scripts (including any old or saved copies of them) should be owned by root and not be writable by anyone but the owner. In some contexts, protecting them against any non-root access is appropriate.

Guidelines for writing initialization scripts

System boot scripts often provide both good and bad shell programming examples. If you write boot scripts or add commands to existing ones, keep these recommended programming practices in mind:

• Use full pathnames for all commands (or use one of the other methods for ensuring that the proper command executable is run).

• Explicitly test for the conditions under which the script is run if it is relying on the system being in some known state. Don't assume, for example, that there are no users on the system or that a daemon the script will be starting isn't already running; have the script check to make sure. Initialization scripts often get run in other contexts and at times other than those for which their writers originally designed them.

• Handle all cases that might arise from any given action, not just the ones that you expect to result. This includes handling invalid arguments to the script and providing a usage message.

• Provide lots of informational and error messages for the administrators who will see the results of the script.

• Include plenty of comments within the script itself.

Shutting Down a Unix System

From time to time, you will need to shut thesystem down. This is necessary for scheduled maintenance, running diagnostics, hardware changes or additions, and other administrative tasks.

During a clean system shutdown, the following actions take place:

• All users are notified that the system will be going down, preferably giving them some reasonable advance warning.

• All running processes are sent a signal telling them to terminate, allowing them time to exit gracefully, provided the program has made provisions to do so.

• All subsystems are shut down gracefully, via the commands they provide for doing so.

• All remaining users are logged off, and remaining processes are killed.

• Filesystem integrity is maintained by completing all pending disk updates.

• Depending on the type of shutdown, the system moves to single-user mode, the processor is halted, or the system is rebooted.

After taking these steps, the administrator can turn the power off, execute diagnostics, or perform other maintenance activities as appropriate.

Unix provides the shutdown command to accomplish all of this. Generally, shutdown sends a series of timed messages to all users who are logged on, warning them that the system is going down; after sending the last of these messages, it logs all users off the system and places the system in single-user mode.

There are two main variations of the shutdown command. The System V version is used by Solaris and HP-UX (the latter slightly modified from the standard), and the BSD version is used under AIX, FreeBSD, Linux, Solaris (in /usr/ucb), and Tru64.

NOTE

On systems that provide it, the telinit command also provides a fast way to shut down (telinit S), halt (telinit 0) or reboot the system (telinit 6).

The System V shutdown Command

The standard System V shutdown command has the following form:

# shutdown [-y] [-g grace] [-i new-level]  message

where -y says to answer all shutdown prompts with yes automatically, grace specifies the number of seconds to wait before starting the process (the default is 60), new-level is the new run level in which to place the system (the default is single-user mode) and message is a text message sent to all users. This is the form used on Solaris systems.

Under HP-UX, the shutdown command has the following modified form:

# shutdown [-y] grace

where -y again says to answer prompts automatically with yes, and grace is the number of seconds to wait before shutting down. The keyword now may be substituted for grace. The shutdown command takes the system to single-user mode.

Here are some example commands that take the system to single-user mode in 15 seconds (automatically answering all prompts):

# shutdown -y -g 15 -i s "system going down"     Solaris
# shutdown -y 15                                 HP-UX

The HP-UX shutdown also accepts two other options, -r and -h, which can be used to reboot the system immediately or to halt the processor once the shutdown is complete (respectively).

For example, these commands could be used to reboot the system immediately:

# shutdown -y -g 0 -i 6 "system reboot"          Solaris
# shutdown -y -r now                             HP-UX

HP-UX shutdown security

HP-UX also provides the file /etc/shutdown.allow . If this file exists, a user must be listed in it in order to use the shutdown command (and root must be included). If the file does not exist, only root can run shutdown. Entries in the file consist of a hostname followed by a username, as in these examples:

hamlet    chavez     Chavez can shut down hamlet.
+         root       Root can shut down any system.
dalton    +          Anyone can shut down dalton.

As these examples illustrate, the plus sign serves as a wildcard. The shutdown.allow file also supports the percent sign as an additional wildcard character denoting all systems within a cluster; this wildcard is not valid on systems that are not part of a cluster.

The BSD-Style shutdown Command

BSD defines the shutdown command with the following syntax:

# shutdown [options] time message

where time can have three forms:

+m      Shut down in m minutes.
h:m     Shut down at the specified time (24-hour clock). 
now     Begin the shutdown at once.

now should be used with discretion on multiuser systems.

message is the announcement that shutdown sends to all users; it may be any text string. For example, the following command will shut the system down in one hour:

# shutdown +60 "System going down for regular maintenance"

It warns users by printing the message "System going down for regular maintenance" on their screens. shutdown sends the first message immediately; as the shutdown time approaches, it repeats the warning with increasing frequency. These messages are also sent to users on the other systems on the local network who may be using the system's files via NFS.

By default, the BSD-style shutdown command also takes the system to single-user mode, except on AIX systems, where the processor is halted by default. Under AIX, the -m option must be used to specify shutting down to single-user mode.

Other options provide additional variations to the system shutdown process:

• shutdown -r says to reboot the system immediately after it shuts down. The reboot command performs the same function.

• shutdown -h says to halt the processor instead of shutting down to single-user mode. Once this process completes, the power may be safely turned off. You can also use the halt command to explicitly halt the processor once single-user mode is reached.

• shutdown -k inaugurates a fake system shutdown: the shutdown messages are sent out normally, but no shutdown actually occurs. I suppose the theory is that you can scare users off the system this way, but some users can be pretty persistent, preferring to be killed by shutdown rather than log out.

The Linux shutdown Command

The version of shutdown found on mos tLinux systems also has a -t option which may be used to specify the delay period between when the kernel sends the TERM signal to all remaining processes on the system and when it sends the KILL signal. The default is 30 seconds. The following command shuts down the system more rapidly, allowing only 5 seconds between the two signals:

# shutdown -h -t 5 now

The command version also provides a -a option, which provides a limited security mechanism for the shutdown command. When it is invoked with this option, the command determines whether any of the users listed in the file /etc/shutdown.allow are currently logged in on the console (or any virtual console attached to it). If not, the shutdown command fails.

The purpose of this option is to prevent casual passers-by from typing Ctrl-Alt-Delete on the console and causing an (unwanted) system reboot. Accordingly, it is most often used in the inittab entry corresponding to this event.

Ensuring Disk Accuracy with the sync Command

As we've noted previously, one of the important parts of the shutdown process issyncing the disks. The sync command finishes all disk transactions and writes out all data to disk, guaranteeing that the system can be turned off without corrupting the files. You can execute this command manually if necessary:

# sync
# sync

Why is sync executed two or three times ? I think this is a bit of Unix superstition. The sync command schedules but does not necessarily immediately perform the required disk writes, even though the Unix prompt returns immediately. Multiple sync commands raise the probability that the write will take place before you enter another command (or turn off the power) by taking up the time needed to complete the operation. However, the same effect can be obtained by waiting a few seconds for disk activity to cease before doing anything else. Typing "sync" several times gives you something to do while you're waiting.

There is one situation in which you do not want sync to be executed, either manually or automatically: when you have run fsck manually on the root filesystem. If you sync the disks at this point, you will rewrite the bad superblocks stored in the kernel buffers and undo the fixing fsck just did. In such cases, on BSD-based systems and under HP-UX, you must use the -n option to reboot or shutdown to suppress the usual automatic sync operation.

FreeBSD and System V are smarter about this issue. The fsck command generally will automatically remount the root filesystem when it has modified the root filesystem. Thus, no special actions are required to avoid syncing the disks.

Aborting a Shutdown

On most systems, the only way to abort a pending system shutdown is to kill the shutdown process. Determine the shutdown process' process ID by using a command like the following:

# ps -ax | grep shutdown     BSD-style
# ps -ef | grep shutdown     System V-style

Then use the kill command to terminate it:

# ps -ef | grep shutdown 
25723 co S     0:01 /etc/shutdown -g300 -i6 -y 
25800 co S     0:00 grep shutdown
# kill -9 25723

It's only safe to kill a shutdown command during its grace period; once it has actually started closing down the system, you're better off letting it finish and then rebooting.

The Linux version of shutdown includes a -c option that cancels a pending system shutdown. Every version should be so helpful.

Troubleshooting: Handling Crashes and Boot Failures

Even the best-maintained systems crash from time to time. A crash occurs when the system suddenly stops functioning. The extent of system failure can vary quite a bit, from a failure affecting every subsystem to one limited to a particular device or to the kernel itself. System hang-ups are a related phenomenon in which the system stops responding to input from any user or device or stops producing output, but the operating system nominally remains loaded. Such a system also may be described as frozen.

There are many causes of system crashes and hangups. These are among the most common:

• Hardware failures: failing disk controllers, CPU boards, memory boards, power supplies, disk head crashes, and so on.

• Unrecoverable hardware errors, such as double-bit memory errors. These sorts of problems may indicate hardware that is about to fail, but they also just happen from time to time.

• Power failures or surges due to internal power supply problems, external power outages, electrical storms, and other causes.

• Other environmental problems: roof leaks, air conditioning failure, etc.

• I/O problems involving a fatal error condition rather than a device malfunction.

• Software problems, ranging from fatal kernel errors caused by operating system bugs to (much less frequently) problems caused by users or third-party programs.

• Resource overcommitment (for example, running out of swap space). These situations can interact with bugs in the operating system to cause a crash or hang-up.

Some of these causes are easier to identify than others. Rebooting the system may seem like the most pressing concern when the system crashes, but it's just as important to gather the available information about why the system crashed while the data is still accessible.

Sometimes it's obvious why the system crashed, as when the power goes out. If the cause isn't immediately clear, the first source of information is any messages appearing on the system console. They are usually still visible if you check immediately, even if the system is set to reboot automatically. After they are no longer on the screen, you may still be able to find them by checking the system error log file, usually stored in /var/log/messages (see Chapter 3 for more details), as well as any additional, vendor-supplied error facilities.

Beyond console messages lie crash dumps. Most systems automatically write a dump of kernel memory when the system crashes (if possible). These memory images can be examined using a debugging tool to see what the kernel was doing when it crashed. Obviously, these dumps are of use only for certain types of crashes in which the system state at the time of the crash is relevant. Analyzing crash dumps is beyond the scope of this book, but you should know where crash dumps go on your system and how to access them, if only to be able to save them for your field service engineers or vendor technical support personnel.

Crash dumps are usually written to the system disk swap partition. Since this area may be overwritten when the system is booted, some provisions need to be made to save its contents. The savecore command solves this problem, as we have seen ( the command is called savecrash under HP-UX).

NOTE

If you want to be able to save crash dumps, you need to ensure that the primary swap partition is large enough. Unless your system has the ability to compress crash dumps as they are created (e.g., Tru64) or selectively dump only the relevant parts of memory, the swap partition needs to be at least as large as physical memory.

If your system crashes and you are not collecting crash dumps by default, but you want to get one, boot the system to single-user mode and execute savecore by hand. Don't let the system boot to multiuser mode before saving the crash dump; once the system reaches multiuser mode, it's too late.

AIX also provides the snap command for collecting crash dump and other system data for later analysis.

4.4.1 Power-Failure Scripts

There are two other action keywords available for inittab that we've not yet considered: powerfail and powerwait. They define entries that are invoked if a SIGPWR signal is sent to the init process, which indicates an imminent power failure. This signal is generated only for detectable power failures: those caused by faulty power supplies, fans, and the like, or via a signal from an u ninterruptable power supply (UPS). powerwait differs from powerfail in that it requires init to wait for its process to complete before going on to the next applicable inittab entry.

The scripts invoked by these entries are often given the name rc.powerfail. Their purpose is to do whatever can be done to protect the system in the limited time available. Accordingly, they focus on syncing the disks to prevent data loss that might occur if disk operations are still pending when the power does go off.

Linux provides a third action, powerokwait, that is invoked when power is restored and tells init to wait for the corresponding process to complete before going on to any additional entries.

When the System Won't Boot

As with system crashes, there can be many reasons why a system won't boot. To solve such problems, you first must figure out what the specific problem is. You'll need to have a detailed understanding of what a normal boot process looks like so that you can pinpoint exactly where the failure is occurring. Having a hard copy of normal boot messages is often very helpful. One thing to keep in mind is that boot problems always result from some sort of change to the system; systems don't just stop working. You need to figure out what has changed. Of course, if you've just made modifications to the system, they will be the prime suspects.

This section lists some of the most common causes of booting problems, along with suggestions for what to do in each case.

Bad or flaky hardware

Check the obvious first.The first thing to do when there is a device failure is to see if there is a simple problem that is easily fixed. Is the device plugged in and turned on? Have any cables connecting it to the system come loose? Does it have the correct SCSI ID (if applicable)? Is the SCSI chain terminated? You get the idea.

Try humoring the device. Sometimes devices are just cranky and can be coaxed back to life. For example, if a disk won't come on line, try power-cycling it. If that doesn't work, try shutting off the power to the entire system. Then power up the devices one by one, beginning with peripherals and ending with the CPU if possible, waiting for each one to settle down before going on to the next device. Sometimes this approach works on the second or third try even after failing on the first. When you decide you've had enough, call field service. When you use this approach, once you've turned the power off, leave it off for a minute or so to allow the device's internal capacitors to discharge fully.

Device failures. If a critical hardware device fails, there is not much you can do except call field service. Failures can occur suddenly, and the first reboot after the system power has been off often stresses marginal devices to the point that they finally fail.

Unreadable filesystems on working disks

You can distinguish this case from the previous one by the kind of error you get. Bad hardware usually generates error messages about the hardware device itself, as a whole. A bad filesystem tends to generate error messages later in the boot process, when the operating system tries to access it.

Bad root filesystem. How you handle this problem depends on which filesystem is damaged. If it is the root filesystem, then you may be able to recreate it from a bootable backup/recovery tape (or image on the network) or by booting from alternate media (such as the distribution tape, CD-ROM, or diskette from which the operating system was installed), remaking the filesystem and restoring its files from backup. In the worst case, you'll have to reinstall the operating system and then restore files that you have changed from backup.

Restoring other filesystems. On the other hand, if the system can still boot to single-user mode, things are not nearly so dire. Then you will definitely be able to remake the filesystem and restore its files from backup.

Damage to non-filesystem areas of a disk

Damaged boot areas. Sometimes, it is the boot partition or even the boot blocks of the root disk that are damaged. Some Unix versions provide utilities for restoring these areas without having to reinitialize the entire disk. You'll probably have to boot from a bootable backup tape or other distribution media to use them if you discover the problem only at boot time. Again, the worst-case scenario is having to reinstall the operating system.

Corrupted partition tables. On PCs, it is possible to wipe out a disk's partition tables if a problem occurs while you are editing them with the fdisk disk partitioning utility. If the power goes off or fdisk hangs, the disk's partition information can be incorrect or wiped out entirely. This problem can also happen on larger systems as well, although its far less common to edit the partition information except at installation (and often not even then).

The most important thing to do in this case is not to panic. This happened to me on a disk where I had three operating systems installed, and I really didn't want to have to reinstall all of them. The fix is actually quite easy: simply rerun fdisk and recreate the partitions as they were before, and all will be well again. However, this does mean that you need to have complete, detailed, and accessible (e.g., hardcopy) records of how the partitions were set up.

Incompatible hardware

Problems with a new device. Sometimes, a system hangs when you try to reboot it after adding new hardware. This can happen when the system does not support the type of device that you've just added, either because the system needs to be reconfigured to do so or because it simply does not support the device.

In the first case, you can reconfigure the system to accept the new hardware by building a new kernel or doing whatever else is appropriate on your system. However, if you find out that the device is not supported by your operating system, you will probably have to remove it to get the system to boot, after which you can contact the relevant vendors for instructions and assistance. It usually saves time in the long run to check compatibility before purchasing or installing new hardware.

Problems after an upgrade.Hardware incompatibility problems also crop up occasionally after operating system upgrades on systems whose hardware has not changed, due to withdrawn support for previously supported hardware or because of undetected bugs in the new release. You can confirm that the new operating system is the problem if the system still boots correctly from bootable backup tapes or installation media from the previous release. If you encounter sudden device-related problems after an OS upgrade, contacting the operating system vendor is usually the best recourse.

Device conflicts. On PCs, devices communicate with the CPU using a variety of methods: interrupt signals, DMA channels, I/O addresses/ports, and memory addresses (listed in decreasing order of conflict likelihood). All devices that operate at the same time must have unique values for the items relevant to it (values are set via jumpers or other mechanisms on the device or its controller or via a software utility provided by the manufacturer for this purpose). Keeping detailed and accurate records of the settings used by all of the devices on the system will make it easy to select appropriate ones when adding a new device and to track down conflicts should they occur.

System configuration errors

Errors in configuration files. This type of problem is usually easy to recognize. More than likely, you've just recently changed something, and the boot process dies at a clearly identifiable point in the process. The solution is to boot to single-user mode and then correct the erroneous configuration file or reinstall a saved, working versions of it.

Unbootable kernels. Sometimes, when you build a new kernel, it won't boot. There are at least two ways that this can occur: you may have made a mistake building or configuring the kernel, or there may be bugs in the kernel that manifest themselves on your system. The latter happens occasionally when updating the kernel to the latest release level on Linux systems and when you forget to run lilo after building a new kernel.

In either case, the first thing to do is to reboot the system using a working, saved kernel that you've kept for just this contingency. Once the system is up, you can track down the problem with the new kernel. In the case of Linux kernels, if you're convinced that you haven't made any mistakes, you can check the relevant newsgroups to see if anyone else has seen the same problem. If no information is available, the best thing to do is wait for the next patch level to become available (it doesn't take very long) and then try rebuilding the kernel again. Frequently, the problem will disappear.

Errors in initialization files are a very common cause of boot problems. The moral of this story is, of course, test, test, test. Note once more that obsessive prudence is your best hope every time.

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March, 12, 2019