Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-) Softpanorama Search

Linux Multipath IO

Linux multipathing provides IO failover and path load sharing for multipathed block devices. The multipath IO support in SUSE Linux Enterprise Server is based on the Device Mapper multipath module of the Linux kernel and the multipath-tools userspace package.

Note that multipathing protects against the failure of path(s) and not the failure of a specific storage device.

A common example of multipath connectivity is a SAN connected storage device. Usually one or more Fibre Channel HBAs from the host will be connected to the fabric switch and the storage controllers will be connected to the same switch.

A simple example of multipath connectivity could be: two HBAs connected to a switch to which the storage controllers are connected. In this case the storage controller can be accessed from either of the HBAs and hence we have multipath connectivity.

In the following diagram each host has two HBAs and each storage device has two controllers. With the given configuration setup each host will have four paths to each of the LUNs in each of the storage devices.

Device mapping multipath IO features automatic configuration of the subsystem for a large variety of setups. Active/passive or active/active (with round-robin load balancing) configurations of up to eight paths to each device are supported.

multipath-tools take care of automatic path discovery and grouping as well as automated path retesting, so that a previously failed path is automatically reinstated when it becomes healthy again. This minimizes the need for administrator attention in a production environment.

Device mapping multipath IO supports partitions (with limitations) and LVM2. Software RAID is also supported, but automatic discovery is not available. To use software RAID with mdadm, /etc/mdadm.conf must be set up correctly.

Currently, device mapping multipath IO is not available for the boot partition, because the boot loader cannot handle multipath IO. Therefore it is recommended to set up a separate boot partition when using multipath IO.

System Configuration

The system must be manually configured to automatically load the device drivers for the controllers to which the multipath IO devices are connected within the INITRD. Therefore add the needed driver module to the variable INITRD_MODULES in the file /etc/sysconfig/ kernel.

If your system, for example, contains a RAID controller accessed by the cciss driver and multipath IO devices connected to a Qlogic controller accessed by the driver qla2xxx, this entry would look like:

   INITRD_MODULES="cciss"

Because the Qlogic driver is not automatically loaded on start-up, add it here:

   INITRD_MODULES="cciss qla2xxx"

After having changed /etc/sysconfig/ kernel, recreate the INITRD on your system with the command mkinitrd.

Configuring multipath-tools

If you are using a storage subsystem that is automatically detected, no further configuration of the multipath-tools is required. Otherwise create /etc/multipath.conf and add an appropriate device entry for your storage subsystem. See /usr/share/doc/packages/multipath-tools/multipath.conf.annotated for a template with extensive comments.

After having set up the configuration, you can perform a dry-run with multipath -v2 -d, which only scans the devices and prints what the setup would look like. The output is similar to the following:

3600601607cf30e00184589a37a31d911 
[size=127 GB]  [features="0"]  [hwhandler="1
    emc"]  
\_ round-robin 0 [first] 
  \_ 1:0:1:2 sdav 66:240  [ready ]
  \_ 0:0:1:2 sdr  65:16   [ready ]
\_ round-robin 0 
  \_ 1:0:0:2 sdag 66:0    [ready ]
  \_ 0:0:0:2 sdc  8:32    [ready ]    
   

•  Name of the device
•  Size of the device
•  Features of the device
•  Hardware handlers involved
•  Priority group 1
•  Priority group 2

Paths are grouped into priority groups. There is only ever one priority group in active use. To model an active/active configuration, all paths end up in the same group. To model active/passive, the paths that should not be active in parallel are placed in several distinct priority groups. This normally happens completely automatically on device discovery.

The output shows the order, the scheduling policy used to balance IO within the group, and the paths for each priority group. For each path, its physical address ( host:bus:target:lun), device node name, major:minor number, and state is shown.

 Enabling the Multipath Deamon

To start the multipath IO services, run the following commands:

/etc/init.d/boot.multipath start
/etc/init.d/multipathd start   

The multipath devices should now show up automatically under /dev/disk/by-name/. The default name is the WWN (World Wide Name) of the logical unit, which you can override using /var/lib/multipath/bindings by setting user_friendly_names in /etc/multipath.conf to yes.

To permanently add multipath IO services to the boot sequence, run the following command:

insserv boot.multipath multipathd

Querying the Status

Querying the multipath IO status outputs the current status of the multipath maps. To query the current MPIO status, run multipath -l.

3600601607cf30e00184589a37a31d911
[size=127 GB][features="0"][hwhandler="1 emc"]
\_ round-robin 0 [active][first]
  \_ 1:0:1:2 sdav 66:240  [ready ][active]
  \_ 0:0:1:2 sdr  65:16   [ready ][active]
\_ round-robin 0 [enabled]
  \_ 1:0:0:2 sdag 66:0    [ready ][active]
  \_ 0:0:0:2 sdc  8:32    [ready ][active]

 Tuning the Failover with Specific Host Bus Adapters

Host bus adapter time-outs are typically set up for non-multipath IO environments, because the only alternative would be to error out the IO and propagate the error to the application. However, with Multipath IO, some faults (like cable failures) should be propagated upwards as fast as possible so that the multipath IO layer can quickly take action and redirect the IO to another, healthy path.

To configure time-outs for your host bus adapter, add the appropriate options to /etc/modprobe.conf.local. For the QLogic 2xxx family of host bus adapters, for example, the following settings are recommended:

options qla2xxx qlport_down_retry=1 ql2xfailover=0 ql2xretrycount=5

Managing IO in Error Situations

In certain scenarios where the driver, the host bus adapter, or the fabric experiences errors leading to loss of all paths, all IO should be queued instead of being propagated upwards.

This can be achieved with the following setting in /etc/multipath.conf.

defaults {
          default_features "1 queue_if_no_path"

}

Because this leads to IO being queued forever unless a path is reinstated, make sure that multipathd is running and works for your scenario. Otherwise, IO might be stalled forever on the affected MPIO device until reboot or until you manually issue

dmsetup message <NAME> 0 fail_if_no_path

This immediately cause all queued IO to fail (replace <NAME> with the the correct map name). You can reactivate queueing by issuing the following command:

dmsetup message <NAME> 0 queue_if_no_path

You can also use these two commands to switch between modes for testing before committing the command to /etc/multipath.conf.

Using the Devices

MPIO devices can be used directly, with LVM, and with mdadm.

 Using the Devices Directly

If you want to use the entire LUNs directly (for example, if you are using the SAN features to partition your storage), you can simply use the /dev/disk/by-name/xxx names directly for mkfs, fstab, your application, etc.

Using LVM2

To make LVM2 recognize the MPIO devices as possible physical volumes, you must modify /etc/lvm/lvm.conf. It is important to modify it in a way that it does not scan and use the physical paths, but only accesses the multipath IO storage through the multipath IO layer. To do so, change the filter and types entry in /etc/lvm/lvm.conf as follows:

filter = [ "a|/dev/disk/by-name/.*|", "r|.*|" ]
types = [ "device-mapper", 253 ]

This allows LVM2 to scan only the by-name paths and reject everything else. If you are also using LVM2 on non-multipath IO devices, make the necessary adjustments to suit your setup.

Using mdadm

The same as for LVM2 applies to mdadm as well—the devices must be accessed by name rather than by physical path. Therefore the DEVICE entry in /etc/mdadm.conf must be modified:

DEVICE /dev/disk/by-name/*

Partitions

Currently it is not possible to partition multipath IO devices themselves. If the underlying physical device is already partitioned, the multipath IO device reflects those partitions and the layer provides /dev/disk/by-name/>name<p1 ... pN devices so you can access the partitions through the multipath IO layer.

As a consequence, the devices need to be partitioned prior to enabling multipath IO. If you change the partitioning in the running system, Multipath IO does not automatically detect and reflect these changes. It must be reinitialized, which usually requires a reboot.

Understanding Multipathing

• Section 5.1.1, What Is Multipathing?

• Section 5.1.2, Benefits of Multipathing

5.1.1 What Is Multipathing?

Multipathing is the ability of a server to communicate with the same physical or logical block storage device across multiple physical paths between the host bus adapters in the server and the storage controllers for the device, typically in Fibre Channel (FC ) or iSCSI SAN environments. You can also achieve multiple connections with direct attached storage when multiple channels are available.

5.1.2 Benefits of Multipathing

Linux multipathing provides connection fault tolerance and can provide load balancing across the active connections. When multipathing is configured and running, it automatically isolates and identifies device connection failures, and reroutes I/O to alternate connections.

Typical connection problems involve faulty adapters, cables, or controllers. When you configure multipath I/O for a device, the multipath driver monitors the active connection between devices. When the multipath driver detects I/O errors for an active path, it fails over the traffic to the device’s designated secondary path. When the preferred path becomes healthy again, control can be returned to the preferred path.

Planning for Multipathing

• Section 5.2.1, Guidelines for Multipathing

• Section 5.2.2, Supported Architectures for Multipath I/O

• Section 5.2.3, Supported Storage Arrays for Multipathing

5.2.1 Guidelines for Multipathing

Use the guidelines in this section when planning your multipath I/O solution.

• Prerequisites

• Vendor-Provided Multipath Solutions

• Disk Management Tasks

• Software RAIDs

• High-Availability Solutions

• Volume Managers

• Virtualization Environments

Prerequisites

• Multipathing is managed at the device level.

• The storage array you use for the multipathed device must support multipathing. For more information, see Section 5.2.3, Supported Storage Arrays for Multipathing.

• You need to configure multipathing only if multiple physical paths exist between host bus adapters in the server and host bus controllers for the block storage device. You configure multipath for the logical device as seen by the server.

Vendor-Provided Multipath Solutions

For some storage arrays, the vendor will provide its own multipathing software to manage multipathing for the array’s physical and logical devices. In this case, you should follow the vendor’s instructions for configuring multipathing for those devices.

Disk Management Tasks

Perform the following disk management tasks before you attempt to configure multipathing for a physical or logical device that has multiple paths:

• Use third-party tools to carve physical disks into smaller logical disks.

• Use third-party tools to partition physical or logical disks. If you change the partitioning in the running system, the Device Mapper Multipath (DM-MP) module does not automatically detect and reflect these changes. DM-MP must be reinitialized, which usually requires a reboot.

• Use third-party SAN array management tools to create and configure hardware RAID devices.

• Use third-party SAN array management tools to create logical devices such as LUNs. Logical device types that are supported for a given array depend on the array vendor.

Software RAIDs

The Linux software RAID management software runs on top of multipathing. For each device that has multiple I/O paths and that you plan to use in a software RAID, you must configure the device for multipathing before you attempt to create the software RAID device. Automatic discovery of multipathed devices is not available. The software RAID is not aware of the multipathing management running underneath.

High-Availability Solutions

High-availability solutions for clustering typically run on top of the multipathing server. For example, the Distributed Replicated Block Device (DRBD) high-availability solution for mirroring devices across a LAN runs on top of multipathing. For each device that has multiple I/O paths and that you plan to use in a DRDB solution, you must configure the device for multipathing before you configure DRBD.

Volume Managers

Volume managers such as LVM2 and EVMS run on top of multipathing. You must configure multipathing for a device before you use LVM2 or EVMS to create segment managers and file systems on it.

Virtualization Environments

When using multipathing in a virtualization environment, the multipathing is controlled in the host server environment. Configure multipathing for the device before you assign it to a virtual guest machine.

5.2.2 Supported Architectures for Multipath I/O

The multipathing drivers and tools in SUSE® Linux Enterprise Server 10 support all seven of the supported processor architectures: IA32, AMD64/EM64T, IPF/IA64, p-Series (32-bit/64-bit), z-Series (31-bit and 64-bit).

5.2.3 Supported Storage Arrays for Multipathing

The multipathing drivers and tools in SUSE Linux Enterprise Server 10 support most storage arrays. The storage array that houses the multipathed device must support multipathing in order to use the multipathing drivers and tools. Some storage array vendors provide their own multipathing management tools. Consult the vendor’s hardware documentation to determine what settings are required.

• Storage ArraysThat Are Automatically Detected for Multipathing

• Tested Storage Arrays for Multipathing Support

• Storage Arrays that Require Specific Hardware Handlers

Storage ArraysThat Are Automatically Detected for Multipathing

The multipath-tools package automatically detects the following storage arrays:

• 3PARdata VV
• Compaq* HSV110
• Compaq MSA1000
• DDN SAN MultiDirector
• DEC* HSG80
• EMC* CLARiiON* CX
• EMC Symmetrix
• FSC CentricStor*
• Hewlett Packard* (HP*) A6189A
• HP HSV110
• HP HSV210
• HP Open-
• Hitachi* DF400
• Hitachi DF500
• Hitachi DF600
• IBM* 3542
• IBM ProFibre 4000R
• NetApp*
• SGI* TP9100
• SGI TP9300
• SGI TP9400
• SGI TP9500
• STK OPENstorage DS280
• Sun* StorEdge 3510
• Sun T4

In general, most other storage arrays should work. When storage arrays are automatically detected, the default settings for multipathing apply. If you want non-default settings, you must manually create and configure the /etc/multipath.conf file. For information, see Section 5.4.5, Creating and Configuring the /etc/multipath.conf File.

Hardware that is not automatically detected requires an appropriate entry for configuration in the DEVICES section of the /etc/multipath.conf file. In this case, you must manually create and configure the configuration file. For information, see Section 5.4.5, Creating and Configuring the /etc/multipath.conf File.

Consider the following caveats:

• Not all of the storage arrays that are automatically detected have been tested on SUSE Linux Enterprise Server. For information, see Tested Storage Arrays for Multipathing Support.

• Some storage arrays might require specific hardware handlers. A hardware handler is a kernel module that performs hardware-specific actions when switching path groups and dealing with I/O errors. For information, see Storage Arrays that Require Specific Hardware Handlers.

Tested Storage Arrays for Multipathing Support

The following storage arrays have been tested with SUSE Linux Enterprise Server:

• EMC
• Hitachi
• Hewlett-Packard/Compaq
• IBM
• NetApp
• SGI

Most other vendors’ storage arrays should also work. Consult your vendor’s documentation for guidance. For a list of the default storage arrays recognized by the multipath-tools package, see Storage ArraysThat Are Automatically Detected for Multipathing.

Storage Arrays that Require Specific Hardware Handlers

Storage arrays that require special commands on failover from one path to the other or that require special nonstandard error handling might require more extensive support. Therefore, the Device Mapper Multipath service has hooks for hardware handlers. For example, one such handler for the EMC CLARiiON CX family of arrays is already provided.

IMPORTANT:Consult the hardware vendor’s documentation to determine if its hardware handler must be installed for Device Mapper Multipath.

The multipath -t command shows an internal table of storage arrays that require special handling with specific hardware handlers. The displayed list is not an exhaustive list of supported storage arrays. It lists only those arrays that require special handling and that the multipath-tools developers had access to during the tool development.

IMPORTANT:Arrays with true active/active multipath support do not require special handling, so they are not listed for the multipath -t command.

A listing in the multipath -t table does not necessarily mean that SUSE Linux Enterprise Server was tested on that specific hardware. For a list of tested storage arrays, see Tested Storage Arrays for Multipathing Support.

Multipath Management Tools

The multipathing support in SUSE Linux Enterprise Server 10 is based on the Device Mapper Multipath module of the Linux 2.6 kernel and the multipath-tools userspace package. You can use mdadm to view the status of multipathed devices.

• Section 5.3.1, Device Mapper Multipath Module

• Section 5.3.2, Multipath I/O Management Tools

• Section 5.3.3, Using mdadm for Multipathed Devices

• Section 5.3.4, The Linux multipath(8) Command

5.3.1 Device Mapper Multipath Module

The Device Mapper Multipath (DM-MP) module provides the multipathing capability for Linux. DM-MP is the preferred solution for multipathing on SUSE Linux Enterprise Server 10. It is the only multipathing option shipped with the product that is completely supported by Novell® and SUSE.

DM-MP features automatic configuration of the multipathing subsystem for a large variety of setups. Configurations of up to 8 paths to each device are supported. Configurations are supported for active/passive (one path active, others passive) or active/active (all paths active with round-robin load balancing).

The DM-MP framework is extensible in two ways:

• Using specific hardware handlers. For information, see Storage Arrays that Require Specific Hardware Handlers.

• Using more sophisticated load-balancing algorithms than round-robin

The user-space component of DM-MP takes care of automatic path discovery and grouping, as well as automated path retesting, so that a previously failed path is automatically reinstated when it becomes healthy again. This minimizes the need for administrator attention in a production environment.

DM-MP protects against failures in the paths to the device, and not failures in the device itself. If one of the active paths is lost (for example, a network adapter breaks or a fiber-optic cable is removed), I/O is redirected to the remaining paths. If the configuration is active/passive, then the path fails over to one of the passive paths. If you are using the round-robin load-balancing configuration, the traffic is balanced across the remaining healthy paths. If all active paths fail, inactive secondary paths must be waked up, so failover occurs with a delay of approximately 30 seconds.

If a disk array has more than one storage processor, make sure that the SAN switch has a connection to the storage processor that owns the LUNs you want to access. On most disk arrays, all LUNs belong to both storage processors, so both connections are active.

NOTE:On some disk arrays, the storage array manages the traffic through storage processors so that it presents only one storage processor at a time. One processor is active and the other one is passive until there is a failure. If you are connected to the wrong storage processor (the one with the passive path) you might not see the expected LUNs, or you might see the LUNs but get errors when trying to access them.

Table 5-1 Multipath I/O Features of Storage Arrays

Device Mapper Multipath detects every path for a multipathed device as a separate SCSI device. The SCSI device names take the form /dev/sdN, where N is an autogenerated letter for the device, beginning with a and issued sequentially as the devices are created, such as /dev/sda, /dev/sdb, and so on. If the number of devices exceeds 26, the letters are duplicated so that the next device after /dev/sdz will be named /dev/sdaa, /dev/sdab, and so on.

If multiple paths are not automatically detected, you can configure them manually in the /etc/multipath.conf file. The multipath.conf file does not exist until you create and configure it. For information, see Section 5.4.5, Creating and Configuring the /etc/multipath.conf File.

5.3.2 Multipath I/O Management Tools

The multipath-tools user-space package takes care of automatic path discovery and grouping. It automatically tests the path periodically, so that a previously failed path is automatically reinstated when it becomes healthy again. This minimizes the need for administrator attention in a production environment.

Table 5-2 Tools in the multipath-tools Package

For a list of files included in this package, see the multipath-tools Package Description.

1. Ensure that the multipath-tools package is installed by entering the following at a terminal console prompt:

   rpm -q multipath-tools

   If it is installed, the response repeats the package name and provides the version information, such as:

   multipath-tools-04.7-34.23

   If it is not installed, the response reads:

   package multipath-tools is not installed
   

Using mdadm for Multipathed Devices

In SUSE Linux Enterprise Server 10, Udev is the default device handler, and devices are automatically known to the system by the Worldwide ID instead of by the device node name. This resolves problems in previous releases where mdadm.conf and lvm.conf did not properly recognize multipathed devices.

Just as for LVM2, mdadm requires that the devices be accessed by the ID rather than by the device node path. Therefore, the DEVICE entry in /etc/mdadm.conf should be set as follows:

DEVICE /dev/disk/by-id/*

This is the default handling in SUSE Linux Enterprise Server 10 and later, as noted above.

To verify that mdadm is installed:

1. Ensure that the mdadm package is installed by entering the following at a terminal console prompt:

   rpm -q mdadm
   

   If it is installed, the response repeats the package name and provides the version information. For example:

   mdadm-2.6-0.11
   

   If it is not installed, the response reads:

   package mdadm is not installed
   

For information about modifying the /etc/lvm/lvm.conf file, see Section 5.12.2, Using LVM2.

5.3.4 The Linux multipath(8) Command

Use the Linux multipath(8) command to configure and manage multipathed devices.

General syntax for the multipath(8) command:

multipath [-v verbosity] [-d] [-h|-l|-ll|-f|-F] [-p failover|multibus|group_by_serial|group_by_prio|group_by_node_name] 

General Examples

Configure multipath devices:

multipath

Configure a specific multipath device:

multipath devicename

Replace devicename with the device node name such as /dev/sdb (as shown by udev in the $DEVNAME variable), or in the major:minor format.

Selectively suppress a multipath map, and its device-mapped partitions:

multipath -f

Display potential multipath devices, but do not create any devices and do not update device maps (dry run):

multipath -d

Configure multipath devices and display multipath map information:

multipath -v2 <device> 

multipath -v3

The -v2 option in multipath -v2 -d shows only local disks. Use the -v3 option to show the full path list. For example:

multipath -v3 -d

Display the status of all multipath devices, or a specified multipath device:

multipath -ll 

multipath -ll <device>

Flush all unused multipath device maps (unresolves the multiple paths; it does not delete the device):

multipath -F

multipath -F <device> 

Set the group policy:

multipath -p [failover|multibus|group_by_serial|group_by_prio|group_by_node_name] 

Specify one of the group policy options that are described in Table 5-3:

Table 5-3 Group Policy Options for the multipath -p Command

Old News ;-)

[Oct 6, 2008] IBM Systems Information Center

[Oct 5, 2008] Crash course on multipath

Setup procedure

• Install the multipath-tools package.
• Edit /etc/init.d/multipathd: add iscsi to Required-Start line. [SUSE]
• Activate on boot [SUSE]:

chkconfig boot.multipath on
chkconfig multipathd on

• Activate now:

service multipathd start

• find out WWIDs:

multipath -v2 -d

• Set up aliases in /etc/multipath.conf:

        multipaths {
                multipath {
                        wwid    averybiglonghexstringthatsimpossibletoread
                        alias   something_readable
                }
                multipath {
                        wwid    anotherverybiglonghexstringequallyhardtoread
                        alias   something_else_readable
                }
        }

• activate multipaths:

multipath -v2

• verify:

multipath -l

• verify (you should see sensible device names in the output):

ls -la /dev/mapper

Links

• SUSE article on multipath setup

[Oct 5, 2008] Linux Multipath Howto (RHAS4)

Steps

1. up2date device-mapper-multipath
2. Edit /etc/multipath.conf
   For detailed information, see: "SAN Persistent Binding and Multipathing in the 2.6 Kernel"
3. modprobe dm-multipath
4. modprobe dm-round-robin
5. service multipathd start
6. multipath -v2
   Will show multipath luns and groups. Look for the multipath group number, this is the dm-# listed in /proc/partitions. The multipath lun is accessed via the /dev/dm-# device entry.
7. Format each SCSI DEVICE:
   • sfdisk /dev/sdX
8. (Optional) Create multipath devices for each partition:
   (not needed if using LVM, since you will just mount the logical volume device)
   • kpartx -a /dev/dm-#
9. Enable multipath to start on bootup:
   • chkconfig multipathd on

Other useful cmds

multipath -F

Clear all multipath bindings. Useful when making changes to /etc/multipath.conf (clear multipath map, then run multipath to read the config file and build a new map).

multipath -v3 -ll

List lots of information about all known disks, what multipath groups they belong to, settings, etc...
(NOTE: the -ll also seems to force multipathd to pick up (rescan for) new devices that might have been added to the system but not recognized by mutlipathd yet.)

dmsetup ls --target=multipath

Determine multipath device assignments on a system.

Links & References

Device Mapper Resource Page

List of sources and related packages, etc...

How do I setup device-mapper multipathing in Red Hat Enterprise Linux 4?

Redhat Knowledge Base, article id: 7170
Basically the same as this page, but more generic.

How do I make device mapper multipath ignore my local disks when generating the multipath maps in Red Hat Enterprise Linux 4?

Redhat Knowledge Base, article id: 7319

How can I add more products into the multipathing database?

Redhat Knowledge Base, article id: 8131

Recommended Links

IBM Systems Information Center/Multipath installation on SUSE Linux Enterprise Server 10 SP2

How to setup - use multipathing on SLES SLES 9