|
Softpanorama
(slightly skeptical)
Open Source Software Educational Society |
May the
source be with you,
but remember the KISS principle ;-)
|
Solaris Multipathing
Summary
To verify the system’s failover configuration, or to change the operational status
of IPMP interfaces, use the if_mpadm
Introduction
Network interfaces are exposed to failure because they connect to network cables
and hardware components in the form of switches or hubs. Failure of any of these
interfaces results in network failure, even if the network interface card (NIC)
that is in place does not fail. Sun offers two features that address customer network
bandwidth demands: IPMP and Sun Trunking software.
IPMP enables multiple interfaces with different IP addresses on the same subnet
to be connected to the same network segment. If any one of these interfaces fail,
current network connections through that interface will be migrated to another interface
automatically: physical interface on failed adapter will became logical on another
(working) LAN adapter. This is a pretty neat idea.
In general, multipathing is a method of using redundancy and automatic
fail-over that provides at least two physical paths to a target resource. It allow
recovery from a single network path failure by re-routing data traffic. That is
important for network storage. Varian of multipathing called trunking also
allows for the parallel transmission of data, which can result in faster throughput
and increased scalability.
Solaris 10 supports Solaris network multipathing via:
- IPMP load spreading—outgoing network traffic is able
to utilize several network interfaces
- network trunking — multiple physical network interfaces
are treated as one; combining these interfaces, which is accomplished in the
TCP/IP stack, allows link aggregation and availability
IPMP in Solaris has the following features/capabilities:
- Eliminates a single network adapter as a single point-of-failure in
these cases:
- Network adapter failure detection (failover)
- Network adapter repair detection (failback)
- Provides outbound load spreading when traffic is flowing to multiple destinations.
- Enables interfaces to failover within approximately 10 seconds when using
the default configuration. Can be configured by adjusting the parameters in
the /etc/default/mpathd file.
- Can be configured for both IPv4 and IPv6.
- Allows interfaces to be configured as Standby Interfaces.
These types of interfaces are only used for failover and are not used for outbound
load spreading, unless they are explicitly chosen by an application.
The Sun multipathing software (MPxIO) allows the merging of multiple SCSI layer
paths, such as an iSCSI device exposing the same LUN via two different iSCSI target
names (or the same name with different target portal group tags). In addition, one
FC path and one iSCSI path can be merged by MPxIO if the target device supports
these options. Additional functionality, such as iSCSI Multiple Connections / Session
(MC/S), might be supported in future Solaris releases.
Multiple physical connections between a host and a target provides two major
benefits: availability and link aggregation.
When multiple physical connections exist between host and storage, if one connection
is lost, then I/O can be rerouted through other paths. To maximize availability
and eliminate single points of failure, the following components in a storage architecture
must be made redundant:
- portals on the initiator and target switches
- IP network (various components)
- array controllers
For more information about configuring a high availability network, see
Enterprise Network Design Patterns: High Availability (Sun
BluePrints Online—December, 2003) at
http://www.sun.com/blueprints/1203/817-4683.pdf
Link Aggregation
Multipathing drivers can route data through multiple paths in parallel, resulting
in increased throughput between host and storage. Current commodity Ethernet implementations
support 1Gb/S full duplex, for an aggregate throughput of 2Gb/S (if transmit and
receive traffic is balanced (if, for example, the network supports 1Gb incoming
and 1Gb outgoing traffic). Combining multiple links can scale performance.
IPMP is a product that was first included with the Solaris 8. IPMP provides enhanced
network availability. IPMP enables the Solaris to recover from network path failures.
IPMP also provides increased throughput by spreading the outbound load across interfaces
when multiple network adapters are connected to the same IP link; for example, to
the same Ethernet switch.
If a failure occurs in the network link and an alternate adapter is configured,
the IP address fails over. The network access changes automatically from the failed
adapter to the alternate adapter, allowing uninterrupted access to the network.
When there are multiple network adapters that are connected to the same IP link,
increased throughput can be achieved by spreading the outbound load across multiple
network interfaces.
IPMP has the following features:
- Eliminates a single network adapter as a single point-of-failure in
these cases:
- Network adapter failure detection (failover)
- Network adapter repair detection (failback)
- Provides outbound load spreading when traffic is flowing to multiple destinations.
- Enables interfaces to failover within approximately 10 seconds when
using the default configuration.
- Can be configured by adjusting the parameters in the /etc/default/mpathd
file.
- Can be configured for both IPv4 and IPv6.
- Allows interfaces to be configured as Standby Interfaces. These types
of interfaces are only used for failover and are not used for outbound
load spreading, unless they are explicitly chosen by an application.
IPMP Requirements
- Unique media access control (MAC) addresses must be configured on each
network interface (the default configuration for old Sun network adapters
has all network interfaces on a specific server using the same MAC address).
IPMP requires that all LAN adapters have unique MAC addresses. Switched configurations
use MAC addresses when making network decisions. Therefore for old Sun hardware,
you must change the system’s default configuration to ensure that each LAN adapter
has a unique MAC address to avoid a MAC address conflict.
- Multiple network adapter interfaces must be connected on each subnet.
You can configure IPMP with a single network interface to take advantage of
network failure detection. To use the full benefit of IPMP, make sure that two
or more network interfaces are connected to the same subnet.
- A network adapter group name must be assigned to IPMP interfaces.
Interfaces that are to be deployed as multipath interfaces must belong to a
multipath group. The in.mpathd multipath
process uses the multipath group. Use a meaningful name that does not include
spaces when you choose a group name. The multipath name is local to the system
and is not used across the network.
- A test address is assigned to an interface. The multipath process
uses test addresses, which must be routable addresses, to monitor the status
of each individual interface. Use the test addresses to detect failure and recovery
of an interface. These addresses are deprecated at configuration time to make
sure that they cannot be used to pass network traffic from other applications.
- Additional hosts must exist on the same subnet. The test interfaces
use ICMP echo request, reply, or both to hosts that they reach by addressing
the 224.0.0.1 multicast group or the default router, as listed in the
/etc/defaultrouter file.
Interface Failure Detection and Repair
The in.mpathd process can detect both the
failure and the repair of an interface by:
- Sending ICMP echo requests to the special host (for example default
router) and monitoring responses through the interface test address.
- Monitoring the internal IFF_RUNNING flag on the interface
An interface has failed if either of these two detection methods indicates a
failure. An interface is considered repaired only if both methods report that the
interface is operational and can send and receive packets through the interface.
To detect the failure or repair of interfaces that belong to the multipath group,
the in.mpathd process sends ICMP echo requests
from the logical IPMP interfaces to targets connected to the local network. The
in.mpathd process determines which targets to
probe dynamically. If five consecutive probes do not receive replies, the interface
is considered failed.
Adjust the failure detection time by editing the FAILURE_DETECTION_TIME
variable from the default value of 10,000 milliseconds (10 seconds) in the
/etc/default/mpathd file.
When responses to the ICMP echo requests are not received and a specific time
period has elapsed, the physical interface is considered failed. The IP address
that is associated with the failed address is moved to a new
logical interface associated with another physical interface in the same IPMP group.
Communications that were taking place continue to function as though the original
interface is still working properly.
ICMP echo requests are still attempted through the failed NIC to detect if a
physical interface is repaired.
If all the NICs or targets appear to fail at the same time, this is a group failure,
and no failover is performed. The in.mpathd
process flushes all of the current targets and attempts to discover new targets.
Because in.mpathd process dynamically determines
what targets to probe, you cannot configure the targets. Routers connected to the
link are chosen as targets for probing. If no routers exist on the link, arbitrary
hosts on the link are chosen by sending a multicast packet to the “all hosts” multicast
address. When you configure IPMP, be sure to have at least one additional system
on the network.
You can configure multipathing by changing configuration files and rebooting,
or you can work at the command line to avoid rebooting the system.
Configuring Multipathing Using Configuration Files
This example shows IPMP configuration on an existing configured
qfe0 interface and on an existing but unconfigured
qfe1 interface on the myhost (192.168.1.1) system.
The multipath group is called mpgrp-one. The
test addresses are:
- 192.168.1.50 for the qfe0 interface
- 192.168.1.51 for the qfe1 interface
The data address for the qfe0 interface remains 192.168.1.1, and the data address
for the qfe1 interface is 192.168.1.45.
To configure IPMP, complete the following steps, which are described in greater
detail in the next sections.
1. Verify the Solaris OE release. The /etc/release file contains information
about the installed version of the Solaris. It should be 9 or higher.
2. Configure unique MAC addresses. To determine if unique MAC addresses
are allowed, use the eeprom utility to view the contents of the flash
code electrically erasable programmable read-only memory (EEPROM):
# eeprom local-mac-address?
local-mac-address?=false
The preceding output indicates that the system is still in its default mode and
uses the same MAC address for each interface. This is indicated by the setting
of the local-mac-address? variable to false.
You now use
the eeprom utility to change the
local-mac-address? variable to true:
# eeprom local-mac-address?=true
Verify that the local-mac-address? variable is set to true:
# eeprom local-mac-address?
Note – Depending on the combination of your
system’s firmware and hardware architecture, you must either plumb the interface
or reboot the system to enable unique MAC address assignment after changing the
eeprom variable.
3. Define data and test IP addresses. You can add the data and test IP
addresses to the /etc/inet/hosts file for the
sake of clarity. After editing the /etc/inet/hosts
file, use the tail utility to view the new information:
# tail -5 /etc/inet/hosts
# Modifications made for IPMP
192.168.1.1 myhost # Data address
for qfe0
192.168.1.45 myhost-dat-qfe1 # Data address for qfe1
192.168.1.50 myhost-test0 # qfe0:1 Test address for qfe0
192.168.1.51 myhost-test1 # qfe1:1 Test address for qfe1
4. Configure the interfaces. Multipath information is placed in
the /etc/hostname.qfe0 and
/etc/hostname.qfe1 files. Modify the
/etc/hostname.qfe0 file to contain contents similar to the following:
# cat /etc/hostname.qfe0
myhost netmask + broadcast + group mpgrp-one up \
addif myhost-test0 deprecated netmask + broadcast + -failover up
addif myhost-test0 Creates the next
unused logical interface, and assigns it the IP address associated
with the myhost-test0 name in
/etc/hosts. deprecated Marks the address
as a deprecated address. Addresses that are marked as deprecated are not used
as source addresses for outbound packets unless either there are no other addresses
available on this interface or the application is bound to this address explicitly.
The output from the ifconfig -a command shows
DEPRECATED as one of the flags associated with this interface.
-failover Marks the address as a non-failover
address. Addresses that are marked in this way do not fail
over when the interface fails. The output from the ifconfig
-a command shows NOFAILOVERas
one of the flags associated with this interface.
Create the /etc/hostname.qfe1 file to contain contents similar to the following:
# cat /etc/hostname.qfe1
myhost-dat-qfe1 netmask + broadcast + group mpgrp-one up \
addif myhost-test1 deprecated netmask + broadcast + -failover up
5. Reboot the system, for example using init 6.
6. View the interface configuration. Use ifconfig
-a command. You should get something like
# ifconfig -a
lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 192.168.30.31 netmask ffffff00 broadcast 192.168.30.255
ether 8:0:20:b9:72:23
qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 192.168.1.1 netmask ffffff00 broadcast 192.168.1.255
groupname mpgrp-one
ether 8:0:20:ac:9b:20
qfe0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>
mtu 1500 index 3
inet 192.168.1.50 netmask ffffff00 broadcast 192.168.1.255
qfe1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
inet 192.168.1.45 netmask ffffff00 broadcast 192.168.1.255
groupname mpgrp-one
ether 8:0:20:ac:9b:21
qfe1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>
mtu 1500 index 4
inet 192.168.1.51 netmask ffffff00 broadcast 192.168.1.255
This information above includes the following:
- The interface’s index number is 3, the same as the physical interface that
supports this logical interface.
- The qfe0:1 interface MAC address is
not shown because logical interfaces use the same MAC address as the physical
interface that supports the logical interface.
- The DEPRECATED and NOFAILOVER flags indicate that the interface is not to
be used by any application (other than the in.mpathd
process), and the interface must not be failed if a communication failure
occurs.
- The RUNNING flag is also monitored by the in.mpathd
process to ensure that communications are functioning as expected.
The system remains available to users if either of the multipath network interfaces
fail or become unusable for any reason.
You must know the state of the system if you need to restore it. Before making
any changes to the system, view the system’s interface configuration by performing
the command:
# ifconfig -a
lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4>
mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 192.168.30.31 netmask ffffff00 broadcast 192.168.30.255
ether 8:0:20:b9:72:23
qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 192.168.1.1 netmask ffffff00 broadcast 192.168.1.255
ether 8:0:20:ac:9b:20
The system remains available to users if either of the multipath network interfaces
fail or become unusable for any reason. As a precaution, reboot the system when
it is convenient to verify that multipathing is properly configured during system
boot.
Command Line Utilities-based Configuration
Configure the qfe0 Interface as Part of a Multipath Group.
To configure the qfe0 interface as part of a multipath group, specify the
name of the group, mpgrp-one, of which the qfe0 interface will be a part:
# ifconfig qfe0 group mpgrp-one
View the changes to the interface using ifconfig
-a
Configure a Test Address for the qfe0 Interface
# ifconfig qfe0 addif 192.168.1.50 deprecated netmask
+ broadcast + -failover up
Configure the qfe1 Interface as Part of the qfe0 Interface Multipath Group
Now, you configure the qfe1interface and make it part of the same IPMP group
as the qfe0 interface. Perform the command:
# ifconfig qfe1 plumb myhost-dat-qfe1 netmask +
broadcast + group mpgrp-one up
Now, you configure a test address for the qfe1interface.
# ifconfig qfe1 addif 192.168.1.51 deprecated netmask
+ broadcast + -failover up
Viewing Multipath Operation
To verify the system’s failover configuration, or to change the operational status
of IPMP interfaces, use the if_mpadm utility.
You can use this utility to take an interface offline (detach), by forcing a failover,
and verifying that an alternate interface takes over as expected. If configuration
errors occur, they appear at this stage. Also, use the
if_mpadm utility to reattach a detached interface.
For example, to detach the qfe0 interface,
perform the command:
# if_mpadm -d qfe0
Nov 17 12:40:46 myhost in.mpathd[541]: Successfully failed over from NIC
qfe0 to NIC qfe1
The message indicates that the failover was successful.
Note – This message appears in the console window and is not seen if you are
using an xterm or dtterm window.
To view the status of the interfaces, use the ifconfig
command:
# ifconfig -a
lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 192.168.30.31 netmask ffffff00 broadcast 192.168.30.255
ether 8:0:20:b9:72:23
qfe0: flags=89000842<BROADCAST,RUNNING,MULTICAST,IPv4,NOFAILOVER,OFFLINE> mtu
0 index 3
inet 0.0.0.0 netmask 0
groupname mpgrp-one
ether 8:0:20:ac:9b:20
qfe0:1: flags=89040842<BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,OFFLINE>
mtu 1500 index 3
inet 192.168.1.50 netmask ffffff00 broadcast 192.168.1.255
qfe1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
inet 192.168.1.45 netmask ffffff00 broadcast 192.168.1.255
groupname mpgrp-one
ether 8:0:20:ac:9b:21
qfe1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>
mtu 1500 index 4
inet 192.168.1.51 netmask ffffff00 broadcast 192.168.1.255
qfe1:2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
inet 192.168.1.1 netmask ffffff00 broadcast 192.168.1.255
The detached interface is assigned an IP address of 0.0.0.0, and a new logical
interface, qfe1:2, is automatically created over the functional qfe1 physical interface.
The new logical interface has the IP address that was assigned to the physical qfe0
interface while it was working. To reattach an offline interface, perform the command:
# if_mpadm -r qfe0
Nov 17 12:52:03 myhost in.mpathd[541]: Successfully failed back to NIC qfe0
Note – This message appears in the console window and is not seen if you are
using an xterm or dtterm window.
The message indicates that the failback was successful. To view the status of
the interfaces, use the ifconfig utility:
# ifconfig -a
lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 192.168.30.31 netmask ffffff00 broadcast 192.168.30.255
ether 8:0:20:b9:72:23
qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 192.168.1.1 netmask ffffff00 broadcast 192.168.1.255
groupname mpgrp-one
ether 8:0:20:ac:9b:20
qfe0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>
mtu 1500 index 3
inet 192.168.1.50 netmask ffffff00 broadcast 192.168.1.255
qfe1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
inet 192.168.1.45 netmask ffffff00 broadcast 192.168.1.255
groupname mpgrp-one
ether 8:0:20:ac:9b:21
qfe1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>
mtu 1500 index 4
inet 192.168.1.51 netmask ffffff00 broadcast 192.168.1.255
#
The qfe0 interface is reassigned its original IP address, and the qfe1:2 logical
interface is automatically removed.
Troubleshooting a Multipath Network Configuration
Incorrectly configured network interfaces might not properly fail over
when connectivity to an interface fails for any reason. It is important to thoroughly
test your network interface after you configure IPMP.
Carefully read messages in the /var/adm/messages
file or in the console window to take the proper troubleshooting steps when you
configure and test the IPMP. For example:
# Nov 17 23:19:51 myhost in.mpathd[475]: Failures cannot
be detected
on qfe0 as no IFF_NOFAILOVER address is available
The message indicates that the in.mpathd
process with a process number of 475 senses that IPMP is not properly configured.
To investigate further, use the ifconfig command:
# ifconfig -a
lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 192.168.30.31 netmask ffffff00 broadcast 192.168.30.255
ether 8:0:20:b9:72:23
qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 192.168.1.1 netmask ffffff00 broadcast 192.168.1.255
groupname mpgrp-one
ether 8:0:20:ac:9b:20
The output indicates that the configuration process is not complete. Recall that
IPMP requires a test address on a logical interface for each physical interface.
After defining a test interface with the ifconfig
command, the following message appears:
# Nov 17 23:17:41 myhost in.mpathd[475]: Failure detection
restored
on qfe0 as an IFF_NOFAILOVER address is available
To configure a test interface, use the ifconfig
utility:
# ifconfig qfe0 addif 192.168.1.50
deprecated netmask + \
broadcast + -failover up
Created new logical interface qfe0:1
Setting netmask of qfe0:1 to 255.255.255.0
# Nov 17 23:17:41 myhost in.mpathd[475]: Failure detection restored
on qfe0 as an IFF_NOFAILOVER address is available
The in.mpathd process reports that it can
now perform failure detection. Be aware that more than one interface is required
to provide effective failover. To view the interface configuration, use the
ifconfig command:
# ifconfig -a
lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 192.168.30.31 netmask ffffff00 broadcast 192.168.30.255
ether 8:0:20:b9:72:23
qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 192.168.1.1 netmask ffffff00 broadcast 192.168.1.255
groupname mpgrp-one
ether 8:0:20:ac:9b:20
qfe0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>
mtu 1500 index 3
inet 192.168.1.50 netmask ffffff00 broadcast 192.168.1.255
Both the physical and logical interface are properly configured.
In case of broken links
please try to use Google search. If you find the page please notify
us about new location
docs.sun.com: IP Network Multipathing Administration Guide
Sun documentation. See also PDF version
816-5249
Chapter 27IP Network Multipathing (Overview)
IP Network Multipathing provides both load spreading
and failover when you have multiple network interface cards that
are connected to the same IP link (for example, Ethernet).
This chapter contains the following information:
|
Guide
IP Services
This is very practical discussion that adds a lot to Sun provided information.
Network failover
using Solaris multipathing
Q1: Which command will define an IPMP test address?
- ifconfig hme1:1 plumb 192.11.16.2 netmask + broadcast
+ up
- ifconfig hme0 addif deprecated group one up
- ifconfig hme0 addif 192.11.16.2 deprecated netmask
+ broadcast + -failover up
- eeprom local-mac-address?=true
A: c
Copyright © 1996-2008 by Dr. Nikolai Bezroukov.
www.softpanorama.org was
created as a service to the UN Sustainable Development Networking Programme (SDNP)
in the author free time.
Submit
comments This document is an industrial compilation designed and created
exclusively for educational use and is placed under the copyright of the
Open Content License(OPL).
Original materials copyright belong to respective owners. Quotes are made
for educational purposes only in compliance with the fair use doctrine.
Standard disclaimer: The statements, views and opinions presented on
this web page are those of the author and are not endorsed by, nor do they necessarily
reflect, the opinions of the author present and former employers, SDNP or any other
organization the author may be associated with. We do not warrant the correctness
of the information provided or its fitness for any purpose.
Last modified:
June 02, 2008