Softpanorama
May the source be with you, but remember the KISS principle ;-)

Contents Bulletin Scripting in shell and Perl Network troubleshooting History Humor

Installing Mellanox InfiniBand Driver
on RHEL 6.5

News

InfiniBand

Recommended Links

Setting up a basic infiniband network

 Troubleshooting InfiniBand connection issues using OFED tools

Message Passing Interface
Clustered Parallel File Systems Infiniband HOWTO Loopback filesystem Burning CD and DVD on Linux Linux Swap filesystem RHEL6 registration on proxy protected network
Networking NTP configuration Managing Disks by UUID Linux Multipath Linux Software RAID udev
Booting from DVD problem Move config files from one server to another Modifying ISO image Oracle Linux Installation Linux Disk Management Installation of Red Hat from a USB drive

Mounting Linux filesystems

Restoration of RHEL

Installing X11 and Gnome Desktop in RHEL

Admin Horror Stories

Humor

Etc

By default, the Mellanox ConnectX-3 card is not natively supported by CentOS 6.x or RHEL 6.x

Mellanox offers  set of protocol software and driver for Linux with the ConnectX®-2 / ConnectX®-3 EN NICs with Ethernet.

-rw-r--r--.  1 root root 194222080 Feb  7 17:20 MLNX_OFED_LINUX-2.1-1.0.0-rhel6.5-x86_64.tar
NOTE:
RHEL 6.5 support is provided with version 2.1 of MLNX OFED 

IMPORTANT: Register and patch OS before install of  MLNX OFED 

Two issues:

Components of OpenFabrics Enterprise Distribution for Linux

As the popularity of InfiniBand technology increases, the number of Linux distributions and open source organizations producing drivers and tools will increase. For up-to-date information, check with open source organizations and your current vendors.

The OpenFabrics organization is the Open Software solution in the InfiniBand software space and OpenFabrics Enterprise Distribution (OFED) is the InfiniBand suite of software produced by this organization. Various vendors contribute their drivers (and other software components) to OFED.

OFED contains the following components:

Installing the InfiniBand OFED Package Linux

View the matrix of MLNX_OFED driver versions vs. supported hardware and firmware for Mellanox products.

Warning: this is an old info extracted form Sun documentation. Despite discrepancies with version 2.1 it is still useful... See MLNX_OFED 2.1-X Release Notes

1. Verify what card you have:

lspci |grep -i Mellanox 
03:00.0 Network controller [0207]: Mellanox Technologies MT27500 Family [ConnectX-3]

2. From root, extract the files by typing:

tar -zxvf MLNX_OFED_LINUX-2.1-1.0.0-rhel6.5-x86_64.tar
3. From the create by tar  directory initiate the installation process by typing ./mlnxofedinstall - This is the MLNX_OFED_LINUX installation script.
# ./mlnxofedinstall
Logs dir: /tmp/MLNX_OFED_LINUX-2.1-1.0.0.3038.logs
Error: One or more required packages for installing MLNX_OFED are missing.
Please install the following packages using your Linux distribution Package Management tool:
tk
libnl-devel
 
# yum install tk
Loaded plugins: product-id, refresh-packagekit, rhnplugin, security, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
This system is receiving updates from RHN Classic or RHN Satellite.
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package tk.x86_64 1:8.5.7-5.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================
 Package             Arch                    Version                        Repository                             Size
========================================================================================================================
Installing:
 tk                  x86_64                  1:8.5.7-5.el6                  rhel-x86_64-server-6                  1.4 M

Transaction Summary
========================================================================================================================
Install       1 Package(s)

Total download size: 1.4 M
Installed size: 0
Is this ok [y/N]: y
Downloading Packages:
tk-8.5.7-5.el6.x86_64.rpm                                                                        | 1.4 MB     00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : 1:tk-8.5.7-5.el6.x86_64                                                                              1/1
  Verifying  : 1:tk-8.5.7-5.el6.x86_64                                                                              1/1

Installed:
  tk.x86_64 1:8.5.7-5.el6

Complete!
 
# yum install libnl-devel
Loaded plugins: product-id, refresh-packagekit, rhnplugin, security, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
This system is receiving updates from RHN Classic or RHN Satellite.
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package libnl-devel.x86_64 0:1.1.4-2.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================
 Package                    Arch                  Version                     Repository                           Size
========================================================================================================================
Installing:
 libnl-devel                x86_64                1.1.4-2.el6                 rhel-x86_64-server-6                707 k

Transaction Summary
========================================================================================================================
Install       1 Package(s)

Total download size: 707 k
Installed size: 0
Is this ok [y/N]: y
Downloading Packages:
libnl-devel-1.1.4-2.el6.x86_64.rpm                                                               | 707 kB     00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : libnl-devel-1.1.4-2.el6.x86_64                                                                       1/1
  Verifying  : libnl-devel-1.1.4-2.el6.x86_64                                                                       1/1

Installed:
  libnl-devel.x86_64 0:1.1.4-2.el6

Complete!
---: /home/bezroun/IB
[0]root@lustwzb1: # yum install libnl-devel
Loaded plugins: product-id, refresh-packagekit, rhnplugin, security, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
This system is receiving updates from RHN Classic or RHN Satellite.
Setting up Install Process
Package libnl-devel-1.1.4-2.el6.x86_64 already installed and latest version
Nothing to do
 
# ./mlnxofedinstall
Logs dir: /tmp/MLNX_OFED_LINUX-2.1-1.0.0.3995.logs
This program will install the MLNX_OFED_LINUX package on your machine.
Note that all other Mellanox, OEM, OFED, or Distribution IB packages will be removed.
Do you want to continue?[y/N]:y


/bin/rpm --nosignature -e --allmatches --nodeps libibverbs librdmacm infinipath-psm libibverbs librdmacm compat-opensm-libs infinipath-psm compat-openmpi compat-openmpi-psm

Starting MLNX_OFED_LINUX-2.1-1.0.0 installation ...

Installing mlnx-ofa_kernel RPM
Preparing...                ##################################################
mlnx-ofa_kernel             ##################################################
Installing kmod-mlnx-ofa_kernel 2.1 RPM
Preparing...                ##################################################
kmod-mlnx-ofa_kernel        ##################################################
Installing mlnx-ofa_kernel-devel RPM
Preparing...                ##################################################
mlnx-ofa_kernel-devel       ##################################################
Installing kmod-kernel-mft-mlnx any RPM
Preparing...                ##################################################
kmod-kernel-mft-mlnx        ##################################################
Installing knem-mlnx RPM
Preparing...                ##################################################
knem-mlnx                   ##################################################
Installing kmod-knem-mlnx 1.1.90mlnx2 RPM
Preparing...                ##################################################
kmod-knem-mlnx              ##################################################
Installing ummunotify-mlnx RPM
Preparing...                ##################################################
ummunotify-mlnx             ##################################################
Installing kmod-ummunotify-mlnx 1.0 RPM
Preparing...                ##################################################
kmod-ummunotify-mlnx        ##################################################
Installing mpi-selector RPM
Preparing...                ##################################################
mpi-selector                ##################################################
Installing user level RPMs:
Preparing...                ##################################################
ofed-scripts                ##################################################
Preparing...                ##################################################
libibverbs                  ##################################################
Preparing...                ##################################################
libibverbs-devel            ##################################################
Preparing...                ##################################################
libibverbs-devel-static     ##################################################
Preparing...                ##################################################
libibverbs-utils            ##################################################
Preparing...                ##################################################
libmlx4                     ##################################################
Preparing...                ##################################################
libmlx4-devel               ##################################################
Preparing...                ##################################################
libmlx5                     ##################################################
Preparing...                ##################################################
libmlx5-devel               ##################################################
Preparing...                ##################################################
libcxgb3                    ##################################################
Preparing...                ##################################################
libcxgb3-devel              ##################################################
Preparing...                ##################################################
libcxgb4                    ##################################################
Preparing...                ##################################################
libcxgb4-devel              ##################################################
Preparing...                ##################################################
libnes                      ##################################################
Preparing...                ##################################################
libnes-devel-static         ##################################################
Preparing...                ##################################################
libipathverbs               ##################################################
Preparing...                ##################################################
libipathverbs-devel         ##################################################
Preparing...                ##################################################
libibcm                     ##################################################
Preparing...                ##################################################
libibcm-devel               ##################################################
Preparing...                ##################################################
libibumad                   ##################################################
Preparing...                ##################################################
libibumad-devel             ##################################################
Preparing...                ##################################################
libibumad-static            ##################################################
Preparing...                ##################################################
libibmad                    ##################################################
Preparing...                ##################################################
libibmad-devel              ##################################################
Preparing...                ##################################################
libibmad-static             ##################################################
Preparing...                ##################################################
ibsim                       ##################################################
Preparing...                ##################################################
ibacm                       ##################################################
Preparing...                ##################################################
librdmacm                   ##################################################
Preparing...                ##################################################
librdmacm-utils             ##################################################
Preparing...                ##################################################
librdmacm-devel             ##################################################
Preparing...                ##################################################
opensm-libs                 ##################################################
Preparing...                ##################################################
opensm                      ##################################################
Preparing...                ##################################################
opensm-devel                ##################################################
Preparing...                ##################################################
opensm-static               ##################################################
Preparing...                ##################################################
infiniband-diags            ##################################################
Preparing...                ##################################################
fca                         ##################################################
INFO: updating ...

IMPORTANT NOTE:
===============

- The FCA Manager and FCA MPI Runtime library are installed in /opt/mellanox/fca directory.
- The FCA Manager will not be started automatically.
- To start FCA Manager now, type:
    /etc/init.d/fca_managerd start

- There should be single process of FCA Manager running per fabric.

- To start FCA Manager automatically after boot, type:
    /etc/init.d/fca_managerd install_service

- Check /opt/mellanox/fca/share/doc/fca/README.txt for quick start instructions.

Preparing...                ##################################################
dapl                        ##################################################
Preparing...                ##################################################
dapl-devel                  ##################################################
Preparing...                ##################################################
dapl-devel-static           ##################################################
Preparing...                ##################################################
dapl-utils                  ##################################################
Preparing...                ##################################################
perftest                    ##################################################
Preparing...                ##################################################
mstflint                    ##################################################
Preparing...                ##################################################
mft                         ##################################################
Preparing...                ##################################################
srptools                    ##################################################
Preparing...                ##################################################
rds-tools                   ##################################################
Preparing...                ##################################################
rds-devel                   ##################################################
Preparing...                ##################################################
ibutils2                    ##################################################
Preparing...                ##################################################
ibutils                     ##################################################
Preparing...                ##################################################
cc_mgr                      ##################################################
Preparing...                ##################################################
dump_pr                     ##################################################
Preparing...                ##################################################
ar_mgr                      ##################################################
Preparing...                ##################################################
ibdump                      ##################################################
Preparing...                ##################################################
infiniband-diags-compat     ##################################################
Preparing...                ##################################################
qperf                       ##################################################
Preparing...                ##################################################
mxm                         ##################################################
Preparing...                ##################################################
openmpi                     ##################################################
Preparing...                ##################################################
openmpi                     ##################################################
Preparing...                ##################################################
bupc                        ##################################################
Preparing...                ##################################################
infinipath-psm              ##################################################
Preparing...                ##################################################
infinipath-psm-devel        ##################################################
Preparing...                ##################################################
mvapich2                    ##################################################
Preparing...                ##################################################
openshmem                   ##################################################
Preparing...                ##################################################
hcoll                       ##################################################
Preparing...                ##################################################
libibprof                   ##################################################
Preparing...                ##################################################
libvma                      ##################################################
- Changing max locked memory to unlimited (in /etc/security/limits.conf)
  Please log out from the shell and login again in order to update this change
  Read more about this topic in the VMA's User Manual

- VMA README.txt is installed at: /usr/share/doc/libvma-6.5.9-0/README.txt
- Please refer to VMA journal for the latest changes: /usr/share/doc/libvma-6.5.9-0/journal.txt
Preparing...                ##################################################
mlnxofed-docs               ##################################################
Preparing...                ##################################################
mpitests_mvapich2__1_9      ##################################################
Preparing...                ##################################################
mpitests_openmpi__1_6_5     ##################################################
Preparing...                ##################################################
mpitests_openmpi__1_7_4     ##################################################
Device (03:00.0):
        03:00.0 Network controller [0207]: Mellanox Technologies MT27500 Family [ConnectX-3]
        Link Width: 8x
        PCI Link Speed: Unknown

Device (03:00.0):
        03:00.0 Network controller [0207]: Mellanox Technologies MT27500 Family [ConnectX-3]
        WARNING - device 03:00.0 The MaxReadRequest size is set too low (512 bytes) and will affect performance.
        Please consult your server's vendor and if possible change BIOS settings or use setpci to configure MaxReadReq to 4096 bytes.
        # /sbin/setpci -s 03:00.0 68.W
        2xxx
        Change to 4096 bytes:
        # /sbin/setpci -s 03:00.0 68.W=5xxx


Installation finished successfully.

Attempting to perform Firmware update...
Querying Mellanox devices firmware ...

Device #1:
----------

  Device:        0000:03:00.0
  Part Number:
  Description:
  PSID:          DEL0A40000028

  Versions:      Current        Available
     FW          2.11.0550      N/A
     PXE         3.4.0000       N/A

  Status:        No matching image found


Log File: /tmp/MLNX_OFED_LINUX-2.1-1.0.0.3995.logs/fw_update.log
Failed to update Firmware.
See /tmp/MLNX_OFED_LINUX-2.1-1.0.0.3995.logs/fw_update.log
Please reboot your system for the changes to take effect.
 

IMPORTANT: After a successful installation, reboot the Server

IMPORTANT: After a successful installation, reboot the Server

The Sun Blade Server Module is configured now to start up the InfiniBand software on reboot (ONBOOT=yes).

If this is not the desired behavior, you can edit the /etc/infiniband/openib.conf file, changing ONBOOT to equal no. You can also manually control basic InfiniBand behavior by entering the following command:

/etc/init.d/openibd option 

where option can be start, stop, or status.

 

After the reboot, the Server Module should come up as a functional member of the InfiniBand fabric.

To Verify Driver Installation on Linux

1. Verify that the Linux software driver is installed and attached to the IB EM by typing the openibd status command.


Note - When using the openibd command, type the entire path as shown in the example.


The following example shows the IB driver installed, running and presenting one IB HCA channel or network device (ibn) to the OS. In the example, the Linux network device appears as ib0.
> /etc/init.d/openibd status
     HCA driver loaded
Configured devices:
ib0
Currently active devices:
ib0
     The following modules are also loaded: 
ib_cm
ip_ipoib
.
.
.

2. To view details of operational status, type the ibstat command.

The following example shows one operational IB channel into the IB fabric (or network). The LinkUp state indicates active participation in an IB fabric. It is present as lid 69 and it is being managed by lid 2.

	> ibstat CA 'mthca0' CA type: MT25204 Number of 
	ports: 1 Firmware version: 1.1.0 Hardware version: a0 Node GUID: 0x001b00000ca72620 
	System image GUID: 0x001b00000ca72623 Port 1 State: Active Physical state: LinkUp 
	Rate: 20 Base lid: 69 LMC: 0 SM lid: 2 Capability mask: 0x02510a68 Port GUID: 
	0x001b00000ca72621

You can also verify that the InfiniBand fabric is operational by entering the ibnetdiscover command. The output from this command will list all the nodes, as shown in the following sample output.

> ibnetdiscover
#
# Topology file: generated on Thu Jan 11 15:19:59 2007
#
# Max of 4 hops discovered
# Initiated from node 001b00000ca72620 port 001b00000ca72621 
 
vendid=0x8f1
devid=0x5a31
sysimgguid=0x8f10400411ef9
switchguid=0x8f10400411ef8
 
Switch  24 "S-0008f10400411ef8"    # Switch port 0 lid 9
[21]       "H-0002c90109761ea0"[2]
[12]       "S-0005ad00000161ba"[5]
[7]        "H-001b00000ca72630"[1]
[6]        "H-001b00000ca72620"[1]
vendid=0x5ad
devid=0xa87c
sysimgguid=0x5ad01010161b6
switchguid=0x5ad00000161ba 
Switch  8 "S-0005ad00000161ba"    # Switch - U3 port 0 lid 3
[4]"       H-0005ad0000011310"[1]
[3]       "S-0005ad00000161b6"[1]
[2]       "S-0005ad00000161b6"[2] 
[1]       "S-0005ad00000161b8"[3]
[5]       "S-0008f10400411ef8"[12]
.
.
.
vendid=0x2c9
devid=0x6274
sysimgguid=0x1b00000ca72633
caguid=0x1b00000ca72630 
Ca  1 "H-001b00000ca72630"  # 4x DDR IB 10-Port PCIe Network Express Module
[1]     "S-0008f10400411ef8"[7]     # lid 68 lmc 0n

Top updates

Softpanorama Switchboard
Softpanorama Search


NEWS CONTENTS

Old News ;-)

ConnectX-3 support for RHEL 6.5 Mellanox Interconnect Community

[Feb 06, 2014]  CentOS 6.4 Mellanox 10GB Driver Install on the Open Compute Windmill V3 server

Our build out of the Open Compute Windmill V3 server utilizes the Mellanox MCX342A XCAN 10GbE, 2-p SFP+ mezzanine card. By default, the Mellanox card is not natively supported by CentOS 6.4.

[root@server /]# lspci | grep Mellanox
06:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]

That being said, I had to download the appropriate driver from Mellanox here.

That being said, I also needed quite a few RPM’s to do the driver install, I did:

yum -y install kernel-source kernel headers gcc rpm-build  gcc make patch redhat-rpm-config

Even then, I was getting an error:

No kernel sources/headers found for 2.6.32-358.el6.x86_64 kernel.

As it turns out, the symbolic link of /lib/modules/2.6.32-358.el6.x86_64/build to ../../../usr/src/kernels/xxxxxxx was incorrect. I had to do:

ln -s ../../../usr/src/kernels/2.6.32-358.23.2.el6.x86_64 build

within /lib/modules/2.6.32-358.el6.x86_64/

Only then was I able to correctly install the Mellanox driver.

ConnectX-3 support for RHEL 6.5

This question is Not Answered.

I have the following ethernet card and after the upgrade to RHEL 6.,5 it stopped to work and I had to roll back to previous kernel.

Even downloading the latest driver it gets the following error.

MLNX_OFED_LINUX-2.0-3.0.0-rhel6.4-x86_64]# ./mlnxofedinstall

Error: The current MLNX_OFED_LINUX is intended for rhel6.4

# cat /etc/redhat-release

Red Hat Enterprise Linux Server release 6.5 (Santiago)

Where is the plan to release a driver for RHEL 6.5?

More information regarding the ethernet card.

Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]

Product Name: CX312A - ConnectX-3 SFP+

Part number: 00W0054 

Engineering changes: A1

Vendor specific: PCIe Gen3 x8 

Thank you in advance.

  • Re: ConnectX-3 support for RHEL 6.5
     

    I was able to it to work with 6.5 by reverting grub back to kernel version 2.6.32-358, doing a yum downgrade for the kernel-{headers,devel,firmware} rpms, and then editing a couple of files in the iso and building a new one. The diffs for those are

     

    # diff iso1/distro iso2/distro

    1c1

    < rhel6.5

    ---

    > rhel6.4

     

    # diff iso1/mlnxofedinstall iso2/mlnxofedinstall

    158a159

    >

    260,262d260

    < } elsif ($dist_rpm =~ /redhat-release-.*-6.5|sl-release-6.5|centos-release-6-5/) {

    <     $DISTRO = "RHEL6.5";

    <     $rpm_distro = "rhel6u5";

    654d651

    <     print "dist = $MLNX_OFED_DISTRO\ndist = $DISTRO\nrpmdist = $rpm_distro\n";

    779,780c776

    <     if (($DISTRO eq "RHEL6.5" and $kernel !~ /2.6.32-358.*el6\.$arch/) or

    <         ($DISTRO eq "RHEL6.4" and $kernel !~ /2.6.32-358.*el6\.$arch/) or

    ---

    >     if (($DISTRO eq "RHEL6.4" and $kernel !~ /2.6.32-358.*el6\.$arch/) or

     

    # diff iso1/mlnx_add_kernel_support.sh iso2/mlnx_add_kernel_support.sh

    339,341d338

    <         redhat-release*-6.5*|centos-release-6-5*|sl-release-6.5*)

    <         distro=rhel6.5

    <         ;;

docs.oracle.com

If you have installed current releases of Red Hat Enterprise Linux Advanced Server (RHEL AS 4-U3 or later) or SUSE Linux Enterprise Server (SLES9 SP3 or later, SLES10) on a Sun Blade Server Module and you have installed the bundled drivers and OFED Release 1.2.5 or later, you do not need to install or configure additional drivers to support the IB ExpressModule (IB EM).

Specifically, RHEL AS 4-U4 contains support in the kernel for HCA hardware produced by Mellanox (mthca driver). The kernel also includes core InfiniBand modules, which provide the interface between the lower-level hardware driver and the upper-layer InfiniBand protocol drivers. The InfiniBand modules provide user space access to InfiniBand hardware.

The kernel also includes the Sockets Direct Protocol (SDP) driver, IP over Infiniband (IPoIB) and the SCSI RDMA Protocol (SRP) driver.

RHEL AS 4-U4 includes the following user space packages:


 
Note - These package names can change, depending on the Linux OS.


 

The packages selected to support any given configuration will vary. TABLE 3-1 lists the packages considered the absolute minimum needed to support the environment described in this guide.


 
TABLE 3-1 Required Packages for InfiniBand Support

Package

Command Enabled

Description

kernel-ib

openibd

IB master control script

openib-diags

ibstat

IB utility to display HCAs

openib-diags

ibnetdiscover

IB utility to probe and show the fabric

mstflint

mstflint

Mellanox utility to update HCA FLASHRAM

libibcommon

NA

IB support package

libibmad

NA

IB support package

libibumad

NA

IB support package

OFED Release 1.2.5 or later

NA

IB support package


 

If you elected not to install these packages when installing the Linux OS or if you want to upgrade your drivers, you can install these packages at any time from the OS distribution source or by downloading the required files from OpenFabrics.org. For information on both of these procedures, see Installing the InfiniBand Drivers on Linux.

OpenFabrics Enterprise Distribution for Linux

As the popularity of InfiniBand technology increases, the number of Linux distributions and open source organizations producing drivers and tools will increase. For up-to-date information, check with open source organizations and your current vendors.

The OpenFabrics organization is the Open Software solution in the InfiniBand software space and OpenFabrics Enterprise Distribution (OFED) is the InfiniBand suite of software produced by this organization. Various vendors contribute their drivers (and other software components) to OFED.

TABLE 3-2 lists the tested Linux platforms and the corresponding OFED release.


 
TABLE 3-2 Linux Platforms and OFED Release

Linux Platform

OFED Release

RHEL AS 4-U3 or later

For RHEL AS 4-U3, Sun has tested OFED Release 1.2.5 of the OpenFabrics stack.

Note: RHEL AS 4-U4 includes an older version of OFED, so you must install OFED Release 1.2.5 or a later version.

SLES9 SP3 or later, SLES10

Sun has tested OFED Release 1.2.5 for the SLES10 platform. Note: You must have OFED Release 1.2.5 or a later version.


 

OFED contains the following components:

Installing the InfiniBand Drivers on Linux

If you did not install the InfiniBand drivers when installing the Linux OS, you can install them at any time from the OS distribution source or by downloading the necessary files from OpenFabrics.org.

To do so, choose one of the following procedures:

If you need to determine whether or not the drivers are already installed, see To Verify Driver Installation on Linux.


 

procedure icon  To Install IB Drivers From Linux Distribution Source

To install the InfiniBand drivers, you need access to the Red Hat Package Manager (RPM) files. Access to these files is dependent on your individual installation configuration (net boot, CD/DVD boot, .iso files, and so on). When you decide on the appropriate access method and package selection, you can add the packages to the KickStart configuration file (on RHEL) for automatic inclusion in future installations.


Note - On a 32-bit RHEL4 system, all packages have a .i386.rpm extension (as shown in the following procedure). On a 64-bit RHEL4 system, all packages have a .x86_64.rpm extension instead.


 

1. Enter the rpm -ivh command for each InfiniBand package that you need to install.

Packages must be installed in the following order:

The following example shows the installation of one package (libibcommon) and the resulting dialog on an RHEL AS 4-U4 32-bit system:


 
 > rpm -ivh libibcommon-1.0-1.i386.rpm
 warning: libibcommon-1.0-1.i386.rpm: V3 DSA signature: NOKEY, key ID db42a60e
 Preparing...     ##################################### [100%]
1:libibcommon  ########################################### [100%]
> rpm -ivh libibumad-1.0-1.i386.rpm
.
.
.

 

2. If you are running the CSH or TCSH shell, enter the rehash command to rebuild the shell’s view of available executables.

3. Enter the ibstat command to verify that the OS sees the IB em.


 
> ibstat
CA 'mthca0'
     CA type: MT25204
     Number of ports: 1
     Firmware version: 1.1.0
     Hardware version: a0
     Node GUID: 0x001b00000ca72640 
     System image GUID: 0x001b00000ca72643
     Port 1
         State: Active
         Physical state: LinkUp
         Rate: 20
         Base lid: 71
         LMC: 0 
         SM lid: 2
         Capability mask: 0x02510a68
         Port GUID: 0x001b00000ca72641

 

4. (Optional) You can enter the ibnetdiscover command to verify the presence of an operational IB fabric.

For an example of the output of this command, see To Verify Driver Installation on Linux.

5. (Optional) You can check the status of the ib0 network interface to determine whether the ib_ipoib driver is installed.

For details on this step, see To Install IPoIB Driver.


 

procedure icon  To Install the OFED Package


 
Note - The Sun Dual Port 4x DDR IB Host Channel Adapter PCIe ExpressModule requires OFED Release 1.2.5 or later.


 

1. On the Sun Blade Server Module, log in as root and copy the required files from the following location:

http://www.openfabrics.org/downloads.htm

In the following example, OFED-1.2.5.tar is used only as an example.


 
Note - You need Write access to the files to execute the install script.


 

2. From root, extract the files by typing:


 
> tar -zxvf OFED-1.2.5.tar

 

3. From the OFED-1.2.5 directory, initiate the installation process by typing:


 
> ./install.sh

 

4. When the InfiniBand OFED Distribution Software Installation Menu appears, select option 2 (Install OFED Software).

5. When the Select OFED Software menu appears, select option 3 (All packages).

6. When you are asked if you wish to create/install an MPI RPM with gcc,
enter n.


 
The following compiler(s) on your system can be used to build/install MPI:  gcc 
Do you wish to create/install an MPI RPM with gcc? [Y/n]: n

 

7. Next, you are asked if you wish to create/install an openmpi RPM with gcc. Again, type n.


 
The following compiler(s) on your system can be used to build/install openmpi:  gcc 
Do you wish to create/install an openmpi RPM with gcc? [Y/n]: n

 

The installation script lists the OFED packages that it will build. See the following sample output.


 
Following is the list of OFED packages that you have chosen (somemay have been added by the installation program due to package dependencies):
ib_ipath
ib_ipoib
...
mpitests
ibutils
 
WARNING: This installation program will remove any previouslyinstalled IB packages on your machine.
 
Do you want to continue? [Y/n]: Y

 

8. Type Y to continue, as shown above.

Next, you are prompted to configure InfiniBand IP support.

9. Type Y when asked if you want to include IPoIB configuration files.


 
Do you want to include IPoIB configuration files (ifcfg-ib*)? [Y/n]: Y

 

10. Press Enter to accept the default when prompted to enter a temporary directory for OFED.


 
RPM build process requires a temporary directory.
Please enter the temporary directory [/var/tmp/OFED]: 

 

11. Press Enter to accept the default when prompted for the OFED installation directory.


 
Please enter the OFED installation directory [/usr/local/ofed]:

 

At this point, the installer begins compiling InfiniBand packages. The process of building packages takes approximately 15-20 minutes.

The system displays output like the following:


 
The MPI_COMPILER_openmpi variable is not defined. Trying the default compiler: gcc 
 
The following compiler(s) will be used to build the openmpi RPM(s): gcc
 
Checking dependencies. Please wait ...
 
Building InfiniBand Software RPMs. Please wait...
 
Building openib RPMs. Please wait... 
.
.
.
33 packages were built
 
Build process finished ...

 

Installation begins. See the following message.


 
Removing previous InfiniBand Software installation
Running /bin/rpm -e libibverbs libibverbs-devel libibverbs-utils...

 

The actual installation takes about one minute.

Assuming the IB EM hardware is installed (and, therefore, an InfiniBand HCA is present), you are prompted to configure InfiniBand IP support.

12. Enter Y in response to the following prompt:


 
Do you want to configure IPoIB interfaces [Y/n]? Y

 

The default IPoIB interface configuration is based on DHCP. A special patch for DHCP is required for supporting IPoIB. The patch is available under:

OFED-1.2.5/docs/dhcp

If you do not have DHCP, you must change this configuration in the following steps.

The system next displays the current configuration.

13. When asked if you want to change the configuration as displayed, type y.


 
The current IPOIB configuration for ib0 is:
DEVICE=ib0
BOOTPROTO=dhcp
ONBOOT=yes
Do you want to change this configuration? [y/N]: Y

 

The configuration script guides you through the changes one at a time. See the following as an example.


 
Enter an IP Address:10.0.0.52
Enter the Netmask: 255.255.255.0
Enter the Network:10.0.0.0
Enter the Broadcast Address:10.0.0.255
Start Device On Boot? [Y/n]:Y
 
Selected configuration:
 
IPADDR=10.0.0.52
NETMASK=255.255.255.0
NETWORK=10.0.0.0
BROADCAST=10.0.0.255
ONBOOT=yes
 
Do you want to save the selected configuration? [Y/n]: Y

 

14. Type Y to save the configuration.

If you have entered a valid IP configuration for ib0, you are now properly configured for IPoIB operations.

15. Iterate the InfiniBand configuration over all InfiniBand interfaces.

Enter a valid IP configuration for each network interface.

Once all IPoIB interfaces have been configured, you are prompted as follows to configure OpenSM for the blade.


 
Do you want to configure OpenSM [Y/n]? n

 

16. Enter n to complete this part of the installation.

You should see a message like the following.


 
Installation finished successfully...
Press Enter to continue...

 

17. Press Enter.

The InfiniBand OFED Distribution Software Installation Menu is displayed.

18. Type Q to exit.

The Sun Blade Server Module is configured now to start up the InfiniBand software on reboot (ONBOOT=yes).

If this is not the desired behavior, you can edit the /etc/infiniband/openib.conf file, changing ONBOOT to equal no. You can also manually control basic InfiniBand behavior by entering the following command:


 
/etc/init.d/openibd option 

 

where option can be start, stop, or status.

19. After a successful installation, reboot the Server Module.

After the reboot, the Server Module should come up as a functional member of the InfiniBand fabric.


 

procedure icon  To Verify Driver Installation on Linux

1. Verify that the Linux software driver is installed and attached to the IB EM by typing the openibd status command.


 
Note - When using the openibd command, type the entire path as shown in the example.


 

The following example shows the IB driver installed, running and presenting one IB HCA channel or network device (ibn) to the OS. In the example, the Linux network device appears as ib0.


 
> /etc/init.d/openibd status
     HCA driver loaded
Configured devices:
ib0
Currently active devices:
ib0
     The following modules are also loaded: 
ib_cm
ip_ipoib
.
.
.

 

2. To view details of operational status, type the ibstat command.

The following example shows one operational IB channel into the IB fabric (or network). The LinkUp state indicates active participation in an IB fabric. It is present as lid 69 and it is being managed by lid 2.


 
> ibstat
CA 'mthca0'
     CA type: MT25204
     Number of ports: 1
     Firmware version: 1.1.0
     Hardware version: a0
     Node GUID: 0x001b00000ca72620 
     System image GUID: 0x001b00000ca72623
     Port 1
         State: Active
         Physical state: LinkUp
         Rate: 20
         Base lid: 69
         LMC: 0 
         SM lid: 2
         Capability mask: 0x02510a68
         Port GUID: 0x001b00000ca72621

 

You can also verify that the InfiniBand fabric is operational by entering the ibnetdiscover command. The output from this command will list all the nodes, as shown in the following sample output.


 
> ibnetdiscover
#
# Topology file: generated on Thu Jan 11 15:19:59 2007
#
# Max of 4 hops discovered
# Initiated from node 001b00000ca72620 port 001b00000ca72621 
 
vendid=0x8f1
devid=0x5a31
sysimgguid=0x8f10400411ef9
switchguid=0x8f10400411ef8
 
Switch  24 "S-0008f10400411ef8"    # Switch port 0 lid 9
[21]       "H-0002c90109761ea0"[2]
[12]       "S-0005ad00000161ba"[5]
[7]        "H-001b00000ca72630"[1]
[6]        "H-001b00000ca72620"[1]
vendid=0x5ad
devid=0xa87c
sysimgguid=0x5ad01010161b6
switchguid=0x5ad00000161ba 
Switch  8 "S-0005ad00000161ba"    # Switch - U3 port 0 lid 3
[4]"       H-0005ad0000011310"[1]
[3]       "S-0005ad00000161b6"[1]
[2]       "S-0005ad00000161b6"[2] 
[1]       "S-0005ad00000161b8"[3]
[5]       "S-0008f10400411ef8"[12]
.
.
.
vendid=0x2c9
devid=0x6274
sysimgguid=0x1b00000ca72633
caguid=0x1b00000ca72630 
Ca  1 "H-001b00000ca72630"  # 4x DDR IB 10-Port PCIe Network Express Module
[1]     "S-0008f10400411ef8"[7]     # lid 68 lmc 0n

Recommended Links

Top Visited

Bulletin Latest Past week Past month
Google Search



Infiniband HOWTO

SourceForge.net Managing the Mellanox Infiniband Network - xcat

IBM Redbooks Mellanox ConnectX-2 VPI Single-port and Dual-port QDR InfiniBand Host Channel Adapters 17 August 2010



Etc

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes.   If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner. 

ABUSE: IPs or network segments from which we detect a stream of probes might be blocked for no less then 90 days. Multiple types of probes increase this period.  

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least


Copyright © 1996-2016 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: November 16, 2014