Softpanorama
May the source be with you, but remember the KISS principle ;-)

Contents Bulletin Scripting in shell and Perl Network troubleshooting History Humor

Solaris Multipathing

News See Also Recommended Links Humor Quiz

Summary

To verify the system’s failover configuration, or to change the operational status of IPMP interfaces, use the if_mpadm

Introduction

Network interfaces are exposed to failure because they connect to network cables and hardware components in the form of switches or hubs. Failure of any of these interfaces results in network failure, even if the network interface card (NIC) that is in place does not fail. Sun offers two features that address customer network bandwidth demands: IPMP and Sun Trunking software.

IPMP enables multiple interfaces with different IP addresses on the same subnet to be connected to the same network segment. If any one of these interfaces fail, current network connections through that interface will be migrated to another interface automatically: physical interface on failed adapter will became logical on another (working) LAN adapter. This is a pretty neat idea. 

In general, multipathing is a method of using redundancy and automatic fail-over that provides at least two physical paths to a target resource. It allow recovery from a single network path failure by re-routing data traffic. That is important for network storage.  Varian of multipathing called trunking also allows for the parallel transmission of  data, which can result in faster throughput and increased scalability.

Solaris 10 supports Solaris network multipathing via:

IPMP in Solaris has the following features/capabilities: 

The Sun multipathing software (MPxIO) allows the merging of multiple SCSI layer paths, such as an iSCSI device exposing the same LUN via two different iSCSI target names (or the same name with different target portal group tags). In addition, one FC path and one iSCSI path can be merged by MPxIO if the target device supports these options. Additional functionality, such as iSCSI Multiple Connections / Session (MC/S), might be supported in future Solaris releases.

Multiple physical connections between a host and a target provides two major benefits: availability and link aggregation.

When multiple physical connections exist between host and storage, if one connection is lost, then I/O can be rerouted through other paths. To maximize availability and eliminate single points of failure, the following components in a storage architecture must be made redundant:

For more information about configuring a high availability network, see Enterprise Network Design Patterns: High Availability (Sun BluePrints Online—December, 2003) at http://www.sun.com/blueprints/1203/817-4683.pdf

Link Aggregation

Multipathing drivers can route data through multiple paths in parallel, resulting in increased throughput between host and storage. Current commodity Ethernet implementations support 1Gb/S full duplex, for an aggregate throughput of 2Gb/S (if transmit and receive traffic is balanced (if, for example, the network supports 1Gb incoming and 1Gb outgoing traffic). Combining multiple links can scale performance.

Implementing Multipathing

IPMP is a product that was first included with the Solaris 8. IPMP provides enhanced network availability. IPMP enables the Solaris to recover from network path failures. IPMP also provides increased throughput by spreading the outbound load across interfaces when multiple network adapters are connected to the same IP link; for example, to the same Ethernet switch.

If a failure occurs in the network link and an alternate adapter is configured, the IP address fails over. The network access changes automatically from the failed adapter to the alternate adapter, allowing uninterrupted access to the network. When there are multiple network adapters that are connected to the same IP link, increased throughput can be achieved by spreading the outbound load across multiple network  interfaces.

IPMP has the following features:

IPMP Requirements


Interface Failure Detection and Repair

The in.mpathd process can detect both the failure and the repair of an interface by:

An interface has failed if either of these two detection methods indicates a failure. An interface is considered repaired only if both methods report that the interface is operational and can send and receive packets through the interface.

To detect the failure or repair of interfaces that belong to the multipath group, the in.mpathd process sends ICMP echo requests from the logical IPMP interfaces to targets connected to the local network. The in.mpathd process determines which targets to probe dynamically. If five consecutive probes do not receive replies, the interface is considered failed.

Adjust the failure detection time by editing the  FAILURE_DETECTION_TIME variable from the default value of 10,000 milliseconds (10 seconds) in the /etc/default/mpathd file.

When responses to the ICMP echo requests are not received and a specific time period has elapsed, the physical interface is considered failed. The IP address that is associated with the failed address is moved to a new
logical interface associated with another physical interface in the same IPMP group. Communications that were taking place continue to function as though the original interface is still working properly.

ICMP echo requests are still attempted through the failed NIC to detect if a physical interface is repaired.

If all the NICs or targets appear to fail at the same time, this is a group failure, and no failover is performed. The in.mpathd process flushes all of the current targets and attempts to discover new targets. Because in.mpathd process dynamically determines what targets to probe, you cannot configure the targets. Routers connected to the link are chosen as targets for probing. If no routers exist on the link, arbitrary hosts on the link are chosen by sending a multicast packet to the “all hosts” multicast address. When you configure IPMP, be sure to have at least one additional system on the network.

You can configure multipathing by changing configuration files and rebooting, or you can work at the command line to avoid rebooting the system.

Configuring Multipathing Using Configuration Files

This example shows IPMP configuration on an existing configured qfe0 interface and on an existing but unconfigured qfe1 interface on the myhost (192.168.1.1) system. The multipath group is called mpgrp-one. The test addresses are:

The data address for the qfe0 interface remains 192.168.1.1, and the data address for the qfe1 interface is 192.168.1.45.

To configure IPMP, complete the following steps, which are described in greater detail in the next sections.

1. Verify the Solaris OE release. The /etc/release file contains information about the installed version of the Solaris. It should be 9 or higher.

2. Configure unique MAC addresses.  To determine if unique MAC addresses are allowed, use the eeprom  utility to view the contents of the flash code electrically erasable programmable read-only memory (EEPROM):

# eeprom local-mac-address?
local-mac-address?=false

The preceding output indicates that the system is still in its default mode and uses the same MAC address for each interface. This is indicated by  the setting of the local-mac-address? variable to false. You now use
the eeprom utility to change the local-mac-address? variable to true:

# eeprom local-mac-address?=true

Verify that the local-mac-address? variable is set to true:

# eeprom local-mac-address?

Note – Depending on the combination of your system’s firmware and hardware architecture, you must either plumb the interface or reboot the system to enable unique MAC address assignment after changing the eeprom variable.

3. Define data and test IP addresses. You can add the data and test IP addresses to the /etc/inet/hosts file for the sake of clarity. After editing the /etc/inet/hosts file, use the  tail utility to view the new information:

# tail -5 /etc/inet/hosts
# Modifications made for IPMP

192.168.1.1 myhost # Data address for qfe0
192.168.1.45 myhost-dat-qfe1 # Data address for qfe1
192.168.1.50 myhost-test0 # qfe0:1 Test address for qfe0
192.168.1.51 myhost-test1 # qfe1:1 Test address for qfe1

 

4. Configure the interfaces.  Multipath information is placed in the /etc/hostname.qfe0 and /etc/hostname.qfe1 files. Modify the /etc/hostname.qfe0 file to contain contents similar to the following:

# cat /etc/hostname.qfe0
myhost netmask + broadcast + group mpgrp-one up \
addif myhost-test0 deprecated netmask + broadcast + -failover up

addif myhost-test0 Creates the next unused logical interface, and  assigns it the IP address associated with the myhost-test0 name in /etc/hosts.  deprecated Marks the address as a deprecated address. Addresses that are marked as deprecated are not  used as source addresses for outbound packets unless either there are no other addresses  available on this interface or the application is bound to this address explicitly.

The output from the ifconfig -a command shows DEPRECATED as one of the flags associated with this interface.

-failover Marks the address as a non-failover address. Addresses that are marked in this way do not fail
over when the interface fails. The output from the ifconfig -a command shows NOFAILOVERas
one of the flags associated with this interface.

Create the /etc/hostname.qfe1 file to contain contents similar to the following:

# cat /etc/hostname.qfe1
myhost-dat-qfe1 netmask + broadcast + group mpgrp-one up \
addif myhost-test1 deprecated netmask + broadcast + -failover up

5. Reboot the system, for example using init 6. 

6. View the interface configuration.  Use ifconfig -a command. You should get something like

# ifconfig -a
lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000


hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 192.168.30.31 netmask ffffff00 broadcast 192.168.30.255
ether 8:0:20:b9:72:23

qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 192.168.1.1 netmask ffffff00 broadcast 192.168.1.255
groupname mpgrp-one
ether 8:0:20:ac:9b:20

qfe0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
inet 192.168.1.50 netmask ffffff00 broadcast 192.168.1.255


qfe1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
inet 192.168.1.45 netmask ffffff00 broadcast 192.168.1.255
groupname mpgrp-one
ether 8:0:20:ac:9b:21


qfe1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 4
inet 192.168.1.51 netmask ffffff00 broadcast 192.168.1.255

 

This information above includes the following:

You must know the state of the system if you need to restore it. Before making any changes to the system, view the system’s interface configuration by performing the command:

# ifconfig -a

lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000

hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 192.168.30.31 netmask ffffff00 broadcast 192.168.30.255
ether 8:0:20:b9:72:23


qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 192.168.1.1 netmask ffffff00 broadcast 192.168.1.255
ether 8:0:20:ac:9b:20

The system remains available to users if either of the multipath network interfaces fail or become unusable for any reason. As a precaution, reboot the system when it is convenient to verify that multipathing is properly configured during system boot.

Command Line Utilities-based Configuration

Configure the qfe0 Interface as Part of a Multipath Group.

To configure the qfe0 interface as part of a multipath group, specify the  name of the group, mpgrp-one, of which the qfe0 interface will be a part:

# ifconfig qfe0 group mpgrp-one

View the changes to the interface using ifconfig -a

Configure a Test Address for the qfe0 Interface

# ifconfig qfe0 addif 192.168.1.50 deprecated netmask + broadcast + -failover up
 

Configure the qfe1 Interface as Part of the qfe0 Interface Multipath Group

Now, you configure the qfe1interface and make it part of the same IPMP group as the qfe0 interface. Perform the command: 

# ifconfig qfe1 plumb myhost-dat-qfe1 netmask + broadcast + group mpgrp-one up

Now, you configure a test address for the qfe1interface.

# ifconfig qfe1 addif 192.168.1.51 deprecated netmask + broadcast + -failover up
 

Viewing Multipath Operation

To verify the system’s failover configuration, or to change the operational status of IPMP interfaces, use the if_mpadm utility. You can use this utility to take an interface offline (detach), by forcing a failover, and verifying that an alternate interface takes over as expected. If configuration errors occur, they appear at this stage. Also, use the if_mpadm utility to reattach a detached interface.

For example, to detach the qfe0 interface, perform the command:

# if_mpadm -d qfe0
Nov 17 12:40:46 myhost in.mpathd[541]: Successfully failed over from NIC
qfe0 to NIC qfe1
 

The message indicates that the failover was successful.

Note – This message appears in the console window and is not seen if you are using an xterm or dtterm window.

To view the status of the interfaces, use the ifconfig command:

# ifconfig -a
lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000


hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 192.168.30.31 netmask ffffff00 broadcast 192.168.30.255
ether 8:0:20:b9:72:23


qfe0: flags=89000842<BROADCAST,RUNNING,MULTICAST,IPv4,NOFAILOVER,OFFLINE> mtu 0 index 3
inet 0.0.0.0 netmask 0
groupname mpgrp-one
ether 8:0:20:ac:9b:20

qfe0:1: flags=89040842<BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,OFFLINE> mtu 1500 index 3
inet 192.168.1.50 netmask ffffff00 broadcast 192.168.1.255


qfe1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
inet 192.168.1.45 netmask ffffff00 broadcast 192.168.1.255
groupname mpgrp-one
ether 8:0:20:ac:9b:21
qfe1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 4
inet 192.168.1.51 netmask ffffff00 broadcast 192.168.1.255
qfe1:2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
inet 192.168.1.1 netmask ffffff00 broadcast 192.168.1.255

 

The detached interface is assigned an IP address of 0.0.0.0, and a new logical interface, qfe1:2, is automatically created over the functional qfe1 physical interface. The new logical interface has the IP address that was assigned to the physical qfe0 interface while it was working. To reattach an offline interface, perform the command:

# if_mpadm -r qfe0
Nov 17 12:52:03 myhost in.mpathd[541]: Successfully failed back to NIC qfe0

 

Note – This message appears in the console window and is not seen if you are using an xterm or dtterm window.

The message indicates that the failback was successful. To view the status of the interfaces, use the ifconfig utility:

# ifconfig -a


lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000


hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 192.168.30.31 netmask ffffff00 broadcast 192.168.30.255
ether 8:0:20:b9:72:23


qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 192.168.1.1 netmask ffffff00 broadcast 192.168.1.255
groupname mpgrp-one
ether 8:0:20:ac:9b:20


qfe0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
inet 192.168.1.50 netmask ffffff00 broadcast 192.168.1.255


qfe1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
inet 192.168.1.45 netmask ffffff00 broadcast 192.168.1.255
groupname mpgrp-one
ether 8:0:20:ac:9b:21


qfe1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 4
inet 192.168.1.51 netmask ffffff00 broadcast 192.168.1.255
#

The qfe0 interface is reassigned its original IP address, and the qfe1:2 logical interface is automatically removed.

Troubleshooting a Multipath Network Configuration

Incorrectly configured network interfaces might not properly fail over  when connectivity to an interface fails for any reason. It is important to thoroughly test your network interface after you configure IPMP.

Carefully read messages in the /var/adm/messages file or in the console window to take the proper troubleshooting steps when you configure and test the IPMP. For example:

# Nov 17 23:19:51 myhost in.mpathd[475]: Failures cannot be detected
on qfe0 as no IFF_NOFAILOVER address is available

The message indicates that the in.mpathd process with a process number of 475 senses that IPMP is not properly configured. To investigate further, use the ifconfig command:

# ifconfig -a
lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 192.168.30.31 netmask ffffff00 broadcast 192.168.30.255
ether 8:0:20:b9:72:23
qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 192.168.1.1 netmask ffffff00 broadcast 192.168.1.255
groupname mpgrp-one
ether 8:0:20:ac:9b:20

The output indicates that the configuration process is not complete. Recall that IPMP requires a test address on a logical interface for each physical interface. After defining a test interface with the ifconfig command, the following message appears:

# Nov 17 23:17:41 myhost in.mpathd[475]: Failure detection restored
on qfe0 as an IFF_NOFAILOVER address is available

To configure a test interface, use the ifconfig utility:

# ifconfig qfe0 addif 192.168.1.50 deprecated netmask + \
broadcast + -failover up

Created new logical interface qfe0:1
Setting netmask of qfe0:1 to 255.255.255.0
# Nov 17 23:17:41 myhost in.mpathd[475]: Failure detection restored
on qfe0 as an IFF_NOFAILOVER address is available

The in.mpathd process reports that it can now perform failure detection. Be aware that more than one interface is required to provide effective failover. To view the interface configuration, use the ifconfig command:

# ifconfig -a

lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000

hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 192.168.30.31 netmask ffffff00 broadcast 192.168.30.255
ether 8:0:20:b9:72:23


qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 192.168.1.1 netmask ffffff00 broadcast 192.168.1.255
groupname mpgrp-one
ether 8:0:20:ac:9b:20


qfe0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
inet 192.168.1.50 netmask ffffff00 broadcast 192.168.1.255


Both the physical and logical interface are properly configured.

Recommended Links

Softpanorama Top Visited

Softpanorama Recommended

docs.sun.com: IP Network Multipathing Administration Guide
 Sun documentation. See also PDF version  816-5249
System Administration

Guide IP Services

Solaris IP Multipathing made easy

This is very practical discussion that adds a lot to Sun provided information.
[PDF] Internet Protocol Network Multipathing (Updated)
Old Sun Blueprint on the topic (dated 2002)

Internet Protocol Network Multipathing (Update) Functional Overview Updated summary of the blueprint.

Network failover using Solaris multipathing

[PDF] Using iSCSI Multipathing in the Solaris(tm) 10 Operating System

Sys Admin > Reliable Network with SolarisTM
Internet Protocol Network Multipathing (Update) > Functional
 

Quiz

Q1: Which command will define an IPMP test address?
 

  1. ifconfig hme1:1 plumb 192.11.16.2 netmask + broadcast + up
  2. ifconfig hme0 addif deprecated group one up
  3. ifconfig hme0 addif 192.11.16.2 deprecated netmask + broadcast + -failover up
  4. eeprom local-mac-address?=true

A: c




Etc

Society

Groupthink : Understanding Micromanagers and Control Freaks : Toxic Managers : BureaucraciesHarvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Two Party System as Polyarchy : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

Skeptical Finance : John Kenneth Galbraith : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes : Oscar Wilde : Talleyrand : Somerset Maugham : War and Peace : Marcus Aurelius : Eric Hoffer : Kurt Vonnegut : Otto Von Bismarck : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce : Oscar Wilde : Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 26, No.1 (January, 2013) Object-Oriented Cult : Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks: The efficient markets hypothesis : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Vol 23, No.10 (October, 2011) An observation about corporate security departments : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

 

The Last but not Least


Copyright © 1996-2014 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Site uses AdSense so you need to be aware of Google privacy policy. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine. This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting hosting of this site with different providers to distribute and speed up access. Currently there are two functional mirrors: softpanorama.info (the fastest) and softpanorama.net.

Disclaimer:

The statements, views and opinions presented on this web page are those of the author and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: July 07, 2013