TIP: When the user reporting the problem first of all verify that you and the user
are on the same page and that his usage of words is correct. This will eliminate
problems in which the user reports a problem but uses technical terms incorrectly;
for example, "my system crashes." Your version could be "a specific application
terminates unexpectedly." Attempt to locate the lowest level of the problem,
for example, applications that appear to be failing may be impacted by underpinning
network problems.
The ping,
traceroute,
ngrep and other network tools are indispensable
tools for troubleshooting networking problems.
Network troubleshooting means recognizing and diagnosing networking problems
with the goal of keeping your network running optimally. As a network administrator,
your primary concern is maintaining connectivity of all devices (a process
often called fault management). You may also continually evaluate and improve
your network's performance. Because serious networking problems can sometimes
begin as performance problems, paying attention to performance can help you address
issues before they become serious.
Like in any investigation you need to avoid jumping to conclusion and calmly
collect all relevant facts. You can use famous "How
to solve it "approach. Among more network specific issues:
Cut though communication barrier to the level of clear understanding
of real symptoms.
If this is a remote problem try to define the problem in your own words and
check with the user reporting the problem if your usage of words is correct.
This will eliminate problems in which the user reports a problem but uses technical
terms incorrectly of use fuzzy terms like "my system crashes." Your version
of the same problem could be "a specific application terminates unexpectedly."
Work bottom-up using TCP/IP networking layers and attempt to trace the
problem to the lowest level of TCP/IP stack. For example, if networked
applications appear to be failing to connect using symbolic name of the host,
it is prudent to check if they can work OK with raw IP addresses. If yes, this
suggests that they may be impacted by underpinning DNS problems.
Do not take anything for granted. For example, the link LED (light
emitting diode) on a hub may light when a cable is connected, leading you to
believe that the link has been established and that the network cabling is functional.
But the transmit wire or connector could be broken causing loss of communications.
The link LED lights because the link signal is being received from the receive
line.
In general, there is no one correct way to determine the root cause of a networking
problem. Like any troubleshooting of complex systems this is more art then science
and the success depends both on your IQ and the level of experience with the environment.
However, there are a heuristics that you can follow:
Concentrate on the problem in hand. Networks are a lot like
cars: You can start out investigating one problem and find 10 other things that
may need attention. Make a note of any non-related problems, but focus on investigating
the primary problem.
Focus on one area at a time. If several users are reporting problems
from different areas of the network at the same time, there is a good chance
that they are reporting elements of the same problem. It can be overwhelming
to have 1,000 or more users down at once, but if the same problem is simply
reoccurring in multiple parts of the network, you only have to figure it out
once.
If possible use the lab. Whenever possible, try to duplicate the
problem in a lab and troubleshoot it there. Sometimes changes performed
during troubleshooting has a greater negative impact on the end-user population
than the original problem.
Use intelligent testing strategies:
If any test requires reconfiguring a device, ensure that you can roll
back the change after the test, or you may find that you have backed yourself
into a corner and cannot proceed.
Use as few tests as possible to isolate and define the problem working
top-down TCP stack or bottom-up. Ensure that the results of the tests
are unambiguous before concentrating on the next level.
Validate the test results by repeating each one at least twice. Note
that running a command to verify a configuration parameter is not considered
a test in this sense and therefore doesn't need to be performed twice.
Document changes as your proceed not as an afterthought:
Document the tests performed and the results in case a bug is found.
Document any changes made to the network during the troubleshooting
procedure so that the network can be properly restored to its original condition.
Document any workarounds that were left in place so that other support
personnel will be able to understand how and why the network changed.
Troubleshooting Commandments
Create a backup of the faulty system before fixing anything. Backup
can be done only for configuration files or for the complete system. Complete
backup is important as troubleshooting is a high stress activity and it is easy
accidentally to destroy some files.
Such backup
also can be indispensable if the fault disappears on its own: faults that fix
themselves often come back on their own too. On modern servers you often can
allocate /backup partition and put cpio images of other partitions into it.
Before changing file always create a baseline of /etc partition.
Also backup each individual configuration file with the timestamp before
changing it. That prevents
you from the most typical mistake in troubleshooting:
losing the initial configuration.
Simplify your environment, if possible. Where possible shutdown
internal firewall and try to remove
routers and firewalls from the networking path affected. Often problems are
introduced by the internal firewall (on both end) or other network devices
in the path.
Problem with routers are more common is home and small business
environment were cheaper routers are used. In the past Linksys routers
were frequent source of grief.
In enterprise environment left hand often does not know what right is doing
and similar effects can be due the fact that someone may have upgraded a router's
operating system or a firewall's rule set.
Patches are just special kind of upgrade and can introduce problems too.
Have a testing plan. Make sure that you can replicate the reported
fault at will. This is important because you should always attempt to re-create
the reported fault after effecting any changes. You need to be sure that you
are not changing or adding to the problem.
Document all steps and results. This is important because you could
forget exactly what you did to fix or change the problem. This is especially
true when someone interrupts you as you are about to test a configuration change.
You can always revert the system to the faulty state if you backed it up as
suggested earlier. If you work outside office in the server room you can use
tablet or netbook or even your smartphone. Remember that many changes and
ideas during troubleshooting might be lost and cause problems later because
they are forgotten, unless they are carefully documented.
Where possible, make permanent changes to the configuration settings.
Temporary changes may be faster to implement, but cause tremendous confusion when the
system reboots after a power failure months or even years later and the fault
occurs again. At this time, nobody will remember what was done by whom.
Using ping as a Troubleshooting Tool
The ping utility sends ICMP echo request packets to the target host or hosts.
Once ICMP echo responses are received, the message target is considered to be alive, where target
is the hostname of the device receiving the ICMP echo requests, is displayed.
# ping problem.host.com
problem.host.com is alive
The -s option is useful when attempting to connect to a remote host that is down
or not available. No output will be produced until an ICMP echo response is received
from the target host. The -R option can be useful if the traceroute utility is not
available.
Statistics are displayed when the ping -s command is terminated.
# ping -s problem.host.com
Another useful troubleshooting technique using ping is to send ICMP echo requests
to the entire network by using the broadcast address as the target host. Using
the -s option with the broadcast address provides good information about which systems
are available on the network:
# ping -s 172.20.4.255
Using ifconfig as a Troubleshooting Tool
The ifconfig utility is useful when
troubleshooting networking problems. For example when you have several poert on the server and
it is unlear which is connected tot he swithc it can show where traffic is.
You can use it to display an interface's current
status including the settings for the following:
MTU
Address family
IP address
Netmask
Broadcast address
Ethernet address (MAC address)
Be aware that there are two ifconfig commands. The two versions differ in how
they use name services. The /sbin/ifconfig is called by the /etc/rc2.d/S30sysid.net
startup script. This version is not affected by the configuration of the
/etc/nsswitch.conf file.
The /usr/sbin/ifconfig is called by the /etc/rc2.d/S69inet and the
/etc/rc2.d/S72inetsvc startup scripts. This version of the ifconfig command is affected
by the name service settings in the /etc/nsswitch.conf file.
Use the plumb switch when troubleshooting interfaces that have been
manually added and configured. Often an interface will report that it is up and
running yet a snoop session from another host shows that no traffic is flowing out
of the suspect interface. Using the plumb switch resolves the misconfiguration problem.
Using arp as a Troubleshooting Tool
The arp utility can be useful when attempting to locate network problems related
to duplicate IP address usage. Determine the Ethernet address of the target host.
You can do this by using the banner utility at the ok prompt, or the ifconfig utility
at a shell prompt on a Sun system. Armed with the Ethernet address (also known as
the MAC address) use the ping utility to determine if the target host can be reached.
Use the arp utility immediately after using the ping utility and verify that
the arp table reflects the expected (correct) Ethernet address.
The following example demonstrates this technique.
Working from the system three, use the ping and arp utilities to determine if
the system one is really responding to system three.
First, determine the Ethernet address of the host called one.
problem.host.com# ifconfig -a
lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1 inet
127.0.0.1 netmask ff000000
hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST, IPv4> mtu 1500 index 6 inet
128.50.2.1 netmask ffffff00 broadcast 128.50.2.255 ether 8:0:20:76:6:b
problem.host.com#
The ifconfig utility shows that the Ethernet address of the hme0 interface is 8:0:20:76:6:b.
The first half of the address, 08:00:20 shows that the system is a Sun computer.
The last half of the address, 76:06:0b is the unique part of the system's Ethernet
address.
Search the Internet to determine the manufacturer of devices with unknown Ethernet
addresses.
2. Use the ping utility to send ICMP echo requests from system three to system
problem.host.com.
ping problem.host.com problem.host.com is alive
3. View the arp table to determine if the device that sent the ICMP echo response
is the correct system, 76:06:0b.
# arp -a
Net to Media Table: IPv4
Device IP Address Mask Flags Phys Addr ------ -------------------- ---------------
----- ---------------
08: 00 : 20 : 76: 06: 0b 08: 00 : 20:
8e : ee : 18 08: 00 : 20: 7a: 0b:b8 08:00:20:78:54:90 00: 60:97:7f:4f:dd 01:
00 : 5e: 00 : 00: 00
Output from the arp utility will appear to hang if name resolution fails because
the arp utility attempts to resolve names. Use the netstat
-pn utility to obtain similar output.
The table displayed in step 3 proved that the correct device responded. If the
wrong system responded, it could have been quickly tracked down by using the Ethernet
address. Once located, it can be configured with the correct IP address.
Many hubs and switches will report the Ethernet address of the attached device,
making it easier to track down incorrectly configured devices.
The first half of the Ethernet address can also be used to refine the search.
The previous example showed a device, presumably a personal computer, as it reported
an Ethernet address of 00:60:97:7f:4f:dd. A quick search on the Internet reveals
that the 00:60:97 vendor code is assigned to the 3COM corporation.
Using snoop as a Troubleshooting Tool
The snoop utility can be particularly
useful when troubleshooting virtually any networking problems. The traces that are
produced by the snoop utility can be most helpful when attempting remote troubleshooting
because an end-user (with access to the root password) can capture a snoop trace
and email it or send it using ftp to a network troubleshooter for remote diagnosis.
You can use the snoop utility to display packets on the fly or to write to a
file. Writing to a file using the -o switch is preferable because each packet can
be interrogated later.
problem.host.com# snoop -o tracefile
Using device /dev/le (promiscuous mode)
You can view the snoop file by using the -i switch and the filename in any of
the standard modes, namely:
Terse mode - No option switch is required.
Summary verbose mode - Use the -V switch.
Verbose mode - Use the -v switch.
Verbose is most useful when you are troubleshooting routing, network booting,
Trivial File Transport Protocol (TFTP), and any network-related problems that require
diagnosis at the packet level. Each layer of the packet is clearly defined by the
specific headers.
View the snoop output file in terse mode and locate a packet or range of packets
of interest. Use the -p switch to view these packets. For example, if packet two
is of interest, type:
problem.host.com# snoop -p2,2 -v -i
tracefile
Using ndd as a Troubleshooting Tool
Use extreme caution when using the Solaris ndd utility
because the system could be rendered inoperable if you set parameters incorrectly.
Use an escaped question mark (\?) to determine which parameters a driver supports.
For example, to determine which parameters the 100-Mbit Ethernet (hme) device supports,
type:
... ... lance_mode
ipg0
#
(read and write) (read and write)
Routing/IP Forwarding
Many systems configured as multi-homed hosts or firewalls may have IP forwarding
disabled. A fast way to determine the state of IP forwarding is to use the ndd utility.
problem.host.com# ndd /dev/ip ip_forwarding
0
This example shows that the system is not forwarding IP packets between its interfaces.
The value for ip_forwarding is 1 when the system is routing or forwarding IP packets.
Interface Speed
The hme (100-Mbit Ethernet) Ethernet card can operate at two speeds, 10 or 100
Mbits per second. You can use the ndd utility to quickly display the speed at which
the interface is running.
ndd /dev/hme link_speed 1
A one (1) indicates that the interface is running at 100 MBits per second. A
zero (0) indicates that the interface is running at 10 MBits per second.
Interface Mode
The hme interface can run in either full-duplex or half-duplex mode. Again, the
ndd utility provides a fast way to determine the mode of the interface.
# ndd /dev/hme link_mode
One (1) indicates that the interface is running in full-duplex mode. A zero (0)
indicates that the interface is running in half-duplex mode.
Using netstat as a Troubleshooting Tool
You can use the netstat utility to display
the status of the system's network interfaces. Of particular interest when troubleshooting
networks are the routing tables of all the systems in question. You can use the
-r switch to display a system's routing tables.
# netstat -r
Although interesting, the displayed routing table is not of much use unless you
are familiar with the name resolution services, be they the /etc/hosts, NIS, or
NIS+ services. The problem is that it is difficult to concentrate on routing issues
when any doubt can be cast on the name services. For example, someone could have
modified the name service database, and the system msbravo may no longer be the
IP address that you expected. Using the -n switch eliminates this uncertainty.
# netstat -rn
# ifconfig -a
This routing table is much easier to translate and troubleshoot, especially when
combined with the information from the ifconfig -a utility.
lo0: flags=849<UP,LOOPBACK,RUNNING,MULTICAST>
mtu 8232
inet 127.0.0.1 netmask ff000000
hme0: flags=863<UP, BROADCAST, NOTRAILERS,
RUNNING, MULTICAST> mtu 1500
The verbose mode switch, -v displays additional information, including the MTU size
configured for the interface:
# netstat -rnv
Using traceroute as a Troubleshooting Tool
The traceroute utility is useful when
you perform network troubleshooting. You can quickly determine if the expected route
is being taken when communicating or attempting to communicate with a target network
device. As with most network troubleshooting, it is useful to have a benchmark against
which current traceroute output can be compared. The traceroute output can report
network problems to other network troubleshooters. For example, you could say, "Our
normal route to a host is from our router called router1-ISP to your routers called
rtr-a1 to rtr-c4. Today, however, users are complaining that performance is very
slow. Screen refreshes are taking more than 40 seconds when they normally take less
than a second. The output from traceroute shows that the route to the host is from
our router router1-ISP to your routers called rtr-a1, rtr-d4 rtr-x5, and then to
rtr-c4. What is going on?"
The traceroute utility uses the IP TTL and tries to force ICMP TIME_EXCEEDED
responses from all gateways and routers along the path to the target host. The traceroute
utility also tries to force a PORT_UNREACHABLE message from the target host. The
traceroute utility can also attempt to force an ICMP ECHO_REPLY message from the
target host by using the -I (ICMP ECHO) option when issuing the traceroute command.
The traceroute utility will, by default, resolve IP addresses as shown in the following
example:
# traceroute 172.20.4.110
traceroute to 172.20.4.110 (172.20.4.110), 30 hops max, 40 byte packets
1 129.147.11.253 (129.147.11.253) 1.037 ms 0.785 ms 0.702 ms
2 129.147.3.249 (129.147.3.249) 1.452 ms 1.569 ms 0.766 ms
3 * dungeon (129.147.11.59) 1.320 ms *
You can display IP addresses instead of hostnames by using the -n switch as shown
in the following example. In this example, the hostname dungeon for IP address 129.147.11.59
on line 3 is no longer resolved.
# traceroute -n 172 .20.4. 110
traceroute to 172.20.4.110 (172.20.4.110), 30 hops max, 40 byte packets
1
129.147.11.253
0.954 ms
0.657 ms
0.695 ms
2
129.147.3.249
0.844 ms
0.745 ms
0.771 ms
3
129.147.11.59
0.534 ms *
0.640 ms
Common Network Problems
Following is a list of some common problems that occur:
Faulty RJ-45 - The network connection fails intermittently.
Faulty wiring on patch cable - No network communications.
mdi to mdi (no mdi-x) - Media data interfaces, such as hubs, are
not connected to another mdi device. Many hubs have a port that can be switched
to become an mdi-x mdi crossover port.
Badly configured encryption - Once encryption is configured, things
are not as they appear. Standard tools such as ifconfig, and netstat will not
locate the problem. Use the snoop utility to view the contents of packets to
determine if all is normal.
Hub or switch configured to block the MAC - Modern hubs and switches
are configured to block specific MAC addresses or any addresses if the connection
is tampered with. Access to the console of the hub or switch is necessary to
unblock a port.
Bad routing tables or rogue router - Routing tables can be corrupted.
Sometime a rogue router can appear of the network due to installation of multihomed
host.
Rogue DHCP server is present in DHCP environment. Often happens when
somebody installs Windows server on the network without understanding what they
are doing.
Protocol not being routed - for example if jumpstart or bootp is
being used across routers.
Interface not plumbed - Additional interfaces, when configured, are
not plumbed. The interface will appear to be functioning, but it will not pass
traffic.
Connection to the wrong interface on multihoned host.
Bad information in the /etc/hosts or NIS database - The IP address
of systems is incorrect or missing.
The user statement, "My application does not work" is just a tip of an iceberg
and the user often does not understand what exactly is not working by jumping to
conclusions that can mislead you in troubleshooting. Never believe the user story.
You need ask the user very specific question to uncover the real story. Among questions
to consider:
Is the server up and functioning normally?
Can other users access the server?
Is the client system up and functioning normally?
Has anything changed on the server?
Has anything changed on the client?
Layers-based troubleshooting
When troubleshooting networks, some people prefer to think in layers, similar to
the TCP/IP Model while others prefer to think in terms of functionality.
Using the TCP/IP Model layered approach, you could start at either the Physical
or Application layer. Start at either end of the model and test, draw conclusions,
move to the next layer and so on.
The Application Layer
A user complains that an application is not functioning. Assuming the application
has everything that it needs, such as disk space, name servers, and the like, determine
if the Application layer is functional by using another system.
Application layer programs often have diagnostic capabilities and may report
that a remote system is not available. Use the snoop command to determine if the
application program is receiving and sending the expected data.
The Transport Layer and the Internet Layer
These two layers can be bundled together for the purposes of troubleshooting.
Determine if the systems can communicate with each other. Look for ICMP messages
that can provide clues as to where the problem lies. Could this be a router or switching
problem? Are the protocols (TFTP, BOOTP) being routed? Are you attempting to use
protocols that cannot be routed? Are the hostnames being translated to the correct
IP addresses? Are the correct netmask and broadcast addresses being used? Tests
between the client and server can include
using ping, traceroute, arp, and snoop.
The Network Interface Layer
Use snoop to determine if the network interface is actually functioning. Use
the arp command to determine if the arp cache has the expected Ethernet or MAC address.
Fourth generation hubs and some switches can be configured to block certain MAC
addresses.
When troubleshooting connectivity problems here are some useful questions:
Have any changes been made to the network devices?
Can the client contact the server using ping?
Can the client contact any system using ping?
Can the server contact the client using ping?
Can the client system use ping to contact any other hosts on the local network
segment?
Can the client use ping to contact the far interface of the router?
Can the client use ping to contact any hosts on the server's subnet?
Is the server in the client's arp cache?
Can snoop be used to determine what happens to the service or arp request
Is the client's interface correctly configured? (Has it been plumbed?)
Has any encryption software installation been attempted?
The Physical Layer
Check that the link status LED is lit. Test it with a known working cable. The
link LED will be lit even if the transmit line is damaged. Verify that a mdi-x connection
or crossover cable is being used if connecting hub to hub.
Selected Troubleshooting Scenarios
Multi-Homed System Acts as Rogue Router
For example system A can use telnet to contact system B, but system B cannot
use telnet to contact system A. Further questioning of the user revealed that this
problem appeared shortly after a power failure.
For troubleshooting use the traceroute utility to show the route that network
traffic takes from system B to system A. If the traceroute output reveals route
that goes via additional system (let's call it system C) you have a rogue router
problem.
Often that happens due to the fact that system C had been modified by an end-user.
For example an additional interface was added, bit the user did not add
/etc/notrouter file to the system. In this case,
after rebooting the system, it came up as a router and started advertising routes,
which confuses the core routers and disrupts network traffic patterns.
Faulty Cable
For example users on network A could not reach hosts on network B even though
routers R1 and E2 appeared to be functioning normally.
First you need to verify that the routers R1and R2 were configured correctly and
that the interfaces are up.
They you need to verify that systems A and B were up and configured correctly.
They you need to use the traceroute utility to discover the actual route from
system A to system B.
For example the traceroute output shows that the attempted route from system
A on network net-1 goes through router R1 as expected. But the trafficnever reaches
router R2 though.
Investigate the router R2 log files. For example is they show that the interface
to network net-2 is flapping (going up and down at a very high rate) and corrupt
routing tables you can suspect that the cable is a problem,
To solve this problem, replace the network net-2 cable to router R2. If it fixes
the problem then it was faulty and causes intermittent connections.
Duplicate IP Address
Reported Problem: Systems on network net-1 could not use ping past
router R1 to a recently configured network, net-2.
You must be "root" or the sys to perform some of the other troubleshooting step
in the previous examples. Suggested steps:
Verify that the T1 link between the routers R1 and R2 is functioning properly.
Verify that router R1 can use ping to contact router R2.
Verify that system A can use ping to reach the close interface of router
R2. System A cannot use ping on the far interface of router R2, though.
Confirm that systems on network net-1 can use ping to reach router R1.
Check that systems on network net-2 can use ping to reach router R2.
Determine that the routers are configured correctly.
Verify that the systems on network net-1 and network net-2 are configured
correctly.
Make sure the systems on network net-1 can communicate with each other.
Verify that systems on network net-2 can communicate with each other.
Log onto router R1 and use traceroute to display how the data is routed
from router R1 to router R2.
traceroute reported that the traffic from router
R1 to router R2 was going out the network net-1 side interface of the router
instead of the network net-2 side as expected. This indicates that the IP address
for router R2 may also exist on network net-1.
Check the Ethernet address of router R2; compare the actual address with
the contents of router R1's arp cache. The arp cache revealed that the device
was of a different manufacturer than expected.
To solve the problem track down the device on network net-1, system C, that has
an illegal IP address (one that is the same as the network-net-1-side interface
of router R2). This resulted in a routing loop as the routers had multiple best-case
paths to take to the same location (which were actually in two different sites).
Correct the duplicate IP address problem on system C and make sure communications
work as expected.
Duplicate MAC Address (Mostly Sun environment problem)
Reported Problem: After adding an additional Ethernet interface to your host,
the system performance is very poor.
Troubleshooting (as user root):
Use arp -a to view the address table on the host. If the MAC address appears
on more than one host that is on the same physical network, this may be the
problem.
Use ifconfig -a to check the IP address and MAC address.
If host with the same MAC address are on the same subnet you found the problem
Notice from the previous ifconfig output that all the interfaces have the same
MAC address. Host C is on different subnet, so this is not a problem. This
would cause problems because packets that leave either qfe0 or qfe1 would not be
guaranteed to receive a response since both interfaces are broadcasting themselves
as the source for those packets.
This How-To covers some basic networking setup and troubleshooting on the
Solaris 9 OS.
Two Tips for Network Performance Checking
a. Use FTP to copy a large file between hosts. Make sure you copy the file
in both directions, as network performance problems can be directional. A possible
cause of performance issues is autonegotiation being enabled at either the host
or the router/switch. (See the Checking Network Settings
section of this article for more details.)
b. Use ping with small (1Kbyte) and large (10K) packet sizes: Sometime routers
in the network can have issues depending upon the size of the packet, as some
use different queues within the router depending upon packet size.
Network Connectivity Troubleshooting
Here is a checklist to help you locate and resolve network connectivity problems.
1. Use ifconfig -a to check that interfaces are
plumbed; that is, that they exist in the output. Also, check the network address
and netmask of the interface.
To plumb an interface, run the command ifconfig <interface><instance>
plumb, for example:
# ifconfig ce1 plumb
Use ifconfig to see if the interface now exists.
# ifconfig -a
lo0:
flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4>
mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
ce0:
flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4>
mtu 1500 index 2
inet 444.555.666.7 netmask ffffff00 broadcast
444.555.666.255
ether 5:3:de:de:de:de
ce1:
flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4>
mtu 1500 index 6
inet 0.0.0.0 netmask 0
ether 3:4:aa:bb:cc:dd
Give the interface its ipaddress and
netmask.
ifconfig ce1 555.66.77.88 netmask 255.255.255.0 up
2. Ping the interface address; it should work!
3. Ping your router/switch. If you see => fail,
then check your network settings. (See the Checking Network
Settings section of this article.)
4. Ping a host on another network. If that doesn't work, check the routing
table. (See the Checking Routing Settings section of
this document.)
Checking Network Settings
You can check or set the status and configuration
of a network interface with the ndd command. To use
the ndd command, you first need to use the
ifconfig command to find out what the device file
is for the network interface in which you are interested.
# ifconfig -a
lo0:
flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4>
mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
ce0:
flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4>
mtu 1500 index 2
inet 444.555.666.7 netmask ffffff00 broadcast
444.555.666.255
ether 5:3:de:de:de:de
This example shows only one interface plumbed, ce0,
so the interface type is ce, and it is instance
0.
Now sometimes the ndd command uses a generic device
file, e.g. /dev/ce, and you need to set the instance
number first to know which instance is being interrogated. Otherwise, an instance-specific
device file, e.g. /dev/bge0, is used. You can use
ls -ld /dev/<interface>* to see if instance-specific
device files exist, for example, ls -ld /dev/ce*.
# ls -l /dev/ce*
lrwxrwxrwx 1 root root 28 Mar 3 2006 /dev/ce ->
.../devices/pseudo/clone@0:ce
or
# ls -ld /dev/bge*
lrwxrwxrwx 1 root root 29 Oct 14 2005 /dev/bge ->
.../devices/pseudo/clone@0:bge
lrwxrwxrwx 1 root root 39 Oct 14 2005 /dev/bge0 ->
.../devices/pci@1f,700000/network@2:bge0
lrwxrwxrwx 1 root root 41 Oct 14 2005 /dev/bge1 ->
.../devices/pci@1f,700000/network@2,1:bge1
lrwxrwxrwx 1 root root 39 Oct 14 2005 /dev/bge2 ->
.../devices/pci@1d,700000/network@2:bge2
lrwxrwxrwx 1 root root 41 Oct 14 2005 /dev/bge3 ->
.../devices/pci@1d,700000/network@2,1:bge3
The following scripts print out all variables available via
ndd. One is for a generic device file, and the other
is for an instance-specific device file. The script is run with the instance
required as a command-line option, that is, <script> 0.
You would need to change the script to have the correct interface device type.
For example, you may need to replace ce with
eri if your interface device file is
/dev/eri.
I have found it best to set the network devices to disable autonegotiation
on both the host and the router to which the host is connected. This is done
by setting the following parameters: autoneg_cap,
1000fdx_cap, 1000hdx_cap,
100T4_cap, 100fdx_cap,
100hdx_cap, 10fdx_cap,
10hdx_cap, and adv_autoneg_cap.
I use this script in the /etc/rc2 directory to
set the network parameters.
The bge interface shows you the currently running
status with the following parameters: link_status,
link_speed, and link_duplex.
For information on interface device drivers, look in the man pages, in Section
7: Devices and Network Interfaces.
Checking Routing Settings
To see your current routing configuration, use netstat
-r, and add the -n option depending on whether
you want to see DNS names or IP addresses. For example:
One of the first signs of trouble on the network is a loss of communications
by one or more
hosts.
If a host
refuses to come up at all the first time it is added to the network, the problem
might lie in one of the configuration files, or in the
network
interface. If a single host suddenly develops a problem, the
network
interface might be the cause. If the
hosts
on a network can communicate with each other but not with other
networks,
the problem could lie with the
router,
or it could lie in another network.
You can use the
ifconfig
program to obtain information on
network
interfaces and
netstat
to display routing tables and protocol statistics. Third-party network diagnostic
programs provide a number of troubleshooting utilities. Refer to third-party
documentation for information.
Less obvious are the causes of problems that degrade performance on the network.
For example, you can use tools like
ping to
quantify problems like the loss of packets by a
host.
Running Software Checks
If the network has trouble, some actions that you can take to diagnose and
fix software-related problems include:
Using the
netstat
command to display network information.
Checking the
hosts
database (and ipnodes if you are using
IPv6) to make sure that the entries are correct and up to
date.
If you are running RARP, checking the Ethernet addresses in the
ethers
database to make sure that the entries are correct and up to
date.
where host
is the host name of the machine in question. The optional timeout argument indicates
the time in seconds for
ping to
keep trying to reach the machine-20 seconds by default. The
ping(1M)
manpage describes
additional syntaxes and options. When you run
ping, the
ICMP protocol sends a datagram
to the host
you specify, asking for a response. (ICMP
is the protocol responsible for error handling on a TCP/IP network. See ICMP
Protocol for details.)
How to Determine if a Host Is Losing Packets
If you suspect that a machine might be losing packets even though it is running,
you can use the s option of
ping to
try to detect the problem. On the command
line, type
the following command.
ping
continually sends packets to
hostname
until you send an interrupt character or a timeout occurs. The responses on
your screen will resemble:
PING elvis: 56 data bytes
64 bytes from 129.144.50.21: icmp_seq=0. time=80. ms
64 bytes from 129.144.50.21: icmp_seq=1. time=0. ms
64 bytes from 129.144.50.21: icmp_seq=2. time=0. ms
64 bytes from 129.144.50.21: icmp_seq=3. time=0. ms
.
.
.
----elvis PING Statistics----
4 packets transmitted, 4 packets received, 0% packet loss
round-trip (ms) min/avg/max = 0/20/80
The packet-loss statistic indicates whether the
host has
dropped packets.
If ping
fails, check the status of the network reported by
ifconfig
and netstat,
as described in ifconfig Command and
netstat
Command
ifconfig Command
The
ifconfig command displays information about the configuration of an interface
that you specify. (Refer to the
ifconfig(1M)
manpage for
details.) The syntax of ifconfig is:
The flags section just given shows that the interface is configured "up,"
capable of broadcasting, and not using "trailer"
link
level encapsulation. The mtu field tells you that this interface has a maximum
transfer size of 1500 octets. Information on the second
line
includes the IP address of the
host
you are using, the
netmask
being currently used, and the IP broadcast address of the interface. The
third line gives the machine address (Ethernet, in this case) of the
host.
How to Get Information About All Interfaces on a Network
A useful
ifconfig
option is -a, which provides information on all interfaces on your network.
le0: flags=49 mtu 8232
inet 127.144.44.140 netmask ff000000
le0:flags=863 mtu 1500
inet 129.144.44.140 netmask ffffff00 broadcast 129.144.44.255
ether 8:0:20:8:el:fd
Output that indicates an interface is not running might mean a problem
with that interface. In this case, see the
ifconfig(1M)
manpage.
netstat Command
The netstat
command generates displays that show network status and protocol statistics.
You can display the status of
TCP and
UDP endpoints
in table format, routing table information, and interface information.
netstat
displays various types of network data depending on the command
line option
selected. These displays are the most useful for system administration. The
syntax for this form is:
The result resembles the display shown in the example below. (Parts of the
output have been truncated.) The information can indicate areas where a protocol
is having problems. For example, statistical information from
ICMP can indicate where this
protocol has found errors.
UDP
udpInDatagrams = 39228 udpOutDatagrams = 2455
udpInErrors = 0
TCP
tcpRtoAlgorithm = 4 tcpMaxConn = -1
tcpRtoMax = 60000 tcpPassiveOpens = 2
tcpActiveOpens = 4 tcpEstabResets = 1
tcpAttemptFails = 3 tcpOutSegs = 315
.
.
IP
ipForwarding = 2 ipDefaultTTL = 255
ipInReceives = 4518 ipInHdrErrors = 0
.
.
ICMP
icmpInMsgs = 0 icmpInErrors = 0
icmpInCksumErrs = 0 icmpInUnknowns = 0
.
.
IGMP:
0 messages received
0 messages received with too few bytes
0 messages received with bad checksum
0 membership queries received
0 membership queries received with invalid field(s)
0 membership reports received
0 membership reports received with invalid field(s)
0 membership reports received for groups to which we belong
0 membership reports sent
How to Display Network Interface Status
The i option of
netstat
shows the state of the
network
interfaces that are configured with the machine where you ran the command.
On the command
line, type
the following command:
Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis Queue
le0 1500 b5-spd-2f-cm tatra 14093893 8492 10174659 1119 2314178 0
lo0 8232 loopback localhost 92997622 5442 12451748 0 775125 0
Using this display, you can
find out
how many packets a machine thinks it has transmitted and received on each network.
For example, the input packet count (Ipkts) displayed for a server can increase
each time a client tries to
boot, while
the output packet count (Opkts) remains steady. This suggests that the server
is seeing the
boot request
packets from the client, but does not realize it is supposed to respond to them.
This might be caused by an incorrect address in the
hosts,
ipnodes, or
ethers
database.
On the other hand, if the input packet count is steady over time, it means
that the machine does not see the packets at all. This suggests a different
type of failure, possibly a hardware problem.
How to Display Routing Table Status
The -r option of
netstat
displays the IP routing table. On the command
line, type
the following command.
Here is a sample display produced by
netstat
-r run on machine tenere:
Routing tables
Destination Gateway Flags Refcnt Use Interface
temp8milptp elvis UGH 0 0
irmcpeb1-ptp0 elvis UGH 0 0
route93-ptp0 speed UGH 0 0
mtvb9-ptp0 speed UGH 0 0
.
mtnside speed UG 1 567
ray-net speed UG 0 0
mtnside-eng speed UG 0 36
mtnside-eng speed UG 0 558
mtnside-eng tenere U 33 190248 le0
The first column shows the destination network, the second the
router
through which packets are forwarded. The U flag indicates that the
route
is up; the G flag indicates that the route is to a
gateway.
The H flag indicates that the destination is a fully qualified
host address,
rather than a network.
The Refcnt column shows the number of active uses per
route,
and the Use column shows the number of packets sent per
route.
Finally, the Interface column shows the
network
interface that the
route
uses.
How to Log Network Problems
Become superuser.
Create a
log
file of routing daemon actions by typing the following command at a command
line
prompt.
# /usr/sbin/in.routed /var/logfilename
Caution: On a busy network, this can generate almost continuous output.
Displaying Packet Contents
You can use
snoop
to capture network packets and display their contents. Packets can be displayed
as soon as they are received, or saved to a file. When snoop writes to an intermediate
file, packet loss under busy trace conditions is unlikely. snoop itself is then
used to interpret the file. For information about using the snoop command,
refer
to the snoop(1M)
manpage.
The snoop
command must be run by root (#) to capture packets to and from the default interface
in
promiscuous mode. In summary form, only the data pertaining to the highest-level
protocol is displayed. For example, an NFS packet only displays NFS information.
The underlying RPC,
UDP, IP,
and Ethernet frame information is suppressed but can be displayed if either
of the verbose options is chosen.
The snoop
capture file format is described in RFC 1761. To access, use your favorite web
browser with the URL: http://ds.internic.net/rfc/rfc1761.txt.
snoop
server client
rpc rstatd
collects all RPC traffic between a client and server, and filters it for rstatd.
How to Check All Packets from Your System
Type the following command at the command
line
prompt to
find
the interfaces attached to the system.
# snoop
Using device /dev/le (promiscuous mode)
maupiti -> atlantic-82 NFS C GETATTR FH=0343
atlantic-82 -> maupiti NFS R GETATTR OK
maupiti -> atlantic-82 NFS C GETATTR FH=D360
atlantic-82 -> maupiti NFS R GETATTR OK
maupiti -> atlantic-82 NFS C GETATTR FH=1A18
atlantic-82 -> maupiti NFS R GETATTR OK
maupiti -> (broadcast) ARP C Who is 120.146.82.36, npmpk17a-82 ?
Interpret the results.
In the example, client maupiti transmits to server atlantic-82 using
NFS file handle 0343. atlantic-82 acknowledges with OK. The conversation
continues until maupiti broadcasts an ARP request asking who is 120.146.82.36?
This example demonstrates the format of
snoop.
The next step is to filter
snoop
to capture packets to a file.
Interpret the capture file using details described in RFC 1761. To access,
use your favorite web browser with the URL: http://ds.internic.net/rfc/rfc1761.txt
This has captured 30 packets in a file /tmp/cap. The file can be anywhere
with enough disk space. The number of packets captured is displayed on the
command
line, enabling you to press Ctl-C to abort at any time.
snoop
creates a noticeable networking load on the
host
machine, which can distort the results. To see reality at work, run
snoop
from a third system, (see the next section).
On the command
line,
type the following command to inspect the file.
# snoop -i /tmp/cap
1 0.00000 frmpk17b-082 -> 224.0.0.2 IP D=224.0.0.2 S=129.146.82.1 LEN=32, ID=0
2 0.56104 scout -> (broadcast) ARP C Who is 129.146.82.63, grail ?
3 0.16742 atlantic-82 -> (broadcast) ARP C Who is 129.146.82.76, honeybea ?
4 0.77247 scout -> (broadcast) ARP C Who is 129.146.82.63, grail ?
5 0.80532 frmpk17b-082 -> (broadcast) ARP C Who is 129.146.82.92, holmes ?
6 0.13462 scout -> (broadcast) ARP C Who is 129.146.82.63, grail ?
7 0.94003 scout -> (broadcast) ARP C Who is 129.146.82.63, grail ?
8 0.93992 scout -> (broadcast) ARP C Who is 129.146.82.63, grail ?
9 0.60887 towel -> (broadcast) ARP C Who is 129.146.82.35, udmpk17b-82 ?
10 0.86691 nimpk17a-82 -> 129.146.82.255 RIP R (1 destinations)
Refer to specific protocol documentation for detailed analysis and recommended
parameters for ARP, IP, RIP and so forth. Searching the Web is a good place
to look
at RFCs.
How to Check Packets Between Server and Client
Establish a
snoop
system off a hub connected to either the client or server.
The third system (the
snoop
system) sees all the intervening traffic, so the snoop trace reflects reality
on the wire.
Become superuser.
On the command
line,
type snoop
with options and save to a file.
Inspect and interpret results.
Look at RFC 1761 for details of the
snoop
capture file. To access, use your favorite web browser with the URL: http://ds.internic.net/rfc/rfc1761.txt
Use
snoop frequently and consistently to get a feel for normal system behavior.
For assistance in analyzing packets,
look
for recent white papers and RFCs, and seek the advice of an expert in a
particular area, such as NFS or YP. For details on using snoop and its options,
refer
to the
snoop(1M)
manpage.
Displaying Routing Information
Use the
traceroute
utility to trace the
route
an IP packet follows to some internet
host. The
traceroute utility utilizes the IP protocol time(to live) ttl field and attempts
to elicit an ICMP TIME_EXCEEDED
response from each
gateway
along the path, and the response PORT_UNREACHABLE (or
ECHO_REPLY) from the destination host. The traceroute utility starts sending
probes with a ttl of one and increases by one until it gets to the intended
host or
has passed through a maximum number of intermediate
hosts.
The
traceroute
utility is especially useful for determining routing misconfiguration and routing
path failures. If a particular
host is
unreachable, you can use the traceroute utility to see what path the packet
follows to the intended host and where possible failures might occur.
The
traceroute
utility also displays the round trip time for each
gateway
along the path to the target
host. This
information can be useful for analyzing where traffic is slow between the two
hosts.
The following sample of the
traceroute
command shows the 7-hop path a packet follows from the
host istanbul
to the host sanfrancisco along with the times for a packet to traverse each
hop.
istanbul% traceroute sanfrancisco
traceroute: Warning: Multiple interfaces found; using 172.31.86.247 @ le0
traceroute to sanfrancisco (172.29.64.39), 30 hops max, 40 byte packets
1 frbldg7c-86 (172.31.86.1) 1.516 ms 1.283 ms 1.362 ms
2 bldg1a-001 (172.31.1.211) 2.277 ms 1.773 ms 2.186 ms
3 bldg4-bldg1 (172.30.4.42) 1.978 ms 1.986 ms 13.996 ms
4 bldg6-bldg4 (172.30.4.49) 2.655 ms 3.042 ms 2.344 ms
5 ferbldg11a-001 (172.29.1.236) 2.636 ms 3.432 ms 3.830 ms
6 frbldg12b-153 (172.29.153.72) 3.452 ms 3.146 ms 2.962 ms
7 sanfrancisco (172.29.64.39) 3.430 ms 3.312 ms 3.451 ms
The Last but not LeastTechnology is dominated by
two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt.
Ph.D
FAIR USE NOTICEThis site contains
copyrighted material the use of which has not always been specifically
authorized by the copyright owner. We are making such material available
to advance understanding of computer science, IT technology, economic, scientific, and social
issues. We believe this constitutes a 'fair use' of any such
copyrighted material as provided by section 107 of the US Copyright Law according to which
such material can be distributed without profit exclusively for research and educational purposes.
This is a Spartan WHYFF (We Help You For Free)
site written by people for whom English is not a native language. Grammar and spelling errors should
be expected. The site contain some broken links as it develops like a living tree...
You can use PayPal to to buy a cup of coffee for authors
of this site
Disclaimer:
The statements, views and opinions presented on this web page are those of the author (or
referenced source) and are
not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society.We do not warrant the correctness
of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be
tracked by Google please disable Javascript for this site. This site is perfectly usable without
Javascript.
