Softpanorama

May the source be with you, but remember the KISS principle ;-)
Contents Bulletin Scripting in shell and Perl Network troubleshooting History Humor

Troubleshooting TCP/IP Communication Issues

News

Linux Network Troubleshooting

Recommended Books

Recommended Links

Network Utilities

Redhat Network Configuration Suse 10 network configuration
Ethernet Protocol ifconfig ethtool How to change IP address in RHEL netstat Xinetd Remote Syslog
route command Linux Routing DNS nslookup hostname Changing hostname Procedure for installing Qlogic QLE2460 cards
NFS vsftp pure ftpd rsync NTP RHEL NTP configuration Troubleshooting NTP on Red Hat Linux
Telnet Protocol VNC on Linux   SSH Autonegotiation Samba Sendmail on RHEL Postfix
TCP Performance Tuning Bonding Multiple Network Interfaces Linux multipath InfiniBand Installing Mellanox InfiniBand Driver on RHEL 6.5 Setting up a basic infiniband network Troubleshooting InfiniBand connection issues using OFED tools
USB to Ethernet Adapter in Linux USB to wireless adapter Network Manager overwrites resolv.conf Tips Admin Horror Stories Humor Etc
Neil Cashell
Proactive Resolution Team
Novell Worldwide Support
ncashell@novell.com
 

15 May 2000

This document addresses communication issues that generate about a third of the support calls coming into the TCP/IP group at Novell Technical Support. We recommend that anyone who is implementing TCP/IP in a NetWare 5.x environment read and understand the information presented here.

This article is divided into two parts: understanding the concepts behind IP routing, and troubleshooting common TCP/IP problems. A follow-up article will explain some of the TCP/IP tools that are available for use in troubleshooting problems in a TCP/IP environment.

Concepts Behind TCP/IP Routing

The majority of connectivity issues involve problems with routing table entries. Every packet being processed by a TCP/IP host has a source and destination IP address. Upon receiving each packet, the IP protocol examines the destination address of the packet, compares it with entries in its local routing table, and then decides what action to take:

  1. If the destination IP address is itself (that is, to a local application such as GroupWise, BorderManager Proxy Server, etc.), the packet is passed up to a protocol layer above IP.
  2. If the packet is destined for another known network, the packet is forwarded through one of the locally-attached network adapters. (This assumes that the TCP/IP host has multiple interfaces and has routing enabled.)
  3. If neither of the above apply, the packet is discarded.

The TCP/IP routing table can maintain four different types of routes, listed below in the order that they are searched for a match:

IP compares the destination IP address of the packet that it is processing with the entries in the table. If IP finds that a host entry exists and matches the destination IP address, it will forward the packet to the next hop associated with that host entry. Host entries are usually found in routing tables when ICMP (Internet Control Message Protocol) has added the entry because of the pathMTU algorithm, or from an "ICMP redirect" call. To check this, load the TCPCON utility at the server console prompt and look at the IP Routing Table option to verify if the protocol associated with that route is ICMP.

IP has three classes of addresses: Class A, Class B and Class C. Each class contains a default subnet mask (for instance, Class A has 255.0.0.0. as a default subnet) until a class of addresses is broken into extra networks (i.e., subnetted). However, once the network is subnetted, the IP address will not have the default subnet mask.

So if IP doesn't find a host entry, but does find a subnet entry that matches the packet's destination IP address, IP will forward the packet to the next hop associated with that subnet entry. Subnet entries exist when RIP2 (Routing Internet Protocol v2), OSPF (Open Shortest Path First), or static entries have been added to the routing table through a non-default subnet mask.

If IP doesn't find a subnet entry in the TCP/IP routing table but does find a network entry that matches the destination IP address, IP will forward the packet to the next hop associated with that network entry. (Customers running in default NetWare TCP/IP mode will have network entries.)

Finally, if IP doesn't find a network entry, but does find that a default route entry exists, IP will forward the packet to the next hop associated with that default entry. The default route is most commonly inserted as a static route through NetWare's server console INETCFG utility. However, the route may also be learned via RIP or OSPF. Failure to at least have a default route can often lead to communication problems on the network.

If an IP packet match has not been found in the TCP/IP routing table at this stage, the packet is simply dropped and an ICMP "destination unreachable" message is triggered to notify the sender that the host or network is unreachable.

When a TCP/IP communication problem occurs, the most common reason is that a route entry doesn't exist for the network or host with which you are trying to communicate. When this is the case, you can either add a route entry or try to figure out why the route is missing.

Troubleshooting Common TCP/IP Problems

When troubleshooting any networking problem, it is helpful to take a logical approach. Some questions to ask are:

Troubleshooting a problem "from the bottom up" is often a good way to quickly isolate what's wrong and come up with a solution. The "bottom up" approach from an IP routing perspective is to start by verifying that the problem is not related to the physical layer (cabling, hubs, switches, and so on) or ARP (Address Resolution Protocol). Next, you ensure that the IP routing table is functioning correctly. Finally, you check to see whether the problem is at a generic TCP/UDP or application level.

To better understand the TCP/IP troubleshooting scenarios covered in this article, we'll use a small example network to illustrate some of the most common IP problems. This example network is shown in Figure 1.

Figure 1: Example network for TCP/IP troubleshooting scenarios.

 

 

In this network, Workstation 1 accesses the Internet/WAN through a NetWare server which contains two network adapters, each with its own IP address: 137.65.43.1 and 137.40.3.1. Workstation 2 accesses the Internet/WAN through the Internet Router with the IP address of 137.40.3.4. The NetWare server also communicates to the Internet/WAN through the Internet Router, as well as the Unix box (whose IP address is 137.40.3.3), which also communicates to the Internet/WAN through the Internet Router (137.40.3.4). The Internet Router's IP internet address is 137.30.1.254.

It's also important that you understand the terms "local host" and "remote host" in an IP network environment:

From the point of view of Workstation 1 in Figure 1, the NetWare server is considered a local host because its network adapter is attached to the same IP subnet as Workstation 1. Workstation 2, whose IP subnet address is different than that of Workstation 1, can be considered a remote host.

The following scenarios, which represent six of the most common IP problems, use the example network in Figure 1 as a reference. The most common solutions are given for each of these problems. While this is not a comprehensive list of solutions, they cover most of the routing issues that customers face.

Scenario 1: Cannot PING or Communicate with Local Router

Symptom: The user cannot PING from Workstation 1 (137.65.43.2) to the local segment side of the NetWare server (137.65.43.1).

Solutions: If two nodes on the same subnet cannot PING each other successfully, you can use the "ARP _A" command at a Windows workstation to check the ARP table entries. The -A parameter displays the ARP entries by interrogating the current protocol data. If more than one network adapter uses the Address Resolution Protocol, you'll see entries for each ARP table.

You can also use the TCPCON utility on the NetWare server to view the IP Address Translations Table. Select the Protocol Information | IP | IP Address Translation options, and see if the computers have the correct MAC addresses listed for each other.

Note: You can use the IPConfig utility (for Windows NT), the WINIPCFG utility (for Windows 95/98), or type CONFIG <Enter> at the NetWare server console to determine a host's MAC address (displayed as Node Address).

  1. If an ARP entry exists for the default router's IP address, perform the following troubleshooting steps.
  2. If no ARP entry exists for the default router's IP address, this usually indicates that there is a hardware problem with the devices on the network. Perform the following troubleshooting steps.

Scenario 2: Cannot PING or Communicate with Remote Interface of Local Router

Symptom: The user can PING from Workstation 1 (137.65.43.2) to the local segment side of the NetWare server (137.65.43.1), but not from Workstation 1 to the other side of the NetWare server (137.40.3.1).

Solutions:
  1. In this scenario, Workstation 1 needs to know which IP router to send the IP packet to when the destination network is on a different subnet (to a remote host, according to our earlier definition). This procedure is not required if Workstation 1 wants to communicate with hosts only on its local subnet (local host). Each TCP/IP stack configuration (whether client or server) has a parameter for a default router or gateway. (See TID #10018660 for information on configuring and troubleshooting client issues on Windows 95/98 and NT.)

    In this scenario, Workstation 1 would need to configure as its default router the IP address of the server's network adapter that is local to the workstation. The IP address would be 137.65.43.1. This implies that any packets that Workstation 1 will transmit to any remote hosts will be sent through this IP address.

  2. The NetWare server must be configured as an IP router so that it can forward packets from one network interface board (137.65.43.1) to the other (137.40.3.1). For this to happen, TCP/IP must have been loaded with the parameter "forward=yes" as part of the configuration.

    The best way to verify that TCP/IP has been loaded with forwarding enabled is through the TCPCON utility. Load TCPCON at the server console. You will see the "IP Forwarded: numbers" entry in the lower left-hand corner of the top window. If this entry has numbers after it (even if it is 0), then this server is configured as an IP router. If this entry has DISABLED after the statistic, it is not set to gateway mode. To enable this, load the INETCFG utility at the server console, select the Protocols entry, the TCP/IP entry, and then ensure that the "IP Packet Forwarding" parameter is set to ENABLED. (See TID #10013002 for more details.)

  3. Check to see whether the "Local Errors" field in TCPCON | Statistics | IP entry increases as your PING requests fail. This field increments anytime IP drops an incoming packet for any reason. If this field is increasing, perform the following diagnostic steps:

Scenario 3: Cannot Ping or Communicate with Internet Router

Symptom: From Workstation 1 (137.65.43.2) the user can ping both IP addresses that are bound to the network adapters in the NetWare server (137.65.43.1 and 137.40.3.1), but cannot ping the Internet Router (137.40.3.4).

Solutions:
  1. By default, the NetWare server uses RIP as its routing protocol. However, most IP Routers use either OSPF (Open Shortest Path First) or IGRP (Interior Gateway Routing Protocol) as the routing protocol of choice. Since the routing protocols are different on both routers, they will not update each other's routing table. The IP Router will not have a route back to the 137.65.0.0 segment, and will therefore not know how to respond to Workstation 1's PING.

    To fix this problem, insert a static route entry at the IP Router. On a NetWare server, this can be done using INETCFG by selecting Protocols| TCPIP| Static Routes. This entry tells the IP Router that in order to get to the 137.65.0.0 subnet, packets must go through the 137.40.3.1 gateway, which is the IP address of the NetWare server for the segment local to the IP Router. This implies that any time the Internet Router has a packet destined for 137.65.0.0, it will send it to the 137.40.3.1 gateway.

  2. Another possible solution is to synchronize the routing protocols at the NetWare server or IP Router so that they both understand either RIP or OSPF. You do this by enabling the same routing protocol on all routers in the network. This will guarantee that the routes being advertised by both sides will dynamically enter the necessary routing tables. Note that an ASBR (Autonomous System Boundary Router) can also be set up on either router to act as a conversion gateway between OSPF and non-OSPF (static, ICMP, or RIP) routes.
  3. Check to see whether an ARP entry exists for the Internet Router (137.40.3.4 ) in the NetWare server. To do this, go to the TCPCON | Protocol Information | IP | IP Address Translations screen.
  4. One other possible problem is that another device is responding to the ARPs using the Internet Router's IP address. In this case, there is either an IP address conflict or a bad switch.

Scenario 4: Cannot PING or Communicate with Remote Workstation

Symptom:

From Workstation 1 (137.65.43.2), the user can PING both IP addresses that are bound to the network adaptersin the NetWare server (137.65.43.1 and 137.40.3.1), and the Internet Router (137.40.3.4), but cannot PING Workstation 2 (137.40.3.2).

Solutions:

As described in Scenario 2, the workstation must have its default router or gateway set in order to reply or send packets to segments other than its local segment gateway. (See TID #10018660 for information on configuring and troubleshooting client issues on Windows 95/98 and NT.)

  1. From a Windows workstation DOS prompt, type the "NETSTAT -R" command. This command displays protocol statistics and current TCP/IP network connections, while the -R parameter displays the routing table. You can use this information to verify whether a default route exists from Workstation 1 to Workstation 2, and whether the route points to the next correct hop router for that subnet.
  2. Configure the default gateway on Workstation 2. In this scenario, the default route should point to the IP address of the Internet Router (137.40.3.4), or to the server network adapter that is local to Workstation 2's segment (137.40.3.1) Then reboot the workstation (unless you used the "ROUTE ADD" command as mentioned in TID #10018660 to insert the static route at the workstation).

Scenario 5: Cannot PING or Communicate with Remote UNIX Host

Symptom:

From Workstation 1 (137.65.43.2), the user can ping both IP addresses that are bound to the network adapters in the NetWare server (137.65.43.1 and 137.40.3.1) and the Internet Router (137.40.3.4), but cannot PING the UNIX box (137.40.3.3).

Solutions:
  1. At the UNIX box, use the "NETSTAT -R" command to see if a default route (0.0.0.0) exists on that box. If no static route exists, you must enter one in order for the UNIX box to have a route to the 137.65.0.0 subnet. The syntax for adding a static route on the UNIX box in this scenario should be similar to the following:
    route add net 137.65.0.0 137.40.3.1 1

    (For more information on the route command for UNIX, refer to the documentation that comes with your UNIX software.)

  2. Synchronize the routing protocols at the NetWare server or at the Unix box so they will both understand either RIP or OSPF protocols (as explained in Scenario 3, Step 2). This will guarantee that the routes being advertised by each side will dynamically enter both routing tables. Note that an ASBR (Autonomous System Boundary Router) can also be set up on either the NetWare router or the UNIX box to act as a conversion gateway between OSPF and non-OSPF (static, ICMP, or RIP) routes.
  3. Check to see whether an ARP entry exists for the UNIX box (137.40.3.3 ) in the NetWare server. To do this, go to the TCPCON | Protocol Information | IP | IP Address Translations screen.
  4. One other possible problem is that another device is responding to the ARPs using the Unix box's IP address. In this case, there is either an IP address conflict or a bad switch.

Scenario 6: Cannot PING or Communicate with Remote Hosts Beyond the Internet Router

Symptom:

From Workstation 1 (137.65.43.2), the user can ping both IP addresses that are bound to the network adapters in the NetWare server (137.65.43.1 and 137.40.3.1) and the Internet Router (137.40.3.4). The user can also ping Workstation 2 (137.40.3.2) and the UNIX box (137.40.3.3), but cannot PING past the Internet Router.

Solutions:
  1. In this scenario, the NetWare server knows about both the 137.65.0.0, and the 137.40.0.0 subnet segments, but it does not know where to route the packet if the destination is not on either of these segments. To fix this, you must add a default route to the NetWare server. Load the INETCFG utility at the server console and go to the Protocols | TCP/IP | Static Routing entry (be sure it is Enabled), then go to the Static Routing Table entry. Press <Insert> to add Default Route with an IP address of Network/Host 0.0.0.0 and with the Next Hop Router on Route (Gateway) of 137.40.3.4 Metric 1 Passive. This information is then written to the SYS:ETC\GATEWAYS file. (See TID #2911404, "Set LAN Default Route NW 4.x, 3.x, WEB, Proxy," for more details.) Then use the Reinitialize System command from INETCFG's initial "Internetworking Configuration" window to add the static route to the routing tables located in the server's memory.
  2. Run commands, such as the "show ip route summary" for Cisco IOS at the Internet Router's console prompt to view the Internet Router's routing table and see if Workstation 1's network (137.65.0.0) has an entry. Because of dynamic routing protocols such as RIP and OSPF, this should normally be the case. Problems here may indicate that the Internet Router's routing table is not being updated correctly. If no route exists, insert a static route for the 137.65.0.0 network and investigate why the 137.65.0.0 network is not being advertised by the dynamic routing protocols.

    To troubleshoot this problem, you first need to understand the network layout. Having the layout in mind will enable you to identify other routers in the network that should be advertising the route. You can use LAN traces to verify whether or not these other routers are advertising the missing network, and if so, with the proper parameters, such as hop count. In some cases, invalid hop counts may be advertised and the routes are being dropped accordingly.

  3. Verify that the packet is not being blocked through some filtering mechanism, such as the IPFLT.NLM. If this NLM is loaded, type "Unload IPFLT.NLM" at the server prompt, then check to see if the behavior is the same.

In the next column, we'll look into an extension of this troubleshooting scenario dealing with subnets and a couple of the more common problems that users face with subnetting.



Etc

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes.   If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner. 

ABUSE: IPs or network segments from which we detect a stream of probes might be blocked for no less then 90 days. Multiple types of probes increase this period.  

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Haterís Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least


Copyright © 1996-2016 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: August, 02, 2017