Softpanorama

May the source be with you, but remember the KISS principle ;-)
Contents Bulletin Scripting in shell and Perl Network troubleshooting History Humor

Classic Network Utilities

News See Also Recommended Links Recommended Articles Reference Network Troubleshooting Tools TCPreplay
ifconfig ethtool ndd solaris route Linux route command netstat Wireshark
ping traceroute mtr Snoop Tcpdump ngrep netcat
nmap ntop rsync snort Packet Generation Tools Humor Etc

There are at lease a dozen of classic network tools. Among them are:

Several network utilities are can pipable (netcat can create pipelines across network):

In Jan 2007 I added the page about TCP/IP troubleshooting tools to complement this page

A very good list of free network tools can be found at Top 100 Network Security Tools

Dr. Nikolai Bezroukov


Top Visited
Switchboard
Latest
Past week
Past month

NEWS CONTENTS

Old News ;-)

[Jun 10, 2010] Deep-protocol analysis of UNIX networks

[Jun 08, 2010 | developerWorks
Parsing the raw data to understand the content

Another way to process the content from tcpdump is to save the raw network packet data to a file and then process the file to find and decode the information that you want.

There are a number of modules in different languages that provide functionality for reading and decoding the data captured by tcpdump and snoop. For example, within Perl, there are two modules: Net::SnoopLog (for snoop) and Net::TcpDumpLog (for tcpdump). These will read the raw data content. The basic interfaces for both of these modules is the same.

To start, first you need to create a binary record of the packets going past on the network by writing out the data to a file using either snoop or tcpdump. For this example, we'll use tcpdump and the Net::TcpDumpLog module: $ tcpdump -w packets.raw.

Once you have amassed the network data, you can start to process the network data contents to find the information you want. The Net::TcpDumpLog parses the raw network data saved by tcpdump. Because the data is in it's raw binary format, parsing the information requires processing this binary data. For convenience, another suite of modules, NetPacket::*, provides decoding of the raw data.

For example, Listing 8 shows a simple script that prints out the IP address information for all of the packets.

Listing 8. Simple script that prints out the IP address info for all packets
use Net::TcpDumpLog;
    
use NetPacket::Ethernet;
    
use NetPacket::IP;

    
my $log = Net::TcpDumpLog->new();
 
$log->read("packets.raw");
 
 
foreach my $index ($log->indexes)
       
{
    
    my $packet = $log->data($index);
           

    my $ethernet = NetPacket::Ethernet->decode($packet);

  
    if ($ethernet->{type} == 0x0800)
       
    {
    
        my $ip = NetPacket::IP->decode($ethernet->{data});
          

    
        printf("  %s to %s protocol %s \n",
               $ip->{src_ip},$ip->{dest_ip},$ip->{proto});
   }

} 
The first part is to extract each packet. The Net::TcpDumpLog module serializes each packet, so that we can read each packet by using the packet ID. The data() method then returns the raw data for the entire packet.

As with the output from snoop, we have to extract each of the blocks of data from the raw network packet information. So in this example, we first need to extract the ethernet packet, including the data payload, from the raw network packet. The NetPacket::Ethernet module does this for us.

Since we are looking for IP packets, we can check for IP packets by looking at the Ethernet packet type. IP packets have an ID of 0x0800.

The NetPacket::IP module can then be used to extract the IP information from the data payload of the Ethernet packet. The module provides the source IP, destination IP and protocol information, among others, which we can then print.

Using this basic framework you can perform more complex lookups and decoding that do not rely on the automated solutions provided by tcpdump or snoop. For example, if you suspect that there is HTTP traffic going past on a non-standard port (i.e., not port 80), you could look for the string HTTP on ports other than 80 from the suspected host IP using the script in Listing 9.


Listing 9. Looking for strong HHTP on ports other than 80
use Net::TcpDumpLog;
    
use NetPacket::Ethernet;
    
use NetPacket::IP;
    
use NetPacket::TCP;
    

    
my $log = Net::TcpDumpLog->new();
       
$log->read("packets.raw");
       

    
foreach my $index ($log->indexes)
       
{
    
    my $packet = $log->data($index);
       

    
    my $ethernet = NetPacket::Ethernet->decode($packet);
       

    
    if ($ethernet->{type} == 0x0800)
       
    {
    
        my $ip = NetPacket::IP->decode($ethernet->{data});
          

    
        if ($ip->{src_ip} eq '192.168.0.2')
       
        {
    
            if ($ip->{proto} == 6)
       
            {
    
                my $tcp = NetPacket::TCP->decode($ip->{data});
       
                if (($tcp->{src_port} != 80) &&
               
                    ($tcp->{data} =~ m/HTTP/))
       
                {
    
                    print("Found HTTP traffic on non-port 80\n");
    
                    printf("%s (port: %d) to %s (port: %d)\n%s\n",
    
                           $ip->{src_ip},
       
                           $tcp->{src_port},
       
                           $ip->{dest_ip},
       
                           $tcp->{dest_port},
       
                           $tcp->{data});
 
                }
    
            }
    
        }
    
   }
    
}

Running the above script on a sample packet set returned the following shown in Listing 10.


Listing 10. Running the script on a sample packet set
$ perl http-non80.pl
Found HTTP traffic on non-port 80
192.168.0.2 (port: 39280) to 168.143.162.100 (port: 80)
GET /statuses/user_timeline.json HTTP/1.1
Found HTTP traffic on non-port 80
192.168.0.2 (port: 39282) to 168.143.162.100 (port: 80)
GET /statuses/friends_timeline.json HTTP/1

In this particular case we're seeing traffic from the host to an external website (Twitter).

Obviously, in this example, we are dumping out the raw data, but you could use the same basic structure to decode and the data in any format using any public or proprietary protocol structure. If you are using or developing a protocol using this method, and know the protocol format, you could extract and monitor the data being transferred.

Using a protocol analyzer

Although, as already mentioned, tools like tcpdump, iptrace and snoop provide basic network analysis and decoding, there are GUI-based tools that make the process even easier. Wireshark is one such tool that supports a vast array of network protocol decoding and analysis.

One of the main benefits of Wireshark is that you can capture packets over a period of time (just as with tcpdump) and then interactively analyze and filter the content based on the different protocols, ports and other data. Wireshark also supports a huge array of protocol decoders, enabling you to examine in minute detail the contents of the packets and conversations.

You can see the basic screenshot of Wireshark showing all of the packets of all types being listed in Figure 1. The window is divided into three main sections: the list of filtered packets, the decoded protocol details, and the raw packet data in hex/ASCII format.

[Aug 6, 2009] Xplico 0.5.2

The goal of Xplico is to extract the applications data from an Internet traffic capture. For example, from a pcap file Xplico extracts each email (POP, IMAP, and SMTP protocols), all HTTP contents, each... VoIP call (SIP), and so on. Xplico isn't a packet sniffer or a network protocol analyzer; it's an IP/Internet traffic decoder or network forensic analysis tool (NFAT).

[Jan 2, 2008] vnStat

About:
vnStat is a console-based network traffic monitor that keeps a log of hourly, daily, and monthly network traffic for the selected interface(s). However, it isn't a packet sniffer. The traffic information is analyzed from the /proc filesystem. That way, vnStat can be used even without root permissions.

Release focus: Minor bugfixes

Changes:
This release fixes a bug that caused a segmentation fault if the environment variable "HOME" wasn't defined, which in turn caused most PHP/CGI scripts using vnStat to malfunction. Some minor feature enhancements are also included.

[Apr 15, 2007] freshmeat.net Project details for Tcpreplay by Aaron Turner

Tcpreplay 3.0.RC1 (stable)

This release improves OpenBSD, HP-UX, Cygwin/Win32, x86_64, and little endian support. Enhancements were made to allow editing packets with tcpreplay. libpcap detection was improved.

[Mar 24, 2007] freshmeat.net Project details for Tcpreplay

Tcpreplay 3.0.beta13 released

Tcpreplay is a set of Unix tools which allows the editing and replaying of captured network traffic in pcap (tcpdump) format. It can be used to test a variety of passive and inline network devices, including IPS's, UTM's, routers, firewalls, and NIDS.

Release focus: Major bugfixes

Changes:
This release fixes some serious regression bugs that prevented tcprewrite from editing most packets on Intel and other little-endian systems. Some smaller bugfixes and tweaks to improve replay performance were made.

Author:
Aaron Turner [contact developer]

[Mar 3rd 2006 ] freshmeat.net Project details for netrw by Jiri Denemark

netrw 1.3.1

About: netrw is a simple (but powerful) tool for transporting data over the Internet. Its main purpose is to simplify and speed up file transfers to hosts without an FTP server. It can also be used for uploading data to some other user. It is something like one-way netcat (nc) with some nice features concerning data transfers. It can compute and check message digest (MD5, SHA-1, and some others) of all the data being transferred. It can also print information on progress and average speed. At the end, it sums up the transfer.

Changes: A bug causing netread to sometimes end up in an endless loop after receiving all data was fixed.



Etc

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes.   If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner. 

ABUSE: IPs or network segments from which we detect a stream of probes might be blocked for no less then 90 days. Multiple types of probes increase this period.  

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Haterís Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least


Copyright © 1996-2016 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: September, 12, 2017