Softpanorama University Classic Unix Tools Page

One incarnation of the so called 80/20 rule has been associated with software systems. It has been observed that 80% of a user population regularly uses only 20% of a system's features. Without backing this up with hard statistics, my 20+ years of building and using software systems tells me that this hypothesis is probably true. The collection of Linux command-line programs is no exception to this generalization. Of the dozens of shell-level commands offered by Linux, perhaps only ten commands are commonly understood and utilized, and the remaining majority are virtually ignored.

Which of these dogs of the Linux shell have the most value to offer? I'll briefly describe ten of the less popular but useful Linux shell commands, those which I have gotten some mileage from over the years. Specifically, I've chosen to focus on commands that parse and format textual content.

The working examples presented here assume a basic familiarity with command-line syntax, simple shell constructs and some of the not-so-uncommon Linux commands. Even so, the command-line examples are fairly well commented and straightforward. Whenever practical, the output of usage examples is presented under each command-line execution.

The following eight commands parse, format and display textual content. Although not all provided examples demonstrate this, be aware that the following commands will read from standard input if file arguments are not presented.

Table 1. Summary of Commands

Head/Tail

As their names imply, head and tail are used to display some amount of the top or bottom of a text block. head presents beginning of a file to standard output while tail does the same with the end of a file. Review the following commented examples:

## (1) displays the first 6 lines of a file
   head -6 readme.txt
## (2) displays the last 25 lines of a file
   tail -25 mail.txt

Here's an example of using head and tail in concert to display the 11th through 20th line of a file.

# (3)
head -20 file | tail -10

Manual pages show that the tail command has more command-line options than head. One of the more useful tail option is -f. When it is used, tail does not return when end-of-file is detected, unless it is explicitly interrupted. Instead, tail sleeps for a period and checks for new lines of data that may have been appended since the last read.

## (4) display ongoing updates to the given
##     log file 

tail -f /usr/tmp/logs/daemon_log.txt

Imagine that a dæmon process was continually appending activity logs to the /usr/adm/logs/daemon_log.txt file. Using tail -f at a console window, for example, will more or less track all updates to the file in real time. (The -f option is applicable only when tail's input is a file).

If you give multiple arguments to tail, you can track several log files in the same window.

## track the mail log and the server error log
## at the same time.

tail -f /var/log/mail.log /var/log/apache/error_log

tac--Concatenate in Reverse

What is cat spelled backwards? Well, that's what tac's functionality is all about. It concatenates file order and their contents in reverse. So what's its usefulness? It can be used on any task that requires ordering elements in a last-in, first-out (LIFO) manner. Consider the following command line to list the three most recently established user accounts from the most recent through the least recent.

# (5) last 3 /etc/passwd records - in reverse
$ tail -3 /etc/passwd | tac
curly:x:1003:100:3rd Stooge:/homes/curly:/bin/ksh
larry:x:1002:100:2nd Stooge:/homes/larry:/bin/ksh
moe:x:1001:100:1st Stooge:/homes/moe:/bin/ksh

nl--Numbered Line Output

nl is a simple but useful numbering filter. I displays input with each line numbered in the left margin, in a format dictated by command-line options. nl provides a plethora of options that specify every detail of its numbered output. The following commented examples demonstrate some of of those options:

# (6) Display the first 4 entries of the password
#     file - numbers to be three columns wide and 
#     padded by zeros.
$ head -4 /etc/passwd | nl -nrz -w3
001	root:x:0:1:Super-User:/:/bin/ksh
002	daemon:x:1:1::/:
003	bin:x:2:2::/usr/bin:
004	sys:x:3:3::/:
#
# (7) Prepend ordered line numbers followed by an
#     '=' sign to each line -- start at 101.
$ nl -s= -v101 Data.txt
101=1st Line ...
102=2nd Line ...
103=3rd Line ...
104=4th Line ...
105=5th Line ...
  .......

fmt--Format

The fmt command is a simple text formatter that focuses on making textual data conform to a maximum line width. It accomplishes this by joining and breaking lines around white space. Imagine that you need to maintain textual content that was generated with a word processor. The exported text may contain lines whose lengths vary from very short to much longer than a standard screen length. If such text is to be maintained in a text editor (like vi), fmt is the command of choice to transform the original text into a more maintainable format. The first example below shows fmt being asked to reformat file contents as text lines no greater than 60 characters long.

# (8) No more than 60 char lines
$ fmt -w 60 README.txt > NEW_README.txt
# 
# (9) Force uniform spacing:
#     1 space between words, 2 between sentences
$ echo "Hello   World. Hello Universe." | fmt -u -w80 

Hello World.  Hello Universe.

fold--Break Up Input

fold is similar to fmt but is used typically to format data that will be used by other programs, rather than to make the text more readable to the human eye. The commented examples below are fairly easy to follow:

# (10) Format text in 3 column width lines
$ echo oxoxoxoxo | fold -w3 
oxo
xox
oxo
# (11) Parse by triplet-char strings - 
#      search for 'xox'
$ echo oxoxoxoxo | fold -w3 | grep "xox"
xox
# (12) One way to iterate through a string of chars
$ for i in $(echo 12345 | fold -w1)
> do
> ### perform some task ...
> print $i
> done
1
2
3
4
5

pr

pr shares features with simpler commands like nl and fmt, but its command-line options make it ideal for converting text files into a format that's suitable for printing. pr offers options that allow you to specify page length, column width, margins, headers/footers, double line spacing and more.

Aside from being the best suited formatter for printing tasks, pr also offers other useful features. These features include allowing you to view multiple files vertically in adjacent columns or columnizing a list in a fixed number of columns (see Listing 2).

Listing 2. Using pr

Miscellaneous

The following two commands are specialized parsers used to pick apart file path pieces.

Basename/Dirname

The basename and dirname commands are useful for presenting portions of a given file path. Quite often in scripting situations, it's convenient to be able to parse and capture a file name or the containing-directory name portions of a file path. These commands reduce this task to a simple one-line command. (There are other ways to approach this using the Korn shell or sed "magic", but basename and dirname are more portable and straightforward).

basename is used to strip off the directory, and optionally, the file suffix parts of a file path. Consider the following trivial examples:

:# (21) Parse out the Java Class name
$ basename
/usr/local/src/java/TheClass.java .java 
TheClass 
# (22) Parse out the file name.  
$ basename srcs/C/main.c 
main.c

dirname is used to display the containing directory path, as much of the path as is provided. Consider the following examples:

# (23) absolute and relative directory examples
$ dirname /homes/curly/.profile 
/homes/curly 
$ dirname curly/.profile
curly 
# 
# (24) From any korn-shell script, the following
#  line will assign the directory from where 
#  the script was launched 
SCRIPT_HOME="$(dirname $(whence $0))" 
# 
# (25)
# Okay, how about a non-trivial practical example?
#  List all directories (under $PWD that contain a  
#  file called 'core'.
$ for i in $(find $PWD -name core )^
> do 
> dirname $i
> done | sort -u
bin 
rje/gcc 
src/C

News	Recommended Books	Recommended Links	See Also	Reference	shell	Other Unix utilities	Solaris	Selected Manpages
regex	pipes	grep	find	awk	sed	ed	mail	pax
sort	uniq	tee	join	split	paste	cat	col	xargs
exec	head	tail	col	fmt	printenv	man	fold	fc
m4	expand unexpand	dos2unix unix2dos	ls	locate	cd	wc	ngrep	Netcat
tar	dd	diff	gzip	cut	tr	Beautifiers	Humor	Etc

Softpanorama University
Classic (pipable) Unix Tools

Old News ;-)

UnixPower Tools excepts

Sys Admin Magazine

[13 Jan 2004] The art of writing Linux utilities Peter Seebach (developerworks@seebs.plethora.net)

[Jul 3, 2003] dunne.dyn.dhs.org/Using the m4 Macro Processor - updated link

[Apr 17, 2003] Exploring processes with Truss: Part 1 By Sandra Henry-Stocker

Linux.ie Using the ps command.

3.2. Displaying all processes owned by a specific user

3.3. Own output format

developerWorks Concatenating files with cat Cat has two useful options:

Dogs of the Linux Shell Posted on Saturday, October 19, 2002 by Louis J. Iacona

Could the command-line tools you've forgotten or never knew save time and some frustration?

Table 1. Summary of Commands

Head/Tail

tac--Concatenate in Reverse

nl--Numbered Line Output

fmt--Format

fold--Break Up Input

pr

Listing 2. Using pr

Miscellaneous

Basename/Dirname

ttyrec a tty recorder

Understanding Archivers

Online Gnu Documentation

Slashdot Articles Free Books Online

[Jan 30, 2000] Use the Source, Luke Compiling and installing from source code LG #49

[Jan 3, 2000] Advanced Programming in Expect A Bulletproof Interface LG #48

A very interesting and useful paper. See also: Ext2- Automating interactive tasks with expect and crontab
QCad the user-friendly CAD system for Linux is now open source PC Week PC Week Labs evaluates open-source apps

See Also

Recommended Links

Reference

Humor

Softpanorama University Classic (pipable) Unix Tools

Old News ;-)

[13 Jan 2004] The art of writing Linux utilities Peter Seebach (developerworks@seebs.plethora.net)

[Jul 3, 2003] dunne.dyn.dhs.org/Using the m4 Macro Processor - updated link

[Apr 17, 2003] Exploring processes with Truss: Part 1 By Sandra Henry-Stocker

3.2. Displaying all processes owned by a specific user

3.3. Own output format

developerWorks Concatenating files with cat Cat has two useful options:

Dogs of the Linux Shell Posted on Saturday, October 19, 2002 by Louis J. Iacona

Could the command-line tools you've forgotten or never knew save time and some frustration?

Table 1. Summary of Commands

Head/Tail

tac--Concatenate in Reverse

nl--Numbered Line Output

fmt--Format

fold--Break Up Input

pr

Listing 2. Using pr

Miscellaneous

Basename/Dirname

[Jan 30, 2000] Use the Source, Luke Compiling and installing from source code LG #49

[Jan 3, 2000] Advanced Programming in Expect A Bulletproof Interface LG #48

A very interesting and useful paper. See also: Ext2- Automating interactive tasks with expect and crontab QCad the user-friendly CAD system for Linux is now open source PC Week PC Week Labs evaluates open-source apps

Softpanorama University
Classic (pipable) Unix Tools

A very interesting and useful paper. See also: Ext2- Automating interactive tasks with expect and crontab
QCad the user-friendly CAD system for Linux is now open source PC Week PC Week Labs evaluates open-source apps