Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
May the source be with you, but remember the KISS principle ;-)

Shell debugging

News See Also Recommended Papers Summary of the BASH Debugger   Command history reuse Etc
    Sysadmin Horror Stories Unix shells history Tips Humor Etc

One of the major innovations in bash 3.0 was built-in debugger:

New variables to support the bash debugger:  BASH_ARGC, BASH_ARGV,

i.  FUNCNAME has been changed to support the debugger: it's now an array

j.  for, case, select, arithmetic commands now keep line number information
    for the debugger.

k.  There is a new `RETURN' trap executed when a function or sourced script
    returns (not inherited child processes; inherited by command substitution
    if function tracing is enabled and the debugger is active).

l.  New invocation option:  --debugger.  Enables debugging and turns on new
    `extdebug' shell option.

m.  New `functrace' and `errtrace' options to `set -o' cause DEBUG and ERR
    traps, respectively, to be inherited by shell functions.  Equivalent to
    `set -T' and `set -E' respectively.  The `functrace' option also controls
    whether or not the DEBUG trap is inherited by sourced scripts.

n.  The DEBUG trap is run before binding the variable and running the action
    list in a `for' command, binding the selection variable and running the
    query in a `select' command, and before attempting a match in a `case'

o.  New `--enable-debugger' option to `configure' to compile in the debugger
    support code.

p.  `declare -F' now prints out extra line number and source file information
    if the `extdebug' option is set.

q.  If `extdebug' is enabled, a non-zero return value from a DEBUG trap causes
    the next command to be skipped, and a return value of 2 while in a
    function or sourced script forces a `return'.

r.  New `caller' builtin to provide a call stack for the bash debugger.

s.  The DEBUG trap is run just before the first command in a function body is
    executed, for the debugger.

t.  `for', `select', and `case' command heads are printed when `set -x' is

Top Visited
Past week
Past month


Old News ;-)

Quick Hacks by frodo from middle ea (602941) on Wednesday March 10, @02:06PM (#8523604)

My 2 cent tips on budding shell script authors.

If the script is not working as you want, put a

set -x

on the fist line and

set +x

on the last line.

You will see the exact execution path and variable expansion, very neat for debugging

Recommended Papers

Debugging Bash scripts

2.3.1. Debugging on the entire script

When things don't go according to plan, you need to determine what exactly causes the script to fail. Bash provides extensive debugging features. The most common is to start up the subshell with the -x option, which will run the entire script in debug mode. Traces of each command plus its arguments are printed to standard output after the commands have been expanded but before they are executed.

This is the script ran in debug mode. Note again that the added comments are not visible in the output of the script.

willy:~/scripts> bash -x
+ clear

+ echo 'The script starts now.'
The script starts now.
+ echo 'Hi, willy!'
Hi, willy!
+ echo

+ echo 'I will now fetch you a list of connected users:'
I will now fetch you a list of connected users:
+ echo

+ w
  4:50pm  up 18 days,  6:49,  4 users,  load average: 0.58, 0.62, 0.40
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU  WHAT
root     tty2     -                Sat 2pm  5:36m  0.24s  0.05s  -bash
willy	 :0       -                Sat 2pm   ?     0.00s   ?     -
willy	 pts/3    -                Sat 2pm 43:13  36.82s 36.82s  BitchX willy ir
willy    pts/2    -                Sat 2pm 43:13   0.13s  0.06s  /usr/bin/screen
+ echo

+ echo 'I'\''m setting two variables now.'
I'm setting two variables now.
+ COLOUR=black
+ echo 'This is a string: '
This is a string:
+ echo 'And this is a number: '
And this is a number:
+ echo

+ echo 'I'\''m giving you back your prompt now.'
I'm giving you back your prompt now.
+ echo

2.3.2. Debugging on part(s) of the script

Using the set Bash built-in you can run in normal mode those portions of the script of which you are sure they are without fault, and display debugging information only for troublesome zones. Say we are not sure what the w command will do in the example, then we could enclose it in the script like this:

set -x			# activate debugging from here
set +x			# stop debugging from here

Output then looks like this:

willy: ~/scripts>
The script starts now.
Hi, willy!

I will now fetch you a list of connected users:

+ w
  5:00pm  up 18 days,  7:00,  4 users,  load average: 0.79, 0.39, 0.33
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU  WHAT
root     tty2     -                Sat 2pm  5:47m  0.24s  0.05s  -bash
willy    :0       -                Sat 2pm   ?     0.00s   ?     -
willy    pts/3    -                Sat 2pm 54:02  36.88s 36.88s  BitchX willyke
willy    pts/2    -                Sat 2pm 54:02   0.13s  0.06s  /usr/bin/screen
+ set +x

I'm setting two variables now.
This is a string:
And this is a number:

I'm giving you back your prompt now.

willy: ~/scripts>

You can switch debugging mode on and off as many times as you want within the same script.

The table below gives an overview of other useful Bash options:

Table 2-1. Overview of set debugging options
Short notation Long notation Result
set -f set -o noglob Disable file name generation using metacharacters (globbing).
set -v set -o verbose Prints shell input lines as they are read.
set -x set -o xtrace Print command traces before executing command.

The dash is used to activate a shell option and a plus to deactivate it. Don't let this confuse you!

In the example below, we demonstrate these options on the command line:

willy:~/scripts> set -v

willy:~/scripts> ls

willy:~/scripts> set +v
set +v

willy:~/scripts> ls *

willy:~/scripts> set -f

willy:~/scripts> ls *
ls: *: No such file or directory

willy:~/scripts> touch *

willy:~/scripts> ls

willy:~/scripts> rm *

willy:~/scripts> ls

Alternatively, these modes can be specified in the script itself, by adding the desired options to the first line shell declaration. Options can be combined, as is usually the case with UNIX commands:

#!/bin/bash -xv

Once you found the buggy part of your script, you can add echo statements before each command of which you are unsure, so that you will see exactly where and why things don't work. In the example script, it could be done like this, still assuming that the displaying of users gives us problems:

echo "debug message: now attempting to start w command"; w

In more advanced scripts, the echo can be inserted to display the content of variables at different stages in the script, so that flaws can be detected:

echo "Variable VARNAME is now set to $VARNAME."

Example of debugging a shell script

To print commands and their arguments as they are executed:

   cat example
   if [ $TEST1 = "result2" ]
     echo $TEST1
   if [ $TEST1 = "result1" ]
     echo $TEST1
   if [ $test3 = "whosit" ]
     echo fail here cos it's wrong

This is a script called example which has an error in it; the variable $test3 is not set so the 3rd and last test [command will fail.

Running the script produces:

   [: argument expected

The script fails and to see where the error occurred you would use the -x option like this:

   sh -x example
   + [ result1 = result2 ]
   + [ result1 = result1 ]
   + echo result1
   + [ = whosit ]
   example: [: argument expected

The error occurs in the command [ = whosit ] which is wrong as the variable $test3 has not been set. You can now see where to fix it.

Debugging shell scripts

To see where a script produces an error use the command:

   sh -x script argument

The -x option to the sh command tells it to print commands and their arguments as they are executed.

You can then see what stage of the script has been reached when an error occurs.

11.4.4 Debugging Programs

At times you may need to debug a program to find and correct errors. Two options to the sh command (listed below) can help you debug a program:  

sh -v shellprogramname
prints the shell input lines as they are read by the system
sh -x shellprogramname
prints commands and their arguments as they are executed

To try these two options, create a shell program that has an error in it. For example, create a file called bug that contains the following list of commands:


$ cat bug<CR>
echo enter person
read person
mail $1
When you log off come into my office please.

Notice that
today equals the output of the date command, which must be enclosed in grave accents for command substitution to occur.

The mail message sent to Tom ($1) at login tommy($1) should look like the following screen:

$ mail<CR>
From mlh Mon Apr 10  11:36  CST  1989
When you log off come into my office please.
Mon Apr 10  11:36:32  CST  1989

To execute
bug, you have to press the BREAK or DELETE key to end the program.


To debug this program, try executing bug using sh -v. This will print the lines of the file as they are read by the system, as shown below:


$ sh -v bug tommy<CR>
echo enter person
enter person
read person
mail $1

Notice that the output stops on the
mail command, since there is a problem with mail. You must use the here document to redirect input into mail.

Before you fix the bug program, try executing it with sh -x, which prints the commands and their arguments as they are read by the system.


$ sh -x bug tommy<CR>
today=Mon Apr 10  11:07:23 CST 1989
+ echo enter person
enter person
+ read person
+ mail tom

Once again, the program stops at the
mail command. Notice that the substitutions for the variables have been made and are displayed.

The corrected bug program is as follows:

$ cat bug<CR>
echo enter person
read person
mail $1 <<!
When you log off come into my office please.

The tee command is a helpful command for debugging pipelines. While simply passing its standard input to its standard output, it also saves a copy of its input into the file whose name is given as an argument.


The general format of the tee command is:


command1 | tee saverfile | command2<CR>

saverfile is the file that saves the output of command1 for you to study.


For example, suppose you want to check on the output of the grep command in the following command line:


who | grep $1 | cut -c1-9<CR>

You can use tee to copy the output of grep into a file called check, without disturbing the rest of the pipeline.


who | grep $1 | tee check | cut -c1-9<CR>

The file check contains a copy of the grep output, as shown in the following screen:


$ who | grep mlhmo | tee check | cut -c1-9<CR>
$ cat check<CR>
mlhmo   tty61  Apr  10  11:30

Advanced Bash-Scripting Guide Chapter 30. Debugging

The Bash shell contains no debugger, nor even any debugging-specific commands or constructs. [1] Syntax errors or outright typos in the script generate cryptic error messages that are often of no help in debugging a non-functional script.

Example 30-1. A buggy script

# This is a buggy script.


if [$a -gt 27 ]
  echo $a

exit 0

Output from script:

./ [37: command not found

What's wrong with the above script (hint: after the if)?


Example 30-2. Missing keyword
# What error message will this generate?

for a in 1 2 3
  echo "$a"
# done     # Required keyword 'done' commented out in line 7.

exit 0   

Output from script: line 10: syntax error: unexpected end of file

Note that the error message does not necessarily reference the line in which the error occurs, but the line where the Bash interpreter finally becomes aware of the error.


Error messages may disregard comment lines in a script when reporting the line number of a syntax error.

What if the script executes, but does not work as expected? This is the all too familiar logic error.

Example 30-3. test24, another buggy script

#  This is supposed to delete all filenames in current directory
#+ containing embedded spaces.
#  It doesn't work.  Why not?

badname=`ls | grep ' '`

# echo "$badname"

rm "$badname"

exit 0

Try to find out what's wrong with Example 30-3 by uncommenting the echo "$badname" line. Echo statements are useful for seeing whether what you expect is actually what you get.

In this particular case, rm "$badname" will not give the desired results because $badname should not be quoted. Placing it in quotes ensures that rm has only one argument (it will match only one filename). A partial fix is to remove to quotes from $badname and to reset $IFS to contain only a newline, IFS=$'\n'. However, there are simpler ways of going about it.

# Correct methods of deleting filenames containing spaces.
rm *\ *
rm *" "*
rm *' '*
# Thank you. S.C.

Summarizing the symptoms of a buggy script,

  1. It bombs with a "syntax error" message, or
  2. It runs, but does not work as expected (logic error).
  3. It runs, works as expected, but has nasty side effects (logic bomb).

Tools for debugging non-working scripts include

  1. echo statements at critical points in the script to trace the variables, and otherwise give a snapshot of what is going on.
  2. using the tee filter to check processes or data flows at critical points.
  3. setting option flags -n -v -x

    sh -n scriptname checks for syntax errors without actually running the script. This is the equivalent of inserting set -n or set -o noexec into the script. Note that certain types of syntax errors can slip past this check.

    sh -v scriptname echoes each command before executing it. This is the equivalent of inserting set -v or set -o verbose in the script.

    The -n and -v flags work well together. sh -nv scriptname gives a verbose syntax check.

    sh -x scriptname echoes the result each command, but in an abbreviated manner. This is the equivalent of inserting set -x or set -o xtrace in the script.

    Inserting set -u or set -o nounset in the script runs it, but gives an unbound variable error message at each attempt to use an undeclared variable.

  4. Using an "assert" function to test a variable or condition at critical points in a script. (This is an idea borrowed from C.)

    Example 30-4. Testing a condition with an "assert"

    assert ()                 #  If condition false,
    {                         #+ exit from script with error message.
      if [ -z "$2" ]          # Not enough parameters passed.
        return $E_PARAM_ERR   # No damage done.
      if [ ! $1 ]
        echo "Assertion failed:  \"$1\""
        echo "File \"$0\", line $lineno"
        exit $E_ASSERT_FAILED
      # else
      #   return
      #   and continue executing script.
    condition="$a -lt $b"     # Error message and exit from script.
                              #  Try setting "condition" to something else,
                              #+ and see what happens.
    assert "$condition" $LINENO
    # The remainder of the script executes only if the "assert" does not fail.
    # Some commands.
    # ...
    echo "This statement echoes only if the \"assert\" does not fail."
    # ...
    # Some more commands.
    exit 0
  5. trapping at exit.

    The exit command in a script triggers a signal 0, terminating the process, that is, the script itself. [2] It is often useful to trap the exit, forcing a "printout" of variables, for example. The trap must be the first command in the script.

Trapping signals
Specifies an action on receipt of a signal; also useful for debugging.  
A signal is simply a message sent to a process, either by the kernel or another process, telling it to take some specified action (usually to terminate). For example, hitting a Control-C, sends a user interrupt, an INT signal, to a running program.
trap '' 2
# Ignore interrupt 2 (Control-C), with no action specified.

trap 'echo "Control-C disabled."' 2
# Message when Control-C pressed.


Example 30-5. Trapping at exit

trap 'echo Variable Listing --- a = $a  b = $b' EXIT
# EXIT is the name of the signal generated upon exit from a script.



exit 0
#  Note that commenting out the 'exit' command makes no difference,
#+ since the script exits in any case after running out of commands.
Example 30-6. Cleaning up after Control-C
# A quick 'n dirty script to check whether you are on-line yet.

# Note that $LOGFILE must be readable (chmod 644 /var/log/messages).
# Create a "unique" temp file name, using process id of the script.
# At logon, the line "remote IP address"
#                     appended to /var/log/messages.
# How many lines in log file to check.

# Cleans up the temp file if script interrupted by control-c.


while [ $TRUE ]  #Endless loop.
  # Saves last 100 lines of system log file as temp file.
  # Necessary, since newer kernels generate many log messages at log on.
  search=`grep $KEYWORD $TEMPFILE`
  # Checks for presence of the "IP address" phrase,
  # indicating a successful logon.

  if [ ! -z "$search" ] # Quotes necessary because of possible spaces.
     echo "On-line"
     rm -f $TEMPFILE    # Clean up temp file.
     exit $ONLINE
     echo -n "."        # -n option to echo suppresses newline,
                        # so you get continuous rows of dots.

  sleep 1 

# Note: if you change the KEYWORD variable to "Exit",
# this script can be used while on-line to check for an unexpected logoff.

# Exercise: Change the script, as per the above note,
#           and prettify it.

exit 0

# Nick Drage suggests an alternate method:

while true
  do ifconfig ppp0 | grep UP 1> /dev/null && echo "connected" && exit 0
  echo -n "."   # Prints dots (.....) until connected.
  sleep 2

# Problem: Hitting Control-C to terminate this process may be insufficient.
#          (Dots may keep on echoing.)
# Exercise: Fix this.

# Stephane Chazelas has yet another alternative:


while ! tail -1 "$LOGFILE" | grep -q "$KEYWORD"
do echo -n .
echo "On-line"

# Exercise: Discuss the strengths and weaknesses
#           of each of these various approaches.
The DEBUG argument to trap causes a specified action to execute after every command in a script. This permits tracing variables, for example.

Example 30-7. Tracing a variable


trap 'echo "VARIABLE-TRACE> \$variable = \"$variable\""' DEBUG
# Echoes the value of $variable after every command.


echo "Just initialized \"\$variable\" to $variable."

let "variable *= 3"
echo "Just multiplied \"\$variable\" by 3."

# The "trap 'commands' DEBUG" construct would be more useful
# in the context of a complex script,
# where placing multiple "echo $variable" statements might be
# clumsy and time-consuming.

# Thanks, Stephane Chazelas for the pointer.

exit 0


trap '' SIGNAL (two adjacent apostrophes) disables SIGNAL for the remainder of the script. trap SIGNAL restores the functioning of SIGNAL once more. This is useful to protect a critical portion of a script from an undesirable interrupt.


	trap '' 2  # Signal 2 is Control-C, now disabled.
	trap 2     # Reenables Control-C


[1] Rocky Bernstein's Bash debugger partially makes up for this lack.
[2] By convention, signal 0 is assigned to exit.


FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes.   If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner. 

ABUSE: IPs or network segments from which we detect a stream of probes might be blocked for no less then 90 days. Multiple types of probes increase this period.  


Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy


War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes


Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law


Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Haterís Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least

Copyright © 1996-2016 by Dr. Nikolai Bezroukov. was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting development of this site and speed up access. In case is down you can use the at


The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: June 04, 2016