Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-) Softpanorama Search

Shell debugging

One of the major innovations in bash 3.0 was built-in debugger:

New variables to support the bash debugger:  BASH_ARGC, BASH_ARGV,
    BASH_SOURCE, BASH_LINENO, BASH_SUBSHELL, BASH_EXECUTION_STRING,
    BASH_COMMAND

i.  FUNCNAME has been changed to support the debugger: it's now an array
    variable.

j.  for, case, select, arithmetic commands now keep line number information
    for the debugger.

k.  There is a new `RETURN' trap executed when a function or sourced script
    returns (not inherited child processes; inherited by command substitution
    if function tracing is enabled and the debugger is active).

l.  New invocation option:  --debugger.  Enables debugging and turns on new
    `extdebug' shell option.

m.  New `functrace' and `errtrace' options to `set -o' cause DEBUG and ERR
    traps, respectively, to be inherited by shell functions.  Equivalent to
    `set -T' and `set -E' respectively.  The `functrace' option also controls
    whether or not the DEBUG trap is inherited by sourced scripts.

n.  The DEBUG trap is run before binding the variable and running the action
    list in a `for' command, binding the selection variable and running the
    query in a `select' command, and before attempting a match in a `case'
    command.

o.  New `--enable-debugger' option to `configure' to compile in the debugger
    support code.

p.  `declare -F' now prints out extra line number and source file information
    if the `extdebug' option is set.

q.  If `extdebug' is enabled, a non-zero return value from a DEBUG trap causes
    the next command to be skipped, and a return value of 2 while in a
    function or sourced script forces a `return'.

r.  New `caller' builtin to provide a call stack for the bash debugger.

s.  The DEBUG trap is run just before the first command in a function body is
    executed, for the debugger.

t.  `for', `select', and `case' command heads are printed when `set -x' is
    enabled.
 

Old News ;-)

My 2 cent tips on budding shell script authors.

If the script is not working as you want, put a

set -x

on the fist line and

set +x

on the last line.

You will see the exact execution path and variable expansion, very neat for debugging

Recommended Papers

Debugging Bash scripts

2.3.1. Debugging on the entire script

When things don't go according to plan, you need to determine what exactly causes the script to fail. Bash provides extensive debugging features. The most common is to start up the subshell with the -x option, which will run the entire script in debug mode. Traces of each command plus its arguments are printed to standard output after the commands have been expanded but before they are executed.

This is the commented-script1.sh script ran in debug mode. Note again that the added comments are not visible in the output of the script.

willy:~/scripts> bash -x script1.sh + clear + echo 'The script starts now.' The script starts now. + echo 'Hi, willy!' Hi, willy! + echo + echo 'I will now fetch you a list of connected users:' I will now fetch you a list of connected users: + echo + w 4:50pm up 18 days, 6:49, 4 users, load average: 0.58, 0.62, 0.40 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT root tty2 - Sat 2pm 5:36m 0.24s 0.05s -bash willy :0 - Sat 2pm ? 0.00s ? - willy pts/3 - Sat 2pm 43:13 36.82s 36.82s BitchX willy ir willy pts/2 - Sat 2pm 43:13 0.13s 0.06s /usr/bin/screen + echo + echo 'I'\''m setting two variables now.' I'm setting two variables now. + COLOUR=black + VALUE=9 + echo 'This is a string: ' This is a string: + echo 'And this is a number: ' And this is a number: + echo + echo 'I'\''m giving you back your prompt now.' I'm giving you back your prompt now. + echo 2.3.2. Debugging on part(s) of the script Using the set Bash built-in you can run in normal mode those portions of the script of which you are sure they are without fault, and display debugging information only for troublesome zones. Say we are not sure what the w command will do in the example commented-script1.sh, then we could enclose it in the script like this: set -x # activate debugging from here w set +x # stop debugging from here Output then looks like this: willy: ~/scripts> script1.sh The script starts now. Hi, willy! I will now fetch you a list of connected users: + w 5:00pm up 18 days, 7:00, 4 users, load average: 0.79, 0.39, 0.33 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT root tty2 - Sat 2pm 5:47m 0.24s 0.05s -bash willy :0 - Sat 2pm ? 0.00s ? - willy pts/3 - Sat 2pm 54:02 36.88s 36.88s BitchX willyke willy pts/2 - Sat 2pm 54:02 0.13s 0.06s /usr/bin/screen + set +x I'm setting two variables now. This is a string: And this is a number: I'm giving you back your prompt now. willy: ~/scripts>

You can switch debugging mode on and off as many times as you want within the same script.

The table below gives an overview of other useful Bash options:

Table 2-1. Overview of set debugging options

Short notation	Long notation	Result
set -f	set -o noglob	Disable file name generation using metacharacters (globbing).
set -v	set -o verbose	Prints shell input lines as they are read.
set -x	set -o xtrace	Print command traces before executing command.

The dash is used to activate a shell option and a plus to deactivate it. Don't let this confuse you!

In the example below, we demonstrate these options on the command line:

willy:~/scripts> set -v willy:~/scripts> ls ls commented-scripts.sh script1.sh willy:~/scripts> set +v set +v willy:~/scripts> ls * commented-scripts.sh script1.sh willy:~/scripts> set -f willy:~/scripts> ls * ls: *: No such file or directory willy:~/scripts> touch * willy:~/scripts> ls * commented-scripts.sh script1.sh willy:~/scripts> rm * willy:~/scripts> ls commented-scripts.sh script1.sh

Alternatively, these modes can be specified in the script itself, by adding the desired options to the first line shell declaration. Options can be combined, as is usually the case with UNIX commands:

#!/bin/bash -xv

Once you found the buggy part of your script, you can add echo statements before each command of which you are unsure, so that you will see exactly where and why things don't work. In the example commented-script1.sh script, it could be done like this, still assuming that the displaying of users gives us problems:

echo "debug message: now attempting to start w command"; w

In more advanced scripts, the echo can be inserted to display the content of variables at different stages in the script, so that flaws can be detected:

echo "Variable VARNAME is now set to $VARNAME."

 

Example of debugging a shell script

To print commands and their arguments as they are executed:

   cat example
   #!/bin/sh
   TEST1=result1
   TEST2=result2
   if [ $TEST1 = "result2" ]
   then
     echo $TEST1
   fi
   if [ $TEST1 = "result1" ]
   then
     echo $TEST1
   fi
   if [ $test3 = "whosit" ]
   then
     echo fail here cos it's wrong
   fi

This is a script called example which has an error in it; the variable $test3 is not set so the 3rd and last test [command will fail.

Running the script produces:

   example
   result1
   [: argument expected

The script fails and to see where the error occurred you would use the -x option like this:

   sh -x example
   TEST1=result1
   TEST2=result2
   + [ result1 = result2 ]
   + [ result1 = result1 ]
   + echo result1
   result1
   + [ = whosit ]
   example: [: argument expected

The error occurs in the command [ = whosit ] which is wrong as the variable $test3 has not been set. You can now see where to fix it.

Debugging shell scripts

To see where a script produces an error use the command:

   sh -x script argument

The -x option to the sh command tells it to print commands and their arguments as they are executed.

You can then see what stage of the script has been reached when an error occurs.

• Other debugging options
• Example

11.4.4 Debugging Programs

At times you may need to debug a program to find and correct errors. Two options to the sh command (listed below) can help you debug a program:  

sh -v shellprogramname

prints the shell input lines as they are read by the system

sh -x shellprogramname

prints commands and their arguments as they are executed

To try these two options, create a shell program that has an error in it. For example, create a file called bug that contains the following list of commands:

 

$ cat bug<CR>
today=`date`
echo enter person
read person
mail $1
$person
When you log off come into my office please.
$today.
MLH
$

Notice that today equals the output of the date command, which must be enclosed in grave accents for command substitution to occur.

The mail message sent to Tom ($1) at login tommy($1) should look like the following screen:

$ mail<CR>
From mlh Mon Apr 10  11:36  CST  1989
Tom
When you log off come into my office please.
Mon Apr 10  11:36:32  CST  1989
MLH
?
.

To execute bug, you have to press the BREAK or DELETE key to end the program.

 

To debug this program, try executing bug using sh -v. This will print the lines of the file as they are read by the system, as shown below:

 

$ sh -v bug tommy<CR>
today=`date`
echo enter person
enter person
read person
tom
mail $1

Notice that the output stops on the mail command, since there is a problem with mail. You must use the here document to redirect input into mail.

Before you fix the bug program, try executing it with sh -x, which prints the commands and their arguments as they are read by the system.

 

$ sh -x bug tommy<CR>
+date
today=Mon Apr 10  11:07:23 CST 1989
+ echo enter person
enter person
+ read person
tom
+ mail tom
$

Once again, the program stops at the mail command. Notice that the substitutions for the variables have been made and are displayed.

The corrected bug program is as follows:

$ cat bug<CR>
today=`date`
echo enter person
read person
mail $1 <<!
$person
When you log off come into my office please.
$today
MLH
!
$

The tee command is a helpful command for debugging pipelines. While simply passing its standard input to its standard output, it also saves a copy of its input into the file whose name is given as an argument.

 

The general format of the tee command is:

 

command1 | tee saverfile | command2<CR>

saverfile is the file that saves the output of command1 for you to study.

 

For example, suppose you want to check on the output of the grep command in the following command line:

 

who | grep $1 | cut -c1-9<CR>

You can use tee to copy the output of grep into a file called check, without disturbing the rest of the pipeline.

 

who | grep $1 | tee check | cut -c1-9<CR>

The file check contains a copy of the grep output, as shown in the following screen:

 

$ who | grep mlhmo | tee check | cut -c1-9<CR>
mlhmo
$ cat check<CR>
mlhmo   tty61  Apr  10  11:30
$

Advanced Bash-Scripting Guide Chapter 30. Debugging

The Bash shell contains no debugger, nor even any debugging-specific commands or constructs. [1] Syntax errors or outright typos in the script generate cryptic error messages that are often of no help in debugging a non-functional script.

Example 30-1. A buggy script

#!/bin/bash
# ex74.sh

# This is a buggy script.

a=37

if [$a -gt 27 ]
then
  echo $a
fi 

exit 0

Output from script:

./ex74.sh: [37: command not found

What's wrong with the above script (hint: after the if)?

 

Example 30-2. Missing keyword

#!/bin/bash # missing-keyword.sh: What error message will this generate? for a in 1 2 3 do echo "$a" # done # Required keyword 'done' commented out in line 7. exit 0

Output from script:

missing-keyword.sh: line 10: syntax error: unexpected end of file

Note that the error message does not necessarily reference the line in which the error occurs, but the line where the Bash interpreter finally becomes aware of the error.

 

Error messages may disregard comment lines in a script when reporting the line number of a syntax error.

What if the script executes, but does not work as expected? This is the all too familiar logic error.

Example 30-3. test24, another buggy script

#!/bin/bash # This is supposed to delete all filenames in current directory #+ containing embedded spaces. # It doesn't work. Why not? badname=`ls | grep ' '` # echo "$badname" rm "$badname" exit 0

Try to find out what's wrong with Example 30-3 by uncommenting the echo "$badname" line. Echo statements are useful for seeing whether what you expect is actually what you get.

In this particular case, rm "$badname" will not give the desired results because $badname should not be quoted. Placing it in quotes ensures that rm has only one argument (it will match only one filename). A partial fix is to remove to quotes from $badname and to reset $IFS to contain only a newline, IFS=$'\n'. However, there are simpler ways of going about it.

# Correct methods of deleting filenames containing spaces. rm *\ * rm *" "* rm *' '* # Thank you. S.C.

 

Summarizing the symptoms of a buggy script,

 

1. It bombs with a "syntax error" message, or
2. It runs, but does not work as expected (logic error).
3. It runs, works as expected, but has nasty side effects (logic bomb).

 

Tools for debugging non-working scripts include

 

1. echo statements at critical points in the script to trace the variables, and otherwise give a snapshot of what is going on.
2. using the tee filter to check processes or data flows at critical points.
3. setting option flags -n -v -x

   sh -n scriptname checks for syntax errors without actually running the script. This is the equivalent of inserting set -n or set -o noexec into the script. Note that certain types of syntax errors can slip past this check.

   sh -v scriptname echoes each command before executing it. This is the equivalent of inserting set -v or set -o verbose in the script.

   The -n and -v flags work well together. sh -nv scriptname gives a verbose syntax check.

   sh -x scriptname echoes the result each command, but in an abbreviated manner. This is the equivalent of inserting set -x or set -o xtrace in the script.

   Inserting set -u or set -o nounset in the script runs it, but gives an unbound variable error message at each attempt to use an undeclared variable.

4. Using an "assert" function to test a variable or condition at critical points in a script. (This is an idea borrowed from C.)
   Example 30-4. Testing a condition with an "assert"

   #!/bin/bash # assert.sh assert () # If condition false, { #+ exit from script with error message. E_PARAM_ERR=98 E_ASSERT_FAILED=99 if [ -z "$2" ] # Not enough parameters passed. then return $E_PARAM_ERR # No damage done. fi lineno=$2 if [ ! $1 ] then echo "Assertion failed: \"$1\"" echo "File \"$0\", line $lineno" exit $E_ASSERT_FAILED # else # return # and continue executing script. fi } a=5 b=4 condition="$a -lt $b" # Error message and exit from script. # Try setting "condition" to something else, #+ and see what happens. assert "$condition" $LINENO # The remainder of the script executes only if the "assert" does not fail. # Some commands. # ... echo "This statement echoes only if the \"assert\" does not fail." # ... # Some more commands. exit 0

5. trapping at exit.

   The exit command in a script triggers a signal 0, terminating the process, that is, the script itself. [2] It is often useful to trap the exit, forcing a "printout" of variables, for example. The trap must be the first command in the script.

Trapping signals

trap

Specifies an action on receipt of a signal; also useful for debugging.

 

A signal is simply a message sent to a process, either by the kernel or another process, telling it to take some specified action (usually to terminate). For example, hitting a Control-C, sends a user interrupt, an INT signal, to a running program.

trap '' 2 # Ignore interrupt 2 (Control-C), with no action specified. trap 'echo "Control-C disabled."' 2 # Message when Control-C pressed.

 

Example 30-5. Trapping at exit

#!/bin/bash trap 'echo Variable Listing --- a = $a b = $b' EXIT # EXIT is the name of the signal generated upon exit from a script. a=39 b=36 exit 0 # Note that commenting out the 'exit' command makes no difference, #+ since the script exits in any case after running out of commands.

Example 30-6. Cleaning up after Control-C

#!/bin/bash # logon.sh: A quick 'n dirty script to check whether you are on-line yet. TRUE=1 LOGFILE=/var/log/messages # Note that $LOGFILE must be readable (chmod 644 /var/log/messages). TEMPFILE=temp.$$ # Create a "unique" temp file name, using process id of the script. KEYWORD=address # At logon, the line "remote IP address xxx.xxx.xxx.xxx" # appended to /var/log/messages. ONLINE=22 USER_INTERRUPT=13 CHECK_LINES=100 # How many lines in log file to check. trap 'rm -f $TEMPFILE; exit $USER_INTERRUPT' TERM INT # Cleans up the temp file if script interrupted by control-c. echo while [ $TRUE ] #Endless loop. do tail -$CHECK_LINES $LOGFILE> $TEMPFILE # Saves last 100 lines of system log file as temp file. # Necessary, since newer kernels generate many log messages at log on. search=`grep $KEYWORD $TEMPFILE` # Checks for presence of the "IP address" phrase, # indicating a successful logon. if [ ! -z "$search" ] # Quotes necessary because of possible spaces. then echo "On-line" rm -f $TEMPFILE # Clean up temp file. exit $ONLINE else echo -n "." # -n option to echo suppresses newline, # so you get continuous rows of dots. fi sleep 1 done # Note: if you change the KEYWORD variable to "Exit", # this script can be used while on-line to check for an unexpected logoff. # Exercise: Change the script, as per the above note, # and prettify it. exit 0 # Nick Drage suggests an alternate method: while true do ifconfig ppp0 | grep UP 1> /dev/null && echo "connected" && exit 0 echo -n "." # Prints dots (.....) until connected. sleep 2 done # Problem: Hitting Control-C to terminate this process may be insufficient. # (Dots may keep on echoing.) # Exercise: Fix this. # Stephane Chazelas has yet another alternative: CHECK_INTERVAL=1 while ! tail -1 "$LOGFILE" | grep -q "$KEYWORD" do echo -n . sleep $CHECK_INTERVAL done echo "On-line" # Exercise: Discuss the strengths and weaknesses # of each of these various approaches.

 

The DEBUG argument to trap causes a specified action to execute after every command in a script. This permits tracing variables, for example. Example 30-7. Tracing a variable #!/bin/bash trap 'echo "VARIABLE-TRACE> \$variable = \"$variable\""' DEBUG # Echoes the value of $variable after every command. variable=29 echo "Just initialized \"\$variable\" to $variable." let "variable *= 3" echo "Just multiplied \"\$variable\" by 3." # The "trap 'commands' DEBUG" construct would be more useful # in the context of a complex script, # where placing multiple "echo $variable" statements might be # clumsy and time-consuming. # Thanks, Stephane Chazelas for the pointer. exit 0

 

trap '' SIGNAL (two adjacent apostrophes) disables SIGNAL for the remainder of the script. trap SIGNAL restores the functioning of SIGNAL once more. This is useful to protect a critical portion of a script from an undesirable interrupt.

 

trap '' 2 # Signal 2 is Control-C, now disabled. command command command trap 2 # Reenables Control-C

Notes

[1]	Rocky Bernstein's Bash debugger partially makes up for this lack.
[2]	By convention, signal 0 is assigned to exit.