Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-)

Care and Feeding of Functions in Shell

Functions in shell are really scripts which run in the current context of the current shell instance (no secondary shell is forked to run the function; it's run within the current shell.)

Functions provide a lot more flexibility that aliases.  Here are two simplest functions possible: "do nothing" function and "Hello world" function:

          function quit {
               exit
           }
           function hello {
               print "Hello world !"
           }
           hello
           quit
       

Declaring a function is just a matter of writing function my_func { my_code }. Functions can be declared in arbitrary order. But they need to be declared before they are invoked in the script.  Calling a function is just like calling another program, you just write its name and (optionally) arguments.

The best way to create a set of  useful shell functions in a separate file. Then you can source the file with the dot (.) command or in your startup scripts. You can also just enter the function at the command line.

To create a function from the command line, you would do something like this:

$ psgrep() {
> ps -ef | grep $1
> }

This is a pretty simple function, and could be implemented as an alias as well. Let's try to solve the problem of displayed files over 1K.  awk can be used to find any matching files that are larger than 100K bytes:

largesize() {
ls -la |  awk ' { if ( $5 gt 100000 ) print $1 } '
}

As in almost any programming language, you can use functions to group pieces of code in a more logical way or practice the divine art of recursion.

Arguments

NOTE: The "$#" macro is  expanded to contain the number of arguments.

Here is an example of a function that takes one parameter and prints it:

           function arg1_echo {
                    print "The first argument is:" $1 
           }
	   function all_arg_echo {
                    print "List of argumnets is:" $@ 
           }
           function arg_num_echo {
                    print "Number of arguments is:" $# 
           }                

    arg1_echo test

    all_arg_echo  test 1 test 2

Checking the loaded functions

the set command displays all the loaded functions available to the shell.

$ set
USER=dave
findit=()
{
if [ $# -lt 1 ]; then
  echo "usage :findit file";
  return 1;
fi;
find / -name $1 -print
}
...

You can use  unset command to remove functions:

unset function_name

The idea of  .functions file

Traditionally kshrc contained aliases. So for functions it might be beneficial to use a separate file called, for example, .functions.

Let's consider a very artificial (bad) example of creating a function that will call find for each argument so that we can find several different files with one command (useless exersize) The function will now look like this:

$ pg .functions 







#!/bin/sh
findit()
{

# findit
if [ $# -lt 1 ]; then
  echo "usage :findit file"
  return 1
fi
for member
do
  find / -name $member -print
done 

} 

Now source the file again:

. ./.functions

Using the set command to see that it is indeed loaded, you will notice that the shell has correctly interpreted the for loop to take in all parameters.

$ set
findit=()
{
if [ $# -lt 1 ]; then
  echo "usage :`basename $0` file";
  return 1;
fi;
for loop in "$@";
do
  find / -name $loop -print;
done
} 







... 

Now to execute the changed  Supplying a couple of files to find:

$ findit LPSO.doc passwd
/usr/local/accounts/LPSO.doc
/etc/passwd

... 

Notes: Those pages are written by people for whom English is not a native language. Some amount of grammar and spelling errors should be expected. This is a Spartan WHYFF (We Help You For Free) site. It cannot replace the best teachers and the best books. The site contain some obsolete pages as it develops like a living tree... Some links on older pages are broken. Please try to use Google, Open directory, etc. to find a replacement link (see HOWTO search the WEB for details). We would appreciate if you can mail us a correct link. Search Amazon by keywords:     Open directory Research Index

Old News ;-)

[Feb 1, 2007] Functions and aliases in bash

[Apr 1, 2006] BigAdmin Submitted Article/ Converting a ksh Function to a ksh Script by William R. Seppeler, April 2006

Here is a simple way to create a script that will behave both as an executable script and as a ksh function. Being an executable script means the script can be run from any shell. Being a ksh function means the script can be optimized to run faster if launched from a ksh shell. This is an attempt to get the best of both worlds.

Procedure

Start by writing a ksh function. A ksh function is just like a ksh script except the script code is enclosed within a function name { script } construct.

Take the following example:

# Example script

function fun {
  print "pid=$$ cmd=$0 args=$*" opts="$-"
}

Save the text in a file. You'll notice nothing happens if you try to execute the code as a script:

ksh ./example

In order to use a function, the file must first be sourced. Sourcing the file will create the function definition in the current shell. After the function has been sourced, it can then be executed when you call it by name:

.. ./example
fun

To make the function execute as a script, the function must be called within the file. Add the bold text to the example function.

# Example script

function fun {
  print "pid=$$ cmd=$0 args=$*" opts="$-"
}

fun $*

Now you have a file that executes like a ksh script and sources like a ksh function. One caveat is that the file now executes while it is being sourced.

There are advantages and disadvantages to how the code is executed. If the file was executed as a script, the system spawns a child ksh process, loads the function definition, and then executes the function. If the file was sourced, no child process is created, the function definition is loaded into the current shell process, and the function is then executed.

Sourcing the file will make it run faster because no extra processes are created, however, loading a function occupies environment memory space. Functions can also manipulate environment variables whereas a script only gets a copy to work with. In programming terms, a function can use call by reference parameters via shell variables. A shell script is always call by value via arguments.

Advanced Information

When working with functions, it's advantageous to use ksh autoloading. Autoloading eliminates the need to source a file before executing the function. This is accomplished by saving the file with the same name as the function. In the above example, save the example as the file name "fun". Then set the FPATH environment variable to the directory where the file fun is. Now, all that needs to be done is type "fun" on the command line to execute the function.

Notice the double output the first time fun is called. This is because the first time the function is called, the file must be sourced, and in sourcing the file, the function gets called. What we need is to only call the function when the file is executed as a script, but skip calling the function if the file is sourced. To accomplish this, notice the output of the script when executing it as opposed to sourcing it. When the file is sourced, arg0 is always -ksh. Also, note the difference in opts when the script is sourced. Test the output of arg0 to determine if the function should be called or not. Also, make the file a self-executing script. After all, no one likes having to type "ksh" before running every ksh script.

#!/bin/ksh
# Example script

function fun {
  print "pid=$$ cmd=$0 args=$*" opts="$-"
}















 [[ "${0##*/}" == "fun" ]] && fun $*

Now the file is a self-executing script as well as a self-sourcing function (when used with ksh autoloading). What becomes more interesting is that since the file can be an autoload function as well as a stand-alone script, it could be placed in a single directory and have both PATH and FPATH point to it.

# ${HOME}/.profile

FPATH=${HOME}/bin
PATH=${FPATH}:${PATH}

In this setup, fun will always be called as a function unless it's explicitly called as ${HOME}/bin/fun.

Considerations

Even though the file can be executed as a function or a script, there are minor differences in behavior between the two. When the file is sourced as a function, all local environment variables will be visible to the script. If the file is executed as a script, only exported environment variables will be visible. Also, when sourced, a function can modify all environment variables. When the file is executed, all visible environment variables are only copies. We may want to make special allowances depending on how the file is called. Take the following example.

#!/bin/ksh

# Add arg2 to the contents of arg1

function addTo {
  eval $1=$(($1 + $2))
}

if [[ "${0##*/}" == "addTo" ]]; then
  addTo $*
  eval print \$$1
fi

The script is called by naming an environment variable and a quantity to add to that variable. When sourced, the script will directly modify the environment variable with the new value. However, when executed as a script, the environment variable cannot be modified, so the result must be output instead. Here is a sample run of both situations.

# called as a function
var=5
addTo var 3
print $var

# called as a script
var=5
export var
var=$(./addTo var 3)
print $var

Note the extra steps needed when executing this example as a script. The var must be exported prior to running the script or else it won't be visible. Also, because a script can't manipulate the current environment, you must capture the new result.

Extra function-ality

It's possible to package several functions into a single file. This is nice for distribution as you only need to maintain a single file. In order to maintain autoloading functionality, all that needs to be done is create a link for each function named in the file.

#!/bin/ksh

function addTo {
  eval $1=$(($1 + $2))
}

function multiplyBy {
  eval $1=$(($1 * $2))
}

if [[ "${0##*/}" == "addTo" ]] \
|| [[ "${0##*/}" == "multiplyBy" ]]; then
  ${0##*/} $*
  eval print \$$1
fi

if [[ ! -f "${0%/*}/addTo" ]] \
|| [[ ! -f "${0%/*}/multiplyBy" ]]; then
  ln "${0}" "${0%/*}/addTo"
  ln "${0}" "${0%/*}/multiplyBy"
  chmod u+rx "${0}"
fi

Notice the extra code at the bottom. This text could be saved in a file named myDist. The first time the file is sourced or executed, the appropriate links and file permissions will be put in place, thus creating a single distribution for multiple functions. Couple that with making the file a script executable and you end up with a single distribution of multiple scripts. It's like a shar file, but nothing actually gets unpacked.

The only downside to this distribution tactic is that BigAdmin will only credit you for each file submission, not based on the actual number of executable programs...

Time to Run

Try some of the sample code in this document. Get comfortable with the usage of each snippet to understand the differences and limitations. In general, it's safest to always distribute a script, but it's nice to have a function when speed is a consideration. Do some timing tests.

export var=8
time ./addTo var 5
time addTo var 5

If this code were part of an inner-loop calculation of a larger script, that speed difference could be significant.

This document aims to provide the best of both worlds. You can have a script and retain function speed for when it's needed. I hope you have enjoyed this document and its content. Thanks to Sun and BigAdmin for the hosting and support to make contributions like this possible.

Recommended Links

Linux tip Bash parameters and parameter expansions

Marco's Bash Functions Library - Summary [Gna!]

This package is an attempt to make GNU bash a viable solution for medium sized scripts. A problem with bash is that it doesn't provide encapsulation of any sort, beside the feature of providing functions. This problem is partly solved by writing subscripts and invoking them in the main script, but this is not always the best solution.

A set of modules implementing common operations and a script template are provided by this package and the author has used them with success in implementing non-small scripts.

The philosophy of MBFL is to do the work as much as possible without external commands. For example: string manipulation is done using the special variable substitution provided by bash, and no use is done of utilities like sed, grep and ed.

The library is better used if our script is developed on the template provided in the package (examples/template.sh). This is because with MBFL some choices have been made to reduce the application dependent part of the script to the smallest dimension; if we follow another schema, MBFL modules may be indequate. This is especially true for the options parsing module.

The best way to use the library is to include at runtime the library file libmbfl.sh i; this is possible by installing MBFL on the system and using this code in the scripts:

mbfl_INTERACTIVE='no'
source "${MBFL_LIBRARY:=`mbfl-config`}"

after the service variables have been declared (Service Variables for details). This code will read the full pathname of the library from the environment variable MBFL_LIBRARY; if this variable is not set: the script mbfl-config is invoked with no arguments to acquire the pathname of the library. mbfl-config is installed in the bin directory with the library.

Another solution is to include the library directly in the script; this is easy if we preprocess our scripts with GNU m4:

m4_changequote([[, ]])
m4_include(libmbfl.sh)

is all we need to do. We can preprocess the script with:

$ m4 --prefix-builtins --include=/path/to/library \
         script.sh.m4 >script.sh

easy to do in a Makefile; we can take the MBFL's Makefile as example of this method.

It is also interesting to process the script with the following rule:

M4      = ...
M4FLAGS = --prefix-builtins --include=/path/to/library

%.sh: %.sh.m4
        $(M4) $(M4FLAGS) $(<) | \
        grep --invert-match -e '^#' -e '^$$' | \
        sed -e "s/^ \\+//" >$(@)

this will remove all the comments and blank lines, decreasing the size of the script significantly if one makes use of verbose comments; note that this will wipe out the #!/bin/bash first line also.

Usually we want the script to begin with #!/bin/bash followed by a comment describing the license terms.

Bash by example, Part 3

Encoding and decoding strings

The purpose of this module is to let an external process invoke a bash script with damncommand line arguments: strings including blanks or strange characters that may trigger quoting rules.

This problem can arise when using scripting languages with some sort of eval command.

The solution is to encode the argument string in hexadecimal or octal format strings, so that all the damn characters are converted to "good" ones. The the bash script can convert them back.

Example:
 

mbfl_decode_hex 414243
-> ABC

Manipulating files and pathnames

• File Names: Manipulating file names.
• File Commands: Manipulating files with external commands.
• File Testing: Testing file existence and the like.
• File Misc: Miscellaneous commands.

File names

• File Name Parts: Splitting a file name into its components.
• File Name Path: Handling relative pathnames.
• File Name System: Finding pathnames on the system.

Splitting a file name into its components

Handling relative pathnames

Finding pathnames on the system

File Commands

• File Commands Listing: Retrieving informations.
• File Commands Mkdir: Creating directories.
• File Commands Copy: Copying files.
• File Commands Removing: Removing files and directories.
• File Commands Tar: Manipulating tar archives.

Retrieving informations

readlink -fn $pathname

Creating directories

Copying files

Removing files and directories

Files removal is forced: the --force option to rm is always used. It is responsibility of the caller to validate the operation before invoking these functions.

Some functions test the existence of the pathname before attempting to remove it: this is done only if test execution is disabled; if test execution is enabled the command line is echoed to stderr to make it easier to debug scripts.

Manipulating tar archives

Remember that when we execute a script with the --test option: the external commands are not executed: a command line is echoed to stdout. It is recommended to use this mode to fine tune the command line options required by tar.

Testing file existence and the like

Miscellaneous commands

Parsing command line options

The getopt module defines a set of procedures to be used to process command line arguments with the following format:

-a

brief option a with no value;
 

-a123

brief option a with value 123;
 

--bianco

long option bianco with no value;
 

--color=bianco

long option color with value bianco.

Requires the message module (Message for details).

• Arguments:
• Using the module:
• Predefined options:
• Interface functions:
• Querying Options:

Arguments

The module contains, at the root level, a block of code like the following:
 

ARGC=0
declare -a ARGV ARGV1

for ((ARGC1=0; $# > 0; ++ARGC1)); do
    ARGV1[$ARGC1]="$1"
    shift
done

this block is executed when the script is evaluated. Its purpose is to store command line arguments in the global array ARGV1 and the number of command line arguments in the global variable ARGC1.

The global array ARGV and the global variable ARGC are predefined and should be used by the mbfl_getopts functions to store non-option command line arguments.

Example:
 

$ script --gulp wo --gasp=123 wa

if the script makes use of the library, the strings wo and wa will go into ARGV and ARGC will be set to 2. The option arguments are processed and some action is performed to register them.

We can access the non-option arguments with the following code:
 

for ((i=0; $i < $ARGC; ++i)); do
    # do something with ${ARGV[$i]}
done

Using the module

To use this module we have to declare a set of script options; we declare a new script option with the function mbfl_declare_option. Options declaration should be done at the beginning of the script, before doing anything; for example: right after the MBFL library code.

In the main block of the script: options are parsed by invoking mbfl_getopts_parse: this function will update a global variable and invoke a script function for each option on the command line.

Examples

Example of option declaration:
 

mbfl_declare_option ALPHA no a alpha noarg "enable alpha option"

this code declares an option with no argument and properties:

• global variable script_option_ALPHA, which will be set to no by default and to yes if the option is used;
• brief flag -a;
• long flag --alpha;
• description enable alpha option, to be shown in the usage output.

If the option is used: the function script_option_update_alpha is invoked (if it exists) with no arguments, after the variable script_option_ALPHA has been set to yes. Valid option usages are:
 

$ script.sh -a
$ script.sh --alpha

Another example:
 

mbfl_declare_option BETA 123 b beta witharg "select beta value"

this code declares an option with argument and properties:

• global variable script_option_BETA, which will be set to 123 by default and to the value selected on the command line if the option is used;
• brief flag -b;
• long flag --beta;
• description select beta value, to be shown in the usage output.

If the option is used: the function script_option_update_beta is invoked (if it exists) with no arguments, after the variable script_option_BETA has been set to the selected value. Valid option usages are:
 

$ script.sh -b456
$ script.sh --beta=456

Predefined options

A set of predefined options is recognised by the library and not handed to the user defined functions.

--encoded-args

Signals to the library that the non-option arguments and the option values are encoded in hexadecimal strings. Encoding is useful to avoid quoting problems when invoking a script from another one.

If this option is used: the values are decoded by mbfl_getopts_parse before storing them in the ARGV array and before being stored in the option's specific global variables.
 

-v

--verbose

Turns on verbose messages. The fuction mbfl_option_verbose returns true (Message, for details).
 

--silent

Turns off verbose messages. The fuction mbfl_option_verbose returns false.
 

--verbose-program

If used the --verbose option is added to the command line of external programs that support it. The fuction mbfl_option_verbose_program returns true or false depending on the state of this option.
 

--show-program

Prints the command line of executed external programs.
 

--debug

Turns on debugging messages (Message, for details).
 

--test

Turns on test execution (Program Testing, for details).
 

--null

Signals to the script that it has to use the null character to separate values, instead of the common newline. The global variable mbfl_option_NULL is set to yes.
 

-f

--force

Signals to the script that it does not have to query the user before doing dangerous operations, like overwriting files. The global variable mbfl_option_INTERACTIVE is set to no.
 

-i

--interactive

Signals to the script that it does have to query the user before doing dangerous operations, like overwriting files. The global variable mbfl_option_INTERACTIVE is set to yes.
 

--validate-programs

Validates the existence of all the programs needed by the script; then exits. The exit code is zero if all the programs were found, one otherwise.
 

--version

Prints to the standard output of the script the contents of the global variable mbfl_message_VERSION, then exits with code zero. The variable makes use of the service variables (Service Variables, for details).
 

--version-only

Prints to the standard output of the script the contents of the global variable script_VERSION, then exits with code zero.
 

--license

Prints to the standard output of the script the contents of one of the global variables mbfl_message_LICENSE_*, then exits with code zero. The variable makes use of the service variables (Service Variables, for details).
 

-h

--help

--usage

Prints to the standard output of the script: the contents of the global variable script_USAGE; a newline; the string options:; a newline; an automatically generated string describing the options declared with mbfl_declare_option; a string describing the MBFL default options. Then exits with code zero.

The following options may be used to set, unset and query the state of the predefined options.

Interface functions

Querying Options

Some feature and behaviour of the library is configured by the return value of the following set of functions. All of these functions are defined by the Getopts module, but they can be redefined by the script.

Printing messages to the console

This module allows one to print messages on an output channel. Various forms of message are supported.

All the function names are prefixed with mbfl_message_. All the messages will have the forms:
 

<progname>: <message>
<progname>: [error|warning]: <message>

The following global variables are declared:

mbfl_message_PROGNAME

must be initialised with the name of the script that'll be displayed at the beginning of each message;
 

mbfl_message_VERBOSE

yes if verbose messages should be displayed, else no;

Using external programs

This module declares a set of global variables all prefixed with mbfl_program_. We have to look at the module's code to see which one are declared.

• Program Testing: Testing a script and running programs.
• Program Checking: Checking programs existence.
• Program Executing: Executing a program.
• Program Declaring: Declaring the intention to use a program.

Testing a script and running programs

MBFL allows a script to execute a "dry run", that is: do not perform any operation on the system, just print messages describing what will happen if the script is executed with the selected options. This implies, in the MBFL model, that no external program is executed.

When this feature is turned on: mbfl_program_exec does not execute the program, instead it prints the command line on standard error and returns true.

Checking programs existence

The simpler way to test the availability of a program is to look for it just before it is used. The following function should be used at the beginning of a function that makes use of external programs.

type -ap program

Executing a program

Declaring the intention to use a program

To make a script model simpler, we assume that the unavailability of a program at the time of its execution is a fatal error. So if we need to execute a program and the executable is not there, the script must be aborted on the spot.

Functions are available to test the availability of a program, so we can try to locate an alternative or terminate the process under the script control. On a system where executables may vanish from one moment to another, no matter how we test a program existence, there's always the possibility that the program is not "there" when we invoke it.

If we just use mbfl_program_exec to invoke an external program, the function will try and fail if the executable is unavailable: the return code will be false.

The vanishing of a program is a rare event: if it's there when we look for it, probably it will be there also a few moments later when we invoke it. For this reason, MBFL proposes a set of functions with which we can declare the intention of a script to use a set of programs; a command line option is predefined to let the user test the availability of all the declared programs before invoking the script.

mbfl_program_validate_declared || mbfl_exit_program_not_found

Catching signals

MBFL provides an interface to the trap builtin that allows the execution of more than one function when a signal is received; this may sound useless, but that is it.

Manipulating strings

• String Quote: Quoted characters.
• String Inspection: Inspecting a string.
• String Splitting: Splitting a string.
• String Case: Converting between upper and lower case.
• String Class: Matching a string with a class.
• String Misc: Miscellaneous functions.

Quoted characters

Inspecting a string

Splitting a string

Example of usage for mbfl_string_chars:
 

string="abcde\nfghilm"
mbfl_string_chars "${string}"
# Now:
# "${#string}" = $SPLITCOUNT
#  a = "${SPLITFIELD[0]}"
#  b = "${SPLITFIELD[1]}"
#  c = "${SPLITFIELD[2]}"
#  d = "${SPLITFIELD[3]}"
#  e = "${SPLITFIELD[4]}"
#  \n = "${SPLITFIELD[5]}"
#  f = "${SPLITFIELD[6]}"
#  g = "${SPLITFIELD[7]}"
#  h = "${SPLITFIELD[8]}"
#  i = "${SPLITFIELD[9]}"
#  l = "${SPLITFIELD[10]}"
#  m = "${SPLITFIELD[11]}"

Converting between upper and lower case

Matching a string with a class

$'char'

Miscellaneous functions

Interacting with the user

Manipulating variables

• Variables Arrays:
• Variables Colon:

Manipulating arrays

Manipulating colon variables

Main function

MBFL declares a function to drive the execution of the script; its purpose is to make use of the other modules to reduce the size of scripts depending on MBFL. All the code blocks in the script, with the exception of global variables declaration, should be enclosed in functions.

type -t FUNCNAME = function

Building test suites

MBFL comes with a little library of functions that may be used to build test suites; its aim is at building tests for bash functions/commands/scripts.

The ideas at the base of this library are taken from the tcltest package distributed with the TCL core ¹; this package had contributions from the following people/entities: Sun Microsystems, Inc.; Scriptics Corporation; Ajuba Solutions; Don Porter, NIST; probably many many others.

The library tries to do as much as possible using functions and aliases, not variables; this is an attempt to let the user redefine functions to his taste.

• Testing Intro: A way to organise a test suite.
• Testing Config: Configuring the package.
• Testing Running: Running tests.
• Testing Compare: Validating results by comparing.
• Testing Output: Validating results by output.
• Testing Messages: Printing messages from test functions.
• Testing Files: Handling files in tests.

A way to organise a test suite

A useful way to organise a test suite is to split it into a set of files: one for each module to be tested.

The file mbfltest.sh must be sourced at the beginning of each test file.

The function dotest should be invoked at the end of each module in the test suite; each module should define functions starting with the same prefix. A module should be stored in a file, and should look like the following:
 

# mymodule.test --

source mbfltest.sh
source module.sh

function module-featureA-1.1 () { ... }
function module-featureA-1.2 () { ... }
function module-featureA-2.1 () { ... }
function module-featureB-1.1 () { ... }
function module-featureB-1.2 () { ... }

dotest module-

### end of file

the file should be executed with:
 

$ bash mymodule.test

To test just "feature A":
 

$ TESTMATCH=module-featureA bash mymodule.test

Remember that the source builtin will look for files in the directories selected by the PATH environment variables, so we may want to do:
 

$ PATH="path/to/modules:${PATH}" \
TESTMATCH=module-featureA bash mymodule.test

It is better to put such stuff in a Makefile, with GNU make:
 

top_srcdir      = ...
builddir        = ...
BASHPROG        = bash
MODULES         = moduleA moduleB

testdir         = $(top_srcdir)/tests
test_FILES      = $(foreach f, $(MODULES), $(testdir)/$(f).test)
test_TARGETS    = test-modules

test_ENV        = PATH=$(builddir):$(testdir):$(PATH) TESTMATCH=$(TESTMATCH)
test_CMD        = $(test_ENV) $(BASHPROG)

.PHONY: test-modules

test-modules:
ifneq ($(strip $(test_FILES)),)
        @$(foreach f, $(test_FILES), $(test_CMD) $(f);)
endif

Configuring the package

Running test functions