Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-)

String Operations in Shell

String operators allow you to manipulate the contents of a variable without resorting to AWK or Perl. 

There are three kinds of variable substitution:

• Pattern matching (only bash 3.2 has =~ operator with "normal regular expressions )
• Substitution
• Command substitution

I'll talk about the first two and leave command substitution for another article.

Please note that double quoted string in shell is not equal to double quoted string in Perl.  If you want strings expansion in Perl style you need to use $'string'

#!/bin/bash # String expansion. # Introduced with version 2 of Bash. # Strings of the form $'xxx' #+ have the standard escaped characters interpreted. echo $'Ringing bell 3 times \a \a \a' # May only ring once with certain terminals. echo $'Three form feeds \f \f \f' echo $'10 newlines \n\n\n\n\n\n\n\n\n\n' echo $'\102\141\163\150' # Bash # Octal equivalent of characters. exit 0

#!/bin/bash

export INFILE=${1-"infile"}

export OUTFILE=${2-"outfile"}

cat $INFILE $OUTFILE

#!/bin/bash
# quickly convert html filenames for use on a dossy system
# only handles file extensions, not filenames

for i in *.html; do
  if [ -f ${i%l} ]; then
    echo ${i%l} already exists
  else
    mv $i ${i%l}
  fi
done

#!/bin/bash

export INFILE=${1-"infile"}

export OUTFILE=${2-"outfile"}

cat $INFILE $OUTFILE

KSH Pattern-matching Operators

 Korn shell's pattern-matching operators.

These can be hard to remember, so here's a handy mnemonic device: # matches the front because number signs precede numbers; % matches the rear because percent signs follow numbers.

The expression ${DIRSTACK :-$PWD} evaluates to $DIRSTACK if it is non-null or $PWD (the current directory) if it is null.

The first two of these operators are ideal for setting defaults for command-line arguments in case the user omits them. We'll use the first one in our first programming task.

The bash shell has many features that are sufficiently obscure you almost never see them used. One of the problems is that the man page offers no examples.

Here I'm going to show how to use some of these features to do the sorts of simple string manipulations that are commonly needed on file and path names.

Background

In traditional Bourne shell programming you might see references to the basename and dirname commands. These perform simple string manipulations on their arguments. You'll also see many uses of sed and awk or perl -e to perform simple string manipulations.

Often these machinations are necessary perform on lists of filenames and paths. There are many specialized programs that are conventionally included with Unix to perform these sorts of utility functions: tr, cut, paste, and join. Given a filename like /home/myplace/a.data.directory/a.filename.txt which we'll call $f you could use commands like:

	dirname $f 
	basename $f 
	basename $f .txt
	

... to see output like:

	/home/myplace/a.data.directory
	a.filename.txt
	a.filename 

Notice that the GNU version of basename takes an optional parameter. This handy for specifying a filename "extension" like .tar.gz which will be stripped off of the output. Note that basename and dirname don't verify that these parameters are valid filenames or paths. They simple perform simple string operations on a single argument. You shouldn't use wild cards with them -- since dirname takes exactly one argument (and complains if given more) and basename takes one argument and an optional one which is not a filename.

Despite their simplicity these two commands are used frequently in shell programming because most shells don't have any built-in string handling functions -- and we frequently need to refer to just the directory or just the file name parts of a given full file specification.

Usually these commands are used within the "back tick" shell operators like TARGETDIR=`dirname $1`. The "back tick" operators are equivalent to the $(...) construct. This latter construct is valid in Korn shell and bash -- and I find it easier to read (since I don't have to squint at me screen wondering which direction the "tick" is slanted).

A Better Way

Although the basename and dirname commands embody the "small is beautiful" spirit of Unix -- they may push the envelope towards the "too simple to be worth a separate program" end of simplicity.

Naturally you can call on sed, awk, TCL or perl for more flexible and complete string handling. However this can be overkill -- and a little ungainly.

So, bash (which long ago abandoned the "small is beautiful" principal and went the way of emacs) has some built in syntactical candy for doing these operations. Since bash is the default shell on Linux systems then there is no reason not to use these features when writing scripts for Linux.

If your concerned about portability to other shells and systems -- you may want to stick with dirname, basename, and sed

The bash Man Page

The bash man page is huge. In contains a complete reference to the "readline" libraries and how to write a .inputrc file (which I think should all go in a separate man page) -- and a run down of all the csh "history" or bang! operators (which I think should be replaced with a simple statement like: "Most of the bang! tricks that work in csh work the same way in bash").

However, buried in there is a section on Parameter Substitution which tells us that $foo is really a shorthand for ${foo} which is really the simplest case of several ${foo:operators} and similar constructs.

Are you confused, yet?

Here's where a few examples would have helped. To understand the man page I simply experimented with the echo command and several shell variables. This is what it all means:

Given:

foo=/tmp/my.dir/filename.tar.gz

We can use these expressions:

path = ${foo%/*}

To get: /tmp/my.dir (like dirname)

file = ${foo##*/}

To get: filename.tar.gz (like basename)

base = ${file%%.*}

To get: filename

ext = ${file#*.}

To get: tar.gz

Note that the last two depend on the assignment made in the second one

Here we notice two different "operators" being used inside the parameters (curly braces). Those are the # and the % operators. We also see them used as single characters and in pairs. This gives us four combinations for trimming patterns off the beginning or end of a string:

${variable%pattern}

Trim the shortest match from the end

${variable##pattern}

Trim the longest match from the beginning

${variable%%pattern}

Trim the shortest match from the end

${variable#pattern}

Trim the shortest match from the beginning

It's important to understand that these use shell "globbing" rather than "regular expressions" to match these patterns. Naturally a simple string like "txt" will match sequences of exactly those three characters in that sequence -- so the difference between "shortest" and "longest" only applies if you are using a shell wild card in your pattern.

A simple example of using these operators comes in the common question of copying or renaming all the *.txt to change the .txt to .bak (in MS-DOS' COMMAND.COM that would be REN *.TXT *.BAK).

This is complicated in Unix/Linux because of a fundamental difference in the programming API's. In most Unix shells the expansion of a wild card pattern into a list of filenames (called "globbing") is done by the shell -- before the command is executed. Thus the command normally sees a list of filenames (like "foo.txt bar.txt etc.txt") where DOS (COMMAND.COM) hands external programs a pattern like *.TXT.

Under Unix shells, if a pattern doesn't match any filenames the parameter is usually left on the command like literally. Under bash this is a user-settable option. In fact, under bash you can disable shell "globbing" if you like -- there's a simple option to do this. It's almost never used -- because commands like mv, and cp won't work properly if their arguments are passed to them in this manner.

However here's a way to accomplish a similar result:

for i in *.txt; do cp $i ${i%.txt}.bak; done

... obviously this is more typing. If you tried to create a shell function or alias for it -- you have to figure out how to pass this parameters. Certainly the following seems simple enough:

function cp-pattern { for i in $1; do cp $i ${i%$1}$2; done

... but that doesn't work like most Unix users would expect. You'd have to pass this command a pair of specially chosen, and quoted arguments like:

cp-pattern '*.txt' .bak

... note how the second pattern has no wild cards and how the first is quoted to prevent any shell globbing. That's fine for something you might just use yourself -- if you remember to quote it right. It's easy enough to add check for the number of arguments and to ensure that there is at least one file that exists in the $1 pattern. However it becomes much harder to make this command reasonably safe and robust. Inevitably it becomes less "unix-like" and thus more difficult to use with other Unix tools.

I generally just take a whole different approach. Rather than trying to use cp to make a backup of each file under a slightly changed name I might just make a directory (usually using the date and my login ID as a template) and use a simple cp command to copy all my target files into the new directory.

Another interesting thing we can do with these "parameter expansion" features is to iterate over a list of components in a single variable.

For example, you might want to do something to traverse over every directory listed in your path -- perhaps to verify that everything listed therein is really a directory and is accessible to you.

Here's a command that will echo each directory named on your path on it's own line:

p=$PATH until [ $p = $d ]; do d=${p%%:*}; p=${p#*:}; echo $d; done

... obviously you can replace the echo $d part of this command with anything you like.

Another case might be where you'd want to traverse a list of directories that were all part of a path. Here's a command pair that echos each directory from the root down to the "current working directory":

p=$(pwd) until [ $p = $d ]; do p=${p#*/}; d=${p%%/*}; echo $d; done

... here we've reversed the assignments to p and d so that we skip the root directory itself -- which must be "special cased" since it appears to be a "null" entry if we do it the other way. The same problem would have occurred in the previous example -- if the value assigned to $PATH had started with a ":" character.

Of course, its important to realize that this is not the only, or necessarily the best method to parse a line or value into separate fields. Here's an example that uses the old IFS variable (the "inter-field separator in the Bourne, and Korn shells as well as bash) to parse each line of /etc/passwd and extract just two fields:

		cat /etc/passwd | ( \
			IFS=: ; while read lognam pw id gp fname home sh; \
				do echo $home \"$fname\"; done \
				)
			

Here we see the parentheses used to isolate the contents in a subshell -- such that the assignment to IFS doesn't affect our current shell. Setting the IFS to a "colon" tells the shell to treat that character as the separater between "words" -- instead of the usual "whitespace" that's assigned to it. For this particular function it's very important that IFS consist solely of that character -- usually it is set to "space," "tab," and "newline.

After that we see a typical while read loop -- where we read values from each line of input (from /etc/passwd into seven variables per line. This allows us to use any of these fields that we need from within the loop. Here we are just using the echo command -- as we have in the other examples.

My point here has been to show how we can do quite a bit of string parsing and manipulation directly within bash -- which will allow our shell scripts to run faster with less overhead and may be easier than some of the more complex sorts of pipes and command substitutions one might have to employ to pass data to the various external commands and return the results.

Many people might ask: Why not simply do it all in perl? I won't dignify that with a response. Part of the beauty of Unix is that each user has many options about how they choose to program something. Well written scripts and programs interoperate regardless of what particular scripting or programming facility was used to create them. Issue the command file /usr/bin/* on your system and and you may be surprised at how many Bourne and C shell scripts there are in there

In conclusion I'll just provide a sampler of some other bash parameter expansions:

${parameter:-word}

Provide a default if parameter is unset or null.
Example:

echo ${1:-"default"}

Note: this would have to be used from within a functions or shell script -- the point is to show that some of the parameter substitutions can be use with shell numbered arguments. In this case the string "default" would be returned if the function or script was called with no $1 (or if all of the arguments had been shifted out of existence. ${parameter:=word}

Assign a value to parameter if it was previously unset or null.

Example:

echo ${HOME:="/home/.nohome"}

${parameter:?word}

Generate an error if parameter is unset or null by printing word to stdout.

Example:

: ${HOME:="/home/.nohome"}

${TMP:?"Error: Must have a valid Temp Variable Set"}

This one just uses the shell "null command" (the : command) to evaluate the expression. If the variable doesn't exist or has a null value -- this will print the string to the standard error file handle and exit the script with a return code of one.

Oddly enough -- while it is easy to redirect the standard error of processes under bash -- there doesn't seem to be an easy portable way to explicitly generate message or redirect output to stderr. The best method I've come up with is to use the /proc/ filesystem (process table) like so:

function error { echo "$*" > /proc/self/fd/2 }

... self is always a set of entries that refers to the current process -- and self/fd/ is a directory full of the currently open file descriptors. Under Unix and DOS every process is given the following pre-opened file descriptors: stdin, stdout, and stderr.

${parameter:+word}

Alternative value. ${TMP:+"/mnt/tmp"}
use /mnt/tmp instead of $TMP but do nothing if TMP was unset. This is a weird one that I can't ever see myself using. But it is a logical complement to the ${var:-value} we saw above.

${#variable}

Return the length of the variable in characters.
Example:

echo The length of your PATH is ${#PATH}

Bash supports a surprising number of string manipulation operations. Unfortunately, these tools lack a unified focus. Some are a subset of parameter substitution, and others fall under the functionality of the UNIX expr command. This results in inconsistent command syntax and overlap of functionality, not to mention confusion.

#!/bin/bash
# substring-extraction.sh

String=23skidoo1
#      012345678    Bash
#      123456789    awk
# Note different string indexing system:
# Bash numbers first character of string as '0'.
# Awk  numbers first character of string as '1'.

echo ${String:2:4} # position 3 (0-1-2), 4 characters long
                                         # skid

# The awk equivalent of ${string:pos:length} is substr(string,pos,length).
echo | awk '
{ print substr("'"${String}"'",3,4)      # skid
}
'
#  Piping an empty "echo" to awk gives it dummy input,
#+ and thus makes it unnecessary to supply a filename.

exit 0

Notes

Notes: Those pages are written by people for whom English is not a native language. Some amount of grammar and spelling errors should be expected. This is a Spartan WHYFF (We Help You For Free) site. It cannot replace the best teachers and the best books. The site contain some obsolete pages as it develops like a living tree... Some links on older pages are broken. Please try to use Google, Open directory, etc. to find a replacement link (see HOWTO search the WEB for details). We would appreciate if you can mail us a correct link. Search Amazon by keywords:     Open directory Research Index

News

David Korn Tells All

# More to the point, thanks to the way ksh works, you can do this:

# make an array, words, local to the current function
typeset -A words

# read a full line
read line

# split the line into words
echo "$line" | read -A words

# Now you can access the line either word-wise or string-wise - useful if you want to, say, check for a command as the Nth parameter,
# but also keep formatting of the other parameters...

Miscellaneous commands

Parsing command line options

The getopt module defines a set of procedures to be used to process command line arguments with the following format:

-a

brief option a with no value;
 

-a123

brief option a with value 123;
 

--bianco

long option bianco with no value;
 

--color=bianco

long option color with value bianco.

Requires the message module (Message for details).

• Arguments:
• Using the module:
• Predefined options:
• Interface functions:
• Querying Options:

Arguments

The module contains, at the root level, a block of code like the following:
 

ARGC=0
declare -a ARGV ARGV1

for ((ARGC1=0; $# > 0; ++ARGC1)); do
    ARGV1[$ARGC1]="$1"
    shift
done

this block is executed when the script is evaluated. Its purpose is to store command line arguments in the global array ARGV1 and the number of command line arguments in the global variable ARGC1.

The global array ARGV and the global variable ARGC are predefined and should be used by the mbfl_getopts functions to store non-option command line arguments.

Example:
 

$ script --gulp wo --gasp=123 wa

if the script makes use of the library, the strings wo and wa will go into ARGV and ARGC will be set to 2. The option arguments are processed and some action is performed to register them.

We can access the non-option arguments with the following code:
 

for ((i=0; $i < $ARGC; ++i)); do
    # do something with ${ARGV[$i]}
done

Using the module

To use this module we have to declare a set of script options; we declare a new script option with the function mbfl_declare_option. Options declaration should be done at the beginning of the script, before doing anything; for example: right after the MBFL library code.

In the main block of the script: options are parsed by invoking mbfl_getopts_parse: this function will update a global variable and invoke a script function for each option on the command line.

Examples

Example of option declaration:
 

mbfl_declare_option ALPHA no a alpha noarg "enable alpha option"

this code declares an option with no argument and properties:

• global variable script_option_ALPHA, which will be set to no by default and to yes if the option is used;
• brief flag -a;
• long flag --alpha;
• description enable alpha option, to be shown in the usage output.

If the option is used: the function script_option_update_alpha is invoked (if it exists) with no arguments, after the variable script_option_ALPHA has been set to yes. Valid option usages are:
 

$ script.sh -a
$ script.sh --alpha

Another example:
 

mbfl_declare_option BETA 123 b beta witharg "select beta value"

this code declares an option with argument and properties:

• global variable script_option_BETA, which will be set to 123 by default and to the value selected on the command line if the option is used;
• brief flag -b;
• long flag --beta;
• description select beta value, to be shown in the usage output.

If the option is used: the function script_option_update_beta is invoked (if it exists) with no arguments, after the variable script_option_BETA has been set to the selected value. Valid option usages are:
 

$ script.sh -b456
$ script.sh --beta=456

Predefined options

A set of predefined options is recognised by the library and not handed to the user defined functions.

--encoded-args

Signals to the library that the non-option arguments and the option values are encoded in hexadecimal strings. Encoding is useful to avoid quoting problems when invoking a script from another one.

If this option is used: the values are decoded by mbfl_getopts_parse before storing them in the ARGV array and before being stored in the option's specific global variables.
 

-v

--verbose

Turns on verbose messages. The fuction mbfl_option_verbose returns true (Message, for details).
 

--silent

Turns off verbose messages. The fuction mbfl_option_verbose returns false.
 

--verbose-program

If used the --verbose option is added to the command line of external programs that support it. The fuction mbfl_option_verbose_program returns true or false depending on the state of this option.
 

--show-program

Prints the command line of executed external programs.
 

--debug

Turns on debugging messages (Message, for details).
 

--test

Turns on test execution (Program Testing, for details).
 

--null

Signals to the script that it has to use the null character to separate values, instead of the common newline. The global variable mbfl_option_NULL is set to yes.
 

-f

--force

Signals to the script that it does not have to query the user before doing dangerous operations, like overwriting files. The global variable mbfl_option_INTERACTIVE is set to no.
 

-i

--interactive

Signals to the script that it does have to query the user before doing dangerous operations, like overwriting files. The global variable mbfl_option_INTERACTIVE is set to yes.
 

--validate-programs

Validates the existence of all the programs needed by the script; then exits. The exit code is zero if all the programs were found, one otherwise.
 

--version

Prints to the standard output of the script the contents of the global variable mbfl_message_VERSION, then exits with code zero. The variable makes use of the service variables (Service Variables, for details).
 

--version-only

Prints to the standard output of the script the contents of the global variable script_VERSION, then exits with code zero.
 

--license

Prints to the standard output of the script the contents of one of the global variables mbfl_message_LICENSE_*, then exits with code zero. The variable makes use of the service variables (Service Variables, for details).
 

-h

--help

--usage

Prints to the standard output of the script: the contents of the global variable script_USAGE; a newline; the string options:; a newline; an automatically generated string describing the options declared with mbfl_declare_option; a string describing the MBFL default options. Then exits with code zero.

The following options may be used to set, unset and query the state of the predefined options.

Interface functions

Querying Options

Some feature and behaviour of the library is configured by the return value of the following set of functions. All of these functions are defined by the Getopts module, but they can be redefined by the script.

Printing messages to the console

This module allows one to print messages on an output channel. Various forms of message are supported.

All the function names are prefixed with mbfl_message_. All the messages will have the forms:
 

<progname>: <message>
<progname>: [error|warning]: <message>

The following global variables are declared:

mbfl_message_PROGNAME

must be initialised with the name of the script that'll be displayed at the beginning of each message;
 

mbfl_message_VERBOSE

yes if verbose messages should be displayed, else no;

Using external programs

This module declares a set of global variables all prefixed with mbfl_program_. We have to look at the module's code to see which one are declared.

• Program Testing: Testing a script and running programs.
• Program Checking: Checking programs existence.
• Program Executing: Executing a program.
• Program Declaring: Declaring the intention to use a program.

Testing a script and running programs

MBFL allows a script to execute a "dry run", that is: do not perform any operation on the system, just print messages describing what will happen if the script is executed with the selected options. This implies, in the MBFL model, that no external program is executed.

When this feature is turned on: mbfl_program_exec does not execute the program, instead it prints the command line on standard error and returns true.

Checking programs existence

The simpler way to test the availability of a program is to look for it just before it is used. The following function should be used at the beginning of a function that makes use of external programs.

type -ap program

Executing a program

Declaring the intention to use a program

To make a script model simpler, we assume that the unavailability of a program at the time of its execution is a fatal error. So if we need to execute a program and the executable is not there, the script must be aborted on the spot.

Functions are available to test the availability of a program, so we can try to locate an alternative or terminate the process under the script control. On a system where executables may vanish from one moment to another, no matter how we test a program existence, there's always the possibility that the program is not "there" when we invoke it.

If we just use mbfl_program_exec to invoke an external program, the function will try and fail if the executable is unavailable: the return code will be false.

The vanishing of a program is a rare event: if it's there when we look for it, probably it will be there also a few moments later when we invoke it. For this reason, MBFL proposes a set of functions with which we can declare the intention of a script to use a set of programs; a command line option is predefined to let the user test the availability of all the declared programs before invoking the script.

mbfl_program_validate_declared || mbfl_exit_program_not_found

Catching signals

MBFL provides an interface to the trap builtin that allows the execution of more than one function when a signal is received; this may sound useless, but that is it.

Manipulating strings

• String Quote: Quoted characters.
• String Inspection: Inspecting a string.
• String Splitting: Splitting a string.
• String Case: Converting between upper and lower case.
• String Class: Matching a string with a class.
• String Misc: Miscellaneous functions.

Quoted characters

Inspecting a string

Splitting a string

Example of usage for mbfl_string_chars:
 

string="abcde\nfghilm"
mbfl_string_chars "${string}"
# Now:
# "${#string}" = $SPLITCOUNT
#  a = "${SPLITFIELD[0]}"
#  b = "${SPLITFIELD[1]}"
#  c = "${SPLITFIELD[2]}"
#  d = "${SPLITFIELD[3]}"
#  e = "${SPLITFIELD[4]}"
#  \n = "${SPLITFIELD[5]}"
#  f = "${SPLITFIELD[6]}"
#  g = "${SPLITFIELD[7]}"
#  h = "${SPLITFIELD[8]}"
#  i = "${SPLITFIELD[9]}"
#  l = "${SPLITFIELD[10]}"
#  m = "${SPLITFIELD[11]}"

Converting between upper and lower case

Matching a string with a class

$'char'

Miscellaneous functions

Interacting with the user

Manipulating variables

• Variables Arrays:
• Variables Colon:

Manipulating arrays