Softpanorama

May the source be with you, but remember the KISS principle ;-)
Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and  bastardization of classic Unix

KSH Substitutions

News

String Operations in Shell Recommended Links Length Substring   Substitution Operators

Pattern Matching

${foo:-bar}
(default)
${foo:=bar}   # ## / and // % %%
Bash Built-in Variables Annotated List of Bash Enhancements     Shell scripts collections   Humor Etc

For in depth coverage of shell string manipulation capabilities see String Operations in Shell

The curly-bracket syntax allows for the shell's string operators. String operators in shell use unique among programming language curly-bracket syntax In particular, string operators let you do the following:

In shell any variable can be displayed as ${foo} without changing its meaning. This notation was extended to allow expressions inside curvy brackets., for example ${foo=moo}. Each operation is encoded using special symbol or two symbols ("digram", for example :-, :=, etc) . An argument that the operator may need is posited after the symbol of the operation.

The first group of string-handling operators tests for the existence of variables and allows substitutions of default values under certain conditions. These are listed in

Note: The colon (:) in each of these operators is actually optional. If the colon is omitted, then change "exists and isn't null" to "exists" in each definition, i.e., the operator tests for existence only.

Any variable can be displayed as ${foo} without changing its meaning. This functionality often is used to protect a variable name from surrounding characters.

$ export foo=foo 
$ echo ${foo}bar # foo exists so this works as expected
foobar
$ echo $foobar # foobar doesn't exist, so this doesn't

Three kinds of variable substitution are available for use: pattern matching, substitution and command substitution. I talk about the first two variables here and leave command substitution for another time.

Note:  Although the # and % operators may not seem obvious, they have a convenient mnemonic. The # key is on the left side of the $ key and operates from the left. The % key is on the right of the $ key and operates from the right. (This is true, at least, for US qwerty keyboards.)

Length Operator

One is ${#varname} returns the length of the value of the variable as a character string.

var="Hello world"
echo ${#var} 

Substr

It's useful to be able to take strings apart and put them together in different ways. Here is how to select a substring from a string:

string="this is a test"
substring=${string:1:9}              

In this example, the variable "substring" contains the word "substring". Remember this rule:


substring=${string_variable_name:starting_position:length}

The string starting position is zero-based.

Substitution Operators (Default value assignments)

Substitution operators mainly allow to check if the variable has null value of is undefined and return default value with or without assignment of this value to the variable. Here are major types fo substitution operators:

Operator Substitution
${varname:-word} If varname  exists and isn't null, return its value; otherwise return word.${count:-0}  evaluates to 0 if count  is undefined.
${varname:=word} If varname  exists and isn't null, return its value; otherwise set it to word  and then return its value.${count:=0}  sets count  to 0 if it is undefined.
${varname:?message} If varname  exists and isn't null, return its value; otherwise print varname:  followed by message, and abort the current command or script. Omitting message  produces the default message parameter null or not set.{count:?" undefined!" }  prints "count: undefined!"  and exits if count  is undefined.
${varname:+word} If varname  exists and isn't null, return word; otherwise return null.${count:+1}  returns 1 (which could mean "true") if count  is defined.

NOTE:  The ":" in the above digrams can be omitted. Such operator simply  tests for the existence of the variable.

Operator: ${foo:-bar}

Function: If $foo exists and is not null, return $foo. If it doesn't exist or is null, return bar.

Example:

$ export foo=""
$ echo ${foo:-one}
one
$ echo $foo

More complex example:

sort -nr $1 | head -${2:-10}

A typical usage include situations when you need to check if arguments were passed to the script and if not assign some default values::

#!/bin/bash 
export FROM=${1:-"~root/.profile"}
export TO=${2:-"~my/.profile"}
cp -p $FROM $TO

Operator: ${foo:=bar}

Function: If $foo exists and is not null, return $foo. If it doesn't exist or is null, set $foo to bar and return bar.

Example:

$ export foo=""
$ echo ${foo:=one}
one

 

Operator: ${foo:+bar}

Function: If $foo exists and is not null, return bar. If it doesn't exist or is null, return a null.

Example:

$ export foo="this is a test"
$ echo ${foo:+bar}
bar

Operator: ${foo:?"error message"}

Function: If $foo exists and isn't null, return its value. If it doesn't exist or is null, print the error message. If no error message is given, it prints parameter null or not set. In a non-interactive shell, this aborts the current script. In an interactive shell, this simply prints the error message.

filename=${1:?"filename missing."}

Example:

$ export foo="one"
$ for i in foo bar baz; do
> eval echo \${$i:?}
> done
one
bash: bar: parameter null or not set
bash: baz: parameter null or not set

 This, in turn, causes the creation of a variable, for example:

$ export foo="this is a test"
$ echo $bar

$ echo ${foo=bar}
this is a test
$ echo ${bar=bar}
bar
$ echo $bar
bar

Pattern Matching

Pattern-matching operators were introduced in ksh88 in a very idiosyncratic way. The notation is different from used by  Perl or utilities such as grep. That's a shame, but that's how it it. Life is not perfect. They are hard to remember, but there is a handy mnemonic tip: #  matches the front because number signs precede  numbers; %  matches the rear because percent signs follow  numbers.

Operator Meaning
${variable#pattern} If the pattern matches the beginning of the variable's value, delete the shortest part that matches and return the rest.
${variable##pattern} If the pattern matches the beginning of the variable's value, delete the longest part that matches and return the rest.
${variable%pattern} If the pattern matches the end of the variable's value, delete the shortest part that matches and return the rest.
${variable%%pattern} If the pattern matches the end of the variable's value, delete the longest part that matches and return the rest.

The classic use for pattern-matching operators is stripping off components of pathnames, such as directory prefixes and filename suffixes. With that in mind, here is an example that shows how all of the operators work. Assume that the variable path  has the value /home /billr/mem/long.file.name; then:

Expression Result
${path##/*/}                       long.file.name
${path#/*/}              billr/mem/long.file.name
$path              /home/billr/mem/long.file.name
${path%.*}         /home/billr/mem/long.file
${path%%.*}        /home/billr/mem/long

 

Operator #:  ${foo#t*is}  deletes the shortest possible match from the left

Example:

$ export foo="this is a test"
$ echo ${foo#t*is}
is a test

Operator ##: ${foo##t*is} deletes the longest possible match from the left

Example:

$ export foo="this is a test"
$ echo ${foo##t*is}
a test

Operator %: ${foo%t*st} Function: deletes the shortest possible match from the right

Example:

$ export foo="this is a test"
$ echo ${foo%t*st}
this is a
for i in *.htm*; do
if [ -f ${i%l} ]; then
echo "${i%l} already exists"
else
mv $i ${i%l}
fi
done

Operator %%: ${foo%%t*st} deletes the longest possible match from the right

Example:

$ export foo="this is a test"
$ echo ${foo%%t*st}

Shell Regular expressions

Think of regular expressions as strings that match patterns more powerfully than the standard shell wildcard schema. Regular expressions began as an idea in theoretical computer science, but they have found their way into many nooks and crannies of everyday, practical computing. The syntax used to represent them may vary, but the concepts are very much the same.

A shell regular expression can contain regular characters, standard wildcard characters, and additional operators that are more powerful than wildcards. Each such operator has the form x(exp), where x  is the particular operator and exp  is any regular expression (often simply a regular string). The operator determines how many occurrences of exp  a string that matches the pattern can contain.

Operator Meaning
*(exp) 0 or more occurrences of exp
+(exp) 1 or more occurrences of exp
?(exp) 0 or 1 occurrences of exp
@(exp1|exp2|...) exp1  or exp2  or...
!(exp) Anything that doesn't match exp 
Expression Matches
x x
*(x) Null string, x, xx, xxx, ...
+(x) x, xx, xxx, ...
?(x) Null string, x
!(x) Any string except x
@(x) x  (see below)

The following section compares Korn shell regular expressions to analogous features in awk  and egrep. If you aren't familiar with these, skip to the section entitled "Pattern-matching Operators."

shell basic regex vs awk/egrep regular expressions

Shell egrep/awk Meaning
*(exp) exp* 0 or more occurrences of exp
+(exp) exp+ 1 or more occurrences of exp
?(exp) exp? 0 or 1 occurrences of exp
@(exp1|exp2|...) exp1|exp2|... exp1  or exp2  or...
!(exp) (none) Anything that doesn't match exp

These equivalents are close but not quite exact. Actually, an exp  within any of the Korn shell operators can be a series of exp1|exp2|... alternates. But because the shell would interpret an expression like dave|fred|bob  as a pipeline of commands, you must use @(dave|fred|bob)  for alternates

For example:

It is worth re-emphasizing that shell regular expressions can still contain standard shell wildcards. Thus, the shell wildcard ?  (match any single character) is the equivalent to .  in egrep  or awk, and the shell's character set operator [...]  is the same as in those utilities.  For example, the expression +([0-9])  matches a number, i.e., one or more digits. The shell wildcard character * is equivalent to the shell regular expression * (?).

A few egrep  and awk  regexp operators do not have equivalents in the Korn shell. These include:

The first two pairs are hardly necessary, since the Korn shell doesn't normally operate on text files and does parse strings into words itself.


Top Visited
Switchboard
Latest
Past week
Past month

NEWS CONTENTS

Old News ;-)

Recommended Links

Google matched content

Softpanorama Recommended

Top articles

Sites



Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D


Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to to buy a cup of coffee for authors of this site

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: September 04, 2019