Softpanorama

May the source be with you, but remember the KISS principle ;-)
Contents Bulletin Scripting in shell and Perl Network troubleshooting History Humor

KSH Substitutions

News

String Operations in Shell Recommended Links Length Substring   Substitution Operators

Pattern Matching

${foo:-bar}
(default)
${foo:=bar}   # ## / and // % %%
Bash Built-in Variables Annotated List of Bash Enhancements     Shell scripts collections   Humor Etc

For in depth coverage of shell string manipulation capabilities see String Operations in Shell

The curly-bracket syntax allows for the shell's string operators. String operators in shell use unique among programming language curly-bracket syntax In particular, string operators let you do the following:

In shell any variable can be displayed as ${foo} without changing its meaning. This notation was extended to allow expressions inside curvy brackets., for example ${foo=moo}. Each operation is encoded using special symbol or two symbols ("digram", for example :-, :=, etc) . An argument that the operator may need is posited after the symbol of the operation.

The first group of string-handling operators tests for the existence of variables and allows substitutions of default values under certain conditions. These are listed in

Note: The colon (:) in each of these operators is actually optional. If the colon is omitted, then change "exists and isn't null" to "exists" in each definition, i.e., the operator tests for existence only.

Any variable can be displayed as ${foo} without changing its meaning. This functionality often is used to protect a variable name from surrounding characters.

$ export foo=foo
$ echo ${foo}bar # foo exists so this works as expected
foobar
$ echo $foobar # foobar doesn't exist, so this doesn't

By the end of this article, you'll be able to use it for a whole lot more.

Three kinds of variable substitution are available for use: pattern matching, substitution and command substitution. I talk about the first two variables here and leave command substitution for another time.

Note:  Although the # and % operators may not seem obvious, they have a convenient mnemonic. The # key is on the left side of the $ key and operates from the left. The % key is on the right of the $ key and operates from the right. (This is true, at least, for US qwerty keyboards.)

Length Operator

One is ${#varname} returns the length of the value of the variable as a character string.

var="Hello world"
echo ${#var} 

Substr

It's useful to be able to take strings apart and put them together in different ways. Here is how to select a substring from a string:

string="this is a test"
substring=${string:1:9}            
              

In this example, the variable "substring" contains the word "substring". Remember this rule:


substring=${string_variable_name:starting_position:length}

The string starting position is zero-based.

Substitution Operators (Default value assignments)

Substitution operators mainly allow to check if the variable has null value of is undefined and return default value with or without assignment of this value to the variable. Here are major types fo substitution operators:

Operator Substitution
${varname:-word} If varname  exists and isn't null, return its value; otherwise return word.${count:-0}  evaluates to 0 if count  is undefined.
${varname:=word} If varname  exists and isn't null, return its value; otherwise set it to word  and then return its value.${count:=0}  sets count  to 0 if it is undefined.
${varname:?message} If varname  exists and isn't null, return its value; otherwise print varname:  followed by message, and abort the current command or script. Omitting message  produces the default message parameter null or not set.{count:?" undefined!" }  prints "count: undefined!"  and exits if count  is undefined.
${varname:+word} If varname  exists and isn't null, return word; otherwise return null.${count:+1}  returns 1 (which could mean "true") if count  is defined.

Note:  The ":" in the above digramms can be omitted. Such operator simply  tests for the existence of the variable.

Operator: ${foo:-bar}

Function: If $foo exists and is not null, return $foo. If it doesn't exist or is null, return bar.

Example:

$ export foo=""
$ echo ${foo:-one}
one
$ echo $foo

More complex example:

sort -nr $1 | head -${2:-10}

A typical usage include situations when you need to check if arguments were passed to the script and if not assign some default values::

#!/bin/bash 
export FROM=${1:-"~root/.profile"}
export TO=${2:-"~my/.profile"}
cp -p $FROM $TO

Operator: ${foo:=bar}

Function: If $foo exists and is not null, return $foo. If it doesn't exist or is null, set $foo to bar and return bar.

Example:

$ export foo=""
$ echo ${foo:=one}
one

$ echo $foo
one

Operator: ${foo:+bar}

Function: If $foo exists and is not null, return bar. If it doesn't exist or is null, return a null.

Example:

$ export foo="this is a test"
$ echo ${foo:+bar}
bar

Operator: ${foo:?"error message"}

Function: If $foo exists and isn't null, return its value. If it doesn't exist or is null, print the error message. If no error message is given, it prints parameter null or not set. In a non-interactive shell, this aborts the current script. In an interactive shell, this simply prints the error message.

filename=${1:?"filename missing."}

Example:

$ export foo="one"
$ for i in foo bar baz; do
> eval echo \${$i:?}
> done
one
bash: bar: parameter null or not set
bash: baz: parameter null or not set

 This, in turn, causes the creation of a variable, for example:

$ export foo="this is a test"
$ echo $bar

$ echo ${foo=bar}
this is a test
$ echo ${bar=bar}
bar
$ echo $bar
bar

Pattern Matching

Pattern-matching operators were introduced in ksh88 in a very idiosyncratic way. The notation is different from used by  Perl or utilities such as grep. That's a shame, but that's how it it. Life is not perfect. They are hard to remember, but there is a handy mnemonic tip: #  matches the front because number signs precede  numbers; %  matches the rear because percent signs follow  numbers.

Operator Meaning
${variable#pattern} If the pattern matches the beginning of the variable's value, delete the shortest part that matches and return the rest.
${variable##pattern} If the pattern matches the beginning of the variable's value, delete the longest part that matches and return the rest.
${variable%pattern} If the pattern matches the end of the variable's value, delete the shortest part that matches and return the rest.
${variable%%pattern} If the pattern matches the end of the variable's value, delete the longest part that matches and return the rest.

The classic use for pattern-matching operators is stripping off components of pathnames, such as directory prefixes and filename suffixes. With that in mind, here is an example that shows how all of the operators work. Assume that the variable path  has the value /home /billr/mem/long.file.name; then:

Expression Result
${path##/*/}                       long.file.name
${path#/*/}              billr/mem/long.file.name
$path              /home/billr/mem/long.file.name
${path%.*}         /home/billr/mem/long.file
${path%%.*}        /home/billr/mem/long

 

Operator #:  ${foo#t*is}  deletes the shortest possible match from the left

Example:

$ export foo="this is a test"
$ echo ${foo#t*is}
is a test

Operator ##: ${foo##t*is} deletes the longest possible match from the left

Example:

$ export foo="this is a test"
$ echo ${foo##t*is}
a test

Operator %: ${foo%t*st} Function: deletes the shortest possible match from the right

Example:

$ export foo="this is a test"
$ echo ${foo%t*st}
this is a
for i in *.htm*; do
if [ -f ${i%l} ]; then
echo "${i%l} already exists"
else
mv $i ${i%l}
fi
done

Operator %%: ${foo%%t*st} deletes the longest possible match from the right

Example:

$ export foo="this is a test"
$ echo ${foo%%t*st}

Shell Regular expressions

Think of regular expressions as strings that match patterns more powerfully than the standard shell wildcard schema. Regular expressions began as an idea in theoretical computer science, but they have found their way into many nooks and crannies of everyday, practical computing. The syntax used to represent them may vary, but the concepts are very much the same.

A shell regular expression can contain regular characters, standard wildcard characters, and additional operators that are more powerful than wildcards. Each such operator has the form x(exp), where x  is the particular operator and exp  is any regular expression (often simply a regular string). The operator determines how many occurrences of exp  a string that matches the pattern can contain.

Operator Meaning
*(exp) 0 or more occurrences of exp
+(exp) 1 or more occurrences of exp
?(exp) 0 or 1 occurrences of exp
@(exp1|exp2|...) exp1  or exp2  or...
!(exp) Anything that doesn't match exp 
Expression Matches
x x
*(x) Null string, x, xx, xxx, ...
+(x) x, xx, xxx, ...
?(x) Null string, x
!(x) Any string except x
@(x) x  (see below)

The following section compares Korn shell regular expressions to analogous features in awk  and egrep. If you aren't familiar with these, skip to the section entitled "Pattern-matching Operators."

shell basic regex vs awk/egrep regular expressions

Shell egrep/awk Meaning
*(exp) exp* 0 or more occurrences of exp
+(exp) exp+ 1 or more occurrences of exp
?(exp) exp? 0 or 1 occurrences of exp
@(exp1|exp2|...) exp1|exp2|... exp1  or exp2  or...
!(exp) (none) Anything that doesn't match exp

These equivalents are close but not quite exact. Actually, an exp  within any of the Korn shell operators can be a series of exp1|exp2|... alternates. But because the shell would interpret an expression like dave|fred|bob  as a pipeline of commands, you must use @(dave|fred|bob)  for alternates

For example:

It is worth re-emphasizing that shell regular expressions can still contain standard shell wildcards. Thus, the shell wildcard ?  (match any single character) is the equivalent to .  in egrep  or awk, and the shell's character set operator [...]  is the same as in those utilities.  For example, the expression +([0-9])  matches a number, i.e., one or more digits. The shell wildcard character * is equivalent to the shell regular expression * (?).

A few egrep  and awk  regexp operators do not have equivalents in the Korn shell. These include:

The first two pairs are hardly necessary, since the Korn shell doesn't normally operate on text files and does parse strings into words itself.



Etc

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes.   If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner. 

ABUSE: IPs or network segments from which we detect a stream of probes might be blocked for no less then 90 days. Multiple types of probes increase this period.  

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Haterís Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least


Copyright © 1996-2016 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: September 12, 2017