Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-) Softpanorama Search

Shellorama

Ksh93 and Bash Shells for Unix System Administrators

This collection of links is oriented on students (initially it was provided as a reference material to my shell programming university course) and is designed to emphasize usage of  advanced shell constructs and pipes in shell programming (mainly in the context of ksh93 and bash 3.2+ which have good support for those constructs).  An introductory paper  Slightly Skeptical View on Shell discusses the shell as a scripting language and as one of the earliest examples of very high level languages.  The page might also be useful for system administrators who constitute the considerable percentage of  shell users and lion part of shell programmers.

This page is the main page to a set of sub-pages devoted to shell that collectively are known as Shellorama. The most important are:

• Shell Papers,E-books and Tutorials

• Language page - describes some of the exotic shell contracts and provides links to web resources about them.  Bash 3.x added several useful extensions. Among them (Bash Reference Manual ):

  • Arithmetic expansion allows the evaluation of an arithmetic expression and the substitution of the result. The format for arithmetic expansion is:

         $(( expression ))

     Older (( ... )) construct borrowed from ksh93 is also supported.
     

  • Process substitution. It takes the form of  <(list) or   >(list). See Process substitution

  • =~ operator. An additional binary operator, ‘=~’, is available, with the same precedence as ‘==’ and ‘!=’. When it is used, the string to the right of the operator is considered an extended regular expression and matched accordingly (as in regex3)). The return value is 0 if the string matches the pattern, and 1 otherwise. If the regular expression is syntactically incorrect, the conditional expression's return value is 2. If the shell option nocasematch (see the description of shopt in Bash Builtins) is enabled, the match is performed without regard to the case of alphabetic characters. Substrings matched by parenthesized sub-expressions within the regular expression are saved in the array variable BASH_REMATCH. The element of BASH_REMATCH with index 0 is the portion of the string matching the entire regular expression. The element of BASH_REMATCH with index n is the portion of the string matching the nth parenthesized sub-expression. 
     

  • C-style for loop. Bash implements the for (( expr1 ; expr2 ; expr3 )) loop, similar to the C language (see Looping Constructs).
     

  • Brace expansion -- a mechanism for generation of a sequence of similar strings (see also Filename Expansion), but the file names generated need not exist. Patterns to be brace expanded take the form of an optional preamble, followed by either a series of comma-separated strings or a sequnce expression between a pair of braces, followed by an optional postscript. Brace expansions may be nested.  For example,

         bash$ echo a{d,c,b}e
         ade ace abe

  • Tilde notation (csh an tcsh feature). Content of DIRSTACK array can be referenced positionally:
    • ~N  -- The string that would be displayed by ‘dirs +N’
    • ~+N -- The string that would be displayed by ‘dirs +N’
    •  ~-N  -- The string that would be displayed by ‘dirs -N’

    which is rather convenient for implementing "directory favorites" concept with "push/pop/dirs" troika
     

• Debugging Current bash has the best debugger and from this point of view represents the best shell.  Until  Actually for ksh93 absence of the debugger is more a weakness, it is a blunder and it is strange that such talented person David Korn did not realize this.

• Command history reuse which is devoted to one of the most important command line features
• Dotfiles
• Command completion
• pushd/pops/dirs troika
• Advanced Unix filesystem navigation
• History Substitution
• Functions in Shell
• BASH Debugging
• Vi editing mode
• Bash Built-in Variables
• Arithmetic Expressions in BASH
• bash Control Structures
• Shell Input and Output Redirection
• Readline and inputrc

I strongly recommend getting a so-called orthodox file manager (OFM). This tool can immensely simplify Unix filesystem navigation and file operations (Midnight Commander  while defective in handling command line can be tried first as this is an active project and it provides fpt and sftp virtual filesystem in remote hosts)

Actually filesystem navigation in shell is an area of great concern as there are several serious problems with the current tools for Unix filesystem navigation.  I would say that usage of cd command (the most common method) is conceptually broken and deprives people from the full understanding of Unix filesystem; I doubt that it can be fixed within the shell paradigm (C-shell made an attempt to compensate for this deficiency by introducing history and popd/pushd/dirs troika, but this proved to be neither necessary nor sufficient for compensating problems with the in-depth understanding of the classical Unix hierarchical filesystem  inherent in purely command line navigation ;-). Paradoxically sysadmins who use OFMs usually have much better understanding of the power and flexibility of the Unix filesystem then people who use command line.   All-in-all usage of OFM is system administration represents Eastern European school of administration and it might be a better way to administer system that a typical "North American Way".

The second indispensable tool for shell programmer is Expect. This is a very flexible application that can be used for automation of interactive sessions as well as automation of testing of applications.

Usually people who know shell and awk and/or Perl well are usually considered to be advanced Unix system administrators (this is another way to say the system administrators who does not know shall/awk/Perl troika well are essentially a various flavors of entry-level system administrators no matter how many years of experience they have). I would argue that no system administrator can consider himself to be a senior Unix system administrator without in-depth knowledge of both one of the OFMs and Expect.

An OFM tends to educate the user about the Unix filesystem in some subtle, but definitely  psychologically superior way. Widespread use of OFMs in Europe, especially in Germany and Eastern Europe, tend to produce specialists with substantially greater skills at handling Unix (and Windows) file systems than users that only have experience with a more primitive command line based navigational tools.

And yes, cd navigation is conceptually broken. This is not a bizarre opinion of the author,  this is a fact:  when you do not even suspect that a particular part of the tree exists something is conceptually broken.  People using command line know only fragments of the file system structure like blinds know only the parts of the elephant.  Current Unix file system with, say,  13K directories for a regular Solaris installation, are just unsuitable for the "cd way of navigation"; 1K directories was probably OK. But when there are over 10K of directories you need something else. Here quantity turns into quality. That's my point.

The page provides rather long quotes as web pages as web pages are notoriously unreliable medium and can disappear without trace.  That makes this page somewhat difficult to browse, but it's not designed for browsing; it's designed as a supplementary material to the university shell course and for self-education.

A complementary page with Best Shell Books Reviews is also available. Although the best book selection is to a certain extent individual, the selection of a bad book is not: so this page might at least help you to avoid most common bad books (often the book recommended by a particular university are either weak or boring or both;  Unix Shell by Example is one such example ;-). Still the shell literature is substantial (over a hundred of books) and that mean that you can find a suitable textbook. Please be aware of the fact that that few authors of shell programming books have a broad understanding of Unix necessary for writing a comprehensive shell book. 

IMHO the first edition of O'Reilly Learning Korn Shell is probably one of the best and contains nice set of  examples (the second edition is more up to date but generally is weaker). Also the first edition has advantage of being available in HTML form too (O'Reilly Unix CD). It does not cover ksh93 but it presents ksh in a unique way that no other book does. Some useful examples can also be found in UNIX Power Tools Book( see Archive of all shell scripts (684 KB); the book is available in HTML from one of O'Reilly CD bookshelf collections).

That means that outside of interactive usage and small one page scripts they generally outlived their usefulness. That's why for more or less complex tasks Perl is usually used (and should be used) instead of shells. While shells continued to improve since the original C-shell and Korn shell, the shell syntax is frozen in space and time and now looks completely archaic.  There are a large number of problems with this syntax as it does not cleanly separate lexical analysis from syntax analysis.  Bash 3.2 actually made some progress of overcoming most archaic features of old shells but still it has it own share of warts (for example last stage of the pipe does not run in on the same level as encompassing the pipe script)

Some syntax features in shell are idiosyncratic as Steve Bourne played with Algol 68 before starting work on the shell. In a way, he proved to be the most influential bad language designer, the designer who has the most lasting influence on Unix environment (that does not exonerate the subsequent designers which probably can take a more aggressive stance on the elimination of initial shell design blunders by marking them as "legacy").

For example there is very little logic in how  different types of blocks are delimitated in shell scripts. Conditional statements end with (broken) classic Algor-68 the reverse keyword syntax: 'if condition; then echo yes; else echo no; fi', but loops are structured like perverted version of PL/1 (loop prefix do; ... done;) , individual case branches blocks ends with  ';;' . Functions have C-style bracketing "{", "}".  M. D. McIlroy as Steve Borne manager should be ashamed. After all at this time the level of compiler construction knowledge was pretty sufficient to avoid such blunders (David Gries book was published in 1971) and Bell Labs staff were not a bunch of enthusiasts ;-).

rm $file

Variable assignments in Bourne shell are whitespace sensitive. 'foo=bar' is an assignment, but 'foo = bar' is not. It is a function call with "= "and "bar" as two arguments. This is another strange idiosyncrasy.

There is also an overlap between aliases and functions. Aliases are positional macros that are recognized only as the first word of the command like in classic  alias ll='ls -l'.  Because of this, aliases have several limitations:

• You can only redirect input/output to the last command in the alias.
• You can only specify arguments to the last command in the alias.
• Alias definitions are a single text string, this means complex functions are nearly impossible to create.

ll() { ls -l $*; }

Since the use of variables as commands is allowed, it is impossible to reliably check the syntax of a script as substitution can accidentally result in key word as in example that I found in the paper about fish (not that I like or recommend fish):

if true; then if [ $RANDOM -lt 1024 ]; then END=fi; else END=true; fi; $END

Dr. Nikolai Bezroukov

Old News ;-)

[Nov 26, 2009] Bash 4.0

Interesting featured are -l -u options in declare statement (automatic conversion to lower case/upper case).

n.  The -p option to `declare' now displays all variable values and attributes
    (or function values and attributes if used with -f).

o.  There is a new `compopt' builtin that allows completion functions to modify
    completion options for existing completions or the completion currently
    being executed.

p.  The `read' builtin has a new -i option which inserts text into the reply
    buffer when using readline.

s.  Changed format of internal help documentation for all builtins to roughly
    follow man page format.

t.  The `help' builtin now has a new -d option, to display a short description,
    and a -m option, to print help information in a man page-like format.

u.  There is a new `mapfile' builtin to populate an array with lines from a
    given file.  The name `readarray' is a synonym.

w.  There is a new shell option: `globstar'.  When enabled, the globbing code
    treats `**' specially -- it matches all directories (and files within
    them, when appropriate) recursively.

x.  There is a new shell option: `dirspell'.  When enabled, the filename
    completion code performs spelling correction on directory names during
    completion.

dd. The parser now understands `|&' as a synonym for `2>&1 |', which redirects
    the standard error for a command through a pipe.

ee. The new `;&' case statement action list terminator causes execution to
    continue with the action associated with the next pattern in the
    statement rather than terminating the command.


hh. There are new case-modifying word expansions: uppercase (^[^]) and
    lowercase (,[,]).  They can work on either the first character or
    array element, or globally.  They accept an optional shell pattern
    that determines which characters to modify.  There is an optionally-
    configured feature to include capitalization operators.

ii. The shell provides associative array variables, with the appropriate
    support to create, delete, assign values to, and expand them.

jj. The `declare' builtin now has new -l (convert value to lowercase upon
    assignment) and -u (convert value to uppercase upon assignment) options.
    There is an optionally-configurable -c option to capitalize a value at
    assignment.

kk. There is a new `coproc' reserved word that specifies a coprocess: an
    asynchronous command run with two pipes connected to the creating shell.
    Coprocs can be named.  The input and output file descriptors and the
    PID of the coprocess are available to the calling shell in variables
    with coproc-specific names.

[Apr 22, 2009] Advanced Bash Scripting Guide freshmeat.net

Changes: Fairly extensive coverage of the version 4.0 Bash release. A great deal of other new material and bugfixes. This is a very important update.

[Apr 22, 2009] Nanolx.org » bashstyle-ng 7.7

BashStyle-NG is a graphical tool for changing Bash's behavior and look and feel. It can also style Readline, Nano, and Vim.

It ships with a set of scripts, which are used by the styles shipped with BS-NG, but can also be used separately. Since v6.3 you have the opportunity to create your own prompts. For important notes on how to do so, refer back to the documentation.

If you don’t understand an option or something does not work as expected read the documentation. It’s installed in /usr/share/doc/bashstyle-ng/index.html ( /usr is default but may vary if you passed an other prefix to configure).

Currently BS-NG ships 15 pre-defined Styles for your prompt. Most of them can be modified via the Custom Prompt Builder. If you want to save your current configuration and want to re-import it later (or for using it on a different user/machine) you can do so via the Profiler (bs-ng-profiler –help).

Standalone (GConf-Free) Configuration (for faster Bash-Startup) can be created via the RCGenerator (rcgenerator –help).

[Apr 7, 2009] Mastering Unix Shell Scripting: Bash, Bourne, and Korn Shell Scripting for Programmers, System Administrators, and UNIX Gurus (Paperback)

This is one of the few book with some AIX bias. For example,  Chapter 11 is about AIX Logical Volume Manager. That author also wrote a great book on AIX administration

by Randal K. Michael (Author)

• Read Excerpt- Table of Contents (PDF)
• Read Excerpt- Chapter (PDF)  Scripting Quick Start and Review
• Mastering Unix Shell Scripting Google books
• Paperback: 1032 pages
• Publisher: Wiley; 2 edition (June 3, 2008)
• Language: English
• ISBN-10: 0470183012
• ISBN-13: 978-0470183014
• Product Dimensions: 9.2 x 7.4 x 2.3 inches

A must-have for all levels of *nix users. , August 1, 2004 By Bindlestiff (rixtertech.com) - See all my reviews This review is from: Mastering UNIX Shell Scripting (Paperback) The breadth of real-world examples make the difference between this book and most reference texts. It's true that it's written for korn, but I've had little trouble adapting for Bash; many of the scripts run almost unchanged and the ones that don't provide a useful opportunity for exercise in adaptation. The authors prose is clear. His attitude is a bit challenging; he says early on that that his intention is to teach you how to -solve problems- by shell scripting, NOT to present a ream of canned solutions. This is NOT a reference text for any particular shell, you'll still need plenty of O'Reilly books, a web browser & etc. This book has enabled me to write a major project using scripting as the glue to hold together a hefty mass of file-moving daemons, fax/paging engines, python UI code, PostGreSQL database engine, networking/email, SSH, and Expect scripts on a Gnu Linux platform. I absolutely could not have done it without this book and I'm very grateful to Mr Michael for his work. If a later edition could more closely serve the needs of the masses by presenting more Bash examples and maybe throwing in a CD it would be a 5-star text.

[Oct 27, 2008] Bash Debugger 4.0-0.1

About: The Bash Debugger (bashdb) is a debugger for Bash scripts. The debugger command interface is modeled on the gdb command interface. Front-ends supporting bashdb include GNU-Emacs and ddd. In the past, the project has been used as a springboard for other experimental features such as a timestamped history file (now in Bash versions after 3.0).

Changes: This major rewrite and reorganization of code has numerous bugfixes and has been tested on bash 3.1 and 4.0 alpha. With the introduction of a simple debugger command alias mechanism, there are some incompatibilities. Command aliasing of short commands is no longer hard-wired. New commands of note are "set autoeval" and "step+", taken from ruby-debug. Emacs support has been greatly improved. Long option command-processing is guaranteed.

[Oct 27, 2008] Linux.com Teach an old shell new tricks with BashDiff

BashDiff is a patch for the bash shell that can do an amazing number of things. It extends existing bash features, brings a few of awk's tricks into the shell itself, exposes some common C functions to bash shell programming, adds an exception mechanism, provides features of functional programming such as list comprehension and the map function, lets you talk with GTK+2 and databases, and even adds a Web server right into the standard bash shell.

There are no packages of BashDiff in the openSUSE, Fedora, or Ubuntu repositories. I'll build from source using BashDiff 1.45 on an x86 Fedora 9 machine running bash 3.0. While versions 3.1 and 3.2 of bash are available, the 1.45 BashDiff patch does not apply cleanly to either.

[Jul 21, 2008] Project details for Advanced Bash Scripting Guide 5.3

freshmeat.net

Recreational scripts were added, such as an almost full-featured Perquacky clone script, a script that does the "Petals Around the Rose" puzzle, and a crossword puzzle solver script. There is also the usual batch of bugfixes and other new material.

[Apr 3, 2008] LinuxDevCenter.com -- Excerpt from Linux Cookbook, Part 1

[Mar 31, 2008]  Bash Shell Programming in Linux by P. Lutus

Simple but nice intro with good examples

[Mar 30, 2008] Linux, Unix, -etc Unix Shell Scripts

These are simple stand-alone scripts. You are unlikely to find anything very impressive here if you already hack your own scripts, but a novice might find some of the ideas new.

• Handy "Desktop" Thingies
• Dates and Times
• Text Utilities
• File Utilities
• Simple Databases
• Administration
• Networking
• Special Stuff for Unix Haters
• E-mail
• Miscellaneous Handy Things

continue

Recommended Links

---

In case of broken links please try to use Google search. If you find the page please notify us about new location

---

Please visit  Heiner Steven SHELLdorado ; the best shell scripting site on the Internet

***** SHELLdorado -- An excellent site by Heiner Steven. Very cute name that catches the fact that the shell is a crown jewel of Unix ;-) IMHO this is the best shell-related site on Internet. Actively maintained. Highly recommended ! Note: This site now has a bulletin that you can subscribe to [Feb 02, 2002]

***** The official Korn Shell Web Site by David Korn, the author of ksh88 and ksh93. See also his son's Home Page  (actually I think that Tksh implementation is the best free shell available, but ksh93 is better supported).

***+ Open Directory - Computers Operating Systems Unix Shell -- a decent collection of links.  Could be better...

**** Unix shell scripts by Paul Dunne. contains links and several useful scripts.

scripting -- this is a bellatlantic.net member site and can be down

***+ kshweb.html--  a good site by Dana French

*** dotfiles.com home -- several dot files of very uneven (often low) quality. Still better than nothing...

Other Links

• pdksh - Public Domain Korn Shell -- this is still ksh88 clone
• Yahoo! UnixShells -- theoretically this page should be actively maintained as yahoo is a FreeBSD shop and should eat it's own dog food, but this is not the case -- Yahoo's speed of deterioration was proportional to its stock rise in 1997-2000 (and did not improved after the Jan 2001 fall :-)
• Shell Programming Language by Author: Weifeng Shi Not maintained since 1996

---

MAN pages

KornShell 93 Manual Page

docs.sun.com man pages section 1 User Commands - ksh

Man pages from Neosoft

• sh
• ksh
• bash

Bash(1) manual page

pdksh man page

---

FAQs

comp.unix.shell -- the only Usenet group about shells

Note: see first  Unix FAQshell Index -- it's probably more up-to-date than this document

• [gnu.bash.bug] BASH Frequently-Asked Questions -- a must reading for BASH users
• UNIX shell differences and how to change your shell (Monthly Posting)
• Z-shell Frequently-Asked Questions -- Z-shell seems to be the latest attempt to create a decent Unix shell
• UNIX FAQ
  • Unix - Frequently Asked Questions (part 5) [Shells]  -- pretty limited
    • Introduction
    • Can shells be classified into categories?
    • How do I "include" one shell script from within another shell script?
    • Do all shells have aliases? Is there something else that can be used?
    • How are shell variables assigned?
    • How can I tell if I am running an interactive shell?
    • What "dot" files do the various shells use?
    • I would like to know more about the differences ... ?

---

References

See also additional material at

• Shell Command Line History Substitution --  a very important facility to learn !!!

• BASH Debugging

• Annotated List of Bash Enhancements

• Bash Tips and Tricks

• Bash Built-in Variables

• Brace Expansion

• Another interesting trick is the

• Common Unix Commands

• Quick Reference Cards

• 300+ IT Reference Cards, Guides and Manuals

• C++ , Programming Reference Cards and Books

• Bash Quick Reference -- The card, written by Arnold Robbins, includes a comprehensive summary of built-in commands, pre-defined variables, I/O, job control, command-line editing, and more. Ouitdated. Also at multiple other locations
  • bash reference
  • BASH REFERENCE
  • UNIX bash Shell Quick Reference
• Unix commands reference card
•  Unix KornShell memento -- good reference. See also Unix KornShell Quick Reference (may be this is an original)
• FOSSwire Unix-Linux Command Cheat Sheet
• Unix Reference Card
• unix01a simple but useful reference card from rice university
• VIM Quick Reference Card

C-shell and tcsh

Tcsh and C shell

• The C Shell tutorial
• TCSH Manual Page
• tcsh 6.06 manual
• The T in TCSH
• A Resource of TC Shell 6.04

C-shell based tutorials

• ***+ The UNIX programming environment  Edition 2.0, June 1997 by Mark Burgess contains a good introduction to UNIX and shell script programming
• **+Working in the UNIX Shell Environment (VTAIX Guide) -- introduction. Both C-shell and Korn shell are covered.
• The UNIX course the C-shell
• C Shell tutorial
• An Introduction to Shell Scripts --  A Short Course from Rice University's Information Systems (C-shell and Bourne shell)

Etc

A modular Perl shell written, configured, and operated entirely in Perl. It aspires to be a fully operational login shell with all the features one normally expects. But it also gives direct access to Perl objects and data structures from the command line, and allows you to run Perl code within the scope of your command line. And it's named after one of the greatest characters on Futurama, so it must be good...

RC shell

A very interesting shell with advanced piping support.

• Plan 9 -sys-man--1-rc -- man page
• rc - a shell
• Rc ­ The Plan 9 Shell a very interesting paper. Must read.

Lost Links

• UNIX Shell Programming Bourne and Korn Shells  by Steven J. Hayes Department of Computer Science RMIT University & Technisearch