Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-)

Chapter 7: Shell Giants

Introduction to the Unix shell history

The Bourne Shell Family

C-shell family (c-shell and tcsh)

Zsh and other "post-scripting languages" shells

Visual shells and Orthodox file managers

Shells History Webliography

Appendix 1:

Shell History by John Mashey

Appendix 2:

 An Introduction to the C shell by William Joy (revised for 4.3BSD by Mark Seiden)

Appendix 3:

An Introduction to the UNIX Shell by S. R. Bourne.

The book Life with Unix by Don Libes and Sandy Ressler is fascinating reading for anyone interested in Unix in general and scripting history in particular. It covers a lot of the history, interactions, etc. Much in the present section is summarized from this book. Some interesting info is in UNIX FAQ.

As most of the readers know Unix history goes back to 1969 and the famous "little-used PDP-7 in a corner" on which Ken Thompson, Dennis Ritchie (the R in K&R), and others started work on what was to become Unix.

But what is less known and appreciated is the fact that Ken Thompson was also the author of the first UNIX shell and that concepts of this shell greatly influenced the subsequent work of John Mashey and Stephen Bourne. Influenced to the extent that John Mashey actually objected to calling his shell Mashey shell considering it as a direct extension of Thompson shell.

The name "Unix" was intended as a pun on Multics (and was written "Unics" at first, for UNiplexed Information and Computing System). As Ch 9 of the Netizens Netbook  (   On the Early History and Impact of Unix Tools to Build the Tools for a New Millenium) aptly stated:

When AT&T made the decision to pull out of the Multics collaboration, they took the research operating system off their GE 645  computer and put up the GECOS operating system. Though GECOS was adequate  for applications, it was "nowhere near as satisfactory if you were trying  to do things that were technically difficult and imperfectly defined,"  explained Vyssotsky, "which is the main task of research."(23)

 For the pioneering work of Bell Labs research programmers like Ken Thompson and the research purposes of the Labs, an operating system more  like what Multics had promised was needed. Along with the advantages of  immediate feedback which time-sharing provided, the Bell Labs researchers  wanted to continue the ability to work collaboratively which time-sharing  had made possible.

"What we wanted to preserve," one of the creators of Unix, Dennis  Ritchie writes, "was not just a good programming environment in which to  do programming, but a system around which a fellowship could form. We knew from experience that the essence of communal computing, as supplied by remote-access, time-shared machines, is not just to type programs into a terminal instead of a keypunch, but to encourage close  communication."(24)

Ritchie describes how an informal group led by Thompson had begun  investigating alternatives to Multics before the GE-645 Multics machine  had been removed from the Labs.(25) Thompson and Ritchie presented Bell Labs with proposals to buy them a computer so they could build their own interactive, time-sharing operating system. Their proposals weren't acted on. Eventually, Thompson found a little used PDP-7 computer. According to Vyssotsky, the orphaned PDP-7 computer was a machine, "more in the class of a Commodore 64 than the class of a PC-AT."(26)

For the first 10 years, Unix development was essentially confined to Bell Labs and most scripting related work was also done in NJ. The initial versions of Unix were labeled "Version n" or "Nth Edition" (of the manuals) and some milestones in shell history are directly related to particular Unix releases.  Major early Unix implementation were for DEC's PDP-11 (16 bits) which was so tiny by today's hardware standards (typical configuration were limited to 128K memory, 2.4M disc, 64K per-process limit (inc the kernel)) and similar configurations can be found only in palmtop computers and top electronic watches ;-). The fact that they managed to created pretty powerful shells for such a computer is nothing but simply amazing and attests tremendous ingenuity of early creators of Unix extremely well.

• 1971 V1: 1st Unix version written in assembler on a PDP-11/20. Included file system, fork(), roff, ed. Was used as a text processing tool for preparation of patents.
   
• 1972 V3: Pipe() appeared first in V3!
   
• 1973 V4: Unix was rewritten in C.
   
• 1974 V5 The UNIX Time-Sharing System is published in CACM by Ken Thompson and Dennis Ritchie. It is a revision of the 1973 paper. Fifth edition was made freely available to universities for educational purposes. University of California at Berkeley (UCB) gets Version 4 of UNIX.  In January 1974, a Version 4 tape was delivered and Unix was installed in Berkeley by graduate student Keith Standiford.  Berkeley begins making major enhancements to UNIX and sets the stage for becoming a major distribution center for their version of UNIX. The Elements of Programming Style by Kernighan and Plauger is published.
   
• 1975
  • V6: First version of Unix widely available outside Bell Labs. Thompson begins one year sabbatical at Berkeley bringing V6 with him. AT&T officially begins licensing UNIX to universities. Due to its portability and flexibility, UNIX Version 6 (V6) became the minicomputer OS of choice for universities in 1975. In the fall of 1975 were two graduate students, Bill Joy and Chuck Haley start working on Unix. J. Lions published his famous "A commentary on the Unix Operating System" based on V6.
     
  • PWB Unix got Mashey shell -- the first powerful shell for Unix that was close to a scripting language.  As John Mashey recollected:

    The "PWB Shell" first appeared in mid-1975. It derived from a set of requirements and suggestions from me in early 1975 in trying to do serious shell programming. In mid-1975, the shell acquired variables, including 3 that were derived from per-process data. This is where the idea of more generalised path- searching came in. Some of the existing support commands were beefed up, including if/else/endif. Most of this work was done by Alan Glasser or Dick Haight. I ended up taking this over, doing performance analysis, moving most of the control structure commands into the shell, adding switch,, while, etc. etc.  By this time there was serious shell programming in massive amounts. This had mostly stabilised by early/mid 1976, although a few more features were added in early 1977. For various reasons this came to be called the "Mashey shell", a term I universally abhorred, since it was fundamentally still a Thompson shell with selected additions and tunings by various of us, constrained to be mostly upward compatible, leading to some things that I found less than esthetic. Even less than esthetic was the title of a later internal Bell Labs course: "Bourne Shell Programming for Mashey Shell Programmers". See [2] and [3].

    Bell Labs Piscataway used the PWB shell quite heavily for its scripting needs in the mid-1970s and that probably was the first time a scripting language was widely used outside research. Rephrasing John R. Mashey  quote "UNIX is a lever for the intellect"  we can say that "Scripting is a level for the intellect". thirty years later we are taking scripting for granted, but it was pretty amazing breakthrough made on a PDP 11 of those days with typical configuration 128K memory, 2.4M disc, 64K per-process limit (inc the kernel). Unix on PDP 11 was a tight squeeze, and utilities like "sh" had to be extremely tightly coded. Mashey shell was also pretty fast (faster then Bourne shell that replaced it later).
     
• 1976. John R. Mashey publishes the paper "Using a Command Language as a High-Level
  Programming Language", Proc. 2nd Int. Conf on Software Engineering, IEEE (Oct 13-15, 1976), 169-176. -- the first paper about scripting as a very high level programming language.
   
• 1977. The new scripting language awk is designed by Alfred V. Aho, Peter J. Weinberger, and Brian W. Kernighan. It was designed to be used with shell, but in essence it needed to be integrated into shell, which was done later in Perl and ksh93. 
• 1978
  • A Borne shell by Stephen Bourne, simply known as the 'sh', is released. It became the most popular Unix shell for the next two decades.
  • Awk was first described in Software Practice and Experience in July, 1978, and first distributed with Seventh Edition UNIX.  That was the next major step in scripting languages development in Unix after Mashey shell.
  • 1978 Bill Joy produces 2BSD, the "Berkeley Software Distribution", that contains C-shell (2BSD was a set of applications that ran on 6th Edition UNIX, and was not a kernel nor a set of kernel patches). It was implemented as a standalone interpreter. Csh was a big improvement over the original sh in many respects; it had more built-in commands, a different and friendlier syntax with better  flow control constructs and an arithmetic facility and most importantly of all command history.  But its primary contribution was a better command interface. Among key ideas here were the ideas of a history list and an in-line editing. Combined they produced a kind of pseudo-GUI environment that saved users a lot of typing. Naturally, csh took off quickly. A full distribution of the 2BSD tape is available at http://www.tuhs.org/Archive/PDP-11/Distributions/ucb/
  • Famous  Steve Bourne paper, "An Introduction to the UNIX Shell"  was published (BSTJ  57, No 6, Part 2 (Jul-Aug 1978), 2797-2822)
  • Steve Johnson complete first port of UNIX to an Interdata 8/32, the first non-DEC computer to run UNIX. Note that this is nearly ten years after running only on DEC equipment.  UNIX is ported to a DEC VAX, but not by Thompson and Ritchie, since they had become disenchanted by DEC and its unwillingness to support UNIX. DEC's initial refusal to support UNIX must be one of the all time great blunders of the computer industry.
  • The C Programming Language by Kernighan and Ritchie is published and served as the standard for the C language for over a decade, until the ANSI C standard appeared in December 1989.
     
• 1979
  • V7: It included full K&R C, uucp, Bourne shell and awk. V7 was ported to the VAX as 32V. The V7 kernel was a mere 40KB! Here (for reference) are the system calls of V7:

      _exit, access, acct, alarm, brk, chdir, chmod, chown,
      chroot, close, creat, dup, dup2, exec*, exit, fork, fstat,
      ftime, getegid, geteuid, getgid, getpid, getuid, gtty,
      indir, ioctl, kill, link, lock, lseek, mknod, mount,
      mpxcall, nice, open, pause, phys, pipe, pkoff, pkon,
      profil, ptrace, read, sbrk, setgid, setuid, signal, stat,
      stime, stty, sync, tell, time, times, umask, umount,
      unlink, utime, wait, write.

  • T. A. Dolotta and  John Mashey presented a talk and published a paper "Using a Command Language as the Primary Programming Tool" at IFIP Working Conference on Command Languages, Sept 10-14, 1979.
     
• 1981 C-shell was extended into tcsh, which for a long time became the most popular interactive shell. It introduced the idea of command completion.
   
• 1982. David Korn created ksh -- a dominant shell in commercial Unixes till this days.
   
• 1987. The first version of Perl was developed by Larry Wall  at  the end of the year. See excellent  PerlTimeline for more information. Perl was an attempt to combine the capabilities of the shell and awk into a single language.  Because Perl is freely available and performs better than combined shell and awk, Perl rapidly acquired a large user community, primarily at universities. Here is the description from Perl manpage:

  NAME
  perl | Practical Extraction and Report Language
  
  SYNOPSIS
      perl [options] filename args
  
  DESCRIPTION
       Perl is a interpreted language optimized for scanning  arbi
       trary  text  files,  extracting  information from those text
       files, and printing reports based on that information.  It's
       also  a good language for many system management tasks.  The
       language is intended to be practical  (easy  to  use,  effi-
       cient,  complete)  rather  than  beautiful  (tiny,  elegant,
       minimal).  It combines (in  the  author's  opinion,  anyway)
       some  of the best features of C, sed, awk, and sh, so people
       familiar with those languages should have little  difficulty
       with  it.  (Language historians will also note some vestiges
       of csh, Pascal, and  even  BASIC|PLUS.)   Expression  syntax
       corresponds  quite  closely  to C expression syntax.  If you
       have a problem that would ordinarily use sed or awk  or  sh,
       but  it exceeds their capabilities or must run a little fas-
       ter, and you don't want to write the silly thing in C,  then
       perl  may  be  for  you.  There are also translators to turn
       your sed and awk scripts  into  perl  scripts.   OK,  enough
       hype.

• 1988 John K. Ousterhout developed TCL, the first embeddable scripting language for Unix. 

  I began work on Tk in late 1988, but it was a part-time project, so it took about two years before there was enough functionality for Tk to be useful.
  ... By 1989 I was using Tcl in a couple of projects, and I believe that I gave a talk on Tcl during the Berkeley Industrial Liaison Conference, where several hundred people from industry came to Berkeley to hear about research in our department. I believe that this led to invitations to give talks on Tcl at a few interested companies, and that some of the people at those talks asked if they could have copies of Tcl. In any case, I gave away a few copies of Tcl to early adopters in 1989.
  
  In 1994 John Ousterhout joined Sun. Unfortunately Sun blowed away a chance to make TCL a standard Unix macro language the way REXX was in OS/2.

• 1989.
  • As part of the GNU project Chet Ramey developed the Bourne-Again Shell, bash. Initially was rather weak but later became a prominent shell due to Linux.
  • Z Shell was created by Paul Falstad in Princeton University, NJ were he was a student. This was the first of "super-shells" a shell designed for interactive use that doubles as a decent scripting language. Many of the useful features of bash, ksh, and tcsh were incorporated into zsh; many original features were added. Later David Korn created ksh93 with a similar agenda.
     
• 1990 
  • Python was created by Guido van Rossum at CWI in Amsterdam.
  • Don Libes of the National Institute of Standards and Technology created Expect.  He made Expect freely available after presenting his paper in the summer of 1990, and it became an instant hit among system administrators. Expect was not only the first widely distributed Tcl application, but for many years it was the most popular scripting application in Unix universe. Many of the early Tcl users learned about it by using Expect.
  • John Ousterhout presented a paper on Tcl at the USENIX Conference n January.
     
• 1991. Perl 4.000 is released and includes an artistic license as well as the GPL.
   
• 1993. David Korn released ksh93, an attempt to backport some Perl features into shell framework. Here is a partial list of ksh93 features:
  • Associative arrays - ksh88 already supports indexed arrays.
  • Floating point arithmetic + math library functions.
  • Arithmetic for command similar to C and awk.
  • Complete ANSI C printf formatting with extensions.
  • Run time linking of libraries and builtins.
  • A number of additional substring operations such as offset length.
  • Full extended regular expression matching capabilities
  • Compound variables which can be used to represent data aggregates.
  • Name references for passing aggregate variables by name.
  • Active variables. Users can trap variable assignment and references by associating intercept functions of the form name.get and name.set.
  • Ability to make socket connections to servers by name.
  • read with timeouts.
  • Conformance to POSIX 1003.2.
  • Command completion and variable completion, ksh93 only had file name completion.
  • A key binding mechanism that allows users to bind keys to new editing functions.
     
• 1994. Perl 5.000 is unveiled.

Last modified: February 28, 2008