Softpanorama
(slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-)

Google   


Shell Tips and Tricks

News

Recommended Links AWK one liners Bash tips Command history reuse Filesystem navigation

Command completion

 Shell Prompts Dotfiles Bash Built-in Variables bash Control Structures Vi mode Pushd, popd and dirs BASH Debugging
Debugging Shell debugging Functions in Shell Annotated List of Bash Enhancements  I/O Redirection Strange Files Deletion Shell scripts collections
Shell Aliases Arithmetic expressions Shell scripts collections Pipes in Loops Sequences of commands Humor Etc

Nobody really knows what the Bourne shell's grammar is. Even examination of the source code is little help.
Tom Duff
Most shell scripts are quick 'n dirty solutions to non-complex problems. As such, optimizing them for speed is not much of an issue. Consider the case, though, where a script carries out an important task, does it well, but runs too slowly. The simplest fix would be to rewrite all or the the parts of the script that slow it down in Perl.  Check the loops in the script. Time consumed by repetitive operations adds up quickly. Use the time and times tools to profile computation-intensive commands.

Bash is not particularly efficient at handling files, so consider using more appropriate tools for this within the script, such as awk or Perl.

Try to write your scripts in a structured, coherent form, so they can be reorganized and tightened up as necessary. Some of the optimization techniques applicable to high-level languages may work for scripts, but others, such as loop unrolling, are mostly irrelevant. Above all, use common sense.

The problem with bash is that it pretty baroque and there are just too many features that you need to remember. this command, that command and so on till infinity. As it is difficult to remember details of each bash has an on-line help feature that provides basic information about most of its built-in commands. To see the help description for a particular command, enter

help command

(for example, help alias) at the bash UNIX prompt. To see a list of bash commands for which help is available, type help at the bash UNIX prompt. You may access the manual page for bash by entering man bash at a UNIX prompt, but beware, it is 60 pages long and not very readable.

Please visit  Heiner Steven SHELLdorado  the best shell scripting site on the Internet

With bash 3.x, you can reissue commands like in C-shell using arrow keys. 

If you set emacs mode you can also use CTRL-p and CTRL-n to scroll through commands in your command history and use regular emacs commands to search for strings and edit a command line. For example, CTRL-d deletes a character and CTRL-k erases from the cursor to the end of the line.

Bash also supports "file name completion" which is pretty neat feature that can save some typing.

Like any decent shell Bash also allows you to define aliases and you should avoid retying the same command twice not only by browsing the history but defining aliases from history.  You can define functions that are more powerful tool the aliases and can simplify repetitive jobs. You can have separate dot files with functions and aliases for example an .aliases file, and  .bashrc

Old News ;-)

bash Cookbook Reader - Contributions browse

The dirs command output isn't that readable with many long pathnames. To make it more readable,

you could just use the -p option on dirs:

 
alias dirs="dirs -p"

 

[Apr 4, 2008] bash Cookbook Reader - Contributions browse

the last argument
The arguments to a script (or function) are $1, $2, ... and can be referred to as a group by $* (or $@ ). But is there an easy way to refer to the last argument in list ? Try ${!#} as in:
 
    echo ${!#}
    LAST=${!#}

Bash, version 3

Bash, version 3

Interactive and non-interactive scripts

Alternatively, the script can test for the presence of i in the $- flag.

   1 case $- in
   2 *i*)    # interactive script
   3 ;;
   4 *)      # non-interactive script
   5 ;;
   6 # (Thanks to "UNIX F.A.Q.", 1993)
 

  Scripts may be forced to run in interactive mode with the i option or with a #!/bin/bash -i header. Be aware that this may cause erratic script behavior or show error messages where no error is present.

Assorted Tips

This is just a idea of writing log clumsy worded...

[Jun 25, 2007] IFS variable and Field splitting in the Korn shell

IFS Specifies internal field separators (normally space, tab, and new line) used to separate command words that result from command or parameter substitution and for separating words with the regular built-in command read. The first character of the IFS parameter is used to separate arguments for the $* substitution.

... ... ...

[May 7, 2007] basename and dirname

basename strips off the path leaving only the final component of the name, which is assumed to be the file name. If you specify suffix and the remaining portion of name contains a suffix which matches suffix, basename removes that suffix.  For example

basename src/dos/printf.c .c

produces

printf
dirname returns the directory part of the full path+name combination.

Also can be done directly in bash

basename=${file##*/}

dirname=${file%/*}

 

[May 7, 2007] To strip file extensions in bash, like this.rbl --> this

  fname=${file%.rbl}

More 2 Cent Tips & Tricks LG #37

Date: Tue, 12 Jan 1999 19:18:15 +0200
From: Reuben Sumner, rasumner@iname.com

Here is a two cent tip that I have been meaning to submit for a long long time now.

If you have a large stack of CD-ROMS, finding where a particular file lies can be a time consuming task. My solution uses the locate program and associated utilities to build up a database of the CDs' contents that allows for rapid searching.

First we need to create the database, the following script does the trick nicely.

 
   #!/bin/bash
   onedisk()
   {
      mount /mnt/cdrom
      find /mnt/cdrom -maxdepth 7 -print | sed "s;^/mnt/cdrom;$1;" > $1.find
      eject -u cdrom
   }
   echo Enter name of disk in device:
   read diskname
   while [ -n "$diskname" ]; do
      onedisk $diskname
      echo Enter name of next disk or Enter if done:
      read diskname
   done
   echo OK, preparing cds.db
   cat *.find | sort -f | /usr/lib/findutils/frcode > cds.db
   echo Done...
Start with no CD mounted. Run the script. It will ask for a label for the CD, a short name like "sunsite1" is best. It will then quickly scan the CD, eject it and prompt for another. When you have exhausted your collection just hit enter at the prompt. A file called cds.db will be done. To make it simple to use copy cds.db to /var/lib (or anywhere else, that is where locatedb is on my system). Now create an alias like
 
       alias cdlocate="locate -d /var/lib/cds.db"
Now if I type "cdlocate lyx" I get
 
debian20_contrib/debian/hamm/contrib/binary-i386/text/lyx_0.12.0.final-0.1.deb
debian20_contrib/debian/hamm/contrib/binary-m68k/text/lyx_0.12.0.final-0.1.deb
debian20_contrib/debian/hamm/contrib/source/text/lyx_0.12.0.final-0.1.diff.gz
debian20_contrib/debian/hamm/contrib/source/text/lyx_0.12.0.final-0.1.dsc
debian20_contrib/debian/hamm/contrib/source/text/lyx_0.12.0.final.orig.tar.gz
lsa3/apps/wp/lyx-0.12.0-linux-elf-x86-libc5-bin.tar.gz
lsa3/apps/wp/lyx-0.12.0.lsm
lsa3/apps/wp/lyx-0.12.0.tar.gz
lsa4/docs/french/www.linux-france.com/lgazette/issue-28/gx/lyx
lsa4/powertools/i386/lyx-0.12.0-1.i386.rpm
lsa4/powertools/SRPMS/lyx-0.12.0-1.src.rpm
openlinux12/col/install/RPMS/lyx-0.11.32-1.i386.rpm
openlinux12/col/sources/SRPMS/lyx-0.11.32-1.src.rpm
suse53/suse/contents/lyx
In order to prevent locate from warning you that the database is old try touch -t 010100002020 /var/lib/cds.db to set the modification date to January 1 2020.

--
Reuben

Finding text strings in binary files

Ever wondered what's inside some of those binary files on your system (binary executables or binary data)? Several times I've gotten error messages from some command in the Solaris system, but I couldn't tell where the error was coming from because it was buried in some binary executable file.

The Solaris "strings" command lets you look at the ASCII text buried inside of executable files, and can often help you troubleshoot problems. For instance, one time I was seeing error messages like this when a user was trying to log in:

   
Could not set ULIMIT

I finally traced the problem down to the /bin/login command by running the "strings" command like this:

   
root> strings /bin/login | more

The strings command lists ASCII character sequences in binary files, and help me determine that the "Could not set ULIMIT" error was coming from this file. Once I determined that the error message I was seeing was coming from this file, solving the problem became a simple matter.

Use CDPATH to traverse filesystems faster

If you're like many Solaris users and administrators, you spend a lot of time moving back and forth between directories in similar locations. For instance, you might often work in your home directory (such as "/home/al"), the /usr/local directories, web page directories, or other user's home directories in /home.

If you're often moving back-and-forth between the same directories, and you use the Bourne shell (sh) or Korn shell (ksh) as your login shell, you can use the CDPATH shell variable to save yourself a lot of typing, and quickly move between directories.

Here's a quick demo. First move to the root directory:

    cd /

Next, if it's not set already, set your CDPATH shell variable as follows:

CDPATH=/usr/spool

Then, type this cd command:

    cd cron

What happens? Type this and see what happened:

    pwd

The result should be "/usr/spool/cron".

When you typed "cd cron", the shell looked in your local directory for a sub-directory named "cron". When it didn't find one, it searched the CDPATH variable, and looked for a "cron" sub-directory. When it found a sub-directory named cron in the /usr/spool directory, it moved you there.

You can set your CDPATH variable just like your normal PATH variable:

    CDPATH=/home/al:/usr/local:/usr/spool:/home

Group commands together with parentheses

Have you ever needed to run a series of commands, and pipe the output of all of those commands into yet another command?

For instance, what if you wanted to run the "sar", "date", "who", and "ps -ef" commands, and wanted to pipe the output of all three of those commands into the "more" command? If you tried this:

    sar -u 1 5; date; who; ps -ef | more

you'll quickly find that it won't work. Only the output of the "ps -ef" command gets piped through the "more" command, and the rest of the output scrolls off the screen.

Instead, group the commands together with a pair of parentheses (and throw in a few echo statements for readability) to get the output of all these commands to pipe into the more command:

(sar -u 1 5; echo; who; echo; ps -ef; echo; date; echo) | more

Use the "at" command to run jobs some other time

Many times it's necessary to schedule programs to run at a later time. For instance, if your computer system is very busy during the day, you may need
to run jobs late at night when nobody is logged on the system.

Solaris makes this very easy with the "at" command. You can use the "at" command to run a job at almost any time--later today, early tomorrow...whenever.

Suppose you want to run the program "my_2_hour_program" at ten o'clock tonight. Simply tell the at command to run the job at 10 p.m. (2200):

    /home/al> at 2200
    at> my_2_hour_program > /tmp/2hour.out
    at> <CTRL><D>
warning: commands will be executed using /bin/ksh
job 890193600.a at Tue Mar 17 22:00:00 1998

Or suppose you'd like to run a find command at five o'clock tomorrow morning:

/home/al> at 0500 tomorrow
at> find /home > /tmp/find.out
at> <CTRL><D>
warning: commands will be executed using /bin/ksh
job 890215200.a at Wed Mar 18 05:00:00 1998

When you're at the "at" prompt, just type the command you want to run. Try a few tests with the at command until you become comfortable with the way
it works.

Create a directory and move into it at the same time

Question: How often do you create a new directory and then move into that directory in your next command?  Answer: Almost always.

I realized this trend in my own work habits, so I created a simple shell function to do the hard work for me.

     md () {
          mkdir -p $1  &&  cd $1
     }

This is a Bourne shell function named "md" that works for Bourne and Korn shell users. It can be easily adapted for C shell users.

Taking advantage of the -p option of the mkdir command, the function easily creates multi-level subdirectories, and moves you into the lowest level of the directory structure.  You can use the command to create one subdirectory like this:

     /home/al> md docs
     /home/al/docs> _

or you can create an entire directory tree and move right into the new directory like this:

     /home/al> md docs/memos/internal/solaris8
     /home/al/docs/memos/internal/solaris8>

Searching Multiple CD-ROM

Date: Fri, 15 Jan 1999 19:55:51 +0100 (CET)
From: JL Hopital, cdti94@magic.fr

My English is terrible,so feel free to correct if you decide to publish...

Hello,i am a French linuxer and here is my two cent tips. If you have many CD-ROMs and want to retrieve this_file_I'm_sure_i_have_but_can't_remember_where, it can helps.

It consist of 2 small scripts using gnu utilities: updatedb and locate. Normally 'updatedb' run every night, creating a database for all the mounted file systems and 'locate' is used to query this system-wide database.But you can tell them where are the files to index and where to put the database.That's what my scripts does:

The first script (addcd.sh) create a database for the cd actually mounted.You must run it once for every cdrom.

The second ( cdlocate.sh ) search in the databases created by addcd.sh and display the cdname and full path of the files matching the pattern you give in parameter. So you can search for unmounted files !

To use:

Beware that locate's regular expressions have some peculiarities, 'man locate' will explain.

Hope this help and happy linuxing !

 
---Cut here------------------------------
# addcd.sh
# Author: Jose-Luc.Hopital@ac-creteil.fr
# Create a filename's database in $DATABASEHOME for the cd mounted 
# at $MOUNTPOINT
# Example usage: addcd.sh Linux.Toolkit.Disk3.Oct.1996
# to search the databases use cdlocate.sh
CDNAME=$1
test "$CDNAME" = "" && { echo Usage:$0 name_of_cdrom ; exit 1 ; }
# the mount point for the cd-ROM
MOUNTPOINT=/mnt/cdrom
# where to put the database
DATABASEHOME=/home/cdroms
updatedb --localpaths=$MOUNTPOINT --output=$DATABASEHOME/$CDNAME.updatedb && \
echo Database added for $CDNAME
---Cut here--------------------------------
# cdlocate.sh
# Author : Jose-Luc.Hopital@ac-creteil.fr
# Usage $0 pattern
# search regular expression in $1 in the database's found in $DATABASEHOME
# to add a database for a new cd-rom , use addcd.sh
test "$*" = "" && { echo Usage:$0 pattern ; exit 1 ; }
DATABASEHOME=/home/cdroms
cd $DATABASEHOME
# get ride of locate warning:more than 8 days old
touch *.updatedb
CDROMLIST=`ls *.updatedb`
for CDROM in $CDROMLIST
do
        CDROMNAME=`basename $CDROM .updatedb`
        locate --database=$DATABASEHOME/$CDROM $@ |sed 's/^/'$CDROMNAME:'/'
done

Recommended Links

bash Cookbook Reader - Contributions browse

Etc

String expansion

 
   1 #!/bin/bash
   2 
   3 # String expansion.
   4 # Introduced in version 2 of bash.
   5 
   6 # Strings of the form $'xxx'
   7 # have the standard escaped characters interpreted. 
   8 
   9 echo $'Ringing bell 3 times \a \a \a'
  10 echo $'Three form feeds \f \f \f'
  11 echo $'10 newlines \n\n\n\n\n\n\n\n\n\n'
  12 
  13 exit 
 

Indirect variable references - the new way

   1 #!/bin/bash
   2 
   3 # Indirect variable referencing.
   4 # This has a few of the attributes of references in C++.
   5 
   6 
   7 a=letter_of_alphabet
   8 letter_of_alphabet=z
   9 
  10 # Direct reference.
  11 echo "a = $a"
  12 
  13 # Indirect reference.
  14 echo "Now a = ${!a}"
  15 # The ${!variable} notation is greatly superior to the old "eval var1=\$$var2"
  16 
  17 echo
  18 
  19 t=table_cell_3
  20 table_cell_3=24
  21 echo "t = ${!t}"
  22 table_cell_3=387
  23 echo "Value of t changed to ${!t}"
  24 # Useful for referencing members
  25 # of an array or table,
  26 # or for simulating a multi-dimensional array.
  27 # An indexing option would have been nice (sigh).
  28 
  29 
  30 exit 0

Using arrays and other miscellaneous trickery to deal four random hands from a deck of cards

   1 #!/bin/bash2
   2 # Must specify version 2 of bash, else might not work.
   3 
   4 # Cards:
   5 # deals four random hands from a deck of cards.
   6 
   7 UNPICKED=0
   8 PICKED=1
   9 
  10 DUPE_CARD=99
  11 
  12 LOWER_LIMIT=0
  13 UPPER_LIMIT=51
  14 CARDS_IN_SUITE=13
  15 CARDS=52
  16 
  17 declare -a Deck
  18 declare -a Suites
  19 declare -a Cards
  20 # It would have been easier and more intuitive
  21 # with a single, 3-dimensional array. Maybe 
  22 # a future version of bash will support
  23 # multidimensional arrays.
  24 
  25 
  26 initialize_Deck ()
  27 {
  28 i=$LOWER_LIMIT
  29 until [ $i -gt $UPPER_LIMIT ]
  30 do
  31   Deck[i]=$UNPICKED
  32   let "i += 1"
  33 done
  34 # Set each card of "Deck" as unpicked.
  35 echo
  36 }
  37 
  38 initialize_Suites ()
  39 {
  40 Suites[0]=C #Clubs
  41 Suites[1]=D #Diamonds
  42 Suites[2]=H #Hearts
  43 Suites[3]=S #Spades
  44 }
  45 
  46 initialize_Cards ()
  47 {
  48 Cards=(2 3 4 5 6 7 8 9 10 J Q K A)
  49 # Alternate method of initializing array.
  50 }
  51 
  52 pick_a_card ()
  53 {
  54 card_number=$RANDOM
  55 let "card_number %= $CARDS"
  56 if [ ${Deck[card_number]} -eq $UNPICKED ]
  57 then
  58   Deck[card_number]=$PICKED
  59   return $card_number
  60 else  
  61   return $DUPE_CARD
  62 fi
  63 }
  64 
  65 parse_card ()
  66 {
  67 number=$1
  68 let "suite_number = number / CARDS_IN_SUITE"
  69 suite=${Suites[suite_number]}
  70 echo -n "$suite-"
  71 let "card_no = number % CARDS_IN_SUITE"
  72 Card=${Cards[card_no]}
  73 printf %-4s $Card
  74 # Print cards in neat columns.
  75 }
  76 
  77 seed_random ()
  78 {
  79 # Seed random number generator.
  80 seed=`eval date +%s`
  81 let "seed %= 32766"
  82 RANDOM=$seed
  83 }
  84 
  85 deal_cards ()
  86 {
  87 echo
  88 
  89 cards_picked=0
  90 while [ $cards_picked -le $UPPER_LIMIT ]
  91 do
  92   pick_a_card
  93   t=$?
  94 
  95   if [ $t -ne $DUPE_CARD ]
  96   then
  97     parse_card $t
  98 
  99     u=$cards_picked+1
 100     # Change back to 1-based indexing (temporarily).
 101     let "u %= $CARDS_IN_SUITE"
 102     if [ $u -eq 0 ]
 103     then
 104      echo
 105      echo
 106     fi
 107     # Separate hands.
 108 
 109     let "cards_picked += 1"
 110   fi  
 111 done  
 112 
 113 echo
 114 
 115 return 0
 116 }
 117 
 118 
 119 # Structured programming:
 120 # entire program logic modularized in functions.
 121 
 122 #================
 123 seed_random
 124 initialize_Deck
 125 initialize_Suites
 126 initialize_Cards
 127 deal_cards
 128 
 129 exit 0
 130 #================
 131 
 132 
 133 
 134 # Exercise 1:
 135 # Add comments to thoroughly document this script.
 136 
 137 # Exercise 2:
 138 # Revise the script to print out each hand sorted in suites.
 139 # You may add other bells and whistles if you like.
 140 
 141 # Exercise 3:
 142 # Simplify

 

 


Copyright © 1996-2007 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. Submit comments This document is an industrial compilation designed and created exclusively for educational use and is placed under the copyright of the Open Content License(OPL). Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

Standard disclaimer: The statements, views and opinions presented on this web page are those of the author and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: April 04, 2008