Pipes and Coroutines in Shell

News	Introduction	Best Shell Books	Recommended Links	Papers, ebooks tutorials	Bourne Shell and portability	Man pages	Reference	Scripts Collections
Bash	ksh93	AWK	Sed	Restricted shell	C-shell and tcsh	zsh	TCL	Expect
Language	Debugging	Advanced filesystem navigation	Command completion	Pipes	Vi editing mode	Pushd and popd	Aliases	Functions
Regular Expressions	Shell Prompts	Shell Dotfiles	Command history reuse in shell	Tips	Monitoring scripts	Pretty Printing	Unix shells history	Humor	Etc

Korn shell and its derivates like ksh93 and zsh ( possesses an interesting and unique feature: the ability to pipe a file into a loop. This is effectively an implementation of simple coroutines although shell has also more general form (See Learning Korn Shell by Bill Rosenblat and Arnold Robbins). In bash this capability is limited as bash does not run the last stage of the pipe as the current process... The developers probably think they can reinvent the wheel but got a square.. Googling for "bash pipe subprocesses order" shows the extent of the problem, but I couldn't find what the bash developers' official stand

Let's assume that we need to find all files that contain string "19%" which is a typical for printing commands like "19%2d"

cd/ /usr/bin
ls | while read file
do
echo $file
string $file | grep '19%'
done

Here we use the ls command to generate the list of the file names and this list it piped into a loop. In a loop we echo command and then run strings piped to grep looking for suspicious format strings.

In another example from O'Reilly "Learning Korn Shell" (first edition). Here we will pipe awk output into the loop. This is a function that, given a pathname as argument, prints its equivalent in tilde notation if possible:

function tildize {
    if [[ $1 = $HOME* ]]; then
        print "\~/${1#$HOME}"
        return 0
    fi
    awk '{FS=":"; print $1, $6}' /etc/passwd | 
        while read user homedir; do
            if [[ $homedir != / && $1 = ${homedir}?(/*) ]]; then
                print "\~$user/${1#$homedir}"
                return 0
            fi
        done
    print "$1"
    return 1
}

Pipes can also output data to the loop

Loop can also serve as a source to input for the pipe. For example

{ while read line'?adc> '; do
      print "$(alg2rpn $line)"
  done 
} | dc

The use of pipes

When you use pipes to pass strings along, try to avoid putting the pipes deep down in the control structure. Remember that you are not restricted to putting a pipe after a simple command, but that you can put the pipe after a for-loop, a while-loop, or any other control structure. If you do this properly, you will not spawn more processes than necessary.

As an example; assume that you want to go through all C files of a directory and, if they are readable to you, convert the filenames to contain lowercase letters only (this example may be a little contrived). We can do it it in slightly different ways.

The first script calls tr inside the the for-loop:

#!/bin/sh
for x in *.c
do
  [ -r $x ] && echo $x | tr 'A-Z' 'a-z'
done

and the second script uses coroutine linage (pipe) to feed tr from the loop:

#!/bin/sh
for x in *.c
do
  [ -r $x ] && echo $x 
done | tr 'A-Z' 'a-z'

Old News ;-)

bash Cookbook Reader - Contributions browse

pipeline of commands
We discuss in the book (in the chapter on common mistakes) the fact that a pipeline of commands runs those commands in subshells. The result (or dilema) is that what happens in those subshells (e.g. counting something) is lost to the parent shell script unless the parent captures output from the pipeline, but that isn't always easy or desirable.

The bash man page describes a feature of bash called "Process Substitution" that lets you substitute the output of a pipeline of commands (actually a list of commands) using <(list) as the syntax.

But notice how the feature is described:

The process list is run with its input or output connected to a FIFO or some file in /dev/fd. The name of this file is passed as an argument to the current command as the result of the expansion.

The <(...) is going to be replaced with the name of a fifo. So if you wrote:
wc <(some commands)
the result would be:
wc fifo
that is, the fifo filename is passed to the command. That's fine for commands like wc that can accept a filename. But what about a builtin like while?

It turns out that you can add the redirect from the fifo, but the space between the two less-than signs is crucial to distinguish it from "<<", the "here document" syntax.

So you can write:
  while read a b c
  do
    ...
  done < <(pipeline of commands)

Internal Commands and Builtins

Piping output to a read, using echo to set variables will fail.
Yet, piping the output of cat seems to work.
cat file1 file2 | while read line do echo $line done
However, as Bjön Eriksson shows:
Example 14-8. Problems reading from a pipe
#!/bin/sh # readpipe.sh # This example contributed by Bjon Eriksson. last="(null)" cat $0 | while read line do echo "{$line}" last=$line done printf "\nAll done, last:$last\n" exit 0 # End of code. # (Partial) output of script follows. # The 'echo' supplies extra brackets. ############################################# ./readpipe.sh {#!/bin/sh} {last="(null)"} {cat $0 |} {while read line} {do} {echo "{$line}"} {last=$line} {done} {printf "nAll done, last:$lastn"} All done, last:(null) The variable (last) is set within the subshell but unset outside.
The gendiff script, usually found in /usr/bin on many Linux distros, pipes the output of find to a while read construct.
find $1 $ -name "*$2" -o -name ".*$2" $ -print | while read f; do . . .
It is possible to paste text into the input field of a read. See Example A-39.

ksh vs bash setting variable in piped loops are lost - Ubuntu Forums