Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-)

Unix Pipes -- powerful
and elegant programming paradigm

The concept of pipes is one of the most important Unix innovation (the other two are probably hierarchical filesystem and regular expressions) that had found its way to all other operating systems.

Let me state it again: the pipe is the the most elegant and powerful features of UNIX.  In The Creation of the UNIX Operating System /Connecting streams like a garden hose the authors wrote: 

Another innovation of UNIX was the development of pipes, which gave programmers the ability to string together a number of processes for a specific output.

Doug McIlroy, then a department head in the Computing Science Research Center, is credited for the concept of pipes at Bell Labs, and Thompson gets the credit for actually doing it.

McIlroy had been working on macros in the later 1950s, and was always theorizing to anyone who would listen about linking macros together to eliminate the need to make a series of discrete commands to obtain an end result.

"If you think about macros," McIlroy explained, "they mainly involve switching data streams. I mean, you're taking input and you suddenly come to a macro call, and that says, 'Stop taking input from here and go take it from there.'

"Somewhere at that time I talked of a macro as a 'switchyard for data streams,' and there's a paper hanging in Brian Kernighan's office, which he dredged up from somewhere, where I talked about screwing together streams like a garden hose. So this idea had been banging around in my head for a long time."

 ... ... ... ...
 

While Thompson and Ritchie were at the chalkboard sketching out a file system, McIlroy was at his own chalkboard trying to sketch out how to connect processes together and to work out a prefix notation language to do it.

It wasn't easy. "It's very easy to say 'cat into grep into...,' or 'who into cat into grep,'" McIlroy explained. "But there are all these side parameters that these commands have; they just don't have input and output arguments, but they have all these options."

"Syntactically, it was not clear how to stick the options into this chain of things written in prefix notation, cat of grep of who [i.e. cat(grep(who))]," he said. "Syntactic blinders: I didn't see how to do it."

Although stymied, McIlroy didn't drop the idea. "And over a period from 1970 to 1972, I'd from time to time say, 'How about making something like this?', and I'd put up another proposal, another proposal, another proposal. And one day I came up with a syntax for the shell that went along with the piping, and Ken said, 'I'm going to do it!'"

"He was tired of hearing this stuff," McIlroy explained. "He didn't do exactly what I had proposed for the pipe system call. He invented a slightly better one that finally got changed once more to what we have today. He did use my clumsy syntax."

"Thompson saw that file arguments weren't going to fit with this scheme of things and he went in and changed all those programs in the same night. I don't know how...and the next morning we had this orgy of one-liners."

"He put pipes into UNIX, he put this notation into shell, all in one night," McElroy said in wonder.

Next: Creating a programming philosophy from pipes and a tool box

Here is how Dennis M. Ritchie in his paper Early Unix history and evolution describes how pipes were introduced in Unix:

If you are not familiar with pipes, however, you should study this feature. Pipes are elegant implementation of coroutines on OS shell level and as such they allow the output from one program to be fed as input to another program. Doug McIlroy, the inventor of pipes, is said to point out that both pipes and lazy lists behave exactly like coroutines. Some other operating systems like MS DOS "fake" pipes by writing all the output of the first program to a temporary file, and then using that temporary file as input to the second program. That's not the real thing (suppose the first program produces an enormous amount of output or even does not terminate), but this is not a page where we discuss operating systems.

The simplest way to get you foot wet in this construct is to use shell language. Among various Unix shells  ksh93 contains the best facilities for using pipes. Bash is weaker and more buggy but still pretty decent. Perl is rather weak in this respect (for example it's impossible for an internal subroutine in Perl to produce stream that will be read by another subroutine via pipe, but one can compensate this weakness by using sockets for the implementation of complex pipe-style processing.

Tools like netcat can connect pipes to TCP/IP sockets creating computer-to-computer pipes and thus extending the Unix philosophy of "everything is a file" to networked multiple computers.  Pipes also can be used with tools like rsh and ssh for inter-computer communication.

Historically Modula 2 was the first language that supported pipes as a programming construct. Among scripting languages both modern Python and Ruby support pipes. BTW Perl is pretty disappointing in providing pipe-related functionality. It's really unfortunate but probably will be partially solved in Perl 6. See advanced languages for some languages that support coroutines and piping as a programming concept. To be fair there is a special module IPC-Run by Barrie Slaymaker(barries@slaysys.com). After a user's spun up on bash/ksh, it provides useful piping constructs, subprocesses, and either expect-like or event loop oriented I/O capabilities.

Probably everybody knows that how to use simple pipes: to send the output of one program to another, use the | symbol (known as the pipe symbol) between the two commands as follows:

command1 | command2

For example, if you want to look at a list of your files, but you have too many files to see at once, you can prevent them from scrolling too quickly by piping the ls command into the more command as shown below:

ls | more

Another example would be to pipe the lpc stat command into the more command as follows:

lpc stat | more

But this is pipes for dummies level -- pipes are much more than that. Essentially pipes are a very powerful programming paradigm that is connected with concept of coroutines.  The latter is available in several programming languages including ksh93, Java and Modula 2, Icon.  IMHO it is an extremely  elegant concept that can be considered as one of the most important programming paradigms -- I think that as a paradigm it is more important that that the idea of object oriented programming.  In many cases structuring a program as sequence  of coroutines makes it much more simpler that object oriented approach (or more correctly primitive object oriented approach because a large part of OO blah-blah-blah is just common sense and useful hierarchical structuring of name space).

Paradoxically, but the most widely used programming language for coroutines programming is ksh. Learning Korn Shell contains probably the best explanation of ksh88 coroutines mechanism (ksh93 has better capabilities).

Again I would like to stress that pipes are implementation of coroutines and several languages (Modula, Oberon, Icon) contain mechanisms for using coroutines.  A much underrated feature, coroutines, buys you 80% of what threads give you with none of the hassle. But threads are better than nothing and it is possible to use threads as a substitute for coroutines, for example, in Java.

There are also named pipes. A named pipe (also called FIFO) is a special file that acts as a buffer to connect processes on the same machine. Ordinary pipes also allow processes to communicate, but those processes must have inherited the filehandles from their parents. To use a named pipe, a process need know only the named pipe's filename. In most cases, processes don't even need to be aware that they're reading from a pipe. Use named pipes you need first to create one using mkfifo command:

% mkfifo /path/to/named.pipe

After that you can write to it using one process and read from it using another:

writer:

open(SYSFIFO, "> /mypath/my_named.pipe")  or die $!;
while (<SYSFIFO>) {
   ... ... ...  
}
close(SYSFIFO);

reader

open(SYSFIFO, "< /mypath/my_named.pipe")  or die $!;
while (<SYSFIFO>) {
   ... ... ...
}
close(SYSFIFO);

The writer to the pipe can be a daemon, for example syslogd.  That gives a possibility to process syslog dynamically.

Unfortunately using pipe as a source of input to other program won't always work, because some programs check the size of the file before trying to read it. Because named pipes appear as special files of zero size on the filesystem, such clients and servers will not try to open or read from our named pipe, and the trick will fail.

Nikolai Bezroukov

Old News ;-)

[Apr 6, 2009] Bash Process Substitution Linux Journal

May 22nd, 2008 by Mitch Frazier

In addition to the fairly common forms of input/output redirection the shell recognizes something called process substitution. Although not documented as a form of input/output redirection, its syntax and its effects are similar.

The syntax for process substitution is:

  <(list)
or
  >(list)

where each list is a command or a pipeline of commands. The effect of process substitution is to make each list act like a file. This is done by giving the list a name in the file system and then substituting that name in the command line. The list is given a name either by connecting the list to named pipe or by using a file in /dev/fd (if supported by the O/S). By doing this, the command simply sees a file name and is unaware that its reading from or writing to a command pipeline.

To substitute a command pipeline for an input file the syntax is:

  command ... <(list) ...

To substitute a command pipeline for an output file the syntax is:

  command ... >(list) ...

At first process substitution may seem rather pointless, for example you might imagine something simple like:

  uniq <(sort a)

to sort a file and then find the unique lines in it, but this is more commonly (and more conveniently) written as:

  sort a | uniq

The power of process substitution comes when you have multiple command pipelines that you want to connect to a single command.

For example, given the two files:

  # cat a
  e
  d
  c
  b
  a
  # cat b
  g
  f
  e
  d
  c
  b

To view the lines unique to each of these two unsorted files you might do something like this:

  # sort a | uniq >tmp1
  # sort b | uniq >tmp2
  # comm -3 tmp1 tmp2
  a
        f
        g
  # rm tmp1 tmp2

With process substitution we can do all this with one line:

  # comm -3 <(sort a | uniq) <(sort b | uniq)
  a
        f
        g

Depending on your shell settings you may get an error message similar to:

  syntax error near unexpected token `('

when you try to use process substitution, particularly if you try to use it within a shell script. Process substitution is not a POSIX compliant feature and so it may have to be enabled via:

  set +o posix

Be careful not to try something like:

  if [[ $use_process_substitution -eq 1 ]]; then
    set +o posix
    comm -3 <(sort a | uniq) <(sort b | uniq)
  fi

The command set +o posix enables not only the execution of process substitution but the recognition of the syntax. So, in the example above the shell tries to parse the process substitution syntax before the "set" command is executed and therefore still sees the process substitution syntax as illegal.

 

Of course, note that all shells may not support process substitution, these examples will work with bash.

[Apr 5, 2009] Using Named Pipes (FIFOs) with Bash

[Dec 9, 2008] Slashdot What Programming Language For Linux Development

by dkf (304284) <donal.k.fellows@man.ac.uk> on Saturday December 06, @07:08PM (#26016101) Homepage

C/C++ are the languages you'd want to go for. They can do *everything*, have great support, are fast etc.

Let's be honest here. C and C++ are very fast indeed if you use them well (very little can touch them; most other languages are actually implemented in terms of them) but they're also very easy to use really badly. They're genuine professional power tools: they'll do what you ask them to really quickly, even if that is just to spin on the spot chopping peoples' legs off. Care required!

If you use a higher-level language (I prefer Tcl, but you might prefer Python, Perl, Ruby, Lua, Rexx, awk, bash, etc. - the list is huge) then you probably won't go as fast. But unless you're very good at C/C++ you'll go acceptably fast at a much earlier calendar date. It's just easier for most people to be productive in higher-level languages. Well, unless you're doing something where you have to be incredibly close to the metal like a device driver, but even then it's best to keep the amount of low-level code small and to try to get to use high-level things as soon as you can.

One technique that is used quite a bit, especially by really experienced developers, is to split the program up into components that are then glued together. You can then write the components in a low-level language if necessary, but use the far superior gluing capabilities of a high-level language effectively. I know many people are very productive doing this.

[Jan 7, 2008] freshmeat.net Project details for pmr by Heikki Orsila

About: pmr is a command line filter that displays the data bandwidth and total number of bytes passing through a pipe. It can also limit the rate of data going through the pipe and compute an MD5 checksum of the stream for verifying data integrity on unreliable networks.

It has following features:

Measure data rate on the command line. pmr reads data from standard input and copies it to standard output.

Limit data rate to a specified speed (e.g. 100 KiB/s useful for slow internet connections)
Example: copy files to another host with at most 100 KiB/s speed


tar cv *files* |pmr -l 100KiB |nc -q0 host port


Compute an md5sum of the stream (useful for verifying integrity of network transfers)
Example: copy files to another host and verify checksums on both sides


Sender: tar cv *files* | pmr -m | nc -q0 host port


Receiver: nc -l -p port | pmr -m | tar xv


Calculate time estimate of the copied data when the stream size is known
Example: copy files to another host and calculate an estimated time of completion


tar cv *files* |pmr -s 1GiB |nc -q0 host port

Changes: The man page was missing in release 1.00, and now it is back.
 

The lost art of named pipes by Tony Mancill

04.20.2004  | searchenterpriselinux.techtarget.com

A "named pipe" -- also known as a FIFO (First In, First Out) or just fifo -- is an inter-process communication mechanism that makes use of the filesystem to allow two processes to communicate with each other. In particular, it allows one of these to open one end of the pipe as a reader, and the other to open it as a writer. Let's take a look at the FIFO and how you can use it.

First, here's a real-life example of a named pipe at work. In this instance, you run a shell command like: "ls -al | grep myfile" In that example, the "ls" program is writing to the pipe, and "grep" is reading from it. Well, a named pipe is exactly that, but the processes don't have to be running under the same shell, nor are they restricted to writing to STDOUT and reading from STDIN. Instead, they reference the named pipe via the filesystem.

In the filesystem, it looks like a file of length 0 with a "p" designation for the file type. Example:

tony@hesse:/tmp$ ls -l pdffifo

prw-r--r--    1 tony     tony            0 2004-01-11 17:32 pdffifo

Note that the file never grows; it's not actually a file but simply an abstraction of one type of IPC (Inter-Process Communication) provided by the kernel. As long as the named pipe is accessible, in terms of permissions, to the processes that would like to make use of it, they can read and write from it without having to worry about every using physical disk space or paying any I/O subsystem overhead.

At first glance, the utility of a named pipe is perhaps not immediately obvious, so I've come armed with more examples.

Here's a sample dilemma about dealing with the receipts generated by Web transactions. My browser of choice, Mozilla Firebird, gives me the option of either printing a page to a printer, or writing it as a Postscript file to a directory. I don't want a hardcopy, as it wastes paper, and I'll just have to file it somewhere. (If I wanted to kill a tree, I wouldn't be doing my transactions on the Web!) Also, I don't like the PostScript files, because they're large and because operating systems that don't have a copy of Ghostscript installed can't view them very easily.

Instead, I want to store the page as a PDF file. This is easily accomplished with the command-line utility "ps2pdf", but I'm too lazy to write the file out as PostScript, open a shell, convert the file and then delete the PostScript file. That's no problem because the browser knows how to open a file and write to it. And, ps2pdf knows how to read from STDIN to produce PDF output.

So, in its simplest incarnation:

mkfifo /tmp/pdffifo

ps2pdf - ~/receipts/webprint.pdf </tmp/pdffifo

When I tell my browser to print the PS output to /tmp/pdffifo, the result is PDF output in ~/receipts/webprint.pdf. This is fine, but it's a "one-shot," meaning that you have to set it up each time you want to print. That's because ps2pdf exits after processing one file. For a slightly more general solution, see listing 1 at the end of this tip.

Admittedly, there are other ways to solve that Web print problem. For example, I could let the browser write the PS file and then have a daemon sweep through the directory once a day and convert everything. Unfortunately, then I'd have to wait for my PDF files, and you wouldn't have seen an example of named pipes in action.

A very pragmatic use for these rascals is made in mp3burn (http://sourceforge.net/projects/mp3burn/), a utility written in Perl that is used to burn MP3 tracks onto audio CDs. It takes advantage of named pipes by using them as the conduit between the MP3 decoders (or player) and the cdrecord, which expects WAV audio files turn to the CD. Those WAV files, which can be 700MB or so, never have to be written anywhere to the filesystem. Your system has to be fast enough to decode the MP3 to WAV and run your CD burner. But you're not paying the overhead of writing to and from the hard drive, and you don't have to worry about having almost a GB of available free space to do the burn.

Finally, there is an entire class of applications for system administrators who need to write compressed logfiles in real-time from programs that don't have compression support built-in. Even when you do have the source, do you want to spend all day re-inventing the wheel?

---

Listing 1

#!/usr/bin/perl
#
# webprint_pdf.pl
######################################################

while (1 == 1) {
    open(FIFO, "</tmp/pdffifo")|| die ("unable to open /tmp/pdffifo");
    open(PDF, "|ps2pdf − /tmp/outfile.$$");
    while(<FIFO>) {
        print PDF $_;
    }
    close(FIFO);
    close(PDF);
    rename("/tmp/outfile.$$", "/tmp/webprint_" . localtime() . ".pdf");
}

Tony Mancill is the author of "Linux Routers: A Primer for Network Administrators" from Prentice Hall PTR. He can be reached at tony@mancill.com.

Stuck in the Shell: The Limitations of Unix Pipes by David Glasser

Taking the a Unix guru's “|” key is as crippling as taking away a Windows user's mouse. At the Unix shell, piping is the fundamental form of program combination: pipes connect the standard output and standard input of many small tools together in a virtual “pipeline” to solve problems much more sophisticated than any of the individual programs can deal with. In theory, stringing together command-line programs is only one use of the underlying Unix pipe system call, which simply creates one file descriptor to write data to and another to read it back; these descriptors can be shared with subprocesses spawned with fork. One might think that this very generic system call, which was essentially the only form of inter-process communication in early Unix, could be used in many ways, of which the original “connect processes linearly” is just one example. Unfortunately, pipes have several limitations, such as unidirectionality and a common-ancestor requirement, which prevent pipes from being more generally useful. In practice, the limitations of a “pipe” system call designed for command-line pipelines restrict its use as a general-purpose tool.

Unix pipes are inherently unidirectional channels from one process to another process, and cannot be easily turned into bidirectional or multicast channels. This restriction is exactly what is needed for a shell pipeline, but it makes pipes useless for more complex inter-process communication. The pipe system call creates a single “read end” and a single “write end”. Bidirectional communication can be simulated by creating a pair of pipes, but inconsistent buffering between the pair of pipes can often lead to deadlock, especially if the programmer only has control of the program on one end of the pipe. Programmers can attempt to use pipes as a multicast channel by sharing one read end between many child processes, but because all of the processes share a single descriptor, an extra buffering layer is needed in order for the children to all independently read the message. A manually maintained collection of many pipes is required for pipe-based multicast, and programming that takes much more effort.

Pipes can only be shared between processes with a common ancestor which anticipated the need for the pipe. This is no problem for a shell, which sees a list of programs and can set up all of their pipes at once. But this restriction prevents many useful forms of inter-process communication from being layered on top of pipes. Essentially, pipes are a form of combination, but not of abstraction - there is no way for a process to name a pipe that it (or an ancestor) did not directly create via pipe. Pipes cannot be used for clients to connect to long-running services. Processes cannot even open additional pipes to other processes that they already have a pipe to.

These limitations are not merely theoretical - they can be seen in practice by the fact that no major form of inter-process communication later developed in Unix is layered on top of pipe. After all, the usual way to respond to the concern that a feature of a system is too simple is to add a higher-level layer on top; for example, the fact that Unix pipes send raw, uninterpreted binary data and not high-level data structures can be fixed by wrapping pipes with functions which marshal your structures before putting them through the pipe. But the restriction of pipes to premeditated unidirectional communication between two processes cannot be fixed in this way. Several forms of inter-process communication, such as sockets, named pipes, and shared memory, have been created for Unix to overcome the drawbacks of pipes. None of them have been implemented as layers over pipes; all of them have required the creation of new primitive operations. In fact, the reverse is true - pipes could theoretically be implemented as a layer around sockets, which have grown up to be the backbone of the internet. But poor old pipes are still limited to solving the same problems in 2006 that they were in 1978.

[June 3, 2002] PipeMore 1.0  by Terry Gliedt

About: PipeMore is a utility to be used as the last of a series of piped commands (like 'more'). It displays STDIN data in a scrolled window where it can be searched or saved, and thereby avoids filling your xterm with temporary output.

Homepage:
http://pipemore.sourceforge.net/
Tar/GZ:
[..]nloads.sourceforge.net/pipemore/pipemore-1.0.tar.gz?download
CVS tree (cvsweb):
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/pipemore/

NETPIPES 1 October 28, 1997

NETPIPES 1 October 28, 1997 netpipes – a package to manipulate BSD TCP/IP stream sockets

version 4.2

SYNOPSIS

faucet port (--in|--out|--err|--fd n)+ [--once] [--verbose] [--quiet] [--unix] [--foreignhost addr] [--foreignport port] [--localhost addr] [--serial] [--daemon] [--shutdown (r|w) ] [--pidfile filename] [--noreuseaddr] [--backlog n] [-[i][o][e][#3[,4[,5...]]][v][1][q][u][d][s]] [-p foreign-port] [-h foreign-host] [-H local-host] command args

hose hostname port (--in|--out|--err|--fd n|--slave) [--verbose] [--unix] [--localport port] [--localhost addr] [--retry n] [--delay n] [--shutdown [r|w][a] ] [--noreuseaddr] [-[i][o][e][#3[,4[,5...]]][s][v][u]] [-p local-port] [-h local-host] command args

encapsulate --fd n [ --verbose ] [ --subproc [ --infd n[=sid] ] [ --outfd n[=sid] ] [ --duplex n[=sid] ] [ --Duplex n[=sid] ] [ --DUPLEX n[=sid] ] [ --prefer-local ] [ --prefer-remote ] [ --local-only ] [ --remote-only ] ] [ --client ] [ --server ] -[#n][v][s[in][on][dn][ion][oin][l][r][L][R]] command args ...

ssl-auth --fd n ( --server | --client ) [ --cert file ] [ --key file ] [ --verbose ] [ --verify n ] [ --CApath path/ ] [ --CAfile file ] [ --cipher cipher-list ] [ --criteria criteria-expr ] [ --subproc [ --infd n ] [ --outfd n ] ] [ -[#n][v][s[in][on]] ]

sockdown [fd [how] ]

getpeername [ -verbose ] [ -sock ] [ fd ]

getsockname [ -verbose ] [ -peer ] [ fd ]

timelimit [ -v ] [ -nokill ] time command args

Korn Shell Script Course Notes

• Redirection

Writing agents in sh: conversing through a pipe from Oleg Kiselyov site

[Mar 10, 2002] Unix pipes

Examples of scripts that use pipes. Most examples are pretty trivial.

[Mar 04, 2002] TR 93-045: Sather Iters: Object-Oriented Iteration Abstraction

Sather iters are a powerful new way to encapsulate iteration. We argue that such iteration abstractions belong in a class' interface on an equal footing with its routines. Sather iters were derived from CLU iterators but are much more flexible and better suited for object-oriented programming. We motivate and describe the construct along with several simple examples. We compare it with iteration based on CLU iterators, cursors, riders, streams, series, generators, coroutines, blocks, closures, and lambda expressions. Finally, we describe how to implement them in terms of coroutines and then show how to transform this implementation into efficient code.

[May 15 2001] coroutines for Ruby

[Feb. 24, 2001] HPUX-DEVTOOLS Named Pipes Vs Sockets

I found the answer to my question in one paper:
Implementation and measurements of efficient communication facilities for distributed database systems
Bhargava, B.; Mafla, E.; Riedl, J.; Sauder, B. Data Engineering, 1989. Proceedings. Fifth International Conference on ,1989 Page(s): 200 -207
Here are the sample times as per this paper for different message sizes in msec.  
                             
                    for 10 bytes   1000 bytes

 Sockets                     4.3         9.6
                   
 Named Pipes            2.3         3.9
            
 Message Queues     2.0         2.9

So message queues are the best to implement Inter process communications with in a system

thanks,
Srinivas

[Oct. 07, 2000] A special module IPC-Run by Barrie Slaymaker(barries@slaysys.com).

After a user's spun up on bash/ksh, it provides useful piping constructs, subprocesses, and either expect-like or event loop oriented I/O capabilities.

[Aug 7, 2000]  http://tmtowtdi.perl.org/rfc/27.pod   -- "Tom Scola" <tscola@inch.com> 

A proposal for Pipes in Perl 6. First I would like to state that this is an excellent and timely proposal. Of course with Perl, the lexical sugar is one of the most difficult parts. Here it might be that we need to adopt a trigram symbol like <|> for pipes :-). But the main thing in the proposal was done right. I agree that  "Inside a coroutine, the meanings of "<>" and the default file descriptor for print, printf, etc. are overloaded. ". That's fundamentally right approach. But one of the most important thing is to preserve symmetry between i/o and pipes as long as this is possible. That means that you should be able to open coroutine as file:

open (SYSCO, >coroutine);

print SYSCO, $record1;

co coroutine {
   $record=<>;
   ... ...
}

Syntactically coroutine name is a bareword so it should be OK in open statement.

The second thing is the ability to feed coroutine in a simplified manner. One of
the most important special cases is feeding it from the loop:

for .... {

} <|> stage1 <|> stage2 # here the value of $_ should be piped on each iteration

The third important special case feeding lists into pipe. That can be achieved by special built-in function pipefeed

pipefeed(@array) <|> co1 <|> co2;

or

pipefeed ('mn','ts','wn',...) <|> co1 <|> co2;

The possibilities to split a pipe on two subpipes are also very important (see
VM/CMS pipes). Streams A and B should be defined in co1. For example

co1 <|>(:::A,:::B)
A::: co2 <|>
B::: co3 <|> ...

As for selling IMHO the key question is probably competition. Introduction of
pipes can help to differentiate the language from PHP at least temporarily :-).

It also might simplify the language in several aspects. For example coroutine can be a natural base of exception handling like in PL/1. Currently Perl is weak in this respect.

Actually one flavor of Python already has this capabilities.

Communications  Patch-free User-level Link-time intercepting of system calls and interposing on library functions

Contents

1.Introduction
2.Statement of the Problem
3.Solutions
1.Linux 2.x and GNU ld
2.HP-UX 10.x and a native ld
3.Solaris 2.6 and a native ld
4.FreeBSD 3.2 and a GNU ld
4.Application: Extended File Names and Virtual File Systems

• Uni-, Bi-, and TCP pipes as regular files
• Writing agents in sh- conversing through a pipe
• TCPStream - a C++ standard iostream over a TCP channel

SunWorld: Introduction to pipes, filters, and redirection, Part 1  by Mo Budlong

Other articles of Mo Budlong are still available at http://www.kingcomputerservices.com/unixtuto.htm

"Redirection allows a user to redirect output that would normally go to the screen and instead send it to a file or another process. Input that normally comes from the keyboard can be redirected to come from a file or another process."

XML.com - Pyxie

PYX is based on a concept from the SGML world known as ESIS. ESIS was popularized by James Clark's SGML parsers. (Clarks' first parser was sgmls, a C-based parser built on top of the arcsgml parser developed by Dr. Charles Goldfarb, the inventor of SGML. Then came the hugely popular nsgmls, which was a completely new SGML parsing application implemented in C++.)

The PYX notation facilitates a useful XML processing paradigm that presents an alternative to SAX or DOM based API programming of XML documents. PYX is particularly useful for pipeline processing, in which the output of one application becomes the input to another application. We will see an example of this later on.

Linux Today Overflow 0.1 Released

similar to National Instruments' "G" language used in their LabView product?

We are proud to announce the first release of the Overflow project, version 0.1. Overflow is a free (GPL) "data flow oriented" development environment which allows users to build programs by visually connecting simple building blocks. Though it is primarily designed as a research tool, it can also be used for real-time data processing. This first release includes 5 toolboxes: signal processing, image processing, speech recognition, vector quantization and neural networks.

The visual interface is written for GNOME, but there is also a command-line tool that doesn't use GNOME. Because of the modular design, we hope to have a KDE version in the future. Screenshots can be found here.

Recommended Links

• Pipeline (Unix) - Wikipedia, the free encyclopedia
• Pipeline (software) - Wikipedia, the free encyclopedia
• Coroutine - Wikipedia, the free encyclopedia
• Unix Socket Programming FAQ Homepage
  • UK site (kipper.york.ac.uk)
• W. Richard Stevens' Home Page
• UMass CMPSCI 653/491G: Computer Networks
• Developer's Resource Guide
• UMd's cmsc498p sock links
• Programming the BSD Socket interface, from CSU-Chico
• UNC INLS 210: TCP/IP Networks and Network Programming
• Bjorn's Java Network Tutorial
• McGill 304-427B: Operating Systems
• Temple CIS307: Introduction to Operating Systems
• yangxy's Network Programming Information
• Networking Research at the Pittsburgh Supercomputing Center

Shell level Introductory materials

Unix Pipes -- small introduction (for dummies level)

NACSE - UNIX TOOLS AND TIPS

Input-Output Redirection and Pipes from A Quick Guide for UNIX and the Department Computing Facilities

Advanced UNIX

• Filters
• Pipes
• Redirection

pipe - PC Webopaedia Definition and Links

Computer Guide for MTS

• Pipes. For Effective Unix Users
• Pipe Filters
• Archiving Files

Advanced Piping

Network Access with GAWK

UNIX File System by Michael Lemmon University of Notre Dame

Unix Programming Frequently Asked Questions - Table of Contents

2. General File handling (including pipes and sockets)

Linux Interprocess Communications

NMRPipe.html -- NMRPipe: a multidimensional spectral processing system based on UNIX pipes

Advanced languages

Simula 76 was the first language that implement coroutines as language constructs. Algol 68 and Modula-2 followed the suit. Actually Modula-2 is a really impressive and under appreciated language for system programming. Here is explanatiuon of this concept from OOSC 2 28.9 EXAMPLES

Coroutines emulate concurrency on a sequential computer. They provide a form of functional program unit ("functional" as opposed to "object-oriented") that, although similar to the traditional notion of routine, provides a more symmetric form of communication. With a routine call, there is a master and a slave: the caller starts a routine, waits for its termination, and picks up where it left; the routine, however, always starts from the beginning. The caller calls; the routine returns. With coroutines, the relationship is between peers: coroutine a gets stuck in its work and calls coroutine b for help; b restarts where it last left, and continues until it is its turn to get stuck or it has proceeded as far as needed for the moment; then a picks up its computation. Instead of separate call and return mechanisms, there is a single operation, resume c, meaning: restart coroutine c where it was last interrupted; I will wait until someone e

This is all strictly sequential and meant to be executed on a single process (task) of a single computer. But the ideas are clearly drawn from concurrent computation; in fact, an operating system that provide such schemes as time-sharing, multitasking (as in Unix) and multithreading, mentioned at the beginning of this chapter as providing the appearance of concurrency on a single computer, will internally implement them through a coroutine-like mechanism.

Coroutines may be viewed as a boundary case of concurrency: the poor man's substitute to concurrent computation when only one thread of control is available. It is always a good idea to check that a general-purpose mechanism degrades gracefully to boundary cases; so let us see how we can represent coroutines. The following two classes will achieve this goal.

REFERENCE MATERIAL

1. Tremblay & Sorenson, "The Theory and Practice of Compiler Writing", McGraw Hill, 1985.
2. Aho, Sethi and Ullman, "Compilers Principles, Techniques and Tools", Addison Wesley, 1987.
3. Pratt, "Programming Languages", Prentice Hall, 1984.
4. Sethi, "Programming Languages Concepts and Constructs", Addison- Wesley, 1989.

Coroutines in Modula-2

Famous Dotzel paper:

• Dotzel Using Coroutines in Oberon-2, Part I
• Dotzel- Using Coroutines in Oberon-2, Part II
• Ulm's Oberon Library Coroutines

Coroutines in BETA

• Coroutines

Ice 9 - Coroutines Using Runqs

Coroutines and stack overflow testing

The MT Icon Interpreter

The Aesop System A Tutorial

• Aesop System Home Page

Java Pipes -- not that impresssive

TaskMaster -- see descripting and pointers to Fabrik

Continuations And Stackless Python

3.1 Coroutines in Display PostScript

An Introduction to Scheme and its Implementation - call-with-current-continuation

• Implementing Coroutines

CPS 206 Advanced Programming Languages Fall, 1999 Text: Finkel: Advanced Programming Language Design

(#3 2 lectures, skip 4)

1. Exception Handling

2. Coroutines

Coroutines in Simula

Coroutines in CLU

Embedding CLU Iterators in C

Coroutines in Icon

3. Continuations: Io

4. Power Loops

5. Final Comments

Programming

C Coroutines

CORO(2)                    C Coroutines                   CORO(2)

NAME

       co_create, co_call, co_resume, co_delete, co_exit_to,
       co_exit - C coroutine management


SYNOPSIS

       #include <coro.h>

       extern struct coroutine *co_current;
       extern struct coroutine co_main[];

       struct coroutine *co_create(void *func, void *stack, int stacksize);
       void co_delete(struct coroutine *co);
       void *co_call(struct coroutine *co, void *data);
       void *co_resume(void *data);
       void *co_exit_to(struct coroutine *co, void *data);
       void *co_exit(void *data);

DESCRIPTION

       The coro library implements the low level functionality
       for coroutines.  For a definition of the term coroutine
       see The Art of Computer Programming by Donald E. Knuth.
       In short, you may think of coroutines as a very simple
       cooperative multitasking environment where the switch from
       one task to another is done explicitly by a function call.
       And, coroutines are fast.  Switching from one coroutine to
       another takes only a couple of assembler instructions more
       than a normal function call.

       This document defines an API for the low level handling of
       coroutines i.e. creating and deleting coroutines and
       switching between them.  Higher level functionality
       (scheduler, etc.) is not covered here.

Pipe Tutorial Intro by Faith Fishman

Advanced UNIX Programming -- lecture notes by Michael D. Lemmon

Interprocess Communications in UNIX -- book

Beej's Guide to Unix IPC

Named Pipes

FMTEYEWTK about Perl IPC

Unix Communication Facilities done by: Gerhard Müller (muellerg@informatik.tu-muenchen,de) Supervisor: Dr. N.A. Speirs (Neil.Speirs@ncl.ac.uk) 2nd April 1996

CTC Tutorial on Pipes

6.2 Half-duplex UNIX Pipes

Sockets programming links

Beej's Guide to Network Programming -- Using Internet Sockets Version 1.5.4 (17-May-1998) [http://www.ecst.csuchico.edu/~beej/guide/net]

BSD Sockets A Quick And Dirty Primer -- Jim Frost Software Tool & Die Copyright (c) 1988, 1994 Jim Frost All Rights Reserved Last changed November 4, 1994

Network Programmer's Guide and Reference  -- big e-book

6 Linux Interprocess Communications

Sockets Tutorial

Sockets Tutorial -- Socketpairs  -- Berkeley UNIX 4.4BSD provides a slight generalization of pipes. A pipe is a pair of connected sockets for one-way stream communication. One may obtain a pair of connected sockets for two-way stream communication by calling the routine socketpair(). The program in Figure 3 calls socketpair() to create such a connection. The program uses the link for communication in both directions. Since socketpairs are an extension of pipes, their use resembles that of pipes. Figure 4 illustrates the result of a fork following a call to socketpair().

(most of them from Davin's collection of unix programming links)

• Unix Socket Programming FAQ
  • UK site (kipper.york.ac.uk)
• Jim Frost's BSD Sockets: A Quick And Dirty Primer
  • Master copy from Jim Frost's homepage
  • From umd
  • From softlab.ntua.gr
  • From csuchico (older version)
• BSD Sockets intro
• Beej's Guide to Network Programming
• An Introductory 4.4BSD Interprocess Communication Tutorial [PSD:20]
  • PostScript from umd
• An Advanced 4.4BSD Interprocess Communication Tutorial [PSD:21]
  • PostScript from um
• Advanced 4.3BSD IPC Tutorial [PS1:8] (from csuchico)
• A short tutorial on socket programming, from the University of Wisconsin (from UMass)
• Jim Frost's UNIX Signals and Process Groups
  • Master copy from Jim Frost's homepage
• the UNIX course, by Liam Madden
• An Introduction to Socket Programming, by Reg Quinton
  • Master copy from Reg Quinton's page
• Asynchronous Socket programming, in Python
• Socket Library Programming Reference (for DESQview/X. But including man pages)
• NetCon's Programmer's Guide BSD Sockets Programming interface
• Socket Programming Course (using RealAudio)
• The Network Book (from Columbia University)
• Temple University socket programming intro
• Client Server course from University of Canberra
• Introduction to network functions in C
• A User's Guide to TCP Windows

Books

• Practical Unix Programming
• UNIX Systems Programming for SVR4
• Source code to Steven's UNIX Network Programming (.tar.Z)

Filters

Commonly Used UNIX Commands -- man pages with useful examples section

Unix User Guide -- command mode

UNIX Survival

Etc

CMS-TSO Pipelines Runtime Library Distribution

This Web page serves as a distribution point for files pertaining to CMS/TSO Pipelines.

The files marked as "packed" should be downloaded in binary mode, reblocked to 1024-byte, fixed-length records (e.g., using an "fblock 1024" stage), and then unpacked using an "unpack" stage. The BOOK files should be downloaded in binary mode and reblocked using an "fblock 4096" stage.

The files in VMARC format should be downloaded in binary mode, reblocked using an "fblock 80" stage, and then unpacked using the VMARC command.

The files in LISTING format have ASA carriage control ("FORTRAN carriage control"). On CMS they should be printed with the "CC" option; on most unix systems they can be printed with "lpr -f".

The GNU C Library - Signal Handling

An illustrated explanation of pipes

REXX and CMS pipes CMS-TSO Pipelines Runtime Library Distribution -- interesting product from IBM, similar to Unix, but not exactly ;-)

• REXX for UNIX Regina and Regina FAQ
• REXX WWW home page and FAQ (Mike Cowlishaw IBM UK Labs)
• CMS Pipelines primer
• more on CMS Pipelines
• Personal REXX User's Guide, Version 3.0
• Six Rules of Thumb for beginning Rexx programmers
• THE -- The Hessling Editor (XEDIT Clone for UNIX) -- uses REXX as a macro language
• CMS Pipelines Pages
• CMSTSO Pipelines Runtime Library Distribution