Unix Xargs Piping Toolkit Utility

The xargs utility is one of the most useful and the most underappreciated utilities in Unix piping toolbox. It constructs an argument list for an arbitrary Unix command using standard input and executes this command.

  xargs [options] [command]

The xargs command creates an argument list for command from standard input. It is typically used with a pipe getting its input from commands like ls and find The latter is probably the most common source of xargs input and is covered in examples below.

One of the most common xargs applications in pipes is to execute a command once for each piped record:

$ find . -name '*050106*' print \

   | xargs -n2 grep 'From: Ralph'

cat iplist | xargs -n1 nmap -sV

The find command searches the entire directory structure for filenames that contain 050106 . The xargs command executes grep command for each two argument (so that filename was visible). In the second example cat command supplies the list of IPs for map to scan.

In many Unix shells there is a limit to the number of arguments allowed on a single command line. This is often a problem when you analyze spam blocked by a spam filter. Here xargs can help: if the argument list read by xargs is larger than the maximum allowed by the shell, xargs will bundle the arguments into smaller groups and execute command separately for each argument bundle.

If no command is specified, xargs works similar to the echo command and prints the argument list to standard output.

Options

Option	Description
`-n#`	Execute command once for every # argument. For example, -n2 bundles arguments into groups of two or less and executes command on each argument bundle.
`-l#`	Execute command once for every # lines of input. For example, -l1 creates a bundle of arguments for every one line of input and executes command on each argument bundle.
`-i`	Normally xargs places input arguments at the end of command. Used with the -i option, xargs will replace all instances of {} with input arguments. You need to put them in single brackets or use a backslash (\) before each bracket to keep the shell from interpreting the special characters.
`-t`	Echo each command before executing. Nice for debugging
`-p`	Prompts the user before executing each command. Useful for debugging.

Examples

To use a command on files whose names are listed in a file, enter:
```
xargs lint -a <cfiles
```
If the cfiles file contains the following text:
```
main.c readit.c
gettoken.c
putobj.c
```
the xargs command constructs and runs the following command:
```
lint -a main.c readit.c gettoken.c putobj.c
```
If the cfiles file contains more file names than fit on a single shell command line (up to LINE_MAX), the xargs command runs the lint command with the file names that fit. It then constructs and runs another lint command using the remaining file names. Depending on the names listed in the cfiles file, the commands might look like the following:
```
lint -a main.c readit.c gettoken.c . . .
lint -a getisx.c getprp.c getpid.c . . .
lint -a fltadd.c fltmult.c fltdiv.c . . .
```
This command sequence is not quite the same as running the lint command once with all the file names. The lint command checks cross-references between files. However, in this example, it cannot check between the main.c and the fltadd.c files, or between any two files listed on separate command lines.

For this reason you may want to run the command only if all the file names fit on one line. To specify this to the xargs command use the -x flag by entering:
xargs -x lint -a <cfiles
If all the file names in the cfiles file do not fit on one command line, the xargs command displays an error message.
To construct commands that contain a certain number of file names, enter:
xargs -t -n 2 diff <<EOF starting chap1 concepts chap2 writing chap3 EOF
This command sequence constructs and runs diff commands that contain two file names each (-n 2):
```
diff starting chap1
diff concepts chap2
diff writing chap3
```
The -t flag causes the xargs command to display each command before running it, so you can see what is happening. The <<EOF and EOF pattern-matching characters define a here document, which uses the text entered before the end line as standard input for the xargs command.
To insert file names into the middle of command lines, enter:
ls | xargs -t -I {} mv {} {}.old
This command sequence renames all files in the current directory by adding.old to the end of each name. The -I flag tells the xargs command to insert each line of the ls directory listing where {} (braces) appear. If the current directory contains the files chap1, chap2, and chap3, this constructs the following commands:
```
mv chap1 chap1.old
mv chap2 chap2.old
mv chap3 chap3.old
```
To run a command on files that you select individually, enter:
ls | xargs -p -n 1 ar r lib.a
This command sequence allows you to select files to add to the lib.alibrary. The -p flag tells the xargs command to display each ar command it constructs and to ask if you want to run it. Enter y to run the command. Press the any other key if you do not want to run the command.
Something similar to the following displays:
```
ar r lib.a chap1 ?...
ar r lib.a chap2 ?...
ar r lib.a chap3 ?... 
```
To construct a command that contains a specific number of arguments and to insert those arguments into the middle of a command line, enter:
```
ls | xargs -n6 | xargs -I{} echo {} - some files in the directory
```
If the current directory contains files chap1 through chap10, the output constructed will be the following:
```
chap1 chap2 chap3 chap4 chap5 chap6 - some files in the directory
chap7 chap8 chap9 chap10 - some file in the directory
```

Typically arguments are lists of filenames passed to xargs via a pipe. Please compare:

$ ls 050106* $ ls 050106* | xargs -n2 grep "From: Ralph"

In the first example list of files that starts with 050106 is printed. In the second for each two such files grep is executed.

Additional Examples

John Meister's UNIX Notes

Change permissions on all regular files in a directory subtree to mode 444, and permissions on all directories to 555:

find -type f -print | xargs chmod 444
$ ls * | xargs -n2 head -10
line 1 of f1 line 2 of f1 line 3 of f1
ls * } xarg -n1 wc -1 
(date +%D ; du -s ~) | xargs >> log
ls *.txt | xargs -i basename \{\} .ascii \
| xargs -i mv \{\}.ascii \{\}.ask
(Note that the backslash usage)
Another example. Let's "cat" the Contents of Files Listed in a File, in That Order.

$ cat file_of_files file1 file2 $ cat file1 This is the data in file1 $ cat file 2 This is the data in file2
So there are 3 files here "file_of_files" which contains the name of other files. In this case "file1" and "file2". And the contents of file1" and "file2" is shown above.

$ cat file_of_files | xargs cat This is the data in file1 This is the data in file2

What if you want to find a string in all finds in the current directory and below. Well the following script will do it.

#!/bin/sh SrchStr=$1 shift for i in $*; do find . -name "$i" -type f -print | xargs egrep -n "$SrchStr"/dev/null done
Another quite nice thing, used for updating CVS/Root files on a Zaurus:
find . -name Root | xargs cp newRoot

Just copies the contents of newRoot into every Root file. I think this works too:

find . -name Root | xargs 'echo user@machine.dom:/dir/root >'

as long as the quote are used to avoid the initial interpretation of the >.

These pieces of randomness will look for all .sh files in PWD and print the 41st line of each - don't ask me why I wanted to know. Thanks to Brian R for these.

for f in *.sh; do sed -n '41p' $f; done

or

ls *.sh | xargs -l sed -n '41p' Remove all the files in otherdir that exist in thisdir.

ls -1d ./* | xargs -i rm otherdir/{}

Oracle Tips UNIX find and cat

Locate Oracle files that contain certain strings - This is one of the most common shell command for finding all files that contain a specified string. For example, assume that you are trying to locate a script that queries the v$process table. You can issue the following command, and UNIX will search all subdirectories, looking in all files for the v$process table.

root> find . -print|xargs grep v\$process ./TX_RBS.sql: v$process p, ./UNIX_WHO.sql:from v$session a, v$process b ./session.sql:from v$session b, v$process a

broadband » Forums » All things UNIX » Little Known Tips and Tricks...

To follow on to Steve's xargs madness, let's say you've got some daemon process that is just running away. It's spawning more and more processes and "service blah stop" is not doing anything for you. Here's a cute way to kill all of those processes with the "big hammer":
ps -auxwww | grep "daemonname" | awk '{print $2}' | xargs kill -9

Tips and tricks for Mandrake Linux

If you edit files on linux, chances are you'll end up with a lot of those backup files with names ending with ~, and removing them one by one is a pain.

Luckily, with a simple command you can get them all, and recursively. Simply go into the top of the directory tree you want to clean (be carefull, these commands are recursive, they will run through the subdirectories), and type

find ./ -name '*~' | xargs rm

Some explanations:

the command find ./ -name '*~'will look for all files ending with ~ in the local directory and its subdirectories,

the sign | is a 'pipe' that makes the outout of the previous command becoming the input of the next one,

xargs reads the arguments on the standard input and applies it to the following command

rm deletes whatever file you want.

Print or Query Before Executing Commands
Used with the -t option, xargs echoes each command before executing. For example, the following command moves all files in dir1 into the directory dir2.
$ ls dir1 | xargs -i -t mv dir1/\{\} dir2/\{\}
mv dir1/f1 dir2/f1 
mv dir1/f2 dir2/f2 
mv dir1/f3 dir2/f3
Used with the -p option, xargs prompts the user before executing each command. For example,
$ ls dir1 | xargs -i -p mv dir1/\{\} dir2/\{\}
mv dir1/f1 dir2/f1 ?...y
mv dir1/f2 dir2/f2 ?...n
mv dir1/f3 dir2/f3 ?...y
Files f1 and f3 were moved but file f2 was not.
Use the query (-p) option, to choose which files in the current directory should be compressed.
$ ls | xargs -n1 -p compress
compress largef1 ?...y
compress largef2 ?...y
compress smallf1 ?...n
compress smallf2 ?...n

Notes:

Those pages are written by people for whom English is not a native language. Some amount of grammar and spelling errors should be expected.
This is a Spartan WHYFF (We Help You For Free) site. It cannot replace the best teachers and the best books.
The site contain some obsolete pages as it develops like a living tree... Some links on older pages are broken. Please try to use Google, Open directory, etc. to find a replacement link (see HOWTO search the WEB for details). We would appreciate if you can mail us a correct link.

Search Amazon by keywords:
Open directory	Research Index

Old News ;-)

Re GNU enhancement -- $(xargs cmd, list)

From: Reid Madsen

Subject: Re: GNU enhancement -- $(xargs cmd, list)

Date: Mon, 6 Nov 2000 19:42:10 -0600 (CST)

 > Date: 6 Nov 2000 16:56:30 -0600
 > From: Reid Madsen <address@bogus.example.com>
 > 
 > I also thought about using the _POSIX_ARG_MAX and ARG_MAX symbols to 
 > determine
 > when the xargs approach was really needed.  If the command line is within
 > limits, then
 > 
 >         $(xargs cmd, list)
 > 
 > would expand to:
 > 
 >         cmd list
 > 
 > Do you believe this is doable/portable?
 > 
 > Reid
 > 
 > 
 > FYI, a quick perusal of our systems shows:
 > 
 >         POSIX_ARG_MAX   ARG_MAX
 >         -------------   -------
 > Solaris         4096    1MB
 > HP              4096    2MB
 > AIX             4096    24576   Pathetic!
 > OSF             4096    38912   Pathetic!
 > NT              4096    14500   Pathetic ** 2 !
 > 
 > Gotta love those 2-4 byte per arg limits on AIX, OSF, and NT.

After experimenting with this on Solaris for about 3 hours, I've come to the
conclusion that this is way to complicated.  The 'exec' man page states that
the error E2BIG is returned when the sum of the command length and environment
length exceeds ARG_MAX.  So, armed with that information I when out and
created a Makefile with enough big arguments to push it right up to the edge.
Adding a single character would case make to fail with:

        make: execvp: num_args: Arg list too long

However the 'edge' was nowhere close to the size of the command line +
environment.  It's off by about 100K.  Yuck!

So, instead, I've decided that in addtion to the $(XARGS) variable that
GNUmake should also define a $(XARGS_LEN) variable.  Which will be used to
measure the length of the command line.  When less the $(XARGS_LEN), then a
normal command is launched, otherwise xargs is launched as"

        xargs -s $(XARGS_LEN) cmd << tmpfile

In our build process, based on IMAKE, we can set XARGS_LEN differently for the
different platforms we build on.  IMO, this is probably the most flexible.

If you have a better idea, I'd be glad to hear it.

Reid

-- 
Reid Madsen                             address@bogus.example.com                      
Senior Member, Tech. Staff              (972) 536-3261 (Desk)
I2 Technologies                         (214) 850-9613 (Cellular)
--
Making incrementally (without 'clean') since May 18, 2000.
Long live incremental make!

comp.unix.aix Frequently Asked Questions (Part 2 of 5)

Solution #3
-----------
   mount | grep jfs | cut -c27- | cut -d" " -f1 | \
     xargs -i backup -${LEVEL} -u -f /dev/rmt1.1 {} > ${DATE}.backup 2>&1

News	Books	See also	Recommended Links	Options	Examples	Additional Examples
Shells	Pipes	AWK	Perl	grep	find	Regex
sort	uniq	cut	tee	History	Humor	Etc