Softpanorama
May the source be with you, but remember the KISS principle ;-)

Contents Bulletin Scripting in shell and Perl Network troubleshooting History Humor

Unix find tutorial

Prev | Contents | Next

Contents

  1. Introduction
  2. Find search expressions
  3. Finding files using name or path
  4. Finding files by age
  5. Using -exec option and xargs with find
  6. Finding SUID/SGUID files
  7. Finding World Writable, Abandoned and other Abnormal Files
  8. Finding Files based on size: largest, empty and within certain range
  9. Additional ways of controlling tree traversal
  10. Using find for backups
  11. Examples of Usage of Unix Find Command
  12. Typical Errors in using find
  13. Summary
  14. Webliography

Part 5: Using -exec option and xargs with find

Introduction

Find is capable to perform several actions on the files or directories that are found with -exec and -execdir options (the latter is "GNU find only" feature). At the same time it is a perfect tool to destroy your filesystem or two as option -exec blindly and very quickly executes commands you specified for the set of files provided by find, which might be not the set you expected. The stress here is on "very quickly", which is especially noticeable in case of -exec /bin/rm {} \; command with wrong file set ;-)

Unix system administrators folklore contains many horror stories of wiping out important filesystems by misunderstanding what set of files will be affected. The first rule of running -exec option with more of less complex find command is to replace it with -ls option and visually inspect the result file set.  This will take five or 10 min which is much shorter then several hours (or days) of desperate efforts to recover from the damage inflicted by some unforeseen side effect or complex find command.  Even typos can be deadly in this case.  You are warned.

More specialized options as alternative to -exec in some common cases

The first rule is to be aware about several more specialized options:

Options exec and execdir

Options -exec and -execdir  command execute the specified command for each file found. This is the most powerful (and thus dangerous) options that find provides.  The difference between them is that the latter provides absolute path to the file and as such is safer.

Please be aware that -exec option has some notorious side effects if used incorrectly. Making backup of filesystem before doing something complex is highly recommended. In case of usage of destructive commands consider it to be a surgery and test if the set of files is correct first with -ls or other non-destructive command.

Both  -exec and -execdir  options uses parameterless macro {}  which is expanded to the current file. More correctly, macro {} is expanded to a relative path starting with the name of one of the starting directories, rather than just the basename of the matched file.

You can use several instances of {} in the command: GNU find  replaces {} wherever it appears.

For more complex things post processing of output of find command with xargs  is a safer option as you first write it wont to a file, check the output and only then run xargs on file preventing running some potentially irreversible action on files beyond the subset you intended to process...

The option -execdir  is a more modern option introduced in GNU find  is an attempt to create a more safe version of -exec. It has the same semantic as -exec with two important enhancements:

Here is a relevant quote from the man page:

Execute command; true if zero status is returned. find  takes all arguments after -execto be part of the command until an argument consisting of ; is reached. It replaces the string {} by the current file name being processed everywhere it occurs in the command. Both of these constructions need to be escaped (with a \) or quoted to protect them from expansion by the shell. The command is executed in the directory in which find  was run.

For example, to compare each C header file in or below the current directory with the file /tmp/master:

find . -name '*.h' -execdir /bin/diff -u '{}' /tmp/master ';'

If you use -execdir, you must ensure that the $PATH variable contains only absolute directory names. Having an empty element in $PATH,  explicitly including .(or any other non-absolute name) is insecure. GNU find will refuse to run

That said, old habits die slowly, and the usage of  -exec option dominates in literature including this site. But you need to understand that all examples for find with -exec option will work after substitution -exec option to -execdir, making them safer with so little effort that it worth adopting as you standard practice.

All examples for find with -exec option will work after substitution -exec option to -execdir, making them safer. Please use -execdir option as default.

The -exec option is a powerful and dangerous tool

While -print and -ls options are pretty benign, the -exec option is a powerful and dangerous tool. Unix system administrators folklore contains many horror stories of wiping out important filesystems by misunderstanding what set of files will be affected.

The rule No.1 is using -exec option is very simple: unless you enjoy the situation commonly called SNAFU, always test find command containing exec using -ls option instead of -exec ( or -execdir) to see if the files selected are the files you really wish to process.

Never use -exec or -execdir option in a hurry or under pressure. Always test correctness of selected files with -ls option first before running "destructive" command on them

SEVEN COMMANDMENTS OF USING   -exec or -execdir OPTION

  1. Unless you really need to proceed the whole subtree use -maxdepth 1 to prevent getting extra files in results.
  2. Always use  the option -0  with  xargs command, if you supply the list of files generated by find to it. Correspondingly always use option -print0 of find command to generate such a list. It prevent mistreating files with spaces in the name (which typically comes from Windows environment)  option 
  3. The command that you want to execute needs to contain a special macro argument {}, which will be replaced by the matched filename on each invocation of -exec or -execdir predicate.  You can use {} multiple times in command and each time.  It will evaluate to the same file and path each time you use it.

  4. You need to specify \; (or ';' ) at the end of the command. (If the \ is left out, the shell will interpret the ; as the end of the find command.)

    For example, the following two commands are equivalent:

    find . -name "*rc.conf" -execdir /bin/chmod o+r {} \;
    find . -name "*rc.conf" -execdir /bin/chmod o+r {} ';' 

    NOTE: In case {} macro parameter is the last item in the command then it should be a space between the {} and the \;. For example:

    find . -type d -execdir /bin/ls -ld {} \; 
  5. If your macro file substitution argument is the last in generated command you can use {} + instead of {} \;  . In this case find will process multiple arguments in xargs fashion groupig multiple arguments in a single command.
  6. If you attempt to make changes that involve system directories it is better to do it in two stages. first create a file with the list of changes using find and verify that it is accurate. Then use xargs with option -p (see below) to process this file.
  7. In case of deletion of the file GNU find  has option -delete which is safer then "-execdir /bin/rm {} \;". For example find / -name core -delete 

    In case of deletion of the file GNU find has option -delete which is safer then " -exec /bin/rm {} \; ". For example find / -name core -delete
    There is classic problem of using rm  -- among file you are trying to delete might be one or several  filenames with spaces, for example files that migrated to Unix filesystem from Windows. In windows, unfortunately, using spaces in filenames is a common practice.

    For example you can accidentally delete all the documents that ends with "doc", by writing :

    find /mnt/zip -name "*doc copy" -delete

    If have several hundred of  important Word document in this folder this is akin to fire inthe library, and you only can imagine the size of the disaster if you backup proved to be unreliable. Especially with filesystem for which undelete tools do not exist. You only option is to switch to runlevel 1 and take dump of the disk with dd and then painstakingly try to locate deleted file one by one. See dd  and Recovery of lost files using DD

    There are several way to prevent this nasty error:

Again it is better to experiment first to see if everything is right if you deal with important files. Five minutes of testing can save five or more hours of desperate attempts to recover accidentally deleted files.

Here are examples of "good practices" of using find. We will use chmod as the base of examples. Many people do not think about commands like chmod or chown as particularly dangerous, but applied to root filesystem they can be pretty devastating. Please note that we first get to the target directory using cd and only then are using find  command with "." (dot) argument. This avoids such unpleased situation as typing "/ etc" instead of "/etc".

Or worse "/etc" instead of local etc directory (the intention was to get to local etc directory but string "/etc" is hardwired in sysadmin brains and this slip costs many sysadmins tremendous pain):

Test command:

find . -type f -ls

Final command:

find `pwd` -type f -execdir /bin/chmod 500 {} ';'
The command bellow search in the current directory and all sub directories and change permissions of each file as specified. Here an additional danger is connected with being in a wrong directory.

Test command:

find . -name "*rc.conf"  -ls

Final command:

find `pwd` -name "*rc.conf"  -execdir /bin/chmod o+r {} \;

This command will search in the current directory and all sub directories. All files named *rc.conf will be processed by the chmod -o+r command. The argument {} is a macro that expands to each found file. The \; argument indicates the -exec argument has ended. You can use ';' instead:

find `pwd` -name "*rc.conf" -execdir /bin/chmod o+r {} ';' 

The end results of this command is all *rc.conf files have read bit set in "other" permissions.

The find  command is commonly used to remove core files that are more than a few 24-hour periods (days) old. These core files are copies of the actual memory image of a running program when the program dies unexpectedly. They can be huge, so occasionally trimming them is wise:

Test command:

find . -name core -ctime +4 -ls

Final command:

find `pwd` -name core -ctime +4 -execdir /bin/rm -f {} \;
For grep the /dev/null  argument can by used to show the name of the file along with the text that is found. Without it, only the text found is printed. An equivalent mechanism in GNU find  is to use the "-H" or "--with-filename" option to grep:
find /tmp -execdir /usr/bin/grep "searchstring" {} /dev/null \; -print

An alternative to -exec / -execdir option is piping output into xargs command which we will discuss in the next section.

Limitation on the number of files passed as the parameters  in Unix environment

Many users frequently ask why xargs  should be used when shell command substitution archives the same results. Take a look at this example:

grep foo `find /usr/src/linux -name "*.html"`

The drawback with commands such as this is that if the set of files returned by find  is longer than the system's command-line length limit, the command will fail.

One way to solve this problem is to use xargs. This  approach gets around this problem because xargs  runs the command as many times as is required, instead of just once.

But ability of xargs to use multiple argument can be a source of the problems too. For example

find . -type f -name "*.java" -print0 | xargs -0 /bin/tar cvf myfile.tar

Here the attempt is made to create a backup of all java files in the current tree: But if the list length for xargs to invoke the /bin/tar command is too  big, xargs will split it into multiple command, and subsequent tar commands will overwrite previous tar archives. As a result archive will contain a fraction of files, but without testing you might discover this sad side effect too late.

To solve this problem you can use either file with the list of files to include in the archive (tar can read a list of files from the file using option -T) or option "-r" which tells tar to append to the archive (option '-c' means "create"):.

find . -type f -name "*.java" | xargs /bin/tar rvf myfile.tar

Feeding find  output to pipes with xargs

One of the biggest limitations of the -exec option (or predicate with the side effect to be more correct) is that it can only run the specified command on one file at a time.

Always check the correctness of the list of the files selected by  find command

The xargs command (or parallel  command which can be used as xargs substitute) solves two problems

Note: Find option -print0 prints list of filenames with null character (\0) instead of whitespace as the output delimiter between pathnames found. This is a safer option if files can contain blanks or other special characters if you use find  with xargs (the -0 argument is needed in xargs.). 

You can also filter output using additional grep stage of pipeline before xargs

cat iplist | /usr/bin/grep "^10.192.15"  | xargs -n1 nmap -sV

By default  xargs places input arguments at the end of each generated command. In this case you do not need to use file placeholder macro '{}' like in option -exec.

Option - i

Option  -i  in xargs provides explicit specification of a symbol or a grup of symbols used to denote a parametlesss macro (macto substitution string). In find as we know this is fixed to '{}'. In xargs you can specify you own macro substitution string. Option -i  requires one parameter -- current file macro placeholderIf you use as this macro special characters that have special meaning in shell, you need to put them in single brackets or use a backslash (\) before each bracket to keep the shell from interpreting the special characters.   

For example "^" is more convenient in most cases then {} as this is rarely used symbol: 

find . -maxdepth 1 -type d -print | xargs -i '^' echo Directory: '^'

Another useful option that is missing in find is the -n  option, which makes xargs run the specified command each tine with exactly N arguments.

Problem of files with with names that contain spaces

One common problem is that without special precautions files with names that contain spaces will be treated by default will be treated as multiple arguments. 

As we mentioned before the option -0 prevent mistreating files with spaces in the name (which typically comes from windows environment) and should be used option  -print0 of find command

As we mentioned before the option -0 prevent mistreating files with spaces in the name (such files typically come from Windows environment) and should be used with option  -print0 of find command. You should always use this option with xargs. I mean always.

I would like to stress it again and again that this is a vital option which not only prevent "mistreatment" of  filenames with spaces, but also filenames with a single quote (for example, can't_open_display.txt ) and several other important cases.  As there is a pretty high chance to encounter such a file in any large set of files in modern Unix environment, you should always use this option. I mean, always

In you feed xargs from find command  you need to use  -print0  in find and option -0 to xargs command.

This way you can avoid the danger to processing wrong file name with blanks as multiple files with potential catastrophic consequences if you use some destruction command iether in  -exec or xargs:

find /mnt/zip -name "*prefs copy" -print0 | xargs -0 -p /bin/rm

Using option -p you can provide manual confirmation of each action. The reason is that xargs runs the specified command on the filenames from its standard input, so interactive commands such as cp -i, mv -i, and rm -i (which are often aliased as cp, mv and rm, respectively) don't work right. For the same reason you need to provide the path to the executable, such as rm to make find work right. 

So when you run the command first time you can use this option as a safety valve. After several operations with confirmation to which you answered NO you can cancel the command and run without option -p. The -p option solves the problem of somee typo that you do not noticed but that dramatically affects what find or xargs is doing.  In the preceding example, the -p option would have makes the initial run safer because you  could answer no to each prompt and then rerun the command  without option -p

People are doing pretty complex staff this way. For example (Ubuntu Forums, March 23rd, 2010)

FakeOutdoorsman

I'm trying to convert Nikon NEF images to jpg. Usually I use find and xargs for batch processes like this for example:

Code:

find . -name "*.flac" -exec basename \{\} .flac \; | xargs -i ffmpeg \
	-i \{\}.flac -acodec libvorbis -aq 3 \{\}.ogg

However, my latest attempt is giving me no output because I can't seem to get xargs to work with pipes. An individual set of commands works:

Code:

dcraw -w -c MEY_7046.NEF | convert - -resize 25% MEY_7046.jpg exiftool \
	-overwrite_original -TagsFromFile MEY_7046.NEF MEY_7046.jpg dcraw -z MEY_7046.jpg

A nice set of commands, but not yet practical for converting a DVD with multiple directories. My truncated find-isized version does nothing:

Code:

find . -name "*.NEF" -exec basename \{\} .NEF \; | xargs -i dcraw -w -c \{\}.NEF | convert - -resize 25% \{\}.jpg

Any ideas of where I'm going wrong?

diesch:

That pipes the output of all the dcraw runs together into one convert call.

Try

Code:

find . -name "*.NEF" -exec basename \{\} .NEF \; | xargs -i sh -c 'dcraw -w -c $0.NEF | convert - resize 25% $0.jpg'

In this example you can also use -0 argument to xargs.

The problem of the last file when using grep

As we mentioned when the xargs is used with grep, or other command the latter it will be getting multiple filenames. If grep gets multiple arguments it automatically includes the filename of any file that contains a match. Still for grep you do need  option -H (or addition /dev/null to the list of files) as the last "chunk" of filenames can contain a single file.

Searching for "lost" files

When used in combination, find, grep, and xargs are a potent team to help find files lost or misplaced anywhere in the UNIX file system. This is important and recurrent problem with modern filesystem, which often contain thousands of files and I strongly encourage you to experiment further. This is a vital sysadmin skill that is really necessary in the current environment. Even directories like /etc in modern Unixes contain way to many files and often you do not remember whether  the necessary config file in /etc  directory or in one of subdirectories like /etc/ssh 

If a regular file is lost it is important to think about distinctive criteria that you can use to find it. It might be not only name but some string within it, date of last modification, size or any other attribute. the more precise in search the better are your chances to find the file.  Often you need to experiment with different criteria to achieve a useful result. Even if file is not found because it was accidentally deleted the content of it might still be present on the disk. In this case just do  dd dump of the whole disk and search it for some unique string. 

Timing your search

With SSD disks find command for root filesystem is almost instant.  In other cases searching using find is an interesting indicator of the speed of the filesystem. It shown clear difference between 15K RPM drives and 10K RPM drives of the same size. It also shows that large size 10K RPM drives beat smaller size 15 RPM drives. In any case it is a very interesting and  revealing test of the i/o subsystem and the filesystem used (ext3 is actually nor a bad filesystem for large number of relatively small files).

Also you can use time command to see the dramatic difference in speed of find with the -exec option vs. results piped to xargs. In the simplest form you can do it the following way:

time find /usr/src -name "*.html" -execdir /usr/bin/grep -H "foo" {} ';' | wc -l
time find /usr/src -name "*.html" | xargs /usr/bin/grep -l "foo" | wc -l

On any substantial set of files xargs works considerably faster. The difference becomes even greater when more complex commands are run and the list of files is longer.

Gotchas

The -exec option in find  command is a very sharp tool. Below we'll present some of the horror stories (see also Typical Errors In Using Find). Such errors are often made under time pressure or when the person is very tied and situation awareness is low.

Please remember that five minutes of testing usually can save five or more hours of desperate attempts to recover from the results of incorrectly run find  command with the option -exec  that contain some 'destructive' action.

Please remember that five minutes of testing usually can save five or more hours of desperate attempts to recover from the results of incorrectly run find  command.

Typically "find blunders" are committed when a complex find  command that changes the files in a certain subtree using rm, chown, or chmod command is constructed and run without any testing. So in many cases this is a direct result of recklessness of sysadmin. Sometimes it is result of time pressure, or being extremely tired (in this situation people often try to cut corners, even if they understand the risk). It also can be result of the lack of situational awareness (like many errors of pilots) due to information overload or other factors.

Often you just can't foresee the results of particular find command without testing. For example, sometimes the directories that are used contain symbolic links to directories in other part of filesystem and "find start running wild" on subtree that you never intended it to run. Sometimes the pattern that you use has unintended side effect. Sometimes it just a silly typo.

It's always safer to create list of files to which you apply the particular command inspect it carefully and only then as a separate state execute command for each of those files. And by "inspect if carefully" I mean to run several grep commands to detect two the most disastrous "hidden" cases:

Life of sysadmin is a complex one so little testing does wonders in preventing nasty surprises from overconfidence in your own abilities :-).

Here are some pretty telling examples:

xargs gotchas

Both Unix and GNU xargs options procession is badly written and does not use lexical scanner to process them.  This is especially dangerous with option -i

You should already use -print0 option in find and -0 option in xargs to avoid this error. Also this error arise when -i option is used without argument (should be at least -i{}  -- no space between -i and {})

If you cat list of files into xargs, use tr to translate \n to 0, but always use option -0 with xargs.  The problem is that  xargs is somewhat capricious and is sensitive to quotes and spaces in filenames. Without option -0 it will complain about single quote in filename, but will process filenames with blanks possibly leading to disastrous consequences.

 find + xargs argument line too long error

In case you put a space after -i argument this error can is observed with -print0  and -0 options which is pretty unexpected and put your troubleshooting off track. See the discussion below to get better understanding of this gotchas.

Unix & Linux Stack Exchange

 Amelio Vazquez-Reina:

I have a line like the following:

find /foo/bar -name '*.mp4' -print0 | xargs -i {} -0 mv -t /some/path {}

but I got the following error:

xargs: argument line too long

I am confused. Isn't the use of xargs supposed to precisely help with this problem?

Note: I know that I can techincally use -exec in find, but I would like to understand why the above fails, since my understanding is that xargs is supposed to know how to split the input into a manageable size to the argument that it runs. Is this not true?

This is all with zsh.

/ xargs / arguments

slm

Well for one thing the -i switch is deprecated:

-i[replace-str]

This option is a synonym for -Ireplace-str if replace-str is specified.

If the replace-str argument is missing, the effect is the same as -I{}.

This option is deprecated; use -I instead.

So when I changed your command around to this, it worked:

$ find /foo/bar -name '*.mp4' -print0 | xargs -I{} -0 mv -t /some/path {}

Example

$ find . -print0 | xargs -I{} -0 echo {}

./.sshmenu
./The GIT version control system.html
./.vim_SO
./.vim_SO/README.txt
./.vim_SO/.git
./.vim_SO/.git/objects
./.vim_SO/.git/objects/pack
./.vim_SO/.git/objects/pack/pack-42dbf7fe4a9b431a51da817ebf58cf69f5b7117b.idx
./.vim_SO/.git/objects/pack/pack-42dbf7fe4a9b431a51da817ebf58cf69f5b7117b.pack
./.vim_SO/.git/objects/info
./.vim_SO/.git/refs
./.vim_SO/.git/refs/tags
...

Use of -I{}

This approach shouldn't be used since running this command construct:

$ find -print0 ... | xargs -I{} -0 ...

implicitly turns on these switches to xargs, -x and -L 1. The -L 1 configures xargs so that it's calling the commands you want it to run the files through in a single fashion.

So this defeats the purpose of using xargs here since if you give it 1000 files it's going to run the mv command 1000 times.

So which approach should I use then?

You can do it using xargs like this:

$ find /foot/bar/ -name '*.mp4' -print0 | xargs -0 mv -t /some/path

Or just have find do it all:

$ find /foot/bar/ -name '*.mp4' -exec mv -t /some/path {} +

slm

Amelio Vazquez-Reina Jul 21 '13 at 4:20

Thanks! When you said "This approach shouldn't be used" which approach should be used instead then? Would

 "find /foot/bar/ -name '*.csv' -print0 | xargs -0 mv -t some_dir'"

be a better solution?

If so, how does xargs know in this case where in the mv command to feed in the arguments it gets from the pipe? (does it always place them last?)

slm, Jul 21 '13 at 6:54

@user815423426

Doing it with just the find ... -exec ... is a better way or if you want to use xargs the find ... | xargs ... mv -t ... is fine too.

Yup it always puts them last. That's why that method needs the -t.

Gilles

The option -i takes an optional argument. Since you put a space after -i, there was no argument to the -i option and therefore the subsequent -0 was not an option to xargs but the second of 6 operands {} -0 mv -t /some/path {}.

With only the option -i, xargs expected a newline-separated list of file names. Since there was probably no newline in the input, xargs received what looked like a huge file name (with embedded null bytes, but xargs didn't check that). This single string containing the whole output of find was longer than the maximum command line length, hence the error "command line too long".

Your command would have worked with -i{} instead of -i {}. Alternatively, you could have used -I {}: -I is similar to -i, but takes a mandatory argument, so the next argument passed to the xargs is used as the argument of the -I option. Then the argument after that is -0 which is interpreted as an option, and so on.

However, you shouldn't use -I {} at all. Using -I has three effects:

Either drop -I and -i altogether

find /foo/bar -name '*.mp4' -print0 | xargs -0 mv -t /some/path {}

or drop xargs and use -exec:

find /foo/bar -name '*.mp4' -exec mv -t /some/path {} +

Here is another post where pretty basic users discuss the same problem and steo on the same rake How to resolve the '-bin-rm Argument list too long' error

February 18, 2009 | Simple Help

Sukrit Dhandhania on 

root@dwarf /var/spool/clientmqueue # rm spam-*
/bin/rm: Argument list too long.

Ever seen this error in Linux when you have too many files in a directory and you are unable to delete them with a simple rm -rf *? I have run into this problem a number of times. After doing a bit of research online I came across a neat solution to work around this issue.

find . -name 'spam-*' | xargs rm

In the above instance the command will forcefully delete all files in the current directory that begin with spam-. You can replace the spam-* with anything you like. You can also replace it with just a * if you want to remove all files in the folder.

find . -name '*' | xargs rm

We have covered the Linux find command in great detail earlier. Xargs is Linux command that makes passing a number of arguments to a command easier.

LetsTalkTexasTurkeyHere

I got this error from my RaspberryPi trying to erase a large amount of jpeg images from the current working directory.

ls | grep ".jpg" | awk '{print "sudo rm "$0}' | /bin/sh

worked fine for me.

redesigned

ls | xargs rm

(works even for those shared hosts that block access to find, like nexus)

Kevin Polley

Big thanks - find . -type f -print0 | xargs -0 /bin/rm saved the day for me with an overflowing pop acct

Michael T

Good catch using the print0 option, that's an important one.

Most find commands do not require the "-name" predicate. What's usually more important is to make sure you're deleting *files* and not something else you might not have intended. For this use "-type f" inplace of the "-name" option....

find . -type f -print0 | xargs -0 /bin/rm

A) Use the full path to the 'rm' command so your aliases don't muck with things.
B) Check your xargs command, you can sometimes, if needed, tell it to use one "result" at a time, such as (if you didn't use print0 but regular print) "-l1"

Problem of files with with names that contain spaces

One common problem is that without special precautions files with names that contain spaces will be treated by default will be treated as multiple arguments.

As we mentioned before the option -0 prevent mistreating files with spaces in the name (which typically comes from windows environment) and should be used option  -print0 of find command

As we mentioned before the option -0 prevent mistreating files with spaces in the name (such files typically come from Windows environment) and should be used with option  -print0 of find command

I would like to stress it again and again that this is a vital option if you can have filenames with spaces in you filesystem. As there is a pretty high chance to encounter such a file in any large set of files in modern Unix environment.

I recommend using it as the default option. That means always.   If you add option -print0 to find command and option -0 to xargs command, you can avoid the danger to processing wrong file name with blanks as multiple files with potential catastrophic consequences if you use some destruction option in -exec or xargs:

find /mnt/zip -name "*prefs copy" -print0 | xargs    -0 -p /bin/rm

Using option -p you can provide manual confirmation of each action. The reason is that xargs runs the specified command on the filenames from its standard input, so interactive commands such as cp -i, mv -i, and rm -i (which are often aliased as cp, mv and rm, respectively) don't work right. For the same reason you need to provide the path to the executable, such as rm to make find work right. 

The problem of the last file when using grep

As we mentioned when the xargs is used with grep, or other command the latter it will be getting multiple filenames. If grep gets multiple arguments it automatically includes the filename of any file that contains a match. Still for grep you do need  option -H (or addition /dev/null to the list of files) as the last "chunk" of filenames can contain a single file.

Limitation on the number of files passed as the parameters  in Unix environment

Many users frequently ask why xargs  should be used when shell command substitution archives the same results. Take a look at this example:

grep foo `find /usr/src/linux -name "*.html"`

The drawback with commands such as this is that if the set of files returned by find  is longer than the system's command-line length limit, the command will fail.

One way to solve this problem is to use xargs. This  approach gets around this problem because xargs  runs the command as many times as is required, instead of just once.

But ability of xargs to use multiple argument can be a source of the problems too. For example

find . -type f -name "*.java" | xargs    /bin/tar cvf myfile.tar

Here the attempt is made to create a backup of all java files in the current tree: But if the list length for xargs to invoke the tar command is too  big, xargs will split it into multiple command, and subsequent tar commands will overwrite previous tar archives. As a result archive will contain a fraction of files, but without testing you might discover this sad side effect too late.

To solve this problem you can use either file with the list of files to include in the archive (tar can read a list of files from the file using option -T) or option "-r" which tells tar to append to the archive (option '-c' means "create"):.

find . -type f -name "*.java" | xargs    /bin/tar rvf myfile.tar

Important Tips

Most of those were already mentioned. But there is a vlue of grouping them in one place for reference:

Tip 1: Always you absolute path to command you are trying to execute

TIP 2: Always put {} macro in double quotes.

Tip 3: Use tr command instead of  cat command to pipe input file into xargs

tr "\n" "\0" < infile | xargs -0 .... 

Tip 4: Always try to use -execdir instead of -exec. It is safer option as it provide absolute path to the file

Tip 5: For large number of processed file inspect the list of file before running any "destructive" command

Tip 6: If your macro file substitution argument is the last in generated command you can use "{} +" instead of {} \;  . In this case find will process multiple arguments in xargs fashion.

 How to use the exec option in find with examples

Linuxaria
Geoff, 09/25/2012 at 08:43

All of the -exec example end with "{} \;" which means they would be more efficient and faster if they ended with "{} +" instead. Using ‘+’ instead of ‘;’ makes find aggregate pathnames and execute far fewer commands, instead of one command for each pathname.

You can’t use a ‘+’ if the last command argument is not "{}", for example you can’t do:

find . -name "*.old" -exec mv {} oldfiles + # doesn’t work

but there is a way around that involving the shell:

find . -name "*.old" -exec sh -c ‘mv "$@" oldfiles’ sh {} +

This uses two process per aggregated set of pathnames, but is still way more efficient than:

find . -name "*.old" -exec mv {} oldfiles \;

if there are more than a couple of files.

James Youngman, 09/27/2012 at 00:40

Ed wrote:

I am surprised you didn’t mention that that the -exec option can over flow the command line if find returns too many objects

That should not happen, which is the whole point of -exec. If you can reproduce this problem with GNU find, please report it (with clear, reproducible instructions on how to reproduce the problem!) as a bug.

Geoff, 09/27/2012 at 08:05

Ed, you are mixing up two different problems. The overflow problem is with -print and command substitution.

The problem with -exec, as stated in the article you referred to, was efficiency. The original solution to that was xargs, but these days find has that functionality built in – you use ‘+’ instead of ‘;’ to terminate the -exec command – and is preferred because of the problems xargs has with spaces and other special characters in filenames.

(GNU solved the xargs problem a different way by inventing find -print0 and xargs -0, but those aren’t as widely implemented as find’s "-exec command {} +", and they are less efficient because of the extra xargs process and the I/O through the pipe.)

Prev | Contents | Next




Etc

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes.   If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner. 

ABUSE: IPs or network segments from which we detect a stream of probes might be blocked for no less then 90 days. Multiple types of probes increase this period.  

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least


Copyright © 1996-2015 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down currently there are two functional mirrors: softpanorama.info (the fastest) and softpanorama.net.

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: February 04, 2015;