Softpanorama

May the source be with you, but remember the KISS principle ;-)
Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and  bastardization of classic Unix

Softpanorama Bulletin. Vol 31, No. 01 (January, 2019))

Bulletin 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007
2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018
2019                    
Jan Feb Mar Apr May Jun Jul Sept Oct Nov Dec

Creative uses of rm

Unintended consequences, or wiping out much more then you intended to do

Version 1.8 (Aug 27, 2019)

Copyright 2006-2019, Nikolai Bezroukov

NOTE: This is copyrighted unpublished work based on original research by the author and presented as a preprint. All right reserved.


Introduction

The most enduring principles in the universe are accident and error

SNAFU is an acronym that is widely used to stand
for the sarcastic expression Situation Normal: All Fucked Up.
It is a well-known example of military acronym slang.

SNAFU - Wikipedia

FUBAR (Fucked/Fouled Up Beyond All Recognition/Any Repair/All Reason)
List of military slang terms

It is much better to read this page before you do something with rm then after you committed some inexcusable blunder :-)

It is much better to read this before you do something with rm then after you committed some inexcusable blunder :-).

Many Unix sysadmin horror stories are related to unintended consequences such as side effects of this classic Unix command. Also rm has several idiosyncrasies which are iether easily forgotten or unknown by novice sysadmins.  Usually the disaster strikes due to some unfortunate and unique confluence of factors.  And, of course, 99.999% deletions using rm are successful and do not produce any side effects. This page is about remaining  0.001% or 0.0001% which Anatoly Ivasyuk aptly named "Creative uses of rm" .  They are both very rare and very destructive.

The main problem is that mishaps/SNAFU with rm happen very rarely and people generally do not expect any troubles. Of course, seasoned sysadmins with decades of experience have their set of scars from blunders connected with misused  rm commands, but the problem is that as such incidents happen very rarely (once in a couple of years or even at larger interval), the lessons learned are soon forgotten.  As Benjamin Franklin  quipped that "Experience keeps a dear school, but fools will learn in no other." But this saying is not applicable in this particular case: fool or no fool, if something bad happened three years ago you probably do not remember critical details now, or at the time you write your "one-time" scripts after running of which you discover some strange side effects ;-) In such cases you simply can't learn from experience -- you need specialized and regular safety training.  Please consider rm as dangerous as a sharp blade, or a chainsaw.

You can commit rm blunders and experience corresponding SNAFU both directly on the command line and in your "one-time" scripts (usually shell or Perl scripts). The latter can be even more destructive then the command line blunders because on command line you usually realize that something is deeply wrong in a couple of seconds. In script it can take much longer than that, or not detected until its execution finished. 

One aggravating factor is that rm -r is very fast on modern servers. It is I/O bound so the speed and number of cores does not matter much but presence of SSD greatly enhanced the damage. Even if the resulting list of files and directories is very long (which suspicious in itself and logically would need specifying an additional option like -f to execute without prompt) , rm starts working of it immediately one by one, left to right, unless you specify additional parameters (see -I below). On a server with SSD disks you can easily wipe something between 200GB to one terabyte (depending on the average size of files)  just in several seconds which takes you to realize that the command that you submitted was very wrong.  Around 10-20K files can be wiped in 5 sec -- the time usually needed to realize that submitted command is wrong and cancel it.

As rm actions change data stamp on directories affected by its actions. In case of deeply nested trees with large number of files in deeply nested directories you can see the damage in directories. Damaged directories  with wiped out subdirectories  now have current modification timestamp. Sometimes that to analyze the net result of this wild ride of rm against your data and helps to understand the scope of damage. 

On a server with SSD disks you can easily wipe something between 200GB to one terabyte (depending on the average size of files) just in several seconds which takes you to realize that the command that you submitted was very wrong.  Around 10-20K files can be wiped in 5 sec -- the time usually needed to realize that submitted command is wrong and cancel it.

NOTE: it make sense to name directories with valuable data in large data trees Xcritical not Critical so that hopefully in case of SNAFY they were picked up later. 

In any case the only way to keep your "safety training" in rm blunders current is to use your own log and such pages as this one. they should be at the core of self study that you need to repeat at least once a year or before writing script with recognizes important data (for example converts folders in tar.gz files). 

In any case the only way to keep your "safety training" in rm blunders current is to use your own log and such pages as this one. they should be at the core of self study that you need to repeat at least once a year or before writing script with recognizes important data (for example converts folders in tar.gz files).

It is much better to read this before you do something with rm then after you committed some inexcusable blunder :-)

In any case the only way to keep your "safety training" in rm blunders current is to use your own log and such pages as this one. they should be at the core of self study that you need to repeat at least once a year or before writing script with recognizes important data (for example converts folders in tar.gz files).

One of the key problem here is too simplistic logic of rm command which definitely should  be re-designed in view of 50 year experience of using it in Unix. Some minor steps in this direction were made and  GNU rm does not allow to recursively deleted the root directory. But more can be done. That's why it makes sense to improve this  situation by using a wrapper, and alias it to rm command as those mishaps happened only in interactive sessions. I wrote my own Perl script saferm, while was initially based on safe-rm, but  then completely rewritten for the project were we need to deleted about 8TB of data located in a more then 1K of directories.   

Expansion of basic regular expression .* is one such case (its expansion includes .. which is a parent directory). This specifics of the behavior of rm -r .* can easily be forgotten from one encounter to another, especially if sysadmin works both with Unix and Windows servers. Consequences if executed as root are often tragic... And can cost a week of intensive work to fix.   

It is easy to forget that expansion of .* includes .. which is a parent directory, as such operation is performed rather rarely.

While most, probably 99.99%, or even 99.999% of operations with rm are successful, it is the remaining .001% that create a lot of damage and is subject to sysadmin folklore.

Generally, if you have enough space, it is better to move files first to the Trash folder then directly deleting them. 

In cases where there is not spare space to keep the files for a while using proper options also can help to limit the damage.  For example, as of April 2018Linux uses GNU implementation that has relatively new, and very useful option -I, which can replace older option -i as a better solution to the same problems. 

The utility rm  remains pretty Spartan, despite typical "option bloat" of many Linux utilities (see ls as an example). It has just a dozen of options, which each sysadmin should know by heart ;-). Here is the list:

-f, --force
ignore nonexistent files, never prompt
-i
prompt before every removal
-I
prompt once before removing more than three files, or when removing recursively. Less intrusive than -i, while still giving protection against most mistakes
--interactive[=WHEN]
prompt according to WHEN: never, once (-I), or always (-i). Without WHEN, prompt always
--one-file-system
when removing a hierarchy recursively, skip any directory that is on a file system different from that of the corresponding command line argument
--no-preserve-root
do not treat '/' specially
--preserve-root
do not remove '/' (default)
-r, -R, --recursive
remove directories and their contents recursively
-v, --verbose
explain what is being done
--help
display this help and exit
--version
output version information and exit

Two of those are unique to GNU version of rm and  are especially important:

TIP: never run rm without the option -I during the first execution of your one-time scripts

NOTE: By default rm -r does not follow symlinks to directories. But if used in  find -exec it does this via find traversing of directories.

Steps that you can do to reduce the chances of the major damage to the data

 Above all, not too much zeal! (Talleyrand’s warning to young diplomats)

Talleyrand

Taleyrand

Wiping out useful data by using rm with wrong parameters or options is probably the most common blunder committed both by novices and experienced sysadmins. The  only difference that experienced sysadmins usually have a backup.

In both case such a mishap happens very rarely, so unless you pay attention and organize for yourself some kind of monthly or quarterly "safety minutes" (which are typically a semi-useless monthly meetings typical for large corporate environment; they can be  made more useful for sysadmins, and IT staff in general,  if  they concentrate on sysadmin horror stores).

Without monthly or quarterly "safety minutes" you usually forget about the dangers that rm has and can accidentally step on the same rake again in a couple of years ;-).

Without monthly "safety minutes" you usually forget about the dangers that  rm has and can accidentally step on the same rake again ;-).  As working with any sharp instrument, this danger can't be completely avoided, but in addition to having an up-to-date backup, there are several steps that you can do to reduce the chances of the major damage to the data:

  1. If possible, do backup before any dangerous operation. Always do the backup of /etc on the server that you plan to work at the beginning of your working day. This can and probably should be done from dot scripts (such as .bash_profile)  on login.  Especially if you plan to delete files from such system directories such  as /etc,  /boot, /root, etc.  Or if the operation you intend to do is complex and requires traversing filesystem using of find -exec option or find -delete option. That's Rule No. 1
  2. Use wrapper such as saferm of safe-rm.  It really helps to prevent  many typical cases of accidental deletion of important system and user files.
  3. If you can't or do not want to use wrapper and plan to delete multiple files/directories using rm -r or  find -exec (see also Typical Errors In Using Find) please use non-destructive command like ls, tree, or find command   to check what files you are deleting first. Then visually verify correctness of your list.
  4. The same is true if you use regular expressions or traverse multiple filesystems. Please run ls, tree, or find command with the argument you intend to use in rm command  and  Wrappers for rm command do it automatically.
  5. It is much safer to delete individual files and directories using file manager like WinSCP, or Midnight Commander, then do this directly by typing on the command line.  In this case you have a very important visual feedback, and this less likely to make a blunder. Also you do not need to type the name of file or directory to be deleted, which also help to avoid the whole class of errors.
  6. When you are programming a simple script for file deletion, try to type in generated rm command (or any other destructive command) an absolute path to the directory on which you working with at least part of it specified expesidly like in rm /etc/myprogram/confi/*, not as rm $what_i_need_to_delete   In most cases writing commands like  rm $MYDIR is a very bad idea. Using some "safe" prefix is much better deal.  Something like rm /home/joeuser/$MYDIR This protects from cases when your variable for some reason is left undefined or you simply mistyped it pointing to the wrong directory.  One large class of rm blunders is when rm mistakenly operate on root,  or some system directory, or current directory.  Similarly, "chmod -R 755 /Backup/$mypath" is much safer then chmod -R 755 $mybackup
     
  7. For small amount of files you can use move to Trash folder instead of deletion, if you have enough space for files that you are deleting if makes sense to delay actual wiping files out as long as possible. You can delete them from the Trash folder later on.
     
  8. Use option -I instead of -i when deleting multiple files. First of all -i option (which it default alias in Red Hat and other distributions) is really annoying.  Moreover after a while the answer Yes became automatic, ingrained in your brain,  and that's a huge danger.  Many users start eventually simply disable it either by using option -f  or by specifying path, and thus avoiding using this alias. As option  -I is less intrusive it can be recommended as a better replacement. Unfortunately it does not solve that problem of reckless actions, but nothing probably can solve it. It prompts you only once before removing more than three files, or when removing recursively. See rm(1) remove files-directories:

    If the -I or --interactive=once option is given, and there are more than three files or the -r, -R, or --recursive are given, then rm prompts the user for whether to proceed with the entire operation. If the response is not affirmative, the entire command is aborted.

  9. To delete subdirectories and files starting with dot you can use "shopt -s dotglob". 
  10. Do not improvise complex, potentially destructive commands on the command line without necessity. History of command is your friend. If the command is complex or destructive it makes sense to write in the editor first, not directly improvise it on the command line.  Sometime while typing on command line  you can hit the Enter key or inserts a blank where you do not  intend. Using the editor help to avoid those blunders.

NOTES:

Use wrapper like saferm

There are two known to the author wrappers for rm:

Both need to be  aliased to rm to be more effective. At least for the root.  Aliasing rm to the wrapper is safe because aliases work only in interactive sessions.

Wrapper saferm was written based on horror stories contained in this page and was inspired by safe-rm. It uses a set of Perl regular expression  to protect system directories and user files, which are more powerful and flexible then prefix strings used is safe-rm.  Both are better then nothing. Experience  has  shown that the set of regular expression that you can supply to it in configuration file is pretty powerful tool. It also makes some common sense checks like presence of ".." in the expansion of the arguments.

All key system directories should be protected from accidental deletion, as removal some of them make recovery process more complex. In RHEL7 that list includes also symlinks in the root directory like /lib64. And it goes without saying  that /etc or /usr should be protected better than other directories, with more protected regex defined. 

In addition, using saferm you can block the deletion of directories that are in your private "protected directories and files" list (by default stored in  ~/Saferm.saferm.conf ) .  There are also some indirect methods of checking if the directory you are trying to delete is a wrong directory. For example, if you use level 2 directory /Fav for links some often used directories, you can convert this list into regular expression and that guarantee that those directories will be protected from accidental deletion.

Recovery from the situation in RHEL7, when you accidentally deleted all symlinks in root directory. For example using rm  [a-z]* command is a very good test of your sysadmin skills in this area.

Additional random observations:

Classic blunders with RM command

There are several classic blunders committed using the rm command :

  1. Running rm - R command with regular expression, without testing regex expansion first by using ls . You should always use ls -Rl command to test complex rm -R  commands (  -R, --recursive means process  subdirectories recursively).
  2. Mistyping complex path or file name in rm commands. It's always better to use copy and paste operation for directories and files used in rm command as it helps to avoid various typos. If you do recursive deletion is useful to have a "dry run" using find or ls to see the files.

    Here are to example of typos:

    rm -rf /tmp/foo/bar/ *

    instead of

    rm -rf /tmp/foo/bar/*

    ===

    Let's assume that you are in directory /backup that has  a subdirectory etc that you want to delete. As path "/etc" is ingrained in sysadmin mind it is very easy automatically/subconsciously type

    rm -r /etc

    instead of

    rm -r etc

    And realize  what you have done in a second or so. That's why for commands which include names OS system directories it is safer to type them in the editor, inspect them and only then execute that command on the command line.  If you are using terminal emulator on Windows desktop then Notepad++ or similar editor is OK.

    This is also why it is prudent to alias rm to the saferm script of your choice, which should prevent such mishaps (aliases are not used in non interactive sessions)

    This actually is an interesting type of "induced" error because /etc is typed daily so often that it kind of ingrained in sysadmin head and can be typed subconsciously instead of etc

  3. Using option -r with  .*  regex without understanding its consequences.   That essentially make rm -r recursive and it traverses down to parent directories as this basic regular expression matches ".." (parent directory). 
  4. You can accidentally hit Enter by mistake before finishing typing the line containing rm command.  If this is after the first character and the first character is /, or after several characters and the first are the  name of system directory like  /boot in older versions of Linux  you  would find yourself in trouble. Sometimes this happens with the new keyboard when you still do not fully adapted to new keys "feel".
  5. You can accidentally insert a blank  after *, this separating the argument into two: "*" and the rest, with corresponding consequences.  Argument * probably should be allowed if and only if it is a single argument on the command  like unless -f is specified.  

Those cases can be viewed as shortcomings of rm implementation (For example, * should be allowed only as a single argument, not as one of several arguments, but Unix expands argument  before passing  them to command  (expansion is done  by shell and there is not simple access to the  original command line) so it is tricky to check; rm also should automatically block deletion of system directories like //etc/ and the list of "protected" directories specified in its config file unless -f  flag is specified. Using has no system attribute so it is difficult to distinguish system files form the rest, but files owned by root:root probably deserve a special treatment as system files, even if one is working as root.

In other works implementation of rm does not take into account complexity of the current Unix environment and its power created an opportunity to wipe our more then you intended in way too many circumstances.

Also Unix does not have system attribute for files although sticky bit on files can be used instead along with ownership of sys instead of root).

Classic blunders in scripts with rm command

The most classic blunder with rm is deletion of wrong directly due to uninitialized variable. For example

TARGET_DIR=/etc
dir_name='Myscript'
cd $TARGET_DIR && rm -r $TARGET_DIR/$DIR_NAME

As you can see the variable used in RM command was types in all caps, but was defined in low case.

Here you need to use what is called defencisve programming. The main idea of which is to introduce addtional "sanity" test before any important operation

For example

TARGET_DIR=/etc
dir_name='Myscript'
if [[ "$TARGET_DIR/$DIR_NAME" = "$TARGET_DIR/" ]] ; then 
   echo [ABNORMAL_SITUATION] Variable $DIR_NAME is not defined
   exit 16
fi   
cd $TARGET_DIR && rm -r $TARGET_DIR/$DIR_NAME
It is highly recommended to make first runs in of such scripts in special test directly and to introduce the DEBUG variable in such scripts which block execution of rm command until some human checks are performed.  In debg mode you can use saferm for checking the list. But even simple addition of "I" flag greatly halt to avoid blunders.

Any script that use rm command but do not have a debug mode is an open invitation to disaster.

Any script that use rm command but do not have a debug mode is an open  invitation to disaster. 

debug=1
if (( $debug > 0 )) ; then 
  SAFETY='I'
else 
  SAFETY=''
fi    
TARGET_DIR=/etc
dir_name='Myscript'
... ... ...
if [[ -n $TARGET_DIR ]] ; then 
   echo [ABNOMAL SITUATION] Target directory is not defied or is a null string
   exit 128
fi
if [[ "$TARGET_DIR/$DIR_NAME" = "$TARGET_DIR/" ]] ; then 
   echo [ABNORMAL_SITUATION] Variable $DIR_NAME is not defined
   exit 16
fi   
cd $TARGET_DIR && rm -r$SAFETY $TARGET_DIR/$DIR_NAME

It is also very important to check the return code from each command

debug=1
if (( $debug > 0 )) ; then 
  SAFETY='I'
else 
  SAFETY=''
fi    
TARGET_DIR=/etc
dir_name='Myscript'
... ... ...
if [[ -n $TARGET_DIR ]] ; then 
   echo [ABNOMAL SITUATION] Target directory is not defied or is a null string
   exit 128
fi
if [[ "$TARGET_DIR/$DIR_NAME" = "$TARGET_DIR/" ]] ; then 
   echo [ABNORMAL_SITUATION] Variable $DIR_NAME is not defined
   exit 16
fi   
cd $TARGET_DIR && rm -r$SAFETY $TARGET_DIR/$DIR_NAME
if (( $? > 0 )) ; then 
   echo "Abnormal return code from rm $?
fi

See also shell - How to check if a variable is set in Bash - Stack Overflow

Generalized approach to checking important parameters in subroutines and before passing them to external programs like rm is called defensive programming.unfortunatly Wipipedia article is exremly weak, but some ideas applicable for bash can be derived from articles devotes to defensive programming in Java. For example  Defensive programming the good, the bad and the ugly - Enterprise Craftsmanship

Checks above are example of assertions about the value of parameters. 

The second common blunder in using rm command in scripts is absence of checking of return code of command that prepare rm environment (if such commands are used)

For example

cd /Etc/mc; rm *.conf
As you can see there is a typo in cd command and as the result it will be ignored and the rm command will be executed in the current directory possibly with disastrous results, if this directory is /etc.

It is always safer to check the return code. For all Unix command return code 0 means success. So classic rewrite of the command above will be

cd /Etc/mc && rm *.conf
In this case && plays the role of a check of the return code. The same,  of course,  can be done explicitly
cd Etc
if (( $? == 0 )) ; then 
   rm *.conf
   if (( $? > 0 )) ; then 
      echo [FAILURE] rm return code is $?
   fi
fi
Of course you need also check the return of code of rm which can fail due of insufficient permission, absent file and other circumstance
cd Etc
if (( $? == 0 )) ; then 
   rm *.conf
   if (( $? > 0 )) ; then 
      echo [FAILURE] rm return code is $?
   fi
fi

Unfortunately not all GNU utilities document return code in man pages. You need to experiment  is such case to determines which situation result in which return code.

NOTE: The examples above are presented just for the illustration of the concept. Generally you should avoid executing rm command on the current directory in your scripts, because that increases possibility to execute the command in the current directory which is the one you intend to use.  Always provide and verify the directory as a parameter to rm

Backup files before executing any "large scale" rm command, especially if you deleting files in system directory or an important files in use of data directories and the total size is not that large

 

Yesterday,
All those backups seemed a waste of pay.
Now my database has gone away.
Oh I believe in yesterday.

Suddenly,
There's not half the files there used to be,
And there's a milestone hanging over me
The system crashed so suddenly.

I pushed something wrong
What it was I could not say.
Now all my data's gone
and I long for yesterday-ay-ay-ay.

Yesterday,
The need for back-ups seemed so far away.
I knew my data was all here to stay,
Now I believe in yesterday.

John Lennon's Yesterday -- variation for sysadmins

"Make back of /etc first"  is the Rule No.1 if you delete something  from /etc directory or other system directories (/boot, /root, etc). Generally you should create the tarball of /etc directory at the beginning of the first root session for the particular day. Making tarball of the /etc directory on modern servers takes a couple of seconds.

"Make back of /etc first"  is the Rule No.1 if you delete something  from /etc directory.

Backups of subtrees affected before execution of rm command are also important. can save quite a lot of nerves in situations that otherwise can s can be devastating. For example, in the example above you would erase all your files and subdirectories in the /etc directory. Modern flavors of Unix usually prevent erasing /,  but not /etc. I think Red hat version  of GNU rm prevents erasing all level 2 system directories, but I am not sure.

Any attempt to remove a file whose last file name component is . or .. is rejected without any prompting, as mandated by POSIX.

If we are talking about more then 1GB of files, you can also move directories to /Trash folder and delete them after, say 7 days or so.  Later you can delete files that are say, older then 7 days using a cron script, or you can delay deletion until the total size of this directory exceeds a certain threshold. 

Within 10TB limit, You can always buy USB 3.0 drive and use it for such operations. Even if you buy it on your own money it is much cheaper than a couple of sleepless nights trying to recover lost information. 

There no excuse not to provide a backup in case of any large scale deletion.

There no excuse not to provide a backup in case of any large scale deletion of important files (for example in case directory to gzipped tarball conversion) when the affected files are within 10TB limit, You can always buy USB 3.0 drive and use it for such operations. Even if you buy it on your own money it is much cheaper than a couple of sleepless nights trying to recover lost information.

More on blunders with the deleting a directory from the backup

Experience keeps a dear school, but fools will learn in no other. ~Benjamin Franklin

The saying is that experience keeps the most expensive school but fools are unable to learn in any other ;-).

The saying is that experience keeps the most expensive school but fools are unable to learn in any other ;-). Please read classic sysadmin horror stories. 

It is dangerous to type destructive command directly on the command line as there are some idiosyncrasies that can play with you a very bad joke. Often backup has directories with the same name as systems directories simply because this is a backup of system directories. I once automatically typed /etc instead of etc trying to delete directory /backup/etc or something like that  to free space on a backup directory on a production server (/etc probably in engraved in sysadmin head as it is typed so often and can be substituted for etc subconsciously).  I realized that it was mistake and cancelled the command, but it was a fast server and one third of /etc was gone. The rest of the day was spoiled...  Actually not completely: I learned quite a bit about the behavior of AIX in this situation and the structure of AIX /etc directory this day, so each such disaster is actually a great learning experience, almost like one day training course ;-).

But it's much less nerve wracking to get this knowledge from the course... Another interesting thing is having backup was not enough is this case -- enterprise backup software stopped working on a damaged server. The same was true for telnet and ssh. And this was a remote server in a datacenter across the country.  I restored the directory on the other non-production server (first overwriting its /etc directory in this second box with the help of operations, tell me about cascading errors and Murphy law :-). In this case one ssh  session survives and helped to transfer the tar file.  As netcat does not requires authentication and if it is statically linked  it might help in such cases as you might be able to transfer missing files using it.

If you are working as root and perform dangerous operations never type a path of the command, copy it from the screen. If you can copy command from history instead of typing, do it !

In general it is safer to assemble the command  in editor then on the command line. If you type the command on t he command like prefix it with echo or # and  remove  them only when you verify that the command  is OK.

Such blunders are really persistent as often used directories are types in a "semiconscious" fashion, in "autopilot" mode and you realize the blunder only after you hit Enter. 

In such cases network services with authentication stop working the only way to transfer files is using CD/DVD, USB drive (if cp works, which in RHEL is not given as it is dynamically linked) or netcat. That's why it is useful to have netcat on servers: netcat is the last resort file transfer program when  services with authentication like ftp or scp stop working.  It is especially useful to have it if the datacenter is remote.

netcat is the last resort file transfer program when  services with authentication like ftp or scp stop working.  It is especially useful to have it if the datacenter is remote.

A simple extra space often produced horrible consequences:

cd /usr/lib
ls /tmp/foo/bar

I typed

rm -rf /tmp/foo/bar/ *

instead of

rm -rf /tmp/foo/bar/*

The system doesn’t run very will without all of it’s libraries……

NOTE: You can block such behavior requiring that if -r option was given you should have one and only one argument to rm. That probably should be a part of functionality of your "saferm" script.

Important class of subtle Unix errors: dot-star errors

A popular class of recursive rm errors are so called dot-star-errors. They happen when rm is using with -r option or when rm is used  in find -exec option.  Novice sysadmins usually do not realize that '.*' also matches '..' often with disastrous consequences.  If you are in any doubt as for how a wildcard will expand use echo command to see the result.

Using "convenience" symlinks to other (typically deeply nested) directories inside a directories and forgetting about them

Command  rm does not follows symlinks but find does.  So if you have a symlink to some system on important directory from the directory you are deleting using

find -exec rm {} \;

you are up to nasty surprise

That's the main danger of "convenience symlinks" which are used to simplify access to deeply nested directories.  They are more dangerous then useful. Using aliases or functions is a better deal.

If you use them, do this only from the level two directory created specially for this purpose like /Fav and protect this directory in your saferm script.

Disastrous typos in name of the directory or regex -- unintended space

There are some exotic typos that can lean you to troubles, especially with -r  option. One of them is unintended space:

 rm /home/joeuser/old *

instead of

rm /home/joeuser/old*

Dumb expression... Gradual realization of what you have done. Cold sweat..

Wrapper can catch some of those errors if  /home/joeuser/old does not exists.

Or, similarly:

 rm * _old

instead of

rm *_old
In all cases when you are deleting the directory directly copying its name from the listing obtained via ls or find command is much safer that typing it by yourself.

Such a mistake is even more damaging  is you use -r option. For example:

rm -rf /tmp/foo/bar/ *

instead of

rm -rf /tmp/foo/bar/*

So it is prudent to block execution of rm commands with multiple argument your saferm script, if option  -r is specified.

Root deletion protection

To remove a file you must have write permission on the folder where it is stored. Sun introduced "rm -rf /" protection in Solaris 10, first released in 2005. Upon executing the command, the system now reports that the removal of / is not allowed.

Shortly after, the same functionality was introduced into FreeBSD version of rm  utility.

GNU rm  refuses to execute rm -rf /  if the --preserve-root  option is given, which has been the default since version 6.4 of GNU Core Utilities was released in 2006.

No such protection exists for critical system directories like /etc /bin, and so on but you can imitate it putting file "-i" into such directories or using a wrapper for the interactive usage of the command. 

Using echo command to see expansion of your wildcards, if any

To understand what will be actually being executed after shell expansion, preface your rm command with echo. For example, if there is a filename starting with the dash you will receive very confusing message from rm

	$ rm *
	rm: unknown option -- -
	usage: rm [-f|-i] [-dPRrvW] file ...
This is connected with  the fact that in Unix regular expressions on the command line are expnaded by shell and the program gets already expended list, which in this case will contain  the name of the file which looks like an option.

That side effect can be used for protection of system directories. To find out what caused it prefix the command with echo

echo rm *

You can generate list of files dynamically for rm (piping it via xargs if it is too large.

But a better way is to include  generation and analysis of the list of files be  deleted into your saferm script and switch to generation of the list instead of actual deletion is any "suspicious files or circumstances are detected.

alias rm="saferm"

Please note that alias used in RHEL (alias rm="rm -i"), unfortunately, encourages a tendency to mindlessly pound y  and the return key to affirm removes - until you realize that got just past the file you need to keep.

Using a file starting with - as stoppers for unintended use of rm. Undeletable files

The rm command accepts the -- option which will cause it to stop processing flag options from that point forward. This allows the removal of file names that begin with a dash (-).

rm -- -filename

Actually accidental creation of files starting with dash is a common problem for novice sysadmin.  The problem is that if I type rm -foo, the rm command treats the filename as an option.

There are two simple ways to delete such a file. One was shown above and based on supplying an empty option argument. The second to use a relative or absolute pathname including at least one directory in  front:

rm ./--protected
rm /home/user/--protected

Here the debugging trick that we mentioned above -- prefacing rm command with echo really helps to understand what happens.

Deletion of files with non-printable characters often lead to unintended consequences

The safest way to do this is to use inodes for such files. You can also specify regular expression replacing non-printable characters in the name with ?  for each bad character. For example:

rm bad?file?name

But trying " a shortcut" which does not require meticulously putting ? in the name like" 

rm bad* 
Might lead to unintended consequences.

Another way to delete files that have control characters in the name is to  use  -i option and an asterisk, which gives you the option of removing each file in the directory — even the ones that you can't type.

% rm -i *
rm: remove faq.html (y/n)? n
rm: remove foo (y/n)? y
%

The -i option may also be helpful when you are dealing with files with Unicode characters that appear to be regular letters in your locale, but that don't match patterns or names you use otherwise.

You can also create the function for deletion of files with special characters. Something along the lines:

# cat  /root/del_special_characters.sh
   if [ ! -d /tmp/Special_files ] ; then
      mkdir /tmp/Special_files
   fi
   genfile=/tmp/Special_files/list`date +"%y%m%d"`
   echo dev/null > $genfile
   find . -maxdepth 1  -type f -ls | fgrep 'a' | while  read myline
   do
      echo Processing: $myline
      IFS=' ' # Warning: the IFS variable means split by one of characters in a set
      set $myline
      inode=$1
      echo inode=$inode
      new_name=${!#};
      new_name=`tr 'a' '_' <<< "$new_name"`
      echo "new_name=$new_name"
      echo mv $inode $new_name
      echo mv $inode $new_name >> $genfile
   done
   echo "Generated commands:"
   cat $genfile

A great way to discover files with control characters in them is to use the -q option to the Unix ls command (some systems also support a useful -b option). You can, for example, alias the ls command:

alias lsq='ls -liq '

Files that have control characters in their filenames will then appear with question marks:

lqs f*
faq.html                fo?o

Blunders in using rm with find

This is class or blunders that deserves a page of its own. See Typical Errors In Using Find  for more information about this SNAFU

You need to be extra careful running rm in an -exec option of find command. There are many horror stories in which people delete large parts of their filesystem due to typos of sloppy find command parameters. Again it is prudent first to test it using ls command to see the actual list of files to be deleted. Sometimes it is quite different from what you intended to do :-).

you can also generate the list using ls command, inspect it in a separate step. And only then pipe it into rm :

cat rm.lst | xargs -t -n1 rm 
The -t flag causes the xargs command to display each command before running it, so you can see what is happening.

Dr. Nikolai Bezroukov

Recommended Links

Google matched content

Softpanorama Recommended

Saferm implementations



Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D


Copyright © 1996-2020 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to to buy a cup of coffee for authors of this site

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: September 04, 2019