Softpanorama
May the source be with you, but remember the KISS principle ;-)

Contents Bulletin Scripting in shell and Perl Network troubleshooting History Humor

rdist - remote file distribution client program

News

See Also Recommended Links Directory Synchronization Unix Configuration Management Tools rsync
Unix Configuration Management Tools Curl LFTP -- Scriptable FTP client wget Config files distribution: copying a file to multiple hosts Slurping: copying a file from multiple hosts
Parallel command execution C3 Tools Admin Horror Stories Unix History Humor Etc

Utility rdist is a classic Unix utility to maintain identical copies of files over multiple hosts. It originated in BSD 4.3.

It preserves the owner, group, mode, and mtime of files if possible and can update programs that are executing. It can use SSH as transport protocol and in this sense can be viewed as more flexible and powerful form of scp. Unlike its more modern analog it can distribute files to multiple host, not just copy form one host to another like rsync.

It reads commands from do called distfile  which contains the set of instructions to direct the updating of files and/or directories. If distfile is '-', then the standard input is used. If no -f option is present, the program looks first for distfile, then the file with the name  'Distfile' to use as the input. If no names are specified on the command line, rdist  will update all of the files and directories listed in distfile. Otherwise, the argument is taken to be the name of a file to be updated or the label of a command to execute. If label and file names conflict, it is assumed to be a label. These may be used together to update specific files using specific commands.

The -c option forces rdist to interpret the remaining arguments as a small distfile. The equivalent distfile is as follows.

(name ... ) -> [login@]host

To use a transport program other than rsh(1c) you need to specify the -P option or define an alias

alias rdist='rdist -P /usr/bin/ssh '

Whatever transport program is used, must be compatible with the above specified syntax for rsh(1c). If the transport program is not, it should be wrapped in a shell script which does understand this command line syntax and which then executes the real transport program. Here's an example which uses ssh(1) as the transport:

       rdist -P /usr/local/bin/ssh -f myDistfile

On each target host   rdist  will attempt to run the command

rdistd -S

This command launch rdistd daemon (see separate description)

Syntax

rdist [ -DFn ] [ -A num ] [ -a num ] [ -d var=value ] [ -l <local logopts> ] [ -L <remote logopts> ] [ -f distfile ] [ -M maxproc ] [ -m host ] [ -o distopts ] [ -t timeout ] [ -p <rdistd-path> ] [ -P <transport-path> ] [ name ... ]

rdist -DFn -c name ... [login@]host[:dest]

rdist -Server

rdist -V  

Options

-A num
Set the minimum number of free files (inodes) on a filesystem that must exist for rdist to update or install a file.
-a num
Set the minimum amount of free space (in bytes) on a filesystem that must exist for rdist to update or install a file.
-D
Enable copious debugging messages.
-d var=value
Define var to have value. This option is used to define or override variable definitions in the distfile. Value can be the empty string, one name, or a list of names surrounded by parentheses and separated by tabs and/or spaces.
-F
Do not fork any child rdist processes. All clients are updated sequentially.
-f distfile
Set the name of the distfile to use to be distfile . If distfile is specified as ``-'' (dash) then read from standard input (stdin).
-l logopts
Set local logging options. See the section MESSAGE LOGGING for details on the syntax for logopts.
-L logopts
Set remote logging options. logopts is the same as for local logging except the values are passed to the remote server (rdistd). See the section MESSAGE LOGGING for details on the syntax for logopts.
-M num
Set the maximum number of simultaneously running child rdist processes to num. The default is 4.
-m machine
Limit which machines are to be updated. Multiple -m arguments can be given to limit updates to a subset of the hosts listed in the distfile.
-n
Print the commands without executing them. This option is useful for debugging distfile.
-o distopts
Specify the dist options to enable. distopts is a comma separated list of options which are listed below. The valid values for distopts are:
 
verify
Verify that the files are up to date on all the hosts. Any files that are out of date will be displayed but no files will be changed nor any mail sent.
whole
Whole mode. The whole file name is appended to the destination directory name. Normally, only the last component of a name is used when renaming files. This will preserve the directory structure of the files being copied instead of flattening the directory structure. For example, rdisting a list of files such as /path/dir1/f1 and /path/dir2/f2 to /tmp/dir would create files /tmp/dir/path/dir1/f1 and /tmp/dir/path/dir2/f2 instead of /tmp/dir/dir1/f1 and /tmp/dir/dir2/f2.
noexec
Automatically exclude executable files that are in a.out(5) format from being checked or updated.
younger
Younger mode. Files are normally updated if their mtime and size (see stat(2)) disagree. This option causes rdist not to update files that are younger than the master copy. This can be used to prevent newer copies on other hosts from being replaced. A warning message is printed for files which are newer than the master copy.
compare
Binary comparison. Perform a binary comparison and update files if they differ rather than comparing dates and sizes.
follow
Follow symbolic links. Copy the file that the link points to rather than the link itself.
ignlnks
Ignore unresolved links. Rdist will normally try to maintain the link structure of files being transferred and warn the user if all the links cannot be found.
chknfs
Do not check or update files on target host that reside on NFS filesystems.
chkreadonly
Enable check on target host to see if a file resides on a read-only filesystem. If a file does, then no checking or updating of the file is attempted.
chksym
If the target on the remote host is a symbolic link, but is not on the master host, the remote target will be left a symbolic link. This behavior is generally considered a bug in the original version of rdist, but is present to allow compatibility with older versions.
quiet
Quiet mode. Files that are being modified are normally printed on standard output. This option suppresses this.
remove
Remove extraneous files. If a directory is being updated, any files that exist on the remote host that do not exist in the master directory are removed. This is useful for maintaining truly identical copies of directories.
nochkowner
Do not check user ownership of files that already exist. The file ownership is only set when the file is updated.
nochkgroup
Do not check group ownership of files that already exist. The file ownership is only set when the file is updated.
nochkmode
Do not check file and directory permission modes. The permission mode is only set when the file is updated.
nodescend
Do not descend into a directory. Normally rdist will recursively check directories. If this option is enabled, then any files listed in the file list in the distfile that are directories are not recursively scanned. Only the existence, ownership, and mode of the directory are checked.
numchkgroup
Use the numeric group id (gid) to check group ownership instead of the group name.
numchkowner
Use the numeric user id (uid) to check user ownership instead of the user name.
savetargets
Save files that are updated instead of removing them. Any target file that is updates is first rename from file to file.OLD.
sparse
Enable checking for sparse (aka wholely) files. One of the most common types of sparse files are those produced by ndbm(3). This option adds some additional processing overhead so it should only be enabled for targets likely to contain sparse files.
-p <rdistd-path>
Set the path where the rdistd server is searched for on the target host.
-P <transport-path>
Set the path to the transport command to be used. This is normally rsh(1c) but can be any other program - such as ssh(1) - which understands rsh(1c) command line syntax and which provides an appropriate connection to the remote host. The transport-path may be a colon separated list of possible pathnames. In this case, the first component of the path to exist is used. i.e. /usr/ucb/rsh:/usr/bin/remsh, /usr/bsd/rsh.
-t timeout
Set the timeout period (in seconds) for waiting for responses from the remote rdist server. The default is 900 seconds.
-V
Print version information and exit.

Message Logging

Rdist uses a collection of predefined message facilities that each contain a list of message types specifying which types of messages to send to that facility. The local client (rdist) and the remote server (rdistd) each maintain their own copy of what types of messages to log to what facilities.

The -l logopts option to rdist tells rdist what logging options to use locally. The -L logopts option to rdist tells rdist what logging options to pass to the remote rdistd server.

The form of logopts should be of form

facility=types:facility=types...

The valid facility names are:

stdout
Messages to standard output.
file
Log to a file. To specify the file name, use the format ``file=filename=types''. e.g. ``file=/tmp/rdist.log=all,debug''.
syslog
Use the syslogd(8) facility.
notify
Use the internal rdist notify facility. This facility is used in conjunction with the notify keyword in a distfile to specify what messages are mailed to the notify address.

types should be a comma separated list of message types. Each message type specified enables that message level. This is unlike the syslog(3) system facility which uses an ascending order scheme. The following are the valid types:

change
Things that change. This includes files that are installed or updated in some way.
info
General information.
notice
General info about things that change. This includes things like making directories which are needed in order to install a specific target, but which are not explicitly specified in the distfile.
nerror
Normal errors that are not fatal.
ferror
Fatal errors.
warning
Warnings about errors which are not as serious as nerror type messages.
debug
Debugging information.
all
All but debug messages.

Here is a sample command line option:

-l stdout=all:syslog=change,notice:file=/tmp/rdist.log=all
This entry will set local message logging to have all but debug messages sent to standard output, change and notice messages will be sent to syslog(3), and all messages will be written to the file /tmp/rdist.log.  

Distfiles

The distfile contains a sequence of entries that specify the files to be copied, the destination hosts, and what operations to perform to do the updating. Each entry has one of the following formats.

<variable name> = <name list> 
[ label: ] <source list> -> <destination list> <command list> 
[ label: ] <source list> :: <time_stamp file> <command list> 
The first format is used for defining variables. The second format is used for distributing files to other hosts. The third format is used for making lists of files that have been changed since some given date. The source list specifies a list of files and/or directories on the local host which are to be used as the master copy for distribution. The destination list is the list of hosts to which these files are to be copied. Each file in the source list is added to a list of changes if the file is out of date on the host which is being updated (second format) or the file is newer than the time stamp file (third format).

Labels are optional. They are used to identify a command for partial updates.

Newlines, tabs, and blanks are only used as separators and are otherwise ignored.

Comments begin with # and end with a newline.

Variables to be expanded begin with $ followed by one character or a name enclosed in curly braces (see the examples at the end).

The source and destination lists have the following format:

<name>
or
(<zero or more names separated by white-space> )

These simple lists can be modified by using one level of set addition, subtraction, or intersection like this:
list - list
or
list + list
or
list & list
If additional modifications are needed (e.g., ``all servers and client machines except for the OSF/1 machines'') then the list should be explicitly constructed in steps using "temporary" variables.

The shell meta-characters `[', `]', `{', `}', `*', and `?' are recognized and expanded (on the local host only) in the same way as csh(1). They can be escaped with a backslash. The `~' character is also expanded in the same way as csh but is expanded separately on the local and destination hosts. When the -o whole option is used with a file name that begins with `~', everything except the home directory is appended to the destination name. File names which do not begin with `/' or `~' use the destination user's home directory as the root directory for the rest of the file name.

The command list consists of zero or more commands of the following format.

`install'         opt_dest_name `;'
`notify'        `;'
`except'        `;'
`except_pat'  `;'
`special'       string `;'
`cmdspecial'    string `;'

The install command is used to copy out of date files and/or directories. Each source file is copied to each host in the destination list. Directories are recursively copied in the same way. Opt_dest_name is an optional parameter to rename files. If no install command appears in the command list or the destination name is not specified, the source file name is used. Directories in the path name will be created if they do not exist on the remote host.

The -o distopts option as specified above under OPTIONS, has the same semantics as on the command line except they only apply to the files in the source list. The login name used on the destination host is the same as the local host unless the destination name is of the format ``login@host".

The notify command is used to mail the list of files updated (and any errors that may have occurred) to the listed names. If no `@' appears in the name, the destination host is appended to the name (e.g., name1@host, name2@host, ...).

The except command is used to update all of the files in the source list except for the files listed in name list. This is usually used to copy everything in a directory except certain files.

The except_pat command is like the except command except that pattern list is a list of regular expressions (see ed(1) for details). If one of the patterns matches some string within a file name, that file will be ignored. Note that since `\' is a quote character, it must be doubled to become part of the regular expression. Variables are expanded in pattern list but not shell file pattern matching characters. To include a `$', it must be escaped with `\'.

The special command is used to specify sh(1) commands that are to be executed on the remote host after the file in name list is updated or installed. If the name list is omitted then the shell commands will be executed for every file updated or installed. String starts and ends with `"' and can cross multiple lines in distfile. Multiple commands to the shell should be separated by `;'. Commands are executed in the user's home directory on the host being updated. The special command can be used to rebuild private databases, etc. after a program has been updated. The following environment variables are set for each special command:

FILE
The full pathname of the local file that was just updated.
REMFILE
The full pathname of the remote file that was just updated.
BASEFILE
The basename of the remote file that was just updated.

The cmdspecial command is similar to the special command, except it is executed only when the entire command is completed instead of after each file is updated. The list of files is placed in the environment variable $FILES. Each file name in $FILES is separated by a `:' (colon).

If a hostname ends in a ``+'' (plus sign), then the plus is stripped off and NFS checks are disabled. This is equivalent to disabling the -o chknfs option just for this one host.

The following is a small example.

HOSTS = ( matisse root@arpa) 
FILES = ( /bin /lib /usr/bin /usr/games
         /usr/include/{*.h,{stand,sys,vax*,pascal,machine}/*.h} 
        /usr/lib /usr/man/man? /usr/ucb /usr/local/rdist )
EXLIB = ( Mail.rc aliases aliases.dir aliases.pag crontab dshrc 
        sendmail.cf sendmail.fc sendmail.hf sendmail.st uucp vfont ) 
${FILES} -> ${HOSTS} 
        install -oremove,chknfs ; 
        except /usr/lib/${EXLIB} ; 
        except /usr/games/lib ; 
        special /usr/lib/sendmail "/usr/lib/sendmail -bz" ; 
srcs: 
        /usr/src/bin -> arpa 
        except_pat ( \\.o\$ /SCCS\$ ) ; 
IMAGEN = (ips dviimp catdvi) 

imagen: /usr/local/${IMAGEN} -> arpa 
        install /usr/local/lib ;
        notify ralph ; 
${FILES} :: stamp.cory
        notify root@cory ; 


Top updates

Softpanorama Switchboard
Softpanorama Search


NEWS CONTENTS

Old News ;-)

[Mar 16, 2017] rdist vs rsync Open Source Web Hosting

www.omniweb.com

Posted on June 30, 2011 by admin

OK why is there even any debate? I just tried rsync’ing a directory with a couple hundred gigs to a USB drive, I had a few gigabytes that weren’t on the USB yet. After several hours there didn’t seem to be any progress. I set up rdist to copy to a different directory (/mnt/usb) on the localhost, and the copy is completed within 5 minutes!

Despite this, when I google around, I find comments going back a decade declaring rsync to be the superior copying tool. hmmm. hours = incomplete vs minutes = success; RDIST winning.

PS here is a great reference for using rdist

Introduction to Rdist by Benedikt Stockebrand

This is the only really valuable article about rdist that I have found on Internet.
Benedikt Stockebrand
Diplom-Informatiker Contact and Legalese
Terms and Conditions
DeutschDeutsch
Home » Resources » Tech. Documentations » rdist

Introduction

"Using rdist doesn't make sense unless you are dealing with a sufficiently large number of machines---let's say at least two." (Markus Moster)

In a Nutshell

Synchronizing files and file hierarchies across multiple machines is an ever-recurring task in almost any reasonably-sized computer network environment. Since the arrival of BSD 4.3 a widely ignored tool called rdist is available to deal with this task. At least among machines running Un*x-style operating systems, that is.

Simple Real World Examples

What does rdist provide? The basic functionality is best demonstrated with some simplified real-world examples.

First consider a WAN connected through low-bandwidth (64kbit/s) links. At about 40 sites there exist general-purpose servers. These servers provide some software repository to say twenty clients each. Whenever a user feels like it s/he may install software from that repository on his/her desktop client. The software in the repository is frequently updated and contains about 500 MB worth of software. Using rdist you simply define one server as the "master" or "reference" machine. Whenever you've modified the repository on that machine you run rdist, for large modifications preferably overnight, and it updates the remote servers without further sysadmin intervention. Since rdist will only update files that have changed this provides a very bandwidth and time preserving way to deal with this distribution issue. If you ever fear that for some reason a repository got out of sync you re-run rdist. Unless things actually have changed this will have little impact on the servers or the bandwidth available in between. Of course the same approach works for the document roots of multiple web servers behind a load balancer, too.

Consider the same servers again. Most of their configuration is the same on all machines, or at least should be. If you set them up the same and then use rdist to distribute all configuration files that shouldn't be adjusted individually you won't have to update 40 /etc/profile's by hand (eventually getting things out of sync anyway) but rather do that only once and then run rdist. If you want to test new configurations you do so on only one machine. If things work out you run rdist from that machine, otherwise you can easily do a rollback by running rdist from one of the remaining functional machines. This has proven to be particularly useful in cluster environments where cluster members must be kept identically configured and no excuses. Guess what Markus Moster (see the quote above) is habitually working on.

If you are responsible for quality control in a software development team you may want to test the installation packages you get from the development team in a clearly defined test environment. Of course you might want to reinstall those machines either from backup or through an install server (like a Sun[TM] Solaris[TM] JumpStart[TM]). You might however want to find out what system files the package installation has messed up. If you define a reference file system somewhere rdist lets you do all this with a reasonable amount of trouble. And imagine the educational value of test machines that are always in a clearly defined state every morning.

In a security-sensitive environment where you need to maintain a consistent user base on multiple machines but can afford neither the security issues of NIS nor the hassles and Solaris-onliness of NIS+ you can easily set up one machine as the "master" passwd server and distribute whatever /etc/passwd, /etc/shadow, /etc/groups and such it has through cron to all other machines. New user accounts get created on that machine or people change their password on that machine and maybe a minute later the new account or password is active on all machines.

How rdist Works

How does rdist provide this functionality? Well, assume that you have a machine master that wants to copy some files to a list of machines slave_1 to slave_n. Here's what happens:

  1. You invoke rdist, possibly with some options.
  2. If you didn't specify directly what to copy where the first thing rdist does is to read a Distfile. This file tells rdist what to copy where.
  3. Now rdist starts a "daemon" program called rdistd on all slaves. By default it does so using rsh but anybody working in a remotely security-sensitive environment will tell it to use the Secure Shell (ssh) instead.
  4. Once the connection between the master rdist and slaves rdistds is up and running the master starts to stat(2) all files and file hierarchies specified in the distfile. It asks the slaves to do so as well. Whenever the stat(2) results prove different the file is sent from the master to the slave where it is installed.
  5. Once the master is through it tells the slave rdistd to terminate which will also shut down the connection between the machines.

This general behaviour can be modified in various ways. Among others you can

Setting Up rdist

Here's an outline how to set up rdist to work in your environment. If you want to use it in conjunction with ssh make sure that ssh is running first.

  1. Get, compile and install an up-to-date version of rdist. Sources should be available from your favourite FTP site. At the time of this writing the latest version is 6.1.5. Note that many Un*xen come with an older version you shouldn't use. Make sure you install the "daemon" binary rdistd on all slave machines, preferably somewhere within the default PATH or at least in the same directory on all machines. If you install rdistd outside the default PATH you need to specify where to find it with the "-p path-to-rdistd" option.
  2. Set up the underlying transport and authentication mechanism. Choose between standard security-free rsh and non-standard secure ssh.
    1. By default rdist uses the traditional rlogin/rsh for this. If that's what you want you need to add the master machine to the ~/.rhosts or /etc/hosts.equiv or whatever file on all slave machines. And of course the slave machines have to run the rshd, which is usually started via inetd. Make sure that an "rsh slave hostname" will output the hostname of the slave.
    2. If you're in any way concerned about security you should set up ssh so that you can connect from the master to the slave machines. Make sure that the master knows about all slaves in its known_hosts file by ssh'ing to all slaves once. To use rdist with ssh you need to specify ssh as the underlying transport mechanism using the option "-P path-to-ssh".
  3. Now see if rdist works. First create some temporary file in /tmp to distribute:
    	  master$ cd /tmp
    	  master$ echo "Hello, world!" >distribute-me
    	

    To test our setup use the "-c" option which allows to specify a mini "distfile" on the command line and send it to a slave. Don't forget to add the "-p path-to-rdistd" and/or "-P path-to-ssh" options here if necessary.

    	  master$ rdist -c distribute-me slave:/var/tmp
    	

    This will distribute our file /tmp/distribute-me from the master to the /var/tmp directory on our slave.

  4. If you want to make life easier you may want to create a simple shell script that takes care of those "-p" and "-P" options. Try this one:
    	  #! /bin/sh
    	  SSH="`which ssh`"
    	  RDISTD="`which rdistd`"
    	  
    	  rdist -p "$RDISTD" -P "$SSH" "$@"
    	

    Adjust as needed, especially if you don't have ssh and/or rdistd in your path at all.

Now the environment is ready to use rdist. At least for one-shot jobs, that is. The real power of rdist is in its distfiles however, so read on.

Elementary Distfiles

Basics

A distfile defines what files and file hierarchies to copy where and with which options. You may specify a particular distfile using the "-f distfile" option. Otherwise rdist will default to a default filename "Distfile" in the current working directory.

The syntax is somewhat similar to a standard makefile. Here's a distfile that'd copy our previous test file /tmp/distribute-me to some slave machine slave_1.

      # Distfile 1
      /tmp/distribute-me -> slave_1
          install /var/tmp ;
    

This distfile tells rdist to install our file on slave_1 in its /var/tmp directory. In detail the first line specifies the source file and the destination machine. The second line tells that the file specified needs to be installed in the /var/tmp directory of the slave.

Now we may want to specify multiple files and/or multiple destination machines. Doing so we need to put parentheses around them, like this:

      # Distfile 2
      ( /var/www /var/ftp ) -> ( slave_1 slave_2 slave_3 )
          install -o remove /var ;
    

This will copy the contents of both /var/www and /var/ftp to all three slave machines. The -o option to install allows us to specify a wide range of additional options. In this particular case we tell rdist to remove all files on the slaves that don't exist on the master. Use this option with care, a typo in the wrong spot may wipe out lots of data on the slaves.

If you just want to see what would happen if you ran rdist without actually doing the update you may add an option "verify" to the install command. That'll make rdist do a dry run that doesn't modify anything on the target machine. See below how you add this option on the command line.

Synchronizing directory trees with lots of small files will take some time even if no files need to be copied at all. To speed things up we may define makefile-style targets that can be invoked on the command line. So we redo our previous distfile again.

      # Distfile 3
      www: /var/www -> ( slave_1 slave_2 slave_3 )
               install -o remove /var ;
      
      ftp: /var/ftp -> ( slave_1 slave_2 slave_3 )
               install -o remove /var ;
    

Different to make an invocation of rdist that doesn't specify a particular target or set of targets will have all targets installed, not just the first one. So if we invoke rdist with this distfile it'll behave exactly like No.2. But if we apply an additional argument "www" or "ftp" to the end of our rdist invocation only the target of that name will be installed.

There are other commands besides the "install" command we've seen. You may exclude files and/or file patterns from distribution using the "exclude" and "exclude_pat" commands. The former takes a list of file names as arguments and excludes them from installation:

      # Distfile 4
      ftp: /var/ftp -> ( slave_1 slave_2 slave_3 )
               install -o remove /var ;
               except /var/ftp/Distfile /var/ftp/incoming ;
    

This will prevent rdist to install Distfile and the incoming directory.

Even more powerful is the except_pat command. It will exclude files according to a ed(1) style regular expression:

      # Distfile 5
      etc: /etc -> ( archivehost )
               install -o remove /archive/master/etc ;
               except_pat ~$ ;
    

Note that you need to escape the dollar character (marking the end of line) in this pattern. It will skip all files ending with a tilde character.

Variables

Similar to makefiles we may use variables in our distfiles. These variables provide a means to "recycle" definitions of file and target groups to simplify the maintenance of distfiles. Here's a very simple example that shows what variables may be good for:

      # Distfile 6
      HOSTS= ( slave_1 slave_2 slave_3 )
      FILES= (
               /etc/hosts
               /etc/inet/ntp.conf 
               /etc/services 
             ) 
      
      base-configs: ${FILES} -> ${HOSTS}
      install -owhole / ;
    

So far this doesn't look particularly useful. It does however show how we use variables---we'll see what they're good for in the following section. The syntax for both variable definition and variable expansion is very much like in a makefile again, except that we must use curly braces around the variable name, not parentheses. Yes, you may skip the braces if your variable name is only a single character long.

Set Operations

The distfiles so far have basically distributed files using a one-to-one scheme. Especially in the last example we may however see a hint of the problems to come: What if we want to maintain configuration files for multiple machines running different Un*x versions? We may use set operations to define which files are host specific, OS specific or generally used. Here is another (simplified) example:

      # Distfile 7
      
      # Consider your lists of hosts here
      SOLARISHOSTS= ( sun1 sun2 sun3 )
      FREEBSDHOSTS= ( chuck1 chuck2 )
      NETBSDHOSTS=  ( pdp1 pdp2 )
      OPENBSDHOSTS= ( marvin1 marvin2 )
      
      LEAFNODES=    ( sun1 sun2 )
      
      # The list of files going to /etc (BSD) or /etc/inet (Solaris) here.
      INETCFG=         ( /etc/services /etc/protocols /etc/hosts /etc/ntp.conf )
      INSECUREINETCFG= ( /etc/inetd.conf )

      
      # No serviceable parts beyond this line (read on, though)
      
      BSDHOSTS=     ( ${FREEBSDHOSTS} + ${NETBSDHOSTS} + ${OPENBSDHOSTS} )
      HOSTS=        ( ${SOLARISHOSTS| + ${BSDHOSTS} )
      
      etc: ( ${INETCFG} + ${INSECUREINETCFG} ) -> ${SOLARISHOSTS}
               install /etc/inet ;
      
      etc: ${INETCFG} -> ${BSDHOSTS}
               install /etc;
      
      etc: ${INSECUREINETCFG} -> ( ${BSDHOSTS} - ${OPENBSDHOSTS} )
               install /etc ; 
      
      etc: /etc/notrouter -> ( ${SOLARISHOSTS} & ${LEAFNODES} )
               install /etc ; 
    

This example shows what you can do with those set operations. Even though we maintain a single master copy of various configuration files we place them on on the destination machines in different directories according to their architecture and/or operating system. In the case of the "inetd.conf" we explicitly exclude the extra-paranoid OpenBSD installations because they don't run an inetd in our example. And for the Solaris workstations we make sure they don't try to route anything by installing the file "/etc/notrouter" on them.

The exact syntax for set operations is this: Every individual object (like a file or target machine name) is considered a set. So is a sequence of objects surrounded in parentheses. So is a variable reference referring to an set. We may use the set operations "+", "&" and "-" on multiple sets to compute the union, intersection and exclusion of those sets, respectively.

Use of these set operations may not be nested within an individual expression. You may however use intermediate variables to work around this. Yes, this is considered a silly limitation even within the man page.

The "special" And "cmdspecial" Commands

It may happen that you want to execute some sort of command on the target machines whenever you have changed a file. There are two commands to do so. The "special" command is executed for every file that has been updated by the rule it has been given while the "cmdspecial" command will be executed once when at least one of the files specified in a rule has been updated.

The command specified is run on the target machine from within the home directory of the user you connect to. It is run through the Bourne shell so you may use whatever the shell has to offer. The "special" command has the environment variable "FILE" set to the path of the file on the master machine and "REMFILE" variable set to the path of the file on the target machine. The "cmdspecial" executes with the environment variable "FILES" containing a colon-separated list of remote file names that have been updated.

The command string needs to be quoted using double quotes. I haven't bothered to figure out the exact details about quoting mechanisms. Instead I rather send a full-blown shell script to the target machine and then execute it.

Here is an example. It logs whatever file is updated to the targets syslogd using logger. Once it has finished it also appends all the new files to a tar file.

    ( /usr /bin /sbin ) -> ( test1 test2 test3 )
	install / ;
        special "logger Updated $REMFILE" ;
	cmdspecial "tar rf /tmp/rdist-update.tar `echo $FILES | tr : ' '`"
    

Command Line Options

You may specify several command line options to rdist. Among the more useful are these:

-D
Create debugging output.
-d var=value
Set a variable. This overrides variable definitions in the distfile.
-f distfile
Specify an alternate distfile.
-M num
Specify the number of target machines to be updated in parallel. Default value is 4.
-m machine
Restrict updates to the machine given. May be used multiple times.
-n
Only show the rules that would be applied to the machines. Best used in conjunction with -m.
-o distopts
Specify additional options to all install commands. Multiple options may be separated by comma. This is particularly useful with the verify and compare options.
-p remote-path-to-rdistd
Tell rdist where to look for rdistd on the target machines.
-P path-to-rsh
Tell rdist where to find rsh or whatever replacement (like ssh) on the master machine.
-V
Make rdist output its version. Use this in shell scripts where it depends on the PATH environment variable which version of rdist is being run.

There are several other options to rdist which are of less importance than the ones shown here. Consult the manual pages for the rest of them.

Advanced Features

Now that we've mastered the basics of rdist we take a look at some additional features that are not of fundamental importance to the everyday use of rdist but still prove useful for more special purposes.

Additional install Options

The "quiet" Option

If you want to make the install command be less talkative you specify an additional option "quiet". As a personal preference however I rather keep the output from an rdist run in a file and either "grep -v" that file afterwards or run rdist again to get a short "everything is up to date"-style output.

Modifying the Update Conditions

By default rdist always updates files whenever the stat(2) syscall returns differences in file size, time stamps, permissions or owners. In case of the former two criteria the file content is updated, in case of the latter two only the permissions and ownership. This default behaviour can be modified using the following options to install.

"younger"
Don't update the destination file if it is younger than the source file. So this option should really be named "notyounger" or "older" or whatever...
"compare"
Force a content comparison of the source and destination file. This is obviously bandwidth intensive so only use it if absolutely necessary.
"ignlnks"
By default rdist will complain about stale symlinks. This option will turn these warnings off.
"chknfs"
Do not update files located on an NFS-mounted volume.
"chkreadonly"
If the destination file is on a RO-mounted volume or file system then don't bother about it.
"nodescend"
Don't recursively update directories. Normally, rdist will update the whole directory tree if it updates a directory. With this option it will only update the directory itself, i.e. its attributes.
"nochkowner" and "nochkgroup"
Don't check the owner and group of a file, respectively. This is particularly useful if you run rdist as ordinary user on the destination machines.
"numchkowner" and "numchkgroup"
By default rdist checks the user and group names. These options make it compare the numeric UID and GID, respectively. This is particularly useful if you want to provide a cross-architecture installation server or such.
"nochkmode"
Don't check the permissions on the target files.

Modifying the Update Behaviour

"whole"
Normally the destination file path is the concatenation of the destination directory specified (or "~" if none was specified) and the last component of the source path. A rule of the form
          # Distfile 8
          /var/tmp/distribute-me -> slave_1
              install /export/home ;
	

would have the file installed as /export/home/distribute-me on slave_1. If however you gave the install command the whole option you would install the file as /export/home/var/tmp/distribute-me.

Whenever you use this option make double sure things go where you want. Especially if you're using it in conjunction with "remove"!

"savetargets"
Whenever a file is updated, keep the old file as a backup copy with an extension ".OLD".
"sparse"
Deal sparse files efficiently. Don't forget this if you distribute ndbm files and such.

Message Logging and Change Notification

It is possible to make rdist generate various kinds of logging output. By default it is already writing copious output to stdout when running. You may however want it to log even more things to stdout, the master or slave machines syslog facility or log file or an e-mail address. Since these features are somewhat beyond the scope of an "introductory" text and I personally rather use sed or perl to extract whatever I want from the output I won't go into detail about these. Take a look at the man page to find out about this---it has a section dedicated to this issue.

There is also a mechanism available to generate lists of files that have changed since some reference file has been last modified. Again, see the man page. I personally rather run rdist with the -overify option instead.

Strategic Considerations and Best Practices

Now that we've made it through the features of rdist it is time to consider approaches how to use it best. Here comes a list of issues worth to consider:

Conclusion

The features rdist provides are simple but extremely useful to handle groups of machines that share some common files that occasionally need to be updated in a systematic way. Considering the simplicity and scope of rdist it is surprising that it isn't way more widely known and used. I sincerely hope that this little documentation (which has become slightly longer than the original man page) will help to change this.

Recommended Links

Softpanorama hot topic of the month

Softpanorama Recommended

Introduction to Rdist by Benedikt Stockebrand  -- this is the only really valuable article about rdisk that I have found on Internet.

Man pages

Etc



Etc

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes.   If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner. 

ABUSE: IPs or network segments from which we detect a stream of probes might be blocked for no less then 90 days. Multiple types of probes increase this period.  

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least


Copyright © 1996-2016 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: March, 16, 2017