Softpanorama

Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
May the source be with you, but remember the KISS principle ;-)
Skepticism and critical thinking is not panacea, but can help to understand the world better

Rsnapshot -- a very elegant approach to incremental backups

News rsync Recommended Links Missing backup horror stories Perl Backup Scripts Relax-and-Recover Disaster Recovery
FIT USB flash drives Unix tape archiver (Tar) Tar options for bare metal recovery Cloning systems, disks and partitions Gzip Unix cpio Dump and restore
dd Remote backups using dd  Recovery of lost files using DD Mount a partition from dd disk image rsync Backing Up And Restoring Linux With SystemImager Baseliners
 Missing backup HP Data Protector Dell DRAC and VFLASH Disk Backup and Cloning Alternatives to Norton Ghost Sysadmin Horror Stories Humor Etc

Introduction

Rsnapshot is a very elegant filesystem snapshot utility for making backups of local and remote systems.  It is written in Perl is a one large Perl script (~7K lines). The key idea is to use  rsync for copying files and hard links to maintaining multiple versions. If you hardlink the most recent version  (say day.0) of a file to a  new directory (say, day.1)  and then run rsync to synchronize day.0 and the target, the hardlink is automatically converted to a backup of the file.  Files that overwrite hardlink  leave old content in place.  Identical files remain hardlinked saving tremendous amount of space in case of OS backup, where using  tiny fraction of files changes from one backup to another.

It is essentially a poor man version control system that can automatically provide  you with backup of your programs and scripts. But it works for large binary trees as well  providing incremental backup at a very low cost.

The idea is very elegant and non trivial: if a file that is hardlink is updated, hardlink is broken and the second hardlink automatically became a backup.  This approach is scalable to a very large number of files as on modern filesystems it is difficult to run  out of inodes

"Braking hard links on update" approach allows to keep multiple, full backups instantly available but  saving space to store then probably 10 times or more.  The disk space required is just a little more than the space of one full backup, plus incremental backups.   But if incremental backups needs to restored in reverse chronological order to get a correct result, here any backup looks like full backup -- you can copy it directly.

The current implementation of Rsnapshot has the following key features

Only rsync are required for local backups. for remote backup you need ssh in addition to rsync.

The utility is well documented. Options are mostly directly translated to rsync options. There is way to see the generated rsync command with -t option, for example

rsnapshot -t hourly

There is way to see the generated rsync command with -t option  for example rsnapshot -t hourly

Along with man page there is the rsnapshot HOWTO. The HOWTO will give you a detailed walk-through on how to get rsnapshot up and running in explicit detail.

There is also FAQ FAQ rsnapshot

Installation

All major distros have Rsnapshot already packaged. Here are a few links:

On RHEL installation is possible via RPM.  But in general as this Perl script you can install it yourself without any RPM.

There is an extension with GUI called Rsnap.

Configuration

The first step after the installation is to configure /etc/rsnapshort.conf.

Attention: you editor should not convert tabs into spaces: configuration file requires tabs.

Here is a very simple example when you backup the whole root filesystem on a  flash drive

[root@centos68 ~]# egrep -v "^#" /etc/rsnapshot.conf

config_version  1.2
snapshot_root   /media/Seagate/.snapshots
cmd_rm          /bin/rm
cmd_rsync       /usr/bin/rsync
cmd_logger      /usr/bin/logger
retain  daily   30
retain  monthly 12
verbose         2
loglevel        3
lockfile        /var/run/rsnapshot.pid
one_fs          1
exclude		/proc
exclude		/sys
backup  	/	/home/

There is a couple of gotcha here:

Scheduling

Typically Rsnapshot runs as root by a cron job, or, optionally,  series of cron jobs.  For example

0  4 * * *       /usr/local/bin/rsnapshot daily
0 23 1 * *       /usr/local/bin/rsnapshot monthly

It is possible, however, to run as it as arbitrary user as permissions it requires are just read/write access to target files.  Of course you can't backup OS as a regular user as you will not be able to recreate permissions.  But all you scripts and programs can be backuped this way

It is also possible to run it with an alternate configuration file. So there can be multiple cron jobs each with own configuration file.

The net result on backup media will be series of directories

[root@localhost]# ls -l /mnt/Seagate/.snapshots
drwxr-xr-x    7 root     root         4096 Dec 28 00:00 daily.0
drwxr-xr-x    7 root     root         4096 Dec 27 00:00 daily.1
drwxr-xr-x    7 root     root         4096 Dec 26 00:00 daily.2
drwxr-xr-x    7 root     root         4096 Dec 25 00:00 daily.3
drwxr-xr-x    7 root     root         4096 Dec 24 00:00 daily.4
drwxr-xr-x    7 root     root         4096 Dec 23 00:00 daily.5
drwxr-xr-x    7 root     root         4096 Dec 22 00:00 daily.6
... ... 

Inside each of these directories is a "full" backup of that point in time but it borrows pre-existing files via hardlinks which saves a lot of space.  The destination directory paths you specified under the backup keyword in the configuration file.

The directory in which that will be create is $RSNAPSHOT_HOME, which by default is /.snapshot):

backup          /etc/           localhost/

If you specified just /etc directory for your backup, the /etc/ directory will initially get backed up into /.snapshots/hourly.0/localhost/etc/

History

The original article by Mark Rubel (that inspired rsnapshot) Easy Automated Snapshot-Style Backups with Rsync (2004) contains some interesting information. One such tip is how to have backup mounted rw for root and read-only for regular users

Making the backup as read-only as possible

In the previous section, we discussed ways to keep your backup data physically separate from the data they're backing up. In this section, we discuss the other side of that coin--preventing user processes from modifying backups once they're made.

We want to avoid leaving the snapshot backup directory mounted read-write in a public place. Unfortunately, keeping it mounted read-only the whole time won't work either--the backup process itself needs write access. The ideal situation would be for the backups to be mounted read-only in a public place, but at the same time, read-write in a private directory accessible only by root, such as /root/snapshot.

There are a number of possible approaches to the challenge presented by mounting the backups read-only. After some amount of thought, I found a solution which allows root to write the backups to the directory but only gives the users read permissions. I'll first explain the other ideas I had and why they were less satisfactory.

It's tempting to keep your backup partition mounted read-only as /snapshot most of the time, but unmount that and remount it read-write as /root/snapshot during the brief periods while snapshots are being made. Don't give in to temptation!.

Bad: mount/umount

A filesystem cannot be unmounted if it's busy -- that is, if some process is using it. The offending process need not be owned by root to block an unmount request. So if you plan to umount the read-only copy of the backup and mount it read-write somewhere else, don't -- any user can accidentally (or deliberately) prevent the backup from happening. Besides, even if blocking unmounts were not an issue, this approach would introduce brief intervals during which the backups would seem to vanish, which could be confusing to users.

Better: mount read-only most of the time

A better but still-not-quite-satisfactory choice is to remount the directory read-write in place:

mount -o remount,rw /snapshot
[ run backup process ]
mount -o remount,ro /snapshot

Now any process that happens to be in /snapshot when the backups start will not prevent them from happening. Unfortunately, this approach introduces a new problem--there is a brief window of vulnerability, while the backups are being made, during which a user process could write to the backup directory. Moreover, if any process opens a backup file for writing during that window, it will prevent the backup from being remounted read-only, and the backups will stay vulnerable indefinitely.

Tempting but doesn't seem to work: the 2.4 kernel's mount --bind

Starting with the 2.4-series Linux kernels, it has been possible to mount a filesystem simultaneously in two different places. "Aha!" you might think, as I did. "Then surely we can mount the backups read-only in /snapshot, and read-write in /root/snapshot at the same time!"

Alas, no. Say your backups are on the partition /dev/hdb1. If you run the following commands,

mount /dev/hdb1 /root/snapshot
mount --bind -o ro /root/snapshot /snapshot

then (at least as of the 2.4.9 Linux kernel--updated, still present in the 2.4.20 kernel), mount will report /dev/hdb1 as being mounted read-write in /root/snapshot and read-only in /snapshot, just as you requested. Don't let the system mislead you!

It seems that, at least on my system, read-write vs. read-only is a property of the filesystem, not the mount point. So every time you change the mount status, it will affect the status at every point the filesystem is mounted, even though neither /etc/mtab nor /proc/mounts will indicate the change.

In the example above, the second mount call will cause both of the mounts to become read-only, and the backup process will be unable to run. Scratch this one.

Update: I have it on fairly good authority that this behavior is considered a bug in the Linux kernel, which will be fixed as soon as someone gets around to it. If you are a kernel maintainer and know more about this issue, or are willing to fix it, I'd love to hear from you!

My solution: using NFS on localhost

This is a bit more complicated, but until Linux supports mount --bind with different access permissions in different places, it seems like the best choice. Mount the partition where backups are stored somewhere accessible only by root, such as /root/snapshot. Then export it, read-only, via NFS, but only to the same machine. That's as simple as adding the following line to /etc/exports:

/root/snapshot 127.0.0.1(secure,ro,no_root_squash)

then start nfs and portmap from /etc/rc.d/init.d/. Finally mount the exported directory, read-only, as /snapshot:

mount -o ro 127.0.0.1:/root/snapshot /snapshot

And verify that it all worked:

mount
...
/dev/hdb1 on /root/snapshot type ext3 (rw)
127.0.0.1:/root/snapshot on /snapshot type nfs (ro,addr=127.0.0.1)

At this point, we'll have the desired effect: only root will be able to write to the backup (by accessing it through /root/snapshot). Other users will see only the read-only /snapshot directory. For a little extra protection, you could keep mounted read-only in /root/snapshot most of the time, and only remount it read-write while backups are happening.

Damian Menscher pointed out this CERT advisory which specifically recommends against NFS exporting to localhost, though since I'm not clear on why it's a problem, I'm not sure whether exporting the backups read-only as we do here is also a problem. If you understand the rationale behind this advisory and can shed light on it, would you please contact me? Thanks!


Top Visited
Switchboard
Latest
Past week
Past month

NEWS CONTENTS

Old News ;-)

[Jul 15, 2017] How To Backup Your Entire Linux System Using Rsync

Apr 25, 2017 | ostechnix.com

First, insert your backup medium (Pend drive or External hard disk). Then find the drive letter using 'fdisk -l' command. In my case, my Pen drive id is /dev/sdb1. Mount your drive to any location of your choice.

sudo mount /dev/sdb1 /mnt

To backup the entire system, all you have to do is open your Terminal and run the following command as root user:

sudo rsync -aAXv / --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} /mnt

This command will backup the entire / directory, excluding /dev, /proc, /sys, /tmp, /run, /mnt, /media, /lost+found directories.

Let us break down the above command and see what each argument does.

Please be mindful that you must exclude the destination directory, if it exists in the local system. It will avoid the an infinite loop.

[Nov 21, 2008] rsnapshot - Local-Remote Filesystem backups utility in openSUSE SUSE & openSUSE

Perl codebase might be reused.
October 4, 2008
rsnapshot is a filesystem backup utility based on rsync. Using rsnapshot, it is possible to take snapshots of your filesystems at different points in time. Using hard links, rsnapshot creates the illusion of multiple full backups, while only taking up the space of one full backup plus differences. When coupled with ssh, it is possible to take snapshots of remote filesystems as well.

rsnapshot is written in Perl, and depends on rsync. OpenSSH, GNU cp, GNU du, and the BSD logger program are also recommended, but not required. rsnapshot is written with the lowest common denominator in mind. It only requires at minimum Perl 5.004 and rsync. As a result of this, it works on pretty much any UNIX-like system you care to throw at it.

rsnapshot can run almost out of the box with very little configuration changes although advanced configurations can be done with little more effort.

rsnapshot

Official website
rsnapshot is a filesystem snapshot utility based on rsync. rsnapshot makes it easy to make periodic snapshots of local machines, and remote machines over ssh. The code makes extensive use of hard links whenever possible, to greatly reduce the disk space required.

Depending on your configuration, it is quite possible to set up in just a few minutes. Files can be restored by the users who own them, without the root user getting involved.

There are no tapes to change, so once it's set up, your backups can happen automatically untouched by human hands. And because rsnapshot only keeps a fixed (but configurable) number of snapshots, the amount of disk space used will not continuously grow.

It is written entirely in Perl with no module dependencies, and has been tested with versions 5.004 through 5.16.3. It should work on any reasonably modern UNIX compatible OS.

rsnapshot was originally based on an article called Easy Automated Snapshot-Style Backups with Linux and Rsync, by Mike Rubel.

If this is your first experience with rsnapshot, you may want to read the rsnapshot HOWTO at http://www.rsnapshot.org/ . The HOWTO will give you a detailed walk-through on how to get rsnapshot up and running in explicit detail.

rsnapshot comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. See the GNU General Public Licence for details.

Recommended Links

Google matched content

Softpanorama Recommended

Top articles

Sites

Top articles

Sites

General

Distribution specific pages



Etc

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D


Copyright © 1996-2018 by Dr. Nikolai Bezroukov. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) in the author free time and without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: October, 25, 2019