Relax-and-Recover on RHEL
Version 2.1, Dec 14, 2020
Relax-and-Recover a bare metal disaster recovery and system migration software. It is a complex, somewhat brittle because of this
complexity, and not well documented (as usual for open source packages).
- RHEL6 packages really old version in RPM for RHEL6 -- 1.17 (in comparison RHEL 5.x uses version 2.0 ;-), The trick is that after
installation you can overwrite Rear with version from RHEL7. It will work.
- RHEL7 packages version 2.4 which is more usable.
Stable releases are available as tgz files from the
Sourceforge Downloads page. Each tar file contain the
main script that should go into /usr/sbin, the set of directories in /usr/shar/rear as well as documentation and sample
config file. There is also a make file for installation.
At the core, ReaR is a a set of scripts written in BASH. Scripts themselves are well written, demonstrate a good BASH style
and as such represent ultimate documentation to the package, so they do compensate for the low quality of the documentation. As the
package is designed for sysadmin it is natural to assume that they are cable to study and debug, if necessary, bash scritps.
In addition to that, rear provides several debugging options (see below) that
provides additional insight into its workflow and allow to debug individual modules by simulation invocation environment for them.
Version 1.17 which Red Hat 6 supplies is broken for G7 version of HP servers: you can backup, but you can't restore. Interestingly
enough it does work for HP G6 servers. If you discover this situation one option is to try to upgrade to version 2.x. If version 1.17
is installed, the upgrade can be done by simply copying /usr/sbin/rear executable and /usr/sbin/share tree from tar
of version 2.x. Please note, that version 2.x has three digit numeric prefixes for scripts, while scripts in version 1.x have a two
digit prefix. If this hack helps and you can restore the test server, you can write a script or make file to propagate those updates
to all relevant servers.
|
If version 1.17 is installed, the upgrade can be done by simply copying /usr/sbin/rear executable
and /usr/sbin/share tree from tar of version 2.x.
|
The quality of console messages in ReaR is rather low. They do not include the name of script that generated them and they often
miss fatal errors. You can't rely on them, but the quality of log written to /var/log/rear is good. It is better to watch ReaR
log then console messages. The package has a useful debugging mode (see Learning
rear internals and debugging ) which can and should be used to learn more about this system.
There are several YouTube videos about this package which helps to understand it (with two of them being quite informative):
- First steps with Relax-and-Recover (ReaR)
A very good presentation . Probably the best I found on YouTube. Published on Mar 7, 2017 included git installation and experimentation
with Rear
- 2017 Relax-and-Recover Automated Testing
- Other that might be useful but mostly flaff
In any bare metal recovery utilities the devil is in details and relax-and-restore managed to account for most important details
right. That's why despite it fragility it is a valuable backup solution:
- Bootable image on USB should not prevent normal boot. There should 30 sec timeout after which normal boot should continue.
Actually I uncounted a nasty bug in rear 1.17.2 on DL580 G7 server -- on this server after 30 sec timeout rear produces that red
screen with the message invalid opcode or something like that. That stresses the importance of testing as this software
is pretty brittle. You still have the tarball, so this is just a nuisance, not a total failure.
- Backup solution should be able to add itself to grub as an entry in boot menu. In Rear this is achieved via
GRUB_RESCUE=1
in your local configuration. The entry in the boot loader is password protected. The default password is REAR. Change it in
your own local.conf. For example GRUB_RESCUE_PASSWORD="SECRET"
- Ability to use NFS for backup and presence of NFS in rescue image
Storing on a central NFS server
The most straightforward way to store your DR images is using a central NFS server. The configuration below will store both
a backup and the rescue CD in a directory on the share.
OUTPUT=ISO
BACKUP=NETFS
BACKUP_URL="nfs://192.168.122.1/nfs/rear/"
- Linux is now split into half dozen major distributions, each with its quirks and ReaR supposedly covers most important of
them correctly. That adds complexity, but this is not a fault of relax-and-recover developers, but of multiple personalities
of Linux, which seized to be a single OS long ago, although still by inertia is viewed as such. We cover only RHEL, but other popular
distributions such as Debian and SLES exist. Relax-and-restore covers them all. It can be used for Suse, Ubuntu, Debian, ArchLinux,
etc. Please note that distributions with systemd present additional problems. As systems with LVM for root partition (you
better avoid this trap, recommended by Red Hat)
- For many types of servers local harddrives contains more then 256GB of data (of those OS directory consume less then 6GB --
really a rounding error) and in such cases flash drive in not large enough for full backup and you need to switch to NFS, or SSD
drive or rotational USB drive. Currently flash drives in FIT form factor flash drive are limited to 256GB (Sandisk has 256GB
model). You can also switch to NFS or rsync and put the backup on a remote server were there is a space for all backups and then
boot with the right IP and network setting to restore it.
-
Flash Drive - SDCZ430-256G-G46 Computers & Accessories
- Bash is not very suitable language (or let's say it politely;-) for writing such programs. The total program exceed the
complexity level for a typical bash program. That create its own set of difficulties, although I like the style of programming used
in rear. Although the resulting system probably is easier to understand by curious sysadmin, then a similar system written, say,
in Perl, the debugging of it is not. Also the question is: "What is the percentage of curious sysadmins among linux sysadmin population?
" is a good question. With the current trend of "Windowization" of Linux advocated by Red Hat (systemd and friends) this
is dying specie, replaced by "click-click-click" types.
By default ReaR uses GNU tar for actual backup. The main advantage of using tar is that it is a well known by all
sysadmins utility which is very reliable. Another advantage is that you do not need to learn something new, as "stack overflow" for
sysadmins is a real problem (especially in RHEL7 which is way too complex for mere mortals): too many, too complex packages to maintain.
And that greatly increases the confidence that your backup will be usable when the disaster strike -- you can use it even if rear
breaks during recovery due to some bugs (happened to me). Just boot from RHEL ISO into rescue mode and you can perform restore.
Wrapper scripts included in ReaR are quite complex. You need to learn how to experiment with them as the documentation is dismal.
The key problem with the packages like Relax and Recover is that they create there own "universe" with complex, non-obvious rules of
submitting additional parameters (for example tar exclude list). This idiotism
is so widespread that it can be called a new normal for open source. Just look at systemd -- no respect to their predecessors whatsoever.
Some of those implicit rules are never documented and require experimentation to understand; most can be understood only looking at
the code and running test cases.
| The key problem with the packages like Relax and Recover is that they create there own "universe" with complex, non-obvious
rules of submitting additional parameters (for example tar exclude list).
This idiotism is so widespread that it can be called a new normal for open source. Just look at systemd -- no respect to their
predecessors whatsoever. |
So outside simplistic usage (format/mkrescue/mkbackup troika with default /etc/rear/local.conf file) you are in
Alice in Wonderland environment and often it is unclear what ReaR
will do in particular case without experimentation. One trick that I have found useful it look at Rear tmp directory on the
fly after rear started. You will see multiple files in it some of them should include the parameters you supplied.
Another common sense option is to look at transferred bytes indicator. It is does not look right -- for example the amount of transfered
bytes is bigger that you the total of your local filesystems -- you can be sure that some parameters you supplied (typically the
list of excluded filesystems) are wrong.
Note that even simple operation as format of USB drive has gotchas -- you need to supply the device name, not the partition name
to it (as you boot form the device not a partition ;-). And it should be unmounted. More sophisticated script would accept
both and ask via prompt is this your real intention to create bootable partion on the device.
In other words Rear if far from AKM47 type simplicity and reliability of backup (while plain vanilla Tar is close ;-). And unless
you are willing to look at the code (you should -- it is well written code; a rear example of a good style of Bash programming ;-) you
might never fully understand the capabilities of this software (not that everybody need to; simplistic usage is OK for many).
Still despite all those warts ReaR can be used, as default configuration that create the rescue disk alone is worth some studing
of this software. Moreover more complex configuration can be debugged using try and error process. As linux does not have any built-in
bare metal recovery utility other then DD (but with it you face problem
writing zeros on unused parts of filesystem before making backup with it in order
to save space) you actually do not have too many alternatives ;-).
The other important thing is that because it is well written BASH scripts you can learn a lot of useful staff about BASH programming
experimenting with it. If you are willing to look at the code, of course.
While "added complexity" often kills such packages, the issues in bare metal recovery are complex and at least as a "poor man" bare
metal backup this package is a good candidate. It allows you to understand what is involved and how to solve certain problems inherent
in bare metal backup. And what is important is that in simple cases it works as expected. So if you do not need to deviate from typical
operations too much, for example, just adding the exclusion of some filesystems from the backup, you are OK. But you need to test whether
it is able to restore the system (which, of course, also is not given). but if you tarball is right, you can always restore it
without rear scripts. That's huge advantage as stakes in such cases are very high.
In a simplified way Rear is equivalent to booting to the rescue mode on your RHEL distribution DVD, restoring MBR (which should
be a part of tarball anyway, most sysadmins write it into /root with dd), and then using a tar file on a USB drive to restore the system.
But the devil is always in details.
I would like to stret again importance of the fact that at the core of Relax-and-Recover is tar -- an already known by you utility.
That means that the backup created by ReaR can be restored outside Relax and Recover framework. This is a very positive feature. Again,
if restore fails (as it does with the version 1.17.x shipped with RHEL 6.9 on HP Intel servers like DL580 G7) you still can use tarball
for recovery. In other words all this edifice of configuration files and settings which make rear very convenient (and very brittle)
can be thrown out in case of need.
The key idea of Relax-and-Recover is that it is possible to construct bootable image with mini OS on the USB drive using files already
available on the harddrive of installed Linux system. If you add to this bootable mini disk a tar archive of system directories and
some scripts, you can restore the system from a USB drive. But implementation is wrong. The right approach would be to copy the OS from
the system that is backuped (which just 6GB or less) as all the necessary drivers are present and this installation will definitely
works. With the approach that Relax-and-Recover uses, the constructed bootable image sometimes work, and sometime (on HP servers) does
not. In the latter case you need to boot to the recovery option in the RHEL ISO and you will be able to continue the restoration process.
So bugs in the restore script and configuration is not the end.
| By default rear uses /tmp filesystem for creation of the backup image. If it is not sufficient you need to specify alternative
location for the image for rear to work |
Still when it work, the recovery image represents a pretty capable mini-OS. You can SSH to it, you can add utilities missing on the
fly via SCP fro other servers, you can chroot the restored image. When it does not work the fastest way is to create a second USB with
mini OS (no GUI) installed from DVD and use it instead (two USB ports are not the problem on most servers, even HP ;-). If some USB
ports need to be used for keyboard and mouse use a USB hub.
You can't relax until you test whether the restore mode works on your server or not. It can and does blue screen on some servers.
You still have the tarball so this is just a nuisance, but it kills "relax" part ;-)
|
You can't relax until you test whether the restore mode works on your server or not. It can and does blue
screen on some servers ;-)
|
When it works you have a bare metal recovery solution created using basic Linux components. Of course you can always install minimal
OS on the USB drive and create tar manually. On systems where relax and recover blue screen on recovery this in the next option to try.
But it requires shutting the server down and rebooting it from installation DVD or remotely mounted ISO to create the image. You can
also do it on the fly by copying the OS manually and then installing grub and modifying grub menu to appropriate harddrive. But this
requires some knowledge of grub.
Relax-and-Recover uses tar with the set of tar options and exclude files to created the tarball which later would be used
to restore the system. If backup is not too large it can be put on permanently inserted FIT form factor USB drive. The current upper
size for this low profile form factor is 256 GB (SanDisk
256GB Ultra Fit USB 3.1 ). The same upper limit has metal bar form factor (SanDisk
Ultra Flair). Larger size "bar" form factor like
PNY Pro Elite 512GB scale to 512GB, but that does not make much sense as the SSD drive of the same capacity costs less.
That means that you can backup on 256GB flash drive filesystem with approximately 512GB of used space (assuming 50% compression). After
that it is better to use USB 3.0 External hard drives such as
Samsung T1 or
Samsung T3, which are almost twice cheaper.
Relax-and-Recover produces a bootable image (rescue image) + tar file (backup). This image restore partitioning of the system, including
LVM based partitioning. Once that is done it initiates a restore from backup. Restores to different size of harddrives and slightly
different hardware are possible. Relax-and-Recover can therefore be used:
- As partition resizing tool.
- A migration tool between two similar servers as well.
Currently Relax-and-Recover supports various boot media (incl. ISO, PXE, OBDR tape, USB or eSATA storage), a large variety of network
protocols (sftp, ftp, http, nfs, rsync, cifs).
USB and NFS are probably the most important of them (low profile USBs like FIT have are generally limited to 256GB, so larger backups
should go to NFS, as the most transparent way of storing backups.
It also can be used with pre-existing backup tools such as Bacula, HP DataProtector, IBM TSM, Symantec NetBackup, but that place
it outside of its main advantages area -- simple and reliable bare-metal backup solution. In this combination it is neither simple,
nor reliable ;-)
|
Relax-and-Recover recovery process is brittle. The setup from RPM requires installation of additional
RPMs.
|
Installation of RHEL7 package is pretty straightforward. There are some nuances with the installation of rear of CentOS and
RHEL6
- Please not that in CentOs7 you need explicitly specify installation of syslinux-extlinux. (that might be the case
of Debian too Package dependency error when backing up Debian 8.2
system to USB device · Issue #720 · rear-rear · GitHub ) see For some reason is is not installed as a dependency:
yum install syslinux-extlinux rear
- Installation of RHEL6 is more involved (see below)
Here is protocol of installation for RHEL7 (6 dependent packages are installed):
[0]d620@ROOT:~ # yum install rear
... ... ...
=============================================================================================
Package Arch Version Repository Size
=============================================================================================
Installing:
rear x86_64 2.4-2.el7 base 636 k
Installing for dependencies:
attr x86_64 2.4.46-13.el7 base 66 k
bc x86_64 1.06.95-13.el7 base 115 k
genisoimage x86_64 1.1.11-25.el7 base 299 k
libusal x86_64 1.1.11-25.el7 base 136 k
mtools x86_64 4.0.18-5.el7 base 203 k
syslinux x86_64 4.05-15.el7 base 990 k
Transaction Summary
=============================================================================================
Install 1 Package (+6 Dependent packages)
Total download size: 2.4 M
Installed size: 6.5 M
Is this ok [y/d/N]: y
... ... ...
I spend several hours trying to install in of RHEL 6.8 until I figured out what to do. This is not how decent open source software
should be documented and developers should be ashamed. The installation on RHEL fails if the supplementary software channel
is not enabled before you attempt the installation. But it fails in its own particular way. Software is installed successfully, you
can format USB drive, but when you try to make backup you get them message that syslinux-extlinux package is missing.
Also it you download those packages (for batch installation on many servers) and try to install them using yum for some reason on RHEL
6.x you face incompatibility of libraries problem and installation fails. Among packages you need
- syslinux-extlinux (from Red Hat supplementary software channel). This is a show stopper. Without it backup can't be
created.
- syslinux
- mingetty or agetty
- sfdisk or parted
- grub2-efi-modules or similar package that provides Grub2 modules ( /usr/lib/grub(2)/ ) if USB recovery on EFI is
used
On RHEL 6.5-6.8 you can install rear using the commands
yum-config-manager --enable repository rhel-6-server-optional-rpms
yum install rear syslinux-extlinux genisoimage syslinux
The most typical and the simplest setup for recovery with ReaR is when your backup is on bootable USB drive or on rather small ISO
file (most servers can boot from ISO file). You backup generally consist of two parts
- Bootable mini-OS
- Tar archive with the backup of your server
To recover from a failed drive just boot from USB. If you use NDF backup you need to boot from this ISO on NFS filesystem.
If needed this ISO can be transferred to your PC and mounted as bootable image via ILO or DRAC.
You will be presented with the inital menu, click of the first (recover server name option). You will be presented with
submenu that contain two recover option (many in several variants corresponding to different dates):
Manual gives you far more flexibility. The process is described in some details in the instructions provided
Red Hat for RHEL and it will work on CentOS and other clones. Please note that RHEL the recovery is complicated by existence of
SE Linux, if it is enabled.
|
Choosing manual recovery option you get into to the command prompt of a pretty capable mini-OS
Networking is enabled and SSH daemon is running, so you can SSH to it, which is huge advantage if you are
working with old ILO or DRAC with split cursor.
|
Choosing manual recovery option you get into to the command prompt of a pretty capable mini-OS. Networking is enabled and SSH daemon
is running, so you can SSH to it, which is huge advantage if you are working with old ILO or DRAC with split cursor.
As you have access to your network you can add any utilities you want but which are missing on the fly via SCP.
You can also chroot the the harddrive with root filesystem and /boot filesystem. In the latter case you can edit grub settings.
You can also reinstall grub (not sure how it works, if it works at all if /boot is a separate filesystem).
When the recovery does not work probably because mini-OS created does not boot correctly, the fastest workaround to create a second
USB with mini OS (no GUI) installed from DVD and use it instead (two USB ports are not the problem on most servers, even HP). If you
have insufficient USB ports or no USB ports ( on HP blades you need some stupid dongle) you can always use your PC to load the
initial ISO from DRAC or ILO. If you have just
one spare USB port you can use a USB hub.
Tar archive with data can be located on USB disk, of better on NFS filesystem. It can moved to hard drive and then unpacked (if
you have enough disk space). Or some files can be extracted from it for any purpose you wish, including
recovery of some lost files.
|
Tar archive with data located on USB disk or, better, on NFS filesystem. It can moved to hard drive and
then unpacked (if you have enough disk space). Or some files can be extracted from it for any purpose you wish, including
recovery of some lost files.
|
There are some interesting possibilities to use Relax-and-Recover for migrating systems to slightly different hardware (which is
often the case if computer is damaged beyond repair, for example some chip on motherboard failed. In this case Relax-and-Recover provides
step-by-step menus which help make decisions to restore the OS and data in new server.
As source code is available you can look into it and experiment with running it via bash with -x option. It also has logging and
optionally "extended output" mode which helps to understand the concepts behind Relax-and-Recover, troubleshoot during initial configuration
and help debug during integration.
Notes:
In almost all circumstances you have to configure two main settings and their parameters: The BACKUP method and the
OUTPUT method.
The backup method defines, how your data was saved and whether Relax-and-Recover should backup your data as part of the mkrescue
process or whether you use an external application, e.g. backup software to archive your data.
The output method defines how the rescue system is written to disk and how you plan to boot the failed computer from the rescue system.
See /usr/share/rear/conf/default.conf for an overview of the possible methods and their options. For example
BACKUP=NETFS
OUTPUT=ISO
ReaR uses the following four variables to define how it backup particular system:
- BACKUP variable defines the method. We will discuss only NETFS method. The other useful method is RSYNC.
- BACKUP_URL variable defines the location where to store the backup archive. When using BACKUP=NETFS you should provide
the backup target location through the BACKUP_URL variable. Possible BACKUP_URL settings are:
- BACKUP_URL=file:// To backup to local disk, use BACKUP_URL=file:///directory/path/
- BACKUP_URL=nfs:// To backup to NFS disk, use BACKUP_URL=nfs://nfs-server-name/share/path
- BACKUP_URL=usb:// To backup to USB storage device, use BACKUP_URL=usb:///dev/disk/by-label/REAR-000 or use a real
device node or a specific filesystem label. Alternatively, you can specify the device using USB_DEVICE=/dev/disk/by-label/REAR-000.
If you combine this with OUTPUT=USB you will end up with a bootable USB device.
- Other target that we will not discuss
- BACKUP_URL=tape:// To backup to tape device, use BACKUP_URL=tape:///dev/nst0 or alternatively, simply define TAPE_DEVICE=/dev/nst0
- BACKUP_URL=cifs:// To backup to a Samba share (CIFS), use BACKUP_URL=cifs://cifs-server-name/share/path. To provide
credentials for CIFS mounting use a /etc/rear/cifs credentials file and define BACKUP_OPTIONS="cred=/etc/rear/cifs"
and pass along:
- username=_username_
password=_secret password_
domain=_domain_
- BACKUP_URL=sshfs:// To backup over the network with the help of sshfs. You need the fuse-sshfs package before
you can use FUSE-Filesystem to access remote filesystems via SSH. An example of defining the BACKUP_URL could be:
BACKUP_URL=sshfs://root@server/export/archives
- OUTPUT variable defines the Rescue media (OUTPUT). We will discuss only two out of possible OUTPUT settings:
- OUTPUT=ISO Create a bootable ISO9660 image on disk as rear-$(hostname).iso
- OUTPUT=USB Create a bootable USB disk (using extlinux). In this case you should use BACKUP_URL=usb:// as the target
location.
For NETFS method you have to define at least three variables for backup to be successful
- BACKUP=NETFS
- BACKUP_URL= (can be file:/// or usb:/// of nfs:///
- OUTPUT= (can be USB or ISO in examples below)
Useful additional variables depends of your BACKUP_URL. We will discuss then for each of three BACKUP_URLs we cover in this
page. One general useful setting is:
NETFS_KEEP_OLD_BACKUP_COPY=y # If you want to keep the previous backup archive
Another is excluded filesystems. We will discuss it later (see Excluding
large filesystems from the backup )
This is the most popular initial configuration of ReaR. You should proceed as following:
- Prepare your USB media. We assume that USB media is at /dev/sdb.
|
Before formatting the device should be unmounted. The current version of rear does not unmount it and
simply fails with a cryptic message
|
Again, you better check this. The device should unmounted before you can start (USB disk are typically mounted on insertion
to /media folder)
Like always the case with format existing data will be destroyed. Formatting is very slow and can take for 32GB flash drive ten
minutes on USB 2.0 connection. The drive is formatted as Ext3 filesystem.
/usr/sbin/rear format /dev/sdb # YOU NEED the DEVICE NAME NOT A PARTITION NAME, IT ALSO SHOULD BE CORRECT DEVICE ;)
NOTES:
- Here you need device name, not the partition name /dev/sdb1
- You can format the drive yourself without using REAR. That's probably the "best practice" with the current version.
After you type Enter, Relax-and-Recover asks you to confirm that you want to format the device. If everything is correct and you
do not accidentally specify you primary harddrive :-), then answer
Yes
The most common reason for this command to abort with error is that the device is still mounted.
If operation is successful, the device will be labeled REAR-000 by the format workflow. This is the default label
and you change it via configuration file if you object to it.
- Edit the /etc/rear/local.conf configuration file. For example you can write initial set of pretty generic
parameters using cat:
cat > /etc/rear/local.conf <<EOF
### write the rescue initramfs to USB and update the USB bootloader
OUTPUT=USB
### create a backup using the internal NETFS method, using 'tar'
BACKUP=NETFS
### write both rescue image and backup to the device labeled REAR-000
BACKUP_URL=usb:///dev/disk/by-label/REAR-000
EOF
If you have multiple boxes you can distribute this file to other boxes: the content of this file is server independent. If you
already have such a file on other server, you can scp it to the new server you are configuring.
Typically you also can benefit from the exclusion of certain directories, but not at this stage of learning.
- Make a backup of your /boot directory (see below). Normally sysadmin usually have such a backup, but
on some servers they might nor. If not just do it (see below)
- CHECK that your USB drive is mountable and is not READ ONLY. That sometimes happens with USB drives that they switch
to read-only mode and in this case Rear might happily delete /boot directory on your system.
You need to verify that your USB drive used for backup can be mounted and is writable.
That's important as Rear misbehaves if USB drive can't be mounted
View the log immediately after the operation you specified ended and browse it for error messages
|
- Create a rescue image. This is essentially a test run as this is pretty quick operation. The size of rescue image
is about 100MB. We want verbose output (-v option).
sudo /usr/sbin/rear -v mkrescue
You will get output similar to the following:
Relax-and-Recover <version>
Using log file: /var/log/rear/rear-<hostname>.log
Creating disk layout
Creating root filesystem layout
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Creating initramfs
Writing MBR to /dev/sdb
Copying resulting files to usb location
If you can create rescue image that means your rear installation works at least in this part, but you need to restore it at least
once to be sure that respore part is OK too.
NOTE: if you forgot to create configuration file rear will write rescue ISO image to /var/lib/rear/output/rear-`hostname`.iso
- Check the log file for possible errors. This is a VERY important step and should not be skipped -- rear is a brittle
software.
And it can play jokes with you.
Checking the log is a very important step and should not be skipped -- rear is a brittle software.
And it can play jokes with you when you need to recover your server
|
I would also recommend to use some "we do not care about server" and try to restore it using your version of ReaR to make sure
that it does not contain catastrophic bugs. Because you can encounter one at the most inopportune moment. But the structure of rear
backup is useful even if restore for a particular version is not working. And the backup tar archive is usable too.
That means that you can restore the system step by step yourself following the scripts rear provides. All the necessary information
is still present that that&#rire39; the most important part of the whole exercise.
- Reboot your system and try to boot from the USB device. You should get a self-explaining boot menu. Verify that the server
can boot itself after 30 sec delay. On HP 580 G7 with rear 1.17.2 I encountered a bug due to which it blow up with the message invalid
opcode of something like that. You can try restore the OS if this is a test computer to ensure that the version of Relax and Recover
you use does not contain some glaring errors.
- If your USB device has enough space, initiate a backup. Backup of filesystem containing, say 32GB on the flash drive
is a long operation which take an hour or so so you need to submit it either in screen session (I experienced problems with
running it with nohup prefix; also you will not see any output until it ends which is a bad idea) You can use the following
command:
/usr/sbin/rear -v mkbackup
Important:
As tar is gziped you get (depending on the content on you filesystems) approximately 2/3 smaller image the the size of filesystem.
That means that 64GB flash can store backup of ~100GB data. With 256GB FIT drive (the max size of thise type of derive as of December
2017) you probably can backup half a terabyte of data.
If you have more then that you need to use iether NFS or rsync for storing backup on remote server.
You can also use USB drive instead of flash drive. 6TB Seagate USB 3.0 SMR drive (so it's supposed to be slow and not to be used
as system drive - for archive purposes only) is now about $130 (at Amazon); A more traditional drive will cost you $180; one terabyte
SSD drive is around $300.
One of the positive things about Relax and Restore is that the rescue image configures internet for you, so all shares that are available
on "normal" server are accessible from the rescue image boot-up. It is not that difficult to configure network manually, but when it
is done automatically for the you it is even better.
After you have such an image you are now better better prepared for catastrophic failure of harddrives or other server components.
But if server has large partitions you need to use NFS of rsync to put the image on a remote filesystem. that would be the next step
in learning rear.
You can put your backup on any locally mounted filesystem. Similar to previous, but you define in configuration OUTPUT=ISO
and directory to store your backup in BACKUP_URL (ATTENTION: this is a directory, not a file)
BACKUP=NETFS
OUTPUT=ISO
BACKUP_URL=file:///DIRECTRY_WITH_YOUR_BACKUP
NOTE: Within this target directory rear will create subdirectory with the name of the server. So ISO will be in this subdirectory,
not in the specified directory.
Of course, you can backup to any mounted NFS share using file method (see above), but in this case your backup will not be available
during the restore operation as this filesystem will not be mounted. So in such cases you need to define BACKUP_URL (ATTENTION:
this is a directory, not a file) in the form that can be used during the restore operation and provide IP parameters to
endure that this server is assessable.BACKUP_URL=nfs://server/path
For example:BACKUP_URL="nfs://192.168.122.1/nfs/rear/"
And since typically all servers use NTP for time synchronization, you should probably also add these lines to iether
/etc/rear/local.conf or to /etc/rear/site.conf as this setting is typically site wise:
TIMESYNC=NTP
During restore operation, you need to have access to correct network settings. If you have DHCP on this segment you can use
USE_DHCLIENT=yes
Otherwise you need to ask Rear to save your static networking settings via the directive:
USE_STATIC_NETWORKING=y
Example: NFS Share (osbconf.org)
# Create ReaR rescue media as ISO image
OUTPUT=ISO
# optionally define backup software, e.g. TSM, NBU, DP, BACULA
BACKUP=NETFS
NETFS_URL=nfS://e6500osl23/srv/backup
NETFS_KEEP_OLD_BACKUP_COPY=Y
# the following is required on older VMware VMs
# MODULES_LOAD=( vmxnet )
# line below was automatically added by 21_include_dhclient.sh
DHCLIENT_BIN=dhcpcd
# line below was automatically added by 21_include_dhclient.sh
DHCLIENT6_BIN=
# Enable the use of cfg2html (set to non-empty to disable). cfg2html must be
installed independently from ReaR.
SKIP_CFG2HTML=N
# Warn about NETFS not being a professional backup solution
NETFS_SKIP_WARNING=Y
# additional tools
PROGS=( "${PROGS[@]}" lsof parted fdisk cfdisk )
You need always check your logs. The fact that you have no warnings on console does not means that your backup is successful. And
if you do not read logs you might discover this sad fact only when you try to revocer failed system from backup.
|
Always check your logs. The fact that you have not warnings on console does not means that your backup is
successful
|
During backup to USB drive the most typical error is the situation when there backup is lager then free space on the device left.
If flash drive is used, most often the size of the backup simply exceeds the size of the USB drive used. If ReaR is aborted, it does
not clean the space on the USB drive, so you need to do it manually or subsequent backup will fail.
What exactly happened often in not clear and only analysis of the log file can tell you exact situation and point to the root case.
So the analysis of the log file is a must.
To configure Relax-and-Recover you have to edit two configuration files in /etc/rear/
Settings that are common to all your server in particular segment of datacenter should generally go to site.conf (which
is intended to be distributed to all servers in your backup scheme)
Unique for a particular server settings should go to local.conf (which is intended to capture unique settings which are
necessary for some servers and which should not be propagated to other servers) .
Settings in local.conf overwrite settings in site.conf.
All other configuration files hold defaults for various distributions and should not be changed.
In /usr/share/rear/conf/ there are also some configuration files which you can study. They have extension .conf
The most useful for study is default.conf (which is also the largest). It is well commented and contains ton of variable
that you can set (and hope that they work and you understand their semantics correctly, which is not given ;-). This "preoccupation
with variables" is a weak point of ReaR, but we have what we have.
Using the site.conf does not makes any sense if you do not distribute it to all your systems. If you have identical configuration
on all servers you can also distribute local.conf t o all servers instead.
In reality you are better off creating subdirectories for each of your rear setup and specifying this directory via option -c.
For example you can create directory
/etc/rear/USB_backup
And use it for backup to USB
And you can create directory
/etc/rear/NFS_backup
and use this directory for creating ISO on NFS
Often you need to exclude some filesystems from backup (typically large data filesystems that are backuped separately via rsync,
or some other method)
|
By default ReaR excludes NFS mounted partitions from backup
|
The most useful for study is /usr/share/rear/conf/default.conf
Source view -etc-rear-default.conf
# Custom command backup stuff
# BACKUP=EXTERNAL
# examples for external backup. In this mode your external program must do EVERYTHING
# In the example below we backup / to the vms host via tar and netcat
# NOTE: The EXTERNAL_* commands can be also defined as an array () to better protect
# arguments with blanks
# NOTE: The EXTERNAL_* commands will be run inside eval like this:
# eval "${EXTERNAL_BACKUP[@]}"
#
# Command to backup the required data
# This example saves the data via SSH to a remote system called vms
EXTERNAL_BACKUP="tar -c -l -z / | ssh vms 'cat >rear64/backup.tar.gz'"
# Command to restore the data
EXTERNAL_RESTORE="ssh vms cat rear64/backup.tar.gz | tar -C /mnt/local -x -z"
# Command to verify the availability of the backup resource, will be executed only if PING=1
# NOTE: This command will be run within the ProgressBar system !! You should therefore avoid
# output on STDOUT by rerouting that to FD 8, the progress bar
EXTERNAL_CHECK="ssh vms date 1>&8"
# How to exclude something ----- EXCLUDES -------
#
# You cannot exclude a device (e.g. /dev/sdg) directly. Instead you have to exclude everything
# ON that device and then the dependancy tracker will automaticall exclude the device from the
# recovery (because there won't be any recovery information for that "unnecessary" device).
#
# Furthermore, you have to exclude MD devices and LVM2 volume groups separately as there is no
# automatic detection of these dependancies (yet, please write and submit it !)
# Exclude filesystems by specifying their mountpoints. Will be automatically added to the
# $BACKUP_PROG_EXCLUDE array during backup to prevent the excluded filesystems' data to
# be backed up
# examples: /tmp
# /media/bigdisk
EXCLUDE_MOUNTPOINTS=()
# Exclude MD devices
# examples: /dev/md0
# /dev/md/0
EXCLUDE_MD=()
# Exclude LVM2 volume groups. This will automatically exclude also the creation of the corresponding
# physical and logical volumes that belong to the excluded volume group.
#
# NOTE: YOU MUST ALSO EXCLUDE THE CORRESPONDING MOUNTPOINTS IN EXCLUDE_MOUNTPOINTS (see above)
# OTHERWISE THE RECOVERY ***WILL*** TRY TO RECREATE THE FILESYSTEMS ONTO NON-EXISTING LVs
#
# Y O U H A V E B E E N W A R N E D ! ! !
EXCLUDE_VG=()
You can use option EXCLUDE_MOUNTPOINTS to exclude some mounted filesystems. For example:
EXCLUDE_MOUNTPOINTS=( '/Usbdrive' '/Apps' '/Scratch' )
This is a BASH array which supposedly is added to "main exclusion array" which is BACKUP_PEXCLUDE (and which already contains
several components such as /tmp, so you should not overwrite it)
BACKUP_PROG_EXCLUDE
You could also try running rear with the -d switch. In your /tmp/rear-XXX/ folder should be a backup-exclude.txt
file. It's in tar exclude syntax. That's an interesting file to study, because if it is incorrect your tar backup will fail
miserably. Using rear dump is not that helpful as it does not include those parameters
To illustrate a mess we are dealing with look at the discussion that I have found
Excluding a folder in a rear backup · Issue #216 · rear-rear · GitHub
cocampbe commented
on Mar 27 2013
I've been perusing the docs and have not come up with a solution yet. I am trying to exclude a folder that contains a large amount
of logs. Here is what I have in the local.conf.ONLY_INCLUDE_VG=( VolGroup00 )
EXCLUDE_COMPONENTS=( "fs:/usr/openv/logs/nbftsrvr" )
So far this does not work. I am not sure if the order matters, or if I am using the wrong exclude directive. Any insight is greatly
appreciated.
jhoekx on Mar 27 2013
Member
EXCLUDE_COMPONENTS only works for real system components like filesystems or logical volumes. If you just
want to exclude a directory you have to use the BACKUP_PROG_EXCLUDE array.
Cool. I will do that. I was about to try that, but I just wasn't sure. I also thought about just trying EXCLUDE_MOUNTPOINTS even
though it technically is not.
jhoekx
on Mar 27 2013
Can you show us the local.conf you're using? To make sure the BACKUP_PROG_EXCLUDE syntax is OK.
OUTPUT=ISO
BACKUP=NETFS
BACKUP_URL="nfs://some_server/rear/"
NETFS_KEEP_OLD_BACKUP_COPY=1
ONLY_INCLUDE_VG=( VolGroup00 )
BACKUP_PROG_EXCLUDE=( /usr/openv/logs/nbftsrvr )
I just added single quotes to BACKUP_PROG_EXCLUDE=( '/usr/openv/logs/nbftsrvr' ). trying again.
jhoekx
on Mar 27 2013
The line is not entirely correct. It should be:
BACKUP_PROG_EXCLUDE=( "${BACKUP_PROG_EXCLUDE[@]}" '/usr/openv/logs/nbftsrvr' )
You could also try running rear with the -d switch. In you /tmp/rear-XXX/ folder should be a backup-exclude.txt
file. It's in tar exclude syntax. That's an interesting file to share.
I checked the file and it contains /usr/openv/logs/nbftsrvr.
jhoekx
on Mar 27 2013
The file is sent to tar as-is. Maybe try /usr/openv/logs/nbftsrvr/*?
File is now showing /usr/openv/logs/nbftsrvr/*. I am running rear again.
I just tailed the backup.log file and saw that it is not copying those files. So it is working. Looks like I may be backing up something
else that is making the archive rather large. Thanks for your help.
Yep. even though I have ONLY_INCLUDE_VG=( VolGroup00 ), it was backing up a san attached volume. I added it to the exclude list.
Manual excludes
It seems prudent to prevent the external drives from ever being backed-up or overwritten. The default configuration contains these
lines:# Exclude components from being backed up, recreation information is active
EXCLUDE_BACKUP=()
# Exclude components during component recreation
# will be added to EXCLUDE_BACKUP (it is not backed up)
# recreation information gathered, but commented out
EXCLUDE_RECREATE=()
# Exclude components during the backup restore phase
# only used to exclude files from the restore.
EXCLUDE_RESTORE=()
To prevent an inadvertently mounted backup filesystem being added to the restore list, the easiest way is to add the filesystem to
the EXCLUDE_RECREATE array.EXCLUDE_RECREATE=( "${EXCLUDE_RECREATE[@]}" "fs:/media/backup" )
While the motivation of the creators of this backup software were noble, the net result is questionable because complexity
of the resulting scripts bite them in the back. There was not real effort to ensure transparency of the interaction and parameter passing
between the scripts and tar.
The architecture chosen (complex scripts driven by million of env variables) is questionable and ability to correct errors in generated
commands (for example in the tar commend used for the backup) is limited. There are also complex interdependencies, and not all situations
are diagnosed on the console
In pretty common case when that tarball overflow USB drive, there no explicit diagnostics about this important event on the console
and if you do not check logs you can be royally screwed.
When you exceed certain level of complexity with bash scripts the resulting "mess" is not only unmanageable, it is outright dangerous.
Unless you are happy with the default "mkrestore" option, you can be royally screwed (especially if you fail to look at the
logs). Backup can fail due to the lack of space on, say, USB drive, but there will be no diagnostic messages. Or backup can succeed,
but the content of the tarball will be not what you expect. As the result, if you do not check that content of the tarball beforehand,
you can destroy some filesystems during restore operation by restoring filesystems that you did not intent to restore. There are several
other horror stories that can happen, when the complexity of the scripts far exceed you level of understanding and generated command
are not explicitly listed.
Here is one example. In most real cases you need to exclude some filesystems and tar provides this capability. But with the maze
of scripts in ReaR this capability is not available directly in ReaR. You need to populate a special array EXCLUDE_MOUNTPOINTS
:
EXCLUDE_MOUNTPOINTS=( '/Usbdrive' '/Apps' '/Scratch' )
You can't provide you own exclude file, you can't correct generated command with you own script, unless you want to mess with the
source code. And you need to rely on variable provided by ReaR, which depends on the processing logic of corresponding script so without
looking at the script code you will be trying to find a black can in dark room. The logic in the script that govern processing of this
array 40_create_include_exclude_files.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
|
# backup all local filesystems, as defined in mountpoint_device
for k in "${BACKUP_PROG_INCLUDE[@]}" ; do
test "$k" && echo "$k"
done > $TMP_DIR/backup-include.txt
# add the mountpoints that will be recovered to the backup include list
while read mountpoint device junk ; do
if ! IsInArray "$mountpoint" "${EXCLUDE_MOUNTPOINTS[@]}" ; then
echo "$mountpoint"
fi
done <"$VAR_DIR/recovery/mountpoint_device" >> $TMP_DIR/backup-include.txt
# exclude list
for k in "${BACKUP_PROG_EXCLUDE[@]}" ; do
test "$k" && echo "$k"
done > $TMP_DIR/backup-exclude.txt
# add also the excluded mount points to the backup exclude list
for k in "${EXCLUDE_MOUNTPOINTS[@]}" ; do
test "$k" && echo "$k/"
done >> $TMP_DIR/backup-exclude.txt
|
Note that the script adds a trailing backlash to each excluded mount point at line 19. So you should not include
trailing backslash in your spec.
Generally this idea of using approximately a hundred variables (yes around a hundred) that guide the behaviour of ReaR sucks. To
add the insult to injury this capability is not well documented and the logic is too simplistic and requires some additional work
If you put additional slash at the end, or make some other minor mistake the configuration will not work, but no errors will be generated.
the way to check the result is to run manually your exclude file with tar. The backup-exclude.txt file in can be found in
tmp directory used by ReaR and if it is not right you can adjust you input. Here is the list of file in this directory (/tmp/rear.8T1ZsZOFSXWwA6g/tmp/
in our case)
-rwxrwx---+ 1 nnb None 35646232 Apr 3 10:08 backup.log
-rwxrwx---+ 1 nnb None 41 Apr 3 05:37 backup-exclude.txt
-rwxrwx---+ 1 nnb None 18 Apr 3 05:37 backup-include.txt
drwxrwx---+ 1 nnb None 0 Apr 3 16:51 boot
-rwxrwx---+ 1 nnb None 43 Apr 3 05:36 bootloader
-rwxrwx---+ 1 nnb None 28 Apr 3 05:36 copy-as-is-exclude
-rwxrwx---+ 1 nnb None 179899 Apr 3 05:36 copy-as-is-filelist
-rwxrwx---+ 1 nnb None 82642591 Apr 3 05:37 initrd.cgz
drwxrwx---+ 1 nnb None 0 Apr 3 16:51 mappings
-rwxrwx---+ 1 nnb None 93 Apr 3 05:36 parted
-rwxrwx---+ 1 nnb None 31 Apr 3 05:36 partitions
-rwxrwx---+ 1 nnb None 18 Apr 3 05:36 partitions_unsorted
-rwxrwx---+ 1 nnb None 18 Apr 3 05:36 partitions-data
The default content is as following
/tmp/*
/dev/shm/*
/var/lib/rear/output/*
Unlike the default content, all generated entries does not have * at the end, only a slash is added, for example
/Usbdrive/
If you did something wrong it will be visible in log stored in temp directory (/tmp/rear.8T1ZsZOFSXWwA6g/tmp/ in our case)
in the structure of tar command and in this case you need to cancel the backup and change the configuration so that it works as intended
here is an example of botched tar command which included "undesirable" /Usbdrive filesystem into the backup:
2018-04-03 05:37:48.851184722
tar --warning=no-xdev --sparse --block-number --totals --verbose --no-wildcards-match-slash \
--one-file-system --ignore-failed-read --anchored --gzip -X /tmp/rear.8T1ZsZOFSXWwA6g/tmp/backup-exclude.txt \
-C / -c -f - /home / /Usbdrive /var/log/rear/rear-z17.log \
| cat BACKUP_PROG_CRYPT_KEY | dd of=/tmp/rear.8T1ZsZOFSXWwA6g/outputfs/rear/z17/20180403.0536/backup.tar.gz
Tar option -C in tar means change to directory DIR, so this command is running from the root directory. You can
generally test that command separately (in this case you do not need to pipe it to dd of course)
At this point your enthusiasm about this package is probably somewhat deflated. But I would like to stress that mkrecover
option is very useful and works. And you get the tar command that failed, modify it and rerun getting the tarball you need. That's the
advantage of using tar, although architectural decisions taken in ReaR are simplistic (usage of huge number of env variable for driving
the scripts) are definitely wrong and the package is suffering from overcomplexity.
For more complex cases you do need to know bash well to be able to debut this beast.
Another problem is that ReaR does not allow just generate commands it uses, which would allow correction of some subtle errors by
your own scripts. But the fact that you can extract the generated tar command from the log is hugely positive and greatly simplifies
the debugging.
You can also run you own tar command based on the example you have found in the log instead of mkbackup and incorporate such a command
in your own script
This is a BASH array which supposedly is added to to "main exclusion array" which is BACKUP_PROG_EXCLUDE (and which already
contains sevral components such as /tmp, so you should not overwrite it)
Rear has a cron job installed when RHEL package is installed. It adds the file rear to the /etc/cron.d. directory
The idea is to update rescue image automatically each time the layout of the box has been changed (for example you updated the version
of OS) which is a good idea:
30 1 * * * root /usr/sbin/rear checklayout || /usr/sbin/rear mkrescue
ReaR RPM is more or less LSB complaint and puts files in expected places:
- /etc/rear/ Configuration
- /usr/sbin/rear Main program
- /usr/share/rear/ Internal scripts. Scripts are grouped according work-flows
- • mkrescue (only make rescue image)
• mkbackup (includes make rescue image)
• mkbackuponly (excludes make rescue image)
• recover (the actual recovery part; presupposed that you already booted from rescue ISO or USB)
- /var/lib/rear/ Recovery and disk and file system layout information
- /var/log/rear/ Log files of ReaR is kept here
- /tmp/rear.$$/ Build area for the tarball or other type of backup image
ReaR uses several locations during backup process (linuxtag.org
):
- /usr/share/rear/... selects that mkbackup method/scripts:
- /etc/rear/*.conf - configuration files (directory can be changes using option -c )
- /var/lib/rear/
- prep/ - preparation work; checking the environment
- layout/ -- save the disk layout - modules, network, storage,...
- build/ - populate the initial ramdisk for our rescue image
- pack/ - create the initrd and copy kernel
- output/ - create the ISO image and copy to OUTPUT URL
- backup/ - make the backup archive to BACKUP URL
There is two presentations that help you to understand how to experiment with ReaR:
The simplest way is to create two virtual machines and backup on one and recover on the another.
There are several options that help you with the debugging:
- -D -- dump use the -D option to dump every function call to the log file; this is very convenient during development
or when troubleshooting Relax-and-Recover
- -s -- simulation mode; show what scripts rear would include. use the -s option to see what scripts Relax-and-Recover
would be using; this is useful to understand how Relax-and-Recover is working internally
- -S -- step-by-step mode; acknowledge each script individually. use the -S option to step through each individual
script when troubleshooting
- -v -- verbose mode; show more output use the -v option to show progress output during execution
- -V --version version information
To view/verify your configuration, run
rear dump
It will print out the current settings for BACKUP and OUTPUT methods and some system information.
Relax-and-Recover is built as a modular framework. A call of rear <command> will invoke the following general workflow:
- Configuration: Collect system information to assemble a correct configuration (default, arch, OS, OS_ARCH, OS_VER, site, local).
See the output of rear dump for an example. + Read config files for the combination of system attributes. Always read 'default.conf'
first and 'site.conf', 'local.conf' last.
- Create work area in '/tmp/rear.$$/' and start logging to '/var/log/rear/rear-hostname.log'
- Run the workflow script for the specified command: '/usr/share/rear/lib/<command>-workflow.sh'
- Cleanup work area
The result of the analysis is written into configuration files under '/etc/rear/recovery/'. This directory is copied together with
the other Relax-and-Recover directories onto the rescue system where the same framework runs a different workflow - the recovery workflow.
The recovery workflow is triggered by the fact that the root filesystem is mounted in a ram disk or tmpfs. Alternatively a "demo"
recovery workflow can be started manually. This will simply recover all data into a subdirectory and not touch the hard disks (Phase
2).
The recovery workflow consists of these parts (identically named modules are indeed the same):
- Config: By utilizing the same configuration module, the same configuration variable are available for the recovery,
too. This makes writing pairs of backup/restore modules much easier.
- Verify: Verify the integrity and sanity of the recovery data and check the hardware found to determine, whether
a recovery will be likely to succeed. If not, then we abort the workflow so as not to touch the hard disks if we don’t believe that
we would manage to successfully recover the system on this hardware.
- Recreate: Recreate the FS layout (partitioning, LVM, raid, filesystems, …) and mount it under /mnt/local
- Restore: Restore files and directories from the backup to '/mnt/local/'. This module is the analog to the Backup
module
- Finalize: Install boot loader, finalize system, dump recovery log onto '/var/log/rear/' in the recovered system.
FS layout
Relax-and-Recover tries to be as much LSB compliant as possible. Therefore ReaR will be installed into the usual locations:
- /etc/rear/
- Configurations
- /usr/sbin/rear
- Main program
- /usr/share/rear/
- Internal scripts
- /tmp/rear.$$/
- Build area
Layout of /etc/rear
- default.conf
- Default configuration - will define EVERY variable with a sane default setting. Serves also as a reference for the available
variables 'site.conf' site wide configuration (optional)
- local.conf
- local machine configuration (optional)
- $(uname -s)-$(uname -i).conf
- architecture specific configuration (optional)
- $(uname -o).conf
- OS system (e.g. GNU/Linux.conf) (optional)
- $OS/$OS_VER.conf
- OS and OS Version specific configuration (optional)
- templates/
- Directory to keep user-changeable templates for various files used or generated
- templates/PXE_per_node_config
- template for pxelinux.cfg per-node configurations
- templates/CDROM_isolinux.cfg
- isolinux.cfg template
- templates/…
- other templates as the need arises
- recovery/…
- Recovery information
Layout of /usr/share/rear
- skel/default/
- default rescue FS skeleton
- skel/$(uname -i)/
- arch specific rescue FS skeleton (optional)
- skel/$OS_$OS_VER/
- OS-specific rescue FS skeleton (optional)
- skel/$BACKUP/
- Backup-SW specific rescue FS skeleton (optional)
- skel/$OUTPUT/
- Output-Method specific rescue FS skeleton (optional)
- lib/*.sh
- function definitions, split into files by their topic
- prep/default/*.sh
- prep/$(uname -i)/*.sh
- prep/$OS_$OS_VER/*.sh
- prep/$BACKUP/*.sh
- prep/$OUTPUT/*.sh
- Prep scripts. The scripts get merged from the applicable directories and executed in their alphabetical order. Naming conventions
are: + _name.sh + where 00 < < 99
- layout/compare/default/
- layout/compare/$OS_$OS_VER/
- Scripts to compare the saved layout (under /var/lib/rear/layout/) with the actual situation. This is used by workflow
rear checklayout and may trigger a new run of rear mkrescue or rear mkbackup
- layout/precompare/default/
- layout/precompare/$OS_$OS_VER/
- layout/prepare/default/
- layout/prepare/$OS_$OS_VER/
- layout/recreate/default/
- layout/recreate/$OS_$OS_VER/
- layout/save/default/
- layout/save/$OS_$OS_VER/
- Scripts to capture the disk layout and write it into /var/lib/rear/layout/ directory
- rescue/…
- Analyse-Rescue scripts: …
- build/…
- Build scripts: …
- pack/…
- Pack scripts: …
- backup/$BACKUP/*.sh
- Backup scripts: …
- output/$OUTPUT/*.sh
- Output scripts: …
- verify/…
- Verify the recovery data against the hardware found, whether we can successfully recover the system
- recreate/…
- Recreate file systems and their dependancies
- restore/$BACKUP/…
- Restore data from backup media
- finalize/…
- Finalization scripts
Inter-module communication
The various stages and modules communicate via standardized environment variables:
| NAME |
TYPE |
Descriptions |
Example |
| CONFIG_DIR |
STRING (RO) |
Configuration dir |
'/etc/rear/' |
| SHARE_DIR |
STRING (RO) |
Shared data dir |
'/usr/share/rear/' |
| BUILD_DIR |
STRING (RO) |
Build directory |
'/tmp/rear.$$/' |
| ROOTFS_DIR |
STRING (RO) |
Root FS directory for rescue system |
'/tmp/rear.$$/initrd/' |
| TARGET_FS_ROOT |
STRING (RO) |
Directory for restore |
'/mnt/local' |
| PROGS |
LIST |
Program files to copy |
bash ip route grep ls … |
| MODULES |
LIST |
Modules to copy |
af_unix e1000 ide-cd … |
| COPY_AS_IS |
LIST |
Files (with path) to copy as-is |
'/etc/localtime' … |
RO means that the framework manages this variable and modules and methods shouldn't change it.
You can add additional tools to your mini OS using the directive:
# additional tools
PROGS=( "${PROGS[@]}" lsof parted fdisk cfdisk )
Relax-and-Recover can add its own GRUB menu entry to your local system's GRUB, which is convenient to restore a system without the
need for an additional boot media. This only works if your system can (still) boot from the local disk and the disaster didn't destroy
that disk.
Relax-and-Recover automatically detects and enables serial console support. This is extremely useful if the only way to access the
console during disaster is a Java-based console riddled with keyboard bugs and slow screen refreshes.
Relax-and-Recover ships with a set of useful commands in its shell history in recover mode. This makes it possible to quickly
look for a command to help troubleshoot, or modify an important file during recovery.
If the original system was configured to log on remotely through the use of SSH keys, Relax-and-Recover preserved those keys on the
rescue environment and you can access the rescue environment from the network as you were used to before.
During recovery at any stage you can re-run Relax-and-Recover, modify the layout file for recreating the structure and intervene
when restoring the backup.
During restore Relax-and-Recover uses the saved system layout as the basis for recreating a workable layout on your new system. If
your new hardware is very different, it's advised to copy the layout file /var/lib/rear/layout/disklayout.conf to /etc/rear
and modify it according to what is required.
cp /var/lib/rear/layout/disklayout.conf /etc/rear/
vi /etc/rear/disklayout.conf
Then restart the recovery process: rear recover. Relax-and-Recover translates this layout file into a shell procedure (/var/lib/rear/layout/diskrestore.sh)
that contains all the needed instructions for recreating your desired layout.
HP StartARray controller presents some problems. You can find more information about your HP SmartArray setup by running one of the
following commands:
# hpacucli ctrl all show detail
# hpacucli ctrl all show config
# hpacucli ctrl all show config detail
You can find these commands as part of the history of the Relax-and-Recover shell.
To use Relax-and-Recover you always call the main script /usr/sbin/rear:
# rear help
Usage: rear [-h|--help] [-V|--version] [-dsSv] [-D|--debugscripts SET] [-c DIR] [-r KERNEL] [--] COMMAND [ARGS...]
Relax-and-Recover comes with ABSOLUTELY NO WARRANTY; for details see
the GNU General Public License at: http://www.gnu.org/licenses/gpl.html
Available options:
-h --help usage information
-c DIR alternative config directory; instead of /etc/rear
-d debug mode; log debug messages
-D debugscript mode; log every function call (via 'set -x')
--debugscripts SET same as -d -v -D but debugscript mode with 'set -SET'
-r KERNEL kernel version to use; current: '3.12.49-3-default'
-s simulation mode; show what scripts rear would include
-S step-by-step mode; acknowledge each script individually
-v verbose mode; show more output
-V --version version information
-
checklayout check if the disk layout has changed
-
dump dump configuration and system information
-
format format and label media for use with rear
-
mkbackup create rescue media and backup system
-
mkbackuponly backup system without creating rescue media
-
mkrescue create rescue media only
-
recover recover the system
-
validate submit validation information
Use 'rear -v help' for more advanced commands.
To view/verify your configuration, run rear dump. It will print out the current settings for BACKUP and OUTPUT
methods and some system information.To create a new rescue environment, simply call rear mkrescue. Do not forget to copy
the resulting rescue system away so that you can use it in the case of a system failure. Use rear mkbackup instead if you are
using the built-in backup functions (like BACKUP=NETFS)
To recover your system, start the computer from the rescue system and run rear recover. Your system will be recovered and you
can restart it and continue to use it normally.
To perform a restore or migration:
- Boot the rescue system on the new hardware. For example, burn the ISO image to a DVD and boot from the DVD.
- In the console interface, select the "Recover" option:
Figure 27.1. Rescue system: menu (omitted)
- You are taken to the prompt:
Figure 27.2. Rescue system: prompt
Warning: Once you have started recovery in the next step, it probably cannot be undone and you may lose anything
stored on the physical disks of the system.
- Run the
rear recover command to perform the restore or migration. The rescue system then recreates the partition
layout and filesystems:
Figure 27.3. Rescue system: running "rear recover"
- Restore user and system files from the backup into the
/mnt/local/ directory.
Example 27.3. Restoring User and System Files
In this example, the backup file is a tar archive created per instructions in
Section 27.2.1.1, "Configuring the Internal Backup Method". First, copy the archive from its storage, then unpack the files into
/mnt/local/, then delete the archive:
~]# scp [email protected]:/srv/backup/rhel7/backup.tar.gz /mnt/local/
~]# tar xf /mnt/local/backup.tar.gz -C /mnt/local/
~]# rm -f /mnt/local/backup.tar.gz
The new storage has to have enough space both for the archive and the extracted files.
- Verify that the files have been restored:
~]# ls /mnt/local/
Figure 27.4. Rescue system: restoring user and system files from the backup
- Ensure that SELinux relabels the files on the next boot:
~]# touch /mnt/local/.autorelabel
Otherwise you may be unable to log in the system, because the /etc/passwd file may have the incorrect SELinux context.
- Finish the recovery by entering
exit. ReaR will then reinstall the boot loader. After that, reboot the system:
Figure 27.5. Rescue system: finishing recovery
Upon reboot, SELinux will relabel the whole filesystem. Then you will be able to log in to the recovered system.
IMPORTANT: If SELinux is enabled on your system you need to ensure that SELinux relabels the files on the next boot:
~]# touch /mnt/local/.autorelabel
Otherwise you may be unable to log in the system, because the /etc/passwd file may have the incorrect SELinux context.
"FIT" form factor is the best as almost does not protrude from the USB port. There is a small (orange in case of SanDisk) indicator
light and I kind of like it. When it is writing or reading, the light will blink. They are supposed to be plugged in and seldom pulled
out, or pulled out fairly rarely. Perfect for local backup of OS.
Bar form factor protrude one inch or so. Which in many case is acceptable but still carry some risks.
USB 3.0 FIT
USB 3.0 BAR
USB 2.0
- 20200821 : ReaR- Backup and Recover your Linux server with confidence by Sreejith Anujan ( Aug 19, 2020 , www.redhat.com )
- 20200323 : Relax-and-Recover - Backup and Recover a Linux System ( Mar 23, 2020 , www.tecmint.com )
- 20180403 : Relax and Recover ( Apr 03, 2018 , blog.nasmart.me )
- 20161105 : Relax and Recover – How Did I Do That ( www.howdididothat.info )
- 20161105 : Relax-and-Recover – Freecode ( Nov 05, 2016 , freecode.com )
- 20161105 : Affordable compact (FIT or bar form factor) 128 GB Flash drives ( Nov 05, 2016 , www.amazon.com )
- 20161104 : Coding Style rear-rear Wiki ( Nov 04, 2016 , github.com )
The article discusses several scarios of using ReaR. Not much new or interesting but looks
like OK overview of version 2.4
... ...
... Deploy ReaR on the server to be backed up
On the production server, install the rear , genisoimage , and
syslinux packages. In RHEL, these packages are part of the base repository.
... ... ...
As suggested by the login banner, run rear recover to restore the system by
connecting to the storage server. From there, retrieve the backup.tar.gz and
restore it to the right destination with appropriate permissions.
RESCUE production: ~ # rear -v -d recover
... ... ...
...ReaR is an integral part of many Linux-based backup solutions. OpenStack and the Red Hat
OpenStack Platform use ReaR for the undercloud and control plane backup and restore. Watch for
future articles in this space addressing the patching and rollback options for applications and
operating systems.
For more on using ReaR in a Red Hat Enterprise Linux production environment, be sure to
consult this solution
in the Red Hat Customer Portal.
Sreejith Anujan is a cloud technology professional with more than 14 years of experience in
on-premise and public cloud providers. He enjoys working with customers on their enablement
plans to upskill the technical team on container and automation tooling. More about me
A good overview
...
Relax-and-Recover Key Features:
- It has a modular design written in Bash and can be extended using custom
functionality.
- Supports various boot media including ISO, PXE, OBDR tape, USB or eSATA storage.
- Supports a variety of network protocols including FTP, SFTP, HTTP, NFS, and CIFS for
storage and
backup .
- Supports disk layout implementation such as LVM, DRBD, iSCSI, HWRAID (HP SmartArray),
SWRAID, multipathing, and LUKS (encrypted partitions and filesystems).
- Supports both third-party and internal backup tools including IBM TSM, HP DataProtector,
Symantec NetBackup, Bacula; tar and rsync
.
- Supports booting via PXE, DVD/CD, bootable tape or virtual provisioning.
- Supports a simulation model that shows what scripts are run without executing them.
- Supports consistent logging and advanced debugging options for troubleshooting
purposes.
- It can be integrated with monitoring tools such as Nagios and Opsview.
- It can also be integrated with job schedulers such as cron .
- It also supports various virtualization technologies supported (KVM, Xen, VMware).
In this article, you will learn how to install and configure ReaR to create a rescue system
and/or system backup using a USB stick and rescue or restore a bare-metal Linux system after a
disaster.
... ... ...
The main purpose of ReaR is to create a bootable image, based on what is currently installed
on a Linux host, that can be used to partition disks and retrieve a backup of the system. There
are options for where to create the bootable image and what to do with it after it has been
created.
The bootable image can be a USB device, an ISO file or a number of other options.
If you create a bootable image on a USB device then you may also wish to create a backup of
your system on the same device, which ReaR will support.
When creating a bootable image as an ISO file you have a multitude of options for what do to
with the file in order to get it off the box so that it can be used for recovery. The two
options I have used are rsync and TSM .
The misconception I mentioned earlier is the belief that ReaR will backup your system. It
can do that, but it is not a given and depends on your configuration acheter du cialis 5 .
... ... ...
Example Procedure
The following is an example of the produce to protect a system with ReaR and TSM during some
operating system patching activities (assumes TSM is already installed):
- Install ReaR (rpms are available here ).
- Configure ReaR to use TSM and to create an ISO file by updating /etc/rear/local.conf with
a line of OUTPUT=ISO and another with BACKUP=TSM.
- Run "rear -v mkrescue" to create the bootable ISO and send it to TSM (mkbackup would have
the same effect in this case as TSM will be handling the file system backups independently
– I feel mkrescue makes it clearer what you're doing).
- Perform a incremental backup of your file systems with TSM using "dsmc inc ".
- Do your patching activity.
If all goes well then you don't need to boot from the ReaR ISO and restore you operating
system. But, let's say it didn't go well. Your system will no longer boot and there's no
immediately obvious way forward. You decide to restore. The procedure is:
- Restore the ReaR ISO to a location that will allow you to present it to the server. This
is most likely to be your desktop so you can present the ISO file as a virtual CD-ROM over
the ILOM interface.
- Present the ISO to the host to be recovered.
- Boot the host from the ISO – It is highly likely that you'll need to change the
boot order or get a pop-up menu to select the ISO as the boot media.
- Select "Recover <hostname>" at the grub prompt.
- Log in as root (password not required).
- Run "rear -v recover" and answer the interactive prompts.
Issues
Since starting to use ReaR I have encountered two problems:
- When recovering a host that used an ext4 file system for /boot I found myself facing at
message of "Error 16: Inconsistent filesystem structure." from grub. After a bit of digging
around and trying to understand what the issue was I ended up modifying the
/var/lib/rear/layout/disklayout.conf ReaR file to change the file system type for /boot from
ext4 to ext2. I initially tried ext3, but as the system did not use ext3 for any of the file
systems the module was not available.
- The version of ReaR that I was using had a bug ( tracked on GitHub ) that affected systems that
do not have a separate /boot partition. There is a patch for the bug available, but if like
me you're happy to have a manual workaround, you need to perform the following actions after
the restore completes:
# chroot /mnt/local
# PATH=/bin:/sbin:/usr/bin
# grub-install <disk path>
# exit
# reboot
Finally, it's worth mentioning that ReaR is written in shell and is open source.
21 August 2014
Start a backup on the CentOS machine
Add the following lines to /etc/rear/local.conf:
OUTPUT=ISO
BACKUP=NETFS
BACKUP_TYPE=incremental
BACKUP_PROG=tar
FULLBACKUPDAY="Mon"
BACKUP_URL="nfs://NFSSERVER/path/to/nfs/export/servername"
BACKUP_PROG_COMPRESS_OPTIONS="--gzip"
BACKUP_PROG_COMPRESS_SUFFIX=".gz"
BACKUP_PROG_EXCLUDE=( '/tmp/*' '/dev/shm/*' )
BACKUP_OPTIONS="nfsvers=3,nolock"
OUTPUT=ISO
BACKUP=NETFS
BACKUP_TYPE=incremental
BACKUP_PROG=tar
FULLBACKUPDAY="Mon"
BACKUP_URL="nfs://NFSSERVER/path/to/nfs/export/servername"
BACKUP_PROG_COMPRESS_OPTIONS="--gzip"
BACKUP_PROG_COMPRESS_SUFFIX=".gz"
BACKUP_PROG_EXCLUDE=( '/tmp/*' '/dev/shm/*' )
BACKUP_OPTIONS="nfsvers=3,nolock"
Now make a backup
[root@centos7 ~]# rear mkbackup -v
Relax-and-Recover 1.16.1 / Git
Using log file: /var/log/rear/rear-centos7.log
mkdir: created directory '/var/lib/rear/output'
Creating disk layout
Creating root filesystem layout
TIP: To login as root via ssh you need to set up /root/.ssh/authorized_keys or SSH_ROOT_PASSWORD
in your configuration file
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Creating initramfs
Making ISO image
Wrote ISO image: /var/lib/rear/output/rear-centos7.iso (90M)
Copying resulting files to nfs location
Encrypting disabled
Creating tar archive '/tmp/rear.QnDt1Ehk25Vqurp/outputfs/centos7/2014-08-21-1548-F.tar.gz'
Archived 406 MiB [avg 3753 KiB/sec]OK
Archived 406 MiB in 112 seconds [avg 3720 KiB/sec]
Now look on your NFS server
You'll see all the files you'll need to perform the disaster recovery.
total 499M
drwxr-x- 2 root root 4.0K Aug 21 23:51 .
drwxr-xr-x 3 root root 4.0K Aug 21 23:48 ..
-rw--- 1 root root 407M Aug 21 23:51 2014-08-21-1548-F.tar.gz
-rw--- 1 root root 2.2M Aug 21 23:51 backup.log
-rw--- 1 root root 202 Aug 21 23:49 README
-rw--- 1 root root 90M Aug 21 23:49 rear-centos7.iso
-rw--- 1 root root 161K Aug 21 23:49 rear.log
-rw--- 1 root root 0 Aug 21 23:51 selinux.autorelabel
-rw--- 1 root root 277 Aug 21 23:49 VERSION
Author: masterdam79
You can also connect with me on Google+ View all posts by masterdam79
Author masterdam79/
Posted on 21 August 2014/
dheeraj says:
31 August 2016 at 02:26
is it possible to give list of directories or mount points while giving mkbackup to exclude
from backup. Like giving a file with list of all directories that need to be excluded ??
masterdam79 says:
26 September 2016 at 21:50
Have a look at
https://github.com/rear/rear/issues/216
Should be possible if you ask me.
Relax-and-Recover (Rear) is a bare metal disaster recovery and system migration solution, similar
to AIX mksysb or HP-UX ignite. It is composed of a modular framework and ready-to-go workflows
for many common situations to produce a bootable image and restore from backup using this image.
It can restore to different hardware, and can therefore be used as a migration tool as well. It
supports various boot media (including tape, USB, or eSATA storage, ISO, PXE, etc.), a variety
of network protocols (including SFTP, FTP, HTTP, NFS, and CIFS), as well as a multitude of backup
strategies (including IBM TSM, HP DataProtector, Symantec NetBackup, Bacula, and rsync). It was
designed to be easy to set up, requires no maintenance, and is there to assist when disaster strikes.
Recovering from disaster is made very straight-forward by a 2-step recovery process so that it
can be executed by operational teams when required. When used interactively (e.g. when used for
migrating systems), menus help make decisions to restore to a new (hardware) environment.
Release Notes: Integrated with duply/duplicity support. systemd support has been added. Various
small fixes and improvements to tape support, Xen, PPC, Gentoo, Fedora, multi-arch, storage ...
layout configuration, and serial console integration.
(more)
Release Notes: This release adds support for multipathing, adds several improvements to distribution
backward compatibility, improves ext4 support, makes various bugfixes, migrates HWADDR ... after
rescovery, and includes better systemd support.
(more)
Release Notes: Multi-system and multi-copy support on USB storage devices. Basic rsync backup
support. More extensive exclude options. The new layout code is enabled by default. Support ...
for Arch Linux. Improved multipath support. Experimental btrfs support.
(more)
Release Notes: Standardization of the command line. The default is quiet output; use the option
-v for the old behavior. Boot images now have a comprehensive boot menu. Support for IPv6 ...
addresses. Restoring NBU backup from a point in time is supported. Support for Fedora 15 (systemd)
and RHEL6/SL6. Improved handling of HP SmartArray. Support for ext4 on RHEL5/SL5. Support for
Xen paravirtualization. Integration with the local GRUB menu. Boot images can now be centralized
through network transfers. Support for udev on RHEL4. Many small improvements and performance
enhancements.
(more)
Release Notes: This release supports many recent distributions, including "upstart" (Ubuntu
7.10). It has more IA-64 support (RHEL5 only at the moment), better error reporting and catching,
... Debian packages (mkdeb), and improved TSM support.
(more)
[Nov 05, 2016] Affordable compact (FIT or bar form factor) 128 GB Flash drives
"FIT" form factor is the best as almost does not protrude from the USB port.
There is a small (orange in case of SanDisk) indicator light and I kind of like it. When it is writing or reading, the light will blink.
They are supposed to be plugged in and seldom pulled out, or pulled out fairly rarely. Perfect for
local backup of OS.
Bar form factor protrude one inch or so. Which in many case is acceptable but still carry some
risks.
USB 3.0 FIT
USB 3.0 BAR
USB 2.0
Reading rear sources is an interesting exercise. It really demonstrates attempt to use
"reasonable' style of shell programming and you can learn a lot.
Relax-and-Recover is written in
Bash (at least bash version
3 is needed), a language that can be used in many styles. We want to make it easier for everybody
to understand the Relax-and-Recover code and subsequently to contribute fixes and enhancements.
Here is a collection of coding hints that should help to get a more consistent code base.
Don't be afraid to contribute to Relax-and-Recover even if your contribution does not fully match
all this coding hints. Currently large parts of the Relax-and-Recover code are not yet in compliance
with this coding hints. This is an ongoing step by step process. Nevertheless try to understand the
idea behind this coding hints so that you know how to break them properly (i.e. "learn the rules
so you know how to break them properly").
The overall idea behind this coding hints is:
Make yourself understood
Make yourself understood to enable others to fix and enhance your code properly as needed.
From this overall idea the following coding hints are derived.
For the fun of it an extreme example what coding style should be avoided:
#!/bin/bash for i in `seq 1 2 $((2*$1-1))`;do echo $((j+=i));done
Try to find out what that code is about - it does a useful thing.
Code must be easy to read
Code should be easy to understand
Do not only tell what the code does (i.e. the implementation details) but also explain what the
intent behind is (i.e. why ) to make the code maintainable.
- Provide meaningful comments that tell what the computer should do and also explain why it
should do it so that others understand the intent behind so that they can properly fix issues
or adapt and enhance it as needed.
- If there is a GitHub issue or another URL available for a particular piece of code provide
a comment with the GitHub issue or any other URL that tells about the reasoning behind current
implementation details.
Here the initial example so that one can understand what it is about:
#!/bin/bash # output the first N square numbers # by summing up the first N odd numbers 1 3 ...
2*N-1 # where each nth partial sum is the nth square number # see https://en.wikipedia.org/wiki/Square_number#Properties
# this way it is a little bit faster for big N compared to # calculating each square number on its
own via multiplication N=$1 if ! [[ $N =~ ^[0-9]+$ ]] ; then echo "Input must be non-negative integer."
1>&2 exit 1 fi square_number=0 for odd_number in $( seq 1 2 $(( 2 * N - 1 )) ) ; do (( square_number
+= odd_number )) && echo $square_number done
Now the intent behind is clear and now others can easily decide if that code is really the best
way to do it and easily improve it if needed.
Try to care about possible errors
By default bash proceeds with the next command when something failed. Do not let your code blindly
proceed in case of errors because that could make it hard to find the root cause of a failure when
it errors out somewhere later at an unrelated place with a weird error message which could lead to
false fixes that cure only a particular symptom but not the root cause.
Maintain Backward Compatibility
Implement adaptions and enhancements in a backward compatible way so that your changes do not
cause regressions for others.
- One same Relax-and-Recover code must work on various different systems. On older systems as
well as on newest systems and on various different Linux distributions.
- Preferably use simple generic functionality that works on any Linux system. Better very simple
code than oversophisticated (possibly fragile) constructs. In particular avoid special bash version
4 features (Relax-and-Recover code should also work with bash version 3).
- When there are incompatible differences on different systems distinction of cases with separated
code is needed because it is more important that the Relax-and-Recover code works everywhere than
having generic code that sometimes fails.
Dirty
hacks welcome
When there are special issues on particular systems it is more important that the Relax-and-Recover
code works than having nice looking clean code that sometimes fails. In such special cases any dirty
hacks that intend to make it work everywhere are welcome. But for dirty hacks the above listed coding
hints become mandatory rules:
- Provide explanatory comments that tell what a dirty hack does together with a GitHub issue
or any other URL that tell about the reasoning behind the dirty hack to enable others to properly
adapt or clean up a dirty hack at any time later when the reason for it had changed or gone away.
- Try as good as you can to foresee possible errors or failures of a dirty hack and error out
with meaningful error messages if things go wrong to enable others to understand the reason behind
a failure.
- Implement the dirty hack in a way so that it does not cause regressions for others.
For example a dirty hack like the following is perfectly acceptable:
# FIXME: Dirty hack to make it work # on "FUBAR Linux version 666" # where COMMAND sometimes inexplicably
fails # but always works after at most 3 attempts # see http://example.org/issue12345 # Retries should
have no bad effect on other systems # where the first run of COMMAND works. COMMAND || COMMAND ||
COMMAND || Error "COMMAND failed."
Character
Encoding
Use only traditional (7-bit) ASCII charactes. In particular do not use UTF-8 encoded multi-byte
characters.
- Non-ASCII characters in scripts may cause arbitrary unexpected failures on systems that do
not support other locales than POSIX/C. During "rear recover" only the POSIX/C locale works (the
ReaR rescue/recovery system has no support for non-ASCII locales) and /usr/sbin/rear sets the
C locale so that non-ASCII characters are invalid in scripts. Have in mind that basically all
files in ReaR are scripts. E.g. also /usr/share/rear/conf/default.conf and /etc/rear/local.conf
are sourced (and executed) as scripts.
- English documentation texts do not need non-ASCII characters. Using non-ASCII characters in
documentation texts makes it needlessly hard to display the documentation correctly for any user
on any system. When non-ASCII characters are used but the user does not have the exact right matching
locale set arbitrary nonsense can happen, cf.
https://en.opensuse.org/SDB:Plain_Text_versus_Locale
Text Layout
- Indentation with 4 blanks, not tabs.
- Block level statements in same line:
if CONDITION ; then
Variables
- Curly braces only where really needed:
$FOO instead of ${FOO} , but ${FOO:-default_foo} .
- All variables that are used in more than a single script must be all-caps:
$FOO instead of $foo or $Foo .
- Variables that are used only locally should be lowercased and should be marked with
local like:
local $foo="default_value"
Functions
- Use the
function keyword to define a function.
- Function names are lower case, words separated by underline (
_ ).
Relax-and-Recover functions
Use the available Relax-and-Recover functions when possible instead of re-implementing basic functionality
again and again. The Relax-and-Recover functions are implemented in various
lib/*-functions.sh
files .
is_true and is_false :
See
lib/global-functions.sh how to use them.
For example instead of using
if [[ ! "$FOO" =~ ^[yY1] ]] ; then
use
if ! is_true "$FOO" ; then
test, [, [[,
((
- Use
[[ where it is required (e.g. for pattern matching or complex conditionals)
and [ or test everywhere else.
(( is the preferred way for numeric comparison, variables don't need to be prefixed
with $ there.
Paired
parenthesis
- Use paired parenthesis for
case patterns as in
case WORD in (PATTERN) COMMANDS ;; esac
so that editor commands (like '%' in 'vi') that check for matching opening and closing parenthesis
work everywhere in the code.
See also
Softpanorama Recommended
Contract support and subscription from developers (Gratien D'haese (IT3 Consultants)
Presentations and videos on Youtube:
linuxtag.org
ReaR- Backup and Recover your Linux server with
confidence - Enable Sysadmin (Red hat)
For more on using ReaR in a Red Hat Enterprise Linux production environment, be sure to consult
this solution in the Red Hat Customer Portal.
Society
Groupthink :
Two Party System
as Polyarchy :
Corruption of Regulators :
Bureaucracies :
Understanding Micromanagers
and Control Freaks : Toxic Managers :
Harvard Mafia :
Diplomatic Communication
: Surviving a Bad Performance
Review : Insufficient Retirement Funds as
Immanent Problem of Neoliberal Regime : PseudoScience :
Who Rules America :
Neoliberalism
: The Iron
Law of Oligarchy :
Libertarian Philosophy
Quotes
War and Peace
: Skeptical
Finance : John
Kenneth Galbraith :Talleyrand :
Oscar Wilde :
Otto Von Bismarck :
Keynes :
George Carlin :
Skeptics :
Propaganda : SE
quotes : Language Design and Programming Quotes :
Random IT-related quotes :
Somerset Maugham :
Marcus Aurelius :
Kurt Vonnegut :
Eric Hoffer :
Winston Churchill :
Napoleon Bonaparte :
Ambrose Bierce :
Bernard Shaw :
Mark Twain Quotes
Bulletin:
Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient
markets hypothesis :
Political Skeptic Bulletin, 2013 :
Unemployment Bulletin, 2010 :
Vol 23, No.10
(October, 2011) An observation about corporate security departments :
Slightly Skeptical Euromaydan Chronicles, June 2014 :
Greenspan legacy bulletin, 2008 :
Vol 25, No.10 (October, 2013) Cryptolocker Trojan
(Win32/Crilock.A) :
Vol 25, No.08 (August, 2013) Cloud providers
as intelligence collection hubs :
Financial Humor Bulletin, 2010 :
Inequality Bulletin, 2009 :
Financial Humor Bulletin, 2008 :
Copyleft Problems
Bulletin, 2004 :
Financial Humor Bulletin, 2011 :
Energy Bulletin, 2010 :
Malware Protection Bulletin, 2010 : Vol 26,
No.1 (January, 2013) Object-Oriented Cult :
Political Skeptic Bulletin, 2011 :
Vol 23, No.11 (November, 2011) Softpanorama classification
of sysadmin horror stories : Vol 25, No.05
(May, 2013) Corporate bullshit as a communication method :
Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law
History:
Fifty glorious years (1950-2000):
the triumph of the US computer engineering :
Donald Knuth : TAoCP
and its Influence of Computer Science : Richard Stallman
: Linus Torvalds :
Larry Wall :
John K. Ousterhout :
CTSS : Multix OS Unix
History : Unix shell history :
VI editor :
History of pipes concept :
Solaris : MS DOS
: Programming Languages History :
PL/1 : Simula 67 :
C :
History of GCC development :
Scripting Languages :
Perl history :
OS History : Mail :
DNS : SSH
: CPU Instruction Sets :
SPARC systems 1987-2006 :
Norton Commander :
Norton Utilities :
Norton Ghost :
Frontpage history :
Malware Defense History :
GNU Screen :
OSS early history
Classic books:
The Peter
Principle : Parkinson
Law : 1984 :
The Mythical Man-Month :
How to Solve It by George Polya :
The Art of Computer Programming :
The Elements of Programming Style :
The Unix Hater’s Handbook :
The Jargon file :
The True Believer :
Programming Pearls :
The Good Soldier Svejk :
The Power Elite
Most popular humor pages:
Manifest of the Softpanorama IT Slacker Society :
Ten Commandments
of the IT Slackers Society : Computer Humor Collection
: BSD Logo Story :
The Cuckoo's Egg :
IT Slang : C++ Humor
: ARE YOU A BBS ADDICT? :
The Perl Purity Test :
Object oriented programmers of all nations
: Financial Humor :
Financial Humor Bulletin,
2008 : Financial
Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related
Humor : Programming Language Humor :
Goldman Sachs related humor :
Greenspan humor : C Humor :
Scripting Humor :
Real Programmers Humor :
Web Humor : GPL-related Humor
: OFM Humor :
Politically Incorrect Humor :
IDS Humor :
"Linux Sucks" Humor : Russian
Musical Humor : Best Russian Programmer
Humor : Microsoft plans to buy Catholic Church
: Richard Stallman Related Humor :
Admin Humor : Perl-related
Humor : Linus Torvalds Related
humor : PseudoScience Related Humor :
Networking Humor :
Shell Humor :
Financial Humor Bulletin,
2011 : Financial
Humor Bulletin, 2012 :
Financial Humor Bulletin,
2013 : Java Humor : Software
Engineering Humor : Sun Solaris Related Humor :
Education Humor : IBM
Humor : Assembler-related Humor :
VIM Humor : Computer
Viruses Humor : Bright tomorrow is rescheduled
to a day after tomorrow : Classic Computer
Humor
The Last but not Least Technology is dominated by
two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt.
Ph.D
Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org
was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP)
without any remuneration. This document is an industrial compilation designed and created exclusively
for educational use and is distributed under the Softpanorama Content License.
Original materials copyright belong
to respective owners. Quotes are made for educational purposes only
in compliance with the fair use doctrine.
FAIR USE NOTICE This site contains
copyrighted material the use of which has not always been specifically
authorized by the copyright owner. We are making such material available
to advance understanding of computer science, IT technology, economic, scientific, and social
issues. We believe this constitutes a 'fair use' of any such
copyrighted material as provided by section 107 of the US Copyright Law according to which
such material can be distributed without profit exclusively for research and educational purposes.
This is a Spartan WHYFF (We Help You For Free)
site written by people for whom English is not a native language. Grammar and spelling errors should
be expected. The site contain some broken links as it develops like a living tree...
Disclaimer:
The statements, views and opinions presented on this web page are those of the author (or
referenced source) and are
not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness
of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be
tracked by Google please disable Javascript for this site. This site is perfectly usable without
Javascript.
Last modified: January, 04, 2021
