Softpanorama
May the source be with you, but remember the KISS principle ;-)

Contents Bulletin Scripting in shell and Perl Network troubleshooting History Humor

Usage of Relax-and-Recover on RHEL

News Disaster Recovery Recommended Links  FIT USB flash drives Perl Backup Scripts Bare metal recovery of Linux systems Cloning systems, disks and partitions
Unix tape archiver (Tar) Tar options for bare metal recovery Baseliners Cron and Crontab commands Gzip Unix cpio Dump and restore
dd Remote backups using dd  Recovery of lost files using DD Mount a partition from dd disk image rsync Backing Up And Restoring Linux With SystemImager Partimage
 Missing backup HP Data Protector Dell DRAC and VFLASH Data deduplication Sysadmin Horror Stories Humor Etc

Introduction

Relax-and-Recover a bare metal disaster recovery and system migration software. It is a modular framework with many ready-to-go "workflows" for common situations.

The key idea of Relax-and-Recover is that it is possible to construct bootable image with mini OS on the USB drive using files already available on the harddrive of installed Linux system.  If you add to this bootable mini disk tar archive of system directories and some scripts, you can restore the system from  a USB drive.  In other words you have a bare metal recovery solution created using basic Linux components.  Of course you can always install minimal Os on the USB drive and create tar manually. If relax and recover does not work this in the next option to try. But it requires shutting the server down and rebooting it from installation DVD or remotely mounted ISO.

The devil is in details and due to its long history of development and despite that it is written in bash relax-and-restore make most details right.

Relax-and-Recover uses tar with the set of tar options and exclude files that tarball that is serving as a backup. If backup is not too large it can be put on permannetly inserted FIT form factor USB drive. the current upper size for this form factor currently is 128 GB.  But flash drives now scale up to 1 TB (Kingston DataTraveler HyperX Predator 1TB USB 3.0 Flash Drive ) although it is better to use USB 3.0 External hard drive such as Samsung T1 or Samsung T3 which is twice cheaper.

Relax-and-Recover produces a bootable image (rescue image) + tar file (backup).  This image restore partitioning of the system. Once that is done it initiates a restore from backup. Restores to different hardware are possible. Relax-and-Recover can therefore be used as a migration tool between two similar servers as well.

Currently Relax-and-Recover supports various boot media (incl. ISO, PXE, OBDR tape, USB or eSATA storage), a large variety of network protocols (incl. sftp, ftp, http, nfs, rsync,  cifs). NFS is probably the most important of them, the most transpatent of them  and should be used whenever possible.

It also can be used with pre-existing backup tools such as  Bacula,  HP DataProtector, IBM TSM,  Symantec NetBackup, Borg.

Relax-and-Recover is brittle and setup from RPM requires installation of additional RPMs.

To recover from a failed drive just boot from USB and using the menu select  recover option. Then follow instructions provided Red Hat for RHEL and its clones (on REHL the recovery is complicated by existence of SE Linux, if it is enabled), and generic instructions for all other flavors. 

Tar archive with data located on USB disk. It can  moved to hard drive unpacked and used manually for any purpose you which, including recovery of some lost files. 

There are some interesting possibilities to use Relax-and-Recover for migrating systems to slightly different hardware (which is often the case if computer is damaged beyond repair, for example some chip on motherboard failed.  In this case Relax-and-Recover provides step-by-step menus which help make decisions to restore the OS and data in new server.  

Consistent logging and optionally extended output helps to understand the concepts behind Relax-and-Recover, troubleshoot during initial configuration and help debug during integration.

Notes:

Required RHEL packages

Relax-and-Recover is written entirely in Bash and theoretically should does not require any external programs beside classic Unix utilities (cat, cut, sort, dd, grep, etc.). But this is not true. It has its own and pretty convoluted set dependences with several packages required to make the  rescue system wor

This is where good simple idea became slightly perverted. Unless you are patient  you might even abandon the package. I spend several hours trying to install in of RHEL 6.8 until I figured out what to do. This is not how decent open source software should be documented and developers should be ashamed.

The installation on RHEL fails if supplementary software channel is not enabled before you attempt the installation. But it fails in its own particular way. Software is installed successfully, you can format USB drive but when you try to make backup you get them message that syslinux-extlinux package is missing.

Also it you download those packages (for batch installation on many servers) and try to install them using yum for some reason on RHEL 6.x you face incompatibility of libraries problem and installation fails.  Among packages you you need

On RHEL 6.5-6.8 you can install rear using the commands

yum-config-manager --enable repository rhel-6-server-optional-rpms
yum install rear syslinux-extlinux genisoimage syslinux

Quick Start Guide

  1. Prepare your USB media. We assume that USB media is at /dev/sdb.

    Before formatting the  device should be unmounted. The current version of rear does not unmount it and simply fails with a cryptic message

    Again, you better check this. The device should unmounted before you can start (USB disk are typically mounted on insertion to /media folder)

    Like always the case with format existing data will be destroyed. Formatting is very slow and can take for 32GB flash drive ten minutes on USB 2.0 connection. The drive is formatted as Ext3 filesystem.

    /usr/sbin/rear format /dev/sdb
    NOTES:
    1. here you need device name, not the partition name /dev/sdb1
    2. You can format the drive yourself without using REAR. That's probably  the "best practice" with the current version.

    After you type Enter, Relax-and-Recover asks you to confirm that you want to format the device. If everything is correct and you do not accidentally specify you primary harddrive :-), then answer

    Yes

    The most common reason for this command to abort with error is that the device is still mounted.

    If operation is successful, the device will be labeled REAR-000 by the format workflow.  This is the default label and you change it via configuration file if you object to it.
     

  2. Edit the /etc/rear/local.conf configuration file. For example you can write initial set of pretty generic parameters using  cat:
    cat > /etc/rear/local.conf <<EOF
    ### write the rescue initramfs to USB and update the USB bootloader
    OUTPUT=USB
    
    ### create a backup using the internal NETFS method, using 'tar'
    BACKUP=NETFS
    
    ### write both rescue image and backup to the device labeled REAR-000
    BACKUP_URL=usb:///dev/disk/by-label/REAR-000
    EOF

    If you have multiple boxes you can distribute this file to other boxes: the content of this file is server independent. If you already have such a file on other server, you can scp it to the new server you are configuring.

    Please make sure you have at least defined three variables OUTPUT, BACKUP  and a BACKUP_URL

    Typically you also can benefit from the exclusion of certain directories, but not at this stage of learning.
     

  3. Create a rescue image. This is essentially a test run as this is pretty quick operation. The size of rescue image is about 100MB. We want verbose output (-v option).
    sudo /usr/sbin/rear -v mkrescue
    You will get output similar to the following:
    Relax-and-Recover <version>
    Using log file: /var/log/rear/rear-<hostname>.log
    Creating disk layout
    Creating root filesystem layout
    Copying files and directories
    Copying binaries and libraries
    Copying kernel modules
    Creating initramfs
    Writing MBR to /dev/sdb
    Copying resulting files to usb location

    If you can create rescue image that means you rear installation works.

    NOTE: if you forgot to create configuration file rear will write rescue ISO image to /var/lib/rear/output/rear-`hostname`.iso
     

  4. Check the log file for possible errors. This is an important step and should not be skipped -- rear is a brittle software. 
    And it can play jokes with you. 

    This is an important step and should not be skipped -- rear is a brittle software.  And it can play jokes with you.

    I would also recommend to use some "we do not care about server" and try to restore it using your version of rear to make sure that it does not contain catastrophic bugs. because you can encounter one at the most inopportune moment. But the structure of rear backup is useful even if restore is not working.

    You can restore the system step by step yourself following the scripts rear provides.  all the necessary information is still present that that' the most important part of the whole exercise.
     

  5. Reboot your system and try to boot from the USB device. You should get a self-explaining boot menu. Verify that the server can boot itself after 30 sec delay. On HP 580 G7 with rear 1.17.2  I encountered a bug due to which it blow up with the message invalid opcode of something like that. You can try restore the OS if this is a test computer to ensure that the version of Relax and Recover you use does not contain some glaring errors. 
     
  6. If your USB device has enough space, initiate a backup.  Backup of filesystem containing, say 32GB on the flash drive is a long operation which take an hour or so so you need to submit it either in screen session (I experienced problems with running it with nohup prefix; also you will not see any output until it ends which is a bad idea) You can use the following command:
    /usr/sbin/rear -v mkbackup
If backup operation is working you will see the running message of how much information was archived. On USB2 drives watching this is like watching the paint dry, so if you have 64 of more GB to backup it is better performed as an overnight operation. For example:
Archived 1320 MiB (avg 7025 KiB/sec)

As tar is gzip you get (depending on the content on you filesystems) approximately 2/3 smaller image the the size of filesystem, That means that 64GB flash can store backup of 100GB data.  with 126Gb FIT drive you probably can backup 200GB of data. If you have more then that you need to use NFS of rsync for storing backup on remote server or use USB drive instead of flash drive. 5TB USB drive is now about $100 (Seagate at Amazon). One of the positive things about Relax and Restore is that the rescue image configure internet for you so all shares that are available on "normal" server are accessible from the rescue image boot-up. It is not that difficult to configure network manually but it is done automatically for the you it is even better. 

After you have such an image  you are now better better prepared for catastrophic failure of harddrives or other server components.  But if server has large partitions you need to use NFS of rsync to put the image on a remote filesystem. that would be the next step in learning rear.

More advanced configuration -- backup using NFS

The next level of complexity is probably using NFS to store backup, while USB store only rescue system.  you need to add to the configuration file the line that specify the target for the backup file
BACKUP_URL=nfs://server/path
And since typically servers use NTP for time synchronization, you should also add these lines to /etc/rear/site.conf:
TIMESYNC=NTP
If you use static networking the you need to add
USE_STATIC_NETWORKING=y
Dhcp client activation can be forced via the variable
USE_DHCLIENT=yes

•Save layout and compare layouts for easy automation of making Relax-and-Recover snapshots (checklayout option)

local.conf vs site.conf

To configure Relax-and-Recover you have to edit the configuration files in /etc/rear/

All *.conf files there are part of the configuration, but only site.conf and local.conf are intended for the user configuration. All other configuration files hold defaults for various distributions and should not be changed.

In /etc/rear/templates/ there are also some template files which are use by Relax-and-Recover to create configuration files (mostly for the boot environment). You can use these templates to prepend your own configurations to the configuration files created by Relax-and-Recover, for example you can edit PXE_pxelinux.cfg to add some general pxelinux configuration you use.

In almost all circumstances you have to configure two main settings and their parameters: The BACKUP  method and the OUTPUT  method.

The backup method defines, how your data was saved and whether Relax-and-Recover should backup your data as part of the mkrescue process or whether you use an external application, e.g. backup software to archive your data.

The output method defines how the rescue system is written to disk and how you plan to boot the failed computer from the rescue system.

See /usr/share/rear/conf/default.conf for an overview of the possible methods and their options. An example to use TSM for backup and PXE for output and would be to add these lines to /etc/rear/local.conf:

BACKUP=TSM
OUTPUT=PXE
Don’t forget to distribute the site.conf to all your systems.

The resulting PXE files (kernel, initrd and pxelinux configuration) will be written to files in /var/lib/rear/output/. You can now modify the behaviour by copying the appropriate configuration variables from default.conf to local.conf and changing them to suit your environment.

Rear cron job which is installed by RHEL package

Rear has cron job installed when RHEL package is installed. It adds the file rear to /etc/cron.d.

It updates rescue image, if layout of the box has been changed which is a good idea: 

30 1 * * * root /usr/sbin/rear checklayout || /usr/sbin/rear mkrescue 
 

Debugging rear

There are several options that help you with the debugging:

To view/verify your configuration, run rear dump. It will print out the current settings for BACKUP  and OUTPUT  methods and some system information.

Relax-and-Recover is built as a modular framework. A call of rear <command> will invoke the following general workflow:

1.Configuration: Collect system information to assemble a correct configuration (default, arch, OS, OS_ARCH, OS_VER, site, local). See the output of rear dump for an example. + Read config files for the combination of system attributes. Always read 'default.conf' first and 'site.conf', 'local.conf' last.


2.Create work area in '/tmp/rear.$$/' and start logging to '/var/log/rear/rear-hostname.log'


3.Run the workflow script for the specified command: '/usr/share/rear/lib/<command>-workflow.sh'


4.Cleanup work area

 

The result of the analysis is written into configuration files under '/etc/rear/recovery/'. This directory is copied together with the other Relax-and-Recover directories onto the rescue system where the same framework runs a different workflow - the recovery workflow.

The recovery workflow is triggered by the fact that the root filesystem is mounted in a ram disk or tmpfs. Alternatively a "demo" recovery workflow can be started manually. This will simply recover all data into a subdirectory and not touch the hard disks (Phase 2).

The recovery workflow consists of these parts (identically named modules are indeed the same):

  1. Config: By utilizing the same configuration module, the same configuration variable are available for the recovery, too. This makes writing pairs of backup/restore modules much easier.
  2. Verify: Verify the integrity and sanity of the recovery data and check the hardware found to determine, whether a recovery will be likely to succeed. If not, then we abort the workflow so as not to touch the hard disks if we don’t believe that we would manage to successfully recover the system on this hardware.
  3. Recreate: Recreate the FS layout (partitioning, LVM, raid, filesystems, …​) and mount it under /mnt/local
  4. Restore: Restore files and directories from the backup to '/mnt/local/'. This module is the analog to the Backup module
  5. Finalize: Install boot loader, finalize system, dump recovery log onto '/var/log/rear/' in the recovered system.

FS layout

Relax-and-Recover tries to be as much LSB compliant as possible. Therefore ReaR will be installed into the usual locations:

/etc/rear/
Configurations
/usr/sbin/rear
Main program
/usr/share/rear/
Internal scripts
/tmp/rear.$$/
Build area

Layout of /etc/rear

default.conf
Default configuration - will define EVERY variable with a sane default setting. Serves also as a reference for the available variables 'site.conf' site wide configuration (optional)
local.conf
local machine configuration (optional)
$(uname -s)-$(uname -i).conf
architecture specific configuration (optional)
$(uname -o).conf
OS system (e.g. GNU/Linux.conf) (optional)
$OS/$OS_VER.conf
OS and OS Version specific configuration (optional)
templates/
Directory to keep user-changeable templates for various files used or generated
templates/PXE_per_node_config
template for pxelinux.cfg per-node configurations
templates/CDROM_isolinux.cfg
isolinux.cfg template
templates/…​
other templates as the need arises
recovery/…​
Recovery information

Layout of /usr/share/rear

skel/default/
default rescue FS skeleton
skel/$(uname -i)/
arch specific rescue FS skeleton (optional)
skel/$OS_$OS_VER/
OS-specific rescue FS skeleton (optional)
skel/$BACKUP/
Backup-SW specific rescue FS skeleton (optional)
skel/$OUTPUT/
Output-Method specific rescue FS skeleton (optional)
lib/*.sh
function definitions, split into files by their topic
prep/default/*.sh
prep/$(uname -i)/*.sh
prep/$OS_$OS_VER/*.sh
prep/$BACKUP/*.sh
prep/$OUTPUT/*.sh
Prep scripts. The scripts get merged from the applicable directories and executed in their alphabetical order. Naming conventions are: + _name.sh + where 00 < < 99
layout/compare/default/
layout/compare/$OS_$OS_VER/
Scripts to compare the saved layout (under /var/lib/rear/layout/) with the actual situation. This is used by workflow rear checklayout and may trigger a new run of rear mkrescue or rear mkbackup
layout/precompare/default/
layout/precompare/$OS_$OS_VER/
layout/prepare/default/
layout/prepare/$OS_$OS_VER/
layout/recreate/default/
layout/recreate/$OS_$OS_VER/
layout/save/default/
layout/save/$OS_$OS_VER/
Scripts to capture the disk layout and write it into /var/lib/rear/layout/ directory
rescue/…​
Analyse-Rescue scripts: …​
build/…​
Build scripts: …​
pack/…​
Pack scripts: …​
backup/$BACKUP/*.sh
Backup scripts: …​
output/$OUTPUT/*.sh
Output scripts: …​
verify/…​
Verify the recovery data against the hardware found, whether we can successfully recover the system
recreate/…​
Recreate file systems and their dependancies
restore/$BACKUP/…​
Restore data from backup media
finalize/…​
Finalization scripts

Inter-module communication

The various stages and modules communicate via standardized environment variables:

NAME TYPE Descriptions Example
CONFIG_DIR STRING (RO) Configuration dir '/etc/rear/'
SHARE_DIR STRING (RO) Shared data dir '/usr/share/rear/'
BUILD_DIR STRING (RO) Build directory '/tmp/rear.$$/'
ROOTFS_DIR STRING (RO) Root FS directory for rescue system '/tmp/rear.$$/initrd/'
TARGET_FS_ROOT STRING (RO) Directory for restore '/mnt/local'
PROGS LIST Program files to copy bash ip route grep ls …​
MODULES LIST Modules to copy af_unix e1000 ide-cd …​
COPY_AS_IS LIST Files (with path) to copy as-is '/etc/localtime' …​

RO means that the framework manages this variable and modules and methods shouldn’t change it.

 

Tips

Relax-and-Recover can add its own GRUB menu entry to your local system’s GRUB, which is convenient to restore a system without the need for an additional boot media. This only works if your system can (still) boot from the local disk and the disaster didn’t destroy that disk.

Relax-and-Recover automatically detects and enables serial console support. This is extremely useful if the only way to access the console during disaster is a Java-based console riddled with keyboard bugs and slow screen refreshes.

Relax-and-Recover  ships with a set of useful commands in its shell history in recover mode. This makes it possible to quickly look for a command to help troubleshoot, or modify an important file during recovery.

If the original system was configured to log on remotely through the use of SSH keys, Relax-and-Recover preserved those keys on the rescue environment and you can access the rescue environment from the network as you were used to before.

During recovery at any stage you can re-run Relax-and-Recover, modify the layout file for recreating the structure and intervene when restoring the backup.

During restore Relax-and-Recover uses the saved system layout as the basis for recreating a workable layout on your new system. If your new hardware is very different, it’s advised to copy the layout file /var/lib/rear/layout/disklayout.conf to /etc/rear and modify it according to what is required.

cp /var/lib/rear/layout/disklayout.conf /etc/rear/
vi /etc/rear/disklayout.conf

Then restart the recovery process: rear recover. Relax-and-Recover translates this layout file into a shell procedure (/var/lib/rear/layout/diskrestore.sh) that contains all the needed instructions for recreating your desired layout.



You can find more information about your HP SmartArray setup by running one of the following commands:


# hpacucli ctrl all show detail
# hpacucli ctrl all show config
# hpacucli ctrl all show config detail

You can find these commands as part of the history of the Relax-and-Recover shell.
 

Options

To use Relax-and-Recover you always call the main script /usr/sbin/rear:

# rear help

Usage: rear [-h|--help] [-V|--version] [-dsSv] [-D|--debugscripts SET] [-c DIR] [-r KERNEL] [--] COMMAND [ARGS...]

Relax-and-Recover comes with ABSOLUTELY NO WARRANTY; for details see
the GNU General Public License at: http://www.gnu.org/licenses/gpl.html

Available options:
 -h --help           usage information
 -c DIR              alternative config directory; instead of /etc/rear
 -d                  debug mode; log debug messages
 -D                  debugscript mode; log every function call (via 'set -x')
 --debugscripts SET  same as -d -v -D but debugscript mode with 'set -SET'
 -r KERNEL           kernel version to use; current: '3.12.49-3-default'
 -s                  simulation mode; show what scripts rear would include
 -S                  step-by-step mode; acknowledge each script individually
 -v                  verbose mode; show more output
 -V --version        version information

List of commands

  1.  checklayout     check if the disk layout has changed
  2.  dump            dump configuration and system information
  3.  format          format and label media for use with rear
  4.  mkbackup        create rescue media and backup system
  5.  mkbackuponly    backup system without creating rescue media
  6.  mkrescue        create rescue media only
  7.  recover         recover the system
  8.  validate        submit validation information
  9. Use 'rear -v help' for more advanced commands.
To view/verify your configuration, run rear dump. It will print out the current settings for BACKUP  and OUTPUT  methods and some system information.

To create a new rescue environment, simply call rear mkrescue. Do not forget to copy the resulting rescue system away so that you can use it in the case of a system failure. Use rear mkbackup  instead if you are using the built-in backup functions (like BACKUP=NETFS)

To recover your system, start the computer from the rescue system and run rear recover. Your system will be recovered and you can restart it and continue to use it normally.

Restoring RHEL-based system

IMPORTANT: If SELinux is enabled on your system you need to ensure that SELinux relabels the files on the next boot:
~]# touch /mnt/local/.autorelabel
Otherwise you may be unable to log in the system, because the /etc/passwd file may have the incorrect SELinux context.

Performing a System Rescue (from Red Hat documentation)

To perform a restore or migration:
  1. Boot the rescue system on the new hardware. For example, burn the ISO image to a DVD and boot from the DVD.
     
  2. In the console interface, select the "Recover" option:

    Rescue system: menu

    Figure 27.1. Rescue system: menu
     

  3. You are taken to the prompt:

    Rescue system: prompt

    Figure 27.2. Rescue system: prompt

    Warning

    Once you have started recovery in the next step, it probably cannot be undone and you may lose anything stored on the physical disks of the system.
     
  4. Run the rear recover command to perform the restore or migration. The rescue system then recreates the partition layout and filesystems:

    Rescue system: running "rear recover"

    Figure 27.3. Rescue system: running "rear recover"
     

  5. Restore user and system files from the backup into the /mnt/local/ directory.

    Example 27.3. Restoring User and System Files

    In this example, the backup file is a tar archive created per instructions in Section 27.2.1.1, “Configuring the Internal Backup Method”. First, copy the archive from its storage, then unpack the files into /mnt/local/, then delete the archive:
    ~]# scp root@192.168.122.7:/srv/backup/rhel7/backup.tar.gz /mnt/local/
    ~]# tar xf /mnt/local/backup.tar.gz -C /mnt/local/
    ~]# rm -f /mnt/local/backup.tar.gz
    The new storage has to have enough space both for the archive and the extracted files.
     
  6. Verify that the files have been restored:
    ~]# ls /mnt/local/

    Rescue system: restoring user and system files from the backup

    Figure 27.4. Rescue system: restoring user and system files from the backup
     

  7. Ensure that SELinux relabels the files on the next boot:
    ~]# touch /mnt/local/.autorelabel
    Otherwise you may be unable to log in the system, because the /etc/passwd file may have the incorrect SELinux context.
     
  8. Finish the recovery by entering exit. ReaR will then reinstall the boot loader. After that, reboot the system:

    Rescue system: finishing recovery

    Figure 27.5. Rescue system: finishing recovery

    Upon reboot, SELinux will relabel the whole filesystem. Then you will be able to log in to the recovered system.

 

Affordable compact (FIT or bar form factor) 128 GB Flash drives

"FIT" form factor is the best as almost does not protrude from the USB port. There is a small (orange in case of SanDisk) indicator light and I kind of like it. When it is writing or reading, the light will blink. They are supposed to be plugged in and seldom pulled out, or pulled out fairly rarely. Perfect for local backup of OS.
Bar form factor protrude one inch or so. Which in many case is acceptable but still carry some risks.

USB 3.0 FIT

USB 3.0 BAR

USB 2.0


Top updates

Softpanorama Switchboard
Softpanorama Search


NEWS CONTENTS

Old News ;-)

linuxtag.org

BACKUP=NETFS

  • /usr/share/rear/conf/default.conf
  • By default is BACKUP_PROG=tar
  • However, BACKUP_PROG=rsync is possible for local attached storage
  • BACKUP_PROG_COMPRESS_OPTIONS="--
    gzip"
  • BACKUP_PROG_COMPRESS_SUFFIX=".gz"
  • BACKUP_PROG_EXCLUDE=( '/tmp/*&#n /etc/rear/local.conf (or /etc/rear/site.conf)

  • # grep -v -E '(^#|^$)' /etc/rear/local.conf
    OUTPUT=ISO
    MODULES_LOAD=( vmxnet )
  • Add:
    BACKUP=NETFS
    BACKUP_URL=nfs://server/path
  • On NFS server backup => /path/$(hostname)/

    Usage: rear [-dDsSvV] [-r KERNEL] COMMAND [--ARGS...]

  • Available options:
  • -d debug mode; log debug messages
  • -D debugscript mode; log every function call
  • -r KERNEL kernel version to use; current: '2.6.42.3-
    2.fc15.i686.PAE'
  • -s simulation mode; show what scripts rear would include
  • -S step-by-step mode; acknowledge each script individually
  • -v verbose mode; show more output
  • -V version information

    Usage: rear [-dDsSvV] [-r KERNEL] COMMAND [--ARGS...]

  • List of commands:
    – checklayout check if the disk layout has changed
    – format format and label media for use with rear
    – mkbackup create rescue media and backup system
    – mkbackuponly backup system without creating rescue media
    – mkrescue create rescue media only
    – recover recover the system; only valid during rescue
    – savelayout save the disk layout of the system
    – shell start a bash within rear; development tool

    Gather system information

  • Store the disk layout
  • Partitioning, LVM and RAID configuration
  • File systems, file system labels ...
  • Boot loader (GRUB, LILO, ELILO)
  • Make a system backup (OS and user data)
  • Create boot-able rescue media with system configuration (and
    optional with backup data)
  • All steps are done “online”

    Will create an ISO image stored as

  • /tmp/rear-$(hostname).iso
  • On NFS server as /path/$(hostname)/rear-\
    $(hostname).iso
  • Inspect file /var/lib/rear/layout/disklayout.conf
  • Try to boot from the ISO image into the RESCUE system
  • Use 'dmesg' to check if devices were found

    Boot rescue image and select 'recover

    '

    mkbackup method: /usr/share/rear/...

  • conf/ - configuration files (/etc/rear/*.conf read last)
  • prep/ - preparation work; checking the environment
  • layout/save/ - save the disk layout /var/lib/rear/layout
  • rescue/ - modules, network, storage,...
  • build/ - populate the initial ramdisk for our rescue image
  • pack/ - create the initrd and copy kernel
  • output/ - create the ISO image and copy to
    OUTPUT_URL
  • backup/ - make the backup archive to BACKUP_URL

    A simple script to save basic system requirements – sysreqs.sh

  • OS version; rear version
  • CPU, memory
  • Disk space requirements
  • IP addresses in use; routes
  • Copy sysreqs.sh to a flow, e.g. rescue is a good choice
  • # cp /tmp/sysreqs.sh \
    /usr/share/rear/rescue/GNU/Linux/96_sysreqs.sh

  • [Nov 05, 2016] Relax and Recover – How Did I Do That

    www.howdididothat.info

    21 August 2014

    Start a backup on the CentOS machine

    Add the following lines to /etc/rear/local.conf:

    OUTPUT=ISO
    BACKUP=NETFS
    BACKUP_TYPE=incremental
    BACKUP_PROG=tar
    FULLBACKUPDAY="Mon"
    BACKUP_URL="nfs://NFSSERVER/path/to/nfs/export/servername"
    BACKUP_PROG_COMPRESS_OPTIONS="--gzip"
    BACKUP_PROG_COMPRESS_SUFFIX=".gz"
    BACKUP_PROG_EXCLUDE=( '/tmp/*' '/dev/shm/*' )
    BACKUP_OPTIONS="nfsvers=3,nolock"

    OUTPUT=ISO
    BACKUP=NETFS
    BACKUP_TYPE=incremental
    BACKUP_PROG=tar
    FULLBACKUPDAY="Mon"
    BACKUP_URL="nfs://NFSSERVER/path/to/nfs/export/servername"
    BACKUP_PROG_COMPRESS_OPTIONS="--gzip"
    BACKUP_PROG_COMPRESS_SUFFIX=".gz"
    BACKUP_PROG_EXCLUDE=( '/tmp/*' '/dev/shm/*' )
    BACKUP_OPTIONS="nfsvers=3,nolock"


    Now make a backup

    [root@centos7 ~]# rear mkbackup -v
    Relax-and-Recover 1.16.1 / Git
    Using log file: /var/log/rear/rear-centos7.log
    mkdir: created directory '/var/lib/rear/output'
    Creating disk layout
    Creating root filesystem layout
    TIP: To login as root via ssh you need to set up /root/.ssh/authorized_keys or SSH_ROOT_PASSWORD in your configuration file
    Copying files and directories
    Copying binaries and libraries
    Copying kernel modules
    Creating initramfs
    Making ISO image
    Wrote ISO image: /var/lib/rear/output/rear-centos7.iso (90M)
    Copying resulting files to nfs location
    Encrypting disabled
    Creating tar archive '/tmp/rear.QnDt1Ehk25Vqurp/outputfs/centos7/2014-08-21-1548-F.tar.gz'
    Archived 406 MiB [avg 3753 KiB/sec]OK
    Archived 406 MiB in 112 seconds [avg 3720 KiB/sec]

    Now look on your NFS server

    You'll see all the files you'll need to perform the disaster recovery.

    total 499M
    drwxr-x- 2 root root 4.0K Aug 21 23:51 .
    drwxr-xr-x 3 root root 4.0K Aug 21 23:48 ..
    -rw--- 1 root root 407M Aug 21 23:51 2014-08-21-1548-F.tar.gz
    -rw--- 1 root root 2.2M Aug 21 23:51 backup.log
    -rw--- 1 root root 202 Aug 21 23:49 README
    -rw--- 1 root root 90M Aug 21 23:49 rear-centos7.iso
    -rw--- 1 root root 161K Aug 21 23:49 rear.log
    -rw--- 1 root root 0 Aug 21 23:51 selinux.autorelabel
    -rw--- 1 root root 277 Aug 21 23:49 VERSION


    Author: masterdam79

    You can also connect with me on Google+ View all posts by masterdam79


    Author masterdam79/
    Posted on 21 August 2014/

    dheeraj says:

    31 August 2016 at 02:26


    is it possible to give list of directories or mount points while giving mkbackup to exclude from backup. Like giving a file with list of all directories that need to be excluded ??

    masterdam79 says:

    26 September 2016 at 21:50

    Have a look at https://github.com/rear/rear/issues/216
    Should be possible if you ask me.

    [Nov 05, 2016] Relax-and-Recover – Freecode

    Nov 05, 2016 | freecode.com

    Relax-and-Recover (Rear) is a bare metal disaster recovery and system migration solution, similar to AIX mksysb or HP-UX ignite. It is composed of a modular framework and ready-to-go workflows for many common situations to produce a bootable image and restore from backup using this image. It can restore to different hardware, and can therefore be used as a migration tool as well. It supports various boot media (including tape, USB, or eSATA storage, ISO, PXE, etc.), a variety of network protocols (including SFTP, FTP, HTTP, NFS, and CIFS), as well as a multitude of backup strategies (including IBM TSM, HP DataProtector, Symantec NetBackup, Bacula, and rsync). It was designed to be easy to set up, requires no maintenance, and is there to assist when disaster strikes. Recovering from disaster is made very straight-forward by a 2-step recovery process so that it can be executed by operational teams when required. When used interactively (e.g. when used for migrating systems), menus help make decisions to restore to a new (hardware) environment.

    Release Notes: Integrated with duply/duplicity support. systemd support has been added. Various small fixes and improvements to tape support, Xen, PPC, Gentoo, Fedora, multi-arch, storage ... layout configuration, and serial console integration.

    (more)

    Release Notes: This release adds support for multipathing, adds several improvements to distribution backward compatibility, improves ext4 support, makes various bugfixes, migrates HWADDR ... after rescovery, and includes better systemd support.

    (more)

    Release Notes: Multi-system and multi-copy support on USB storage devices. Basic rsync backup support. More extensive exclude options. The new layout code is enabled by default. Support ... for Arch Linux. Improved multipath support. Experimental btrfs support.

    (more)

    Release Notes: Standardization of the command line. The default is quiet output; use the option -v for the old behavior. Boot images now have a comprehensive boot menu. Support for IPv6 ... addresses. Restoring NBU backup from a point in time is supported. Support for Fedora 15 (systemd) and RHEL6/SL6. Improved handling of HP SmartArray. Support for ext4 on RHEL5/SL5. Support for Xen paravirtualization. Integration with the local GRUB menu. Boot images can now be centralized through network transfers. Support for udev on RHEL4. Many small improvements and performance enhancements.

    (more)
    • 1.6
    • 11 Dec 2007 22:07

    Release Notes: This release supports many recent distributions, including "upstart" (Ubuntu 7.10). It has more IA-64 support (RHEL5 only at the moment), better error reporting and catching, ... Debian packages (mkdeb), and improved TSM support.

    (more)

    [Nov 05, 2016] Affordable compact (FIT or bar form factor) 128 GB Flash drives

    "FIT" form factor is the best as almost does not protrude from the USB port. There is a small (orange in case of SanDisk) indicator light and I kind of like it. When it is writing or reading, the light will blink. They are supposed to be plugged in and seldom pulled out, or pulled out fairly rarely. Perfect for local backup of OS.
    Bar form factor protrude one inch or so. Which in many case is acceptable but still carry some risks.
    Nov 05, 2016 | www.amazon.com

    USB 3.0 FIT

    USB 3.0 BAR

    USB 2.0

    [Nov 04, 2016] Coding Style rear-rear Wiki

    Reading rear sources is an interesting exercise. It really demonstrates attempt to use "reasonable' style of shell programming and you can learn a lot.
    Nov 04, 2016 | github.com

    Relax-and-Recover is written in Bash (at least bash version 3 is needed), a language that can be used in many styles. We want to make it easier for everybody to understand the Relax-and-Recover code and subsequently to contribute fixes and enhancements.

    Here is a collection of coding hints that should help to get a more consistent code base.

    Don't be afraid to contribute to Relax-and-Recover even if your contribution does not fully match all this coding hints. Currently large parts of the Relax-and-Recover code are not yet in compliance with this coding hints. This is an ongoing step by step process. Nevertheless try to understand the idea behind this coding hints so that you know how to break them properly (i.e. "learn the rules so you know how to break them properly").

    The overall idea behind this coding hints is:

    Make yourself understood

    Make yourself understood to enable others to fix and enhance your code properly as needed.

    From this overall idea the following coding hints are derived.

    For the fun of it an extreme example what coding style should be avoided:

    #!/bin/bash for i in `seq 1 2 $((2*$1-1))`;do echo $((j+=i));done

    
    
       

    Try to find out what that code is about - it does a useful thing.

    Code must be easy to read Code should be easy to understand

    Do not only tell what the code does (i.e. the implementation details) but also explain what the intent behind is (i.e. why ) to make the code maintainable.

    Here the initial example so that one can understand what it is about:

    #!/bin/bash # output the first N square numbers # by summing up the first N odd numbers 1 3 ... 2*N-1 # where each nth partial sum is the nth square number # see https://en.wikipedia.org/wiki/Square_number#Properties # this way it is a little bit faster for big N compared to # calculating each square number on its own via multiplication N=$1 if ! [[ $N =~ ^[0-9]+$ ]] ; then echo "Input must be non-negative integer." 1>&2 exit 1 fi square_number=0 for odd_number in $( seq 1 2 $(( 2 * N - 1 )) ) ; do (( square_number += odd_number )) && echo $square_number done

    Now the intent behind is clear and now others can easily decide if that code is really the best way to do it and easily improve it if needed.

    Try to care about possible errors

    By default bash proceeds with the next command when something failed. Do not let your code blindly proceed in case of errors because that could make it hard to find the root cause of a failure when it errors out somewhere later at an unrelated place with a weird error message which could lead to false fixes that cure only a particular symptom but not the root cause.

    Maintain Backward Compatibility

    Implement adaptions and enhancements in a backward compatible way so that your changes do not cause regressions for others.

    Dirty hacks welcome

    When there are special issues on particular systems it is more important that the Relax-and-Recover code works than having nice looking clean code that sometimes fails. In such special cases any dirty hacks that intend to make it work everywhere are welcome. But for dirty hacks the above listed coding hints become mandatory rules:

    For example a dirty hack like the following is perfectly acceptable:

    # FIXME: Dirty hack to make it work # on "FUBAR Linux version 666" # where COMMAND sometimes inexplicably fails # but always works after at most 3 attempts # see http://example.org/issue12345 # Retries should have no bad effect on other systems # where the first run of COMMAND works. COMMAND || COMMAND || COMMAND || Error "COMMAND failed."

    Character Encoding

    Use only traditional (7-bit) ASCII charactes. In particular do not use UTF-8 encoded multi-byte characters.

    Text Layout Variables Functions Relax-and-Recover functions

    Use the available Relax-and-Recover functions when possible instead of re-implementing basic functionality again and again. The Relax-and-Recover functions are implemented in various lib/*-functions.sh files .

    test, [, [[, (( Paired parenthesis See also

    Recommended Links

    Softpanorama hot topic of the month

    Softpanorama Recommended

    ...



    Etc

    FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes.   If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner. 

    ABUSE: IPs or network segments from which we detect a stream of probes might be blocked for no less then 90 days. Multiple types of probes increase this period.  

    Society

    Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

    Quotes

    War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

    Bulletin:

    Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

    History:

    Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

    Classic books:

    The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

    Most popular humor pages:

    Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

    The Last but not Least


    Copyright © 1996-2016 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.

    The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

    Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

    FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

    This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

    You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info

    Disclaimer:

    The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

    Last modified: January, 13, 2017