Solaris Backup/Restore

"Native" backup and recovery in Solaris show signs of 30 years of Unix development. Frankly speaking it is potpourri of almost a dozen partially incompatible utilities plus equal amount of GNU clones if you install them (gzip is installed by default). Many utilities duplicate each other and none is very competitive with the best Windows backup and recovery tools (as exemplified by rar and Ghost to name a few; there are close of Ghost for linux).

Most native Solaris tools support ACLs, most GNU tools don't (although in some combinations like Solaris tar + GNU gzip (tar.gz or tgz archives) can support them. Solaris' approach to handling ACLs in cpio and tar archives is using two files of the same name. The first file held the ACL info, and the second file was the actual file. If you weren't running on Solaris, the second file would simply overwrite the first one, so the format was, for all intents and purposes, 100% backward compatible.

Here is a "slightly skeptical" characterization of some:

Comparing tar, cpio, and dump

There is a very old paper by John Pezzano from Hewlett-Packard comparing three backup utilities:

Feature	tar	cpio	dump
Simplicity of invocation	Very simple (tar c `files`)	Needs find to specify filenames	Simple—few options
Recovery from I/O errors	None—write your own utility	Resync option on HP-UX will cause some data loss	Automatically skips over bad section
Back up special files	Later revisions	Yes	Yes
Multivolume backup	Later revisions	Yes	Yes
Back up across network	Using rsh only	Using rsh only	Yes
Append files to backup	Yes (tar -r)	No	No
Multiple independent backups on single tape	Yes	Yes	Yes
Ease of listing files on the volume	Difficult—must search entire backup (tar -t)	Difficult—must search entire backup (cpio -it)	Simple—index at front (restore -t)
Ease and speed of finding a particular file	Difficult—no wildcards, must search entire volume	Moderate—wildcards, must search entire volume	Interactive—very easy with commands like cd, ls
Incremental backup	No	Must use find to locate new/modified files	Incremental of whole filesystem only, multiple levels
List files as they are being backed up	tar cvf 2>`logfile`	cpio -v 2>`logfile`	Only after backup with restore -t >`logfile` (dump can show % complete, though)
Back up based on other criteria	No	find can use multiple criteria	No
Restore absolute pathnames to relative location	Only by using chroot	Limited with cpio -I	Always relative to current working directory
Interactive decision on restore	Yes or no possible with tar -w	Can specify new path or name on each file	Specify individual files in interactive mode
Compatibility	Multiple platform	Multiple platform with ASCII header, not always portable	Readable between some platforms, but cannot be relied on
Primary usefulness	Individual user backup, transfer files between filesystems	System backup, transfer files between filesystems	System backup
Volume efficiency	Medium, usually limited to 10 K block size	Medium, usually only 5 K block size, but can specify larger size on some OSes	High, can usually specify up to maximum block size of device
Wildcards on restore	No	Yes	Only in interactive mode
Simplicity of selecting files for backup from numerous directories	Low—must specify each independent directory, subdirectories included	Medium—find options	None—will back up one and only one filesystem
Specifying directory on restore get files in that directory	Yes	No—must use `path`/*	Yes
Stop reading tape after a restored file is found	No	No	Will stop reading tape as soon as last file is found
Track deleted files	No	No	If you restore with -r, files deleted before last incremental dump will be deleted
Filesystem efficiency	Better	Worst (files get a stat from both find and cpio)	Best
Limit on path length(tests done with Solaris native utilities 7/99)	155 characters. Complains "prefix is greater than 155 characters." gtar has slight workaround	255 characters. Doesn't complain. Just truncates pathname to 255 chars	1056 characters.
Likelihood that file exists in TOC but not in archive	Low	Low	Medium (since TOC is made first)

Old News ;-)

Admin's Choice - Solaris & Unix Discussion Forums BTIPS Backup commands - ufsdump , tar , cpio-B

The Solaris Companion: Reliable and Practical Root Disk MirroringSys Admin Magazinecolumn by Peter Baer Galvin

A Tool for Cold Mirroring of Solaris System Disks

Linux and Solaris ACLs - Backup

Recommended Links

Tutorials

Tips

The Solaris version of tar includes extra options. The -I option allows a list of files and directories that are backed up to be put into a text file. The -X option allows an exclusion file to be specified that lists the names of files and directories that should be skipped.

The Solaris version of mt supports an asf subcommand which moves the tape to the nth file. n being the number of the file.

Archiving

tar

The standard UNIX archiving utility. Originally a Tape ARchiving program, it has developed into a general purpose package that can handle all manner of archiving with all types of destination devices, ranging from tape drives to regular files to even stdout (see Example 4-3). GNU tar has long since been patched to accept gzip compression options, such as tar czvf archive-name.tar.gz *, which recursively archives and compresses all files (except dotfiles) in a directory tree.

Some useful tar options:

-c create (a new archive)
--delete delete (files from the archive)
-r append (files to the archive)
-t list (archive contents)
-u update archive
-x extract (files from the archive)
-z gzip the archive

It may be difficult to recover data from a corrupted gzipped tar archive. When archiving important files, make multiple backups.

shar

Shell archiving utility. The files in a shell archive are concatenated without compression, and the resultant archive is essentially a shell script, complete with #!/bin/sh header, and containing all the necessary unarchiving commands. Shar archives still show up in Internet newsgroups, but otherwise shar has been pretty well replaced by tar/gzip. The unshar command unpacks shar archives.

ar

Creation and manipulation utility for archives, mainly used for binary object file libraries.

cpio

This specialized archiving copy command is rarely seen any more, having been supplanted by tar/gzip. It still has its uses, such as moving a directory tree.

Example 12-21. Using cpio to move a directory tree

#!/bin/bash

# Copying a directory tree using cpio.

ARGS=2
E_BADARGS=65

if [ $# -ne "$ARGS" ]
then
  echo "Usage: `basename $0` source destination"
  exit $E_BADARGS
fi  

source=$1
destination=$2

find "$source" -depth | cpio -admvp "$destination"
# Read the man page to decipher these cpio options.

exit 0

Example 12-22. Unpacking an rpm archive

#!/bin/bash
# de-rpm.sh: Unpack an 'rpm' archive

E_NO_ARGS=65
TEMPFILE=$$.cpio                         # Tempfile with "unique" name.
                                         # $$ is process ID of script.

if [ -z "$1" ] 
then
  echo "Usage: `basename $0` filename"
exit $E_NO_ARGS
fi


rpm2cpio < $1 > $TEMPFILE                # Converts rpm archive into cpio archive.
cpio --make-directories -F $TEMPFILE -i  # Unpacks cpio archive.
rm -f $TEMPFILE                          # Deletes cpio archive.

exit 0

Compression

gzip

The standard GNU/UNIX compression utility, replacing the inferior and proprietary compress. The corresponding decompression command is gunzip, which is the equivalent of gzip -d.

The zcat filter decompresses a gzipped file to stdout, as possible input to a pipe or redirection. This is, in effect, a cat command that works on compressed files (including files processed with the older compress utility). The zcat command is equivalent to gzip -dc.

On some commercial UNIX systems, zcat is a synonym for uncompress -c, and will not work on gzipped files.

See also Example 7-6.

bzip2

An alternate compression utility, usually more efficient than gzip, especially on large files. The corresponding decompression command is bunzip2.

compress, uncompress

This is an older, proprietary compression utility found in commercial UNIX distributions. The more efficient gzip has largely replaced it. Linux distributions generally include a compress workalike for compatibility, although gunzip can unarchive files treated with compress.

The znew command transforms compressed files into gzipped ones.

sq

Yet another compression utility, a filter that works only on sorted ASCII word lists. It uses the standard invocation syntax for a filter, sq < input-file > output-file. Fast, but not nearly as efficient as gzip. The corresponding uncompression filter is unsq, invoked like sq.

The output of sq may be piped to gzip for further compression.

zip, unzip

Cross-platform file archiving and compression utility compatible with DOS PKZIP. "Zipped" archives seem to be a more acceptable medium of exchange on the Internet than "tarballs".

File Information

file

A utility for identifying file types. The command file file-name will return a file specification for file-name, such as ascii text or data. It references the magic numbers found in /usr/share/magic, /etc/magic, or /usr/lib/magic, depending on the Linux/UNIX distribution.

The -f option causes file to run in batch mode, to read from a designated file a list of filenames to analyze. The -z option, when used on a compressed target file, forces an attempt to analyze the uncompressed file type.

bash$ file test.tar.gz
test.tar.gz: gzip compressed data, deflated, last modified: Sun Sep 16 13:34:51 2001, os: Unix

bash file -z test.tar.gz
test.tar.gz: GNU tar archive (gzip compressed data, deflated, last modified: Sun Sep 16 13:34:51 2001, os: Unix)

Example 12-23. stripping comments from C program files

#!/bin/bash
# strip-comment.sh: Strips out the comments (/* COMMENT */) in a C program.

E_NOARGS=65
E_ARGERROR=66
E_WRONG_FILE_TYPE=67

if [ $# -eq "$E_NOARGS" ]
then
  echo "Usage: `basename $0` C-program-file" >&2 # Error message to stderr.
  exit $E_ARGERROR
fi  

# Test for correct file type.
type=`eval file $1 | awk '{ print $2, $3, $4, $5 }'`
# "file $1" echoes file type...
# then awk removes the first field of this, the filename...
# then the result is fed into the variable "type".
correct_type="ASCII C program text"

if [ "$type" != "$correct_type" ]
then
  echo
  echo "This script works on C program files only."
  echo
  exit $E_WRONG_FILE_TYPE
fi  


# Rather cryptic sed script:
#--------
sed '
/^\/\*/d
/.*\/\*/d
' $1
#--------
# Easy to understand if you take several hours to learn sed fundamentals.


# Need to add one more line to the sed script to deal with
# case where line of code has a comment following it on same line.
# This is left as a non-trivial exercise for the reader.

# Also, the above code deletes lines with a "*/" or "/*",
# not a desirable result.

exit 0


# ----------------------------------------------------------------
# Code below this line will not execute because of 'exit 0' above.

# Stephane Chazelas suggests the following alternative:

usage() {
  echo "Usage: `basename $0` C-program-file" >&2
  exit 1
}

WEIRD=`echo -n -e '\377'`   # or WEIRD=$'\377'
[[ $# -eq 1 ]] || usage
case `file "$1"` in
  *"C program text"*) sed -e "s%/\*%${WEIRD}%g;s%\*/%${WEIRD}%g" "$1" \
     | tr '\377\n' '\n\377' \
     | sed -ne 'p;n' \
     | tr -d '\n' | tr '\377' '\n';;
  *) usage;;
esac

# This is still fooled by things like:
# printf("/*");
# or
# /*  /* buggy embedded comment */
#
# To handle all special cases (comments in strings, comments in string
# where there is a \", \\" ...) the only way is to write a C parser
# (lex or yacc perhaps?).

exit 0

which

which command-xxx gives the full path to "command-xxx". This is useful for finding out whether a particular command or utility is installed on the system.

$bash which rm

/usr/bin/rm

whereis

Similar to which, above, whereis command-xxx gives the full path to "command-xxx", but also to its manpage.

$bash whereis rm

rm: /bin/rm /usr/share/man/man1/rm.1.bz2

whatis

whatis filexxx looks up "filexxx" in the whatis database. This is useful for identifying system commands and important configuration files. Consider it a simplified man command.

$bash whatis whatis

whatis               (1)  - search the whatis database for complete words

Example 12-24. Exploring /usr/X11R6/bin

#!/bin/bash

# What are all those mysterious binaries in /usr/X11R6/bin?

DIRECTORY="/usr/X11R6/bin"
# Try also "/bin", "/usr/bin", "/usr/local/bin", etc.

for file in $DIRECTORY/*
do
  whatis `basename $file`   # Echoes info about the binary.
done

exit 0
# You may wish to redirect output of this script, like so:
# ./what.sh >>whatis.db
# or view it a page at a time on stdout,
# ./what.sh | less

See also Example 10-3.

vdir

Show a detailed directory listing. The effect is similar to ls -l.

This is one of the GNU fileutils.

bash$ vdir
total 10
 -rw-r--r--    1 bozo  bozo      4034 Jul 18 22:04 data1.xrolo
 -rw-r--r--    1 bozo  bozo      4602 May 25 13:58 data1.xrolo.bak
 -rw-r--r--    1 bozo  bozo       877 Dec 17  2000 employment.xrolo

bash ls -l
total 10
 -rw-r--r--    1 bozo  bozo      4034 Jul 18 22:04 data1.xrolo
 -rw-r--r--    1 bozo  bozo      4602 May 25 13:58 data1.xrolo.bak
 -rw-r--r--    1 bozo  bozo       877 Dec 17  2000 employment.xrolo

shred

Securely erase a file by overwriting it multiple times with random bit patterns before deleting it. This command has the same effect as Example 12-31, but does it in a more thorough and elegant manner.

This is one of the GNU fileutils.

Using shred on a file may not prevent recovery of some or all of its contents using advanced forensic technology.

locate, slocate

The locate command searches for files using a database stored for just that purpose. The slocate command is the secure version of locate (which may be aliased to slocate).

$bash locate hickson

/usr/lib/xephem/catalogs/hickson.edb

strings

Use the strings command to find printable strings in a binary or data file. It will list sequences of printable characters found in the target file. This might be handy for a quick 'n dirty examination of a core dump or for looking at an unknown graphic image file (strings image-file | more might show something like JFIF, which would identify the file as a jpeg graphic). In a script, you would probably parse the output of strings with grep or sed. See Example 10-7 and Example 10-8.

Utilities

basename

Strips the path information from a file name, printing only the file name. The construction basename $0 lets the script know its name, that is, the name it was invoked by. This can be used for "usage" messages if, for example a script is called with missing arguments:

echo "Usage: `basename $0` arg1 arg2 ... argn"

dirname

Strips the basename from a filename, printing only the path information.

basename and dirname can operate on any arbitrary string. The argument does not need to refer to an existing file, or even be a filename for that matter (see Example A-6).

Example 12-25. basename and dirname

#!/bin/bash

a=/home/bozo/daily-journal.txt

echo "Basename of /home/bozo/daily-journal.txt = `basename $a`"
echo "Dirname of /home/bozo/daily-journal.txt = `dirname $a`"
echo
echo "My own home is `basename ~/`."         # Also works with just ~.
echo "The home of my home is `dirname ~/`."  # Also works with just ~.

exit 0

split

Utility for splitting a file into smaller chunks. Usually used for splitting up large files in order to back them up on floppies or preparatory to e-mailing or uploading them.

sum, cksum, md5sum

These are utilities for generating checksums. A checksum is a number mathematically calculated from the contents of a file, for the purpose of checking its integrity. A script might refer to a list of checksums for security purposes, such as ensuring that the contents of key system files have not been altered or corrupted. The md5sum command is the most appropriate of these in security applications.

Encoding and Encryption

uuencode

This utility encodes binary files into ASCII characters, making them suitable for transmission in the body of an e-mail message or in a newsgroup posting.

uudecode

This reverses the encoding, decoding uuencoded files back into the original binaries.

Example 12-26. uudecoding encoded files

#!/bin/bash

lines=35        # Allow 35 lines for the header (very generous).

for File in *   # Test all the files in the current working directory...
do
  search1=`head -$lines $File | grep begin | wc -w`
  search2=`tail -$lines $File | grep end | wc -w`
  #  Uuencoded files have a "begin" near the beginning,
  #+ and an "end" near the end.
  if [ "$search1" -gt 0 ]
  then
    if [ "$search2" -gt 0 ]
    then
      echo "uudecoding - $File -"
      uudecode $File
    fi  
  fi
done  

#  Note that running this script upon itself fools it
#+ into thinking it is a uuencoded file,
#+ because it contains both "begin" and "end".

# Exercise:
# Modify this script to check for a newsgroup header.

exit 0

The fold -s command may be useful (possibly in a pipe) to process long uudecoded text messages downloaded from Usenet newsgroups.

crypt

At one time, this was the standard UNIX file encryption utility. [1] Politically motivated government regulations prohibiting the export of encryption software resulted in the disappearance of crypt from much of the UNIX world, and it is still missing from most Linux distributions. Fortunately, programmers have come up with a number of decent alternatives to it, among them the author's very own cruft (see Example A-4).

Miscellaneous

make: Utility for building and compiling binary packages. This can also be used for any set of operations that is triggered by incremental changes in source files.
The make command checks a Makefile, a list of file dependencies and operations to be carried out.
install: Special purpose file copying command, similar to cp, but capable of setting permissions and attributes of the copied files. This command seems tailormade for installing software packages, and as such it shows up frequently in Makefiles (in the make install : section). It could likewise find use in installation scripts.
more, less: Pagers that display a text file or stream to stdout, one screenful at a time. These may be used to filter the output of a script.

Softpanorama May the source be with you, but remember the KISS principle ;-)
	Contents	Bulletin	Latest	Last year	Top visited	Scriptorama

Top updates <p>Your browser does not support iframes.</p>	Top visited <p>Your browser does not support iframes.</p>
	Google Search
	Bulletin	Latest	Past week	Past month

News	Recommended Links	Recommended Books	Backup media and utilities	Recovering file systems	Flash archives
tar	dd	cpio	pax	ufsdump	ufsrestore	`fssnap`
volcopy	compress /uncomress	gzip	zip	bzip2	rar	Tips