|
Softpanorama
(slightly skeptical)
Open Source Software Educational Society |
May the
source be with you,
but remember the KISS principle ;-)
|
Unix DD Command and Image Creation
The dd command has been around since
the 1970s, ported to many systems, rewritten many times, and proved to be
an indispensable Unix tool. The name is an allusion to mainframe JCL
DD statement. It served as an inspiration to the most important recent
backup program for Windows
Ghost. The
key idea is to get an image of the partition in the form of the file.
It can be used to recovery of deleted files and in computer forensics when
the contents of a disk need to be preserved as a byte-exact copy. In the
latter case using cp command would not be sufficient because
data from deleted files still physically present on a disk but are
visible in the file system naming space.
The GNU clone of dd is part of
fileutils package and was written by Paul Rubin, David MacKenzie,
and Stuart Kemp.
dd is also available for Windows
as part of Microsoft
Unix toolkit (SFU 3.5) and can be used in Windows environment too. At
least for reading raw partitions. It is also part of Cygwin.
There is also natural Windows port of dd
.
Unlike most commands, dd uses a
keyword=value format for its parameters. This was modeled
after IBM System/360 JCL, which had an elaborate DD 'Dataset Definition'
specification for I/O devices in JCP language.
A complete listing of all keywords is available via
dd --help
The dd command in UNIX and its ported
variants for windows (for example the implementation from
SFU 3.5 )
can be used to do direct dumps from one device to file or vise versa. It
is a useful tool create a disk image or to make a disk from an image but
in windows environment it was by and large displaced
Ghost and its
alternatives. On Linux Partimage is especially useful alternative
to dd. See for example
Backup and Restore Linux Partitions Using Partimage -- Debian Admin
Partition Image is a Linux/UNIX utility which saves partitions in many formats
(see below) to an image file. The image file can be compressed in the GZIP/BZIP2
formats to save disk space, and split into multiple files. Partitions can
be saved across the network since version 0.6.0.When using Partimage, the
partitions must be unmounted.
For more detailed information about capabilities of Classic Unix
dd see Solaris
man page:
The dd utility copies the specified input
file to the specified output with possible conversions. The standard
input and output are used by default. The input and output block sizes
may be specified to take advantage of raw physical I/O. Sizes are specified
in bytes; a number may end with k, b, or w
to specify multiplication by 1024, 512, or 2, respectively. Numbers
may also be separated by x to indicate multiplication.
The dd utility reads the input one block
at a time, using the specified input block size. dd then
processes the block of data actually returned, which could be smaller
than the requested block size. dd applies any conversions
that have been specified and writes the resulting data to the output
in blocks of the specified output block size.
cbs is used only if
ascii, asciib, unblock, ebcdic,
ebcdicb, ibm, ibmb, or block conversion
is specified. In the first two cases, cbs characters are copied
into the conversion buffer, any specified character mapping is done,
trailing blanks are trimmed, and a NEWLINE is added before
sending the line to output. In the last three cases, characters up to
NEWLINE are read into the conversion buffer and blanks are
added to make up an output record of size cbs. ASCII files
are presumed to contain NEWLINE characters. If cbs
is unspecified or 0, the ascii, asciib,
ebcdic, ebcdicb, ibm, and ibmb options
convert the character set without changing the input file's block structure.
The unblock and block options become a simple file
copy.
After completion, dd reports the number
of whole and partial input and output blocks.
OPERANDS
The following operands are supported:
- if=file
- Specifies the input path. Standard input is the
default.
- of=file
- Specifies the output path. Standard output is
the default. If the seek=expr conversion is
not also specified, the output file will be truncated before the
copy begins, unless conv=notrunc is specified. If seek=expr
is specified, but conv=notrunc is not, the effect of the
copy will be to preserve the blocks in the output file over which
dd seeks, but no other portion of the output file will
be preserved. (If the size of the seek plus the size of the input
file is less than the previous size of the output file, the output
file is shortened by the copy.)
- ibs=n
- Specifies the input block size in n
bytes (default is 512).
- obs=n
- Specifies the output block size in n
bytes (default is 512).
- bs=n
- Sets both input and output block sizes to
n bytes, superseding ibs= and obs=.
If no conversion other than sync, noerror, and
notrunc is specified, each input block is copied to the
output as a single block without aggregating short blocks.
- cbs=n
- Specifies the conversion block size for block
and unblock in bytes by n (default is 0).
If cbs= is omitted or given a value of 0, using
block or unblock produces unspecified results.
This option is used only if ASCII or EBCDIC conversion
is specified. For the ascii and asciib operands,
the input is handled as described for the unblock operand
except that characters are converted to ASCII before the trailing
SPACE characters are deleted. For the ebcdic,
ebcdicb, ibm, and ibmb operands, the
input is handled as described for the block operand except
that the characters are converted to EBCDIC or IBM EBCDIC after
the trailing SPACE characters are added.
- files=n
- Copies and concatenates n input files
before terminating (makes sense only where input is a magnetic tape
or similar device).
- skip=n
- Skips n input blocks (using the specified
input block size) before starting to copy. On seekable files, the
implementation reads the blocks or seeks past them. On non-seekable
files, the blocks are read and the data is discarded.
- iseek=n
- Seeks n blocks from beginning of input
file before copying (appropriate for disk files, where skip
can be incredibly slow).
- oseek=n
- Seeks n blocks from beginning of output
file before copying.
- seek=n
- Skips n blocks (using the specified
output block size) from beginning of output file before copying.
On non-seekable files, existing blocks are read and space from the
current end-of-file to the specified offset, if any, is filled with
null bytes. On seekable files, the implementation seeks to the specified
offset or reads the blocks as described for non-seekable files.
- count=n
- Copies only n input blocks.
- conv=value[,value.
. . ]
- Where values are comma-separated symbols
from the following list:
- ascii
- Converts EBCDIC to ASCII.
- asciib
- Converts EBCDIC to ASCII using BSD-compatible
character translations.
- ebcdic
- Converts ASCII to EBCDIC. If converting fixed-length
ASCII records without NEWLINEs, sets up a pipeline with
dd conv=unblock beforehand.
- ebcdicb
- Converts ASCII to EBCDIC using BSD-compatible
character translations. If converting fixed-length ASCII records
without NEWLINEs, sets up a pipeline with dd conv=unblock
beforehand.
- ibm
- Slightly different map of ASCII to EBCDIC.
If converting fixed-length ASCII records without NEWLINEs,
sets up a pipeline with dd conv=unblock beforehand.
- ibmb
- Slightly different map of ASCII to EBCDIC
using BSD-compatible character translations. If converting fixed-length
ASCII records without NEWLINEs, sets up a pipeline
with dd conv=unblock beforehand.
The ascii (or asciib), ebcdic
(or ebcdicb), and ibm (or ibmb) values
are mutually exclusive.
- block
- Treats the input as a sequence of NEWLINE-terminated
or EOF-terminated variable-length records independent
of the input block boundaries. Each record is converted to a
record with a fixed length specified by the conversion block
size. Any NEWLINE character is removed from the input
line. SPACE characters are appended to lines that are
shorter than their conversion block size to fill the block.
Lines that are longer than the conversion block size are truncated
to the largest number of characters that will fit into that
size. The number of truncated lines is reported.
- unblock
- Converts fixed-length records to variable
length. Reads a number of bytes equal to the conversion block
size (or the number of bytes remaining in the input, if less
than the conversion block size), delete all trailing SPACE
characters, and append a NEWLINE character.
The block and unblock values
are mutually exclusive.
- lcase
- Maps upper-case characters specified by the
LC_CTYPE keyword tolower to the corresponding
lower-case character. Characters for which no mapping is specified
are not modified by this conversion.
- ucase
- Maps lower-case characters specified by the
LC_CTYPE keyword toupper to the corresponding
upper-case character. Characters for which no mapping is specified
are not modified by this conversion.
The lcase and ucase symbols
are mutually exclusive.
- swab
- Swaps every pair of input bytes. If the current
input record is an odd number of bytes, the last byte in the
input record is ignored.
- noerror
- Does not stop processing on an input error.
When an input error occurs, a diagnostic message is written
on standard error, followed by the current input and output
block counts in the same format as used at completion. If the
sync conversion is specified, the missing input is
replaced with null bytes and processed normally. Otherwise,
the input block will be omitted from the output.
- notrunc
- Does not truncate the output file. Preserves
blocks in the output file not explicitly written by this invocation
of dd. (See also the preceding of=file
operand.)
- sync
- Pads every input block to the size of the
ibs= buffer, appending null bytes. (If either block
or unblock is also specified, appends SPACE
characters, rather than null bytes.)
If operands other than conv= are specified
more than once, the last specified operand=value
is used.
For the bs=, cbs=, ibs=,
and obs= operands, the application must supply an expression
specifying a size in bytes. The expression, expr, can be:
- a positive decimal number
- a positive decimal number followed by k,
specifying multiplication by 1024
- a positive decimal number followed by b,
specifying multiplication by 512
- two or more positive decimal numbers (with or
without k or b) separated by x, specifying
the product of the indicated values.
All of the operands will be processed before any input
is read.
USAGE
See
largefile(5) for the description of the behavior of dd
when encountering files greater than or equal to 2 Gbyte ( 231
bytes).
EXAMPLES
Example 1 Copying from one tape drive to another
The following example copies from tape drive 0
to tape drive 1, using a common historical device naming convention.
example% dd if=/dev/rmt/0h of=/dev/rmt/1h
Example 2 Stripping the first 10 bytes from standard
input
The following example strips the first 10 bytes from
standard input:
example% dd ibs=10 skip=1
Example 3 Reading a tape into an ASCII file
This example reads an EBCDIC tape blocked ten 80-byte
EBCDIC card images per block into the ASCII file x:
example% dd if=/dev/tape of=x ibs=800 cbs=80 conv=ascii,lcase
Example 4 Using conv=sync to write to tape
The following example uses conv=sync when
writing to a tape:
example% tar cvf - . | compress
| dd obs=1024k of=/dev/rmt/0 conv=sync
ENVIRONMENT VARIABLES
See
environ(5) for descriptions of the following environment variables
that affect the execution of dd: LANG, LC_ALL,
LC_CTYPE, LC_MESSAGES, and NLSPATH.
EXIT STATUS
The following exit values are returned:
- 0
- The input file was copied successfully.
- >0
- An error occurred.
If an input error is detected and the noerror
conversion has not been specified, any partial output block will be
written to the output file, a diagnostic message will be written, and
the copy operation will be discontinued. If some other error is detected,
a diagnostic message will be written and the copy operation will be
discontinued.
SEE ALSO
DIAGNOSTICS
- f+p records in(out)
- numbers of full and partial blocks read(written)
NOTES
Do not use dd to copy files between file
systems having different block sizes.
Using a blocked device to copy a file will result
in extra nulls being added to the file to pad the final block to the
block boundary.
When dd reads from a pipe, using the
ibs=X and obs=Y operands, the output will always be
blocked in chunks of size Y. When bs=Z is used, the output
blocks will be whatever was available to be read from the pipe at the
time.
When using dd to copy files to a tape device,
the file size must be a multiple of the device sector size (for example,
512 Kbyte). To copy files of arbitrary size to a tape device, use
tar(1) or
cpio(1).
For SIGINT, dd writes status
information to standard error before exiting. It takes the standard
action for all other signals.
In Linux, the command is invoked as follows: bash# dd if=<image
file> of=/dev/fd0. For example
dd if=/dev/fd0 of=floppy.img bs=18k
where /dev/fd0 should be the device for your raw floppy drive (_not_
/dev/floppy) and floppy.img the file you want to save the info to. You can
then copy that file to somewhere you can read it with DOS, or maybe even
zip it so it will fit onto a floppy ;). You should see something like the
following to indicate that the image transfer was successful:
2880+0 records in
2880+0 records out |
If you see a smaller block count, your image did not transfer correctly.
If this is the case, it will usually be accompanied by a disk error.
After you make a disk, make sure to label it according to its contents.
The dd command transfers raw data between devices. For example,
dd can be used to create a boot floppy under Linux using the
following command:
dd if=boot.img of=/dev/fd0
bs=1440
where
if= is the input file or device
of= is the output file or device
bs= specifies the block size for both input and output
files
To change the input and output block sizes to different values, you could
use the arguments:
or
|
|
Notes:
- This is a Spartan WHYFF (We Help
You For Free) site written by people for whom English
is not a native language.
Some amount of grammar and spelling errors should be
expected.
- The site contain some broken links
as it develops like a living tree...
Please try to use Google, Open directory,
etc. to find a replacement link (see
HOWTO search the WEB for details). We would appreciate
if you can
mail us a correct link.
|
|
|
Feb 11, 2009 | unstableme.blogspot.com
"The command is:
dd if=/dev/zero of=testfile_10MB
bs=10485760 count=1
"1+0 records in
1+0 records out
10485760 bytes (10 MB) copied, 0.312 s, 33.6 MB/s"
Looks like dd was slightly faster with 128K blocks. Python program
might be reused for other purposes
Block Size: 128 Throughput: 62.8 MB/s
Block Size: 256 Throughput: 61.8 MB/s
Block Size: 512 Throughput: 57.1 MB/s
Block Size: 1024 Throughput: 56.5 MB/s
We benchmarked the throughput of the disk by running
the dd command with various block sizes from 128 KB
to 1 MB. (Note: If you want to run the script on your
own machine, make sure that the volume you use doesn’t
contain any valuable data, because the data will be
erased by the dd command. Remember, data loss makes
grandpappy mad!)
For the benchmark, we wrote a Python
script that uses the commands module to run and capture
the output of the dd command. The script also uses the
csv module to generate a comma-separated values file
so that we can graph the results later. For this example,
we chose to graph the results using the Google Chart
API.
dd_chart
on Flickr - Photo Sharing!
Automated Image & Restore (AIR) is an open source application that provides
a GUI front end to the dd/dcfldd (Dataset Definition (dd)) command.
AIR is designed to easily create forensic disk/partition images. It
supports MD5/SHAx hashes, SCSI tape drives, imaging over a TCP/IP network,
splitting images, and detailed session logging. To date, the AIR utility
has only been developed for use on Linux distributions. In its simplest
form, AIR provides a convenient interface to execute the dd set of commands.
It eliminates the risk of "fat fingering" an error in the shell terminal
and ultimately makes using the dd command more user-friendly for those
who are not as experienced. Please note that using the AIR front end
still requires some basic knowledge of how the dd (or dcfldd) commands
work.The dd command has been around for quite a while. It is well
known throughout the Unix/Linux community, well documented, and as I
can only imagine extensively used. A dd image is a bit by bit image
of a source device or file. The uses for dd range from creating and
maintaining system backups and restore images to the forensic application
of imaging evidence that will be returned to the lab and examined.
This tutorial is not designed to teach the use of the dd command;
this is well documented and a simple internet search will yield a plethora
of results. Instead, the intent of this mini "how-to" is to introduce
users to the AIR front end application, increase overall awareness of
the utility, and provide a brief example of creating a dd image using
this tool.
DISCLAIMER: I do not claim to be expert at using dd or Automated
Image & Restore.
dd_rescue Advantages
Imagine, one of your partitions is crashed,
and as there are some hard errors, you don’t
want to write to this hard disk any more.
Just getting all the data off it and
retiring it seems to be suitable. However,
you can’t access the files, as the
file
system is damaged.
Now, you want to copy the whole partition
into a file. You burn it on
CD-Rom, just to never lose
it again. You can setup a loop device, and
repair (fsck) it and hopefully are able to
mount it.
Copying this partition with normal Un*x
tools like cat or dd will fail, as those
tools abort on error. dd_rescue instead will
try to read and if it fails, it will go on
with the next sectors. The output file
naturally will have holes in it, of course.
You can write a log file, to see, where all
these errors are located.
The data rate drops very low, when errors
are encountered. If you interrupt the
process of copying, you don’t lose anything.
You can just continue at any position later.
The output file will just be filled in
further and not truncated as with other Un*x
tools.
If you have one spot of bad sectors
within the partition, it might be a good
idea, to approach this spot from both sides.
Reverse direction copy is your friend.
The two block sizes are a
performance
optimization. Large block
sizes result in superior
performance, but in case
of errors, you want to try to salvage every
single sector. So hardbs is best be set to
the
hardware sector size (most
often 512 bytes) and softbs to a large
value, such as the default 16k.
Install dd_rescue in Debian
Install ddrescue using the following
command
#apt-get install ddrescue
Install ddrescue in Ubuntu
sudo apt-get install ddrescue
This will complete the installation
ddrescue Syntax
dd_rescue [options] infile outfile
Now we will see how to use ddrescue under
damaged disk
If you have a damaged hard disk /dev/sda1
and you have an empty space hard disk
/dev/sda2 Now if you want to copy data from
/dev/sda1 to /dev/sda2 use the following
commnd
# dd_rescue /dev/sda1
/dev/sda2/backup.img
If you are using ubuntu
linux use the following
command
sudo dd_rescue /dev/sda1
/dev/sda2/backup.img
This copies an image of /dev/sda1 to sda2
Now you need to check the backup image
consistency this will check for is there any
problems with this image.
#fsck -y /dev/sda2/backup.img
If you are using ubuntu linux use the
following command
sudo fsck -y /dev/sda2/backup.img
After finishing this checking you need to
mount your disk image in to your other hard
disk
#mount /dev/sda2/backup.img
/mnt/recoverydata
If you are using ubuntu linux use the
following command
sudo mount /dev/sda2/backup.img
/mnt/recoverydata
This will mount all the data from the
backup.img under /mnt/recoverydata now you
can try to access the data it should work
without any problem.
Restore image
If you want to restore this image use the
following command
#dd_rescue /dev/sda2/backup.img /dev/sda1
If you are using ubuntu linux use the
following command
sudo dd_rescue /dev/sda2/backup.img
/dev/sda1
Copy Disk Image to remote machine
using SSH
If you want to copy your disk image to
remote machine over ssh you need to use the
following command
#dd_rescue /dev/sda1 - | ssh
username@machineip ‘cat
/datarecovery/backup.img’
If you are using ubuntu linux use the
following command
sudo dd_rescue /dev/sda1 - | ssh
username@machineip ‘cat
/datarecovery/backup.img’
This will be prompetd for password of the
username you have menctioned in the above
command after entering the password
dd_rescue strats copying obviously it will
take some time to copy over the network.
Possible Error
If you see the following error at the
time of copying you can ignore this error
dd_rescue: (warning): output file is not
seekable!
dd_rescue: (warning): Illegal seek
If you want to take this image in
compressed format you can use the following
command format
#tar zcvf - /dev/sda1 | ssh
username@machineip ‘cat@@>/tmp
/datarecovery/backup.tar.gz’
If you are using ubuntu linux use the
following command
sudo tar zcvf - /dev/sda1 | ssh
username@machineip ‘cat@@>/tmp
/datarecovery/backup.tar.gz’
If you want to know more available
options check dd_rescue
man page
Like dd, dd_rescue does copy data from one file or block device
to another.dd_rescue is a tool to help you to save data from crashed
partition. It tries to read and if it fails, it will go on with the
next sectors where tools like dd will fail. If the copying process
is interrupted by the
user it is possible to continue at any position
later. It can copy backwards.
Creating a hard drive backup directly to another hard drive
# dd if=/dev/hda of=/dev/sda conv=noerror,sync bs=4k
This command is used often to create a backup of a drive (/dev/hda)
directly to another hard drive (/dev/sda). (The device name /dev/hda
is typical of an IDE hard drive, the device /dev/sda is typical of a
USB disk.) This works only if the hard drive has enough storage to accommodate
the source drive's filesystem. The advantage of this is that you do
not have to mount the hard drive to make a backup and the only reference
to hda is in /dev and in the command which is usually in a script in
cron.
The option "bs=4k" is used to specify the block size used in the
copy. The default for the dd command is 512 bytes: use of this small
block size can result in significantly slower copying. However, the
tradeoff with larger block sizes is that when an error is encountered,
the remainder of the block is filled with zero-bytes. So if you increase
your block size when copying a failing device, you'll lose more data
but also spend less time trying to read broken sectors. Tools like
dd_rescue and
dd_rhelp can provide a more flexible solution in such cases, combining
the speed of a large block size for the regions without errors with
finer-grained block-copies for regions with errors.
Creating a hard drive backup image
# dd if=/dev/hda | gzip > /mnt/hdb1/system_drive_backup.img.gz
Here dd is making an image of the first harddrive, and piping
it through the
gzip
compression program. The compressed image is then placed in a file on
a seperate drive. To reverse the process:
# gzip -dc /mnt/hdb1/system_drive_backup.img.gz | dd of=/dev/hda
Here, gzip is decompressing (the -d switch) the file, sending the
results to stdout (the -c switch), which are piped to dd, and then written
to /dev/hda.
Copy floppy
# dd if=/dev/fd0 of=/tmp/floppy.img bs=10240
That will copy the contents of the floppy to a file. Then, to put
the image onto a new floppy, swap "if" and "of" params.
# dd if=/tmp/floppy.img of=/dev/fd0 bs=10240
Backing up your Master Boot Record (MBR).
You should do this before you edit your partition table so that you
can put it back if you mess things up.
# dd if=/dev/hda of=/root/hda.boot.mbr bs=512 count=1
If things mess up, you can boot with
Knoppix, mount the partition containing /root (hda1 in this example)
and put back the MBR with the command:
# dd if=/mnt/hda1/root/hda.boot.mbr of=/dev/hda bs=512 count=1
Obviously, if you have a
GPT
system (like the intel mac for instance) this will need some adjustment.
see:
http://forum.onmac.net/showthread.php?t=136
You can backup only the MBR and exclude the partition table with
the command:
# dd if=/dev/hda of=/root/hda.mbr.noparttab bs=446 count=1
Getting around file size limitations
using split
When making images, it's quite easy to run up against various file
size limitations. One way to work around a given file size limitation
is to use the
split command.
# dd if=/dev/hda1 | gzip -c | split -b 2000m - /mnt/hdc1/backup.img.gz.
- This example is using dd to take an image of the first partition
on the first harddrive.
- The results are passed through to
gzip for compression
- The -c option switch is used to output the result to
stdout.
- The compressed image is then piped to the
split tool
- The -b 2000m switch tells split how big to make the individual
files. You can use k and m to tell switch kilobytes and megabytes
(this option uses bytes by default).
- The - option tells split to read from
stdin. Otherwise, split would interpret the /mnt/hdc1...
as the file to be split.
- The /mnt/hdc1... is the prefix for the created files. Split
will create files named backup.img.gz.aa, backup.img.gz.ab,
etc.
To restore the multi-file backup, do the following:
# cat /mnt/hdc1/backup.img.gz.* | gzip -dc | dd of=/dev/hda1
- Cat recombines contents of the compressed and split image files
to
stdout, in order.
- Results are piped through gzip for decompression.
- And are then written to the first partition of the hard drive
with dd.
Creating empty disk images
To create an empty disk image, to be used as the disk for an emulator
for example, one can get data from /dev/zero. To create a 10mb image:
$ dd if=/dev/zero of=myimage bs=1024 count=10240
A clever alternative is:
$ dd of=myimage bs=1024 count=0 seek=10240
Here we don't write anything, not even zeroes, we just seek 10mb
into the file and close it. The result is a sparse file that is implicitly
full of 10mb of zeroes, but that takes no disk space.
ls -l
will report 10mb, while
du
and
df will report 0. When the file is written to, either as an
emulator disk or a loopback device, Linux will allocate disk space for
the data. ls will still show 10mb, while du will gradually
approach 10mb.
For swap images, where it's more important to reserve the data than
to save disk space, a non-sparse file is better.
this article,
Sam Chessman explains the use of the dd command with a lot of
useful examples. This article is not aimed at absolute beginners. Once
you are familiar with the basics of Linux, you would be in a better
position to use the dd command.
The ' dd
' command is one of the original Unix utilities and should be in everyone's
tool box. It can strip headers, extract parts of binary files and write
into the middle of floppy disks; it is used by the Linux kernel Makefiles
to make boot images. It can be used to copy and convert magnetic tape
formats, convert between ASCII and EBCDIC, swap bytes, and force to
upper and lowercase.
For
blocked I/O, the dd command has no competition in the standard tool
set. One could write a custom utility to do specific I/O or formatting
but, as dd is already available almost everywhere, it makes sense to
use it.
Like
most well-behaved commands, dd reads from its standard input and writes
to its standard output, unless a command line specification has been
given. This allows dd to be used in pipes, and remotely with the rsh
remote shell command.
Unlike
most commands, dd uses a keyword=value format for its parameters. This
was reputedly modeled after IBM System/360 JCL, which had an elaborate
DD 'Dataset Definition' specification for I/O devices. A complete listing
of all keywords is available from GNU dd with
$
dd --help
Some
people believe dd means ``Destroy Disk'' or ``Delete Data'' because
if it is misused, a partition or output file can be trashed very quickly.
Since dd is the tool used to write disk headers, boot records, and similar
system data areas, misuse of dd has probably trashed many hard disks
and file systems.
In essence,
dd copies and optionally converts data. It uses an input buffer, conversion
buffer if conversion is specified, and an output buffer. Reads are issued
to the input file or device for the size of the input buffer, optional
conversions are applied, and writes are issued for the size of the output
buffer. This allows I/O requests to be tailored to the requirements
of a task. Output to standard error reports the number of full and short
blocks read and written.
Example
1
A typical
task for dd is copying a floppy disk. As the common geometry of a 3.5"
floppy is 18 sectors per track, two heads and 80 cylinders, an optimized
dd command to read a floppy is:
Example
1-a : Copying from a 3.5" floppy
dd bs=2x80x18b if=/dev/fd0 of=/tmp/floppy.image
1+0 records in
1+0 records out
The
18b specifies 18 sectors of 512 bytes, the 2x multiplies the sector
size by the number of heads, and the 80x is for the cylinders--a total
of 1474560 bytes. This issues a single 1474560-byte read request to
/dev/fd0 and a single 1474560 write request to /tmp/floppy.image, whereas
a corresponding cp command
cp /dev/fd0
/tmp/floppy.image
issues
360 reads and writes of 4096 bytes. While this may seem insignificant
on a 1.44MB file, when larger amounts of data are involved, reducing
the number of system calls and improving performance can be significant.
This
example also shows the factor capability in the GNU dd number specification.
This has been around since before the Programmers Work Bench and, while
not documented in the GNU dd man page, is present in the source and
works just fine, thank you.
To finish
copying a floppy, the original needs to be ejected, a new diskette inserted,
and another dd command issued to write to the diskette:
Example
1-b : Copying to a 3.5" floppy
dd bs=2x80x18b < /tmp/floppy.image > /dev/fd0
1+0 records in
1+0 records out
Here
is shown the stdin/stdout usage, in which respect dd is like most other
utilities.
Example
2
The
original need for dd came with the 1/2" tapes used to exchange data
with other systems and boot and install Unix on the PDP/11. Those days
are gone, but the 9-track format lives. To access the venerable 9-track,
1/2" tape, dd is superior. With modern SCSI tape devices, blocking and
unblocking are no longer a necessity, as the hardware reads and writes
512-byte data blocks.
However,
the 9-track 1/2" tape format allows for variable length blocking and
can be impossible to read with the cp command. The dd command allows
for the exact specification of input and output block sizes, and can
even read variable length block sizes, by specifying an input buffer
size larger than any of the blocks on the tape. Short blocks are read,
and dd happily copies those to the output file without complaint, simply
reporting on the number of complete and short blocks encountered.
Then
there are the EBCDIC datasets transferred from such systems as MVS,
which are almost always 80-character blank-padded Hollerith Card Images!
No problem for dd, which will convert these to newline-terminated variable
record length ASCII. Making the format is just as easy and dd again
is the right tool for the job.
Example
2 : Converting EBCDIC 80-character fixed-length record to ASCII variable-length
newline-terminated record
dd bs=10240 cbs=80 conv=ascii,unblock if=/dev/st0 of=ascii.out
40+0 records in
38+1 records out
The fixed record length is specified by the cbs=80 parameter, and the
input and output block sizes are set with bs=10240. The EBCDIC-to-ASCII
conversion and fixed-to-variable record length conversion are enabled
with the conv=ascii,noblock parameter.
Notice
the output record count is smaller than the input record count. This
is due to the padding spaces eliminated from the output file and replaced
with newline characters.
Example
3
Sometimes
data arrives from sources in unusual formats. For example, every time
I read a tape made on an SGI machine, the bytes are swapped. The dd
command takes this in stride, swapping the bytes as required. The ability
to use dd in a pipe with rsh means that the tape device on any *nix
system is accessible, given the proper rlogin setup.
Example
3 : Byte Swapping with Remote Access of Magnet Tape
rsh sgi.with.tape dd bs=256b if=/dev/rmt0 conv=swab | tar xvf -
The dd runs on the SGI and swaps the bytes before writing to the tar
command running on the local host.
Example
4
Murphy's
Law was postulated long before digital computers, but it seems it was
specifically targeted for them. When you need to read a floppy or tape,
it is the only copy in the universe and you have a deadline past due,
that is when you will have a bad spot on the magnetic media, and your
data will be unreadable. To the rescue comes dd, which can read all
the good data around the bad spot and continue after the error is encountered.
Sometimes this is all that is needed to recover the important data.
Example
4 : Error Handling
dd bs=265b conv=noerror if=/dev/st0 of=/tmp/bad.tape.image
Example
5
The
Linux kernel Makefiles use dd to build the boot image. In the Alpha
Makefile /usr/src/linux/arch/alpha/boot/Makefile, the srmboot target
issues the command:
Example
5 : Kernel Image Makefile
dd if=bootimage of=$(BOOTDEV) bs=512 seek=1 skip=1
This
skips the first 512 bytes of the input bootimage file (skip=1) and writes
starting at the second sector of the $(BOOTDEV) device (seek=1). A typical
use of dd is to skip executable headers and begin writing in the middle
of a device, skipping volume and partition data. As this can cause your
disk to lose file system data, please test and use these applications
with care.
December 19, 2006 (debianadmin.com) by Admin
@ 4:59 pm. Filed under
Backup
dd_rhelp is a bash script that handles a very usefull program written
in C which is called dd_rescue, it roughly act as the dd linux command
with the characteristic to NOT stop when it falls on read/write errors.
This makes dd_rescue the best tool for recovering
hard drive having bad sectors. In short, it’ll use dd_rescue on your
entire disc, but will try to gather the maximum valid data before trying
for ages on badsectors. So if you leave dd_rhelp work for infinite time,
it’ll have the same effect as a simple dd_rescue. But because you might
not have this infinite time , dd_rhelp will jump over bad sectors and
rescue valid data. In the long run, it’ll parse all your device with
dd_rescue.
You can Ctrl-C it whenever you want, and rerun-it at will, it’ll
resume it’s job as it depends on the log files dd_rescue creates.In
addition, progress will be shown in a ASCII picture of your device beeing
rescued.
Why use dd_rhelp and not dd_rescue ?
dd_rhelp uses dd_rescue to compute a recovery path through the device
that will focus on valid data recovering. This recovery path will go
through all the device, exactly as dd_rescue could do it on its own
without any path. This means that dd_rhelp will save you time ONLY IF
YOU INTEND TO CANCEL ITS JOB BEFORE THE END of a full recovery.
Because a considerable amount of time is taken to try to rescue badsectors.
This amount of time can be mesured in days,
month, years, depending on your device capacity and its defectiveness.
You might not want to spend this time knowing that 99 percent of this
time will be taken to look at badsector and won’t lead to any more data
recovering.
dd_rhelp shifts this useless waiting time to the end. Using dd_rescue
strait throughout your device make your waiting time dependent on the
badsector distribution.
Install dd_rhelp
First you need to download latest version from
here
Preparing your system
#apt-get install make gcc g++ ddrescue
If you are using ubuntu linux use the following command
sudo apt-get install make gcc g++ ddrescue
Download dd_rhelp using the follwoing command
wget http://www.kalysto.org/pkg/dd_rhelp-0.0.6.tar.gz
Now you have dd_rhelp-0.0.6.tar.gz file
Extract this file using the following command
tar xzvf dd_rhelp-0.0.6.tar.gz
cd dd_rhelp-0.0.6
Run the following commands
# ./configure
# make
# make install
If you are using ubuntu linux use the following command
sudo ./configure
sudo make
sudo make install
This will complete the installation and this will install program
under /usr/local/bin/dd_rhelp
dd_rhelp Syntax
dd_rhelp {filename|device} {output-file} [{info}]
or dd_rhelp --help
or dd_rhelp --version
Now we will see how to use ddrescue under damaged disk
If you have a damaged hard disk /dev/sda1 and you have an empty space
hard disk /dev/sda2 Now if you want to copy data from /dev/sda1 to /dev/sda2
use the following command
#dd_rhelp /dev/sda1 /dev/sda2/backup.img
If you are using ubuntu linux use the following command
sudo dd_rhelp /dev/sda1 /dev/sda2/backup.img
Output looks like below
/usr/bin/tail: Warning: “+number” syntax is deprecated, please use
“-n +number”
=== launched via ‘dd_rhelp’ at 0k, 0 >>> ===
dd_rescue: (info): ipos: 104384.0k, opos: 104384.0k, xferd: 104384.0k
errs: 0, errxfer: 0.0k, succxfer: 104384.0k
+curr.rate: 10178kB/s, avg.rate: 8532kB/s, avg.load: 31.3%
dd_rescue: (info): /dev/sda2 (104391.0k): EOF
Summary for /dev/sda2 -> backup.img:
dd_rescue: (info): ipos: 104391.0k, opos: 104391.0k, xferd: 104391.0k
errs: 0, errxfer: 0.0k, succxfer: 104391.0k
+curr.rate: 355kB/s, avg.rate: 8519kB/s, avg.load: 31.3%
/usr/bin/tail: Warning: “+number” syntax is deprecated, please use “-n
+number”
This copies an image of /dev/sda1 to sda2
Now you need to check the backup image consistency this will check
for is there any problems with this image.
#fsck -y /dev/sda2/backup.img
If you are using ubuntu linux use the following command
sudo fsck -y /dev/sda2/backup.img
After finishing this checking you need to mount your disk image in
to your other hard disk
#mount /dev/sda2/backup.img /mnt/recoverydata
If you are using ubuntu linux use the following command
sudo mount /dev/sda2/backup.img /mnt/recoverydata
This will mount all the data from the backup.img under /mnt/recoverydata
now you can try to access the data it should
work without any problem.
Restore image
If you want to restore this image use the following command
#dd_rhelp /dev/sda2/backup.img /dev/sda1
If you are using ubuntu linux use the following command
sudo dd_rhelp /dev/sda2/backup.img /dev/sda1
Copy Disk Image to remote machine using SSH
If you want to copy your disk image to remote machine over ssh you
need to use the following command
#dd_rhelp /dev/sda1 - | ssh username@machineip ‘cat /datarecovery/backup.img’
If you are using ubuntu linux use the following command
sudo dd_rhelp /dev/sda1 - | ssh username@machineip ‘cat /datarecovery/backup.img’
This will be prompetd for password of the username you have menctioned
in the above command after entering the password
dd_rhelp strats copying obviously it will take some time to copy
over the network.
Example 1
A typical task for dd is copying a floppy disk. As the common geometry
of a 3.5" floppy is 18 sectors per track, two heads and 80 cylinders,
an optimized dd command to read a floppy is:
Example 1-a : Copying from a 3.5" floppy
dd bs=2x80x18b if=/dev/fd0 of=/tmp/floppy.image
1+0 records in
1+0 records out
The 18b specifies 18 sectors of 512 bytes, the 2x multiplies the sector
size by the number of heads, and the 80x is for the cylinders--a total
of 1474560 bytes. This issues a single 1474560-byte read request to
/dev/fd0 and a single 1474560 write request to /tmp/floppy.image, whereas
a corresponding cp command
cp /dev/fd0 /tmp/floppy.image
issues 360 reads and writes of 4096 bytes. While this may seem insignificant
on a 1.44MB file, when larger amounts of data are involved, reducing
the number of system calls and improving performance can be significant.
This example also shows the factor capability in the GNU dd number specification.
This has been around since before the Programmers Work Bench and, while
not documented in the GNU dd man page, is present in the source and
works just fine, thank you.
To finish copying a floppy, the original needs to be ejected, a new
diskette inserted, and another dd command issued to write to the diskette:
Example 1-b : Copying to a 3.5" floppy
dd bs=2x80x18b < /tmp/floppy.image
> /dev/fd0
1+0 records in
1+0 records out
Here is shown the stdin/stdout usage, in which respect dd is like most
other utilities.
Example 2
The original need for dd came with the 1/2" tapes used to exchange data
with other systems and boot and install Unix on the PDP/11. Those days
are gone, but the 9-track format lives. To access the venerable 9-track,
1/2" tape, dd is superior. With modern SCSI tape devices, blocking and
unblocking are no longer a necessity, as the hardware reads and writes
512-byte data blocks.
However, the 9-track 1/2" tape format allows for variable length blocking
and can be impossible to read with the cp command. The dd command allows
for the exact specification of input and output block sizes, and can
even read variable length block sizes, by specifying an input buffer
size larger than any of the blocks on the tape. Short blocks are read,
and dd happily copies those to the output file without complaint, simply
reporting on the number of complete and short blocks encountered.
Then there are the EBCDIC datasets transferred from such systems as
MVS, which are almost always 80-character blank-padded Hollerith Card
Images! No problem for dd, which will convert these to newline-terminated
variable record length ASCII. Making the format is just as easy and
dd again is the right tool for the job.
Example 2 : Converting EBCDIC 80-character fixed-length record to ASCII
variable-length newline-terminated record
dd bs=10240 cbs=80 conv=ascii,unblock
if=/dev/st0 of=ascii.out
40+0 records in
38+1 records out
The fixed record length is specified by the cbs=80 parameter, and the
input and output block sizes are set with bs=10240. The EBCDIC-to-ASCII
conversion and fixed-to-variable record length conversion are enabled
with the conv=ascii,noblock parameter.
Notice the output record count is smaller than the input record count.
This is due to the padding spaces eliminated from the output file and
replaced with newline characters.
Example 3
Sometimes data arrives from sources in unusual formats. For example,
every time I read a tape made on an SGI machine, the bytes are swapped.
The dd command takes this in stride, swapping the bytes as required.
The ability to use dd in a pipe with rsh means that the tape device
on any *nix system is accessible, given the proper rlogin setup.
Example 3 : Byte Swapping with Remote Access of Magnet Tape
rsh sgi.with.tape dd bs=256b if=/dev/rmt0
conv=swab | tar xvf -
The dd runs on the SGI and swaps the bytes before writing to the tar
command running on the local host.
Example 4
Murphy's Law was postulated long before digital computers, but it seems
it was specifically targeted for them. When you need to read a floppy
or tape, it is the only copy in the universe and you have a deadline
past due, that is when you will have a bad spot on the magnetic media,
and your data will be unreadable. To the rescue comes dd, which can
read all the good data around the bad spot and continue after the error
is encountered. Sometimes this is all that is needed to recover the
important data.
Example 4 : Error Handling
dd bs=265b conv=noerror if=/dev/st0
of=/tmp/bad.tape.image
Example 5
The Linux kernel Makefiles use dd to build the boot image. In the Alpha
Makefile /usr/src/linux/arch/alpha/boot/Makefile, the srmboot target
issues the command:
Example 5 : Kernel Image Makefile
dd if=bootimage of=$(BOOTDEV)
bs=512 seek=1 skip=1
This skips the first 512 bytes of the input bootimage file (skip=1)
and writes starting at the second sector of the $(BOOTDEV) device (seek=1).
A typical use of dd is to skip executable headers and begin writing
in the middle of a device, skipping volume and partition data. As this
can cause your disk to lose file system data, please test and use these
applications with care.BigAdmin
Submitted Article Backup and Recovery Utilities for the Solaris OS
dd -- Convert and Copy a File
The dd command is most commonly used to
copy a complete file system to another file system or to copy a hard
disk drive to another disk drive. dd can
also be used to copy a file system to tape, and vice versa.
dd is a relatively quick copy tool: It creates
an exact copy (byte for byte) as it transfers the data. Several options
are used with dd to specify buffer sizes,
block sizes, and data conversions. The basic syntax for
dd is:
dd <inputfile>= <outputfile>= options
The following example copies the entire contents of c0t1d0s2 to a
second disk, c0t4d0s2, using a block size of 128. This works great if
you have a spare disk available and want to have a backup disk ready
to swap out in case of a disaster. Slice 2 is specified in this example
because it represents the entire disk in the Solaris OS.
% dd if=/dev/rdsk/c0t1d0s2 of=/dev/rdsk/c0t4d0s2 bs=128
To copy the contents of one tape device to another, use the following
example. (Note: You must have two drives available.)
% dd if=/dev/rmt/#1drive of=/dev/rmt/#2drive
The man pages on dd give many other options
available for use. I suggest viewing the man pages for
dd and also reading the Notes section. There
you will find some limitations to dd, as
well as a reminder to use the raw character device when copying data
from disk devices.
For quick review, let's
create an image based on the following:
We have a 2GB hard disk we want to image. We connect
the 2GB disk as a slave in our PC. Upon boot, we mount the 2GB disk
read only. We are now ready to image! Using DD, how 'bout:
dd if=/dev/hdb of=/images/hdbApril42001.img
dd
does low-level data transfer, byte-by-byte or block-by-block, with adjustable
block sizes. It can also skip specified numbers of blocks in the input
and/or output files, as well as converting data formats. All of those
are handy for working with magnetic tape and disks. But it's also useful
for many types of data transfers.
By default, dd reads the
standard input and writes to the standard output. Input
and output filenames, and other options too, are given in an unusual
syntax without leading dash (-)
characters.
For instance, to read a floppy disk and
write its image to a file, you could type:
$ dd if=/dev/fd0 of=dosboot.img
2880+0 records in
2880+0 records out
$ ls -l dosboot.img
-rw-rw-r- ... 1474560 Nov 2 12:59 dosboot.img
The dd command line says,
"Reading from the input file /dev/fd0, write all of the data
to the file dosboot.img." dd doesn't try to find lines
of data or individual files on the disk; it does a binary copy of the
bytes from first to last. dd always tells you (on the standard
error) how many times it read and wrote data. Above, it read 2,880
512-byte blocks. If you don't want to see this information -- or any
error messages, either -- you can redirect dd's standard error
to the Linux "bit bucket," /dev/null, by adding the Bourne shell
operator 2>/dev/ null
to the command line.
It's more efficient to specify
a larger block size so the device drivers do a single read and write.
There are lots of other options, and many of them start with
conv=, like
conv=unblock
to replace trailing spaces in a block with a newline, and
conv= swap to
swap pairs of input bytes (which is needed with some tapes written on
other types of hardware). But we'll leave that sort of optimization
to you and the dd man page. Let's look at some less-obvious uses
of this handy utility.
Stupid dd Tricks
Need a file with 100 arbitrary
bytes -- for testing, for instance? The Linux device /dev/urandom
(available since Linux 1.3.30) can supply as many pseudo-random
bytes as you can read from it. To get just 100 bytes, set a block size
of 1 byte with bs=1
and tell dd to stop after copying 100 "blocks" (here, that's
100 bytes):
$ dd if=/dev/urandom of=myrand bs=1 count=100
What's in that myrand file? The
od utility can show you. (See the sidebar "What's In That File?")
If you need more-random data, try
/dev/random instead. Reading data from /dev/random can take
some time, though, as the random(4) man page explains. When you
read from /dev /random, set a block size of 1.
Another use for dd is for
"wiping" a text file before you delete it. Simply removing a Linux file
(with rm, for instance) only deletes the inode that points to
the data. A cracker with root access might read the raw disk
(with dd!) and find the "deleted" file. We can use dd
to write random data over the file before deleting it. Normally dd
truncates a file before writing, so use
conv=notrunc to make
it write over the existing data. Set bs
to the file size and count
to 1. For
example:
% ls -l afile
-rw------- ... 3769 Nov 2 13:41 afile
% dd if=/dev/urandom of=afile \
bs=3769 count=1 conv=notrunc
1+0 records in
1+0 records out
% rm afile
If you want to, you can repeat
the "wiping" command several times with the C shell repeat command,
the Z shell repeat loop, or simply use the history operator
!!.
The dd command
can be used to put data on a disk, or get it off again,
depending on the given input and output devices. An example:
gaby:~>dd if=images-without-dir.tar.gz of=/dev/fd0H1440
98+1 records in
98+1 records out
gaby~>dd if=/dev/fd0H1440 of=/var/tmp/images.tar.gz
2880+0 records in
2880+0 records out
gaby:~>ls /var/tmp/images*
/var/tmp/images.tar.gz
|
Note that the dumping is done on an unmounted
device. Floppies created using this method will not be mountable
in the file system, but it is of course the way to go for
creating boot or rescue disks. For more information on the
possibilities of dd, read the man
pages.
This tool is part of the GNU fileutils
package.
One would think this would be a simple task.
Half the commerical packages proclaim that they can do it. But
alas, for me, none of them worked.
The task was to create a system installer that would boot a sun, load
up the appropriate system images and tools required for a server in
our environment. Yes, this is a perfect job for jumpstart, but
the systems would be distributed across the state and not connected
via a fast enough pipe to use jumpstart effectivly. The other
option was to build external disks with the proper images on them and
boot from there. The cost of producing a few hundrad of these
was prohibitive, thus the CDROM approach was taken.
Of the few responses I received from Sun Managers readers, all of them
basicly said 'Its easy to clone the install disc!' or 'it just cant
be done!'
Well. It is possible.
First, a breakdown of the process. When the command > boot cdrom
< is givin to openboot the system actually looks at slice 2 to 5 for
its bootblks depending on the machine type. A sun4c is slice 2
to an sun4u at slice 5. These bootblks redirect the system to
slice 1 to load its kernel. The root partition is also stored
in slice 1 and slice 0 is usr as well as packages. All slices save 0
are ufs, and 0 is hsfs.
This brings us to a small problem point.
First we must boot from a UFS partition, as the bootblks all require
that. Second, a CDROM has no label by default, thus its kinda
hard to make partitions. And third, even if we do flush a UFS filesystem
image off to the cdrom, the geometries will be all wrong, unless your
staging disk just happens to have the same
geometries as the CDROM.
My solution is probally not the best. What would be required is
an application that simply converts the UFS geometries from the staging
disk uses to those used by the cdrom. I didnt really have time
to create this so I did it as follows. Im sure I will get lots
of flak for this
solution, but it does work.
1) Use dd to grab the first cylinder off the solaris boot cdrom.
This contains a valid disk label and VTOC for the cdrom. Once
this is created, our limitation is that we must work within the defines
of this VTOC. You should be able to use prtvtoc on the cdrom to get
a look at this VTOC, but this dosnt work if volmgr is running.
dd if=/dev/dsk/c0t6d0s0 of=cdrom.vtoc bs=512 count=1
2) Now use dd to grab the UFS slices from 1 to 5
for slice in 1 2 3 4 5
do
dd if=/dev/dsk/c0t6d0${slice} of=cdrom.s${slice}
done
3) Create a staging area and copy the parts of the usr filesystem (slice
0) off the cdrom into it. I started by copying cdrom:/export to
it and then trimmed out the parts I didnt need like X and openwindows.
4) Add in all the things you need for your disc. For me, this
ment a shell script that automated the build process, and images of
all the data I wanted to move out. Make sure you donot go beyond
te size of the slice 0 on the cdrom you started with. prtvtoc
will should you this, if you cant get prtvtoc to work on the cdrom (sometimes
it does, sometimes it dosnt) then use
> dd if=/dev/dsk/c0t6d0s0 of=/dev/null bs=512
<
Recordthe exact size of the partition in
blocks as you will need it later.
5) Patch the slice 1 image (cdrom.s1) to start your custom application
rather then the suninstall. This can be done by finding the break
point you wish to use in the file cdrom:/sbin/sysconfig, selecting a
unique set of chars in this file ( I used the string #***** S30sysid.net
) and then searching via a hexeditor or emacs in bin mode for that string.
Then find a comment line, change the first
# and chars after to point to your script, then add a # line after.
Example, I patched my image so the line
#***** S30sysid.net
became
exec build #ysid.net
I then put a script in my staging area into the /usr/bin dir. (the staging
area will be made into slice 0 which is hsfs)
6) Using mkisofs or the tools that came with your cdrom burning package
(HyCD worked well, as well as Gear) turn your staging area into a hsfs
filesystem image, making sure that symbloic links are unmodified. HyCD
required changing a default option that would have modified all the
links.
7) Using dd, throw away the first block of this image.
dd if=image of=image.data bs=512 skip=1
8) Subtract the block count of slice 0 from the solaris cdrom from the
image size above, add one to the answer. Say dd reported for the
above step that your image was 500000 blocks, and your solaris cdrom
has a size of 787840 for slice 0
787840 - ( 500000 + 1 ) = 287839
9) Feed this number into dd reading from /dev/zero to build a pad file.
dd if=/dev/zero of=image.pad bs=512 count=287839
10) Cat all of the image files together with the VTOC and the UFS slices.
cat cdrom.vtoc > image
cat image.data >> image
cat image.pad >> image
cat cdrom.s1 >> image
..
cat cdrom.s5 >> image
11) Burn this image to the cdrom drive using cdrecord, HyCD, Gear, etc
12) put it in a machine and test boot it and make sure it does what
you need.
Thats about it. We have to go through the gyrations because UFS
is geometry sensitive. We cannot take a image of a hard disk built
UFS as all the cylinder groups will be off. The RIGHT way of doing this
would be to build a tool that did the conversion for you and built an
image up for burning. But that requires a bit more work, this
gets the job done with a
minimal amount of strain. I am working on the above mentioned
tool, but so far my progress has been to create lots of coasters. Sun
has a tool called MakeDisc that does this job, or something similar,
but I do not have a copy of it, so had to develop a method, while under
a big gun, to do it in a very compressed amount of time. The only
real limitation this
has is you can only store about 400 megs of information, of which around
70 or so are needed by usr in the hsfs partition. Plus you do
not have to go through the pain of figuring out which parts of the system
need to be moved to the memfs filesystem (cdrom is readonly, so dev,
devices, etc need to be linked to /tmp)
If you wish to flame the procedure and tell me that I did it really
stupid, please correct me! I would willing stop development of
my tools to do same and do it the right way :) But my original
question on how to do this went unanswered.
Have Fun
James
In case of broken links
please try to use Google search. If you find the page please notify
us about new location
dd
The name dd stands for "copy and convert".
Don't see it? Well, "cc" was already taken for the C compiler, so the
author chose the next letter in the alphabet. The syntax has sort of
an evil, JCL-like quality to it. According to
The Jargon File, the interface was a prank.
Using dd
Most people use dd incorrectly. This
is because dd is a piece of junk that should be replaced.
For example, one common misusage of dd
is to try and get 64k blocks written to the tape with this command:
tar -cf - args... | dd of=/dev/rmt8 bs=64k
This won't work because (as you will
see below), the bs argument gives you only one buffer. The dd process
will attempt to read 64k chunks from the pipe into this buffer, but
will only receive a maximum of PIPE_BUF bytes (usually 4 or 8k). It
will then write this buffer out to the tape as a single record (it will
not pad this block to 64k, fortunately).
GNU dd
This data taken from GNU fileutils 3.12
When dd starts up, it parses all the
arguments on the command line in order. Note that the bs= argument will
override any previous ibs= or obs= arguments. If neither the obs nor
the ibs argument is presented, and bs is given, and no character-translation
conversions are performed, then only one buffer will be used (more on
this later). In all other cases, two buffers (input and output) are
used. If you don't specify any *bs args, ibs and obs default to 512.
Next, based on the translations that
you have specified, dd builds a translation table. This table is a 256
entry array, specifying a character-by-character mapping that is the
composite of all specified translations. The actual order of application
of translations is not the same as what is on the command line. It is:
- ebcidic_to_ascii
- lower_to_upper
- upper_to_lower
- ascii_to_ebcidic
- ascii_to_ibm
Note that not all conversions can be
specified at once. You have your choice of only one conv in {ascii,ebcdic,ibm},
{lcase,ucase}, {block,unblock}, {unblock,sync}.
Finally, dd enters the copy stage. It
allocates enough room for the input buffer, and if using a two-buffer
scenario, allocates an output buffer as well. It performs any skips
on the input, then performs any seeks on the output.
The main loop of the copy stage occurs
now. It attempts to read input_blocksize characters into the input buffer.
Errors here may be trapped, depending on command line options. If a
full input block is not read (for example, when reading from a communication
line, the end of a file, a pipe or special file, especially tapes),
the partial block count is incremented. If the sync option is in effect,
partial input blocks are NUL padded and treated as full input blocks.
At this point, if we are single-buffering,
we write the block out. TODO: finish up here (I got bored)
AIX dd
AIX dd is broken. From their manpage:
3. Use the backup, tar, or cpio command instead of the dd com-
mand whenever possible to copy files to tape. These commands are
designed for use with tape devices. For more information on us-
ing tape devices see the rmt special file.
6. To ensure that only whole blocks are written to the output
device (such as an 8mm tape in fixed-block mode), specify the ibs
flag, the obs flag, and the conv=sync flag. The ibs flag must
be a multiple of the obs flag.
This was hard-won knowledge for me. When
dding a tar file directly out to tape, I ended up using:
dd if=foo.tar of=/dev/rmt1.5 ibs=1 obs=10240 conv=sync
Alternatively, you can use
catblock, which is more efficient.
Note that their comment about ibs being a multiple of obs is simply
wrong, as my example demonstrates.
Solaris dd
Solaris dd is also not necessarily broken,
but somewhat unintuitive. From the manpage:
When dd reads from a pipe, using the
ibs=X and obs=Y operands, the output will always be blocked in chunks
of size Y. When bs=Z is used, the output blocks will be whatever
was available to be read from the pipe at the time.
In other words, don't expect "bs=Z" to
be the same as "obs=Z ibs=Z". That's because, like GNU tar, it probably
uses one buffer if you put "bs=Z", whereas "obs=Z ibs=Z" forces it to
use two buffers. GNU dd may be subject to the same deficiency here.
(TODO: check) I suppose it depends on how it treats a short read.
In the most basic
sense, the DD command is used for copying in the UNIX environment.
For simplicity, we will consider 'copy' to mean 'to duplicate exactly.'
The DD command is used in the Forensics Arena to perform
a physical backup of the evidence. DD can be thought of as
tool - in the sense that using it is a means of building an evidence
file. There are other tools which can be used when making a physical
backup, such as EnCase and SafeBack. However, the intent of this
paper is to give some insight on what DD is and how to use
it.
What is special about the DD copy command is that it has
special flags available to it that make it suitable for copying
block-oriented devices, such as tapes. DD is capable of addressing
these block devices sequentially. We will discuss this later. But,
for now, it is good to note that this is why DD can be a
powerful tool when acquiring and copying tapes for cases.
I do not want to describe each and every flag option available to
DD ('man DD' can show you them). I do, however, want to detail
some key flags that are very useful when copying evidence. Before
we can get into these, it is imperative to understand the basic
syntax of the DD command:
dd if=/*source* of=/*destination*
where:
if = infile, or evidence you are copying (a hard disk, tape, etc.)
source = source of evidence
of = outfile, or copy of evidence
destination = where you want to put the copy
For example, if our acquired evidence is /dev/hda, the following
would produce an exact copy with the name of 'case10img1':
dd if=/dev/hda of=/dev/case10img1
Now that we see the basic use of DD we can look at the options
which make it very suitable for copying in the UNIX environment.
As mentioned earlier, DD is very useful when copying and/or
restoring block-oriented devices, such as tapes. (NOTE: DD
is an excellent tool to use when copying hard disks as well. I am
stressing the usage with regards to tapes because it has proved
quite useful in reducing the amount of time required to copy tapes
of large sizes.) There are a few options available when copying
tapes (or any device). Of the options available, I have found some
more useful than others. These are shown below:
ibs = input block size
obs = output block size
count = number of blocks to copy
skip = number of blocks to skip at start of input
seek = number of blocks to skip at start of output
conv = conversion
Let's say we have a 2GB hard disk seized as evidence. We will use
DD to make a complete physical backup of the hard disk:
dd if=/dev/hda of=/dev/case5img1
Now let's say we have an unknown tape to examine. If we are unsure
of the block size used on the tape, we could use the ibs/obs flags
to find the correct size. Finding the correct size speeds up the
copying process - sometimes dramatically!
dd if=/dev/st0 ibs=128 of=/dev/case10img1 obs=1 count=1
The above usage will attempt to take 1 block with size of 128 from
'st0' and create 'case10img1' output with a block size of 1. The
'count' flag is used so that only 1 block is read. We do this because
we want to limit DD to just the 1 block. If we did not set
a count size DD would continue on and a whole lot of time
would be wasted! What this example attempts to show is that by setting
the input block size to 128 we can effectively find what the real
block size is (unless, of course, it is 128!). With 512 as the standard
block size, assuming 128 is virtually a failproof way to find the
real block size. The output of the above command would most likely
be an 'error' message (which was our intent) with the real block
size revealed (say 1024, for example).
Another example of DD usage is the following. Let's say we
have an image which we need to chop up into smaller pieces. Perhaps
our backup media is limited to 4 1GB discs and the evidence is 4GB
in size. We could use DD with the flags below to create 4
images of the evidence, each 1GB in size.
dd if=/dev/st0 count=1000000 of=/dev/case10img1
dd if=/dev/st0 count=1000000 skip=1000000 of=/dev/case10img2
dd if=/dev/st0 count=1000000 skip=2000000 of=/dev/case10img3
dd if=/dev/st0 count=1000000 skip=3000000 of=/dev/case10img4
Now, we have taken the 4GB evidence tape and chopped it into 4 separate
1GB images. Each image is 1GB in size. Let's look at this example
more closely. Notice that the first command takes 1GB (count=1000000)
and copies it, naming the copy 'case10img1.' The second command
skips the first 1GB (skip=1000000) and then copies the next
1GB (count=1000000), naming this image 'case10img2.' We can now
see exactly what the 'count' and 'skip' flags do.
As you can see, DD is a very resourceful tool to use when
performing physical backups of evidence. It is especially useful
when working with large hard disks and/or tapes. The examples above
were created to show you different ways you can get DD to
work for you. As you become more familiar with it, you will find
that you can do more than what I've shown above. You may even find
out that DD is also quite useful when restoring evidence!
I recommend that you create some evidence disks and tapes and play
with DD. Read the man page on it and try the different flags.
The learning curve is not steep, and the cost (free) can't be beat!
Hi everyone,
I have tried time and time again to make images of my NTFS drives
via the
dd command in windows. I use the FIRE cd forensic shell on the windows
box and:
dd.exe if=\\.\f: |nc.exe <forensic machine IP> <port>
On my linux box I run:
nc -l -p <port> |dd of=/home/user/ntfs.dd
That all works fine and it makes and transfers the file but then
I try to add the file in autopsy and it tells me its not an
NTFS image and consequently doesn't add it.
I tried conv=noerrors and I tried just dumping the file on the linux
box without dd on the of= side. I tried different NTFS partitions
of different sizes as well. My linux box has the NTFS support
kernel mod and everything else about autopsy works fine. Just
these NTFS images. I have no probs using dd with linux partitions
at all. I'd like to find a solution to this because commerical
ware like Encase is outrageously expensive and dd is free making
it perfect for my situation.
Thanks,
Sakaba
Anytime we think about installing OS on more than one system
'cloning' comes to our attention. Because we are too lazy :-). Well
that is one of the important characteristics of Systems Administrator
so that he/she is forced to automate. In this document we
will try to exploit the power of low level data transfer command
popularly known as 'dd' and netcat. These programs are available
for all major UNIX, Linux and Windows platforms. These commands
are fairly popular among Forensics Analysis professionals.
Problem Description:
You got more than 1 machine with almost same hardware. i.e.
same hard drive, SCSI card, graphics etc. You setup one single
machine from top to bottom and now it is time to replicate OS
setup on other machines. Commercial Software such as Ghost does
a good job in cloning Windows based machines and now many of
these software support Linux ext2 file system also. dd
although very crude but gives you enough flexibility to manipulate
cloning as you wish. We have demonstrated cloning of hard drives
in machines loaded with Linux, Win2K, Solaris, HPUX machines
using dd . This document is not a single click
solution so you may have to juggle through here first. Once
you get a hold of this process then it is very powerful to create
your own disk cloning schemes save lots of time and hassle.
Basic concept:
'dd' command can copy any data bit by bit from one
location to another location. So a simple command
dd if=<src> of=<dst>
where, <src> and <dst> can be a file, file
system partition or whole hard drive so anything which can be
read/write in binary form, dd can handle it. dd however
is not a network program. In order to support dd with networking
feature we use another nice command 'netcat'.
netcat can be used to connect any TCP/UDP servers and a very
good tool for diagnostics also. A typical netcat can run
both into client server mode. such as:
server% nc -l -p 30000
==> (Listen for port 30000 on <server> )
client% nc <server> 30000
==> (Connect to <server> at port 30000,
ready to communicate)
This document will explain cloning under Linux, but concept
is very similar for all other operating systems also for which
'dd' and 'netcat' binaries are available.
Operating System
Cloning (Using STANDALONE machine):
Let us assume we have two drives (sda) and
(sdb) attached to the system ( Example:
Linux box, but can be any other OS). (sda) is drive with
Master OS (let's call it Master OS drive) and (sdb) is
a drive (slave drive) where we have to clone data
from (sda).
- IMPORTANT (IF you
are cloning RAID/root devices):
If slave drive has already
RAID and/or root partition setup before. (especially
/ (root) partition), make sure you run fdisk (Use
some 3rd Linux box if required) on slave hard
drives , remove any partitions on slave hard
drives and make it a plain new disk. If you don't do this
and connect both drives with Master Linux box, it is very
likely that Master linux box may come up with slave drive
as (sda) i.e. primary device rather than expected (sdb)
because how they appear in SCSI scan list. While following
steps below you may destroy
contents of actual Master Linux drive. So
you have been warned already :-).
- CAUTION (IF you are
using SCSI drives) : In case slave drive is connected
to external SCSI controllers and if external SCSI
controllers appear first in SCSI scan list during boot and
BIOS scan, then external drives will get first device name
such as /dev/sda , /dev/sdb and your master Linux box may
not boot from desired hard drive. In order to avoid this,
DISABLE BIOS scan for external SCSI controller. This
will cause not to scan SCSI drives connected with external
SCSI controller, so Linux will get internal harddrive as
/dev/sda and boot from desired disk. Drives
on external controller will be available when Linux Kernel
try to scan all SCSI bus while booting. For a typical Adaptec
SCSI controller:
- During Boot time, Go to SCSI controller BIOS screen
(Ctrl-A)
- Select Controller Configuration -> Configure/View
Host Adapter Settings -> Advance Configuration options
-> Host Adapter Bios = (Disabled: Not Scan) .
- Save options and reboot box, During next boot, you
will see SCSI BIOS Not Installed for this card.
DO NOT DISABLE BIOS SCAN FOR INTERNAL SCSI BUS. THIS
IS NECESSARY TO FIND PRIMARY BOOT DISK.
Basically in any case IDE ot SCSI make sure you are
booting from correct harddrive and that should appear as
/dev/sda (Master drive) before you start any cloning
process.
Let's assume we have to clone a harddrive (sda). Which has
a partition table setup below. It has 1 NTFS partition loaded
with WinNT/Win2K and rest Linux partition. (swap, Linux
and Raid partition). Assuming second (slave) harddrive (sdb)
is also attached to the same system.
Device Boot Start End Blocks Id System
/dev/sda1 1
9 72261 83 HPFS/NTFS
/dev/sda2
10 75 530145 82 Linux swap
/dev/sda3
76 467 3148740 fd Linux raid autodetect
/dev/sda4
468 2200 13920322+ 83 Linux
A simple way to clone this drive (/dev/sda) to another drive
attached to this system (/dev/sdb) is to use dd command.
dd if=/dev/sda of=/dev/sdb
This command will copy each bit from sda (Master drive)
to sdb (Slave drive) including MBR (Master
Boot record). Thus after cloning new drive (sdb) is ready for
deployment. This will also copy any information like File System
IDs etc.
Since these days drive size is getting bigger and may run
upto 100+ GB, this whole dd process may take long time and obviously
there is no point in cloning Linux swap area or empty partitions
which doesn't contain any useful data yet. Hence in this situation
it is best to clone only relevant partitions. For this you
need to partition second drive beforehand.
Note: Both drives
are partitioned exactly same. If you have different brand harddrives,
make sure each partition on second drive must be equal
to or greater than first drive partitions. Also make sure
File system ID should match for second drive also.
Device
Boot Start End Blocks Id System
/dev/sdb1 1
9 72261 83 HPFS/NTFS
/dev/sdb2
10 75 530145 82 Linux swap
/dev/sdb3
76 467 3148740 fd Linux raid autodetect
/dev/sdb4
468 2200 13920322+ 83 Linux
Now cloning process partition by partition will look
like:
- First step is to copy MBR (Master Boot Record)
to second drive. MBR is read right after BIOS in PCs bootstrap
process. In case of Linux this will store LiLo ( Linux
Loader) setup to find out Linux or Windows kernel.
MBR is located with in first 446 bytes in harddrive
(or partition) selected to store MBR during Linux install
time or Windows install time. In our Linux installation
we have selected MBR to be stored on first harddrive.
Steps below will make second drive as a bootable drive.
Note:
here we are using whole drive sda and sdb as input
and output arguments of dd. (This process of making Solaris,
HPUX drives bootable may be different but they allow you to
setup boot record also just like PC's MBR)
dd if=/dev/sda of=/dev/sdb
bs=446 count=1
- There is no reason to clone swap partition. Swap is
raw area.
- Now you can clone other relevant partitions. Let's say
/dev/sda3 will contain Linux OS and mounted as / (root)
having ext2 or some other file system and /dev/sda4
is mounted as /home but doesn't contain any data. (May or
may not contain any file system)
dd if=/dev/sda1 of=/dev/sdb1
==> Clone NTFS partition
dd if=/dev/sda3 of=/dev/sdb3
==> Clone RAID-1 partition having ext2 FS or some other.
- So in this case we can save time just by cloning desired
partitions.
Operating System Cloning
over network:
One major bottleneck in above process is we have to
physically open boxes, connect harddrive to Master box and the
run clone process. This is easier in case of desktops where
you have a liberty to connect external drives (IDE, SCSI bus).
But Laptop can hardly house one IDE drive in general and there
are no easy way to open and connect second drive for cloning.
Thus above process will be highly useful if cloning process
can be used over network. There are several possible combination
presented here. Idea here is we have Master Linux box
up and running over network and we boot slave box having harddrive
which is to be cloned but we use some alternate media
such as boot CD and boot slave linux using root file system
on CD itself *NOT* on harddrive so that we are free to write
on slave hardrive.
Master Box-----------network-----------Slave
box
[]
[NOT
* using slave drive]
One of Following 3 methods can be used to boot slave box
using alternative media.
Method [1]
Making your own root filesystem on ext2 CDROM.
(Not Scalable )
One can make a small Linux distribution (less
than 650MB) which can fit into CDROM. Burn this CDROM
with ext2 filesystem (not ISO9660) and then use Linux
boot floppy to boot from and use CDROM ext2 file system as /
(root) file system (read only) (instead of root file system
on Harddrive). This process although is doable but has issues
like you need to have all possible drivers for network, SCSI
etc. Making your custom ext2 read only file system on CD and
booting from it would be quite a trial and error issue. If you
are interested in making such Cds or bootable CDs see
reference section for links. I once did that to clone HP
Omnibook 6000 laptops loaded with Linux+Win2K OS together
and it worked pretty okay but this is not a scalable solution
though.
Method [2]
Using popular Linux distribution and floppy combination.
On a similar line Linux distribution such as
RedHat/SuSe boot CDROM at OS install
time will allow you to boot into some kind of rescue system.
In case of RedHat boot from RedHat OS CD and at initial
OS install prompt type 'linux
rescue ' at the boot time and this will let you use
CDROM as root file system and provide you a shell prompt. Linux
distribution uses this facility to repair problematic Linux
install but we will use this for getting just shell prompt.
Great thing about this is most Linux distribution comes up with
lots of popular SCSI, network drivers so you don't have to worry
about cooking your custom bootable CD.
Many common utilities including 'dd' command usually available
in rescue mode. However you need netcat (static binary not dynamically
linked) command. You can download netcat distribution
and recompile it as a static binary (use -static flag).
When I compiled it is small enough to fit into one floppy.
So you can copy this into floppy. (I formatted floppy
in ext2 format and then mounted in Linux system, copied netcat
binary there.)
mkfs /dev/fd0
mount /dev/fd0 /mnt/floppy
cp nc /mnt/floppy
umount /mnt/floppy
So with 'linux rescue' mode and netcat
binary on floppy you can use dd and netcat to clone
your system over network. As we will see below.
Method [3] Modifying
popular Linux distribution CDs and recreating your personal
bootable ISO image:
If for some reason netcat won't fit in 1 floppy
or you need more utility/binaries. Then you can change Linux
distribution (SuSe/RedHat CD). This is a little hack but works.
NOTE: ISO images
are read-only file systems. Even if you have an iso image (Say
by using dd command )
dd if=/dev/cdrom of=redhat-boot-cd.iso
and if you try to mount this iso file using loopback device
with option read/write (-o rw) (you need to have
loopback device support (CONFIG_BLK_DEV_LOOP=y) compiled in
kernel to do that)
mount -o loop -o rw ./redhat-boot-cd.iso /mnt/cdrom
This won't allow you to write/modify ISO filesystem.
I haven't found any good solutions to edit iso
image directly , One such tool is winISO (http://www.winiso.com
) this is a shareware package so you have to pay for it. But
you can use this to add more files in your ISO image and burn
new image back to new CD. If you know any better solution let
me know also :-)
Following steps are useful for adding additional files in
RedHat bootable ISO image and burning a new CDs with additional
files as of your choice.
- First mount RedHat CD (say RedHat 7.1).
mount /dev/cdrom /mnt/cdrom
- Create a directory where you store stuff what goes on
new CD. mkdir /home/cdburn
- cd /mnt/cdrom
- (tar cbf 20 - *) |
(cd /home/cdburn | tar xvbf 20 -)
- This tar command will copy whole CD (~650MB to your
Harddrive). In most cases you do not need all CD. Important
stuff you need from directories, dosutils/,
images/, RedHat/base. But if you do not
want to play much simply copy the whole CD as above tar
command and then you can delete
RedHat/RPMS directory. These
are simply RPM packages and since our intention is not to
install OS from CD so we don't need that.
- Create subdirectory directory
mkdir /home/cdburn/mystuff/
and add all your stuff such as static version of netcat
binary etc. there.
- Now delete all
TRANSLATION TABLES (TRANS.TBLS) files, otherwise
mkisofs command will complain.
find ./ -name "TRANS.TBL" -exec /bin/rm
{} \;
- Make bootable iso image out of above distribution (/home/cdburn).
Use mkisofs (part of mkisofs-1.9-6 package). The
command below will create a bootable ISO image using initial
boot image specified by -b, the -c
option is for the boot catalog file. The -r
option will make appropriate file ownership and modes. This
iso image redhat-bootcd.iso is very similar
to what is provided by RedHat except it will have our stuff
also and we may have deleted any unnecessary contents
such as RedHat/RPMS directory.
mkisofs -r -b images/boot.img
-c boot.catalog -o /tmp/redhat-bootcd.iso ./
- Finally burn this iso image redhat-bootcd.iso
using your cd-burner.
Now the Real drill:
Whatever method you choose to boot slave machine (
RedHat bootable CD + floopy or custom bootable RedHat
CD), ultimate aim is to obtain shell, dd and netcat binary
after 'linux rescue'. After you get shell you can access
files stored on boot CD by changing directory to
/mnt/sources/mystuff .
Hopefully your ethernet card has been detected by now. (as most Linux
distributions allow OS install over network) if not then you
have to load drivers for your ethernet card. Linux distribution
documentation usually tells that how and sometimes they provide
extra drivers floppy. In case of RedHat these floppy images
are generally stored under directory images/ and you can copy
these images to your floppy using commands like
dd if=<floppy-image> of=/dev/fd0
On Slave machine:
Run netcat command first on slave linux box (that to be cloned and
booted using Linux boot CDROM as 'linux rescue' (See also
Shell script
case [1] in automation section below).
Once ethernet card has been detected. (Use ifconfig -a command
to check) assign IP address to this interface now on slave
machine. Define loopback interface also. (You may choose
different IP address for eth0). Also you may need to define
/etc/hosts file before you can
assign IP address. Use following commands to create your new
/etc/hosts. (These are actually created in ram file system RAMFS).
rm /etc/hosts
echo "127.0.0.1 localhost" >
/etc/hosts
echo "192.168.0.254 fakehost"
>> /etc/hosts
ifconfig lo 127.0.0.1 up
ipconfig eth0 192.168.0.254 up
Assuming Master Linux box (from where you want to clone)
is up and running with IP 192.168.0.1.
slave% nc -l -p 9000 | dd of=/dev/sda
(Replace /dev/sda with actual drive on your
slave machine)
This will listen at port 9000 and whatever it gets at port
9000 will hand over to dd command and dd will simply write
that to on slave harddrive (sda) bit by bit. Here I am assuming
dd and netcat (nc) are available either through floppy (/mnt/floppy/nc
or through /mnt/sources/mystuff/nc). In case of floppy you need
to mount floppy first using command:
mount /dev/fd0 /mnt/floppy
On Master machine:
Now Login on master linux box and run following command. (It
is advisable that Master Linux box should be in calm state ,
i.e no major jobs running on the machine). This command below
will read master disk bit by bit and throw this bit stream to
netcat command which is connected to netcat command at port
9000 on <slave> box.
master% dd if=/dev/sda | nc 192.168.0.254
9000
That's it. You may have to wait for long time
depending upon network speed and size of your harddrive. Typically
36GB drive may take 50 minutes over 100Mbps link. Again
rather than cloning complete drive we can clone only relevant
partitions and MBR only. That will make cloning much faster
like we saw in above section.
Automating process and Case
studies:
One of the primary reason for using dd and netcat way of
cloning OS instead of using commercial software such as Ghost
is we have a liberty to automate process as we like. Following
scripts may help in automating cloning process.
Case [1]: Script
for Slave machine (netcat and dd cloning) on the fly.
- Make sure you have netcat command available either /mnt/floppy
or /mnt/sources/mystuff area.
=================================================
cloneme.sh :: Shell script for slave machine.
=================================================
#!/bin/sh
############### Edit variables below ######################
FLOPPY_PATH=/mnt/floppy
MYSTUFF_PATH=/mnt/sources/mystuff
# Uncomment only One of the
options below.
#### OPTION ==> 1 if using floppy ################
#NC=$FLOPPY_PATH/nc
#### OPTION ==> 2 if using mystuff/ on
CD #########
NC=$MYSTUFF_PATH/nc
LPORT=9000
DEST=/dev/sda
SRC=$DEST
############# No need to edit after this
in general ###########
if [ $# -eq 1 ]
then
IPADDR=$1
echo "###############################################################"
echo " If there are
no errors here. You need to run following"
echo " command on Master
Box."
echo ""
echo "dd if=$SRC | nc $IPADDR $LPORT"
echo "###############################################################"
echo ""
echo "##>> Preparing /etc/hosts
##"
rm /etc/hosts
echo "127.0.0.1 localhost" > /etc/hosts
echo "$IPADDR fakehost" >> /etc/hosts
echo "#===================================================================="
echo "NOTE:: If you need to create
routes"
echo " #route add -net <DEST_NET>
netmask 255.255.255.0 gw $IPADDR metric 0"
echo "#===================================================================="
echo "##>> Preparing
interfaces lo and eth0 ##"
ifconfig lo 127.0.0.1 up
ifconfig eth0 $IPADDR up
echo ""
echo ">>> Now start listening(at
$LPORT) for traffic from Master :-)"
echo "$NC -l -p $LPORT | dd of=$DEST"
$NC -l -p $LPORT | dd of=$DEST
echo ""
echo "%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%"
echo " Cloning Process completed.....
:-) Reboot Now"
echo "%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%"
else
echo "Usage:: cloneme <IP_ADDR_OF_THIS_MACHINE>"
fi
Case [2] Saving Disk
Images (Export Image for later use):
Although you can clone running machine over the network
anytime. But it is sometimes desirable to store base installation
as a reference image and you may want to clone from this pristine
image later. With the help of dd you can image disks also. But
let's discuss some issues first.
Most 32 bit operating system (Linux for IA32 , Windows etc.)
will have physical limitation on max file size. In general practical
limit is 2GB as a max. file size. 64 bit OS (Solaris8, HPUX
11.0, Linux for IA64, etc.) will not have this limitation. So
if you use dd to copy harddrive image you can maximum image
2GB harddrive. That is pretty useless these days. Fortunately
dd can image in chunks and you can specify start and end blocks,
skip blocks etc. So idea here is to image your big harddrives
in chunks of approx. 2GB files over network. Although
I noticed RedHat 7.1 with Linux 2.4.x kernels will allow fie
size even bigger than 4GB on ext2 FS.
Also if you want to store images in compressed format (to save
space) it is desirable to have each image file size not too
large.
Following perl script (export-image.pl) can be used to
image local Linux harddrive /dev/hda to remote machine over
NFS using dd.
If you are not running NFS you can implement same thing using
dd and netcat. For now that would be a manual process.
If somebody knows a better way to run netcat and transfer multiple
files automatically between two machines please let me
know and I will cook up some automation script here.
This perl script is actually use dd command something as
described below. This is imaging your big harddrive into chunks
of 1950 MB files named (1, 2, 3, 4, .....) over NFS to remote
machine.
($NFS is NFS destination directory on another server having
plenty of space)
For 1st Image:
dd if=/dev/hda of=$NFS/1 bs=1024k count=1950
skip=0
For 2nd image: (Skipping the part of harddrive used for 1st
image.)
dd if=/dev/hda of=$NFS/2 bs=1024k count=1950
skip=1950
For 3rd image: (Skipping the part of Harddrive used for 1st+2nd
image)
dd if=/dev/hda of=$NFS/3 bs=1024k count=1950
skip=3900
and so on.
In case you want to use netcat you can simply pipe above
dd commands manually to netcat and listen using netcat and dd
on remote machine, just like we used netcat and dd to clone
hardisks above. For example imaging harddrive on machineA and
saving image on machineB.
For 1st image:
machineB% nc -l -p 9000 | dd of=1
machineA(master)% dd if=/dev/hda
bs=1024k count=1950 skip=0 | nc machineB 9000
For 2nd image:
machineB% nc -l -p 9001 | dd of=2
machineA(master)% dd if=/dev/hda
bs=1024k count=1950 skip=1950 | nc machineB 9001
For 3rdimage:
machineB% nc -l -p 9002 | dd of=2
machineA(master)% dd if=/dev/hda
bs=1024k count=1950 skip=3900 | nc machineB 9002
and so on.
Once you have images (1, 2, 3, 4 ....) stored on network
then you can boot your slave Linux box using bootable CD and
pull these images to slave box as described in case [3].
========================================================
export-image.pl :: Perl script to image big harddrive using
dd and NFS.
========================================================
#!/usr/bin/perl
#####################################################
#This script will run dd command (in serial)
and dump
#1950 blocks (1.9GB) file for each.
#Run script as perl export-image.pl
#####################################################
################ Edit variables
below #########################
#device is raw device name for
harddrive to be cloned (imaged).
$device="/dev/hda";
#mount NFS file system with large
space available which can hold images.
$nfs_path="/nfs/remote/home/tmp";
#Image name (read from user) (Make
sure you have $nfs_path/$image directory)
#on remote machine.
$image="ob6000";
############################################################
$dd="/bin/dd";
#For compressing image
$bzip2="/usr/bin/bzip2";
$suffix=".bz2";
############## No need to edit after this
#########################
$bs="1024k";
$block_count=1950;
$image_dir="$nfs_path/$image";
$compress=$bzip2;
$proceed=0;
if(!(-d $image_dir) )
{ die "\nOops!! Image Directory
$image_dir must exist with chmod 777 permission\n"; }
system("clear");
print <<MSG1;
###########################################################
NOTE:: COMPRESSION TAKE TOO MUCH
TIME(Many HOURS) OVER NFS.
So better compress manually latter
on server itself.
###########################################################
\n\n Do you want to compress images
using $compress [y/n] (Default n) = \t
MSG1
$compress_flag=<STDIN>;
if(($compress_flag eq "y") or ($compress_flag
eq "Y"))
{ $compress_flag=1; }
else
{ $compress_flag=0; }
print "\n\n";
print "***************************************************\n";
print " Local Device = $device [SOURCE]
\n";
print " Image Dir = $image_dir [TARGET]
\n";
print "***************************************************\n\n\n";
print "Dude! I hope you understand what
are you doing by pressing [y/Y] here :-) \n";
print " Press [y/Y] if you
want to continue .. ";
$con=<STDIN>; chomp($con);
if(($con eq "y") or ($con
eq "Y"))
{
$i=0;
$image_size=1; #Some fake value greater
than zero.
print "\n\nDisk Imaging starts...\n";
system("date");
while($image_size > 0)
{
$image_name="$image_dir/$i";
print "##############################################\n";
print "Creating Image $image_name\n";
print "##############################################\n";
$skip=$i*$block_count;
print "$dd if=$device of=$image_name
bs=$bs count=$block_count skip=$skip \n";
system("$dd if=$device of=$image_name
bs=$bs count=$block_count skip=$skip");
if($compress_flag)
{
print "Compressing Image:
$bzip2 $image_name => $image_name$suffix\n";
system("$bzip2 $image_name");
$image_name .= "$suffix";
}
++$i;
$image_size=(stat($image_name))[7];
system("date");
}
}
else
{
print "Bye Bye ...\n";
}
Case [3] Importing Disk
Images (1, 2, 3, 4 ...) created in Case [2] using netcat, dd
and cat
This part is little tricky in the sense we want all images
(1, 2, 3, 4, ...) to be imported on slave machine and use dd
to write these images serially on slave drive. A very simple
set of commands can be used as below.
On Slave machine:
(booted through linux rescue). Run following netcat command
to capture incoming data stream.
machineC(slave)% nc -l -p 9000 |
dd of=/dev/hda
On machineB machine:
(where images 1, 2, 3, 4 .... are stored). Run following cat
and netcat command. Make sure you cat images in the same sequence
as they were imported in case [2]. cat command will simply
join these images and throw data stream to netcat which slave
machine will pick up and copy bit by bit on slave
harddrive.
machineB% cat 1 2 3 4 .... | nc
machineC 9000
Case [4] Importing
Disk images created in Case[2]:
Most likely 'linux rescue' system won't have NFS support. Which
means when you boot slave box using such method you can not
access resources over NFS. But if you cook your own CD and that
has NFS support and perl the following perl script can be used
to fetch images stored earlier from machineB using NFS.
This script is actually doing:
($NFS is NFS source directory on another server machineB where
you have images 1, 2, 3, 4, ... stored earlier)
For image 1:
dd if=$NFS/1 of=/dev/hda bs=1024k conv=notrunc
seek=0
For image 2:
dd if=$NFS/2 of=/dev/hda bs=1024k conv=notrunc
seek=1950
For image 3:
dd if=$NFS/3 of=/dev/hda bs=1024k
conv=notrunc seek=3900
In any case if you are interested in using perl script below
(if you have perl and NFS client support on slave
linux box).
import-image.pl
#!/usr/bin/perl
#####################################################
#This script will run dd command (in serial)
and dump
#and import image.
#####################################################
##############################################################################
#device is target raw device name
for harddrive to be cloned.
$device="/dev/hda";
#mount NFS file system with large
space available which can hold images.
$nfs_path="/mnt/images";
#Image name (read from user)
$image="ob6000";
###############################################################################
$dd="/bin/dd";
#$bzcat="/usr/bin/bzcat";
#$suffix=".bz2";
$bs="1024k";
$block_count=1950;
###############################################################################
$image_dir="$nfs_path/$image";
$proceed=0;
if(!(-d $image_dir) )
{ die "\nOops!! No Image Directory
$image_dir\n"; }
system("clear");
print "***************************************************\n";
print " Local Device = $device [TARGET]\n";
print " Image Dir = $image_dir [SOURCE]\n";
print "***************************************************\n\n\n";
print "Dude! I hope you understand what
are you doing by pressing [y/Y] here :-) \n";
print " Press [y/Y] if you
want to continue .. ";
$con=<STDIN>; chomp($con);
print " Once Again!!! Press
[y/Y] if you want to continue .. ";
$con=<STDIN>; chomp($con);
system("date");
if(($con eq "y") or ($con eq "Y"))
{
print "\n\nDisk Imaging import starts...\n";
$i=0;
$image_name="$image_dir/$i";
while(-f $image_name )
{
print "##############################################\n";
print "Importing Image $image_name\n";
print "##############################################\n";
$seek=$i*$block_count;
print "##############################################\n";
$seek=$i*$block_count;
print "$dd if=$image_name of=$device
bs=$bs conv=notrunc seek=$seek \n";
#system("$bzcat $image_name | $dd
of=$device bs=$bs conv=notrunc seek=$seek");
system("$dd if=$image_name
of=$device bs=$bs conv=notrunc seek=$seek");
++$i;
$image_name="$image_dir/$i";
system("date");
}
}
else
{
print "Bye Bye ...\n";
}
Other Operating Systems
Tips:
You can pretty much do same in other operating systems also.
This section quickly list few tips that may be useful.
Windows:
- You can find GNU utilities (
http://unxutils.sourceforge.net/ ) for Win32 platforms
which includes dd.exe command.
- dd.exe syntax is similar to as you use on Linux side.
For physical partition you may have to use devicename something
like \\.\PhysicalDrive0 etc. For example
dd if=\\.\PhysicalDrive0
of=<target>
- You can download netcat for windows (
http://www.l0pht.com/~weld/netcat/ )
- If you are cloning WinNT/2K system , you need to change
SID for the new cloned system if machine is participating
in Windows domain. You can use Ghostwalker program from
Ghost distribution or can use newsid.exe from
http://www.sysinternals.com
Solaris:
Others: (Make disk bootable)
- Irix : dvhtool
- HP-UX: mkboot
- Tru64: disklabel
Conclusion:
Few possible uses of netcat and dd shown in this document.
Although methods presented here are very simple and easy to
use but have few pros and cons also. This technique is very
good for on the fly OS cloning. When we image the whole drive
we need the equivalent harddrive space on other
machine. This may not be very much practical. You can try compressing
those images which will save lots of space. I noticed dd image
can be compressed upto 30-80% depending upon real data on the
drive using gzip/compress program. This cloning and imaging
method may be very effective in forensic analysis where sometimes
you need an exact snapshot of harddrive including swap space
partitions. You can always break your images in small pieces
(may be compress them) transfer over network to somewhere else
and reproduce data. As mentioned above one of the great advantage
here is to custom your own cloning scheme.
References:
- GNU utilities for Win32.
http://unxutils.sourceforge.net/
- netcat for Windows.
http://www.l0pht.com/~weld/netcat
- First Attempt at Creating a Bootable Live Filesystem on
a CDROM
http://www.linuxgazette.com/issue54/nielsen.html
- Good Site for Windows utilities such as newsid.exe:
http://www.sysinternals.com
- Modifying ISO image
http://www.winiso.com
- Solaris Bootable CD creation:
http://www.lka.ch/projects/solcdburn/solcdburn.html
- Sun Blueprint:
http://www.sun.com/software/solutions/blueprints/0301/BuildBoot.pdf
- Linux on Floppy:
http://www.toms.net/rb/
- Static binaries for Linux.
Take Command dd -- http://www.linuxjournal.com/article.php?sid=1320
The ' dd ' command is one of the original Unix utilities and should
be in everyone's tool box. It can strip headers, extract parts of binary
files and write into the middle of floppy disks; it is used by the Linux
kernel Makefiles to make boot images. It can be used to copy and convert
magnetic tape formats, convert between ASCII and EBCDIC, swap bytes,
and force to upper and lowercase. For blocked I/O, the dd command
has no competition in the standard tool set.
One could write a custom utility to do specific I/O or formatting
but, as dd is already available almost everywhere, it makes sense to
use it. Like most well-behaved commands, dd reads from its standard
input and writes to its standard output, unless a command line specification
has been given. This allows dd to be used in pipes, and remotely with
the rsh remote shell command.
Unlike most commands, dd uses a keyword=value format for its parameters.
This was reputedly modeled after IBM System/360 JCL, which had an elaborate
DD 'Dataset Definition' specification for I/O devices. A complete listing
of all keywords is available from GNU dd with
$ dd --help
Some people believe dd means ``Destroy Disk'' or ``Delete Data''
because if it is misused, a partition or output file can be trashed
very quickly. Since dd is the tool used to write disk headers, boot
records, and similar system data areas, misuse of dd has probably trashed
many hard disks and file systems. In essence, dd copies and optionally
converts data. It uses an input buffer, conversion buffer if conversion
is specified, and an output buffer. Reads are issued to the input file
or device for the size of the input buffer, optional conversions are
applied, and writes are issued for the size of the output buffer. This
allows I/O requests to be tailored to the requirements of a task. Output
to standard error reports the number of full and short blocks read and
written.
Example 1
A typical task for dd is copying a floppy disk. As the common geometry
of a 3.5" floppy is 18 sectors per track, two heads and 80 cylinders,
an optimized dd command to read a floppy is:
Example 1-a : Copying from a 3.5" floppy
dd bs=2x80x18b if=/dev/fd0 of=/tmp/floppy.image
1+0 records in
1+0 records out
The 18b specifies 18 sectors of 512 bytes, the 2x multiplies the
sector size by the number of heads, and the 80x is for the cylinders--a
total of 1474560 bytes. This issues a single 1474560-byte read request
to /dev/fd0 and a single 1474560 write request to /tmp/floppy.image,
whereas a corresponding cp command
cp /dev/fd0 /tmp/floppy.image
issues 360 reads and writes of 4096 bytes. While this may seem insignificant
on a 1.44MB file, when larger amounts of data are involved, reducing
the number of system calls and improving performance can be significant.
This example also shows the factor capability in the GNU dd number
specification. This has been around since before the Programmers Work
Bench and, while not documented in the GNU dd man page, is present in
the source and works just fine, thank you.
To finish copying a floppy, the original needs to be ejected, a new
diskette inserted, and another dd command issued to write to the diskette:
Example 1-b : Copying to a 3.5" floppy
dd bs=2x80x18b < /tmp/floppy.image > /dev/fd0
1+0 records in
1+0 records out
Here is shown the stdin/stdout usage, in which respect dd is like
most other utilities.
Example 2
The original need for dd came with the 1/2" tapes used to exchange
data with other systems and boot and install Unix on the PDP/11. Those
days are gone, but the 9-track format lives. To access the venerable
9-track, 1/2" tape, dd is superior. With modern SCSI tape devices, blocking
and unblocking are no longer a necessity, as the hardware reads and
writes 512-byte data blocks.
However, the 9-track 1/2" tape format allows for variable length
blocking and can be impossible to read with the cp command. The dd command
allows for the exact specification of input and output block sizes,
and can even read variable length block sizes, by specifying an input
buffer size larger than any of the blocks on the tape. Short blocks
are read, and dd happily copies those to the output file without complaint,
simply reporting on the number of complete and short blocks encountered.
Then there are the EBCDIC datasets transferred from such systems
as MVS, which are almost always 80-character blank-padded Hollerith
Card Images! No problem for dd, which will convert these to newline-terminated
variable record length ASCII. Making the format is just as easy and
dd again is the right tool for the job.
Example 2 : Converting EBCDIC 80-character fixed-length record
to ASCII variable-length newline-terminated record
dd bs=10240 cbs=80 conv=ascii,unblock if=/dev/st0 of=ascii.out
40+0 records in
38+1 records out
The fixed record length is specified by the cbs=80 parameter, and
the input and output block sizes are set with bs=10240. The EBCDIC-to-ASCII
conversion and fixed-to-variable record length conversion are enabled
with the conv=ascii,noblock parameter.
Notice the output record count is smaller than the input record count.
This is due to the padding spaces eliminated from the output file and
replaced with newline characters.
Example 3
Sometimes data arrives from sources in unusual formats. For example,
every time I read a tape made on an SGI machine, the bytes are swapped.
The dd command takes this in stride, swapping the bytes as required.
The ability to use dd in a pipe with rsh means that the tape device
on any *nix system is accessible, given the proper rlogin setup.
Example 3 : Byte Swapping with Remote Access of Magnet Tape
rsh sgi.with.tape dd bs=256b if=/dev/rmt0 conv=swab | tar xvf -
The dd runs on the SGI and swaps the bytes before writing to the tar
command running on the local host.
Example 4
Murphy's Law was postulated long before digital computers, but it
seems it was specifically targeted for them. When you need to read a
floppy or tape, it is the only copy in the universe and you have a deadline
past due, that is when you will have a bad spot on the magnetic media,
and your data will be unreadable. To the rescue comes dd, which can
read all the good data around the bad spot and continue after the error
is encountered. Sometimes this is all that is needed to recover the
important data.
Example 4 : Error Handling
dd bs=265b conv=noerror if=/dev/st0 of=/tmp/bad.tape.image
Example 5
The Linux kernel Makefiles use dd to build the boot image. In the
Alpha Makefile /usr/src/linux/arch/alpha/boot/Makefile, the srmboot
target issues the command:
Example 5 : Kernel Image Makefile
dd if=bootimage of=$(BOOTDEV) bs=512 seek=1 skip=1
This skips the first 512 bytes of the input bootimage file (skip=1)
and writes starting at the second sector of the $(BOOTDEV) device (seek=1).
A typical use of dd is to skip executable headers and begin writing
in the middle of a device, skipping volume and partition data. As this
can cause your disk to lose file system data, please test and use these
applications with care.
Credits
The dd command has been around since the 1970s, ported to many systems,
rewritten many times, and tested by time as a useful tool. The current
Linux version is GNU dd GNU fileutils 3.12, written by Paul Rubin, David
MacKenzie, and Stuart Kemp, Copyright © 1985, 1990, 1991 Free Software
Foundation, Inc. GNU dd is found in the fileutils collection, with the
current version at the URL ftp://prep.ai.mit.edu/pub/gnu/fileutils-3.12.tar.gz
or a mirror near you.
It is often neccesary to backup into a remote machine's tape drive. Here
are the commands that can be used to achieve this. Execute this command
on the machine you want to backup.
$ tar cvf - $DIRNAME | rsh $SYS dd of=$TAPEDEV
Substitute
$DIRNAME with the directory to backup,
$SYS with the machine name with the tape drive,
$TAPEDEV with the tape device.
Note: You must be able to rlogin into the remote machine without
a password. To do this add the name of your local machine with your user
name in the .rhost file in your home directory on the remote machine.
To retrieve the backed up info...
rsh $REM dd if=$TAPEDEV | tar xvf -
Tips For Linux - How and when to use the dd command In this article,
Sam Chessman explains the use of the dd command with a lot of useful
examples. This article is not aimed at absolute beginners. Once you are
familiar with the basics of Linux, you would be in a better position to
use the dd command.
The ' dd ' command is one of the original Unix utilities and should be
in everyone's tool box. It can strip headers, extract parts of binary files
and write into the middle of floppy disks; it is used by the Linux kernel
Makefiles to make boot images. It can be used to copy and convert magnetic
tape formats, convert between ASCII and EBCDIC, swap bytes, and force to
upper and lowercase.
For blocked I/O, the dd command has no competition in the standard tool
set. One could write a custom utility to do specific I/O or formatting but,
as dd is already available almost everywhere, it makes sense to use it.
Like most well-behaved commands, dd reads from its standard input and
writes to its standard output, unless a command line specification has been
given. This allows dd to be used in pipes, and remotely with the rsh remote
shell command.
Unlike most commands, dd uses a keyword=value format for its parameters.
This was reputedly modeled after IBM System/360 JCL, which had an elaborate
DD 'Dataset Definition' specification for I/O devices. A complete listing
of all keywords is available from GNU dd with
$ dd --help
Some people believe dd means ``Destroy Disk'' or ``Delete Data'' because
if it is misused, a partition or output file can be trashed very quickly.
Since dd is the tool used to write disk headers, boot records, and similar
system data areas, misuse of dd has probably trashed many hard disks and
file systems.
In essence, dd copies and optionally converts data. It uses an input
buffer, conversion buffer if conversion is specified, and an output buffer.
Reads are issued to the input file or device for the size of the input buffer,
optional conversions are applied, and writes are issued for the size of
the output buffer. This allows I/O requests to be tailored to the requirements
of a task. Output to standard error reports the number of full and short
blocks read and written.
Example 1
A typical task for dd is copying a floppy disk. As the common geometry
of a 3.5" floppy is 18 sectors per track, two heads and 80 cylinders, an
optimized dd command to read a floppy is:
Example 1-a : Copying from a 3.5" floppy
dd bs=2x80x18b if=/dev/fd0 of=/tmp/floppy.image
1+0 records in
1+0 records out
The 18b specifies 18 sectors of 512 bytes, the 2x multiplies the sector
size by the number of heads, and the 80x is for the cylinders--a total of
1474560 bytes. This issues a single 1474560-byte read request to /dev/fd0
and a single 1474560 write request to /tmp/floppy.image, whereas a corresponding
cp command
cp /dev/fd0 /tmp/floppy.image
issues 360 reads and writes of 4096 bytes. While this may seem insignificant
on a 1.44MB file, when larger amounts of data are involved, reducing the
number of system calls and improving performance can be significant.
This example also shows the factor capability in the GNU dd number specification.
This has been around since before the Programmers Work Bench and, while
not documented in the GNU dd man page, is present in the source and works
just fine, thank you.
To finish copying a floppy, the original needs to be ejected, a new diskette
inserted, and another dd command issued to write to the diskette:
Example 1-b : Copying to a 3.5" floppy
dd bs=2x80x18b < /tmp/floppy.image > /dev/fd0
1+0 records in
1+0 records out
Here is shown the stdin/stdout usage, in which respect dd is like most
other utilities.
Example 2
The original need for dd came with the 1/2" tapes used to exchange data
with other systems and boot and install Unix on the PDP/11. Those days are
gone, but the 9-track format lives. To access the venerable 9-track, 1/2"
tape, dd is superior. With modern SCSI tape devices, blocking and unblocking
are no longer a necessity, as the hardware reads and writes 512-byte data
blocks.
However, the 9-track 1/2" tape format allows for variable length blocking
and can be impossible to read with the cp command. The dd command allows
for the exact specification of input and output block sizes, and can even
read variable length block sizes, by specifying an input buffer size larger
than any of the blocks on the tape. Short blocks are read, and dd happily
copies those to the output file without complaint, simply reporting on the
number of complete and short blocks encountered.
Then there are the EBCDIC datasets transferred from such systems as MVS,
which are almost always 80-character blank-padded Hollerith Card Images!
No problem for dd, which will convert these to newline-terminated variable
record length ASCII. Making the format is just as easy and dd again is the
right tool for the job.
Example 2 : Converting EBCDIC 80-character fixed-length record to
ASCII variable-length newline-terminated record
dd bs=10240 cbs=80 conv=ascii,unblock if=/dev/st0 of=ascii.out
40+0 records in
38+1 records out
The fixed record length is specified by the cbs=80 parameter, and the input
and output block sizes are set with bs=10240. The EBCDIC-to-ASCII conversion
and fixed-to-variable record length conversion are enabled with the conv=ascii,noblock
parameter.
Notice the output record count is smaller than the input record count.
This is due to the padding spaces eliminated from the output file and replaced
with newline characters.
Example 3
Sometimes data arrives from sources in unusual formats. For example,
every time I read a tape made on an SGI machine, the bytes are swapped.
The dd command takes this in stride, swapping the bytes as required. The
ability to use dd in a pipe with rsh means that the tape device on any *nix
system is accessible, given the proper rlogin setup.
Example 3 : Byte Swapping with Remote Access of Magnet Tape
rsh sgi.with.tape dd bs=256b if=/dev/rmt0 conv=swab | tar xvf -
The dd runs on the SGI and swaps the bytes before writing to the tar command
running on the local host.
Example 4
Murphy's Law was postulated long before digital computers, but it seems
it was specifically targeted for them. When you need to read a floppy or
tape, it is the only copy in the universe and you have a deadline past due,
that is when you will have a bad spot on the magnetic media, and your data
will be unreadable. To the rescue comes dd, which can read all the good
data around the bad spot and continue after the error is encountered. Sometimes
this is all that is needed to recover the important data.
Example 4 : Error Handling
dd bs=265b conv=noerror if=/dev/st0 of=/tmp/bad.tape.image
Example 5
The Linux kernel Makefiles use dd to build the boot image. In the Alpha
Makefile /usr/src/linux/arch/alpha/boot/Makefile, the srmboot target issues
the command:
Example 5 : Kernel Image Makefile
dd if=bootimage of=$(BOOTDEV) bs=512 seek=1 skip=1
This skips the first 512 bytes of the input bootimage file (skip=1) and
writes starting at the second sector of the $(BOOTDEV) device (seek=1).
A typical use of dd is to skip executable headers and begin writing in the
middle of a device, skipping volume and partition data. As this can cause
your disk to lose file system data, please test and use these applications
with care.
size="-2">Credits
size="-2">
The dd command has been around since the 1970s, ported to many systems,
rewritten many times, and tested by time as a useful tool. The current Linux
version is GNU dd GNU fileutils 3.12, written by Paul Rubin, David MacKenzie,
and Stuart Kemp, Copyright © 1985, 1990, 1991 Free Software Foundation,
Inc.
GNU dd is found in the fileutils collection, with the current version at
the URL ftp://prep.ai.mit.edu/pub/gnu/fileutils-3.12.tar.gz or a mirror
near you.
Other major versions include SYSV and BSD, with the BSD source version 5.16
4/28/93 derived from software contributed to Berkeley by Keith Muller of
the University of California, San Diego and Lance Visser of Convex Computer
Corporation, Copyright © 1991 The Regents of the University of California.
#!/bin/sh
tar -cvf my.tar $(for i in `cat list`
do
echo $i
done)
exit
This script is backing up to tape using dump command. Logging date and all the messages
to a log file.
#!/bin/sh
#
echo "$DATE"backup.log
filenumber=`/usr/bin/mt stat|/usr/bin/grep "File Number"|/usr/bin/awk '{print $3}'`
echo "Backing up / to tape location: $filenumber"backup.log
/sbin/dump -0ua -f /dev/nrsa0 / &2backup.log
if [ $? -eq 0 ];then
echo "/ backup successful"$HOME/log/backup.log
fi
Copy files ( even complete filesystem) from remote to local system
Note: You must be able to rlogin into the remote machine without a password. To do this add the name of your local machine with your user name in the .rhost file in your home directory on the remote machine.
#!/bin/sh
#
# Copies files from Remote System to the local current directory
#
name=`basename $0`
if [ $# -ne 2 ];then
echo "Usage: $name <remote-system> <dir-to-copy>"
exit
fi
system=$1
dir_to_cp=$2
rsh $system "cd $dir_to_cp; find . -print|cpio -ocB"|dd ibs=5k obs=5k|cpio -iducmvB
Humor
It is jokingly said that dd stands for "destroy disk" or "delete data",
since, being used for low-level operations on hard disks, a small mistake,
such as reversing the if and of parameters, may accidentally
render the entire disk unusable:
Tips
For Linux - How and when to use the dd command
Also Murphy's Law was formulated long before digital computers, but it
seems it was specifically targeted for them. When you need to read a floppy
or tape, and it is the only copy in the universe the effect of a bad spot
on the magnetic media is devastating. dd can read all the good data around
the bad spot and continue after the error is encountered. Sometimes this
is all that is needed to recover the important data.
Copyright © 1996-2009 by Dr. Nikolai Bezroukov.
www.softpanorama.org was
created as a service to the UN Sustainable Development Networking Programme (SDNP)
in the author free time.
Submit
comments This document is an industrial compilation designed and created
exclusively for educational use and is placed under the copyright of the
Open Content License(OPL).
Site uses AdSense so you need to be aware of Google privacy policy. Original materials copyright belong to respective owners. Quotes are made
for educational purposes only in compliance with the fair use doctrine.
Disclaimer:
- The statements, views and opinions presented on
this web page are those of the author and are not endorsed by, nor do they necessarily
reflect, the opinions of the author present and former employers, SDNP or any other
organization the author may be associated with.
- We do not warrant the correctness of the information provided or its
fitness for any purpose
- In no way this site is associated with or endorse cybersquatters
using
the term "softpanorama" with other main or country domains (e.g. softpanorama.com) with
bad faith intent to profit from the goodwill belonging to
someone else.
Last modified:
August 08, 2009