Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-) Softpanorama Search

Unix DD Command and Image Creation

The dd command has been around since the 1970s, ported to many systems, rewritten many times, and proved to be an indispensable Unix tool.  The name is an allusion to mainframe JCL DD statement.  It served as an inspiration to the most important recent backup program for Windows Ghost. The key idea is to get an image of the partition in the form of the file.  It can be used to recovery of deleted files and in computer forensics when the contents of a disk need to be preserved as a byte-exact copy. In the latter case using cp command would not be sufficient because data from deleted files still physically present on a disk but are  visible in the file system naming space.

The GNU clone of dd is part of  fileutils package and was written by Paul Rubin, David MacKenzie, and Stuart Kemp.

dd is also available for Windows as part of Microsoft Unix toolkit (SFU 3.5) and can be used in Windows environment too. At least for reading raw partitions.  It is also part of Cygwin.  There is also natural Windows port of dd .

Unlike most commands, dd uses a keyword=value format for its parameters. This was modeled after IBM System/360 JCL, which had an elaborate DD 'Dataset Definition' specification for I/O devices in JCP language.

A complete listing of all keywords is available via dd --help

The dd command in UNIX and its ported variants for windows (for example the implementation from SFU 3.5 ) can be used to do direct dumps from one device to file or vise versa. It is a useful tool create a disk image or to make a disk from an image but in windows environment it was by and large displaced Ghost and its alternatives.  On Linux Partimage is especially useful alternative to dd. See for example Backup and Restore Linux Partitions Using Partimage -- Debian Admin Partition Image is a Linux/UNIX utility which saves partitions in many formats (see below) to an image file. The image file can be compressed in the GZIP/BZIP2 formats to save disk space, and split into multiple files. Partitions can be saved across the network since version 0.6.0.When using Partimage, the partitions must be unmounted.

For more detailed information about capabilities of Classic Unix dd see Solaris man page:

The dd utility copies the specified input file to the specified output with possible conversions. The standard input and output are used by default. The input and output block sizes may be specified to take advantage of raw physical I/O. Sizes are specified in bytes; a number may end with k, b, or w to specify multiplication by 1024, 512, or 2, respectively. Numbers may also be separated by x to indicate multiplication.

The dd utility reads the input one block at a time, using the specified input block size. dd then processes the block of data actually returned, which could be smaller than the requested block size. dd applies any conversions that have been specified and writes the resulting data to the output in blocks of the specified output block size.

cbs is used only if ascii, asciib, unblock, ebcdic, ebcdicb, ibm, ibmb, or block conversion is specified. In the first two cases, cbs characters are copied into the conversion buffer, any specified character mapping is done, trailing blanks are trimmed, and a NEWLINE is added before sending the line to output. In the last three cases, characters up to NEWLINE are read into the conversion buffer and blanks are added to make up an output record of size cbs. ASCII files are presumed to contain NEWLINE characters. If cbs is unspecified or 0, the ascii, asciib, ebcdic, ebcdicb, ibm, and ibmb options convert the character set without changing the input file's block structure. The unblock and block options become a simple file copy.

After completion, dd reports the number of whole and partial input and output blocks.

OPERANDS

The following operands are supported:

if=file

Specifies the input path. Standard input is the default.

of=file

Specifies the output path. Standard output is the default. If the seek=expr conversion is not also specified, the output file will be truncated before the copy begins, unless conv=notrunc is specified. If seek=expr is specified, but conv=notrunc is not, the effect of the copy will be to preserve the blocks in the output file over which dd seeks, but no other portion of the output file will be preserved. (If the size of the seek plus the size of the input file is less than the previous size of the output file, the output file is shortened by the copy.)

ibs=n

Specifies the input block size in n bytes (default is 512).

obs=n

Specifies the output block size in n bytes (default is 512).

bs=n

Sets both input and output block sizes to n bytes, superseding ibs= and obs=. If no conversion other than sync, noerror, and notrunc is specified, each input block is copied to the output as a single block without aggregating short blocks.

cbs=n

Specifies the conversion block size for block and unblock in bytes by n (default is 0). If cbs= is omitted or given a value of 0, using block or unblock produces unspecified results.

This option is used only if ASCII or EBCDIC conversion is specified. For the ascii and asciib operands, the input is handled as described for the unblock operand except that characters are converted to ASCII before the trailing SPACE characters are deleted. For the ebcdic, ebcdicb, ibm, and ibmb operands, the input is handled as described for the block operand except that the characters are converted to EBCDIC or IBM EBCDIC after the trailing SPACE characters are added.

files=n

Copies and concatenates n input files before terminating (makes sense only where input is a magnetic tape or similar device).

skip=n

Skips n input blocks (using the specified input block size) before starting to copy. On seekable files, the implementation reads the blocks or seeks past them. On non-seekable files, the blocks are read and the data is discarded.

iseek=n

Seeks n blocks from beginning of input file before copying (appropriate for disk files, where skip can be incredibly slow).

oseek=n

Seeks n blocks from beginning of output file before copying.

seek=n

Skips n blocks (using the specified output block size) from beginning of output file before copying. On non-seekable files, existing blocks are read and space from the current end-of-file to the specified offset, if any, is filled with null bytes. On seekable files, the implementation seeks to the specified offset or reads the blocks as described for non-seekable files.

count=n

Copies only n input blocks.

conv=value[,value. . . ]

Where values are comma-separated symbols from the following list:

ascii

Converts EBCDIC to ASCII.

asciib

Converts EBCDIC to ASCII using BSD-compatible character translations.

ebcdic

Converts ASCII to EBCDIC. If converting fixed-length ASCII records without NEWLINEs, sets up a pipeline with dd conv=unblock beforehand.

ebcdicb

Converts ASCII to EBCDIC using BSD-compatible character translations. If converting fixed-length ASCII records without NEWLINEs, sets up a pipeline with dd conv=unblock beforehand.

ibm

Slightly different map of ASCII to EBCDIC. If converting fixed-length ASCII records without NEWLINEs, sets up a pipeline with dd conv=unblock beforehand.

ibmb

Slightly different map of ASCII to EBCDIC using BSD-compatible character translations. If converting fixed-length ASCII records without NEWLINEs, sets up a pipeline with dd conv=unblock beforehand.

The ascii (or asciib), ebcdic (or ebcdicb), and ibm (or ibmb) values are mutually exclusive.

block

Treats the input as a sequence of NEWLINE-terminated or EOF-terminated variable-length records independent of the input block boundaries. Each record is converted to a record with a fixed length specified by the conversion block size. Any NEWLINE character is removed from the input line. SPACE characters are appended to lines that are shorter than their conversion block size to fill the block. Lines that are longer than the conversion block size are truncated to the largest number of characters that will fit into that size. The number of truncated lines is reported.

unblock

Converts fixed-length records to variable length. Reads a number of bytes equal to the conversion block size (or the number of bytes remaining in the input, if less than the conversion block size), delete all trailing SPACE characters, and append a NEWLINE character.

The block and unblock values are mutually exclusive.

lcase

Maps upper-case characters specified by the LC_CTYPE keyword tolower to the corresponding lower-case character. Characters for which no mapping is specified are not modified by this conversion.

ucase

Maps lower-case characters specified by the LC_CTYPE keyword toupper to the corresponding upper-case character. Characters for which no mapping is specified are not modified by this conversion.

The lcase and ucase symbols are mutually exclusive.

swab

Swaps every pair of input bytes. If the current input record is an odd number of bytes, the last byte in the input record is ignored.

noerror

Does not stop processing on an input error. When an input error occurs, a diagnostic message is written on standard error, followed by the current input and output block counts in the same format as used at completion. If the sync conversion is specified, the missing input is replaced with null bytes and processed normally. Otherwise, the input block will be omitted from the output.

notrunc

Does not truncate the output file. Preserves blocks in the output file not explicitly written by this invocation of dd. (See also the preceding of=file operand.)

sync

Pads every input block to the size of the ibs= buffer, appending null bytes. (If either block or unblock is also specified, appends SPACE characters, rather than null bytes.)

If operands other than conv= are specified more than once, the last specified operand=value is used.

For the bs=, cbs=, ibs=, and obs= operands, the application must supply an expression specifying a size in bytes. The expression, expr, can be:

1. a positive decimal number
2. a positive decimal number followed by k, specifying multiplication by 1024
3. a positive decimal number followed by b, specifying multiplication by 512
4. two or more positive decimal numbers (with or without k or b) separated by x, specifying the product of the indicated values.

All of the operands will be processed before any input is read.

USAGE

See largefile(5) for the description of the behavior of dd when encountering files greater than or equal to 2 Gbyte ( 2³¹ bytes).

EXAMPLES

Example 1 Copying from one tape drive to another

The following example copies from tape drive 0 to tape drive 1, using a common historical device naming convention.


example% dd if=/dev/rmt/0h  of=/dev/rmt/1h
 

Example 2 Stripping the first 10 bytes from standard input

The following example strips the first 10 bytes from standard input:

example% dd ibs=10  skip=1

Example 3 Reading a tape into an ASCII file

This example reads an EBCDIC tape blocked ten 80-byte EBCDIC card images per block into the ASCII file x:

example% dd if=/dev/tape of=x ibs=800 cbs=80 conv=ascii,lcase

Example 4 Using conv=sync to write to tape

The following example uses conv=sync when writing to a tape:

example% tar cvf - . | compress | dd obs=1024k of=/dev/rmt/0 conv=sync
 

ENVIRONMENT VARIABLES

See environ(5) for descriptions of the following environment variables that affect the execution of dd: LANG, LC_ALL, LC_CTYPE, LC_MESSAGES, and NLSPATH.

EXIT STATUS

The following exit values are returned:

0

The input file was copied successfully.

>0

An error occurred.

If an input error is detected and the noerror conversion has not been specified, any partial output block will be written to the output file, a diagnostic message will be written, and the copy operation will be discontinued. If some other error is detected, a diagnostic message will be written and the copy operation will be discontinued.

SEE ALSO

cp(1), sed(1), tr(1), attributes(5), environ(5), largefile(5), standards(5)

DIAGNOSTICS

f+p records in(out)

numbers of full and partial blocks read(written)

NOTES

Do not use dd to copy files between file systems having different block sizes.

Using a blocked device to copy a file will result in extra nulls being added to the file to pad the final block to the block boundary.

When dd reads from a pipe, using the ibs=X and obs=Y operands, the output will always be blocked in chunks of size Y. When bs=Z is used, the output blocks will be whatever was available to be read from the pipe at the time.

When using dd to copy files to a tape device, the file size must be a multiple of the device sector size (for example, 512 Kbyte). To copy files of arbitrary size to a tape device, use tar(1) or cpio(1).

For SIGINT, dd writes status information to standard error before exiting. It takes the standard action for all other signals.

 In Linux, the command is invoked as follows: bash# dd if=<image file> of=/dev/fd0. For example

dd if=/dev/fd0 of=floppy.img bs=18k

where /dev/fd0 should be the device for your raw floppy drive (_not_ /dev/floppy) and floppy.img the file you want to save the info to. You can then copy that file to somewhere you can read it with DOS, or maybe even zip it so it will fit onto a floppy ;). You should see something like the following to indicate that the image transfer was successful:

If you see a smaller block count, your image did not transfer correctly. If this is the case, it will usually be accompanied by a disk error.  After you make a disk, make sure to label it according to its contents.

The dd command transfers raw data between devices. For example, dd can be used to create a boot floppy under Linux using the following command:

dd if=boot.img of=/dev/fd0 bs=1440

where

if= is the input file or device
of= is the output file or device
bs= specifies the block size for both input and output files

To change the input and output block sizes to different values, you could use the arguments:

ibs= input block size

or

obs= output block size

Old News ;-)

[Feb 11, 2009] UNIX BASH scripting Create a file with given size - Linux dd command

Feb 11, 2009 |  unstableme.blogspot.com

"The command is:

dd if=/dev/zero of=testfile_10MB bs=10485760 count=1

"1+0 records in
1+0 records out
10485760 bytes (10 MB) copied, 0.312 s, 33.6 MB/s"

[Oct 3, 2008] Red Hat Magazine This isn’t your grandpappy’s dd command

Looks like dd was slightly faster with  128K blocks. Python program might be reused for other purposes

Block Size: 128 Throughput: 62.8 MB/s

Block Size: 256 Throughput: 61.8 MB/s

Block Size: 512 Throughput: 57.1 MB/s

Block Size: 1024 Throughput: 56.5 MB/s
 

We benchmarked the throughput of the disk by running the dd command with various block sizes from 128 KB to 1 MB. (Note: If you want to run the script on your own machine, make sure that the volume you use doesn’t contain any valuable data, because the data will be erased by the dd command. Remember, data loss makes grandpappy mad!)

For the benchmark, we wrote a Python script that uses the commands module to run and capture the output of the dd command. The script also uses the csv module to generate a comma-separated values file so that we can graph the results later. For this example, we chose to graph the results using the Google Chart API.

dd_chart on Flickr - Photo Sharing!

Creating a dd-dcfldd Image Using Automated Image & Restore (AIR) HowtoForge - Linux Howtos and Tutorials

Automated Image & Restore (AIR) is an open source application that provides a GUI front end to the dd/dcfldd (Dataset Definition (dd)) command. AIR is designed to easily create forensic disk/partition images. It supports MD5/SHAx hashes, SCSI tape drives, imaging over a TCP/IP network, splitting images, and detailed session logging. To date, the AIR utility has only been developed for use on Linux distributions. In its simplest form, AIR provides a convenient interface to execute the dd set of commands. It eliminates the risk of "fat fingering" an error in the shell terminal and ultimately makes using the dd command more user-friendly for those who are not as experienced. Please note that using the AIR front end still requires some basic knowledge of how the dd (or dcfldd) commands work.

The dd command has been around for quite a while. It is well known throughout the Unix/Linux community, well documented, and as I can only imagine extensively used. A dd image is a bit by bit image of a source device or file. The uses for dd range from creating and maintaining system backups and restore images to the forensic application of imaging evidence that will be returned to the lab and examined.

This tutorial is not designed to teach the use of the dd command; this is well documented and a simple internet search will yield a plethora of results. Instead, the intent of this mini "how-to" is to introduce users to the AIR front end application, increase overall awareness of the utility, and provide a brief example of creating a dd image using this tool.

DISCLAIMER: I do not claim to be expert at using dd or Automated Image & Restore.

dd_rescue Advantages

Imagine, one of your partitions is crashed, and as there are some hard errors, you don’t want to write to this hard disk any more. Just getting all the data off it and retiring it seems to be suitable. However, you can’t access the files, as the file system is damaged.

Now, you want to copy the whole partition into a file. You burn it on CD-Rom, just to never lose it again. You can setup a loop device, and repair (fsck) it and hopefully are able to mount it.

Copying this partition with normal Un*x tools like cat or dd will fail, as those tools abort on error. dd_rescue instead will try to read and if it fails, it will go on with the next sectors. The output file naturally will have holes in it, of course. You can write a log file, to see, where all these errors are located.

The data rate drops very low, when errors are encountered. If you interrupt the process of copying, you don’t lose anything. You can just continue at any position later. The output file will just be filled in further and not truncated as with other Un*x tools.

If you have one spot of bad sectors within the partition, it might be a good idea, to approach this spot from both sides. Reverse direction copy is your friend.

The two block sizes are a performance optimization. Large block sizes result in superior performance, but in case of errors, you want to try to salvage every single sector. So hardbs is best be set to the hardware sector size (most often 512 bytes) and softbs to a large value, such as the default 16k.

Install dd_rescue in Debian

Install ddrescue using the following command

#apt-get install ddrescue

Install ddrescue in Ubuntu

sudo apt-get install ddrescue

This will complete the installation

ddrescue Syntax

dd_rescue [options] infile outfile

Now we will see how to use ddrescue under damaged disk

If you have a damaged hard disk /dev/sda1 and you have an empty space hard disk /dev/sda2 Now if you want to copy data from /dev/sda1 to /dev/sda2 use the following commnd

# dd_rescue /dev/sda1 /dev/sda2/backup.img

If you are using ubuntu linux use the following command

sudo dd_rescue /dev/sda1 /dev/sda2/backup.img

This copies an image of /dev/sda1 to sda2

Now you need to check the backup image consistency this will check for is there any problems with this image.

#fsck -y /dev/sda2/backup.img

If you are using ubuntu linux use the following command

sudo fsck -y /dev/sda2/backup.img

After finishing this checking you need to mount your disk image in to your other hard disk

#mount /dev/sda2/backup.img /mnt/recoverydata

If you are using ubuntu linux use the following command

sudo mount /dev/sda2/backup.img /mnt/recoverydata

This will mount all the data from the backup.img under /mnt/recoverydata now you can try to access the data it should work without any problem.

Restore image

If you want to restore this image use the following command

#dd_rescue /dev/sda2/backup.img /dev/sda1

If you are using ubuntu linux use the following command

sudo dd_rescue /dev/sda2/backup.img /dev/sda1

Copy Disk Image to remote machine using SSH

If you want to copy your disk image to remote machine over ssh you need to use the following command

#dd_rescue /dev/sda1 - | ssh username@machineip ‘cat /datarecovery/backup.img’

If you are using ubuntu linux use the following command

sudo dd_rescue /dev/sda1 - | ssh username@machineip ‘cat /datarecovery/backup.img’

This will be prompetd for password of the username you have menctioned in the above command after entering the password dd_rescue strats copying obviously it will take some time to copy over the network.

Possible Error

If you see the following error at the time of copying you can ignore this error

dd_rescue: (warning): output file is not seekable!
dd_rescue: (warning): Illegal seek

If you want to take this image in compressed format you can use the following command format

#tar zcvf - /dev/sda1 | ssh username@machineip ‘cat@@>/tmp /datarecovery/backup.tar.gz’

If you are using ubuntu linux use the following command

sudo tar zcvf - /dev/sda1 | ssh username@machineip ‘cat@@>/tmp /datarecovery/backup.tar.gz’

If you want to know more available options check dd_rescue man page

recover-data-from-a-dead-hard-drive-using-ddrescue

Like dd, dd_rescue does copy data from one file or block device to another.dd_rescue is a tool to help you to save data from crashed partition. It tries to read and if it fails, it will go on with the next sectors where tools like dd will fail. If the copying process is interrupted by the user it is possible to continue at any position later. It can copy backwards.

Dd - LQWiki

Creating a hard drive backup directly to another hard drive

# dd if=/dev/hda of=/dev/sda conv=noerror,sync bs=4k

This command is used often to create a backup of a drive (/dev/hda) directly to another hard drive (/dev/sda). (The device name /dev/hda is typical of an IDE hard drive, the device /dev/sda is typical of a USB disk.) This works only if the hard drive has enough storage to accommodate the source drive's filesystem. The advantage of this is that you do not have to mount the hard drive to make a backup and the only reference to hda is in /dev and in the command which is usually in a script in cron.

The option "bs=4k" is used to specify the block size used in the copy. The default for the dd command is 512 bytes: use of this small block size can result in significantly slower copying. However, the tradeoff with larger block sizes is that when an error is encountered, the remainder of the block is filled with zero-bytes. So if you increase your block size when copying a failing device, you'll lose more data but also spend less time trying to read broken sectors. Tools like dd_rescue and dd_rhelp can provide a more flexible solution in such cases, combining the speed of a large block size for the regions without errors with finer-grained block-copies for regions with errors.

Creating a hard drive backup image

# dd if=/dev/hda | gzip > /mnt/hdb1/system_drive_backup.img.gz

Here dd is making an image of the first harddrive, and piping it through the gzip compression program. The compressed image is then placed in a file on a seperate drive. To reverse the process:

# gzip -dc /mnt/hdb1/system_drive_backup.img.gz | dd of=/dev/hda

Here, gzip is decompressing (the -d switch) the file, sending the results to stdout (the -c switch), which are piped to dd, and then written to /dev/hda.

Copy floppy

# dd if=/dev/fd0 of=/tmp/floppy.img bs=10240

That will copy the contents of the floppy to a file. Then, to put the image onto a new floppy, swap "if" and "of" params.

# dd if=/tmp/floppy.img of=/dev/fd0 bs=10240

Backing up your Master Boot Record (MBR).

You should do this before you edit your partition table so that you can put it back if you mess things up.

# dd if=/dev/hda of=/root/hda.boot.mbr bs=512 count=1

If things mess up, you can boot with Knoppix, mount the partition containing /root (hda1 in this example) and put back the MBR with the command:

# dd if=/mnt/hda1/root/hda.boot.mbr of=/dev/hda bs=512 count=1

Obviously, if you have a GPT system (like the intel mac for instance) this will need some adjustment.

see: http://forum.onmac.net/showthread.php?t=136

You can backup only the MBR and exclude the partition table with the command:

 # dd if=/dev/hda of=/root/hda.mbr.noparttab bs=446 count=1

Getting around file size limitations using split

When making images, it's quite easy to run up against various file size limitations. One way to work around a given file size limitation is to use the split command.

# dd if=/dev/hda1 | gzip -c | split -b 2000m - /mnt/hdc1/backup.img.gz.

1. This example is using dd to take an image of the first partition on the first harddrive.
2. The results are passed through to gzip for compression
   • The -c option switch is used to output the result to stdout.
3. The compressed image is then piped to the split tool
   • The -b 2000m switch tells split how big to make the individual files. You can use k and m to tell switch kilobytes and megabytes (this option uses bytes by default).
   • The - option tells split to read from stdin. Otherwise, split would interpret the /mnt/hdc1... as the file to be split.
   • The /mnt/hdc1... is the prefix for the created files. Split will create files named backup.img.gz.aa, backup.img.gz.ab, etc.

To restore the multi-file backup, do the following:

# cat /mnt/hdc1/backup.img.gz.* | gzip -dc | dd of=/dev/hda1

1. Cat recombines contents of the compressed and split image files to stdout, in order.
2. Results are piped through gzip for decompression.
3. And are then written to the first partition of the hard drive with dd.

Creating empty disk images

To create an empty disk image, to be used as the disk for an emulator for example, one can get data from /dev/zero. To create a 10mb image:

$ dd if=/dev/zero of=myimage bs=1024 count=10240

A clever alternative is:

$ dd of=myimage bs=1024 count=0 seek=10240

Here we don't write anything, not even zeroes, we just seek 10mb into the file and close it. The result is a sparse file that is implicitly full of 10mb of zeroes, but that takes no disk space. ls -l will report 10mb, while du and df will report 0. When the file is written to, either as an emulator disk or a loopback device, Linux will allocate disk space for the data. ls will still show 10mb, while du will gradually approach 10mb.

For swap images, where it's more important to reserve the data than to save disk space, a non-sparse file is better.

[Sep 12, 2008] Tips For Linux - How and when to use the dd command

this article, Sam Chessman explains the use of the dd command with a lot of useful examples. This article is not aimed at absolute beginners. Once you are familiar with the basics of Linux, you would be in a better position to use the dd command.

The ' dd ' command is one of the original Unix utilities and should be in everyone's tool box. It can strip headers, extract parts of binary files and write into the middle of floppy disks; it is used by the Linux kernel Makefiles to make boot images. It can be used to copy and convert magnetic tape formats, convert between ASCII and EBCDIC, swap bytes, and force to upper and lowercase.

For blocked I/O, the dd command has no competition in the standard tool set. One could write a custom utility to do specific I/O or formatting but, as dd is already available almost everywhere, it makes sense to use it.

Like most well-behaved commands, dd reads from its standard input and writes to its standard output, unless a command line specification has been given. This allows dd to be used in pipes, and remotely with the rsh remote shell command.

Unlike most commands, dd uses a keyword=value format for its parameters. This was reputedly modeled after IBM System/360 JCL, which had an elaborate DD 'Dataset Definition' specification for I/O devices. A complete listing of all keywords is available from GNU dd with

$ dd --help

Some people believe dd means ``Destroy Disk'' or ``Delete Data'' because if it is misused, a partition or output file can be trashed very quickly. Since dd is the tool used to write disk headers, boot records, and similar system data areas, misuse of dd has probably trashed many hard disks and file systems.

In essence, dd copies and optionally converts data. It uses an input buffer, conversion buffer if conversion is specified, and an output buffer. Reads are issued to the input file or device for the size of the input buffer, optional conversions are applied, and writes are issued for the size of the output buffer. This allows I/O requests to be tailored to the requirements of a task. Output to standard error reports the number of full and short blocks read and written.

Example 1

A typical task for dd is copying a floppy disk. As the common geometry of a 3.5" floppy is 18 sectors per track, two heads and 80 cylinders, an optimized dd command to read a floppy is:

Example 1-a : Copying from a 3.5" floppy

dd bs=2x80x18b if=/dev/fd0 of=/tmp/floppy.image
1+0 records in
1+0 records out

The 18b specifies 18 sectors of 512 bytes, the 2x multiplies the sector size by the number of heads, and the 80x is for the cylinders--a total of 1474560 bytes. This issues a single 1474560-byte read request to /dev/fd0 and a single 1474560 write request to /tmp/floppy.image, whereas a corresponding cp command

cp /dev/fd0 /tmp/floppy.image

issues 360 reads and writes of 4096 bytes. While this may seem insignificant on a 1.44MB file, when larger amounts of data are involved, reducing the number of system calls and improving performance can be significant.

This example also shows the factor capability in the GNU dd number specification. This has been around since before the Programmers Work Bench and, while not documented in the GNU dd man page, is present in the source and works just fine, thank you.

To finish copying a floppy, the original needs to be ejected, a new diskette inserted, and another dd command issued to write to the diskette:

Example 1-b : Copying to a 3.5" floppy
dd bs=2x80x18b < /tmp/floppy.image > /dev/fd0
1+0 records in
1+0 records out

Here is shown the stdin/stdout usage, in which respect dd is like most other utilities.

 

Example 2

 

The original need for dd came with the 1/2" tapes used to exchange data with other systems and boot and install Unix on the PDP/11. Those days are gone, but the 9-track format lives. To access the venerable 9-track, 1/2" tape, dd is superior. With modern SCSI tape devices, blocking and unblocking are no longer a necessity, as the hardware reads and writes 512-byte data blocks.

 

However, the 9-track 1/2" tape format allows for variable length blocking and can be impossible to read with the cp command. The dd command allows for the exact specification of input and output block sizes, and can even read variable length block sizes, by specifying an input buffer size larger than any of the blocks on the tape. Short blocks are read, and dd happily copies those to the output file without complaint, simply reporting on the number of complete and short blocks encountered.

 

Then there are the EBCDIC datasets transferred from such systems as MVS, which are almost always 80-character blank-padded Hollerith Card Images! No problem for dd, which will convert these to newline-terminated variable record length ASCII. Making the format is just as easy and dd again is the right tool for the job.

Example 2 : Converting EBCDIC 80-character fixed-length record to ASCII variable-length newline-terminated record
dd bs=10240 cbs=80 conv=ascii,unblock if=/dev/st0 of=ascii.out
40+0 records in
38+1 records out

 

The fixed record length is specified by the cbs=80 parameter, and the input and output block sizes are set with bs=10240. The EBCDIC-to-ASCII conversion and fixed-to-variable record length conversion are enabled with the conv=ascii,noblock parameter.

 

Notice the output record count is smaller than the input record count. This is due to the padding spaces eliminated from the output file and replaced with newline characters.

 

Example 3

 

Sometimes data arrives from sources in unusual formats. For example, every time I read a tape made on an SGI machine, the bytes are swapped. The dd command takes this in stride, swapping the bytes as required. The ability to use dd in a pipe with rsh means that the tape device on any *nix system is accessible, given the proper rlogin setup.

Example 3 : Byte Swapping with Remote Access of Magnet Tape
rsh sgi.with.tape dd bs=256b if=/dev/rmt0 conv=swab | tar xvf -

The dd runs on the SGI and swaps the bytes before writing to the tar command running on the local host.

 

Example 4

 

Murphy's Law was postulated long before digital computers, but it seems it was specifically targeted for them. When you need to read a floppy or tape, it is the only copy in the universe and you have a deadline past due, that is when you will have a bad spot on the magnetic media, and your data will be unreadable. To the rescue comes dd, which can read all the good data around the bad spot and continue after the error is encountered. Sometimes this is all that is needed to recover the important data.

Example 4 : Error Handling
dd bs=265b conv=noerror if=/dev/st0 of=/tmp/bad.tape.image

 

Example 5

 

The Linux kernel Makefiles use dd to build the boot image. In the Alpha Makefile /usr/src/linux/arch/alpha/boot/Makefile, the srmboot target issues the command:

Example 5 : Kernel Image Makefile
dd if=bootimage of=$(BOOTDEV) bs=512 seek=1 skip=1

This skips the first 512 bytes of the input bootimage file (skip=1) and writes starting at the second sector of the $(BOOTDEV) device (seek=1). A typical use of dd is to skip executable headers and begin writing in the middle of a device, skipping volume and partition data. As this can cause your disk to lose file system data, please test and use these applications with care.

[Feb 15, 2007] Recover Data from a Damaged hard disk using dd_rhelp -- Debian Admin

December 19, 2006 (debianadmin.com) by Admin @ 4:59 pm. Filed under Backup

dd_rhelp is a bash script that handles a very usefull program written in C which is called dd_rescue, it roughly act as the dd linux command with the characteristic to NOT stop when it falls on read/write errors.

This makes dd_rescue the best tool for recovering hard drive having bad sectors. In short, it’ll use dd_rescue on your entire disc, but will try to gather the maximum valid data before trying for ages on badsectors. So if you leave dd_rhelp work for infinite time, it’ll have the same effect as a simple dd_rescue. But because you might not have this infinite time , dd_rhelp will jump over bad sectors and rescue valid data. In the long run, it’ll parse all your device with dd_rescue.

You can Ctrl-C it whenever you want, and rerun-it at will, it’ll resume it’s job as it depends on the log files dd_rescue creates.In addition, progress will be shown in a ASCII picture of your device beeing rescued.

Why use dd_rhelp and not dd_rescue ?

dd_rhelp uses dd_rescue to compute a recovery path through the device that will focus on valid data recovering. This recovery path will go through all the device, exactly as dd_rescue could do it on its own without any path. This means that dd_rhelp will save you time ONLY IF YOU INTEND TO CANCEL ITS JOB BEFORE THE END of a full recovery.

Because a considerable amount of time is taken to try to rescue badsectors. This amount of time can be mesured in days,
month, years, depending on your device capacity and its defectiveness. You might not want to spend this time knowing that 99 percent of this time will be taken to look at badsector and won’t lead to any more data recovering.

dd_rhelp shifts this useless waiting time to the end. Using dd_rescue strait throughout your device make your waiting time dependent on the badsector distribution.

Install dd_rhelp

First you need to download latest version from here

Preparing your system

#apt-get install make gcc g++ ddrescue

If you are using ubuntu linux use the following command

sudo apt-get install make gcc g++ ddrescue

Download dd_rhelp using the follwoing command

wget http://www.kalysto.org/pkg/dd_rhelp-0.0.6.tar.gz

Now you have dd_rhelp-0.0.6.tar.gz file

Extract this file using the following command

tar xzvf dd_rhelp-0.0.6.tar.gz

cd dd_rhelp-0.0.6

Run the following commands

# ./configure

# make

# make install

If you are using ubuntu linux use the following command

sudo ./configure

sudo make

sudo make install

This will complete the installation and this will install program under /usr/local/bin/dd_rhelp

dd_rhelp Syntax

dd_rhelp {filename|device} {output-file} [{info}]
or dd_rhelp --help
or dd_rhelp --version

Now we will see how to use ddrescue under damaged disk

If you have a damaged hard disk /dev/sda1 and you have an empty space hard disk /dev/sda2 Now if you want to copy data from /dev/sda1 to /dev/sda2 use the following command

#dd_rhelp /dev/sda1 /dev/sda2/backup.img

If you are using ubuntu linux use the following command

sudo dd_rhelp /dev/sda1 /dev/sda2/backup.img

Output looks like below

/usr/bin/tail: Warning: “+number” syntax is deprecated, please use “-n +number”
=== launched via ‘dd_rhelp’ at 0k, 0 >>> ===
dd_rescue: (info): ipos: 104384.0k, opos: 104384.0k, xferd: 104384.0k
errs: 0, errxfer: 0.0k, succxfer: 104384.0k
+curr.rate: 10178kB/s, avg.rate: 8532kB/s, avg.load: 31.3%
dd_rescue: (info): /dev/sda2 (104391.0k): EOF
Summary for /dev/sda2 -> backup.img:
dd_rescue: (info): ipos: 104391.0k, opos: 104391.0k, xferd: 104391.0k
errs: 0, errxfer: 0.0k, succxfer: 104391.0k
+curr.rate: 355kB/s, avg.rate: 8519kB/s, avg.load: 31.3%
/usr/bin/tail: Warning: “+number” syntax is deprecated, please use “-n +number”

This copies an image of /dev/sda1 to sda2

Now you need to check the backup image consistency this will check for is there any problems with this image.

#fsck -y /dev/sda2/backup.img

If you are using ubuntu linux use the following command

sudo fsck -y /dev/sda2/backup.img

After finishing this checking you need to mount your disk image in to your other hard disk

#mount /dev/sda2/backup.img /mnt/recoverydata

If you are using ubuntu linux use the following command

sudo mount /dev/sda2/backup.img /mnt/recoverydata

This will mount all the data from the backup.img under /mnt/recoverydata now you can try to access the data it should
work without any problem.

Restore image

If you want to restore this image use the following command

#dd_rhelp /dev/sda2/backup.img /dev/sda1

If you are using ubuntu linux use the following command

sudo dd_rhelp /dev/sda2/backup.img /dev/sda1

Copy Disk Image to remote machine using SSH

If you want to copy your disk image to remote machine over ssh you need to use the following command

#dd_rhelp /dev/sda1 - | ssh username@machineip ‘cat /datarecovery/backup.img’

If you are using ubuntu linux use the following command

sudo dd_rhelp /dev/sda1 - | ssh username@machineip ‘cat /datarecovery/backup.img’

This will be prompetd for password of the username you have menctioned in the above command after entering the password

dd_rhelp strats copying obviously it will take some time to copy over the network.

[Feb 14, 2007] linux-noob.com-forums How and when to use the dd command

Example 1


A typical task for dd is copying a floppy disk. As the common geometry of a 3.5" floppy is 18 sectors per track, two heads and 80 cylinders, an optimized dd command to read a floppy is:

Example 1-a : Copying from a 3.5" floppy

dd bs=2x80x18b if=/dev/fd0 of=/tmp/floppy.image
1+0 records in
1+0 records out

The 18b specifies 18 sectors of 512 bytes, the 2x multiplies the sector size by the number of heads, and the 80x is for the cylinders--a total of 1474560 bytes. This issues a single 1474560-byte read request to /dev/fd0 and a single 1474560 write request to /tmp/floppy.image, whereas a corresponding cp command

cp /dev/fd0 /tmp/floppy.image

issues 360 reads and writes of 4096 bytes. While this may seem insignificant on a 1.44MB file, when larger amounts of data are involved, reducing the number of system calls and improving performance can be significant.

This example also shows the factor capability in the GNU dd number specification. This has been around since before the Programmers Work Bench and, while not documented in the GNU dd man page, is present in the source and works just fine, thank you.


To finish copying a floppy, the original needs to be ejected, a new diskette inserted, and another dd command issued to write to the diskette:

Example 1-b : Copying to a 3.5" floppy
dd bs=2x80x18b < /tmp/floppy.image > /dev/fd0
1+0 records in
1+0 records out

Here is shown the stdin/stdout usage, in which respect dd is like most other utilities.


Example 2


The original need for dd came with the 1/2" tapes used to exchange data with other systems and boot and install Unix on the PDP/11. Those days are gone, but the 9-track format lives. To access the venerable 9-track, 1/2" tape, dd is superior. With modern SCSI tape devices, blocking and unblocking are no longer a necessity, as the hardware reads and writes 512-byte data blocks.


However, the 9-track 1/2" tape format allows for variable length blocking and can be impossible to read with the cp command. The dd command allows for the exact specification of input and output block sizes, and can even read variable length block sizes, by specifying an input buffer size larger than any of the blocks on the tape. Short blocks are read, and dd happily copies those to the output file without complaint, simply reporting on the number of complete and short blocks encountered.


Then there are the EBCDIC datasets transferred from such systems as MVS, which are almost always 80-character blank-padded Hollerith Card Images! No problem for dd, which will convert these to newline-terminated variable record length ASCII. Making the format is just as easy and dd again is the right tool for the job.

Example 2 : Converting EBCDIC 80-character fixed-length record to ASCII variable-length newline-terminated record
dd bs=10240 cbs=80 conv=ascii,unblock if=/dev/st0 of=ascii.out
40+0 records in
38+1 records out

The fixed record length is specified by the cbs=80 parameter, and the input and output block sizes are set with bs=10240. The EBCDIC-to-ASCII conversion and fixed-to-variable record length conversion are enabled with the conv=ascii,noblock parameter.

Notice the output record count is smaller than the input record count. This is due to the padding spaces eliminated from the output file and replaced with newline characters.

Example 3


Sometimes data arrives from sources in unusual formats. For example, every time I read a tape made on an SGI machine, the bytes are swapped. The dd command takes this in stride, swapping the bytes as required. The ability to use dd in a pipe with rsh means that the tape device on any *nix system is accessible, given the proper rlogin setup.

Example 3 : Byte Swapping with Remote Access of Magnet Tape
rsh sgi.with.tape dd bs=256b if=/dev/rmt0 conv=swab | tar xvf -

The dd runs on the SGI and swaps the bytes before writing to the tar command running on the local host.

Example 4

Murphy's Law was postulated long before digital computers, but it seems it was specifically targeted for them. When you need to read a floppy or tape, it is the only copy in the universe and you have a deadline past due, that is when you will have a bad spot on the magnetic media, and your data will be unreadable. To the rescue comes dd, which can read all the good data around the bad spot and continue after the error is encountered. Sometimes this is all that is needed to recover the important data.

Example 4 : Error Handling
dd bs=265b conv=noerror if=/dev/st0 of=/tmp/bad.tape.image

Example 5

The Linux kernel Makefiles use dd to build the boot image. In the Alpha Makefile /usr/src/linux/arch/alpha/boot/Makefile, the srmboot target issues the command:

Example 5 : Kernel Image Makefile
dd if=bootimage of=$(BOOTDEV) bs=512 seek=1 skip=1

This skips the first 512 bytes of the input bootimage file (skip=1) and writes starting at the second sector of the $(BOOTDEV) device (seek=1). A typical use of dd is to skip executable headers and begin writing in the middle of a device, skipping volume and partition data. As this can cause your disk to lose file system data, please test and use these applications with care.BigAdmin Submitted Article Backup and Recovery Utilities for the Solaris OS

dd -- Convert and Copy a File

The dd command is most commonly used to copy a complete file system to another file system or to copy a hard disk drive to another disk drive. dd can also be used to copy a file system to tape, and vice versa. dd is a relatively quick copy tool: It creates an exact copy (byte for byte) as it transfers the data. Several options are used with dd to specify buffer sizes, block sizes, and data conversions. The basic syntax for dd is:

dd <inputfile>= <outputfile>= options

The following example copies the entire contents of c0t1d0s2 to a second disk, c0t4d0s2, using a block size of 128. This works great if you have a spare disk available and want to have a backup disk ready to swap out in case of a disaster. Slice 2 is specified in this example because it represents the entire disk in the Solaris OS.

% dd if=/dev/rdsk/c0t1d0s2 of=/dev/rdsk/c0t4d0s2 bs=128

To copy the contents of one tape device to another, use the following example. (Note: You must have two drives available.)

% dd if=/dev/rmt/#1drive of=/dev/rmt/#2drive

The man pages on dd give many other options available for use. I suggest viewing the man pages for dd and also reading the Notes section. There you will find some limitations to dd, as well as a reminder to use the raw character device when copying data from disk devices.

Network Security Blog Using DD to clone a HD

Thomas Rude - DD and Computer Forensics - He's Worth a Deuce!

For quick review, let's create an image based on the following:
     We have a 2GB hard disk we want to image. We connect the 2GB disk as a slave in our PC. Upon boot, we mount the 2GB disk read only. We are now ready to image! Using DD, how 'bout:

      dd if=/dev/hdb of=/images/hdbApril42001.img

Linux Magazine October 2004 POWER TOOLS Performing Data Surgery Data Dumping with dd

dd does low-level data transfer, byte-by-byte or block-by-block, with adjustable block sizes. It can also skip specified numbers of blocks in the input and/or output files, as well as converting data formats. All of those are handy for working with magnetic tape and disks. But it's also useful for many types of data transfers.

By default, dd reads the standard input and writes to the standard output. Input and output filenames, and other options too, are given in an unusual syntax without leading dash (-) characters.

For instance, to read a floppy disk and write its image to a file, you could type:

$ dd if=/dev/fd0 of=dosboot.img
2880+0 records in
2880+0 records out
$ ls -l dosboot.img
-rw-rw-r- ... 1474560 Nov 2 12:59 dosboot.img

The dd command line says, "Reading from the input file /dev/fd0, write all of the data to the file dosboot.img." dd doesn't try to find lines of data or individual files on the disk; it does a binary copy of the bytes from first to last. dd always tells you (on the standard error) how many times it read and wrote data. Above, it read 2,880 512-byte blocks. If you don't want to see this information -- or any error messages, either -- you can redirect dd's standard error to the Linux "bit bucket," /dev/null, by adding the Bourne shell operator 2>/dev/ null to the command line.

It's more efficient to specify a larger block size so the device drivers do a single read and write. There are lots of other options, and many of them start with conv=, like conv=unblock to replace trailing spaces in a block with a newline, and conv= swap to swap pairs of input bytes (which is needed with some tapes written on other types of hardware). But we'll leave that sort of optimization to you and the dd man page. Let's look at some less-obvious uses of this handy utility.

Stupid dd Tricks

Need a file with 100 arbitrary bytes -- for testing, for instance? The Linux device /dev/urandom (available since Linux 1.3.30) can supply as many pseudo-random bytes as you can read from it. To get just 100 bytes, set a block size of 1 byte with bs=1 and tell dd to stop after copying 100 "blocks" (here, that's 100 bytes):

$ dd if=/dev/urandom of=myrand bs=1 count=100

What's in that myrand file? The od utility can show you. (See the sidebar "What's In That File?")

If you need more-random data, try /dev/random instead. Reading data from /dev/random can take some time, though, as the random(4) man page explains. When you read from /dev /random, set a block size of 1.

Another use for dd is for "wiping" a text file before you delete it. Simply removing a Linux file (with rm, for instance) only deletes the inode that points to the data. A cracker with root access might read the raw disk (with dd!) and find the "deleted" file. We can use dd to write random data over the file before deleting it. Normally dd truncates a file before writing, so use conv=notrunc to make it write over the existing data. Set bs to the file size and count to 1. For example:

% ls -l afile
-rw------- ... 3769 Nov  2 13:41 afile
% dd if=/dev/urandom of=afile \
  bs=3769 count=1 conv=notrunc
1+0 records in
1+0 records out
% rm afile

If you want to, you can repeat the "wiping" command several times with the C shell repeat command, the Z shell repeat loop, or simply use the history operator !!.

Moving your data to a backup device

SUMMARY How to build a Solaris bootable CDROM

One would think this would be a simple task.  Half the commerical packages proclaim that they can do it.  But alas, for me, none of them worked.

The task was to create a system installer that would boot a sun, load up the appropriate system images and tools required for a server in our environment.  Yes, this is a perfect job for jumpstart, but the systems would be distributed across the state and not connected via a fast enough pipe to use jumpstart effectivly.  The other option was to build external disks with the proper images on them and boot from there.  The cost of producing a few hundrad of these was prohibitive, thus the CDROM approach was taken.

Of the few responses I received from Sun Managers readers, all of them basicly said 'Its easy to clone the install disc!' or 'it just cant be done!'

Well.  It is possible.

First, a breakdown of the process.  When the command > boot cdrom < is givin to openboot the system actually looks at slice 2 to 5 for its bootblks depending on the machine type.  A sun4c is slice 2 to an sun4u at slice 5.  These bootblks redirect the system to slice 1 to load its kernel.  The root partition is also stored in slice 1 and slice 0 is usr as well as packages. All slices save 0 are ufs, and 0 is hsfs.

This brings us to a small problem point.  First we must boot from a UFS partition, as the bootblks all require that.  Second, a CDROM has no label by default, thus its kinda hard to make partitions. And third, even if we do flush a UFS filesystem image off to the cdrom, the geometries will be all wrong, unless your staging disk just happens to have the same
geometries as the CDROM.

My solution is probally not the best.  What would be required is an application that simply converts the UFS geometries from the staging disk uses to those used by the cdrom.  I didnt really have time to create this so I did it as follows.  Im sure I will get lots of flak for this
solution, but it does work.

1) Use dd to grab the first cylinder off the solaris boot cdrom.  This contains a valid disk label and VTOC for the cdrom.  Once this is created, our limitation is that we must work within the defines of this VTOC. You should be able to use prtvtoc on the cdrom to get a look at this VTOC, but this dosnt work if volmgr is running.

dd if=/dev/dsk/c0t6d0s0 of=cdrom.vtoc bs=512 count=1

2) Now use dd to grab the UFS slices from 1 to 5

for slice in 1 2 3 4 5
do
dd if=/dev/dsk/c0t6d0${slice} of=cdrom.s${slice}
done

3) Create a staging area and copy the parts of the usr filesystem (slice 0) off the cdrom into it.  I started by copying cdrom:/export to it and then trimmed out the parts I didnt need like X and openwindows.

4) Add in all the things you need for your disc.  For me, this ment a shell script that automated the build process, and images of all the data I wanted to move out.  Make sure you donot go beyond te size of the slice 0 on the cdrom you started with.  prtvtoc will should you this, if you cant get prtvtoc to work on the cdrom (sometimes it does, sometimes it dosnt) then use

> dd if=/dev/dsk/c0t6d0s0 of=/dev/null bs=512 <

Recordthe exact size of the partition in blocks as you will need it later.

5) Patch the slice 1 image (cdrom.s1) to start your custom application rather then the suninstall.  This can be done by finding the break point you wish to use in the file cdrom:/sbin/sysconfig, selecting a unique set of chars in this file ( I used the string #*****  S30sysid.net ) and then searching via a hexeditor or emacs in bin mode for that string.

Then find a comment line, change the first # and chars after to point to your script, then add a # line after.  Example, I patched my image so the line

#*****  S30sysid.net

became

exec build #ysid.net

I then put a script in my staging area into the /usr/bin dir. (the staging area will be made into slice 0 which is hsfs)

6) Using mkisofs or the tools that came with your cdrom burning package (HyCD worked well, as well as Gear) turn your staging area into a hsfs filesystem image, making sure that symbloic links are unmodified. HyCD required changing a default option that would have modified all the links.

7) Using dd, throw away the first block of this image.

dd if=image of=image.data bs=512 skip=1

8) Subtract the block count of slice 0 from the solaris cdrom from the image size above, add one to the answer.  Say dd reported for the above step that your image was 500000 blocks, and your solaris cdrom has a size of 787840 for slice 0

  787840 - ( 500000 + 1 ) = 287839

9) Feed this number into dd reading from /dev/zero to build a pad file.

dd if=/dev/zero of=image.pad bs=512 count=287839

10) Cat all of the image files together with the VTOC and the UFS slices.

cat cdrom.vtoc > image
cat image.data >> image
cat image.pad >> image
cat cdrom.s1 >> image
..
cat cdrom.s5 >> image

11) Burn this image to the cdrom drive using cdrecord, HyCD, Gear, etc

12) put it in a machine and test boot it and make sure it does what you need.

Thats about it.  We have to go through the gyrations because UFS is geometry sensitive.  We cannot take a image of a hard disk built UFS as all the cylinder groups will be off. The RIGHT way of doing this would be to build a tool that did the conversion for you and built an image up for burning.  But that requires a bit more work, this gets the job done with a
minimal amount of strain.  I am working on the above mentioned tool, but so far my progress has been to create lots of coasters. Sun has a tool called MakeDisc that does this job, or something similar, but I do not have a copy of it, so had to develop a method, while under a big gun, to do it in a very compressed amount of time.  The only real limitation this
has is you can only store about 400 megs of information, of which around 70 or so are needed by usr in the hsfs partition.  Plus you do not have to go through the pain of figuring out which parts of the system need to be moved to the memfs filesystem (cdrom is readonly, so dev, devices, etc need to be linked to /tmp)

If you wish to flame the procedure and tell me that I did it really stupid, please correct me!  I would willing stop development of my tools to do same and do it the right way :)  But my original question on how to do this went unanswered.

Have Fun
James

Recommended Links

---

In case of broken links please try to use Google search. If you find the page please notify us about new location

• dd (Unix) - Wikipedia, the free encyclopedia
• Marc's realm - Creating and using disk images mini-HOWTO
• test results for the disk imaging tool dd from the U.S. National Insitute of Justice
• link collection about forensics with opensource tools
• File system and disk images for testing digital forensic analysis and acquisition tools by Brian Carrier
• Using dd (instead of cdrecord and dvdrecord) to write bootable ISO images to DVD-RAM disks - CCP14 Linux Internet Server - CCP14 Web-Config Administration Information - CCP14 Homepage - Single Crystal and Powder Diffraction
• Dd - LQWiki
• Tips For Linux - How and when to use the dd command
• How to use floppy disks with Solaris
• BigAdmin Submitted Article- Backup and Recovery Utilities
• BigAdmin - Solaris (x86 Platform Edition) DCA Boot Diskette
• TAR DD or CPIO command
• Solaris Systems Administrator’s Guide, Second EditionAdministering Devices Try Copying Data from the Disk
• Burning a bootable CDRom for Solaris

Reference

docs.sun.com man pages section 1M System Administration Commands - dd

The Source of All Tape Knowledge

dd The name dd stands for "copy and convert". Don't see it? Well, "cc" was already taken for the C compiler, so the author chose the next letter in the alphabet. The syntax has sort of an evil, JCL-like quality to it. According to The Jargon File, the interface was a prank.

Using dd

Most people use dd incorrectly. This is because dd is a piece of junk that should be replaced.

For example, one common misusage of dd is to try and get 64k blocks written to the tape with this command:

tar -cf - args... | dd of=/dev/rmt8 bs=64k

This won't work because (as you will see below), the bs argument gives you only one buffer. The dd process will attempt to read 64k chunks from the pipe into this buffer, but will only receive a maximum of PIPE_BUF bytes (usually 4 or 8k). It will then write this buffer out to the tape as a single record (it will not pad this block to 64k, fortunately).

GNU dd

This data taken from GNU fileutils 3.12

When dd starts up, it parses all the arguments on the command line in order. Note that the bs= argument will override any previous ibs= or obs= arguments. If neither the obs nor the ibs argument is presented, and bs is given, and no character-translation conversions are performed, then only one buffer will be used (more on this later). In all other cases, two buffers (input and output) are used. If you don't specify any *bs args, ibs and obs default to 512.

Next, based on the translations that you have specified, dd builds a translation table. This table is a 256 entry array, specifying a character-by-character mapping that is the composite of all specified translations. The actual order of application of translations is not the same as what is on the command line. It is:

1. ebcidic_to_ascii
2. lower_to_upper
3. upper_to_lower
4. ascii_to_ebcidic
5. ascii_to_ibm

Note that not all conversions can be specified at once. You have your choice of only one conv in {ascii,ebcdic,ibm}, {lcase,ucase}, {block,unblock}, {unblock,sync}.

Finally, dd enters the copy stage. It allocates enough room for the input buffer, and if using a two-buffer scenario, allocates an output buffer as well. It performs any skips on the input, then performs any seeks on the output.

The main loop of the copy stage occurs now. It attempts to read input_blocksize characters into the input buffer. Errors here may be trapped, depending on command line options. If a full input block is not read (for example, when reading from a communication line, the end of a file, a pipe or special file, especially tapes), the partial block count is incremented. If the sync option is in effect, partial input blocks are NUL padded and treated as full input blocks.

At this point, if we are single-buffering, we write the block out. TODO: finish up here (I got bored)

AIX dd

AIX dd is broken. From their manpage:

3. Use the  backup,  tar, or  cpio command instead of the dd com-
  mand whenever possible to copy files to tape.  These commands are
  designed for use with  tape devices.  For more information on us-
  ing tape devices see the  rmt special file.

6. To ensure that only whole blocks are written to the output
  device (such as an 8mm tape in fixed-block mode), specify the ibs
  flag, the obs flag, and the  conv=sync flag.  The ibs flag must
  be a multiple of the obs flag.

This was hard-won knowledge for me. When dding a tar file directly out to tape, I ended up using:

dd if=foo.tar of=/dev/rmt1.5 ibs=1 obs=10240 conv=sync

Alternatively, you can use catblock, which is more efficient. Note that their comment about ibs being a multiple of obs is simply wrong, as my example demonstrates.

Solaris dd

Solaris dd is also not necessarily broken, but somewhat unintuitive. From the manpage:

When dd reads from a pipe, using the ibs=X and obs=Y operands, the output will always be blocked in chunks of size Y. When bs=Z is used, the output blocks will be whatever was available to be read from the pipe at the time.

In other words, don't expect "bs=Z" to be the same as "obs=Z ibs=Z". That's because, like GNU tar, it probably uses one buffer if you put "bs=Z", whereas "obs=Z ibs=Z" forces it to use two buffers. GNU dd may be subject to the same deficiency here. (TODO: check) I suppose it depends on how it treats a short read.

Recommended Articles

Take Command dd -- http://www.linuxjournal.com/article.php?sid=1320

The ' dd ' command is one of the original Unix utilities and should be in everyone's tool box. It can strip headers, extract parts of binary files and write into the middle of floppy disks; it is used by the Linux kernel Makefiles to make boot images. It can be used to copy and convert magnetic tape formats, convert between ASCII and EBCDIC, swap bytes, and force to upper and lowercase. For blocked I/O, the dd command has no competition in the standard tool set.

One could write a custom utility to do specific I/O or formatting but, as dd is already available almost everywhere, it makes sense to use it. Like most well-behaved commands, dd reads from its standard input and writes to its standard output, unless a command line specification has been given. This allows dd to be used in pipes, and remotely with the rsh remote shell command.

Unlike most commands, dd uses a keyword=value format for its parameters. This was reputedly modeled after IBM System/360 JCL, which had an elaborate DD 'Dataset Definition' specification for I/O devices. A complete listing of all keywords is available from GNU dd with

$ dd --help

Some people believe dd means ``Destroy Disk'' or ``Delete Data'' because if it is misused, a partition or output file can be trashed very quickly. Since dd is the tool used to write disk headers, boot records, and similar system data areas, misuse of dd has probably trashed many hard disks and file systems. In essence, dd copies and optionally converts data. It uses an input buffer, conversion buffer if conversion is specified, and an output buffer. Reads are issued to the input file or device for the size of the input buffer, optional conversions are applied, and writes are issued for the size of the output buffer. This allows I/O requests to be tailored to the requirements of a task. Output to standard error reports the number of full and short blocks read and written.

Example 1

A typical task for dd is copying a floppy disk. As the common geometry of a 3.5" floppy is 18 sectors per track, two heads and 80 cylinders, an optimized dd command to read a floppy is:

Example 1-a : Copying from a 3.5" floppy

dd bs=2x80x18b if=/dev/fd0 of=/tmp/floppy.image
1+0 records in
1+0 records out

The 18b specifies 18 sectors of 512 bytes, the 2x multiplies the sector size by the number of heads, and the 80x is for the cylinders--a total of 1474560 bytes. This issues a single 1474560-byte read request to /dev/fd0 and a single 1474560 write request to /tmp/floppy.image, whereas a corresponding cp command

cp /dev/fd0 /tmp/floppy.image

issues 360 reads and writes of 4096 bytes. While this may seem insignificant on a 1.44MB file, when larger amounts of data are involved, reducing the number of system calls and improving performance can be significant.

This example also shows the factor capability in the GNU dd number specification. This has been around since before the Programmers Work Bench and, while not documented in the GNU dd man page, is present in the source and works just fine, thank you.

To finish copying a floppy, the original needs to be ejected, a new diskette inserted, and another dd command issued to write to the diskette:

Example 1-b : Copying to a 3.5" floppy
dd bs=2x80x18b < /tmp/floppy.image > /dev/fd0
1+0 records in
1+0 records out

Here is shown the stdin/stdout usage, in which respect dd is like most other utilities.

Example 2

The original need for dd came with the 1/2" tapes used to exchange data with other systems and boot and install Unix on the PDP/11. Those days are gone, but the 9-track format lives. To access the venerable 9-track, 1/2" tape, dd is superior. With modern SCSI tape devices, blocking and unblocking are no longer a necessity, as the hardware reads and writes 512-byte data blocks.

However, the 9-track 1/2" tape format allows for variable length blocking and can be impossible to read with the cp command. The dd command allows for the exact specification of input and output block sizes, and can even read variable length block sizes, by specifying an input buffer size larger than any of the blocks on the tape. Short blocks are read, and dd happily copies those to the output file without complaint, simply reporting on the number of complete and short blocks encountered.

Then there are the EBCDIC datasets transferred from such systems as MVS, which are almost always 80-character blank-padded Hollerith Card Images! No problem for dd, which will convert these to newline-terminated variable record length ASCII. Making the format is just as easy and dd again is the right tool for the job.

Example 2 : Converting EBCDIC 80-character fixed-length record to ASCII variable-length newline-terminated record
dd bs=10240 cbs=80 conv=ascii,unblock if=/dev/st0 of=ascii.out
40+0 records in
38+1 records out

The fixed record length is specified by the cbs=80 parameter, and the input and output block sizes are set with bs=10240. The EBCDIC-to-ASCII conversion and fixed-to-variable record length conversion are enabled with the conv=ascii,noblock parameter.

Notice the output record count is smaller than the input record count. This is due to the padding spaces eliminated from the output file and replaced with newline characters.

Example 3

Sometimes data arrives from sources in unusual formats. For example, every time I read a tape made on an SGI machine, the bytes are swapped. The dd command takes this in stride, swapping the bytes as required. The ability to use dd in a pipe with rsh means that the tape device on any *nix system is accessible, given the proper rlogin setup.

Example 3 : Byte Swapping with Remote Access of Magnet Tape
rsh sgi.with.tape dd bs=256b if=/dev/rmt0 conv=swab | tar xvf -

The dd runs on the SGI and swaps the bytes before writing to the tar command running on the local host.

Example 4

Murphy's Law was postulated long before digital computers, but it seems it was specifically targeted for them. When you need to read a floppy or tape, it is the only copy in the universe and you have a deadline past due, that is when you will have a bad spot on the magnetic media, and your data will be unreadable. To the rescue comes dd, which can read all the good data around the bad spot and continue after the error is encountered. Sometimes this is all that is needed to recover the important data.

Example 4 : Error Handling
dd bs=265b conv=noerror if=/dev/st0 of=/tmp/bad.tape.image

Example 5

The Linux kernel Makefiles use dd to build the boot image. In the Alpha Makefile /usr/src/linux/arch/alpha/boot/Makefile, the srmboot target issues the command:

Example 5 : Kernel Image Makefile
dd if=bootimage of=$(BOOTDEV) bs=512 seek=1 skip=1

This skips the first 512 bytes of the input bootimage file (skip=1) and writes starting at the second sector of the $(BOOTDEV) device (seek=1). A typical use of dd is to skip executable headers and begin writing in the middle of a device, skipping volume and partition data. As this can cause your disk to lose file system data, please test and use these applications with care.

Credits

The dd command has been around since the 1970s, ported to many systems, rewritten many times, and tested by time as a useful tool. The current Linux version is GNU dd GNU fileutils 3.12, written by Paul Rubin, David MacKenzie, and Stuart Kemp, Copyright © 1985, 1990, 1991 Free Software Foundation, Inc. GNU dd is found in the fileutils collection, with the current version at the URL ftp://prep.ai.mit.edu/pub/gnu/fileutils-3.12.tar.gz or a mirror near you.

Remote backup

It is often neccesary to backup into a remote machine's tape drive. Here are the commands that can be used to achieve this. Execute this command on the machine you want to backup.

$ tar cvf - $DIRNAME | rsh $SYS dd of=$TAPEDEV

Substitute

$DIRNAME with the directory to backup,

$SYS with the machine name with the tape drive,

$TAPEDEV with the tape device.

Note: You must be able to rlogin into the remote machine without a password. To do this add the name of your local machine with your user name in the .rhost file in your home directory on the remote machine.

To retrieve the backed up info...

rsh $REM dd if=$TAPEDEV | tar xvf - Tips For Linux - How and when to use the dd command In this article, Sam Chessman explains the use of the dd command with a lot of useful examples. This article is not aimed at absolute beginners. Once you are familiar with the basics of Linux, you would be in a better position to use the dd command.

The ' dd ' command is one of the original Unix utilities and should be in everyone's tool box. It can strip headers, extract parts of binary files and write into the middle of floppy disks; it is used by the Linux kernel Makefiles to make boot images. It can be used to copy and convert magnetic tape formats, convert between ASCII and EBCDIC, swap bytes, and force to upper and lowercase.

For blocked I/O, the dd command has no competition in the standard tool set. One could write a custom utility to do specific I/O or formatting but, as dd is already available almost everywhere, it makes sense to use it.

Like most well-behaved commands, dd reads from its standard input and writes to its standard output, unless a command line specification has been given. This allows dd to be used in pipes, and remotely with the rsh remote shell command.

Unlike most commands, dd uses a keyword=value format for its parameters. This was reputedly modeled after IBM System/360 JCL, which had an elaborate DD 'Dataset Definition' specification for I/O devices. A complete listing of all keywords is available from GNU dd with

$ dd --help

Some people believe dd means ``Destroy Disk'' or ``Delete Data'' because if it is misused, a partition or output file can be trashed very quickly. Since dd is the tool used to write disk headers, boot records, and similar system data areas, misuse of dd has probably trashed many hard disks and file systems.

In essence, dd copies and optionally converts data. It uses an input buffer, conversion buffer if conversion is specified, and an output buffer. Reads are issued to the input file or device for the size of the input buffer, optional conversions are applied, and writes are issued for the size of the output buffer. This allows I/O requests to be tailored to the requirements of a task. Output to standard error reports the number of full and short blocks read and written.

Example 1

A typical task for dd is copying a floppy disk. As the common geometry of a 3.5" floppy is 18 sectors per track, two heads and 80 cylinders, an optimized dd command to read a floppy is:

Example 1-a : Copying from a 3.5" floppy

dd bs=2x80x18b if=/dev/fd0 of=/tmp/floppy.image
1+0 records in
1+0 records out

The 18b specifies 18 sectors of 512 bytes, the 2x multiplies the sector size by the number of heads, and the 80x is for the cylinders--a total of 1474560 bytes. This issues a single 1474560-byte read request to /dev/fd0 and a single 1474560 write request to /tmp/floppy.image, whereas a corresponding cp command

cp /dev/fd0 /tmp/floppy.image

issues 360 reads and writes of 4096 bytes. While this may seem insignificant on a 1.44MB file, when larger amounts of data are involved, reducing the number of system calls and improving performance can be significant.

This example also shows the factor capability in the GNU dd number specification. This has been around since before the Programmers Work Bench and, while not documented in the GNU dd man page, is present in the source and works just fine, thank you.

To finish copying a floppy, the original needs to be ejected, a new diskette inserted, and another dd command issued to write to the diskette:

Example 1-b : Copying to a 3.5" floppy
dd bs=2x80x18b < /tmp/floppy.image > /dev/fd0
1+0 records in
1+0 records out

Here is shown the stdin/stdout usage, in which respect dd is like most other utilities.

Example 2

The original need for dd came with the 1/2" tapes used to exchange data with other systems and boot and install Unix on the PDP/11. Those days are gone, but the 9-track format lives. To access the venerable 9-track, 1/2" tape, dd is superior. With modern SCSI tape devices, blocking and unblocking are no longer a necessity, as the hardware reads and writes 512-byte data blocks.

However, the 9-track 1/2" tape format allows for variable length blocking and can be impossible to read with the cp command. The dd command allows for the exact specification of input and output block sizes, and can even read variable length block sizes, by specifying an input buffer size larger than any of the blocks on the tape. Short blocks are read, and dd happily copies those to the output file without complaint, simply reporting on the number of complete and short blocks encountered.

Then there are the EBCDIC datasets transferred from such systems as MVS, which are almost always 80-character blank-padded Hollerith Card Images! No problem for dd, which will convert these to newline-terminated variable record length ASCII. Making the format is just as easy and dd again is the right tool for the job.

Example 2 : Converting EBCDIC 80-character fixed-length record to ASCII variable-length newline-terminated record
dd bs=10240 cbs=80 conv=ascii,unblock if=/dev/st0 of=ascii.out
40+0 records in
38+1 records out

The fixed record length is specified by the cbs=80 parameter, and the input and output block sizes are set with bs=10240. The EBCDIC-to-ASCII conversion and fixed-to-variable record length conversion are enabled with the conv=ascii,noblock parameter.

Notice the output record count is smaller than the input record count. This is due to the padding spaces eliminated from the output file and replaced with newline characters.

Example 3

Sometimes data arrives from sources in unusual formats. For example, every time I read a tape made on an SGI machine, the bytes are swapped. The dd command takes this in stride, swapping the bytes as required. The ability to use dd in a pipe with rsh means that the tape device on any *nix system is accessible, given the proper rlogin setup.

Example 3 : Byte Swapping with Remote Access of Magnet Tape
rsh sgi.with.tape dd bs=256b if=/dev/rmt0 conv=swab | tar xvf -

The dd runs on the SGI and swaps the bytes before writing to the tar command running on the local host.

Example 4

Murphy's Law was postulated long before digital computers, but it seems it was specifically targeted for them. When you need to read a floppy or tape, it is the only copy in the universe and you have a deadline past due, that is when you will have a bad spot on the magnetic media, and your data will be unreadable. To the rescue comes dd, which can read all the good data around the bad spot and continue after the error is encountered. Sometimes this is all that is needed to recover the important data.

Example 4 : Error Handling
dd bs=265b conv=noerror if=/dev/st0 of=/tmp/bad.tape.image

Example 5

The Linux kernel Makefiles use dd to build the boot image. In the Alpha Makefile /usr/src/linux/arch/alpha/boot/Makefile, the srmboot target issues the command:

Example 5 : Kernel Image Makefile
dd if=bootimage of=$(BOOTDEV) bs=512 seek=1 skip=1

This skips the first 512 bytes of the input bootimage file (skip=1) and writes starting at the second sector of the $(BOOTDEV) device (seek=1). A typical use of dd is to skip executable headers and begin writing in the middle of a device, skipping volume and partition data. As this can cause your disk to lose file system data, please test and use these applications with care.

size="-2">Credits
size="-2">
The dd command has been around since the 1970s, ported to many systems, rewritten many times, and tested by time as a useful tool. The current Linux version is GNU dd GNU fileutils 3.12, written by Paul Rubin, David MacKenzie, and Stuart Kemp, Copyright © 1985, 1990, 1991 Free Software Foundation, Inc.

GNU dd is found in the fileutils collection, with the current version at the URL ftp://prep.ai.mit.edu/pub/gnu/fileutils-3.12.tar.gz or a mirror near you.

Other major versions include SYSV and BSD, with the BSD source version 5.16 4/28/93 derived from software contributed to Berkeley by Keith Muller of the University of California, San Diego and Lance Visser of Convex Computer Corporation, Copyright © 1991 The Regents of the University of California.
 

Scripts

#!/bin/sh
tar -cvf my.tar $(for i in `cat list`
   do
       echo $i
   done)
exit

This script is backing up to tape using dump command. Logging date and all the messages
to a log file.

#!/bin/sh
#
echo "$DATE"backup.log
filenumber=`/usr/bin/mt stat|/usr/bin/grep "File Number"|/usr/bin/awk '{print $3}'`
echo "Backing up / to tape location: $filenumber"backup.log
/sbin/dump -0ua -f /dev/nrsa0 / &2backup.log
if [ $? -eq 0 ];then
   echo "/ backup successful"$HOME/log/backup.log
fi

Copy files ( even complete filesystem) from remote to local system

Note: You must be able to rlogin into the remote machine without a password. To do this add the name of your local machine with your user name in the .rhost file in your home directory on the remote machine.

      #!/bin/sh                                                      
      #                                                             
      # Copies files from Remote System to the local current directory
      #      
      name=`basename $0`                 
      if [ $# -ne 2 ];then
      echo "Usage: $name <remote-system> <dir-to-copy>"
      exit                                                           
      fi                                                              
      system=$1                                                  
      dir_to_cp=$2                                                
      rsh $system "cd $dir_to_cp; find . -print|cpio -ocB"|dd ibs=5k obs=5k|cpio -iducmvB

Humor

It is jokingly said that dd stands for "destroy disk" or "delete data", since, being used for low-level operations on hard disks, a small mistake, such as reversing the if and of parameters, may accidentally render the entire disk unusable:

Tips For Linux - How and when to use the dd command

Also Murphy's Law was formulated long before digital computers, but it seems it was specifically targeted for them. When you need to read a floppy or tape, and it is the only copy in the universe the effect of a bad spot on the magnetic media is devastating. dd can read all the good data around the bad spot and continue after the error is encountered. Sometimes this is all that is needed to recover the important data.