Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-) Softpanorama Search

Creating External Snapshots for JFS (not JFS2)

AIX 5.3 supports filesystem snapshots. The problem is that they are extremely badly documented. It took me hours just to understand simplest things about this feature So typical junk that IBM documentation represents here is made into incomprehensible mess that probably ensured that this feature is not used. You simply cannot do worse. In this sense Solaris documentation is a masterpiece ;-). Also AIX 5.3 supports splitting mirror drives:

In JFS (and JFS only; not JFS2) in addition to ability to create snapshots chfs command provides freeze function which writes all blocks to disk and convert disk to read-only for a certain number of seconds.  If you are using FlashCopy to backup mounted AIX filesystems, please see the restrictions listed in the following URL: http://www-1.ibm.com/support/docview.wss?

1. shutdown particular application (start of downtime)

2. freeze the drive for 1 second to write all blocks to the disk
     chfs -a freeze=1
3. Create a  snapshot  chfs -a splitcopy=/snapshot_mount_point /filesystem/mount/point
4. Restart the application (end of downtime)

5. Mount snapshot using a new mount point.

6. Backup snapshot using a new mount point.

7. Delete the snapshot  with  rmfs

This "external snapshot" feature is controlled via chfs Command. To create a snapshot type the following:

chfs -a splitcopy=/snapshot_mount_point /filesystem/mount/point

At this point, a read-only copy of the file system is available in //snapshot_mount_point. Any changes made to the original file system after the copy is split off are not reflected in the backup copy.

You can control which mirrored copy is used as the backup by using the copy attribute. The second mirrored copy is the default if a copy is not specified by the user. For example:

 chfs -a splitcopy=/snapshot_mount_point -a copy=1 /filesystem/mount/point

To reintegrate the JFS split image as a mirrored copy at the /backup_point mount point, use the following commands:

umount /backup_point 

rmfs /backup_point

The rmfs command removes the file system copy from its split-off state and allows it to be reintegrated as a mirrored copy.

chfs Parameters Reference

-a freeze={ timeout | 0 | off}

Specifies that the file system must be frozen or thawed, depending on the value of timeout.

The act of freezing a file system produces a nearly consistent on-disk image of the file system, and writes all dirty file system metadata and user data to the disk. In its frozen state, the file system is read-only, and anything that attempts to modify the file system or its contents must wait for the freeze to end. The value of timeout must be either 0, off, or a positive number. If a positive number is specified, the file system is frozen for a maximum of timeout seconds. If timeout is 0 or off, the file system will be thawed, and modifications can proceed.

Note:

Freezing base file systems (/, /usr, /var, /tmp) can result in unexpected behavior.

Splitting mirror drives is available via is available via chfs utility option -a

-a splitcopy=NewMountPointName
 

Splits off a mirrored copy of the file system and mounts it read-only at the new mount point. This provides a copy of the file system with consistent JFS meta-data that can be used for backup purposes. User data integrity is not guaranteed, so it is recommended that file system activity be minimal while this action is taking place. Only one copy may be designated as an online split mirror copy.

For example to split off a copy of a mirrored file system and mount it read-only for use as an online backup, enter:

chfs -a splitcopy=/backup -a copy=2 /testfs

Old News ;-)

[Apr 16, 2009] Freezing filesystems and containers [LWN.net]

By Jake Edge
June 25, 2008

Freezing seems to be on the minds of some kernel hackers these days, whether it is the northern summer or southern winter that is causing it is unclear. Two recent patches posted to linux-kernel look at freezing, suspending essentially, two different pieces of the kernel: filesystems and containers. For containers, it is a step along the path to being able to migrate running processes elsewhere, whereas for filesystems it will allow backup systems to snapshot a consistent filesystem state. Other than conceptually, the patches have little to do with each other, but each is fairly small and self-contained so a combined look seemed in order.

Takashi Sato proposes taking an XFS-specific feature and moving it into the filesystem code. The patch would provide an ioctl() for suspending write access to a filesystem, freezing, along with a thawing option to resume writes. For backups that snapshot the state of a filesystem or otherwise operate directly on the block device, this can ensure that the filesystem is in a consistent state.

Essentially the patch just exports the freeze_bdev() kernel function in a user accessible way. freeze_bdev() locks a file system into a consistent state by flushing the superblock and syncing the device. The patch also adds tracking of the frozen state to the struct block_device state field. In its simplest form, freezing or thawing a filesystem would be done as follows:

    ioctl(fd, FIFREEZE, 0);

    ioctl(fd, FITHAW, 0);

Where fd is a file descriptor of the mount point and the argument is ignored.

 

In another part of the patchset, Sato adds a timeout value as the argument to the ioctl(). For XFS compatibility—though courtesy of a patch by David Chinner, the XFS-specific ioctl() is removed—a value of 1 for the pointer argument means that the timeout is not set. A value of 0 for the argument also means there is no timeout, but any other value is treated as a pointer to a timeout value in seconds. It would seem that removing the XFS-specific ioctl() would break any applications that currently use it anyway, so keeping the compatibility of the argument value 1 is somewhat dubious.

If the timeout occurs, the filesystem will be automatically thawed. This is to protect against some kind of problem with the backup system. Another ioctl() flag, FIFREEZE_RESET_TIMEOUT, has been added so that an application can periodically reset its timeout while it is working. If it deadlocks, or otherwise fails to reset the timeout, the filesystem will be thawed. Another FIFREEZE_RESET_TIMEOUT after that occurs will return EINVAL so that the application can recognize that it has happened.

Moving on to containers, Matt Helsley posted a patch which reuses the software suspend (swsusp) infrastructure to implement freezing of all the processes in a control group (i.e. cgroup). This could be used now to checkpoint and restart tasks, but eventually could be used to migrate tasks elsewhere entirely for load balancing or other reasons. Helsley's patch set is a forward port of work originally done by Cedric Le Goater.

The first step is to make the freeze option, in the form of the TIF_FREEZE flag, available to all architectures. Once that is done, moving two functions, refrigerator() and freeze_task(), from the power management subsystem to the new kernel/freezer.c file makes freezing tasks available even to architectures that don't support power management.

As is usual for cgroups, controlling the freezing and thawing is done through the cgroup filesystem. Adding the freezer option when mounting will allow access to each container's freezer.state file. This can be read to get the current freezer state or written to change it as follows:

    # cat /containers/0/freezer.state
    RUNNING
    # echo FROZEN > /containers/0/freezer.state
    # cat /containers/0/freezer.state
    FROZEN

It should be noted that it is possible for tasks in a cgroup to be busy doing something that will not allow them to be frozen. In that case, the state would be FREEZING. Freezing can then be retried by writing FROZEN again, or canceled by writing RUNNING. Moving the offending tasks out of the cgroup will also allow the cgroup to be frozen. If the state does reach FROZEN, the cgroup can be thawed by writing RUNNING.

 

In order for swsusp and cgroups to share the refrigerator() it is necessary to ensure that frozen cgroups do not get thawed when swsusp is waking up the system after a suspend. The last patch in the set ensures that thaw_tasks() checks for a frozen cgroup before thawing, skipping over any that it finds.

There has not been much in the way of discussion about the patches on linux-kernel, but an ACK from Pavel Machek would seem to be a good sign. Some comments by Paul Menage, who developed cgroups, also indicate interest in seeing this feature merged.

IBM Redbooks IBM AIX Version 6.1 Differences Guide

2.2 JFS2 internal snapshot

With AIX 5L V5.2, the JFS2 snapshot was introduced. Snapshots had to be created into separate logical volumes. AIX V6.1 offers the ability to create snapshots within the source file system.

Therefore, starting with AIX V6.1, there are two types of snapshots:

• External snapshot
• Internal snapshot

Table 2-1 provides an overview of the differences between the two types of snapshots.

Category	External snapshot	Internal snapshot
Location	Separate logical volume	Within the same logical volume
Access	Must be mounted separately	/fsmountpoint/.snapshot/snapshotname
Maximum generations	15	64
AIX compatibility	>= AIX 5L V5.2	>= AIX V6.1

Comparison of external and internal snapshots

Both the internal and the external snapshots keep track of the changes to the snapped file system by saving the modified or deleted file blocks. Snapshots provide point-in-time (PIT) images of the source file system. Often, snapshots are used to be able to create a consistent PIT backup while the workload on the snapped file system continues.

The internal snapshot introduces the following enhancements:

• No super user permissions are necessary to access data from a snapshot, since no initial mount operation is required.
• No additional file system or logical volume needs to be maintained and monitored.
• Snapshots are easily NFS exported, since they are in held in the same file system.

2.2.1 Managing internal snapshots

A JFS2 file system must be created with the new -a isnapshot=yes option. Internal snapshots require the use of the extended attributes v2 and therefore the crfs command will automatically create a v2 file system.

Existing file systems created without the isnapshot option cannot be used for internal snapshots. They have to be recreated or have to use external snapshots.

There are no new commands introduced with internal snapshots. Use the snapshot, rollback, and backsnap commands to perform operations. Use the new -n snapshotname option to specify internal snapshots. There are corresponding SMIT and Web-based System Manager panels available.

To create an internal snapshot:

# snapshot -o snapfrom=/aix61diff -n snap01

Snapshot for file system /aix61diff created on snap01

To list all snapshots for a file system:

# snapshot -q /aix61diff

Snapshots for /aix61diff

Current  Name         Time

*        snap01       Tue Sep 25 11:17:51 CDT 2007

To list the structure on the file system:

# ls -l /aix61diff/.snapshot/snap01

total 227328

-rw-r--r--    1 root     system     10485760 Sep 25 11:33 file1

-rw-r--r--    1 scott    staff       1048576 Sep 25 11:33 file2

-rw-r--r--    1 jenny    staff     104857600 Sep 25 11:33 file3

drwxr-xr-x    2 root     system          256 Sep 24 17:57 lost+found

The previous output shows:

• All snapshots are accessible in the /fsmountpoint/.snapshot/ directory.
• The data in the snapshot directories are displayed with their original file permission and ownership. The files are read only; no modifications are allowed.

Note: The .snapshot directory in the root path of every snapped file system is not visible to the ls and find command. If the .snapshot directory is explicitly specified as an argument, they are able to display the content.

To delete an internal snapshot:

# snapshot -d -n snap01 /aix61diff

2.2.3 Considerations

The following applies for internal snapshots:

• A snapped file system can be mounted read only on previous AIX 5L versions. The snapshot itself cannot be accessed. The file system must be in a clean state; run the fsck command to ensure that this is true.
• A file system created with the ability for internal snapshots can still have external snapshots.

Once a file system has been enabled to use internal snapshots, this cannot be undone.

If the fsck command has to modify the file system, any internal snapshots for the file system will be deleted by fsck.

Snapped file systems cannot be shrunk.

The defragfs command cannot be run on a file system with internal snapshots.

Existing snapshot Web-based System Manager and SMIT panels are updated to support internal snapshots.

The following items apply to both internal and external snapshots:

• A file system can use exclusively one type of snapshot at the same time.
• Typically, a snapshot will need two to six percent of the space needed for the snapped file system. For a highly active file system, 15 percent is estimated.
• External snapshots are persistent across a system reboot.
• During the creation of a snapshots, only read access to the snapped file system is allowed.
• There is reduced performance for write operations to a snapped file system. Read operations are not affected.
• Snapshots are not replacement for backups. A snapshot depends always on the snapped file system, while backups have no dependencies on the source.
• Neither the mksysb nor alt_disk_install commands will preserve snapshots.
• A file system with snapshots cannot be managed by DMAPI. A file system being managed by DMAPI cannot create a snapshot.

Understanding and exploiting snapshot technology for data protection, Part 1 Snapshot technology overview

Split mirror

Split mirror creates a physical clone of the storage entity, such as the file-system, volume, or LUN for which snapshot is being created, onto another entity of the same kind and the exact same size. The entire contents of the original volume are copied onto a separate volume. Clone copies are highly available, since they are exact duplicates of the original volume that resides on a separate storage space. However, due to the data copy, such snapshots cannot be created instantaneously. Alternatively, a clone can also be made available instantaneously by "splitting" a pre-existing mirror of the volume into two, with the side effect that original volume has one fewer synchronized mirror. This snapshot method requires as much storage space as the original data for each snapshot. This method has the performance overhead of writing synchronously to the mirror copy.

EMC Symmterix and AIX Logical Volume Manager support split mirror. Additionally, any raid system supporting multiple mirrors can be used to create a clone by splitting a mirror.

Copy-on-write with background copy (IBM FlashCopy)

Some vendors offer an implementation where a full copy of the snapshot data is created using copy-on-write and a background process that copies data from original location to snapshot storage space. This approach combines the benefits of copy-on-write and split mirror methods as done by IBM FlashCopy and EMC TimeFinder/Clone. It uses copy-on-write to create an instant snapshot and then optionally starts a background copy process to perform block-level copy of the data from the original volume (source volume) to the snapshot storage (target volume) in order to create an additional mirror of the original volume.

When a FlashCopy operation is initiated, a FlashCopy relationship is created between the source volume and target volume. This type of snapshot is called a COPY type of FlashCopy operation.

IBM incremental FlashCopy

Incremental FlashCopy tracks changes made to the source and target volumes when the FlashCopy relationships are established. This allows the capability to refresh a LUN or volume to the source or target's point in time content using only the changed data. The refresh can occur in either direction, and it offers improved flexibility and faster FlashCopy completion times.

This incremental FlashCopy option can be used to efficiently create frequent and faster backups and restore without the penalty of having to copy entire content of the volume .

developerWorks IBM Systems IBM System Storage Flashcopy on FAStT700 with AIX 5.2 (P ...

Flashcopy on FAStT700 with AIX 5.2 (P Series Server)
Posted: Nov 23, 2006 04:57:45 AM

I am hoping somebody with Flashcopy experience on AIX may be able to help. We have had a few issues with the scripts we have been running to import Flashcopies of our databases to our P series host nightly to dump to tape; mostly centering around repository sizing which I have pretty much resolved.

I am struggling with a couple of legacy issues though. First of all we need to add some more disk (LUNS) from the array, and I am concerned about conflicts with pvids on the scripted Flashcopies.

Currently our import/export runs something like this :

Import:

- Recreate Flashcopy with SMcli(FAStT700)
 

• LUN Mapping with SMcli(FAStT700)
• hot_add (AIX cfgmgr) to import the disk devices to the host
• chdev -l hdisk -a pv=clear (AIX) to clear the source LVM data structures from the FC's
• recreatevg (AIX) to recreate the Flashcopy Volume Group
• mount filesystems

Export:
- umount filesystems (AIX)
 

• varyoffvg (AIX) varyoff the FC volume group
• exportvg (AIX) export the FC volume group
• rmdev -l hdisk -d (AIX) completely remove the FC devices from the ODM
• Remove LUN mapping with SMcli(FAStT700)
• Disable the Flashcopy with SMcli (FAStT700)

Problem is as we completely remove the FC pvs from AIX each time, they only keep the same pvids each time for scripting purposes because we never add any other volumes to the system.

Somebody on another forum suggested that we should only remove the FCs from AIX using rmdev -l hdisk (no -d flag) in order that they remain defined in the ODM. That theory seems to work; but unfortunately when re-importing the volumes as soon as you run 'chdev -l hdisk -a pv=clear'I can not get the 'defined' volumes to switch back to 'available', and subsequently re-running hot_add (cfgmgr) just brings the FCs on line with new pvids, leaving the original ones as 'defined'.

I have tried throwing a 'chdev -l hdisk -a pv=yes' into the equation and even a 'mkdev -l hdisk' into the mix in various orders but nothing seems to work. Has anybody got any ideas how I can effectively 'hardcode' the FC pvids in the ODM and reuse the same ones each time ?

The other issues we have experienced are failures due to either 'chdev -l hdisk -a pv=clear' fails; or corrupt jfs superblocks when mounting the imported volumes on the host, requiring an fsck or re-running the snapshot.

We shut down the database prior to recreating the FCs to quiesce i/o; but I was wondering if additionally we should be (a) running a synch call prior to recreating the FCs to flush the disk buffers and (b) running a full fsck prior to mounting the volumes as a matter of course.

Does anybody have any ideas/experience here ?

Sorry for the long post - but I'm an HP guy historically and I'm just getting to grips with this Flashcopy stuff; and unfortunately IBM won't ratify our 'home grown' scripts under our support agreement.

I will not duplicate this posting at this time but if anybody thinks it would be better in the AIX forum please let me know.
 

fosteria   Posts: 4 Registered: Nov 23, 2006 04:12:26 AM

Re: Flashcopy on FAStT700 with AIX 5.2 (P Series Server) Posted: Nov 23, 2006 05:53:00 AM   in response to: fosteria's post     Reply Further to this; I just found that our recent 'failed to clear pvid' error was actualy due to this error when creating the FC : 20:45:02 SMcli_north_recreate.sh :Created Logical drive for Flash copy 4-1 An internal error, error code 12, has occurred. This is possibly due to initialization problems when loading the necessary internal files. Please check your installation of SMclient. If problems continue to persist, re-install SMcli ent. SMcli failed. Anybody got any ideas ? THX.

jvk   Posts: 23 Registered: Oct 24, 2006 04:15:23 AM

Re: Flashcopy on FAStT700 with AIX 5.2 (P Series Server) Posted: Dec 03, 2006 11:54:19 AM   in response to: fosteria's post     Reply From my notes here are some general hints for a good FlashCopy: . 1) just before FlashCopy appl. should be frozen or stoped. Next step is to unmount the FlashCopy source file systems to ensure that all data is flashed to the disks. If you don't use unmount, then it's required to run "freeze" and "thaw" AIX commands. See the "chfs -a freeze" and "chfs -a freeze=off" in the man pages. 2) after "freeze" run sync; sync; sync; sleep 5 3) if your FlashCopy source logical volumes are spread over more hdisks, it's required to use disk level"consistency group" on your disk subsytem to ensure all IOs are stoped at the same time. Without disk level"consistency group" there is a possibility that the target volumes may not be mountable. The failure to mount is not an error in this case. It results from an inconsistency between the file system data and the file system log that can be introduced during the FlashCopy process. 4) I would suggest to use separate jfs/jfs2 log for every FS. Otherwise it must be ensured that all FS's they use same log are stoped or frozen at the same time. 5) on target node (where FlashCopy disks are imported) run - "recreatevg -f" if you flashed just half of LVM mirred hdisks - "fsck -y" on all file systems before mounting For your SMcli problems I have no hints as I don't know SMcli. Next I don't understand your PVIDs problem... I see nothing wrong if disks are rmdev -dl during "Export". These are only FlashCopy target LUNs, so they will get all LVM and data structure _exactly_the_same_ (including the PVID) as FlashCopy source LUNs. If you want to import target LUNs on some other AIX node, then the "chdev -l hdisk -a pv=clear" and normal recreatevg are not necessary. If you import target LUNs on the same system where are source LUNs, then the "chdev -l hdisk -a pv=clear" and recreatevg are required.

fosteria   Posts: 4 Registered: Nov 23, 2006 04:12:26 AM

Re: Flashcopy on FAStT700 with AIX 5.2 (P Series Server) Posted: Dec 08, 2006 05:21:11 AM   in response to: jvk's post     Reply First of all - thanks for your help - and especially all the tips with ensuring the integrity of the Flashcopy. On the pvid issue (and bear with me here); the problem is that we do import to the same system each night and use recreatevg with the pvids hardcoded in the import script (currently hdisk17 - hdisk22 for instance). Even though we export them using rmdev -dl they always get the same hdisk ids when we run hot_add (cfgmgr) each night to bring them back in. Problem is if I introduce a new device whilst the FC's are off line, that new device is going to get hdisk17 pvid in the ODM when I run cfgmgr; so subsequently when I next run my FC import I'm screwed because the script is trying to use the same pvid. My Flashcopy volumes would be configured in as hdisks 18-23 this time; so the script would have to be changed. What I was trying to do was effectively leave the FC volumes permanently defined as hdisk17 - hdisk22, so when I configure a new volume it would be assigned hdisk23 for instance. It is configuring this functionality that is causing the issue.

Recommended Links

Help - chfs Command