Student Guide - Symm BC Management
Student Guide - Symm BC Management
Management
Student Guide
Education Services
August 2010
Copyright © 2010 EMC Corporation. Do not Copy - All Rights Reserved.
Module 1 - Introduction -1
Copyright © 2010 EMC Corporation. Do not Copy - All Rights Reserved.
Module 1 - Introduction -2
Copyright © 2010 EMC Corporation. Do not Copy - All Rights Reserved.
Module 1 - Introduction -3
Copyright © 2010 EMC Corporation. Do not Copy - All Rights Reserved.
The objectives for this course are shown here. Please take a moment to read them.
Module 1 - Introduction -4
Copyright © 2010 EMC Corporation. Do not Copy - All Rights Reserved.
The Agenda for the week, with general timelines for Module presentation and lab exercises is outlined.
Module 1 - Introduction -5
Copyright © 2010 EMC Corporation. Do not Copy - All Rights Reserved.
The Agenda for the week, with general timelines for Module presentation and lab exercises is outlined.
Module 1 - Introduction -6
Copyright © 2010 EMC Corporation. Do not Copy - All Rights Reserved.
The Agenda for the week, with general timelines for Module presentation and lab exercises is outlined.
Module 1 - Introduction -7
Copyright © 2010 EMC Corporation. Do not Copy - All Rights Reserved.
Module 1 - Introduction -8
Copyright © 2010 EMC Corporation. Do not Copy - All Rights Reserved.
The objectives for this module are shown here. Please take a moment to read them.
TimeFinder/Clone is a local replication solution available on the Symmetrix arrays. It allows users to
create point-in-time copies of the source devices, on to target devices. The source and target devices
can be either Standard or BCVs, provided they are of the emulation (FBA to FBA, CKD to CKD).
Unlike TimeFinder/Mirror, the clone copies are immediately available on activation of a clone session,
while actual copying of data can occur in the background.
The create action defines the copy session requirements and sets the track protection bitmap on the
source device to detect which tracks are being accessed by the target host or written to by the source
host. The target device is made Not Ready to its host and placed on hold status for copy session
activity.
This prevents other control operations from using the device. The activate action makes the clone
ready to the host. It also starts one of the background copying processes. The process, Copy-On-Access
or Full Copy, depends on the argument used when the clone session was created.
The recreate command allows you to incrementally copy all subsequent changes made to the source
device. In other words, it allows you to refresh the contents of the clone with new contents of the
source since the clone session has been activated. The restore action restores contents of the clone
volume to a restore target.
The establish operation creates and then immediately activates a clone session with a single command.
The terminate action terminates the Source-Target relationship.
Full Device Copy Option: When the copy session is activated, data copying begins in the background
so that a full copy of the data will become available on the target device. This is the default with
Solutions Enabler 7.0 and above.
Differential Option: This option creates an SDDF (Symmetrix Data Differential Facility) session for
maintaining changed track information. Creating a differential clone session automatically implies a
Full Device Copy. This is the default with Solutions Enabler 7.0 and above.
CopyOnAccess: After activating the copy session, only those tracks that have been written to the
source or written/read from the target, will be copied to the target device. A full data copy to the target
device will not occur unless all of device tracks are accessed or written to while participating in the
active session.
Precopy Option: The -precopy option can be used with the create operation to start copying tracks in
the background, before the clone session is activated.
CopyOnWrite: CopyOnWrite can be enabled by setting the following parameter in the options file
ENABLE.SYMAPI_CLONE_COPY_ON_WRITE = ENABLE | DISABLE
Once CopyOnWrite as been enabled and activated, all reads will be handled from the source device and
writes to the source device or target device during the active copy session will result in the data being
copied to the target device.
Recreate Before Copied: This feature allows users to recreate the symclone session before the copy
finishes. In prior releases the customer had a problem if there was change in the SRC after the initial
clone session. The customer had to wait until the previous copy finished in order to make differential
changes. Aborting the original session with terminate would destroy the differential information and
require a full copy.
Up to 16 clone copies of a standard source device can be created. The Symmetrix array is currently
limited to 16 sessions per source device, which can be used for Clone, Snap, or SDDF (Symmetrix
Differential Data Facility) operations. This limits the number of available Clone copies that can be
created. A total of 16 concurrent CopyOnAccess copy sessions can be created from a standard device
(sessions created using the -nocopy option).
Copy sessions created using the -copy option to create full data copies are limited to eight concurrent
sessions.
This slide summarizes the session slots used by the different local and remote replication operations
discussed in this course.
When you issue the symclone activate command, the target is made ready again. A user on the target’s
host can then access data on the target immediately:
If the target host attempts to read from, or write to, the tracks on the target that have not been copied
yet, those tracks are first copied from the source, preferentially.
If the source host issues a writes to tracks that are not yet copied, those tracks are copied to the target,
prior to updating them with the new writes.
Not Ready Option: The -not_ready option can be used with the activate action to cause the target
device to remain not ready to its host. The copy session will be activated and the target device will be
placed in the Not Ready state. The Clone copy can later be read/write enabled to the host using the
symld ready command.
Consistent Option: The symclone activate command can be used with the -consistent option to invoke
the Enginuity Consistency Assist (ECA) feature. This feature can be used to create Clone copies that
are consistent up to the point in time that the activation occurs. The feature suspends writes to the
source devices during the activation. When the activation has completed, writes are resumed and the
target device contains a consistent copy of the source device at the time of activation.
For consistent activation, there should be a control host with access to only Gatekeeper (GK) devices,
or a separate HBA on the production host with access only to GKs and not data devices. This is to
ensure that in write intensive environments SYMAPI will be able to freeze and then thaw I/O to the
production devices within the ECA window, regardless of the number of outstanding I/Os held by the
host HBA accessing the data devices. A device group, composite group or a file must be created on the
controlling host. If a single DG, CG or file is created to include all the devices for both Host A and B,
then the copy sessions for both Host A and B can be consistently activated with a single command.
Alternatively DG, CG or file can be created to include only the relevant devices for Host A and for
Host B. In this case copy sessions for each of the hosts can be consistently activated, independent of
each other.
A Device Group (DG) is a user created object for viewing and managing related Symmetrix devices.
All devices in a device group should be on the same Symmetrix array. There are three types of device
groups: REGULAR, R1, and R2. For TimeFinder/Clone and TimeFinder/Snap operations, device
group of type REGULAR should be created. The device group definition is stored in the SYMAPI
database (symapi_db.bin) on the host where the symdg create command was executed.
A Composite Group (CG) follows the same rules as a DG, except that the devices can span multiple
local Symmetrix arrays. If an application uses Symmetrix devices from multiple local Symmetrix
arrays, a CG containing all the devices must be created for consistent activation of TimeFinder
operations.
Note: This concept of consistent activation applies to TimeFinder/Snap as well. In the case of
TimeFinder/Snap, the target devices (TGT) will be Virtual Devices (VDEV).
The symclone establish command sets the target device to Not Ready for a short time. Therefore, you
may want to unmount the target volume before issuing the command.
An establish operation creates and then activates a new clone session. The clone session created by the
establish operation is by default equivalent to a session created with –copy and –differential
options. The establish operation can also be performed on an existing differential clone session. In
which case, the operation performs recreate and activate operations on the clone session.
Terminating a copy session deletes the pairing information in the Symmetrix array and removes any
hold on the target device.
Terminating a session while the device pairs are in the CopyOnAccess or CopyInProg state causes the
session to end. If the application has not finished accessing all of the data, the target copy is not a full
copy. The symclone terminate command is allowed for all TimeFinder/Clone pair states.
A created / activated copy session may be terminated, but the data on the target device becomes invalid
(whether or not data was actually copied to the target). If the state is CopyInProg, then the -symforce
option must be applied to terminate the session. This also leaves the target copy as an incomplete
copy. If the pair state is Copied, then terminating the session will give a full copy of the data of the
target device.
You can use the symclone restore command to copy target data to another device (full restore), or back
to the original source device (incremental restore).
In the case of a full restore (-full), the original session terminates and a copy session to the target of the
restore starts.
In the case of an incremental restore, the original session terminates and an incremental copy session
back to the original source device starts.
To support this operation, the session must have been created with the default options and the device
must be in a fully copied state.
A SYMCLI device group – clonedg is created. Next 2 source devices (13D and 13E) are added to
the group. Two target devices (0F9 and 0FA) are added. Note the use of the –tgt flag. This designates
0F9 and 0FA as Targets for the clone sessions.
Clone session is then created and activated. Again note the use of –tgt flag with the create and
activate commands.
After activation the device group is queried to determine the status of the clone sessions between
devices in the group.
The command has been executed from the Source host. Hence the source devices 13D and 13E have
Physical Device Names (PdevName) and the target devices 0F9 and 0FA do not. As the target devices
were added with the –tgt flag, they have been given Logical Device names of TGT001 and TGT002
respectively. The output from the command has been edited to fit the slide. The complete output is
more verbose than shown here. Also note that the target devices are currently in Read/Write status, as
they have been just added to the device group.
The symclone create action defines the clone copy session requirements and sets the track protection
bitmap on the source device to detect which tracks are being accessed by the target host or written to
by the source host. The target device is made Not Ready to its host and placed on hold status for clone
copy session activity. This prevents other control operations from using the device. The clone copy
does not become accessible to its host until the copy session is activated. The output has been edited to
fit the slide. Note that the target devices have been moved to a Not Ready state (NR) upon Clone
session creation. They will be in a held state until the Clone session is terminated.
Query of the clone session shows that the session has been created. No tracks have yet been copied as
indicated by 0 % copied. With Solutions Enabler 7.0, the default options for the create command are –
copy and –differential. This can be verified under the CGDP columns. As the legend below
indicates, the sessions have been created with background copy and differential options, event though
these were not explicitly specified when creating the session. The source devices and the target devices
belong to the same device group in our example. Hence an X under the G column of the output. If
copying at creation is required then the –precopy flag should have been entered during the create
operation. If PreCopy is desired now, then symclone set mode precopy –tgt can be
executed. This will start the precopy operation. After creating the clone session with the default
options, the set mode precopy is the only operation possible to change the nature of the copy
process. i.e set mode nocopy cannot be issued.
Legend:
(C): X = The background copy setting is active for this pair.
. = The background copy setting is not active for this pair.
(G): X = The Target device is associated with this group.
. = The Target device is not associated with this group.
(D): X = The Clone session is a differential copy session.
. = The Clone session is not a differential copy session.
(P): X = The pre-copy operation has completed one cycle
. = The pre-copy operation has not completed one cycle
The symclone list command will show all the clone session in the Symmetrix.
As the symclone set mode precopy –tgt has been issued, the Status now reflects
PreCopy as well.
Legend:
This activates the copy operation from the source device to the target device.
The symclone activate command can be used with the -consistent option to invoke the Enginuity
Consistency Assist (ECA) feature. This feature can be used to create clone copies that are consistent
with the database up to the point in time that the activation occurs. The feature suspends writes to the
source devices during the activation.
When the activation has completed, writes are resumed and the target device contains a consistent
production database copy of the source device at the time of activation.
Note:
Cloned data is made available as a point-in-time copy at the time of activation and not at the time that
the session was created.
Performing the symclone query command shows the options with which the clone session has been
created and activated. The current state for these pairs is “Copied”.
The symclone recreate command allows you to incrementally copy all subsequent changes
made to the source device (made after the point-in-time copy initiated) to the target device.
To recreate the clone session, it should have been first created with either the -copy or -precopy
option, and the -differential option. While in the Recreated state, the target device remains Not
Ready to the host. After the recreate operation, the session can be activated to initiate a new point-in-
time (differential) copy of the data.
Note that the query output has been truncated to show just the relevant information.
Terminating a copy session deletes the pairing information in the Symmetrix array and removes any
hold on the target device. Terminating a session while the device pairs are in the CopyOnAccess or
CopyInProg state causes the session to end. If the application has not finished accessing all of the data,
the target copy is not a full copy. The symclone terminate command is allowed for all
TimeFinder/Clone pair states.
Note:
A created and activated copy session may be terminated, but the data on the target device is not valid
unless the state had previously been COPIED.
The establish command combines the create and activate steps into a single operation. The clone
session is created and immediately activated. The first time an establish operation is performed, the –
full option has to be specified.
Note:
The symclone establish command sets the target device to Not Ready for a short time, prior to
activation. Therefore, the target volumes should be umounted from the target host, to avoid any
undesired effects. Upon command completion, the target volumes can be mounted back again.
In this example, two additional devices (0FB and 0FC) have been added to the device group. Standard
Devices can now be paired with two sets of Target Devices to obtain two concurrent clone copies.
A clone session is first created between 13D:13E and 0F9:0FA device pairs
With the –concurrent option in the create operation, a second clone session is created between
device pairs 13D:13E and 0FB:0FC. To list all the clone sessions for the standard devices, the –multi
option is used in the query command.
To activate all clone sessions concurrently, the device pairings have to be specified exactly in the
command as shown in this example.
symclone restore command can be used to copy target data to another device, or back to the
original source device. If data is restored to the original source devices then the operation is
incremental as shown in this example.
In the case of an incremental restore, the original session terminates and an incremental copy session
back to the original source device starts. To support this operation, the session must have been created
with the -differential option and the device must be in a fully copied state.
If data is restored to another device, then the original clone session has to be terminated (symclone
terminate –tgt) and a full restore operation to the new device should be initiated (symclone
restore –full –tgt).
To recreate the clone session between the original source and target device pairs, the restored session
should be first split. Next the clone session can be recreated and activated.
symclone recreate –tgt
symclone activate –tgt
In the Clone Emulation mode, the TimeFinder/Mirror reverse split option is not allowed (symmir
split –reverse). Restore operations will always be protected. The
SYMAPI_DEFAULT_BCV_SPLIT_TYPE, SYMAPI_DEFAULT_BCV_RESTORE_TYPE,
SYMAPI_DEFAULT_BCV_ESTABLISH_TYPE option file settings are ignored. Also the maximum
number of BCVs that can be incrementally established with a Standard device is 8, instead of the 16
allowed by TimeFinder/Mirror.
In this example two STD devices 141:142 have been added to the device group mirrordg and one
BCV(RAID-1) device (111) and one BCV(RAID-5) device have been associated with the device
group. Note that the output from symdg show mirrordg has been edited to show just the relevant
information for our purpose.
With Enginuity Version 5874, the TimeFinder/Mirror establish command is automatically converted to
Clone Emulation mode even for a BCV(RAID-1). The only way to display that Emulation mode is
active, is to perform a symdev show on the device(s) of interest. As can be seen in the BCV State
Flags field – Emulation is in action and as mentioned earlier, one cannot perform TimeFinder/Mirror
reverse split operation in Emulation mode {CantRevSpl}.
TimeFinder/Clone can be used to create point-in-time copies of thin and thick devices on other thin
devices. Creating clones of thin devices is exactly the same as creating clones of regular devices. Thin
clones can be presented to a host and accessed just like regular devices or clone source devices. Thin
clone targets can also be updated and the changes on the targets can be restored back to the source thin
devices or to different thin devices.
Thin devices are classified as standard devices and can be added to Symmetrix device groups or put in
device files. When using device groups, clone operations can be performed against devices by
specifying the clone source and target when the clone pairs are created or clone targets can instead be
added as targets by specifying the –tgt option when they are added to the group.
With the new feature, A B, and B C can be simultaneously in a Pre Copy mode. So data can flow
from A B and then on from B C. Hence the term Cascaded TimeFinder/Clone. When both clone
sessions have completed one precopy cycle, A B can be activated. After A B reach the copied state, B
C session can be activated.
With Enginuity 5874, all TimeFinder/Mirror commands will be in emulation mode. Native
TimeFinder/Mirror functionality is removed. As the table shows, whenever a TF/Mirror functionality is
called, it will be translated to emulation.
Continuing with the previous example of Clone Emulation, the BCV devices in the device group
mirrordg can be split. The BCV devices can now be used as Source devices for a Clone session. In this
example BCV devices 111 and 119 act as Source for the Clone Targets 0FD and 0FE. The
TimeFinder/Mirror split is equivalent to the TimeFinder/Clone activate operation. So the Emulated
Clone session between the STD and BCV devices is in an “active” state when the BCV devices are
used as Source for the Clone session. The clone session can now be terminated, data can be
incrementally copied from the STD to the BCV (in Clone Emulation mode again). As the Clone
session from the BCV devices as Source has been terminated, if the BCV devices should again be used
as Clone Source, the Clone session cannot be incremental. In other words, symclone recreate
cannot be used. So the sequence for iteration would be:
symmir establish –full (STD: 141, 142 ; BCV: 111, 119 – in Emulation Mode)
symmir split –consistent (equivalent to symclone activate)
symclone create (BCV: 111, 119 ; TGT: 0FD, 0FE – Native Clone)
symclone terminate (BCV: 111, 119 ; TGT: 0FD, 0FE – Native Clone)
symmir establish (STD: 141, 142 ; BCV: 111, 119 – in Emulation Mode – incremental establish)
symmir split –consistent (equivalent to symclone activate)
symclone create (BCV: 111, 119 ; TGT: 0FD, 0FE – Native Clone)
symclone terminate (BCV: 111, 119 ; TGT: 0FD, 0FE – Native Clone)
The edited output of the symdg show command, shows that STD devices 13D:13E are paired with
STD devices 0F9:0FA (added as targets to the device group), in the device group clone_a_to_b
The edited output of the symdg show command, shows that the Clone targets in device group
clone_a_to_b devices 0F9:0FA are now added as Source devices and are paired with STD devices
0FB:0FC (added as targets to the device group), in the device group clone_b_to_c. Note that in
order to add the same set of devices to multiple device groups (on the same host), the SYMAPI
variable SYMAPI_ALLOW_DEV_IN_MULT_GRPS must be set to ENABLE in the SYMAPI options
file.
As all devices involved are STD devices, the Clone sessions are all Native (not Emulation Mode).
Notice that Enginuity 5874 now permits the Target of one Clone session (F9:FA) to be the Source for
another Clone session, without having to terminate the first session.
After activating the session for devices in clone_a_to_b, if an activate is attempted for the devices
in clone_b_to_c, the operation fails. This is because, while both sessions can simultaneously be in
“Created” state, only on of them can be actively copying at any time. In this case we have to wait for
devices in clone_a_to_b to reach a “Copied” state before successfully activating the session for
devices in clone_b_to_c.
This table indicates that both sessions can be simultaneously in Created or in Precopy mode, when both
sessions are Native Clones. However, if one session is in CopyInProg, the other session cannot be
simultaneously in CopyInProg as well.
This slide lists some of the considerations for cloning a source devices to larger target devices. Logical
Volume Manager structures along with the data will be copied from the Source device to the larger
Target device. As a result, even though the Target device is physically larger, the host may not
recognize this until platform specific LVM actions are completed.
The difference in the behavior of SRDF restore between TimeFinder/Mirror and TimeFinder/Clone restores
to R2, causes an incompatibility. With 5874, all TimeFinder/Mirror commands will now be Clone
emulation. So, the existing functionality of concurrent TimeFinder/Mirror restore to R2 and SRDF restore,
will break. Enginuity 5874 has been changed to accommodate to allow SRDF restore from R2 to R1, while
the TimeFinder/Clone restore to R2 is in progress.
These are the key points covered in this module. Please take a moment to review them.
The objectives for this module are shown here. Please take a moment to read them.
TimeFinder/Snap creates space-saving, logical point-in-time images. The Snaps are not a full copies of
data; they are logical images of the original information, based on the time the Snap was created. A set
of pointers to the source volume data tracks is created upon activation of the Snap. These pointers are
stored in the Virtual Device. Target host is given access to the Virtual Device.
The default maximum number of Snaps is 16. If a restore from a Snap is planned, one session is
reserved for the restore session. This reduces the default maximum number of Snaps to 15.
Setting the following SYMCLI environment variable: SYMCLI_MULTI_VIRTUAL_SNAP = ENABLED,
increases the maximum number of Snap sessions to 128.
TimeFinder/Snap functionality is managed via copy sessions, which pair the source and target devices.
Sessions are maintained on the Symmetrix array and can be queried to verify the current state of
devices.
A copy session must first be Created that defines the Snap devices in the copy operation. When the
session is subsequently activated, the target virtual devices then become accessible to its host. When
sessions are no longer required they can be terminated.
Snap operations are controlled from the host by using the symsnap command to create, activate,
terminate, and restore the Snap copy sessions.
TimeFinder/Snap allows you to make copies of data simultaneously on multiple target devices from a
single source device. The data is available to a target host instantly. This source device can be a
Standard or a BCV device. (BCV in a Split State). A target device is a virtual device (VDEV) that
consumes no physical storage.
TimeFinder/Snap uses a process called “Copy on First Write” when a Snap session is activated. When
a host attempts to write to a track on the Source volume for the first time after activation, the original
track is copied from the Source volume to the Save Area, the pointer is updated to reference the Save
Area, then the Source volume is updated with the new write. Subsequent writes to the same track on
the Source volume will not invoke a data copy. Likewise, when a write is issued to a track on the
VDEV for the first time, the original data is copied from the Source volume to the Save Area, the
pointer is updated, and the new write to the VDEV is stored in the Save Area.
The save pool device is not host accessible, but accessed only through the virtual devices that point to
it. Save devices provide a pool of physical space to store snap-copy data to which virtual devices point.
Tracks are striped in a round-robin manner to save devices to improve performance. When savedevs
are configured across the backend, they are separated first by director, then channel, and then by disk.
As sessions are terminated, the space in the Save Area is released and available for re-use by other
snap sessions.
Symmetrix SAVE devices are specially configured devices (not mapped to the host) that provide
pooled physical storage space used to store pre-update images or changed tracks during a virtual copy
session. Multiple named SAVE device pools can be created and sessions can be specified to use a
specific SAVE pool
Save Area capacity can also be allocated based on application requirements. For example:
Allocate more space for Write-intensive applications
Allocate more space for long duration snapshots
The -svp option can be used with the create action to specify which SAVE device pool to use for an
operation.
There are four types of consistent activation available; each offers a restartable copy on target VDEVs:
PowerPath
ECA
Veritas File System
-both_sides
Additional Cache is required for Snap snapshots. The number of VDEVs must be included in cache
calculations.
VDEVs can be configured using Symmetrix Management Console or SYMCLI. VDEVs take device
IDs.
For best performance the Save Devices should be spread across as many physical volumes as possible.
When the Save Area is full, any active session that requires a free track in the Save Pool will be placed
in a failed state. This is because, the original data has to be copied to the Save Area before changing
the data on the Source devices. How ever if there is no more space in the Save Area, then this operation
cannot complete. Hence the session is placed in a failed state.
Use symsnap monitor to track the amount of available space of the save area. The symsnap monitor
command can also be used to automatically run an action if a predetermined threshold for space has
been crossed. Use -percent nn –action script_pathname to specify the percentage full threshold and
script associated with it.
You can monitor SAVE devices by using the symsnap monitor command to check the percentage
full. When devices reach the specified percentage, an optional action script can be executed by the
application to preserve the data or terminate sessions. The following is an example of the monitor
command:
symsnap monitor -percent 80 -action SaveScript -norepeat -i 60 –svp
Accounting
In the above example, the SAVE device pool “Accounting” will be monitored every minute for
percentage full. When the percentage of the SAVE devices are 80 percent full, the associated action
script SaveScript will be executed each time the threshold of 80 percent is met.
The list of available VDEVs can be obtained with the command shown.
In this example there is a DEFAULT_POOL of device emulation type FBA, with a number of devices
in it.
The individual Symmetrix devices that constitute the Save Pool can be listed as well.
The symconfigure command is used to create a new Save Pool and add Save Devices to it. The
action and the devices are specified in the *.txt files. In this example we are moving devices 17D:180
out of the DEFAULT_POOL, into a new Save pool – appn_a_pool.
A SYMCLI Device Group has been created, two STD devices have been added as Source for Snap
sessions and two VDEVs have been added as Targets for Snap sessions.
Note that the VDEVs are given the Logical Devices Names VDEV001 and VDEV002 respectively.
In this example, Snap sessions have been created between the source and the target VDEVs. The
symsnap create command sets up the track protection bitmap. As can be seen in the output, all
tracks on the Source devices have been protected. The state of the Source – Target device pair goes to
Created.
Legend:
A “hold” is placed on the VDEVs and they remain in a Not Ready state. We can also see that the Save
Pool named appn_a_pool is used for these Snap sessions.
The symsnap activate command places the snap pair in the CopyOnWrite (copy-on-first-write) state
and the target virtual device in a Read/Write (RW) state. The actual copying of data occurs when a
track is written to either on the Source device or on the VDEV.
For this example two additional VDEVs (133:134) were added to the device group snapdg.
symsnap create –svp appn_a_pool – Creates the first Snap session between 13F:131 and
140:132
symsnap activate –consistent – Activates the first Snap session
symsnap create –svp appn_a_pool –concurrent – Creates the second Snap session
between 13F:133 and 140:134
symsnap activate –consistent – Activates the second Snap session
Three types of restore operations can be performed for virtual device copy sessions:
1. Incremental restore back to the original source device.
2. Incremental restore to a TimeFinder/Clone BCV or a TimeFinder/Mirror BCV, which has been
split from its original standard source device but maintains the incremental relationship with the
source.
3. Full restore to any standard or split TimeFinder/Clone BCV or a TimeFinder/Mirror BCv device
outside of the existing copy session. The target device of the restore must the same size and
emulation type as the source device.
All original Snap copy sessions are maintained when performing a restore operation. A new restore
copy session between the source device and VDEV is created. A restore operation can only be
performed if an additional copy session is available for use.
When a restore command is issued, both the VDEVs and the Source devices are set to Not Ready.
Once the restore operation starts the Source devices are automatically set to Ready. But the VDEVs
will remain Not Ready. Hence it would be best practice to stop access to the VDEVs and the Source
devices when initiating a restore operation. After the restore comples the VDEVs can be made ready
again by symdev ready 131 –sid 80; symdev ready 132 –sid 80
The restore command initiates an incremental restore by default.
Note that even after the restore operation, the original Snap session is still maintained. If this original
session should now be recreated or terminated, the restored session should first be terminated with
symsnap terminate –restored operation.
Thin devices can be used with TimeFinder/Snap in the same way that other types of Symmetrix
devices can be used. Point-in-time snap copies of thin devices can be created on virtual devices
(VDEVs), which can subsequently be presented to a host. The VDEVs can also be updated and the
changes on the VDEVs can be restored back to the source thin devices or to different thin devices.
Thin devices are classified as standard devices and can be added to Symmetrix device groups or put in
device files.
Prior to Enginuity 5874, taking a new point-in-time TimeFinder/Snap copy required terminating the
previously activated Snap session between the source volume and the target virtual volume (VDEV).
Recreate operation will now be allowed with TimeFinder/Snap symsnap command. After activation of a
Snap session, the session can be recreated and then activated, as and when new point-in-time image is
required. As the recreate will lead to the replacement of the old image on the VDEV, all used tracks in the
SAVE pool associated with the previous session will be released. The query output will show a recreated
state. The recreate function is not supported with Multi-Virtual snap.
Note that with Solutions Enabler 7.0 and Enginuity 5874, a symsnap recreate operation can also
be performed on the Snap session with Clone target as the source.
These are the key points covered in this module. Please take a moment to review them.
The objectives for this module are shown here. Please take a moment to read them.
Symmetrix Remote Data Facility (SRDF) is a Symmetrix system based business continuance, disaster
recovery, restart, and data mobility solution. In the simplest terms, SRDF is a configuration of multiple
Symmetrix units for remote replication of data. The Symmetrix units can be in the same room, in
different buildings within the same campus, or hundreds and even thousands of miles apart. Data can
be replicated Synchronously (SRDF/S) or Asynchronously (SRDF/A).
The SYMCLI command symrdf list pd gives a list of all SRDF devices accessible to the host. In
this example the host has access to 4 SRDF devices (0145:0148). As can be seen under the RDF
Typ:G column, the devices are of type R1 and they belong to the SRDF Group 100. The of SRDF
operation is Synchronous, and currently all the R1-R2 pairs are in a Synchronized state. The local R1
devices (the Sym Dev column of the output) 0145:0148 are paired with corresponding R2 devices (the
RDev column of the output) 0145:0148
Legend for MODES:
M(ode of Operation) : A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy
D(omino) : X = Enabled, . = Disabled
A(daptive Copy) : D = Disk Mode, W = WP Mode, . = ACp off
(Mirror) T(ype) : 1 = R1, 2 = R2
(Consistency) E(xempt): X = Enabled, . = Disabled, M = Mixed, - = N/A
The inquiry command syminq will give a list of all devices accessible to the host. The type of the device
is will be indicated in the Type column. This command will also help in correlating the host Physical
Device Names with the corresponding Symmetrix Device numbers.
The command symdev list –r1 can be used to list all the SRDF R1 devices configured on the
Symmetrix to which the host is attached.
A device group is a user created object for viewing and managing related Symmetrix devices. All
devices in a device group should be on the same Symmetrix array. There are three types of device
groups: RDF1, RDF2, RDF21 or REGULAR. When creating a device group, if a type is not specified
explicitly, by default a device group of type REGULAR will be created. A device group, with the type
REGULAR cannot contain SRDF devices, a device group of type RDF1 cannot contain R2 devices and
a device group of type RDF2 cannot contain R1 devices. When performing TimeFinder or SRDF/S
operations, SYMCLI commands can be executed for ALL devices in the device group or a subset of
them. For SRDF/A operations the commands should be executed for ALL devices in the SRDF Group
(another construct discussed later).
Storage Administrators must create a device group with RDF1 or RDF2 for SRDF operations, as
appropriate. In this example device group of type R1 (RDF1) is created, so that the R1 devices 0145
and 0146 can be added to it.
The device group definition is stored in the SYMAPI database (symapi_db.bin) on the host where
the symdg create command was executed. By default a Symmetrix device can only belong to one
device group on the host from which the group was created. If the device should belong in multiple
device groups on the same host then the variable SYMAPI_ALLOW_DEV_IN_MULT_GRPS must
be set to ENABLE in the options file.
The symdg show command displays detailed group information for any specific device group. The
device group (srdfsdg), contains 2 local standard devices. The device group type is RDF1. The
Symmetrix serial number is also displayed. If there are any BCVs, VDEVs, etc associated with or
added to the group, this information is also displayed.
Further information on the devices in the group as well as relevant RDF information is also presented
with the symdg show command. The two devices have been assigned (by default) the Logical Device
Names of DEV001 and DEV002. The host (Windows OS in this example) addresses the two devices as
PHYSICALDRIVE13 and PHYSICALDRIVE14.
Users can perform a number of Symmetrix SRDF operations using host-based SYMCLI commands.
Major SRDF operations or actions include: suspend, resume, failover, failback, update, split, establish,
and restore.
The symcfg list command gives the list of all Symmetrix Arrays that are connected to the Local
Symmetrix. In this example Symmetrix with ID 80 is the Local Symmetrix. It is connected via SRDF
to both Symmetrix ID 81 and Symmetrix ID 82. The Num Phys Devices column indicates that the
host from which the command was executed, has physical access to 64 devices on Symmetrix ID 80,
and none on the other two Symmetrix Arrays.
The symrdf ping command verifies that the Local Symmetrix can communicate with the Remote
Symmetrix Arrays over the SRDF Links.
This command shows that Symmetrix ID 80 has a pair of SRDF Adapters configured – 7H and 8H.
SRDF group 100 (RA Group 100) has been created to use both the adapters.
Suspend is a singular operation. Data transfer from the Source devices to the Target devices is stopped.
RA communication path is still available. New writes to the Source devices accumulate as invalid
tracks to the R2 (R2 Inv Tracks). The link is set to Not Ready (NR). The R1s continue to be Read
Write enabled and the R2s continue to be Write Disabled.
To invoke a suspend, the RDF pair(s) must already be in one of the following states:
Synchronized
R1 Updated
Resume is a singular operation. To invoke this operation, the RDF pair(s) must already be in the
Suspended state. Data transfer from R1 to R2 is resumed. The pair state will remain in SyncInProg
until all accumulated invalid tracks for the pair have been transferred. Invalid tracks are transferred to
the R2 in any order – so write serialization will not be maintained. The link is set to Read Write. The
R1s continue to be Read Write enabled and the R2s continue to be Write Disabled.
The “symrdf set mode” command will change the SRDF operation mode. In this example the mode
has been changed to Adaptive Copy Disk for these two R1-R2 pairs. This is indicated by the C in M
column and D in the A column of the output.
Legend for MODES:
The failover operation can be executed from the source side host or the target side. This is true for all
symrdf commands. In the event of an actual disaster, this is helpful as there would be no way of
communicating with the source Symmetrix. The operation assumes disaster situation - makes all effort
to enable data access on target Symmetrix
Will proceed if possible
Will give message for any potential data integrity issue
As can be seen in the output, the R1 devices are write disabled, the SRDF links between the device
pairs are logically suspended, and the R2 devices are read write enabled. Host accessing the R2 devices
can now resume processing the application.
While in a true disaster situation when the source host/Symmetrix/site may be unreachable, the steps
listed below would be recommended if performing a “graceful” failover to the target site.
If failing over for a Maintenance operation: For a clean, consistent, coherent point in time copy which
can be used with minimal recovery on the target side some or all of the following steps may have to be
taken on the source side:
Stop All Applications
Unmount file system (unmount or unassing drive letter to flush the filesystem buffers from the
host memory down to the Symmetrix)
Deactivate the Volume Group
A failover leads to a write disabled state of the R1 devices. If a device suddenly becomes write
disabled from a read/write state the reaction of the host can be unpredictable if the device is in use.
Hence the recommendation to stop applications, unmount filesystem/unassign drive letter, prior to
performing a failover for maintenance operations Module 4 – SRDF/S Operations - 15
Copyright © 2010 EMC Corporation. Do not Copy - All Rights Reserved.
As seen in the output, the R1s are now Write Disabled, the links are set to Not Ready and the R2s are
Read Write enabled. As application processing has been started using the R2 devices, we see that there
are invalid tracks accumulating (R1 Inv Tracks) on the target Symmetrix. These are the changes that
are being made to the R2 devices. When it is time to return to the source Symmetrix, these invalid
tracks will be incrementally synchronized back to the source. The pair state is reflected as Failed Over.
While the target (R2) device is still operational (Read Write Enabled to its local host), an incremental
data copy from the target (R2) device to the source (R1) device can be initiated in order to update the
R1 mirror with changed tracks from the target (R2) device. After an extended outage on the R1 side, a
substantial amount of invalid tracks could have accumulated on the R2. If a failback is now performed,
production starts from the R1. New writes to the R1 have to be transferred to the R2 synchronously.
Any track requested on the R1 that has not yet been transferred from the R2 has to be read from across
the links. This could lead to performance degradation on the R1 devices. The update operation helps to
minimize this impact.
When performing and update, the R1 devices are still “Write Disabled”; the links becomes “Read
Write” enabled because of the “Updated” state. The target devices (R2) remain “Read Write” during
the update process.
The update operation can be used with the –until flag, which represents a skew value assigned to the
update process. For example, one can choose to update until the accumulated invalid tracks is down to
30000. Then a failback operation can be executed.
Note that when the update operation is performed after a failover, the links become Read/Write
enabled, but the Source devices are still Write disabled. Production work continues on the R2 devices.
When the source site has been restored, or if maintenance is completed, one can return production to
the source site. The symrdf failback command will set the R2s Write Disabled, the link Read Write and
the R1s Read Write enabled. Merge of the device track tables between the source and target is done.
The SRDF links are resumed. The accumulated invalid tracks are transferred to the source devices
from the target devices. So all changes made to the data when in a failed over state will be preserved.
As noted earlier, the Primary host can access the R1 devices and start production work as soon as the
command completes. If a track that has not yet been sent over from the R2 is required on the R1,
SRDF can preferentially read that track from across the links.
As the R2s will be set to Write Disabled, it is important to shut down the applications using the R2
devices, and perform the appropriate host dependent steps to unmount filesystem/deactivate volume
groups. If applications still actively access R2s when they are being set to Write Disabled, the reaction
of the host accessing R2s will be unpredictable. In a true disaster, the failover process may not give an
opportunity for a graceful shutdown. But a failback event should always be planned and done
gracefully.
As can be seen in the output the R1s are set to Read Write, R2s are set to Write Disabled and the links
are set to Read Write. The pair states go into Synchronized. The accumulated invalid tracks have been
transferred from the target array to the source array.
The split command suspends the links between source (R1) and Target (R2) volumes. The source
devices continue to be read write enabled. The target devices are set to read write enabled. It enables
read and write operations on the and target volumes.
Changes to the source devices are marked as R2 invalids. Changes to target devices are marked as R1
invalids. SRDF does not provide Disaster Recovery in this state, as changes are being made to both R1
and R2 independently of each other and remote mirroring has been suspended.
Establish operation will resume SRDF remote mirroring. Changes made to the source while in a split
state, will be transferred to the target. Changes made to the target are overwritten. The R2 devices are
set to Write Disable. Hence applications should stop accessing the R2 devices, prior to performing an
establish operation. The links are resumed.
Restore operation will resume SRDF remote mirroring. Changes made to the target while in a split
state, will be transferred to the source. Changes made to the source are overwritten. The R2 devices are
set to Write Disable. Hence applications should stop accessing the R2 devices, prior to performing an
establish operation. The links are resumed. As data on the R1 devices will change without the
knowledge of the host, access to R1 devices should be stopped prior to performing a restore operation.
As soon as the command completes, R1 devices can be accessed again without waiting for
synchronization to be completed. Any required track on the R1 that has not yet been received from the
R2, will be read across the links.
Dynamic RDF Configuration State is Enabled by default with Symmetrix V-Max array and Enginuity
5874. Symmetrix devices can have an attribute set on them which enables them to become a R1, or a
R2, or either. The combination of the ability to dynamically create SRDF groups and the dynamic
device attribute enables one to create, delete and swap SRDF R1-R2 pairs.
SRDF groups define the relationships between the local Symmetrix SRDF director/ports and the
corresponding remote Symmetrix SRDF director/ports. Any Symmetrix device that has been
configured as an SRDF device must be assigned to an SRDF group. It would be convenient if the
SRDF group numbers on the local and the remote Symmetrix are identical, however this is not a
requirement. Static SRDF groups can be explicitly configured in the Symmetrix bin file. Storage
Administrators can dynamically create SRDF groups and assign them to Fibre Channel directors or
GigE directors. Dynamic SRDF group information is not written to the Symmetrix bin file, but they are
persistent through power cycle and IMPL. Symmetrix SRDF groups are also referenced as RA groups
or RDF groups.
Before creating a new SRDF group, some information needs to be gathered. First we need to know
SRDF directors that are configured on the Symmetrix. We also need to know the number of SRDF
groups (RA groups) currently configured and their corresponding group numbers. Symmetrix V-Max
array with Enginuity 5874 can support up to 250 SRDF groups.
The symrdf addgrp command creates an empty Dynamic SRDF group on the source and the target
Symmetrix and logically links them. Note that physical link connectivity and communication between
the two Symmetrix must exist for this command to succeed. Note that the SRDF group number in the
command is in decimal. In the Symmetrix it is converted to hexadecimal. The decimal group numbers
start at 01 but the hexadecimal group numbers start at 00. Hence the hexadecimal group numbers will
be off by one. The symrdf createpair command takes the dynamic capable device pairs listed in
the text file (pairs.txt) and makes them R1-R2 pairs. First we need to determine the devices that
have the Dynamic attribute set on them. The command example is shown in the next slide. By
specifying –establish the newly created R2 devices are synchronized with the data from the newly
created R1 devices. In this example the file contains the following entries:
pairs.txt
149 149
14A 14A
The first column lists the devices on the Symmetrix on which the command is executed and the second
column is the remote Symmetrix. Specifying –type R1 makes the device in the first column to be R1s
and the devices in the second column will become their corresponding R2s.
The symdev show command will display the field shown below for Dynamic capable devices:
Creating dynamic SRDF pairs with establish - Optionally, you can include the establish operation
in the createpair command line by replacing the -invalidate r2 option described earlier
with the –establish option, where the default copy path is R1 to R2 for all the device pairs:
symrdf createpair -file grp5.txt -sid 97 -type RDF1 -rdfg 5 -establish
Creating dynamic SRDF pairs for a restore - One can perform a restore operation to copy data back to
the R1 source devices by including the -restore option in the createpair command line as
follows:
symrdf createpair -file grp5.txt -sid 97 -type RDF1 -rdfg 5 -restore
Creating dynamic SRDF pairs and not bring up the SRDF links - Invalidate, allows creation of
dynamic SRDF pairs, but does not bring up the SRDF links and initiate data copy:
symrdf createpair -file grp5.txt -sid 97 -type RDF1 -rdfg 5 –invalidate
grp5.txt
13D 174
13E 175
The file contains the listing of the pairs of devices that should be changed to R1-R2 pairs. The first
column lists the devices on array with sid 97 and the second column lists the corresponding devices
on the remote array. In the examples shown as the type is specified as RDF1, device 13D and 13E on
sid 97 will become R1s and devices 174 and 175 on the remote array will become R2s. If the type
was specified to be RDF2, then 13D and 13E will become R2s and 174 and 175 will become the
corresponding R1s.
The delete SRDF pairs command cancels SRDF pairs in the device file specified. For example, to
delete the SRDF pairs in an RDF group 5, enter: Before the delete pair can be invoked the pair must
be suspend first.
Example
c:\symrdf suspend -sid 97 -file grp5.txt -rdfg 5
c:\symrdf deletepair -sid 97 -file grp5.txt -rdfg 5
An R1/R2 personality swap (or R1/R2 swap) refers to when the RDF personality of the RDF device
designations of a specified device group are swapped so that source R1 device(s) become target R2
device(s) and target R2 device(s) become source R1 device(s).
Sample scenarios for R1/R2 Swap
Symmetrix Load Balancing
In today’s rapidly changing computing environments, it is often necessary to deploy applications and
storage on a different Symmetrix without having to give up disaster protection. R1/R2 swap can enable
this redeployment with minimal disruption, while offering the benefit of load balancing across two
Symmetrix storage arrays.
Primary Data Center Relocation
Sometimes a primary data center needs to be relocated to accommodate business practices. For
example, several financial institutions in New York City routinely relocate their primary data center
across the Hudson River to New Jersey as part of their disaster drills. R1/R2 swaps allow these
customers to run their primary applications in their New Jersey data centers. The Manhattan data
centers now act as the disaster protection site.
Post-Failover Temporary Protection Measure
If the hosts on the source side are down for maintenance, R1/R2 swap permits the relocation of
production computing to the target site without giving up the security of remote data protection. When
all problems have been solved on the local Symmetrix, you have to failover again and swap the
personality of the devices to go back to the original configuration.
Module 4 – SRDF/S Operations - 33
Copyright © 2010 EMC Corporation. Do not Copy - All Rights Reserved.
The R21 devices are configured and used for Cascaded SRDF environments. The R22 devices are used
in SRDF/Star environments. An R22 device has two R1 mirrors. How ever, it can receive data from
only one of the R1 mirrors at a time.
Concurrent SRDF allows two remote SRDF mirrors of a single R1 device, e.g. use one remote copy
for disaster recovery, and another for decision support or backup. A concurrent R1 device has two R2
devices associated with it. Each of the R2 devices is in a different Symmetrix array. Any combination
SRDF modes is allowed, except for Async and Async. i.e
R1 R2 (Site B) in Synchronous mode and R1 R2 (Site C) in Asynchronous mode
R1 R2 (Site B) in Synchronous mode and R1 R2 (Site C) in Adaptive Copy mode
R1 R2 (Site B) in Synchronous mode R1 R2 (Site C) in Synchronous mode
Each of the R1 R2 pairs are created in different SRDF Groups.
2 Synchronous remote mirrors :
A write IO from the host at the primary device side cannot be returned as completed until both
remote Symmetrix’ signal the local Symmetrix that the SRDF IO is in cache at the remote side.
1 Sync and 1 Adaptive Copy remote mirror:
The SRDF / IO from the secondary device operating in Synchronous mode must present ending
status to the sending Symmetrix before a second host IO can be accepted. The host I/O does not
wait for the secondary device operating in Adaptive Copy mode.
Simultaneous restore from both R2s to the R1 cannot be performed. SRDF swap is not allowed in this
configuration.
RDF Enginuity Consistency Assist (RDF-ECA) provides consistency protection for synchronous mode
devices by performing suspend operations across all SRDF/S devices in a consistency group or a
named subset of all devices in a composite group. SRDF/S with RDF-ECA is supported by an RDF
daemon that performs monitoring and cache recovery operations across all SRDF/S sessions in the
group. If one or more source (R1) devices in an SRDF/S consistency group cannot propagate data to
their corresponding target (R2) devices, the RDF daemon suspends data propagation from all R1
devices in the consistency group, halting all data flow to the R2 targets. This ensures that a consistent
R2 data copy of the database exists at the point-in-time any interruption occurs. The RDF daemon
monitors data copy operations and coordinates the suspension of R1 to R2 data propagation if the
consistency group is tripped.
A Composite group must be created using the RDF consistency protection option (-rdf_consistency)
and must be enabled using the symcg enable command for the RDF daemon to begin monitoring and
managing the RDF-ECA consistency group. A Composite group follows the same rules as a Device
group, except that the devices can span multiple local Symmetrix arrays, as well as multiple SRDF
groups (RA groups). An SRDF Consistency group is a Composite group created with special properties
to allow SRDF devices in separate SRDF groups to maintain dependent write consistency. Starting
with Enginuity 5874, SRDF/S consistency is only supported with RDF-ECA. SRDF/S consistency
using PowerPath is no longer supported.
With Enginuity 5874, the two mirrors of a concurrent SRDF/S device can be managed independently.
These are the key points covered in this module. Please take a moment to review them.
The objectives for this module are shown here. Please take a moment to read them.
SRRD/A’s unique architecture delivers a remote mirroring solution that has no impact on production
applications over extended distance because I/O requests from the host are acknowledged locally.
Changes made to the same data blocks are periodically sent only once to the remote Symmetrix. This
enables significant operational savings through reduced bandwidth requirements. Moreover, SRDF/A
provides an alternative Disaster Recovery solution, in addition to SRDF/S, by maintaining a consistent
image of a RDBMS (Relational DataBase Management System) on the R2 volumes at all times.
SRDF/A is a single solution supporting both Mainframe and Open Systems. It also compliments
SRDF solutions to meet mixed service level requirements.
SRDF/A uses Delta Sets to maintain a group of writes over a short period of time. Delta Sets are
discrete buckets of data that reside in different sections of the Symmetrix cache. There are four types
of Delta Sets to manage the data flow process.
The Capture Delta Set in the source Symmetrix (numbered N in this example), captures (in cache) all
incoming writes to the source volumes in the SRDF/A group.
The Transmit Delta Set in the source Symmetrix (numbered N-1 in this example), contains data from
the immediately preceding Delta Set. This data is being transferred to the remote Symmetrix.
The Receive Delta Set in the target system is in the process of receiving data from the transmit Delta
Set N-1.
The target Symmetrix contains an older Delta Set, numbered N-2, called the Apply Delta Set. Data
from the Apply Delta set is being assigned to the appropriate cache slots ready for de-staging to disk.
The data in the Apply Delta set is guaranteed to be consistent and restartable should there be a failure
of the source Symmetrix.
The Symmetrix performs a cycle switch once data in the N-1 set is completely received, data in the N-
2 set is completely applied, and the 30 second minimum cycle time elapsed. During the cycle switch, a
new delta set (N+1) becomes the capture set, N is promoted to the transmit/receive set and N-1
becomes the apply Delta Set.
All commonly used database management systems are inherently dependent write consistent. For
instance, a DBMS (DataBase Management System) will not perform a log write, indicating that a
transaction is complete, until it has received an acknowledgement from the storage subsystem that the
log data pertaining to the transaction itself was completely written to disk. Symmetrix honors this logic
in SRDF/A by treating any successor I/O, which arrives after a predecessor I/O as a dependent I/O.
As noted in the SRDF/S module, the RDF Configuration States are all Enabled by default on the
Symmetrix V-Max array with Enginuity 5874. The use of Host Throttle, and Maximum Cache Usage
attributes will be explained later in this module.
A partial listing of the output is shown in the slide. None of the groups are in SRDF/A mode at this
point in time. Notice that Transmit Idle is enabled by default when the SRDF Groups (RA groups) are
created. The Minimum Cycle Time for SRDF/A is the default 30 seconds. The default priority for the
SRDF groups is 33. More about the group priority later in the module.
Legend:
RDFA Flags :
(C)onsistency : X = Enabled, . = Disabled, - = N/A
(S)tatus : A = Active, I = Inactive, - = N/A
(R)DFA Mode : S = Single-session, M = MSC, - = N/A
(M)sc Cleanup : C = MSC Cleanup required, - = N/A
(T)ransmit Idle : X = Enabled, . = Disabled, - = N/A
(D)SE Status : A = Active, I = Inactive, - = N/A
DSE (A)utostart : X = Enabled, . = Disabled, - = N/A
The list of devices in SRDF Group 101 is displayed here. The two devices 149 and 14A are R1 devices
on the local Symmetrix 80. They are in Synchronous mode of SRDF operations, and they are currently
Synchronized with their remote mirrors.
Legend for MODES:
SRDF/A can be enabled when the device pairs are operating in any of the listed modes. In the case of
Adaptive Copy to SRDF/A transitions, it takes two additional cycle switches after resynchronization of
data for the R2 devices to be consistent.
Any SRDF/A operation (with the exception of consistency exempt, discussed later in the module) must
be performed on ALL devices in an SRDF group (RA group). This means that all devices in an SRDF
group must be in the same SRDF Device group as well. This is in contrast with SRDF/S, where
operations can be performed on a subset of devices in an SRDF group.
The mode of SRDF operation is set to Asynchronous for the device pairs in the device group
srdfadg. SRDF/A consistency is enabled. symrdf query –rdfa gives detailed information
about the SRDF/A state of the device group. As can be seen, the SRDF/A session has been assigned a
number 100. Five cycle switches have completed and the SRDF/A session is active.
Transition to SRDF/A is immediate (A…X.) and the pair state is immediately Consistent.
In this example, the device pairs are operating in SRDF Adaptive Copy Disk Pending Mode (C.D.)
Note that the transition into SRDF/A is immediate (A..X.), and the group has been enabled for
consistency. However, the pair state is SyncInProg. R2 device does not have consistent data until at
least two cycle switches have occurred.
The transition takes two cycle switches to complete and may not be possible to do if there isn’t enough
cache on the R2 side. When the request to transition from SRDF/A to SRDF/S is received, the
Consistent Deactivation transition happens in three phases.
In the first phase, SRDF/A operates normally. Then at the next cycle switch, which guarantees an
empty capture cycle at the R1 side, the second phase occurs.
In the second phase, new writes at the R1 side are sent directly in synchronous mode to the R2 side,
with one key exception. As these write arrive at the R2, they are stored in receive delta set at the R2
side. The transmit delta set data from the R1 side also flows into the receive delta set on the R2 side. At
the next cycle switch (two switches into the process), a transition to the third phase occurs.
When all data from the transmit delta set has been received on the R2 side, that data as well as the
accumulated writes that came across in synchronous mode are applied to the R2. If a failure occurs
before that final cycle switch, data in the incomplete receive delta set is converted to invalid tracks but
not written to disk. Extra cache is needed on the R2 side to hold the additional synchronous data that
needs to be stored on the R2 Symmetrix.
SRDF/A Transmit Idle is a feature of SRDF/A that provides it with the capability of dynamically and
transparently extending the Capture, Transmit, and Receive phases of the SRDF/A cycle while masking
the effects of an “all SRDF links lost” event.
Without the SRDF/A Transmit Idle feature, an “all SRDF links lost” event would normally result in the
abnormal termination of SRDF/A. SRDF/A would become inactive. The SRDF/A Transmit Idle feature
has been specifically designed to prevent this event from occurring. Transmit Idle is enabled by default
when dynamic SRDF groups are created. When all SRDF links are lost, SRDF/A still stays active. The
capture cycle grows in cache. Eventually if and when a cache full condition occurs, then SRDF/A will
become inactive. A further feature called Delta Set Extension (DSE) discussed later can enable paging-
out and paging-in of data from/to cache, in the event cache threshold is reached.
Note that the session is still active, and that the RDF pair state has become TransIdle.
SRDF/A Delta Set Extension (DSE) provides a mechanism for augmenting the cache-based Delta Set
buffering mechanism of SRDF/A with a disk-based buffering ability. This extended Delta Set
buffering ability may allow SRDF/A to ride through larger and/or longer SRDF/A throughput
imbalances than would be possible with cache-based Delta Set buffering alone.
Delta Set Extension (DSE), extends the cache space available for SRDF/A session cycles by off
loading some or all of its cycle data from cache to preconfigured disk storage, or pools, which are
similar to SAVE device pools.
Using SRDF/A DSE pools provide a safety net for running SRDF/A sessions; however, SRDF/A will
continue to operate normally with this feature disabled.
SRDF/A DSE Pools and Save devices are managed in the same way as TimeFinder/Snap pools.
A SRDF group can have at most one pool of each emulation.
A single rdfa_dse pool can be associated with more than one SRDF group, similar to snap pools
shared by multiple snap sessions.
SRDF/A DSE Threshold sets the percentage of cache used for SRDF/A that will start offloading
cache to disk.
DSE must be enabled on both the source and target arrays. Extension on only one side of the links
would lead to failure of the SRDF/A recovery with SRDF/A dropping because the R2 side would
fail to have enough cache to hold the large and extended Transmit cycle.
SRDF/A has always buffered Delta Set data in cache. However, a buffer full condition causes SRDF/A
to drop. With SRDF/A DSE, delta set data is offloaded to disk buffers (DSE Pools) by the DSE task.
SRDF/A DSE should be used with the Transmit Idle feature. Thus, SRDF/A can ride through a
temporary link loss. Once the DSE threshold is reached, data is paged out to the DSE Pools.
The SRDF/A write pacing feature, available with Enginuity 5874, helps secure the availability of an
SRDF/A session by preventing conditions that cause cache overflow on both the R1 and R2 sides.
SRDF/A detects when the SRDF I/O service rates are lower than the host I/O rates, and then takes
corrective actions to slow down the host I/O rates to the match the SRDF I/O service rates. This is a
dynamic feature applied at the SRDF group level. It can work in conjunction with the SRDF/ADSE
and transmit idle functionality.
SRDF/A write pacing can also monitor and respond to spikes in the host write I/O rates and slowdowns
in data transmittal to R2 and R2 restore rates. By monitoring and pacing the host write I/O rates, the
SRDF/A write pacing feature controls the amount of cache used by SRDF/A. This prevents cache
overflow on both R1 and R2, and helps the SRDF/A session to stay up and running.
You configure SRDF/A write pacing when setting the following parameters in the symconfigure set rdf
group command. These parameters are highlighted in the following command syntax:
• rdfa_wpace_delay sets the maximum host I/O delay, in microseconds, that the SRDF/A write
pacing can cause.
• rdfa_wpace_threshold sets the minimum percentage of write pending cache that can be used before
this feature begins pacing the host write I/Os for the SRDF group.
• rdfa_wpace_autostart specifies whether to automatically enable SRDF/A write pacing when the
SRDF/A session is activated for the SRDF group.
Note: In a cascaded SRDF environment, SRDF/A write pacing is only supported on the R1-->R21 hop
of the relationship.
As noted earlier, during resynchronization, R2 does not have consistent data. A copy of the consistent
R2 data prior to resynchronization can safeguard against unexpected failures during the
resynchronization process. When the link is resumed, if there are a large number of invalid tracks owed
by the R1 to its R2, it is recommended that SRDF/A not be enabled right away. Enabling SRDF/A right
after link resumption causes a surge of traffic on the link due to (a) shipping of accumulated invalid
tracks, and (b) the new data added to the SRDF/A cycles. This would lead to SRDF/A consuming more
cache and reaching the System Write Pending limit. If this happens, SRDF/A would drop again. Like
with SRDF/S, resynchronization should be performed during periods of relatively low production
activity.
Resynchronization in Adaptive Copy Disk mode minimizes the impact on the production host. New
writes are buffered and these, along with the R2 invalids, are sent across the link. The time it takes to
resynchronize is elongated.
Resynchronization in Synchronous mode impacts the production host. New writes have to be sent
preferentially across the link while the R2 invalids are also shipped. Switching to Synchronous is
possible only if the distances and other factors permit. For instance, if the norm is to run in SRDF/S
and toggle into SRDF/A for batch processing (due to higher bandwidth requirement). In this case, if a
loss of links occurs during the batch processing, it might be possible to resynchronize in SRDF/S.
In either case, R2 data is inconsistent until all the invalid tracks are sent over. Therefore, it is advisable
to enable SRDF/A after the two sides are completely synchronized.
In this example there is a workload on the devices in SRDF/A enabled state. A permanent loss of link
place the devices in a Partitioned state. Production work continues on the R1 devices and the new
writes arriving for the R1 devices are marked as invalid or owed to the R2. SRDF/A is dropped and
session is Inactive.
When the links are active again, note that the pair state has moved to Suspended. Even though the
Mode indicates SRDF/A (A..X.), SRDF/A session is inactive.
As mentioned, we will next place the device group in Adaptive Copy Disk mode (C.D..). As
consistency was enabled when the links were lost, we have to first disable consistency before changing
the mode to Adaptive Copy Disk. The RDF pair state is still Suspended. Changes made to the R1s
accumulate as R2 invalid tracks.
The pair state can now be resumed (symrdf resume). Once the RDF pair state goes to
Synchronized, the mode can be changed to Asynchronous and consistency enabled.
symrdf set mode async
symrdf enable
The symrecover command can be run manually from the command line, but more commonly will be
setup to run continuously in the background by using the Windows Scheduled Tasks, the UNIX
CRON/scheduled. The symrecover command can be run from either the R1 or the R2 side, as long as
all the devices in the monitored group are fully viewable from the execution host.
However, when concurrent RDF is used, this command must be run from the R1 side. To manually
start symrecover for an SRDF/A composite group named RDFAmon, using the options file named
cg_mon_opts, enter:
symrecover start -cg RDFAmon -mode async -options cg_mon_opts
Again, it is advisable to make a copy of the R2 prior to executing a failback operation. When workload
is resumed on the R1 devices immediately after a failback, accumulated invalid tracks have to be
synchronized from the R2 to the R1, and new writes must be shipped from the R1 to R2. If there is an
interruption now, data on the R2 is not consistent. Even though SRDF/A can be enabled right after a
failback, for reasons stated earlier, it should be enabled after the SRDF pairs entered the Synchronized
state.
SRDF group 102 is configured to use RF-7G and 8G, SRDF group 101 is configured to use RF-7H and
8H.
Devices in SRDF group 101 are in asynchronous mode and are consistent. Note that the current cycle
number for this SRDF/A session is 155.
Devices in SRDF group 102 are in asynchronous mode and are consistent. Note that the current cycle
number for this SRDF/A session is 28. Cycle switches for the two SRDF groups occur independent of
each other.
Loss of link for RF-7H and RF-8H causes the devices in the SRDF group 101 to go into a partitioned
state. SRDF/A is dropped and the session status is Inactive.
How ever as links for RF-7G and RF-8G are still available, the SRDF group 102 continues to be in
SRDF/A and proceed with its cycle switches.
Since Enginuity 5671, consistency protection for SRDF/Asynchronous devices is provided using Multi
Session Consistency (MSC). If one or more source (R1) devices in an SRDF/A MSC enabled SRDF
consistency group cannot propagate data to their corresponding target (R2) devices. The MSC process
suspends data propagation from all R1 devices in the consistency group, halting all data flow to the R2
targets. RDF daemon (storrdfd) performs cycle-switching and cache recovery operations across all
SRDF/A sessions in the group. This ensures that a consistent R2 data copy of the database exists at the
point-in-time any interruption occurs. If a session has devices from multiple Symmetrix arrays, then
the host running storrdfd must have access to all the arrays to coordinate cycle switches. It would be
recommended to have more than one host with access to all the arrays running the storrdfd daemon. In
the event one host fails, the surviving host can continue with MSC cycle switches.
A composite group must be created using the RDF consistency protection option (-rdf_consistency)
and must be enabled using the symcg enable command before the RDF daemon begins monitoring and
managing the MSC consistency group.
The RDF process daemon maintains consistency for enabled composite groups across multiple
Symmetrix arrays for SRDF/A with MSC. For the MSC option (-rdf_consistency) to work in an RDF
consistency-enabled environment, each locally-attached host performing management operations must
run an instance of the RDF daemon (storrdfd). Each host running storrdfd must also run an instance of
the base daemon (storapid).. Optionally, if the Group Naming Services (GNS) daemon is also running,
it communicates the composite group definitions back to the RDF daemon. If the GNS daemon is not
running, the composite group must be defined on each host individually.
How does Multi Session Consistency work?
In single session, SRDF/A cycle switch occurs when the Transmit cycle on the R1 side AND the Apply
cycle on the R2 side are both empty. The switch is controlled by Enginuity.
In MSC, the Transmit cycles on the R1 side of all participating sessions must be empty, and also the
corresponding Apply cycles on the R2 side. The switch is coordinated and controlled by the RDF
Daemon.
All host writes are held for the duration of the cycle switch. This ensures dependent write consistency.
If one or more sessions in MSC complete their Transmit and Apply cycles ahead of other sessions, they
have to wait for all sessions to complete, prior to a cycle switch.
There are three ways the RDF daemon can be started. If the RDF daemon is enabled, the daemon is
started automatically by the Solutions Enabler libraries the first time they attempt to connect with it,
which can cause a slight delay in performance on that initial connection while the daemon starts and
builds its cache.
Prior to starting storrdfd, ensure that your default SYMAPI configuration database is up-to-date,
since storrdfd uses the stored information to establish contact with your Symmetrix arrays.
There are three ways the RDF daemon can be started. First, if the RDF daemon is enabled, the daemon
is started automatically by the Solutions Enabler libraries the first time they attempt to connect with it,
which can cause a slight delay in performance on that initial connection while the daemon starts and
builds its cache.
Second, the daemon can be started manually via the stordaemon command line utility as follows:
stordaemon start storrdfd [-wait Seconds]
By default, the stordaemon command waits 30 seconds to verify the daemon is running. To override
this, use the -wait option.
Third, the daemon can be set to start automatically every time the local host is booted using the
following command line:
stordaemon install storrdfd -autostart
Pre-starting the daemon, either manually or via the automatic option, is useful because the daemon
may take a while to initially construct its cache - depending on the number of groups and Symmetrix
arrays it has to load.
We wish to control SRDF groups 101 and 102 as one entity. A composite group has been created and
enabled using the following commands:
The cycle numbers for both the SRDF groups are now reset to the same value. Loss of link for one
group suspends propagation of data for the other group as well. When the links are restored, recovering
from this state can be accomplished as usual:
symrdf –cg srdfacg establish
Once the invalid tracks are marked, merged, and synchronized, MSC protection is automatically re-
instated; i.e., user does not have to issue symcg –cg srdfacg enable again.
The first scenario is easy to understand. In this instance, the Receive cycles contain the most recent and
consistent data.
The second situation arises if there is a failure when some Receive cycles are complete while the
others are in transit. In this case, clearly it is only the Apply cycles of all sessions that contain the
consistent data. Therefore, ALL Receive cycles are discarded.
To understand the third scenario, keep in mind the following: For a cycle switch to be initiated, ALL
Transmits must be empty and all Apply’s must be empty. This means that the failure has occurred
DURING the cycle switch process in this case. Receive can only be promoted to Apply on a cycle
switch. In MSC, cycle switch is sent to all sessions at once. So each of them is in the process of
executing a cycle switch. For example, failure occurred prior to promoting some Receives to Apply.
Therefore, the Receives not yet promoted should be committed along with those that have already been
promoted.
Each Symmetrix has an array-wide Max # of System Write Pending Slots limit (generally
calculated as 80% of available cache slots).
The purpose of this limit is to ensure that cache is not filled with Write Pending (WP) tracks,
potentially preventing fast writes from hosts, because there is no place to put the I/O in cache.
SRDF/A creates WP tracks as part of each cycle.
session_priority = The priority used to determine which SRDF/A sessions to drop if cache becomes
full. Values range from 1 to 64, with 1 being the highest priority (last to be dropped).
minimum_cycle_time = The minimum time to wait before attempting an SRDF/A cycle switch. Values
range from 1 to 59 seconds, minimum is 3 for MSC.
rdfa_transmit_idle = Indicates whether this group has transmit idle support enabled.
rdfa_dse_pool = The name of a collection of SAVE devices used for SRDF/A DSE.
emulation = The pool emulation type.
rdfa_dse_threshold = Specifies the percentage of the Symmetrix array’s write pending limit. Once the
cache usage of all active groups in the Symmetrix array exceeds this limit, data tracks for this group
start to spill over to disks. Valid values are from 20 to 100. The default value is 50.
rdfa_dse_autostart = Specifies whether SRDF/A DSE is automatically activated when SRDF/A session
is activated for the group. Valid values are ENABLE or DISABLE. DISABLE is the default.
This output was captured from the R1 side. As displayed, the Minimum cycle time for each of the
SRDF/A session is at the default of 30 seconds. The Active Cycle is the Capture and the Inactive is the
Transmit, as this output is from the R1 (source) perspective.
From the R2 prospective the Active Cycle is Apply and the Inactive is Receive.
Note that the Cache Slots available for all SRDF/A sessions is 94% of the System Write Pending Limit
(121653/129419). Both the sessions have a default priority value of 33.
The dropping of the SRDF/A session can also be displayed via the symevent command.
This process can be cumbersome if the application is growing rapidly and more devices have to be
added to an active SRDF/A group frequently.
Consistency Exempt feature requires Enginuity 5773.150 and above, as well as Solutions Enabler 7.0.
Consistency exempt can be set from either side. R2 will report the consistency exempt state if R1 is not
available. Note that the attribute can only be removed by Enginuity. There is no CLI command to remove
this attribute. Solutions Enabler 7.0 provides a flag to set the consistency exempt attribute, but none for
removing the attribute. With consistency exempt, the existing devices in an active SRDF/A session need not
be suspended when adding new devices to the session. Consistency is maintained for the existing devices.
The new devices are excluded from the consistency calculation until they are synchronized, move into a
consistent state, and the consistency exempt attribute has been removed.
Moving the device will clear the Consistency Exempt indicator from the SRDF mirror in the group from
which it is moved. However, if the –cons_exempt flag is used with movepair operation, then the
Consistency Exempt indicator will be set when the device is moved into the new SRDF group.
Operations such as suspend, resume, and establish can now be performed on a subset of devices in a
SRDF/A session, by using the consistency exempt flag. How ever, operations such as split, failover cannot
be performed on a subset of devices.
It is critical to wait for the new devices to go into a Consistent RDF Pair state, before using the R1 device
for application data. As long as the Consistency Exempt attribute is set, data on the R2 is not guaranteed to
be consistent with the primary data on the R1.
With Solutions Enabler 7.0 and Enginuity 5874, a subset of the devices in the SRDF/A session can be
suspended. This requires the use of the Consistency Exempt flag. The device pairs can then be moved to a
different RDF Group. The movepair operation also requires the Consistency Exempt flag, if the device is
moved to another active SRDF/A group. Otherwise, Consistency Exempt flag is not required.
RDF Group number 9 contains two devices in an active SRDF/A session. The device pairs are consistent
and the SRDF Group is enabled for SRDF Consistency. There is a new field added to the Modes –
(Consistency) (E)xempt
A new SRDF device pair is created into a different SRDF Group, Group number 10 in this example.
After synchronization, the new device pair is suspended and then moved from the current SRDF Group 10,
to the existing SRDF/A session (SRDF Group number 9), with the consistency exempt flag.
Querying the SRDF/A session shows that the new device (device 0152) has the Consistency Exempt flag
set. There is a new field in the output of the query command – RDFA Consistency Exempt
Devices : Yes. Note that the existing devices continue to be Consistent. Adding a new device does not
compromise the Disaster Recovery capability.
With the consistency exempt flag set, the new device pair alone can be resumed in the SRDF/A session.
Once synchronization is complete, and two cycle switches have occurred, Enginuity removes the
Consistency Exempt flag. The SRDF Pair State moves to Consistent. Only now it is safe to use the R1
device for production work.
These are the key points covered in this module. Please take a moment to review them.
The objectives for this module are shown here. Please take a moment to read them.
SRDF/AR allows users to automate the sequence of SRDF and TimeFinder mirror operations. The
automated sequence - cycle - is performed on a user-defined interval called cycle time.
The replication cycles automatically loop indefinitely or to the number of cycles specified by the users.
Users perform all SRDF/AR operations, setup, start, stop, restart, and query, through the
symreplicate command. Even though the SRDF link can be set to all SRDF operational mode,
except Asynchronous, it is usually set to operate in Adaptive Copy mode due to the long distance
between local and remote sites. This allows the users to save on network bandwidth, thus minimizing
the network costs without compromising the integrity of the data.
The copy path for a single-hop configuration is from the local R1/BCV pair (1) to the SRDF pair (2) to
the remote BCV pair (3). The remotely associated BCV holds the DBMS restartable copy. The amount
of data loss is a function of the replication cycle time (period of time between the start of one copy
cycle and the start of another copy cycle). Copy cycle time is affected by distance, bandwidth, I/O
update rate, and locality of reference for the updates. Update rate and locality of reference tend to
equate to changed tracks. The maximum data loss would be one copy cycle, this makes the RPO ~
Twice the Cycle Time.
Single Hop benefits include:
The ability to perform incremental resynchronization between the intermediate SRDF target site
and the final SRDF target site, reducing required network bandwidth
Reduction in communication link cost and improved resynchronization time for long-distance
SRDF implementations
The copy path for a multi-hop configuration is from the local SRDF pair (1) to the remote BCV pair
(2) to the remote SRDF pair (3) to the Target BCV (4). If your configuration does not include Target
BCVs, the path stops at (3).
Automated replication with the BCVs at Target is applicable if you want a zero data loss solution but
cannot risk the loss of both the Local site and Bunker site at the same time. With this configuration,
there are two possible disaster restart possibilities:
If only the Local site is lost, the result is zero data loss at the Target restart site.
If both the Local and Bunker site are lost, the result is a DBMS restartable copy at the Target
restart site with controlled data loss. The amount of data loss is a function of the replicate copy
cycle time between the Bunker site and the Target restart site.
Multi-Hop benefits include:
The ability to perform incremental resynchronization between the intermediate SRDF target site
and the final SRDF target (Multi-Hop) site, reducing required network bandwidth
Reduction in communication link cost and improved resynchronization time for long-distance
SRDF implementations
The ability to use the SRDF Multi-Hop site to provide disaster recovery testing, point-in-time
backups, decision support operations, third-party software testing, and application upgrade testing
or the testing of new applications
Standard devices 139:13A are added to the device group srdfardg. Local R1/BCV devices 155:156
are then associated with this device group. Remote BCV devices 155:156 are then associated with this
device group. Note the use of –rdf and –bcv flags for this operation.
Shown below is an extract from the symdg show srdfardg command. Note that the remote BCV
devices are given the Logical Device Name of BRBCV.
There are six steps to prepare the state of all the mirrors involved in SRDF/AR Single Hop prior to
running the symreplicate command.
To begin a replication session in the single-hop configuration, the mirrors must be in the following
states:
The local STD/BCV pairs must be fully synchronized
The RDF pairs must be in a suspended state
The remote BCV pairs must be in a split state
Note: The user must wait and check for full synchronization before proceeding to the next step and
issuing the next command.
In step (1), if it is the first time that the STD devices are paired with these R1/BCV devices, a full
establish operation is required. The establish operation in step (1) automatically suspends the RDF
links between the R1/BCV – R2 device pairs. After verifying that the STD – R1/BCV pairs are in a
synchronized state, they can be split in step (2). The RDF link between R1/BCV – R2 device pairs is
resumed in step (3). After R1/BCV – R2 device pairs are synchronized, the R2 – BCV device pairs can
be established in step (4). Again as this is the first time these devices are paired, a full establish
operation has to be performed. After synchronization of R2 – BRBCV pairs, they can be split in step
(5). Finally the STD – BCV device pairs can now be incrementally established in step (6).
The symreplicate command set invokes a session that generates automated, recurrent background
copies of the standard data across an RDF link and cascading BCVs. The symreplicate command set
will start, stop, and restart SRDF/AR session.
The option file is a text file that contains parameters such as SRDF/AR hop type and cycle time. This
file is required and is used in conjunction with the symreplicate command to start a replication session.
This slide lists a subset of parameters that can be specified. There are a minimum of two parameters
that must specified for proper session behavior. They are: SYMCLI_REPLICATE_HOP_TYPE
=<single|multi> and one of SYMCLI_REPLICATE_CYCLE or SYMCLI_REPLICATE_CYCLE_DELAY (a
non-zero value must be specified).
Refer to EMC® Solutions Enabler Symmetrix® SRDF® Family CLI, Version 7.0. Product Guide
P/N 300-000-877 REV A11 for all the possible options and their descriptions.
A simple options file (above) specifies a single-hop cycle time of 10 minutes, to be repeated 3 times,
log each step of the cycle and wait for the next cycle time boundary to start a new cycle if the prior
cycle over runs the cycle time.
This slide show different combinations of cycle time, delay time, and overflow behavior to achieve
various results. Reference points on the cycle time lines are marked ACT (actual completion time),
NCS (next cycle start), and MDT (minimum delay time). Short cycle and delay times (in minutes)
were chosen for illustration purposes only.
Copy cycles #1 and #2 have the same cycle time (2) and no delay time. When copy cycle #1 runs
longer than two minutes, the overflow setting of “Next” results in a new copy cycle beginning at the
four-minute mark.
Copy cycle #3 finishes before its scheduled cycle time of three minutes, waits two minutes, and begins
copying again at the next scheduled copy cycle (the six-minute mark).
Copy cycle #4 performs continuous copy cycles when its cycle time is set to zero. At the end of a
cycle, the system waits three minutes and then begins another copy cycle, regardless of the overflow
setting.
It is recommended to be generous with time parameters for cycles, and adjust once more information is
collected for various cycles. The times configured in the options file should be configured for worst
case scenarios.
Beginning with Solutions Enabler 6.1, you can display statistical information for cycle time and invalid
tracks by using the symreplicate stats command. The command can be issued by device group (-g) or
composite group (-cg) for a specified Symmetrix (-sid) and information can optionally be written to a
specified log file (-log).
The -all option is the default and will display both the cycle time and invalid tracks statistics. The
following example will display both cycle time and invalid track for device group srdfardg on
Symmetrix 80, and will output the result to a log file
symreplicate -g srdfardg -sid 80 -all stats –log c:\temp\srdfardg.log
The start operation fails in the devices are not in a valid initial state. We can specify the –setup option
as noted in the previous slide to automatically correct the problem.
A stopped session can be restarted with the –recover option. The “-recover” option is used to recover
the device locks from the previously started session.
The -all option is the default and will display both the cycle time and invalid tracks statistics.
Determination of the states of the devices and deducing the cycle step using the states can be
performed from a host on the target side, using appropriate device files. These operations can be
performed from a different host on the source side, if the failure only affected the primary host, and the
array and site are still available and accessible.
Since Enginuity 5669, Symmetrix arrays support clustered SRDF/AR environments for multiple node
(host) capability. Clustered SRDF/AR provides the capability to start, stop, and restart replication
sessions from any host connected to any local Symmetrix array participating in the replication session.
The clustered SRDF/AR environment allows the replication log file to be written directly to the
Symmetrix File System (SFS) instead of the local host directory of the node that began the session. If
the primary node should fail, then any locally attached host to the Symmetrix array containing the log
file would then be able to restart the SRDF/AR session from where it left off. If you begin a session
and specify a user log file name (-log), you must specify the -log option for all other commands in the
session sequence.
To write the log file to the SFS, you must specify the ID of the Symmetrix array (-sid) where the log
file is to be stored at the start of the replication session, along with a group name (-g, -cg) and an
optional user log filename (-log).
For example:
symreplicate start -log srdfardg.log -sid 80
Note:
Not specifying the Symmetrix ID (-sid) at the start of the session causes the log file to be written to
local disk using the default SYMAPI log directory, which is not restartable from another node.
RW enabling of R2s can be performed via the symrdf failover command with the appropriate device
file definitions.
These are the key points covered in this module. Please take a moment to review them.
The objectives for this module are shown here. Please take a moment to read them.
Open Replicator for Symmetrix delivers key requirements for platform-independent replication by
being fast, simple, and open. Since it is based on the Symmetrix system, management is common
across all connected hosts, and performance and scalability are superior.
With Open Replicator for Symmetrix, you can create point-in-time copies of local Symmetrix volumes
and transfer them to or from any qualified storage array. Open Replicator offers live and incremental
update capabilities. When migrating data from older arrays to a new Symmetrix, you don’t need to
wait for the data to complete the copy before accessing it locally on the Symmetrix.
A protected track is a track that must be moved before a write to that track is allowed. This term is
used for Snaps, Clones and Open Replicator.
All tracks on the control device are protected after creation of an Open Replicator session.
Use of the -precopy option with the create or recreate commands initiate a data copy immediately in
the background, before the session is activated.
If the background copy option was specified at time of creation, copying of protected tracks starts
directly after activation. Otherwise, copying is deferred until a protected track is written to by the local
host.
During a hot push, the control device can stay available for reads and writes. There is a delay in
completing a write to a protected track until it has been moved to the remote device. This penalty, paid
for the first write to a protected track only, is referred to as the COFW (Copy on First Write).
Subsequent writes to the control device occur without delays.
As in every Open Replicator data transfer, the data on the remote device should not be altered until
data copy has finished. Note that most operating systems alter data on a device, even when the file
system is opened for reads.
Since each FA port on the DMX through which I/Os may arrive to the control devices must act to move
protected tracks over to the remote array, each FA port on the local array to which the control devices
are mapped must be able to access the corresponding remote device on the remote array.
Remember to use the –copy option. Otherwise, no tracks are transferred and the session will sit there
forever.
Although it may be typical to use a BCV as the control device in a cold push, this is not an absolute
requirement. As long as the Control volume is write-disabled (WD or NR) to the host, it can participate
as the source for a cold push. Another option is for the Cold Push device to be a Clone Target volume,
which can be incrementally updated like a BCV from the application volume.
Since there is no COFW penalty paid by incoming host writes during a Cold Push Sessions, up to 16
targets are supported in a single session.
A device is limited to 16 total SDDF sessions:
The active (CREATED, RECREATE, COPYINPROG) session uses an SDDF session for the
protection bitmap.
Differential sessions use an additional SDDF session to track changes since ACTIVATE.
The 15 limit is reached with 15 differential sessions, with one currently active (protection bitmap is
the 16th device SDDF session).
Let us say an Open Replicator push is activated at 10 a.m. and the copy takes an hour to complete. At
11:05 a.m. you issue a recreate followed by an activate command. This has the effect of propagating
the changes made to the control device between 10 and 11:05 a.m. to the remote side. Since only
changes are transmitted, the data transfer does not take as much time.
For the purpose of data integrity, the remote device has to stay untouched by a remote side host
between 10 and whenever the second data transfer finishes.
The operation described above is known as an incremental push.
Each Symmetrix logical volume is allotted a quota of 16 SDDF sessions. These sessions allow the
Symmetrix to track changes by using bitmaps which flip from a zero to a one whenever a track that is
being monitored changes.
SDDF sessions are used to monitor changes in BCVs, Clones, Snaps, Change Tracker, Open Replicator
and SRDF/Star.
For incremental push operations only, data can be restored back to the control device by pulling back
only the changed tracks from the remote device. The session must have been created using the -
differential option and must be in the copied state. Hot or cold differential push sessions can be
restored.
For example, if you copied all data from the control device to the remote device(s) and then made
changes to the control device, you could then recover the original data from the remote device by using
the symrcopy restore command. When the command is issued, the session is recreated in restore
mode and automatically activated. At the start of the restore operation, all control devices are set to
Not Ready status. If running a hot session, control devices are returned to Ready status at the end of
the operation (as the data begins copying). If running a cold session, the control devices remain in Not
Ready status.
If a restore is being performed to a hot push session, new writes to the control are only allowed after
the appropriate track has been restored.
Hot pull is a way to perform migrations with minimal application down time. However, there is a risk
of data loss if the hot pull session is prematurely terminated before the data transfer is complete,
because the writes to the local control volumes are not written back to the remote side. There is a
separate license required for hot pulls.
If the target of the copy is larger than the source, the extra tracks are not copied. If the target is smaller,
the copy fails. However, the –force_copy option can be used to override this limitation. In that case,
only as many tracks as the target can hold are copied.
While the Hot Push exhibits Copy on First Write behavior, Hot Pull exhibits Copy on Access behavior.
A read or a write to the control volume triggers the track to be moved from the remote to the control
volume. There is a performance penalty that is paid the first time a track is accessed.
Upon loss of links, host I/O could stall and cause the application to crash or stall. However, the session
persists unless explicitly terminated. When the links come back, copy proceeds where it left off.
To protect against potential data loss due to a SAN failure or other connectivity issue during a hot pull
operation, you can use the donor update option. When enabled, this feature also causes all writes to the
control device from the host to be immediately copied to the remote device. Because the data is fully
copied to the remote and control devices, if a failure occurs, the session can safely be terminated and
created again to fully recover from any mid-copy failure.
Remember to specify the –copy option with cold pulls. Otherwise, no tracks are transferred.
Data transfer is suspended if the links go down, and resumed after they come back.
Throttling is relevant for hot data transfers where FAs must time share between host I/O and Open
Replicator I/O. It is also relevant for cold transfers if FA bandwidth is being shared. However, in the
case of cold transfers, a separate FA can be set aside for the transfer so production I/O is not affected.
The Pace parameter inserts milliseconds of delay between each track write.
The ceiling value limits the Open Replicator throughput for a port of a specific director.
If you really want to go as fast as possible, remember to set the pace to 0 at creation time or thereafter,
since the default pace is 5.
Ceiling is more practical in real life because this way, you can reserve a fraction of available
bandwidth for host I/O.
The only case where the session is failed after a loss of links is a hot push. This is because you do not
want to hold up production work at the control site waiting for network connectivity to be re-
established.
In the case of a hot pull or push, each FA that has access to the control devices must have access to the
corresponding remote devices. This is because the FA is responsible for pulling or pushing a track from
the remote in case there is a write to the protected track by the control side host.
In the case of a cold pull or push, each FA that has access to the control devices need not have access
to the corresponding remote devices. As long as one FA from the control device can reach the remote
devices, the cold push works.
The steps outlined here are a suggested way to zone two Symmetrix arrays. If you are adept at zoning
and masking, you can certainly deviate from this order and still be successful.
Starting with Symmetrix V-Max series array with Enginuity 5874 and Solutions Enabler version 7.0,
host access to devices is set up using Auto Provisioning Groups, using the symaccess command. This
example shows the list of Masking Views that have been created on the Remote Symmetrix (sid 82). A
Masking View consists of one Initiator Group, one Port Group and one Storage Group.
The Initiator Group winControl_Ports_IG contains yet another Initiator Group – linControl_Ports_IG .
The WWN of the Control Array FA dir:p should be placed in an Initiator Group on the Remote Array
(sid 82) in this case. In effect the Control Array FA dir:p acts as a “hba”, accessing devices on the
Remote Array FA dir:p
As can be seen from the output below, 50000972C002D159 is the WWN of FA 7F Port
1 of the Control Array (sid 80). This FA DIR:P has been placed in
the Initiator Group on the Remote Array (sid 82).
Devices 13D:144 mapped to the Front Director 7F Port 1 are included in the storage group.
TF_COUNT_MODIFIED tracks show modified tracks, while a BCV and standard are split and after a
clone has been activated. SNAP_MODIFIED_TRACKS counts modified tracks after activation is for
clones. It shows how many tracks have changed on the source since clone activation.
Devices 13D and 13E will be used for the Hot Push operation. These will be the Control devices and
they are mapped to Front End director 7F, Port 1. Symmetrix with sid 80 is the Controlling array.
The symsan command lists ports visible from a given director, and LUNs visible behind a given
remote port. It allows a user to make sure that the remote devices intended for use with Open
Replicator are reachable through the SAN.
The display indicates that two different arrays are accessible through director 7F port 1. The WWN of
FA 7F:1 on Symmetrix 82 and that of the service processor on the CLARiiON are displayed.
After identifying the WWNs of the remote ports that can be accessed through the control Symmetrix
fibre port 7F:1, the symsan command can be used to identify the LUNs that are accessible through the
remote FA. As is evident from these two displays, the symsan command vastly simplifies the
verification and troubleshooting of SAN connectivity through the use of one simple command.
Legend:
Flags: (I)ncomplete : X = record is incomplete, . = record is
complete.
(C)ontroller : X = record is controller, . = record is not
controller.
(R)eserved : X = record is reserved, . = record is not
reserved.
(T)ype : A = AS400, F = FBA, C = CKD, . = Unknown
t(H)in : X = record is a thin dev, . = record is not a
thin dev.
(S)ymmtrix : X = Symmetrix device, . = not Symmetrix
device.
We could have used WWNs, but since the remote Symmetrix was discovered through SRDF, using
Symmetrix device numbers is allowed. The environment variable specifies a differential, background
copy for the OR session.
Without the environment variable, we would have had to use –copy –diff options.
Note that all tracks on the two devices are marked as protected, as this is the first time a ORS hot push
session has been created for these two devices. From the legend shown below, we see that a hot push
(S=X, H=X) session has been created with differential (D=X) and background copy (C=X) options
Flags:
(C): X = The background copy setting is active for this pair.
. = The background copy setting is not active for this pair.
(D): X = The session is a differential copy session.
. = The session is not a differential copy session.
(S): X = The session is pushing data to the remote device(s).
. = The session is pulling data from the remote device(s).
(H): X = The session is a hot copy session.
. = The session is a cold copy session.
(U): X = The session has donor update enabled.
. = The session does not have donor update enabled.
(*): The failed session can be reactivated.
Note that the background copy is progressing. The default copy pace is 5. This can be increased if so
desired by using the command symrcopy –f rsym.txt set pace 0 (0 being the highest
pace possible). Also note the use of –consistent flag. As this is a hot push operation, the control
devices are pushing data to remote devices and the control devices are currently online for host I/O
operations. Including the Enginuity Consistency Assist (ECA) option (–consistent) in the
command line. temporarily prevents host I/O while the Open Replicator copy session begins. This
begins a consistent point-in-time copy to the remote devices using an ECA window. As seen earlier
with TimeFinder/Clone and TimeFinder/Snap, this temporarily freezes host I/O to the control devices.
After the activation is completed for all devices, the host I/O is thawed.
When the session is recreated, the changes made to the Control devices after the initial activation are
reflected as protected tracks. As the session was created with –copy –diff flags, subsequent activation
will only move these changed tracks (incremental/differential operation).
The incremental restore action was introduced with Enginuity 5772. It copies the contents of the
remote device to the control device incrementally. For restore to work, the initial OR session must
have been created as an incremental push session. Also, the remote device cannot have changed
between the last push and the restore.
The creation of a hot push session with the –precopy option starts data copy right after session
creation. This cuts down on the time to finish copying protected tracks after the session is activated.
Note that the number of protected tracks is lower than on page 37. This is because 33% of the
protected tracks have already been copied to the remote devices when the query was issued.
After the session is activated, only a few tracks must be copied to the remote devices.
The verify action checks whether the copying of data has finished. An interval can be specified with
verify, such as –i 10, which will check every 10 seconds. Once the copy completes and the
devices are in Copied state, the verification stops.
After the status has reached Copied, the session can be terminated. How ever as the session was
created with –donor_update option, the –force flag is required to terminate. Or donor update
can be turned off and then the session can be terminated.
C:\>symrcopy –f rsym.txt set donor_update off
C:\>symrcopy –f rsym.txt terminate
If the devices were not set to NR, cold push creation would fail with the following error message “The
device is not in a valid Ready status. Operation cannot proceed”.
You can set the pace on the create command line. Note that the session flags (S and H) reflect that this
is a Cold Push.
The commands to create and activate Cold Pull are similar to the commands for other actions. The
Control devices again have to be set to Not Ready, else the create operation will fail.
Prior to Enginuity 5874, cold push operations from a BCV, were only allowed from full volume copies such
as a TimeFinder/Mirror or TimeFinder/Clone control volumes or by making the standard volume not ready.
With Enginuity 5874, an Open Replicator cold push can be performed using TimeFinder/Snap snapshot
volume (VDEV) as the control device. This allows users to perform ORS push operations without requiring
equal amount of space, when they do not want or require a local replica of their data. The applications can
remain on line, and the push can be performed from the VDEV. The VDEV has to be mapped to the front
end directors and ports that will participate in the ORS session. In principle for a cold push, the VDEV
needs to be mapped to just one front end director port.
The steps listed here have to be performed in the exact order. The Snap session should be activated with –
not_ready flag, to set the VDEV Not Ready. If the ORS session is not created with a –copy flag, then both
the states of the SRC Target for the Snap session, and the CTL RMT for the ORS session will be in
CopyOnWrite state. A symrcopy set mode copy operation is required to start copying the data. If the ORS
session is created with the –copy flag, then copy operations start immediately on activating the ORS
session.
These are the steps for performing a Virtual ORS Cold Push operation.
1) Create the TimeFinder/Snap session.
2) Create the ORS session
3) Activate the Snap session
This creates a point-in-time copy
4) Activate the ORS session
ORS copy begins at the time of activation
5) Wait for ORS session copy to complete
To repeat the cycle, the Snap session can first be recreated. Enginuity 5874 now supports recreating
TimeFinder/Snap sessions. Then the ORS session can be recreated, followed by the activation of the Snap
session and then the activation of the ORS session.
These are the steps for performing a Virtual ORS Cold Push operation.
Recreate the SnapSession
Recreate the ORS session
Activate the Snap session
Activate the ORS Session
Terminate the ORS session
Terminate the Snap session
In general, the idea of a Snap is to make the VDEV ready at activation. This ensures that the Business
Continuity host can access data from either the Source device or the Save Pool, as might be necessary. How
ever for ORS session with VDEV, the TimeFinder/Snap activate is performed with a –not_ready flag. This
is a cold push, and there should be no changes to the VDEV when the push session is in progress. Open
Replicator supports multiple cold pushes from a single source. But if the control device is a VDEV, as in
this case, there can be only one ORS session per VDEV.
While this is one suggested order of completing the tasks of zoning and masking between a Symmetrix
and a CLARiiON, if you are experienced with zoning and masking, you are free to perform the steps in
a different order that works for you.
If you do not discover the CLARiiON using SYMCLI, you must find out the WWNs from a
CLARiiON based tool such as Navisphere Manager. Since WWNs tend to be longer and more difficult
to enter, it is often more convenient to “discover” the CLARiiON over the network and get the LUNs
through SYMCLI.
Transferring data between a DMX and non-Symmetrix arrays is not significantly different from
transferring data between two Symmetrix arrays. The arrays, which have been qualified by
engineering, can be found documented in the e-lab navigator available on PowerLink. The principal
challenge is to find the WWNs of the devices on the remote array. Once the WWNs are known, the
Open Replicator sessions can be created over the SAN.
While Symmetrix volumes are configured using the number of cylinders as the lowest unit of measure,
other arrays size their LUNs in 512 byte blocks. Open Replicator pays attention to the size of the
remote and the control array measured in 512 byte blocks. Hence, care must be taken to size the LUNs
appropriately if bi-directional data transfer is planned between remote and control arrays.
When the remote smaller than the control device, you cannot push to the remote device without the
force_copy option. If that option is used, extra tracks on the control are not copied to remote. However,
a pull works because the extra tracks on the control device are simply left untouched.
When the remote device is bigger than the control device, you cannot pull data from the remote device
without the force_copy option. If that option is used, extra tracks on the remote are not copied to
control. However, a push works because the extra tracks on the remote device are simply not written
to.
An excerpt from the log file shows the error message related to the device size discrepancy during an
attempted pull from a larger remote device to a smaller control device.
Open Replicator can be used to perform remote replication between thin devices or between thin and
regular devices. Managing thin device replication with Open Replicator is exactly the same as
managing the replication of regular devices.
Thin devices can be used as control devices for hot and cold pull and cold push Open Replicator copy
operations. If a push operation is done using a thin device as the source, zeroes will be sent for any
regions of the thin device that have not been allocated, or that have been allocated but have not been
written to.
Open Replicator can also be used to copy data from a regular device to a thin device. If a pull or push
operation is initiated from a regular device that targets a thin device, then a portion of the target thin
device, equal in size to the reported size of the source volume, will become allocated.
These are the key points covered in this module. Please take a moment to review them.
The objectives for this module are shown here. Please take a moment to read them.
Almost all replication technologies available on the Symmetrix Arrays can be monitored and managed
via SMC. At the present time, SRDF/Star cannot be managed via SMC.
SRDF and TimeFinder operation in SMC require the use of Device Groups. SMC has visibility to all
the device groups that exist in the symapi_db of the SMC Server host and to all device groups, if GNS
is in use. The Device Group Wizard allows the creation of new device groups. Device groups can be
deleted and renamed and devices can be added or removed as necessary.
The Devices types supported in SMC are shown on the slide. The STD Devices can be Regular (non
SRDF Devices), SRDF R1s, or SRDF R2. Devices can be added as Clone Target devices on the local
or the remote array, and remote Virtual devices can be added. The device group dialog displays only
valid devices when choosing a device type. For instance, when adding remote Clone Target devices,
only eligible devices configured on the remote array display in the dialog.
SMC Device Group creation is a multi-step process, similar to creating SYMCLI device groups using
symdg, symld, and symbcv. The wizard is launched by right clicking the Device Group Folder and
choosing the Device Group Management Create Device Group option.
Drop-down to the Symmetrix ID where the devices reside. In this example SID 80. Click on Device
Type STD and select the devices for Source of Clone session. In this example devices 13D and 13E.
Then click Add. Name the device group, such as clonedg in this slide. The group type has been chosen
to be Regular as we are preparing for a TimeFinder/Clone operation
Next select TGT for the device type and add the two devices which will be Clone targets. Note the
Device Group Summary which reflects the selection. In this case device 0F9 and 0FA have been added
as local Clone targets. Then click OK and the message show will be displayed if Device Group
creation succeeds.
On Page 1 of 2, select Local TGT from the menu, Create for the Action and in this example we have
selected Optimize pairs. Local TGT allows you to pair STD device in the DG/CG with the associated
TGT devices in the group. Device pairs can also be defined by using the Edit Pairs button or by using
Set Exact pair. With Edit pairs, the device pairs can be defined in the Add/Remove Clone pairs dialog.
Then click on Next. In Page 2 of 2, we can see that the –copy and –differential options (default with
Solutions Enabler 7.0, as discussed earlier) are automatically selected by default. Clicking Finish now
will create the Clone session. A message is displayed to that effect. For other Clone operations such as
Activate, right-click on the device group Replication TimeFinder/Clone and then select the
desired command from the Action drop down – such as Activate for the next step in starting a Clone
copy session.
Like in the Clone example, a device group named snapdg has been created. Two STD devices 13F and
140, as well as two VDEVs 131 and 132 have been added to the device group.
Right-click on the device group snapdg brings up page 1 of Snap operations. The create action and the
SAVE device pool have been selected from the drop-down menus. Exact pairs have been set as well.
Clicking on the Next button brings up page 2. Clicking Finish on this page will create the Snap session.
For other Snap operations such as Activate, right-click on the device group Replication
TimeFinder/Snap and then select the desired command from the Action drop down – such as Activate
for the next step in starting a Snap copy session.
A device group srdfdg has been created. Right-click on the device group and navigate to Replication,
then SRDF Settings to set mode of remote replication for the device pair. The options are shown in the
drop-down menu for Set Mode. Choose the desired mode of operation and click OK
Right-click on the device group and navigate to Replication, then SRDF Control to perform SRDF
operations. The options are shown in the drop-down menu for Action. Choose the desired Action such
as Fail Over and then proceed to the next page by clicking on Next. This will bring up the next page
were other option can be selected.
Right-click on the Symmetrix and cascade down to Replication Open Replicator Create Copy
Session menu option.
Select Hot or Cold, Push or Pull radio buttons. Hot-Push operation has been selected for this example.
This is page 1 of 5 pages to complete the ORS create operation. In the next page select the Control
FA:Port and the Control device. In this example, FA 7F:1 and device 0141 has been selected.
On page 3, select the remote device. Clicking on Next brings up page 4 where the Control and Remote
can be selected. Then click on Add.
On page 5, select the Action (Create), optionally name the session (smc_ors), set the pace value
(default=5, it has been changed to 0 for this example), select create options (copy and differential in
this example). Once finished, a ORS session will be created.
After session creation, right-click on the session name and navigate to Replication Open Replicator
Session Control, to Activate the session.
SRDF and TimeFinder/Snap sessions can be monitored via the use of Replication Monitor. Replication
Monitoring has to be enabled on a per Device or Composite Group basis and thresholds (on a per
Symmetrix basis) set for SRDF/A Cache usage, cycle time, and for the TimeFinder/Snap and SRDF
DSE Pool Utilization. CG and DF Replication Monitoring has to be enabled from the Tasks view. Click
on the “Config DG Replication Monitor” or “Config CG Replication Monitor” to enable Replication
Monitoring. Set the Monitor Policy “Polling Interval”, and check the “enable” box.
Select on the Device Group and switch to Replication Monitor View to monitor the status of
replication.
These are the key points covered in this module. Please take a moment to review them.
These are the key points covered in this training. Please take a moment to review them.
This concludes the training.