Remote Replication Technologies: Architecture Overview

A White Paper
By Roselinda R. Schulman

May 2007

Executive Summary
Well-planned business continuity and disaster recovery solutions are critical to organizations operating in 24/7 environments. Minimal to zero disruption is the goal of todays enterprises when faced with planned or unplanned outages. In addition to these business needs, recent government recommendations are driving businesses to look at out-of-region replication options, to enable them to recover from larger-scale events. This paper considers technological approaches to meeting disaster recovery needs, and emerging configuration alternatives and technologies that provide cost-effective solutions for disaster recovery and business resilience. This paper does not address traditional tape backup technology or disk-based backup options, but focuses on remote replication approaches. It familiarizes readers with the vocabulary of software copy alternatives and defines currently available remote copy techniques, including network options for replication. It reviews a variety of remote copy technologies that are widely used for in-region and out-of-region replication, offered by Hitachi Data Systems and other major storage suppliers. And it outlines cost-effective replication approachesincluding two data center and three data center configurationsthat meet a wide range of business needs. In each area of replication technology, this paper shows that Hitachi Data Systems provides customers with a wide range of flexibility and choice in technologies and solutions for both open systems and mainframe environments.

Contents
Introd uctio n ........................................................................................................................................ 1 Reco ver y O bje cti ves : All Dat a Is Not Cr ea te d Eq ual ............................................................................... 2 Rolling Dis aster s ................................................................................................................................. 5 Network O ption s for R e plic ation ........................................................................................................... 7 IBM ESCON -based Options ............................................................................................................................................ 7

Fibre Channelbased Options .......................................................................................................................................... 7 Curren tly Avai la ble R emot e Cop y T ec hni que s ........................................................................................ 8 Syn chrono us R emot e Co py .................................................................................................................. 9 Asy nchro nous Remo te Co py ............................................................................................................... 10 Hitachi Compatible Replication Software for IBM XRC .................................................................................................. 10 Hitachi TrueCopy Asynchronous Software..................................................................................................................... 11 Hitachi Universal Replicator for the Universal Storage Platform ..................................................................................... 13 EMC SRDF Asynchronous.............................................................................................................................................. 14 PiT-m edi at ed R emot e Cop y ................................................................................................................ 15 Hitachi 3- and 4-Copy Models ....................................................................................................................................... 15 IBM PPRC Global Mirror ................................................................................................................................................. 15 Cascade Copy Techniques ............................................................................................................................................ 15 Three Dat a C ent er Cop y .................................................................................................................... 16 Multi-hop and Traditional Three Data Center Copy........................................................................................................ 16 Advanced Three Data Center Copy with Universal Replicator Software........................................................................ 17 Cascade.......................................................................................................................................................................... 17 Multi-target...................................................................................................................................................................... 18 Ban dwidt h Consi der ations .................................................................................................................. 18 Sum mar y .......................................................................................................................................... 19

Remote Replication Technologies: Architecture Overview

A White Paper

By Roselinda R. Schulman

Introduction
The world has changed significantly in the past few years. Devastating terrorist acts and threats, the seemingly increased frequency of widespread power-grid disruptions, and the emergence of regulatory requirements for infrastructure protection are all placing stringent, yet necessary, data protection requirements on many organizations. Regardless of the industry, as more and more businesses operate in a 24/7 environmentespecially large enterprises where global operations are the normthey need an increasingly competitive edge to maintain profitability and stay in business. In the complex and challenging global environment, well-planned business continuity or proven disaster recovery practices for nonstop data availability have become critical to organizations if they are to survive any type of outage. Most information technologyrelated disasters are actually logical disasters, such as data corruption, viruses, and human error, as opposed to physical disasters like fire, earthquakes, hurricanes, etc. Logical disasters occur all the time and pose a bigger threat to businesses. However, because they are less visible to the general public, logical disasters tend to be taken less seriously. The real challenge lies in your organizations ability to think proactively and deploy best practices and technologies that can be leveraged to maximize business operations instead of adopting a reactive fix-it posture. The true test lies in the ability to prevent outages from occurring in the first place, and minimizing the effects of those incidents when they do occur. Companies today must follow the continuous business paradigm, which combines highavailability solutions with advanced disaster recovery techniques. The ultimate goal is to be able to manage both planned and unplanned situations with minimal or zero disruption. When an unplanned event does occur, the ideal scenario is:
Recovery will happen almost automatically with no loss of data Costs of the solution and resources are minimal Impact to the production environment is zero

While technology is moving forward at a rapid pace to reach this ideal scenario, many other business and technology concerns exist, including some significant trade-offs dictated by technology, budgets, and personnel resources.

Recovery Objectives: All Data Is Not Created Equal

Recovery time objective (RTO) and recovery point objective (RPO), along with their associated costs, are important criteria when evaluating the right solution.
RTO describes the time frame in which business functions or applications must be restored (includes time

before disaster is declared and time to perform tasks).

RPO describes the point in time to which data must be restored to successfully resume processing (often

thought of as time between last backup and when outage occurred). Many solutions are available depending on your organizations recovery objectives. For example, when looking at your RPO, you may be concerned with the cost of some data loss (typically less than five minutes, depending on the replication methodology). You may prefer having the ability to quickly perform a database restart instead of a no-data-loss option. Or, you may prioritize limiting possible impacts to the production environment and ensuring easy recovery (minimizing rolling disaster and database corruption) at the secondary site. Consider the different tiers of recovery available to you (see Figure 1). While Hitachi Data Systems focuses on the higher recovery tiers in terms of its software and hardware replication offerings, we understand that not all data is created equal when it comes to disaster recovery protection (see Figure 2). Remember that these trade-offs are business- and application-driven. A thorough business impact analysis is a good starting point to determine the best course of action for data protection and business continuity. Figure 1. Available Tiers of Recovery
Technology Tier
Tier 1Tape Backup Tier 2a Virtual Tape4 Tier 2bDisk Point-in-time copies Tier 3aSync Tier 3bSync w/failover Tier 4aAsync Tier 4bAsync w/failover Tier 5Three data center

RPO Range
24168 hours 1248 hours 436 hours 02 minutes 02 minutes 05 minutes4 05 minutes4 02 minutes

RTO Range
48168 hours 124 hours 424 hours 18 hours 560 minutes 18 hours

Minimum # Disk Copies

N/A Disk Pool5 33 23 23 23 23 3-7 3,5

Distance
Any Any Any Limited Limited Any Any Any

Regional Disaster Support

Maybe2 Yes Maybe2 No No Yes Yes Yes

3090 minutes 18 hours

Note 1 The RPO and RTO ranges in this table are Hitachi Data Systems estimates based on customer experience in a real-world enterprise application environment and are also based on using the technology after a system-wide disaster. Note 2 If the tapes or disk backups are relocated to an out-of-region recovery site Note 3 Best practice is one additional copy for doing disaster recovery testing without impacting the ongoing replication session Note 4 Network problems will extend the RPO Note 5 Depends on vendor and method deployed

Many organizations still fall into Tier 1, although some of them will also use techniques such as remote logging of data. Typically, the higher the tier, the greater the cost. You can, however, achieve significant improvements in data currency and recovery time with the higher tiers. Figure 2. Data Types and Disaster Recovery
Completely Duplicated/ Interconnected Hot Site

Remote Disk Mirroring

More

Disk Mirroring Shared Disk Disk Consolidation Single Disk Copy Electronic Vaulting

Value of Data

Tape On-site Tape Backup Off-site (trucks) More

Amount of Data
Less Delayed

Less Immediate

Recovery Time

Different types of data require different levels of protection. A data classification is required to assess business criticality and cost to recover.

First, we should consider your environment. As mentioned earlier, all data is not created equal. It is likely that only a portion of a corporations data is critical to its basic operation and that a variety of techniques could be used to secure that data, depending on the criticality of that particular business function. Depending on how intertwined the data and applications are and the degree to which they are segregated, this can become a big undertaking. Many organizations choose not to take this approach, but rather choose to copy everything. This is a trade-off in terms of the cost of potentially having to re-engineer your environment compared to the cost of using a higher tier for all data. For companies that have a fairly local site for replication and significant bandwidth available, the option to copy everything is very attractive. However, this can change as protection from both local and regional disasters becomes necessary. While evolving network technologies help lower the cost of replicating data over significant distances, the price can still be extremely high. Therefore, it may be necessary to prioritize data or use a less current copy of data (for example, a copy that is four hours old) for the second copy. However, newer approaches such as using Hitachi Universal Replicator software with the Hitachi Universal Storage Platform can help by enabling enterprises to support replication needs without over-provisioning bandwidth based on peak-load traffic.

Before we start focusing on the technology, it is important to understand some of the basic terms we use when discussing remote-copy alternatives. Advances in technology have brought new words and phrases, such as real time, point in time (PiT), and snapshot into the language of enterprise-class storage. Copy products are designed to allow an enterprise to replicate, protect, and share data in dynamic new ways. Some of the terms used in copy technology are:
Remote copyThis refers to the mirroring of data, typically in real time, to provide an I/O-consistent remote

copy of that data. The purpose of remote copy is to protect the data in the event of a business interruption at a production location.
Point-in-time (PiT) copyPiT copy refers to a copy of data that is taken at a specific point in time. Ideally, this

copy should be I/O-consistent. PiT copies are used in many ways, including backups and checkpoints. More recently, PiT copies have been used in architected disaster recovery solutions.
Data duplicationThis software duplicates data, as in remote copy or PiT snapshots. Data duplication differs

from data migration in that with data duplication there are two copies of data at the end of the process, while with data migration there is only one.
Data migrationThis software migrates data from one storage device to another. Data migration differs from

data duplication in that at the end of the process there is only one copy of data. One purpose of data migration is to reduce operational complexity and costs for storage system upgrades or equipment refurbishment.
Synchronous replicationRequires the application to wait for the remote site to confirm each write operation

before sending the next write operation. As the replication distance increases, the time lag between synchronous write operations gets longer, and these delays become intolerable for high-volume, write-intensive transaction processing applications.
Asynchronous replicationAllows the production application to continue, and keeps track of the pending

writes until they are complete. A well-designed asynchronous replication solution maintains an I/O-consistent copy at the remote site. It may deliver a somewhat lengthened RPO and RTO, but it does protect the enterprise from major data loss in case of a regional disaster. For further discussion, see the resources in Appendix A. Over the last few years, many significant new technologies in both the software and hardware arenas have come to market. These technologies can reduce time-to-business resumption from days to hours and shorten the downtime required for backups to near zero. When evaluating alternative copy technologies, there are some important points to consider. One is the consistency or integrity of the copy. While replicating data may sound simple in practice, the ability to recover from that copy of the data can be extremely complex. This depends not only on the technology but also on the processes that were employed. We will discuss this in further detail as we look at available technologies with a focus on Hitachi Data Systems solutions. But first, we look at the impact of rolling disasters on disaster recovery strategies and technologies.

Rolling Disasters
Real-time copy products are designed to maintain a duplicate image of data at a remote location so that if the primary location is lost due to a disaster, processing can continue at the second site. Although the concept of replicating updates sounds simple, surviving a disaster is actually extraordinarily difficult and challenging. To address this, three basic disaster recovery requirements should be satisfied by any disaster recovery solution:
Surviving a rolling disaster Preserving write sequencing Emergency restart capability following a disaster

Two discrete points in time define any catastrophic disaster: when the disaster first strikes (the beginning), and when the disaster finally completes (the end). Many seconds or even minutes may follow the beginning of a disaster. The period of time between these two events is the rolling disaster. Figure 3. Rolling Disaster Window
Data is I/O consistent Data can be unusable

Rolling Disaster
Surviving a rolling disaster is the true test of any disaster recovery solution, because data corruption might occur within this rolling disaster window.

The real objective of any disaster recovery solution is to provide the capability to produce an image or I/Oconsistent copy of data at the secondary location, as it existed at a point in time prior to the beginning of the disaster. This can be likened to the state in which data exists after a server or system crash. If update activity during the rolling disaster is also shadowed to the backup site, the backup copy may also be corrupted as the write order cannot always be preserved during this time. We know that the image of the shadowed data is usable at any point prior to the disaster occurring, but the image may not be immediately usable if the potentially corrupted updates that occur during the rolling disaster are copied. In a rolling disaster, the data image may be corrupted due to write sequencing and write dependency. Write sequencing is the notion that the order or sequence of updates to the primary data structure must be maintained to ensure the integrity of the data (see Figure 4).

Figure 4. Crash Recovery Example

Many applications control update sequence Updates applied out of sequence violate data integrity One transaction may involve updates to multiple volumes Databases have been carefully designed to protect integrity A DBMS has no knowledge of remote copy A DBMS does not control remote copy updates
OK? YES Transaction Complete Update Log

OK? YES

NO

Error Recovery

Update Dbase

OK? YES

NO

Error Recovery

Update Log

NO

Error Recovery

The sequence in which a database and a log are updated allows the database management system (DBMS) to instantly recover the database, with data integrity, following any sudden outage.

That means a remote copy solution must be able to replicate the original sequence of updates; failure to do so will result in corrupted data at the backup site. Write dependency implies that there is a logical relationship among a series of updates, and if there is a particular update failure, the sequence of subsequent updates might change. The application controls this write sequence/dependency, but the application has no knowledge of the remote copy. There are different ways to preserve write dependency, and vendors have chosen various approaches in their remote copy products. Hitachi Data Systems believes that customers should look to proven technologies. These include true synchronous remote copy products with appropriate controls, such as freeze functions. A number of asynchronous replication approaches also satisfactorily solve the write-sequencing problem, including Hitachi Universal Replicator software, Hitachi TrueCopy Asynchronous software, Hitachi Compatible Replication software for IBM XRC (formerly known as Hitachi Extended Remote Copy, or HXRC), IBM XRC, IBM GDPS, IBM PPRC Global Mirror, EMC SRDF Asynchronous, and PiT technologies that give the ability to snap a consistent images. For applications that span multiple volumes, including many production databases, the remote copy technology must also maintain consistency across all related volumes. This can be a challenge in a rolling disaster, in which different storage systems or communication links can cease transmitting remote copy updates at different points in time. Remote copy vendors employ a number of different approaches to define consistency groups and to maintain consistent remote copy status across all the volumes in a consistency group.

Network Options for Replication

Before we take a look at the various remote copy technologies, we will consider some of the network options available.

IBM ESCON-based Options

Hardware-based replication techniques had their origins in the S/390 world, and therefore used ESCON channels as the transport mechanism for links between the systems. Today, most customers employ Fibre Channel to transport data between systems. This is available for both open systems and mainframe environments and provides some additional benefits over ESCON-based copy.

Fibre Channelbased Options

Fibre Channel can be used in a manner similar to ESCON, or it can be directly connected using dark fibre and dense wavelength division multiplexing (DWDM) technology. In this environment, we may see a lower total cost of ownership (TCO) due to improved performance and the need for fewer links. Other benefits accrue if Fibre Channel storage area network (SAN) management expertise is already in place: the customer doesnt have to manage yet another type of interconnection, and this network option can fit into the existing SAN infrastructure environment. When looking at Fibre Channel for extended distance copy, there are several options including replication over IP networks connecting into traditional networks, such as point-to-point OC3s. Newer extender technologies also allow you to connect directly through synchronous optical network technologies (SONET) networks with potentially lower overhead. When comparing the different options, you have to consider cost, ease of procurement, and the ability to control the available pipe and quality of service (QoS), so as not to compromise the real-time application by not having enough bandwidth. However, IP replication does seem to offer the ability to cover extreme distances at a lower cost. Hitachi Data Systems and other leading IT and storage vendors were invited to participate in a transcontinental IP storage demonstration named the Promontory Project. Hitachi storage systems were connected to Fibre Channel SANs in California and New Jersey that were bridged (with gigabit IP pipes) with IP storage switches from Nishan Systems (McData, now part of Brocade). This demonstration was undertaken to show that distance and speed are no longer barriers to mass storage architectures and solutions. Storage locations can now be implemented far enough away from primary centers so as not to share common disaster areas. Hitachi TrueCopy Remote Replication software over Fibre Channel was tested over the gigabit IP links; basic features of the software were run without modifications over the IP backbone. While the Promontory Project demonstration involved Nishan Systems, other vendors such as Ciena are currently qualified for replication over IP networks and over more traditional point-to-point protocols with Fibre Channel. Many other vendors are looking to get into this market as well as other emerging network replication protocols, such as iSCSI.

Currently Available Remote Copy Techniques

Organizations have implemented a variety of replication approaches based on server, storage, and network capabilities. Each has its strengths and weaknesses, but storage-based replication is generally considered the best approach for replication of critical data, particularly in heterogeneous server and application environments. This white paper focuses primarily on storage-based approaches, and considers a number of approaches that can be used alone or in combination to meet enterprise disaster recovery requirements: Synchronous remote copy
Hitachi TrueCopy Synchronous software and similar approaches from EMC and IBM

Asynchronous remote copy

IBM XRC and Hitachi Compatible Replication software for IBM XRC (for z/OS and OS/390

environments only)
Hitachi TrueCopy Asynchronous software Hitachi Universal Replicator software for the Universal Storage Platform EMC SRDF Asynchronous

PiT-mediated remote copy (using point-in-time copies for consistency)

Hitachi 3 and 4 copy Model PiT technology (utilizing local consistency groups and ATTIME split for Universal

Replicator software)
IBM PPRC Global Mirror Cascade copytypically EMC (SRDF/AR) and IBM using SRDF Adaptive or PPRC-XD, respectively.

All of these offeringsexcept IBM XRC and Hitachi Compatible Replication software for IBM XRCare available for UNIX, Microsoft Windows, and z/OS and OS/390 operating systems. Implementations differ widely based on customer requirements, and no single alternative will satisfy every customers objectives. In many circumstances there is little choice; for example, if a customers backup location is hundreds of kilometers away, then only asynchronous or PiT copy technologies may be practical. It is also worth noting that not all options deserve equal merit. The concepts of write dependency and write sequencing are not addressed in all remote copy techniques, notably PPRC-XD (Peer-to-Peer Remote Copy) and adaptive copy. Other techniques, such as multi-hop, cascade, IBM PPRC Global Mirror, and Hitachi 3 or 4 PIT copy models address the issue by creating consistent PiT copies on a fairly regular basis, although they all have different characteristics and cost structures. Hitachi Data Systems has the widest range of offerings of any storage vendor, and, our competitors protests to the contrary, this does not make things confusing for the customer. Hitachi has taken the initiative of providing both IBM-compatible and proprietary technologies to give customers the most flexibility in choosing offerings. We use the concept of building blocks for all our software to architect the right solution, based on the customers goals and objectives, rather than trying to fit a square peg solution into a round hole. For example, our flexibility is evident in long-distance replication. Depending on the requirements, we can offer a solution with which a customer could replicate directly to a site halfway around the world, or, for the ultimate protection, we can offer advanced three data center solutions. Other vendors can only offer cascade-style solutions, as they cannot support direct copy for write-dependent applications over distance.

Synchronous Remote Copy

Hitachi TrueCopy Synchronous software is one of three hardware-based, synchronous remote copy solutions from major storage vendors. The other synchronous solutions include PPRC from IBM and SRDF (Symmetrix Remote Data Facility) Synchronous from EMC. In these solutions, an update to the primary data is not allowed to complete to the application until it has also been successfully secured at the secondary location. It is widely but erroneously believed that with synchronous copy you are protected against a rolling disaster. In fact, if the update does not complete successfully at both locations, the action taken is dependent on how the environment has been set up and what controls are in place. Without the correct procedures, the production environment can be impacted and/or the data at the secondary location can be corrupted (see Figure 5). Figure 5. Synchronous Replication
Primary Host Secondary Host

1 2 5
Primary Logical Volume

3 4

Secondary Logical Volume

1. Write to primary logical volume 2. Disconnect from pathpath free 3. Write to secondary logical volume 4. Write complete on secondary logical volume 5. I/O completionapplication posted

The remote link is storage controller to controller. Remote copy activity is serverless and remote copy is at the LUN/volume level. Issues with this solution include performance, distance, and multiple controller coordination.

Performance is another consideration with synchronous copy, and we often get asked what the distance limitation is. The answer is that it depends on the performance sensitivity of your applications. While newer technologies, such as Fibre Channel and DWDM, provide improvements, you still cannot exceed the speed of light. It takes about 1ms for light to travel 199.64km (124 miles); both ESCON and Fibre Channel protocols support multiple round trips when used for data replication. This means that even at short distances there will be overhead when using synchronous remote copy. Having said this, many satisfied customers are using our synchronous technology very successfully. One other benefit of using Hitachi TrueCopy software is full support of IBM GDPS (an IBM service offering) for system failover, workload balancing, and data mirroring on systems spread across two or more sites up to 40km (25 miles) apart. While this may not be of interest to you today, it is an evolving technology that could be an option at a future time. Other vendors also support GDPS, but nearly all of the production GDPS sites around the world are currently using Hitachi storage systems. Additionally, due to the virtualization capabilities of the Universal Storage Platform, synchronous replication is possible from any externalized storage area attached to the Universal Storage Platform.

Asynchronous Remote Copy

Examples in this category include several remote copy technologies: Hitachi Compatible Replication software for IBM XRC, Hitachi TrueCopy Asynchronous software, Hitachi Universal Replicator software, and EMC SRDF Asynchronous. Both XRC and Hitachi-based asynchronous solutions utilizing the Universal Storage Platform support the attachment of other vendors devices externally and therefore allow Hitachi software to be used on other vendors hardware when virtualized. This reduces the complexity for customers who no longer have to manage multiple replication solutions.

Hitachi Compatible Replication Software for IBM XRC

This offering from Hitachi Data Systems is based on the same architecture as IBMs XRC. Its strength is in its ability to survive a rolling disaster. This asynchronous remote copy technique is designed to provide complete data integrity for primary and remote systems. This is accomplished through the use of time stamps that allow the asynchronous updates to be applied consistently by the system data mover (SDM) host software. Time-stamping and SDM algorithms are the heart of XRC technology. While it is often considered a software offering due to the requirement of host software, the hardware has to be able to interface with the software and is therefore not supported by some vendors (see Figure 6). Figure 6. Hitachi Compatible Replication Software for IBM XRC
Primary Host System Data Mover Host

1 Data + Time Stamp 2

ta Da

3
+T im

ta eS

mp
Secondary Logical Volume

Primary Logical Volume

This option provides great data integrity, long distance capability, and time stamps on all updates. Issues with this option include the requirement of host MIPs and a server at the secondary site, and cost (software license at secondary site).

This solution does require a secondary site server and software, but in order to recover quickly from any disaster, that is a requirement anyway. Recent enhancements have improved performance, reliability, and scalability, as well as adding unplanned outage support. EMC has recently added support for XRC, although it is believed the initial version is a very early one and may not incorporate the recent enhancements.

10

Hitachi TrueCopy Asynchronous Software

This technology is unique to Hitachi Data Systems and was developed based on a requirement to achieve consistent disaster recovery solutions in both open and mainframe environmentswithout host software and the need for intermediate systems. Using this technology, our customers can be confident that they can recover mission-critical business functions at facilities across the world, without the encumbrance and overhead that is inherent in other vendors solutions. In a mainframe environment, TrueCopy Asynchronous software utilizes reliable time stamps, along with other information created by the primary system(s), to enable Hitachi storage systems to transmit updates directly to the secondary system without any host intervention. These writes are then buffered in the secondary systems cache, queued, sorted by the time stamp, and written to the correspondent volumes in the same sequence. These rewrites are issued by the primary system over the remote link, checking sequence numbers embedded in records to ensure no records are missing. In this way, I/O consistency is maintained. In open systems, TrueCopy Asynchronous software uses sequence information to achieve the same level of consistency, though currently an application must reside on a single system. In contrast, an application in the S/390 or z/OS environment can be spread across up to four primary controllers, while recovering to a single secondary. Hitachi Data Systems is working on eliminating some of these restrictions in the future. This approach allows the enterprise to achieve data integrity in a significantly simpler and less costly hardwarebased solution, with no impact on server or application performance. TrueCopy Asynchronous software also supports consistency groups, which enable you to logically group applications together for consistency purposes (see Figure 7).

11

Figure 7. Hitachi TrueCopy Asynchronous Software: Write Sequencing and Consistency Groups
Data (5) Data (4) Data (3) Data (2) Data (1) C/TG = Consistency Group

Hitachi TrueCopy Asynchronous Software] Attach Sequence Control

(2) (1) (3) (5) (4)

Sort

#3: Data #2: Data #1: Data #2: Data #1: Data

Asynchronous Transfer with Out of Order delivery

#3: Data #2: Data #1: Data #2: Data #1: Data

P-VOL 0

P-VOL 1

P-VOL 2

C/TG_0

S-VOL 0

S-VOL 1

S-VOL 2

P-VOL 3

P-VOL 4

P-VOL 5

C/TG_1

S-VOL 3

S-VOL 4

S-VOL 5

Primary Storage System

Secondary Storage System

TrueCopy Asynchronous software uses write sequencing and consistency groups to ensure data integrity and allow users to perform operations on single applications.

This capability allows you to execute operations on single applications, for example, during disaster recovery testing. In areas where customers require long-distance replication to ensure business continuity or for highly performance-sensitive environments, Hitachi Data Systems has demonstrated clear leadership with TrueCopy Asynchronous software:
Data integrity is guaranteed for dependent write applications Excellent performance is achieved for both long- and short-distance requirements, due to its asynchronous

nature
Future enhancements will provide additional configuration flexibility

Lets compare TrueCopy Asynchronous software with other asynchronous implementations:

Other implementations are based on sending changed tracks to the secondary system instead of time-stamped

I/Os
Application of changed tracks by other implementations (and not individually time-stamped I/Os) cannot preserve

the original sequence of writes, and therefore should not be used for real-time disaster recovery unless used as part of a properly architected PiT solution.

12

Hitachi Universal Replicator for the Universal Storage Platform

Hitachi Universal Replicator software is an advanced technology for asynchronously replicating data hosted on the Universal Storage Platform or on externally attached storage systems from Hitachi Data Systems and other vendors1. Universal Replicator builds on the proven algorithms for data consistency of TrueCopy Asynchronous software, but differs from other solutions in that it pulls the data to the remote site, instead of pushing it from the primary site. Universal Replicator software gives customers significant flexibility as depending on the bandwidth and resources deployed for the solution allows RPOs to range from near-synchronous to hours. Figure 8 illustrates the Universal Replicator application of performance-optimized, disk-based journaling. In this illustration, primary data volumes at the primary data center are being replicated to a second Universal Storage Platform at the remote data center. Figure 8. Hitachi Universal Replicator Software: Asynchronous Remote Copy
Primary Data Center
Hitachi Universal Storage Platform

Remote Data Center

Universal Storage Platform

Primary Data Volumes

Journal-Read Requests JNL Journal Data Transfers JNL

Remote Data Volumes (asynchronous)

Hitachi Universal Replicator (asynchronous remote copy)

Universal Replicator uses disk-based journal volumes and a pull-style replication engine to move data from the primary site to the remote site.

When collecting the data to be replicated, the primary Universal Storage Platform writes the designated records to cache and subsequently to special set of journal volumes if cache exceeds safe limits for performance. The remote system then reads the records from either cache or the journal volumes, pulling them across the communication link. For large IBM z/OS customers, Universal Replicator software extends the capabilities provided by TrueCopy Asynchronous software and provides the capability to have one consistency group spread across multiple controllers in z/OS environments at both the primary and secondary sites. This provides large organizations with the ability to have up to 64K volumes be consistent to a single point in time, while maintaining a near-synchronous RPO. This is achieved using advanced time-stamping and sequencing capabilities and using a master/slave relationship at the remote site, to maintain consistency (see Figure 9). Other alternatives freeze I/O at the primary site to maintain consistency, thus impacting application performance. This capability also provides a compelling alternative to current XRC customers, by lowering their TCO as this solution does not require any host software at the remote site. In addition, managing many thousands of replicated volumes is accomplished easily using the Hitachi Business Continuity Manager software, when compared to issuing thousands of z/OS commands.

For a discussion of the Universal Storage Platform and business continuity applications of its capabilities, including Universal Replicator, see Business Continuity and the Hitachi Universal Storage Platform, Hitachi Data Systems white paper WHP-163. Customers should check with Hitachi Data Systems on supported platforms and configurations.

13

Figure 9. Universal Replicator Softwares 4 by 4 Architecture

Deciding most latest point among DKCs = (n-2)
Fix as (n-2) Journal Pointer of RCU (n) n-2 (n-2) (n-1) (n) Fix as (n-2) Journal Pointer of RCU (n-1) n-2 (n-2) (n-1) (n) Fix as (n-2) Journal Pointer of RCU (n-2) n-2 (n-2) (n-1) (n) Fix as (n-2) Journal Pointer of RCU (n-1) n-2 (n-2) (n-1) (n)

MCUs

RCUs

Universal Replicator

Master

Universal Replicator

Slave

Universal Replicator

Slave

Universal Replicator

Slave

Universal Replicators disk-based journaling and pull-style replication engine help reduce resource consumption and costs, while increasing performance and operational resilience. In effect, Universal Replicator restores primary site storage to its intended role as a transaction processing resource, not a replication engine. This can also translate to a lower cost of ownership over the life of the storage, as depending on a customers RPO, bandwidth acquisitions may be delayed. This technology also used significantly less storage than a PIT Mediated copy, while delivering better recovery points

Additionally, by using a disk-based journaling technique, Universal Replicator prevents network issues or spikes in workload from causing the replication process to suspend: the overflow is buffered in the journal disks until the replication process can catch up. This is a significant benefit, as recovery from a suspended state (in cache-based replication methods) requires a destructive resynchronization process that causes RPOs to be significantly elongated. By contrast Universal Replicator allows the RPO to improve at the remote site after a network outage as it catches up, versus falling farther behind until the total catch-up is complete2.

EMC SRDF Asynchronous

This is a new asynchronous replication technology from EMC, available only on the EMC Symmetrix DMX storage series. Its technique for preserving write dependency is essentially just honoring writes, without the use of other proven techniques such as sequencing or time stamping.

For a more detailed discussion of Universal Replicator and its application to advanced disaster-recovery strategies, see Universal Replicator Advanced Technology, Hitachi Data Systems white paper WHP-165.

14

SRDF Asynchronous uses the concept of timed cycles known as delta sets, typically every 30 seconds, and captures all host I/O during that period in cache. If the same record is updated more than once, only the most recent update is kept. Any dependent I/O will in theory be in that delta set or in a subsequent one. Once the time is up, SRDF Asynchronous starts another delta set cycle at the primary site and begins to transmit the previous delta set to the remote side cache. Once all the data is received at the secondary site, it then promotes it to an active cycle and the data can be de-staged to the back-end disk at the secondary site. If there is a problem during transmission or a disaster occurs, then the copy of consistent data at the remote site should be the previous completely transmitted delta set. However, with multiple links and network retries, that may be of concern. This technology also appears to use large amounts of cache, as the data is held there much longer than with other asynchronous techniques. EMC claims significant improvement in required bandwidth, but this will be highly data and network dependent. Additionally, since the time RPO of the data will be at least the 30-second cycle and the time required to transmit it, using too little bandwidth will result in elongated RPOs and the possible dropping of the SRDF Asynchronous environment due to cache overload. A new feature called Delta-Set extension is designed to alleviate temporary spikes or network failures, by directing IOs to a disk pool if cache fills up. However, this is not the same as Universal Replicator journals, which are designed to be part of the technology, rather than a temporary buffer.

PiT-mediated Remote Copy

Examples in this category include Hitachi 3- and 4-copy model PiT technology, IBM PPRC Global Mirror, and various cascade copy approaches such as those offered by EMC and IBM.

Hitachi 3- and 4-Copy Models

Hitachi offers 3- and 4-copy PiT-mediated remote copy solutions that allow customers to take a point-in-time copy (snapshot) of data without having to stop or otherwise interrupt the application accessing the data. This is often used when customers have RPOs of many hours, for example, with Tier 2 and 3 applications or maybe replication of a virtual tape library. As an alternative, however, the journaling capability of Universal Replicator gives customers additional flexibility, choice, and potentially lower TCO to traditional PiT copies, This capability has also been significantly enhanced through the use of the ATTIME feature of Universal Replicator software for z/OS to provide an enhanced solution by taking periodic, nondisruptive PiT copies at the remote site. These copies may be split off every 15 minutes or every few hours, according to requirements.

IBM PPRC Global Mirror

This is essentially IBMs implementation of a 3-copy PiT model, but it uses an adaptive copy type technology PPRC-XD along with its FlashCopy. IBM claims it can cycle rapidly, updating the remote PiT copy every 30 seconds. However, as always, it depends on the type and rate of change of the data in addition to available bandwidth.

Cascade Copy Techniques

Cascade copies refer to a method of creating remote PiT copies of data, using PiT copy methods such as EMC TimeFinder or IBM FlashCopy with remote copy technologies such as SRDF Adaptive or PPRC-XD. The approach is similar to Hitachi 3- or 4-copy PiT offerings or PPRC Global Mirror, but there are a few major differences. First, most solutions in this area require interrupting the application in order to spin off the local PiT copy. As this will affect the application, you may not want to do it very often, as the age of the PiT copy may be affected. Typically, 15

these approaches are designed using four copies of data (i.e., two local copies and two remote copies) instead of just two or three. This technology may seem appropriate for an environment where a customer wants to create a PIT copy every four hours; there is some expectation of lower bandwidth requirements in such an environment, since not every update is replicated to the remote site. We will revisit the bandwidth trade-offs in a later section of this paper.

Three Data Center Copy

Up to this point, we have been describing remote replication solutions involving two data centers: either synchronous replication to an in-region recovery site or asynchronous replication to an out-of-region recovery site. To obtain the benefits of both approachescombining in-region and out-of-region replicationenterprises may consider techniques involving three data center (3DC) replication. The following sections describe traditional and advanced 3DC approaches.

Multi-hop and Traditional Three Data Center Copy

These offerings revolve around creating multiple PiT copies of data and require anywhere from two to six copies of data and two or three storage systems. EMCs multi-hop capability was primarily devised for the express purpose of achieving consistency of data over distance, as they did not support any true data-consistent asynchronous technology. The obvious downside of these solutions is the requirement of many additional copies of data and extra storage systems just to achieve consistency over distance, as is still the case for EMC customers who do not have Symmetrix DMX storage systems or who require consistency across more than a single storage system in a mainframe environment (see the discussion of SRDF Asynchronous, above). This is not the case with Hitachi Data Systems replication offerings. Zero data loss over distance is a typical reason to adopt these solutions. By running synchronous and then asynchronous technology, you can essentially get a synchronous-style replication over distance. Its important to remember that in order to protect your enterprise from a rolling disaster, it is unlikely that you will get zero data loss, but you may get close to it. Additionally, you need to review your RTO, as there is a cyclical catch-up phase to the recovery site before you can use the data. How long this takes will depend on the locality of reference3 of the data and the available bandwidth. This catch-up lag is only true of the traditional 3DC approaches; advanced 3DC solutions like those from Hitachi Data Systems (below) do not use PiT copies and therefore do not need a catch-up cycle, which makes them ideal for those customers whose business requirements demand a very quick recovery. Another issue with the multi-hop and traditional three data center copy approaches is the requirement to run synchronous for the first hop, as you need to be close enough to avoid any synchronous performance issues, but far enough away to avoid the impact of a region-wide disaster, as this could significantly affect the RPO of the recovery copy. To mitigate a potential negative dependence on the middle regional site, the best option is to replicate to two or more sites at the same time, including out of region. This approach can work, for example, in an environment where a customer wants to have two copies of data at different locations. In this scenario where you need two copies of data, one fairly local and one over an extreme distance, multi-hop and traditional three data center copy may seem to be a good choice. The first hop provides the local backup, with an additional copy at an alternate location. However, if a problem arises at the local backup site, you have no disaster recovery at all unless you replicate directly from your production center.

Locality of reference refers to the observation that references to data tend to cluster. A particular piece of data, once referenced, is often referenced again in the near future. Similarly, once a particular piece of data is referenced, nearby data is often referenced in the near future.

16

Multi-hop and traditional three data center copy operations are only really applicable in very specific circumstances and for a limited number of customers, and rely heavily on complex scripts and many copies of data. EMC offered multi-hop prior to the availability of SRDF Asynchronous, and may still offer it to customers who have not moved to the DMX product line.

Advanced Three Data Center Copy with Universal Replicator Software

With its introduction of the Universal Storage Platform, Hitachi Data Systems offers additional replication choices that reduce the cost of a 3DC configuration, making this approach affordable for a broader range of enterprises and applications. Unlike traditional methodologies that were complex and expensive to implement due to large numbers of copies, the Universal Storage Platform reduces the number of copies required and uses simple combinations of TrueCopy software and Universal Replicator software to enable two different 3DC configurations: cascade, and multi-target (see Figure 10). Figure 10. Advanced Three Data Center Strategies
Primary
TrueCopy Synchronous

Intermediate
Universal Replicator

Secondary

Cascade

TrueCopy Synchronous

Multi-target
Universal Replicator

With two configuration options available, three data center copy is now an affordable, realistic alternative with TrueCopy Synchronous and Universal Replicator software.

Cascade
This solution uses TrueCopy Synchronous software to maintain a current copy of the production data at an inregion data center. The Universal Storage Platform at the in-region site also cascades the data to an out-of-region recovery site, using Universal Replicator softwares asynchronous replication capabilities. In comparison with other asynchronous replication technologies, Universal Replicator software does not require an additional point-in-time copy of the data volume at the intermediate site, or at the remote site (except for best practices to have a disaster recovery test copy).

17

Multi-target
TrueCopy Synchronous software maintains a current copy of the production data at an in-region recovery data center. At the same time, the Universal Storage Platform at the primary site replicates the data to an out-of-region recovery site, using Universal Replicator softwares asynchronous replication across a separate replication network. This technology also features a Delta Resync capability, as depicted by the dashed line. Should the primary site fail and operations resume at the intermediate site, it allows for customers to resume disaster recovery capability within minutes to the out-of-region site (remote) by only sending the differential data between the intermediate and remote sites. Figure 11 summarizes the RTO and RPO impact of several different disaster recovery approaches based on two data center (2DC) and 3DC replication. Figure 11. Benefits Comparison for Selected Disaster Recovery Configurations
Data Center Strategy
Replication configuration Primary site failure/failover Speed of recovery (RTO) Data currency (RPO) Regional disaster (RTO) Protection after failure outside primary site
*

1DC*
Onsite Synchronous near

2DC
Asynchronous far

3DC Traditional

3DC Advanced
Cascade Multi-target

bad bad bad n/a

better better bad bad

good good good bad

depends depends depends bad

better better good depends

better better good better

1DC strategy represents a traditional tape-based disaster recovery approach.

Alternative replication strategies provide different levels of protection.

The chart shows the relative benefits of these configurations in terms of recovery speed and data currency after a primary site failureand in terms of recovery speed after a regional disaster. The chart also illustrates the impact of a failure outside the primary site, which could affect the ongoing level of protection and recovery capability in case of an additional site failure. For a more detailed discussion of these 2DC and 3DC configurations and their pros and cons, see Hitachi Universal Replicator Advanced Technology, Hitachi Data Systems white paper WHP-165.

Bandwidth Considerations
One of the biggest considerations for your organization is the cost of the solution, and network bandwidth is one of the biggest contributors to that cost. In real-time remote copy, every update is sent to the remote location. If your application executes 100 writes of 10K blocks/sec you need bandwidth that accommodates the writes, plus any control information that is also sent. In PiT copy solutions for disaster recovery, data is only replicated at preset intervals, such as every 15 minutes, every hour, etc. During the period of time that data is not being replicated, for data that is updated at the primary site, tracks are marked as changed by the storage system. If the same

18

record is updated 100 times, then only the last change to the track will be shipped when the data is sent to the secondary location. This in theory means that the bandwidth requirement may be lower when using PiT technologies; however, there are many factors to consider before coming to that conclusion. If the PiT copy is frequentunder six hours, for exampleas is often required, then the chances are that there will not be significant reduction in bandwidth. In certain circumstances, bandwidth requirements may be the same or not significantly different. The bandwidth requirement will be very dependent on locality of reference of the data and how current you want the PiT to be at the remote location. Careful consideration should be used when architecting remote copy solutions with a view to saving network bandwidth. This includes technologies such as cascade, PiT copies, multi-hop, three data center copy, and even SRDF Asynchronous. It is important to understand your data patterns, to try and determine if there are any benefits, and to consider your RPO and RTO. Using too little bandwidth can elongate your RPO and cause other problems in the environment. Where catch-up is required, the available bandwidth will affect the time you can begin your recovery (RTO). Universal Replicator is unique in that customers can improve bandwidth utilization and lower their communication costs by sizing bandwidth for average bandwidth needsnot for peak usage. This simplifies bandwidth planning, and empowers users to better control their RPO in relation to infrastructure and communication costs. In environments in which RPO of the data can be looked at in hours versus minutes, a PiT solution may be a viable alternative to a real-time solution, however Universal Replicator with its journaling capabilities may be a viable alternative.

Summary
Enterprises today must meet business applications service levelsincluding increased resilience and protection from local and regional disruptionswhile dealing with complex infrastructures and tight budgets. Services Oriented Storage Solutions from Hitachi Data Systems enable organizations to precisely match business application requirements to Hitachi storage system attributes (performance, availability, data value, and cost), leveraging the application-centric management capabilities of the Hitachi Storage Management Suite of software. To meet these requirements, Hitachi Data Systems provides customers with a wide range of flexibility and choice in technologies and solutions for both open systems and mainframe environments.
The focus of Hitachi Data Systems product offerings is on standards and interoperability. Universal Storage Platform and Hitachi Lightning 9900 V Series systems provide fully compatible, high-

performance S/390 (PPRC, GDPS, and XRC) solutions; this positions customers to take advantage of future software enhancements.
Hitachi TrueCopy Asynchronous software, Hitachi Universal Replicator software, and advanced 3DC

configurations provide unique, enterprise-wide, simple, elegant disaster recovery solutions. Many copy technologies today can be considered when implementing business continuity solutions, especially when you include traditional backup methods of copying data sets. This is why it is important to choose only from best-of-breed solutions when addressing disaster recovery objectives. Hitachi Data Systems provides not only a superior range of offerings in this area, but also the expertise necessary to help you achieve your disaster recovery goals.

19

Hitachi Data Systems Corporation

Corporate Headquarters 750 Central Expressway, Santa Clara, California 95050-2627 USA Contact Information: 1 408 970 1000 www.hds.com / [email protected] Asia Pacific and Americas 750 Central Expressway, Santa Clara, California 95050-2627 USA Contact Information: 1 408 970 1000 [email protected] Europe Headquarters Sefton Park, Stoke Poges, Buckinghamshire SL2 4HD United Kingdom Contact Information: + 44 (0) 1753 618000 [email protected]
Hitachi is a registered trademark of Hitachi, Ltd., and/or its affiliates in the United States and other countries. Hitachi Data Systems is a registered trademark and service mark of Hitachi, Ltd., in the United States and other countries. TrueCopy is a registered trademark and Universal Storage Platform and Lightning 9900 are trademarks of Hitachi Data Systems Corporation. IBM is a registered trademark of International Business Machines Corporation. Microsoft is a registered trademark of Microsoft Corporation All other trademarks, service marks, and company names are properties of their respective owners. Notice: This document is for informational purposes only, and does not set forth any warranty, express or implied, concerning any equipment or service offered or to be offered by Hitachi Data Systems. This document describes some capabilities that are conditioned on a maintenance contract with Hitachi Data Systems being in effect, and that may be configuration-dependent, and features that may not be currently available. Contact your local Hitachi Data Systems sales office for information on feature and product availability. Hitachi Data Systems sells and licenses its products subject to certain terms and conditions, including limited warranties. To see a copy of these terms and conditions prior to purchase or license, please go to http://www.hds.com/products_services/support/warranty.html or call your local sales representative to obtain a printed copy. If you purchase or license the product, you are deemed to have accepted these terms and conditions. Hitachi Data Systems Corporation 2007. All Rights Reserved. WHP-117-03 LKW May 2007