Sync Replication Live Volume
Sync Replication Live Volume
December 2020
CML1064
Revisions
Revisions
Date Description
May 2014 Merged synchronous replication and Live Volume documents; updated for Enterprise
Manager 2014 R2 and SCOS 6.5
July 2014 vSphere HA PDL update
November 2015 Updated for SCOS 6.7
July 2016 Updated for SCOS 7.1 and DSM 2016 R2
October 2016 Minor updates
February 2017 Updated guidance on MPIO settings for Windows Server and Hyper-V
July 2017 Minor updates
December 2017 Minor updates to section 3.4.1
April 2018 Updated for SCOS 7.3 and DSM 2018
July 2019 Consistent snapshot and Live Volume updates
September 2019 Updated link
February 2020 Minor updates for vSphere 6.7
December 2020 Updated for DSM 2020 R1 and vSphere 7
Acknowledgments
Authors: Jason Boche, Marty Glaser, Mike Matthews, Dan Tan, Mark Tomczik, and Henry Wong
The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of any kind with respect to the information in this
publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.
Use, copying, and distribution of any software described in this publication requires an applicable software license.
Copyright © 2014–2020 Dell Inc. or its subsidiaries. All Rights Reserved. Dell Technologies, Dell, EMC, Dell EMC and other trademarks are trademarks
of Dell Inc. or its subsidiaries. Other trademarks may be trademarks of their respective owners. [12/1/2020] [Technical White Paper] [CML1064]
Table of contents
Revisions.............................................................................................................................................................................2
Acknowledgments ...............................................................................................................................................................2
Table of contents ................................................................................................................................................................3
Executive summary .............................................................................................................................................................6
1 Introduction to synchronous replication ........................................................................................................................7
1.1 Features of SC Series synchronous replication .................................................................................................7
1.2 Synchronous replication requirements ...............................................................................................................7
2 Data replication primer .................................................................................................................................................9
2.1 Replication methods ...........................................................................................................................................9
3 Synchronous replication features ...............................................................................................................................13
3.1 Modes of operation ...........................................................................................................................................13
3.2 Minimal recopy..................................................................................................................................................14
3.3 Asynchronous replication capabilities ..............................................................................................................15
3.4 Multiple replication topologies ..........................................................................................................................16
3.5 Live Volume ......................................................................................................................................................17
3.6 Dell Storage Manager recommendations .........................................................................................................18
3.7 Dell Storage Manager DR recovery..................................................................................................................19
3.8 Support for VMware vSphere Site Recovery Manager ....................................................................................19
4 Synchronous replication use cases ............................................................................................................................20
4.1 Overview ...........................................................................................................................................................20
4.2 High consistency...............................................................................................................................................20
4.3 High availability .................................................................................................................................................22
4.4 Remote database replicas ................................................................................................................................25
4.5 Disaster recovery ..............................................................................................................................................26
5 Live Volume overview ................................................................................................................................................34
5.1 Reference architecture .....................................................................................................................................34
5.2 Proxy data access ............................................................................................................................................36
5.3 Live Volume ALUA............................................................................................................................................37
5.4 Live Volume connectivity requirements ............................................................................................................40
5.5 Replication and Live Volume attributes ............................................................................................................42
6 Data Progression and Live Volume ............................................................................................................................46
6.1 Primary and secondary Live Volume ................................................................................................................46
7 Live Volume and MPIO ..............................................................................................................................................47
7.1 MPIO policies for Live Volume .........................................................................................................................47
Executive summary
Preventing the loss of data or transactions requires a reliable method of continuous data protection. In the
event of a disaster or unplanned outage, applications and services must be made available at an alternate
site as quickly as possible. A variety of data mobility methods, including asynchronous replication, can
accomplish the task of providing offsite replicas. Synchronous replication sets itself apart from the other
methods by guaranteeing transactional consistency between the protected site and the recovery site.
While remote replicas have traditionally provided a data protection strategy for disaster recovery, the disaster
itself and the execution of a disaster recovery (DR) plan involves a period of downtime for organizations.
Replicas along with storage virtualization can provide other types of data mobility that fit a broader range of
proactive high availability use cases without an outage.
This guide focuses on two of the main data protection and mobility features available with Dell EMC™ SC
Series storage: synchronous replication and Live Volume. In this paper, each feature is discussed and use
cases are highlighted where these technologies fit independently or together.
By definition, synchronous replication ensures data is written and committed to both the replication source
and destination volumes in real time. The data is essentially written to both locations simultaneously. In the
event that the data cannot be written to either of the locations, the write I/O will not be committed to either
location, ensuring transactional consistency, and a write I/O failure will be issued to the storage host and
application where the write request originated. The benefit synchronous replication provides is guaranteed
consistency between replication sites resulting in zero data loss in a recovery scenario.
Dell Technologies advises customers to understand the types of replication available, their applications, and
their business processes before designing and implementing a data protection and availability strategy.
Live Volume support: Live Volumes may leverage any available type of replication offered with SC Series
storage including both modes of synchronous (high consistency or high availability) and asynchronous.
Live Volume managed replication: Live Volume allows an additional synchronous or asynchronous
replication to a third SC Series array that can be DR activated using Dell™ Storage Manager (DSM).
Preserve Live Volume (manual failover): In the event an unplanned outage occurs impacting availability of
a primary Live Volume, the secondary Live Volume can be promoted to the primary Live Volume role
manually using DSM.
Live Volume automatic failover: In the event an unplanned outage occurs impacting availability of a primary
Live Volume, the secondary Live Volume can be promoted to the primary Live Volume role automatically.
Live Volume automatic restore: After Live Volume automatic failover has occurred, Live Volume pairs may
be automatically repaired after the impacted site becomes available.
1.2.3 Licensing
Replication licensing, which includes synchronous replication and asynchronous replication, is required for
each SC Series array participating in volume replication. Additionally, a Live Volume license for each array is
required for all Live Volume features. With Dell EMC SC All-Flash storage arrays such as the SC5020F and
SC7020F, the replication and Live Volume licensing are included.
2.1.1 Synchronous
Synchronous replication guarantees data consistency (zero data loss) between the replication source and
destination. This is achieved by ensuring write I/O commitments at the replication source and destination
before a successful write acknowledgement is sent back to the storage host and the requesting application. If
the write I/O cannot be committed at the source or destination, the write will not be committed at either
location to ensure consistency. Furthermore, a write failure is sent back to the storage host and its
application. Application error handling will then determine the next appropriate step for the pending
transaction. By itself, synchronous replication provides CDP. Coupled with hardware redundancy, application
clustering, and failover resiliency, continuous availability for applications and data can be achieved.
Because of the method used in synchronous replication to ensure data consistency, any issues impacting the
source or destination storage, or the replication link in-between, will adversely impact applications in terms of
latency (slowness) and availability. This applies to Live Volumes built on top of synchronous replications as
well. For this reason, appropriate performance sizing is paramount for the source and destination storage, as
well as the replication bandwidth and any other upstream infrastructure that the storage is dependent on.
Figure 1 demonstrates the write I/O pattern sequence with synchronous replication:
The process is repeated for each write I/O requested by the application or server.
2.1.2 Asynchronous
Asynchronous replication accomplishes the same data protection goal in that data is replicated from source
storage to destination storage. However, the manner and frequency that the data is replicated differs from
synchronous replication. Instead of committing a write at both replication source and destination
simultaneously, the write is committed only at the source and an acknowledgement is then sent to the storage
host and application. The accumulation of committed writes at the source volume are replicated to the
destination volume in one batch at scheduled intervals and committed to the destination volume.
Aside from replicating the active snapshot (semi-synchronous replication is discussed in section 2.1.3),
Asynchronous replication in SC Series storage is tied to the source volume replication schedule. When a
snapshot is created on the source volume, and that volume is configured for asynchronous replication, the
new snapshot is replicated to the destination volume. Snapshots on a volume may be created automatically
according to a schedule or manually created from a variety of integration tools. Regardless, all snapshots
occur on a per-volume basis. As a result, volumes may adhere to their own independent replication schedule,
or they may share a replication schedule with other volumes leveraging the same snapshot profile. This type
of replication is also referred to as a point-in-time replication, which is a type of asynchronous replication that
specifically leverages volume snapshots. Because asynchronously replicated transactions are not required to
wait for write committals at the replica destination volume, the replication link and/or destination storage will
not contribute to application or transaction latency at the source volume.
Figure 2 demonstrates the write I/O pattern sequence with respect to asynchronous replication.
The process is repeated for each write I/O requested by the application or server.
4. Periodically, a batch of write I/Os that have already been committed to the source volume are
transferred to the destination volume.
5. The write I/Os are committed to the destination volume.
6. A batch acknowledgement is sent to the source.
2.1.3 Semi-synchronous
With SC Series storage, semi-synchronous replication behaves like synchronous replication in that application
transactions are immediately sent to the replication destination storage (assuming that the replication link and
destination storage have the bandwidth to support the current rate of change). The difference is that the write
I/O is committed at the source volume and an acknowledgement is sent to the storage host and application
without a guarantee that the write I/O was committed at the destination storage. Semi-synchronous replication
is configured in Dell Storage Manager by creating asynchronous replication between two volumes and
checking the box for Replicate Active Snapshot. A snapshot is an SC Series storage term that describes
frozen data. The Active Snapshot refers to newly written or updated data that has not yet been frozen in a
snapshot. Semi-synchronous offers a synchronous-like recovery point objective (RPO) without application
latency, but the RPO and loss of data in an unplanned outage scenario cannot be guaranteed.
Figure 3 demonstrates the write I/O pattern sequence with semi-synchronous replication.
The process is repeated for each write I/O requested by the application or server.
For each write I/O that completes that process, there is an independent and parallel process:
The commits at the source and destination volumes are not guaranteed to be in lockstep with each other.
3.1.1 Legacy
Synchronous replications created prior to SCOS 6.3 are identified as legacy after upgrading to SCOS 6.3 and
newer. Legacy synchronous replications cannot be created in SCOS 6.3 or newer and do not possess the
newer synchronous replication features currently available. To upgrade a legacy synchronous replication to
synchronous high consistency or synchronous high availability replication, a legacy synchronous replication
must be deleted and recreated after both source and destination SC Series arrays have SCOS 6.3 or newer
installed. Deleting and recreating a synchronous replication will result in data inconsistency between the
replication source and destination volumes until 100% of the initial and journaled replication is completed.
The difference between high consistency and high availability mode is that data availability will not be
sacrificed for data consistency. What this means is that if the replication link or the destination storage either
becomes unavailable or exceeds a latency threshold, the SC Series array will automatically remove the dual
write committal requirement at the destination volume. This allows application write transactions at the source
volume to continue, with no downstream latency impacts, instead of write I/O being halted or slowed, which is
the case with high consistency mode and legacy synchronous replication. This relaxed state is referred to as
being out of date. If and when an SC Series array enters the out-of-date state, inconsistent write I/O will be
journaled at the source volume. When the destination volume becomes available within a tolerable latency
threshold, journaled I/O at the source volume is flushed to the destination volume where it will be committed.
During this process, incoming application writes continue to be written to the journal. After all journaled data is
committed to the destination volume, the source and destination will be in sync and the data on both volumes
will be consistent. When the source and destination volumes are in sync, downstream latency will return
within the application at the source volume. Similar to the high consistency mode, application latency and
data consistency are important points to consider in a design that incorporates synchronous replication in high
availability mode.
Note: This feature is compatible with all replication modes except legacy synchronous replication.
exposure as well as the replication link bandwidth consumed to recover. Minimal recopy is also employed in
high consistency mode should the destination volume become unavailable during initial synchronization or an
administrator invoked a pause operation on the replication.
Note: Consistent snapshots may be created for asynchronous and synchronous replications. However,
consistent snapshots are not supported with Live Volumes.
3.3.2 Pause
Synchronous replications configured in either high consistency or high availability modes can be paused
without impacting availability of applications relying on the replication source volume. Pausing replication can
facilitate multiple purposes. For example, it can be used to relieve replication link bandwidth utilization. In
designs where replication bandwidth is shared, other processes can temporarily be given burstable priority.
Pausing may also be preferred in anticipation of a scheduled replication link or fabric outage.
Mixed topology
For recovery purposes, the replica can be activated and mapped by Dell Storage Manager to a storage host
(for instance, at a disaster recovery site).
Furthermore, in a mixed topology, a replica volume may be configured to replicate to another one of the
replicas (asynchronous or synchronous) without having to reseed a majority of the data both volumes would
already have before the original source volume became unavailable. This may be useful where two or more
disaster recovery sites exist.
After DR activation, a replica volume can be replicated to another replica with efficiency
Cascade topology
Hybrid topology
newer, Live Volume is designed to work in conjunction with asynchronous and synchronous replication types.
In addition, Live Volume supports many of the current synchronous replication features such as modes of
operation and mode migration.
Live Volume managed replication before and after swap role or failover
replica. When using high consistency synchronous replication, data between source and destination must be
consistent for DSM to advise it is safe to use the destination replica for recovery.
When using high availability synchronous replication (or high consistency with the ability to pause replication),
the data between source and destination volumes may or may not be consistent depending on whether the
replication was in sync or out of date at the time of the failure. If at the time of failure replication was in sync,
DSM will advise that the destination replica volume is data consistent and safe to use for recovery.
Conversely, if the synchronous replication was out of date, this means journaled transactions at the source
volume likely have not been replicated to the destination and the destination replica is not data consistent and
not recommend for use. At this point, the data recovery options would be to use a data consistent snapshot
as the recovery point or continue with using the inconsistent replica. In either case, the most recent
transactions will have been lost at the destination but recovering from a snapshot will provide a precise point
in time as the recovery point.
SRM version 6.1 support for stretched storage with Live Volume was added in DSM 2016 R1. Supported
deployment configurations are outlined in the document, Dell EMC SC Series Best Practices with VMware
Site Recovery Manager. For more information on use cases and integrating stretched storage with SRM,
please see the Site Recovery Manager Administration documentation provided by VMware.
4.1 Overview
Array-based replication is typically used to provide upper tier application high availability or disaster recovery,
a data protection process to enable image or file-level backup and recovery, or a development tool to
generate copies of data in near or remote locations for application development or testing purposes. For
many business use cases, asynchronous replication provides a good balance of meeting recovery point
objective (RPO) and recovery time objective (RTO) service level agreements without a cost-prohibitive
infrastructure such as dark fibre, additional networking hardware, or additional storage. This is why
asynchronous replication is often used between data centers where longer distances are involved.
However, there are an increasing number of designs where a strong emphasis is placed on the prevention of
data loss. Regardless of where the need originates, the method of replication that satisfies zero transaction
loss is synchronous. The next few sections highlight examples of synchronous replication with a focus on high
consistency for zero data loss or high availability for relaxed data consistency requirements.
Note that Dell Storage Manager DR plans cannot be predefined with Live Volumes. Predefined DR plans are
supported with regular (asynchronous or synchronous) volume replications or with a managed (cascaded or
hybrid) asynchronous replication from a Live Volume.
A replication link or destination volume issue in St. Paul results in a VM outage in Minneapolis
A replication link or destination volume issue in St. Paul results in database outage in Minneapolis
To summarize, there are high consistency use cases that can be integrated with virtualization as well as
database platforms. The key benefit being provided is data consistency and zero transaction loss. Keep in
mind that the infrastructure supporting synchronous replication between sites must perform sufficiently. In the
case of high consistency, the supporting infrastructure must be highly redundant and immune to outages for
slowness or an outage of the replication link or the destination site is reflected equally at the source site
where the end user applications are running.
An important factor when considering the type of replication to be implemented is that the infrastructure
required to keep two sites well connected, particularly at greater distances, often comes at a premium.
Stakeholders may be skeptical about implementing a design where a failure at the secondary site or a failure
of the connection between sites can have such a large impact on application availability and favor
asynchronous replication over synchronous replication. However, with the high availability synchronous
replication offered with SC series storage, customers have additional flexibility compared to legacy
synchronous replication.
In the following examples, note that using high availability mode in place of high consistency mode does not
automatically allow the design to be stretched over further distances without consideration to application
latency. High availability mode is still a form of synchronous replication and should not be confused with
asynchronous replication. Adding significant distance between sites generally results in latency increases
which will still be observed in the applications at the source side for as long as the high availability replication
is in sync.
Finally, if virtual machines are deployed in a configuration that spans multiple volumes, consider using Replay
Manager or consistency groups. Replay Manager is covered in section 11.6.
A replication link or destination volume issue in St. Paul results in no VM outage in Minneapolis
Note: Consistent snapshots may be created for asynchronous and synchronous replications. However,
Consistent snapshots are not supported with Live Volumes.
For storage hosts with data confined to a single volume, special considerations are not necessary. However,
if the host has application data spread across multiple volumes (for example, a VM with multiple virtual
machine disk files, or a database server with instance or performance isolation of data, logs, and other files)
then it becomes critical to ensure snapshot consistency for the replicated data that will be used as a restore
point. Ensuring all volumes of a dataset are quiesced and then snapped at precisely 8:00, for example,
provides a data consistent restore point across volumes supporting the dataset. This snapshot consistency is
accomplished with Replay Manager (especially recommended for Microsoft products through VSS integration)
or by containerizing volumes by use of consistency groups.
To create consistency across snapshots using consistency groups, a snapshot profile is created with a
snapshot Creation Method of Consistent (Figure 18). This profile is then applied to all volumes containing the
dataset. For virtual machines, the volumes would contain the virtual disks representing the c: drive, d: drive,
or Linux mount points such as / or /tmp. For Microsoft SQL Server database servers, the volumes may
represent system databases, application databases, transaction logs, and tempdb. For Oracle databases, the
dataset must contain all volumes containing any part of the database (data, index, data dictionary, temporary
files, control files, online redo logs, and optionally offline redo logs). For Oracle RAC, OCR files or voting disks
can be added to the dataset. For either database platform, separate volumes for hot dumps, archived redo
logs, or boot from SAN may exist but typically would not need to be included in a consistency group with the
key database files.
Note: Consistent snapshots may be created for asynchronous and synchronous replications. However,
consistent snapshots are not supported with Live Volumes.
Another method of capturing consistency in snapshots across volumes (and perhaps more useful for
customers with Microsoft Windows, SQL Server, Exchange, Hyper-V, or VMware vSphere) would be to use
Replay Manager. Replay Manager has the underlying storage integration and VSS awareness required to
create application consistent snapshots, across volumes if necessary, which can then be replicated
synchronously (either mode) or asynchronously.
Once data is frozen with consistency across volumes using Replay Manager or consistency groups, those
snapshots will be replicated to the destination volume where they can serve as historical restore points for
high availability mode recovery, disaster recovery, or remote replicas which will be discussed in the coming
sections.
A replication link or destination volume issue in St. Paul results in no database outage in
Minneapolis
However, if the replica is to be stored on a different array, whether or not it is in the same building or
geographic region, replication or portable volume must be used to seed the data remotely, and replication
should be used to refresh the data as needed. For the purposes of developer or DBA testing, asynchronous
replication may be timely enough. However, for reporting purposes, synchronous replication will ensure the
data is up to date when the reporting database is refreshed using the replicated volumes. The choice of
providing zero data loss through high consistency mode or a more flexible high availability mode should be
decided ahead of time with the impacts of each mode well understood.
SC Series snapshots, as well as asynchronous and synchronous replication, are natively space and
bandwidth efficient on storage and replication links respectively. Only the changed data is frozen in a
snapshot and replicated to remote SC Series arrays. In the following figure, notice the use of high availability
synchronous replication within the Minneapolis data center. Although the two arrays are well connected, the
risk of internal reporting database inconsistency does not warrant a production outage for the organization.
Recovery point objective (RPO): This is the acceptable amount of data loss measured by time or the
previous point in time at which data is recovered from. An RPO is negotiated with business units and
predefined in a disaster recovery plan. In terms of replication, the keys to achieving an RPO target are
choosing the appropriate replication type, making sure replication is current (as opposed to out of date or
behind), and knowing the tools and processes required to recovery from a given restore point.
Recovery time objective (RTO): This is the elapsed recovery time allowed to bring up a working production
environment. Just like RPO, RTO is negotiated with business units and predefined in a disaster recovery plan
and may also be included in a service level agreement (SLA). The keys to achieving targeted RTO may vary
from data center to data center but they all revolve around process efficiency and automation tools wherever
possible. Replication is a quintessential contributor to meeting RTO, especially at large scale.
By leveraging replication, aggressive RPOs and RTOs can be targeted. Data footprint and rate of change
growth may be continuous, but feasible RPO and RTO goals do not linearly diminish as long as the replication
infrastructure (this includes network, fabric, and storage) can scale to support the amount of data being
replicated and the rate of change.
RTO is a paramount metric to meet in testing and executing a live business continuation plan. In a DR plan,
all steps are predefined and executed in order according to the plan; some steps may be carried out in
parallel. Successfully recovered database servers are a required dependency beginning early in the DR plan.
This includes bringing up application and web servers that have a critical tie to the database server. The more
databases a shared database server hosts, the broader the impact because the number of dependent
application and front-end tiers fan out.
Industry analysis reflects data growing at alarming rates across many verticals. Providing performance and
capacity is not a challenge with current technology, but protecting the data is. Data growth drives changes in
technology and strategy so that SLAs, RTOs, and RPOs can still be maintained even though they were
defined when data was a fraction of the size it is today. Restoring 10 TB of data from tape is probably not
going to satisfy a 24-hour RTO. Data growth on tapes means that there is a growing number of sequential-
access, long-seek time tapes for restoration. This diminishes the chances of meeting RTO, and increases the
chances that one bad tape will cause data recovery to fail. Data replication is a major player in meeting RTO.
Intra-volume consistency is extremely important in a distributed virtual machine disk or database volume
architecture. Comparing the synchronous replication modes, high consistency guarantees data consistency
between sites across all high consistency replicated volumes. Unfortunately, this is at the cost of destination
site latency, or worse, downtime of the production application if the destination volume becomes unavailable
or exceeds latency thresholds.
Outside of use cases that require the textbook definition of synchronous replication, high availability mode (or
asynchronous) may be a lot more attractive for DR purposes. This mode offers data consistency in the proper
conditions, as well as some allowance for latency while in sync. However, consistency is not guaranteed if
production application uptime is jeopardized should the destination volume become unavailable.
Because consistency cannot be guaranteed in high availability mode, it is important to implement VSS-
integrated Replay Manager snapshots or consistency groups with high availability synchronous replication
where a multiple volume relationship exists (this is commonly found in both SQL Server and Oracle
environments). While this will not guarantee active snapshot consistency across volumes, the next set of
frozen snapshots that have been replicated to the remote array should be consistent across volumes.
When choosing to access volumes at the DR site, it is important to consider the purpose. Is this a validation
test of the DR plan, or is this an actual declared disaster? As an active replication destination target
(regardless of asynchronous, synchronous, mode, or topology), destination volumes cannot be mounted for
read/write use to a storage host at the DR site. (For circumstances involving Live Volume, refer to section 5).
To perform a test of the DR plan, present view volumes from the snapshots of each test volume to the storage
hosts. Snapshots and view volumes are available for both asynchronous and synchronous replications in
either high consistency or high availability mode. Snapshots and view volumes are beneficial during DR
testing because replication continues between the source and destination volumes to maintain RPO in case
an actual disaster occurs during the test. Conversely, if a disaster is being declared and the Activate Disaster
Recovery feature is invoked, then replication from source to destination needs to be halted (if it has not been
already by the disaster) if the active volume at the destination site is intended for data recovery in the DR
plan.
Unisphere Central for SC Series in Dell Storage Manager is a unified management suite available to SC
Series and Dell FS8600 customers. Unisphere Central has disaster recovery features built into it that can
create, manage, and monitor replications, as well as automate the testing and execution of a predefined DR
plan.
For virtualization and database use cases alike, Unisphere Central is used to create asynchronous or
synchronous replications. These replications predefine the destination volumes that will be presented to
storage hosts for disaster recovery.
Note: Destination replica volumes are for DR purposes only and should not be used actively in a Microsoft or
VMware vSphere Metro Storage Cluster design.
In most cases, predefined destination volumes are data volumes. Where physical hosts are involved, boot
from SAN volumes can also be included in the pre-defined DR plan to quickly and effortlessly recover
physical hosts and applications as opposed to rebuilding and installing applications from scratch. Rebuilding
takes significant time, is error prone, and may require subject matter expert knowledge of platforms,
applications, and the business depending on how well detailed the build process is in the DR plan. Confusion
or errors during test or DR execution lead to high visibility failure. Detailed and current DR documentation
provides clarity at the DR site. Process inconsistency and errors are mitigated by automation or closely
following DR documentation. With these points in mind, the benefit of automating a DR plan with DSM (or a
similar tool) is clear. From the moment a disaster is declared by the business, the RTO is in jeopardy;
automation of tasks saves time and provides process consistency.
Once the volumes are prepared and presented manually or automatically by DSM or API scripting, the
process of data recovery continues. For SQL Server and Oracle database servers, databases are attached
and various scripts are run to prepare the database server and applications for production use (such as to
resync login accounts). For VMware vSphere and Hyper-V hosts, VM datastores are now visible to the hosts
and VMs need to be added to the inventory so that they can be allocated as compute and storage resources
by the hypervisor and then powered on.
In Hyper-V 2008 R2, the configuration file for each virtual machine must be generated with the correct number
of processors, memory, network card, and attached virtual machine disk files. This is a process that is
documented or scripted prior to the DR event. Hyper-V 2012/R2 and newer includes a virtual machine import
wizard that is able to import the existing virtual machine configuration located on replicated Dell storage,
rather than generating a new configuration for each VM from scratch. Once the VMs are added to inventory in
Hyper-V 2008 R2 or Hyper-V 2012/R2 and newer, they can be powered on. All versions of VMware vSphere
have the same capability as Hyper-V 2012/R2 and newer in that once datastores are presented to the
vSphere hosts, the datastores can be browsed and the virtual machine configuration file that is located in
each VM folder can be added to inventory and then powered on. A manual DR process, especially in an
environment with hundreds or thousands of virtual machines, quickly eats into RTO. The automation of
discovering and adding virtual machines to inventory is covered in the next section.
VMware vSphere Site Recovery Manager is a disaster recovery and planned migration tool for virtual
machines. It bolts onto an existing vSphere environment and leverages Dell Technologies™ certified storage
and array-based replication. Both synchronous and asynchronous replication are supported as well as each
of their native features. Live Volume stretched storage is also supported in specific configurations starting with
SRM 6.1. With this support, customers can strive for RPOs that are more aggressive and maintain
compatibility with third party automation tools, like SRM, to maintain RTOs in large VMware virtualized
environments.
For vSphere environments, SRM invokes the commands necessary for tasks (such as managing replication,
creating snapshots, creating view volumes, and presenting and removing volumes from vSphere hosts) to be
performed at the storage layer without removing DSM from the architecture. The storage-related commands
from SRM flow to the Storage Replication Adapter (SRA) and then to the DSM server. For this reason, a DSM
Data Collector needs to remain available at the recovery site for the automation to be carried out. Outside of
SRM, in a heterogeneous data center, DSM or API scripting would be needed to carry out the DR automation
for Hyper-V or physical hosts.
Beyond the scope of storage, SRM automates other processes of DR testing, DR recovery, and planned
migrations, making it a major contributor to meeting RTO goals. SRM takes care of important, time-
consuming tasks such as adding virtual machines to inventory at the DR site, modifying TCP/IP address
configurations, VM dependency, power-on order, and reprotection of virtual machines.
The DSM server must be available to perform DR testing or an actual DR cutover for an automated DR
solution involving SC Series storage replication. This means making sure that at least one DSM server
resides at the recovery site so that it can be engaged when needed for DR plan execution. The DSM server
labeled as a physical server in Figure 22 also represents a virtualization candidate if it is already powered on
and not required to automate the recovery of the vSphere infrastructure where it resides.
Metro and remote DR sites, mixed topology (1-to-N), Fibre Channel and iSCSI replication
Intra campus and metro DR sites, cascade topology, Fibre Channel and iSCSI replication
Intra-campus, metro, and remote DR sites; hybrid topology; Fibre Channel and iSCSI replication
The examples in Figure 26 through Figure 30 serve to represent physical hosts or virtual machines. Some
notable details in the examples are:
• A 1U DSM server is racked at each recovery site to facilitate DR testing and cutover automation
through DSM, VMware Site Recovery Manager, or both. The DSM shown is for logical representation
only. DSM may be a virtual machine.
• SC Series storage supports Fibre Channel and iSCSI-based replication.
• SC Series storage supports asynchronous replication as well as multiple modes of synchronous
replication.
• Either mode of synchronous replication latency will impact production applications relying on the
replication source volume. The infrastructure design should be sized for adequate replication
bandwidth, controllers, and storage at the recovery site to efficiently absorb throughput.
• The examples provide adequate hardware and data center redundancy at the recovery site when
implementing high consistency synchronous replication. Aside from a pause operation, unavailability
of a destination replica volume leads to unavailability of the source volume and will result in a
production application outage.
Live Volume is a software-defined high availability solution that is integrated into the SC Series controllers. It
is designed to operate in a production environment that allows storage hosts and applications to remain
operational during planned volume migration or data movement regardless of the geographical distance
between arrays. The automatic failover and restore features available for vSphere Metro Storage Cluster
integration with SCOS 6.7 and newer, and for Microsoft environments with SCOS 7.1 and newer, provides
high availability for applications and services during planned or unplanned events in the data center.
Live Volume increases operational efficiency, eliminates the need for planned storage outages, and enables
planned migrations as well as disaster avoidance and disaster recovery. The Live Volume feature provides
powerful options:
• Allows storage to follow mobile virtual machines and applications in virtualized environments within a
data center, campus, or across larger distances
• Supports automatic or manual mechanisms to migrate virtual machine storage as virtual machines
are migrated within or across hypervisor clusters
• Zero application downtime occurs for planned maintenance outages
• Enables all data to be moved non-disruptively between arrays to achieve full planned or unplanned
site shutdown without downtime
• Provides on-demand load balancing; Live Volume enables data to be relocated as desired to
distribute workload between arrays
• Stretches Microsoft clustered volumes between geographically disperse locations
• Allows VMware vSphere and Microsoft Clusters to see the same disk signature on the volume
between data centers and allows the volume to be clustered across arrays
- Snapshots
- High consistency and high availability modes
- Mode migration
- DR activation for Live Volume managed replications
• Supports an additional asynchronous or synchronous Live Volume replication to a third array created
and dynamically managed by Live Volume
• Provides automatic or manual Live Volume failover and restore in the event of an unplanned outage
at a primary Live Volume site
• Includes VMware vSphere Metro Storage Cluster (vMSC) certification with SCOS 6.7 and newer
Live Volume is designed to fit into existing physical and virtual environments without disruption or significant
changes to existing configurations or workflow. Physical and virtual servers see a consistent, unchanging
virtual volume. Volume presentation is consistent and transparent before, during, and after migration. The
Live Volume role swap process can be managed automatically or manually and is fully integrated into the
array and Dell Storage Manager. Live Volume operates asynchronously or synchronously and is designed for
application high availability during planned migration, resource balancing, disaster avoidance, and disaster
recovery use cases.
A Live Volume can be created between two SC Series arrays residing in the same data center or between
two well-connected data centers with Fibre Channel or iSCSI replication connectivity.
Using Unisphere Central for SC Series, a Live Volume and an optional Live Volume managed replication can
be created. For more information on creating a Live Volume, see the appropriate Dell Storage Manager User
Guide provided with the DSM software.
In Figure 33, a mapped server is accessing a Live Volume by proxy access through the secondary Live
Volume system to the primary Live Volume system. This type of proxy data access requires the replication
link between the two arrays to have enough bandwidth and minimum latency to support the I/O operations
and latency requirements of the application data access.
ALUA can be enabled on Live Volumes when both SC Series arrays are running SCOS 7.3 or newer. For new
Live Volumes, select the Live Volume option to enable Report Non-optimized Paths.
For pre-existing Live Volumes, once both SC Series arrays are upgraded to SCOS 7.3, a banner will be
displayed allowing these Live Volumes to upgraded to support ALUA capability.
Banner in DSM 2018 showing Live Volumes can be upgraded for ALUA capability
Dell Storage Manager 2018 and newer provides guidance through the upgrade process. All Live Volumes can
be upgraded, or just a partial subset of Live Volumes can be upgraded if not all storage host operating system
types meet the requirements.
Microsoft Windows, Hyper-V, or vSphere Live Volumes can be upgraded for ALUA capability
The next task in the wizard provides an option to unmap and remap the secondary Live Volume mappings to
the storage host. Depending on the operating system of the storage host, resetting the secondary Live
Volume mappings, or alternatively rebooting the storage host, may be necessary to cleanly pick up the ALUA
state changes on the Live Volume. This is a flexible option available to the administrator performing the
upgrade. Be aware of the impact of either operation. Testing revealed that vSphere hosts require a reboot.
The default action in the workflow is to not reset the secondary server mappings.
Reset the secondary Live Volume mappings or reboot the storage host as necessary
After the Live Volume is upgraded for ALUA capability, the last step is to choose whether or not to Report
Non-optimized Paths. The default action is to report non-optimized paths. However, if the storage host
operating system does not support ALUA but there is a desire to upgrade the Live Volume for ALUA support,
this option allows leaving the ALUA feature disabled from the viewpoint of the storage host operating system.
Live Volumes which were upgraded for ALUA capability or created in SCOS 7.3 and natively have ALUA
capability cannot be downgraded or have their ALUA capability removed. However, the ALUA feature can be
disabled at any time by editing the Live Volume and unchecking the Report Non-Optimized Paths option.
From the Live Volume perspective and outside of automatic failover with vSphere Metro Storage Cluster or
Microsoft Server/Hyper-V clusters, there are no firm restrictions on bandwidth or latency. However, to proxy
data access from one SC Series array to another requires the Live Volume arrays to be connected through a
high-bandwidth, low-latency replication link. Some operating systems and applications require disk latency
under 10 ms for optimal performance. However, performance impact may not be effectively realized until disk
latency reaches 25 ms or greater. Some applications are more latency sensitive. This means that if average
latency in the primary data center to the storage is 5 ms for the volume and the connection between the two
data centers averages 30 ms of latency, the storage latency writing data to the primary Live Volume from the
secondary Live Volume proxy across the link is probably going to be 35 ms or greater. While this may be
tolerable for some applications, it may not be tolerable for others.
If the Live Volume proxy communication or synchronous replication is utilized, it is strongly recommended to
leverage site-to-site replication connectivity which offers consistent bandwidth and the least amount of
latency. The amount of bandwidth required for the connectivity is highly dependent on the amount of changed
data that requires replication, as well as the amount of other traffic on the same wire. If a site is not planning
to proxy data access between arrays with asynchronous replication, then latency is not as much of a concern.
It is recommended to use dedicated VLANs or fabrics to isolate IP-based storage traffic from other types of
general-purpose LAN traffic, especially when spanning data centers. While this is not a requirement for Live
Volume, it is a general best practice for IP-based storage.
For hypervisor virtualization products such as VMware vSphere, Microsoft Hyper-V, and Citrix® XenServer®, a
site must have at least a 1 GB connection with 10 ms or less latency between servers to support VMware
vMotion® Metro or live migration activities. Standard vMotion requires 5 ms or less latency between the
source and destination host.
5.5.1 Replication
Live Volume is built on standard SC Series storage replicated volumes in which each replicated volume can
individually be configured as asynchronous, synchronous high availability, or synchronous high consistency.
Note that Live Volume automatic failover requires synchronous high availability mode. More information about
these attributes can be found throughout this document and in the Dell Storage Manager Administrator’s
Guide.
Sync Mode: If the replication type is synchronous, Sync Mode describes the mode of synchronous replication
that may be either high consistency or high availability. Sync Mode is not displayed for asynchronous Live
Volumes.
Sync Status: If the replication type is synchronous, Sync Status describes the current state of the
synchronous replication as Current or Out Of Date. When Sync Status is Current, synchronous replication is
in sync. This means that both the source and destination volumes are consistent and the cumulative latency
to the secondary Live Volume will be observed at the primary Live Volume application. An out-of-date status
indicates that the data on the source and destination volumes is not consistent. Changed or inconsistent data
is tracked in a journal where the primary Live Volume resides until the two volumes are once again Current.
Sync Status is not displayed for asynchronous Live Volumes.
Deduplication: The Deduplication feature replicates only the changed portions of the snapshot history on the
source volume, rather than all data captured in each snapshot. While this is a more processor-intensive
activity, it may reduce the amount of replication traffic and bandwidth required. If sufficient bandwidth is
present on the connection, Dell Storage recommends disabling Deduplication for Live Volumes in order to
preserve controller CPU time for other processes.
Replicate Active Snapshot: It is recommended that Replicate Active Snapshot is enabled for asynchronous
Live Volumes. This ensures that data is replicated in real time as quickly as possible which decreases the
amount of time required to perform a Live Volume Swap role. For Live Volumes configured with either mode
of synchronous replication, the Active Snapshot is effectively replicated in real time providing the replication is
in sync (HA mode) and not paused by an administrator (HA and HC modes).
Replicate Storage to Lower Tier: The Replicate Storage to Lowest Tier feature is automatically enabled for
a new Live Volume. Disable this option to replicate data to tier 1 on the destination array. Many users perform
the initial Live Volume replication to the lowest tier, and then deselect this option once the initial replication
completes. This strategy aids in preserving tier 1 storage capacity, which is useful when using SSD or 15K
drives. For more information on Data Progression with Live Volume, see the section, Data Progression and
Live Volume.
QoS Nodes: A pair of QoS Nodes depicts the desired egress traffic shaping to be applied when replicating
from the primary to the secondary Live Volume. Although labeled a secondary QoS Node, it does not provide
ingress traffic shaping. Instead, it provides egress traffic shaping after a role swap occurs and it becomes the
primary Live Volume. QoS Nodes apply to replication traffic only. The Live Volume proxy traffic between the
arrays is not governed by QoS Nodes. If the link between the Live Volume storage controllers is shared by
other traffic, it may be necessary to throttle the replication traffic using QoS Nodes to prevent it from flooding
the replication link. However, throttling synchronous replication traffic will produce latency for applications
which are dependent on the Live Volume. For this reason, replication links and QoS Nodes should be sized
appropriately taking into account the amount of data per Live Volume, rate of change, application latency
requirements, and any other applications or services that may be sharing the replication link. This is
especially important when using synchronous replication. Note that vSphere Metro Storage Cluster latency
between sites should not exceed 10 ms.
For instance, if a 20 Gbps replication link exists between data centers that is shared by all intra-data-center
traffic, a replication QoS could be set at 10 Gbps and thereby limits the amount of bandwidth used by
replication traffic to half of the pipe capacity. This allows the other non-Live-Volume replication traffic to
receive a reasonable share of the replication pipe, but could cause application latency if synchronous
replication traffic exceeds 1,000 MBps.
As a best practice, common QoS Nodes should not be shared between a single SC Series source and
multiple SC Series destinations, particularly where Live Volume managed replications are in use. For
instance, if volume A is replicating to volume B synchronously to support a Live Volume and volume A is also
replicating to volume C asynchronously to support a Live Volume managed replication, an independent QoS
Node should be created and used for each of these replications.
Swap Roles Automatically: When Swap Roles Automatically is selected, the primary Live Volume will be
automatically swapped to the array serving the most I/O load to that Live Volume as long as it meets the
conditions for a swap. The Live Volume logic gathers I/O samples to determine the primary access to the Live
Volume (from either servers accessing it directly on the primary Live Volume array, or servers accessing from
a secondary Live Volume array). Samples are taken every 30 seconds and automated role-swap decisions
are based on the last ten samples (five minutes total). This occurs continuously on the primary Live Volume
array (it does not start once the 30-minute delay timer expires).
The autoswap design is meant to make intelligent decisions on the autoswap movement of Live Volume
primary systems while preventing role swap movement from occurring rapidly back and forth between arrays.
Min Amount Before Swap: This attribute is the amount of data accessed from a secondary system. If there
is light, infrequent access to a Live Volume from a secondary array, consider if this makes sense to move the
primary to that system. If so, set this value to a very small value. The criteria for this aspect are defined by the
Min Amount Before Swap attribute for the Live Volume. The value specifies an amount equal to the read/write
access divided by the second-per-sample value. If a sample shows that the secondary array access exceeds
this value, this sample/aspect has been satisfied.
Min Secondary Percent Before Swap: This is the percentage of total access of a Live Volume from the
secondary array on a per-sample basis. The criteria for this aspect are defined by the Min Secondary Percent
for Swap attribute for the Live Volume. If a sample shows the secondary array accessed the Live Volume
more than the defined setting for this aspect, this sample/aspect has been satisfied. The default setting for
this option is 60%. The SC Series array takes samples every 30 seconds and keeps the most recent ten
samples (five minutes total) for analysis. This means that the secondary Live Volume has to have more I/O
than the primary system for six out of ten samples (60%).
Min Time As Primary Before Swap: Each Live Volume has a TimeAsPrimary timer (default setting of 30
minutes) that will prohibit an autoswap from occurring after a role swap has completed. This means that
following a role swap of a Live Volume (either auto or user specified), the SC Series array waits the specified
amount of time before analyzing data for autoswap conditions to be met again. The purpose of this is to
prevent thrashing of autoswap in environments where the primary access point could be dynamic or where a
Live Volume is shared by applications that can be running on servers both at the primary and secondary sites.
Failover Automatically: This indicates whether or not the Live Volume is configured to automatically fail over
and remain available if there is an unplanned outage of the primary Live Volume.
ALUA Optimized: This specifies whether the Live Volume is capable of supporting ALUA Non-Optimized
path reporting. All new Live Volumes created between SC Series arrays running SCOS 7.3 or newer will
automatically be ALUA Optimized. Live Volumes created on SC Series arrays with SCOS 7.2 or earlier can
be updated to be ALUA Optimized once both arrays are running SCOS 7.3 or newer. Live Volumes running
on SCOS 7.2 or earlier will not be ALUA Optimized. Live Volumes which are not ALUA Optimized will report
an Optimal path status for all paths to both the primary and secondary Live Volume.
Report Non-Optimized Paths: This indicates whether or not the Live Volume is configured to report Non-
Optimized paths to the secondary Live Volume (enabled by default). This feature is used in conjunction with
uniform storage presentation and a Round Robin path selection policy where optimal MPIO paths will be
preferred over non-optimal paths for I/O when both types are available. Live Volumes must be ALUA
Optimized and the box must be checked in the Live Volume settings in order to report Non-Optimized paths to
the secondary Live Volume.
For instance, if an array has two tiers of storage and the storage profiles write data at RAID 10 on tier 1,
snapshot data at tier 1 is at RAID 5, and tier 3 is at RAID 5. The blocks of data that were written during the
day will progress from tier 1, RAID 10 to tier 1, RAID 5 on the first night.
If a primary Live Volume is frequently swapped between the Live Volume arrays, then the Data Progression
pattern will be determined by how often the data is accessed on both systems.
Note: Live Volume ALUA support was added in SCOS 7.3 for VMware vSphere and Microsoft Windows
operating systems only.
A Round Robin MPIO policy may also be used with Live Volume. There may be slight deviations based on the
storage host, but in general, with the ALUA support added in SCOS 7.3, Round Robin utilizes all available
optimal paths to a device. MPIO paths leading to the primary Live Volume are advertised by the SC Series
array as optimal. In the event optimal paths are not available, Round Robin will then utilize non-optimal paths
that are available. MPIO paths leading to the secondary Live Volume are advertised by the SC Series array
as non-optimal.
Standalone servers or clustered servers accessing shared storage may use uniform storage presentation.
Uniform storage presentation provides host to Live Volume storage paths through both arrays. With the Live
Volume ALUA feature enabled, a Round Robin path selection policy can quickly and easily be configured with
uniform storage presentation. With the ALUA intelligence built into both the storage host and the arrays,
optimal MPIO paths will be used for I/O and the storage fabric will be both well utilized and well balanced.
For non-vSphere or Microsoft storage hosts in a uniform Live Volume configuration, all MPIO paths will be
reported as optimal. When using a Round Robin path selection policy, half of the I/O will go through the
primary Live Volume and the other half with be proxied through the replication link through the secondary Live
Volume. This may or may not be a good design choice. Also, the automatic role swap feature will not be able
to make an accurate determination of where the primary Live Volume should be, and the feature will be
ineffective if enabled.
Clustered servers accessing shared storage may also use non-uniform storage presentation. Non-uniform
means that each cluster node will access a Live Volume through one array or the other, but not both. Non-
uniform typically applies to a stretched cluster configuration where each node in the cluster accesses the Live
Volume through paths that are local in fabric proximity only. Each cluster node would have read/write access
to either the primary or the secondary Live Volume, but not both simultaneously. For the cluster nodes which
have access to the primary Live Volume, their front-end I/O will remain local in proximity. For the cluster
nodes which have access to the secondary Live Volume, their front-end I/O will be proxied to the primary Live
Volume through the replication link. Implementing Round Robin with non-uniform presentation is typically
preferred to provide automated port and fabric balancing, but fixed paths may also be used. Automatic role
swap would be preferred with non-uniform storage presentation so that a role swap will automatically follow
the vMotion of virtual machines between sites.
Regardless of storage presentation, MPIO path selection, and role swap policies, synchronous replication
latency between arrays will impact applications. Additional information on configuring Live Volume MPIO can
be found in each of the application-specific sections of this document, such as VMware vSphere (section 8),
Microsoft Windows/Hyper-V (section 9), and Linux/Unix (section 10).
Note: ALUA information used by the Round Robin PSP for a device can also be viewed using esxcli as
shown in the following example. TPG_state identifies the active/optimal and active/non-optimal ALUA state
for each target port group. Working Paths identifies each active/optimal path for Round Robin as derived
from the target port groups.
If a Live Volume role swap occurs, the SC Series array is designed to report the ALUA path state changes to
the vSphere host using the following process (according to the T10 SCSI-3 specification SPC-3):
1. A Live Volume role swap occurs between optimal and non-optimal paths.
2. The vSphere hosts send Round Robin I/O to what is now the secondary Live Volume through non-
optimal paths.
3. The SC Series array fails the I/O with a Unit Attention Check Condition.
4. The vSphere host requests Report Target Port Groups.
5. The SC Series array responds with new ALUA state changes.
6. vSphere begins Round Robin I/O over new optimal paths.
Note: The Unit Attention Check Condition can be observed in /var/log/vmkernel.log in the following
example. For more information, see the VMware KB article, Interpreting SCSI sense codes in VMware ESXi
and ESX (289902).
During extensive testing, it was discovered that vSphere did not consistently or reliably begin using the new
optimal primary Live Volume paths immediately after a role swap as designed. Instead, up to five minutes
elapsed before vSphere recognized the ALUA path state change. This means that vSphere may continue to
follow the Round Robin policy over the non-optimal paths to the secondary Live Volume for up to five minutes.
Dell Technologies is working with VMware to understand this behavior.
For customers concerned about this behavior occurring in their environments, there are two workarounds:
• After the Live Volume role swap, perform a vSphere storage rescan.
• Reduce the vSphere host advanced setting Disk.PathEvalTime from the default of 300 seconds
down to an acceptable automatic storage rescan interval. This would need to be performed on each
vSphere host the Live Volume is mapped to.
8.3 Fixed
The Fixed PSP may be desirable when a preferred path on the storage fabric should be used. The Fixed PSP
may also be useful in a uniform Live Volume storage presentation, if the ALUA feature (added in SCOS 7.3) is
not yet available, in order to avoid sending read and write I/O down a non-optimal path to the secondary Live
Volume. As shown in Figure 44, the MPIO policy on the vSphere host should be set to Fixed with the
preferred path leading to the primary Live Volume. Using the Fixed PSP generally requires more
administrative effort to implement, maintain, and document.
Fixed policy
Alternatively, the vSphere Fixed MPIO policy with preferred path can be used to ensure front-end I/O traffic is
going through the primary Live Volume array. In the event a Live Volume role swap occurs, the preferred path
will need to be updated on each vSphere host to point to an optimal primary Live Volume path. This is where
additional administrative effort is involved to maintain a well-balanced fabric.
As depicted in Figure 45, a Live Volume replication exists between SC Series A and SC Series B. Two
vSphere hosts are mapped to the Live Volume on each array. The Primary Live Volume is located on SC
Series A, and the Fixed preferred path (Figure 44) on each vSphere host is configured to use a path to SC
Series A as the preferred path.
If planned downtime will impact SC Series A, the preferred path for the vSphere hosts could be configured for
SC Series B. When the Live Volume is configured for automatic role swap, this change in preferred paths will
trigger the Live Volume to swap roles making SC Series B the Primary Live Volume controller, allowing SC
Series A to be taken offline without a disruption to virtual machines on that Live Volume.
In this configuration, the MPIO policy for the primary Live Volume can be configured as either Round Robin
(preferred) or Fixed. Enabling Live Volume automatic role swap would be optimal in this configuration where
vMotion is used to migrate virtual machines between sites (for example, from Site A to Site B). After the
vMotion occurs, read and write I/O stemming from VMware Host B to the secondary Live Volume is proxied to
the Primary Live Volume controller through the asynchronous or synchronous replication link(s) between the
arrays. Automatic role swap allows SC Series storage to optimize the I/O after vMotion so that all or the
majority is flowing through the primary Live Volume.
Multiple sites and stretched clusters may also be configured with uniform storage presentation. For more
information on vMSC storage presentation, see section 8.9.
Live Volume configured with Synchronous, High Availability, and Failover Automatically enabled
High Availability
Live Volume
VMFS datastores or RDM
Synchronous High
Primary Availability replication Secondary
Secondary Primary
Fibre Channel
or iSCSI
Tiebreaker
site
SC Series SC Series
array array
High Availability
Live Volume
VMFS datastores or RDM
Synchronous High
Primary Availability replication Secondary
Secondary Primary
Fibre Channel
or iSCSI
Tiebreaker
site
SC Series SC Series
array array
Tiebreaker failure
In vSphere 4.1 and later, DRS host groups and VM groups can be used to benefit a vSphere Metro Storage
Cluster environment.
• VM groups: Virtual machines that share a common Live Volume datastore can be placed into VM
groups. Movement of virtual machines and management of their respective Live Volume datastore,
VM groups can be assigned to host groups using the DRS Groups Manager. This will ensure all virtual
machines that share a common Live Volume datastore are consistently running from the same datastore. The
virtual machines can be vMotioned as a group from one site to another. After a polling threshold is met, SC
Series storage can perform an automatic role swap of the primary Live Volume datastore to the site where the
VMs were migrated. The infrastructure can also be designed with separate DRS-enabled clusters existing at
both sites, keeping automatic migration of virtual machines within the respective site where the primary Live
Volume resides. In the event of a Live Volume role swap, all virtual machines associated with the Live Volume
can be vMotioned from the site A cluster to the site B cluster.
vSphere High Availability (HA) is also a cluster-centric configuration. In the event of a host or storage failure,
HA will attempt to restart virtual machines on a host candidate within the same cluster that may be local or
stretched in a vMSC deployment. In a non-uniform storage configuration, if an outage impacts primary Live
Volume availability and Live Volume automatic failover occurs, vSphere HA can be configured to restart
impacted virtual machines on the surviving array.
The following vSphere advanced tuning should be configured for non-uniform stretched cluster configurations.
This tuning allows HA to power off and migrate virtual machines during storage-related availability events
after the primary Live Volume becomes available again using the Preserve Live Volume or Live Volume
Failover Automatically feature.
Note: The Live Volume Failover Automatically feature is only supported with vSphere 5.5 and newer.
Configuring HA to restart VMs impacted by a non-uniform storage outage was improved in vSphere 6. In
addition to supporting HA restart for Permanent Device Loss (PDL) events, HA restart can also react to All-
Paths-Down (APD) events. The tuning becomes easier and more intuitive using the Web Client or HTML5
client in newer versions of vSphere.
1. It is recommended to configure VM Component Protection for PDL and APD events. For PDL events,
select Power off and restart VMs. For APD events, VMware recommends selecting Power off and
restart VMs (conservative). For the advanced APD settings, refer to VMware documentation or the
VMware vSphere Metro Storage Cluster Recommended Practices available on the VMware
Documentation site.
The Disk.AutoremoveOnPDL advanced setting is not configurable in VMCP and should remain at its default
value of 1 for each vSphere 6 host in the cluster. For more information on the Disk.AutoremoveOnPDL
feature, refer to VMware KB article 2059622, PDL AutoRemove feature in vSphere 5.5 and vSphere 6.0.
In a non-uniform configuration, if an outage impacts primary Live Volume availability and Live Volume
automatic failover cannot occur, vSphere HA will not be able to immediately restart impacted virtual
machines.
If the VMware virtual infrastructure is version 4.0 or earlier, vSphere Metro Storage Cluster is not supported
and other steps should be taken to prevent virtual machines from unexpectedly running from the secondary
Live Volume. An individual VM or group of VMs may be associated with a DRS rule that keeps them together
but this does not guarantee they will stay over a period of time on the same host or group of hosts where the
primary Live Volume is located. As a last resort, DRS can be configured for manual mode or disabled when
using Live Volume in a multi-site configuration that will prevent the automatic migration of VMs to Secondary
Live Volume hosts in the same cluster.
VMware vSphere monitors SCSI sense codes sent by an array to determine if a device is in a PDL state.
These SCSI sense codes are outlined in VMware KB article 2004684, Permanent Device Loss (PDL) and All-
Paths-Down (APD) in vSphere 5.x and 6.x. SC Series storage supports two SCSI sense codes (see Table 1)
which will be sent to vSphere hosts when a PDL condition is met.
• Dell SCOS 6.7 or newer with both SC Series arrays having the same firmware version
• Dell Storage Manager 2015 R2 or newer with tiebreaker service located at a site physically
independent of SC Series arrays
Although managed replications are controlled by Live Volume, they fundamentally work the same way as
standard synchronous or asynchronous volume replication. This includes the recovery options available with
view volumes and DR activation.
One differentiator between standard replication and Live Volume managed replication is that the source
volume of a managed replication is dynamic and changes as Live Volume role swaps occur. The easiest way
to think of this is to understand that the managed replication source is always the primary Live Volume.
Because the primary Live Volume role can shift manually or automatically between arrays, the flow of
replicated data will also seamlessly and automatically follow this pattern. The reason for this is that the
primary Live Volume is the volume where write I/O is first committed. Therefore, and especially in an
asynchronous Live Volume configuration, in the event of an unplanned outage or disaster that disables both
Live Volumes, data should be recovered from the most transaction consistent volume to minimize loss of
data. For synchronous Live Volumes, transaction inconsistency is less of an issue because both primary and
secondary Live Volumes should have a sync status of current and not out of date unless the synchronous
replication was paused or became out of date due to excess latency in high availability mode. Disaster
recovery at a remote site is a good use case for the Live Volume managed replication feature. However, Live
Volume managed replications are not explicitly supported with vSphere Site Recovery Manager. LUN
presentation and data recovery on a managed replication can be handled automatically by DSM, but before
virtual machines can be powered on, they must be registered into vSphere inventory, which may require a
manual DR-documentation or scripted process.
9.1 MPIO
Microsoft Windows or Hyper-V servers running 2008/R2 and newer versions on SC Series arrays can use the
in-box Microsoft MPIO device specific module (DSM). The DSM comes with different MPIO policy options.
The following policies are supported with Live Volume:
• Round Robin
• Failover Only
• Round Robin with Subset, Live Volume ALUA support (see limitations described in sections 9.3 and
9.4)
Note: For more information about Windows Server and MPIO policies, settings, and best practices, review
the Dell EMC SC Series Storage and Microsoft Multipath I/O best practices guide.
• If using non-uniform server mappings (where a host is presented with multiple data paths only to the
SC Series array that is local to it)
• When the workload is local to the SC Series array that hosts the primary Live Volume(s)
• If bandwidth and latency performance are the same for all the available data paths
Round Robin also works well with uniform server mappings in the following cases:
• When both primary and secondary Live Volumes are accessible by high-bandwidth low-latency
primary and secondary data paths.
• If the slight latency penalty caused by proxied data over secondary data paths does not negatively
impact the workload.
In many cases, using non-uniform server mappings with Round Robin provides the best overall design while
minimizing complexity, and it provides a good design baseline.
A Windows server will detect the presence of non-optimal paths when this feature is enabled on a Live
Volume. In the MPIO settings, the MPIO policy indicates Round Robin With Subset. In this example (with
uniform server mappings), half of the paths are listed as Active/Optimized (paths to the primary Live Volume),
and the other half are listed as Active/Unoptimized (paths to the secondary Live Volume). There are four of
each kind of path, eight paths total. Figure 63 shows three paths; the other paths can be viewed by scrolling.
In the process of developing this feature, it was found that the Microsoft implementation of MPIO incorrectly
utilizes non-optimal paths when a stretch cluster node is configured to use non-uniform server mappings to a
secondary Live Volume. In cases where no optimal paths are available, the Windows host will use a single
available non-optimal path, instead of using Round Robin to spread the I/O over all available non-optimal
paths. While a host configured to use only non-optimal paths to a secondary Live Volume (with non-uniform
server mappings) is a legitimate use-case scenario by design with Live Volume ALUA, it is not a typical
configuration encountered in the SAN industry, and this issue was not immediately apparent.
To address this limitation, install the March 2018 monthly cumulative update for Windows Server 2016.
Microsoft does not plan to extend this fix to Windows Server operating systems prior to Windows Server
2016.
Customers should consider their design and decide whether to implement Live Volume ALUA support for their
Windows Server hosts when using non-uniform server mappings with Live Volume. The following details the
support for Windows Server:
• Windows Server 2016 with the March 2018 cumulative update and newer fully supports Live Volume
ALUA.
• Windows Server 2012 R2 and prior: When using non-uniform server mappings for stretch cluster
nodes, the recommendation is to uncheck the Report Non-Optimal Paths option, and the host
server will detect and utilize all available paths as active-optimized.
9.6 Uniform server mappings with Live Volume and Round Robin
In Figure 65, a Round Robin Live Volume MPIO configuration is depicted. In this scenario, the server has two
adapters and is mapped to both SC Series arrays that host the primary and secondary Live Volume pair. This
configuration is referred to as uniform server mapping since the server is mapped to both SC Series arrays.
Since all four paths are included in an MPIO Round Robin policy, about half of the storage traffic has to
traverse the proxy link between the two SC Series arrays. In cases where the two SC Series arrays are well
connected (for example, if they reside within the same data center), the slight latency penalty incurred for
proxied data may be negligible for the workload, and this design would perform well. However, this
configuration would be sub-optimal if there were significant latency or bandwidth limitations for the secondary
paths, in which case, Round Robin with Subset or Failover Only would be a better choice.
Round Robin with uniform server mappings also prevents a Live Volume from automatically swapping roles
(when the Swap Roles Automatically feature is enabled on a Live Volume) because about 50% of the traffic
will always be going through each array and therefore the thresholds that trigger a role swap will not be
exceeded.
The decision to use uniform or non-uniform server mappings in conjunction with MPIO Round Robin, Round
Robin with Subset (with non-optimal paths reported to the host server), or Failover Only is a function of
environmental variables that are unique to each customer. Because there are so many different configuration
possibilities, administrators have great flexibility to tailor a solution that is best for their environment.
Note: Live Volume automatic failover (LV-AFO) supports host and guest clusters running Windows Server
2012 or newer, or Hyper-V 2012 or newer. More on LV-AFO is covered in section 9.9.
If Live Volume automatic role swap is enabled, the VM placement and workload determines which array will
own the primary Live Volume, since the Live Volume will follow the workload, based on the auto-swap
thresholds. The scenario in Figure 66 depicts a virtual machine that is live migrated from Host A to Host B.
The secondary SC Series array will proxy I/O to the primary array for short time and then automatically swap
the primary Live Volume role from the primary array to the secondary array.
In a multi-site Live Volume Hyper-V cluster, the virtual machines will typically be kept running on nodes within
the same site as their respective primary Live Volume(s). If PRO is activated on a Hyper-V cluster with nodes
in each site, it could automatically migrate some of the virtual machines running on Cluster Shared Volumes
(CSVs) that are also Live Volumes to a node that resides in the other data center, thereby splitting the I/O
between data centers.
One of the best practices with CSVs is utilizing the ability to control which node is the CSV owner. Set the
CSV to be owned by a cluster node that is in the primary site and mapped directly to the SC Series array. In
this way, if the CSV goes into Network Redirected mode, the CSV owner is in the same site and downtime
can be eliminated or reduced.
Figure 67 depicts a multi-site Hyper-V cluster with Live Volume. In this figure, the SC Series storage at Site B
is offline. CSV network redirection can take over and proxy all the data traffic through the CSV owner on Host
A.
If a failure happens that takes down SC Series B, Hyper-V can redirect access to the volume over the network
using CSV Network Redirected access.
With the release of SCOS 7.1 and Dell Storage Manager 2016 R2, support for LV-AFO was extended to
Microsoft Window Server cluster and Hyper-V cluster environments in addition to VMware. LV-AFO functions
essentially the same with Microsoft or VMware hosts, regarding triggers that cause a Live Volume to
automatically fail over given a DR situation (see section 8.12). The main difference is that VMware offers
some additional resiliencies with VM and workload recovery given a disaster.
To learn more about how LV-AFO works with Microsoft, including a deep dive into LV-AFO functionality and a
lab demo, see the three-part video series, Live Volume with Auto Failover Support for Microsoft.
• A pair of SC Series arrays that support Live Volume (SCv2000 Series is not supported)
• Replication and Live Volume feature licenses applied to both SC Series arrays
• SCOS 7.1 or newer on both SC Series arrays; both SC Series arrays should be running matching
SCOS versions
• At least 1 Gbps of available bandwidth between the SC Series arrays for replication and proxied data
(iSCSI or FC)
• No more than 5 to 10 ms of latency between the SC Series arrays and the host servers (higher
latencies may be tolerable for some workloads but generally anything over 10 ms will start to be
noticeable)
• An instance of the Dell Storage Manager at a third site to act as a tiebreaker/quorum witness for LV-
AFO
- DSM version 2016 R2 or newer is required to support LV-AFO for Microsoft; DSM 2018 or newer
is required to support Live Volume ALUA.
- DSM can be configured as a physical host or as a VM.
- DSM supports running from a cloud-based service if the customer does not have third site.
- This DSM instance should be dedicated to a tiebreaker/quorum witness role (run other DSM
instances at the primary or secondary site for day-to-day management, monitoring, and
reporting).
- No more than 200 ms of round-trip latency is required between the DSM tiebreaker and each SC
Series array.
• Guest VM clusters running Windows Server 2012 and newer, that use the following methods for
guest VM clustering:
- Shared virtual hard disks with VHDX format (host servers require Hyper-V 2012 R2 in this case)
- In-guest iSCSI (guest VMs running on either Hyper-V or VMware hosts are supported)
- Physical raw device mappings (pRDMs) on VMware 5.5 or 6.0 hosts
Note: Each customer needs to make an informed choice about how they configure the quorum witness for a
clustered environment that utilizes LV-AFO. The risk of an outage due to a temporary loss of quorum due to a
less resilient design might be permissible in some cases, such as for test or development environments.
• Quorum witness: With Windows/Hyper-V clusters (physical or virtual), administrators can choose
between a SAN-based quorum disk witness or a file share witness to serve as a tiebreaker. This is
particularly important when there are equal numbers of nodes at each site, given a stretch cluster with
LV-AFO. Configuring a quorum witness with Windows Server 2012 R2 Hyper-V and newer is
recommended, regardless of the number of cluster nodes.
- The optimal quorum configuration for LV-AFO is a file share witness that is located at a third site
to ensure that the surviving node(s) at either site A or site B (given a complete site failure at either
location) always have continuous, uninterrupted access to the quorum. This is true for both
physical node clusters and guest VM clusters.
- If a third site is not available for a file share witness, then the next best configuration is to use a
SAN-based quorum disk that is also a Live Volume, configured for LV-AFO, so it can
automatically fail over given a DR situation.
However, this may not prevent loss of quorum given a complete site failure if the site that goes
down hosts the quorum disk (as a primary LV), and it also hosts the node that owns the quorum
disk. Given a complete site failure, if the quorum disk is configured for LV-AFO, it will fail over to
the other site and become available as a primary LV to a surviving node. However, with a
complete site failure, the combination of a surviving node needing to take ownership of the
quorum disk, concurrent with the short pause in I/O to the quorum disk (20 to 30 seconds) while
LV-AFO completes, may result in a failure of a surviving node to take ownership of the quorum
disk. If this occurs, workloads dependent on cluster resources at the surviving site may go offline
if the surviving nodes require a tiebreaker to achieve quorum, as would be the case if there are
an equal number of cluster nodes at each site. To avoid this scenario, configure a file share
witness at a third location.
- Configuring a file share witness that is local to either site A or site B given a stretch cluster
configuration with LV-AFO should be avoided. If the site that hosts the file share witness suffers a
site outage, access to the file share witness by the surviving nodes at the surviving site will fail. If
the file share witness is unavailable to the surviving nodes as a tiebreaker, and its vote is needed
to maintain quorum, then cluster resources will go offline.
Note: There are two additional methods of guest VM clustering that have not been tested with LV-AFO:
Windows Server guest VM clusters on Hyper-V that use pass-through disks, or virtual Fibre Channel disks as
cluster disks. While they may work, they are not supported configurations with LV-AFO.
In general, VMware (when configured) may offer more HA resiliency with its ability to automatically move and
recover VMs and workloads at another location given a disaster.
While nothing will prevent a customer from co-locating the DSM tiebreaker role at the same site as the
primary or secondary Live Volume, cutting corners with this design puts the customer at an elevated risk of an
unintended outage due to loss of LV-AFO quorum. To avoid confusion, the reference to LV-AFO quorum is
separate, in addition to the Windows Server/Hyper-V quorum, as discussed in section 9.10.2.
For example, if the DSM tiebreaker is co-located at the same site as the secondary Live Volume, and this site
suffers an outage so that the primary Live Volume loses connectivity to both the secondary Live Volume and
DSM tiebreaker, the primary Live Volume will also go offline due to loss of quorum. The result is that there will
be a complete outage for any workloads that use this Live Volume. This is by design. When the primary Live
Volume loses connectivity to both the DSM tiebreaker and the secondary Live Volume concurrently
(assuming the Live Volume pair is in sync at the time), the primary Live Volume has to assume that by
becoming completely isolated (loss of quorum), that the DSM tiebreaker and secondary Live Volume are still
online, and that the DSM tiebreaker is now promoting the secondary Live Volume to primary status.
Placing the DSM tiebreaker at the primary site with the primary Live Volume will help prevent an unintended
loss of quorum, but only as long as the primary Live Volume stays at that site. If the primary Live Volume ever
fails over (for any reason, manually or automatically) to the other site, then the customer is again vulnerable
to an unintended loss of quorum and an outage due to site bias. To work around this, the customer could
install another instance of DSM at the secondary site to manually seize the local tiebreaker role for that
particular Live Volume if it ever fails over to that site, but this is not a very practical situation from an
administration standpoint.
If the customer does not have a third site of their own for the DSM tiebreaker role, then a cloud service can be
used. DSM can be installed on a physical host or a VM, as long as the OS is supported.
Each customer will need to make an informed choice about DSM tiebreaker placement. The risk of an outage
due to unintended loss of quorum might be permissible in some cases, such as for test or development
environments.
Replay Manager cannot be used to protect SQL Server data stored on Live Volumes. Snapshots on a Live
Volume will always be crash consistent. Unfortunately, database recovery from crash-consistent snapshots is
not always successful, especially when database files are spread across multiple volumes. While generally
not a best practice, placing all files for a given database on the same volume will increase the odds of a
successful recovery from crash-consistent snapshots.
As long as the primary and secondary volumes are synchronized, database recovery should be reliable from
the active snapshot on the secondary volume, if the primary volume fails. Since asynchronous Live Volume
does not keep the primary and secondary volumes synchronized, synchronous Live Volume should be used
for SQL Server database files. If synchronous Live Volume is used in high availability mode, the secondary
volume may not always be in sync with the primary. A successful recovery from the active snapshot on the
secondary volume may not be possible if it is out of sync. Synchronous Live Volume in high consistency
mode provides the best protection.
Live Volume with automatic failover can be used to create a multi-site failover cluster instance of SQL Server.
As long as the sites are synchronized, SQL Server will automatically bring the instance online at the
secondary site if the primary site were to fail. If Live Volume is set up in a uniform configuration, it is possible
for SQL Server to be running in one site and the primary volume in another. Using Live Volume in a non-
uniform configuration, where each node of the cluster only has volume mappings to the local array and with
automatic role swap enabled, will help ensure that SQL Server and the primary volume are running at the
same site. Since Live Volume with automatic failover uses synchronous replication in high availability mode,
the secondary volume is not guaranteed to always be in sync with the primary. If a failure occurs while the
secondary is out of sync, manual intervention will be required to recover the SQL Server instance.
If automatic failover is not needed, regular replication may be a better choice to protect SQL Server data.
Replication alone offers recovery times comparable to Live Volume without losing the protection benefits of
application-consistent snapshots provided by Replay Manager, reducing the risk of having to recover from
backup in the event of a failure. The entire recovery can be scripted using Microsoft PowerShell®, reducing
recovery time and minimizing mistakes.
Note: The serial number of the replicated volume on the destination SC Series array is unique, and different
from that of the source volume.
When Live Volume is coupled with synchronous replication, it allows customers to deploy and manage
identical data integrity guaranteed copies of data to a secondary SC Series array at an alternate location. The
use of Live Volume creates value in this scenario, by masking the serial number of the replicated volume on
the destination SC Series array and making the serial number appear identical to that of the source volume.
The ability to maintain an identical serial number volume (or volumes) on the destination SC Series array
simplifies and expedites the ability to recover from this volume (or volumes) into the destination application
stack.
The remainder of this section discusses some of these use cases along with certain technical considerations
in these scenarios, respectively.
An LVMR volume (in asynchronous mode with active snapshot/Replay enabled, or in synchronous mode) can
be used to manage and provide an offsite copy of business-critical data for disaster recovery or data
distribution use cases (where locating data closer to the audience could significantly reduce data access
latency). Figure 69 depicts this scenario.
As of SCOS 7.3, the Live Volume automatic failover feature has not been fully tested on the Linux platform
and therefore it is not a supported configuration.
It is recommended to examine the settings closely especially when Live Volumes are involved because the
built-in default settings might not be appropriate. The following sample configuration demonstrates a version
of the multipath configuration:
defaults {
find_multipaths yes
user_friendly_names yes
polling_interval 5
}
devices {
device {
vendor "COMPELNT"
product "Compellent Vol"
path_grouping_policy "multibus"
path_checker "tur"
features "0"
hardware_handler "0"
prio "const"
failback immediate
rr_weight "uniform"
no_path_retry "24"
fast_io_fail_tmo 5
dev_loss_tmo infinity
path_selector "round-robin 0"
}
}
• The storage arrays are fully redundant and are protected for complete outage.
• Storage paths are expected to be highly available within an SC Series array and across remote
arrays.
• In the event of path failovers, I/O is queued and retried for up to 24 times. Each retry interval is 5
seconds. Therefore, it tolerates a maximum of 2 minutes (24 x 5 seconds) of path unavailability
before it stops retrying. This timeout needs to be long enough to cover the controller failover within an
SC Series array and the Live Volume swap role between the primary and secondary array. Adjust this
value if necessary to adapt to your environment.
• It is recommended to explicitly include the COMPELNT device settings in /etc/multipath.conf file
even though the built-in defaults might cover it. Several reasons for this recommendation are:
• While device path_selector policies service-time and round-robin are supported by DM-Multipath,
Dell Technologies has performed extensive testing with round-robin policy only. See Figure 65.
• In a uniform configuration where primary and secondary paths are mapped to the Linux host, these
paths are grouped into a single group and the round-robin policy spreads I/O to these paths with
equal weight even though the secondary paths might have higher latency. This is because I/O issued
to the secondary paths is proxied through the replication link to the primary SC Series array as
described in section 5.2. Hence, the overall application performance would be limited by the slowest
paths.
Note: As of SCOS 7.3, the ALUA optimization feature has not been fully tested on the Linux platform.
Therefore, it is not recommended to enable the feature.
After SCOS is upgraded to 7.3, DSM displays a banner in the Live Volume tab indicating that there are
existing Live Volumes that can be optimized for ALUA. Select any Live Volume on the list to see the existing
ALUA Optimized status. See Figure 70.
Live Volumes created before SCOS 7.3 do not automatically become ALUA optimized. The storage
administrator must click the Update to ALUA Optimized link to initiate the optimization process. The link
opens a pop-up window that shows all Live Volumes that are eligible for optimization. See Figure 71. To
ensure Linux Live Volumes do not get ALUA optimized, unselect them from the list.
Note: By default, all eligible Live Volumes are selected. It is important to make sure the Linux Live Volumes
are unselected from the list. These Live Volumes will remain on the list and need to be unselected the next
time the storage administrator optimizes other Live Volumes.
In the event that Linux Live Volumes created before SCOS 7.3 are optimized, the Linux hosts might
experience temporary I/O interruption during the optimization process. The length of the interruption varies
depending on the host's DM-Multipath configuration. Internal testing showed this can range from a few
seconds to a couple minutes.
The Report Non-optimized Paths setting can be found under the Edit Live Volume Settings screen (Figure
72). By unchecking this option, SCOS stops reporting the non-optimized ALUA status of the storage path to
the Linux host. As a result, all paths on the Linux host appear as active and optimized, regardless of the
primary and secondary SC Series arrays. This is the same behavior on a system with SCOS prior to 7.3.
Disabling the Report Non-optimized Paths option does not interrupt host I/O, and therefore it is safe to
execute at any time.
Because the Live Volumes are created as already ALUA-optimized on the SC Series array, even though it is
not reporting the non-optimized status, these Live Volumes would not show up in the Update to ALUA
optimized list (Figure 71).
a) Show the paths information with multipath command. Note the H:C:T (Host adapter, Controller,
Target) and the sd devices information. In this example, a Live Volume has eight storage paths.
b) Execute the lsscsi command to show the transport information. Note the transport column (third field)
contains the SC Series target port number.
# lsscsi --transport
[0:0:0:1] disk fc:0x5000d31000ed1f220x1b0b02 /dev/sdb
[0:0:3:1] disk fc:0x5000d31000ed21210x1b1201 /dev/sdc
[0:0:4:1] disk fc:0x5000d31000ed21220x1b1301 /dev/sdd
[0:0:15:1] disk fc:0x5000d31000ed1f210x1b0a02 /dev/sdh
[1:0:0:0] disk sata: /dev/sda
[5:0:0:0] cd/dvd sata: /dev/sr0
[7:0:0:1] disk fc:0x5000d31000ed1f260x1c0b02 /dev/sde
[7:0:3:1] disk fc:0x5000d31000ed21250x1c1201 /dev/sdf
[7:0:4:1] disk fc:0x5000d31000ed21260x1c1301 /dev/sdg
[7:0:15:1] disk fc:0x5000d31000ed1f250x1c0a02 /dev/sdi
c) In DSM, navigate to the Fibre Channel fault domain tabs. If the array is configured in Virtual Port
mode, go to the Virtual Ports screen. In this example, the Virtual Ports information for both primary
and secondary SC Series arrays are examined.
d) The following relationships can be established after analyzing the information from steps a, b, and c.
Linux multipath device mpatha consists of eight storage paths that span across SC 22 and SC 23.
10.7.1 Single-site
In this single-site scenario, the Linux hosts and SC Series arrays are geographically located within the same
site boundaries though different buildings or labs if necessary. Figure 73 depicts this scenario.
A volume (or multiple volumes) is first created and mapped to a Linux host. The volumes are scanned,
identified and brought into multipath awareness as shown in the following example.
These volumes are then converted into Live Volumes and synchronously (either HA or HC modes) replicated
to the alternate array. The volumes on the alternate array are then mapped back to the same Linux host. The
volumes are once again scanned, identified, and brought into multipath awareness as shown in the following.
Each volume is now represented by four additional paths, these paths are the mappings from the alternate
array.
It should be noted that even though these additional paths are shown as active and will be actively used for
I/O requests (Round Robin), any I/O requests sent to these paths will be proxied through the replication link to
the primary array for commits. The use of these paths would thus introduce unintended latency to any
applications that may be latency-adverse; at this time, path priority definitions and grouping (for example, all
paths are prio=1 by default and used in equal fashion) are not configurable between Linux hosts and SC
Series arrays. Therefore, it is not recommended to use this approach for any critical production use cases.
That being said, this scenario can still apply in certain use cases. In situations where a maintenance event
requires SC Series arrays to be powered down, the volumes on the primary array can be Live Volume
replicated to an alternate array in a different building or lab. The roles of the arrays can then be swapped,
making the alternate array adopt the primary array role (and the paths from them). The latent paths (from the
formerly primary array) can then be removed from multipath for the duration of these maintenance events
while maintaining uptime and zero disruption to any and all functions and applications that may reside on the
Linux hosts. Upon completion of these maintenance events, the volumes and roles of the arrays can then be
swapped back or left in place to the discretion of the business requirements.
10.7.2 Multi-site
In this multi-site scenario, the Linux hosts and SC Series arrays are geographically dispersed within a
metropolitan (or multi-state) region; sites may be connected with MAN or WAN technologies. Figure 74
depicts this scenario. It should be noted that this scenario can also be scaled down and applied towards
single-site deployments as well.
The Live Volumes are synchronously (in either HA or HC modes) replicated across SC Series arrays. In this
scenario, the alternate array volumes are mapped to a secondary Linux host instead. It should be noted that
the volumes mapped to the secondary Linux host are not shared volumes and do not possess shared I/O
management and locking mechanisms. These secondary volumes would need to be remounted to reflect any
data changes that were written to the primary volumes. For this reason, the integrity of data across both
primary and secondary volumes is guaranteed (in either HA or HC modes) as long as the replication link is in
a known good state.
The secondary Linux host can be used in various ways including, but not limited to, the following use cases:
• These volumes can be used to present a consistent read-only copy of this data to a remotely located
site; this can apply not only for data distribution reasons, but to also manage and minimize any data
access latency concerns.
• The consistency of these volumes (and the integrity of the data that it guarantees) also lends itself
towards database replication use. Databases (and applications) can be brought online at the
remotely-located site, in either read-only mode or used in a disaster recovery after the roles of the SC
Series arrays are swapped (the secondary array becomes the primary array).
• These volumes can also be used to complement virtualization technologies such as Red Hat
Enterprise Virtualization (RHEV) to allow for the replication of virtual machine workgroups from one
site or hypervisor to another site or hypervisor. RHEV hosts its virtual machines on a storage domain
that in turn is correlated through a one-to-many relationship to one or multiple backing SC Series
volumes. These volumes are enabled with Live Volume or are synchronously replicated to an
alternate site or secondary array. At the alternate site, these storage volumes can then be used to
reconstruct or import the storage domain into the alternate data center object, and from which object,
reconstruct the virtual machine workgroup.
The following technical and performance considerations should be taken when exploring these use cases.
/etc/multipath.conf: This keeps the contents of /etc/multipath.conf file consistent across all hosts sharing
these Live Volumes; this file contains the definitions of volume WWIDs and their respective aliases and will
ensure that volumes are identified across the different hosts as the same device, and contain the same data
to maintain application integrity.
/etc/fstab: Use the /etc/fstab file in conjunction with /etc/multipath.conf to ensure that volumes are accurately
mounted to their respective and proper mount points in the filesystem. Use the multipath device aliases (if
defined) or default multipath device naming (mpathX) in /etc/fstab to maintain this consistency.
/etc/ntp.conf: Always use the /etc/ntp.conf file to maintain a certain degree of time-based integrity across all
infrastructure. The use of /etc/ntp.conf file becomes even more critical when attempting to maintain data
integrity across multiple cluster nodes/sites dispersed across larger geographic deployments (MAN, WAN).
Cluster configurations: In addition to the considerations mentioned above, take into account cluster-specific
configuration files as well and the best practices of their respective vendors. The discussion of cluster-specific
configuration remains outside the scope of this paper. The list of enterprise-class clustering technologies
include but is not limited to Red Hat Cluster Suite, IBM® PowerHA®, Oracle® Solaris Cluster, Oracle RAC, HP-
UX ServiceGuard, and Symantec™ Veritas™ Cluster Server to name a few.
Performance considerations: It should be noted that the dual commit nature of synchronous replication (I/O
writes must be committed to the secondary SC Series volume before it is committed to the primary volume)
may introduce latency to the applications generating these write requests. The use of synchronous replication
guarantees the consistency and integrity of the data at both SC Series sites when the acknowledgment is
received by the requesting application. The use of synchronous replication should be made upon the detailed
and thorough analysis and understanding of the applications and its I/O needs.
The following demonstration has a small dataset (40 MB) that is written to a primary Live Volume and is
synchronously replicated to an alternate array. The filesystem is formatted as ext4 and mounted (@ /vol_00)
with the discard and sync (the sync option disables filesystem buffer caching) flags.
In this next demonstration, the synchronously replicated Live Volume at the alternate site is mounted to the
secondary Linux host and writes are applied to the secondary Linux host. This demonstrates additional
latency as all write I/O requests are proxied through the secondary SC Series array to the primary array for
processing.
In this final demonstration, the replication link is quickly changed from synchronous replication to
asynchronous replication and write I/O requests are applied to the primary Live Volume. Note the reduction in
time required to commit these writes compared to the previous use of synchronous replication.
real 0m18.266s
user 0m0.104s
sys 0m1.328s
The third site and data in this scenario is replicated asynchronously. One of its uses may include the remotely
located disaster recovery copy of business-critical data. It should be noted that the LVMR copy is always
associated or paired with the primary SC Series array. If the primary and secondary array roles are swapped,
the LVMR copy automatically transfers and associates or pairs itself with the array that assumes the primary
role. Additionally, this asynchronously replicated copy (unlike a synchronously replicated) is current if the
replication link has sufficient bandwidth and the Replicate Active Snapshot (Replay) feature is enabled, or
current as of the last captured snapshot if the feature is disabled. Consequently, discussions should also be
conducted around acceptable RPO and RTO thresholds and maintaining agreeable SLAs with all business
entities involved.
The technical considerations discussed in the section 10.7.2 should be kept in consideration when setting up
this use case.
Summary: In advance of a planned outage, Live Volume can non-disruptively migrate volumes from one SC
Series array to another, enabling continuous operation for all applications — even after one array has
completely powered down.
Operation: In an on-demand, operator-driven process, Live Volume can transparently move volumes from
one SC Series array to another. The applications operate continuously. This enables several options for
improved system operation:
Live Volume continuously monitors for changes in I/O traffic for each volume and non-disruptively moves the
primary storage to the optimal SC Series array for optimum efficiency. This involves the following:
100 Dell EMC SC Series: Synchronous Replication and Live Volume | CML1064
Live Volume use cases
Operation: Live Volume can transparently move volumes from one SC Series array to another. The
applications operate continuously. This enables several options for improved system operation:
101 Dell EMC SC Series: Synchronous Replication and Live Volume | CML1064
Live Volume use cases
Configuration: SC Series arrays must be connected using high-bandwidth and low-latency connections,
especially when synchronous replication is used with Live Volume.
Operation: In an on-demand, operator-driven process, Live Volume can transparently move volumes from
one SC Series array to another. The applications operate continuously. This enables several options for
improved system operation:
Configuration: Live Volumes can be created between any two SC Series arrays in a data center. Each array
can have many Live Volumes, each potentially connecting to a different array.
102 Dell EMC SC Series: Synchronous Replication and Live Volume | CML1064
Live Volume use cases
Operation: In an on-demand, operator-driven process, Live Volume can transparently move volumes from
one SC Series array to another. The applications operate continuously. This enables several options for
improved system operation:
Cloud computing
103 Dell EMC SC Series: Synchronous Replication and Live Volume | CML1064
Technical support and additional resources
Storage solutions technical documents and videos provide expertise that helps to ensure customer success
on Dell EMC storage platforms.
104 Dell EMC SC Series: Synchronous Replication and Live Volume | CML1064