Server to Server Storage Replication 2016-10-11 + 13 min to read + Contributors @O@ In this article Terms Prerequisites Provision operating system, features, roles, storage, and network Configure Server to Server Replication using Windows PowerShell Manage replication Replacing DFS Replication with Storage Replica Related Topics See Also Applies To: Windows Server 2016 You can use Storage Replica to configure two servers to sync data so that each has an identical copy of the same volume. This topic provides some background of this server-to-server replication configuration, as well as how to set it up and manage the environment. To manage Storage Replica you can use PowerShell, or the Azure Server management tools © Important In this scenario, each server should be in a different physical or logical site. Each server must be able to communicate with the other via a network Terms This walkthrough uses the following environment as an example: * Two servers, named SR-SRVOS5 and SR-SRVO6. * A pair of logical “sites” that represent two different data centers, with one called Redmond and one called Bellevue.Building 9 Ssax= Ss Figure 1: Server to server replication Prerequisites * Active Directory Domain Services forest (does not need to run Windows Server 2016). ‘* Two servers with Windows Server 2016 Datacenter Edition installed. * Two sets of storage, using SAS JBODs, fibre channel SAN, iSCSI target, or local SCSI/SATA storage. The storage should contain a mix of HDD and SSD media. You will make each storage set available only to each of the servers, with no shared access. * Each set of storage must allow creation of at least two virtual disks, one for replicated data and one for logs. The physical storage must have the same sector sizes on all the data disks. The physical storage must have the same sector sizes on all the log disks. * At least one ethernet/TCP connection on each server for synchronous replication, but preferably RDMA. * Appropriate firewall and router rules to allow ICMP, SMB (port 445, plus $445 for SMB Direct) and WS-MAN (port 5985) bi-directional traffic between all nodes. * network between servers with enough bandwidth to contain your 10 write workload and an average of =5ms round trip latency, for synchronous replication. Asynchronous replication does not have a latency recommendation. * The replicated storage cannot be located on the drive containing the Windows operating system folder. Many of these requirements can be determined by using the Test-skTopology cndlet . You get access to this tool if you install Storage Replica or the Storage Replica Management Tools features on at least one server. There is no need to configure Storage Replica to use this tool, only to install the cmdlet. More information is included in the steps below. Provision operating system, features, roles, storage, and network1. Install Windows Server 2016 on both server nodes with an installation type of Windows Server 2016 Datacenter (Desktop Experience). Do not choose Standard Edition if it is available, as it does not contain Storage Replica 2. Add network information and join them to the domain, then restart them. D Note From this point on, always logon as a domain user who is a member of the built-in administrator group on all servers. Always remember to elevate your PowerShell and CMD prompts going forward when running on a graphical server installation or on a Windows 10, computer. 3. Connect first set of JBOD storage enclosure, iSCSI target, FC SAN, or local fixed disk (DAS) storage to the server in site Redmond 4. Connect second set of storage to the server in site Bellevue. 5. As appropriate, install latest vendor storage and enclosure firmware and drivers, latest vendor HBA drivers, latest vendor BIOS/UEFI firmware, latest vendor network drivers, and latest motherboard chipset drivers on both nodes. Restart nodes as needed. D Note Consult your hardware vendor documentation for configuring shared storage and networking hardware, 6, Ensure that BIOS/UEFI settings for servers enable high performance, such as disabling C-State, setting QP! speed, enabling NUMA, and setting highest memory frequency. Ensure power management in Windows Server is set to high performance. Restart as required. 7. Configure roles as follows: * Graphical method a, Run ServerManager.exe and create a Server Group, adding all server nodes b. Install the File Server and Storage Replica roles and features on each of the nodes and restart them. * Windows PowerShell method(On SR-SRV06 or a remote management computer, run the following command in a Windows PowerShell console to install the required features and roles and restart them: Copy 2 $_ -Name Storage-Replica,FS-FileServer -IncludeManagenentTools -restart } For more information on these steps, see Install or Uninstall Roles, Role Services, or Features 8. Configure storage as follows © Important * You must create two volumes on each enclosure: one for data and one for logs. * Log and data disks must be initialized as GPT, not MBR * The two data volumes must be of identical size. * The two log volumes should be of identical size. * All replicated data disks must have the same sector sizes. * All log disks must have the same sector sizes. * The log volumes should use flash-based storage, such as SSD. Microsoft recommends that the log storage be faster than the data storage. Log volumes must never be used for other workloads. * The data disks can use HDD, SSD, or a tiered combination and can use either mirrored or parity spaces or RAID 1 o 10, or RAID S or RAID 50. * The log volume must be at least 9GB by default and may be larger or smaller based on log requirements. * The File Server role is only necessary for Test-SRTopology to operate, as it opens the necessary firewall ports for testing, * For JBOD enclosures: a. Ensure that each server can see that site's storage enclosures only and that the SAS connections are correctly configured. b, Provision the storage using Storage Spaces by following Steps 1-3 provided in the Deploy Storage Spaces on a Stand-Alone Server using Windows PowerShell or Server Manager. * For iSCSI storage:a. Ensure that each cluster can see that site's storage enclosures only. You should use more than one single network adapter if using iSCSI. b. Provision the storage using your vendor documentation. If using Windows-based iSCSI Targeting, consult iSCSI Target Block Storage, How To. * For FC SAN storage: a, Ensure that each cluster can see that site's storage enclosures only and that you have properly zoned the hosts. b. Provision the storage using your vendor documentation © For local fixed k (DAS) storage: © Ensure the storage does not contain a system volume, page file, or dump files. © Provision the storage using your vendor documentation 9, Start Windows PowerShell and use the Test-SRTopology cmdlet to determine if you meet all the Storage Replica requirements. You can use the cmadlet in a requirements- only mode for a quick test as well as a long running performance evaluation mode For example, to validate the proposed nodes that each have a F: and G: volume and run the test for 30 minutes: D Copy MD c:\temp Test-SRTopology -SourceComputerName SR-SRV@5 -SourceVolumeName f: -SourceLogVolur © Important When using a test server with no write 1O load on the specified source volume during the evaluation period, consider adding a workload or it will not generate a useful report. You should test with production-like workloads in order to see real numbers and recommended log sizes. Alternatively, simply copy some files into the source volume during the test or download and run DISKSPD to generate write IOs, For instance, a sample with a low write 1 workload for ten minutes to the D: volume: Diskspd.exe -clg -d600 -W5 -C5 -b8k -t2 -02 -r -w5 -i100 d:\test10. Examine the TestSrTopologyReport.html report to ensure that you meet the Storage Replica requirements. ° * (Since etree octet RSE (0 saws ir Tau erng en omar Rat RR es re ce (0 nthe cer 8 a pent ye Initial Synchronization Performance Tests Configure Server to Server Replication using Windows PowerShell Now you will configure server-to-server replication using Windows PowerShell. You must perform all of the steps below on the nodes directly or from a remote management computer that contains the Windows Server 2016 RSAT management tools, Graphical Management of Storage Replica is supported using the free Server Manager Too! (SMT), Please examine Azure documentation for Preview availability of this set of management tools. This document will be updated afer SMT reaches general availability. 1. Ensure you are using an elevated Powershell console as an administrator. 2. Configure the server-to-server replication, specifying the source and destination disks, the source and destination logs, the source and destination nodes, and the log size. PowerShell Copy New-SRPartnership -SourceComputerNane sr-srv@5 -SourceRGName rg@1 -SourceVolumeNarOutput: PowerShell 2 Copy DestinationComputerName : SR-SRV@6 DestinationkGNane 2 rgo2 SourceComputerNane SR-SRVOS PscomputerName : © Important The default log size is 8GB. Depending on the results of the | Test-SRTopology |cmdlet, you may decide to use -LogSizeInBytes with a higher or lower value. 3. To get replication source and destination state, use cet-srcroup and Get-sRPartnership _as follows’ Powershell 2 Copy Get-SRGroup Get-SRPartnership (Get -SRGroup) replicas Output: PowerShell 1 Copy Currentisn :e DataVolume F:\ LastInsyncTime LastknownPrimaryLsn 1 LastoutofsyncTime NumofBytesRecovered 37731958784 NumofBytesRemaining 30851203072 Partitiontd + €3999F10-dbc9-daBe-8F9c-dd2ee6eF3e9F PartitionSize 68583161856 ReplicationMode : synchronous ReplicationStatus : InitialBlockCopy PSComputerName : 4, Determine the replication progress as follows: a. On the source server, run the following command and examine events 5015, 5002, 5004, 1237, 5001, and 2200: PowerShell CopyGet-WinEvent -ProviderName Microsoft-Windows-StorageReplica -max 20 b. On the destination server, run the following command to see the Storage Replica events that show creation of the partnership. This event states the number of copied bytes and the time taken. Example: PowerShell copy Get-winEvent -ProviderName Microsoft-Windows-StorageReplica | where-Object { TimeCreated : 4/8/2016 4:12:37 PM ProviderName : Microsoft-Windows-StorageReplica 1d : 121s Message Block copy completed for replica. ReplicationGroupName: rge2 ReplicationGroupld: (616F1E00-5A68-4447-830F -BOB0EFED359C) ReplicaNam Replicald: {eee99eee-0eae-0eaa-0eae-oeee00e00000) End LSN in bitmap: LogGeneration: (0@e00000-2000-2000-2000-e0eeaaa00a00} LogFilerd: © CLSFLsn: OxFFFFFFFF Aunber of Bytes Recovered: 68583161856 Elapsed Time (ms): 117 D Note Storage Replica dismounts the destination volumes and their drive letters or mount points, This is by design c Alternatively, the destination server group for the replica states the number of byte remaining to copy at all example: es, and can be queried through PowerShell. For copy (Get-SRGroup) .Replicas | Select-Object numofbytesrenaining As a progress sample (that will not terminate): & copy while($true) ($v = (Get-SRGroup -Name "RGO2").replicas | Select-Object numofbytesremainir [System.Console]::Write("Number of bytes remaining: (@}"r", $v.numofoytesre Start-Sleep -s 5 } d. On the destination server, run the following command and examine events 5009, 1237, 5001, 5015, 5005, and 2200 to understand the processing progress. There should be no warnings of errors in this sequence. There will be many 1237 events; these indicate progress. "Copy Get-WinEvent -ProviderName Microsoft-Windows-StorageReplica | FL Manage replication Now you will manage and operate your server-to-server replicated infrastructure. You can perform all of the steps below on the nodes directly or from a remote management computer that contains the Windows Server 2016 RSAT management tools. 1, Use Get-sRPartnership and Get-sRGroup to determine the current source and destination of replication and their status. 2. To measure replication performance, use the | Get-counter_cmdlet on both the source and destination nodes. The counter names are: * \Storage Replica Partition |/O Statistics(*)\Number of times flush paused \storage Replica Partition I/O Statistics(*)\Number of pending flush 1/0 * \Storage Replica Partition |/O Statistics(*)\Number of requests for last log write # \Storage Replica Partition 1/0 Statistics(")\Avg. Flush Queue Length * \Storage Replica Partition {/O Statistics(*)\Current Flush Queue Length # \Storage Replica Partition I/O Statistics(*)\Number of Application Write Requests * \Storage Replica Partition [/O Statistics(*)\Avg. Number of requests per log write# \Storage Replica Partition 1/0 Statistics(*)\Avg. App Write Latency * \Storage Replica Partition I/O Statistics(*)\Avg. App Read Latency # \Storage Replica Statistics(*)\Target RPO # \Storage Replica Statistics(*)\Current RPO # \Storage Replica Statistics(*)\Avg. Log Queue Length * \Storage Replica Statistics(*)\Current Log Queue Length * \Storage Replica Statistics(*)\Total Bytes Received # \Storage Replica Statistics(*)\Total Bytes Sent * \Storage Replica Statistics(*)\Avg. Network Send Latency # \Storage Replica Statistics(*)\Replication State * \Storage Replica Statistics(*)\Avg. Message Round Trip Latency « Storage Replica Statistics(*)\Last Recovery Elapsed Time * \Storage Replica Statistics(*)\Number of Flushed Recovery Transactions * \Storage Replica Statistics(*)\Number of Recovery Transactions * \Storage Replica Statistics(*)\Number of Flushed Replication Transactions # \Storage Replica Statistics(*)\Number of Replication Transactions * \Storage Replica Statistics(*)\Max Log Sequence Number * \Storage Replica Statistics(*)\Number of Messages Received \Storage Replica Statistics(*)\Number of Messages Sent For more information on performance counters in Windows PowerShell, see Get- Counter. 3. To move the replication direction from one site, use the set-srpartnership cmdlet. & Copy Set-sRPartnership -NewSourceComputerName sr-srv@6 -SourceRGName rg@2 -Destination$ Warning Windows Server 2016 prevents role switching when the initial sync is ongoing, as it can lead to data loss if you attempt to switch before allowing initial replication to complete. Do not force switch directions until the initial sync is complete Check the event logs to see the direction of replication change and recovery mode occur, and then reconcile. Write IOs can then write to the storage owned by the new source server. Changing the replication direction will block write IOs on the previous source computer. 4,To remove replication, use Get-srsroup , Get-SRPartnership , Renove-sRGroup , and Renove-SRPartnership on each node. Ensure you run the Renove-SRPartnership |cmdlet on the current source of replication only, not on the destination server. Run Renove-Group on both servers, For example, to remove all replication from two servers: 2 Copy Get-srPartnership Get-sRPartnership | Renove-SRPartnership Get-SRGroup | Renove-SRGroup Replacing DFS Replication with Storage Replica Many Microsoft customers deploy DFS Replication as a disaster recovery solution for unstructured user data like home folders and departmental shares. DFS Replication has shipped in Windows Server 2003 R2 and all later operating systems and operates on low bandwidth networks, which makes it attractive for high latency and low change environments with many nodes. However, DFS Replication has notable limitations as a data replication solution: * Itdoes not replicate in-use or open files. * It does not replicate synchronously. * Its asynchronous replication latency can be many minutes, hours, or even days. * It relies on a database that can require lengthy consistency checks after a power interruption. * Itis generally configured as multi-master, which allows changes to flow in both directions, possibly overwriting newer data. Storage Replica has none of these limitations, It does, however, have several that might make it less interesting in some environments:* It only allows one-to-one replication between volumes, It is possible to replicate different volumes between multiple servers. ‘© While it supports asynchronous replication, it is not designed for low bandwidth, high latency networks. # It does not allow user access to the protected data on the de: ongoing ination while replication is If these are not blocking factors, Storage Replica allows you to replace DFS Replication servers with this newer technology. The process is, at a high level: 1. Install Windows Server 2016 on two servers and configure your storage. This could mean upgrading an existing set of servers or cleanly installing. 2. Ensure that any data you want to replicate exists on one or more data volumes and not on the C: drive, a. You can also seed the data on the other server to save time, using a backup or file copies, as well as use thin provisioned storage. Making the metadata-like security match perfectly is unnecessary, unlike DFS Replication, 3, Share the data on your source server and make it accessible through a DFS Namespace. This is important, to ensure that users can still access it if the server name changes to one ina disaster site a. You can create matching shares on the destination server, which will be unavailable during normal operations, b. Do not add the destination server to the DFS Namespaces namespace, or if you do, ensure that all its folder targets are disabled. 4, Enable Storage Replica replication and complete initial sync. Replication can be either synchronous or asynchronous. a. However, synchronous is recommended in order to guarantee IO data consistency on the destination server. b, We strongly recommend enabling Volume Shadow Copies and periodically taking snapshots with VSSADMIN or your other tools of choice. This will guarantee applications flush their data files to disk consistently. In the event of a disaster, you can recover files from snapshots on the destination server that, asynchronously. Snapshots replicate along with files. 5, Operate normally until there is a disaster. 6, Switch the destination server to be the new source, which surfaces its replicated volumes to users, ight have been partially replicated 7. If using synchronous replication, no data restore will be necessary unless the user was using an application that was writing data without transaction protection (this isirrespective of replication) during loss of the source server. If using asynchronous replication, the need for a VSS snapshot mount is higher but consider using VSS in all circumstances for application consistent snapshots. 8, Add the server and its shares as a DFS Namespace folder target. 9. Users can then access their data, DNote Disaster Recovery planning is a complex subject and requires great attention to detail Creation of runbooks and the performance of annual live failover drills is highly recommended. When an actual disaster strikes, chaos will rule and experienced personnel may be unavailable, Related Topics * Storage Replica Overview * Stretch Cluster Replication Using Shared Storage * Cluster to Cluster Storage Replication Storage Replica: Known Issues * Storage Replica: Frequently Asked Questions See Also © Windows Server 2016 * Storage Spaces Direct in Windows Server 2016