Hyper-V 2012 CSV Support Considerations

v2 updated 7/26/2013 to add additional resources and antivirus concerns

Purpose
This article is intended to discuss known software defects in Microsoft Hyper-V 2012
related to Cluster Shared Volumes (CSV), and Microsoft’s supported/recommended
configurations and patches required to allow successful 3rd party backups of Hyper-
V 2012 systems that utilize CSV. Additionally, this article includes information
necessary to ensure the default “system” writer is used when backing up Hyper-V
guests using the Unitrends Agent.

Description
Symptoms of an improperly patched Hyper-V 2012 server with CSV can include but
are not limited to:
 CSV completely going offline when a backup is attempted using VSS
 Windows System events 5120 or 5142 are logged during heavy disk activity
cycles or when taking a backup
 Hyper-V nodes are removed from active cluster membership, System Event
ID 1135 is logged
 You have Event ID 4 Kerberos KRB_AP_ERR_MODIFIED errors appearing on
2003 Server and XP workstations when attempting to access file shares
hosted on Server 2012 systems using CSV.
 The Windows Systems Log on Server 2012 system has event IDs 1228 or
1196 when attempting to bring a cluster resource online.
 Live migration of VMs between hyper-V nodes fails with event ID 21502.
 An attempt to back up a Hyper-V guest is interrupted or fails and a
VSS_E_SNAPSHOT_SET_IN_PROGRESS error is generated in the event logs.
 Memory usage on a Hyper-V node increases dramatically during snapshot
corresponding to read requests from other non-owning nodes
 When under heavy load a Hyper-V node may crash and provide the following
Stop error message: STOP: 0x000000C2 (parameter1, parameter2,
parameter3, parameter4)
 When using thin provisioned guest disks, I/O delays or I/O stalls occur
frequently during volume unmounting
 Agent backups attempt to use a VSS provider other than the Microsoft
System Writer
  Unitrends Agent backups of Hyper-V 2012 servers configured with remote
storage fail.

Cause
The production release of Hyper-V 2012 Server was released with numerous bugs
related to a new implementation of CSV. These bugs do have current known
hotfixes, however most of these hotfixes are NOT part of any current patch
available through Windows Update Services. Many of these bugs can cause a CSV
to go offline completely causing data loss in most scenarios, when snapshots are
being taken of the volume or when high load is applied to the storage system.

Additionally, Hyper-V 2012 has known configuration requirements Microsoft has

conveyed concerning network topology when iSCSI storage is in use that could lead
to further complications and unreliability.

Finally, Hyper-V guests by default are not snapshotted in all cases by the default
system provider, but by the SAN storage provider. Such configurations are not
always compatible with the Unitrends Agent for Hyper-V guest backup and
recovery.

An initial issue with the CSV implementation in Hyper-V 2012, which is a new
architecture from previous CSV implementations, was first discovered with
Microsoft’s own DPM backup solution. This issue impacts all backup solutions, 1st
and 3rd party for hyper-V 2012.

Resolution
This article pulls together all resources current as of July 22, 2013. Updates to this
article will be posted as more information is made available. As this has been an
ongoing saga including more than 10 total KBs to resolve a single issue, neither
Unitrends nor Microsoft can confirm this solution will work for all parties in all
scenarios and cannot guarantee unresolved issues still remain unreported that may
impact CSV stability. However, at this time numerous Unitrends clients have
confirmed successful backups of Hyper-V 2012 using CSV without incident after
following the patch process provided by Microsoft detailed in this article.
By following the information below, Hyper-V 2012 systems using remote or CSV
storage should be able to be backed up successfully using the Unitrends Agent vs
the Microsoft System Provider via VSS.
1) Install the following hotfix by clicking the “Hotfix Download Available” image
at the top of this KB on every Hyper-V 2012 node. The link will direct you to
a download link for a fix for Windows 8 RTM (this is correct for Server 2012
64bit systems although not intuitive). You must reboot after installing this
hotfix. http://support.microsoft.com/kb/2870270
This hotfix includes all hotfixes from KBs 2870270, 2848344, 2796995, 279728,
2801054, 2813630, and 2848727.
2) Install the following hotfix by clicking the “Hotfix Download Available” image
at the top of this KB on every Hyper-V 2012 node. A reboot is required after
installing this hotfix. http://support.microsoft.com/kb/2838043
Further, if you have experienced any of the symptoms noted in the KB’s
symptoms section previously then you must change the password on the
named cluster resources following the procedures in the KB article.

3) Ensure the patch rollup for 2012 servers from March 2013 via KB 2811660 is
installed. You must restart each node in the cluster after installing this
update rollup. The download link is included here:
http://www.microsoft.com/en-us/download/details.aspx?id=36916
4) Ensure your SAN architecture is running current firmware and storage drivers
(and if necessary fiber hardware drivers and switch firmware). Preferably
drivers and firmware which have been released after May 10 2013 as those
are likely to include or be in reflection of Microsoft’s own hotfix efforts
themselves to tolerate I/O latency issues caused by CSV under load. Ensure
your SAN hardware is openly supported and certified by Microsoft for use
with 2012 CSV.
5) Ensure the mainboards in your server are running the latest BIOS or EFI
firmware, and that all hardware drivers have been updated, specifically
noting tcpip.sys, netio.sys, mrxsmb.sys, mrxsmb20.sey, mrxsmb10.sys and
srv.sys. Ensure your server hardware is Certified for Windows 2012 systems
by Microsoft.
6) Ensure the networking requirements for number of NICs and required
dedicated subnets for Storage, Heartbeat, and public data access have been
reviewed and are correct.
7) Run the Microsoft “Validate a Configuration Wizard” or the PowerShell
equivalent scripts following the information in the following article for 2012
clusters ensuring to include the full storage tests. This test can only be run
during a maintenance window as it requires guests to be powered down
during some tests. http://technet.microsoft.com/en-
us/library/jj134244.aspx If any deficiencies are reported, resolve them
before attempting backups using any VSS aware tools.
8) Ensure the 7.1 Unitrends agent or higher is deployed to all Hyper-V 2012
nodes and on the backup appliance.
 9) Edit the file C:/PCBP/master.ini on all nodes. Locate the section [Usnap]
near the bottom of the file and add the line: “UsnapProvider=System” (note
this is CASE SENSITIVE but does not include the quotes). Save the
Master.ini file. Agents do not need to be restarted after this change. If
backups previously ran using a provider other than system a new full backup
of each Hyper-V guest is required.
10) Ensure any antivirus engine excludes all scans of C:\Windows\cluster
directory
11)

After patching your system, confirming proper network routing and number of
network cards are used, and if running on certified platform for Hyper-V 2012
clusters, your cluster validation should be successful. With the change to step 9,
Unitrends backups should also be successful using our agent.

Note: some limitations of Windows and Hyper-V 2012 prevent certain types of
guest configurations being supported for backup and recovery through VSS. Please
review the links below and ensure your host and guest architecture is compliant
with Microsoft’s backup requirements. Some guest configurations are possible that
are not supported under VSS for backup and will require the deployment of an
agent inside the guest for protection by Unitrends.

Third-Party Sources
Links of particular value in diagnosing and configuring your Hyper-V 2012 CSV
environment are highlighted in red below.

A technet forum post containing a detailed and routinely updated history of the
problems with Hyper-V 2012 CSV backups can be found here:
http://social.technet.microsoft.com/Forums/en-US/604409df-ada1-47d1-bdfb-
3f938cde0b59/dpm-2012-sp1-beta-causing-server-2012-hyperv-cluster-hang-iscsi-
problems

PowerShell Script to check for all related Hyper-V 2012 hotfixes!

http://blogs.technet.com/b/cedward/archive/2013/05/24/validating-hyper-v-2012-
and-failover-clustering-2012-hotfixes-and-updates-with-powershell.aspx
List of Failover Cluster hotfixes for 2012 server from the MS cluster team Wiki:
http://social.technet.microsoft.com/wiki/contents/articles/15577.list-of-failover-
cluster-hotfixes-for-windows-server-2012.aspx
Official Microsoft KB for failover cluster hotfixes: (this is expanded upon greatly by
the link above and is included for reference only)
http://support.microsoft.com/?id=2784261
Windows Server 2012 Hyper-V Best Practices (EASY CHECKLIST )
http://blogs.technet.com/b/askpfeplat/archive/2013/03/10/windows-server-2012-
hyper-v-best-practices-in-easy-checklist-form.aspx
Network configuration requirements for Hyper-V 2012 (Very Important)
http://blogs.technet.com/b/gavinmcshera/archive/2011/03/27/3416313.aspx
http://blogs.msdn.com/b/clustering/archive/2011/06/17/10176338.aspx
http://blogs.technet.com/b/askcore/archive/2010/02/12/windows-server-2008-
failover-clusters-networking-part-1.aspx

http://blogs.technet.com/b/schadinio/archive/2010/11/06/virtualization-network-
card-settings-for-hyper-v-cluster.aspx
http://social.technet.microsoft.com/wiki/contents/articles/9711.hyper-v-network-
design-configuration-and-prioritization-guidance.aspx

Antivirus software and clustering

http://support.microsoft.com/kb/250355
http://support.microsoft.com/kb/309422
Information about the changes to Failover Clustering in 2012 for Hyper-V:
http://blogs.technet.com/b/askcore/archive/2012/07/23/windows-server-2012-
failover-cluster-sessions-at-teched.aspx
More info about 2012 failover clustering:
http://technet.microsoft.com/en-in/library/hh831414.aspx
Best practices for Live-Migration and its configuration
http://blogs.technet.com/b/askcore/archive/2009/12/10/windows-server-2008-r2-
live-migration-the-devil-may-be-in-the-networking-details.aspx
Known limitations of Hyper-V 2012 with Passthru Disks in use:
http://blogs.technet.com/b/askcore/archive/2013/01/24/behavior-change-when-
working-with-pass-through-disks-in-windows-server-2012-failover-clusters.aspx
Windows 2012 clustering news group:
http://social.technet.microsoft.com/Forums/windowsserver/en-
US/home?forum=winserverClustering
Microsoft support policy for 4k sector hard disks
http://support.microsoft.com/kb/2510009#Application
windows server 2012 Backup features (Supported/Not Supported)
http://technet.microsoft.com/en-us/library/jj614621.aspx
Windows Backup of Hyper-V Supported in 2012 (unlike 2008 R2)
http://blogs.msdn.com/b/virtual_pc_guy/archive/2013/02/18/windows-backup-
and-hyper-v-in-server-2012.aspx
Backup of Hyper-v VMs using CMD
http://blogs.msdn.com/b/virtual_pc_guy/archive/2013/02/25/backing-up-hyper-v-
virtual-machines-from-the-command-line.aspx
Features added/Updated in Hyper-V
http://technet.microsoft.com/en-us/library/hh831531.aspx
Features added/Updated in Failover Clustering 2012
http://technet.microsoft.com/en-us/library/hh831414.aspx
Features added/Updated in Failover Clustering 2012 R2 (Preview)
http://technet.microsoft.com/en-us/library/dn265972.aspx
2012 Failover Clustering
http://technet.microsoft.com/en-us/library/hh831579.aspx
List of Supported OS on Hyper-v 2012
http://blogs.technet.com/b/schadinio/archive/2012/06/26/windows-server-2012-
hyper-v-list-of-supported-client-os.aspx

Meta tags: Hyper-V 2012, CSV, system provider, KB2870270, ID5120, ID5142