MICROSOFT SQL SERVER

HIGH AVAILABILITY
AND DISASTER RECOVERY
Michael Poremba // October 2008
 Database HA & DR Experience…
2

 Work with business to determine HA or DR

requirements for applications and data?

 Design HA or DR solutions?

 Administer HA or DR process?

 Still learning MS SQL Server HA & DR capabilities?

 Scope of this Presentation
3

Presentation Focus Beyond Scope of Presentation

 Data Availability  In-depth how-to

(available elsewhere)
 Data recovery  Partitioned views (federated)
 High availability  Advanced DBA techniques
 Custom application logic
 Disaster recovery  3rd-party software solutions
 Technology Focus  Alternate DBMS engines
(e.g. Oracle; DB2)
 MS SQL Server  HA on virtual machines
 Physical servers  Complex scenarios & solutions
 SANs  Load balancing
4 Introduction to Data Availability
So, you need to make your
production database bulletproof…
 Data Availability Continuum
5

Degrees of protection for information systems:

Business Risk Solution
Data Recovery Data loss Redundant data
High Availability Downtime of Redundant system
database service components
Disaster Recovery Downtime of Redundant systems
business operations and facilities
 Business Case for Availability
6

High Availability Disaster Recovery

 Keep business-critical  Protect against loss of

applications available data center
 Secondary:  Secondary:
 Server maintenance  Application upgrades
 Infrastructure
upgrades
 Service Level Agreement (SLA)
7

 Permitted downtime (planned vs. unplanned?)

Uptime SLA Downtime Downtime
per Year per Month
99.9% 8.76 hours 43.8 minutes
99.99% 52.6 minutes 4.38 minutes
99.999% 5.26 minutes 0.438 minutes
 Acceptable data/transaction loss
 Application response times
 Mean time to recovery

Note: Database uptime is not equivalent to application availability

 Failures of other application services
 Network outages
 Protect What?
8

 Application data stores

 Databases
 Files
 Other data repositories
 Database services
 DBMS availability for applications
 Application services
 Application availability for users and external systems

Databases are the heart of most information systems;

they deserve the highest affordable protection.
 Database Failure Scenarios
9

Physical Infrastructure Failures Logical Data Failures

 Storage subsystem  Operator errors

 Disk  DBMS interruption
 Controller  Drops / deletes
 Network  Application defects
 Server  DBMS defects
 Power  Data corruption
 Service Recovery Strategies
10

Standby Failover Behavior SQL Server Feature

Mode
Cold • Manual intervention • Backup and restore
standby required to restore offline
data copy
Warm • Data copy online and ready • Transaction log
standby • Manual failover required shipping
• Database mirroring
Hot • Automatic failover • Database mirroring
standby • Failover clustering
 Data Recovery—Terminology
11

Terminology varies for source vs. copy

High Availability Data Source Data Copy
Strategy
Backup and Restore Database Backup
Log Shipping Primary Secondary
Standby
Database Mirroring Principal Mirror
Failover Clustering Primary Secondary
Active Passive
Standby
Inactive
12 Data Recovery
[Briefly…]
 Database Backups
13

 Traditional backup types

 Full backup
 Differential backup
 Transaction log backup
 Disk is better than tape
 First backup to disk (separate physical disk volume)
 Detect exceptions encountered during backup
 Verify backup files
 Copy backup files to tape or remote disk
 Data retention policy for backup files
 Database Backup Strategy
14

Backup of user databases not sufficient for recovery

 System database

 Master database

 MSDB database

 Model database

 External data stores…

 Synch with External Data Stores
15

Synchronize recovered database with external data

stores:
Identity column seeds

Full-text indexes
(SQL Server 2000)
LDAP entries
File system objects

Other databases
 Backup Retention Policy
16

 Location of backup files

 Duration of retention
 Protection of sensitive data
 Sarbanes/Oxley (SOX)
 HIPAA
 Internal policies for data management and protection
 Access to backups from offsite data storage
 Data Recovery Process
17

 Backup file sets  Recovery strategy depends on

 Full baseline, differential, and failure scenario
transaction logs  Create comprehensive failure
 Retrieving backup files matrix
 Offsite storage
 Devise recovery strategy for
each scenario
 Tape  Does worst-case recovery
 Network copy scenario fit within SLA
 Dependency on multiple parameters?
people to get access to  Recovery time; SLA
backup files
 Include future data growth in
recovery plan
 Fully test recovery strategies
—practice is essential
18 High Availability
 High Availability
19

 Minimize or avoid service downtime

 Whether planned or unplanned
 When components fail,
service interruption is brief or non-existent
 Automatic failover
 Eliminate single points of failure (as affordable)
 Redundant components
 Fault-tolerant servers
 Redundant Components
20

Objective: Avoid single points of failure (where affordable)

Approach: Use redundant components for database service
 Database server nodes

 Server components
 ECC RAM; failure-tolerant HW & OS
 DBMS instance
 User databases
 Storage devices
 Storage unit components
 MPIO: Interfaces; paths; switches; controllers
 RAID: Disks
 Networking
 MPIO: Interfaces; paths; switches
 Data copies
 E.g. Recovering torn page from mirror in SQL Server 2008
 Transaction Log Shipping
21

 Warm standby solution

 Duplicate user database
 Copy transaction logs to standby server & restore
 Database available for read-only access
 Users must disconnect for logs to be applied
 Two database licenses required if querying standby
 Manual application failover
 Supported on standard hardware
 Possible data loss (unapplied transactions)
 Database Mirroring
22

 Redundancy at user database level

 Duplicate copy of user database
 Independent storage devices
 Multiple copies of instance databases
witness
 Mirrored over private network channel (optional)
 Mirror always redoing transactions from principal
 Negligible impact on transaction throughput
 Multiple mirroring modes: node A node B
 High-availability: commit @ log on mirror; automatic failover
 High-protection: commit @ log on mirror; manual failover
 High-performance: commit when logged on principal
 Very fast automatic failover—seconds Local Storage Local Storage
· local sys DBs · local sys DBs
 Requires witness server · source user DB · mirror user DB
 Mirror-aware application client connection
 Provided by client library
 Database connection string must specify both servers
 Mirror may be available for read-only access (snapshots)
 Works with standard hardware
 Mirror Witness
23

 With mirroring, more than one server is required to

decide on failover
 Witness automates failover from primary to mirror
 Watches database availability
 Reports observations back to principal and mirror
 Runs in separate SQL Server instance (Express is OK)
 Prevents “split brain” scenario
 Very low resource consumption
 Can be witness for multiple databases
 Not a single point of failure
 SQL Server Failover Clustering
24

 Two clustered nodes

 Active/Passive config
 MS SQL services
 Running on virtual server node A node B

 Shared storage device Shared Storage

 User databases · system DBs
· user DBs
· quorum
 System databases
 Quorum drive
 Redundant internal
components
 Active/Passive Failover Clustering
25

 Redundancy at database instance level

 All databases fail over together
 Shared copy of system databases
 Single data copy on shared storage
device
 No I/O overhead reducing throughput
 Storage unit is single point of failure for node A node B
cluster
 All database services are clustered
 SQL Agent; Analysis Services; Full-Text Shared Storage
engine, MS DTC · system DBs
· user DBs
 Automatic failover (up to minutes)
· quorum
 DBMS accessed over virtual IP
 Database not available from inactive
node for DB client connections
 Storage is controlled by one cluster node
at a time
 Requires hardware certified by Microsoft
for Microsoft Cluster Service
 HA Comparison
26

Database Mirroring Failover Clustering

 Scope: user DB  Scope: DBMS instance
 Standard hardware  Certified hardware
 One SQL license  One SQL license
(unless querying snapshots on (only one node can access
mirror) database)
 Very fast failover (seconds)  Automatic failover (up to minutes)
 OS flexible (e.g. 32/64)
 Enterprise OS
 Independent storage
 Shared storage
 Independent services
 Clustered services
 Standby not available
 Reporting on mirror
 Servers are usually co-located
 Geographic separation OK
 Considerations for HA
27

 HA complements backup and recovery strategy

 Does not replace data recovery plan
 Application service availability is often determined by
a network of interdependent services
 Availability can be difficult to define (e.g. partial failures)
 Failure probability difficult to measure or compute
 Increased system complexity could lead to lower
service availability!
 Operator error a leading cause of availability issues
 Increased number/types of system components
 More complex to configure and administer
 Data Recovery Requirements
28
29 Disaster Recovery
 Disaster Recovery
30

 Minimize downtime of business operations

 Redundant systems and facilities
 SQL Server features:
 Transaction log shipping
 Database mirroring
 Failover clustering
 Other technologies
 Storage-based mirroring
 Disaster Recovery Planning
31

 Data security requirements

 Clarify SLA, data loss allowance
 Evaluate system cost vs. data protection
 Failure analysis
 System redundancy
 Process validation
 Training for personnel
 Prevention practices
 Executing disaster recovery and business continuity
 Practice, practice, practice
 Business Continuity Facility
32

 System redundancy
 Systems: Web servers app servers; database, etc.
 Data: Databases; data files on OS; security info, etc.
 Networking: Domain, routing, subnet, VIPs, etc.
 Alternate facilities
 Network bandwidth
 Physical or network access by operations staff
 Failover
 Often a deliberate decision, using manual failover
 Data Redundancy
33

 Synchronous redundancy
 Network bandwidth cost
 Network latency and application performance
 Network reliability
 Asynchronous redundancy
 Risk of data loss
 More cost-effective
 Resilient to network latency issues
 Candidate Technologies
 SQL Server database mirroring
 Failover clustering with SAN-based mirroring
 DR Using Database Mirroring
34

 Two sites: Primary and DR location

 Separate failover clusters at each site
 SQL Server database mirroring between sites
witness
(optional)

failover cluster at site A failover cluster at site B

node A1 node A2 database node B1 node B2

mirroring

Shared Storage A Shared Storage B

· local sys DBs · local sys DBs
· local quorum · local quorum
· source user DB · mirror user DB
 DR Using SAN-Based Mirroring
35

 Two sites: Primary and DR location

 Four-node failover cluster; one virtual IP address
 SAN-based mirroring between sites
 Manual cluster failover
failover cluster nodes at site A failover cluster nodes at site B

node A1 node A2 node B1 node B2

storage-
based
Shared Storage A mirroring Shared Storage B
· system DBs · system DBs
· quorum · quorum
· user DBs · user DBs
36 Complimentary Technologies
[Skip if time is running short.]
 SAN-Based Data Mirroring
37

 Data blocks duplicated at storage level

 Similar to transaction log shipping
 Copy performed in sequence and coordinated with
database checkpoint
 Ensures consistency of mirrored data files
 Synchronous or asynchronous mirroring
 Co-located or geographically dispersed—both are OK
 SAN link bandwidth must support database I/O rate
 May require extra feature support from SAN vendor
 Could rely on Failover Clustering for HA
 SQL Server Database Snapshots
38

 Read-only point-in-time database snapshot

 No data is copied—instantaneous
 Historical snapshot pages tracked separately from
changing pages
 Snapshots can be maintained indefinitely
 Limited only by available storage
 Snapshot copy can be used for reporting
 Read-only, so no locking issues
 SQL Server Replication
39

 Transactional replication  Subscriber databases

 High transaction volume available for reporting
 Low data latency required  Replicate data subsets
 Mixed technologies:  Some data loss is possible
Integrates with other DBMS
 Merge replication
 Periodically validate
replicated data
 Bi-directional data changes
 Typically server-to-client
 Snapshot replication
 Large, infrequent data
changes
 Data change latency OK
 Best for smaller data sets
40 App Development and Admin
 Considerations for App Developers
41

 App services tolerant to database service interruptions

 Application transactions must be handled in code—data consistency
 Exception handling for transaction retry, connection recovery
 Requires coding standards, code reviews, and testing
 Bulk data operations
 Transaction volume impacts rollback time during failover
 Batch jobs must be run on alternate nodes
 Don’t bypass transaction logging
 Synchronization with external data sources?
 Be aware of database recovery model
 Mirroring uses FailoverPartner in connection string
 Use TCP/IP as client protocol
 Considerations for Admins
42

 Use identical server hardware, when possible

 Design network redundancies, when feasible
 Consider network latency for geographic separation
 Always manage through virtual cluster, not individual cluster nodes
 Retest failover/failback after HA maintenance
 Diagnose after failover
 Repair alternate node
 Resynchronize data, as necessary
 Be aware of primary/secondary locations
 Ensure application services are connected and functioning properly
 Keep server node configurations synchronized:
 Service pack and patch levels
 Duplicate non-redundant resources
 Jobs; logins and permissions; OS & sys objects
 HA Risks
43

 System performance degradation

 HA system complexity leads to availability issues
 Some system failures not planned for
 Backup and recovery planning incomplete
 Administrators not fully trained or informed
 User databases not synchronized with other data
sources
 Common Admin Use Cases
44

 Maintain HA nodes
 Hardware maintenance
 Rolling upgrades and software patches
 Resynchronize the redundant copy
 Re-synch mirror
 Restart log shipping
 Diagnose and repair
 Diagnose cause of failover
 Repair failed node and restore failover capabilities
 Test failover and failback
 Common Admin Actions
45

Train and practice administrators to:

 Initiate a database mirror

 Manually failover mirror database or cluster node

 Add/remove passive node from mirror or cluster

 Upgrade/patch servers nodes

 Restart or redirect application services

46 More Information
 References—Books
47

High Availability Related Topics

 Microsoft SQL Server 2008 High  Pro SQL Server 2005 Replication
Availability with Clustering & by Sujoy Paul, 2006.
Database Mirroring  Pro SQL Server 2005 Service
by Michael Otey, 2009. Broker
 Microsoft SQL Server High by Klaus Aschenbrenner, 2007.
Availability  The Rational Guide to SQL Server
by Paul Bertucci, 2004. 2005 Service Broker
 Pro SQL Server 2005 High by Roger Wolter, 2006.
Availability
by Allan Hirt, 2007.
 References—Presentations
48

 Microsoft Load Balancing and Clustering

http://ce.sharif.edu/courses/84-85/2/ce317/resources/root/lecture%20slides/14.%20Microsoft%20L
oad%20Balancing%20and%20Clustering.ppt
 SQL Server 2005 High Availability
http://www.atlantamdf.com/Presentations/AtlantaMDF_111207HA.ppt
 High Availability Technologies In SQL Server 2000 And SQL Server 2005
http://202.181.238.2/hk/teched2004/ppt/Day_2_Rm407/DAT431(1330-1445).ppt
 Meeting the Availability Challenge
http://download.microsoft.com/download/E/D/C/EDCF54DB-19CD-4882-9FC4-
4F7D46FCEAA6/HighAvailability.ppt
 Disaster Recovery Mistakes
http://www.sqlsig.org/Oct%2011%20DASSUG%20-%20Jason%20Hall%2010-11-07%20MM.ppt
 SQL Server 2005 High Availability
http://blogs.msdn.com/sql2005event/attachment/564303.ashx
 Effective Usage of SQL Server 2005 Database Mirroring
http://www.sqlserver-qa.net/SSQA-Effective%20Usage%20of%20SQL%20Server
%202005%20Database%20Mirroring_show.ppt
 References—Articles
49

 Achieve High Availability for SQL Server

http://technet.microsoft.com/en-us/magazine/cc162477.aspx
 Geographically Dispersed Clusters in Windows
Server 2003
http://www.microsoft.com/windowsserver2003/techinfo/overview/clustergeo.mspx
 Restoring file and filegroup backups
http://support.microsoft.com/kb/281122/en-us
 Restoring specific tables or rows from backups
http://support.microsoft.com/kb/321836/en-us
 Maintaining Availability During Upgrades
http://msdn.microsoft.com/en-us/library/ms191449.aspx