Best Practices for protecting Oracle RAC with NetBackup

Issue Date: 29th July 2009 Version number: 1.0 Applies to: All supported versions of NetBackup up to version 6.5.4 Note: This is a living document and will be subject to periodic updates. Please check the data and version number to ensure you are referencing the latest version. If you have any feedback or questions about this whitepaper please email them to [email protected]

This document is provided for informational purposes only and is intended for distribution only by Symantec employees to selected partners and customers. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice. Copyright 2009 Symantec Corporation. All rights reserved. Symantec, the Symantec logo and NetBackup are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners.

Contents
1.0 INTRODUCTION .................................................................................................................. 1

1.1 INTENDED AUDIENCE ........................................................................................................... 1 1.2 SOLUTION OVERVIEW ........................................................................................................... 1 1.2.1 Oracle Recovery Manager (RMAN) ........................................................................... 1 1.2.2 NetBackup for Oracle................................................................................................. 1 1.3 GLOSSARY ............................................................................................................................... 1 1.4 ADDITIONAL READING................................................................................................................ 2 2.0 3.0 3.1 3.2 3.3 3.4 3.5 4.0 5.0 OPERATIONAL CHARACTERISTICS ................................................................................ 2 RAC BACKUP OPTIONS .................................................................................................... 3 FAILOVER VIP EXISTS, BACKUP IS NOT LOAD BALANCED......................................................... 3 FAILOVER VIP EXISTS, BACKUP IS LOAD BALANCED ............................................................... 4 VIP DOES NOT FAILOVER, BACKUP IS NOT LOAD BALANCED .................................................... 6 VIP DOES NOT FAILOVER, BACKUP IS LOAD BALANCED, 1 POLICY WITH CUSTOM SCRIPT ........... 7 VIP DOES NOT FAILOVER, BACKUP IS LOAD BALANCED, SEPARATE AUTOMATIC POLICIES .......... 8 CATALOG CONSIDERATIONS .......................................................................................... 9 STEPS TO CONFIGURE LOAD BALANCED BACKUPS ................................................ 10

Best Practices for protecting Oracle RAC with NetBackup Page i

Issue date: 29th July 2009

1.0 Introduction
Oracle database backup and recovery becomes more difficult as databases grow in size and as increasing demands on database availability limit the time available to perform backups. Often the backup window is too short to accommodate a complete backup process. Database administrators are looking for methods that will enable them to complete these large backups in the time available. For the Oracle Real Application Clusters (RAC) database, NetBackup provides the ability to split backups in pieces and send them in parallel across multiple nodes, therefore shortening the time needed to complete a backup of the whole database. NetBackup software (NBU) is an enterprise wide backup and recovery solution. Symantec Software has worked closely with Oracle Corporation to develop a highly scalable and reliable online backup and recovery solution for Oracle databases. NetBackup for Oracle software protects both data and the availability of Oracle applications. This paper will discuss methods that can be used to balance the backup load across multiple nodes of an Oracle RAC database. This is an advanced technical paper. Familiarity with Oracle RAC, Oracle RMAN and NetBackup is expected from the reader.

1.1 Intended Audience

Backup administrators and database administrators can use this paper to determine the best solution for protecting their Oracle RAC environments.

1.2 Solution overview

Solutions described in this paper his paper utilize Oracle Recovery Manager and NetBackup for Oracle. Oracle System Backup to Tape (SBT) API is used to enable direct access to the functionality of NetBackup (NBU).

1.2.1 Oracle Recovery Manager (RMAN)

Oracle databases can be protected with the out-of-the-box Oracle Recovery Manager (RMAN) utility. RMAN provides the interface to the databases, and performs backup and recovery functions. It controls data streams going in or out of a database for the purpose of backup and recovery. RMAN by itself can save to or restore from disk only. RMAN can only connect to one target instance in a RAC database at a time. However, this does not preclude load balancing a backup across all instances. This is because RMAN makes a distinction between target connections and subsequent channel allocation connections. RMAN connections initiated via the allocate channel command can be distributed among all instances in a RAC database. This is the key to the load balancing of data files and archive logs between instance and channels.

1.2.2 NetBackup for Oracle

NetBackup (NBU) provides a comprehensive data protection solution, including centralized administration and reporting, media management, automated policy based backups, and simplified restores. NetBackup for Oracle Agent (NBU Oracle) extends the capabilities of NetBackup to include online backups and restores of Oracle databases. NetBackup for Oracle tightly integrates with RMAN and tailors NetBackup capabilities to the uniqueness of an Oracle environment. Through the GUI or command line interface it provides a complete, flexible database protection solution for a variety of platforms. NBU Oracle provides a media management to RMAN enabling backup to media other than disk.

1.3 Glossary
The following descriptive terms are used throughout this document: Backup piece - The physical file format used to store RMAN backup sets. Best Practices for protecting Oracle RAC with NetBackup Page 1 Issue date: 29th July 2009

Backup set - A backup of one or more datafiles, control files, SPFILEs and archived redo log files. Each backup set consists of one or more binary files called backup pieces. Backup pieces are written in a proprietary format that can only be created or restored by RMAN. Instance - A system global area (SGA) and the Oracle background processes constitute an Oracle database instance. Every time a database is started, a system global area is allocated and Oracle background processes are started. The SGA is de-allocated when the instance shuts down. Real Application Clusters (RAC) - Option that allows multiple concurrent instances to share a single physical database. Recovery Manager (RMAN) - A utility that backs up, restores, and recovers Oracle databases. It can be used with or without the central information repository called a recovery catalog. If you do not use a recovery catalog, RMAN uses the database's control file to store information necessary for backup and recovery operations. You can use RMAN in conjunction with a media manager to back up files to tertiary storage. SBT - System Backup to Tape. This term specifies a nondisk backup device type, typically a tape library or tape drive. RMAN supports channels of type disk and SBT. TNSPING Oracle utility that determines whether or not a service (for example, an Oracle database, an Oracle Names server, or any other Oracle service) on a Oracle Net network can be successfully reached. VIP Virtual IP address is an IP address that is not connected to a specific computer or network interface card (NIC) on a computer. VIP address will still be available if a computer or NIC fails because an alternative computer or NIC replies to connections.

1.4 Additional reading

NetBackup 5.1 for Oracle System Administrator's Guide (http://support.veritas.com/docs/268099). NetBackup 6.0 for Oracle System Administrator's Guide (http://support.veritas.com/docs/279281). NetBackup 6.5 for Oracle System Administrator's Guide for Windows (http://support.veritas.com/docs/290215). NetBackup 6.5 for Oracle System Administrator's Guide for UNIX and Linux (http://support.veritas.com/docs/290216).

2.0 Operational Characteristics

This section lists background information for NetBackup for ORACLE general operation. Initiating RMAN: The NBU Oracle policy can contain one or more client names and one or more backup scripts to execute. The NBU master is going to use the Automatic schedules in the Oracle policy to determine when the scripts should be executed on the client(s). The NBU scheduler will start one Automatic job for each client in the policy, the jobs can run concurrently. The NBU scheduler will execute each script, in sequence specified, on each client. The all the scripts for one client will run in the same Automatic job. The script(s) start RMAN which gathers the data, sends a user-directed backup request to NBU, and then transfers the data to NBU for storage. If an Automatic schedule and script does not exist in the policy, a process on the client can still initiate RMAN when necessary.

RMAN Requests the Backup: RMAN will connect to the appropriate Oracle instance(s) for the backup. Issue date: 29th July 2009

Best Practices for protecting Oracle RAC with NetBackup Page 2

Hence the script may execute on one host, but the backup may take place on a different host. RMAN will allocate one or more channels per the backup script. RMAN will send one or more backupset pieces on each channel, in sequence. Each channel will send a user-directed backup request to the NBU master for each backupset piece. Each request will become a separate NBU Application Backup job. Hence there can be one NBU Application Backup job queued or active, concurrently, per allocated channel. This will be in addition to an active job if an Automatic schedule initiated the script execution. For the Application Backup jobs, by default the NBU master will select the first policy it finds of type Oracle for this client and use the first schedule of type Application Backup within the policy. The NBU master must be able to match the requesting client name to an Oracle policy and Application Backup schedule or the job will fail. RMAN can SEND the variables NB_ORA_CLIENT and/or NB_ORA_POLICY and/or NB_ORA_SCHED to the NBU master to override the defaults otherwise selected for the job.

Receiving the Data from RMAN: Once the Application Backup job(s) go active, they will connect back to the provided client host to receive the data. Hence the client name that is sent in the user-directed request must bring the data connection back to the requesting host. RMAN will then send the appropriate data on the appropriate channel and it will be transferred to storage.

3.0 RAC backup options

The following sections describe five potential RAC configurations and presents various options for the backup of each one.

3.1 Failover VIP exists, backup is not load balanced

In this configuration, a failover VIP name exists such that the NBU media server can always reach a host which is up to execute the backup script. Further, since load balancing is disabled, RMAN will allocate the channels on a single host, typically the same host where the script executes. The configuration is as follows: The policy should specify the failover VIP name as the client name so that the Automatic schedule executes the backup script on a host that is currently operational. The backup script, or an identical copy, must be accessible to all hosts in the cluster. The clustered file system makes a good location. The backup script should be configured so that RMAN sends to the NBU master (regardless of active host) the same client name to be used to both store the images and for the connect-back from the NBU media server to the client for the data transfer. The VIP name from the policy is ideal since that name is already in the policy, will float to the active instance/host and ensure successful data transfer, and all the backups will be stored under that single client name.
ALLOCATE CHANNEL ... ; SEND 'NB_ORA_CLIENT=<vip_name_from_policy>'; BACKUP ... ;

The NBU master should be configured to know about the relationship between the physical and virtual names. This is required for backups, such as this one, where the user-directed request originates from a hostname that does not match the virtual name Issue date: 29th July 2009

Best Practices for protecting Oracle RAC with NetBackup Page 3

used for NB_ORA_CLIENT. This configuration also makes it possible to perform alternate client/instance restores from either node in the cluster.
cd /usr/openv/netbackup/db/altnames echo "hostname1" >> hostname1 echo "vip_name1" >> hostname1 echo "hostname2" >> hostname1 echo "vip_name2" >> hostname1 echo "vip_name" >> hostname1 cp hostname1 hostname2

If REQUIRED_INTERFACE or another means is being utilized on the client hosts to force NBU to use the IP addresses associated with the VIP names for the outbound userdirected backup requests, then the NBU master configuration must be extended in the reverse direction.
cd /usr/openv/netbackup/db/altnames cp hostname1 vip_name1 cp hostname1 vip_name2

The net result is that the backup script will run on the active host currently hosting the VIP name, RMAN will allocate the channels on that host to perform the backup, the Application Backup jobs will queue to the VIP name, the NBU media server will connect back to the VIP name for the data transfer. The backup images will be stored under the VIP name regardless of which host performed the backup. Restores can take place from either host as long as the restore request is configured to SEND 'NB_ORA_CLIENT=vip_name';

3.2 Failover VIP exists, backup is load balanced

In this configuration, a failover VIP name exists such that the NBU master can always reach an active host to execute the backup script. However, RMAN will allocate the channels on both hosts and the NBU media server must connect-back to the correct host to obtain the data for each request. Because of this, the backup images will be stored under two different client names which also differ from the VIP name used to execute the backup script. The policy should specify the failover VIP name as the client name so that the Automatic schedule executes the backup script on a host that is currently operational. The backup script, or an identical copy, must be accessible to both hosts in the cluster. The clustered file system makes a good location. The backup script must NOT be configured to send a single value for NB_ORA_CLIENT because the NBU media server needs to connect back to the correct host depending on which channel/host originated the user-directed backup request. By default, the backup will use the CLIENT_NAME from the bp.conf file which should be distinct for each host and is typically the physical hostname.

If desired, the persistent parameters in the Oracle configuration can be utilized to bind specific channels to specific hosts.
CONFIGURE CONFIGURE CONFIGURE CONFIGURE CONFIGURE DEVICE TYPE 'SBT_TAPE' PARALLELISM 4; CHANNEL 1 DEVICE TYPE 'SBT_TAPE' CONNECT CHANNEL 2 DEVICE TYPE 'SBT_TAPE' CONNECT CHANNEL 3 DEVICE TYPE 'SBT_TAPE' CONNECT CHANNEL 4 DEVICE TYPE 'SBT_TAPE' CONNECT 'sys/passwd@tns1'; 'sys/passwd@tns2'; 'sys/passwd@tns1'; 'sys/passwd@tns2';

Alternatively, RMAN can be configured to bind specific channels to specific instances and send specific client names on each channel for backup image storage and for connect-back to the requesting host for the data transfer. The failover VIP name should not be used since it will be active on only one of the hosts.
ALLOCATE CHANNEL 1 ... PARMS='ENV=(NB_ORA_CLIENT=hostname1)' CONNECT='sys/passwd@tns1'; ALLOCATE CHANNEL 2 ... PARMS='ENV=(NB_ORA_CLIENT=hostname2)' CONNECT='sys/passwd@tns2';

Best Practices for protecting Oracle RAC with NetBackup Page 4

Issue date: 29th July 2009

ALLOCATE CHANNEL 3 ... PARMS='ENV=(NB_ORA_CLIENT=hostname1)' CONNECT='sys/passwd@tns1'; ALLOCATE CHANNEL 4 ... PARMS='ENV=(NB_ORA_CLIENT=hostname2)' CONNECT='sys/passwd@tns2';

Because the CLIENT_NAME or NB_ORA_CLIENT differs from the VIP name in the policy, the NBU master will not accept the user-directed backup request unless one of the following two configurations is implemented: Configuration A: 1. Coordinate the policy and backup script to handle multiple client names. 2. Add both host names to the policy along with the failover VIP name. 3. Modify the script so that it exits with status 0 if the client name is not the VIP name. Configuration B: 1. Create a second policy to receive the backup requests from RMAN. 2. The policy needs to be of type Oracle. 3. The policy must contain the client/host names used by each host in the backup requests from RMAN. 4. The Application Backup schedule must have an open window to accept the backups. 5. The policy does not need either a backup script or an Automatic schedule. 6. The backup script executed by the first policy should be configured to send the name of the second policy with each user-directed backup request;
ALLOCATE CHANNEL ... PARMS='ENV=(NB_ORA_POLICY=<second_policy_name>)';

or
SEND 'NB_ORA_POLICY=<second_policy_name>';

The NBU master should be configured to know about the relationship between the physical and virtual names to facilitate browsing and restoring images from the other host.
cd /usr/openv/netbackup/db/altnames echo "vip_name" >> hostname1 echo "hostname1" >> hostname1 echo "vip_name1" >> hostname1 echo "hostname2" >> hostname1 echo "vip_name2" >> hostname1 cp hostname1 hostname2If REQUIRED_INTERFACE or another means is being utilized

on the client hosts to force NBU to use the IP addresses associated with the VIP names for the outbound user-directed backup requests, then the NBU master configuration must be extended in the reverse direction.
cd /usr/openv/netbackup/db/altnames cp hostname1 vip_name1 cp hostname1 vip_name2

For Configuration A, the net result is that the NBU scheduler will start three automatic jobs, and each will execute the backup script (two of them on the same host). The two jobs started using the host names will exit immediately with status 0 and not retry. The third job will execute the backup script and RMAN will send the data for backup using the appropriate client name for the host originating each channel. The backup images will be stored under the initiating policy using both host names. For Configuration B, the net result is that the first policy executes the backup script using the failover VIP name. RMAN sends the local hostname with the user-directed request from each host. The second policy stores the backup images using both host names. Either client can initiate a restore. RMAN can be configured with 'SET AUTOLOCATE ON;' to request the backupset pieces from the appropriate instance/host that performed the backup. Alternatively, restores can take place from either host/instance as long as the restore request is configured to include the same client name used at the time the backupset piece was transferred to storage.
SEND 'NB_ORA_CLIENT=backup_hostnameX_or_vip_nameX'

Best Practices for protecting Oracle RAC with NetBackup Page 5

Issue date: 29th July 2009

3.3 VIP does not failover, backup is not load balanced

In this configuration, hostname1 and VIP name1 both allow connections to one host in the cluster and hostname2 and VIP name2 both allow connections to the other host in the cluster. Special configuration will need to ensure that the backup script executes on at least one of the hosts but not on both hosts. Otherwise either a backup will not occur (1 of 1 specified instances is down) or a redundant backup will occur (2 of 2 specified instances are active). For ease of discussion, the term primary refers the instance on which the backup would normally occur and secondary refers to the other instance which could be used for the backup of the primary is down. In addition, because the backup may occur on either host, the backup images have the potential to be stored under both client names depending on the host which is active at the time of the backup. The configuration is as follows: The policy should specify both hosts, either hostname1 & hostname2 or VIP name1 & VIP name2 to ensure that the backup is attempted on a host which is currently operational. The backup script must be accessible to both hosts in the cluster, the clustered file system makes a good location. The backup script should be customized so that it will perform the backup on only one of the two instance if they are both active. If executed on the primary, then start RMAN and perform the backup. If executed on the secondary and the primary is up, then exit with status 0 so that the NBU scheduler does not retry this client. If executed on the secondary and the primary is down, then start RMAN and perform the backup. The script customization could be built around a ping to the primary or even a query of the database to see if the other instance is open and able to perform the backup, e.g.
$ select INST_ID,STATUS,STARTUP_TIME, HOST_NAME from gv$instance; INST_ID STATUS STARTUP_T HOST_NAM ---------- ------------ --------- --------1 OPEN 13-JAN-06 CAMS1 2 OPEN 13-JAN-06 CAMS2

The user-directed backup request must utilize a client name which allows the NBU media server to connect back to the correct host for the data transfer. By default, the backup will use the CLIENT_NAME from the bp.conf file which will be distinct for each host. Alternatively, RMAN can be configured to send the appropriate hostname or VIP name from the policy. If executing on hostname1:
SEND 'NB_ORA_CLIENT=<hostname1_or_vip_name1_per_policy_client)';

If executing on hostname2:
SEND 'NB_ORA_CLIENT=<hostname2_or_vip_name2_per_policy_client)';

The NBU master should be configured to know about the relationship between the physical and virtual names. This is required for backups, such as this one, where the user-directed request originates from a hostname that may not match the virtual name used for NB_ORA_CLIENT. This configuration also makes it possible to perform alternate client/instance restores.
cd /usr/openv/netbackup/db/altnames echo "hostname1" >> hostname1 echo "vip_name1" >> hostname1 echo "hostname2" >> hostname1 echo "vip_name2" >> hostname1 cp hostname1 hostname2

If REQUIRED_INTERFACE or another means is being utilized on the client hosts to force NBU to use the IP addresses associated with the VIP names for the outbound userdirected backup requests, then the NBU master configuration must be extended in the reverse direction. Issue date: 29th July 2009

Best Practices for protecting Oracle RAC with NetBackup Page 6

cd /usr/openv/netbackup/db/altnames cp hostname1 vip_name1 cp hostname1 vip_name2

The net result is that the backup script will run on all hosts that are currently active but will only start RMAN on one host. The user-directed backup requests from the host running RMAN will include the hostname/VIP name of itself so that the connect-back for data is successful and the backup image will be stored under that name. Restores can take place from either host or instance as long as the restore request is configured to specify the same client name as used at the time the backupset piece was transferred to storage.
SEND 'NB_ORA_CLIENT=backup_hostnameX_or_vip_nameX';

3.4 VIP does not failover, backup is load balanced, 1 policy with custom script
This configuration encounters the same hurdles as Option #2 and Option #3. Because a failover VIP name does not exist, the NBU scheduler must attempt to execute the backup script on both hosts and the script must start RMAN on only one of the hosts. Because RMAN may allocate channels on both instances, the user-directed requests must specify host specific names so that the connect-back from the NBU media server is able to retrieve the data from the correct host. The policy should specify both hosts, either hostname1/hostname2 or VIP name1/VIP name2 to ensure that the backup is attempted on a host which is currently operational. The backup script must be accessible to both hosts in the cluster. The clustered file system makes a good location. The backup script should be customized so that it will perform the backup on only one of the two instance if they are both active. If executed on the primary, then start RMAN and perform the backup. If executed on the secondary and the primary is up, then exit with status 0 so that the NBU scheduler does not retry this client. If executed on the secondary and the primary is down, then start RMAN and perform the backup. The script customization could be built around a tnsping to the primary or even a query of the database to see if the other instance is open and able to perform the backup, e.g.
$ select INST_ID,STATUS,STARTUP_TIME, HOST_NAME from gv$instance; INST_ID STATUS STARTUP_T HOST_NAM ---------- ------------ --------- --------1 OPEN 13-JAN-06 CAMS1 2 OPEN 13-JAN-06 CAMS2

The backup script must NOT be configured to send a single value for NB_ORA_CLIENT because the NBU media server needs to connect back to the correct host depending on which channel/host originated the user-directed backup request. By default, the backup will use the CLIENT_NAME from the bp.conf file which will be distinct for each host. Alternatively, Oracle/RMAN can be configured to bind specific channels to specific instances/hosts. Then each channel can specify the client name to be used by the NBU media server to store the backup image and for connect-back to the requesting host. The name specified must match the host specific name in the policy. ...Oracle persistent parameter configuration for each channel from this host...
ALLOCATE CHANNEL ... PARMS='ENV=(NB_ORA_CLIENT=<host_specific_name_from_policy)';

The NBU master should be configured to know about the relationship between the physical and virtual names to facilitate browsing and restoring images from the other host.
cd /usr/openv/netbackup/db/altnames echo "hostname1" >> hostname1 echo "vip_name1" >> hostname1 echo "hostname2" >> hostname1

Best Practices for protecting Oracle RAC with NetBackup Page 7

Issue date: 29th July 2009

echo "vip_name2" >> hostname1 cp hostname1 hostname2

If REQUIRED_INTERFACE or another means is being utilized on the client hosts to force NBU to use the IP addresses associated with the VIP names for the outbound userdirected backup requests, then the NBU master configuration must be extended in the reverse direction.
cd /usr/openv/netbackup/db/altnames cp hostname1 vip_name1 cp hostname1 vip_name2

The net result is that the backup script will run on all of the currently active hosts but will only start RMAN on one host. RMAN will allocate channels across the hosts for load balancing. The userdirected backup requests will include a CLIENT_NAME/NB_ORA_CLIENT specific to the requesting host and which matches the policy. The connect-back for data transfer and the backup image will be stored under that name. Either client can initiate a restore and RMAN can be configured with 'SET AUTOLOCATE ON;' to request the backupset pieces from the appropriate instance/host that performed the backup. Alternatively, restores can take place from either host or instance as long as the restore request is configured to specify the same client name as used at the time of the backup.
SEND 'NB_ORA_CLIENT=backup_hostnameX_or_VIP nameX';

3.5 VIP does not failover, backup is load balanced, separate automatic policies
Some implementations of RAC (Linux/Windows) do not include a failover VIP. On sites which do not need a robust backup script which determines the active instance in real-time, the following ad-hoc configuration can be utilized to manually initiate a backup via the secondary host when the primary host is down. Create one Oracle policy with an Application Backup schedule to receive the backup images from both hosts. Configure the hostname/VIP name of both hosts in the policy. Do not configure an Automatic schedule or a Backup Selection (script). Create a second Oracle policy to execute the backup script on the primary host. Configure the hostname/VIP name of the primary host in the policy. Configure the pathname of the backup script in the policy. Create an Automatic schedule WITH AN OPEN WINDOW in the policy. Optionally create a third Oracle policy to execute the backup script on the secondary host. Configure the hostname/VIP name of the secondary host in the policy. Configure the pathname of the backup script in the policy. Create an Automatic schedule WITHOUT AN OPEN WINDOW in the policy.

The backup script must be accessible to both hosts in the cluster, the clustered file system makes a good location. The user-directed backup request originating from the RMAN clients must utilize a client name which allows the NBU media server to connect back to the correct host for the data transfer. By default, the backup will use the CLIENT_NAME from the bp.conf file which will be distinct for each host. Alternatively, RMAN can be configured to send the appropriate hostname or VIP name from the policy. If executing on hostname1:
SEND 'NB_ORA_CLIENT=<hostname1_or_vip_name1_per_policy_client)';

If executing on hostname2:
SEND 'NB_ORA_CLIENT=<hostname2_or_vip_name2_per_policy_client)';

Best Practices for protecting Oracle RAC with NetBackup Page 8

Issue date: 29th July 2009

The NBU master should be configured to know about the relationship between the physical and virtual names. This is required for backups, such as this one, where the user-directed request originates from a hostname that may not match the virtual name used for NB_ORA_CLIENT. This configuration also makes it possible to perform alternate client/instance restores.
cd /usr/openv/netbackup/db/altnames echo "hostname1" >> hostname1 echo "vip_name1" >> hostname1 echo "hostname2" >> hostname1 echo "vip_name2" >> hostname1 cp hostname1 hostname2

If REQUIRED_INTERFACE or another means is being utilized on the client hosts to force NBU to use the IP addresses associated with the VIP names for the outbound user-directed backup requests, then the NBU master configuration must be extended in the reverse direction.
cd /usr/openv/netbackup/db/altnames cp hostname1 vip_name1 cp hostname1 vip_name2

The net result is that the second policy will execute the backup script on the primary host when scheduled and the RMAN script will send back the appropriate CLIENT_NAME or NB_ORA_CLIENT specific to that host. In the event the primary is down, the policy for the secondary can be initiated manually from the NBU master and perform a similar backup. Restores can take place from either host or instance as long as either 'SET AUTOLOCATE ON'; is enabled for the instance or the restore request is configured to specify the same client name as used at the time the backupset piece was transferred to storage.
SEND 'NB_ORA_CLIENT=backup_hostnameX_or_vip_nameX';

4.0 Catalog considerations

If virtual names are available to be used with NB_ORA_CLIENT, then the following technique can be used to simplify catalog maintenance and restores. Before performing the initial backup of the RAC cluster, create a single directory in the NetBackup image database to hold the backup images for all the client hosts and then create symbolic links to that directory for each of the virtual names.
cd /usr/openv/netbackup/db/images mkdir racname ln -s racname vip_name1 ln -s racname vip_name2

In this configuration, the backup from each host via a virtual name will be placed into the common directory. Subsequent restore and maintenance functions can than use either virtual name to access all the images. In addition, if altnames is configured appropriately, the common directory name can be used as a client name for alternate client restores with or without AUTOLOCATE enabled.
cd /usr/openv/netbackup/db/altnames echo "hostname1" >> racname echo "vip_name1" >> racname echo "hostname2" >> racname echo "vip_name2" >> racname SEND 'NB_ORA_CLIENT=racname';

NOTE: The technique above is recommended for use only when virtual names are available. If utilized with physical names, then the backup images from standard backups using the physical names of the two hosts will be commingled within a single image directory with the Oracle backup images, resulting in two potential problems. First, either host may restore the files that normally reside only on the other host. Second, if the same file exists on both hosts but with different content, care must be taken at restore time to select the correct backup image from which to affect the restore.

Best Practices for protecting Oracle RAC with NetBackup Page 9

Issue date: 29th July 2009

NOTE: A similar configuration is possible with NTFS on a Windows master server (Windows 2000 and above) using junctions in place of symbolic links. See this document for details. http://www.microsoft.com/technet/sysinternals/FileAndDisk/Junction.mspx

5.0 Steps to configure load balanced backups

1) Determine if the RAC is configured to load balance the backups. The Oracle persistent parameters will show that parallelism is greater than 1 for backups using the SBT API. This example is for a two node cluster which will allocate two channels for each host.
CONFIGURE DEVICE TYPE 'SBT_TAPE' PARALLELISM 4;

If the backups are not load balanced, then use the recommended configuration in Appendix A of the NetBackup for Oracle System Administrator's Guide. It is not necessary to follow the remaining steps below. 2) Configure the master/media server to accept backup job requests from all of the instances in the RAC cluster. First, make sure the network allows connectivity to the VIP names used by all of the instances in the RAC. The VIP names must be reachable from the master and media servers. Next, create an policy that will receive the backup job requests from the instances in the cluster. Select Oracle as the policy type. Specify the VIP names (preferred) or client names as the policy clients. See Note #1 for details. If desired, narrow the backup window in the schedule of type Application backup to exclude those hours during which backup requests from the RAC cluster should not be allowed. Automatic schedules and Backup Selections may not be needed. See Step 7 for further details.

3) Create an RMAN backup script to perform the backup. Several samples, that are not RAC specific, were installed on the client host along with the NetBackup Oracle extension. The samples can be modified for use with RAC. NOTE: The NetBackup for Oracle Template Wizard is intended for simple backup and restore scenarios. Hence it does not support load balanced RAC configurations at this time due to E-Track 1086611 unless they are client initiated. 4) Make the backup script accessible to all hosts which could be used for failover. An ideal location is the Clustered File System (CFS) or other file system mounted by all hosts in the cluster. That way there is only one copy of the script and it is accessible to all hosts. Alternatively, copy the script to the same pathname on each host. In the future, be sure to propagate any changes to all copies.

5) Coordinate the RMAN and NetBackup configurations to ensure that the backup job that will be started on each host will use a client name that will allow the master and media servers to connect back to that same host to receive the backup image. The values for NB_ORA_CLIENT or defaulting CLIENT_NAME must match the client names placed into the policy during Step 2. Option 1 Allow the backup process on each host to use the local CLIENT_NAME found in the bp.conf. This is a simple solution, but results in the backup images for the RAC cluster

Best Practices for protecting Oracle RAC with NetBackup Page 10

Issue date: 29th July 2009

being stored under three client names and co-mingled with the standard backups that were taken using those same client names. Option 2 Configure the Oracle persistent parameters to bind each channel to a specific instance and matching client name.
CONFIGURE CHANNEL 1 ... CONNECT "ENV=(NB_ORA_CLIENT=vip1)"; CONFIGURE CHANNEL 2 ... CONNECT "ENV=(NB_ORA_CLIENT=vip2)"; CONFIGURE CHANNEL 3 ... CONNECT "ENV=(NB_ORA_CLIENT=vip1)"; CONFIGURE CHANNEL 4 ... CONNECT "ENV=(NB_ORA_CLIENT=vip2)"; 'sys/passwd@vip1' PARMS 'sys/passwd@vip2' PARMS 'sys/passwd@vip1' PARMS 'sys/passwd@vip2' PARMS

Option 3 Configure RMAN to bind each channel to a specific instance and matching client name at runtime, for example.
ALLOCATE CHANNEL 1 ... PARMS='ENV=(NB_ORA_CLIENT=vip1)' CONNECT='sys/passwd@vip1'; ALLOCATE CHANNEL 2 ... PARMS='ENV=(NB_ORA_CLIENT=vip2)' CONNECT='sys/passwd@vip2'; ALLOCATE CHANNEL 3 ... PARMS='ENV=(NB_ORA_CLIENT=vip1)' CONNECT='sys/passwd@vip1'; ALLOCATE CHANNEL 4 ... PARMS='ENV=(NB_ORA_CLIENT=vip2)' CONNECT='sys/passwd@vip2';

Note: Configuring RMAN to send a single NB_ORA_CLIENT value for all channels on all hosts will cause application backup jobs to fail with status 54 and the overall job to fail with status 6. This is because the media server processes for all of the jobs will connect back to the one NB_ORA_CLIENT host to receive the backup images, subsequently the jobs that were requested by the other host will timeout and fail. 6) Make sure that the backup requests from each host are queued to the policy configured in Step 2. Option 1 Configure the Oracle persistent parameters to bind each channel to a specific policy.
CONFIGURE CHANNEL 1 ... PARMS "ENV=(NB_ORA_CLIENT=vip1,NB_ORA_POLICY=OracleBackup)"; CONFIGURE CHANNEL 2 ... PARMS "ENV=(NB_ORA_CLIENT=vip2,NB_ORA_POLICY=OracleBackup)"; CONFIGURE CHANNEL 3 ... PARMS "ENV=(NB_ORA_CLIENT=vip1,NB_ORA_POLICY=OracleBackup)"; CONFIGURE CHANNEL 4 ... PARMS "ENV=(NB_ORA_CLIENT=vip2,NB_ORA_POLICY=OracleBackup)";

Option 2 Configure RMAN to bind each channel to a specific policy at runtime.

ALLOCATE CHANNEL 1 ... PARMS='ENV=(NB_ORA_CLIENT=vip1,NB_ORA_POLICY=OracleBackup)'; ALLOCATE CHANNEL 2 ... PARMS='ENV=(NB_ORA_CLIENT=vip2,NB_ORA_POLICY=OracleBackup)'; ALLOCATE CHANNEL 3 ... PARMS='ENV=(NB_ORA_CLIENT=vip1,NB_ORA_POLICY=OracleBackup)'; ALLOCATE CHANNEL 4 ... PARMS='ENV=(NB_ORA_CLIENT=vip2,NB_ORA_POLICY=OracleBackup)';

Option 3 Configure RMAN to send the policy name after the channels have been allocated.
SEND 'NB_ORA_POLICY=OracleBackup';

Best Practices for protecting Oracle RAC with NetBackup Page 11

Issue date: 29th July 2009

If the policy is not specified explicitly, NetBackup will choose the first available Oracle policy for the specified client, which may not be appropriate. 7) Configure the environment to execute the backup script on at least one host that is online but also prevent the redundant execution of RMAN on multiple hosts. The goal is to ensure that a backup gets taken regardless of how many instances are up, and to also ensure that a redundant backup does not also occur. Option 1 Does a virtual fail-over hostname exists, and will it always be active on one host in the cluster? If so, that virtual hostname provides a good mechanism to launch the backup script. Make sure the virtual hostname is reachable from the master and media servers and then configure as follows: Select Oracle as the policy type. Specify the virtual hostname as the policy client. Specify the pathname to the backup script in the Backup Selections and confirm Step 4 was completed correctly. Create one or more Automatic schedules to execute the backup script when desired. It is not necessary to configure an Application Backup schedule. It is not necessary to delete the default Application Backup schedule as it should not be used. It is essential that the RMAN backup script set NB_ORA_POLICY correctly per Step 6. Otherwise this policy may be randomly selected, and it will only accept backup jobs from one host in the cluster resulting in failure on the other host(s).

Option 2 Execute the backup script on all hosts, but customize the script to only perform the backup on one host. The script would need to be aware of a relative ranking between the hosts (primary/secondary) and also aware of which instances are online. If executed on the primary, start RMAN and perform the backup. If executed on the secondary and the primary is up, exit with status 0 so that the NetBackup scheduler does not retry this client. If executed on the secondary and the primary is down, start RMAN and perform the backup. The script customization could be built around a ping to the primary or even a query of the database to see if the other instance is open and able to perform the backup, e.g.
$ select INST_ID, STATUS, STARTUP_TIME, HOST_NAME from gv$instance; INST_ID ---------1 2 STATUS -----------OPEN OPEN STARTUP_T --------13-JAN-06 13-JAN-06 HOST_NAM --------vip1 vip2

This type of customization would allow the policy configured in Step 2 to also be used to initiate the backup. Simply add a Backup Selection list which points to the backup script and create the desired automatic backup schedules. Option 3 Configure for a manual failover by creating two or more policies to execute the backup script. All of the policies should be of type Oracle and have a Backup Selection that lists the pathname to the backup script.

Best Practices for protecting Oracle RAC with NetBackup Page 12

Issue date: 29th July 2009

One policy should designate the preferred client and be configured with the desired automatic schedules that will be used to initiate the backup script. This policy should be made active. The other policy or policies should contain the non-preferred clients. These policies can either be active and without an active window or they can be inactive but with an open window. The former can be used to initiate an immediate backup from the associated client whenever desired (anticipating that the preferred client will be back online shortly). The latter can be activated to take over backup duties until the preferred client/policy is restored to operation (anticipating that the preferred client will be offline for an extended period of time).

8) Configure the master server to allow each host to have access to the backup images for all of the instances. This access is granted based on the peernames by which the master server knows the client hosts. The master server obtains the peernames from the source IP address that the client hosts use when sending the backup/list/restore request.
cd /usr/openv/netbackup/db/altnames echo "vip1" >> peername1 echo "vip2" >> peername1 echo "peername1" >> peername1 echo "peername2" >> peername1 cp peername1 peername2

Please note that static routes or the use of REQUIRED_INTERFACE on the client hosts may cause the traffic from the client hosts to the NBU servers to originate from an unexpected IP address. The hostnames to which those IPs resolve are the peernames to use. 9) Image storage, catalog maintenance, and restore operations The above configuration will have RAC backup images stored under multiple client names. This is not of concern if Oracle is configured to autolocate when restores or catalog reconciliation occurs between NetBackup and Oracle.
SET AUTOLOCATE ON;

However, this can present challenges when performing alternate client restores or restores utilizing a different number of instances. In that case, it is often advantageous to have all of the RAC backup images stored under a single client name. For adhoc operations this can be accomplished by creating a new netbackup/db/image directory using a temporary virtual name to house a temporary copy of the backup image files which are then copied from the original client directories. A more flexible permanent solution is to create a linked image directory structure ahead of time so that the standard backups are distinct from the Oracle backups and all the RAC images are aggregated into a single directory but accessible from any instance.
cd /usr/openv/netbackup/db/images # Create the image directories for the standard backups of both hosts. mkdir client1 client2 # Create an image directory to house all the RAC images. mkdir rac # Create links to redirect the vips to the RAC directory. ln -s rac vip1 ln -s rac vip2

The same structure and functionality is available in NTFS, since Windows 2000, via junctions. http://www.microsoft.com/technet/sysinternals/FileAndDisk/Junction.mspx Once the above image db structure exists, catalog maintenance and alternate client restores can occur using NB_ORA_CLIENT=rac. However, the requesting peernames must be allowed access to the new client name. Best Practices for protecting Oracle RAC with NetBackup Page 13 Issue date: 29th July 2009

cd /usr/openv/netbackup/db/altnames echo "rac" >> peername1 echo "rac" >> peername2 echo "rac" >> peername_for_alternate_client

Best Practices for protecting Oracle RAC with NetBackup Page 14

Issue date: 29th July 2009