Table of Contents

Rac11gR1OnLinux.............................................................................................................................................1 1. *Introduction.......................................................................................................................................1 1.1. *What you need to know....................................................................................................1 1.1.1. Software required for install...............................................................................1 1.1.1.1. 32-Bit Required RPM's .......................................................................2 1.1.1.2. 64-Bit Required RPM's .......................................................................2 1.1.2. Processor Model........................................................................................................................................3 . 1.2. *Installation steps ..........................................................................................................................................3 1.3. *Schematic....................................................................................................................................................4 1.3.1. Hardware/software configuration BEFORE Oracle software install............................................4 1.3.2. Hardware/software configuration AFTER Oracle software install..............................................4 1.4. *Installation Method.....................................................................................................................................5 2. *Prepare the cluster nodes for Oracle RAC.....................................................................................................6 2.1. *User Accounts................................................................................................................................6 2.1.1. *Creating the OSDBA (DBA) Group..............................................................................6 2.1.2. *Creating an OSOPER Group (Optional) ........................................................................6 2.1.3. *Creating an OSASM Group...........................................................................................7 2.1.4. *Creating an OINSTALL Group.....................................................................................7 2.1.5. *Determining Whether an Oracle Software Owner User Exists.....................................7 2.1.6. *Creating an Oracle Software Owner User.....................................................................8 2.1.7. *Modifying an Oracle Software Owner User..................................................................8 2.1.8. *Configuring SSH on Cluster Member Nodes................................................................8 2.1.9. *Checking Existing SSH Configuration on the System..................................................8 2.1.10. *To configure SSH, Create RSA Keys On Each Node using the following steps:.......9 2.1.11. *Add All Keys to a Common authorized_keys File......................................................9 2.1.12. *Enabling SSH User Equivalency on Cluster Member Nodes....................................10 2.1.13. *Setting Display and X11 Forwarding Configuration.................................................11 2.2. *Networking...............................................................................................................................................12 2.2.1. *Network Ping Tests...................................................................................................................13 2.2.2. *Network Adapter configuration................................................................................................14 2.3. *Time Sync.................................................................................................................................................14 2.4. *Configuring Kernel Parameters................................................................................................................14 2.5. *Setting Shell Limits for the oracle User....................................................................................................15 2.6. *Configuration of the Hangcheck-timer Module........................................................................................16 2.6.1. *Hangcheck-timer Module verification procedure.....................................................................16 2.7. *Platform Specific Setup............................................................................................................................17 2.8. *Stage the Oracle Software.........................................................................................................................18 2.9. *Cluster Verification Utility stage check....................................................................................................18 3. *Prepare the shared storage for Oracle RAC.................................................................................................20 3.1. Example of Configuring Block Device Storage for Oracle Clusterware .......................................21 3.1.1. Example of Creating a Udev Permissions File for Oracle Clusterware........................21 3.1.2. Platform Specific Settings.............................................................................................22 4. Oracle Clusterware Installation and Configuration.......................................................................................22 4.1. CVU Pre Oracle Clusterware install check ....................................................................................22 4.2. Oracle Clusterware Install.............................................................................................................26 . 4.2.0.1. Verify cluster resources are online.............................................................................41 4.2.0.2. Use CVU to verify the Oracle Clusterware install........................................41 5. Oracle Clusterware patching..........................................................................................................................43 5.1. Oracle Clusterware patching complete..........................................................................................50 6. Oracle ASM Home Software Install..............................................................................................................50 6.1. CVU check before creating ASM instance....................................................................................50 6.2. Oracle ASM Home Software Install..............................................................................................56 6.2.1. Oracle ASM software home has been installed.............................................................64 7. Oracle ASM Software Home Patching..........................................................................................................64 7.1. Start the OUI for Oracle ASM Software Home Patching..............................................................64 i

Table of Contents
Rac11gR1OnLinux 8. Oracle RAC Database Listener Creation.......................................................................................................71 8.1. Create Node specific network listeners ..........................................................................................71 8.2. Verify the Listener resources are online........................................................................................76 8.3. Completed Node specific network listeners creation .....................................................................76 9. Oracle ASM Instance and diskgroup Creation..............................................................................................77 9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups using the Database Assistant................................................................................................................................................77 9.2. Verify ASM instances are online...................................................................................................91 9.3. Completed creation of ASM Instance and add of the +DATA and +FLASH diskgroups............92 10. Oracle RAC Database Home Software Install.............................................................................................92 10.1. CVU check -pre Database Instance Install..................................................................................92 10.2. Oracle RAC Database Home Software Install.............................................................................97 10.2.1. Oracle RAC Database software home has been installed..........................................106 11. Oracle RAC Software Home Patching......................................................................................................106 11.1. Using the installer to patch the Oracle RAC Software Home...................................................106 12. Oracle RAC Database Creation.................................................................................................................113 12.1. CVU Database Verification Check............................................................................................113 12.2. use dbca to create the RAC database.........................................................................................115 12.3. Verify all database resource are online......................................................................................133 12.4. RAC database creation is complete...........................................................................................134

ii

Rac11gR1OnLinux

1. *Introduction
1.1. *What you need to know
For those of you starting an Oracle RAC install for the first time, all this will be new. For those of you with Oracle 10g RAC on Linux experience, there are just a few changes. Disk requirements for Oracle Clusterware devices The storage requirement for the OCR & vote devices used by Oracle Clusterware has been increased to 280 MB (256 MB, plus an additional buffer to allow for fdisk variation in space allocation). Better support for IP address The previous release had some restrictions that could be easily worked around, with regard to IP addresses from the non-routable range. These restrictions have been removed in the Oracle Clusterware 11g version and the workarounds are no longer required.

1.1.1. Software required for install

Linux x86 Base Oracle 11gR1 DVD Linux x86-64 (AMD64/EM64T) Base Oracle 11gR1 DVD Oracle Enterprise Linux: as part of the Oracle Unbreakable Linux Program, Oracle offers for download or on CD: Oracle Enterprise Linux 4 fully compatible with Red Hat Enterprise Linux 4 AS/ES Oracle Enterprise Linux 5 fully compatible with Red Hat Linux 5 Server and Advanced Platform. If you install the Oracle Validated Configuration RPM, then it sets and verifies system parameters based on recommendations from the Oracle Validated Configurations program, and installs any additional packages needed for installing Oracle Clusterware and Oracle Database. It also updates sysctl.conf settings, system startup parameters, user limits, and driver parameters to values that testing shows will provide better performance. Note: Additional information on the Oracle Validated Configurations can be found at the following URL: http://www.oracle.com/technology/tech/linux/validated-configurations/index.html This release is certified for the following distributions: Oracle Enterprise Linux 4 RedHat Enterprise Linux 4 AS/ES Oracle Enterprise Linux 5 Red Hat Linux 5 Server and Advanced Platform. SLES-10 Asianux Server 3 Asianux 2.0 SP2 Do not mix Operating Systems between nodes in a cluster. For example, do not configure a cluster with Enterprise Linux 4 on one node and Enterprise Linux 5 on another node. Rac11gR1OnLinux 1

Install your Linux operating system with the default software packages (RPMs). This installation includes most of the required packages and helps you limit manual checks of package dependencies. Do not customize RPMs during installation. Oracle continues to work with Linux vendors to certify Oracle products running on Red Hat Enterprise Linux SUSE Linux Enterprise Server Asianux For additional information on Linux Operating System support refer to Metalink Note 266043.1.

1.1.1.1. 32-Bit Required RPM's

The following packages (or later versions) must be installed: Asianux 2, Enterprise Linux 4 Red Hat Enterprise Linux 4 binutils-2.15.92.0.2-18 compat-libstdc++-33-3.2.3-47.3 elfutils-libelf-0.97-5 elfutils-libelf-devel-0.97-5 gcc-3.4.5-2 gcc-c++-3.4.5-2 glibc-2.3.4-2.19 glibc-common-2.3.4-2.19 glibc-devel-2.3.4-2.19 glibc-headers-2.3.4-2.19 libaio-devel-0.3.105-2 libaio-0.3.105-2 libgcc-3.4.5 libstdc++-3.4.5-2 libstdc++-devel-3.4.5-2 make-3.80-5 sysstat-5.0.5 unixODBC-2.2.11 unixODBC-devel-2.2.11 Asianux 3, Enterprise Linux 5, Red Hat Enterprise Linux 5 binutils-2.17.50.0.6-2.el5 compat-libstdc++-33-3.2.3-61 elfutils-libelf-0.125 elfutils-libelf-devel-0.125 gcc-4.1.1-52 gcc-c++-4.1.1-52 glibc-2.5-12 glibc-common-2.5-12 glibc-devel-2.5-12 glibc-headers-2.3.4-2 libaio-devel-0.3.106 libaio-0.3.106 libgcc-4.1.1-52 libstdc++-4.1.1 libstdc++-devel-4.1.1 make-3.81-1.1 sysstat-7.0.0 unixODBC-2.2.11 unixODBC-devel-2.2.11 SUSE 10 Packages binutils-2.16.91.0.5 compat-libstdc++-5.0.7 libelf-0.8.5 gcc-4.1.0 glibc-2.4-31.2 glibc-devel-2.4-31.2 libaio-devel-0.3.104 libaio-0.3.104 libgcc-4.1.0 libstdc++-4.1.0 libstdc++-devel-4.1.0 make-3.80 sysstat-6.0.2 unixODBC-2.2.11 unixODBC-devel-2.2.11 ksh-93r-12.9

1.1.1.2. 64-Bit Required RPM's

The following packages (or later versions) must be installed: Asianux 2, Enterprise Linux 4 Red Hat Enterprise Linux 4 binutils-2.15.92.0.2 compat-libstdc++-33-3.2.3 compat-libstdc++-33-3.2.3 (32 bit) elfutils-libelf-0.97 elfutils-libelf-devel-0.97 gcc-3.4.5 Asianux 3, Enterprise Linux 5 Red Hat Enterprise Linux 5 binutils-2.17.50.0.6 compat-libstdc++-33-3.2.3 compat-libstdc++-33-3.2.3 (32 bit) elfutils-libelf-0.125 elfutils-libelf-devel-0.125 gcc-4.1.1 SUSE 10 Packages binutils-2.16.91.0.5 compat-libstdc++-5.0.7-22.2 libelf-0.8.5 gcc-4.1.0

1.1.1. Software required for install

gcc-c++-3.4.5 glibc-2.3.4-2.19 glibc-2.3.4-2.19 (32 bit) glibc-common-2.3.4 glibc-devel-2.3.4 glibc-devel-2.3.4 (32-bit) libaio-0.3.105 libaio-0.3.105 (32 bit) libaio-devel-0.3.105 libgcc-3.4.5 libgcc-3.4.5 (32-bit) libstdc++-3.4.5 libstdc++-3.4.5 (32 bit) libstdc++-devel 3.4.5 make-3.80 sysstat-5.0.5

gcc-c++-4.1.1 glibc-2.5-12 glibc-2.5-12 (32 bit) glibc-common-2.5 glibc-devel-2.5 glibc-devel-2.5-12 (32 bit) libaio-0.3.106 libaio-0.3.106 (32 bit) libaio-devel-0.3.106 libgcc-4.1.1 libgcc-4.1.1 (32 bit) libstdc++-4.1.1 libstdc++-4.1.1 (32 bit) libstdc++-devel 4.1.1 make-3.81 sysstat-7.0.0

gcc-c++-4.1.0 glibc-2.4-31.2 glibc-32bit-2.4-31.2 (32 bit) glibc-devel-2.4 glibc-devel-32bit-2.4 (32 bit) libaio-0.3.104 libaio-32bit-0.3.104 (32 bit) libgcc-4.1.0

libstdc++-4.1.0 libstdc++-devel-4.1.0 make-3.80 sysstat-6.0.2

To determine whether the required packages are installed, enter commands similar to the following:
# rpm -q package_name

If a package is not installed, then install it.

1.1.2. Processor Model

This paper covers both 32bit and 64bit install. Please note that The OS Version of Oracle you install must match the OS Chip Version. Lunix x86 32bit - you must install the 32bit versions of the Oracle software. Linux x86-64 (AMD64/EM64T)- you must install the 64bit X64 versions of the Oracle software. Lunix 64bit Itanium IA64 - NOT AVAILABLE - When available you must install the 64bit Itanium IA64 versions of the Oracle software.

1.2. *Installation steps

To summarise the install steps are: Preparation Pre-reqs. to make sure the cluster is setup OK. Stage all the software on one node, typically Node1 Establish Oracle Clusterware Install the Oracle Clusterware (using the push mechanism to install on the other nodes in the cluster) Patch the Clusterware to the latest patchset Establish ASM Install an Oracle Software Home for ASM Patch the ASM Home to the latest patchset Create the Listeners Create the ASM Instances and initial Disk Group Establish RAC Database 1.1.1.2. 64-Bit Required RPM's 3

 Install an Oracle Software Home for RAC Database Patch the RAC Database Home to the latest patchset Create the RAC Database Instances

1.3. *Schematic
The following is a schematic of the software & hardware layout of a 2node RAC cluster. As explained in this document the actual number of LUNs required will vary depending on your mirroring requirements.

1.3.1. Hardware/software configuration BEFORE Oracle software install

1.3.2. Hardware/software configuration AFTER Oracle software install

1.2. *Installation steps

Notice that there are a number of LUNs present to both machines from the shared storage. 5 of 290MB LUNs 2 are used for the OCR devices (/dev/sdb1,/dev/sdc1) 3 are used for the vote devices (/dev/sdb5,/dev/sdc5, /dev/sdd5) 16 of 13358MB LUNs 8 are used for the +DATA ASM diskgroup (/dev/sdc[8-15]) 8 are used for the +FLASH diskgroup. (/dev/sdd[8-15]) Also notice there is a dedicated switch on the cluster interconnect.

1.4. *Installation Method

This document details one method of installing a 2-node Oracle 11gR1 RAC cluster on Linux: Block Device: The Oracle Clusterware, ASM & Oracle Database Homes binaries are established on the local disk of each of the RAC nodes. The files required by Oracle Clusterware are on Block devices. The database data-files are on ASM. The local Homes for Oracle Clusterware, ASM and RAC take up a total of 6.5GB on each nodes local disk filesystem. It should be noted that there are other installation configurations. This document does not cover these methods: OCFS: The Oracle Cluster, ASM & Oracle Database Homes are established on the local disk of each of the RAC nodes. The files required by Oracle Clusterware are on OCFS. The database data-files are also on OCFS. 1.3.2. Hardware/software configuration AFTER Oracle software install 5

 NFS: The Oracle Cluster, ASM & Oracle Database Homes are established on the local disk of each of the RAC nodes. The files required by Oracle Clusterware are on Block devices. The database data-files are also on NFS File Server. There are other possible combinations although Oracle recommends one of the above methods. You cannot place the Clusterware devices on NFS File Server. NFS File Server is not covered here

2. *Prepare the cluster nodes for Oracle RAC

The guides include hidden sections, use the and image for each section to show/hide the section or you can Expand all or Collapse all by clicking these buttons. This is implemented using the Twisty Plugin which requires Java Script to be enabled on your browser. This installation routine presumes that you have a 2-node Linux cluster. There are a number of items that require checking before the install commences. Getting this bit right will enhance your install experience. 1. User Accounts 2. Networking 3. Time Sync 4. Stage the Oracle Software 5. Run CVU It is essential that these items are checked and are configured correctly before the install commences.

2.1. *User Accounts

2.1.1. *Creating the OSDBA (DBA) Group
You must create an OSDBA group in the following circumstances: An OSDBA group does not exist, for example, if this is the first installation of Oracle Database software on the system An OSDBA group exists, but you want to give a different group of operating system users database administrative privileges in a new Oracle installation To determine whether the OSDBA group exists, enter the following command:
# grep OSDBA_group_name /etc/group

If the OSDBA group does not exist or if you require a new OSDBA group, then create it as follows. In the following command, use the group name dba unless a group with that name already exists.
# /usr/sbin/groupadd dba

Note: The default OSDBA group name is dba.

2.1.2. *Creating an OSOPER Group (Optional)

Create an OSOPER group only if you want to identify a group of operating system users with a limited set of database administrative privileges (SYSOPER operator privileges). For most installations, it is sufficient to create only the OSDBA group. If you want to use an OSOPER group, then you must create it in the following circumstances: 1.4. *Installation Method 6

 If an OSOPER group does not exist, for example, if this is the first installation of Oracle Database software on the system If an OSOPER group exists, but you want to give a different group of operating system users database operator privileges in a new Oracle installation If you require a new OSOPER group, then create it as follows. In the following command, use the group name oper unless a group with that name already exists.
# /usr/sbin/groupadd oper

2.1.3. *Creating an OSASM Group

Create an OSASM group only if you want SYSASM as a system privilege that enables the separation of the SYSDBA database administration privilege from the ASM storage administration privilege. If you want to use an OSASM group, then you must create it in the following circumstances: If an OSASM group does not exist, for example, if this is the first installation of Oracle Database software on the system If an OSASM group exists, but you want to give a different group of operating system users database operator privileges in a new Oracle installation To determine whether the OSASM group exists, enter the following command:
# grep OSASM_group_name /etc/group

If the OSASM group does not exist or if you require a new OSASM group, then create it as follows. In the following command, use the group name asadmin unless a group with that name already exists.
# /usr/sbin/groupadd asmadmin

2.1.4. *Creating an OINSTALL Group

You must create an OINSTALL group in the following circumstances: An OINSTALL group does not exist, for example, if this is the first installation of Oracle Database software on the system An OINSTALL group exists, but you want to give a different group of operating system users database administrative privileges in a new Oracle installation To determine whether the OINSTALL group exists, enter the following command:
# grep OINSTALL_group_name /etc/group

If the OINSTALL group does not exist or if you require a new OINSTALL group, then create it as follows. In the following command, use the group name dba unless a group with that name already exists.
# /usr/sbin/groupadd oinstall

Note: The default OINSTALL group name is oinstall.

2.1.5. *Determining Whether an Oracle Software Owner User Exists

To determine whether an Oracle software owner user named oracle exists, enter the following command:
# id oracle

If the oracle user exists, then the output from this command is similar to the following: 2.1.2. *Creating an OSOPER Group (Optional) 7

uid=440(oracle) gid=200(oinstall) groups=201(dba),202(oper)

If the user exists, then determine whether you want to use the existing user or create another oracle user. If you want to use the existing user, then ensure that the users primary group is the Oracle Inventory group and that it is a member of the appropriate OSDBA and OSOPER groups.

2.1.6. *Creating an Oracle Software Owner User

In the following procedure, use the user name oracle unless a user with that name already exists. If the Oracle software owner user does not exist or if you require a new Oracle software owner user, then create it as follows: 1.To create the oracle user, enter a command similar to the following:
# /usr/sbin/useradd -g oinstall -G dba[,oper] oracle

In this command: The -g option specifies the primary group, which must be the Oracle Inventory group, for example oinstall The -G option specifies the secondary groups, which must include the OSDBA group and if required, the OSOPER group.dba or dba,oper 2.Set the password of the oracle user:
# passwd oracle

2.1.7. *Modifying an Oracle Software Owner User

If the oracle user exists, but its primary group is not oinstall or it is not a member of the appropriate OSDBA or OSOPER groups, then enter a command similar to the following to modify it. Specify the primary group using the -g option and any required secondary group using the -G option:
# /usr/sbin/usermod -g oinstall -G dba[,oper] oracle

2.1.8. *Configuring SSH on Cluster Member Nodes

To configure SSH, you must first create RSA or DSA keys on each cluster node, and then copy all the keys generated on all cluster node members into an authorized keys file that is identical on each node. Note that the SSH files must be readable only by root and by the software installation user (oracle, crs, asm), as SSH ignores a private key file if it is accessible by others. When this is done, then start the SSH agent to load keys into memory. In the examples that follow, the RSA key is used. You must configure SSH separately for each Oracle software installation owner that you intend to use for installation.

2.1.9. *Checking Existing SSH Configuration on the System

To determine if SSH is running, enter the following command:
$ pgrep sshd

If SSH is running, then the response to this command is one or more process ID numbers. In the home directory of the software owner that you want to use for the installation (crs, oracle), use the command ls -al to ensure that the .ssh directory is owned and writable only by the user. You need either an RSA or a DSA key for the SSH protocol. RSA is used with the SSH 1.5 protocol, while 2.1.5. *Determining Whether an Oracle Software Owner User Exists 8

DSA is the default for the SSH 2.0 protocol. With OpenSSH, you can use either RSA or DSA. The instructions that follow are for SSH1. If you have an SSH2 installation, and you cannot use SSH1, then refer to your SSH distribution documentation to configure SSH1 compatibility or configure SSH2 with DSA.

2.1.10. *To configure SSH, Create RSA Keys On Each Node using the following steps:
1. Log in as the software owner (in this example, the crs user). 2. To ensure that you are logged in as the Oracle user, and that the user ID matches the expected user ID you have assigned to the Oracle user, enter the commands id and id oracle. Ensure that Oracle user group and user and the terminal window process group and user IDs are identical. For example:
[crs@stnsp001 ~]$ id uid=502(crs) gid=501(oinstall) groups=501(oinstall),502(crs) [crs@stnsp001 ~]$ id crs uid=502(crs) gid=501(oinstall) groups=501(oinstall),502(crs)

3. If necessary, create the .ssh directory in the crs user's home directory, and set permissions on it to ensure that only the crs user has read and write permissions:
[crs@stnsp001 ~]$ mkdir ~/.ssh [crs@stnsp001 ~]$ chmod 700 ~/.ssh

4. Enter the following command:

[crs@stnsp001 ~]$ /usr/bin/ssh-keygen -t rsa

At the prompts: Accept the default location for the key file (press Enter). Enter and confirm a pass phrase unique for this installation user. This command writes the RSA public key to the ~/.ssh/id_rsa.pub file and the private key to the ~/.ssh/id_rsa file. Never distribute the private key to anyone not authorized to perform Oracle software installations. 5. Repeat steps 1 through 4 on each node that you intend to make a member of the cluster, using the RSA key.

2.1.11. *Add All Keys to a Common authorized_keys File

Complete the following steps: 1. Combine the contents of the id_rsa.pub files from each server. (You can do this all on one node) run the command below for each node that is a part of the cluster, replacing "stnsp001" with a different node name each time. As the oracle user:
$ ssh stnsp001 "cat ~/.ssh/id_rsa.pub" >> ~/.ssh/authorized_keys $ ssh stnsp002 "cat ~/.ssh/id_rsa.pub" >> ~/.ssh/authorized_keys

* Each time you connect from any node to a new hostname for the first time, you will see a message similar to:
"The authenticity of host 'stnsp001 (10.137.8.215)' can't be established. Are you sure you want to continue connecting (yes/no)?" RSA key fingerprint is ...

Type "yes" and press ENTER. You will then see the message: 2.1.9. *Checking Existing SSH Configuration on the System 9

"Warning: Permanently added 'stnsp001,10.137.8.215' (RSA) to the list of known hosts." "oracle@stnsp001's password:"

Type password and press ENTER. In the .ssh directory, you should see the id_rsa.pub keys that you have created, and the file authorized_keys. 2. On the local node, use SCP (Secure Copy) or SFTP (Secure FTP) to copy the authorized_keys file to the oracle user .ssh directory on a remote node. The following example is with SCP, on a node called stnsp002, with the Oracle Clusterware owner crs, where the crs user path is /home/crs:
[[crs@stnsp001 crs]$ scp ~/.ssh/authorized_keys stnsp002:~/.ssh/

You are prompted to accept an RSA key. Enter Yes, and you see that the node you are copying to is added to the known_hosts file. When prompted, provide the password for the oracle user, which should be the same on all nodes in the cluster. The authorized_keys file is copied to the remote node. Your output should be similar to the following, where xxx represents parts of a valid IP address:
[crs@stnsp001 .ssh]$ ~/.ssh/scp authorized_keys stnsp002:~/.ssh/ crs@stnsp002's password: authorized_keys 100% 828 7.5MB/s

00:00

3. Repeat steps 2 from each node to each other member node in the cluster.

2.1.12. *Enabling SSH User Equivalency on Cluster Member Nodes

After you have copied the authorized_keys file that contains all keys to each node in the cluster, complete the following procedure, in the order listed. In this example, the Oracle Clusterware software owner is named crs: 1. On the system where you want to run OUI, log in as the crs user. 2. Use the following command syntax, where hostname1, hostname2, and so on, are the public hostnames (alias and fully qualified domain name) of nodes in the cluster to run SSH from the local node to each node, including from the local node to itself, and from each node to each other node:
[crs@stnsp001 ~]$ ssh stnsp001 date [crs@stnsp001 ~]$ ssh stnsp002 date . . . For example: [crs@stnsp001 crs]$ ssh stnsp001 date The authenticity of host 'stnsp001 (xxx.xxx.8.215)' can't be established. RSA key fingerprint is 7z:60:60:zz:48:48:z1:a0:f7:4e. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'stnsp001,xxx.xxx.8.215' (RSA) to the list of known hosts. Enter passphrase for key '/home/crs/.ssh/id_rsa': Mon May 19 11:08:13 PST 2008 [crs@stnsp001 crs]$ ssh stnsp001.us.oracle.com date The authenticity of host 'stnsp001.us.oracle.com (xxx.xxx.8.215)' can't be established. RSA key fingerprint is 7z:60:60:zz:48:48:z1:a0:f7:4e. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'stnsp001.us.oracle.com,xxx.xxx.8.215' (RSA) to the list of known hosts. Enter passphrase for key '/home/crs/.ssh/id_rsa': Mon May 19 11:08:13 PST 2008

2.1.11. *Add All Keys to a Common authorized_keys File

10

[crs@stnsp001 crs]$ ssh stnsp002 date Enter passphrase for key '/home/crs/.ssh/id_rsa': Mon May 19 11:08:35 PST 2008 . .

At the end of this process, the public hostname for each member node should be registered in the known_hosts file for all other cluster member nodes. If you are using a remote client to connect to the local node, and you see a message similar to
"Warning: No xauth data; using fake authentication data for X11 forwarding,"

then this means that your authorized keys file is configured correctly, but your ssh configuration has X11 forwarding enabled. To correct this, proceed to "Setting Display and X11 Forwarding Configuration" below. 3. Repeat step 2 on each cluster node member. 4. (Optional) If you entered passphrase above, and desire password-less login (needed for 10g RAC install), you need to inform the ssh-agent (already running for desktop) of the passphrase so that ssh clients would not be prompted for a passphrase. Once you notify ssh-agent the passphrase is cached for duration of the GUI desktop login, and further client requests are satisfied by the ssh-agent.
$ ssh-add Enter passphrase for /home/oracle/.ssh/id_dsa: *TYPE YOUR PASSPHRASE* Identity added: /home/oracle/.ssh/id_dsa (/home/oracle/.ssh/id_dsa)

At the prompt, enter the pass phrase for each key that you generated. For example:
[crs@stnsp001 .ssh]$ exec /usr/bin/ssh-agent $SHELL [crs@stnsp001 .ssh]$ ssh-add Enter passphrase for /home/crs/.ssh/id_rsa Identity added: /home/crs/.ssh/id_rsa (/home/crs/.ssh/id_rsa)

These commands start the ssh-agent on the node, and load the RSA keys into memory so that you are not prompted to use pass phrases when issuing SSH commands. If you have configured SSH correctly, then you can now use the ssh or scp commands without being prompted for a password or a pass phrase. For example:
[crs@stnsp001 ~]$ ssh stnsp002 date Mon May 19 23:34:42 UTC 2008 [crs@stnsp001 ~]$ ssh stnsp001 date Mon May 19 23:34:48 UTC 2008 [crs@stnsp001 ~]$ ssh stnsp002

If any node prompts for a password or pass phrase, then verify that the ~/.ssh/authorized_keys file on that node contains the correct public keys, and that you have created an Oracle software owner with identical group membership and IDs.

2.1.13. *Setting Display and X11 Forwarding Configuration

If you are on a remote terminal, and the local node has only one visual (which is typical), then use the following syntax to set the DISPLAY environment variable: Bourne, Korn, and Bash shells
[crs@stnsp001 ~]$ export DISPLAY=hostname:0

2.1.12. *Enabling SSH User Equivalency on Cluster Member Nodes

11

Note: You must run OUI from this session, or make a note of your SSH pass phrase, and remember to repeat step 4 before you start OUI from a different terminal session. C shell:
[crs@stnsp001 ~]$ setenv DISPLAY hostname:0

For example, if you are using the Bash shell, and if your hostname is stnsp001, then enter the following command:
$ export DISPLAY=stnsp001:0

To ensure that X11 forwarding will not cause the installation to fail, create a user-level SSH client configuration file for the Oracle software owner user, as follows: A. Using any text editor, edit or create the software installation owners
~/.ssh/config file.

B. Make sure that the ForwardX11 attribute is set to no. For example:
Host * ForwardX11 no

If you are not plannig to run the installer on the console you may need to run the following command to allow the X session to be accessed remotely
xhost +

2.2. *Networking
You will need a minimum of 2 network adapters per machine, 1 adapter is the public interface. The other adapter, the private interface, is used for Oracle RAC node membership and inter-instance traffic. The private interface adapters should be connected to each other via a network switch which is used exclusively for the interconnect traffic. In 2 node environments do not be tempted to use a 'Crossover Cable'. Best practice is to use more than 2 adapters, Before installation of the Oracle software you take 2 of the adapters on the private interface and team them together to create what looks like a single, highly available adapter for the interconnect traffic. You can then take the remaining 3rd adapter and use it for the public interface; some customers even team their public adapters as well. In that case you would need to start with 4 adapters, 2 public & 2 private. Ensure you team the adapters before starting installation. Please be aware that even with 2-node cluster you must use a dedicated network switch on the interconnect, Crossover cables are not supported You will need a total of 3 IP addresses per node: The public IP address, which should be recorded in hosts file on each node and, if available, DNS. This IP Address should be bound to the public adapter before starting the install. It should be a static, not DHCP, address The private IP address, which should be from a different subnet than the public IP address. This address does not require registering in DNS but you should place an entry in the hosts file on each node. This IP Address should be bound to the private adapter before starting the install. It should be a static, not DHCP, address A VIP address, which should be from the same subnet as the public IP address and should be recorded in DNS and the hosts file on each node. This IP Address should NOT be bound to the public adapter before starting the install. Oracle Clusterware is responsible for binding this address. It should 2.1.13. *Setting Display and X11 Forwarding Configuration 12

be a static, not DHCP, address If you do not have a DNS server then make sure both the public and the VIP addresses are entered into all the relevant hosts files that are normally found in the /etc/hosts file. Show Networking setup information Hide Networking setup information You should use ifconfig on each node to confirm the correct IP addresses are bound to the correct adapters /sbin/ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:0E:0C:08:08:F2 inet addr:10.137.8.215 Bcast:10.137.15.255 Mask:255.255.248.0 inet6 addr: fe80::20e:cff:fe08:8f2/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:111936568 errors:0 dropped:0 overruns:0 frame:0 TX packets:617958 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3027847550 (2.8 GiB) TX bytes:138428916 (132.0 MiB) Base address:0x2880 Memory:fe7c0000-fe7e0000 Link encap:Ethernet HWaddr 00:0E:0C:08:08:F3 inet6 addr: fe80::20e:cff:fe08:8f3/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1041506 errors:0 dropped:0 overruns:0 frame:0 TX packets:5 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:86613582 (82.6 MiB) TX bytes:398 (398.0 b) Base address:0x2840 Memory:fe7e0000-fe800000 Link encap:Ethernet HWaddr 00:04:23:A6:CD:61 inet addr:10.137.20.171 Bcast:10.137.23.255 Mask:255.255.252.0 inet6 addr: fe80::204:23ff:fea6:cd61/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:41400340 errors:0 dropped:0 overruns:0 frame:0 TX packets:37403509 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:844975237 (805.8 MiB) TX bytes:43154372 (41.1 MiB) Base address:0x2800 Memory:fe880000-fe8a0000

eth1

eth2

A sample /etc/hosts file

# Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost 10.137.8.215 stnsp001.us.oracle.com stnsp001 # Also in DNS if available 10.137.8.233 stnsp001-vip.us.oracle.com stnsp001-vip # Also in DNS if available 10.137.20.171 stnsp001-rac.us.oracle.com stnsp001-rac

It is important to confirm that networking is configured correctly before commencing install. Show all Hide all

2.2.1. *Network Ping Tests

There are a series of 'ping' tests that should be completed, and then the network adapter binding order should be checked. You should ensure that the public IP addresses resolve correctly and that the private addresses are of the form 'nodename-priv' and resolve on both nodes via the hosts file. Public Ping test Pinging stnsp001 from stnsp001 should return stnsp001's public IP address Pinging stnsp002 from stnsp001 should return stnsp002's public IP address Pinging stnsp001 from stnsp002 should return stnsp001's public IP address Pinging stnsp002 from stnsp002 should return stnsp002's public IP address

2.2. *Networking

13

 Private Ping test Pinging stnsp001 private from stnsp001 should return stnsp001's private IP address Pinging stnsp002 private from stnsp001 should return stnsp002's private IP address Pinging stnsp001 private from stnsp002 should return stnsp001's private IP address Pinging stnsp002 private from stnsp002 should return stnsp002's private IP address VIP Ping test Pinging the VIP address at this point should fail. VIPs will be activated at the end of the Oracle Clusterware install. If any of the above tests fail you should fix name/address resolution by updating the DNS or local hosts files on each node before continuing with the installation.

2.2.2. *Network Adapter configuration

If your network adapters allow configuration you should make sure that they are configured for Full Duplex and at the 'fastest' speed consistent among nodes. They should not be left to 'auto-negotiate'.

2.3. *Time Sync

There is a general requirement for Oracle RAC that the time on all the nodes be the same. Typically this can be achieved using Network Time Server protocol. At the very least you should ensure that the system clocks on all nodes are as close as possible.

2.4. *Configuring Kernel Parameters

On all cluster nodes, verify that the kernel parameters shown in the following table are set to values greater than or equal to the recommended value shown. The procedure following the table describes how to verify and set the values. Parameter Value File semmsl 250 /proc/sys/kernel/sem semmns 32000 semopm 100 semmni 128 shmmax Half the size of physical memory (in bytes) /proc/sys/kernel/shmmax shmmni 4096 /proc/sys/kernel/shmmni shmall 2097152 /proc/sys/kernel/shmall file-max 65536 /proc/sys/fs/file-max ip_local_port_range Minimum: 9000 Maximum:65500 /proc/sys/net/ipv4/ip_local_port_range rmem_default 262144 /proc/sys/net/core/rmem_default rmem_max 4194304 /proc/sys/net/core/rmem_max wmem_default 262144 /proc/sys/net/core/wmem_default wmem_max 1048576 /proc/sys/net/core/wmem_max aio-max-nr 1048576 /proc/sys/fs/aio-max-nr To view the current value specified for these kernel parameters, and to change them if necessary, follow these steps: 1. Enter the commands shown in the following table to view the current values of the kernel parameters: Parameters semmsl, semmns,semopm, and semmni for shmall, shmmax, and shmmni file-max 2.2.1. *Network Ping Tests Command # /sbin/sysctl -a | grep sem # /sbin/sysctl -a | grep shm # /sbin/sysctl -a | grep file-max 14

ip_local_port_range # /sbin/sysctl -a | grep ip_local_port_range rmem_default, rmem_max, wmem_default, and wmem_max # /sbin/sysctl -a | grep net.core aio-max-nr # /sbin/sysctl -a | grep aio-max-nr 2. If the value of any kernel parameter is less than the recommended value, then complete the following process: Using any text editor, create or edit the /etc/sysctl.conf file, and add or edit lines similar to the following:
kernel.shmall = 2097152 kernel.shmmax = 2147483648 kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 fs.file-max = 65536 net.ipv4.ip_local_port_range = 9000 65500 net.core.rmem_default = 262144 net.core.rmem_max = 4194304 net.core.wmem_default = 262144 net.core.wmem_max = 1048576

By specifying the values in the /etc/sysctl.conf file, they persist when you restart the system. Note: Include lines only for the kernel parameter values that you want to change. For the semaphore parameters (kernel.sem), you must specify all four values. However, if any of the current system parameter values are greater than the recommended values, then keep using the larger values. On Red Hat systems, to have these changes take effect immediately so that you do not have to restart the system, enter the following command: /sbin/sysctl -p 3. Repeat steps 1 and 2 on all other nodes in the cluster. On SUSE systems only, enter the following command to cause the system to read the /etc/sysctl.conf file when it restarts: # /sbin/chkconfig boot.sysctl on 4. On SUSE Linux Enterprise Server 9.0 only, set the kernel parameter disable_cap_mlock as follows: disable_cap_mlock = 1 5. After updating the values of kernel parameters in the /etc/sysctl.conf file, either restart the computer, or run the command sysctl -p to make the changes in the /etc/sysctl.conf file available in the active kernel memory.

2.5. *Setting Shell Limits for the oracle User

To improve the performance of the software on Linux systems, you must increase the following shell limits for the oracle user |Shell Limit |Item in limits.conf |Hard Limit| Maximum number of open file descriptors nofile 65536 Maximum number of processes available to a single user nproc 16384 To increase the shell limits: 1. Add the following lines to the /etc/security/limits.conf file:
oracle oracle oracle oracle soft hard soft hard nproc 2047 nproc 16384 nofile 1024 nofile 65536

2. Add or edit the following line in the /etc/pam.d/login file, if it does not already exist:
session required /lib/security/pam_limits.so

3. Depending on the oracle user's default shell, make the following changes to the default shell startup file: For the Bourne, Bash, or Korn shell, add the following lines to the /etc/profile file (or the file /etc/profile.local on SUSE systems): 2.4. *Configuring Kernel Parameters 15

if [ $USER = "oracle" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi fi

For the C shell (csh or tcsh), add the following lines to the /etc/csh.login file (or the file /etc/csh.login.local on SUSE systems):
if ( $USER == "oracle" ) then limit maxproc 16384 limit descriptors 65536 endif

1. Repeat this procedure on all other nodes in the cluster.

2.6. *Configuration of the Hangcheck-timer Module

Before installing Oracle Real Application Clusters on Linux systems, verify that the hangcheck-timer module (hangcheck-timer) is loaded and configured correctly. hangcheck-timer monitors the Linux kernel for extended operating system hangs that could affect the reliability of a RAC node and cause a database corruption. If a kernel/device driver hang occurs, then the module restarts the node in seconds. There are 3 parameters used to control the behavior of the module: 1. The hangcheck_tick parameter: it defines how often, in seconds, the hangcheck-timer checks the node for hangs. The default value is 60 seconds. Oracle recommends to set it to 1 (hangcheck_tick=1). 2. The hangcheck_margin parameter: it defines how long the timer waits, in seconds, for a response from the kernel. The default value is 180 seconds. Oracle recommends to set it to 10 (hangcheck_margin=10) 3. The hangcheck_reboot parameter: If the value of hangcheck_reboot is equal to or greater than 1, then the hangcheck-timer module restarts the system. If the hangcheck_ reboot parameter is set to zero, then the hangcheck-timer module will not restart the node. It should always be set to 1. If the kernel fails to respond within the sum of the hangcheck_tick and hangcheck_margin parameter values, then the hangcheck-timer module restarts the system.

2.6.1. *Hangcheck-timer Module verification procedure

1. Log in as root, and enter the following command to check the kernel version:
# uname -a

2. Enter the following command on each node to determine which kernel modules are loaded:
# /sbin/lsmod

3. If the hangcheck-timer module is not listed for any node: on Kernel 2.4 enter a command similar to the following to start the module on that node: # /sbin/insmod hangcheck-timer hangcheck_tick=1 hangcheck_margin=10 hangcheck_reboot=1 on Kernel 2.6 enter a command similar to the following to start the module located in the directories of the current kernel version:

2.5. *Setting Shell Limits for the oracle User

16

# insmod /lib/modules/kernel_version/kernel/drivers/char/hangcheck-timer.ko hangcheck_tick=1 hangcheck_margin=10 hangcheck_reboot=1 In the preceding command example, the variable kernel_version is the kernel version running on your system that you have got from uname -a. 4. To confirm that the hangcheck module is loaded, enter the following command:
# lsmod | grep hang

The output should be similar to the following:

hangcheck_timer 3289 0

5. To ensure that the module is loaded every time the system restarts, verify that the local system startup file contains the command shown in the previous step, or add it if necessary: on Red Hat: On Red Hat Enterprise Linux systems, add the command to the /etc/rc.d/rc.local file. on SUSE: On SUSE systems, add the command to the /etc/init.d/boot.local file.

2.7. *Platform Specific Setup

Show How to Disable RUN_PARALLEL on SLES (SuSE Linux): Hide Disable RUN_PARALLEL on SLES: As per Metalink Note:464061.1, on SuSE Linux disable RUN_PARALLEL by setting it to "no" in /etc/sysconfig/boot. The default setting of "yes" prevents the Oracle Clusterware to start in most cases. For a different workaround refer to the note for details. Show How to Configure SELinux on OEL5 and RHEL5: Hide Configure SELinux on OEL5 and RHEL5: As per Metalink Note:454196.1, on Oracle Enterprise Linux 5 & Redhat Enterprise Linux 5 it's required to either disable or set SELinux to permissive mode using one of the methods below: Editing /etc/selinux/config: Change the SELINUX value to "SELINUX=disabled" or "SELINUX=permissive". Appending kernel boot options: Edit the kernel boot line and append "selinux=0" (to completely disable), or "enforcing=0" (to set to permissive mode) to the kernel boot options. For example: title Red Hat Enterprise Linux AS (2.6.9-42.ELsmp) root (hd0,0) kernel /vmlinuz-2.6.9-42.ELsmp ro root=LABEL=/ rhgb quiet selinux=0 initrd /initrd-2.6.9-42.ELsmp.img Do one of the above and then reboot the server for change to take effect, or if a reboot is not possible, issue as root: # setenforce 0 This will put SELinux into permissive mode immediately. Show How to Ensure latest glibc is installed: Hide Ensure latest glibc is installed: As per Metalink Note:731599.1, to avoid false node reboots, ensure your glibc rpm is updated as follows: 2.6.1. *Hangcheck-timer Module verification procedure 17

 EL4 customers: install glibc-2.3.4-2.40 (or above) or upgrade to EL4u7 EL5 customers: install glibc-2.5-24 (or above) or upgrade to EL5u2 SLES10 customers: bug 416838 filed for this problem, contact Novell for fix. Unbreakable Linux customers or Redhat Network subscribers can issue: # up2date glibc Refer to above note for further details. Show How to disable deletion of critical Clusterware files: Hide deleting critical Clusterware files: The Oracle Clusterware places important socket files in /var/tmp/.oracle directory, which can be removed by Linux crontab cleanup jobs. As per Metalink Note: 391790.1, ensure that such deletion does not occur. The supplied builtin tmpwatch cronjob: /usr/sbin/tmpwatch 720 /var/tmp

is safe since it does not delete any socket or special files, only regular files. However custom scripts or 3rd party cleanup jobs might cleanup such files and cause Clusterware failures.

2.8. *Stage the Oracle Software

It is recommended that you stage the required software onto a local drive on Node 1 of your cluster. Important. Ensure that you use only 32 bit versions of the Oracle Software on a 32bit OS and 64 bit versions of the Oracle Software on a 64bit OS A suggested stage is as follows Oracle 11g Release 1 EE (11.1.0.6) (1 DVD ) /mnt The 11gR1 now integrates the Oracle Clusterware, Database and Client install into one DVD with one runInstaller program. If you download the software from OTN you will not get the integrated installer. You will have separate downloads for : Oracle Clusterware and ASM/Database

2.9. *Cluster Verification Utility stage check

Now you can run the CVU (Cluster Verification Utility) to check the state of the Operating System configuration. CVU can be run from the installation media, but it is recommended to download the latest version from: http://www.oracle.com/technology/products/database/clustering/cvu/cvu_download_homepage.html After the hardware and OS has been configured, it is recommended to run CVU to verify the nodes are configured correctly: runcluvfy.sh stage -post hwos -n stnsp001,stnsp002 -verbose Show CVU -post hwos output Hide CVU output

./runcluvfy.sh stage -post hwos -n stnsp001,stnsp002 -verbose Performing post-checks for hardware and operating system setup

2.7. *Platform Specific Setup

18

Checking node reachability... Check: Node reachability from node "stnsp001" Destination Node Reachable? ------------------------------------ -----------------------stnsp001 yes stnsp002 yes Result: Node reachability check passed from node "stnsp001".

Checking user equivalence... Check: User equivalence for user "oracle11" Node Name Comment ------------------------------------ -----------------------stnsp002 passed stnsp001 passed Result: User equivalence check passed for user "oracle11". Checking node connectivity...

Interface information for node "stnsp002" Interface Name IP Address Subnet ---------------- ------------ -----------eth0 10.137.8.216 10.137.8.0 eth1 10.137.24.207 10.137.24.0 eth2 10.137.20.172 10.137.20.0

Subnet Gateway Default Gateway Hardware Address ------------ ------------ -----------0.0.0.0 10.137.8.1 00:0E:0C:08:06:4E 0.0.0.0 10.137.8.1 00:0E:0C:08:06:4F 0.0.0.0 10.137.8.1 00:04:23:A6:CD:BD

Interface information for node "stnsp001" Interface Name IP Address Subnet ---------------- ------------ -----------eth0 10.137.8.215 10.137.8.0 eth1 10.137.24.206 10.137.24.0 eth2 10.137.20.171 10.137.20.0

Subnet Gateway Default Gateway Hardware Address ------------ ------------ -----------0.0.0.0 10.137.8.1 00:0E:0C:08:08:F2 0.0.0.0 10.137.8.1 00:0E:0C:08:08:F3 0.0.0.0 10.137.8.1 00:04:23:A6:CD:61

Check: Node connectivity of subnet "10.137.8.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------stnsp002:eth0 stnsp001:eth0 yes Result: Node connectivity check passed for subnet "10.137.8.0" with node(s) stnsp002,stnsp001. Check: Node connectivity of subnet "10.137.24.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------stnsp002:eth1 stnsp001:eth1 yes Result: Node connectivity check passed for subnet "10.137.24.0" with node(s) stnsp002,stnsp001. Check: Node connectivity of subnet "10.137.20.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------stnsp002:eth2 stnsp001:eth2 yes Result: Node connectivity check passed for subnet "10.137.20.0" with node(s) stnsp002,stnsp001. Interfaces found on subnet "10.137.8.0" that are likely candidates for a private interconnect: stnsp002 eth0:10.137.8.216 stnsp001 eth0:10.137.8.215 Interfaces found on subnet "10.137.24.0" that are likely candidates for a private interconnect: stnsp002 eth1:10.137.24.207 stnsp001 eth1:10.137.24.206 Interfaces found on subnet "10.137.20.0" that are likely candidates for a private interconnect: stnsp002 eth2:10.137.20.172 stnsp001 eth2:10.137.20.171 WARNING: Could not find a suitable set of interfaces for VIPs. Result: Node connectivity check passed.

Checking shared storage accessibility...

2.9. *Cluster Verification Utility stage check

19

WARNING: Package cvuqdisk not installed. stnsp001 NFS -----------------------------------/nas Sharing Nodes (2 in count) -----------------------stnsp002 stnsp001

Shared storage check was successful on nodes "stnsp002,stnsp001". Post-check for hardware and operating system setup was successful.

Show all

Hide all

3. *Prepare the shared storage for Oracle RAC

This section describes how to prepare the shared storage for Oracle RAC For all installations, you must choose the storage option that you want to use for Oracle Clusterware files, Automatic Storage Manager (ASM) and Oracle Real Application Clusters databases (Oracle RAC). You do not have to use the same storage option for each file type. Oracle Clusterware files include: Voting disks, used to monitor cluster node status. Oracle Cluster Registry (OCR) which contains configuration information about the cluster. There are two ways of storing Oracle Clusterware files: Block or Raw Devices: Oracle Clusterware files can be placed on either Block or RAW devices based on shared disk partitions. Oracle recommends using Block devices for easier usage. A supported shared file system: Supported file systems include the following: A supported cluster file system Network File System (NFS): A file-level protocol that enables access and sharing of files When you create partitions with fdisk by specifying a device size, such as +256M, the actual device created may be smaller than the size requested, based on the cylinder geometry of the disk. This is due to current fdisk restrictions. Oracle configuration software checks to ensure that devices contain a minimum of 256MB of available disk space. Therefore, Oracle recommends using at least 280MB for the device size. You can check partition sizes by using the command syntax fdisk -s partition. For example: [root@node1]$ fdisk -s /dev/sdb1 281106 As root, now configure storage for cluster registry, voting disk and database files. You are presented with a bunch of disks from the storage array. The output of fdisk s /dev/sd[b-e] command may look as follows.
/dev/sdb: /dev/sdc: /dev/sdd: /dev/sde: 116924416 116924416 116924416 116924416

In this example we have four 116Gig LUNs:

/dev/sdb /dev/sdc /dev/sdd /dev/sde 116G 116G 116G 116G

3. *Prepare the shared storage for Oracle RAC

20

3.1. Example of Configuring Block Device Storage for Oracle Clusterware

The procedure to create partitions for Oracle Clusterware files on block devices is asfollows: 1. log in as root 2. Enter the fdisk command to format a specific storage disk (for example,/sbin/fdisk /dev/sdb) 3. Create a new partition, and make the partition 280 MB in size for both OCR andvoting disk partitions. 4. Use the command syntax /sbin/partprobe diskpath on each node in thecluster to update the kernel partition table for the shared storage device on eachnode. The following is an example of how to use fdisk to create one partition on a shared storage block disk device for an OCR file:
[root@stnsp001] # /sbin/fdisk /dev/sdb The number of cylinders for this disk is set to 1024. Command (m for help): n Command action e extended P primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-1024, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-1024, default 1) Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-1024, default 1024):+280m Command (m for help):w The partition table has been altered! Calling ioctl () to re-read partition table. Synching disks.

Login as the root user on the remote nodes and execute the following: [root@stnsp002] # /sbin/partprobe /dev/sdb1 Note: Oracle recommends that you create partitions for Oracle Clusterware files on physically separate disks. The user account with which you perform the installation (oracle or crs) must have write permissions to create the files in the path that you specify.

3.1.1. Example of Creating a Udev Permissions File for Oracle Clusterware

The procedure to create a permissions file to grant oinstall group members write privileges to block devices is as follows: 1. Log in as root. 2. Change to the /etc/udev/permissions.d directory:
# cd /etc/udev/permissions.d

1. Start a text editor, such as vi, and enter the partition information where you want to place the OCR and voting disk files, using the syntax
device_partitions:root:oinstall:0640

3.1. Example of Configuring Block Device Storage for Oracle Clusterware

21

Note that Oracle recommends that you place the OCR and the voting disk files on separate physical disks. For example, to grant oinstall members access to SCSI disks to place OCR files on sdb1 and sdc1, and to grant the Oracle Clusterware owner (in this example crs) permissions to place voting disks on sdb5, sdc5 and sda5, add the following information to the file:
# OCR disks sdb1:root:oinstall:0640 sdc1:root:oinstall:0640 # Voting disks sdb5:crs:oinstall:0640 sdc5:crs:oinstall:0640 sdd5:crs:oinstall:0640

1. Save the file: On Asianux 2, Enterprise Linux 4, and Red Hat Enterprise Linux 4 systems, save the file as 49-oracle.permissions. On Asianux 3, Enterprise Linux 5, Red Hat Enterprise Linux 5, and SUSE Enterprise Server 10 systems, save the file as 51-oracle.permissions. 1. Using the following command, assign the permissions in the udev file to the devices: # /sbin/udevstart Use the following procedure above to create additional partitions to use for the OCR, Voting, and ASM disks. Refer to the OS documentation for additional information on using the fdisk command.

3.1.2. Platform Specific Settings

Show How to Disable HOTPLUG_USE_SUBFS on SLES9 (SuSE Linux): Hide Disable HOTPLUG_USE_SUBFS on SLES9: As per Metalink Note:357472.1, on SuSE Linux SLES9 disable HOTPLUG_USE_SUBFS by setting it to "no" in /etc/sysconfig/hotplug. The default setting of "yes" causes problems for multipath devices.

4. Oracle Clusterware Installation and Configuration

This page includes hidden sections, use the and image for each section to show/hide the section Show all Hide all

or you can 'Show' or 'Hide' all by selecting here :

Next we will install the Oracle Clusterware layer. Oracle Clusterware is an essential component of the Oracle RAC database infrastructure. Oracle Clusterware does not require any other clustering software.

4.1. CVU Pre Oracle Clusterware install check

Prior to installing CRS, verify the nodes are configured correctly for the CRS install
runcluvfy.sh stage -pre crsinst -n stnsp001,stnsp002 -r 11gR1 -verbose

Show CVU -pre crsinst output

Hide CVU output

runcluvfy.sh stage -pre crsinst -n stnsp001,stnsp002 -r 11gR1 -verbose Performing pre-checks for cluster services setup Checking node reachability... Check: Node reachability from node "stnsp001" Destination Node Reachable?

3.1.1. Example of Creating a Udev Permissions File for Oracle Clusterware

22

------------------------------------ -----------------------stnsp001 yes stnsp002 yes Result: Node reachability check passed from node "stnsp001".

Checking user equivalence... Check: User equivalence for user "oracle11" Node Name Comment ------------------------------------ -----------------------stnsp002 passed stnsp001 passed Result: User equivalence check passed for user "oracle11". Checking administrative privileges... Check: Existence of user "oracle11" Node Name User Exists Comment ------------ ------------------------ -----------------------stnsp002 yes passed stnsp001 yes passed Result: User existence check passed for "oracle11". Check: Existence of group "oinstall" Node Name Status Group ID ------------ ------------------------ -----------------------stnsp002 exists 1422 stnsp001 exists 1422 Result: Group existence check passed for "oinstall". Check: Membership of user "oracle11" in group "oinstall" [as Primary] Node Name User Exists Group Exists User in Group Primary Comment ---------------- ------------ ------------ ------------ ------------ -----------stnsp002 yes yes yes yes passed stnsp001 yes yes yes yes passed Result: Membership check for user "oracle11" in group "oinstall" [as Primary] passed. Administrative privileges check passed. Checking node connectivity...

Interface information for node "stnsp002" Interface Name IP Address Subnet ---------------- ------------ -----------eth0 10.137.8.216 10.137.8.0 eth2 10.137.20.172 10.137.20.0

Subnet Gateway Default Gateway Hardware Address ------------ ------------ -----------0.0.0.0 10.137.8.1 00:0E:0C:08:06:4E 0.0.0.0 10.137.8.1 00:04:23:A6:CD:BD

Interface information for node "stnsp001" Interface Name IP Address Subnet ---------------- ------------ -----------eth0 10.137.8.215 10.137.8.0 eth2 10.137.20.171 10.137.20.0

Subnet Gateway Default Gateway Hardware Address ------------ ------------ -----------0.0.0.0 10.137.8.1 00:0E:0C:08:08:F2 0.0.0.0 10.137.8.1 00:04:23:A6:CD:61

Check: Node connectivity of subnet "10.137.8.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------stnsp002:eth0 stnsp001:eth0 yes Result: Node connectivity check passed for subnet "10.137.8.0" with node(s) stnsp002,stnsp001. Check: Node connectivity of subnet "10.137.20.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------stnsp002:eth2 stnsp001:eth2 yes Result: Node connectivity check passed for subnet "10.137.20.0" with node(s) stnsp002,stnsp001. Interfaces found on subnet "10.137.8.0" that are likely candidates for a private interconnect: stnsp002 eth0:10.137.8.216 stnsp001 eth0:10.137.8.215 Interfaces found on subnet "10.137.20.0" that are likely candidates for a private interconnect: stnsp002 eth2:10.137.20.172 stnsp001 eth2:10.137.20.171

4.1. CVU Pre Oracle Clusterware install check

23

WARNING: Could not find a suitable set of interfaces for VIPs. Result: Node connectivity check passed.

Checking system requirements for 'crs'... Check: Total memory Node Name Available ------------ -----------------------stnsp002 5.95GB (6234256KB) stnsp001 5.95GB (6234256KB) Result: Total memory check passed. Check: Free disk space in "/tmp" dir Node Name Available ------------ -----------------------stnsp002 235.83GB (247283204KB) stnsp001 201.14GB (210907900KB) Result: Free disk space check passed. Check: Swap space Node Name Available ------------ -----------------------stnsp002 5.91GB (6192456KB) stnsp001 5.91GB (6192456KB) Result: Swap space check passed. Check: System architecture Node Name Available ------------ -----------------------stnsp002 i686 stnsp001 i686 Result: System architecture check passed. Check: Kernel version Node Name Available ------------ -----------------------stnsp002 2.6.9-34.0.1.0.11.ELsmp stnsp001 2.6.9-34.0.1.0.11.ELsmp Result: Kernel version check passed.

Required -----------------------1GB (1048576KB) 1GB (1048576KB)

Comment ---------passed passed

Required -----------------------400MB (409600KB) 400MB (409600KB)

Comment ---------passed passed

Required -----------------------1.5GB (1572864KB) 1.5GB (1572864KB)

Comment ---------passed passed

Required -----------------------i686 i686

Comment ---------passed passed

Required -----------------------2.6.9 2.6.9

Comment ---------passed passed

Check: Package existence for "make-3.80" Node Name Status ------------------------------ -----------------------------stnsp002 make-3.80-5 stnsp001 make-3.80-5 Result: Package existence check passed for "make-3.80".

Comment ---------------passed passed

Check: Package existence for "binutils-2.15.92.0.2" Node Name Status Comment ------------------------------ ------------------------------ ---------------stnsp002 binutils-2.15.92.0.2-18 passed stnsp001 binutils-2.15.92.0.2-18 passed Result: Package existence check passed for "binutils-2.15.92.0.2". Check: Package existence for "gcc-3.4.5" Node Name Status ------------------------------ -----------------------------stnsp002 gcc-3.4.5-2 stnsp001 gcc-3.4.5-2 Result: Package existence check passed for "gcc-3.4.5". Check: Package existence for "libaio-0.3.105" Node Name Status ------------------------------ -----------------------------stnsp002 libaio-0.3.105-2 stnsp001 libaio-0.3.105-2 Result: Package existence check passed for "libaio-0.3.105". Check: Package existence for "libaio-devel-0.3.105" Node Name Status ------------------------------ -----------------------------stnsp002 libaio-devel-0.3.105-2

Comment ---------------passed passed

Comment ---------------passed passed

Comment ---------------passed

4.1. CVU Pre Oracle Clusterware install check

24

stnsp001 libaio-devel-0.3.105-2 passed Result: Package existence check passed for "libaio-devel-0.3.105". Check: Package existence for "libstdc++-3.4.5" Node Name Status ------------------------------ -----------------------------stnsp002 libstdc++-3.4.5-2 stnsp001 libstdc++-3.4.5-2 Result: Package existence check passed for "libstdc++-3.4.5".

Comment ---------------passed passed

Check: Package existence for "elfutils-libelf-devel-0.97" Node Name Status Comment ------------------------------ ------------------------------ ---------------stnsp002 elfutils-libelf-devel-0.97-5 passed stnsp001 elfutils-libelf-devel-0.97-5 passed Result: Package existence check passed for "elfutils-libelf-devel-0.97". Check: Package existence for "sysstat-5.0.5" Node Name Status ------------------------------ -----------------------------stnsp002 sysstat-5.0.5-7.rhel4 stnsp001 sysstat-5.0.5-7.rhel4 Result: Package existence check passed for "sysstat-5.0.5". Check: Package existence for "libgcc-3.4.5" Node Name Status ------------------------------ -----------------------------stnsp002 libgcc-3.4.5-2 stnsp001 libgcc-3.4.5-2 Result: Package existence check passed for "libgcc-3.4.5".

Comment ---------------passed passed

Comment ---------------passed passed

Check: Package existence for "libstdc++-devel-3.4.5" Node Name Status Comment ------------------------------ ------------------------------ ---------------stnsp002 libstdc++-devel-3.4.5-2 passed stnsp001 libstdc++-devel-3.4.5-2 passed Result: Package existence check passed for "libstdc++-devel-3.4.5". Check: Package existence for "unixODBC-2.2.11" Node Name Status ------------------------------ -----------------------------stnsp002 unixODBC-2.2.11-1.RHEL4.1 stnsp001 unixODBC-2.2.11-1.RHEL4.1 Result: Package existence check passed for "unixODBC-2.2.11".

Comment ---------------passed passed

Check: Package existence for "unixODBC-devel-2.2.11" Node Name Status Comment ------------------------------ ------------------------------ ---------------stnsp002 unixODBC-devel-2.2.11-1.RHEL4.1 passed stnsp001 unixODBC-devel-2.2.11-1.RHEL4.1 passed Result: Package existence check passed for "unixODBC-devel-2.2.11". Check: Package existence for "glibc-2.3.4-2.19" Node Name Status ------------------------------ -----------------------------stnsp002 glibc-2.3.4-2.19 stnsp001 glibc-2.3.4-2.19 Result: Package existence check passed for "glibc-2.3.4-2.19". Check: Group existence for "dba" Node Name Status Comment ------------ ------------------------ -----------------------stnsp002 exists passed stnsp001 exists passed Result: Group existence check passed for "dba". Check: Group existence for "oinstall" Node Name Status Comment ------------ ------------------------ -----------------------stnsp002 exists passed stnsp001 exists passed Result: Group existence check passed for "oinstall". Check: User existence for "nobody" Node Name Status ------------ ------------------------

Comment ---------------passed passed

Comment ------------------------

4.1. CVU Pre Oracle Clusterware install check

25

stnsp002 exists passed stnsp001 exists passed Result: User existence check passed for "nobody". System requirement passed for 'crs' Pre-check for cluster services setup was successful.

Show all

Hide all

4.2. Oracle Clusterware Install

Start the installer by running "runInstaller" from the staged installation media.
./runInstaller

Notes Notice that the Oracle 11g Installer now combines the Oracle Database, Client and Clusterware components Action Select the Oracle Clusterware radio button Click Next

4.2. Oracle Clusterware Install

26

 Notes The OUI will name the Oracle Clusterware Home 'OraCrs11g_home'. If you change this you should make sure that the name you use is unique Actions Specify a location for the Oracle Clusterware Home Click Next

4.2. Oracle Clusterware Install

27

 Notes The installer will validate the state of the cluster before continuing. If there are issues you should rectify them before continuing Actions Click Next

Notes Each Cluster requires a name, this should be unique within your organisation, The default is a substring of the node name followed by _cluster This is where you specify details of all the nodes in the cluster. The installer will default names for the node it is running on. You must add other nodes manually Oracle defaults the names to 'nodename', 'nodename-priv', 'nodename-vip' Actions Confirm the Cluster Name selected is acceptable Confirm the Details for the current node are OK. The defaults are: Public Node Name : must resolve via hosts and or DNS to the public IP address and must be live Private Node Name : must resolve via hosts to the interconnect IP address and must be live Virtual Host Name : must resolve via hosts and or DNS to a new IP address and must not be live If these are not correct select the node entry and click Edit... to modify OR Click Add... to add more nodes to the cluster

4.2. Oracle Clusterware Install

28

 Notes Here you specify the details of the node you wish to add to the cluster nodes list Actions Enter the new node details Public Node Name : must resolve via hosts and or DNS to the public IP address and must be live Private Node Name : must resolve via hosts to the interconnect IP address and must be live Virtual Host Name : must resolve via hosts and or DNS to a new IP address and must not be live Click OK to return to the node list for the cluster

4.2. Oracle Clusterware Install

29

 Notes If you have more nodes repeast the Add... cycle Actions Click Next

Notes 4.2. Oracle Clusterware Install 30

 The installer lists all the Network adapters. You should have one Adapter correctly identified as type 'Public' and at least one adapter correctly identified as type 'Private'. The installer will try and guess the use of an adapter based on the IP address bound. If it guesses incorrectly you must change the usage. Here it has guessed that all adapters are Private, which is incorrect. Actions Select the Adapter eth0 Click Edit...

Notes Here you specify the Interface type Oracle will use this specific network for Actions Select the Interface Type : Public radio button Click OK

4.2. Oracle Clusterware Install

31

 Notes Here you can see we have successfully configured the network adapter usage. Ideally you will have only 1 adapter set as public and 1 adapter set as private. Other adapters, if available, set to do not use. If you have multiple public or multiple private adapters it is better to team them at the OS adapter driver level before commencing the install.Type : Private Actions Click Next

4.2. Oracle Clusterware Install

32

 Notes Here we specify the shared storage devices that will be used by Oracle Clusterware. Ideally you will have 2 devices for the OCR, Oracle will mirror to these devices to protect you from a single OCR device failure. Also you will have an additional 3 vote devices to protect your cluster from the failure of a single vote device. Actions Select the Normal radio button Enter the device to be used for the First OCR Enter the device to be used for the Second OCR Click Next

4.2. Oracle Clusterware Install

33

 Notes Next we specify the devices to be used for the Oracle Clusterware vote disks Actions Select the Normal radio button Enter the device to be used for the First vote disk Enter the device to be used for the Second vote disk Enter the device to be used for the Third vote disk Click Next

4.2. Oracle Clusterware Install

34

 Notes the installer lists a summary of the planned actions Actions Click Install

Notes 4.2. Oracle Clusterware Install 35

 the installer installs the software onto the local node Actions none required

Notes the installer installs the software onto the remote node Actions none required

4.2. Oracle Clusterware Install

36

 Notes The installer requires commands to be run as root on each of the nodes Be sure to follow directions carefully, running scripts one at a time, on each node Action On the first node open a root shell window and run the root.sh script
bash-3.00# /scratch/11.1.0/crs/root.sh WARNING: directory '/scratch/11.1.0' is not owned by root Checking to see if Oracle CRS stack is already configured Setting the permissions on OCR backup directory Setting up Network socket directories Oracle Cluster Registry configuration upgraded successfully The directory '/scratch/11.1.0' is not owned by root. Changing owner to root Successfully accumulated necessary OCR kegs. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node <nodenunber>: <nodenane> <private interconnect name> <hostnane> node 1: stnsp001 stnsp001-rac stnsp001 node 2: stnsp002 stnsp002-rac stnsp002 Creating OCR kegs for user 'root', privgrp 'root'.. Operation successful. Now formatting voting device: /dev/sdb5 Now formatting voting device: /dev/sdc5 Now formatting voting devices /dev/sdd5 Format of 3 voting devices complete. Startup will be queued to init within 30 seconds. Adding daemons to inittab Expecting the CRS daemons to be up within 600 seconds. Cluster Synchronization Services is active on these nodes. stnsp001 Cluster Synchronization Services is inactive on these nodes. stnsp002 Local node checking complete. Run root.sh on remaining nodes to start CRS daemons.

Notes The output from the root.sh script should be similar to this 4.2. Oracle Clusterware Install 37

 here the Oracle Clusterware is configured This may take some time to run Action open a root shell on the first node run the identified command as root on the first node You must wait for the command to complete before continuing
bash-3.00# /scratch/11.1.0/crs/root.sh WARNING: directory '/scratch/11.1.0' is not owned by root Checking to see if Oracle CRS stack is already configured Setting the permissions on OCR backup directory Setting up Network socket directories Oracle Cluster Registry configuration upgraded successfully The directory '/scratch/11.1.0' is not owned be root. Changing owner to root clscfg: EXISTING configuration version 4 detected. clscfg: version 4 is 11 Release 1. Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node <nodenunber>: <nodenane> <private interconnect name> <hostnane> node 1: stnsp001 stnsp001-rac stnsp001 node 2: stnsp002 stnsp002-rac stnsp002 clscfg: Arguments check out successfully. NO KEYS WERE WRITTEN. Supply -force parameter to override. -force is destructive and will destroy any previous cluster configuration. Oracle Cluster Registry for cluster has already been initialized Startup will be queued to init within 30 seconds. Adding deamons to inittab Expecting the CRS daemons to be up within 600 seconds. Cluster Synchronization Services is active on these nodes. stnsp001 stnsp002 Cluster Synchronization Services is active on all the nodes. Waiting for the Oracle CRSD and EVMD to start Waiting for the Oracle CRSD and EVMD to start Oracle CRS stack installed and running under init(lM) Running vipca(silent) for configuring nodeapps Creating Creating Creating Creating Creating Creating Done. VIP GSD ONS VIP GSD ONS application application application application application application resource resource resource resource resource resource on on on on on on (2) (2) (2) (2) (2) (2) nodes... nodes... nodes... nodes... nodes... nodes...

Notes The output from the root.sh script should be similar to this here the Oracle Clusterware is configured This may take some time to run Action open a root shell on the second node run the identified command as root on the second node You must wait for the command to complete before continuing

4.2. Oracle Clusterware Install

38

 Notes You can now contnue with the install Action return to the installer Click OK

4.2. Oracle Clusterware Install

39

 Notes a series of configuration assistants are run at the end of the install Actions none required

Notes The installer has completed installation and configuration of Oracle Clusterware Actions Click Exit

4.2. Oracle Clusterware Install

40

 Actions Click Yes

4.2.0.1. Verify cluster resources are online

[oracle11@stnsp001 bin]$ cd /scratch/11.1.0/crs/bin [oracle11@stnsp001 bin]$ ./crs_stat -t Name Type Target State Host -----------------------------------------------------------ora....001.gsd application ONLINE ONLINE stnsp001 ora....001.ons application ONLINE ONLINE stnsp001 ora....001.vip application ONLINE ONLINE stnsp001 ora....002.gsd application ONLINE ONLINE stnsp002 ora....002.ons application ONLINE ONLINE stnsp002 ora....002.vip application ONLINE ONLINE stnsp002 Notes You can see the resources configured inside Oracle Clusterware Action change directory to Oracle Clusterware home bin directory run the ./crs_stat -t comand

4.2.0.2. Use CVU to verify the Oracle Clusterware install

Verify the Oracle Clusterware installation using the Oracle Cluster Verification Utility: ./cluvfy stage -post crsinst -n stnsp001,stnsp002 -verbose

4.2.0.1. Verify cluster resources are online

41

Show CVU -post crsinst output

Hide CVU output

-verbose

./cluvfy stage -post crsinst -n stnsp001,stnsp002 Performing post-checks for cluster services setup Checking node reachability...

Check: Node reachability from node "stnsp001" Destination Node Reachable? ------------------------------------ -----------------------stnsp001 yes stnsp002 yes Result: Node reachability check passed from node "stnsp001".

Checking user equivalence... Check: User equivalence for user "oracle11" Node Name Comment ------------------------------------ -----------------------stnsp002 passed stnsp001 passed Result: User equivalence check passed for user "oracle11". Checking Cluster manager integrity...

Checking CSS daemon... Node Name Status ------------------------------------ -----------------------stnsp002 running stnsp001 running Result: Daemon status check passed for "CSS daemon". Cluster manager integrity check passed. Checking cluster integrity... Node Name -----------------------------------stnsp001 stnsp002 Cluster integrity check passed

Checking OCR integrity... Checking the absence of a non-clustered configuration... All nodes free of non-clustered, local-only configurations. Uniqueness check for OCR device passed. Checking the version of OCR... OCR of correct Version "2" exists. Checking data integrity of OCR... Data integrity check for OCR passed. OCR integrity check passed. Checking CRS integrity... Checking daemon liveness... Check: Liveness for "CRS daemon" Node Name Running ------------------------------------ -----------------------stnsp002 yes stnsp001 yes Result: Liveness check passed for "CRS daemon".

Checking daemon liveness...

4.2.0.2. Use CVU to verify the Oracle Clusterware install

42

Check: Liveness for "CSS daemon" Node Name Running ------------------------------------ -----------------------stnsp002 yes stnsp001 yes Result: Liveness check passed for "CSS daemon". Checking daemon liveness... Check: Liveness for "EVM daemon" Node Name Running ------------------------------------ -----------------------stnsp002 yes stnsp001 yes Result: Liveness check passed for "EVM daemon". Liveness of all Node Name -----------stnsp002 stnsp001 the daemons CRS daemon -----------------------yes yes

CSS daemon -----------------------yes yes

EVM daemon ---------yes yes

Checking CRS health... Check: Health of CRS Node Name -----------------------------------stnsp002 stnsp001 Result: CRS health check passed. CRS integrity check passed. Checking node application existence... Checking existence of VIP node application Node Name Required Status ------------ ------------------------ -----------------------stnsp002 yes exists stnsp001 yes exists Result: Check passed. Checking existence of ONS node application Node Name Required Status ------------ ------------------------ -----------------------stnsp002 no exists stnsp001 no exists Result: Check passed. Checking existence of GSD node application Node Name Required Status ------------ ------------------------ -----------------------stnsp002 no exists stnsp001 no exists Result: Check passed.

CRS OK? -----------------------yes yes

Comment ---------passed passed

Comment ---------passed passed

Comment ---------passed passed

Post-check for cluster services setup was successful.

Show all

Hide all

5. Oracle Clusterware patching

At this point we have installed Oracle Clusterware 11.1.0.6. In this section we will patch the Oracle Clusterware to the latest release of Oracle 11gR1 - 11.1.0.7. The patchset can be downloaded from metalink. Note: The Oracle Clusterware software must be at the same or newer level as any other ORACLE_HOME directory. Therefore, you should always upgrade Oracle Clusterware before you upgrade any other 5. Oracle Clusterware patching 43

ORACLE_HOME directory. This patch does not allow you to upgrade other ORACLE_HOME directories, before you upgrade Oracle Clusterware home directory. We will apply this patchset in a non rolling upgrade fashion. Before we start we can query the clusterware versions
#/scratch/11.1.0/crs/bin/crsctl query crs softwareversion Oracle Clusterware version on node [stnsp001] is [11.1.0.6.0] #/scratch/11.1.0/crs/bin/crsctl query crs activeversion Oracle Clusterware active version on the cluster is [11.1.0.6.0]

Enter the following commands to start Oracle Universal Installer, where patchset_directory is the directory where you unpacked the patch set software:
$ cd patchset_directory/Disk1 $ ./runInstaller

Notes The installer appears Actions Click Next

5. Oracle Clusterware patching

44

 Notes You should ensure that the Clusterware home is selected in the first drop down list box The installer should default the directory to the correct location Actions Ensure the correct home is selected Click Next

5. Oracle Clusterware patching

45

 Notes the installer detects that this is a clustered home and automatically selects all the nodes in the cluster Actions Click Next

Notes Some parameters are validated by the installer Actions Click Next

5. Oracle Clusterware patching

46

 Notes This is a summary of the actions the installer will complete Actions Click Install

Notes 5. Oracle Clusterware patching 47

 the installer stages the patch on all the nodes in the cluster Actions No action required

Notes At the end the installer lists the mandatory steps that must be completed to apply this patch Actions Log in as the root user and enter the following command to shut down the Oracle Clusterware:
# CRS_home/bin/crsctl stop crs

Run the root111.sh script. It will automatically start the Oracle Clusterware on the patched node:
# CRS_home/install/root111.sh

Show root111.sh output (all nodes except last)

Hide root111.sh output (all nodes except last)

bash-3.00# /scratch/11.1.0/crs/install/root111.sh Creating pre-patch directory for saving pre-patch clusterware files Completed patching clusterware files to /opt/crs Relinking some shared libraries. Relinking of patched files is complete. Preparing to recopy patched init and RC scripts. Recopying init and RC scripts. Startup will be queued to init within 30 seconds. Starting up the CRS daemons. Waiting for the patched CRS daemons to start. This may take a while on some systems. . 11107 patch successfully applied. clscfg: EXISTING configuration version 4 detected. clscfg: version 4 is 11 Release 1.

5. Oracle Clusterware patching

48

Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node <nodenumber>: <nodename> <private interconnect name> <hostname> node 1: stnsp001 stnsp001-priv stnsp001 Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. clscfg -upgrade completed successfully Creating '/scratch/11.1.0/crs/install/paramfile.crs' with data used for CRS configuration Setting CRS configuration values in /scratch/11.1.0/crs/install/paramfile.crs Setting cluster unique identifier Restarting Oracle clusterware Stopping Oracle clusterware Stopping resources. This could take several minutes. Successfully stopped Oracle Clusterware resources Stopping Cluster Synchronization Services. Shutting down the Cluster Synchronization Services daemon. Shutdown request successfully issued. Waiting for Cluster Synchronization Services daemon to stop Cluster Synchronization Services daemon has stopped Starting Oracle clusterware Attempting to start Oracle Clusterware stack Waiting for Cluster Synchronization Services daemon to start Waiting for Cluster Synchronization Services daemon to start Waiting for Cluster Synchronization Services daemon to start Waiting for Cluster Synchronization Services daemon to start Cluster Synchronization Services daemon has started Event Manager daemon has started Cluster Ready Services daemon has started

Show root111.sh output on last node

Hide root111.sh output on last node

bash-3.00# /scratch/11.1.0/crs/install/root111.sh Creating pre-patch directory for saving pre-patch clusterware files Completed patching clusterware files to /opt/crs Relinking some shared libraries. Relinking of patched files is complete. Preparing to recopy patched init and RC scripts. Recopying init and RC scripts. Startup will be queued to init within 30 seconds. Starting up the CRS daemons. Waiting for the patched CRS daemons to start. This may take a while on some systems. . 11107 patch successfully applied. clscfg: EXISTING configuration version 4 detected. clscfg: version 4 is 11 Release 1. Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node <nodenumber>: <nodename> <private interconnect name> <hostname> node 2: stnsp002 stnsp002-priv stnsp002 Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. clscfg -upgrade completed successfully Creating '/scratch/11.1.0/crs/install/paramfile.crs' with data used for CRS configuration Setting CRS configuration values in /scratch/11.1.0/crs/install/paramfile.crs

Continue with these two steps on all nodes, one by one, this will achieve a rolling upgrade of the Oracle Clusterware. When done, verify the Oracle Clusterware is running on all nodes before exiting the installer.
$ CRS_home/bin/crsctl check crs Cluster Synchronization Services appears healthy Cluster Ready Services appears healthy Event Manager appears healthy # /scratch/11.1.0/crs/bin/crsctl query crs softwareversion Oracle Clusterware version on node [stnsp001] is [11.1.0.7.0] # /scratch/11.1.0/crs/bin/crsctl query crs activeversion Oracle Clusterware active version on the cluster is [11.1.0.7.0]

5. Oracle Clusterware patching

49

 Notes Once the Oracle Clusterware is running on all nodes, exit the installer Actions Click Yes

5.1. Oracle Clusterware patching complete

6. Oracle ASM Home Software Install

6.1. CVU check before creating ASM instance
Verify the cluster is configured correctly for an instance creation ./runcluvfy.sh stage -pre dbinst -n stnsp001,stnsp002 -r 11gR1 -verbose Show CVU -pre dbinst output Hide CVU output

./runcluvfy.sh stage -pre dbinst -n stnsp001,stnsp002 -r 11gR1 -verbose Performing pre-checks for database installation Checking node reachability... Check: Node reachability from node "stnsp001" Destination Node Reachable? ------------------------------------ -----------------------stnsp001 yes stnsp002 yes Result: Node reachability check passed from node "stnsp001".

Checking user equivalence...

5.1. Oracle Clusterware patching complete

50

Check: User equivalence for user "oracle11" Node Name Comment ------------------------------------ -----------------------stnsp002 passed stnsp001 passed Result: User equivalence check passed for user "oracle11". Checking administrative privileges... Check: Existence of user "oracle11" Node Name User Exists Comment ------------ ------------------------ -----------------------stnsp002 yes passed stnsp001 yes passed Result: User existence check passed for "oracle11". Check: Existence of group "oinstall" Node Name Status Group ID ------------ ------------------------ -----------------------stnsp002 exists 1422 stnsp001 exists 1422 Result: Group existence check passed for "oinstall". Check: Membership of user "oracle11" in group "oinstall" [as Primary] Node Name User Exists Group Exists User in Group Primary Comment ---------------- ------------ ------------ ------------ ------------ -----------stnsp002 yes yes yes yes passed stnsp001 yes yes yes yes passed Result: Membership check for user "oracle11" in group "oinstall" [as Primary] passed. Check: Existence of group "dba" Node Name Status Group ID ------------ ------------------------ -----------------------stnsp002 exists 8500 stnsp001 exists 8500 Result: Group existence check passed for "dba". Check: Membership of user "oracle11" in group "dba" Node Name User Exists Group Exists User in Group Primary ---------------- ------------ ------------ ------------ -----------stnsp002 yes yes yes no stnsp001 yes yes yes no Result: Membership check for user "oracle11" in group "dba" passed. Administrative privileges check passed. Checking node connectivity...

Comment -----------passed passed

Interface information for node "stnsp002" Interface Name IP Address Subnet ---------------- ------------ -----------eth0 10.137.8.216 10.137.8.0 eth0 10.137.8.234 10.137.8.0 eth1 10.137.24.207 10.137.24.0 eth2 10.137.20.172 10.137.20.0

Subnet Gateway Default Gateway Hardware Address ------------ ------------ -----------0.0.0.0 10.137.8.1 00:0E:0C:08:06:4E 0.0.0.0 10.137.8.1 00:0E:0C:08:06:4E 0.0.0.0 10.137.8.1 00:0E:0C:08:06:4F 0.0.0.0 10.137.8.1 00:04:23:A6:CD:BD

Interface information for node "stnsp001" Interface Name IP Address Subnet ---------------- ------------ -----------eth0 10.137.8.215 10.137.8.0 eth0 10.137.8.233 10.137.8.0 eth1 10.137.24.206 10.137.24.0 eth2 10.137.20.171 10.137.20.0

Subnet Gateway Default Gateway Hardware Address ------------ ------------ -----------0.0.0.0 10.137.8.1 00:0E:0C:08:08:F2 0.0.0.0 10.137.8.1 00:0E:0C:08:08:F2 0.0.0.0 10.137.8.1 00:0E:0C:08:08:F3 0.0.0.0 10.137.8.1 00:04:23:A6:CD:61

Check: Node connectivity of subnet "10.137.8.0" Source Destination ------------------------------ -----------------------------stnsp002:eth0 stnsp002:eth0 stnsp002:eth0 stnsp001:eth0 stnsp002:eth0 stnsp001:eth0 stnsp002:eth0 stnsp001:eth0 stnsp002:eth0 stnsp001:eth0 stnsp001:eth0 stnsp001:eth0

Connected? ---------------yes yes yes yes yes yes

6.1. CVU check before creating ASM instance

51

Result: Node connectivity check passed for subnet "10.137.8.0" with node(s) stnsp002,stnsp001. Check: Node connectivity of subnet "10.137.24.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------stnsp002:eth1 stnsp001:eth1 yes Result: Node connectivity check passed for subnet "10.137.24.0" with node(s) stnsp002,stnsp001. Check: Node connectivity of subnet "10.137.20.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------stnsp002:eth2 stnsp001:eth2 yes Result: Node connectivity check passed for subnet "10.137.20.0" with node(s) stnsp002,stnsp001. Interfaces found on subnet "10.137.8.0" that are likely candidates for a private interconnect: stnsp002 eth0:10.137.8.216 eth0:10.137.8.234 stnsp001 eth0:10.137.8.215 eth0:10.137.8.233 Interfaces found on subnet "10.137.24.0" that are likely candidates for a private interconnect: stnsp002 eth1:10.137.24.207 stnsp001 eth1:10.137.24.206 Interfaces found on subnet "10.137.20.0" that are likely candidates for a private interconnect: stnsp002 eth2:10.137.20.172 stnsp001 eth2:10.137.20.171 WARNING: Could not find a suitable set of interfaces for VIPs. Result: Node connectivity check passed.

Checking system requirements for 'database'... Check: Total memory Node Name Available ------------ -----------------------stnsp002 5.95GB (6234256KB) stnsp001 5.95GB (6234256KB) Result: Total memory check passed. Check: Free disk space in "/tmp" dir Node Name Available ------------ -----------------------stnsp002 240.93GB (252637044KB) stnsp001 240.93GB (252637044KB) Result: Free disk space check passed. Check: Swap space Node Name Available ------------ -----------------------stnsp002 5.91GB (6192456KB) stnsp001 5.91GB (6192456KB) Result: Swap space check passed. Check: System architecture Node Name Available ------------ -----------------------stnsp002 i686 stnsp001 i686 Result: System architecture check passed. Check: Kernel version Node Name Available ------------ -----------------------stnsp002 2.6.9-34.0.1.0.11.ELsmp stnsp001 2.6.9-34.0.1.0.11.ELsmp Result: Kernel version check passed.

Required -----------------------1GB (1048576KB) 1GB (1048576KB)

Comment ---------passed passed

Required -----------------------400MB (409600KB) 400MB (409600KB)

Comment ---------passed passed

Required -----------------------1.5GB (1572864KB) 1.5GB (1572864KB)

Comment ---------passed passed

Required -----------------------i686 i686

Comment ---------passed passed

Required -----------------------2.6.9 2.6.9

Comment ---------passed passed

Check: Package existence for "make-3.80" Node Name Status ------------------------------ -----------------------------stnsp002 make-3.80-5 stnsp001 make-3.80-5 Result: Package existence check passed for "make-3.80".

Comment ---------------passed passed

6.1. CVU check before creating ASM instance

52

Check: Package existence for "binutils-2.15.92.0.2" Node Name Status Comment ------------------------------ ------------------------------ ---------------stnsp002 binutils-2.15.92.0.2-18 passed stnsp001 binutils-2.15.92.0.2-18 passed Result: Package existence check passed for "binutils-2.15.92.0.2". Check: Package existence for "gcc-3.4.5" Node Name Status ------------------------------ -----------------------------stnsp002 gcc-3.4.5-2 stnsp001 gcc-3.4.5-2 Result: Package existence check passed for "gcc-3.4.5". Check: Package existence for "libaio-0.3.105" Node Name Status ------------------------------ -----------------------------stnsp002 libaio-0.3.105-2 stnsp001 libaio-0.3.105-2 Result: Package existence check passed for "libaio-0.3.105".

Comment ---------------passed passed

Comment ---------------passed passed

Check: Package existence for "libaio-devel-0.3.105" Node Name Status Comment ------------------------------ ------------------------------ ---------------stnsp002 libaio-devel-0.3.105-2 passed stnsp001 libaio-devel-0.3.105-2 passed Result: Package existence check passed for "libaio-devel-0.3.105". Check: Package existence for "libstdc++-3.4.5" Node Name Status ------------------------------ -----------------------------stnsp002 libstdc++-3.4.5-2 stnsp001 libstdc++-3.4.5-2 Result: Package existence check passed for "libstdc++-3.4.5".

Comment ---------------passed passed

Check: Package existence for "elfutils-libelf-devel-0.97" Node Name Status Comment ------------------------------ ------------------------------ ---------------stnsp002 elfutils-libelf-devel-0.97-5 passed stnsp001 elfutils-libelf-devel-0.97-5 passed Result: Package existence check passed for "elfutils-libelf-devel-0.97". Check: Package existence for "sysstat-5.0.5" Node Name Status ------------------------------ -----------------------------stnsp002 sysstat-5.0.5-7.rhel4 stnsp001 sysstat-5.0.5-7.rhel4 Result: Package existence check passed for "sysstat-5.0.5". Check: Package existence for "libgcc-3.4.5" Node Name Status ------------------------------ -----------------------------stnsp002 libgcc-3.4.5-2 stnsp001 libgcc-3.4.5-2 Result: Package existence check passed for "libgcc-3.4.5".

Comment ---------------passed passed

Comment ---------------passed passed

Check: Package existence for "libstdc++-devel-3.4.5" Node Name Status Comment ------------------------------ ------------------------------ ---------------stnsp002 libstdc++-devel-3.4.5-2 passed stnsp001 libstdc++-devel-3.4.5-2 passed Result: Package existence check passed for "libstdc++-devel-3.4.5". Check: Package existence for "unixODBC-2.2.11" Node Name Status ------------------------------ -----------------------------stnsp002 unixODBC-2.2.11-1.RHEL4.1 stnsp001 unixODBC-2.2.11-1.RHEL4.1 Result: Package existence check passed for "unixODBC-2.2.11".

Comment ---------------passed passed

Check: Package existence for "unixODBC-devel-2.2.11" Node Name Status Comment ------------------------------ ------------------------------ ---------------stnsp002 unixODBC-devel-2.2.11-1.RHEL4.1 passed stnsp001 unixODBC-devel-2.2.11-1.RHEL4.1 passed Result: Package existence check passed for "unixODBC-devel-2.2.11".

6.1. CVU check before creating ASM instance

53

Check: Package existence for "glibc-2.3.4-2.19" Node Name Status ------------------------------ -----------------------------stnsp002 glibc-2.3.4-2.19 stnsp001 glibc-2.3.4-2.19 Result: Package existence check passed for "glibc-2.3.4-2.19". Check: Kernel parameter for "semmsl" Node Name Configured ------------ -----------------------stnsp002 250 stnsp001 250 Result: Kernel parameter check passed for Check: Kernel parameter for "semmns" Node Name Configured ------------ -----------------------stnsp002 32000 stnsp001 32000 Result: Kernel parameter check passed for Check: Kernel parameter for "semopm" Node Name Configured ------------ -----------------------stnsp002 100 stnsp001 100 Result: Kernel parameter check passed for Check: Kernel parameter for "semmni" Node Name Configured ------------ -----------------------stnsp002 128 stnsp001 128 Result: Kernel parameter check passed for Check: Kernel parameter for "shmall" Node Name Configured ------------ -----------------------stnsp002 2097152 stnsp001 2097152 Result: Kernel parameter check passed for Check: Kernel parameter for "shmmax" Node Name Configured ------------ -----------------------stnsp002 2147483648 stnsp001 2147483648 Result: Kernel parameter check passed for Check: Kernel parameter for "shmmni" Node Name Configured ------------ -----------------------stnsp002 4096 stnsp001 4096 Result: Kernel parameter check passed for Check: Kernel parameter for "file-max" Node Name Configured ------------ -----------------------stnsp002 6553600 stnsp001 6553600 Result: Kernel parameter check passed for

Comment ---------------passed passed

Required -----------------------250 250 "semmsl".

Comment ---------passed passed

Required -----------------------32000 32000 "semmns".

Comment ---------passed passed

Required -----------------------100 100 "semopm".

Comment ---------passed passed

Required -----------------------128 128 "semmni".

Comment ---------passed passed

Required -----------------------2097152 2097152 "shmall".

Comment ---------passed passed

Required -----------------------536870912 536870912 "shmmax".

Comment ---------passed passed

Required -----------------------4096 4096 "shmmni".

Comment ---------passed passed

Required -----------------------65536 65536 "file-max".

Comment ---------passed passed

Check: Kernel parameter for "rmem_default" Node Name Configured Required ------------ ------------------------ -----------------------stnsp002 4194304 4194304 stnsp001 4194304 4194304 Result: Kernel parameter check passed for "rmem_default". Check: Kernel parameter for "rmem_max" Node Name Configured ------------ -----------------------stnsp002 4194304 stnsp001 4194304

Comment ---------passed passed

Required -----------------------4194304 4194304

Comment ---------passed passed

6.1. CVU check before creating ASM instance

54

Result: Kernel parameter check passed for "rmem_max". Check: Kernel parameter for "wmem_default" Node Name Configured Required ------------ ------------------------ -----------------------stnsp002 524288 262144 stnsp001 524288 262144 Result: Kernel parameter check passed for "wmem_default". Check: Kernel parameter for "wmem_max" Node Name Configured Required ------------ ------------------------ -----------------------stnsp002 1048576 262144 stnsp001 1048576 262144 Result: Kernel parameter check passed for "wmem_max". Check: Group existence for "dba" Node Name Status Comment ------------ ------------------------ -----------------------stnsp002 exists passed stnsp001 exists passed Result: Group existence check passed for "dba". Check: User existence for "nobody" Node Name Status Comment ------------ ------------------------ -----------------------stnsp002 exists passed stnsp001 exists passed Result: User existence check passed for "nobody". System requirement passed for 'database' Checking CRS integrity... Checking daemon liveness... Check: Liveness for "CRS daemon" Node Name Running ------------------------------------ -----------------------stnsp002 yes stnsp001 yes Result: Liveness check passed for "CRS daemon". Checking daemon liveness... Check: Liveness for "CSS daemon" Node Name Running ------------------------------------ -----------------------stnsp002 yes stnsp001 yes Result: Liveness check passed for "CSS daemon". Checking daemon liveness... Check: Liveness for "EVM daemon" Node Name Running ------------------------------------ -----------------------stnsp002 yes stnsp001 yes Result: Liveness check passed for "EVM daemon". Liveness of all Node Name -----------stnsp002 stnsp001 the daemons CRS daemon -----------------------yes yes

Comment ---------passed passed

Comment ---------passed passed

CSS daemon -----------------------yes yes

EVM daemon ---------yes yes

Checking CRS health... Check: Health of CRS Node Name -----------------------------------stnsp002 stnsp001 Result: CRS health check passed.

CRS OK? -----------------------yes yes

6.1. CVU check before creating ASM instance

55

CRS integrity check passed. Pre-check for database installation was successful.

Show all

Hide all

6.2. Oracle ASM Home Software Install

Start the installer by running "runInstaller" from the staged installation media.
./runInstaller

In this section we establish the Oracle software for the ASM home.

Notes Here we will create a ASM Software home on all the nodes in the cluster Actions Select the Oracle Database 11g radio button Click Next

6.2. Oracle ASM Home Software Install

56

 Notes Here we can choose the type of database software to be installed Actions Select the Enterprise Edition radio button Click Next

6.2. Oracle ASM Home Software Install

57

 Notes Here we specify the location of various components. An 11g install makes more use of the ORACLE_BASE. Most logs will be stored in sub directories under the oracle base. The oracle base will be common to all installs. Modify as required. Also this is where you specify the location of the ASM software home. I usually modify the Home name and Home Path to include the word ASM. This makes it easier to identify later on. If you change the path you should ensure that you do not use the exact same path as the Oracle Clusterware home Actions Confirm entries are OK Click Next

Notes The installer has detected the presence of Oracle Clusterware and uses this to populate this dialog box. To build a cluster which includes all nodes you must ensure that there are check-boxes next to the node names Actions Click Select All Click Next

6.2. Oracle ASM Home Software Install

58

 Notes The installer will then complete some Product-Specific Prerequisite checks. These should all pass OK as you have already run the CVU check Actions Click Next

6.2. Oracle ASM Home Software Install

59

 Notes We are going to install a Software only home and then subsequently configure the software Actions Select the Install Software Only radio button Click Next

Notes Assign the Unix groups for the privileged ASM users (default is oinstall) Action Click Next

6.2. Oracle ASM Home Software Install

60

 Notes Here we can see a summary of the install. Actions Click Install

Notes 6.2. Oracle ASM Home Software Install 61

 Here the installer copies the software to all nodes in the cluster Actions none required

Notes The installer pauses, some scripts need to be run as root on all nodes of the cluster Be sure to follow directions carefully, running scripts one at a time, on each node Action open a shell window on each node run the root.sh script
bash-3.00# /scratch/product/11.1.0/asm/root.sh Running Oracle 11g root.sh script... The following environment variables are set as: ORACLE_OWNER= oracle11 ORACLE_HOME= /scratch/product/11.1.0/asm Enter the full pathname of the local bin directory: [/usr/local/bin]: /usr/local/bin is read only. Continue without copy (y/n) or retry (r)? [y]: y Warning: /usr/local/bin is read only. No files will be copied.

Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed. Finished product-specific root actions.

Notes

6.2. Oracle ASM Home Software Install

62

 The output from the root.sh script should be similar to this it should only take a few seconds to run on each node Action run the scripts on all nodes then return to the installer and Click OK

Notes After the software install completes you will see this End of Installation dialog Actions Click Exit

6.2. Oracle ASM Home Software Install

63

 Notes Are you sure ? Actions Click Yes

6.2.1. Oracle ASM software home has been installed

7. Oracle ASM Software Home Patching

Once ASM software has been installed, the next step in the process is to apply the 11.1.0.7 patchset to the ASM ORACLE_HOME.

7.1. Start the OUI for Oracle ASM Software Home Patching
Enter the following commands to start Oracle Universal Installer, where patchset_directory is the directory where you unpacked the patch set software:
$ cd patchset_directory/Disk1 $ ./runInstaller

6.2.1. Oracle ASM software home has been installed

64

 Notes the Universal Installer screen appears Action On the Welcome screen, click Next.

Notes 7.1. Start the OUI for Oracle ASM Software Home Patching 65

 Specify the name and the location of the asm home Action Check that the name and location are correct

Notes: here you can specify your metalik credentials for this install. If you leave both fields blank you can opt out of notifications (see next screen). Action Enter your email address Enter your Metalink password Click Next

7.1. Start the OUI for Oracle ASM Software Home Patching

66

 Notes:* If you leave both fields blank, you will opt out of notifications and get the above screen. Action Click Yes

Notes 7.1. Start the OUI for Oracle ASM Software Home Patching 67

 the installer detects that this is a clustered home and automatically selects all the nodes in the cluster Actions Click Next

Notes Some parameters are validated by the installer Action Click Next

7.1. Start the OUI for Oracle ASM Software Home Patching

68

 Notes This is a summary of the actions the installer will complete Actions Click Install

Notes 7.1. Start the OUI for Oracle ASM Software Home Patching 69

 The installer copies the patch to all the nodes in the cluster Actions No action required

Notes The installer pauses, root.sh need to be run as root on both nodes of the cluster Action open a shell window on each node and run root.sh (one after the other) then return to the installer and Click OK
bash-3.00# /scratch/product/app/oracle11/product/11.1.0/asm/root.sh Running Oracle 11g root.sh script... The following environment variables are set as: ORACLE_OWNER= oracle11 ORACLE_HOME= /scratch/product/11.1.0/asm Enter the full pathname of the local bin directory: [/usr/local/bin]: /usr/local/bin is read only. Continue without copy (y/n) or retry (r)? [y]: y Warning: /usr/local/bin is read only. No files will be copied.

Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed. Finished product-specific root actions.

Notes The output from the root.sh script should be similar to this 7.1. Start the OUI for Oracle ASM Software Home Patching 70

 it should only take a few seconds to run on each node Action run the scripts on all nodes then return to the installer and Click OK

Notes After the software install completes you will see this End of Installation dialog, exit the installer Actions Click Yes

8. Oracle RAC Database Listener Creation

8.1. Create Node specific network listeners
The Oracle network listeners traditionally run from the ASM home. Here we are going to create the listeners using netca from the ASM home.
[oracle11@stnsp001 bin]$ cd /scratch/product/11.1.0/asm/bin [oracle11@stnsp001 bin]$ ./netca

Notes run the Network Configuration Assistant Action Set environment variable ORACLE_HOME to ASM home location 8. Oracle RAC Database Listener Creation 71

 change directory to the ASM home bin directory run ./netca

Notes Netca detects that the Oracle Clusterware layer is running and offers Cluster or Single Node configuration Actions Select the Cluster configuration radio button Click Next

Notes Netca uses Oracle Clusterware to determine all the nodes in the cluster Actions Click Select all nodes Click Next

8.1. Create Node specific network listeners

72

 Notes You get various options we need to configure listeners Actions Select the Listener configuration radio button Click Next

Notes We need to add a listener Actions Select the Add radio button Click Next

8.1. Create Node specific network listeners

73

 Notes Here you get the opportunity to name the listener Do not change this. The listeners will eventually be called LISTENER_nodename1 & LISTENER_nodename2. This is important for RAC Actions Click Next

Notes Oracle Net supports various network protocols, although TCP is the most common. Actions Ensure the Selected Protocols list includes TCP Click Next

8.1. Create Node specific network listeners

74

 Notes It is possible to choose a non-default Port I do not recommend this Actions Ensure the Use the Standard port number of 1521 radio button is set Click Next

Notes After configuring the node listeners you get the opportunity to configure more network components Actions Select the No radio button Click Next

8.1. Create Node specific network listeners

75

 Notes You get the opportunity to configre other networking components Action Click the Finish button to exit the tool

8.2. Verify the Listener resources are online

[oracle11@stnsp001 bin]$ cd /scratch/11.1.0/crs/bin [oracle11@stnsp001 bin]$ ./crs_stat -t Name Type Target State Host -----------------------------------------------------------ora....01.lsnr application ONLINE ONLINE stnsp001 ora....001.gsd application ONLINE ONLINE stnsp001 ora....001.ons application ONLINE ONLINE stnsp001 ora....001.vip application ONLINE ONLINE stnsp001 ora....02.lsnr application ONLINE ONLINE stnsp002 ora....002.gsd application ONLINE ONLINE stnsp002 ora....002.ons application ONLINE ONLINE stnsp002 ora....002.vip application ONLINE ONLINE stnsp002 Notes You can see the listener resources inside Oracle Clusterware Action change directory to the Oracle Clusterware home bin directory run the ./crs_stat -t comand

8.3. Completed Node specific network listeners creation

You have now completed the RAC cluster nodes Network listener configuration. A new managed resource will have been added to Oracle Clusterware OCR for each listener.

8.2. Verify the Listener resources are online

76

9. Oracle ASM Instance and diskgroup Creation

9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups using the Database Assistant
cd /scratch/product/11.1.0/asm/bin ./dbca

Notes We use the dbca from the ASM install to create the ASM instances Actions ensure the ORACLE_HOME environment variable is set to the ASM home directory run ./dbca from the ASM home bin directory

Notes dbca detects the Oracle Clusterware layer is running and offers to create either cluster or single instance database Actions Select the Oracle Real Application Clusters database radio button Click Next

9. Oracle ASM Instance and diskgroup Creation

77

 Notes We use dbca to create RAC databases and ASM instances Actions Select the Configure Automatic Storage Management radio button Click Next

9.1. Create ASM Instance and add the +DATA and +FLASH diskgroupsusing the Database Assistant

78

 Notes You need to make sure you create ASM instances on all the cluster nodes Actions Click Select All Click Next

9.1. Create ASM Instance and add the +DATA and +FLASH diskgroupsusing the Database Assistant

79

 Notes Here we specify the password for the ASM Oracle SYS user Actions Enter the SYS password Enter the same password for the Confirm SYS password Click Next

9.1. Create ASM Instance and add the +DATA and +FLASH diskgroupsusing the Database Assistant

80

 Notes dbca will create and start ASM instances Actions Click OK

9.1. Create ASM Instance and add the +DATA and +FLASH diskgroupsusing the Database Assistant

81

 Notes This should take a few seconds Actions no action

Notes ASM requires disks to be group together into diskgroups. This section will be used to create 2 disk groups +DATA and +FLASH Actions Click Create New

9.1. Create ASM Instance and add the +DATA and +FLASH diskgroupsusing the Database Assistant

82

 Notes At the moment no disks are visible Actions Click Change Discovery Path

9.1. Create ASM Instance and add the +DATA and +FLASH diskgroupsusing the Database Assistant

83

 Notes here we specifya filter to allow us to see the disks on the shared array Action Enter a filter to allow the installer to see the disks Click OK

Notes Here you can now see thr disksw are visible Action none requried

9.1. Create ASM Instance and add the +DATA and +FLASH diskgroupsusing the Database Assistant

84

 Notes Now we will assign disks to specific disk groups and create the DATA diskgroup Actions In the Disk Group Name enter DATA Select the External Redundancy radio button Select the Show All radio button Select the 6 disks to be used for the DATA diskgroup Click OK

9.1. Create ASM Instance and add the +DATA and +FLASH diskgroupsusing the Database Assistant

85

 Notes A progress message this should not take very long Actions none

9.1. Create ASM Instance and add the +DATA and +FLASH diskgroupsusing the Database Assistant

86

 Notes Here we can see the DATA diskgroup has been created and is mounted on 2/2 instances. We now need to create the FLASH diskgroup Actions Click Create New

Notes We need to allow the installer to see the disks we haev reserved for the FLASH disk group Action Click Change Discovery Path

9.1. Create ASM Instance and add the +DATA and +FLASH diskgroupsusing the Database Assistant

87

 Notes We need to modify the disk discovery string Action Modify the string Click OK

9.1. Create ASM Instance and add the +DATA and +FLASH diskgroupsusing the Database Assistant

88

* Notes Now we will assign disks to specific disk groups and create the FLASH diskgroup For a change we will create this diskgroup with Normal redundancy Actions In the Disk Group Name enter FLASH Select the Normal radio button Select the Show Candidate disks radio button Select the remaining disks allocated for the FLASH diskgroup Click OK

9.1. Create ASM Instance and add the +DATA and +FLASH diskgroupsusing the Database Assistant

89

 Notes A progress message this should not take very long Actions none

9.1. Create ASM Instance and add the +DATA and +FLASH diskgroupsusing the Database Assistant

90

 Notes Here we can see the DATA and FLASH diskgroups have been created and are mounted on 2/2 instances. This completed the ASM configuration. Actions Click Finish

Notes A confirmation dialog box Actions Click No

9.2. Verify ASM instances are online

[oracle11@stnsp001 bin]$ cd /scratch/11.1.0/crs/bin [oracle11@stnsp001 bin]$ ./crs_stat -t Name Type Target State Host -----------------------------------------------------------ora....SM1.asm application ONLINE ONLINE stnsp001 ora....01.lsnr application ONLINE ONLINE stnsp001 ora....001.gsd application ONLINE ONLINE stnsp001 ora....001.ons application ONLINE ONLINE stnsp001 ora....001.vip application ONLINE ONLINE stnsp001 ora....SM2.asm application ONLINE ONLINE stnsp002 ora....02.lsnr application ONLINE ONLINE stnsp002 ora....002.gsd application ONLINE ONLINE stnsp002 ora....002.ons application ONLINE ONLINE stnsp002 9.2. Verify ASM instances are online 91

ora....002.vip application Notes

ONLINE

ONLINE

stnsp002

Here we can see the asm instances now as part of Oracle Clusterware Action change directory to the Oracle Clusterware bin directory run ./crs_stat -t

9.3. Completed creation of ASM Instance and add of the +DATA and +FLASH diskgroups
Congratulations you have installed ASM, Network listeners and created 2 ASM diskgroups.

10. Oracle RAC Database Home Software Install

10.1. CVU check -pre Database Instance Install
Before installing the RAC Database software you should use the CVU utility to confirm the status of the cluster.
./runcluvfy.sh stage -pre dbinst -n stnsp001,stnsp002 -r 11gR1 -verbose

Note: If you executed the CVU -pre dbinst command during the ASM instance installation, you do not have the execute the command again. Show CVU -pre dbinst output Hide CVU output

./runcluvfy.sh stage -pre dbinst -n stnsp001,stnsp002 -r 11gR1 -verbose Performing pre-checks for database installation Checking node reachability... Check: Node reachability from node "stnsp001" Destination Node Reachable? ------------------------------------ -----------------------stnsp001 yes stnsp002 yes Result: Node reachability check passed from node "stnsp001".

Checking user equivalence... Check: User equivalence for user "oracle11" Node Name Comment ------------------------------------ -----------------------stnsp002 passed stnsp001 passed Result: User equivalence check passed for user "oracle11". Checking administrative privileges... Check: Existence of user "oracle11" Node Name User Exists Comment ------------ ------------------------ -----------------------stnsp002 yes passed stnsp001 yes passed Result: User existence check passed for "oracle11". Check: Existence of group "oinstall" Node Name Status ------------ ------------------------

Group ID ------------------------

9.3. Completed creation of ASM Instance and add of the +DATA and +FLASH diskgroups

92

stnsp002 exists 1422 stnsp001 exists 1422 Result: Group existence check passed for "oinstall". Check: Membership of user "oracle11" in group "oinstall" [as Primary] Node Name User Exists Group Exists User in Group Primary Comment ---------------- ------------ ------------ ------------ ------------ -----------stnsp002 yes yes yes yes passed stnsp001 yes yes yes yes passed Result: Membership check for user "oracle11" in group "oinstall" [as Primary] passed. Check: Existence of group "dba" Node Name Status Group ID ------------ ------------------------ -----------------------stnsp002 exists 8500 stnsp001 exists 8500 Result: Group existence check passed for "dba". Check: Membership of user "oracle11" in group "dba" Node Name User Exists Group Exists User in Group Primary ---------------- ------------ ------------ ------------ -----------stnsp002 yes yes yes no stnsp001 yes yes yes no Result: Membership check for user "oracle11" in group "dba" passed. Administrative privileges check passed. Checking node connectivity...

Comment -----------passed passed

Interface information for node "stnsp002" Interface Name IP Address Subnet ---------------- ------------ -----------eth0 10.137.8.216 10.137.8.0 eth0 10.137.8.234 10.137.8.0 eth1 10.137.24.207 10.137.24.0 eth2 10.137.20.172 10.137.20.0

Subnet Gateway Default Gateway Hardware Address ------------ ------------ -----------0.0.0.0 10.137.8.1 00:0E:0C:08:06:4E 0.0.0.0 10.137.8.1 00:0E:0C:08:06:4E 0.0.0.0 10.137.8.1 00:0E:0C:08:06:4F 0.0.0.0 10.137.8.1 00:04:23:A6:CD:BD

Interface information for node "stnsp001" Interface Name IP Address Subnet ---------------- ------------ -----------eth0 10.137.8.215 10.137.8.0 eth0 10.137.8.233 10.137.8.0 eth1 10.137.24.206 10.137.24.0 eth2 10.137.20.171 10.137.20.0

Subnet Gateway Default Gateway Hardware Address ------------ ------------ -----------0.0.0.0 10.137.8.1 00:0E:0C:08:08:F2 0.0.0.0 10.137.8.1 00:0E:0C:08:08:F2 0.0.0.0 10.137.8.1 00:0E:0C:08:08:F3 0.0.0.0 10.137.8.1 00:04:23:A6:CD:61

Check: Node connectivity of subnet "10.137.8.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------stnsp002:eth0 stnsp002:eth0 yes stnsp002:eth0 stnsp001:eth0 yes stnsp002:eth0 stnsp001:eth0 yes stnsp002:eth0 stnsp001:eth0 yes stnsp002:eth0 stnsp001:eth0 yes stnsp001:eth0 stnsp001:eth0 yes Result: Node connectivity check passed for subnet "10.137.8.0" with node(s) stnsp002,stnsp001. Check: Node connectivity of subnet "10.137.24.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------stnsp002:eth1 stnsp001:eth1 yes Result: Node connectivity check passed for subnet "10.137.24.0" with node(s) stnsp002,stnsp001. Check: Node connectivity of subnet "10.137.20.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------stnsp002:eth2 stnsp001:eth2 yes Result: Node connectivity check passed for subnet "10.137.20.0" with node(s) stnsp002,stnsp001. Interfaces found on subnet "10.137.8.0" that are likely candidates for a private interconnect: stnsp002 eth0:10.137.8.216 eth0:10.137.8.234 stnsp001 eth0:10.137.8.215 eth0:10.137.8.233 Interfaces found on subnet "10.137.24.0" that are likely candidates for a private interconnect:

10.1. CVU check -pre Database Instance Install

93

stnsp002 eth1:10.137.24.207 stnsp001 eth1:10.137.24.206 Interfaces found on subnet "10.137.20.0" that are likely candidates for a private interconnect: stnsp002 eth2:10.137.20.172 stnsp001 eth2:10.137.20.171 WARNING: Could not find a suitable set of interfaces for VIPs. Result: Node connectivity check passed.

Checking system requirements for 'database'... Check: Total memory Node Name Available ------------ -----------------------stnsp002 5.95GB (6234256KB) stnsp001 5.95GB (6234256KB) Result: Total memory check passed. Check: Free disk space in "/tmp" dir Node Name Available ------------ -----------------------stnsp002 209.06GB (219216588KB) stnsp001 240.93GB (252637044KB) Result: Free disk space check passed. Check: Swap space Node Name Available ------------ -----------------------stnsp002 5.91GB (6192456KB) stnsp001 5.91GB (6192456KB) Result: Swap space check passed. Check: System architecture Node Name Available ------------ -----------------------stnsp002 i686 stnsp001 i686 Result: System architecture check passed. Check: Kernel version Node Name Available ------------ -----------------------stnsp002 2.6.9-34.0.1.0.11.ELsmp stnsp001 2.6.9-34.0.1.0.11.ELsmp Result: Kernel version check passed.

Required -----------------------1GB (1048576KB) 1GB (1048576KB)

Comment ---------passed passed

Required -----------------------400MB (409600KB) 400MB (409600KB)

Comment ---------passed passed

Required -----------------------1.5GB (1572864KB) 1.5GB (1572864KB)

Comment ---------passed passed

Required -----------------------i686 i686

Comment ---------passed passed

Required -----------------------2.6.9 2.6.9

Comment ---------passed passed

Check: Package existence for "make-3.80" Node Name Status ------------------------------ -----------------------------stnsp002 make-3.80-5 stnsp001 make-3.80-5 Result: Package existence check passed for "make-3.80".

Comment ---------------passed passed

Check: Package existence for "binutils-2.15.92.0.2" Node Name Status Comment ------------------------------ ------------------------------ ---------------stnsp002 binutils-2.15.92.0.2-18 passed stnsp001 binutils-2.15.92.0.2-18 passed Result: Package existence check passed for "binutils-2.15.92.0.2". Check: Package existence for "gcc-3.4.5" Node Name Status ------------------------------ -----------------------------stnsp002 gcc-3.4.5-2 stnsp001 gcc-3.4.5-2 Result: Package existence check passed for "gcc-3.4.5". Check: Package existence for "libaio-0.3.105" Node Name Status ------------------------------ -----------------------------stnsp002 libaio-0.3.105-2 stnsp001 libaio-0.3.105-2

Comment ---------------passed passed

Comment ---------------passed passed

10.1. CVU check -pre Database Instance Install

94

Result: Package existence check passed for "libaio-0.3.105". Check: Package existence for "libaio-devel-0.3.105" Node Name Status Comment ------------------------------ ------------------------------ ---------------stnsp002 libaio-devel-0.3.105-2 passed stnsp001 libaio-devel-0.3.105-2 passed Result: Package existence check passed for "libaio-devel-0.3.105". Check: Package existence for "libstdc++-3.4.5" Node Name Status ------------------------------ -----------------------------stnsp002 libstdc++-3.4.5-2 stnsp001 libstdc++-3.4.5-2 Result: Package existence check passed for "libstdc++-3.4.5".

Comment ---------------passed passed

Check: Package existence for "elfutils-libelf-devel-0.97" Node Name Status Comment ------------------------------ ------------------------------ ---------------stnsp002 elfutils-libelf-devel-0.97-5 passed stnsp001 elfutils-libelf-devel-0.97-5 passed Result: Package existence check passed for "elfutils-libelf-devel-0.97". Check: Package existence for "sysstat-5.0.5" Node Name Status ------------------------------ -----------------------------stnsp002 sysstat-5.0.5-7.rhel4 stnsp001 sysstat-5.0.5-7.rhel4 Result: Package existence check passed for "sysstat-5.0.5". Check: Package existence for "libgcc-3.4.5" Node Name Status ------------------------------ -----------------------------stnsp002 libgcc-3.4.5-2 stnsp001 libgcc-3.4.5-2 Result: Package existence check passed for "libgcc-3.4.5".

Comment ---------------passed passed

Comment ---------------passed passed

Check: Package existence for "libstdc++-devel-3.4.5" Node Name Status Comment ------------------------------ ------------------------------ ---------------stnsp002 libstdc++-devel-3.4.5-2 passed stnsp001 libstdc++-devel-3.4.5-2 passed Result: Package existence check passed for "libstdc++-devel-3.4.5". Check: Package existence for "unixODBC-2.2.11" Node Name Status ------------------------------ -----------------------------stnsp002 unixODBC-2.2.11-1.RHEL4.1 stnsp001 unixODBC-2.2.11-1.RHEL4.1 Result: Package existence check passed for "unixODBC-2.2.11".

Comment ---------------passed passed

Check: Package existence for "unixODBC-devel-2.2.11" Node Name Status Comment ------------------------------ ------------------------------ ---------------stnsp002 unixODBC-devel-2.2.11-1.RHEL4.1 passed stnsp001 unixODBC-devel-2.2.11-1.RHEL4.1 passed Result: Package existence check passed for "unixODBC-devel-2.2.11". Check: Package existence for "glibc-2.3.4-2.19" Node Name Status ------------------------------ -----------------------------stnsp002 glibc-2.3.4-2.19 stnsp001 glibc-2.3.4-2.19 Result: Package existence check passed for "glibc-2.3.4-2.19". Check: Kernel parameter for "semmsl" Node Name Configured ------------ -----------------------stnsp002 250 stnsp001 250 Result: Kernel parameter check passed for Check: Kernel parameter for "semmns" Node Name Configured ------------ -----------------------stnsp002 32000

Comment ---------------passed passed

Required -----------------------250 250 "semmsl".

Comment ---------passed passed

Required -----------------------32000

Comment ---------passed

10.1. CVU check -pre Database Instance Install

95

stnsp001 32000 32000 Result: Kernel parameter check passed for "semmns". Check: Kernel parameter for "semopm" Node Name Configured ------------ -----------------------stnsp002 100 stnsp001 100 Result: Kernel parameter check passed for Check: Kernel parameter for "semmni" Node Name Configured ------------ -----------------------stnsp002 128 stnsp001 128 Result: Kernel parameter check passed for Check: Kernel parameter for "shmall" Node Name Configured ------------ -----------------------stnsp002 2097152 stnsp001 2097152 Result: Kernel parameter check passed for Check: Kernel parameter for "shmmax" Node Name Configured ------------ -----------------------stnsp002 2147483648 stnsp001 1073741824 Result: Kernel parameter check passed for Check: Kernel parameter for "shmmni" Node Name Configured ------------ -----------------------stnsp002 4096 stnsp001 4096 Result: Kernel parameter check passed for Check: Kernel parameter for "file-max" Node Name Configured ------------ -----------------------stnsp002 6553600 stnsp001 6553600 Result: Kernel parameter check passed for

passed

Required -----------------------100 100 "semopm".

Comment ---------passed passed

Required -----------------------128 128 "semmni".

Comment ---------passed passed

Required -----------------------2097152 2097152 "shmall".

Comment ---------passed passed

Required -----------------------536870912 536870912 "shmmax".

Comment ---------passed passed

Required -----------------------4096 4096 "shmmni".

Comment ---------passed passed

Required -----------------------65536 65536 "file-max".

Comment ---------passed passed

Check: Kernel parameter for "rmem_default" Node Name Configured Required ------------ ------------------------ -----------------------stnsp002 4194304 4194304 stnsp001 4194304 4194304 Result: Kernel parameter check passed for "rmem_default". Check: Kernel parameter for "rmem_max" Node Name Configured ------------ -----------------------stnsp002 4194304 stnsp001 4194304 Result: Kernel parameter check passed for

Comment ---------passed passed

Required -----------------------4194304 4194304 "rmem_max".

Comment ---------passed passed

Check: Kernel parameter for "wmem_default" Node Name Configured Required ------------ ------------------------ -----------------------stnsp002 524288 262144 stnsp001 524288 262144 Result: Kernel parameter check passed for "wmem_default". Check: Kernel parameter for "wmem_max" Node Name Configured ------------ -----------------------stnsp002 1048576 stnsp001 524288 Result: Kernel parameter check passed for Check: Group existence for "dba" Node Name Status ------------ ------------------------

Comment ---------passed passed

Required -----------------------262144 262144 "wmem_max".

Comment ---------passed passed

Comment ------------------------

10.1. CVU check -pre Database Instance Install

96

stnsp002 exists passed stnsp001 exists passed Result: Group existence check passed for "dba". Check: User existence for "nobody" Node Name Status Comment ------------ ------------------------ -----------------------stnsp002 exists passed stnsp001 exists passed Result: User existence check passed for "nobody". System requirement passed for 'database' Checking CRS integrity... Checking daemon liveness... Check: Liveness for "CRS daemon" Node Name Running ------------------------------------ -----------------------stnsp002 yes stnsp001 yes Result: Liveness check passed for "CRS daemon". Checking daemon liveness... Check: Liveness for "CSS daemon" Node Name Running ------------------------------------ -----------------------stnsp002 yes stnsp001 yes Result: Liveness check passed for "CSS daemon". Checking daemon liveness... Check: Liveness for "EVM daemon" Node Name Running ------------------------------------ -----------------------stnsp002 yes stnsp001 yes Result: Liveness check passed for "EVM daemon". Liveness of all Node Name -----------stnsp002 stnsp001 the daemons CRS daemon -----------------------yes yes

CSS daemon -----------------------yes yes

EVM daemon ---------yes yes

Checking CRS health... Check: Health of CRS Node Name -----------------------------------stnsp002 stnsp001 Result: CRS health check passed. CRS integrity check passed. Pre-check for database installation was successful.

CRS OK? -----------------------yes yes

Show all

Hide all

10.2. Oracle RAC Database Home Software Install

Start the installer by running "runInstaller" from the staged installation media. ./runInstaller

10.2. Oracle RAC Database Home Software Install

97

In this section we establish the Oracle software for the RAC Database home.

Notes We use the same installer for Database homes as Clusterware homes Actions Select the Oracle Database 11g radio button Click Next

10.2. Oracle RAC Database Home Software Install

98

 Notes Here we can choose the type of database software to be installed Actions Select the Enterprise Edition radio button Click Next

10.2. Oracle RAC Database Home Software Install

99

 Notes Here we specify the location of various components. If you change the path you should ensure that you do not use the exact same path as the Oracle Clusterware or ASM homes Actions Confirm entries are OK Click Next

Notes The installer has detected the presence of Oracle Clusterware and uses this to populate this dialog box. To build a cluster which includes all nodes you must ensure that there are check-boxes next to the node names Actions Click Select All Click Next

10.2. Oracle RAC Database Home Software Install

100

 Notes The installer will then complete some Product-Specific Prerequisite checks. These should all pass OK - as you have already run the CVU check Actions Click Next

10.2. Oracle RAC Database Home Software Install

101

 Notes We are going to install a Software only home and then subsequently configure the software Actions Select the Install Software Only radio button Click Next

Notes Assign the Unix groups for the privileged db users (default is oinstall) Action Click Next

10.2. Oracle RAC Database Home Software Install

102

 Notes Here we can see a summary of the install. Actions Click Install

Notes 10.2. Oracle RAC Database Home Software Install 103

 The installer copies the software to all nodes in the cluster Action Wait, The installer will pause and display instructions for running scripts, such as root.sh

Notes The installer pauses, some scripts need to be run as root on all ndes of the cluster Be sure to follow directions carefully, running scripts one at a time, on each node Action open a shell window on each node and execute the root script
bash-3.00# /scratch/product/11.1.0/db/root.sh Running Oracle 11g root.sh script... The following environment variables are set as: ORACLE_OWNER= oracle11 ORACLE_HOME= /scratch/product/11.1.0/asm Enter the full pathname of the local bin directory: [/usr/local/bin]: /usr/local/bin is read only. Continue without copy (y/n) or retry (r)? [y]: y Warning: /usr/local/bin is read only. No files will be copied. Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed. Finished product-specific root actions.

Notes The output from the root.sh script should be similar to this it should only take a few seconds to run on each node 10.2. Oracle RAC Database Home Software Install 104

 Action run the scripts on all nodes then return to the installer and Click OK

Notes After the software install completes you will see this End of Installation dialog Actions Click Exit

10.2. Oracle RAC Database Home Software Install

105

 Notes Are you sure ? Actions Click Yes

10.2.1. Oracle RAC Database software home has been installed

11. Oracle RAC Software Home Patching

Once RAC software has been installed, the next step in the process is to apply the 11.1.0.7 patchset to the RAC ORACLE_HOME.

11.1. Using the installer to patch the Oracle RAC Software Home
Enter the following commands to start Oracle Universal Installer, where patchset_directory is the directory where you unpacked the patch set software:
$ cd patchset_directory/Disk1 $ ./runInstaller

10.2.1. Oracle RAC Database software home has been installed

106

 Notes the Universal Installer screen appears Actions On the Welcome screen, click Next.

Notes 11.1. Using the installer to patch the Oracle RAC Software Home 107

 Specify the name and the location of the Database home Actions Check that the name and location are correct, Click Next

Notes here you can specify your metalik credentials for this install. If you leave both fields blank you can opt out of notifications (see next screen). Actions Enter your email address Enter your Metalink password Click Next

11.1. Using the installer to patch the Oracle RAC Software Home

108

 Notes If you leave both fields blank, you will opt out of notifications and get the above screen. Actions Click Yes

Notes 11.1. Using the installer to patch the Oracle RAC Software Home 109

 the installer detects that this is a clustered home and automatically selects all the nodes in the cluster Actions Click Next

Notes Some parameters are validated by the installer Actions Click Next

11.1. Using the installer to patch the Oracle RAC Software Home

110

 Notes This is a summary of the actions the installer will complete Actions Click Install

Notes 11.1. Using the installer to patch the Oracle RAC Software Home 111

 The installer copies the patch to all the nodes in the cluster Actions No action required

Notes The installer pauses, root.sh need to be run as root on both nodes of the cluster Actions open a shell window on each node and run root.sh (one after the other), output is shown below then return to the installer and Click OK
bash-3.00# /scratch/product/app/oracle11/product/11.1.0/db/root.sh Running Oracle 11g root.sh script... The following environment variables are set as: ORACLE_OWNER= oracle11 ORACLE_HOME= /scratch/product/app/oracle11/product/11.1.0/db Enter the full pathname of the local bin directory: [/usr/local/bin]: /usr/local/bin is read only. Continue without copy (y/n) or retry (r)? [y]: y Warning: /usr/local/bin is read only. No files will be copied. Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed. Finished product-specific root actions.

11.1. Using the installer to patch the Oracle RAC Software Home

112

 Notes Exit the installer Actions Click Yes

12. Oracle RAC Database Creation

12.1. CVU Database Verification Check
Verify the cluster before creating a cluster database
./runcluvfy.sh stage -pre dbcfg -n stnsp001,stnsp002 -d <DB:ORACLE_HOME> -verbose

Show CVU -pre dbinst output

Hide CVU output

./runcluvfy.sh stage -pre dbcfg -n stnsp001,stnsp002 -d /scratch/product/app/oracle11/product/11.1.0/db -verbose Performing pre-checks for database configuration Checking node reachability... Check: Node reachability from node "stnsp001" Destination Node Reachable? ------------------------------------ -----------------------stnsp001 yes stnsp002 yes Result: Node reachability check passed from node "stnsp001".

Checking user equivalence... Check: User equivalence for user "oracle11" Node Name Comment ------------------------------------ -----------------------stnsp002 passed stnsp001 passed Result: User equivalence check passed for user "oracle11".

12. Oracle RAC Database Creation

113

Checking administrative privileges... Check: Existence of user "oracle11" Node Name User Exists Comment ------------ ------------------------ -----------------------stnsp002 yes passed stnsp001 yes passed Result: User existence check passed for "oracle11". Check: Existence of group "oinstall" Node Name Status Group ID ------------ ------------------------ -----------------------stnsp002 exists 1422 stnsp001 exists 1422 Result: Group existence check passed for "oinstall". Check: Membership of user "oracle11" in group "oinstall" [as Primary] Node Name User Exists Group Exists User in Group Primary Comment ---------------- ------------ ------------ ------------ ------------ -----------stnsp002 yes yes yes yes passed stnsp001 yes yes yes yes passed Result: Membership check for user "oracle11" in group "oinstall" [as Primary] passed. Check: Existence of group "dba" Node Name Status Group ID ------------ ------------------------ -----------------------stnsp002 exists 8500 stnsp001 exists 8500 Result: Group existence check passed for "dba". Check: Membership of user "oracle11" in group "dba" Node Name User Exists Group Exists User in Group Primary ---------------- ------------ ------------ ------------ -----------stnsp002 yes yes yes no stnsp001 yes yes yes no Result: Membership check for user "oracle11" in group "dba" passed. Administrative privileges check passed. Checking node connectivity... Interface information for node "stnsp002" Interface Name IP Address Subnet ---------------- ------------ -----------eth0 10.137.8.216 10.137.8.0 eth0 10.137.8.234 10.137.8.0 eth1 10.137.24.207 10.137.24.0 eth2 10.137.20.172 10.137.20.0

Comment -----------passed passed

Subnet Gateway Default Gateway Hardware Address ------------ ------------ -----------0.0.0.0 10.137.8.1 00:0E:0C:08:06:4E 0.0.0.0 10.137.8.1 00:0E:0C:08:06:4E 0.0.0.0 10.137.8.1 00:0E:0C:08:06:4F 0.0.0.0 10.137.8.1 00:04:23:A6:CD:BD

Interface information for node "stnsp001" Interface Name IP Address Subnet ---------------- ------------ -----------eth0 10.137.8.215 10.137.8.0 eth0 10.137.8.233 10.137.8.0 eth1 10.137.24.206 10.137.24.0 eth2 10.137.20.171 10.137.20.0

Subnet Gateway Default Gateway Hardware Address ------------ ------------ -----------0.0.0.0 10.137.8.1 00:0E:0C:08:08:F2 0.0.0.0 10.137.8.1 00:0E:0C:08:08:F2 0.0.0.0 10.137.8.1 00:0E:0C:08:08:F3 0.0.0.0 10.137.8.1 00:04:23:A6:CD:61

Check: Node connectivity of subnet "10.137.8.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------stnsp002:eth0 stnsp002:eth0 yes stnsp002:eth0 stnsp001:eth0 yes stnsp002:eth0 stnsp001:eth0 yes stnsp002:eth0 stnsp001:eth0 yes stnsp002:eth0 stnsp001:eth0 yes stnsp001:eth0 stnsp001:eth0 yes Result: Node connectivity check passed for subnet "10.137.8.0" with node(s) stnsp002,stnsp001. Check: Node connectivity of subnet "10.137.24.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------stnsp002:eth1 stnsp001:eth1 yes Result: Node connectivity check passed for subnet "10.137.24.0" with node(s) stnsp002,stnsp001. Check: Node connectivity of subnet "10.137.20.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------stnsp002:eth2 stnsp001:eth2 yes Result: Node connectivity check passed for subnet "10.137.20.0" with node(s) stnsp002,stnsp001. Interfaces found on subnet "10.137.8.0" that are likely candidates for a private interconnect: stnsp002 eth0:10.137.8.216 eth0:10.137.8.234 stnsp001 eth0:10.137.8.215 eth0:10.137.8.233 Interfaces found on subnet "10.137.24.0" that are likely candidates for a private interconnect: stnsp002 eth1:10.137.24.207

12.1. CVU Database Verification Check

114

stnsp001 eth1:10.137.24.206 Interfaces found on subnet "10.137.20.0" that are likely candidates for a private interconnect: stnsp002 eth2:10.137.20.172 stnsp001 eth2:10.137.20.171 WARNING: Could not find a suitable set of interfaces for VIPs. Result: Node connectivity check passed.

Checking CRS integrity... Checking daemon liveness... Check: Liveness for "CRS daemon" Node Name Running ------------------------------------ -----------------------stnsp002 yes stnsp001 yes Result: Liveness check passed for "CRS daemon". Checking daemon liveness... Check: Liveness for "CSS daemon" Node Name Running ------------------------------------ -----------------------stnsp002 yes stnsp001 yes Result: Liveness check passed for "CSS daemon". Checking daemon liveness... Check: Liveness for "EVM daemon" Node Name Running ------------------------------------ -----------------------stnsp002 yes stnsp001 yes Result: Liveness check passed for "EVM daemon". Liveness of all Node Name -----------stnsp002 stnsp001 the daemons CRS daemon -----------------------yes yes

CSS daemon -----------------------yes yes

EVM daemon ---------yes yes

Checking CRS health... Check: Health of CRS Node Name -----------------------------------stnsp002 stnsp001 Result: CRS health check passed. CRS integrity check passed. Pre-check for database configuration was successful.

CRS OK? -----------------------yes yes

Show all

Hide all

12.2. use dbca to create the RAC database

[oracle11@stnsp001 bin]$ cd /scratch/product/11.1.0/db/bin [oracle11@stnsp001 bin]$ ./dbca and

Notes You must launch dbca from the RAC database home Action Ensure the ORACLE_HOME environment variable is set to the new RAC Home change to the RAC Home bin directory run ./dbca

12.2. use dbca to create the RAC database

115

 Notes dbca detects the Oracle Clusterware layer is running and offers to create either cluster or single instance database Actions Select the Oracle Real Application Clusters database radio button Click Next

12.2. use dbca to create the RAC database

116

 Notes We use dbca to create RAC databases Actions Select the Create a Database radio button Click Next

12.2. use dbca to create the RAC database

117

 Notes You need to make sure you create RAC database instances on all the cluster nodes Actions Click Select All Click Next

12.2. use dbca to create the RAC database

118

 Notes Here you get to select the type of database Actions Select the General Purpose... radio button Click Next

12.2. use dbca to create the RAC database

119

 Notes Now you specify the preface for the SIDs for the RAC database instances recommendation is to keep it simple Actions Enter APX in the Global Database name, The SID should autofill Click Next

12.2. use dbca to create the RAC database

120

 Notes Here you get to configure Oracle Enterprise Manager. If you have no Grid Control server then the best method is to use dbcontrol as detailed here. Actions Check the Configure Enterprise manager checkbox Select the Configure Database Control for local management radio button Click Next

12.2. use dbca to create the RAC database

121

 Notes Here we specify the password for the Oracle Database users. In this example we are setting the same password for all users. You should set a password scheme that meets your requirements. Actions Select the Use the Same Administrative Password radio button Enter the password Enter the same password for the Confirm password Click Next

12.2. use dbca to create the RAC database

122

 Notes Here you specify where you would like your database datafiles stored. We are going to use the ASM diskgroups we created earlier. Actions Select the Automatic Storage Management (ASM) radio button Click Next

12.2. use dbca to create the RAC database

123

 Notes dbca displays the diskgroups we previously created. We are going to use the DATA diskgroup Actions Check the box on the DATA diskgroup line Click Next

12.2. use dbca to create the RAC database

124

 Notes Here we can adjust where various files are placed Actions Select the Use Oracle-managed Files radio button Ensure +DATA is in the Database Area field Click Next

12.2. use dbca to create the RAC database

125

 Notes Here we specify recovery configuration information. We are going to use a flash recovery area. Actions Check the Specify Flash Recovery Area checkbox In the Flash recovery area enter +FLASH Set the flash recovery size remember the size of the +FASH diskgroup Check the Enable Archiving checkbox Click Next

12.2. use dbca to create the RAC database

126

 Notes Here you can choose to add the Oracle Sample schemas Actions Check the Sample schemas checkbox Click Next

12.2. use dbca to create the RAC database

127

 Notes Here you can modify the default initialisation parameters. Actions Click Next

12.2. use dbca to create the RAC database

128

 Notes 11g introduces enhanced security settings Actions Select the Keep the enhanced radio button Click Next

Notes 11g will set up some automatic maintenance tasks Actions Check the Enable automatic maintenance tasks checkbox Click Next

12.2. use dbca to create the RAC database

129

 Notes Here you can review the placement of various database files Actions Click Next

12.2. use dbca to create the RAC database

130

 Notes You can decide to generate database creation scripts Actions Check the Create Database checkbox Click Finish

Notes A summary screen Actions Click OK

12.2. use dbca to create the RAC database

131

 Notes The database is being created Actions none

12.2. use dbca to create the RAC database

132

 Notes After the database is created the summary screen is displayed note the URL for the Database Control Actions Click Exit

Notes The Database instances will be started on all nodes Actions none

12.3. Verify all database resource are online

[oracle11@stnsp001 bin]$ cd /scratch/11.1.0/crs/bin [oracle11@stnsp001 bin]$ ./crs_stat -t Name Type Target State Host -----------------------------------------------------------ora.APX.db application ONLINE ONLINE stnsp001 ora....X1.inst application ONLINE ONLINE stnsp001 ora....X2.inst application ONLINE ONLINE stnsp002 ora....SM1.asm application ONLINE ONLINE stnsp001 ora....01.lsnr application ONLINE ONLINE stnsp001 ora....001.gsd application ONLINE ONLINE stnsp001 ora....001.ons application ONLINE ONLINE stnsp001 ora....001.vip application ONLINE ONLINE stnsp001 ora....SM2.asm application ONLINE ONLINE stnsp002 12.3. Verify all database resource are online 133

ora....02.lsnr ora....002.gsd ora....002.ons ora....002.vip Notes

application application application application

ONLINE ONLINE ONLINE ONLINE

ONLINE ONLINE ONLINE ONLINE

stnsp002 stnsp002 stnsp002 stnsp002

You can see that Oracle clusterware has started the database instances on each node Action change directory to the Oracle Clusterware home bin directory type ./crs_stat -t

12.4. RAC database creation is complete

12.4. RAC database creation is complete

134