Step By Step: Upgrade 10gR2 RAC to 11gR1 RAC on Oracle Enterprise Linux 5 (32 bit) Pla orm.

By Bhavin Hingu

CLICK HERE for the”Step By Step: Upgrade 10gR2 RAC to 11gR2 RAC on Linux”

<<HOME>>

This Document shows the step by step of upgrading 3‐Node 10gR2 RAC database on ASM to 11gR1 RAC. The below upgrade path allows upgrading 10gR2 CRS to 11gR1 CRS
with no down me using rolling upgrade. The rolling upgrade of ASM is possible star ng from Release 11gR1 (11.1.0.6) and is not backward compa ble. That means we cannot
use Rolling upgrade method for upgrading 10gR2 ASM to 11gR1 ASM.

Upgrading 10gR2 RAC database will need outage and the total down me may further be avoided or minimized by using Logical Standby Database in the upgrade process (not
covered here).

Existing 10gR2 RAC setup (Before Upgrade) Target 11gR1 RAC Setup (After Upgrade)
Clusterware Oracle 10g R2 Clusterware 10.2.0.3 Oracle 11gR1 Clusterware 11.1.0.6
ASM Binaries 10g R2 RAC 10.2.0.3 11gR1 RAC 11.1.0.6
Cluster Name Lab Lab
Cluster Nodes node1, node2, node3 node1, node2, node3
Clusterware Home /u01/app/oracle/crs (CRS_HOME) /u01/app/oracle/crs (CRS_HOME)
Clusterware Owner oracle:(oinstall, dba) oracle:(oinstall, dba)
VIPs node1­vip, node2­vip, node3­vip node1­vip, node2­vip, node3­vip
SCAN N/A N/A
SCAN_LISTENER Host/port N/A N/A
OCR and Vo ng Disks Storage Type Raw Devices Raw Devices
OCR Disks /dev/raw/raw1, /dev/raw/raw2 /dev/raw/raw1, /dev/raw/raw2

Vo ng Disks /dev/raw/raw3, /dev/raw/raw4, /dev/raw/raw5 /dev/raw/raw3, /dev/raw/raw4, /dev/raw/raw5

Vo ng Disks /dev/raw/raw3, /dev/raw/raw4, /dev/raw/raw5 /dev/raw/raw3, /dev/raw/raw4, /dev/raw/raw5
ASM_HOME /u01/app/oracle/asm /u01/app/oracle/asm11gr1
ASM_HOME Owner oracle:(oinstall, dba) oracle:(oinstall, dba)
ASMLib user:group oracle:oinstall oracle:oinstall

DB Binaries Oracle 10gR2 RAC (10.2.0.3) Oracle 11gR1 RAC (11.1.0.6)

DB_HOME /u01/app/oracle/db /u01/app/oracle/db11gr1
DB_HOME Owner oracle:(oinstall, dba) oracle:(oinstall, dba)
DB LISTENER LAB_LISTENER LAB_LISTENER
DB Listener Host/port node1­vip, node2­vip, node3­vip (port 1530) node1­vip, node2­vip, node3­vip (port 1530)
DB Storage Type, File Management ASM with OMFs ASM with OMFs
ASM diskgroups for DB and FRA DATA, FRA DATA, FRA
OS Pla orm Oracle Enterprise Linux 5.5 (32 bit) Oracle Enterprise Linux 5.5 (32 bit)

HERE’s the Exis ng 10gR2 RAC Setup in Detail

The Upgrade Process is composed of below 4 Stages:

 Pre‐Upgrade Tasks
 Upgrade 10gR2 (10.2.0.1) Clusterware to 11gR1 (11.1.0.6)  Rolling Upgrade No Down me.
 Upgrade 10gR2 ASM_HOME (10.2.0.1) to 11gR1 RAC (11.1.0.6)  ASM and Database Down me required.
 Upgrade 10gR2 Database to 11gR1 RAC (11.1.0.6)  Database Down me required.

Pre‐Upgrade tasks:

Install/Upgrade RPMs required for 11gR1 RAC and 11gR2 RAC Installa on
Set 11gR1 specific Kernel Parameters
Update the TIMEZONE file version
Backing up ORACLE_HOMEs/database

Minimum Required RPMs for 11gR1 RAC on OEL 5.5 (All the 3 RAC Nodes):

binutils­2.17.50.0.6­2.el5
compat­libstdc++­33­3.2.3­61
elfutils­libelf­0.125
elfutils­libelf­devel­0.125
 elfutils­libelf­devel­0.125
glibc­2.5­12
glibc­common­2.5­12
glibc­devel­2.5­12
glibc­headers­2.3.4­2
gcc­4.1.1­52
gcc­c++­4.1.1­52
libaio­0.3.106
libaio­devel­0.3.106
libgcc­4.1.1­52
libstdc++­4.1.1
libstdc++­devel­4.1.1­52.e15
make­3.81­1.1
sysstat­7.0.0
unixODBC­2.2.11
unixODBC­devel­2.2.11

Below command verifies whether the specified rpms are installed or not. Any missing rpms can be installed from the OEL Media Pack

rpm ­q binutils compat­libstdc++­33 elfutils­libelf elfutils­libelf­devel \

gcc gcc­c++ glibc glibc­common glibc­devel glibc­headers libaio libaio­devel \
libgcc libstdc++ libstdc++­devel make sysstat unixODBC unixODBC­devel

NOTE: The cvuqdisk rpm in 11gR1 has the same version as one available in 10gR2 (cvuqdisk­1.0.1­1), so I did not have to replace the 10gR2 cvuqdisk by
11gR1 one.

Kernel Parameter Requirements (All the 3 RAC Nodes):

/etc/sysctl.conf:
# Kernel sysctl configuration file for Oracle Enterprise Linux
#
# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and
# sysctl.conf(5) for more details.

# Controls IP packet forwarding

net.ipv4.ip_forward = 0

# Controls source route verification

net.ipv4.conf.default.rp_filter = 1

# Do not accept source routing

net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel

kernel.sysrq = 1

# Controls whether core dumps will append the PID to the core filename
# Useful for debugging multi­threaded applications
kernel.core_uses_pid = 1

# Controls the use of TCP syncookies

net.ipv4.tcp_syncookies = 1

# Controls the maximum size of a message, in bytes

 # Controls the maximum size of a message, in bytes
kernel.msgmnb = 65536

# Controls the default maxmimum size of a mesage queue

kernel.msgmax = 8192

# Controls the maximum shared segment size, in bytes

kernel.shmmax = 4294967295

# Controls the maximum number of shared memory segments, in pages

kernel.shmall = 1073741824
# For 11g, recommended value for file­max is 6815744
fs.file­max = 6815744
# For 10g, uncomment 'fs.file­max 327679', comment other entries for this parameter and re­run sysctl ­p
# fs.file­max:327679
kernel.msgmni = 2878
kernel.sem = 250 32000 100 142
kernel.shmmni = 4096
net.core.rmem_default = 262144
# For 11g, recommended value for net.core.rmem_max is 4194304
net.core.rmem_max = 4194304
# For 10g, uncomment 'net.core.rmem_max 2097152', comment other entries for this parameter and re­run sysctl ­p
# net.core.rmem_max=2097152
net.core.wmem_default = 262144
# For 11g, recommended value for wmem_max is 1048576
net.core.wmem_max = 1048576
# For 10g, uncomment 'net.core.wmem_max 262144', comment other entries for this parameter and re­run sysctl ­p
# net.core.wmem_max:262144
fs.aio­max­nr = 3145728
# For 11g, recommended value for ip_local_port_range is 9000 65500
net.ipv4.ip_local_port_range = 9000 65500
# For 10g, uncomment 'net.ipv4.ip_local_port_range 1024 65000', comment other entries for this parameter and re­run sysctl ­p
# net.ipv4.ip_local_port_range:1024 65000
# Added min_free_kbytes 50MB to avoid OOM killer on EL4/EL5
vm.min_free_kbytes = 51200

Enable these changes in the kernel.

sysctl ­p

Update the TIMEZONE file Version:

The below database patches were applied in order to update the mezone file version to 4 if it is less than 4. DBUA may fail with the “ORA­01722: invalid
number” error during the upgrade process.

Verify the current version of the timezone file in the 10gR2 database.

select * from v$timezone_file;

Patches to update the timezone file to version 4:

5746875
5632264

Backing Up ORACLE_HOMEs/database:
Steps I followed to take the Backup of ORACLE_HOMEs before the upgrade: (This can be applied to 11gR1 and 10g databases)
On node1:

mkdir backup
cd backup

dd if=/dev/dev/raw1 of=ocr_disk_10gr2.bkp
dd if=/dev/dev/raw3 of=voting_disk_10gr2.bkp

tar cvf node1_crs_10gr2.tar /u01/app/oracle/crs/*

tar cvf node1_asm_10gr2.tar /u01/app/oracle/asm/*
tar cvf node1_db_10gr2.tar /u01/app/oracle/db/*
tar cvf node1_etc_oracle /etc/oracle/*

cp /etc/inittab etc_inittab

mkdir etc_init_d
cd etc_init_d
cp /etc/init.d/init* .

On node2:

mkdir backup
cd backup

tar cvf node2_crs_10gr2.tar /u01/app/oracle/crs/*

tar cvf node2_asm_10gr2.tar /u01/app/oracle/asm/*
tar cvf node2_db_10gr2.tar /u01/app/oracle/db/*
tar cvf node2_etc_oracle /etc/oracle/*

cp /etc/inittab etc_inittab

mkdir etc_init_d
cd etc_init_d
cp /etc/init.d/init* .

On node3:

mkdir backup
cd backup

tar cvf node3_crs_10gr2.tar /u01/app/oracle/crs/*

tar cvf node3_asm_10gr2.tar /u01/app/oracle/asm/*
tar cvf node3_db_10gr2.tar /u01/app/oracle/db/*
tar cvf node3_etc_oracle /etc/oracle/*

cp /etc/inittab etc_inittab
 mkdir etc_init_d
cd etc_init_d
cp /etc/init.d/init* .

RMAN full backup was taken.

Step By Step: Upgrade Clusterware, ASM and Database from 10.2.0.3 to 11.1.0.6.

Upgrade 10gR2 CRS to 11gR1 CRS in Rolling Fashion (No Down me):

On Node1:

 Stop the labdb1 instance, asm instance, nodeapps.

 Relocate any db or cluster service resources to the other nodes.
 Stop and disable the CRS on this node
 Reboot the node to clean up any oracle/CRS related processes on this node.
 Enable the CRS a er the node comes back up.
 Run the preupdate.sh script from 11gR1 clusterware Media to unlock the CRS_HOME (so that oracle user can modify some of the root owned files).
 Start the runInstaller from the 11gR1 Clusterware so ware directory.
srvctl stop instance ­d labdb –i labdb1
srvctl stop asm –n node1
srvctl stop nodeapps –n node1
crs_relocate ora.labdb.db
crs_relocate ora.labdb.oltp.cs
crsctl stop crs
crsctl disable crs
reboot

crsctl enable crs  once the node comes back up after the reboot.
Crsctl check crs  CRS stack will not be up as CRS was disabled before the reboot.
/home/oracle/11gR1/clusterware/upgrade/preupdate.sh –crshome /u01/app/oracle/crs –crsuser oracle ­noshutdown

./runInstaller –ignoreSysPrereqs
/u01/app/oracle/crs/install/rootupgrade  run this script at the end as root
Verify the CRS

crsctl check crs (on all the RAC nodes)  it should be up on all the nodes
crsctl query crs activeversion node1  The active version should be 10.2.0.3
crsctl query crs activeversion node2
crsctl query crs activeversion node3
crsctl query crs softwareversion node1  The Software Version should show 11.1.0.6
crsctl query crs softwareversion node2
crsctl query crs softwareversion node3

On Node2:

 Stop the database instance labdb2, asm instance, nodeapps.

 Relocate any db or cluster service resources to the other nodes.
 Stop and disable the CRS on this node
  Stop and disable the CRS on this node
 Reboot the node to clean up any oracle related processes on this node.
 Enable the crs a er the node comes back up.
 Run the preupdate.sh script from 11gR1 clusterware Media to unlock the CRS_HOME (so that oracle user can modify some of the root owned files).
 Start the runInstaller from the 11gR1 Clusterware so ware directory.

srvctl stop instance ­d labdb –i labdb2

srvctl stop asm –n node2
srvctl stop nodeapps –n node2
crs_relocate ora.labdb.db
crsctl stop crs
crsctl disable crs
reboot

crsctl enable crs

crsctl check crs
crsctl check crs
/home/oracle/11gR1/clusterware/upgrade/preupdate.sh –crshome /u01/app/oracle/crs –crsuser oracle ­noshutdown

./runInstaller –ignoreSysPrereqs
/u01/app/oracle/crs/install/rootupgrade

Verify the CRS

crsctl check crs (on all the RAC nodes)

crsctl query crs activeversion node1  The active version should be 10.2.0.3
crsctl query crs activeversion node2  The active version should be 10.2.0.3
crsctl query crs activeversion node3
crsctl query crs softwareversion node1  The Software Version should show 11.1.0.6
crsctl query crs softwareversion node2  The Software Version should show 11.1.0.6
crsctl query crs softwareversion node3
On Node3:

 Stop the database instance labdb3, asm instance, nodeapps.

 Relocate any db or cluster service resources to the other nodes.
 Stop and disable the CRS on this node
 Reboot the node to clean up any oracle related processes on this node.
 Enable the crs a er the node comes back up.
 Run the preupdate.sh script from 11gR1 clusterware Media to unlock the CRS_HOME (so that oracle user can modify some of the root owned files).
 Start the runInstaller from the 11gR1 Clusterware so ware directory.

srvctl stop instance ­d labdb –i labdb3

srvctl stop asm –n node3
srvctl stop nodeapps –n node3
crs_relocate ora.labdb.oltp.cs
crs_relocate ora.labdb.oltp.cs
crsctl stop crs
crsctl disable crs
reboot

crsctl enable crs

crsctl check crs
/home/oracle/11gR1/clusterware/upgrade/preupdate.sh –crshome /u01/app/oracle/crs –crsuser oracle ­noshutdown

./runInstaller –ignoreSysPrereqs
/u01/app/oracle/crs/install/rootupgrade

Verify the CRS

crsctl check crs (on all the RAC nodes)

crsctl query crs activeversion node1
crsctl query crs activeversion node2
crsctl query crs activeversion node3
crsctl query crs softwareversion node1
crsctl query crs softwareversion node2
crsctl query crs softwareversion node3

The active Version and Software version should show 11.1.0.6 at this stage.
 HERE’s the detailed Screen Shots of Upgrading 10gR2 CRS to 11gR1 CRS

Upgrade 10gR2 ASM to 11gR1 ASM:

 Start the runInstaller from the 11gR1 database so ware Install Media and select the “Install Software Only” Configura on Op on on the OUI screen.
./runInstaller –ignoreSysPrereqs  From the 11gR1 Software for Database

Run the root.sh on all the RAC nodes to finish the Installa on.

Invoke the dbua from the new 11gR1 ASM_HOME to upgrade the 10gR2 ASM to 11gR1 ASM. Upgrading ASM instance is pre y much like Star ng the ASM Instances
and ASM Listeners using the newly Installed 11gR1 ASM_HOME. DBUA simply copies the asm pfile and password files from 10gR2 dbs folder to 11gR1 dbs folder. It also
copies the tnsnames.ora and listener.ora files as well. At the end, it modifies the ASM related CRS resources (asm and all the listeners belonging to 10gR2 ASM_HOME)
to start from new 11gR1 ASM_HOME by modifying the ACTION_SCRIPT parameter to point it to the 11gR1 racgwrap.

export ORACLE_HOME=/u01/app/oracle/asm11gr1
export ORACLE_SID=+ASM1
/u01/app/oracle/asm11gr1/bin/dbua
HERE’s the detailed Screen Shots of Upgrading 10gR2 ASM to 11gR1 ASM.

Issues Encountered and Troubleshoo ng:

Issue #1:

During the ASM Upgrade by DBUA, I was prompted for migra ng DB listeners running from 10gR2 exis ng Database HOME to 11gR1 ASM HOME. I did not want the DB
Listener to migrate to ASM HOME but the Installer was not moving ahead if pressed “No”. So, I decided to move forward with the DB listener migra on. I had to
manually migrate this DB listener back to 10gR2 DB home at the end of the upgrade of ASM process.
Here’s how the DB listener LAB_LISTENER was moved back to 10gR2 DB HOME from 11gR1 ASM_HOME.

I made sure that the 10gR2 TNS_ADMIN s ll have the listener.ora and tnsnames.ora files with informa on about LAB_LISTENER.
Updated the CRS resources related to LAB_LISTENER as shown below to point the ACTION_SCRIPT to /u01/app/oracle/asm/bin/racgwrap

Generate the CAP files:

cd /u01/app/oracle/crs/crs/public
/u01/app/oracle/crs/bin/crs_stat –p ora.node1.LAB_LISTENER_NODE1.lsnr > ora.node1.LAB_LISTENER_NODE1.lsnr.cap
/u01/app/oracle/crs/bin/crs_stat –p ora.node2.LAB_LISTENER_NODE2.lsnr > ora.node2.LAB_LISTENER_NODE2.lsnr.cap
/u01/app/oracle/crs/bin/crs_stat –p ora.node3.LAB_LISTENER_NODE3.lsnr > ora.node3.LAB_LISTENER_NODE3.lsnr.cap
/u01/app/oracle/crs/bin/crs_stat –p ora.node3.LAB_LISTENER_NODE3.lsnr > ora.node3.LAB_LISTENER_NODE3.lsnr.cap

The ACTION_SCRIPT parameter was modified in these cap files.

Replaced the line…

ACTION_SCRIPT=/u01/app/oracle/asm11gr1/bin/racgwrap

WITH…

ACTION_SCRIPT=/u01/app/oracle/asm/bin/racgwrap

Ran the below commands to update the OCR with these changes.

/u01/app/oracle/crs/bin/crs_register –u ora.node1.LAB_LISTENER_NODE1.lsnr
/u01/app/oracle/crs/bin/crs_register –u ora.node2.LAB_LISTENER_NODE2.lsnr
/u01/app/oracle/crs/bin/crs_register –u ora.node3.LAB_LISTENER_NODE3.lsnr

Issue #2:

The DBUA could not start the ASM on node 2 during post upgrade steps at the end of the upgrade process and so it terminated with the below error in trace.log file.
I tried to manually start the ASM instance and it worked just fine but the OCR was s ll showing the “UNKNOWN” status for this resource. Even using
crs_start/crs_stop to start or stop the ASM on node2 could not change the status from UNKNOWN. This means that the ASM resource entry for node2 in OCR
was somehow logically corrupted by DBUA. I had to unregister this entry and register back with the same parameter to fix the issue. A er doing that, the ASM was
successfully started and stopped by srvctl command.

Generate the CAP file for ASM resource on node2.

ora.node2.ASM2.asm.cap:

NAME=ora.node2.ASM2.asm
TYPE=application
ACTION_SCRIPT=/u01/app/oracle/asm11gr1/bin/racgwrap
ACTIVE_PLACEMENT=0
AUTO_START=1
CHECK_INTERVAL=300
DESCRIPTION=CRS application for ASM instance
FAILOVER_DELAY=0
FAILURE_INTERVAL=0
 FAILURE_INTERVAL=0
FAILURE_THRESHOLD=0
HOSTING_MEMBERS=node2
OPTIONAL_RESOURCES=
PLACEMENT=restricted
REQUIRED_RESOURCES=
RESTART_ATTEMPTS=5
SCRIPT_TIMEOUT=600
START_TIMEOUT=900
STOP_TIMEOUT=180
UPTIME_THRESHOLD=7d
USR_ORA_ALERT_NAME=
USR_ORA_CHECK_TIMEOUT=0
USR_ORA_CONNECT_STR=/ as sysasm
USR_ORA_DEBUG=0
USR_ORA_DISCONNECT=false
USR_ORA_FLAGS=
USR_ORA_IF=
USR_ORA_INST_NOT_SHUTDOWN=
USR_ORA_LANG=
USR_ORA_NETMASK=
USR_ORA_OPEN_MODE=mount
USR_ORA_OPI=false
USR_ORA_PFILE=
USR_ORA_PRECONNECT=none
USR_ORA_SRV=
USR_ORA_START_TIMEOUT=0
USR_ORA_STOP_MODE=immediate
USR_ORA_STOP_TIMEOUT=0
USR_ORA_VIP=

Unregister the ASM instance resource from OCR for node2

srvctl remove instance –d labdb –i labdb2

crs_unregister ora.node2.ASM2.asm

Registered the same ASM resource back to OCR

crs_register ora.node2.ASM2.asm
srvctl add instance –d labdb –i labdb2 –n node2
srvctl modify instance –d labdb –I labdb2 –t +ASM2
srvctl modify service ­d labdb ­s oltp ­n ­i "labdb1,labdb2,labdb3"

Upgrade 10gR2 database to 11gR1 Database:

 Start the runInstaller from the 11gR1 database so ware Install Media and select the “Install Software Only” Configura on Op on on the OUI screen.

 Invoke the dbua to upgrade the database.

  Invoke the dbua to upgrade the database.
 There may be warnings about INVALID objects and Stale Sta s cs which can safely be ignored.

./runInstaller –ignoreSysPrereqs  From the 11gR1 Software for Database

Invoke the DBUA from the newly installed 11gR1 HOME for the database using X terminal

/u01/app/oracle/db11gr1/bin/dbua

DBUA invalidates some of the objects in the database during the upgrade of database and so DBUA runs the utlrcmp.sql script at the end as part of post upgrade steps
to make them valid. I would start running this script when the DBUA progress is around 75% so that when the DBUA runs this script to recompile the INVALID objects, it
has to recompile fewer objects and helps reduce the overall ming of upgrade process as a whole.
Move the listener LAB_LISTENER from 10gR2 DB home to 11gR1 HOME.

The DBUA did not move the DB listener LAB_LISTENER from 10gR2 HOME to the 11gR1 Home and I had to manually moved it as below:

Copy the 10gR2 listener.ora and tnsnames.ora files to the 11gR2 TNS_ADMIN