Practice 12 Apply Patch Set using DBMS_ROLLING | page: 1

Practice Apply Patch Set using DBMS_ROLLING

Title

Purpose In this practice we will use DBMS_ROLLING package to apply patch set
number 22646084 on a Data Guard environment.

Software Oracle database version 12.1.0.2 on Oracle Linux 6.7 64-bit.

version

Document 1.0, Aug-2016

version

Required VirtualBox Appliance

Software
The practice has been implemented on an Oracle virtual appliances that have
/ Files
been in practice number 3 “Configuring the Data Guard Broker”.
Patch Set 22646084 for Linux x86 64-bit.
Latest version of Opatch, which is available for download from My Oracle
Support patch 6880880.

Hardware About 125 GB is required for the appliances used in this practice.

Downloading Opatch
Observe the options selected to download the Opatch from support.oracle.com in the following
screenshot:

Download Patch Set 22646084

Download Patch Set number 22646084 for Linux x86-64bit. The file name is
p22646084_121020_Linux-x86-64.zip and it is of size 1.27 GB.

Oracle 12c Data Guard Administration Course by Ahmed Baraka

Practice 12 Apply Patch Set using DBMS_ROLLING | page: 2

Data Guard Configuration Specifications

Protection Mode Maximum Performance

fast-start failover Enabled

The management interface Broker

Standby Database Type Physical Standby

Standby Database Unique Name ORADB_S2

Standby Database Hostname srv2

Oracle 12c Data Guard Administration Course by Ahmed Baraka

Practice 12 Apply Patch Set using DBMS_ROLLING | page: 3

The Practice Overview

The Practice Environment
• You will work on the appliances that were created in the “Practice 3 Configure the
Broker”.

Apply Patch Set using DBMS_ROLLING package

• Download Patch Set 22646084 and copy it to both servers
• Make sure the prerequisites apply
• List objects that are unsupported by the logical standby
• Disable the Broker configuration
• Set the VALID_FOR attribute
• Plan the rolling upgrade
• Use the DBMS_ROLLING package to perform the rolling upgrade

Oracle 12c Data Guard Administration Course by Ahmed Baraka

Practice 12 Apply Patch Set using DBMS_ROLLING | page: 4

Get the Environment Ready

1. Make a copy of the appliances that your created in the “Practice 3 Configure the Broker”. Give the
new folder the name “Practice 12 Rolling Upgrade”.

2. In VirtualBox open the two appliances in the new folder.

3. Start the appliances.

4. Start the databases in each appliance.

5. Start the redo apply.

dgmgrl sys/oracle@oradb
edit database oradb_s2 set state=apply-on;

6. Check the Broker configuration health.

show configuration
show database oradb
show database oradb_s2

7. Open Putty sessions to monitor the alert logs in both servers.

# on srv1
tail -f /u01/app/oracle/diag/rdbms/oradb/ORADB/trace/alert_ORADB.log

# on srv2
tail -f /u01/app/oracle/diag/rdbms/oradb_s2/ORADB_S2/trace/alert_ORADB_S2.log

Oracle 12c Data Guard Administration Course by Ahmed Baraka

Practice 12 Apply Patch Set using DBMS_ROLLING | page: 5

Download Patch Set 22646084 and copy it to both servers

8. Login to Oracle Support site (support.oracle.com) and download the patch set number 22646084 for
Linux x86-64. It is about 1.27 GB size.

This patch set will be applied on both the GI home and Oracle database home.

9. Make directory in both servers to copy the patch set file to it.
# srv1 and srv2
su - grid
mkdir ~/patches

10. To each host, copy the Opatch upgrade zip file (p6880880_121010_Linux-x86-64) and the Patch Set
zip file (p22646084_121020_Linux-x86-64.zip) to the directory /home/grid/patches using the WinScp
utility.

11. In each host, make the grid user the owner of the newly created directory.
su -
chown -R grid:oinstall /home/grid/patches/

Make sure the prerequisites apply

12. We have already configured all the prerequisites in those machines in practice number 3. Refer to the
previous lecture to obtain the list of the prerequisites. In our case, no action is actually needed at this
stage.

List objects that are unsupported by the logical standby

13. EDS functionality requires enabling Supplemental Logging in the primary database. Run the following
command in each database.
su - oracle

sqlplus sys/oracle@oradb as sysdba

ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY, UNIQUE INDEX) COLUMNS;

conn sys/oracle@oradb_s2 as sysdba

ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY, UNIQUE INDEX) COLUMNS;

14. Check which objects have datatypes unsupported by the logical standby database:

Our demo database has more than 100 unsupported tables. In a production database, you need to put on
a plan to handle all of them. Refer to the lecture “Managing Logical Standby Database” for more
information. In this practice, for demo purposes, we will implement EDS on a single table (CUSTOMERS).
conn sys/oracle@oradb as sysdba

Oracle 12c Data Guard Administration Course by Ahmed Baraka

Practice 12 Apply Patch Set using DBMS_ROLLING | page: 6

set pagesize 200

set linesize 100
col TABLE_NAME for a30
col COLUMN_NAME for a21
col DATA_TYPE for a28
SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE FROM DBA_LOGSTDBY_UNSUPPORTED ;

15. Make sure the objects are supported by EDS.

SELECT * FROM DBA_LOGSTDBY_EDS_SUPPORTED WHERE TABLE_NAME= 'CUSTOMERS';

16. Make note of the tables to be added to the EDS. You will add them when the LGM is converted to a
transient logical standby database.

Disable the Broker configuration

17. Login to DGMGRL and disable the Broker.

dgmgrl sys/oracle@oradb
disable configuration

Set the VALID_FOR attributes

18. Set the VALID_FOR attribute in both databases. This is required for the switchover to the logical
standby database to succeed. Refer to the lecture “Creating a Logical Standby Database” for further
details.
# in the primary database (oradb):
# create the destination of the archived redo log files of the standby redo log files:
su - grid
asmcmd
cd FRA/ORADB/ARCHIVELOG
mkdir ARCHREDOLOG

# make sure the VALID_FOR is set for the LOG_ARCHIVE_DEST_* parameters

su – oracle
sqlplus sys/oracle@oradb as sysdba
show parameter LOG_ARCHIVE_DEST_1
ALTER SYSTEM SET LOG_ARCHIVE_DEST_1='LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(
ONLINE_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=ORADB' scope=both;

show parameter LOG_ARCHIVE_DEST_2

Oracle 12c Data Guard Administration Course by Ahmed Baraka

Practice 12 Apply Patch Set using DBMS_ROLLING | page: 7

ALTER SYSTEM SET LOG_ARCHIVE_DEST_3='LOCATION=+FRA/ORADB/ARCHIVELOG/ARCHREDOLOG/

VALID_FOR=(STANDBY_LOGFILES,STANDBY_ROLE) DB_UNIQUE_NAME=ORADB' scope=both;
ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_3=ENABLE;

# in oradb_s2:
su - grid
asmcmd
cd FRA/ORADB_S2/ARCHIVELOG
mkdir ARCHREDOLOG

su - oracle
sqlplus sys/oracle@oradb_s2 as sysdba
show parameter LOG_ARCHIVE_DEST_1
ALTER SYSTEM SET LOG_ARCHIVE_DEST_1='LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(
ONLINE_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=ORADB_S2' scope=both;

# this parameter must be manually set in the standby database because it has been
# reset by the Broker when it was disabled:
show parameter LOG_ARCHIVE_DEST_2
ALTER SYSTEM SET LOG_ARCHIVE_DEST_2= 'SERVICE=ORADB ASYNC NOAFFIRM delay=0 reopen=300
VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=ORADB' SCOPE=BOTH ;
ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=ENABLE;

ALTER SYSTEM SET LOG_ARCHIVE_DEST_3='LOCATION=+FRA/ORADB_S2/ARCHIVELOG/ARCHREDOLOG/

VALID_FOR=(STANDBY_LOGFILES,STANDBY_ROLE) DB_UNIQUE_NAME=oradb_s2' scope=both;
ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_3=ENABLE;

Plan the rolling upgrade

19. Initialize the rolling upgrade plan with defining which standby database will be the Transient Logical
standby database.
conn sys/oracle@oradb as sysdba
exec DBMS_ROLLING.INIT_PLAN(FUTURE_PRIMARY=>'ORADB_S2');
# the status should be READY
SELECT REVISION, STATUS, PHASE FROM DBA_ROLLING_STATUS;

20. View the current upgrade parameter values:

set linesize 180
col scope for a10
col name for a35
col curval for a35
SELECT SCOPE, NAME, CURVAL FROM DBA_ROLLING_PARAMETERS ORDER BY SCOPE, NAME;

Oracle 12c Data Guard Administration Course by Ahmed Baraka

Practice 12 Apply Patch Set using DBMS_ROLLING | page: 8

21. Set values for the following parameters:

exec DBMS_ROLLING.SET_PARAMETER('SWITCH_LGM_LAG_WAIT', '1');
exec DBMS_ROLLING.SET_PARAMETER('SWITCH_LGM_LAG_TIME', '800');

22. Build the Upgrade Plan. Any issue reported by this procedure must be resolved before you proceed
with the rolling upgrade plan.
exec DBMS_ROLLING.BUILD_PLAN;
# View the Current Upgrade Plan:
set linesize 180
set pagesize 50
col TARGET for a10
col PHASE for a10
col DESCRIPTION for a65
SELECT INSTID, TARGET, PHASE, DESCRIPTION FROM DBA_ROLLING_PLAN ORDER BY 1;

# check the status of the plan:

col PHASE for a15
SELECT REVISION, STATUS, PHASE FROM DBA_ROLLING_STATUS;

# check the status of the databases included in the plan:

SELECT DBUN,ROLE,OPEN_MODE,ENGINE_STATUS,UPDATE_PROGRESS FROM DBA_ROLLING_DATABASES;

Troubleshooting Tool

This section is only a tip, not a required action.

While implementing the rest of the practice, if you an error is reported by any DBMS_ROLLING procedure,
to dig into the issue, first query DBA_ROLLING_EVENTS to know which step in the plan failed:
SELECT EVENTID, EVENT_TIME, TYPE, MESSAGE, STATUS, INSTID
FROM DBA_ROLLING_EVENTS ORDER BY EVENT_TIME DESC

Obtain Instruction ID “INSTID” from the query above. You can then obtain further details about that
instruction by using the following query:
SELECT INSTID, TARGET, PHASE, DESCRIPTION FROM DBA_ROLLING_PLAN WHERE INSTID=<INSTID>;

Furthermore, when doing the rolling upgrade, always keep eye on the Alert log files. In almost all the
rolling upgrade cases, they provide useful details to diagnose the issues.

Oracle 12c Data Guard Administration Course by Ahmed Baraka

Practice 12 Apply Patch Set using DBMS_ROLLING | page: 9

Perform the rolling upgrade

23. Configure the LGM database. This basically takes actions required to convert the standby database
into transient logical standby database. A GRP will also be created. It takes a few minutes to finish.
exec DBMS_ROLLING.START_PLAN;
# check the status of the plan and databases:
SELECT REVISION, STATUS, PHASE FROM DBA_ROLLING_STATUS;
SELECT DBUN, ROLE, ENGINE_STATUS,UPDATE_PROGRESS FROM DBA_ROLLING_DATABASES;

24. Observe the role of the LGM database.

# in srv2
SELECT DATABASE_ROLE FROM V$DATABASE ;

25. Shutdown the LGM database (oradb_s2)

# in srv2
srvctl stop database -d oradb_s2 -o immediate

26. Shutdown the ASM in srv2.

su – grid
crsctl stop has

27. Install the downloaded version of the Opatch in GI and DB homes. The PSU that you are applying
requires the Opatch utility of version 12.1.0.1.7 or later.
At the time of this writing, the latest version was 12.1.0.1.12. You have made a copy of the patch zip
file in /home/grid/patches in both servers.
# install the Opatch in the GI home
# in srv2 as grid:
# check the current version:
$ORACLE_HOME/OPatch/opatch version

# remove the current Opatch

rm -rf $ORACLE_HOME/OPatch/*

# unzip the patch zip file into Opatch directory:

cd ~/patches
unzip p6880880_121010_Linux-x86-64.zip -d /u01/app/12.1.0/grid

# verify the installed version:

$ORACLE_HOME/OPatch/opatch version

# install Opatch in DB home

# in srv2 as root:

Oracle 12c Data Guard Administration Course by Ahmed Baraka

Practice 12 Apply Patch Set using DBMS_ROLLING | page: 10

rm -rf /u01/app/oracle/product/12.1.0/db_1/OPatch/*
mkdir /home/oracle/patches
mv /home/grid/patches/p6880880_121010_Linux-x86-64.zip /home/oracle/patches/
chown -R oracle:oinstall /home/oracle/patches/
su - oracle
cd ~/patches
unzip p6880880_121010_Linux-x86-64.zip -d /u01/app/oracle/product/12.1.0/db_1
$ORACLE_HOME/OPatch/opatch version
rm p6880880_121010_Linux-x86-64.zip

28. Apply the patch to the GI home in srv2.

Note: do not use opatchauto utility. It does not support patching in Data Guard environments.

Note: do not apply the SQL installation part of the patch. This step will be performed only after software
homes of the primary and all the standby databases have been patched.

Note: Our PSU installation reference document is Doc ID 1591616.1, section 5: “Manual Steps for
Apply/Rollback Patch”
# in srv2
su - grid
# validate Oracle inventory
$ORACLE_HOME/OPatch/opatch lsinventory -oh $ORACLE_HOME

# decompress the psu zip file

cd ~/patches
unzip p22646084_121020_Linux-x86-64.zip > /dev/null

# as root
su -
/u01/app/12.1.0/grid/crs/install/roothas.pl -prepatch

# as grid, apply the patch:

su - grid
export PATH=$PATH:/u01/app/12.1.0/grid/OPatch
/u01/app/12.1.0/grid/OPatch/opatch apply -oh /u01/app/12.1.0/grid -local
/home/grid/patches/22646084/21436941

/u01/app/12.1.0/grid/OPatch/opatch apply -oh /u01/app/12.1.0/grid -local

/home/grid/patches/22646084/22291127

/u01/app/12.1.0/grid/OPatch/opatch apply -oh /u01/app/12.1.0/grid -local

/home/grid/patches/22646084/22502518

Oracle 12c Data Guard Administration Course by Ahmed Baraka

Practice 12 Apply Patch Set using DBMS_ROLLING | page: 11

/u01/app/12.1.0/grid/OPatch/opatch apply -oh /u01/app/12.1.0/grid -local

/home/grid/patches/22646084/22502555

# as root, run the post script (it will eventually startup the has resources):
su -
/u01/app/12.1.0/grid/crs/install/roothas.pl -postpatch

# verify, as grid:
su - grid
$ORACLE_HOME/OPatch/opatch lsinventory -detail
srvctl status asm

29. Apply the patch to the DB home in srv2:

# in srv2: apply the patch on DB home
# as root: move the PSU folder from grid to oracle home
mv /home/grid/patches/22646084/ /home/oracle/patches/
chown -R oracle:oinstall /home/oracle/patches/

su - oracle
# validate Oracle inventory
$ORACLE_HOME/OPatch/opatch lsinventory -oh $ORACLE_HOME

# as oracle, apply the patch:

export PATH=$PATH:/u01/app/oracle/product/12.1.0/db_1/OPatch
/u01/app/oracle/product/12.1.0/db_1/OPatch/opatch apply -oh /u01/app/oracle/product/12.1.0/db_1 -
local /home/oracle/patches/22646084/22291127

/u01/app/oracle/product/12.1.0/db_1/OPatch/opatch apply -oh /u01/app/oracle/product/12.1.0/db_1 -

local /home/oracle/patches/22646084/22502555

# verify:
opatch lsinventory -detail

30. Startup the LGM database (oradb_s2).

# in our environment the standby database will startup in MOUNT state by default, when it is
# started using srvctl. That is why the startoption parameter must be used in our case.
srvctl start database -d oradb_s2 -startoption OPEN

31. Start the SQL Apply process in SQL*Plus (remember the Broker is disabled at this stage).
sqlplus sys/oracle@oradb_s2 as sysdba
ALTER DATABASE START LOGICAL STANDBY APPLY NODELAY;

Oracle 12c Data Guard Administration Course by Ahmed Baraka

Practice 12 Apply Patch Set using DBMS_ROLLING | page: 12

32. In srv2, run the SQL installation part of the PSU in the database oradb_s2. When this step is done,
you consider the database oradb_s2 upgrade is finished. Changes done in this step will automatically
be applied later in the other databases by the redo recovery.
# srv2:
su - oracle
export PATH=$PATH:/u01/app/oracle/product/12.1.0/db_1/OPatch
cd $ORACLE_HOME/OPatch
./datapatch -verbose

# I recommend starting the LGM database at this stage:

conn sys/oracle@oradb_s2 as sysdba
shutdown immediate
startup open
ALTER DATABASE START LOGICAL STANDBY APPLY NODELAY;

33. Switchover to the standby database.

conn sys/oracle@oradb as sysdba
execute DBMS_ROLLING.SWITCHOVER;

# check the status of the plan and databases:

SELECT REVISION, STATUS, PHASE FROM DBA_ROLLING_STATUS;
col dbun for a10
SELECT DBUN, ROLE, ENGINE_STATUS,UPDATE_PROGRESS FROM DBA_ROLLING_DATABASES;
SELECT DB_UNIQUE_NAME,DATABASE_ROLE,OPEN_MODE FROM V$DATABASE;

Upgrade the TGM Database (former primary database)

34. Shutdown the old primary database (oradb):
# in srv1
srvctl stop database -d oradb -o immediate

35. Shutdown the ASM in srv1:

su - grid
crsctl stop has

36. Install the downloaded Opatch in both the GI and DB homes.

# in srv1 as grid:
# remove the current Opatch
su - grid
rm -rf $ORACLE_HOME/OPatch/*

Oracle 12c Data Guard Administration Course by Ahmed Baraka

Practice 12 Apply Patch Set using DBMS_ROLLING | page: 13

# unzip the patch zip file into Opatch directory:

cd ~/patches
unzip p6880880_121010_Linux-x86-64.zip -d /u01/app/12.1.0/grid >/dev/null

# verify the installed version:

$ORACLE_HOME/OPatch/opatch version

# in srv1 as root:
su -
rm -rf /u01/app/oracle/product/12.1.0/db_1/OPatch/*
mkdir /home/oracle/patches
mv /home/grid/patches/p6880880_121010_Linux-x86-64.zip /home/oracle/patches/
chown -R oracle:oinstall /home/oracle/patches/
su - oracle
cd ~/patches
unzip p6880880_121010_Linux-x86-64.zip -d /u01/app/oracle/product/12.1.0/db_1 >/dev/null
# make sure the new Opatch has been installed:
$ORACLE_HOME/OPatch/opatch version
rm p6880880_121010_Linux-x86-64.zip

37. Apply the patch to the GI home in srv1:

# in srv1: apply the patch on GI home
su - grid
# validate Oracle inventory
$ORACLE_HOME/OPatch/opatch lsinventory -oh $ORACLE_HOME

# decompress the psu zip file

cd ~/patches
unzip p22646084_121020_Linux-x86-64.zip >/dev/null

# as root
su -
/u01/app/12.1.0/grid/crs/install/roothas.pl -prepatch

# as grid, apply the patch:

su - grid
export PATH=$PATH:/u01/app/12.1.0/grid/OPatch
/u01/app/12.1.0/grid/OPatch/opatch apply -oh /u01/app/12.1.0/grid -local
/home/grid/patches/22646084/21436941

/u01/app/12.1.0/grid/OPatch/opatch apply -oh /u01/app/12.1.0/grid -local

/home/grid/patches/22646084/22291127

Oracle 12c Data Guard Administration Course by Ahmed Baraka

Practice 12 Apply Patch Set using DBMS_ROLLING | page: 14

/u01/app/12.1.0/grid/OPatch/opatch apply -oh /u01/app/12.1.0/grid -local

/home/grid/patches/22646084/22502518

/u01/app/12.1.0/grid/OPatch/opatch apply -oh /u01/app/12.1.0/grid -local

/home/grid/patches/22646084/22502555

# as root, run the post script (the has resources will take a few seconds after running the
# command to finish startup):
su -
/u01/app/12.1.0/grid/crs/install/roothas.pl -postpatch

# verify, as grid:
su - grid
$ORACLE_HOME/OPatch/opatch lsinventory -detail
srvctl status asm

38. Apply the patch to the Oracle database home in srv1:

# as root, move the PSU folder from grid to oracle home
mv /home/grid/patches/22646084/ /home/oracle/patches/
chown -R oracle:oinstall /home/oracle/patches/

su - oracle
# validate Oracle inventory
$ORACLE_HOME/OPatch/opatch lsinventory -oh $ORACLE_HOME

# as oracle, apply the patch:

export PATH=$PATH:/u01/app/oracle/product/12.1.0/db_1/OPatch
/u01/app/oracle/product/12.1.0/db_1/OPatch/opatch apply -oh /u01/app/oracle/product/12.1.0/db_1 -
local /home/oracle/patches/22646084/22291127

/u01/app/oracle/product/12.1.0/db_1/OPatch/opatch apply -oh /u01/app/oracle/product/12.1.0/db_1 -

local /home/oracle/patches/22646084/22502555

# verify:
opatch lsinventory

39. Startup the old primary database (oradb) in MOUNT state:

srvctl start database -d oradb -startoption mount

40. Convert the former primary database to physical standby database.

FINISH_PLAN flashes back the database to GRP DBMSRU_INITIAL and converts it from logical into physical
standby database. It must be run from the current primary database. It takes some time to finish.
conn sys/oracle@oradb_s2 as sysdba

Oracle 12c Data Guard Administration Course by Ahmed Baraka

Practice 12 Apply Patch Set using DBMS_ROLLING | page: 15

# have a look at the alert log file after running the command:
execute DBMS_ROLLING.FINISH_PLAN;
# from the alert log file, observe the media recovery operations after the switchover is finished

# verify the status of the current standby database:

SELECT DBUN, ROLE, ENGINE_STATUS,UPDATE_PROGRESS FROM DBA_ROLLING_DATABASES;

conn sys/oracle@oradb as sysdba

SELECT DB_UNIQUE_NAME,DATABASE_ROLE,OPEN_MODE FROM V$DATABASE;

41. Enable Broker configuration:

dgmgrl sys/oracle@oradb_s2
enable configuration
# wait for a few seconds before you run the commands below:
SHOW CONFIGURATION

42. Check on the StaticConnectIdentifier database property for the databases in the Broker
configuration. Make sure it is pointing to the right host.
SHOW DATABASE ORADB StaticConnectIdentifier
SHOW DATABASE ORADB_S2 StaticConnectIdentifier
EDIT DATABASE ORADB_S2 RESET PROPERTY StaticConnectIdentifier ;

43. Restart the standby database (oradb)

srvctl stop database -d oradb
srvctl start database -d oradb -startoption mount
SHOW CONFIGURATION

44. Switchover to oradb database (optional)

switchover to oradb

Oracle 12c Data Guard Administration Course by Ahmed Baraka