6 

Manual Multi-Node Patching of

Grid Infrastructure and RAC DB
Environment Using OPatchAuto
Patch orchestration is the automated execution of the patching steps, such as the
execution of pre-patch checks, stopping services, applying the binary patches, and starting
the services. Patch orchestration for Oracle Database 12c applies the patch to the GI/RAC
configuration on that machine, including all of its databases. The OPatchAuto patch
orchestration utility has been made available with version 12.1 of the OPatch utility.

This chapter covers the following topics:

 Running OPatchAuto on a Single Node

 Patching Session Output
 Sample Console Output
 OPatchAuto Apply
 Example: Manual OPatchAuto Patching Steps
Important:
This chapter applies to Oracle Database 12c only.

6.1 Running OPatchAuto on a Single
Node
Prerequisites: Node Availaibility During Patching (Rolling vs. Non-rolling)

In order to start a new patching session, the following conditions must be met.

Note:
The following conditions apply only for the first node, such as when the session is first
started on the cluster.

 Local node must be up for both Rolling and Non-rolling modes.

 At least one of the remote nodes must be up in order to start a Rolling mode session.
 ALL the remote nodes must be down in order to start a Non-rolling session.

To ensure successful patching, Oracle recommends that a patching session always consist of the four
following steps:

1. Run opatch lsinventory

When to run: During cluster uptime.

Before you begin a new patching session you must ensure that all nodes of the cluster are at
the same patch level.

$GRID_HOME/OPatch/opatch lsinventory
 Example Output

Patch level status of Cluster nodes :

Patching Level Nodes

-------------- -----

12345678 node123, node456

2. Run opatchauto apply -analyze

When to run: During cluster uptime.

The apply -analyze command simulates an opatchauto apply session by running all
prerequisite checks, when possible, without making changes to the system (either bits or
configurations).

3. Run opatchauto apply

When to run: During cluster downtime.

The apply command applies all patches within a specified system patch to the GI or RAC
home from which the opatchauto command is run.

4. Run opatch lsinventory

When to run: During cluster uptime.

After successfully running opatchauto apply on each individual node, you must

run opatch lsinventory on that node to confirm that the patch level has been correctly
updated.

6.2 Patching Session Output

The following patching session output examples illustrate successful OPatchAuto apply and rollback
sessions. Note: This session applies to both single and multi-node environments.

Example 6-1 OPatchAuto Apply/Rollback Session in Analyze Mode

--------------------------------
Summary--------------------------------

Analysis for applying patches has completed successfully:

Host:myhostq

CRS Home:/scratch/aime_ordb_myhostq/crso1/crshome_crso1

==Following patches were SUCCESSFULLY analyzed to be applied:

Patch: /tmp/patch_gipsu_12024/patch/22191349/21436941
Log:
/scratch/aime_ordb_myhostq/crso1/crshome_crso1/cfgtoollogs/opatchauto/
core/opatch/opatch2016-03-08_23-21-21PM_1.log

Patch: /tmp/patch_gipsu_12024/patch/22191349/21948341

Log:
/scratch/aime_ordb_myhostq/crso1/crshome_crso1/cfgtoollogs/opatchauto/
core/opatch/opatch2016-03-08_23-21-21PM_1.log

Patch: /tmp/patch_gipsu_12024/patch/22191349/21948344

Log:
/scratch/aime_ordb_myhostq/crso1/crshome_crso1/cfgtoollogs/opatchauto/
core/opatch/opatch2016-03-08_23-21-21PM_1.log

Patch: /tmp/patch_gipsu_12024/patch/22191349/21948354

Log:
/scratch/aime_ordb_myhostq/crso1/crshome_crso1/cfgtoollogs/opatchauto/
core/opatch/opatch2016-03-08_23-21-21PM_1.log

Host:myhostr

CRS Home:/scratch/aime_ordb_myhostq/crso1/crshome_crso1

==Following patches were SUCCESSFULLY analyzed to be applied:

Patch:
/scratch/aime_ordb_myhostq/crso1/crshome_crso1/OPatch/auto/dbtmp/22191
349/21436941

Log:
/scratch/aime_ordb_myhostq/crso1/crshome_crso1/cfgtoollogs/opatchauto/
core/opatch/opatch2016-03-08_23-24-56PM_1.log

Patch:
/scratch/aime_ordb_myhostq/crso1/crshome_crso1/OPatch/auto/dbtmp/22191
349/21948341

Log:
/scratch/aime_ordb_myhostq/crso1/crshome_crso1/cfgtoollogs/opatchauto/
core/opatch/opatch2016-03-08_23-24-56PM_1.log
Patch:
/scratch/aime_ordb_myhostq/crso1/crshome_crso1/OPatch/auto/dbtmp/22191
349/21948344

Log:
/scratch/aime_ordb_myhostq/crso1/crshome_crso1/cfgtoollogs/opatchauto/
core/opatch/opatch2016-03-08_23-24-56PM_1.log

Patch:
/scratch/aime_ordb_myhostq/crso1/crshome_crso1/OPatch/auto/dbtmp/22191
349/21948354

Log:
/scratch/aime_ordb_myhostq/crso1/crshome_crso1/cfgtoollogs/opatchauto/
core/opatch/opatch2016-03-08_23-24-56PM_1.log

OPatchAuto successful.

Example 6-2 OPatchAuto Apply Session

--------------------------------
Summary--------------------------------

Patching is completed successfully. Please find the summary as

follows:

Host:mymachineemq

CRS Home:/scratch/aime_ordb_mymachineemq/crso1/crshome_crso1

Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /tmp/patch_gipsu_12024/patch/22191349/21436941

Log:
/scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/cfgtoollogs/opatch
auto/core/opatch/opatch2016-03-08_23-41-38PM_1.log

Patch: /tmp/patch_gipsu_12024/patch/22191349/21948341

Log:
/scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/cfgtoollogs/opatch
auto/core/opatch/opatch2016-03-08_23-41-38PM_1.log

Patch: /tmp/patch_gipsu_12024/patch/22191349/21948344
Log:
/scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/cfgtoollogs/opatch
auto/core/opatch/opatch2016-03-08_23-41-38PM_1.log

Patch: /tmp/patch_gipsu_12024/patch/22191349/21948354

Log:
/scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/cfgtoollogs/opatch
auto/core/opatch/opatch2016-03-08_23-41-38PM_1.log

Host:mymachineemr

CRS Home:/scratch/aime_ordb_mymachineemq/crso1/crshome_crso1

Summary:

==Following patches were SUCCESSFULLY applied:

Patch:
/scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/OPatch/auto/dbtmp/
22191349/21436941

Log:
/scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/cfgtoollogs/opatch
auto/core/opatch/opatch2016-03-08_23-59-15PM_1.log

Patch:
/scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/OPatch/auto/dbtmp/
22191349/21948341

Log:
/scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/cfgtoollogs/opatch
auto/core/opatch/opatch2016-03-08_23-59-15PM_1.log

Patch:
/scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/OPatch/auto/dbtmp/
22191349/21948344

Log:
/scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/cfgtoollogs/opatch
auto/core/opatch/opatch2016-03-08_23-59-15PM_1.log

Patch:
/scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/OPatch/auto/dbtmp/
22191349/21948354

Log:
/scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/cfgtoollogs/opatch
auto/core/opatch/opatch2016-03-08_23-59-15PM_1.log

OPatchAuto successful.

Example 6-3 OPatchAuto Rollback Session

--------------------------------
Summary--------------------------------
Patching is completed successfully. Please find the summary as
follows:

Host:mymachineemm

CRS Home:/scratch/aime_ordb_mymachineemm/crso1/crshome_crso1

Summary:

==Following patches were SUCCESSFULLY rolled back:

Patch: /tmp/patch_gipsu_12019/patch/22191492/17077442

Log:
/scratch/aime_ordb_mymachineemm/crso1/crshome_crso1/cfgtoollogs/opatch
auto/core/opatch/opatch2016-03-08_19-25-46PM_1.log

Patch: /tmp/patch_gipsu_12019/patch/22191492/17303297

Log:
/scratch/aime_ordb_mymachineemm/crso1/crshome_crso1/cfgtoollogs/opatch
auto/core/opatch/opatch2016-03-08_19-25-46PM_1.log

Patch: /tmp/patch_gipsu_12019/patch/22191492/21951844

Log:
/scratch/aime_ordb_mymachineemm/crso1/crshome_crso1/cfgtoollogs/opatch
auto/core/opatch/opatch2016-03-08_19-25-46PM_1.log

OPatchAuto successful.

Example 6-4 OPatchAuto Apply/Rollback Failure

---------------------------Patching
Failed---------------------------------

Command execution failed during patching in home:

/scratch/aime/app/aime/product/11.2.0/dbhome_2, host: mymachineemg.

Command failed: /bin/sh -c

'ORACLE_HOME=/scratch/aime/app/aime/product/11.2.0/dbhome_2
/scratch/aime/app/aime/product/11.2.0/dbhome_2/bin/srvctl stop home -o
/scratch/aime/app/aime/product/11.2.0/dbhome_2 -n mymachineemg -f -t
TRANSACTIONAL -s
/scratch/aime/app/aime/product/11.2.0/dbhome_2/cfgtoollogs/opatchautod
b/statfile/mymachineemg/OracleHome-eca39d53-5b51-4cdf-9c79-
ce9d9312d86a_mymachineemg.stat'

Command failure output:

PRCH-1000 : Failed to stop resources running from Oracle home

/scratch/aime/app/aime/product/11.2.0/dbhome_2

PRCH-1029 : One or more resources failed to stop: PRCH-1006 : Failed

to stop Listener

PRCR-1014 : Failed to stop resource ora.LISTENER2.lsnr

PRCR-1065 : Failed to stop resource ora.LISTENER2.lsnr

CRS-5016: Process
"/scratch/aime/app/aime/product/11.2.0/dbhome_2/bin/lsnrctl" spawned
by agent
"/scratch/aime_ordb_mymachineemg/crso1/crshome_crso1/bin/oraagent.bin"
for action "stop" failed: details at "(:CLSN00010:)" in
"/scratch/aime_ordb_mymachineemg/crso1/crshome_crso1/log/mymachineemg/
agent/crsd/oraagent_aime/oraagent_aime.log"

CRS-2675: Stop of 'ora.LISTENER2.lsnr' on 'mymachineemg' failed

After fixing the cause of failure Run opatchauto resume with session
id "J5A3"

OPATCHAUTO-68061: The orchestration engine failed.

OPATCHAUTO-68061: The orchestration engine failed with return code 1

OPATCHAUTO-68061: Check the log for more details.