How an Oracle Database 12c Upgrade

Works in a Multitenant Environment

Joe Errede

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | How Database Upgrade works in a Multitenant Environment
 Introduction & Overview

What changed in upgrade

Working with Oracle Multitenant

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | How Database Upgrade works in a Multitenant Environment 2
Joe Errede from Boston Mass.

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | How Database Upgrade works in a Multitenant Environment 3
 Introduction & Overview

What changed in upgrade

Working with Oracle Multitenant

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | How Database Upgrade works in a Multitenant Environment 4
Upgrade SQL Automation

PREUPGRD.SQL produces either TEXT or XML output.

New Pre-Upgrade Script
upgrade.xml is used by (DBUA) catcon.pl -n 1 -e -b preupgrade -d
• preupgrd.sql '''.''' preupgrd.sql

• Executes pre-upgrade checks upgrade.xml contains ID, Creates preupgrade0.log

Status, Cause and Action to containing textual information
• Runs in source environment perform that is reviewed by DBA.

• Generates fixup scripts

All Output is written to
– preupgrade_fixups.sql $ORACLE_BASE/cfgtoollogs/sid/preupgrade
– postupgrade_fixups.sql
Generates preupgrade_fixups.sql
• MOS Note:884522.1 Executed in source home before the upgrade

Generates postupgrade_fixups.sql
Executed in target home after the upgrade

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

preupgrd.sql in a Multitenant Database

New Pre-Upgrade Script

IF (PDB1 )THEN
.
• Pre-Upgrade tools runs in every container .
using catcon.pl .
• catcon.pl -n 1 -e -b preupgrade -d '''.''' IF (PDB2) THEN
preupgrd.sql .
.
• Results are logged to preupgrd0.log base .
on the –b parameter IF (PDB3 )THEN
• The pre and post upgrade fix-up scripts run .
.
changes specific to each PDB .
• Intelligence has been built into the script

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Faster Upgrade – Less Downtime

New Parallel Upgrade catctl.pl

• catctl.pl
script1.sql script2.sql script3.sql
• Runs database upgrade in parallel
• Up to 40% faster upgrade script4.sql script7.sql script9.sql

• Used and proven by selected Oracle

Database 11g global customers script5.sql script8.sql script10.sql

– Telco billing
script6.sql script11.sql

– >100 SAP systems

– Large DWH script6.sql

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Faster Upgrade – Less Downtime

New Parallel Upgrade

• Parallel upgrade is performed in phases.
These phases are determined at run time by
parsing CATCTL tags within the SQL files
• Each phase builds dependencies for next
phase
• Each phase must complete before moving
onto the next phase
• Phases can be run in serial or in parallel

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

 Introduction & Overview

What changed in upgrade

Working with Oracle Multitenant

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | How Database Upgrade works in a Multitenant Environment 9
One Interface Used By All
catctl.pl
catctl.pl catcon.pl
sqlpatch.pl
• In 12.1.0.2.0 we broke out catctl.pl into a
common shared library called catcon.pm
• catcon.pm is used in upgrade, datapatch
and with catcon.pl catcon.pm
• catctl.pl calls packages within
catcon.pm to perform the upgrade.
• catctl.pl works with DBUA, MASS, RAC
and STANDBY upgrades.
DATABASE

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

How Upgrade Works in a Multitenant Database

catctl.pl CATCTL.PL
• Upgrade the CDB$ROOT first
• EXIT if there is any error in the upgrade and
abort the upgrade UPGRADE EXIT IF
• Upgrade the PDB’s next CDB$ROOT ERROR

UPGRADE UPGRADE
PDB1 PDB2

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

How Upgrade Works in a Multitenant Database

MAIN PROCESS
catctl.pl CATCTL.PL
• Each PDB upgrade is run as a child catctl.pl process
• Each child process is started by the main catctl
process
• Each child process run independently from main
CHILD CHILD
process PROCESS 1 PROCESS 2
• The main process waits for child processes to CATCTL.PL CATCTL.PL
complete
• The main process dispatches more child processes if
more PDB’s are in need of upgrading
• Upgrade processing is ordered by con_id’s
PDB1 PDB2

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Multiple PDB Upgrades

catctl.pl $ORACLE_HOME/perl/bin/perl catctl.pl -n catupgrd.sql

• Multiple PDB upgrades is controlled by -n

parameter
• -n tells catctl.pl how many PDB upgrades to run
together
• -n defaults to cpu_count/2 for example: -n PDB1 PDB2 PDB3 PDB n
Upgrade TO
Upgrade Upgrade Upgrade
– A cpu_count equal to 24 equates to 12 pdb
upgrades
• -n is divided by 2 and then rounded down. This
yields the number of PDB’s upgraded together
• Maximum concurrent upgrades is 32 minimum is 2

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

 Parallel Processing Within a PDB Upgrade

catctl.pl $ORACLE_HOME/perl/bin/perl catctl.pl -N catupgrd.sql

• Parallel processing is controlled by -N SQL PROC 1 SQL PROC 3 SQL PROC 5 SQL PROC N
parameter SQL PROC 2 SQL PROC 4 SQL PROC 6 SQL PROC N+1
• -N tells catctl .pl how many SQL processes are
used to upgrade the PDB database dictionary
• -N defaults to 2 for example:
– cpu_count equal to 24 equates to 12 -N PDB1 PDB2 PDB3 TO PDBn
pdb upgrades. Each pdb upgrade runs
2 SQL processes.
• Maximum value is 8 minimum value is 1

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Putting It All Together
$ORACLE_HOME/perl/bin/perl catctl.pl –n 8 catupgrd.sql CATCTL.PL

catctl.pl CATCON.PM

• Start catctl.pl with –n 8 CDB$ROOT UPGRADE EXIT IF

• Upgrade Root first FIRST ERROR
• Startup child catctl processes to upgrade 4 pdb’s at
a time START CHILD CATCTL.PL PROCESSES
• Startup 2 SQL processes per PDB because we took
the default CATCTL.PL CATCTL.PL CATCTL.PL CATCTL.PL
• Perform the upgrade
• As soon as one PDB is upgraded then the next PDB CATCON.PM CATCON.PM CATCON.PM CATCON.PM
can be processed until all the PDB’s have been
upgraded
SQL PROC 1&2 SQL PROC 3&4 SQL PROC 5&6 SQL PROC 7&8

PDB$SEED PDB1 PDB2 PDB3

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Upgrade Success or Failure

catctl.pl SQL> describe registry$error;

Name Null? Type
----------------------------------------- -------- ----------------------------
• All errors are written to registry$error. USERNAME VARCHAR2(256)
• Oracle Server component determines TIMESTAMP TIMESTAMP(6)
SCRIPT VARCHAR2(1024)
upgrade success or failure by looking at the
IDENTIFIER VARCHAR2(256)
registry$error table.
MESSAGE CLOB
• Components have their own validation STATEMENT CLOB
routines to determine status.

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

How to Evaluate Upgrade Logs in a Multitenant Database
Find Errors in
upg_summary.rpt
catctl.pl and identify the
• View the upgrade summary report: PDB Name
$ORACLE_BASE/cfgtoollogs/<sid>/upgrade/upg_summary.rpt

– Summary report is also included in

catupgrd0.log
Edit
• Report contains the status, errors and total catupgrdpdb_name
upgrade times for all the PDB’s that ran during 0..1.log
the upgrade
• PDB log files are named as follows
– catupgrdPDBNAME0…n.log (0..7 –N option)
• CDB$ROOT log files are named as follows Search for error
noted in summary
– catupgrd0…n.log (-n) (0…7 –n option)
report

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Container Lists Inclusion and Exclusion

catctl.pl o Container lists provide a way to prioritizing upgrades.

o Controls which upgrades are done first.
o Container lists support re-upgrades to specific PDB(s).
• Inclusion List
o Used in conjunction with the –p parameter you can restart a
– -c ‘CDB$ROOT PDB$SEED’ single PDB upgrade from a specific phase.
o Catctl.pl –p 74 –P 74
– Upgrades CDB$ROOT and PDB$SEED only o Upgrade log file identified each phase.
PHASE_TIME___START 74 PHASE_TIME___END 74
• Exclusion List
o Container lists are also used after we plug a PDB into a
– -C ‘CDB$ROOT’ container, where the container is at a higher release
– Upgrades everything but CDB$ROOT than the PDB.
o Create pluggable database PDB1 using
(‘PDB1.XML’) nocopy tempfile reuse;
o $ORACLE_HOME/perl/bin/perl catctl.pl –c ‘PDB1’
catupgrd.sql

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Debugging catctl.pl

CATCON.PM
catctl.pl catconInit: base for log and spool file names
= catupgrd
• Catcon.pm debugging. running catconInit(User = 0,
InternalUser = ,
– Catctl.pl -z catupgrd.sql SrcDir = 0,
• Catctl.pl tracing. This is done using perl to generate LogDir = 0,
trace files. Trace files are generated in the format LogBase = catupgrd,
catctl_YYYYMMDDHHMNSC_pid_trace.log CATCTL.PL
– Catctl.pl –Z 1 catupgrd.sql Package catctl.pl.
269: my @gArgs;
– All code is log along with entry and exit 270: foreach $argnum (0 .. $#ARGV)
function values. 271: {
272: push (@gArgs, $ARGV[$argnum]);
• By default catctl.pl will generate a stack trace 272: push (@gArgs, $ARGV[$argnum]);
should some catastrophic event occurs.
272: push (@gArgs, $ARGV[$argnum]);

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Other Parameters of Interest

catctl.pl o -l /tmp
o All log files will be placed in the /tmp directory.
• -l directory to use for spool log files o -I orcl
o All log files will be created with a prefix of
• -i identifier to use when creating catupgrdorcl
spool log files
o -d $ORACLE_HOME/rdbms/admin
• -d directory containing the files to o Tells catctl.pl where the upgrade files live.
be run
o -x
• -x Postpone running the post o Not advised postpones post upgrade
upgrade procedure procedures (datapatch and catuppst.sql).
o Must be done manually after upgrade the
completes.

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

 Normal Mode for Multitenant Databases

catctl.pl RUN UPGRADE

• The default behavior for Multitenant upgrades is
the following RESTART IN NORMAL MODE CDB$ROOT

• Upgrade CDB$ROOT in upgrade mode RUN POST UPGRADE

– Restart CDB$ROOT in normal mode
• Run post upgrade procedures
• Upgrade the PDB’s RUN UPGRADE
– Restart PDB in normal restricted mode RESTART IN NORMAL RESTRICTED MODE
PDB1 PDB2
• Run post upgrade procedures RUN POST UPGRADE
• Shutdown PDB SHUTDOWN

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Normal Mode for Multitenant databases

catctl.pl
RUN UPGRADE
• Advantages to default behavior RESTART IN NORMAL MODE CDB$ROOT
– As PDB’s become available they can be
brought online while other PDB’s are still RUN POST UPGRADE
being upgraded
• Disadvantages to default behavior
– More background processes are running
while upgrade is in progress RUN UPGRADE
– Online PDB’s will experience slow down RESTART IN NORMAL RESTRICTED MODE
while other PDB’s are being upgraded PDB1 PDB2
RUN POST UPGRADE
– Upgrades of the PDB’s tend to be slower
SHUTDOWN

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Upgrade Mode for Multitenant Databases

catctl.pl RUN UPGRADE

• Catctl –M catupgrd.sql RESTART IN UPGRADE MODE CDB$ROOT
• Upgrade CDB$ROOT in upgrade mode
RUN POST UPGRADE
– Restart CDB$ROOT in upgrade mode
• Run post upgrade procedures
• Upgrade the PDB’s RUN UPGRADE
– Restart PDB in normal restricted mode RESTART IN NORMAL RESTRICTED MODE
• Run post upgrade procedures PDB1 PDB2
RUN POST UPGRADE
• Shutdown PDB SHUTDOWN

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Upgrade Mode for Multitenant Databases

catctl.pl RUN UPGRADE

• Advantages to –M behavior RESTART IN UPGRADE MODE CDB$ROOT
– No background processes are running RUN POST UPGRADE
while PDB upgrades are in progress
– Upgrades of the PDB’s are faster
• Disadvantages to -M behavior RUN UPGRADE
– All PDB’s have to complete their RESTART IN NORMAL RESTRICTED MODE
PDB2
PDB1
upgrades before they can become RUN POST UPGRADE
available SHUTDOWN

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Upgrade Mode Versus Normal Mode

catctl.pl
• Can I have a show of hands on which
method you prefer?
• Upgrade Mode (“Total Time”)
– Faster upgrades but waiting until all the
upgrades have completed before you
can bring online
• Normal Mode (“Fast First”)
– Slower upgrades but able to bring your
PDB online as soon as the PDB upgrade
completes

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

In House Case Study 1

catctl.pl
• Upgraded 252 Pdb’s + CDB$ROOT+ PDB$SEED
• Linux Machine with 12 cores (Hyperthreading turn
on for 24 CPUs)
• 20 Gig Redo Log size with autoextend turn on
• 30 Gig sga_target
• Defaults were used when running catctl.pl
– 24/2 = 12 PDBS upgraded at a time
– 2 Sql processes per PDB
– Upgrade Mode Verses Normal Mode
• Upgrade mode was twice as fast

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

In House Case Study 2

catctl.pl
• Upgraded 25 Pdb’s + CDB$ROOT+ PDB$SEED
• Linux Exadata machine V1 with 2 physical CPU’s
(8 Hyperthreading cores)
• 20 Gig Redo Log size
• catctl.pl –n 18
– 18/2 = 9 PDBS upgraded at a time gave best
performance
– 2 Sql processes per PDB
– Upgrade Mode Verses Normal Mode
• Upgrade Mode 3-5% faster

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

In Conclusion…

catctl.pl
• The more PDB’s you have the better performance
you will get from upgrade mode (catctl –M)
• Different machines perform better with different
parameters
• Before you upgrade see what works best for your
system
– catctl.pl –n –N –M
• In most cases just taking the defaults maybe good
enough

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Resources
• Download all slides from:
– blogs.oracle.com/UPGRADE

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | How Database
Upgrade, Upgrade
Migrate works intoa Oracle
& Consolidate Multitenant Environment
Database 12c |• 29
Demo Booth Presence
• Discuss your upgrade at our demo booth
–Moscone South – SD-183

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | How Database
Upgrade, Upgrade
Migrate works intoa Oracle
& Consolidate Multitenant Environment
Database 12c |• 30
Recommended Upgrade Related Talks September 28 –
October 2, 2014
San Francisco

• Oracle Database 12c Upgrade: Tools and Best Practices from Oracle Support
– Tuesday, Sep 30, 5:00 PM - Moscone South - 310
• How and Why to Migrate from Schema Consolidation to Pluggable Databases
– Wednesday, Oct 1, 11:30 AM - Moscone South – 306
• Trouble Free Upgrade to Oracle Database 12c with Oracle Real Application Testing
– Wednesday, Oct 1, 2:00 PM - Moscone South – 303
• Database Patching Best Practices II CON7748
– Wednesday, Oct 1, 5:00 PM - Moscone South 104

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | How Database
Upgrade, Upgrade
Migrate works intoa Oracle
& Consolidate Multitenant Environment
Database 12c |• 31
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | How Database Upgrade works in a Multitenant Environment 32