Upgrade Procedure DB & OGG from 11g to 19C

Current Env we have Bi-directional replication setup between Window and Linux Server both servers have DB & OGG in 11g
version.

We need to upgrade DB & OGG from 11g to 19c.Considering both Source and Target OGG cannot be upgraded at same time

Window Server
DB : 11.2.0.4

OGG : 11.2

Linux Server
DB : 11.2.0.4
OGG : 11.2

We are going to divide Upgrade procedure of DB & OGG into different phases. As in most of cases we will not be able to
upgrade both Source and Target DB to 19C during same downtime period.

1st Phase
A. Upgrade Oracle DB in Linux Machine to 19c
B. Upgrade OGG in Linux Machine to 19c
C. Setup OGG in Linux machine to support Linux-OGG 19c with Window-OGG 11g

2nd Phase
A. Upgrade Oracle DB in Window Machine to 19c
B. Upgrade OGG in Window Machine to 19c
C. Setup OGG in Window machine to support Linux-OGG 19c with Window-OGG 19c
3rd Phase

A. Change compatibility parameter of DB to 19c and do necessary changes in OGG to support 19c DB

Assumption : -
1. 19c RDBMS home is already created in both Servers
2. 19c OGG software is download and copied to Servers
1st Phase
A. Upgrade Oracle DB in Linux Machine to 19c
Stop OGG process
Stop all OGG process in Linux machine & Extract Pump & Replicat in Window machine.
Disable OGG Trigger
If you fail to disable DDL Trigger your autougrade will fail with below error

DATABASE NAME: test01

CAUSE: ERROR at Line 227 in
[/tmp/autoupgrd/upg_logs/test01/test01/105/dbupgrade/test01_autocompile20201119113217test010.log]
REASON: ORA-04088: error during execution of trigger 'SYS.GGS_DDL_TRIGGER_BEFORE'
ACTION: [MANUAL]
DETAILS: 04088, 00000, "error during execution of trigger '%s.%s'"
// *Cause: A runtime error occurred during execution of a trigger.
// *Action: Check the triggers which were involved in the operation.

Upgrade Linux machine DB to 19c using autoupgrade utility

Download latest autoupgrade utility from AutoUpgrade MOS : Doc ID 2485457.1 and copy to staging directory in Server.

Please note autoupgrade utility require java 1.8+, I am using JAVA Home from 19c Oracle home.

Create sample config file and edit to as per your requirement

Run autoupgrade in analyze mode and check test01_preupgrade.html for any potential issue which require manual fix

Precheck file will be available in staging directory : /tmp/autoupgrd/upg_logs/test01/test01/100/prechecks

Run autoupgrade in deploy mode, it will take nearly 40 to 50 mins to preform upgrade.

Once DB is upgrade drop the GRP (Guaranteed restore point) & enable OGG Trigger and do not change compatible to 19 after DB upgrade
B. Upgrade OGG in Linux Machine to 19c, While DB is Upgrade running
Make sure to exit from ggsci command prompt before OGG upgrade otherwise you will end up corrupting ggsci utility while upgrading OGG to 19c
Below is the example if ggsci got corrupted and MOS Note for resolution.

[oracle@test-machine01 ucas]$ ./ggsci

./ggsci: symbol lookup error: ./ggsci: undefined symbol: _ZN3ggs5gglib9ggunicode7UString4nposE
[oracle@test-machine01 ucas]$

MOS : ./extract: symbol lookup error: ./extract: undefined symbol: _ZN3ggs5gglib9ggunicode7UString4nposE (Doc ID 2566438.1)

Navigate to directory where OGG 19c software is copied and invoke runInstaller

Select 19c option

I am using inplace method, So I am providing my current 11g OGG path and unselected “Start Manager option”

Click Yes
Once installation is finished check the version.

C. Setup OGG in Linux machine to support Linux-OGG 19c with Window-OGG 11g
Once both DB and OGG upgrade to 19c we need to do below steps in Linux Machine to run Linux-OGG 19c with Window-OGG 11g

1. Do not change DB compatible parameter to 19c after DB upgrade otherwise you will not be able to start Linux machine Extract in 11g trail
format and will receive below error if you try to start extract

ERROR OGG-02246 Source redo compatibility level 19.0.0 requires trail FORMAT 12.2 or higher.
ERROR OGG-01668 PROCESS ABENDING.

2 . If you are receiving remote trail in different location other than OGG_HOME add below parameter GLOBALS file ALLOWOUTPUTDIR
/u01/ggate/p_file_t169 otherwise you will receive below error

ERROR OGG-01031 There is a problem in network communication, a remote file problem, encryption keys for target and source do not match (if using
ENCRYPT) or an unknown error. (Reply received is GSOutput file /u01/ggate/p_file_t169/ra000001 is not in any allowed output directories.).

ERROR OGG-01668 PROCESS ABENDING.

3. Upgrade checkpoint table otherwise you will receive below error

ERROR OGG-05302 An error occurred while doing commit handling.

4. Create directory dirdmp in OGG_HOME.

5. Delete and Add Extract and Pump process with Extract parameter : FORMAT RELEASE 11.2 & TRANLOGOPTIONS EXCLUDETAG 00
Please note in OGG 19c TRANLOGOPTIONS EXCLUDEUSER GGS parameter will not work if you are using same user in GLOBALS file
GGSCHEMA GGS , Primary Extract will capture changes done by Replicat, Please refer below MOS note
(I decided to delete and add extract process because without these steps newly created remote trail file was corrupted)
TRANLOGOPTIONS EXCLUDEUSER Doesn't Work For The Schema Used in GLOBALS (Doc ID 2581689.1)
6. Start all process in Linux machine now.
 7. Alter Window machine replicate with below commands to reposition Replicat process with new generated trail file

2nd Phase
A. Upgrade Oracle DB in Window Machine to 19c
1. I am going to use manual method for Window machine DB Upgrade as with autoupgrade had facing many issues and bugs.
Use same steps as above to stop all Window machine OGG process and Linux Pump & Replicat process & disable OGG Trigger.

2. Purge recyclebin, Collect dictionary stats.

3. Run preupgrade tool , And review preupgrade.log file and execute preupgrade_fixups, Try to fix all manual preupgrade check.

4. Stop the database service, using either command-line commands, or PowerShell.

Delete the database service from the earlier release Oracle home.
Stop the listener from the earlier release Oracle home
Set the environment variables to the new Oracle home.
Copy database files, such as PFILE tnsnames.ora, listener.ora, password files, wallets, and other similar files to the new Oracle home.
5. After you create service in 19c home during DB startup you may receive below error. Just change db service logon user in service.msc
and restart service and try to login again, Start DB with upgrade option once db is started in upgrade mode execute dbupgrade

6. Once dbupgrade command finish, Start the database and check version and run postupgrade_fixups.sql and gather FIXED OBJECTS
STATS , Enable OGG Trigger and do not change compatible to 19 after DB upgrade. Otherwise you will not be able to generate 11.2
version trail file.
 B. Upgrade OGG in Window Machine to 19c, while db is getting upgrade
Make sure to exit ggsci command prompt before start of OGG upgrade. Navigate to directory where OGG 19c software is copied and invoke runInstaller

Select 19c option

I am using inplace method, So I am providing my current 11g OGG path and unselected “Start Manager option”

Once installation is finished check the version. If after OGG upgrade you encounter below error follow MOS 2584593.1 to fix issue.

The code execution cannot be proceed because MSVCR120.dll was not found.

GGSCI Fails to Run On Windows Server 2016 With Missing MSVCR120.dll Error (Doc ID 2584593.1)

C. Setup OGG in Window machine to support Linux-OGG 19c

Steps are same as we did in Linux machine.

1. Add ALLOWOUTPUTDIR D:\Oracle\ggate\p_file_t230 if you are receiving remote trail file in different location other than OGG_HOME

2. Upgrade REPLICAT Checkpoint table

3. Create dirdmp directory in OGG_HOME

4. Delete and Add Primary Extract and Pump process with Extract parameter FORMAT RELEASE 11.2 & TRANLOGOPTIONS EXCLUDETAG 00
And start all OGG process in Window machine and Pump process in Linux machine
5. Alter Linux machine replicate with below commands to reposition Replicat process with new generated trail file
3rd Phase
When you are ready to change DB compatible to '19.0.0' follow below steps, you can follow below steps for both Source and Target or only Source and
then target as per your requirement and downtime availability

1. Change DB compatible parameter to '19.0.0' – This step require DB restart so stop all OGG process as per above steps

2. Execute $OGG_HOME/ulg.sql (execute ulg.sql only if you are using trandata for supplemental logging, For schematrandata no need to
execute this script)

3. Edit Extract parameter file and remove FORMAT RELEASE 11.2 Start Extract and Pump with ETROLLOVER option, If you try to start Extract or
PUMP Process without ETROLLOVER option you will encounter below error.

INFO OGG-02922 File /u01/ggate/ext_file/ea000000009, with format RELEASE 11.2, does not match default format specification of RELEASE
19.1. Extract is going to write new trail file in the format RELEASE 11.2 to keep compatibility.
ERROR OGG-02246 Source redo compatibility level 19.0.0 requires trail FORMAT 12.2 or higher.
ERROR OGG-01668 PROCESS ABENDING.
Alter Window machine REPLICAT to start with newly created trail file.

If you can upgrade DB & OGG from 11g to 19c in same Downtime You can follow below steps
1. Stop all OGG process in both Source and Target DB and disable OGG Trigger
2. Upgrade both Source and Target DB to 19C
3. Upgrade both OGG to 19C
4. Change compatible parameter to '19.0.0' in both Source and Target DB
5. Run @$OGG_HOME/ulg.sql in both Source and Target DB
6. Follow below steps for in both Source & Target OGG
a. Add parameter ALLOWOUTPUTDIR in GLOBALS file if you are getting remote tail in different location other than OGG_HOME
b. Upgrade Checkpoint Table with command UPGRADE CHECKPOINTTABLE
c. Create dirdmp directory in OGG_HOME
d. Add parameter TRANLOGOPTIONS EXCLUDETAG 00 in Extract prm file
e. Start Extract and PUMP with ETROLLOVER option
f. Alter Replicat process in both machines to start with new trail file.
7. Enable OGG Trigger