Business / Technical Brief

Oracle GoldenGate
Microservices Architecture
with Oracle Real Application
Clusters Configuration Best
Practices

April, 2023, Version 2.32

Copyright © 2023, Oracle and/or its affiliates
Public

1 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

Purpose statement
This document describes the best practices for configuring Oracle GoldenGate Microservices Architecture, to work
with Oracle Real Application Clusters (RAC), Oracle Clusterware, and Oracle Database File System (DBFS) or Oracle
Advanced Cluster File System (ACFS).

Disclaimer
This document in any form, software or printed matter, contains proprietary information that is the exclusive property
of Oracle. Your access to and use of this confidential material is subject to the terms and conditions of your Oracle
software license and service agreement, which has been executed and with which you agree to comply. This
document and information contained herein may not be disclosed, copied, reproduced or distributed to anyone
outside Oracle without prior written consent of Oracle. This document is not part of your license agreement nor can it
be incorporated into any contractual agreement with Oracle or its subsidiaries or affiliates.

This document is for informational purposes only and is intended solely to assist you in planning for the
implementation and upgrade of the product features described. It is not a commitment to deliver any material, code,
or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing
of any features or functionality described in this document remains at the sole discretion of Oracle. Due to the nature
of the product architecture, it may not be possible to safely include all features described in this document without
risking significant destabilization of the code.

2 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

Table of contents

Purpose statement 2
Disclaimer 2
Introduction 5
Configuration Overview 5
Oracle GoldenGate 5
Oracle Real Application Clusters 6
Oracle Clusterware 7
Oracle Grid Infrastructure Agents 7
Oracle Database File System (DBFS) 7
Oracle Advanced Cluster File System (ACFS) 7
Configuration Best Practices 8
Step 1: Configure the Oracle Database for Oracle GoldenGate 8
Step 2: Create the Database Replication Administrator User 8
Step 3: Create the Database Services 8
Step 4: Set Up a File System on Oracle RAC 9
Oracle Database File System (DBFS) 9
Oracle Advanced Cluster File System (ACFS) 11
Step 5: Install Oracle GoldenGate 13
Step 6: Create the Oracle GoldenGate Deployment 13
Step 7: Oracle Clusterware Configuration 17
Step 8: Configure NGINX Reverse Proxy 22
Step 9: Create Oracle Net TNS Alias for Oracle GoldenGate Database Connections 23
Step 10: Configure Oracle GoldenGate Processes 24
Extract Configuration 24
Distribution Path Configuration 26
Replicat Configuration 28
Step 11: Configure Autostart of Extract and Replicat Processes 29
Summary of Recommendations when Deploying n Oracle RAC 31
References 32
Appendix A: Troubleshooting Oracle GoldenGate on Oracle RAC 33
1. XAG log file 33
2. XAG GoldenGate instance trace file 33
3. CRS trace file 34
4. GoldenGate deployment log files 35
5. GoldenGate report files 35
Example Configuration Problems 36
1. Incorrect parameter settings in the mount-dbfs.conf file 36
3 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

 2. Problems with file locking on DBFS 38
3. Critical Extract/Replicat process preventing GoldenGate startup 39

4 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

Introduction
This technical brief describes best practices for configuring Oracle GoldenGate Microservices Architecture, to work
with Oracle Real Application Clusters (RAC), Oracle Clusterware, and Oracle Database File System (DBFS) or Oracle
Advanced Cluster File System (ACFS). The target Oracle RAC system that hosts Oracle GoldenGate Microservices
Architecture can act as the source database, as the target database, or in some cases as both source and target
databases, for Oracle GoldenGate operations.

This technical brief is applicable for configuring Oracle GoldenGate Microservices Architecture with on-premises
systems, including Oracle Exadata. A separate technical brief will cover implementation with Exadata
Cloud@Customer and Exadata Cloud Services.

Oracle GoldenGate is instrumental in Oracle RAC configurations for many reasons, including the following:

• To support Oracle Platinum MAA reference architecture resulting in zero or near zero database and
application downtime for planned and unplanned outages. Refer to Oracle MAA Reference Architectures or
Oracle Exadata MAA - Platinum Tier Focused Presentation
• To migrate to an Oracle Database with minimal, or zero downtime
• To implement a near real-time data warehouse or consolidated database on Oracle RAC, sourced from
various, possibly heterogeneous, source databases, populated by Oracle GoldenGate
• To capture data from an OLTP application running on Oracle RAC to support further downstream
consumption, such as middleware integration

Configuration Overview
This section introduces Oracle GoldenGate Microservices Architecture, Oracle RAC, Oracle Clusterware, Oracle
Database File System (DBFS), and Oracle Advanced Cluster File System (ACFS). For more information about these
features, see the References section at the end of this technical brief.

Oracle GoldenGate
Oracle GoldenGate provides real-time, log-based change data capture and delivery between homogenous and
heterogeneous systems. This technology enables you to construct a cost-effective and low-impact real-time data
integration and continuous availability solution.

Oracle GoldenGate replicates data from committed transactions with transaction integrity and minimal overhead on
your existing infrastructure. The architecture supports multiple data replication topologies such as one-to-many,
many-to-many, cascading, and bidirectional. Its wide variety of use cases includes real-time business intelligence;
query offloading; zero-downtime upgrades and migrations; and active-active databases for data distribution, data
synchronization, and high availability.

Oracle GoldenGate Microservices Architecture provides REST-enabled services. The REST-enabled services provide
remote configuration, administration, and monitoring through HTML5 web pages, command line interfaces, and
APIs. Figure 1 shows the Oracle GoldenGate Microservices Architecture referenced throughout this technical brief.

Recommended Oracle GoldenGate 21c (and higher releases) introduces unified build support so a single software
installation supports capturing and applying replicated data to multiple major Oracle Database versions (11g Release
2 to 21c). This is possible because an Oracle GoldenGate installation includes the required Oracle Database client
libraries without requiring a separate database ORACLE_HOME installation.

5 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

Figure 1: Oracle GoldenGate Microservices Architecture

More information about Oracle GoldenGate Microservices Architecture can be found in the Oracle GoldenGate
documentation central hub by referencing the highest production GoldenGate release:

https://docs.oracle.com/en/middleware/goldengate/core/index.html

This technical brief is solely based on Oracle GoldenGate 21c. The latest version of Oracle GoldenGate can be
downloaded from:

http://www.oracle.com/technetwork/middleware/goldengate/downloads/index.html

For information about configuring Oracle GoldenGate Classic Architecture with Oracle RAC, see the “Oracle
GoldenGate with Oracle Real Application Clusters Configuration” technical brief:

http://www.oracle.com/technetwork/database/features/availability/maa-goldengate-rac-2007111.pdf

Oracle Real Application Clusters

Oracle Real Application Clusters (Oracle RAC) enables multiple instances that are linked by an interconnect to share
access to an Oracle database. In an Oracle RAC environment, the database runs on two or more systems in a cluster
while concurrently accessing a single shared database. The result is a single database that spans multiple hardware

6 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

systems, enabling Oracle RAC to provide high availability and redundancy during failures in the cluster. Oracle RAC
accommodates all system types, from read-only data warehouse systems to update-intensive online transaction
processing (OLTP) systems.

Oracle RAC and Oracle Clusterware allow Oracle Database to run any packaged or custom application across a set of
clustered servers. This capability provides continual database service uptime for node and instance failures, most
planned maintenance activities, and for Oracle RAC expansion. If an Oracle RAC clustered node or instance fails, the
Oracle Database service continues running on the surviving nodes and instances. When more processing power is
needed, you can add another node without interrupting user access to the database or data.

Oracle Clusterware
Oracle Clusterware is a cluster manager that is designed specifically for Oracle Database. In an Oracle RAC
environment, Oracle Clusterware monitors all Oracle resources (such as database instances and listeners). If a failure
occurs, Oracle Clusterware automatically attempts to restart the failed resource. During outages, Oracle Clusterware
relocates the processing performed by the inoperative resource to a backup resource. For example, if a node fails,
then Oracle Clusterware relocates the database services used by the application to surviving RAC nodes and instances
in the RAC cluster.

Oracle Grid Infrastructure Agents

Oracle Grid Infrastructure Agents (XAG) are Oracle Grid Infrastructure components that provide the high availability
(HA) framework to application resources and resource types managed through the agent management interface,
AGCTL. This framework provides a complete, ready to use solution that contains pre-defined Oracle Grid
Infrastructure resource configurations and agents to integrate applications for complete application HA.

The Oracle Grid Infrastructure Agents provide pre-defined Oracle Clusterware resources for Oracle GoldenGate,
Siebel, Oracle PeopleSoft, JD Edwards, and Oracle WebLogic Server, as well as Apache and MySQL applications. Using
the agent for Oracle GoldenGate simplifies the creation of dependencies on the source/target database, the
application VIP, and the file system (ACFS or DBFS) mount point. The agent command line utility (AGCTL) is used to
start and stop Oracle GoldenGate and can also be used to relocate Oracle GoldenGate between the nodes in the
cluster.

Oracle Database File System (DBFS)

The Oracle Database File System (DBFS) creates a file system interface to files stored in the database. DBFS is similar
to NFS in that it provides a shared network file system that looks like a local file system. Because the data is stored in
the database, the file system inherits all the high availability and disaster recovery capabilities provided by Oracle
Database.

With DBFS, the server is the Oracle Database. Files are stored as SecureFiles LOBs. PL/SQL procedures implement file
system access primitives such as create, open, read, write, and list directory. The implementation of the file system in
the database is called the DBFS SecureFiles Store. The DBFS SecureFiles Store allows users to create file systems that
can be mounted by clients. Each file system has its own dedicated tables that hold the file system content.

Oracle Advanced Cluster File System (ACFS)

Oracle Advanced Cluster File System (Oracle ACFS) is a multi-platform, scalable file system, and storage management
technology that extends Oracle Automatic Storage Management (Oracle ASM) functionality to support all customer
files.

Oracle ACFS leverages Oracle Clusterware for cluster membership state transitions and resource-based high
availability. Oracle ACFS is bundled into the Oracle Grid Infrastructure (GI) allowing for integrated optimized
management of databases, resources, volumes, and file systems.

7 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

Configuration Best Practices

Step 1: Configure the Oracle Database for Oracle GoldenGate

The source and target Oracle GoldenGate databases should be configured using the following recommendations.

• Enable Oracle GoldenGate replication by setting the database initialization parameter

ENABLE_GOLDENGATE_REPLICATION=TRUE.
• Run the Oracle GoldenGate source database in ARCHIVELOG mode.
• Enable FORCE LOGGING mode in the Oracle GoldenGate source database.
• Enable minimal supplemental logging in the source database. Additionally, add schema or table level logging for
all replicated objects.
• Configure the streams pool in the System Global Area (SGA) on the source database using the
STREAMS_POOL_SIZE initialization parameter. The streams pool is only needed on the target database if
integrated Replicat will be used. The value of STREAMS_POOL_SIZE should be set to the following value:
STREAMS_POOL_SIZE = (#Extracts and #Integrated Replicats * 1GB) * 1.25

For example, in a database with 2 Extracts and 2 integrated Replicats:

STREAMS_POOL_SIZE = 4GB * 1.25 = 5GB

When adding Extract or Replicat processes, it is important to recalculate, and configure the new streams pool size
requirement.

For further information on preparing the database for Oracle GoldenGate, refer to the Using Oracle GoldenGate with
Oracle Database Guide at

https://docs.oracle.com/en/middleware/goldengate/core/21.3/oracle-db/preparing-database-oracle-
goldengate.html#GUID-E06838BD-0933-4027-8A6C-D4A17BDF4E41

Step 2: Create the Database Replication Administrator User

The source and target Oracle databases need a GoldenGate Administrator user created, with appropriate privileges
assigned. For single tenant databases, refer to the Using Oracle GoldenGate with Oracle Database guide at

https://docs.oracle.com/en/middleware/goldengate/core/21.3/oracle-db/establishing-oracle-goldengate-
credentials.html#GUID-F9EBB989-E22F-4355-BE60-40F957B8515E

For a multitenant source database, GoldenGate Extract must be configured to connect to a user in the root container
database, using a c## account. For a multitenant target database, a separate GoldenGate administrator user is
needed for each PDB that a Replicat applies data to. For further details on creating a GoldenGate Administrator in and
Oracle Multitenant Database, refer to the Using Oracle GoldenGate with Oracle Database guide at

https://docs.oracle.com/en/middleware/goldengate/core/21.3/oracle-db/configuring-oracle-goldengate-
multitenant-container-database-1.html#GUID-0B0CEB35-51C6-4319-BEE1-FA208FF4DE05

Step 3: Create the Database Services

A database service is required so that the Oracle Grid Infrastructure Agent will automatically start the GoldenGate
deployment when the database is opened. When DBFS is used for the shared file system, the database service is also
used to mount DBFS to the correct RAC instance.

8 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

When using a source multitenant database, a separate service is required for the root container database (CDB) and
the pluggable database (PDB) that contains the schema being replicated. For a target multitenant database, a single
service is required for the PDB.

Create the service using the following command, as the oracle user.

$ srvctl add service -db <db_name> -service <service name> -preferred <instance_1>
-available <instance_2, instance_3 etc.> -pdb <PDB name>

Example:

$ srvctl add service -db ggdb -service oggserv_pdb -preferred ggdb1 -available ggdb2
–pdb GGPDB01

It not using Oracle Multitenant Database, omit the -pdb parameter.

Refer to the Real Application Clusters Administration and Deployment Guide for further details on creating a database
service at https://docs.oracle.com/en/database/oracle/oracle-database/19/racad/index.html.

Step 4: Set Up a File System on Oracle RAC

Oracle GoldenGate Microservices Architecture is designed with a simplified installation and deployment directory
structure. The installation directory should be placed on local storage on each Oracle RAC node to provide minimized
downtime during software patching. The deployment directory, which is created during deployment creation using
the Oracle GoldenGate Configuration Assistant (oggca.sh), must be placed on a shared file system. The deployment
directory contains configuration, security, log, parameter, trail, and checkpoint files. Placing the deployment in DBFS
or ACFS provides the best recoverability and failover capabilities in the event of a system failure. Ensuring the
availability of the checkpoint files cluster-wide is essential so that after a failure occurs the GoldenGate processes can
continue running from their last known position.

If Oracle GoldenGate will be configured along with Oracle Data Guard, the recommended file system is DBFS. DBFS is
contained in the database protected by Data Guard, and can be fully integrated with XAG. In the event of a Data Guard
role transition, the file system can be automatically mounted on the new primary server, followed by automated
startup of Oracle GoldenGate. This is currently not possible with ACFS, because it is not part of the Oracle Data Guard
configuration.

Follow the instructions below to configure the chosen file system.

Oracle Database File System (DBFS)

It is required that you create the DBFS tablespace inside the same database that the Oracle GoldenGate processes are
connected to. For example, if a GoldenGate integrated Extract process is extracting from a database called GGDB, the
DBFS tablespace would be located in the same GGDB database.

Follow instructions in My Oracle Support note 869822.1 to install the required FUSE libraries if they are not already
installed.

Use the instructions in My Oracle Support note 1054431.1 to configure the database, tablespace, database user,
tnsnames.ora Oracle Net connection alias, and permissions on source or target GoldenGate environments required
for DBFS.

NOTE: When using an Oracle Multitenant Database, the DBFS tablespace MUST be created in a Pluggable Database
(PDB). It is recommended that you use the same PDB that the GoldenGate Extract or Replicat processes are connecting
to, allowing DBFS to use the same database service, created above in step 2, for its database dependency.

9 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

Create a file system for storing the GoldenGate deployment files. It is recommended that you allocate enough trail file
disk space to permit storage of up to 12 hours of trail files. Doing this will give sufficient space for trail file generation
should a problem occur with the target environment that prevents it from receiving new trail files. The amount of
space needed for 12 hours can only be determined by testing trail file generation rates with real production data.

Example DBFS creation:

$ cd $ORACLE_HOME/rdbms/admin
$ sqlplus dbfs_user/dbfs_password@<database_tns_alias>
SQL> start dbfs_create_filesystem dbfs_gg_tbs goldengate

Follow the instructions in My Oracle Support note 1054431.1 for configuring the newly created DBFS file system so
that the DBFS instance and mount point resources are automatically started by Cluster Ready Services (CRS) after a
node failure, with the following DBFS configuration and script file modifications.

1. Change the mount-dbfs.conf parameters to reflect your database environment.

Modify the MOUNT_OPTIONS parameter to the following:

MOUNT_OPTIONS=allow_other,direct_io,failover,nolock

The failover option forces all file writes to be committed to the DBFS database in an IMMEDIATE WAIT mode.
This prevents data getting lost when it has been written into the dbfs_client cache but not yet written to the
database at the time of a database or node failure.

The nolock mount option is required if you are using Oracle Database 18c or later versions, due to a change in
the DBFS file locking, which can cause issues for GoldenGate processes after a RAC node failure when a file is
currently locked.

If you are using a dbfs_client from Oracle Database 12c Release 2 (12.2), make sure you have applied the
latest release update that includes the fix for bug 27056711. Once the fix has been applied, the MOUNT_OPTIONS
should also include the nolock option.

2. Modify the mount-dbfs.sh script to force unmounting of DBFS when the CRS resource is stopped.
Change two occurrences of:

$FUSERMOUNT -u $MOUNT_POINT

To the following:

$FUSERMOUNT -uz $MOUNT_POINT

3. When registering the resource with Oracle Clusterware, be sure to create it as a cluster_resource instead of a
local_resource as specified in the My Oracle Support note. The reason for using cluster_resource is so
the file system can only be mounted on a single node at one time, preventing mounting of DBFS from concurrent
nodes, which creates the potential for concurrent file writes, causing file corruption problems.
Make sure to use the database service name created in a previous step for the DBFS service dependency.

Example:

DBNAME=ggdb
DEPNAME=ora.$DBNAME.oggserv.svc

crsctl add resource $RESNAME \

-type cluster_resource \

10 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

 -attr "ACTION_SCRIPT=$ACTION_SCRIPT, \
CHECK_INTERVAL=30,RESTART_ATTEMPTS=10, \
START_DEPENDENCIES='hard($DEPNAME)pullup($DEPNAME)',\
STOP_DEPENDENCIES='hard($DEPNAME)',\
SCRIPT_TIMEOUT=300"

Once the DBFS resource has been created, the file system should be mounted and tested.

$ crsctl start res dbfs_mount

$ crsctl stat res dbfs_mount

After the file system is mounted, create the directory for storing the GoldenGate files.

$ cd /mnt/dbfs/goldengate
$ mkdir deployments

NOTE: Leave the shared file system mounted. It is required for creating the GoldenGate deployment in a later step.

Oracle Advanced Cluster File System (ACFS)

Oracle ACFS is an alternative to DBFS for the shared GoldenGate files in an Oracle RAC configuration. Refer to My
Oracle Support note 1929629.1 for more information about ACFS configuration requirements for Oracle Exadata
Database Machine.

Create a single ACFS file system for storing the Oracle deployment files.

It is recommended that you allocate enough trail file disk space to permit storage of up to 12 hours of trail files. Doing
this provides sufficient space for trail file generation should a problem occur with the target environment that
prevents it from receiving new trail files. The amount of space needed for 12 hours can only be determined by testing
trail file generation rates with real production data.

1. Create the file system using ASMCMD as the Oracle ASM administrator user.
ASMCMD [+] > volcreate -G datac1 -s 1200G ACFS_GG

Note: Modify the file system size according to the determined size requirements.

ASMCMD> volinfo -G datac1 acfs_gg

Diskgroup Name: DATAC1

Volume Name: ACFS_GG
Volume Device: /dev/asm/acfs_gg-151
State: ENABLED
Size (MB): 1228800
Resize Unit (MB): 64

Redundancy: MIRROR
Stripe Columns: 8
Stripe Width (K): 1024

11 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

 Usage:
Mountpath:

Make the file system with the following mkfs command.

$ /sbin/mkfs -t acfs /dev/asm/acfs-gg-151

2. Create the CRS resource for the newly created ACFS file system, if not already created.
Check to see if the file system resource was already created.

$ srvctl status filesystem -volume ACFS_GG -diskgroup DATAC1

ACFS file system /mnt/acfs_gg is mounted on nodes oggadm07, oggadm08

If not already created, create the ACFS mount point on all RAC nodes.

# mkdir -p /mnt/acfs_gg

Create the file system resource as the root user. Due to the implementation of distributed file locking on ACFS,
unlike DBFS, it is acceptable to mount ACFS on more than one RAC node at any one time.

Create the ACFS resource using srvctl from the Oracle Grid Infrastructure ORACLE_HOME.

# srvctl add filesystem -device /dev/asm/acfs_gg-151 -volume ACFS_GG

-diskgroup DATAC1 -path /mnt/acfs_gg -user oracle -autostart RESORE

To verify the currently configured ACFS file systems, use the following command to view the file system details.

$ srvctl config filesystem

Volume device: /dev/asm/acfs_gg-151

Diskgroup name: datac1
Volume name: ACFS_GG
Canonical volume device: /dev/asm/acfs_gg-151
Accelerator volume devices:
Mountpoint path: /mnt/acfs_gg
Mount point owner: oracle

Check the status of the ACFS resource and mount it.

$ srvctl status filesystem -volume ACFS_GG -diskgroup DATAC1

ACFS file system /mnt/acfs is not mounted

$ srvctl start filesystem -volume ACFS_GG -diskgroup DATAC1 –node dc1north01

The CRS resource that is created is named using the format ora.diskgroup_name.volume_name.acfs. Using
the above file system example, the CRS resource is called ora.datac1.acfs_gg.acfs.

To see all ACFS file system CRS resources that currently exist, use the following command.

$ crsctl stat res -w "((TYPE = ora.acfs.type) OR (TYPE = ora.acfs_cluster.type))"

12 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

 NAME=ora.datac1.acfs_gg.acfs
TYPE=ora.acfs.type
TARGET=ONLINE , OFFLINE
STATE=ONLINE on dc1north01, OFFLINE

3. Create a GoldenGate deployment directory on ACFS.

After the file system is mounted, create the directory for storing the GoldenGate deployments.

$ cd /mnt/acfs_gg
$ mkdir deployments

Refer to the Oracle Automatic Storage Management Administrator’s Guide for more information about ACFS:

https://docs.oracle.com/en/database/oracle/oracle-database/19/ostmg/index.html

NOTE: Leave the shared file system mounted. It is required for creating the GoldenGate deployment in a later step.

Step 5: Install Oracle GoldenGate

Download the Oracle GoldenGate 21c Microservices software, or higher, from Oracle Technology Network (OTN) at:

http://www.oracle.com/technetwork/middleware/goldengate/downloads/index.html

Install the Oracle GoldenGate software locally on all nodes in the Oracle RAC configuration that will be part of the
GoldenGate configuration. Make sure the installation directory is the identical on all nodes.

Follow the generic installation instructions detailed in Installing Oracle GoldenGate:

https://docs.oracle.com/en/middleware/goldengate/core/21.3/coredoc/index.html

Step 6: Create the Oracle GoldenGate Deployment

Once the Oracle GoldenGate software has been installed, the next step is to create a deployment using the Oracle
GoldenGate Configuration Assistant (oggca).

There are two limitations that currently exist with Oracle GoldenGate and XAG:

1. A Service Manager that is registered with XAG can only manage a single deployment. If multiple deployments are
required, each deployment must use their own Service Manager. Oracle GoldenGate release 21c simplifies this
requirement because it uses a single deployment to support Extract and Relicat processes connecting to different
versions of the Oracle Database.
2. Each Service Manager registered with XAG must belong to separate OGG_HOME software installation directories.
Instead of installing Oracle GoldenGate multiple times, the recommended approach is to install Oracle
GoldenGate one time, and then create a symbolic link for each Service Manager OGG_HOME.
Example:

$ echo $OGG_HOME
/u01/oracle/goldengate/gg21c_MS

$ ln –s /u01/oracle/goldengate/gg21c_MS /u01/oracle/goldengate/gg21c_MS_ggnorth

$ export OGG_HOME=/u01/oracle/goldengate/gg21c_MS_ggnorth
$ $OGG_HOME/bin/oggca.sh

13 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

 The symbolic link and OGG_HOME environment variable must be configured before running the Oracle
GoldenGate Configuration Assistant on all Oracle RAC nodes.

Recommendations for creating the GoldenGate deployment in the Oracle GoldenGate Configuration Assistant are as
follows.

1. In Service Manager Options, specify the following for the creation of a new Service Manager.
a. In the Service Mnaager Details pane, select ‘Create New Service Manager’.
b. Enter the ‘Service Manager Deployment Home’ location on the shared DBFS or ACFS file system.
c. Select to ‘Integrate with XAG’
d. In the Service Manager Connection Details pane, specify localhost in the Listening
hostname/address field. Using localhost allows the deployment to be started on all of the RAC
nodes without the need for a Virtual IP address (VIP).
e. Enter the port number in Listening port

Below is a screenshot of the recommended configuration.

Figure 2: Oracle GoldenGate Microservices Service Manager configuration

2. In Deployment Directories, specify the Deployment home directory on the shared DBFS or ACFS file system:

14 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

Figure 3: Deployment home directory configuration

3. In Environment Variables, specify a correct TNS_ADMIN directory.

Starting with Oracle GoldenGate release 21.3, a database ORACLE_HOME is no longer required because the
required database libraries are installed as part of the Oracle GoldenGate installation. It is recommended that
you use TNS_ADMIN directory outside of any existing ORACLE_HOME directories.

In the example screenhost below, the entry is configured as /u01/oracle/goldengate/network/admin

Figure 4: Deployment environment variables

4. In Security Options, do not select ‘SSL/TLS Security’.

External access to GoldenGate Microservices server is achieved by using NGINX Reverse Proxy SSL-
termination. Secure access and communication to the GoldenGate deployments will be exclusively through
the SSL port 443. Internal connectivity within the same local host between NGINX and GoldenGate does not
require SSL.

5. In Port Settings, if the Management Pack for Oracle GoldenGate has been licensed, select ‘Enable Monitoring’
to use the performance metric server using either Berkeley Database (BDB) or Lightening Memory Database
(LMDB).

15 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

Figure 5: Selecting the Metric Server Datastore type

For both BDB and LMDB Metrics Service DataStore types, set the ‘Metrics Service DataStore home’ directory
to a local directory that exists on all RAC nodes. For example:
/u01/oracle/goldengate/datastores/<deployment name>

Continue through the Oracle GoldenGate Configuration Assistant until the deployment is created.

After the deployment has been created, if you are using DBFS for the shared file system and the database version is
lower than Oracle Database Release 21c (21.3), run the following commands to move the GoldenGate deployment
temp directory from DBFS to local storage.

On the first node:

$ cd <DBFS GoldenGate deployment home directory/var
$ mkdir -p <local storage directory>/<deployment_name>
$ mv temp <local storage directory>/<deployment_name>
$ ln -s <local storage directory>/<deployment_name>/temp temp

On all other nodes:

$ mkdir <local storage directory>/<deployment_name>/temp

First node example:

$ cd /mnt/dbfs/goldengate/deployments/ggnorth/var
$ mkdir –p /u01/oracle/goldengate/deployments/ggnorth
$ mv temp /u01/oracle/goldengate/deployments/ggnorth
$ ln -s /u01/oracle/goldengate/deployments/ggnorth/temp temp

On all other nodes:

$ mkdir /u01/oracle/goldengate/deployments/ggnorth/temp

16 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

Step 7: Oracle Clusterware Configuration
The following procedure shows you how to configure Oracle Clusterware to manage Oracle GoldenGate using the
Oracle Grid Infrastructure Standalone Agent (XAG). Using XAG automates the mounting of the shared file system
(DBFS or ACFS) and the stopping and starting of the GoldenGate deployment when relocating between Oracle RAC
nodes.

1. Install the Oracle Grid Infrastructure Standalone Agent.

It is recommended that you install the XAG software as a standalone agent outside of the Grid Infrastructure
ORACLE_HOME. This allows you to use the latest XAG release available, and the software can be updated without
impact to the Grid Infrastructure.

When using Oracle GoldenGate Microservices Architecture you MUST use XAG version 10.2 or later.

The latest agent software is available for download from the following location:

http://www.oracle.com/technetwork/database/database-technologies/clusterware/downloads/xag-agents-
downloads-3636484.html

Install the XAG standalone agent outside of the Oracle Grid Infrastructure home directory. XAG must be installed
in the same directory on all RAC nodes in the cluster where Oracle GoldenGate is installed.

For example, as the Oracle Grid Infrastructure user, the default of oracle:

$ ./xagsetup.sh --install --directory /u01/oracle/xag --all_nodes

Add the location of the newly installed XAG software to the PATH variable so that the location of agctl is known
when the oracle user logs on to the machine.

$ cat .bashrc
export PATH=/u01/oracle/xag/bin:$PATH

NOTE: It is important to make sure that the XAG bin directory is specified BEFORE the Grid Infrastructure bin directory,
to ensure the correct agctl binary is found. Set this location in the oracle user environment to take effect at time of
logging on, such as in the .bashrc file when the Bash shell is in use.

2. Prepare for Application Virtual IP Address (VIP) Creation.

A dedicated application VIP is required to allow access to the GoldenGate Microservices using the same host
name, regardless of which RAC node is hosting the services. An application VIP also ensures that the GoldenGate
Distribution Server can communicate with the Distribution Receiver running the current RAC node.

The VIP is a cluster resource that Oracle Clusterware manages. The VIP is assigned to a cluster node and is
automatically migrated to another node in the event of a node failure.

There are two pieces of information needed before creating the application VIP:

• The network number, which can be identified using the following command.
$ crsctl status resource -p -attr ADDRESS_TYPE,NAME,USR_ORA_SUBNET -w "TYPE =
ora.network.type" |sort | uniq

ADDRESS_TYPE=IPV4
NAME=ora.net1.network

17 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

 USR_ORA_SUBNET=10.133.16.0

The net1 in NAME=ora.net1.network indicates that this is network 1, and it is of type IPV4.

• The IP address for the new Application VIP, provided by your system administrator. This IP address must be
in the same subnet of the cluster environment as determined above.
The VIP will be created in the next step, when configuring the Oracle Grid Infrastructure Agent.

3. Configure Oracle Grid Infrastructure Agent (XAG).

Oracle GoldenGate must be registered with XAG so that the deployment is started and stopped automatically
when the database is started and the file system is mounted.

To register Oracle GoldenGate Microservices Architecture with XAG use the following command format.

agctl add goldengate <instance_name>

--gg_home <GoldenGate_Home>
--service_manager
--config_home <GoldenGate_SvcMgr_Config>
--var_home <GoldenGate_SvcMgr_Var Dir>
--port <port number>
--oracle_home <$OGG_HOME/lib/instantclient>
--adminuser <OGG admin user>
--user <GG instance user>
--group <GG instance group>
--network <network_number>
--ip <ip_address>
--vip_name <vip_name>
--filesystems <CRS_resource_name>
--db_services <service_name>
--use_local_services
--nodes <node1, node2, ... ,nodeN>

Where:

--gg_home specifies the location of the Oracle GoldenGate software. Specify the OGG_HOME symbolic link for
the OGG_HOME if registering multiple Service Managers (see Step 6: Create the Oracle GoldenGate Deployment).

--service_manager indicates this is a GoldenGate Microservices instance.

--config_home specifies the GoldenGate Service Manager deployment configuration home directory.

--var_home specifies the GoldenGate Service Manager deployment variable home directory.

--port specifies the deployment Service Manager port number.

--oracle_home specifies the location of the Oracle database libraries that are included as part of Oracle
GoldenGate 21c and later releases. Example: $OGG_HOME/lib/instantclient

--adminuser specifies the Oracle GoldenGate Microservices administrator account name.

--user specifies the name of the operating system user that owns the GoldenGate deployment.

--group specifies the name of the operating system group that owns the GoldenGate deployment.

--network specifies the network subnet for the VIP, determined above in step 7.2.

18 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

 --ip specifies the IP address for the VIP, which was determined above in step 7.2. If you have already created a
VIP, then specify it using the --vip_name <vip_name> parameter in place of --network and --ip.
--vip_name specifies a CRS resource name for an application VIP that has previously been created. This
parameter replaces --network and –ip (optional).

--filesystems specifies the DBFS or ACFS CRS file system resource that must be mounted before the
deployment is started.

--db_services specifies the ora.<database>.<service_name>.svc service name that was created in the previous
step. If using Oracle Multitenant Database, specify the PDB database service for Replicat, or the CDB database
service for an Extract. If using both Replicat and Extract, specify both services names, separated by a comma.

--use_local_services specifies that the GoldenGate instance must be co-located on the same RAC node
where the db_services service is running.

--nodes specifies which of the RAC nodes this GoldenGate instance can run on. If GoldenGate is configured to
run on any of the RAC nodes in the cluster, this parameter should still be used to determine the preferred order of
nodes to run Oracle GoldenGate.

Notes:

• The GoldenGate instance registration with XAG MUST be run as the root user.
• The user and group parameters are mandatory because the GoldenGate registration with XAG is run as
the root user.

Below are some examples of registering Oracle GoldenGate with XAG.

RAC cluster using DBFS, using an already created application VIP

# agctl add goldengate GGNORTH \

--gg_home /u01/oracle/goldengate/gg21c_MS \
--service_manager \
--config_home /mnt/dbfs/goldengate/deployments/ggsm01/etc/conf \
--var_home /mnt/dbfs/goldengate/deployments/ggsm01/var \
--port 9100 \
--oracle_home /u01/oracle/goldengate/gg21c_MS/lib/instantclient
--adminuser oggadmin
--user oracle \
--group oinstall \
--vip_name gg_vip_prmy \
--filesystems dbfs_mount \
--db_services ora.ds19c.oggserv.svc \
--use_local_services \
--nodes dc1north01,dc1north02

Where:

• GoldenGate instance is GGNORTH

• GoldenGate home directory is /u01/oracle/goldengate/gg21c_MS
• This is an Oracle GoldenGate Microservices Architecture instance (--service_manager)

19 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

 • GoldenGate deployment configuration home directory is
/mnt/dbfs/goldengate/deployments/ggsm01/etc/conf
• GoldenGate deployment variable home directory is /mnt/dbfs/goldengate/deployments/ggsm01/var
• Deployment Service Manager port number is 9100
• Oracle GoldenGate Microservices administrator account name is oggadmin
• The GoldenGate user is oracle in the group oinstall
• Application VIP name, managed by CRS, is called gg_vip_prmy
• The CRS resource name for the file system the deployment depends on is dbfs_mount
• The GoldenGate instance will be started on the same RAC node as the CRS service called
ora.ds19c.oraserv.svc will be co-located on the same RAC node as this GoldenGate instance.

RAC cluster, using ACFS, with an application VIP running on a subset of the nodes in the cluster.

# agctl add goldengate GGNORTH \

--gg_home /u01/oracle/goldengate/gg21c_MS \
--service_manager \
--config_home /mnt/acfs/goldengate/deployments/ggsm01/etc/conf \
--var_home /mnt/acfs/goldengate/deployments/ggsm01/var \
--port 9100 \
--oracle_home /u01/oracle/goldengate/gg21c_MS/lib/instantclient
--adminuser admin \
--user oracle \
--group oinstall \
--network 1 --ip 10.13.11.203 \
--filesystems ora.datac1.acfs_gg.acfs \
--db_services ora.ds19c.oraserv.svc \
--use_local_services \
--nodes dc1north01,dc1north02

Where:

• GoldenGate instance is GGNORTH

• GoldenGate home directory is /u01/oracle/goldengate/gg21c_MS
• This is an Oracle GoldenGate Microservices Architecture instance (--service_manager)
• GoldenGate deployment configuration home directory is
/mnt/acfs/goldengate/deployments/ggsm02/etc/conf
• GoldenGate deployment variable home directory is /mnt/acfs/goldengate/deployments/ggsm02/var
• Deployment Service Manager port number is 9100
• Oracle GoldenGate Microservices administrator account name is admin
• GoldenGate user is oracle in the group oinstall
• The network is the default ora.net1.network and the VIP is 10.13.11.203
• The CRS resource name for the file system the deployment depends on is ora.datac1.acfs_gg.acfs
• This GoldenGate instance will be started on the same RAC node as the CRS service called
ora.ds19c.oraserv.svc will be co-located on the same RAC node as this GoldenGate instance

20 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

 • Oracle GoldenGate will only run on RAC nodes dc1north01 and dc1north02, listed in priority order.

Below are some example agctl commands that are used to manage the GoldenGate deployment with XAG.

To check the status of Oracle GoldenGate:

% agctl status goldengate

Goldengate instance 'GGNORTH' is running on dc1north01

To start the GoldenGate deployment, and all Extract/Replicat processes that have been configured to autostart
(instructions in a later step):

% agctl start goldengate GGNORTH --node dc1north02

To stop the GoldenGate deployment:

% agctl stop goldengate GGNORTH

To manually relocate the GoldenGate deployment to another node:

% agctl relocate goldengate GGNORTH --node dc1north02

To view the configuration parameters for the GoldenGate resource:

% agctl config goldengate GGNORTH

Instance name: GGNORTH

Application GoldenGate location is: /u01/oracle/goldengate/gg21c_MS
Goldengate MicroServices Architecture environment: yes
Goldengate Service Manager configuration directory:
/mnt/dbfs/goldengate/deployments/ggsm01/etc/conf

Goldengate Service Manager var directory: /mnt/dbfs/goldengate/deployments/ggsm01/var

Service Manager Port: 9100

Goldengate Administration User: oggadmin
Autostart on DataGuard role transition to PRIMARY: no
Configured to run on Nodes: dc1north01 dc1north02
ORACLE_HOME location is: /u01/oracle/goldengate/gg21c_MS/lib/instantclient
Database Services needed: ora.cdb1.oggcdb.svc [use_local_services]
File System resources needed: ora.datac1.acfs_gg.acfs
Network: 1, IP: 10.13.11.203, User:oracle, Group:oinstall

To delete the GoldenGate XAG resource:

$ actl stop goldengate GGNORTH

# agctl remove goldengate GGNORTH

21 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

Further information on the Oracle Grid Infrastructure Bundled Agent:

http://www.oracle.com/technetwork/database/database-technologies/clusterware/downloads/xag-agents-
downloads-3636484.html

Step 8: Configure NGINX Reverse Proxy

Follow the instructions provided in My Oracle Support note 2826001.1 to install and configure NGINX Reverse Proxy
with SSL connection, and to ensure all external communication is completely secure.

NOTE: When using CA Signed Certificates with NGINX, make sure the NGINX ssl_certificate parameter points to
a certificate file that contains the certificates in the correct order of CA signed certificate, intermediate certificate and root
certificate.

Oracle Clusterware needs to have control over starting the NGINX reverse proxy so that it can be started automatically
before the GoldenGate deployments are started.

The NGINX resource is created with a dependency on the underlying network CRS resource, the name of which can be
determined using the following command:

$ $GRID_HOME/bin/crsctl stat res -w "TYPE == ora.network.type"|grep NAME

NAME=ora.net1.network

As the root user, use the following example command to create a Clusterware resource to manage NGINX.

# $GRID_HOME/bin/crsctl add resource nginx -type generic_application -attr

"ACL='owner:root:rwx,pgrp:root:rwx,other::r--,group:oinstall:r-
x,user:oracle:rwx',EXECUTABLE_NAMES=nginx,START_PROGRAM='/bin/systemctl start -f
nginx',STOP_PROGRAM='/bin/systemctl stop -f nginx',CHECK_PROGRAMS='/bin/systemctl status
nginx' ,START_DEPENDENCIES='hard(ora.net1.network) pullup(ora.net1.network)',
STOP_DEPENDENCIES='hard(intermediate:ora.net1.network)', RESTART_ATTEMPTS=0,
HOSTING_MEMBERS='dc1north01,dc1north02', CARDINALITY=2"

The NGINX resource created in this example run on the named cluster nodes at the same time, specified by
HOSTING_MEMBERS. This is recommended when multiple GoldenGate Service Manager deployments are
configured, and they can independently move between cluster nodes.

Once the NGINX Clusterware resource is created, alter the GoldenGate XAG resources so that NGINX must be started
before the GoldenGate deployments are started.

As the oracle user, modify the XAG resources using the following example commands.

# Determine the current --filesystems parameter:

$ agctl config goldengate SOURCE|grep "File System"

File System resources needed: ora.datac1.acfs_gg.acfs

# Modify the --filesystems parameter:

22 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

$ agctl modify goldengate SOURCE --filesystems ora.datac1.acfs_gg.acfs,nginx

Repeat the above commands for each of the XAG GoldenGate registrations relying on NGINX.

Step 9: Create Oracle Net TNS Alias for Oracle GoldenGate Database Connections
Create a TNS alias on all of the RAC nodes where Oracle GoldenGate may be started to provide local database
connections for the GoldenGate processes when switching between Oracle RAC nodes. Create the TNS alias in the
tnsnames.ora file in the TNS_ADMIN directory specified in the deployment creation.

If the source database is a multitenant database, two TNS alias entries are required: one for the container database
(CDB) and one for the pluggable database (PDB) that is being replicated. For a target multitenant database, the TNS
alias connects the PDB where replicated data is being applied to. The pluggable database SERVICE_NAME should be
set to the database service created in an earlier step (refer Step 1: Create the Database Services).

Below are some example source database TNS alias definitions using the IPC protocol, which must be defined locally
on all RAC nodes.

OGGSOURCE_CDB =
(DESCRIPTION =
(ADDRESS = (PROTOCOL=IPC)(KEY=LISTENER))
(CONNECT_DATA =
(SERVICE_NAME = oggserv_cdb)
)
)

OGGSOURCE_PDB =
(DESCRIPTION =
(ADDRESS = (PROTOCOL=IPC)(KEY=LISTENER))
(CONNECT_DATA =

(SERVICE_NAME = oggserv_pdb)
)
)

NOTE: When the tnsnames.ora or sqlnet.ora, located in the TNS_ADMIN directory for the GoldenGate deployment, are
modified, the deployment needs to be restarted in order to pick up the changes.

With the GoldenGate deployment created, use the Administration Server home page to create the database
credentials using the above TNS alias names. See Figure 6 below for an example of the database credential creation
using the TNS alias appended to the database user name in the ‘User ID” field.

If the source database is a multitenant database, create database credentials for the CDB and PDB. If the target
database is a multitenant database, create a single credential for the PDB.

23 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

Figure 6: Creating a database credential

Step 10: Configure Oracle GoldenGate Processes

When creating Extract, Distribution Paths, and Replicat processes with Oracle GoldenGate Microservices Architecture,
all files that need to be shared between Oracle RAC nodes are already shared with the deployment files stored on a
shared file system (DBFS or ACFS).

Listed below are important configuration details that are recommended for running Oracle GoldenGate Microservices
on Oracle RAC for Extract, Distribution Paths and Replicat processes.

Extract Configuration
1. When creating an Extract using the Oracle GoldenGate Administration Server GUI interface, leave the Trail
SubDirectory parameter blank, so that the trail files are automatically created in the deployment directories
stored on the shared file system. The default location for trail files is the /<deployment
directory>/var/lib/data directory.
In the example shown below, all of the Add Extract Basic Information fields are populated except Trail
Subdirectory:

24 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

 Figure 7: Adding an Extract process with the Administration Server.

2. If you are using DBFS for shared storage, and the deployment var/temp directory was moved to local
storage as described in Step 6: Create the Oracle GoldenGate Deployment, it is recommended that you use
the Extract CACHEMGR parameter to place the temporary cache files on the shared storage.
Create a new directory under the DBFS deployment mount point. For example:

$ mkdir –p /mnt/dbfs/goldengate/deployments/ggnorth/temp_cache

Set the Extract parameter to the new directory:

CACHEMGR CACHEDIRECTORY /mnt/dbfs/goldengate/deployments/ggnorth/temp_cache

Shown below is an example of how the parameters specified for an integrated Extract with the Oracle
GoldenGate Administration Server GUI looks in the UI.

25 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

Figure 8: Extract parameters for defining the temporary cache files.

More instructions about creating an Extract process are available in the Using Oracle GoldenGate Microservices
Architecture guide at https://docs.oracle.com/en/middleware/goldengate/core/21.3/oracle-db/index.html

Distribution Path Configuration

When using GoldenGate distribution paths with the NGINX Reverse Proxy, there are additional steps that must be
performed to ensure that the path server certificates are configured.

Follow the instructions provided in the following video to correctly configure the certificates:
https://apexapps.oracle.com/pls/apex/f?p=44785:112:0::::P112_CONTENT_ID:31380

Configuration highlights presented in this video:

1. Create a client certificate for the source deployment and add the client certificate to the source deployment
Service Manager. This is not required when using Oracle GoldenGate 21c or higher releases.
2. Download the target deployment server’s root certificate and add the CA certificate to the source deployment
Service Manager.
3. Create a user in the target deployment for the distribution path to connect to.
4. Create a credential in the source deployment connecting to the target deployment with the user created in
the previous step. For example, a domain of GGNORTH_to_GGSOUTH and an alias of PathReceiver.
After configuring the client and server certificates, the following configuration options need to be set correctly, as
shown in Figure 9 below.

1. Change the Generated Source URI specifying localhost for the server name. This allows the distribution
path to be started on any of the Oracle RAC nodes.
2. Set the Target Authentication Method to ‘UserID Alias’ and the Target transfer protocol to wss (secure web
socket). Set the Target Host to the target host name/VIP that will be used for connecting to the target
system along with the Port Number that NGINX was configured with (default is 443). The target host
name/VIP should match the common name in the CA signed certificate used by NGINX.
26 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

 3. Set the Domain to the credential domain created above in step 4 presented in the video, for example
GGNORTH_to_GGSOUTH. The Alias is set to the credential alias, also created in step 4 in the video.
4. Set the distribution path to automatically restart when the Distribution Server starts. This is required so that
manual intervention is not required after a RAC node relocation of the Distribution Server. It is recommended
to set the number of Retries to 10. Set the Delay, which is the amount of time in minutes to pause between
restart attempts, to 1.

Figure 9: Creating a distribution path using the Distribution Server GUI

27 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

More instructions about creating distribution paths are available in Using Oracle GoldenGate Microservices
Architecture at https://docs.oracle.com/en/middleware/goldengate/core/21.3/ggmas/working-paths.html#GUID-
7F9F7045-AA27-4007-9852-BC69C2F301A1.

Replicat Configuration
1. The checkpoint table is a required component for GoldenGate Replicat processes. Make sure that a
checkpoint table has been created in the database GoldenGate administrator (GGADMIN) schema.
The checkpoint table can be created using the Oracle GoldenGate Administration Server GUI, clicking on the
‘+’ button and entering the checkpoint table name in the format of schema.tablename. This is shown in
Figure 10.

Figure 10: Creating the checkpoint table for Replicat processes

Refer to Using Oracle GoldenGate with Oracle Database for more information about creating a checkpoint
table at https://docs.oracle.com/en/middleware/goldengate/core/21.3/oracle-db/configuring-oracle-
goldengate-apply.html#GUID-3DFBE2BE-20C5-48AA-B96A-7697126D77FE

2. When creating a Replicat using the Oracle GoldenGate Administration Server GUI interface, set the Trail
SubDirectory parameter to the location where the distribution path or local Extract are creating the trail files.
3. If a checkpoint table was created previously, select the table name from the Checkpoint Table pulldown list.
Below, in Figure 11, a screenshot shows the Trail SubDirectory and Checkpoint Table prompts.

28 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

 Figure 11: Replicat creation with Trail SubDirectory and Checkpoint Table

Step 11: Configure Autostart of Extract and Replicat Processes

Configure the Extract and Replicat processes to automatically start when the Oracle GoldenGate Administration
Server is started, and then to restart if any Extract or Replicat processes abend. With GoldenGate Microservices auto
start and restart is managed by Profiles.

Using the Oracle GoldenGate Administration Server GUI, create a new profile which can be assigned to each of the
GoldenGate processes. Figure 12 shows the recommended settings.

29 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

Figure 12: Creating an auto start profile using the Oracle GoldenGate Administration Server GUI.

After the profile has been created, and set as the default profile, all new GoldenGate processes created are assigned
this profile. For all existing processes, the profile must be assigned to each process. Figure 13 shows an example of
assigning the auto start profile to a GoldenGate process.

30 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

Figure 13: Assigning an auto start profile to a GoldenGate process using the GoldenGate Administration Server GUI.

NOTE: When using Oracle GoldenGate Microservices with XAG, it is strongly recommended not to enable the ‘Critical to
deployment health’ flag for any Extract or Replicat processes. Doing so can cause an entire GoldenGate deployment
outage from a single Extract or Replicat failure, and also prevents XAG from being able to restart GoldenGate. Refer to
Appendix A for an example of troubleshooting an outage caused by setting a Replicat to critical.

Summary of Recommendations when Deploying on Oracle RAC

When configuring Oracle GoldenGate in an Oracle RAC environment, follow these recommendations.

• Install the latest version of Oracle GoldenGate software locally on each Oracle RAC node, making sure that the
software location is the same on all Oracle RAC nodes.
• Use the Oracle Database File System (DBFS) or Oracle Advanced Cluster File System (ACFS) for the file
system where the GoldenGate files are stored (trail, checkpoint, temporary, report, and parameter files).
• Use the same DBFS or ACFS mount point on all of the Oracle RAC nodes that may run Oracle GoldenGate.
• When creating the GoldenGate deployment, specify either DBFS or ACFS for the deployment location.
• Install Grid Infrastructure agent (XAG) version 10 or later on all Oracle RAC nodes that will run Oracle
GoldenGate.
• Configure the GoldenGate processes to automatically start and restart when the deployment is started.

31 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

References
• Oracle GoldenGate 21c Documentation
• Oracle Database SecureFiles and Large Object Developer’s Guide 19c (DBFS)
• Oracle Automatic Storage Management Cluster File System Administrator’s Guide 19c (ACFS)
• Oracle Clusterware Administration and Deployment Guide 19c
• Oracle Maximum Availability Architecture Web site
http://www.otn.oracle.com/goto/maa

32 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

Appendix A: Troubleshooting Oracle GoldenGate on Oracle RAC
There may be occasions when GoldenGate processes are not successfully started on an Oracle RAC node. There are
number of files generated by Oracle GoldenGate, XAG, and CRS that should be reviewed to determine the cause of the
problem.

Below is a list of important log and trace files, along with their example locations and some example output.

1. XAG log file

Location: <XAG installation directory>/log/<hostname>
Example location: /u01/oracle/xag/log/dc1north01

File name: agctl_goldengate_oracle.trc

Contains all commands executed with agctl along with the output from the commands, including those commands
that are executed by CRS.

Example:

2022-04-18 11:52:21: stop resource success

2022-04-18 11:52:38: agctl start goldengate GGNORTH
2022-04-18 11:52:38: executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl status res
xag.GGNORTH.goldengate
2022-04-18 11:52:38: executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl status res
xag.GGNORTH.goldengate –f
2022-04-18 11:52:38: executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl start resource
xag.GGNORTH.goldengate –f
2022-04-18 11:52:45: Command output:
> CRS-2672: Attempting to start 'xag.GGNORTH.goldengate' on 'dc1north01'
> CRS-2676: Start of 'xag.GGNORTH.goldengate' on 'dc1north01' succeeded
>End Command output
2022-04-18 11:52:45: start resource success

2. XAG GoldenGate instance trace file

Location: <XAG installation directory>/log/<hostname>
Example location: /u01/oracle/xag/log/dc1north01

File name: <GoldenGate_instance_name>_agent_goldengate.trc

Contains the output from the commands executed by agctl along with the environment variables used, and any
debug output enabled for the underlying commands.

Example:

2022-04-18 12:14:46: Exported ORACLE_SID ggdg1

2022-04-18 12:14:46: Exported GGS_HOME /u01/oracle/goldengate/gg21c_MS
2022-04-18 12:14:46: Exported OGG_CONF_HOME
/mnt/dbfs/goldengate/deployments/ggsm01/etc/conf
2022-04-18 12:14:46: Exported LD_LIBRARY_PATH
/u01/oracle/goldengate/gg21c_MS:/u01/app/19.0.0.0/grid/lib:/etc/ORCLcluster/lib
2022-04-18 12:14:46: Exported LD_LIBRARY_PATH_64 /u01/oracle/goldengate/gg21c_MS
2022-04-18 12:14:46: Exported LIBPATH /u01/oracle/goldengate/gg21c_MS
2022-04-18 12:14:46: ogg input =

33 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

{"oggHome":"/u01/oracle/goldengate/gg21c_MS","serviceManager":{"oggConfHome":"/mnt/dbfs/
goldengate/deployments/ggsm01/etc/conf","portNumber":9100},"username":"admin","credentia
l":"xyz"}
2022-04-18 12:14:46: About to exec /u01/oracle/goldengate/gg21c_MS/bin/XAGTask
HealthCheck
2022-04-18 12:14:47: XAGTask retcode = 0

3. CRS trace file

Location: /u01/app/oracle/diag/crs/<hostname>/crs/trace
Example location: /u01/app/oracle/diag/crs/dc1north01/crs/trace

File name: crsd_scriptagent_oracle.trc

Contains the output created by any CRS resource action scripts, like XAG or dbfs_mount. This trace file is crucial to
determining why DBFS or GoldenGate did not start on a RAC node.

Example:
2022-04-18 11:52:38.634 : AGFW:549631744: {1:30281:59063} Agent received the message:
RESOURCE_START[xag.GGNORTH.goldengate 1 1] ID 4098:4125749
2022-04-18 11:52:38.634 : AGFW:549631744: {1:30281:59063} Preparing START command for:
xag.GGNORTH.goldengate 1 1
2022-04-18 11:52:38.634 : AGFW:549631744: {1:30281:59063} xag.GGNORTH.goldengate 1 1
state changed from: OFFLINE to: STARTING
2022-04-18 11:52:38.634 :CLSDYNAM:558036736: [xag.GGNORTH.goldengate]{1:30281:59063}
[start] Executing action script: /u01/oracle/XAG_MA/bin/aggoldengatescaas[start]
2022-04-18 11:52:38.786 :CLSDYNAM:558036736: [xag.GGNORTH.goldengate]{1:30281:59063}
[start] GG agent running command 'start' on xag.GGNORTH.goldengate
2022-04-18 11:52:42.140 :CLSDYNAM:558036736: [xag.GGNORTH.goldengate]{1:30281:59063}
[start] ServiceManager fork pid = 265747
2022-04-18 11:52:42.140 :CLSDYNAM:558036736: [xag.GGNORTH.goldengate]{1:30281:59063}
[start] Waiting for /mnt/dbfs/goldengate/deployments/ggsm01/var/run/ServiceManager.pid
2022-04-18 11:52:42.140 :CLSDYNAM:558036736: [xag.GGNORTH.goldengate]{1:30281:59063}
[start] Waiting for SM to start
2022-04-18 11:52:42.140 :CLSDYNAM:558036736: [xag.GGNORTH.goldengate]{1:30281:59063}
[start] ServiceManager PID = 265749
2022-04-18 11:52:43.643 :CLSDYNAM:558036736: [xag.GGNORTH.goldengate]{1:30281:59063}
[start] XAGTask retcode = 0
2022-04-18 11:52:43.643 :CLSDYNAM:558036736: [xag.GGNORTH.goldengate]{1:30281:59063}
[start] XAG HealthCheck after start returned 0
2022-04-18 11:52:43.643 : AGFW:558036736: {1:30281:59063} Command: start for resource:
xag.GGNORTH.goldengate 1 1 completed with status: SUCCESS
2022-04-18 11:52:43.643 :CLSDYNAM:558036736: [xag.GGNORTH.goldengate]{1:30281:59063}
[check] Executing action script: /u01/oracle/XAG_MA/bin/aggoldengatescaas[check]
2022-04-18 11:52:43.644 : AGFW:549631744: {1:30281:59063} Agent sending reply for:
RESOURCE_START[xag.GGNORTH.goldengate 1 1] ID 4098:4125749
2022-04-18 11:52:43.795 :CLSDYNAM:558036736: [xag.GGNORTH.goldengate]{1:30281:59063}
[check] GG agent running command 'check' on xag.GGNORTH.goldengate
2022-04-18 11:52:45.548 :CLSDYNAM:558036736: [xag.GGNORTH.goldengate]{1:30281:59063}
[check] XAGTask retcode = 0
2022-04-18 11:52:45.548 : AGFW:549631744: {1:30281:59063} xag.GGNORTH.goldengate 1 1
state changed from: STARTING to: ONLINE
34 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

4. GoldenGate deployment log files
Location: <Goldengate_deployment_directory>/<deployment_name>/var/log
Example location: /mnt/dbfs/goldengate/deployments/ggnorth/var/log

File names: adminsrvr.log, recvsrvr.log, pmsrvr.log, distsrvr.log

Contains the output of start, stop and status checks of the GoldenGate deployment processes (Administration Server,
Distribution Server, Receiver Server and the Performance Metrics Server).

Example:

2022-04-18T11:52:42.645-0400 INFO | Setting deploymentName to 'ggnorth'. (main)

2022-04-18T11:52:42.665-0400 INFO | Read SharedContext from store for length 19 of file
'/mnt/dbfs/goldengate/deployments/ggnorth/var/lib/conf/adminsrvr-resources.dat'. (main)
2022-04-18T11:52:42.723-0400 INFO | XAG Integration enabled (main)
2022-04-18T11:52:42.723-0400 INFO | Configuring security. (main)
2022-04-18T11:52:42.723-0400 INFO | Configuring user authorization secure store path as
'/mnt/dbfs/goldengate/deployments/ggnorth/var/lib/credential/secureStore/'. (main)
2022-04-18T11:52:42.731-0400 INFO | Configuring user authorization as ENABLED. (main)
2022-04-18T11:52:42.749-0400 INFO | Set network configuration. (main)
2022-04-18T11:52:42.749-0400 INFO | Asynchronous operations are enabled with default
synchronous wait time of 30 seconds (main)
2022-04-18T11:52:42.749-0400 INFO | HttpServer configuration complete. (main)
2022-04-18T11:52:42.805-0400 INFO | SIGHUP handler installed. (main)
2022-04-18T11:52:42.813-0400 INFO | SIGINT handler installed. (main)
2022-04-18T11:52:42.815-0400 INFO | SIGTERM handler installed. (main)
2022-04-18T11:52:42.817-0400 WARN | Security is configured as 'disabled'. (main)
2022-04-18T11:52:42.818-0400 INFO | Starting service listener... (main)
2022-04-18T11:52:42.819-0400 INFO | Mapped 'ALL' interface to address 'ANY:9101' with
default IPV4/IPV6 options identified by 'dc1north01.us.oracle.com'. (main)
2022-04-18T11:52:42.821-0400 INFO | Captured 1 interface host names:
'dc1north01.us.oracle.com' (main)
2022-04-18T11:52:42.824-0400 INFO | The Network ipACL specification is empty. Accepting
ANY address on ALL interfaces. (main)
2022-04-18T11:52:42.826-0400 INFO | Server started at 2022-04-18T11:52:42.827-05:00
(2022-04-18T15:52:42.827Z GMT) (main)

5. GoldenGate report files

Location: <Goldengate_deployment_directory>/<deployment_name>/var/lib/report
Example location: /mnt/dbfs/goldengate/deployments/ggnorth/var/lib/report

The GoldenGate report files contain important information, warning messages and errors for all of the GoldenGate
processes, including the Manager processes. If any of the GoldenGate processes fail to start or abend when they are
running, the process report file will contain important information which can be used to determine the cause of the
failure.

Example errors from an Extract report file:

2022-04-23 13:01:50 ERROR OGG-00446 Unable to lock file "

/mnt/acfs_gg/deployments/ggnorth/var/lib/checkpt/EXT_1A.cpe" (error 95, Operation not

35 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

supported).
2022-04-23 13:01:50 ERROR OGG-01668 PROCESS ABENDING.

Example Configuration Problems

Below are some examples of configuration problems that can be encountered with Oracle GoldenGate in an Oracle
RAC environment along with how to diagnose and resolve them.

1. Incorrect parameter settings in the mount-dbfs.conf file

When XAG fails to mount DBFS the failure will be reported either on the command line (if you are running the manual
agctl command) or in the XAG log file:

$ agctl start goldengate GGNORTH --node dc1north01

CRS-2672: Attempting to start 'dbfs_mount' on ' dc1north01'

CRS-2674: Start of 'dbfs_mount' on ' dc1north01' failed
CRS-2679: Attempting to clean 'dbfs_mount' on ' dc1north01'
CRS-2681: Clean of 'dbfs_mount' on ' dc1north01' succeeded
CRS-4000: Command Start failed, or completed with errors.

The XAG log file (agctl_goldengate_oracle.trc) has the advantage that it shows timestamps that can be used
when looking at other log or trace files:

2022-04-19 15:32:16: executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl start resource

xag.GGNORTH.goldengate -f -n dc1north01
2022-04-19 15:32:19: Command output:
> CRS-2672: Attempting to start 'dbfs_mount' on ' dc1north01'
> CRS-2674: Start of 'dbfs_mount' on ' dc1north01' failed
> CRS-2679: Attempting to clean 'dbfs_mount' on ' dc1north01'
> CRS-2681: Clean of 'dbfs_mount' on ' dc1north01' succeeded
> CRS-4000: Command Start failed, or completed with errors.
>End Command output
2022-04-19 15:32:19: start resource failed rc=1

Next, check the CRS trace file (crsd_scriptagent_oracle.trc) which shows the reason why DBFS failed to
mount. Below are some example errors caused by incorrect parameter settings in the mount-dbfs.conf file.

• Incorrect DBNAME
2022-04-19 15:32:16.679 : AGFW:1190405888: {1:30281:17383} dbfs_mount 1 1 state
changed from: UNKNOWN to: STARTING
2022-04-19 15:32:16.680 :CLSDYNAM:1192507136: [dbfs_mount]{1:30281:17383} [start]
Executing action script: /u01/oracle/scripts/mount-dbfs.sh[start]
2022-04-19 15:32:16.732 :CLSDYNAM:1192507136: [dbfs_mount]{1:30281:17383} [start] mount-
dbfs.sh mounting DBFS at /mnt/dbfs from database ggdg
2022-04-19 15:32:17.883 :CLSDYNAM:1192507136: [dbfs_mount]{1:30281:17383} [start]
ORACLE_SID is
2022-04-19 15:32:17.883 :CLSDYNAM:1192507136: [dbfs_mount]{1:30281:17383} [start] No
running ORACLE_SID available on this host, exiting
2022-04-19 15:32:17.883 : AGFW:1192507136: {1:30281:17383} Command: start for
resource: dbfs_mount 1 1 completed with invalid status: 2
36 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

• Incorrect MOUNT_POINT
2022-04-19 16:45:14.534 : AGFW:1734321920: {1:30281:17604} dbfs_mount 1 1 state
changed from: UNKNOWN to: STARTING
2022-04-19 16:45:14.535 :CLSDYNAM:1736423168: [dbfs_mount]{1:30281:17604} [start]
Executing action script: /u01/oracle/scripts/mount-dbfs.sh[start]
2022-04-19 16:45:14.586 :CLSDYNAM:1736423168: [dbfs_mount]{1:30281:17604} [start] mount-
dbfs.sh mounting DBFS at /mnt/dbfs_gg from database ggdgs
2022-04-19 16:45:15.638 :CLSDYNAM:1736423168: [dbfs_mount]{1:30281:17604} [start]
ORACLE_SID is ggdg1
2022-04-19 16:45:15.738 :CLSDYNAM:1736423168: [dbfs_mount]{1:30281:17604} [start]
spawning dbfs_client command using SID ggdg1
2022-04-19 16:45:20.745 :CLSDYNAM:1736423168: [dbfs_mount]{1:30281:17604} [start] fuse:
bad mount point `/mnt/dbfs': No such file or directory
2022-04-19 16:45:21.747 :CLSDYNAM:1736423168: [dbfs_mount]{1:30281:17604} [start] Start
– OFFLINE
2022-04-19 16:45:21.747 : AGFW:1736423168: {1:30281:17604} Command: start for
resource: dbfs_mount 1 1 completed with status: FAIL

• Incorrect DBFS_USER or DBFS_PASSWD

2022-04-19 16:47:47.855 : AGFW:1384478464: {1:30281:17671} dbfs_mount 1 1 state
changed from: UNKNOWN to: STARTING
2022-04-19 16:47:47.856 :CLSDYNAM:1386579712: [dbfs_mount]{1:30281:17671} [start]
Executing action script: /u01/oracle/scripts/mount-dbfs.sh[start]
2022-04-19 16:47:47.908 :CLSDYNAM:1386579712: [dbfs_mount]{1:30281:17671} [start] mount-
dbfs.sh mounting DBFS at /mnt/dbfs from database ggdgs
2022-04-19 16:47:48.959 :CLSDYNAM:1386579712: [dbfs_mount]{1:30281:17671} [start]
ORACLE_SID is ggdg1
2022-04-19 16:47:49.010 :CLSDYNAM:1386579712: [dbfs_mount]{1:30281:17671} [start]
spawning dbfs_client command using SID ggdg1
2022-04-19 16:47:55.118 :CLSDYNAM:1386579712: [dbfs_mount]{1:30281:17671} [start] Fail
to connect to database server. Error: ORA-01017: invalid username/password; logon denied
2022-04-19 16:47:55.118 :CLSDYNAM:1386579712: [dbfs_mount]{1:30281:17671} [start]
2022-04-19 16:47:56.219 :CLSDYNAM:1386579712: [dbfs_mount]{1:30281:17671} [start] Start
– OFFLINE
2022-04-19 16:47:56.220 : AGFW:1386579712: {1:30281:17671} Command: start for
resource: dbfs_mount 1 1 completed with status: FAIL

• Incorrect ORACLE_HOME
2022-04-19 16:50:38.952 : AGFW:567502592: {1:30281:17739} dbfs_mount 1 1 state
changed from: UNKNOWN to: STARTING
2022-04-19 16:50:38.953 :CLSDYNAM:569603840: [dbfs_mount]{1:30281:17739} [start]
Executing action script: /u01/oracle/scripts/mount-dbfs.sh[start]
2022-04-19 16:50:39.004 :CLSDYNAM:569603840: [dbfs_mount]{1:30281:17739} [start] mount-
dbfs.sh mounting DBFS at /mnt/dbfs from database ggdgs

37 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

2022-04-19 16:50:39.004 :CLSDYNAM:569603840: [dbfs_mount]{1:30281:17739} [start]
/u01/oracle/scripts/mount-dbfs.sh: line 136:
/u01/app/oracle/product/19.0.0.0/rdbms/bin/srvctl: No such file or directory
2022-04-19 16:50:39.004 :CLSDYNAM:569603840: [dbfs_mount]{1:30281:17739} [start]
/u01/oracle/scripts/mount-dbfs.sh: line 139:
/u01/app/oracle/product/19.0.0.0/rdbms/bin/srvctl: No such file or directory
2022-04-19 16:50:39.004 :CLSDYNAM:569603840: [dbfs_mount]{1:30281:17739} [start]
ORACLE_SID is
2022-04-19 16:50:39.004 :CLSDYNAM:569603840: [dbfs_mount]{1:30281:17739} [start] No
running ORACLE_SID available on this host, exiting
2022-04-19 16:50:39.004 : AGFW:569603840: {1:30281:17739} Command: start for
resource: dbfs_mount 1 1 completed with invalid status: 2

To resolve these configuration issues, set the correct parameter values in mount-dbfs.conf.

2. Problems with file locking on DBFS

If you are using Oracle Database 12c Release 2 (12.2) and the nolock DBFS mount option is not used, there can be
problems with GoldenGate processes trying to lock checkpoint or trail files.

The same problem is encountered if you are using Oracle Database 11g Release 2 (11.2.0.4) or 12c Release 1 (12.1)
with patch for bug 22646150 applied. This patch changes the way in which file locking is handled by DBFS to match
Oracle Database 12c Release 2 (12.2).

To add the nolock DBFS mount option, the patch for bug 27056711 must be applied to the database. If the patch for
bug 22646150 has not been applied to the database, the patch for bug 27056711 and the nolock mount option is
not required.

Below is an example of how to diagnose a GoldenGate Microservices Architecture locking problem.

When starting a deployment with XAG, one or more processes may not start due to detecting a locking conflict on one
or more files. This will often occur after a RAC node failover where the deployment did not get a chance to shut down
cleanly.

When one of the deployment server processes fails to start (Administration Server, Performance Metrics Server,
Distribution Server, Receiver Server, or Service Manager) check the log file for the particular server located in the
deployment var/log directory.

For example, the log file /mnt/dbfs/goldengate/deployments/GGNORTH/var/log/pmsrvr.log shows the

following error on startup:
2022-04-11T12:41:57.619-0700 ERROR| SecureStore failed on open after retrying due to
extended file lock. (main)
2022-04-11T12:41:57.619-0700 ERROR| SecureStore failed to close (28771). (main)
2022-04-11T12:41:57.619-0700 INFO | Set network configuration. (main)
2022-04-11T12:41:57.619-0700 INFO | Asynchronous operations are enabled with default
synchronous wait time of 30 seconds (main)
2022-04-11T12:41:57.619-0700 INFO | HttpServer configuration complete. (main)
2022-04-11T12:42:07.674-0700 ERROR| Unable to lock process file, Error is [1454] - OGG-
01454 (main)
2022-04-11T12:42:07.675-0700 ERROR| Another Instance of PM Server is Already Running
(main)

38 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

An Extract process reports start up failures in the ER-events.log logfile located in the deployment log file directory.

For example, /mnt/dbfs/goldengate/deployments/ggnorth/var/log/ER-events.log shows the following

error:
2022-04-11T00:14:56.845-0700 ERROR OGG-01454 Oracle GoldenGate Capture for Oracle,
EXT1.prm: Unable to lock file
"/mnt/dbfs/goldengate/deployments/ggnorth/var/run/EXT1.pce" (error 11, Resource
temporarily unavailable). Lock currently held by process id (PID) 237495.
2022-04-11T00:14:56.861-0700 ERROR OGG-01668 Oracle GoldenGate Capture for Oracle,
EXT1.prm: PROCESS ABENDING.

Next, check to make sure the process failing to start up is not running on any of the RAC nodes.

Example:
$ ps -ef|grep EXT1|grep –v grep

Once you determine that the process is not running, the deployment needs to be shut down cleanly, the file system
unmounted, and the correct DBFS patch applied.

Example:
$ agctl stop goldengate GGNORTH
$ crsctl stop resource dbfs_mount

Check the DBFS mount options:

$ ps -ef|grep dbfs_client

oracle 204017 1 0 14:37 ? 00:00:00

/u01/app/oracle/product/19.1.0.0/dbhome_1/bin/dbfs_client [email protected] -o
allow_other,failover,direct_io /mnt/dbfs

It is clear that the nolock mount option was not used, which leads to the locking errors.

Use the guidelines at the beginning of this topic to determine if a DBFS patch is required. Then add the nolock
mount option to the mount-dbfs.conf file on all RAC nodes that are part of the deployment.

Example:
MOUNT_OPTIONS=allow_other,direct_io,failover,nolock

Finally, restart the deployment:

$ agctl start goldengate GGNORTH

3. Critical Extract/Replicat process preventing GoldenGate startup

When an Extract or Replicat process has been set as critical to the health of the deployment in the Administration
Server, if the process abends, the Service Manager and deployment will be stopped.

When restarting GoldenGate using XAG (agctl start goldengate) it will fail with the following error:

39 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

$ agctl start goldengate GGNORTH --node dc1north01
CRS-2672: Attempting to start 'xag.GGNORTH.goldengate' on 'dc1north01'
CRS-2674: Start of 'xag.GGNORTH.goldengate' on 'dc1north01' failed
CRS-2679: Attempting to clean 'xag.GGNORTH.goldengate' on ' dc1north01'
CRS-2681: Clean of 'xag.GGNORTH.goldengate' on ' dc1north01' succeeded
CRS-2632: There are no more servers to try to place resource 'xag.GGNORTH.goldengate' on
that would satisfy its placement policy
CRS-4000: Command Start failed, or completed with errors.

The CRS trace file (crsd_scriptagent_oracle.trc) does not provide enough information to determine the
reason behind the startup failure.

Example:

2022-06-30 20:20:28.675 : AGFW:521639680: [ INFO] {1:29336:52291} Command: start

for resource: xag.GGNORTH.goldengate 1 1 completed with status: SUCCESS
2022-06-30 20:20:28.675 :CLSDYNAM:521639680: [xag.GGNORTH.goldengate]{1:29336:52291}
[check] Executing action script: /u01/oracle/XAG/bin/aggoldengatescaas[check]
2022-06-30 20:20:28.677 : AGFW:517437184: [ INFO] {1:29336:52291} Agent sending
reply for: RESOURCE_START[xag.GGNORTH.goldengate 1 1] ID 4098:178187
2022-06-30 20:20:28.827 :CLSDYNAM:521639680: [xag.GGNORTH.goldengate]{1:29336:52291}
[check] GG agent running command 'check' on xag.GGNORTH.goldengate
2022-06-30 20:20:28.978 :CLSDYNAM:521639680: [xag.GGNORTH.goldengate]{1:29336:52291}
[check] execute XAGTask HealthCheck
2022-06-30 20:20:29.779 :CLSDYNAM:521639680: [xag.GGNORTH.goldengate]{1:29336:52291}
[check] XAGTask retcode = 3
2022-06-30 20:20:29.780 : AGFW:517437184: [ INFO] {1:29336:52291}
xag.GGNORTH.goldengate 1 1 state changed from: STARTING to: FAILED

In order to determine why the startup failed, increase the CRS logging mode and restart GoldenGate.

Example:

$ crsctl set log res xag.GGNORTH.goldengate=5

$ agctl start goldengate GGNORTH --node dc1north01
$ crsctl set log res xag.GGNORTH.goldengate=1

Check the CRS trace file (crsd_scriptagent_oracle.trc) for more information on the failure.

Example:

2022-06-30 20:24:36.389 :CLSDYNAM:521639680: [xag.GGNORTH.goldengate]{1:29336:52435}

[check] Deployment: GGNORTH
2022-06-30 20:24:36.389 :CLSDYNAM:521639680: [xag.GGNORTH.goldengate]{1:29336:52435}
[check] Service: adminsrvr
2022-06-30 20:24:36.389 :CLSDYNAM:521639680: [xag.GGNORTH.goldengate]{1:29336:52435}
[check] Status: running
2022-06-30 20:24:36.389 :CLSDYNAM:521639680: [xag.GGNORTH.goldengate]{1:29336:52435}
[check] Started: 2022-07-01T03:24:35.911Z
2022-06-30 20:24:36.389 :CLSDYNAM:521639680: [xag.GGNORTH.goldengate]{1:29336:52435}
40 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

[check] Health: Failed
2022-06-30 20:24:36.389 :CLSDYNAM:521639680: [xag.GGNORTH.goldengate]{1:29336:52435}
[check] Resources: Name Type Status Health
2022-06-30 20:24:36.389 :CLSDYNAM:521639680: [xag.GGNORTH.goldengate]{1:29336:52435}
[check] -------------------------------- ---------- ---------- ------
2022-06-30 20:24:36.389 :CLSDYNAM:521639680: [xag.GGNORTH.goldengate]{1:29336:52435}
[check] REP1 replicat abended Failed
2022-06-30 20:24:36.389 :CLSDYNAM:521639680: [xag.GGNORTH.goldengate]{1:29336:52435}
[check] HealthCheck failed

When Extract or Replicat processes are not set to critical, they will not appear in the CRS trace file. Because Replicat
REP1 is shown as failed, this indicates that REP1 is set to critical and is preventing GoldenGate from starting.

To disable the critical setting for Replicat, the GoldenGate Service Manager and deployment must be started
manually.

Example:
$ export OGG_ETC_HOME=/mnt/acfs/goldengate/deployments/ggsm01/etc
$ export OGG_VAR_HOME=/mnt/acfs/goldengate/deployments/ggsm01/var
$ export OGG_HOME=/u01/oracle/goldengate/ggMS_21c
$ $OGG_HOME/bin/ServiceManager --xagEnabled

Using the Administration Server GUI, select the Replicat or Extract details, and unset the critical flag. Figure 14 shows
how to unset the critical flag. Remember to submit the change.

Figure 14: Disabling Extract and Replicat critical setting.

Once the critical setting has been disabled, XAG can be used to start and stop Oracle GoldenGate Microservices.

Example:

$ agctl start goldengate GGNORTH --node dc1north01

$ agctl status goldengate GGNORTH

41 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public

Connect with us

Call +1.800.ORACLE1 or visit oracle.com. Outside North America, find your local office at: oracle.com/contact.

blogs.oracle.com facebook.com/oracle twitter.com/oracle

Copyright © 2023, Oracle and/or its affiliates. All rights reserved. This document is Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be
provided for information purposes only, and the contents hereof are subject to change trademarks of their respective owners.
without notice. This document is not warranted to be error-free, nor subject to any other
Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC
warranties or conditions, whether expressed orally or implied in law, including implied
trademarks are used under license and are trademarks or registered trademarks of SPARC
warranties and conditions of merchantability or fitness for a particular purpose. We
International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or
specifically disclaim any liability with respect to this document, and no contractual
registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open
obligations are formed either directly or indirectly by this document. This document
Group. 0120
may not be reproduced or transmitted in any form or by any means, electronic or
mechanical, for any purpose, without our prior written permission. Disclaimer: If you are unsure whether your data sheet needs a disclaimer, read the revenue
recognition policy. If you have further questions about your content and the disclaimer
This device has not been authorized as required by the rules of the Federal
requirements, e-mail [email protected].
Communications Commission. This device is not, and may not be, offered for sale or
lease, or sold or leased, until authorization is obtained.

42 Business / Technical Brief / Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Copyright © 2023, Oracle and/or its affiliates / Public