15/11/24, 2:46 p.m. Document 2773012.

1
Copyright (c) 2024, Oracle. All rights reserved. Oracle Confidential.

Things to Consider to Avoid Database Performance Problems on 19c (Doc ID 2773012.1)

In this Document

Purpose
Scope
Details
References

APPLIES TO:

Oracle Cloud Infrastructure - Exadata Cloud Service

Gen 2 Exadata Cloud at Customer
Gen 1 Exadata Cloud at Customer (Oracle Exadata Database Cloud Machine) - Version N/A and later
Oracle Database - Enterprise Edition - Version 19.0.0.0 to 19.14.0.0.0 [Release 19]
Oracle Database Cloud Exadata Service
Information in this document applies to any platform.

PURPOSE

The purpose of this document is to publish some recommended fixes in hopes of avoiding database performance related
problems in 19c. This also includes some ORA-600/ORA-7445 and other errors impacting performance.

For SQL Performance, refer the below document:

Document 2773715.1 Things to Consider to Avoid SQL Performance Problems on 19c

SCOPE

This document contains the list of some known bugs impacting database performance. Some of bug fixes are included in
DBRU but some are not yet.

REMEMBER: Database Release Updates (RU) are cumulative in nature. For example, the Apr 2021 Release Update
patch includes the contents of all previous Release Updates. Hence, it is always RECOMMENDED to be in the latest
DBRU (Database Release Update) that contains most of the known bug fixes.

Patch 33515361 - Database Jan 2022 Release Update (19.14) - Latest

Refer to the below document for the latest DBRU information.

Document 2521164.1 Oracle Database 19c Proactive Patch Information

For bugs that are not included to RU yet, please download and install one-off patches for your database version and OS.
you do not find a patch for your specific version and OS from MOS, please open a Service Request with details on the
patch needed. Please include a list of patches already applied (opatch lsinventory -detail) as well as any other patches yo
intend to apply.

For self-guided upgrade and best practices:-

Document 1919.2 19c Database Self-Guided Upgrade with Best Practices

Document 555.1 Oracle Database 19c Important Recommended One-off Patches

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=8m3unzoni_151&id=2773012.1 1/6
15/11/24, 2:46 p.m. Document 2773012.1

DETAILS

List of some known bug fixes for Database Performance in 19c:

Bug Description

Document 30329209.8 High wait on row cache mutex after upgrading to 12.2.0.1 and above

Document 31933451.8 High row cache mutex contention

Document 29523216.8 Major performance bug for dc_users row cache

Document 30712670.8 High row cache mutex contention for queries with dblink (dc_props / dc_cdbprops)

Document 30431274.8 High row cache mutex contention (ktatminextsz) - regression of 22909260

<Document 29628647.8> High CPU for DESCRIBE command due to contention in dc_users rowcache

Document 32043701.8 row cache lock for sequences in RAC due to S-optimization feature for dc_sequences

Document 30489582.8 Hanganalyze trace unable to identify the blocker of "row cache lock" in RAC

Document 30327149.8 GEN0 process in RAC waiting on ktatminextsz while reading rowcache

Document 30720844.8 CLMN process waits on 'library cache: mutex X' and/or might cause ORA-600 [kglrfcl_1]

Document 30384121.8 LCK process in RAC holds mutex in kglHandleMessage causing database hang

Document 32356628.8 Huge waits on 'library cache: mutex X' with audit enabled for 'select any table' privilege

Document 28889389.8 High waits on 'cursor:mutex X' after upgrade

Document 31211220.8 High version count (cursor leaks) due to BIND_EQUIV_FAILURE mismatch

Document 33163187.8 Child Cursor Increase Due To "Bind Mismatch" Even When Using Same Bind Values

Document 34304965.8 [ROW CACHE] 19c Tracking Bug for Row Cache Bug Fixes - Regressed

Document 35778398.8 FURTHER FIXES FOR ROW CACHE ON TOP OF 34304965 - Regressed & replaced with

Document 35925654.8 SESSIONS DEADLOCK ON ROW CACHE MUTEX AND SHARED POOL LATCH

Document 20319830.8 Latch Get and Free Functions Available for Shared Parent-Child Latches

Document 31602782.8 ORA-12850/ORA-12872 or huge waits on 'cursor: pin S wait on X' with parallel execution

Document 31753692.8 High waits on 'cursor: pin S wait on X' and 'cursor: mutex X' with PX_MISMATCH

Document 30293345.8 Waits for latch: MGA Shared Context Latch After Migration to 18c or above

<Document 30614411.8> Huge delay in LGWR BOC (Broadcast-on-commit) processing in RAC

<Document 31827912.8> High waits for 'log file sync' & 'remote write sync' in RAC ADG with Fast-Start Failover (FSFO

Document 31176502.8 LGWR Hangs during log switch after Upgrade to 19c in RAC due to Cache Fusion Write Hang

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=8m3unzoni_151&id=2773012.1 2/6
15/11/24, 2:46 p.m. Document 2773012.1

Document 31331038.8 ORA-600 error in ADG SYNC mode on Exadata with PMEMlog

Document 32249371.8 High 'log file parallel write' waits on Exadata due to single HCA bottleneck

Document 32498752.8 ORA-600 [ksu_get_available_pso: numa mismatch] signaled when using parallel LGWR

Document 30978554.8 GC hang on read-mostly object or ORA-481 in RAC

Document 32035536.8 Sessions Blocked by 'gc current request' in RAC

Document 28697526.8 Session Hangs On 'gc cr request' And Other Sessions Wait On 'cr request retry'

Document 32227352.8 High CPU with set role command

Document 31812824.8 Slow performance with set role command

Document 28889730.8 "Insert As Select" or a "Create Table As Select" command consume huge space

Document 32379140.8 LCK process in RAC crashes due to ORA-07445 [kjcvmsn()+128] [SIGSEGV]

Document 30240930.8 Scheduler Jobs Time Classified as Background Instead of Foreground

Document 29932310.8 AWR Report Generation Takes Long time in 19c

Document 31489731.8 ORA-600/ORA-7445 While Shared Pool Memory is Being Freed

Document 31892767.8 Improvement to Temp Space Shrink (Affects on Cloud 19c)

Document 32465193.8 ORA-4031 Due to high SQL Monitoring allocations in Shared Pool

Document 31820859.8 ORA-4025 Due To 65535 Active Locks Limit Reached On Select NLS_CHARSET_ID

Document 30887989.8 ORA-00001 While Generating AWR Snapshot in non-CDB with Resource Manager enabled

Document 29423227.8 Drop Partition with global indexes hangs on library cache lock

Performance Slow due to High CPU post July 2020 DBRU (19.8) caused by Space Manageme
Document 32234161.8
processes (Wnnn)

Document 29454450.8 High waits on "latch: cache buffers chains" in RAC

Document 31563138.8 Securefile mutex waits when many inserts occurring on Securefile compressed LOB

Document 32103628.8 High Latch Free Waits While Flushing Top Segment Statistics in AWR

Document 33123985.8 DBW0 Process Generate Huge Traces With Dumping DBWR Process State After DBRU 19.11

Significant Contention on "library cache: mutex X" While accessing Interval or Auto-List Part
Document 32936537.8
table Concurrently

Document 32148419.8 High Row Cache Lock Waits on Alter Table Exchange Partition in RAC Environment

Document 30662963.8 High contention for "latch: MGA shared context root latch" When Many Sessions are Logging

Document 34284147.8 High contention for "latch: MGA shared context root latch" When Many Sessions are Logging

Document 32550751.8 ONLY FOR AIX: Performance issues due to MGA related operations in AIX

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=8m3unzoni_151&id=2773012.1 3/6
15/11/24, 2:46 p.m. Document 2773012.1

ONLY FOR AIX: High waits on 'latch: MGA shared context root latch' and 'latch: MGA shared
Document 33352794.8
latch' even with Fix 32550751

<Document 32117253.8> High "enq: RO - fast object reuse" waits & active checkpoint queue latch gets

Cursor: mutex S waits due to High Version Count For SQL Statements using Bind variables a
Document 30710917.8
From 12.2

Document 33025005.8 Waits on 'latch: cache buffers chains' after Database Upgrade from 12.1.0.2 to 19c

Document 31387123.8 High Waits on 'enq: IV - contention' observed in RAC Standby environment

Document 32225742.8 Gathering Statistics in Primary DB Results in ORA-4061/ORA-4065 in Secondary DB

Document 32069508.8 High Latch: Cache Buffers Chains Contention in Standby Database

Document 33803836.8 Regression in 19.14 Due to Bug Fix 32119144

Document 33163187.8 High Version Count due To "Bind Mismatch" Even When Using Same Bind Values

Document 32755517.8 High Version count with USER_BIND_PEEK_MISMATCH for SQLs with bind peeking disabled

Document 33121934.8 Library cache lock / load lock / mutex x during connection storm due to update user$

RESULT CACHE: GLOBAL FLUSH SHOULD ALWAYS CLEAR BYPASS FLAG EVEN IF THE RESUL
Document 36587533.8
IS UNINITIALIZED

Popular Known Issues

Problem 1: Tablespace level IO statistics are missing in 19c AWR reports

Solution: The Bug Document 25416731.8 is fixed from 19. onwards. Apply 19. or above and do the below steps.

To get Tablespace IO stats data back in AWR reports in 19.X versions, please run following
commands as SYS:

$ sqlplus / as sysdba

exec dbms_workload_repository.modify_table_settings(table_name => 'WRH$_FILESTATXS', flush_level

=> 'TYPICAL');
exec dbms_workload_repository.modify_table_settings(table_name => 'WRH$_DATAFILE', flush_level =>
'TYPICAL');
exec dbms_workload_repository.modify_table_settings(table_name => 'Tempfile Group', flush_level =>
'TYPICAL');
exec dbms_workload_repository.modify_table_settings(table_name => 'WRH$_TEMPSTATXS', flush_level
=> 'TYPICAL');
exec dbms_workload_repository.modify_table_settings(table_name => 'WRH$_TEMPFILE', flush_level =>
'TYPICAL');

When new AWR snapshots are generated, you will start getting Tablespace IO stats data for checking
IO performance.

FYI: You may have to run these commands in the PDBs also if you are generating AWR snapshots at
PDB level and they are missing Tablespace IO stats data in AWR reports.

The Bug Document 34733173.8 is fixed from 19.22RU. These commands are not needed if you apply the
patch 34733173.

Document 25416731.8 - Bug 25416731 - Tablespace IO Statistics Missing From AWR Report
Document 34733173.8 - Bug 34733173 - Tablespace IO Stats and File IO Stats Data Must Be Included in AWR Reports
From Oracle 19C, 21C and 23ai
Document 3008056.1 - AWR Report Under File IO Stats Shows No Data Exists For This Section Of The Report.

Problem 2: High Version counts for the SQLs that are consistently above 1024

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=8m3unzoni_151&id=2773012.1 4/6
15/11/24, 2:46 p.m. Document 2773012.1
Solution: Refer below document for controlling Version counts for the SQLs that are consistently above 1024.

Document 2431353.1 High Version Counts For SQL Statements (>1024) Post Upgrade To 12.2 and Above Causing
Database Slow Performance

Problem 3: High CPU for SQL statements with too many columns. The short stack shows functions spinning on
qosdGetOptDir, qosdInitDirCtx, qosdUpdExprExecStatsRws. There could be high latch waits including GES latches in RAC

Solution: Set, _column_tracking_level=1 at system level. Its a dynamic parameter. This is to avoid the extensive column
usage tracking which could incur more CPU. The default value was 1 in 11.2 & 12.1 and changed to 21 and higher from
12.2 onwards.

Problem 4: If Database is upgraded from 11.2 to 19c, then there is a change in LGWR architecture from 12c onwards b
using parallel adaptive LGWR. This might cause slow LGWR throughput sometimes causing 'log file sync' waits in 19c for
foreground sessions. This is due to some bug with adaptive behavior of parallel LGWR to serial LGWR and vice versa.

Solution: Evaluate the new LGWR architecture (enabled by default) in UAT before PROD. To use the old LGWR architectu
set the below parameter:

_use_single_log_writer=TRUE /* default value: ADAPTIVE */

Note: This is NOT a dynamic parameter. It requires database bounce.

Problem 5: High library cache lock waits in 19c database during Partition maintenance

Solution: Document 2619066.1 High Library Cache Lock Waits After Upgrading To 19C During Partition Index
Maintenance

Refer below documents for troubleshooting library cache related waits:-

Document 444560.1 Troubleshooting Library Cache: Lock, Pin and Load Lock
Document 1353015.1 How to Identify Hard Parse Failures Causing Library Cache contention
Document 2746493.1 How To Trace Overall Library Cache Objects Invalidation Happening At Particular Period

19c Best Practices

1. Database Testing

Oracle Real Application Testing option enables you to perform real-world testing of Oracle Database in Development or U
before doing PROD upgrade. By capturing production workloads and assessing the impact of system changes on these
workloads before production deployment, Oracle Real Application Testing minimizes the risk of instabilities associated wit
system changes post upgrade. SQL Performance Analyzer and Database Replay are key components of Oracle Real
Application Testing.

Documentation: https://docs.oracle.com/en/database/oracle/oracle-database/19/ratug/introduction-to-oracle-database
testing.html
Document 1464274.1 Primary Note for Real Application Testing Option

2. Use Performance Hub

Performance Hub (Perf Hub) feature of EM Express provides an active report with a consolidated view of all performance
data for a specified time period. The report is fully interactive and its contents are saved in a HTML file, which you can
access offline using a web browser. Refer the below tutorials for Performance Hub.

Perf Hub using EM Database Express: https://docs.oracle.com/en/database/oracle/oracle-database/tutorial-monitor-perf/index.html?opt-release-

19c#UsePerformanceHub
Perf Hub without using Database Express: Document 2436566.1 Monitoring Database Performance Using Performance Hub Report

3. Monitor Real-time ADDM

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=8m3unzoni_151&id=2773012.1 5/6
15/11/24, 2:46 p.m. Document 2773012.1
A Prognostic tool to foresee the Performance problems proactively in 19c PROD environment and to take corrective actio
to avoid any system down situation.

Document 2763576.1 Proactively Detecting Database Performance Problem Using Real-Time ADDM

4. Download Latest ORAchk / EXAchk

It is recommended to download and install the latest AHF or ORAchk/EXAchk that can be used to check for any anomalie
and also to take corrective actions.

Document 2550798.1 Autonomous Health Framework (AHF) - Including TFA and ORAchk/EXAChk

5. Install OS Watcher

It is recommended to install the latest version of OS Watcher post upgrade to collect OS related information whenever
required.

Document 301137.1 OS Watcher User Guide

Document 461053.1 OS Watcher Analyzer User Guide

For more information about Database Performance:-

Document 402983.1 Primary Note: Database Performance Overview

Document 1306791.2 Information Center: Oracle Exadata Database Machine

DISCLAIMER: To avoid any issues during patching or post-patching, it is recommended to apply the aforesaid patches in UAT environment and validate before PROD.This
document will be revised every quarter(during the release of quarterly RU/RUR) with the new candidates.

Didn't find what you are looking for?

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=8m3unzoni_151&id=2773012.1 6/6