LAD - GroundBreakers - Jul 2019 - Using Oracle Autonomous Health Framework to Troubleshoot and Diagnose Database 19c

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Using Oracle Autonomous
Health Framework to
Troubleshoot and Diagnose
Database 19c
LAD – Oracle Groundbreakers
Sandesh Rao
VP AIOps , Autonomous Database
@sandeshr
https://www.linkedin.com/in/raosandesh/
https://www.slideshare.net/SandeshRao4
1

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, timing, and pricing of any
features or functionality described for Oracle’s products may change and remains at the
sole discretion of Oracle Corporation.
2

Program Agenda
Overview & History
Installation and Configuration
Reactive Usage
Proactive Usage
Centralized Usage
1
2
3
4
5
3

Program Agenda
Overview & History
Reactive Usage
Proactive Usage
Centralized Usage
1
2
3
4
5
4

Introducing…
Autonomous Health
Framework
5
A collection of tools as
components, which work
together autonomously 24x7 to
keep database systems healthy
and running while minimizing
human reaction time.

Avoid the Pitfalls of
Inefficient and
Incomplete
Diagnostics Collection
6
Become Proactive and
Avoid Encountering
Known Issues
Help Us
Help You!

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 7
Real-time fault
detection, diagnostic
collection & diagnosis
via a single interface
Secure log collection
Continuous log lifecycle
management
Top problem detection
& diagnostics
What is TFA?
TFA makes it quicker & easier
to detect & diagnose Database problems
Com
ing..

Automatic proactive warning
of problems before they
impact you
8
Get scheduled health reports
sent to you in email
What is ORAchk/EXAchk?
Health checks for most impactful
reoccurring problems
Runs in your environment
with no need to send
anything to Oracle
Findings can be integrated
into other tools of choiceEngineered
Systems
Non
Engineered
Systems
EXAchk
Common Framework
ORAchk

Lots of Pings
Customer Experience Before TFA
Oracle Grid Infrastructure
& Databases
Oracle Support
1 Open new Service Request
Collect data from all nodes
without regard to relevance2
3 Upload data
Collect more
missing data
(ping)
4 5
Upload more
missing data
6
Download tools/scripts
(ping)
7 Run tools/scripts
Upload results of tools/scripts8
Confidential – Oracle Internal
Multiple
iterations &
pings during
SR resolution

Lots of Pings
Support Experience Before TFA
1 Download all files to laptop or view separately in ISDE
End of shift handover write-ups / discussion
after which the new analysts frequently asked
for some of the same data already uploaded
Navigate 100s of log files to figure out problem areas
3 Search SR’s, bugs, MOS notes to find solutions
4
Oracle
Support
2Multiple
iterations &
pings during
SR resolution

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal 11
Customer Experience Before ORAchk/EXAchk
Oracle Databases
“Hi Oracle Support, my database just fell over”
“ahh yer….we published a
note on that a while ago….
…didn’t you see it?”
“hmmmmm”

Experience Today With TFA & ORAchk/EXAchk
TFA UI (TFA Web)
@ Oracle
Add advice
to SR
1
Diagnose SR &
Recommend
Solution
2
Detect
Issue
3
Notification
of Issues
5
Trim, Capture,
Package &
Optionally
Upload
Diagnostics
6
Integrate with AHF
4
OthersCollection
Manager
Integrate & Display
Health Checks Results
2
Auto
Proactive
Health checks
1
TFA /
ORAchk/EXAchk
@ Customer

TFA UI (TFA Web)
@ Oracle
Add advice
to SR
1
Diagnose SR &
Recommend
Solution
2
Detect
Issue
3
5
Trim, Capture,
Package &
Optionally
Upload
Diagnostics
6
Integrate with AHF
4
OthersCollection
Manager
Integrate & Display
2
@ Customer
Notification
of Issues
TFA /
ORAchk/EXAchk
Auto
Proactive
Health checks
1

TFA UI (TFA Web)
@ Oracle
Add advice
to SR
1
Diagnose SR &
Recommend
Solution
2
5
Trim, Capture,
Package &
Optionally
Upload
Diagnostics
6
OthersCollection
Manager
Integrate & Display
2
Detect
Issue
3
Integrate with AHF
4
@ Customer
Notification
of Issues
TFA /
ORAchk/EXAchk
Auto
Proactive
Health checks
1

TFA UI (TFA Web)
@ Oracle
Add advice
to SR
1
Diagnose SR &
Recommend
Solution
2
1
Detect
Issue
3
Trim, Capture,
Package &
Optionally
Upload
Diagnostics
6
Integrate with AHF
4
OthersCollection
Manager
Integrate & Display
2
Notification
of Issues
5
@ Customer
TFA /
ORAchk/EXAchk
Auto
Proactive
Health checks

TFA UI (TFA Web)
@ Oracle
Add advice
to SR
1
Diagnose SR &
Recommend
Solution
2
1
Detect
Issue
3
5
Integrate with AHF
4
OthersCollection
Manager
Integrate & Display
2
Trim, Capture,
Package &
Optionally
Upload
Diagnostics
6
@ Customer
Notification
of Issues
TFA /
ORAchk/EXAchk
Auto
Proactive
Health checks

TFA UI (TFA Web)
@ Customer
Diagnose SR &
Recommend
Solution
2
1
Detect
Issue
3
5
Trim, Capture,
Package &
Optionally
Upload
Diagnostics
6
Integrate with AHF
4
OthersCollection
Manager
Integrate & Display
2
@ Oracle
Add advice
to SR
1
Notification
of Issues
TFA /
ORAchk/EXAchk
Auto
Proactive
Health checks

@ Customer @ Oracle
Add advice
to SR
1
1
Detect
Issue
3
5
Trim, Capture,
Package &
Optionally
Upload
Diagnostics
6
Integrate with AHF
4
OthersCollection
Manager
Integrate & Display
2
TFA UI (TFA Web)
Diagnose SR &
Recommend
Solution
2
Notification
of Issues
TFA /
ORAchk/EXAchk
Auto
Proactive
Health checks

Program Agenda
Overview & History
Reactive Usage
Proactive Usage
Centralized Usage
1
2
3
4
5
19

• Quarterly release cycle
– Follow similar release number formatting to DB
• Installed by default for Grid Infrastructure
• Available for install from database homes
• Updated via Release Updates
• Also available on My Oracle Support (MOS):
Doc 1513912.1
– Supported to patch GI Homes
20
Getting TFA (Which Includes ORAchk/EXAchk)

Installation / Upgrade Using My Oracle Support Download
Continuous Service Mode (Preferred)
1. Transfer zip to required machine
2. Unzip
3. Execute self extracting install script as root user
• Will install/upgrade on all cluster nodes
• Will auto discover relevant Oracle Software & Exadata Storage Servers
• Will start monitoring all discovered items for significant events & collect
diagnostics when necessary
21
./installTFA-<platform>

Program Agenda
Overview & History
Reactive Usage
Proactive Usage
Centralized Usage
1
2
3
4
5
22

Upload diagnostic
collection to Oracle
Support
23
Reactive Usage
Oracle Grid Infrastructure
& Databases
Oracle
Support
TFA
4
1
Find events
2
Diagnose
with DB
tools
3
Perform
diagnostic
collection

Command line
• Specify all command options at the
command line
Shell
1. Set and change context
2. Run commands from within the shell
24
Menu
• Select menu navigation options then
choose the command you want to run
REST
• Invoke commands over HTTPS
TFA Command Interfaces
tfactl <command> tfactl
tfaclt > database MyDB
MyDB tfactl > oratop
tfactl menu
tfactl rest -start
https://host:port/ords/{api}

TFA Utilities To Detect and Analyze Issues
Tool Description
ORAchk or
EXAchk
Provides health checks for the Oracle stack.
Oracle Trace File Analyzer will install: -
• EXAchk for Engineered Systems, see document 1070954.1 for more details
• ORAchk for all non-Engineered Systems, see document 1268927.2 for more details
OSWatcher Oswatcher Collects and archives OS metrics. These are useful for instance or node evictions
& performance Issues. See document 301137.1 for more details
oratop Provides near real-time database monitoring. See document 1500864.1 for more details.
alertsummary Provides summary of events for one or more database or ASM alert files from all nodes
ls Lists all files TFA knows about for a given file name pattern across all nodes
pstack Generate process stack for specified processes across all nodes
grep Search alert or trace files with a given database and file name pattern, for a search string.
summary Provides high level summary of the configuration
25
tfactl <tool>

TFA Utilities To Detect and Analyze Issues
Tool Description
vi Opens alert or trace files for viewing a given database and file name pattern in the vi editor
tail Runs a tail on an alert or trace files for a given database and file name pattern
param Shows all database and OS parameters that match a specified pattern
dbglevel Sets and unsets multiple CRS trace levels with one command
history Shows the shell history for the tfactl shell
changes Reports changes in the system setup over a given time period. This includes database
parameters, OS parameters and patches applied
calog Reports major events from the Cluster Event log
events Reports warnings and errors seen in the logs
managelogs Shows disk space usage and purges ADR log and trace files
ps Finds processes
triage Summarize oswatcher/exawatcher data
26

Collecting Diagnostics with TFA
Standard Diag Collection Targeted Diag Collection via SRDC
1. Run
2. Upload resulting zip file to SR
tfactl diagcollect –srdc <srdc>1. Run
OR
Run
OR
Run
2. Upload resulting zip file to SR
tfactl diagcollect –last <n><d>|<h>
tfactl diagcollect –from <date> -to <time>
tfactl diagcollect
27

Manual Method
1. Generate ADDM reviewing Document 1680075.1 (multiple steps)
2. Identify “good” and “problem” periods and gather AWR reviewing
Document 1903158.1 (multiple steps)
3. Generate AWR compare report (awrddrpt.sql) using “good” and
“problem” periods
4. Generate ASH report for “good” and “problem” periods reviewing
5. Collect OSWatcher data reviewing Document 301137.1 (multiple
steps)
6. Collect Hang Analyze output at Level 4
7. Generate SQL Healthcheck for problem SQL id using Document
1366133.1 (multiple steps)
8. Run support provided sql scripts – Log File sync diagnostic output using
9. Check alert.log if there are any errors during the “problem” period
10. Find any trace files generated during the “problem” period
11. Collate and upload all the above files/outputs to SR
Automated One Command TFA SRDC
1. Run
28
Targeted Diagnostics – Service Request Data Collections (SRDCs)
tfactl diagcollect –srdc dbperf
[-sr <sr_number>]

Detect and Collect
29Confidential – Oracle Internal

Use ‘tfactl’ to check for recent Errors
bash-4.1# tfactl events
Output from host : myserver69
INFO :2
ERROR :2
WARNING :0
Event Timeline:
[Oct/18/2018 02:38:25.000]: [db.ogg11204.ogg112041]: Incident details in:
/scratch/app/oradb/diag/rdbms/ogg11204/ogg112041/incident/incdir_102702/ogg112041_ora_5001_i102
702.trc
[Oct/18/2018 02:38:25.000]: [db.ogg11204.ogg112041]: ORA-00600: internal error code, arguments:
[ksprcvsp2], [1596993584], [], [], [], [], [], [], [], [], [], []
[Oct/18/2018 02:38:37.000]: [db.ogg11204.ogg112041]: Incident details in:
/scratch/app/oradb/diag/rdbms/ogg11204/ogg112041/incident/incdir_102703/ogg112041_ora_5001_i102
703.trc
[Oct/18/2018 02:38:37.000]: [db.ogg11204.ogg112041]: ORA-00600: internal error code, arguments:
[ktfbtgex-7], [1015817], [1024], [1015816], [], [], [], [], [], [], [], []
30

Check to see if a change may have caused the issue ?
-bash-4.1# tfactl changes
------------------------------
[Oct/17/2018 04:54:15.397]: [RDBMS.myDB1]: Parameter: parallel_max_servers: Value: 8 => 16
[Oct/17/2018 05:12:13.344]: [RDBMS.myDB1]: Parameter: log_archive_dest_1: Value: /var => /opt
31

Metadata search capability
• All metadata stored in the TFA index is searchable:
• Searching for all events for a database between certain dates:
32
tfactl search -showdatatypes|-json [json_details]
tfactl search -json
‘{
"data_type":"event",
"content":"oracle",
"database":"rac11g",
"from":“10/01/2018 00:00:00",
"to":"10/21/2018 00:00:00"
}’

Collect ORA-00600 SRDC
bash-4.1$ ./tfactl diagcollect -srdc ORA-00600
Enter the time of the ORA-00600 [YYYY-MM-DD HH24:MI:SS,=ALL] :
Enter the Database Name [=ALL] :
1. Oct/18/2018 02:38:37 : [ogg11204] ORA-00600: internal error code, arguments: [ktfbtgex-7], [1015817],
[1024], [1015816], [], [], [], [], [], [], [], []
2. Oct/18/2018 02:38:25 : [ogg11204] ORA-00600: internal error code, arguments: [ksprcvsp2],
[1596993584], [], [], [], [], [], [], [], [], [], []
Please choose the event : 1-2 [1]
33

Selected value is : 1 ( Oct/18/2018 02:38:37 )
Scripts to be run by this srdc: ipspack rdahcve1210 rdahcve1120 rdahcve1110
Components included in this srdc: OS CRS DATABASE
Collecting data for local node(s)
Scanning files from Oct/17/2018 20:38:37 to Oct/18/2018 08:38:37
WARNING: End time entered is after the current system time.
Collection Id : 20181018032231myserver69
Detailed Logging at :
/scratch/app/oragrid/tfa/repository/srdc_ora600_collection_Thu_Oct_18_03_22_31_PDT_2018_node_loca
l/diagcollect_20181018032231_myserver69.log
2018/10/18 03:22:36 PDT : NOTE : Any file or directory name containing the string .com will be renamed to
replace .com with dotcom
34

.-----------------------------------------------------.
| Collection Summary |
+----------------+---------------+---------+-------+
| Host | Status | Size | Time |
+----------------+---------------+--------+--------+
| myserver69| Completed| 2MB | 97s |
'-----------------+---------------+--------+---------'
Logs are being collected to:
/scratch/app/oragrid/tfa/repository/srdc_ora600_collection_Thu_Oct_18_03_22_31_PDT_2018_node_local
/scratch/app/oragrid/tfa/repository/srdc_ora600_collection_Thu_Oct_18_03_22_31_PDT_2018_node_local
/myserver69.tfa_srdc_ora600_Thu_Oct_18_03_22_31_PDT_2018.zip
35

Analyze
• Each tool can be run using tfactl in shell mode
• Start tfactl shell with
• Run a tool with the tool name
1. Where necessary set context with database <dbname>
2. Then run tool
3. Clear context with database
36
tfactl
tfactl > database MyDB
MyDB tfactl > database
tfactl > orachk
MyDB tfactl > oratop

Manage logs

Manual Database Log Purge
tfactl managelogs –show usage tfactl managelogs –show variation –older <n><m|h|d>
Use -gi to only
show grid
infrastructure
Use –database to only
show database

Manual Database Log Purge
tfactl managelogs –purge –older n<m|h|d> -dryrun tfactl managelogs –purge –older n<m|h|d>
Use –dryrun
for a “what if”

Summary

Summary
-bash-4.1# tfactl summary
Executing Summary in Parallel on Following Nodes:
Node : myserver6969
Node : myserver70
Node : myserver71
LOGFILE LOCATION :
/scratch/app/oragrid/tfa/repository/suptools/myserv
er69/summary/root/20181204025828/log/summary
_command_20181204025828_myserver69_8963.log
Component Specific Summary collection :
- Collecting CRS details ... Done.
- Collecting ASM details ... Done.
- Collecting ACFS details ... Done.
- Collecting DATABASE details ... Done.
- Collecting PATCH details ... Done.
- Collecting LISTENER details ... Done.
- Collecting NETWORK details ... Done.
- Collecting OS details ... Done.
- Collecting TFA details ... Done.
- Collecting SUMMARY details ... Done.
41

Summary
Remote Summary Data Collection : In-Progress - Please wait ...
- Data Collection From Node - myserver70 .. Done.
- Data Collection From Node - myserver71 .. Done.
Prepare Clusterwide Summary Overview ... Done
cluster_status_summary
COMPONENT STATUS DETAILS
+-----------+---------+---------------------------------------------------------------------------------------------------+
CRS OK .-----------------------------------------------------------------------.
| CRS_SERVER_STATUS : ONLINE |
| CRS_STATE : ONLINE |
| CRS_INTEGRITY_CHECK : PASS |
| CRS_RESOURCE_STATUS : OFFLINE Resources Found|
'-----------------------------------------------------------------------'
42

Summary
ASM PROBLEM .-----------------------------------------------------------------------------------------.
| ASM_DISK_SIZE_STATUS : WARNING - Available Size < 20% |
| ASM_BLOCK_STATUS : PASS |
| ASM_CHAIN_STATUS : PASS |
| ASM_INCIDENTS : PASS |
| ASM_PROBLEMS : FAIL |
'-----------------------------------------------------------------------------------------'
ACFS OFFLINE .---------------------------------.
| ACFS_STATUS : OFFLINE |
'---------------------------------'
43

Summary
DATABASE PROBLEM
.--------------------------------------------------------------------------------------------------------------------------------------------------.
| ORACLE_HOME_DETAILS |ORACLE_HOME_NAME |
+----------------------------------------------------------------------------------------------------------------+-------------------------------+
| .--------------------------------------------------------------------------------------------------------------.| OraDb11g_home1 |
| | PROBLEMS | INCIDENTS | DB_BLOCKS | DATABASE_NAME | STATUS | DB_CHAINS | |
| +-----------+-----------+----------------+------------------+---------------+----------------+ | |
| | PASS | PASS | PROBLEM | apxcmupg | PROBLEM | PROBLEM | | |
| '------------+-----------+----------------+------------------+---------------+----------------’ | |
'-----------------------------------------------------------------------------------------------------------------+--------------------------------'
PATCH OK
.----------------------------------------------------------------------------------.
| CRS_PATCH_CONSISTENCY_ACROSS_NODES : OK |
| DATABASE_PATCH_CONSISTENCY_ACROSS_NODES : OK |
'----------------------------------------------------------------------------------'
LISTENER OK
.--------------------------------.
| LISTNER_STATUS : OK |
'--------------------------------' 44

Summary
NETWORK PROBLEM
.-------------------------------------------------------------.
| NODE_APPLICATION_CHECK : FAIL |
| NODE_CONNECTIVITY : FAIL |
| NTP_DAEMON_SLEW_OPTION_CHECK : FAIL |
'-------------------------------------------------------------'
OS OK
.--------------------------------------.
| MEM_USAGE_STATUS : OK |
'--------------------------------------'
TFA OK
.----------------------------------.
| TFA_STATUS : RUNNING |
'----------------------------------'
SUMMARY OK
.-----------------------------------------------------------.
| SUMMARY_EXECUTION_TIME : 0H:1M:48S |
'-----------------------------------------------------------'
+-----------+---------+---------------------------------------------------------------------------------------------------+ 45

Summary
### Entering in to SUMMARY Command-Line Interface ###
tfactl_summary>list
Components : Select Component - select [component_number|component_name]
1 => overview
2 => crs_overview
3 => asm_overview
4 => acfs_overview
5 => database_overview
6 => patch_overview
7 => listener_overview
8 => network_overview
9 => os_overview
10 => tfa_overview
11 => summary_overview
tfactl_summary>
46

Find events

alertsummary
-bash-4.1# tfactl alertsummary
------------------------------
Reading /scratch/app/oradb/diag/rdbms/apxcmupg/apxcmupg_2/trace/alert_apxcmupg_2.log
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
Oct 29 16:19:37 Database started
Summary: Ora-600=0, Ora-7445=0, Ora-700=0
~~~~~~~
Warning: Only FATAL errors reported
Warning: These errors were seen and NOT reported
Ora-12012 Ora-04063 Ora-06508 Ora-06512 Ora-15064 Ora-03113 Ora-15080
48

alertsummary
Reading /scratch/app/oradb/diag/rdbms/ogg11204/ogg112041/trace/alert_ogg112041.log
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
------------------------------------------------------------------------
Aug 01 08:14:48 Database started
Sep 13 07:08:40 Ora-00700 [kgerev1] ogg112041_ora_31177.trc
Sep 13 07:08:40 Ora-00600 [] ogg112041_ora_31177.trc
Sep 13 08:09:49 Ora-00600 [ktfbtgex-7] ogg112041_ora_8881.trc
Sep 15 04:27:17 SystemState Dumped ogg112041_diag_4271_20180915042717.trc
------------------------------------------------------------------------
Sep 18 14:25:15 Database started
Oct 18 02:38:25 Ora-00600 [ksprcvsp2] ogg112041_ora_5001.trc
Oct 18 02:38:37 Ora-00600 [ktfbtgex-7] ogg112041_ora_5001.trc
49

alertsummary
Sep 18 14:25:15 Database started
Oct 18 02:38:25 Ora-00600 [ksprcvsp2] ogg112041_ora_5001.trc
Oct 18 02:38:37 Ora-00600 [ktfbtgex-7] ogg112041_ora_5001.trc
------------------------------------------------------------------------
Summary: Ora-600=8, Ora-7445=0, Ora-700=1
~~~~~~~
Warning: Only FATAL errors reported
Warning: These errors were seen and NOT reported
Ora-19815 Ora-29913 Ora-29400
Reading /scratch/app/oragrid/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
50

calog

calog
#tfactl calog
2018-12-05 10:36:56.301000 : (:CLSGN01660:) CLSNS-00017: invalid status: 3
CLSGN-00524: NS query for subdomain "myserver370044.us.oracle.com" failed.
An error was received from an operating system API:
CLSU-00107: operating system function: getaddrinfo; failed with error data: 0; at location: SCLSIN01
CLSU-00101: operating system error message: Error 0
CLSU-00104: additional error information: node name or service name not known
CLSGN-00178: Resolution of name "GNSTESTHOST.myserver370044.us.oracle.com" failed. :
15426651416834114/2275/1 :
2018-12-05 10:46:58.421000 : (:CLSGN01660:) CLSNS-00017: invalid status: 3
CLSGN-00524: NS query for subdomain "myserver370044.us.oracle.com" failed.
An error was received from an operating system API:
CLSU-00107: operating system function: getaddrinfo; failed with error data: 0; at location: SCLSIN01
52

ls files

ls files
tfactl ls alert_
------------------------------
/u01/app/crsusr/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log
------------------------------
/u02/app/racusr/diag/rdbms/ratc1c/ratc1c_1/trace/alert_ratc1c_1.log
------------------------------
/u01/app/crsusr/diag/rdbms/_mgmtdb/-MGMTDB/trace/alert_-MGMTDB.log
/u02/app/racusr/diag/rdbms/ratc1c/ratc1c_2/trace/alert_ratc1c_2.log
54

grep files

grep files
# tfactl grep 'ORA-15130: diskgroup "MGMT"' alert_
------------------------------
Searching 'ORA-15130: diskgroup "MGMT' in alert_
Searching /u01/app/crsusr/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
13087:ORA-15130: diskgroup "MGMT" is being dismounted
56

grep files
------------------------------
Searching 'ORA-15130: diskgroup MGMT' in alert_
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
Searching /u02/app/racusr/diag/rdbms/ratc1c/ratc1c_1/trace/alert_ratc1c_1.log
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
57

grep files
------------------------------
Searching 'ORA-15130: diskgroup MGMT' in alert_
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
Searching /u01/app/crsusr/diag/rdbms/_mgmtdb/-MGMTDB/trace/alert_-MGMTDB.log
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
Searching /u02/app/racusr/diag/rdbms/ratc1c/ratc1c_2/trace/alert_ratc1c_2.log
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
58

tail files

tail files
-bash-4.1# tfactl tail alert
------------------------------
==> /scratch/app/11.2.0.4/grid/log/myserver69/alertmyserver69.log <==
2018-11-25 23:28:22.532:
[ctssd(5630)]CRS-2409:The clock on host myserver69 is not synchronous with the mean cluster time. No
action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2018-11-25 23:58:22.964:
2018-11-26 00:28:23.395:
60

tail files
==> /scratch/app/oradb/diag/rdbms/apxcmupg/apxcmupg_2/trace/alert_apxcmupg_2.log <==
Sun Nov 25 06:00:00 2018
VKRM started with pid=82, OS id=4903
Sun Nov 25 06:00:02 2018
Begin automatic SQL Tuning Advisor run for special tuning task "SYS_AUTO_SQL_TUNING_TASK"
Sun Nov 25 06:00:37 2018
End automatic SQL Tuning Advisor run for special tuning task "SYS_AUTO_SQL_TUNING_TASK"
Sun Nov 25 23:00:28 2018
Thread 2 advanced to log sequence 759 (LGWR switch)
Current log# 3 seq# 759 mem# 0: +DATA/apxcmupg/onlinelog/group_3.289.917164707
Current log# 3 seq# 759 mem# 1: +FRA/apxcmupg/onlinelog/group_3.289.917164707
61

tail files
==> /scratch/app/oradb/diag/rdbms/ogg11204/ogg112041/trace/alert_ogg112041.log <==
Clearing Resource Manager plan via parameter
Sun Nov 25 05:59:59 2018
Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
Sun Nov 25 05:59:59 2018
Starting background process VKRM
Sun Nov 25 05:59:59 2018
VKRM started with pid=36, OS id=4901
Sun Nov 25 22:00:31 2018
Thread 1 advanced to log sequence 305 (LGWR switch)
Current log# 1 seq# 305 mem# 0: +DATA/ogg11204/redo01.log
62

tail files
==> /scratch/app/oragrid/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log <==
Thu Nov 22 04:42:22 2018
NOTE: [ocrcheck.bin@myserver69 (TNS V1-V3) 2323] opening OCR file
Fri Nov 23 01:05:39 2018
Fri Nov 23 01:05:41 2018
Fri Nov 23 01:21:12 2018
Fri Nov 23 01:21:12 2018
63

vi files

vi files
-bash-4.1# tfactl vi alert
2018-11-25 19:58:19.481:
2018-11-25 20:28:19.911:
2018-11-25 20:58:20.346:
65

tfactl shell history

tfactl Shell History
tfactl> history
05 Dec 18 02:37:17 PST INFO Started session
05 Dec 18 02:37:35 PST COMMAND param kernel.panic
05 Dec 18 02:37:45 PST COMMAND history
tfactl>
67

Monitor Database
performance

oratop (Support Tools Bundle)
Near Real-Time Database Monitoring
• Single instance & RAC
• Monitoring current database activities
• Database performance
• Identifying contentions and bottleneck

oratop
<Oratop Options>:
-d : real-time (RT) wait events, section 3 (default is Cumulative)
-k : FILE#:BLOCK#, section 4 lt is (EVENT/LATCH)
-m : MODULE/ACTION, section 4 (default is USERNAME/PROGRAM)
-s : SQL mode, section 4 (default is process mode)
-c : database service mode (default is connect string)
-f : detailed format, 132 columns (default: standard, 80 columns)
-b : batch mode (default is text-based user interface)
-n : maximum number of iterations (requires number)
-i : interval delay, requires value in seconds (default: 5s)
e.g:
tfactl oratop -database testdb1
tfactl oratop -database testdb1 -bn1
70

-bash-4.1# tfactl oratop -database ogg11204
oratop
71

-bash-4.1# tfactl oratop -database ogg11204 -d
oratop
72

-bash-4.1# tfactl oratop -database ogg11204 -s
oratop
73

Procwatcher (Support Tools Bundle)
Monitor & Examine Database Processes
• Single instance & RAC
• Generates session wait, lock and latch reports as well as call stacks
from any problem process(s)
• Ability to collect stack traces of specific processes using Oracle Tools
and OS Debuggers
• Typically reduces SR resolution for performance related issues
• Runs on ALL major UNIX Platforms
• MOS Note: 459694.1 – Procwatcher Install Guide

Procwatcher
-bash-4.1# tfactl prw start
Mon Nov 26 05:01:09 PST 2018: Starting Procwatcher as user root
Mon Nov 26 05:01:09 PST 2018: Thank you for using Procwatcher. :-)
Mon Nov 26 05:01:09 PST 2018: Please add a comment to Oracle Support Note 459694.1
Mon Nov 26 05:01:09 PST 2018: if you have any comments, suggestions, or issues with this tool.
Procwatcher files will be written to: /scratch/app/oragrid/tfa/repository/suptools/prw/root
Mon Nov 26 05:01:09 PST 2018: Started Procwatcher
75

Procwatcher
-bash-4.1# tfactl prw log runtime
Mon Nov 26 05:01:44 PST 2018: ..SQL: Running SQLvwaitchains.sql on SID ASM1
Mon Nov 26 05:01:49 PST 2018: Saving SQL report data for SID apxcmupg_2
Mon Nov 26 05:01:50 PST 2018: No contention found on DB instance apxcmupg_2, no additional data
collection needed
Mon Nov 26 05:01:50 PST 2018: Saving SQL report data for SID ogg112041
Mon Nov 26 05:01:51 PST 2018: No contention found on DB instance ogg112041, no additional data
collection needed
Mon Nov 26 05:01:51 PST 2018: Saving SQL report data for SID ASM1
Mon Nov 26 05:01:52 PST 2018: No contention found on DB instance ASM1, no additional data collection
needed
Mon Nov 26 05:01:55 PST 2018: SQL collection complete after 44 seconds (10 SQLs - average seconds: 4)
Mon Nov 26 05:01:55 PST 2018: Cycle complete after 44 seconds
Mon Nov 26 05:01:55 PST 2018: Sleeping 16 seconds until time to run again per the INTERVAL setting (60
seconds)
################################################################################
76

Procwatcher
Mon Nov 26 05:02:12 PST 2018: Collecting SQL Data for SID apxcmupg_2
Mon Nov 26 05:02:14 PST 2018: ..SQL: Running SQLvwaitchains.sql on SID apxcmupg_2
Mon Nov 26 05:02:17 PST 2018: Collecting SQL Data for SID ogg112041
Mon Nov 26 05:02:18 PST 2018: ..SQL: Running SQLvwaitchains.sql on SID ogg112041
Mon Nov 26 05:02:20 PST 2018: Collecting SQL Data for SID ASM1
Mon Nov 26 05:02:23 PST 2018: ..SQL: Running SQLvwaitchains.sql on SID ASM1
Mon Nov 26 05:02:26 PST 2018: Saving SQL report data for SID apxcmupg_2
Mon Nov 26 05:02:27 PST 2018: No contention found on DB instance apxcmupg_2, no additional data
collection needed
Mon Nov 26 05:02:27 PST 2018: Saving SQL report data for SID ogg112041
Mon Nov 26 05:02:29 PST 2018: No contention found on DB instance ogg112041, no additional data
collection needed
Mon Nov 26 05:02:29 PST 2018: Saving SQL report data for SID ASM1
Mon Nov 26 05:02:30 PST 2018: No contention found on DB instance ASM1, no additional data collection
needed
Mon Nov 26 05:02:33 PST 2018: Sleeping 38 seconds until time to run again per the INTERVAL setting (60
seconds)
77

Monitor processes

ps of Processes
-bash-4.1# tfactl ps lmd
------------------------------
oragrid 6143 1 0 Oct29 ? 01:13:45 asm_lmd0_+ASM1
oradb 7903 1 0 Oct29 ? 00:55:38 ora_lmd0_apxcmupg_2
oradb 7905 1 0 Oct29 ? 01:04:42 ora_lmd0_ogg112041
------------------------------
oragrid 6089 1 0 Oct29 ? 01:16:48 asm_lmd0_+ASM2
oradb 7035 1 0 Oct29 ? 01:03:55 ora_lmd0_ogg112042
------------------------------
oragrid 8343 1 0 Dec03 ? 00:03:06 asm_lmd0_+ASM3
79

pstack of Processes
-bash-4.1# tfactl pstack lmd
------------------------------
# pstack output for pid : 6143
#0 0x000000341cedf0d8 in poll () from
/lib64/libc.so.6
#1 0x00007fcd83fd38a8 in ssskgxp_poll () from
/scratch/app/11.2.0.4/grid/lib/libskgxp11.so
#2 0x00007fcd83fcbec2 in sskgxp_selectex () from
#3 0x00007fcd83f78b4a in skgxpiwait () from
#4 0x00007fcd83f7720a in skgxpwaiti () from
#5 0x00007fcd83fb79fe in skgxpwait () from
#6 0x0000000003a27525 in ksxpwait ()
#7 0x00000000082e3bc9 in ksliwat ()
#8 0x00000000082e072d in kslwaitctx ()
#9 0x00000000082ddc3b in kslwait ()
#10 0x0000000003a568b3 in ksxprcv_int ()
#11 0x0000000003a550cc in ksxprcvimd ()
#12 0x00000000041ed075 in kjctr_rksxp ()
#13 0x00000000041f0633 in kjctrcv ()
#14 0x00000000041d04c0 in kjcsrmg ()
#15 0x0000000004265c1f in kjmdm ()
#16 0x00000000021c941f in ksbrdp ()
#17 0x00000000023efdc7 in opirip ()
#18 0x000000000169df21 in opidrv ()
#19 0x0000000001c7591b in sou2o ()
#20 0x0000000000853206 in opimai_real ()
#21 0x0000000001c7bc39 in ssthrdmain ()
#22 0x00000000008530fd in main ()
80

Monitor Processes
#0 0x000000341cedf0d8 in poll () from /lib64/libc.so.6
#1 0x00007fd85dc678a8 in ssskgxp_poll () from
/scratch/app/oradb/product/11.2.0/dbhome_11204/li
b/libskgxp11.so
#2 0x00007fd85dc5fec2 in sskgxp_selectex () from
b/libskgxp11.so
#3 0x00007fd85dc0cb4a in skgxpiwait () from
b/libskgxp11.so
#4 0x00007fd85dc0b20a in skgxpwaiti () from
b/libskgxp11.so
#5 0x00007fd85dc4b9fe in skgxpwait () from
b/libskgxp11.so
#6 0x0000000004ebacf5 in ksxpwait ()
#7 0x00000000094daff9 in ksliwat ()
#8 0x00000000094d7b5d in kslwaitctx ()
#9 0x00000000094d506b in kslwait ()
#10 0x0000000004eea083 in ksxprcv_int ()
#11 0x0000000004ee889c in ksxprcvimd ()
#12 0x00000000055d6049 in kjctr_rksxp ()
#13 0x00000000055d9607 in kjctrcv ()
#14 0x00000000055b9494 in kjcsrmg ()
#15 0x000000000564ebf3 in kjmdm ()
#16 0x00000000026abbe3 in ksbrdp ()
#17 0x0000000002910a9b in opirip ()
#18 0x0000000001afd845 in opidrv ()
#19 0x00000000020db5cf in sou2o ()
#20 0x0000000000a29ab6 in opimai_real ()
#21 0x00000000020e18ed in ssthrdmain ()
#22 0x0000000000a299ad in main ()
81

Monitor Processes
#0 0x000000341cedf0d8 in poll () from /lib64/libc.so.6
#1 0x00007ff6260528a8 in ssskgxp_poll () from
b/libskgxp11.so
#2 0x00007ff62604aec2 in sskgxp_selectex () from
b/libskgxp11.so
#3 0x00007ff625ff7b4a in skgxpiwait () from
b/libskgxp11.so
#4 0x00007ff625ff620a in skgxpwaiti () from
b/libskgxp11.so
#5 0x00007ff6260369fe in skgxpwait () from
b/libskgxp11.so
#9 0x00000000094d506b in kslwait ()
#12 0x00000000055d6049 in kjctr_rksxp ()
#13 0x00000000055d9607 in kjctrcv ()
#14 0x00000000055b9494 in kjcsrmg ()
#15 0x000000000564ebf3 in kjmdm ()
#16 0x00000000026abbe3 in ksbrdp ()
#17 0x0000000002910a9b in opirip ()
#18 0x0000000001afd845 in opidrv ()
#19 0x00000000020db5cf in sou2o ()
#22 0x0000000000a299ad in main ()
82

Monitor Processes
------------------------------
#0 0x000000369a6df0d8 in poll () from /lib64/libc.so.6
#1 0x00007f85fab708a8 in ssskgxp_poll () from
#2 0x00007f85fab68ec2 in sskgxp_selectex () from
#3 0x00007f85fab15b4a in skgxpiwait () from
#4 0x00007f85fab1420a in skgxpwaiti () from
#5 0x00007f85fab549fe in skgxpwait () from
#6 0x0000000003a27525 in ksxpwait ()
#7 0x00000000082e3bc9 in ksliwat ()
#13 0x00000000041f0633 in kjctrcv ()
#14 0x00000000041d04c0 in kjcsrmg ()
#15 0x0000000004265c1f in kjmdm ()
#16 0x00000000021c941f in ksbrdp ()
#17 0x00000000023efdc7 in opirip ()
#18 0x000000000169df21 in opidrv ()
#19 0x0000000001c7591b in sou2o ()
#20 0x0000000000853206 in opimai_real ()
#22 0x00000000008530fd in main ()
83

Monitor Processes
#0 0x000000369a6df0d8 in poll () from /lib64/libc.so.6
#1 0x00007f648acc88a8 in ssskgxp_poll () from
b/libskgxp11.so
#2 0x00007f648acc0ec2 in sskgxp_selectex () from
b/libskgxp11.so
#3 0x00007f648ac6db4a in skgxpiwait () from
b/libskgxp11.so
#4 0x00007f648ac6c20a in skgxpwaiti () from
b/libskgxp11.so
#5 0x00007f648acac9fe in skgxpwait () from
b/libskgxp11.so
#9 0x00000000094d506b in kslwait ()
#12 0x00000000055d6049 in kjctr_rksxp ()
#13 0x00000000055d9607 in kjctrcv ()
#14 0x00000000055b9494 in kjcsrmg ()
#15 0x000000000564ebf3 in kjmdm ()
#16 0x00000000026abbe3 in ksbrdp ()
#17 0x0000000002910a9b in opirip ()
#18 0x0000000001afd845 in opidrv ()
#19 0x00000000020db5cf in sou2o ()
#22 0x0000000000a299ad in main ()
84

Monitor Processes
------------------------------
#0 0x00007f12631d63c8 in poll () from /lib64/libc.so.6
#1 0x00007f12653b18a8 in ssskgxp_poll () from
#2 0x00007f12653a9ec2 in sskgxp_selectex () from
#3 0x00007f1265356b4a in skgxpiwait () from
#4 0x00007f126535520a in skgxpwaiti () from
#5 0x00007f12653959fe in skgxpwait () from
#6 0x0000000003a27525 in ksxpwait ()
#7 0x00000000082e3bc9 in ksliwat ()
#13 0x00000000041f0633 in kjctrcv ()
#14 0x00000000041d04c0 in kjcsrmg ()
#15 0x0000000004265c1f in kjmdm ()
#16 0x00000000021c941f in ksbrdp ()
#17 0x00000000023efdc7 in opirip ()
#18 0x000000000169df21 in opidrv ()
#19 0x0000000001c7591b in sou2o ()
#20 0x0000000000853206 in opimai_real ()
#22 0x00000000008530fd in main ()
85

Analyse OS Metrics

OS Watcher (Support Tools Bundle)
Collect & Archive OS Metrics
• Executes standard UNIX utilities (e.g. vmstat, iostat, ps,
etc) on regular intervals
• Built in Analyzer functionality to summarize, graph and
report upon collected metrics
• Output is Required for node reboot and performance
issues
• Simple to install, extremely lightweight
• Runs on ALL platforms (Except Windows)
• MOS Note: 301137.1 – OS Watcher Users Guide

Analyse OS Metrics
-bash-4.1# tfactl oswbb
Starting OSW Analyzer V8.1.2
OSWatcher Analyzer Written by Oracle Center of
Expertise
Copyright (c) 2017 by Oracle Corporation
Parsing Data. Please Wait...
Scanning file headers for version and platform info...
Parsing file myserver69_iostat_18.11.24.0900.dat ...
Parsing file myserver69_iostat_18.11.24.1000.dat ...
……..
88

Analyse OS Metrics
Enter 1 to Display CPU Process Queue Graphs
Enter 2 to Display CPU Utilization Graphs
Enter 3 to Display CPU Other Graphs
Enter 4 to Display Memory Graphs
Enter 5 to Display Disk IO Graphs
Enter GC to Generate All CPU Gif Files
Enter GM to Generate All Memory Gif Files
Enter GD to Generate All Disk Gif Files
Enter GN to Generate All Network Gif Files
Enter L to Specify Alternate Location of Gif Directory
Enter Z to Zoom Graph Time Scale (Does not change
analysis dataset)
Enter B to Returns to Baseline Graph Time Scale
(Does not change analysis dataset)
Enter R to Remove Currently Displayed Graphs
Enter X to Export Parsed Data to Flat File
Enter S to Analyze Subset of Data(Changes analysis
dataset including graph time scale)
Enter A to Analyze Data
Enter D to Generate DashBoard
Enter Q to Quit Program
Please Select an Option:1
89

Analyse OS Metrics
Confidential – Oracle Internal/Restricted/Highly Restricted 90
myserver69

Analyse OS Metrics
myserver69

Check OS / DB
parameters

Check OS or DB parameters
-bash-4.1# tfactl param kernel.panic
.-------------------------------------------------------------.
| OSPARAM |
+------------------------------------------------+----------+
| PARAM | VALUE |
+------------------------------------------------+----------+
| kernel.panic | 60 |
+------------------------------------------------+----------+
| kernel.panic_on_io_nmi | 0 |
+------------------------------------------------+----------+
| kernel.panic_on_oops | 1 |
+------------------------------------------------+----------+
| kernel.panic_on_unrecovered_nmi | 0 |
+------------------------------------------------+----------+
.-------------------------------------------------------------.
| OSPARAM |
+------------------------------------------------+----------+
| PARAM | VALUE |
+------------------------------------------------+----------+
| kernel.panic | 120 |
+------------------------------------------------+----------+
| kernel.panic_on_io_nmi | 0 |
+------------------------------------------------+----------+
| kernel.panic_on_oops | 1 |
+------------------------------------------------+----------+
| kernel.panic_on_unrecovered_nmi | 0 |
+------------------------------------------------+----------+
93

Diagnose cluster health

-bash-4.1# chactl query diagnosis -db oltpacdb -start "2018-11-26 02:52:50.0" -end "2018-11-26 03:19:15.0"
2018-11-26 01:47:10.0 Database oltpacdb DB Control File IO Performance (oltpacdb_1) [detected]
2018-11-26 01:47:10.0 Database oltpacdb DB Control File IO Performance (oltpacdb_2) [detected]
2018-11-26 02:52:15.0 Database oltpacdb DB CPU Utilization (oltpacdb_2) [detected]
2018-11-26 02:52:50.0 Database oltpacdb DB CPU Utilization (oltpacdb_1) [detected]
2018-11-26 02:59:35.0 Database oltpacdb DB Log File Switch (oltpacdb_1) [detected]
2018-11-26 02:59:45.0 Database oltpacdb DB Log File Switch (oltpacdb_2) [detected]
95

Problem: DB Control File IO Performance
Description: CHA has detected that reads or writes to the control files are slower than expected.
Cause: The Cluster Health Advisor (CHA) detected that reads or writes to the control files were slow
because of an increase in disk IO.
The slow control file reads and writes may have an impact on checkpoint and Log Writer (LGWR)
performance.
Action: Separate the control files from other database files and move them to faster disks or Solid State
Devices.
96

Problem: DB CPU Utilization
Description: CHA detected larger than expected CPU utilization for this database.
Cause: The Cluster Health Advisor (CHA) detected an increase in database CPU utilization
because of an increase in the database workload.
Action: Identify the CPU intensive queries by using the Automatic Diagnostic and Defect Manager
(ADDM)
and follow the recommendations given there. Limit the number of CPU intensive queries
or relocate sessions to less busymachines. Add CPUs if the CPU capacity is insufficent to support the load
without a performance degradation or effects on other databases.
97

Problem: DB Log File Switch
Description: CHA detected that database sessions are waiting longer than expected for log switch
completions.
Cause: The Cluster Health Advisor (CHA) detected high contention during log switches
because the redo log files were small and the redo logs switched frequently.
Action: Increase the size of the redo logs.
98

Find if anything has
changed

Has anything changed recently?
-bash-4.1# tfactl changes
------------------------------
[Oct/17/2018 04:54:15.397]: Parameter: fs.aio-nr: Value: 95488 => 97024
[Oct/17/2018 04:54:15.397]: Parameter: fs.inode-nr: Value: 764974 131561 => 740744 131259
[Oct/17/2018 04:54:15.397]: Parameter: kernel.pty.nr: Value: 2 => 1
[Oct/17/2018 04:54:15.397]: Parameter: kernel.random.entropy_avail: Value: 189 => 158
[Oct/17/2018 04:54:15.397]: Parameter: kernel.random.uuid: Value: 36269877-9bc9-40a3-82e0-
1619865096f2 => 7551c5e7-c59f-40fa-b55f-5bd170e8b1ab
[Oct/17/2018 05:46:15.397]: Parameter: kernel.random.uuid: Value: 37cc31aa-ee31-459e-8f2a-
0766b34b1b64 => f5176cdc-6390-415d-882e-02c4cff2ae4e
100

Has anything changed recently?
------------------------------
[Oct/17/2018 04:54:15.397]: Parameter: kernel.random.entropy_avail: Value: 189 => 158
[Oct/17/2018 04:54:15.397]: Parameter: kernel.random.uuid: Value: 36269877-9bc9-40a3-82e0-
1619865096f2 => 7551c5e7-c59f-40fa-b55f-5bd170e8b1ab
[Oct/17/2018 05:46:15.397]: Parameter: kernel.random.uuid: Value: 37cc31aa-ee31-459e-8f2a-
0766b34b1b64 => f5176cdc-6390-415d-882e-02c4cff2ae4e
101

Program Agenda
Overview & History
Reactive Usage
Proactive Usage
Centralized Usage
1
2
3
4
5
102

Proactively Detect
database issues

ORAchk/EXAchk email Notification
• Automatically started & configured to run Critical Health Checks
• You only need to configure your email for notification
104
tfactl orachk/exachk -set “NOTIFICATION_EMAIL=SOME.BODY@COMPANY.COM

ORAchk/EXAchk
Report

Configure Diagnostic Collection email Notification
• Set notification email for any problem detected:
• To set notification email for specific
ORACLE_HOMEs include the OS home owner:
106
tfactl set notificationAddress=john.doe@oracle.com
tfactl set notificationAddress=oracle:another.person@oracle.com

Event Notification

Analysis in MOS

Program Agenda
Overview & History
Reactive Usage
Proactive Usage
Central Repository and UI
1
2
3
4
5
113

Deploy with Minimum Footprint and Maximum Manageability
Oracle 18c Domain Services Cluster
Application
Member
Cluster
Database
Member
Cluster
Database
Member
Cluster
Oracle Domain Services Cluster
Database
Member
Cluster
Application
Member
Cluster
Database
Member
Cluster
ORACLE CLUSTER DOMAIN
Management Repository Service
Trace File Analyzer Service
Grid Names Service
Storage Services
QoS Management Service
Rapid Home Provisioning Service
Confidential – Oracle Internal/Restricted/Highly Restricted
• Hosts Framework as Services
• Reduces local resource footprint
• Centralizes management
• Speeds deployment and
patching
• Optional Shared Storage
• Supports multiple versions and
platforms going forward

Domain Services Cluster Already Has TFA User Interface
• Central TFA Repository utilizing ACFS Storage
• Member Clusters Send TFA Collections to the TFA Service on DSC
• TFA Service indexes the Collection and runs Analysers.
• New UI will be shipped in 19
116

Standalone User Interface
• TFA Collector will upload to central repository
• TFA UI analyses files and generates
– Events TimeLine
– Anomaly TimeLine using Applied Machine Learning
– Root Cause Analysis and Recommendations where available.
– Interface to easily access all files and analyser reports.
• Already used in Oracle Database Cloud.
117

Maintenance Slot Identification
ORAchk/EXAchk results are
automatically uploaded to TFA &
automatically processed

Maintenance Slot Identification

LAD - GroundBreakers - Jul 2019 - Using Oracle Autonomous Health Framework to Troubleshoot and Diagnose Database 19c

More Related Content

What's hot (20)

Similar to LAD - GroundBreakers - Jul 2019 - Using Oracle Autonomous Health Framework to Troubleshoot and Diagnose Database 19c (20)

More from Sandesh Rao (20)

Recently uploaded (20)

LAD - GroundBreakers - Jul 2019 - Using Oracle Autonomous Health Framework to Troubleshoot and Diagnose Database 19c