Oracle Exadata Monitoring and

Management Best Practices

Session CON9727
October 26, 2015

Ashish Agrawal, Group Product Manager

Swapnil Sinvhal, Sr. Software Dev Manager
Oracle Corporation
Rick Shawver, Infrastructure DBA
Wellsfargo
Om Prakash Seth, Vice President
HDFC Bank LTD

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal/Restricted/Highly Restricted
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal/Restricted/Highly Restricted 3
Program Agenda

1
Top EM 13c Features for Exadata Management
2
Real World Tips & Best Practices
- Wells Fargo Bank
- HDFC Bank

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal/Restricted/Highly Restricted 4
Oracle Exadata Monitoring and Management Best Practices

Presented by :
Rick Shawver

October 2015
 Exadata Monitoring with 12c

Wells Fargo DAN Infrastructure Team

We build, support, and manage standardized, shared environments for Oracle
OID, RMAN, ASM, SAN, NAS, Oracle s/w stack, Grid Control
35000 Targets split between two Grid Control environments (prod and nonprod)
Directly support approximately 640 hosts, 160 clusters, 1756 databases, 5102 instances, etc
Same 11.2.0.4 and 12.1.0.2 versions across the environment
Redhat 5/6, Solaris 10/11, OEL 6 (Exadata), Oracle ZBA
We provide triage support for all of the above to the LOB DBA’s

So what’s new?
We got our first 14 Exadata appliances!
Target usage initially for a few critical applications
Shared environments
Predominately used for OLTP
Now we’re building on our history of lessons learned to develop compatible standards that can be applied to the Exadata
*
architecture
In our new role as DBMA’s we have added patching, upgrading, managing the Exadata infrastructure

6
Exadata Monitoring with 12c

The Exadata plugin exposes the

Appliance Components

* Monitoring complexity is increased if the Exadata platform is used to house databases for
multiple LOB’s and the Exadata infrastructure is maintained by a yet another team.

7
Exadata Monitoring with 12c

The Exadata plugin displays all components of the Appliance Components. Some examples:

* The Exadata cell information is a component that the Application DBA team wanted access to.
This required additional group and incident rule setup.

8
Exadata Monitoring with 12c

9
The 12c target type “oracle_dbmachine” is a composite target that includes ALL targets within the Exadata
environment.

Items to consider when setting up Exadata monitoring:

• In a typical group/role security model, adding the “Exadata Database Machine” to any
group, adds all databases as well as all platform related targets such as cell, iloms, pdus,
network, etc.

• LOB support teams require 12c access to their databases and shared targets such as
host, agents, crs, cells, asm, listeners, but NOT iloms, pdus, network, etc.

• There are cases where databases such as fsdb require monitoring with alerts directed to
the Exadata support team.

10
How Wellsfargo handles this unique separation of duty and alert notifications

• Create a separate 12c group for those Exadata targets that the LOB teams will have access to. This includes the
Exadata Storage Server Grid and the Exadata Storage Servers

• Create two 12c groups for the Exadata support team

• Group 1 contains all platform related Exadata type excluding the DB Machine.
• Group 2 contains only the Top Level DB Machine targets. This is used for the Incident rule alerting on the
overall DB Machine status.

• Place the databases that each LOB and Exadata support teams are responsible for in groups that each team has full
control over.

• Create 12c Roles that contain the appropriate groups for each support team

11
Notice both groups appear to have the same targets. However group 2 only contains:

There is an automatic expansion of the DB Machine targets. If the DB Machines are added
to an Incident Rule, ALL targets on the DB Machine will be included.

12
There are Incident rules for Exadata targets in one rule set, AND
a specific DB Machine Ruleset.

13
The two Exadata rules for the Exadata Support team used two different groups:

14
The typical setup for LOB support groups (Those supporting DBs on Exadata) required view access into the
Exadata Cells

A second LOB group was needed because “view” access is given to the cell targets. The
main LOB group propagates “full” for all targets included.

15
The LOB Exadata group is only given View access to the cells.
The role these groups are in get assigned to the DBA’s in that LOB.

e.g. View Role: LENDING_CORE_ROLE

16
The LOB rules also include a specific rule for cells monitoring.

17
The Exadata Plugin gives a detailed look into the setup and health of an Exadata environment. However, care
must be taken to secure and authorize only those targets that are required for each support role.

Monitoring a shared Exadata environment requires:

• The ability to limit LOB DBA access to those targets they are responsible for.

• The ability to produce alerts for LOB DBAs that exclude specific platform hardware targets that are the sole
responsibility of the Exadata support team.

• Giving the LOB DBA team a view into the Exadata cells, and allow them to receive alerts for the cells.

• The ability to alert on the DB Machine target without inundating the Exadata support team with database alerts
that should only be sent to the LOB DBA teams.

• Allow the Exadata support team to receive alerts for databases they are responsible for (such as fsdb).

18
Enhancing Exadata 12c Monitoring utilizing Metric Extensions
The Exadata Plugin exposes metrics for the Exadata Target Types for
monitoring/alerting.

New incident rules were created with all “out of the box” metrics that Oracle defined
thresholds for.

19
The Exadata Plugin also exposes additional metrics that cannot be used directly for alerting

There are metrics that are collected from the Exadata environment, that thresholds cannot be
set for.

20
In cases where metrics are exposed, but not available for alerting, Repository side Metrics extensions can be
used to produce alerts on those metrics that have not be defined with thresholds.

Application LOB teams desired additional cell alerting that was not available “out of the box”,
but that could be exposed via Metric Extensions.

21
Using Repository side Metric Extensions the desired metrics can be alerted on:

Custom repository side Metric Extensions were created and then added to the LOB DBA
Incident Rules.

22
Repository side Metric Extensions use the metric data that is populated when the Exadata Plugin is deployed:

The Repository side Metric Extensions used data that is being captured by the Exadata
Plugin.
23
Summary:
The Exadata Plugin provides the 12c functionality needed to “See” into and monitor your Appliance.

• The plugin provides an extensive picture of what your Exadata environment looks like. Providing an easy to
understand landscape of all involved components.

• Performance data for all Exadata components is available either through the GUI or 12c reporting.

• Monitoring and alerting for all Exadata Target Types can be enabled to ensure stability and reduce risk of
outages.

• 12c allows for very specific Exadata target security for a variety of support roles.

• 12c Exadata monitoring can easily be extend with Metric Extension to enhance alerting capabilities.

24
Oracle Exadata Monitoring and Management Best
Practices

Om Prakash Seth
Vice President - IT
[email protected]
 HDFC Bank …. Bank aapki Muththi Mein

HDFC Bank Limited, incorporated in 1994, is an Indian banking & financial services company
headquartered in Mumbai, Maharashtra, India

Largest private sector bank in India by market capitalization as of Feb. 2014

Winner of Best Asian Bank award 2015

Top 100 most valuable global brands in 2015 with a value of $14 billion

Ranked as 'Most Valuable Indian Brand’ for second consecutive year

Go Digital ….Bank offers 10-second Personal loan, the 30-minute auto and two wheeler loan, Chillr
app & Payzapp as part of digital banking initiative
About me: Om Prakash Seth is VP – IT & Joint Incident Management Head & manages
production incidents for mission critical Core banking applications. Om has lead the implementation of Oracle Super Cluster in
HDFC Security in 2013, which has been the Word's
first Super Cluster implementation in e-broking segment & has been accredited as "Best technology Implementation of the year"
by Asian Banker's award in 2014.
Evolution of Engineered Systems in HDFC Bank

Extreme High
Performance Availability
Purpose Built
Engineered
Machines

Technology
Scalability
Management
Engineered System’s Journey in HDFC Bank

2012 : Migrated Retail Assets Solution on Exadata

2013 : Migrated Basel GL Solution on Exadata

2013 : Implemented Core Trading Platform on Oracle Super Cluster in

HDFC Securities Ltd

2015 : In Process of Migrating Core Retail Banking Platform on Oracle

Super Cluster
Engineered System’s Footprint in HDFC Bank

10 Exadata database machines running critical workload –

Retail Asset, eBusiness Suites

2 half Rack Oracle Super Cluster machines deployed across

2 data centers for running Core Trading Solution

3 full Rack & 5 half Rack Oracle Super cluster being deployed
across 3 data centers in HA for running Core Retail Banking
System
 Drivers for OEM12c
Real time Centralized
Monitoring Performance
Notification & Optimization & Advisory
Unified
Dashboard

Implemented OEM12c With

ASR & Platinum Support Managing
Automated DB
Fault Tracking operations
and Resolution

Database Life Cycle

Management
 Exadata & Super Cluster Monitoring : Best Practices

Implement IORM for prioritizing application workload

Keep your OEM 12c to latest patch set level

Install OEM 12c Exadata Plug-in for monitoring & enhanced performance
impact detection

Take advantage of performance & tuning, advance notification features of OEM

12c

Implement Monitoring template for Incident creation, notification for hardware &
database

Implement ASR
OEM12c @ HDFC Bank : Technology benefits Achieved

• Integrated Management Console for H/W +S/W

End to End Monitoring • Single Dashboard for all Engineered Systems

Performance & • Comprehensive view including Performance, availability, usage by

databases, services, clusters through OEM 12c Plug-ins
Availability • Enabled out of the box alerts for databases, cluster, ASM, Topology view
Management of DB systems/clusters

• Enabled consolidated Configuration view including Version summary,

Standardization& patch recommendation & Ongoing Database Provisioning
Compliance • Exadata and Super Cluster specific Compliance evaluation, ongoing
Drift tracking across the stack

Cloud Ready • Enabled Private database cloud & smart UAT cloud
• Eliminated need to store multiple copies through Snap Clone on Exadata
Architecture
 Thank You
[email protected]
 34
Oracle Confidential – Internal