Exadata Performance Debugging

Biswaroop Biswal / Ranga Sarvabhouman

Agenda


What basic information one must know about Exadata I/O related performance:


- Check if cells are IO bound
- Flash Cache
- Smart Scan


Check if cells are IO bound
Check if db nodes are CPU or memory bound
Check if smart scan works as expected

Check if HCC/Partitioning/DBFS are used
If none is true, go back to database performance tuning and planning



Check if cells are IO bound?
Differentiate between slowness vs. full utilization
Use OSW iostat and/or CellDisk metrics to compute total HDD and FLASH
throughput (MBPS) and IOPS
Refer to Exadata DBM data sheet for peak numbers:
Watch out for high latency if IOs ever approach peak numbers
High latency does NOT mean slow disks
each IO takes long primarily due to time waiting in disk queue
IO latency can be >100ms (note disks are not slow!)
IO latency depends on disk queue length so can be varied based on
different workloads
Be aware that max MBPS and max IOPS can not be reached
simultaneously
How to evaluate mixed workload?
Examine disk utilization - is it close to 100%?
Run calibrate if needed (requires cells being quiesced)

What happens when cells are IO bound
Consider SAME (Stripe And Mirror Everywhere)
when any disk or cell is maxed out, performance will be
throttled by that disk/cell even with workload parallelization
Be aware of potential performance disparity between system and
data disks
System disks not only have user data but also have cells own file
systems
System disks may run slower than data disks
More pronounced on High Capacity 2TB drives due to lower
IOPS capacity when compared with High Performance
600GB drives



Exadata Flash Cache - overview


Know your Flash:



Flash storage entities and relationships:



When/How and what to measure in Flash cache:


When to measure:
- Missing SLA
- Poor performance across the environment.
- AWR reports
- Users screaming
How to measure:
- V$SYSSTAT/V$SQL/V$SESSION_WAIT/V$SESSION_EVENT and V$SySTEM_EVENT
- Exadata Storage Servers using views
- Exadata Storage Servers using metrics (cellcli and dcli commands)
- Exadata Smart flash log with metrics
What to measure:
- Effective use of flash
- Percentage of I/O requests satisfied by flash
- Number of objects kept on flash
- Size of objects kept on flash

Smart Flash log what to look for?


Smart Flash Logging affects log file parallel write time, not log file sync
time.
Check the AWR report for high log file parallel write times; there should be
very few log file parallel write waits > 32 ms, and no waits > 0.5 seconds.
This can be verified in the following sections of an AWR report:
Wait Event Histogram
Wait Event Histogram Detail (64 msec to 2 sec)
If a cell has a non-zero value for FL_IO_W_SKIP_BUSY, then this means that
the hard disks which contain the database log files (or their mirrored
copies) are not performing well. This can due to throughput or heavy load on
the database. It can be resolved by expanding your system to more cells.
If a cell has a non-zero value for FL_ACTUAL_OUTLIERS and
FL_EFFICIENCY_PERCENTAGE is not 100%, then this means that flash disks
are not performing well. This can be due to load or performance issue. If it is
load issue then try to reduce the load or replace the flash disk.
Besides hard disk and flash disk performance, there are other factors which
can affect redo log write latencies:
Examine database nodes to make sure that LGWR is not experiencing
scheduling hiccups due to factors such as swapping.
Check whether the network is impacting the performance.




Smart Scan works as expected & how to debug
Symptoms
In AWR report you would see these 2 wait events
Cell table smart scan
Cell index smart scan
V$views v$sysstat and v$cell_state, statistics that you need to look for (also found system
statitiscs section of AWR report):
Cell physical IO bytes eligible for predicate offload
Cell physical IO bytes saved by storage index
Cell physical IO bytes send directly to DB node to balance CPU usage
Cell physical IO interconnect bytes returned by smart scan
%cell num smart%
PREDIO (v$cell_state)
Oswatcher logs
Reasons
Smart scan has less filtering/no filtering
Suboptimal storage index usage
Less/No filtering due to quarantine/ time zone upgrade
Less filtering due to CPU pass through
Other reasons
Cell is CPU bound or IO bound
Suboptimal flash usage
How to identify long running transaction that
causes sub optimal Storage Index usage?
Purpose:
Long running transaction prevents min active scn from progressing,
there by causing scans to not use storage index.
Steps
Get global min active scn by setting system event to 55703, level 1; the
min active scn is printed in alert log once every 3 minutes. Unset the
event after you get the min active scn. Convert the min scn from hex
to decimals.
Use scn_to_timestamp to compare scn from alert log and current_scn,
if there is differ by good amount then continue.
Query the X$KTUXE to get the slot number, undo segment number
where the column KTUXESTA is not like INACTIVE.
Query v$process to obtain the instance ID, process ID.
Now either you can kill the process or use pstack to obtain more
information.
Less/No Filtering due to quarantines.
Presence of SQL or DB quarantines result in smart scan not being used.
Look at following v$cell_state statistics to see if filtering is not happening
due to quarantines
Smart IO not used to SQL Step or DB Quarantine.
Smart IO not used to Disk Region Quarantine.
Use dcli or cellcli on cells to check for quarantines, for eg: , list quarantine
where QuarantineType=Database in cellcli prompt.
Quarantines are removed automatically when cell software is upgraded or
use cellcli drop quarantine to remove manually.
Less/No filtering due to timezone upgrade; CPU
passthrough
Look at v$cell_state statistics to see if smart scan is not
happening due to timezone upgrade:
Smart IO uses passthru as timezone file is unavailable. OR
Select value from v$sysstat where name = cell num smart
IO sessions using passthru mode due to timezone;
Smart scans will take place after timezone file is made available.
Reasons for CPU passthrough (storage is CPU bound)
More scan queries is running on the storage server resulting in high
CPU usage.
Suboptimal Storage Index usage results in more physical IO being
performed. Which results in more filtering on storage server which
increases CPU usage.
Scans on encrypted tables increases storage CPU usage.
I/O Resource Management Plans
I/O Resource Management Plans : Example
I/O Resource Management Plans : Example
I/O Resource Management Plans : Example
Setting the IORM Objective
Available IORM objective settings:
basic
IORM does not enforce user-defined plans.
IORM protects against extreme latencies for small I/O requests.
Maximum throughput is maintained.
low_latency
Minimizes latency by limiting the number of concurrent I/O requests
Useful for critical OLTP workloads
Performance of high-throughput workloads may suffer
high_throughput
Maximizes throughput by not limiting concurrent I/O requests
Useful for DSS and data warehouse workloads
Performance of latency-critical workloads may suffer
balanced
Balances low disk latency and high throughput
Useful for mixed workloads
auto
IORM decides the best objective setting based on active plans and workloads
Intradatabase Plan : Example
Interdatabase Plan : Example
Using Share-Based Allocation in the
Interdatabase Plan
Commencing with Exadata Storage Server software release 11.2.3.1.0, I/O allocations in the
Interdatabase plan can be expressed as shares rather than using the level and allocation attributes
shown on the previous page. Each share is a value between 1 and 32, with 1 being the lowest share, and
32 being the highest share. The share value represents the relative importance of each database rather
than specifying an IO allocation percentage.
Share-based allocation is a simplified approach designed to support large numbers of databases. Using
shared-based allocations, an interdatabase plan can support up to 1024 directives.
Setting Database I/O Utilization Limits
Database Roles
Category Plan: Example
IORM and Exadata Storage Server Flash Memory
IORM manages only I/O queues for physical disks.
IORM does not arbitrate requests serviced by Exadata Smart Flash
Cache.
IORM can control whether a database can use Exadata Smart Flash
Cache.
IORM can control whether a database can use Exadata Smart Flash Log.
Exadata Storage Server software release 11.2.2.3, IORM can be used to specify if a
database is allowed to use Exadata Smart Flash Cache. This allows flash cache to be
reserved for the most important databases, which is especially useful in environments
that are used to consolidate multiple databases
Complete Example
Complete Example
Exadata Cells
Exadata Metrics and alerts
Monitoring Exadata Storage Server with
Metrics
Monitoring Exadata Cell Metrics
Metric abbreviation :

CL_ (cell)
CD_ (cell disk)
GD_ (grid disk)
FC_ (flash cache)
DB_ (database)
CG_ (consumer group)
CT_ (category)
N_ (interconnect network)
_R for read or _W for write.
_SM or _LG to identify small or large I/Os
At the end of the name, there could be _SEC to signify per second or _RQ to signify
per request

CD_IO_RQ_R_SM is the number of requests to read small blocks on a cell disk.
GD_IO_BY_W_LG_SEC is the number of MB of large block I/O per second on a
grid disk.
I/O-related metric name :

IO_RQ (number of requests)
IO_BY (number of MB)
IO_TM (I/O latency)
IO_WT (I/O wait time)
Monitoring Exadata Storage Server with Alerts
Isolating Faults with Exadata Storage Server
Quarantine
In addition to metrics and alerts, when prescribed faults are detected in Exadata
Storage Server, a quarantine object is automatically created. By this, the action that
caused the fault can be quarantined, so that the fault can be avoided in the future.
Quarantine reduces the chance of storage server software crashes, and improves
storage availability
Exadata Storage Server Quarantines
Types of automatic quarantine :
SQL PLAN: Created when the cell crashes while performing Smart Scan for a SQL statement. The SQL Plan
for the SQL statement is quarantined, and Smart Scan is disabled for the SQL plan.
DISK REGION: Created when the cell crashes while performing Smart Scan of a disk region. The 1 MB disk
region being scanned is quarantined and Smart Scan is disabled for the disk region.
Database: Created when the cell detects that a particular database causes instability. Instability detection
is based on the number of SQL Plan Quarantines for a database. Smart Scan is disabled for the database.
Cell Offload: Created when the cell detects that some offload feature has caused instability. Instability
detection is based on the number of database quarantines for a cell. Smart Scan is disabled for all
databases
CellCLI commands to manually manipulate quarantines:

LIST QUARANTINE: To show the quarantines currently on the cell
ALTER QUARANTINE: To set the comment attribute. The comment attribute is the only quarantine attribute
that can be modified.
DROP QUARANTINE: To manually remove a quarantine.
CREATE QUARANTINE: To manually create a quarantine object. Manual quarantines are created to proactively
isolate SQL statements that are known to cause problems. Example :
CELLCLI> CREATE QUARANTINE quarantineType="SQLID", sqlid="5xnjp4cutc1s8"
Choosing the Flash Cache Mode
Choosing the Flash Cache Mode
Setting the Flash Cache Mode
Enabling write-back mode
Enabling write-through mode
Exadata specific system statistics
Gather Exadata specific system statistics:


Enables the optimizer to more accurately cost operations using
actual performance information:
CPU speed
IO Performance
Sets multi block read count (MBRC) correctly for Exadata
Requires at least Oracle Database version 11.2.0.2 BP 18 or 11.2.0.3 BP 8
Recommended for all new databases
Test thoroughly before changing existing databases.
Databases with stable good plans do not require a change.