Oracle Exadata Database Machine:

Implementation and Administration

Student Guide – Volume I

D1101652GC10

Learn more from Oracle University at education.oracle.com

Author Copyright © 2021, Oracle and/or its affiliates.

Sean Kim Disclaimer

This document contains proprietary information and is protected by copyright and other
Technical Contributors intellectual property laws. You may copy and print this document solely for your own use in an
Oracle training course. The document may not be modified or altered in any way. Except
and Reviewers where your use constitutes "fair use" under copyright law, you may not use, share, download,
Jerry Lee upload, copy, print, display, perform, reproduce, publish, license, post, transmit, or distribute
this document in whole or in part without the express authorization of Oracle.
Nilesh Choudhury
Jia Shi The information contained in this document is subject to change without notice and is not
warranted to be error-free. If you find any errors, please report them to us in writing.
He Qu
Restricted Rights Notice
Gavin Parish
Ria Mirchandani If this documentation is delivered to the United States Government or anyone using the
documentation on behalf of the United States Government, the following notice is applicable:
Amar Subba
Swapnil Sinvhal U.S. GOVERNMENT END USERS: Oracle programs (including any operating system, integrated
software, any programs embedded, installed or activated on delivered hardware, and
Raman Venkateswaran modifications of such programs) and Oracle computer documentation or other Oracle data
delivered to or accessed by U.S. Government end users are "commercial computer software"
Ganesh Ramakrishnan or "commercial computer software documentation" pursuant to the applicable Federal
Acquisition Regulation and agency-specific supplemental regulations. As such, the use,
Senthilkumar Veeranan reproduction, duplication, release, display, disclosure, modification, preparation of derivative
works, and/or adaptation of i) Oracle programs (including any operating system, integrated
Narayanan software, any programs embedded, installed or activated on delivered hardware, and
Prashant Bhogaraju modifications of such programs), ii) Oracle computer documentation and/or iii) other Oracle
data, is subject to the rights and limitations specified in the license contained in the applicable
Pooja Ajithan contract. The terms governing the U.S. Government's use of Oracle cloud services are defined
Shweta Phabba by the applicable contract for such services. No other rights are granted to the U.S.
Government.
Binny Joseph
Sanjaya Sahu Trademark Notice

MinJi Kim Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be
trademarks of their respective owners.
Peter Fusek
James Womack Intel and Intel Inside are trademarks or registered trademarks of Intel Corporation. All SPARC
trademarks are used under license and are trademarks or registered trademarks of SPARC
International, Inc. AMD, Epyc, and the AMD logo are trademarks or registered trademarks of
Editors Advanced Micro Devices. UNIX is a registered trademark of The Open Group.

Moushmi Mukherjee Third-Party Content, Products, and Services Disclaimer

Aju Kumar
This documentation may provide access to or information about content, products, and
Nikita Abraham services from third parties. Oracle Corporation and its affiliates are not responsible for and
expressly disclaim all warranties of any kind with respect to third-party content, products, and
services unless otherwise set forth in an applicable agreement between you and Oracle. Oracle
Graphic Designers Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred
due to your access to or use of third-party content, products, or services, except as set forth in
Komal Reenu an applicable agreement between you and Oracle.
Lakhan Mandar
Toufik Bagwan
Komal Reenu

Publishers
Sumesh Koshy
Giri Venugopal
Jayanthy Keshavamurthy

1005172021
Contents

1 Introduction
Course Objectives 1-2
Audience and Prerequisites 1-3
Oracle Exadata: 10 Years of Innovation 1-4
Oracle Provides Full Choice of Deployment Model 1-6
Oracle Database Deployment Models 1-7
Additional Resources 1-8
Practice 1: Overview 1-9

2 Exadata Database Machine: Overview

Objectives 2-2
Exadata Database Machine: Introduction 2-3
Why Exadata Database Machine? 2-4
Exadata X8M Hardware 2-6
Exadata Database Machine X8M-2 Database Server Hardware: Overview 2-7
Exadata X8M-2 High Capacity Storage Server Hardware: Overview 2-8
Exadata X8M-2 Extreme Flash Storage Server Hardware: Overview 2-9
Exadata X8M-2 Extended Storage Server Hardware: Overview 2-10
Exadata Capacity and Performance Metrics: Individual Servers 2-11
Exadata 8M-2 Rack Configurations 2-12
Elastic Configuration: Examples 2-13
Exadata Database Machine X8M-8 2-14
Exadata Database Machine X8M-8 Database Server Hardware: Overview 2-15
Exadata Database Machine X8M-8 Use Cases 2-16
Oracle Exadata Virtual Machines 2-17
RDMA Over Converged Ethernet (RoCE): Overview 2-18
Exadata Storage Expansion Racks 2-19
Exadata Database Machine Support: Overview 2-20
Oracle Platinum Services: Enhanced Support at No Additional Cost 2-21
Quiz 2-22
Summary 2-25

3 Exadata Database Machine Architecture

Objectives 3-2
Exadata Architecture: Overview 3-3

iii
 Remote Direct Memory Access (RDMA) 3-5
RDMA over Converged Ethernet (RoCE) 3-6
Exadata Network Architecture 3-7
Leaf Switch Topology 3-9
Scale Performance and Capacity Beyond a Single Rack 3-10
Two Rack Case: Leaf/Spine Topology 3-11
Multiple Rack Case: Up to Eight Racks 3-12
Multiple Rack Case: 9 to 18 Racks 3-13
Exadata Software Architecture: Overview 3-14
Support for Mixed Database Versions 3-17
High Capacity Storage Server: Disk Storage Entities and Relationships 3-18
High Capacity Storage Server: Flash Storage Entities and Relationships 3-19
Extreme Flash Storage Server: Flash Storage Entities and Relationships 3-20
HC and EF Storage Server: Persistent Memory Entities and Relationships 3-21
Disk Group Configuration 3-22
Quiz 3-23
Summary 3-27
Practice 3: Overview 3-28

4 Key Capabilities of Exadata Database Machine

Objectives 4-2
Classic Database I/O and SQL Processing Model 4-3
Exadata Smart Scan Model 4-4
Exadata Storage Index: Overview 4-5
Performance Optimization for SQL Queries with MIN or MAX Functions 4-7
Hybrid Columnar Compression: Overview 4-8
Hybrid Columnar Compression: Data Organization 4-9
Exadata Smart Flash Cache: Overview 4-10
Exadata Smart Flash Cache Intelligent Caching: Overview 4-11
Exadata Smart Flash Cache Intelligent Caching Details 4-12
Conventional Read: Write-Through Cache 4-13
Conventional Write: Write-Back Cache 4-14
Conventional Log Write: Overview 4-16
Persistent Memory (PMEM) 4-17
PMEM Advantages 4-18
Exadata Smart PMEM Cache: Read Operation 4-19
Exadata Smart PMEM Log: Log Write Operation 4-20
In-Memory Columnar Caching on Storage Servers 4-21
Smart Aggregation 4-22
Smart In-Memory Columnar Cache with Row IDs 4-23
Exafusion Direct-to-Wire Protocol 4-24

iv
 Exadata X8M RoCE: High Priority Networking 4-25
Exadata X8M RoCE: Avoiding Packet Loss 4-26
Exadata X8M RoCE: Instant Failure Detection 4-27
Cell-to-Cell Rebalance 4-28
I/O Resource Management: Overview 4-29
Exadata Benefits for Data Warehousing and Analytics 4-30
Exadata Benefits for OLTP 4-32
Quiz 4-33
Summary 4-35
Practice 4: Overview 4-36

5 Exadata Database Machine Initial Configuration

Objectives 5-2
Exadata Implementation: Overview 5-3
Key Documentation 5-5
Exadata Site Preparation 5-6
Exadata Configuration Tool: Overview 5-7
Web-Based Version of OEDA 5-8
Hardware Page: Select Hardware 5-9
Hardware Page: Rack and Customer Details 5-10
Operating System Page 5-12
Rack Networks Page: Admin Network 5-13
Rack Networks Page: Enable VLAN 5-14
Rack Networks Page: Admin network Review and Edit 5-15
Rack Networks Page: Private Network 5-16
Rack Networks Page: Private network Review and Edit 5-17
Users and Groups Page 5-18
Define Clusters Page 5-19
Diskgroup Page 5-20
Quorum Disks on Database Servers 5-22
Database Home Page 5-23
Database Page 5-24
Cluster Network Page: Client Network 5-25
View Network Connections 5-26
Example: IP Address Allocation 5-27
Cluster Network Page: Backup Network 5-28
Alerting Page 5-29
Comments Page 5-30
Exadata Hardware Installation: Overview 5-31
Configuring Exadata: Overview 5-32
Loading the Configuration Information and Installing the Software 5-33

v
 Running the Exadata Deployment Tool 5-34
Result After Installation and Configuration 5-37
Oracle Exadata Deployment Assistant (OEDA) CLI 5-38
OEDACLI Operating Modes 5-39
Using OEDACLI: Manage Exadata Storage Cells 5-41
Using OEDACLI: Manage Clusters 5-43
Example: Use the OEDACLI CREATE CLUSTER Command 5-45
Using OEDACLI: Add Database Homes 5-48
Quiz 5-51
Summary 5-53
Practice 5: Overview 5-54

6 Exadata Storage Server Configuration

Objectives 6-2
Exadata Storage Server Administration: Overview 6-3
Default User Accounts for Oracle Exadata Storage Server 6-4
Exadata Software Users, Roles, and Privileges 6-5
Exadata Software Users, Roles, and Privileges: Examples 6-6
Running CellCLI Commands from Database Servers 6-7
ExaCLI: Examples 6-9
Executing Commands Across Multiple Servers Using dcli 6-10
dcli: Examples 6-11
Executing Commands Across Multiple Servers Using exadcli 6-12
exadcli: Examples 6-13
Testing Storage Server Performance by Using CALIBRATE 6-14
CALIBRATE: Example 6-15
Configuring the Exadata Cell Server Software 6-16
Starting and Stopping Exadata Cell Server Software 6-17
Configuring Cell Disks 6-18
Configuring Grid Disks 6-19
Sparse Grid Disks 6-20
Configuring Hosts to Access Exadata Cells 6-21
Configuring ASM and Database Instances to Access Exadata Cells 6-22
Configuring ASM Disk Groups by Using Exadata Storage 6-23
Specifying Content Type for a Disk Group 6-24
Reconfiguring Exadata Storage 6-26
Optional Configuration Tasks 6-27

vi
 Exadata Storage Security: Overview 6-28
Exadata Storage Security Implementation 6-29
Quiz 6-31
Summary 6-34
Practice 6: Overview 6-35

7 I/O Resource Management

Objectives 7-2
I/O Resource Management: Overview 7-3
I/O Resource Management Concepts 7-5
I/O Resource Management Plans 7-6
I/O Resource Management Plans: Example 7-7
IORM Architecture 7-10
Getting Started with IORM 7-11
Setting the IORM Objective 7-12
Enabling Intradatabase Resource Management 7-14
Intradatabase Plan: Example 7-15
Using Exadata IORM with CDB 7-16
Enabling IORM for Multiple Databases 7-18
Interdatabase Plan: Example 7-19
Category Plan: Example 7-20
Complete Example: Database A 7-21
Complete Example: Database B 7-22
Complete Example: Exadata Cells 7-23
Using Share-Based Allocation in the Interdatabase Plan 7-24
Setting Database I/O Utilization Limits 7-25
Interdatabase Plans and Database Roles 7-26
Flash Cache and Flash Log Resource Control 7-27
Flash Cache Space Resource Management 7-28
PMEM Cache and PMEM Log Resource Control 7-30
PMEM Cache Space Resource Management 7-31
I/O Resource Management Profiles 7-33
Monitoring IORM Using AWR 7-34
Monitoring IORM Using Database Statistics 7-36
Monitoring IORM Using Exadata Metrics 7-37
What to Look for When Monitoring IORM 7-38
Using Database I/O Metrics 7-40
Quiz 7-41
Summary 7-45

vii
8 Recommendations for Optimizing Database Performance
Objectives 8-2
Optimizing Performance 8-3
Persistent Memory Usage 8-4
Setting the PMEM Cache Mode 8-5
Flash Memory Usage 8-10
Influencing Caching Priorities 8-11
Using cachingPolicy attribute 8-12
Choosing the Flash Cache Mode for Non-Extreme Flash Cells 8-13
Setting the Flash Cache Mode 8-14
PMEM/Flash Cache Configurations 8-16
Administering In-Memory Columnar Caching 8-17
Table Compression Usage 8-19
Index Usage 8-21
ASM Allocation Unit Size 8-22
Extent Size 8-23
Exadata Specific System Statistics 8-24
Exadata I/O Latency Capping 8-25
Setting the Exadata Cell I/O Timeout Threshold 8-26
Quiz 8-27
Summary 8-30
Practice 8: Overview 8-31

9 Using Smart Scan

Objectives 9-2
Exadata Smart Scan: Overview 9-3
Smart Scan Requirements 9-4
Situations Preventing Smart Scan 9-5
Monitoring Smart Scan in SQL Execution Plans 9-6
Smart Scan Execution Plan: Example 9-7
Example of a Situation Preventing Smart Scan 9-9
Smart Scan Join Processing with Bloom Filters 9-10
Smart Scan Join Filtering: Example 9-11
Other Situations Affecting Smart Scan 9-12
Exadata Storage Server Statistics: Overview 9-13
Exadata Storage Server Wait Events: Overview 9-14
Smart Scan Statistics: Example 9-15
Smart Scan Wait Events: Example 9-16
Concurrent Transaction: Example 9-17
Extreme Concurrent Transaction: Example 9-18
Migrated Rows: Example 9-19

viii
 I/O Sent Directly to Database Server to Balance CPU Usage: Example 9-20
Column Filtering: Example 9-21
Quiz 9-22
Summary 9-25
Practice 9: Overview 9-26

10 Consolidation Options and Recommendations

Objectives 10-2
Consolidation: Overview 10-3
Methods for Database Consolidation 10-4
Core Principles for Database Consolidation 10-6
Recommended Consolidation Approach 10-7
Recommended Storage Configuration for Consolidation 10-8
Alternative Storage Configurations 10-10
Benefits and Limitations of Partitioned Storage Configurations 10-11
Cluster Configuration Options 10-12
Operating System Parameter Recommendations 10-13
Memory Management Recommendations 10-14
CPU Management Recommendations 10-16
Process Management Recommendations 10-18
Other Recommendations 10-20
Isolating Management Roles 10-21
Consolidation Using Virtual Machines 10-22
Consolidation Using Oracle Multitenant Architecture 10-23
Schema Consolidation Recommendations 10-24
General Maintenance Considerations 10-25
Quiz 10-26
Summary 10-29

11 Migrating Databases to Exadata Database Machine

Objectives 11-2
Migration Best Practices: Overview 11-3
Performing Capacity Planning 11-4
Exadata Migration Considerations 11-5
Choosing the Right Migration Path 11-6
Logical Migration Approaches 11-7
Physical Migration Approaches 11-9
Reducing Down Time for Migration by Using Transportable Tablespaces 11-12

ix
 Other Approaches 11-13
Post-Migration: Best Practices 11-14
Quiz 11-15
Summary 11-18
Practice 11: Overview 11-19

12 Bulk Data Loading

Objectives 12-2
Bulk Data Loading Architectures for Exadata 12-3
Staging Data Files Using DBFS 12-4
Staging Data Files Using ACFS 12-5
Staging Data Files Using External File Systems 12-6
Comparison of Staging Area Options 12-7
Bulk Data Loading Using Oracle DBFS: Overview 12-9
Preparing the Data Files 12-10
Configuring a DBFS Staging Area 12-11
Configuring the Target Database 12-14
Loading the Target Database 12-15
Quiz 12-17
Summary 12-21
Practice 12: Overview 12-22

13 Exadata Database Machine Platform Monitoring: Introduction

Objectives 13-2
Monitoring Technologies and Standards 13-3
Simple Network Management Protocol (SNMP) 13-4
Intelligent Platform Management Interface (IPMI) 13-5
Integrated Lights Out Manager (ILOM) 13-6
Exadata Storage Server: Metrics, Thresholds, and Alerts 13-7
Automatic Diagnostic Repository (ADR) 13-8
Automatic Workload Repository (AWR) 13-9
Enterprise Manager Cloud Control 13-10
Quiz 13-11
Summary 13-13

14 Monitoring Exadata System Software Components

Objectives 14-2
Guidelines for Exadata Monitoring 14-3
Monitoring Oracle Exadata System Software Components 14-4
Monitoring Exadata Smart Flash Cache 14-5
Using AWR 14-6

x
 Using Database Statistics and Wait Events 14-8
Using Exadata Metrics 14-11
What to Look for When Monitoring Exadata Smart Flash Cache 14-12
Monitoring PMEM Cache 14-15
Using AWR 14-16
Using Database Statistics and Wait Events 14-18
Using Exadata Metrics 14-19
What to Look for When Monitoring PMEM Cache 14-20
Monitoring Exadata Smart Flash Log 14-22
Using AWR 14-23
Using Database Statistics and Wait Events 14-24
Using Exadata Metrics 14-25
What to Look for When Monitoring Exadata Smart Flash Log 14-26
Monitoring PMEM Log 14-28
Using Database Statistics and Wait Events 14-29
What to Look for When Monitoring PMEM Log 14-30
Monitoring Cell Disk I/O 14-31
Using AWR 14-32
Using Database Statistics and Wait Events 14-36
Using Exadata Metrics 14-38
What to Look for When Monitoring Cell Disk I/O 14-40
Quiz 14-42
Summary 14-46

15 Configuring Enterprise Manager Cloud Control to Monitor Exadata Database

Machine
Objectives 15-2
Enterprise Manager Cloud Control Architecture: Overview 15-3
Enterprise Manager Cloud Control: Supported Exadata Configurations 15-4
Cloud Control Monitoring Architecture for Exadata 15-5
Configuring Cloud Control to Monitor Exadata 15-6
Pre-Discovery Configuration and Verification 15-7
Deploying the Oracle Management Agent 15-9
Discovering Exadata 15-10
Discovering Additional Targets 15-11
Post-Discovery Configuration and Verification 15-12
Configuring an Exadata Dashboard 15-13
Quiz 15-14
Summary 15-18
Practice 15: Overview 15-19

xi
16 Monitoring Exadata Storage Servers
Objectives 16-2
Lesson Overview 16-3
Exadata Storage Server Metrics and Alerts Architecture 16-4
Monitoring Exadata Storage Server with Metrics 16-5
Monitoring Exadata Cell Metrics: Examples 16-7
Monitoring Exadata Storage Server with Alerts 16-8
Monitoring Cell Alerts and Creating Thresholds: Examples 16-10
Isolating Faults with Exadata Storage Server Quarantine 16-11
Monitoring Exadata Storage Server with Active Requests 16-13
Automatic Hard Disk Scrubbing and Repair 16-14
Adaptive Hard Disk Scrubbing 16-15
Cell Alert Summary 16-16
Cell Diagnostic Packages 16-17
Monitoring Exadata Storage Server with Enterprise Manager: Overview 16-18
Monitoring Metrics in a Storage Server 16-20
Accessing Exadata Metrics 16-21
Aggregated Exadata FlashDisk and HardDisk Metric 16-22
Exadata Storage Server Metric 16-23
Exadata Key Performance Indicators Metrics 16-24
Exadata Key Performance Indicators Metrics: Example 16-25
Comparing Metrics Across Multiple Storage Servers 16-26
Monitoring Hardware Failure and Sensor State 16-27
Checking for Undelivered Alerts 16-28
Checking for Disk I/O Errors 16-29
Checking for Network Errors 16-30
Monitoring File System Free Space 16-31
Third-Party Monitoring Tools 16-32
Quiz 16-33
Summary 16-37
Practice 16: Overview 16-38

17 Monitoring Exadata Database Machine Database Servers

Objectives 17-2
Monitoring Database Servers: Overview 17-3
Monitoring Hardware 17-4
Monitoring the Operating System 17-5
Monitoring Oracle Grid Infrastructure and Database 17-6
Monitoring Oracle Management Agent 17-7
Database Monitoring with Enterprise Manager Cloud Control 17-8
Monitoring Database Servers with MS and DBMCLI: Overview 17-9

xii
 Running DBMCLI 17-10
Starting and Stopping Management Services on Exadata Database Servers 17-11
Configuring Management Services on Exadata Database Servers 17-12
Monitoring Database Server Metrics: Examples 17-13
Quiz 17-14
Summary 17-17
Practice 17: Overview 17-18

18 Monitoring Other Exadata Database Machine Components

Objectives 18-2
Monitoring the RoCE Switches 18-3
Verifying the RoCE Network Fabric Configuration 18-4
Monitoring the Management Ethernet Switch 18-5
Monitoring the Power Distribution Units 18-6
Quiz 18-7
Summary 18-9
Practice 18: Overview 18-10

19 Other Useful Exadata Monitoring Tools

Objectives 19-2
ExaWatcher: Overview 19-3
Using ExaWacher Charts 19-4
Exachk: Overview 19-5
Running Exachk 19-6
Exachk: Sample Reports 19-8
TFA Collector: Overview 19-9
Running TFA Collector on Exadata 19-10
Integrated Lights Out Manager (ILOM): Overview 19-11
Using ILOM 19-12
Accessing the Browser Interface 19-13
Powering the Device On or Off 19-14
Locating the Device 19-15
Viewing Hardware Status and Specifications 19-16
Monitoring Power Consumption 19-17
Accessing the Command-Line Interface (CLI) 19-18
CLI: Examples 19-19
Automated Cloud Scale Performance Monitoring 19-20
Automated Cloud Scale Performance Monitoring: Example – High Memory
Usage 19-21
Quiz 19-22
Summary 19-25

xiii
20 Backup and Recovery
Objectives 20-2
Backup and Recovery: Overview 20-3
Backup Option 1: Existing Exadata Storage Server 20-4
Backup Option 2: Oracle ZFS Storage Appliance 20-5
Example: 100Gb Local Backup Solution 20-6
Example: Sharing Exadata X8M Switches 20-7
Example: Dedicated Top of Rack (ToR) Switches 20-8
Backup Option 3: Zero Data Loss Recovery Appliance 20-9
Backup Option 4: Tape Library 20-10
Backup Option 5: Oracle Database Backup Cloud Service 20-11
Exadata Backup Recommendations 20-12
Configuration Best Practices 20-13
Parallelism Usage with Local Disk on Exadata 20-14
Parallelism Usage with ZDLRA 20-15
Other Recommendations with ZDLRA 20-16
Oracle ZFS Storage Appliance 20-17
Parallelism Usage with Tape Libraries 20-18
Other recommendations with Tape Libraries 20-19
Operational Best Practices 20-20
RMAN Backup Operations 20-22
Archived Log Deletion Policy 20-23
Backup and Recovery of Database Machine Software 20-24
Quiz 20-25
Summary 20-28
Practice 20: Overview 20-29

21 Exadata Database Machine Maintenance Tasks

Objectives 21-2
Exadata Maintenance: Overview 21-3
Powering Exadata Off and On 21-4
Safely Shutting Down a Single Exadata Storage Server 21-5
Do Not Service LED on Storage Servers 21-6
Replacing a Damaged Physical Disk 21-7
Replacing a Hard Disk Proactively 21-9
Moving All Disks from One Cell to Another 21-11
Replacing a Damaged Flash Card 21-13
Replacing a Damaged PMEM Device 21-14

xiv
 Dedicated Disks for System Partitions on Storage Servers 21-15
Replacing a Damaged M.2 Device 21-16
Quiz 21-17
Summary 21-22

22 Patching Exadata Database Machine

Objectives 22-2
Patching and Updating: Overview 22-3
Patching and Updating: Key Information Sources 22-4
Maintaining Exadata Storage Server Software 22-5
Using patchmgr to Orchestrate Storage Server Patching 22-6
Maintaining Database Server Software 22-7
Using DB Node Update Utility 22-8
Using patchmgr to Orchestrate Database Server Patching 22-9
Performing RoCE Network Fabric Switch Updates 22-10
Recommended Patching Process 22-11
Test System Recommendations 22-13
Quiz 22-14
Summary 22-16

23 Exadata Database Machine Automated Support Ecosystem

Objectives 23-2
Auto Service Request: Overview 23-3
ASR Process 23-4
ASR Requirements 23-5
Configuring ASR Manager 23-6
Configuring Exadata for ASR 23-7
Activating ASR Assets 23-8
Verifying the ASR Configuration 23-9
Oracle Configuration Manager: Overview 23-10
Configuring Oracle Configuration Manager 23-11
Quiz 23-12
Summary 23-15

xv