Running E-Business Suite on

Exadata: Technical Details

April 25, 2017
 Gary Gordhamer
Principal Technologist ERP Infrastructure
Power & Water Technology
GE Power & Water
25 years of IT experience
24 years with Oracle (6.x up to 12c)
Worked in many different industries including healthcare,
manufacturing, utilities, banking, and printing
@ggordham

linkedin.com/in/ggordhamer/

oraontap.blogspot.com

Note: The views in this presentation are my own and do represent the views of the company I work for.

Oracle and NLS, #C17LV Session 333 April 25, 2017 2

Topics

1. What is Exadata
2. Networking
3. OS Configuration
4. Cluster Settings
5. Storage settings
6. DB Settings
7. EBS application items
8. Maintain the Exadata

Presentation Title April 25, 2017 3

Notes!

Oracle has lots of technical notes, you need to follow them!

But which ones? My general order of importance:
1. Start with EBS on Exadata notes
2. Then Exadata notes (ignoring most DB specific items)
3. Then EBS notes
The EBS on Exadata notes supersede any other notes!
List at end of presentation to start with

Presentation Title April 25, 2017 4

1. What is Exadata?

Hyper converged Infrastructure

• Compute (DB NODES)
• Storage (Storage NODES)
• Network (10G Ethernet + IB)
• OS + Cluster
• Management tools (ExaCheck, ExaPatch,
etc…)

Presentation Title April 25, 2017 5

 Exadata Architecture, ½ Rack (X6-2)
Complete | Optimized | Standardized | Hardened Database Platform

Standard Database Servers

– 4x 2-socket servers è 176 cores, 3TB DRAM

Unified Ultra-Fast Network

– 40 Gb InfiniBand internal connectivity è all ports active
– 10 Gb or 1 Gb Ethernet data center connectivity

Scale-out Intelligent Storage Servers

– 7x 2-socket servers è 140 cores in storage
– 40 SAS disk drives è 672 TB total capacity HC drives
– 20 Flash PCI cards è ~90 TB Flash

Fully Redundant
Redundancy notes

• All networking is redundant in all configurations (IB and

10GB Ethernet)
• The administrative network is never redundant (1GB ILOM /
ADMIN port on Exadata / eth0)
• If you are doing on-line storage cell patching, I would
highly recommend a minimum of 4 storage nodes
E.G. ability to lose 2 storage nodes

Presentation Title April 25, 2017 7

2. Networking

The network in Exadata can be complex

Minimum of 3 networks
1. Admin network (eth0) – 1GB Ethernet
• Minimal usage

2. IB network (ib0 / ib1) – 40GB InfiniBand

• Storage, Cluster, Cache Fusion, possibly NFS, SQL*NET

3. Public network (bondeth0 / eth4 / eth5) – 10GB Ethernet

• Primary interface for applications
Note: You can have custom configurations beyond this
Presentation Title April 25, 2017 8
Public network / LISTENER

Primary connectivity for DB

Recommend hostname of DB nodes matches PUBLIC
network IP (note, default from Oracle is admin network IP)
LISTENER
VIP LISTENER, SCAN LISTENER will run on this network
Allows for external interfaces (SOA, data loads, DW, etc…)
Make sure LOCAL_LISTENER and REMOTE_LISTENER are set
right for your EBS database
Presentation Title April 25, 2017 9
Admin Network

By default, local LISTENER on the server will listen on this IP

Make sure application connections are not using this
If the ADMIN IP is your hostname:
• Make sure LOCAL + REMOTE_LISTENER are set right
• LISTENERS can re-direct connections to wrong network
• Autoconfig will pick up hostname by default

Presentation Title April 25, 2017 10

IB Network

If you have application servers on IB,

add IB_LISTENER (MOS Note 1580584.1)
BUT
EBS 12.2 does not support both PUBLIC and IB LISTENER
• EBS 12.1 does support this
• You can probably “hack” the DB configuration, but will
break your EBS 12.2 clone process
IB Network MTU should be 65520 (Exadata version 12.1.2.1.0 +)
Presentation Title April 25, 2017 11
NFS

You will probably be using NFS

• Shared storage between APP and DB tier
• Backups (ZFS?)
• DB files for non-production
Oracle Note: 2087231.1 covers NFS settings for DB file and backup usage
For shared storage between APP and DB tier (CSF)
• If supported NFSv4 recommended (ZFS supports this)
• NFSv4 requires shared directory system (NIS or better yet LDAP)
• Your OS users and groups should exist in your shared directory

Presentation Title April 25, 2017 12

NFS Continued

NFS Mount Options

• MOS Note 359515.1 covers NFS v3 options
• NFSv4 options for IB are covered in EBS MAA Whitepaper
nfs4 rw,bg,hard,nointr,rsize=131072,wsize=131072

If not using NFS over IB (ZFS), recommend HA NFS over public

network for “light” usage (shared App storage)
DB backups on NFS, recommend physically segmented network
• Jumbo frames if supported
• 10GB Ethernet if possible
• IB if using ZFS appliance for backup

Presentation Title April 25, 2017 13

Additional IB / ZFS / Network best practices

1. Enable scatter / gather on Exadata (ver 12.1.2.1.1 +)

cat /sys/module/ib_ipoib/parameters/cm_ibcrc_as_csum
1 = enabled

2. Confirm ZFS supports scatter / gather

(version 2013.06.05.3.0,1-1.7 + )
3. Exadata IPoIB receive queue size of 2048
cat /sys/module/ib_ipoib/parameters/recv_queue_size

4. Exadata TCP Segmentation Offload (TSO) is disabled

/sbin/ethtool --show-offload <interface>

5. Exadata, confirm IB MTU of 65520

Note: check Exalogic ZFS IB interface firmware for fencing bug 17951397

Presentation Title April 25, 2017 14

3. OS Configuration - HugePages

HugePages – you need to configure this

/etc/sysctl.conf - vm.nr_hugepages=<number of pages>

Pages are 2MB on Linux x86-64 by default

You can not use all the memory on the server!
Save Non-hugepage memory for:
• Operating System (OS)
• PGA
• DB utilities (RMAN, SQL*Loader, SQL*Plus, etc…)
• Other non database utilities
Recommend setting this higher than minimum memory needed to account for DB
changes over time.
Start with MOS note: 401749.1 for sizing advice
Presentation Title April 25, 2017 15
Memory Management

Exadata uses RAC / GI clustering, memory management is key!

Memory race conditions / out of memory issues = node evictions
• You must use HugePages for DB memory (OS + DB settings MOS ID 1392497.1)
• /etc/sysctl.conf - vm.min_free_kbytes = 524288
Reserves “low memory” for the OS to prevent race conditions (4194304 for X?-8)
Also required for NFS usage
• When consolidating databases:
Keep track of all DB memory usage
Do no over-use memory on individual nodes

Keep enough memory free for sessions + non-database processes

Note: many EBS systems have 1,000+ sessions taking up 1MB each
Presentation Title April 25, 2017 16
OS Users

OS Users need to be consistent across all nodes of the cluster, and on the
application tiers!
• Recommend central directory (NIS or better LDAP)
• UID and GID number matter!
• If you have existing, make sure OEDA has those numbers during Exadata install

OS Limits - /etc/security/limits.conf + /etc/security/limits.d/*

• Settings in limits.d directory override the limits.conf file!
• Be consistent on all nodes (DB + App tiers)
Recommend making a custom configuration file for limits.d with your
needed setup and copy to all nodes (E.G. 98-mylimits.conf)

Presentation Title April 25, 2017 17

Local Storage

You need unique storage for EBS database homes

EBS DB homes on RAC can not be shared across nodes
Two main options
• Use /u01 and increase the size for what you need
• Add a new mountpoint (E.G. /p01)
Carved out of local disk / LVM volumes on each node
“Exadata Database Machine Maintenance Guide”' => “'Resizing LVM
Partition”
Note: be sure to leave room for LVM snapshots for backup!
Presentation Title April 25, 2017 18
4. Cluster Settings – RAC node timeout

CSS miscount - seconds of failure for cluster before restart

• Previously was default to 60 seconds
• For missions critical apps recommend 30 seconds
• New default is 30 seconds
Reduces cluster “stall” or ”hang” during incidents (termed as
brownout)
As GRID owner user:
crsctl get css misscount
Presentation Title April 25, 2017 19
LISTENER – Dead Connection Detection

For critical systems, recommended to enable DCD

Server side verifies the client is alive
SQLNET.ORA setting on server side (LISTENER)
• $GRID_HOME/network/admin/sqlnet.ora
• $ORACLE_HOME/network/admin/sqlnet.ora
• SQLNET.EXPIRE_TIME = <minutes>

MOS Note: 151972.1

Presentation Title April 25, 2017 20

5. Storage Settings - IORM

For EBS on Exadata Oracle recommends

IORM Plan objective=balanced
CellCLI command:
alter iormplan objective=balanced
This setting:
• Allows large IO’s for 90% of disk capacity
• If there is any queuing of IO, putts small requests at the front of
the queue

Presentation Title April 25, 2017 21

Flash Cache settings

Write Through – writes go to disk, Conservative setting

Write Back – writes go to cache, written to disk out of band
You can see more performance, if your EBS has a lot of write
operations, with Write Back turned on
• X5 and bellow defaults to Write Through
• X6 defaults to Write Back
More details and how to change: 1500257.1
Note: Monthly ”brownout” of flash cache for backup battery cycle test

Presentation Title April 25, 2017 22

6. DB Settings – Log Writer

Exadata can accommodate large redo log buffers

• Minimum of 128MB, can go larger
Unset LOG_CHECKPOINT_INTERVAL
• Use FAST_START_MTTR_TARGET instead
• MTTR advisor can provide advice (if licensed)
• Should be > 300 (Exacheck will report this)
• Helps with batch (CM) program performance
Exadata Flash Logging – enabled after Exadata 11.2.2..4 +
Note: flash logging uses only 512MB of the 300+GB of flash, so a small
investment for huge improvement!
Presentation Title April 25, 2017 23
Fixed Object Statistics

Should be done on the new platform (Exadata)

Needs to be done after the database is “warmed up”
Should be done during “off-peak” time
exec dbms_stats.gather_fixed_objects_stats(’ALL’);
Note 1: 12c has automatic collection for this (unless automatic
collection is turned off, which is normal for EBS)
Note 2: This is normally a one time item event
(may need to be repeated after major patches)
Presentation Title April 25, 2017 24
INIT.ORA settings

Exacheck will give you good guidance in this space,

But, it is just guidance, you will need to sort through what is
needed by your system and all the best recommendations.

Presentation Title April 25, 2017 25

Database patches

Must follow specific note: MOS ID 1392527.1

“Database Patches Required by Oracle EBusiness Suite on
Oracle Engineered Systems: Exadata Database Machines
and SuperClusters”
This note supersedes any other note when running EBS on
Exadata

Presentation Title April 25, 2017 26

7. EBS configuration items

Concurrent Managers
• Recommend hosting on application server or dedicated server
• Recommend not running on Exadata tier
Different patching cycles / levels from DB to app tier
• Exadata = quarterly QFSDP or BP
• App Tier = ???
Easier to manage with application technology stack
May still share NFS mounts with DB tier for UTL_FILE access

Presentation Title April 25, 2017 27

8. Maintaining Exadata – Exacheck

Exacheck – built in and free tool with Exadata

• Upgrades at least quarterly, MOS ID 1070954.1
• Has specific checks for EBS
• Recent version = 1,902 checks
(cluster, hardware, software, OS, network, database)
Collection Manager
• Free application express (APEX) application
• Provides central repository for Exacheck reports
• Alerts, trends, graphs, etc..
You should review the reports and take actions!
Presentation Title April 25, 2017 28
April 25, 2017 Presentation Title 29
Patching Exadata

Recommend using the QFSDP

• Quarterly Full Stack Download Patch
• Patches the OS, Network, Storage, Grid Infrastructure
• Database can be patched at same time, or latter time
• Exadata / GI must be at newer or same level as DB patch
Patches come out quarterly, contain critical fixes
• OS CVE – Security bugs (shell shock, HeartBlead , DirtyCOW, etc…)
• Critical fixes – NFS buffers, memory leaks, GI stability
Patching process will depend a lot on your environment
Recommend a N-1 patching level
Presentation Title April 25, 2017 30
Suggested Exadata Patching Process

Exadata support “on-line” patching

EBS does not support “on-line” infrastructure patching
Recommended method:
• Shutdown EBS (outage starts)
• Shutdown DB
• Patch DB server OS / Firmware
• Patch GI
• Bring DB + EBS back up (outage ends)
• Patch Storage cells ”on-line” (if high redundant, and 4 or more storage cells)
• Patch IB switch “on-line”
Total outage depends on number of DB nodes, but should be less than 5 or 6 hours.
There are other methods and models, but they are much more complex.
Presentation Title April 25, 2017 31
Exadata Backups

Recommend OS backups on Exadata

Use in-house OS backup tools (Tivoli, NetBackup, etc…)
Use LVM snapshots + simple OS tools like TAR
• Oracle Exadata Database Machine Maintenance Guide =>
chapter 2.10 “Creating a Snapshot-Based Backup of Oracle Linux
Database Server”
• Example script:
/opt/oracle.SupportTools/dbserver_backup.sh
Note: this script only backups up the OS, not /u01 or any other
added mount points.

Presentation Title April 25, 2017 32

Wrap up

Lots of documents out there

Pretty well documented but not all in one place
Exadata is a great start, but…
Still need to piece together a full “EBS” system

Presentation Title April 25, 2017 33

Questions

April 25, 2017 34

 References
“Oracle E-Business Suite and Oracle MaximumAvailability Architecture Best Practices” April 2014

MOS ID 1392527.1 “Database Patches Required by Oracle EBusiness Suite on Oracle Engineered
Systems: Exadata Database Machines and SuperClusters”

MOS ID 2087231.1 “Guidelines When Using ZFS Storage in an Exadata Environment”

MOS ID 1354980.1 “Oracle ZFS Storage: FAQ: Exadata RMAN Backup with The Oracle ZFS Storage
Appliance”

MOS ID 1580584.1 “Setup Listener on Infiniband Network using both SDP and TCP Protocol”

MOS ID 401749.1 “Oracle Linux: Shell Script to Calculate Values Recommended Linux HugePages /
HugeTLB Configuration”

MOS ID 1392497.1 “USE_LARGE_PAGES To Enable HugePages”

Presentation Title April 25, 2017 35

 References

MOS ID 1500257.1 “Exadata WriteBack Flash Cache FAQ”

MOS ID 1070954.1 “Oracle Exadata Database Machine exachk or HealthCheck”

Presentation Title April 25, 2017 36