Exadata Technical Deep Dive:

Architecture and Internals

Kothanda (Kodi) Umamageswaran

Vice President, Exadata Development

Gurmeet Goindi
Exadata Product Management

Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.

Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 2

The Exadata Database Machine Vision
Best Platform for the Oracle Database – On Premises and in the Cloud

1. State-of-the-art enterprise-grade hardware, refreshed

yearly (processors, flash, disks, network)

2. Sized, tuned and optimized exclusively for Oracle

Database workloads (DW, Analytics, OLTP, Mixed)
3. High-powered intelligent storage servers capable of
offloading database workloads Exadata
Unique
Intellectual
4. “Smart” database protocols and optimizations from Property
servers to network to storage

5. One vendor responsible for all hardware, software and

customer support

Copyright © 2016. Oracle and/or its affiliates. All rights reserved. 3

Proven at Thousands of Critical Deployments since 2008
Half OLTP - Half Analytics - Many Mixed
4 OF THE TOP 5
BANKS, TELCOS, RETAILERS RUN EXADATA
• Petabyte Warehouses
• Online Financial Trading
• Business Applications
– SAP, Oracle, Siebel, PSFT, …
• Massive DB Consolidation
• Public SaaS Clouds
– Oracle Fusion Apps,
Salesforce, SAS, …

Copyright © 2016. Oracle and/or its affiliates. All rights reserved. 4

 On-Premises Cloud at Customer Public Cloud
Preview:
Exadata Database Machine Exadata Cloud Machine Exadata Cloud Service

Customer Data Center Customer Data Center Oracle Cloud

Purchased Subscription Subscription
Customer Managed Oracle Managed
Oracle Managed
Copyright © 2016. Oracle and/or its affiliates. All rights reserved. 5
Exadata Database Machine X6-2
• Scale-Out Database Servers Compute Software
– Oracle Linux 6
– 2 socket x86 processors – Oracle Database Enterprise Edition
– 44 CPU cores – Oracle VM (optional)
– 256 GB - 1.5 TB GB DRAM – Oracle Database options (optional)
• Fastest Internal Fabric
– 40 Gb/s InfiniBand
– Ethernet external connectivity

• Scale-Out Intelligent Storage Storage Server Software

12.8 TB PCI Flash – Smart Scan (SQL Offload)
96 TB disk – Smart Flash Cache
20 CPU cores – Hybrid Columnar Compression
– High-Capacity Storage Server
– I/O Resource Management
25.6 TB PCI Flash
20 CPU cores
– Extreme Flash Storage Server
Copyright © 2016. Oracle and/or its affiliates. All rights reserved. 6
Exadata Database Machine X6-8
• Scale-Out Database Servers Large SMP Processor Model
– 8-socket x86 – Large warehouses
processors – Massive database consolidation
– 144 cores – Big In-Memory databases
– 2-6 TB DRAM

• Fastest Internal Fabric

– 40 Gb/s InfiniBand
– Ethernet external connectivity

• Scale-Out Intelligent Storage Storage Server Software

Same Networking, Storage and
– Smart Scan Software
(SQL Offload)

– High-Capacity Storage Server

as X6-2
– Smart Flash Cache
– Hybrid Columnar Compression
– I/O Resource Management

– Extreme Flash Storage Server

Copyright © 2016. Oracle and/or its affiliates. All rights reserved. 7
Elastic Configurations Incrementally Scale Servers
Achieve any Level of Performance with Minimum Hardware

Database Server

 
Incrementally Add
add DB or Extreme Flash Storage Racks to
Storage Continue
Servers Scaling
High Capacity Storage

 Start Small Full Rack Multi-Rack

2 Database Servers
3 Storage Servers • Enable Database CPU cores as needed with Capacity on Demand
• Expand older Exadata machines with new X6-2 servers

Copyright © 2016. Oracle and/or its affiliates. All rights reserved. 8

Oracle Database Exadata Cloud Service
• Full Oracle Database with all advanced options
– 100% Compatible with on-premises databases
• On fastest and most available database cloud platform
– Scale-Out Compute, Scale-Out Intelligent Storage, InfiniBand, PCIe Flash
– Complete isolation of tenants with no overprovisioning
• All Benefits of Public Cloud
– Fast, elastic, web driven provisioning
– Oracle experts deploy and manage infrastructure
– Monthly or yearly subscription with online capacity bursting

Best of On-Premises with Best of Cloud

Copyright © 2016. Oracle and/or its affiliates. All rights reserved. 9
Preview: Oracle Public Cloud Services @ Customer

• Same PaaS and IaaS hardware and software

as Oracle Public Cloud
• Managed by Oracle and delivered as a service
in your datacenter behind your firewall
• Same cost-effective subscription pricing
model as Oracle Cloud
• Helps conform to business and government
security requirements
• Connect via fast LAN to existing systems

Copyright © 2016. Oracle and/or its affiliates. All rights reserved. 10

Exadata X6 is Much Faster and Cheaper than All-Flash EMC
Analytic Scans OLTP Write IOPS
• One High Capacity Exadata
beats the fastest EMC 350
5
5.2 M
301
XtremIO all-flash array in 300
4
every performance metric 250 2.5X

GB/sec
200 12X 3
– 12X more throughput 150
2M
2
– 2.5X more IOPS 100
24 1
– 2X faster latency 50

0 0
8 X-Brick 8 X-Brick
EMC 8 X-Brick XtremIO: $7.8 M EMC XtremIO 1 Rack HC
Exadata EMC 1 Rack HC
XtremIO Exadata
Exadata X6-2 Full Rack: $1.1 M
EMC Performance does not scale higher - Exadata scales by adding racks

Copyright © 2016. Oracle and/or its affiliates. All rights reserved. 11

Preview:
Exadata SL6

Linux on SPARC Software in Silicon

Copyright © 2016 Oracle and/or its affiliates. All rights reserved. | 12

Database Intelligence Extended into CPU Chip
SPARC M7 Software in Silicon

• Traditional DB algorithms too complex for chips

• Big Change: In-memory algorithms are much simpler
Software • 5 years ago Oracle initiated a revolutionary project
in Silicon – Build fastest ever microprocessor
• Most processing cores (32)
• Most concurrent threads (256)
• Fastest Memory Bandwidth (160 GB/sec)
– Add In-Memory DB operations directly on chip

Copyright © 2016 Oracle and/or its affiliates. All rights reserved. | 13

In-Memory Algorithms Natively Implemented in Silicon

SQL in Silicon Capacity in Silicon

DB Acceleration SPARC M7 Decompression Engines
Software
in Silicon

Database Software
Silicon Secured Memory Already Available
Fine-Grained Memory
Protection

Copyright © 2016 Oracle and/or its affiliates. All rights reserved. | 14

SQL in Silicon: Database In-Memory Acceleration Engines

SPARC M7 • SIMD Vectors instructions are fast, but were designed for
graphics, not database

Core Core Core Core • New SPARC M7 chip has 32 optimized database
acceleration engines (DAX) built on chip
Shared Cache • Independently process streams of columns
– E.g. find all values that match ‘California’
DB DB DB DB
Accel Accel Accel Accel – Up to 170 Billion rows per second!
• Like adding 32 additional specialized cores to chip
– Using less than 1% of chip space

Copyright © 2016 Oracle and/or its affiliates. All rights reserved. |

Capacity in Silicon: Decompression Engines

• Compression is key to putting more data in-memory

• Decompression is far more important for databases than
compression
– Data is loaded once, queried many times

• Bit pattern decompression in normal cores is slow

– 64 CPU cores needed to decompress at full memory speed
Doubles Memory • SPARC M7 adds 32 optimized decompress engines
Capacity
– Run bit-pattern decompress at memory speed

Copyright © 2016 Oracle and/or its affiliates. All rights reserved. |

Silicon Secured Memory: Fine Grained Memory Protection

• Database In-memory places terabytes of data in memory

– More vulnerable to corruption by bugs/attacks than storage

• SPARC M7 locks memory as it is allocated so only the owner

can access it
– Hidden “color” bits added to pointers (key), and content (lock)
– Pointer color (key) must match content color or program is aborted
– Hardware support eliminates performance impact

• Helps prevent access off end of structure, stale pointer access,

malicious attacks, etc. plus improves developer productivity STOP
Memory Memory
Pointers Content
Copyright © 2016 Oracle and/or its affiliates. All rights reserved. |
Exadata SL6: Exadata with Ultra-fast SPARC Linux Servers
• Identical to Exadata with x86 Database servers
replaced by SPARC T7-2 servers
– Ultra-fast 32-core SPARC M7 Processors
– Two-socket T7-2 Servers
• Same elastic configurations as Exadata X6-2
• Storage servers identical as Exadata X6-2
• Runs same Oracle Linux as Exadata X6-2
– Oracle Linux (UEK2) – single domain configuration
• Runs Oracle Database 12.1.0.2

Copyright © 2016 Oracle and/or its affiliates. All rights reserved. | 18

Preview: Exadata SL6
World’s Fastest and Most Secure Linux Database Machine

Massive Memory Bandwidth Fastest Database Processor Silicon Secured Memory

End to End
2.2x Intel x86 1.9x Intel x86
Database Security
Copyright © 2016 Oracle and/or its affiliates. All rights reserved. | 19
Exadata Smart System Software

Copyright © 2016, Oracle and/or its affiliates. All rights reserved.

Smart System Software Highlights
Smart Analytics Smart Storage
• Move queries to storage, not storage to • Hybrid Columnar Compression reduces
queries space usage by 10X
• Automatically offload and parallelize queries • Database-aware Flash Caching gives
across all storage servers speed of flash with
• 100X faster analytics capacity of disk PCI Flash

Smart OLTP Smart Consolidation

• Special InfiniBand protocol enables highest • Workload prioritization from CPU to
speed, lowest latency OLTP network to storage ensures QoS
• Ultra-fast transactions using • 4X more Databases in same hardware
DB optimized flash logging algorithms
• Fault-tolerant In-Memory DB by mirroring
memory across servers

Copyright © 2016. Oracle and/or its affiliates. All rights reserved. 21

Smart System Software Introduced in 2015
Smart Analytics Smart Consolidation
• 5X faster scans by converting data to • Zero overhead VMs
Columnar format in Flash Cache • Snapshots for test/dev
• 3X faster JSON/XML by • Set flash cache min size per DB to ensure QoS
offloading to storage servers VM VM
• InfiniBand partitioning
• IPv6 for Ethernet

Smart OLTP Smart Licensing

• 3X faster OLTP messaging using direct DB to • Capacity-on-Demand reduces license cost by
InfiniBand access disabling unneeded cores
• Instant detection of node failure • Trusted Partitions limit license scope of
• Sub-second capping of I/O latency by specialized options
rerouting I/Os to faster storage

Copyright © 2016. Oracle and/or its affiliates. All rights reserved. 22

Preview: New Smart System Software
Smart Analytics Smart Consolidation
• Database In-Memory columnar format in • Hierarchical snapshots
storage server • 2X application connections*
• Aggregation in storage Products • Automated VLAN creation* Sparse Sparse
Snap Snap
• Set membership using new • Add extra 10g Ethernet Card Base
DB
type of storage index
• 64GB DIMMs for 2X Memory CDB

Smart OLTP Smart Availability

• Smart Fusion Block Transfer eliminates log • Short Range Stretch (Extended) clusters
writes when moving blocks between nodes* • 4X faster software updates*
• Automated rolling upgrade across full stack • High redundancy Quorum disks on Quarter
• 2X faster disk recovery and Eighth racks*
• Storage Index preserved on rebalance*
*Already Released
Copyright © 2016. Oracle and/or its affiliates. All rights reserved. 23
Upcoming: In memory format in Columnar Flash Cache
• In-Memory formats used in Smart Columnar Flash Cache
• Enables vector processing on storage server during smart scans
– Multiple column values evaluated in single instruction
In-Memory
• Faster decompression speed than Hybrid Columnar Compression Columnar scans
• Enables dictionary lookup and avoids processing unnecessary rows
• Smart Scan results sent back to database in In Memory Columnar
format
In-Flash
– Reduces Database node CPU utilization
Columnar scans
• In-memory performance seamlessly extended from DB node DRAM
memory to 10x capacity flash in storage
– Even bigger differentiation against all-flash arrays and other in-memory
databases

Upcoming release of Exadata Software

Copyright © 2016. Oracle and/or its affiliates. All rights reserved. 24
Upcoming: Storage Index Set Membership
• Storage Index ORIGIN DEST NAME ADDRESS
– Currently contains up to 8 columns of min/max summary Sierra Leone AZ Alice …
– Created automatically and kept in memory Sierra Leone UT Bob …
– Used to skip performing I/Os
Sierra Leone VT John
• What about queries with low cardinality columns?
select name, address from travels HASH(AZ) HASH(UT) HASH(VT)
where origin=‘Sierra Leone’ and dest=‘CA’
• Traditional min/max not good enough Create Bloom Filter
• Database gathers stats and find that column has less than
256 distinct values Bloom Filter in Storage Index
• Database requests storage to compute bloom filter First Scan
• Storage will compute distinct values and create a bloom Future Scans
filter
• Smart Scans check value ‘CA’ against bloom filter and HASH(CA) Lookup SAVE I/O
saves performing I/O
Upcoming release of Exadata Software

Copyright © 2016. Oracle and/or its affiliates. All rights reserved. 25

Upcoming: Join and Aggregation Smart Scan
• Extend In-Memory Aggregation technique into storage
• Find Sales per country
SELECT /*+ VECTOR_TRANSFORM */ country_id, sum(amount_sold) amount_sold
FROM customers, sales
WHERE customers.cust_id = sales.cust_id
GROUP BY customers.country_id
ORDER BY customers.country_id;
• Storage cells scanning sales fact table will return tuples
{ country_id, sum_amount_sold }
• Join and Aggregation offloaded to the storage server

12.2 Database and 12.2 Exadata Storage Server Software

Copyright © 2016. Oracle and/or its affiliates. All rights reserved. 26
Upcoming: Smart write bursts and temp IO in flash cache
• Write throughput of four flash cards has become greater than the
write throughput of 12-disks
Write Burts and Temp IO
• When database write throughput exceeds the throughput of disks, in
smart flash cache intelligently caches writes Flash Cache
• When queries write a lot of temp IO and it is bottlenecked on disk,
smart flash cache intelligently caches temp IO
– Writes to flash for temp spill reduces elapsed time
– Reads from flash for temp reduces elapsed time further

• Smart flash cache prioritizes OLTP data and does not remove hot
OLTP lines from the cache
• Smart flash wear management for large writes

Upcoming release of Exadata Software

Copyright © 2016. Oracle and/or its affiliates. All rights reserved. 27
Upcoming: Smart Analytics Software Features

• Compressed Index Fast Full Scan

• Smart Scan VIEWs with LOBs, XML and JSON
– not just tables
• AWR Enhancements
– Diff report for Exadata section
– Flash Cache Metrics
– More granular histograms
• Up to 25% reduction in Storage Server CPU for SPARC
SuperCluster during Smart Scans
– Reduces endianness conversion overhead

12.2 Database and 12.2 Exadata Storage Server Software

Copyright © 2016. Oracle and/or its affiliates. All rights reserved. 28
Upcoming: Snapshots
• Hierarchical Snapshots Nightly Master
– Create snapshots of databases on previously created
snapshots
– Use case example
• Development releases nightly build of the database
• Tester creates a snapshot for himself and finds a bug Test Snapshot
• Tester creates a snapshot of his snapshot
• Tester provides the new copy back to development for analysis
– Syntax and technology remain unchanged
– Works with pluggable and non-pluggable databases
• Sparse backup of snapshots
– RMAN backs up the modified blocks and not the unchanged blocks
from parent Snapshot to Dev

12.2 Database and 12.2 Exadata Storage Server Software

Copyright © 2016. Oracle and/or its affiliates. All rights reserved. 29
Upcoming: Extended Distance Clusters
• Two sites and a quorum site
• InfiniBand connected for high performance
– 100m optical cables in 2016 (best for fire cells)
• Created using ASM Extended Diskgroups
– Nested failure groups InfiniBand
• Compute nodes at each site read data local to that site
• Data is written to all sites
• Smart Scans scan across cells on both sites increasing
throughput
– Row filtering, column projection, storage index, and flash cache
provide extreme performance Quorum Failure Group
• Data Guard continues to be the recommended DR solution
12.2 Database and 12.2 Exadata Storage Server Software
Copyright © 2016. Oracle and/or its affiliates. All rights reserved. 30
Smart Fusion Block Transfer
• OLTP workloads can have hot blocks that are Prior Inter-Instance Block Transfer Protocol
frequently updated (e.g. right-growing index )
– Log file must be written before transferring a hot block 1. Issue log write
between instances so the block can be recovered
– Adds latency and reduces throughput

• On Exadata, Oracle does not wait for the log write 3. Transfer 2. Wait for log
block write completion
– Exadata ensures the log write completes before changes to
block on another instance commit, guaranteeing durability
– Wait for Log I/O during transfer of hot blocks is eliminated Exadata Avoids I/O Wait
– Up to 40% throughput and 33% response time improvement
in some heavily contended OLTP workloads
Available with 12.1.0.2 BP12

Copyright © 2016. Oracle and/or its affiliates. All rights reserved. 31

Upcoming: Super Fast Software Updates

• 4x speed up in Storage Server Software Update

– Parallel firmware upgrades across components such as hard disks, flash,
ILOM/BIOS, InfiniBand card
– Reduced reboots for Software updates
– Use kexec where possible
• Manage a Cloud instead of managing a single rack
– Use single patchmgr utility to upgrade hundreds of racks
• Enable patchmgr to run from a non-Exadata system and run as
low privileged user

Upcoming release of Exadata Software

Copyright © 2016. Oracle and/or its affiliates. All rights reserved. 32
Upcoming: Extreme Manageability
• IPv6 + Virtual machine + VLAN deployments
• Get graphs from Exawatcher
• Make DNS, NTP, and other IP address changes online
• Seamless customer service with Automatic Service
Requests sending diagnostic attachments
• Manage Compute nodes using a RESTful service
– ExaCli enabled for compute nodes in addition to storage cells

• Much faster rebalance with improved flash cache hit

ratio during rebalance
• Secure Erase during hardware retirement

Upcoming release of Exadata Software

Copyright © 2016. Oracle and/or its affiliates. All rights reserved. 33
Exadata Advantages Increase Every Year
• Exadata Cloud Service
Transformational OLTP, • In-Memory Columnar in Flash
• Smart Fusion Block Transfer
Analytics, Consolidation • In-Memory Fault Tolerance
• Direct-to-wire Protocol
• JSON and XML offload
Cloud Without Compromise • Instant failure detection
• Network Resource Management • 3D V-NAND
• Multitenant Aware Resource Mgmt
Flash
• Prioritized File Recovery • Software-in-
• IO Priorities Silicon
a re • Data Mining Offload
ftw • Tiered Disk/ Flash
r t So • Database Aware Flash Cache
• Offload Decrypt on Scans
S ma • Storage Indexes
• PCIe NVMe Flash
• Columnar Compression • Unified InfiniBand
• Smart Scan
• InfiniBand Scale-Out
a re • DB Processors in Storage
rdw
H a • Scale-Out Storage
a rt • Scale-Out Servers
S m

Copyright © 2016. Oracle and/or its affiliates. All rights reserved. 34

Copyright © 2016. Oracle and/or its affiliates. All rights reserved. 35