IBM XIV Gen3 Storage System: Performance and Ease of Use

IBM System Storage
2012 IBM Corporation 2012 IBM Corporation 1

IBM XIV Gen3 Storage System:
Performance and Ease of Use
XIV Gen3 SSD Performance
XIV Gen3 SSD Performance
2012 IBM Corporation
2
Redpaper
http://www.redbooks.ibm.com/abstracts/redp4842.html?Open
3
Reliable
Reliable Reliable
Affordable
Affordable Affordable
XIV SSD Caching: Design Goals
Allow use of SSDs
without increasing
IT complexity
Allow customers to
focus on capacity
and service
management, not
disk technologies
Zero management
Invisible
Invisible Invisible
At least 99.999%
availability, beyond
the reliably of
todays off-the-shelf
SSD technology
Patents
Wide SSD
deployment is
currently limited by
cost
System software
handles SSD wear-
leveling and data
placement, allowing
greater drive choice
flexibility, and
keeping prices low
3
4
SSD as a Cache
SSD as a Cache SSD as a Cache
SSD as a Tier
SSD as a Tier SSD as a Tier
Approaches for Using SSDs in a Storage System
4
Multiple independent disk tiers provide
different levels of service
Data can be relocated between tiers
- Manually
- Automatically, policy driven
A memory cache serves all tiers
SSDs do not change how XIV handles
data
SSDs used as a secondary cache
between the disks and the DRAM cache
Disk access is avoided when host read
resides in the SSD cache
SSD
FC
SATA
D
R
A
M

C
a
c
h
e
D
R
A
M

C
a
c
h
e
Disks
S
S
D

C
a
c
h
e
5
Advantages of Using SSDs as Cache
5
Each module has SSD drive sized for optimum
performance
Data movement is inherent to the architecture
and happens naturally
No separate software to manage - no decisions
Layout
RAID
Tiering policy
How much SSD
Benefits start immediately
No tuning
Data on SSD is for read hits only - no need to
protect - XIV uses the whole SSD
What is between SSD & DRAM? = PCIe bus
.vs. RAID data protection
.vs. RAID layer
No need to relocate data in SSD
If it becomes irrelevant, it is simply dropped
SSD as a Cache
SSD as a Cache SSD as a Cache
D
R
A
M

C
a
c
h
e
Disks
S
S
D

C
a
c
h
e
6
XIV SSD Cache: Best Fit
6
Best IO Profile: Highly Random Reads Small Block
>80% Reads, 8KB Block Sizes and Low Latency requirements
SAP, ERP, OLTP, Cache Data Bases
Where and When to use SSDs?
Where and When to use Where and When to use SSDs SSDs? ?
7
XIV System Components
Host
Host
Host
Host
FC
iSCSI
Host
Host
UPS UPS UPS
Data Module
Data Module
Data Module
Data Module
Inter-module
Connectivity
Full Rack up to 243TB
15 modules
180 1/2/3 TB SAS
disks
360 GB memory
24 8GB FC ports
22 iSCSI ports
6TB SSD cache
8
XIV SSD Implemented
High capacity SSDs used as secondary cache
400GB SSD device can be added to each
module
6TB of cache per rack
Housed in PCI caddy in rear of module
Scales with the system 6 to 15 SSD drives
Must be added to all modules together
No tuning necessary
Performance gain for all applications as soon
as the devices are phased-in
For service-level control, select volumes to
exclude/include
Available to all Gen3 systems
Order with SSD from factory
Field non-disruptive upgrade
9
Management Simplicity
10
10
XIV SSD is for Random Reads
Sequential read performance is already great
Aggressive prefetching into huge DRAM cache
Sequential reads from cache
Random read performance good
Huge DRAM cache can satisfy many
random read requests
Read hit
If data is not already in DRAM
Read miss
Data retrieved from SAS drives
XIV SSDs significantly improve random read hits
SSDs hold a huge amount of small block random
data
Result is great random read hit performance
SSDs are an independent read cache
Just caches data of corresponding module drives
Just caches primary copy of the data (XIV
redundancy scheme)
XIV I/O summary
Random and sequential writes go to cache
Sequential reads are from cache
With SSD, significantly more random reads are from
cache
11
XIV SSD Maintenance
SSD maintenance hot swap
Module with failed SSD
Continues to service read hits and sequential
reads
Small block reads are redirected to secondary
Where is secondary copy of small-block read
data?
Distributes SSD caching duties
Can be phased out, tested, replaced and
phased in
Only contains read data
XIV redundant data scheme unchanged
We do not protect this data in any way - no
need
11
RAM
SSD as
CACHE
Non-SSD
Drives
System Module
12
Random Reads
12
Read data 64 KB
Read Miss Operation (Host requests
random data - its not in DRAM):
Checks hash table for hit on SSD
extended cache
If data is in SSD, data is moved from
SSD to DRAM
Else forwards the read request
unmodified to disk drives
Disk drives send data to DRAM
Data copied onto 512KB buffers page
asynchronously
When buffer fills up, it is destaged to the
SSD
SSD Extended Cache
SSD Extended Cache
If hit, Serves
Data (4k read
hit
granularity)
Check SSD
extended
cache for hit
- hash table
Main Cache (DRAM)
Main Cache (DRAM)
.
.
.
Yes No
Serves data to
host and copies
pages to buffers
for log
structured write
Serves data to
host and copies
pages to buffers
for log
structured write
.
.
512KB Buffers
512KB Buffers
.
SAS (SG/SX) driver
SAS (SG/SX) driver
Sequential I/O detection bypasses SSD
Sequential pre-fetch goes direct to disk
Sequential pre-fetch is fast
No need to put sequential data in SSD
Large block (>64KB) reads - probably sequential anyway
13
Disk Writes
13
Write Operation:
All writes go to DRAM (main cache)
If SSD has a copy of the data being
modified, then the data in SSD is old and
stale and a corresponding hash table entry
on SSD is invalidated to prevent subsequent
hits from the extended cache
Writes are destaged from DRAM to disk
normally and asynchronously
As writes are de-staged from Main Cache
to disk they are asynchronously staged on
to the SSD buffer (when 64KB)
Once buffer is full, data is written to SSD,
hash table is updated
Main Cache
(DRAM)
SSD hash table -
invalidate data
Main Cache
(DRAM)
SSD hash table -
invalidate data
Extended Cache
Extended Cache
SAS (SG/SX) driver
SAS (SG/SX) driver
.
.
512KB Buffers
512KB Buffers
.
Incoming write
Mirror write and send
Ack to host
.
.
14
14
SSD Map - Hash Table
Checks for read hits in extended SSD
cache
If DRAM read miss
Then if read request is random (and
less than 64 KB), then
SSD hash table is checked to see if the
data is in SSD
Hash table is a map to what data is in
SSD
Hash table lives in DRAM (with a copy on
SSD)
Less than 0.3% of the size of the SSD
Persistent SSD cache map during module
reboots (upgrades, maintenance, graceful
shutdown)
We want to retain SSD data during hot
and cold boots so we dont have to re-
populate SSDs
Resident on SSD during shutdown,
validated during shutdown
On boot, repopulated into DRAM after
validation
Hash table is designed to support
concurrent access and supports parallel
access for optimal performance
15
15
SSD Longevity - Structured Writes
Patented algorithms designed for SSD long life
Wear leveling algorithm - even wear on SSD disk blocks
Erasures and writes are distributed evenly
512 KB writes fill SSD from beginning to end sequentially
Log structured writes
Not constantly burning data on same bits
Algorithms further ensure erasures and re-writes are distributed evenly across the medium
Value
What wears out an SSD? Writing to the same block.
Allows flexibility in SSD vendor, technology and size
SSD technology and size can (and will) change easily
512 KB
512 KB
512 KB
512 KB
Log structured write
512KB Buffers
in DRAM
512 KB
512 KB
512KB Buffers
in DRAM
16
Core ERP (IOPS)
CRM and Financial DB Workload
70/30/8k
20 TB DB
Medical Record App Server (RT)
Healthcare EMR Workload
100% random IO
6TB DB
Read latency @ 20K IOPS:
DB2 Brokerage (IOPS)
Securities trading OLTP Workload
Mixed block IO
12TB DB
Websphere Datastore (IOPS)
Web 2.0 OLTP Workload
80/20/4k
20TB DB
Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations
such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will
achieve throughput improvements equivalent to the performance ratios stated here.
Outstanding Applications Performance with SSD Caching
17
Server 1 (800 threads) ~60k 8k OLTP IOPS without SSD, Server 2 At 21:08:00 starts 800 additional OLTP threads
SSD are phased in online at 21:14:00, SSD continues to drop response time as SSD warms. @120k RT is less than 13ms
2x
performance
SSD
Phased in
OLTP Performance (SAP like)
70/30/50 8KB Fixed Random IO (No Compression)
18
XIV SSD Performance over time
SSD 6TB Workload, 100% Random Read Miss, 4KB Block Size
100% Random Read Workload with SSD is warmed up
Almost 150,000
Random IOPS
under ~6ms
6x
performance
19
Database Performance and XIV Gen3 with SSD
Consider this a heat map of all I/O access within
the XIV frame. Within the inner box are XIV cache
hits for reads satisfied by the main cache that can
be up to 360GB. By adding up to 6TB of SSD
memory, which acts as an extension of XIV cache
rather than a storage device, the number of I/Os
satisfied by cache hits is significantly increased as
characterized by the outer box.
The number of I/Os satisfied by cache hits or
SSD hits goes up significantly. The average
response time for those I/Os is less than 1ms.
If your application uses indexes the way it was
designed to do, it will exhibit less table scans and
more index reads. Less sequential I/O and more
random I/O. The random I/O profile is key to SSD
success. SSDs do not help sequential reads.
Those will be satisfied by the XIV main cache.
20
Monitoring via XIV GUI/CLI supporting SSD statistics reporting
The IBM XIV Management GUI/CLI can be downloaded from
http://www-03.ibm.com/systems/storage/disk/xiv/resources.html#demos
or
All of the XIV related downloads:
http://www-
933.ibm.com/support/fixcentral/swg/selectFixes?parent=ibm~Storage_Disk&prod
uct=ibm/Storage_Disk/XIV+Storage+System+%282810,+2812%29&release=All&pla
tform=All&function=all#Management%20Tools
21
21
XIV GUI Performance reporting showing Mem and SSD Read Hit
22
XIV GUI Performance reporting showing Mem and SSD Read Hit
SPC-1 Like OLTP Workload
23
XIV CLI for gathering Performance statistics
xcli -u admin -p password -m 174.226.108.123 -s statistics_get start=2012-11-05.00:00:00 count=720 interval=2
resolution_unit=minute > DABX5265_110512.csv
resolution_unit=minute > DABX5265_110612.csv
resolution_unit=minute > L5BX5045_110512.csv
resolution_unit=minute > L5BX5045_110612.csv
24
Use the CLI Viewer to format the data
25
Example of Hit Rates from XIV CLI data. VDI
26
Example before and after SSD performance. Oracle
Data from XIV statistics_get at 1 minute samples
27
28
29
Will a workload benefit from SSD? XIV Gen 3 with no SSD
30
Will a workload benefit from SSD? 15ms Latency, 40% Read Hit
31
Will a workload benefit from SSD?
32
33
34
35
XIV Block Trace Facility. I/O Analytics.
IO Analytics is focused on the collection of IO traces from XIV storage systems.
There are four separate pieces to this: trace collection, replay, visualization, and
prediction.
36
XIV Block Trace Facility. I/O Analytics.
IBM System Storage
2012 IBM Corporation 2012 IBM Corporation 37
Thank You
38
Disclaimer
Copyright 2012 by International Business Machines Corporation.
This publication is provided AS IS. IBM product information is subject to change without notice.
No part of this document may be reproduced or transmitted in any form without written permission from IBM Corporation.
Product data has been reviewed for accuracy as of the date of initial publication. Product data is subject to change without
notice. This information could include technical inaccuracies or typographical errors. IBM may make improvements and/or
changes in the product(s) and/or program(s) at any time without notice. The information provided in this document is
distributed AS IS without any warranty, either express or implied. IBM EXPRESSLY DISCLAIMS any warranties of
merchantability, fitness for a particular purpose OR INFRINGEMENT. IBM shall have no responsibility to update this
information. IBM products are warranted according to the terms and conditions of the agreements (e.g., IBM Customer
Agreement, Statement of Limited Warranty, International Program License Agreement, etc.) under which they are provided.
Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or
other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the
accuracy of performance, compatibility or any other claims related to non-IBM products.
IBM makes no representations or warranties, expressed or implied, regarding non-IBM products and services, including those
designated as ServerProven.
IBM may not offer the products, services or features discussed in this document in other countries, and the information may be
subject to change without notice. Consult your local IBM business contact for information on the product or services available in
your area.
All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals
and objectives only. Contact your local IBM office or IBM authorized reseller for the full text of the specific Statement of
Direction.

IBM XIV Gen3 Storage System: Performance and Ease of Use

Uploaded by

Copyright:

Available Formats

IBM XIV Gen3 Storage System: Performance and Ease of Use

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

IBM XIV Gen3 Storage System: Performance and Ease of Use

Uploaded by

Copyright:

Available Formats

IBM System Storage

2012 IBM Corporation 2012 IBM Corporation 1

You might also like