MartyCzekalski - SCSI Express - Fast - Reliable - Flash - Storage-Final PDF

SCSI Express – Fast & Reliable Flash Storage
for the Enterprise

Marty Czekalski – President, SCSI Trade Association,
Interface Architecture Initiatives Manager, Seagate
Mike James – Board Member, SCSI Trade Association,

Director, Engineering, SanDisk Corporation
1
SNIA Legal Notice
The material contained in this tutorial is copyrighted by the SNIA unless otherwise
noted.
Member companies and individual members may use this material in presentations
and literature under the following conditions:
Any slide or slides used must be reproduced in their entirety without
modification
The SNIA must be acknowledged as the source of any material used in the
body of any document containing material from these presentations.
This presentation is a project of the SNIA Education Committee.
Neither the author nor the presenter is an attorney and nothing in this
presentation is intended to be, or should be construed as legal advice or an opinion
of counsel. If you need legal advice or a legal opinion please contact your attorney.
The information presented herein represents the author's personal opinion and
current understanding of the relevant issues involved. The author, the presenter,
and the SNIA do not assume any responsibility or liability for damages arising out of
any reliance on or use of this information.
NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK.
SCSI Express Fast and Reliable Flash Storage for the Enterprise
© 2012 Storage Networking Industry Association. All Rights Reserved.
2
Abstract
SCSI Express
Recently announced, SCSI Express represents the natural
evolution of enterprise storage technology building upon
decades of customer and industry experience.
SCSI Express is based on two of the highest volume and widely

deployed, interoperable technologies in the world – SCSI and
PCI Express. These two technologies enable unprecedented
performance gains while maintaining enterprise experience.
This presentation contains an in-depth overview of SCSI

Express including what it is, where it will be developed, why it is
important to the enterprise computing platform, how it is
implemented, and the current status of the timeline.
3
SCSI Logical Abstraction Layer:
A Foundation for Innovation
 Preserves Hardened SCSI Command Set
• Successive Product Generations
• Accommodates Frequent Technology Shifts
• Multiple Vendors
• Multiple Interconnects
 Reduces Time to Market and Integration Costs
 Delivers Enterprise Attributes and Features
• End-to-End Data Protection
• Atomic Writes
• Hinting
• Task Management
• Power Management
• And more on the way
4
SCSI Logical Abstraction Layer:
A foundation for Innovation
 Operates Over Numerous Transport

Layers
• ATAPI (ATA, SATA) • iSCSI
• USB • FC
• Memory sticks • Parallel SCSI
• Firewire • SAS
• Infiniband
 And now, SCSI Over PCIe (SOP, PQI)
SCSI:
The Most Widely Implemented Logical Storage Protocol
5
SCSI Express Overview
What is SCSI Express?

 Proven SCSI protocol combined with PCIe creating an
industry standard path to PCIe-based storage
Why do we need SCSI Express?

 Deliver proven enterprise storage for PCIe based
storage devices in a standardized ecosystem
 Take advantage of lower latency PCIe to improve
performance
 Unified management and programming interface
6
SCSI Express Value Proposition
Performance and Investment

Reliability Protection
Innovation
 Increased performance  Proven enterprise  Coexistence with
through lower latency SCSI ecosystem SAS via Express Bay
for emerging advanced and common
technologies  Architected for command set
nonstop availability
 Enables new storage  Leveraging robust
architectures middleware
ecosystem
7
SCSI Express Components
SCSI
The SCSI storage command set

SCSI Over
PCI Express
PCIe (SOP) Packages SCSI for a PQI queuing layer
Flexible, high-performance queuing

layer
PCIe Queuing Accommodates PCIe, SAS, and SATA

SFF-8639
Connector
Interface drives
(PQI)
Leading server I/O interconnect
Express Bay
Express Bay
 Up to 25 Watts
 SFF-8639 connector
 PCI-SIG electrical specification
Objectives
 Preserve the enterprise storage experience for PCI Express storage
 Meet SSD performance demands
 Serviceable, hot-pluggable Express Bay opens up new possibilities…
9
Connector Compatibility
MultiLink SAS SFF-8630 43 Pins

SATA
SATA 22 Pins
SATA SAS
SAS MultiLink2
Multifunction SFF-8639 68 Pins

SATA
SAS SFF-8680 29 Pins
SATA
SAS SAS
SAS MultiLink1 SAS MultiLink3
SCSI Express
1 Max two links operate
2 Four links operational
3 Two or four links operation depending on host provisioning
10
SCSI Express Hardware/Software
SCSI SCSI SCSI

Express Controllers Express Drive/Device Express Driver
 Supports SOP-PQI  SOP-PQI protocol  Driver supplied by
driver functionality on storage OEMs, IHVs
the controller to the  Connects to SFF- or OSVs
target device on 8639  Open Source Linux
the PCIe lanes driver and IHV
drivers available
 PCIe up to x4
 Typically supports interface
beginning in
SAS/SATA devices Q1CY13
Architecture
Key excerpts from SCSI, SOP, and PQI architecture models
SCSI initiator device: a server with a PCI Express Root Port
SCSI target device: an SSD, HDD, HBA, or RAID controller
SOP target device (SCSI target

device)
SOP initiator device (SCSI initiator
device) SOP domain Logical
(SCSI domain) unit
Application Task manager
client
Device server
SOP initiator port SOP target port

(SCSI initiator port) (SCSI target port)
PQI device PQI device
PCI
Express
PCI function PCI Express port PCI Express port PCI function
SOP service delivery subsystem

(service delivery subsystem)
12
Simple devices
SSDs, etc.
• Usually just a single logical unit with LUN 0
• Any SCSI device type is possible

• SSD, tape drive, optical drive (CD/DVD/BluRay), etc.
SOP target device
SOP PCI
Express SOP Logical unit
initiator target
device port medium
13
Bridges
Interconnect SCSI transport
HBAs protocol
• Bridges from PCI Express to another Serial Attached SCSI Serial SCSI Protocol
(SAS) (SSP)
interconnect supporting SCSI
• Maps SCSI target devices one-for-one Fibre Channel (FC) Fibre Channel
• Typical terms: host bus adapter (HBA), host Protocol (FCP)
controller, host adapter, network interface
Ethernet Internet SCSI (iSCSI)
controller, converged network adapter
• Usually referred to only by the back-end Universal Serial Bus USB Attached SCSI
interconnect (USB) (UAS)
• e.g. “SAS HBA” InfiniBand SCSI RDMA Protocol
• Manage with SOP bridge management (SRP)
functions PCI Express SCSI over PCI Express
(SOP)
SOP target device
SAS target device
PCI SAS SAS
SOP Express SOP SAS SAS target device
initiator SAS
target Bridge initiator
device expander SAS target device
port port
SAS target device
14
RAID controllers
Less complex than bridges from an SOP perspective

• Indirectly bridges from PCI Express to another interconnect supporting SCSI
• Not a one-to-one mapping of SCSI target devices
• Presents logical drives over PCI Express
• Created from physical drives
• Manage with standard SCSI commands
• REPORT LUNS reports the logical units that have been created
• Bridge management not involved (unless it’s a hybrid HBA + RAID
controller
SOP target device

SAS target device
PCI SAS SAS
SOP Express SOP SAS SAS SAS target device
initiator target Logical unit
device initiator expander
port SAS target device
Logical unit port
SAS target device
15
Queuing layer services
SOP expects a queuing layer over PCI Express to

define
inbound queues
transfer IUs from SOP initiator port to SOP target port
outbound queues
transfer IUs from SOP target port to SOP initiator port
SOP architected to support multiple queuing layers
PCI Express Queuing Interface (PQI)
Information Units (IUs)
Messages between a driver or a SCSI Express controller
and a SCSI Express device
16
Information Units (IUs) Types
SCSI Request/Response IUs

Commands, Task Management, Success, Command
Response, Task Management Response, etc.
General Management Request/Response IUs
Report General, Report Configuration, Set Configuration,
Report Event Configuration, Management Response,
Event, Event Acknowledge
Bridge Management Request/Response IUs
Administrator Request/Response IUs
Other
Null IU, etc.
17
IUs and Queues
Smaller, equal, or larger than queue element

• IUs are sent in operational IQs or operational OQs
element
• Each IU starts at the beginning of an element
• No more than one IU in an element element
• Any bytes after the IU are ignored by the recipient
up until the end of the element element
• e.g., OQ element size 64 bytes, with a 16 byte

element
SUCCESS IU IU shorter than element
• IU may span multiple elements element
• e.g., OQ element size 16 bytes, with a 64 byte
COMMAND RESPONSE IU spanning four element
IU spanning
elements
elements
element
• The IU header is only in the first element, not
repeated in each element
IU matching element
18
Circular Queue Basics
Circular queue basics

• Element array
• Fixed size elements (e.g., 64 bytes)
• Producer index (PI)
• Location to which producer writes
elements
• Write to element array[PI++]
• Wrap at size of the element array
• Consumer index (CI)
• Location from which consumer
reads elements
• Read from element array[CI++]
• Wrap at size of the element array
19
Empty and Full Circular Queues
One element is always unused

• Empty
• PI equals CI
• e.g., PI=17 and CI=17
• Entries to consume
• e.g., PI=18 and CI=17
• Full
• PI is one behind CI
• e.g., PI=17 and CI=18
• e.g., PI=62 and CI=63
• e.g., PI=63 and CI=0 (if
element array has 64
elements)
• e.g., PI=0 and CI=1
• One queue element is always unused
• e.g., maximum 63 entries in a
queue of 64
20
Inbound Queues (IQs) and Outbound
Queues (OQs)
Named from the PQI device’s perspective

• Inbound queues (IQs) • Outbound queues (OQs)
• PQI host to PQI device • PQI device to PQI host
• Administrator request IUs • Administrator response IUs
• SCSI request IUs (in SOP) • SCSI response IUs (in SOP)
IQ
PQI host PCI Express PQI device
OQ
21
IQ and OQ Object Locations
Avoids PCI Express memory reads of PIs and CIs

• IQ object locations • OQ object locations
• IQ element array in host memory (typical) • OQ element array in host memory (typical)
• IQ PI in PQI device memory space (always) • OQ PI in host memory (typical)
• IQ CI in host memory (typical) • OQ CI in PQI device memory space (always)
IQ element array MemRd
IQ CI MemWr
PQI device
MemWr IQ PI
PQI host MemWr OQ
CI
OQ MemWr
PI
OQ element array MemRd
MemWr
Interrupt receiver
22
Queue Types
Administrator queues and operational queues

• Administrator queues • Operational queues
• Created via PQI device registers • Created via PQI administrator
• Located in PQI device functions
memory space • Delivered over the
• Single administrator IQ and administrator queues
administrator OQ • Any number of operational IQs and
• i.e., one administrator queue operational OQs
pair • Not in pairs
• IUs defined by PQI • Can be specific to different
cores
• IUs defined by the information unit
layer standard
• e.g., SCSI over PCI Express
(SOP)
23
Key PQI Features
Interrupt Coalescing
Single interrupt for multiple queue entries
Tuning via; count, min and max times
Scatter Gather Lists (SGL)
Describes a data buffer
How it is distributed across non contiguous chunks of memory
Can be embedded in IUs or a separate list
Widely supported method across multiple OSs
24
SCSI – Looking to the Future
• SCSI Express
• 12Gb/s SAS, Multilink, 24Gb/s SAS
• Performance and scalability
• Power Limit Control - up to 25W devices
• Both SAS and SCSI Express
• Extended Copy Feature
• Atomic Writes
• Hinting & other NVM features
25
Extended Copy - Connecting the Tiers
Data Caching/Migration
Application
Token
Offload Return Offload Return

Write Result Read Token
DATA Movement
PCIe Fabric
SOP/PQI RAID SOP/PQI Bridge (HBA)
SCSI Express SSD
Storage
Network
26
Atomic Writes
All or nothing written capability across multiple

commands
For single commands and across non contiguous LBA
ranges
Benefits:
Simplifies resilient system designs
Database, file system, etc.
Improves system performance in these applications
27
Hinting & Other NVM Features
Pass “hints” to devices to make operations more efficient

and increase performance
Targeted at SSDs and hybrid drives, but also useful for HDDs
Device modifies how data is stored based on type
Direct attached devices don’t need to continually OPEN

and CLOSE connections
Can be implemented within the existing standard
Reduces latency on both SSDs and HDDs
NVM features and programming interfaces

Leverages ongoing work in SNIA and T10
28
External Timeline
Approved timeline – updated as of May 2012
CY2012 CY2013 CY2014

1H’12 2H’12 1H’13 2H’13 1H’14 2H’14
SCSI Express
devices/
Today controllers
available
Plugfest #2
1H 2014
Plugfest #1
2H 2013
SCSI Express
Based drivers
1H 2013
SCSI Express Samples

1H 2013
SOP/PQI Letter Ballot

SPEC Stability
2H 2012
Express bay available

2H 2012
SOP proposal complete

2H 2012
© 2012 Storage Networking Industry Association. All Rights Reserved. 29
SCSI Express Summary
Proven SCSI protocol combined with PCIe creating an

industry standard path to PCIe-based storage
Enterprise storage for PCIe based storage devices

Increased performance through lower latency
Coexistence with SAS via Express Bay and common command
set
Unified management and programming interface
30
Attribution & Feedback
The SNIA Education Committee would like to thank the
following individuals for their contributions to this Tutorial.
Authorship History Additional Contributors
8/28/2012 Marty Czekalski David Allen
8/24/2012 Mike James Rob Elliott
Ie-Wei Njoo
Updates: Bret Gibbs
Please send any questions or comments regarding this SNIA Tutorial to

[email protected]
31

MartyCzekalski - SCSI Express - Fast - Reliable - Flash - Storage-Final PDF

Uploaded by

Copyright:

Available Formats

MartyCzekalski - SCSI Express - Fast - Reliable - Flash - Storage-Final PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

MartyCzekalski - SCSI Express - Fast - Reliable - Flash - Storage-Final PDF

Uploaded by

Copyright:

Available Formats

SCSI Express – Fast & Reliable Flash Storage

for the Enterprise

Mike James – Board Member, SCSI Trade Association,

SCSI Express is based on two of the highest volume and widely

This presentation contains an in-depth overview of SCSI

 Operates Over Numerous Transport

• Memory sticks • Parallel SCSI

 And now, SCSI Over PCIe (SOP, PQI)

What is SCSI Express?

Why do we need SCSI Express?

Performance and Investment

The SCSI storage command set

Flexible, high-performance queuing

PCIe Queuing Accommodates PCIe, SAS, and SATA

Leading server I/O interconnect

MultiLink SAS SFF-8630 43 Pins

Multifunction SFF-8639 68 Pins

SAS MultiLink1 SAS MultiLink3

SCSI SCSI SCSI

SOP target device (SCSI target

SOP initiator port SOP target port

SOP service delivery subsystem

• Usually just a single logical unit with LUN 0

• Any SCSI device type is possible

SOP target device

Less complex than bridges from an SOP perspective

SOP target device

SOP expects a queuing layer over PCI Express to

SCSI Request/Response IUs

Smaller, equal, or larger than queue element

• e.g., OQ element size 64 bytes, with a 16 byte

Circular queue basics

One element is always unused

Named from the PQI device’s perspective

PQI host PCI Express PQI device

Avoids PCI Express memory reads of PIs and CIs

IQ element array MemRd

Administrator queues and operational queues

Offload Return Offload Return

SOP/PQI RAID SOP/PQI Bridge (HBA)

SCSI Express SSD

All or nothing written capability across multiple

Pass “hints” to devices to make operations more efficient

Direct attached devices don’t need to continually OPEN

NVM features and programming interfaces

CY2012 CY2013 CY2014

SCSI Express Samples

SOP/PQI Letter Ballot

Express bay available

SOP proposal complete

Proven SCSI protocol combined with PCIe creating an

Enterprise storage for PCIe based storage devices

Coexistence with SAS via Express Bay and common command

Please send any questions or comments regarding this SNIA Tutorial to

You might also like