SAN Technology and Applications

Security Level:
Contents

1 SAN Storage Overview

2 Components and Connectivity of SAN

3 Network Protocol of SAN

4 Application Scenarios of SAN

DAS

DAS (Direct Attached Storage)

DAS is dedicated digital storage attached directly to a server or DAS is used to expand the capacity of a single server.
PC via a cable, as opposed to storage accessed over a computer
network. A typical DAS system is made of a data storage Typical scenarios:
device (for example, JBOD) connected directly to a computer Storing cold data with low cost, for example, backup,
through a host bus adapter (HBA). Between those two points archiving, and video surveillance.
there is no network device (like hub, switch, or router). DAS
provides block-level services to servers.
DAS Types and Evolution
Internal DAS External DAS Intelligent external DAS

Server Server Server

CPU CPU
CPU
RAM RAM
RAM

Disk Disk
Disk RAID
function HBA card HBA card

Controller
RAID function
JBOD (Just a Bunch of Disks)
• Disks are inside the server. • Disks are placed in an • Add controller chips to offload
• A limited number of disks external enclosure. the RAID function on DAS.
deliver small capacity. • More but still a limited • Provide simple management
number of disks functions.
• Just provide disk capacity. The • Another name is DAS-RAID
RAID function is still provided storage.
by servers.
Advantages and Disadvantages of DAS
Advantages
 Simplicity
DAS is connected to servers using cables for expansion. There is no need for complex configuration and it is easy to use.
 Low Cost
The price is more affordable than enterprise SAN or NAS.

Disadvantages
Following the data explosion and the new requirements in modern data centers, DAS has shown a lot of
disadvantages:
 Hard to Scale Requirements:
DAS can be only used by one server (limited ports without network device, limited connected servers, and limited connection
High scalability
distance).
 Inefficient Utilization High utilization
DAS can be only accessed by a few servers and cannot share capacity or performance, resulting in data islands. High reliability
 Low Reliability
Better performance
Lack of DR features as professional enterprise storage
 Poor Performance Easy to manage
Limited ports and bandwidth, consuming the computing capabilities of servers …
 High Management Costs
Managed one by one, cannot be managed in a centralized manner
Storage Evolution
DAS SAN/NAS
Server Server Server
Data Island Data Island Data Island
• Improve scalability by network
Server Server Server devices. … …
• Can be shared between many
servers.
… … • Add rich disaster recovery
FC/IP
features.
• Achieve better performance Switch Switch
• …

JBOD JBOD JBOD

Storage
Emerged in 1970s Emerged in 1990s
Data is scattered and not shared. Data is centralized and shared.
SAN – Overview

SAN (Storage Area Network) 1. A network whose primary purpose is the

transfer of data between computer systems and
storage devices and among storage devices.
LAN App Server File Server Database Server
A SAN consists of a communication
infrastructure, which provides physical
connections, and a management layer, which
organizes the connections, storage devices, and
computer systems so that data transfer is secure
Data Flow and robust. The term SAN is usually (but not
necessarily) identified with block I/O services
FC/IP SAN rather than file access services.

Data Flow 2. A storage system consisting of storage

elements, storage devices, computer systems,
and/or appliances, plus all control software,
communicating over a network.

From: SNIA
Storage Device Storage Device Storage Device
Advantages of SAN Compared with DAS

 Excellent Scalability
Support containing plenty of disks and connecting huge number of servers, support scale-out controllers and scale-up disks to increase
performance and capacity linearly on demand, support long-distance connections.
 Efficient Utilization
Resources can be shared by all servers, eliminating data islands.
 High Reliability
Rich DR and backup features for protecting reliability, such as replication, snapshot, and E2E DIF
 High Performance
High-speed and high-bandwidth network ports, offloading RAID calculation on servers
 Easy to Manage
Provide centralized management and monitoring tools, reducing OPEX.
SAN Types

By Front-End Protocol By Architecture

InfiniBand FCoE Centralized SAN Distributed SAN

NVMe over Fabric

IP FC
Comparison Among Different Protocol SANs
Type FC SAN IP SAN FCoE SAN IB SAN NVMe over Fabric SAN

NVMe over Fabric

Network Fiber Channel Over
Fiber Channel iSCSI InfiniBand using RDMA
Protocol Ethernet
NVMe over Fabric using FC
4 Gbit/s, 8 Gbit/s, 1 Gbit/s, 10 Gbit/s, 25 Gbit/s, 40 Gbit/s, 100
10 Gbit/s, 25 Gbit/s, 40Gbit/s, 56
16 Gbit/s/ 25 Gbit/s, Gbit/s
Bandwidth 40 Gbit/s, or Gbit/s, 100 Gbit/s,
32 Gbit/s, or 128 40 Gbit/s, or 100 using RDMA, or
100Gb or 200 Gbit/s
Gbit/s Gbit/s 32 Gbit/s using FC
Higher than iSCSI,
Speed High Low High Highest
lower than FC
Higher than iSCSI,
Cost High Low High Highest
lower than FC
Distance RDMA for long distance
Short distance Long distance Short distance Short distance
supported FC for short distance
From about 2009,
From 1980s, the From about 2001, allowing the From about 2005, From 2017 to 2019, new
most common the benefit is the transfer of Fiber used in the technologies provide best
Others
type, around 75% lower cost, around Channel frames scenario requiring performance combining
market share 20% market share over Ethernet network high bandwidth with SSD and SCM.
s
Comparison Between Centralized SAN and Distributed SAN
Item Centralized SAN Distributed SAN

Hardware Dedicated: Common:

architecture Uses dedicated and different hardware. Supports common hardware architecture.

Software Closed source: Open source:

Uses closed-source software, which is updated by vendors. Supports open source software, which can be modified
or developed by users.

Network FC/iSCSI/NVMe/IB iSCSI

Scalability Limited scalability: Better scalability:

Supports scale-up and scale-out, but limited, support up to 16/32 Better scalability, support up to thousands of nodes and
controllers and thousands of disks, not elastic to expand or shrink EB-level capacity, elastic and easy to linearly expand or
on demand. shrink on demand

Reliability Better reliability: Evolving:

Mature architecture, dedicated end-to-end design for reliability Evolving architecture, the reliability of common X86
and rich DR features servers is not as good as traditional enterprise storage,
and DR features are also not strong enough

Performance Limited performance but lower latency: Unlimited performance but higher latency:
Limited maximum performance because of the limit in controller Unlimited peak performance and high bandwidth, but
quantity, but lower latency higher latency because of the network interconnection
between distributed clusters

Applicable Mission-critical applications, which need high reliability and low Cloud-based applications, which need elasticity, high
scenario latency, such as databases bandwidth, such as hosting business in ISP
Comparisons Among DAS, SAN, and NAS
DAS SAN NAS
App Server App Server App Server

File System File System NFS/CIFS

FC/IP Switch IP Switch

SCSI, FC

JBOD
RAID RAID RAID RAID

Protocol SCSI/FC/ATA FC/iSCSI TCP/IP

Transmission Object Block Block File
Typical Scenario JBOD for server Databases, VSI File share
Contents

1 SAN Storage Overview

2 Components and Connectivity of SAN

3 Network Protocol of SAN

4 Application Scenarios of SAN

SAN Components
I/O Stack
Application
Windows Server Linux Server AIX Server
File System
Volume Manager
Host … … Multipathing
Layer Software
HBA HBA HBA
Hardware Driver
HBA

Cables Cables
FC/IP

Fabric Cable
SAN Switch SAN Switch
Layer Switch

Front-end Front-end
interface interface

Front-end
Storage Storage Storage Storage interface
Controller
Layer Array Array Storage Cache
SAS/IP SAS/IP Disk
Disk
Enclosure
SAN Connectivity – Switch
Switch is the core component of a SAN, connecting servers and storage devices.
It allows many-to-many automatically connecting senders and receivers, device name lookup, security,
and redundancy.
SAN Switch Design Considerations:
What Type? FC or iSCSI
How many ports? 48 or 96
How about the bandwidth?

IP Switch 8Gb or 16Gb FC, 10GE or 25GE IP

FC Switch Zoning or VLAN?
Cascading or Mesh?
IB Switch

Broadcom

Mellanox CISCO
SAN Connectivity – HBA
Host bus adapter (HBA) is a circuit board and/or integrated circuit adapter which can:
• Provide a physical connection between a host server and compatible devices/network.
• Perform Input/Output (I/O) processing.
• Transfer data between the connected devices and the host computer.
• Free up resources of the host computer by conducting the data storage and retrieval operations on its
own

FC HBA iSCSI HBA IB HBA SFP

(small form-factor pluggable
transceiver)
SAN Connectivity – Cable

Optical Fiber Ethernet Cable (RJ45) AOC/DAC

Transmission mode: Category: AOC (active optical cable):

SMF (single model fiber): Cat 5/5e/6 for 1Gbps an optical module and an optical fiber
used in long distance < 10 km Cat 6e/7/7e for 10Gbps DAC (direct attached cable):
MMF (multi model fiber): a copper cable
used in short distance < 500 m Copper cable
Distance < 100 m Speed: 10GE/25GE/40GE/100GE
Interface type:
LC (for GE,10GE,8G/16G/32G FC)
MPO-PMO, MPO-4*LC (for
40GE/100GE)
Contents

1 SAN Storage Overview

2 Components and Connectivity of SAN

3 Network Protocol of SAN

4 Application Scenarios of SAN

SCSI Is the Basis of ALL
What is SCSI?
Small Computer System Interface (SCSI), is a set of standards for physically connecting and transferring data
between computers and peripheral devices. The SCSI standards define commands, protocols, electrical, optical and
logical interfaces. SCSI is most commonly used for hard disk drives and tape drives, but it can connect a wide
range of other devices, including scanners and CD drives, although not all controllers can handle all devices.

SCSI is better and faster than IDE

Item SCSI IDE
Scenario Interface between computers and disks Interface between personal computers and disks
Performance 320 to 4000 Mbit/s 20 to 40 Mbit/s
Expansion Thousands of disks 2 disks
Disk Type 1000 or 1500 rpm, and even SSD 5400 or 7200 rpm
SCSI History
SCSI-1
Published at 1983-1985
SCSI-2


 Regular SCSI
 7-bit data bus  Approved at 1988-1994
 7 possible devices  Inherited from SCSI-1
 Common command set
SCSI-1  16-bit data bus
 15 possible devices

SCSI-3
Approved at 1993
SCSI-2 

 Compatible with SCSI-2/SCSI-1

 Support both parallel and serial
data transmission
SCSI-3  Support many transport
protocols such as FC, iSCSI …
SCSI Model – Initiator and Target
Initiator Target

SCSI Command SCSI Command

(Block…) (Block…)

Protocol Protocol
(FC, iSCSI…) (FC, iSCSI…)

Physical Interconnect Physical Interconnect

(FC, ETH…) (FC, ETH…)
Cable

Server SAN
SAN Network Protocols
SCSI-3

Mapping Layer FC-4 iSCSI FC-4 SRP

Transport Layer FC-3 FC-3 IBA

TCP Operation
SAR
Network Layer FC-2 FC-2
IP Network
Link Layer FC-1 FCoE

Lossless Link
Physical Layer FC-0 ETH
ETH PHY
FC iSCSI FCoE IB

FC Frame SOF Header Data CRC EOF

(Destination ID, Source ID, Type, Frame Control,…)

iSCSI Frame Ethernet header IP header TCP header iSCSI header Data Ethernet trailer
Fibre Channel Protocol
Fiber Channel (FC) is a transport layer protocol that is used to transfer data between computers and
peripheral devices. It has been primarily used for transporting SCSI packets from servers to SAN.
Fibre Channel started in 1988, with ANSI standard approval in 1994.

Advantages SCSI-3 Command

Low Latency
Light overhead of Fibre Channel improves
the
FC-4 Fiber Channel Protocol
transmission latency.
 High security FC-3 Common Fabric Services
Fibre Channel separates from LAN and is
more secure. FC-2 Framing Protocol/Flow Control
Disadvantages
 High Cost FC-1 Encode/Decode (8b/10b or 64b/66b)
Fibre Channel requires the dedicated 133 256 531
adapter and switch. Generally, it is expansive FC-0 Mb/ 1 2 4 8 16 32
Mb/ Mb/
Gb/s Gb/s Gb/s Gb/s Gb/s Gb/s
to deploy. s s s

Obsolete Data Rates Current Data Rates

Fibre Channel Topology

Popular
Point to Arbitrated Loop Fabric
Point

Support only two Support 126 devices Up to 1,600,000 devices

devices
(Fiber Channel Hub) (Fiber Channel Switch)
(Direct connecting)
iSCSI Protocol
Internet Small Computer System Interface (iSCSI) is a transport layer protocol that describes how SCSI
data should be transported over a TCP/IP network, providing an interoperable solution which can take
advantage of existing Internet infrastructure. iSCSI was pioneered by IBM and Cisco in 1998 and
submitted as a draft standard by Internet Engineering Task Force (IETF) in March 2000.

Advantages
SCSI-3 Command
 Low Cost
Use standard Ethernet and does not require
expensive and complex switches and cards like
iSCSI
Fibre Channel, so it is more affordable.
 High Flexibility TCP
Running on IP, it is easy to connect and expand.

Disadvantages IP

 Low Reliability
Standard Ethernet has issues of packet loss, so it is ETH
not as reliable as Fibre Channel, however, iSCSI is
fit for general-purpose applications. 10 25 40 100
1 Gb/s
Gb/s Gb/s Gb/s Gb/s
FCoE Protocol
Fiber Channel over Ethernet (FCoE) is a storage Advantages
protocol that ensures that Fiber Channel frames are
transmitted directly over Ethernet. FCoE moves Fiber
Channel traffic to the existing high-speed Ethernet  Low Network Complexity
infrastructures and then integrates storage and IP FCoE unifies input/output (I/O) ports, consolidates
protocols into a single cable transport and interface. SAN and other traffic into same network, and
FCoE is approved by ANSI at 2009. reduces the number of interface cards and cables.

FC-4 FC-4
Disadvantages
FC-3 FC-3

FC-2 FC-2
 Customer Reluctance
Base on Lossless Ethernet, customer is reluctant to
FC-1 FCoE Mapping
change or replace the legacy network with FCoE.
FC-0 ETH

FC Stack FCoE Stack

IB Protocol
InfiniBand (IB) is a networking communications protocol
SCSI-3 Command
used in high-performance computing that features very
high throughput and very low latency. It is used for data
interconnect among servers and is also used as either a SRP
direct or switched interconnect between servers and
storage systems, as well as an interconnect between
storage systems. InfiniBand originated in 1999. IBA Operation SAR
Advantages
High Speed Network
40Gb/s (QDR), 56 (FDR), 100 (EDR), 200G (HDR)
 Low Latency
IB reduces OS overhead so data can move through the network Link
quickly.
Disadvantages PHY
 High Cost
IB requires the dedicated adapter and switch. Generally, IB is more 40 56 100 200
expensive than Fibre Channel. Gb/s Gb/s Gb/s Gb/s
NVMe-oF Protocol

What is NVMe? NVMe Replaces SCSI

Non-Volatile Memory Express (NVMe) is an
interface specification between computers and
non-volatile memory (especially SSDs) and
Introduced in 2011 by the NVM Express group NVMe-oF
(www.nvmexpress.org).

Type SCSI NVMe

SAN SAN NVMe-oF

Scenario
with HDD/SAS SSD with NVMe SSD
NVM Express over Fabrics (NVMe-oF), a transport
Throughput Low High protocol, used to communicate between
computers and peripheral devices (especially AFA
Latency High Low SAN) over fabric, such as FC, RoCE, IB, TCP i.e.
Currently, NVMe-oF/RoCE is a focus as well as
Queue Single Multiple
NVMe-oF/FC.
NVMe-oF/RoCE

NVMe Command
Software

RDMA Software Stack

RDMA
IB Transport IB Transport IB Transport IB RoCE v1
Typical Hardware

Protocol Protocol Protocol

IB Network IB Network UDP

Layer Layer IP
RDMA / Flexibility
IB Link Layer Ethernet Link Layer Ethernet Link Layer
IB + IP RoCE v2
InfiniBand RoCE v1 RoCE v2

InfiniBand Lossless Ethernet Lossless Ethernet

NOW
NVMe-oF/RoCE Ecosystem
Category Component Status
 Mellanox
NIC
 Marvell
Server NVMe-oF Driver  Linux: Suse SLES 12 SP3, RedHat RHEL 7.4, Ubuntu
16.04.2 LTS…
 Windows: not ready
Multipathing Software  VMware: not ready

 Cisco
LAN Switch
Network  Huawei
(support RoCE)
 Mellanox

 Huawei
SAN NVMe-oF/RoCE  NetApp
 Pure Storage
Vendors are building interoperability among components of NVMe-oF (FC) as well.
NVMe-oF Protocol Trend
Protocol Latency Scalable Cost

NVMe-oF (FC) ★★ ★★ ★

NVMe-oF (RoCE) ★★★ ★ ★★

NVMe-oF (TCP) ★ ★★★ ★★★

HUAWEI Storage NVMe-oF Roadmap

2021 H1 2021 H2 2022 H1 2022 H2

NVMe/FC
NVMe NVMe-oF/FC NVMe-oF/RoCE NVMe-oF/TCP

Replace Replace Replace NVMe/RoCE

NVMe/TCP
SCSI FC-SCSI IB iSCSI
SAN Protocol Comparison – FC Is Now and NVMe-oF Is Future

Performance Cost Reliability,

Future
Availability Ease of
Choice Upgrade
and Mgmt.
Protocol Latency Throughput Purchase Operational Path
Flexibility

FC

iSCSI

FCoE

IB

NVMe-
oF/
RoCE
Lowest rating Highest rating
Fibre Channel is still popular for lower latency, higher reliability and out of habit, however, NVMe-oF is
developing fast to become the major for higher throughput and higher flexibility.
Contents

1 SAN Storage Overview

2 Components and Connectivity of SAN

3 Network Protocol of SAN

4 Application Scenarios of SAN

Typical Application Scenarios for SAN

Databases VSI VDI

Databases commonly Virtual Server Virtualization desktop

store an enterprise's Infrastructure (VSI) often infrastructure (VDI)
most valuable data. They extends to thousands of environments serve virtual
are frequently business- VMs running a broad desktops to large numbers
critical and require the range of operating of an organization's users.
highest performance and systems and applications, Some VDI environments
availability. with different can easily number in the
performance tens of thousands of virtual
requirements. And desktops. By centralizing
reliability also becomes the virtual desktops,
even more important organizations can more
because a failure can easily manage data
cause multiple application protection and data security.
outages.
SAN Deployment for Databases

Data Center 1 Data Center 2 Key Requirement

 High Performance
IP IP Typically, it is required that SAN
WAN provides more than 200,000 IOPS and
Oracle SQL Oracle SQL
RAC 1 Server
Sysbase
RAC 2 Server
Sysbase satisfies thousands of transactions each
second.
 High Reliability
Generally, database is a mission-critical
Active-Active task. DC failures have impact on
business.
SAN Solution
FC switches FC switches  AFA SAN
Configure AFA SAN which provides
extremely high IOPS and less than 1 ms
latency.
IP  Active-Active/snapshot
FC
Build Active-Active solution with RPO=0
Mirror FC
and RTO≈0, moreover, enable snapshot
SAN SAN
to protect data against logical errors.
SAN Deployment for VSI

Data Center 1 Data Center 2 Key Requirement

 High Performance
IP IP
WAN Each VM requires 20 to 50 IOPS, thousands
VMware FusionSphere VMware FusionSphere of VMs require that SAN provides more or
less 100K IOPS.
 Require Data Protection
Many applications are running on SAN. In
the event of SAN’s failure, data loss is less
than 30 mins.

SAN Solution

IP switches
 AFA SAN
IP switches Configure AFA SAN to support hundreds to
thousands of VMs.
 Remote Replication/snapshot
Snapshots protect data against logical errors
IP or virus infection. Remote replication builds
Replication
Link
disaster recovery solution, in the event of
Remote SAN's failure, services will be manually
SAN Replication SAN switched over to DC 2 and RPO ≤ 30 mins.
SAN Deployment for VDI

Key Requirement

IP  Performance
Typically, regarding to performance of each
VMware desktop, image volume requires 15 to 17 IOPS and
Horizon data volume often requires 3 IOPS.
Citrix  High Scalability
FusionAccess Require to expand capacity and increase
performance as the number of desktops is growing.

SAN Solution
IP switches
 SSD & NL-SAS disks mixed
Configure SSDs for image volumes and NL-SAS
disks for data volumes.
 Scale-Up and Scale-Out
Easy to scale up when expanding capacity. Also,
easy to scale out when increasing performance.
SAN
Thank you. 把数字世界带入每个人、每个家庭、
每个组织，构建万物互联的智能世界。
Bring digital to every person, home, and
organization for a fully connected,
intelligent world.

Copyright©2021 Huawei Technologies Co., Ltd.

All Rights Reserved.

The information in this document may contain predictive

statements including, without limitation, statements regarding
the future financial and operating results, future product
portfolio, new technology, etc. There are a number of factors that
could cause actual results and developments to differ materially
from those expressed or implied in the predictive statements.
Therefore, such information is provided for reference purpose
only and constitutes neither an offer nor an acceptance. Huawei
may change the information at any time without notice.