Storage System Architecture

Foreword

 With the development of cloud computing and big data, the storage
architecture changes from scattered to centralized, and gradually evolves
to network-based, virtualized, and massive cloud storage. Storage not only
needs to provide routine services such as data management, data
replication, snapshot, mirroring, and migration, but also needs to enable
functions such as data disaster recovery, data consistency, virtualized
convergence, elastic computing, and resource expansion. These services and
functions depend on a good storage system architecture.

2 Huawei Confidential
 Objectives

On completion of this course, you will be able to know:

 Storage architecture evolution and expansion methods
 Huawei storage product architecture

3 Huawei Confidential
 Contents

1. Storage System Architecture Evolution

2. Storage System Expansion Methods

3. Huawei Storage Product Architecture

4 Huawei Confidential
Storage System Architecture Evolution

Single controller Dual controllers Multiple controllers Distributed

architecture

5 Huawei Confidential
Single-controller Storage

SCSI interface
Logical disks are presented to hosts as LUNs.

SCSI

Terminator
controller

Controller
OS

Host Single-controller storage

6 Huawei Confidential
Dual-controller Storage
Services are running on Services are running on
only one controller. both controllers.

Controller A Controller B Controller A Controller B

Active-Standby Active-Active

7 Huawei Confidential
Mid-range Storage Architecture Evolution
Fixed storage Dual-controller Flexible hardware Flexible software
system active-active component function
configuration redundancy configuration configuration
configuration

Limited Fibre Channel The single-controller Fast Ethernet (FE) Unified storage that
interfaces are and dual-controller interface module are supports both SAN
provided, and the active-passive supported, greatly and NAS protocols has
flexibility is poor. architecture gradually improving flexibility become a hotspot.
Capacity expansion evolve to the active- and scalability. Users can flexibly
can be implemented active architecture. The number of ports configure multi-
only by cascading disk can be selected as protocol services as
enclosures. required. required.

8 Huawei Confidential
Multi-controller Storage

Controller A Controller B Controller C Controller D

SAS controller SAS controller SAS controller SAS controller

SAS expander SAS expander

Disk Disk
enclosures enclosures

9 Huawei Confidential
Mission-Critical Storage Architecture Evolution
Bus Hi-Star Direct-connection Virtual matrix
architecture architecture architecture architecture

Scale-up multi- Switch-based Front-end interfaces Scale-out expansion

controller architecture connection of front- and back-end disk mode, full switching
based on bus end interfaces, back- interfaces directly mode, x86 platform,
interconnection and end disk interfaces, connected to cache and loose coupling
upgrade by using and cache modules, resources to avoid
more powerful CPUs, and back-end Fibre latency caused by bus
interface modules, Channel connection and switch connection
memory, and
protocols

10 Huawei Confidential
Storage Software Technology Evolution
Improved data Simplified data Improved space Optimized service
reliability management utilization performance

Emergence of Storage devices are Tiered storage Performance is

snapshot, clone, and flexibly managed by promotes storage optimized according
data replication centralized space utilization and to service types.
(synchronous and management service efficiency. The Resources are properly
asynchronous) software. Thin data deduplication allocated according to
technologies provisioning technology reduces service importance.
technology resolves maintenance and Storage resources are
the conflict between capacity expansion balanced and shared
resource investment costs. flexibly and globally.
and optimal use.

11 Huawei Confidential
Distributed Storage Technology Evolution
Automatic metadata extension

Giga+ indexing

Hadoop FS
Universal hardware Universal hardware Metadata separation Public cloud storage
Customized hardware Software-defined Software-defined Metadata extension
Scale-up Scale-up Scale-out EC Coding
Integrated Parallel cluster file Huawei
GoogleFS Azure Storage
box system OceanStor 100D
General purpose Scientific computing Analysis scenario Facebook
Facebook F4
Haystack
Social scenario Warm storage
MooseFS NoSQL
Small files

199x 2000 2008 to 2011 2014

2003
EMC PVFS Facebook/Microsoft/Huawei Facebook
Google
NetApp Lustre

12 Huawei Confidential
Distributed Storage Architecture
iSCSI S3/Swift client
VBS
initiator HDFS client
Compute node Compute node

VBS EDS-F EDS-F EDS-F OBJ OBJ OBJ

service service service
EDS-B EDS-B EDS-B HDFS HDFS HDFS
Block Block Block
OBJ OBJ OBJ
service service service
service service service index index index
Index Index Index Index Index Index

Storage pool (block) Storage pool (HDFS) Storage pool (object)

MDC MDC MDC

OSD OSD OSD OSD OSD OSD OSD OSD OSD

CM CM CM

Server node Server node Server node Server node Server node Server node Server node Server node Server node

Note: Huawei OceanStor 100D is used as an example to describe the distributed storage architecture.

13 Huawei Confidential
 Contents

1. Storage Architecture Evolution

2. Storage System Expansion Methods

3. Huawei Storage Product Architecture

14 Huawei Confidential
Scale-up and Scale-out
 With the continued development of enterprise information systems and the ever-increasing expansion in the
scale of services, service data keeps increasing. The initial configuration of storage systems is often not
enough to meet these demands. Storage system capacity expansion has become the major concern of system
administrators. There are two capacity expansion methods: scale-up and scale-out. The following uses Huawei
storage products as an example to describe the two methods.
Storage
Devices (enclosures and disks) Storage controllers
controllers

Storage Storage
controllers controllers

Scale-out Switch Fabric

Storage
Scale-up controllers Storage
controllers

Scale-up architecture

Scale-out architecture

15 Huawei Confidential
SAS Disk Enclosure Scale-up Networking Principles
Adding a 2 U SAS disk enclosure to
an existing loop
Controller
enclosure

2 U SAS disk
enclosure 0

2 U SAS disk
enclosure 1

16 Huawei Confidential
Smart Disk Enclosure Scale-up Networking Principles
Adding a 2 U smart disk enclosure
to an existing loop
Controller
enclosure

Smart disk
enclosure 0

Smart disk
enclosure 1

17 Huawei Confidential
PCIe Scale-out and IP Scale-out
 PCIe scale-out runs on the PCIe protocol while IP scale-out works based on the IP protocol.

Large Mission-critical storage

enterprises Functions first
PCIe scale-out

Mid-range storage
Medium Functions and prices
enterprises balanced
IP scale-out

Entry-level storage
Small enterprises Price first
IP scale-out

18 Huawei Confidential
Scale-out Technologies Used by Huawei Storage Systems
OceanStor OS

XNET

XNET-PCIe XNET RDMA XNET UDP

iWARP
UDP
IB
PCIE DRV transport TCP

IP

PCIe IB ETH

19 Huawei Confidential
Scale-out Networking
Connects to the user's management network

Controller enclosure
0 (original)

Connects to the user's

management network

Controller enclosure
1 (new)

The figure shows the scale-out

networking of Huawei solid-state
storage Huawei OceanStor Dorado
5000 V6 and 6000 V6.
Four-controller direct connection

20 Huawei Confidential
Local Write Process

1 2

SAN

1
LUN 2 PCIe switched network
4

Engine 0 Engine 1 Engine 2 Engine 3

3 5

21 Huawei Confidential
Non-local Write Process

1 4

SAN

1 4

LUN
2 PCIe switched network
3
6

Engine 0 Engine 1 Engine 2 Engine 3

5 7

22 Huawei Confidential
Local Read Process

1 2

SAN

2 PCIe switched network

LUN

8
Engine 0 Engine 1 Engine 2 Engine 3
3 4 6 7

23 Huawei Confidential
Non-local Read Process

1 4

SAN

1 4

2 PCIe switched network

LUN
3
7

10
Engine 0 Engine 1 Engine 2 Engine 3
5 6 8 9

24 Huawei Confidential
 Contents

1. Storage Architecture Evolution

2. Storage System Expansion Methods

3. Huawei Storage Product Architecture

25 Huawei Confidential
Hardware Architecture with Two Controllers
Interface .. Interface Interface Interface
.. Service subsystem
module A0 . module A7 module B7 module B0
.
PCIe 3.0 x8 PCIe 3.0 x8

Fan module 0 PCIe 3.0 Fan module 0

Control x8 Control
Fan module 1 Fan module 1
module A module B
Fan module 2 Fan module 2

Management
Management Management subsystem
interface module A interface module B

12 V
12 V
Power Power Electromechanical
BBU 0 BBU 1 BBU 2 BBU 3
supply 0 supply 1 subsystem

Service channel
Manage channel
Power supply

26 Huawei Confidential
Hardware Architecture with Multiple Controllers
Interface ... Interface Interface ... Interface
module A0 module A5 module B5 module B0
PCIe 3.0 x8 PCIe 3.0 x8

Fan module 0 PCIe 3.0 x8 Fan module 0

Control Control Service subsystem
Fan module 1 Fan module 1
module C module D
Fan module 2 Fan module 2

... ...
Interface Interface Interface Interface
module A0 module A5 module B5 module B0
PCIe 3.0 x8 PCIe 3.0 x8

Fan module 0 PCIe 3.0 x8 Fan module 0

Control Control
Fan module 1 Fan module 1
module A module B
Fan module 2 Fan module 2

Management
Management Management
subsystem
interface module A interface module B

12 V
12 V
Electromechanical
Power Power Power Power
BBU 0 BBU 1 BBU 2 BBU 3 subsystem
supply 0 supply 1 supply 2 supply 3

27 Huawei Confidential
Full-mesh Architecture
Host I/O
Network adapter Network adapter
Fully-shared FIMs
 Each front-end interconnect I/O module (FIM)
connects to all four controllers through PCIe ports
in a controller enclosure. This module can
FIM FIM FIM FIM simultaneously access the four controllers with
multi-channel technology in active-active mode.

Full interconnection among controllers

192 192  The controllers in a controller enclosure are fully
cores core interconnected using the passive backplane.
s  100 Gbit/s RDMA shared interface modules are
used for expansion across controller enclosures,
BIM BIM BIM BIM 192 192
implementing full interconnection between 8/12/16
cores cores
controllers.

Interconnection of disks and

enclosures across controller enclosures
 A controller enclosure uses the back-end
interconnect I/O module (BIM) to connect a disk
enclosure which can be accessed by all controllers in
the enclosure.
Note: Huawei OceanStor Dorado 8000 and 18000 V6  A smart disk enclosure has two groups of uplink
are used as the example. ports and can connect to two controller enclosures,
implementing full interconnection between the disk
enclosure and eight controllers.

28 Huawei Confidential
Fully Interconnected Disk Enclosures
Controller Controller
enclosure 0 enclosure 1

2 U smart disk
enclosure 0

2 U smart disk
enclosure 1

29 Huawei Confidential
E2E Global Resource Sharing

Host

SAN

Distributed based
on hash results DHT

Global cache

... ...

30 Huawei Confidential
Switchover in Seconds: Mission-Critical Storage with FIMs
Host

 Hosts are unaware of the

Fibre Channel switch switchover, and links are
not interrupted.

Hi1822

I/O process I/O process I/O process I/O process

System mgmt. Device mgmt. Configuration Network

process process mgmt. process mgmt. process

User mode

OS kernel mode

Controller 0 Controller 1 Controller 2 Controller 3

31 Huawei Confidential
Global Cache
Write latency
LUN 0 LUN 1 LUN 2 95 μs
4 KB 4 KB ... 8 KB 4 KB 4 KB ... 8 KB 4 KB 4 KB ... 8 KB
50 μs

Write Ahead Log (WAL)

Cache linear ...
A B C D E
space
Traditional cache OceanStor
Data Dorado V6

write

A B C D E ... Global memory

Virtual address space

RDMA
AddrN1 AddrN2 AddrN3

Memory of controller A Memory of controller B Memory of controller C Memory of controller D

32 Huawei Confidential
Global Pool
A A1 A2 A3

B B1
LUN
C C1 C2

D D1 D2

LUN mapping
and FP mapping

Strip A B C D P Q

Strip A1 B1 C1 D1 P Q

Strip A2 A3 C2 D2 P Q

RAID ... ...

...

33 Huawei Confidential
Back-End Sharing
Engine 0 Engine 1 • A single port is connected via a single
13 0 13 connection in the external system.
• A single port is connected to four
A A controllers in the internal system.
0 13 0 13

B B
100 Gbit/s 100 Gbit/s • Dual physical links, balanced and
100 Gbit/s redundant

100 Gbit/s
A 0 1 2 3
B 0 1 2 3
A B C D
x4 x4 x4 • A single port on the controller
x4 cascades two disk enclosures in a
loop.
• A single expansion module supports
A 0 1 2 3
dual links, achieving load balancing
1822
B 0 1 2 3 and redundancy.

34 Huawei Confidential
Active-Active Architecture with Full Load Balancing

35 Huawei Confidential
Cache Mirroring Technology
 The function is as follows:
Controller enclosure 1 Controller enclosure 2

Controller A Controller B Controller C Controller D Controller A Controller B Controller C Controller D

Hybrid flash storage

36 Huawei Confidential
Key Reliability Technology
 The function is as follows:

Controller A Controller B Controller C Controller D Controller A Controller B Controller C Controller D Controller A Controller B Controller C Controller D

Normal Failure of a controller (controller A)

37 Huawei Confidential
Host Service Switchover upon Failure of a Single
Controller
Host Host

Fibre Channel switch Fibre Channel switch

Interconnect I/O Interconnect I/O

module module

Automatic
switchover

Backplane Backplane

Controller Controller Controller Controller Controller Controller Controller Controller

1 2 3 4 1 2 3 4

38 Huawei Confidential
 Quiz

1. (True or False) Scale-up is a method in which disk enclosures are continuously

added to existing storage systems to cope with increasing data volumes.

2. (Short Answer Question) What are the differences between scale-up and scale-
out?

39 Huawei Confidential
 Summary

Storage Architecture
Evolution

Storage System
Scale-up and Scale-out
Storage System Expansion Methods
Architecture

Hardware System Architecture

Huawei Storage
Product Architecture
Full-mesh Architecture

40 Huawei Confidential
 More Information

Enterprise Huawei
Technical Enterprise
Support App Service App

41 Huawei Confidential
 Recommendations

 Huawei official websites

 Enterprise business: https://enterprise.huawei.com/en/
 Technical support: https://support.huawei.com/enterprise/en/index.html
 Online learning: https://learning.huawei.com/en/

 Popular tools
 HedEx Lite
 Network Document Tool Center
 Information Query Assistant

42 Huawei Confidential
Thank you. 把数字世界带入每个人、每个家庭、
每个组织，构建万物互联的智能世界。
Bring digital to every person, home, and
organization for a fully connected,
intelligent world.

Copyright©2020 Huawei Technologies Co., Ltd.

All Rights Reserved.

The information in this document may contain predictive

statements including, without limitation, statements regarding
the future financial and operating results, future product
portfolio, new technology, etc. There are a number of factors that
could cause actual results and developments to differ materially
from those expressed or implied in the predictive statements.
Therefore, such information is provided for reference purpose
only and constitutes neither an offer nor an acceptance. Huawei
may change the information at any time without notice.