Storage: Cheap or Fast, Pick One
Storage: Cheap or Fast, Pick One
Storage: Cheap or Fast, Pick One
Server Attached
Disk
Server 1 Meta
FC SAN
Server 2 Switch Disk SAN
Disk
Metadata Comms
As an added twist, most implementations send the
metadata over a separate physical network from the
fibrechannel--usually gigabit ethernet. So every
device in the SAN is usually connected by two
networks
Gig E Switch
FC Switch
SAN Disks
As you can see, the SAN “disks” are getting
pretty complex. In reality the “disks” are
usually RAID enclosures
The RAID enclosure has a FC port, an ethernet
port, and a RAID controller for its disks
The SAN is really building on top of RAID
building blocks
SAN
This arrangement has many advantages:
• Servers and storage devices can be spread farther apart
• All the storage devices go into a single, shared pool
• Very high performance; can do serverless backups
• Storage device availability not tied to the uptime of a host
• Can tie applications to specific storage types (ATA RAID for high
volume, SCSI RAID for high traffic transactional databases)
OTOH, SANs probably won’t go commodity any time soon, since
they have limited applicability to the desktop. Which means that
they will remain expensive and complex
iSCSI
The most common SAN network is fibre channel (FC). The
protocol used is often SCSI.
You can send SCSI over other network protocols; an
emerging option is iSCSI, which puts SCSI in IP over
gigabit ethernet. This is slower but exploits the existing
IP infrastructure and enables WAN SANs.
Why recreate another, separate network for data traffic
using a different hardware standard that requires new
training? Why not just use well-understood IP networks
that your people are already trained on?
The drawback is that iSCSI has somewhat higher latency,
which is an issue for disk access
iSCSI
iSCSI simply sends SCSI commands encapsulated inside TCP/IP,
just as FC SANs send SCSI commands encapsulated inside FC
frames
This can enable wide-area SANs (if you have the bandwidth and
are willing to live with the latency) since IP can be routed; a
SAN distributed across an entire state or country is possible
Putting the SCSI commands inside TCP/IP can add a couple layers
to the software stack, which unavoidably increases latency; Gbit
Ethernet also has less bandwidth than FC
Implementations seem a bit immature for the enterprise; might
ony make a splash with 10 gbit ethernet
Another option is ATA over Ethernet (AOE)
Fibre Channel Costs
Uses copper or optical fibre at 2+ Gbits/sec
Dell/EMC AX100, 3 TB SATA disks, 1 8 port fibre
channel switch, 1 FC card = $16K
FC cards approx. $500 each, 8 port FC switch aprox.
$2,500
Filers with hundreds of TB are available if you’ve got
the checkbook
Often SANs use SCSI disk arrays to maximize
performance
Optical FC has a range up to a few KM, so this can be
spread across a campus
SAN and Clusters
FE Fiber Channel Switching Fabric
C1 FC
Interconnect Storage
C2
Network FC FC
C3 Switch Jukebox
FC
C4 Storage
Very high performance--the compute nodes participate in the
SAN and can share in the benefits of the high speed storage
network
SAN and Clusters
If you an afford this you can get very high
performance; you might have four network
interfaces (Gigabit ethernet, Infiniband,
Gigabit ethernet for metadata, and FC) on
each node
May work well in situations with heavy database
access, very heavy image processing
Biological research, etc.
SAN Example
Apple sells XSAN, their OEM’d storage area
network software; we are setting this up here
Beatnik FC Raid 1
Switch
Bongo
Raid2
Ethernet
Switch
XSAN
Each RAID enclosure of 14 disks may be divided up into
multiple Logical Unit Numbers (LUNs). One LUN
corresponds to one set of RAID disks--a single RAID
box may have more than one collection of RAID sets,
which may be of different types.
One of the RAID boxes has three LUNs: One set of
seven disks in a RAID-5 configuration, one set of five
disks in a RAID-5 configuration, and one set of two
disks in a RAID-1 configuration
XSAN
XSAN builds “Volumes” (a virtual disk) out of
“storage pools”. A storage pool consists of
one or more LUNs.
XSAN
A storage pool is simply a collection of LUNs,
and a volume consists of one or more storage
pools
XSAN
The LUNs are themselves used as RAID
elements; XSAN treats each of them as disks
and does a RAID-0 across them. This means
that LUNs in the same storage pool should be
of the same size
FC Switch Config
The FC switch requires some configuration; this is done via a web-based
interface. It’s not quite as easy as plug-and-go. “Initiators” are
computers, and “targets” are RAID enclosures
Non-Apple
Non-Apple computers can participate in the
SAN; there are Linux implementations of the
XSAN software available from third parties
Each computer participating in the SAN must
have the XSAN software; edu price is about
$500 per copy (more for Linux)
Summary
Buy some disk