FICON Over IP – Technology

and Customer Use

Tony Almeida
Cisco Systems
[email protected]

Thursday, March 3, 2011

Session 8487
Abstract

•  Many customers need to send FICON traffic between two datacenter

sites - or between their datacenter and their tape archive. In the past
this meant dedicated SONET circuits but today FICON over IP allows
customers to send this storage data over IP networks that are much
more cost effective. This IP network can be a dedicated circuit or even
be a virtual tunnel within collapsed corporate networks. First in this
session we will discuss the technology to do this and how it insures the
RAS required for storage and then in the second half of the session,
we will discuss a customer network where this was implemented and
is running in production today.
Agenda

•  What is FCIP
•  Where/Why is it mostly used
•  Basic FCIP – What do I need to worry about
•  Advanced FCIP
•  Customer Examples
•  Customer 1
•  Customer 2
 SAN Extension Technology Options

Increasing Distance
Data
Center Campus Metro Regional National Global

Dark Fiber Sync (Any Speed) Limited by Optics (Power Budget)

Optical

CWDM Sync (1,2,4 Gbps) Limited by Optics (Power Budget)

DWDM Sync (1,2,4,10 Gbps per λ) Limited by BB_Credits

SONET/SDH Sync (1,2 Gbps + Subrate) Async

MDS9000 FCIP Sync (Metro Eth)

IP

Async (WAN,1 Gbps)

FCIP: Fibre Channel over IP

IP Network

FC/FICON SAN FCIP Tunnel FC/FICON SAN

FCIP Is a Standard from the IETF IP Storage WG for Linking

FibreChannel SANs over IP (RFCs 3821 and 3643)
•  Point-to-point tunnel between FCIP link end-points
•  Appears as one logical FC fabric with single FSPF routing domain
FICON is just another upper layer protocol that can be transported over IP
•  FICON over FCIP can provide cost-effective channel extension
5
FCIP Frame Detail

Ethernet IP TCP TCP FCIP EISL opt Ethernet

SOF
FC Frame
Header Header Header Opts Header Hdr Hdr CRC32
14 20 20 12 28 4 8 0-16
4

94 Max 2148 (E_Port) + EISL and Opt Headers

FCIP Overhead for EISL and Optional Headers
Ethernet Frames:
If TE_Port, then 8 Bytes
94 Byte Header + 4 Added to FC Frame (after
Byte CRC = 98 Bytes SOF) for VSAN Routing

•  Max FibreChannel frame is 2148 bytes plus optional extras

•  FC frames are segmented and reassembled if MTU too small (TCP
payload on second or subsequent packets)
•  Jumbo frames may increase performance
•  IP MTU of 2300 avoids splitting of TCP frames

6
Why Use FCIP?

•  Network availability:
•  Lambdas or dark fiber not available or too expensive
•  IP network capacity already in place or only alternative

•  Distance:
•  FCIP not limited by BB_Credits
•  Extension only limited by TCP max window (32MB for MDS9000 
20,000km at 1Gbps)

•  Application requirements:
•  Need Acceleration technologies built into FCIP

7
FCIP Configuration Overview
• To create a basic FCIP configuration, follow these steps on each
peer switch:
1.  Configure the Gigabit Ethernet interface
2.  Configure static IP routes
3.  Enable the FCIP feature
4.  Create an FCIP profile
5.  Create an FCIP interface (up to 3 per profile)

FCIP Int
Trunk mode
FCP or Profile (TCP)
FICON Max BW
QoS
Min BW
IP
RTT
FCIP Int GbE Network
IP address
MTU
FCIP Int
Basic FCIP Configuration Example

FCIP configuration on MDS A FCIP configuration on MDS B

interface GigabitEthernet2/1 interface GigabitEthernet2/4
ip address 10.1.21.21 255.255.255.0 ip address 10.2.21.11 255.255.255.0
no shutdown no shutdown

ip route 10.2.21.0 255.255.255.0 ip route 10.1.21.0 255.255.255.0

10.1.21.254 interface gig2/1 10.2.21.254 interface gig2/4

fcip enable fcip enable

fcip profile 1 fcip profile 1
ip address 10.1.21.21 ip address 10.2.21.11

interface fcip1 interface fcip1

use-profile 1 use-profile 1
peer-info ipaddr 10.2.21.11 peer-info ipaddr 10.1.21.21
no shutdown no shutdown
GE2/1 10.1.21.21 GE2/4 10.2.21.11

WAN
MDS A 10.1.21.254 10.2.21.254 MDS B
MDS FCIP SAN Extension Design
FC

•  Same port channeling and

VSAN trunking rules apply as
with FC links
Portchannel •  Port channel individual FCIP
Diverse 2 x 1Gbps
Network over Two links to alternate Ethernet
Paths Diverse switches/routers
Paths
•  Each WAN link carries two FCIP
tunnels
Switch/Router

FC

10
Storage Traffic and TCP

•  Storage traffic:
•  Quite bursty
•  Latency sensitive (sync apps)
•  Requires high, instantaneous throughput
•  Traditional TCP:
•  Tries to be network sociable
•  Tries to avoid congestion (overrunning downstream routers)
•  Backs off when congestion detected
•  Slow to ramp up over long links (slow start and
congestion avoidance)
MDS FCIP TCP Behavior

•  Reduce probability of drops

• Bursts controlled through per flow shaping and congestion
window control  less likely to overrun routers
•  Increased resilience to drops
•  Uses SACK, fast retransmit and shaping
•  Aggressive slow start q
•  Initial rate controlled by min-available-bandwidth
•  Max rate controlled by max-bandwidth
Differences with Normal TCP:
  When congestion occurs with other conventional TCP traffic, FCIP
is more aggressive during recovery ( bullying the other traffic)
Aggression is proportional to the min-available-bandwidth configuration
 The Impact of TCP Congestion Avoidance
Simplified TCP Congestion Avoidance
Linear Congestion
Avoidance
(+1 cwnd per ACK) loss cwnd halved on packet loss;
Packets Sent per Round Trip (Congestion Window)

retransmission signals congestion;

Slow Start Threshold adjusted

Slow Start Threshold loss

Exponential
Slow Start
(2x pkts per RTT)

Low throughput during

this period

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

Round Trips
 FCIP Flow Control Design Factors

12
Round Trip Time: 9 3

4 3 2 1

Maximum Window Size:

ACK

Packet Shaping: Total Bandwidth

•  Maximum bandwidth Maximum

•  Minimum available bandwidth Minimum

Congestion window monitoring

Round Trip Time

Configuring the round-trip-time parameter:

•  Not necessarily symmetric
•  Use ping and measure-rtt commands to calculate
•  Automatically calculated in Cisco MDS

45-Mbps max_bw (dedicated)

gigE 45-Mbps gigE

12-ms

12-ms

End-to-end latency * 2 = 24 ms RTT.

TCP Maximum Window Size

Set the TCP MWS to

RTT 10-ms

Bandwidth To keep the pipe full:

155-Mbps 155-Mbps x 10-ms = 192-KB
(OC-3)

MWS = Maximum bandwidth x RTT

Example: 5-ms latency = 10-ms RTT x 155-Mbps (OC-3) = 192-KB
•  Under dimensioning will throttle throughput
•  Over dimensioning can cause congestion
•  Use the bandwidth of the lowest speed link
TCP Maximum Bandwidth

Configure the TCP max-bandwidth value as follows:

•  No larger than smallest pipe in the path
•  If sharing the pipe, configure to be highest amount available to FCIP
•  In a shared environment, configure QoS in the entire path

max_bw 45-Mbps (dedicated)

GigE 45 Mbps GigE

GigE 155-Mbps GigE

max_bw 45-Mbps (shared)

100-Mbps 100-Mbps
TCP Minimum Available Bandwidth
Configure TCP min-available-bandwidth value as follows:
•  If dedicated path, min-available-bandwidth = max-bandwidth
•  If shared path, use least amount that is always available to FCIP
•  Lower if you see frequent retransmissions in a shared transport
•  Must be at least 1/20 of max-bandwidth

min_bw 45-Mbps (dedicated)

max_bw 45-Mbps (dedicated)

GigE 45-Mbps GigE

GigE 155-Mbps GigE

max_bw 45-Mbps (shared)

Must be higher than
1/20 of TCP max-bw min_bw 20-Mbps (shared)

100-Mbps 100-Mbps
 Results of Packet Shaping
Traditional TCP Congestion Avoidance

Packet Shaping
Congestion Avoidance (+2
cwnd per RTT)
Retransmission
Max_Window Size
Packets Sent per Round Trip

Slow Start Threshold

Slow Start Threshold
(Congestion Window)

initialized to 95% Minimum threshold =

MWS min-available-bw
after one RTT
Shaper engaged during first
RTT = min-available-bw

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Round Trips
MTU Size: Performance Comparison
140.0
Bidirectional
Single Port
120.0

100.0
MTU = 2300
FC Throughput (MBps)

80.0

60.0

40.0

MTU = 1500; Single FC Frame

20.0
broken over two IP Packets

0.0
100 252 500 752 1000 1252 1500 1752 2000 2148
FC Frame Size
FCIP – Multiple FCIP Tunnels

int fcip 11 int fcip 11 VSAN 100

VSAN 100 GigE1/1.100 VSAN 100 GigE1/1.100 Replication
Replication GigE1/1 GigE1/1
int fcip 12 int fcip 12
GigE1/1.200 VSAN 100 GigE1/1.200
int fcip 21 VSAN 200 int fcip 21
GigE1/2.100 GigE1/2.100
VSAN 200 GigE1/2 int fcip 22
VSAN 200
int fcip 22 GigE1/2
Tape Access GigE1/2.200 GigE1/2.200 VSAN 200
Tape Access

Using GE Sub-Interfaces, Multiple FCIP Tunnels and Port Channeling

to Enable High b/w FCIP
•  Use separate VSANs for data replication (100) and tape acces (200)
•  Port channel FCIP tunnels for replication traffic for load balancing
•  Tape access Links are not port channeled

Note – When multiple FCIP tunnels are on the same interface, they use
a different TCPIP port numbers
QoS for FCIP SAN Extension

Most FCIP Implementations Use Dedicated Links, However:

•  Acceptance of FCIP generating interest in converged
IP network for FCIP SAN Extension
•  No standard DSCP values for FCIP traffic
(unlike voice with DSCP EF)
• QoS—define marking and classification
• Mark DSCP according an agreed value
• Separate consideration of FCIP data and control packet
•  Bandwidth reservation
• FCIP has no support for reservation protocol
• Simulated using min/max B/W command
(can be considered as a type of reservation)
FCIP QoS Mapping Proposal

•  Synchronous data replication: bursty, high bandwidth

• Can be mapped into mission critical (AF31/DSCP 26)
•  Asynchronous data: bursty, low to medium b/w
• Can be mapped into transactional data (AF21/DSCP 18)
• Can also mapped into bulk data (AF11/DSCP 10)
•  Backup data: 150 ~ 500 ms, constant (during backup),
medium b/w
• Can be mapped into bulk data (AF11 / DSCP 10)
•  Control packets
• Both control and data traffic can be assigned the same class
• If needed can assign CS6 or DSCP 48
 FCIP QoS Markings
Customer networks can have several types of business-critical traffic, including
voice over IP (VoIP), video, FCIP, business applications, etc…

Traffic is normally classified as it enters the network, where it is marked for

appropriate treatment.

High

Med

Low
IP Quality Of Service

The Cisco MDS 9000 can tag control and data traffic for every FCIP
link with a DSCP value between 0 and 63:
•  QoS-aware WANs can then recognize and treat the tagged traffic according to
enterprise QoS policies
•  FCIP traffic tagged at a higher DiffServ priority is treated more favorably if
congestion occurs on the WAN

SWT1# show int fcip 1 SWT2# show int fcip 1

. . . . . .
QOS control code point is 3 QOS control code point is 3
QOS data code point is 5 QOS data code point is 5

SWT1# show int fcip 2 QoS QoS SWT2# show int fcip 2
. . . Policy Policy . . .
QOS control code point is 3 QOS control code point is 3
QOS data code point is 8 QOS data code point is 8
IPS IPS

Priority Priority
FCIP – Multiple FCIP Tunnels

int fcip 11 int fcip 11 VSAN 100

VSAN 100 GigE1/1.100 VSAN 100 GigE1/1.100 Replication
Replication GigE1/1 GigE1/1
int fcip 12 int fcip 12
GigE1/1.200 VSAN 100 GigE1/1.200
int fcip 21 VSAN 200 int fcip 21
GigE1/2.100 GigE1/2.100
VSAN 200 GigE1/2 int fcip 22
VSAN 200
int fcip 22 GigE1/2
Tape Access GigE1/2.200 GigE1/2.200 VSAN 200
Tape Access

Now, Configure QOS based on business priorities of data

•  VSAN 100 – high priority – disk mirroring
•  VSAN 200 – med priority – Tape backups
•  VSAN 300 (not shown) – low priority (open systems SAN stuff)

Making the assumption that this is a dedicated SAN WAN infrastructure –

but within that, prioritization is needed.

Note: Routers and Switches MUST be QOS aware.

 Large Provider of Business
Outsourcing Services - FCIP

MDS
9000 Mainframe VSAN
Mainframe VSAN

7600 Routers

PtP VTS VSAN

PtP VTS VSAN
MPLS
Network
(8) OC48
Native Tape VSAN Native Tape VSAN

1500 Km

Disk Replication
Disk Replication IP Services Module Provide:
Compression
VSAN
VSAN
Encryption
Tape Acceleration
Disk Write Acceleration

Open Systems Open Systems

VSANs VSANs

27
FCIP Data Compression
•  Cisco uses RFC standard compression algorithms
implemented in both hardware and software
•  MDS 9000 18/4-port Multiservice Module
•  Third Generation IP Services Module
•  Hardware and software-based compression, hardware-based encryption,
and intelligent fabric-based application services
•  Three compression algorithms—modes 1–3 plus auto
mode
•  Compressibility is data stream dependent
•  All nulls or ones → high compression (>30:1)
•  Random data (e.g., encrypted) → low compression (~1:1)
•  Typical rate is around 4:1, but may vary considerably
•  Application throughput is the most important factor
IPSec Encryption for FCIP
FCIP Link Encryption Provides:
•  Data confidentiality—sender can encrypt packets before
transmitting them across a network
•  Data integrity—receiver can authenticate packets sent by
the IPSec sender to ensure that the data has not been
altered during transmission
•  Data origin authentication—receiver can authenticate the
source of the IPSec packets sent; this service is
dependent upon the data integrity service
•  Anti-replay protection—receiver can detect and reject
replayed packets
 Hardware-Based IPSec Encryption
Remote Tape Backup

Primary Site

Remote Replication IP Network

Tape Backup and Remote Replication

Secured with IPsec

•  Hardware-based GigE wire-rate performance with latency

~ 10 µs per packet
•  Standards-based IPSec encryption—implements RFC 2402
to 2410, and 2412
• IKE for protocol/algorithm negotiation and key generation
• Encryption: AES (128 or 256 bit key), DES (56 bit), 3DES (168 bit)
FICON Tape Write Acceleration

•  Accelerates Writes by means of local acknowledgement

•  Command Response
•  Status
•  Data is never fully owned by the FTA
•  Sync command is not emulated – insures data integrity
•  Tape control, label processing, etc are not accelerated
•  Has been shipping for multiple years and numerous
customers in production
FICON Tape Read Acceleration

•  Accelerates Reads by
•  Flowing off the host until data ready
•  Stage data at host side – continue reading at the tape side
•  Start up the host reading the staged data
•  If too much data is pre-read, FTA will reposition the tape
•  Tape control, label processing, etc are not accelerated
•  Currently completing development and test for release mid
2010.
Sun VSM to RTD Extension

VSM at local site writing both a local tape copy as

well as a tape copy at the archived site.

FICON FICON
 FICON Tape Acceleration
MainFrame

Direct Tape Ficon

Over
FCIP

Tape Drives

MainFrame
Direct Library Ficon
Access Over
FCIP

Tape Libraries
FTA Configuration Information

•  Only one active FTA-enabled FCIP link is allowed between

two domains. Port Channels of FTA-enabled links are not
supported.
•  This is due to the fact that state is kept on a per-port basis.
•  Multiple/all ports on a IPS card can run FTA
simultaneously
•  Each of the links must be trunking a different VSAN.
•  8 FICON VSANs are allowed per MDS Chassis – each of
these with its own CUP for management.
•  The number of write chains buffered is automatically
adjusted based on the tape speed and the RTT of the
FCIP connection
FTA – More Details

•  There is support for both 3590 and 3490 real FICON tape
drives. There is support for 3490 Virtual Tapes
•  IBM and STK have both only implemented 3490 Virtual
Tapes in their VTS and VSM platforms respectively.
•  3490 versus 3590 selection is dynamic and no configuration
is needed for device selection. There can be 3490 and 3590
on the same FCIP link at the same time.
•  Multipath is supported from the host to the tape.
•  These multiple paths must still transverse the same FCIP link
but this gives higher host-side redundancy.
 Backup protocol without acceleration …
FCIP
Mainframe Cisco MDS Cisco MDS VTS / Tape Library
ScratchVol mount, Write VolHdrs etc.
Write Chain

Status
TAPE IDLE
Write Chain

Status

SYNC

All data on Media

Status
Rewind Unload
 Backup protocol with acceleration …
Host-side Tape-Side VTS / Tape
Mainframe FCIP FCIP Library
ScratchVol mount, Write VolHdrs etc.

Write Chain 1
Write
Chain 1

New OXID
Write
SYNC Chain 2

All data on
Media

Rewind Unload
 Results: Throughput

18.000

16.000

14.000

12.000
Throughput – MB/s

10.000
FTA Enabled (MB/sec)
FTA Disabled (MB/sec)
8.000

6.000

4.000
1000 Km – 10ms
2.000
5000 Km – 50ms

0.000
0ms 10ms 20ms 30ms 40ms 50ms 60ms 70ms 80ms 90ms 100ms

Site-Site Delay
VSM - RTD Customer Example - EMEA

OC-12

OC-12

40
What is XRC?
•  XRC = eXtended Remote Copy
•  Now officially z/OS Global Mirror
•  Mainframe-based replication SW
Primary •  XRC clients include:
Primary FICON
System z DASD •  Over 300 installations worldwide (source: IBM)
•  Major Banks in Germany, Scotland, Italy,
MAN / Turkey, Greece
WAN •  Major US Banks / Brokerages / Insurance
Co s
•  Major Banks in Taiwan, Japan, China,
Thailand, Korea
FICON
•  Remote System Data Mover (z)
•  Reads data from remote primary DASD
2 •  Writes it to local secondary DASD
System z Secondary
SDM DASD
What is good and bad about XRC ?

•  Disk Vendor Independent

•  No lock into vendor unique implementations
•  Can copy from one vendor disk to another
•  Can be used for migrations from one vendor to another
•  Management and control from the mainframe
•  No reliance on disk-to-disk replication changes
•  Performance management from Z
•  Until now, only one solution for Channel extension
•  Cisco is now supporting XRC!
• Will support up to 20,000 Km with XRC Acceleration
feature
XRC Acceleration – How It Works
•  Acceleration of RRS channel programs
- Read Record Sets (ie. Updates)
- In DSO (first command in RRS set)
we know how to pre-read for the whole
chain of data

- We pre-read the data and send

across .. thereby filling the pipe.
- This works around the IU limitations in
the FICON architecture.

PRIMARY DASD SECONDARY DASD

43
How Fast is it?

•  Some performance testing results:

•  Vs. no Acceleration:
• Almost 5x faster at 1600 km
• Almost 9x faster at 3200 km
•  vs. IBM s new Extended Distance FICON feature (z10
ONLY)
• Comparable up to 3200 km; gets better over longer
distance
XRC Acceleration – Other Facts

•  Works with Cisco port channels

•  Allows for less disruption when loss of WAN occurs
•  Works with all models of Z system
•  Backwards compatible with all older Z systems
•  Fully compatible with the new z10 Extended distance FICON
•  Can utilize all compression/encryption on FCIP hardware
•  Supports all 3 major vendor s disk arrays
•  Supports multi-reader, PAVs, and Hyper PAVs
•  This is a separately licensed feature through IBM
XRC Customer Example – Hi Redundancy

OC-xx

OC-xx

x4

All x2

46
FICON – A Few last comments

  FTA Read and Write supported at the same time

•  Seems obvious but wanted to make sure
  FTA and XRCA are mutually exclusive on an FCIP link
•  Can be on the same physical link but not same virtual
interface
  Remember FTA only supports 1 FCIP link between sites
•  Per FICON VSAN
  Any Questions ?!
FICON Device Support Matrix (4.2.7b)
Without Acceleration
Traffic Type Transport Supported Max Distance
XRC FC ISL (CWDM optics) Yes 100 km / 40 km
XRC FC ISL (DWDM optics) Yes 200-300 km
XRC FCIP ISL Yes 200-300 km
FICON FC ISL (CWDM optics) Yes 100 km
FICON FC ISL (DWDM optics) Yes 300 km
FICON FCIP ISL Yes 300 km
Tape - Host to Oracle*** VSM 4/5 FC ISL (CWDM optics) Yes 100 km / 40 km
Tape - Host to Oracle VSM 4/5 FC ISL (DWDM optics) Yes 300 km
Tape - Host to Oracle VSM 4/5 FCIP ISL Yes 300 km
Tape - Oracle VSM 4/5 to RTD FC ISL (CWDM optics) Yes 100 km / 40 km
Tape - Oracle VSM 4/5 to RTD FC ISL (DWDM optics) Yes 300 km
Tape - Oracle VSM 4/5 to RTD FCIP ISL Yes 300 km
Tape - Host to Oracle Real Tape * FC ISL (CWDM optics) Yes 100 km / 40 km
Tape - Host to Oracle Real Tape * FC ISL (DWDM optics) Yes 300 km
Tape - Host to Oracle Real Tape * FCIP ISL Yes 300 km
Tape - Host to IBM Real Tape ** FC ISL (CWDM optics) Yes 100 km / 40 km
Tape - Host to IBM Real Tape ** FC ISL (DWDM optics) Yes 300 km
Tape - Host to IBM Real Tape ** FCIP ISL Yes 300 km
Tape - Host to IBM TS77x0 FC ISL (CWDM optics) Yes 100 km / 40 km
Tape - Host to IBM TS77x0 FC ISL (DWDM optics) Yes 300 km
Tape - Host to IBM TS77x0 FCIP ISL Yes 300 km

*- includes T9840C, T9840D, T9940B, T10K-A, T10K-B

** - includes all FICON supported real tape models
*** - Oracle products are those formerly sold by Sun and StorageTek (STK)
 FICON Acceleration Distance Support
(4.2.7b)

Traffic Type Transport Max Distance

XRC (z/OS Global Mirror) FCIP w/ XRCA (XRC Acceleration) *** 20,000 km

Tape - Host to Oracle VSM 4/5 FCIP w/ FTA (FICON Tape Acceleration)* 5,000 km

Tape - Oracle VSM 4/5 to RTD FCIP w/ FTA (FICON Tape Acceleration)* 5,000 km

Tape - Host to Oracle Real Tape FCIP w/ FTA (FICON Tape Acceleration)* 5,000 km

Tape - Host to IBM Real Tape FCIP w/ FTA (FICON Tape Acceleration)* 3,000 km **

Tape - Host to IBM TS77x0 FCIP w/ FTA (FICON Tape Acceleration)* 3,000 km
**

*- Includes both Read and Write acceleration on the MSM-18/4 card and MDS 9222i, Write only for 14+2.
SSN-16 not supported (yet)
** - Longer distances can be supported via RPQ
*** - Supported on the MSM-18/4 card, SSN-16 not supported (yet)