0% found this document useful (0 votes)
788 views342 pages

IBM B Type Gen 7 Installation Migration and Best Practices Guide

Uploaded by

Rahul Goyal
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
788 views342 pages

IBM B Type Gen 7 Installation Migration and Best Practices Guide

Uploaded by

Rahul Goyal
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 342

Front cover

Draft Document for Review April 5, 2021 12:31 pm SG24-8497-00

IBM b-type Gen 7


Installation, Migration, and
Best Practices Guide
Brian Larsen
Kameswara Bhaskarabhatla
Matt Brennan
AJ Casamento
Jack Consoli
Mark Detrick
Johnny Fonseca
Tim Jeka

Redbooks
Draft Document for Review April 5, 2021 12:31 pm 8497edno.fm

IBM Redbooks

IBM b-type Gen 7 Installation, Migration, and Best


Practices Guide

April 2021

SG24-8497-00
8497edno.fm Draft Document for Review April 5, 2021 12:31 pm

Note: Before using this information and the product it supports, read the information in “Notices” on
page ix.

First Edition (April 2021)

This edition applies IBM b-type Gen 7 SAN products.

This document was created or updated on April 5, 2021.

© Copyright International Business Machines Corporation 2021. All rights reserved.


Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule
Contract with IBM Corp.
Draft Document for Review April 5, 2021 12:31 pm 8497TOC.fm

Contents

Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .x

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
Authors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
Now you can become a published author, too! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv
Stay connected to IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv

Chapter 1. Introduction to IBM b-type Gen 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1


1.1 Overview of IBM b-type Gen 7 platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.1 Self-learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.2 Self-optimizing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.3 Self-healing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 IBM b-type Gen 7 platforms and naming conventions . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Chapter 2. IBM b-type Gen 7 product review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5


2.1 IBM b-type Gen 7 SAN director overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2 IBM b-type Gen 7 SAN512B-7 (8961-F78) director overview . . . . . . . . . . . . . . . . . . . . . 6
2.2.1 Hardware components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2.2 Port-side view of the device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2.3 Port-side slot numbering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2.4 Non-port side view of the device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2.5 Additional Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3 IBM b-type Gen 7 SAN256B-7 (8961-F74) director overview . . . . . . . . . . . . . . . . . . . . 13
2.3.1 Hardware components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.3.2 Port-side view of the device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.3.3 Port-side slot numbering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3.4 Non-port-side view of the device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.3.5 Additional Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.4 IBM b-type Gen 7 SAN64B-7 (8960-P64/R64) switch overview . . . . . . . . . . . . . . . . . . 18
2.4.1 Software License Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.4.2 Port-side view of the device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.4.3 Nonport-side view of the device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.4.4 Additional Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.5 Upgrading Gen 6 to Gen 7 supported platform overview . . . . . . . . . . . . . . . . . . . . . . . 22
2.5.1 Migration steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.6 IBM b-type Gen7 transceiver module overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Chapter 3. Gen 7 Fabric Operating System (FOS) overview . . . . . . . . . . . . . . . . . . . . . 25


3.1 Autonomous SAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.1.1 Self-learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.1.2 Self-optimizing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.1.3 Self-healing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.2 Automation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.2.1 Why organizations should embrace automation . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.3 FOS 9.x features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.3.1 Port identifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.3.2 Unified addressing mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

© Copyright IBM Corp. 2021. iii


8497TOC.fm Draft Document for Review April 5, 2021 12:31 pm

3.3.3 Fabric zone locking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31


3.3.4 FICON logical switch. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.4 Management tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.4.1 Web Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.5 Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.6 Traffic optimizer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.6.1 Speed-based classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.6.2 Slow drain device quarantine (SDDQ) isolation . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.6.3 Compatibility with QoS zones and CS_CTL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.6.4 Compatibility with previous platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.6.5 Traffic Optimizer and QoS Mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.7 Fabric Notifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.8 The IBM B-Type switch Fabric Notifications solution . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.8.1 Congestion signal primitives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.8.2 Extended link service notifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.8.3 EDC and RDF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.8.4 Congestion signal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.8.5 Congestion notification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.8.6 Congestion peer notification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.8.7 Link Integrity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.8.8 SCSI Command Delivery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

Chapter 4. Virtual Fabrics overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51


4.1 Logical switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.1.1 Default logical switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.1.2 Logical switches and fabric IDs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.1.3 Port assignment in logical switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.1.4 Logical switches and connected devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.1.5 Management model for logical switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.2 Logical fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.2.1 Logical fabric and ISLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.2.2 Base switch and extended ISLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
4.2.3 Account management and Virtual Fabrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.2.4 Setting up IP addresses for a logical switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.2.5 Logical switch login context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
4.2.6 Supported platforms for Virtual Fabrics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4.3 Device sharing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

Chapter 5. Fibre Channel SAN design, implementation, and migration . . . . . . . . . . . 69


5.1 Modernizing SAN infrastructures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
5.1.1 IBM b-type Gen 7 platforms: the intelligent foundation of modern data centers . . 70
5.2 Fibre Channel SAN design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.2.1 SAN design basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.2.2 Capabilities you need to consider for your storage architecture . . . . . . . . . . . . . 73
5.2.3 Architecting a SAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
5.2.4 High-performance workloads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
5.2.5 Redundancy and resiliency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
5.2.6 Switch interconnections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
5.2.7 UltraScale ICL connectivity for IBM b-type Gen 7 platforms . . . . . . . . . . . . . . . . . 79
5.2.8 Best practices for IBM b-type Gen 7 UltraScale ICL connections. . . . . . . . . . . . . 80
5.2.9 Mesh topology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
5.3 Implementation planning for a Fibre Channel SAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
5.3.1 Device placement and traffic locality. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

iv IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497TOC.fm

5.3.2 Scalability and performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86


5.3.3 SAN design for critical workloads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
5.3.4 Capacity planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
5.4 Migration strategies of Fibre Channel SAN infrastructure . . . . . . . . . . . . . . . . . . . . . . . 90
5.4.1 IBM b-type Gen 7 provide infrastructure for today and tomorrow . . . . . . . . . . . . . 90
5.4.2 Tools to assist in migration planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
5.4.3 SAN migration factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
5.4.4 SAN migration – process overview: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
5.5 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

Chapter 6. FICON design, implementation and migration. . . . . . . . . . . . . . . . . . . . . . 101


6.1 Important support notes for Gen 7 platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
6.1.1 Product Support Dates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
6.1.2 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
6.1.3 Link address refresher . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
6.2 FICON SAN design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
6.2.1 Typical redundant pathing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
6.2.2 Cascaded performance and virtual channels . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
6.2.3 Ultra-low latency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
6.2.4 Channel paths: 32 vs 48 port cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
6.2.5 Chassis configuration options using 48 port cards . . . . . . . . . . . . . . . . . . . . . . . 111
6.2.6 Two logical FICON switches per chassis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
6.2.7 ISL B/W sharing with logical switches. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
6.3 FICON SAN implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
6.3.1 Chassis level management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
6.3.2 FICON display mode setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
6.3.3 FICON fabric configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
6.3.4 Managing FICON fabrics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
6.3.5 Converting a production switch to a FICON logical switch . . . . . . . . . . . . . . . . . 127
6.4 FICON SAN migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
6.4.1 Firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
6.4.2 FICON logical switch. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
6.4.3 Inter Chassis Links (ICL). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
6.4.4 Cabling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
6.4.5 Patch room & fibre plant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
6.4.6 SFPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration . . . 137
7.1 Extension platform overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
7.1.1 IBM b-type Gen 7 extension platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
7.2 Extension design and architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
7.2.1 FCIP design and architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
7.2.2 IP Extension design and architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
7.3 Extension implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
7.3.1 The WAN Side . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
7.3.2 The FC/FICON side . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
7.3.3 The LAN Side (IP Extension) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
7.4 Extension migration strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
7.4.1 IBM Extension Platforms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
7.4.2 IBM Extension Solution Designs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
7.4.3 Migration methodology and techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
7.4.4 Migration Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
7.5 Extension validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198

Contents v
8497TOC.fm Draft Document for Review April 5, 2021 12:31 pm

7.6 Extension Monitoring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204


7.6.1 IBM SANnav SAN Management Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
7.6.2 Fabric OS CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
7.6.3 REST API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
7.6.4 SNMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210

Chapter 8. NVMe over FC design, implementation and migration strategies . . . . . . 213


8.1 NVMe overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
8.1.1 Comparing mapping of NVMe versus FCP protocols . . . . . . . . . . . . . . . . . . . . . 216
8.1.2 NVMe over Fibre Channel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
8.2 NVMe over Fibre Channel – SAN design considerations . . . . . . . . . . . . . . . . . . . . . . 220
8.3 NVMe over Fibre Channel –implementation overview . . . . . . . . . . . . . . . . . . . . . . . . 221
8.4 NVMe over Fibre Channel - migration example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
8.5 NVMe over FC - summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226

Chapter 9. Fabric OS administrative tools and diagnostics . . . . . . . . . . . . . . . . . . . . 227


9.1 Command-line interface (CLI) overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
9.1.1 Connecting to fabric OS through the serial port . . . . . . . . . . . . . . . . . . . . . . . . . 228
9.1.2 Connecting to Fabric operating system through SSH or telnet connection. . . . . 229
9.2 9.2 Web tools overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
9.3 Fabric Vision overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
9.3.1 Monitoring, alerting, and performance suite (MAPS) . . . . . . . . . . . . . . . . . . . . . 231
9.3.2 Flow Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
9.3.3 ClearLink Diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
9.3.4 Fabric Vision SANnav Managed Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
9.3.5 IO Insight . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
9.3.6 VM Insight . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
9.3.7 Fabric Vision Diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242

Chapter 10. IBM b-type Gen 7 SAN automation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243


10.1 Motivation to automate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
10.2 Overview of the REST API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
10.3 Simple Automation Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
10.4 Ansible as an alternative . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
10.5 SANnav’s REST API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
10.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250

Chapter 11. IBM SANnav Management Suite overview . . . . . . . . . . . . . . . . . . . . . . . . 251


11.1 IBM SANnav Management Suite 2.1 release overview . . . . . . . . . . . . . . . . . . . . . . 252
11.1.1 IBM SANnav Management Portal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
11.1.2 IBM SANnav Global View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
11.1.3 New Hardware Platforms supported in SANnav 2.1 . . . . . . . . . . . . . . . . . . . . . 253
11.2 IBM SANnav Management Portal 2.1 release overview . . . . . . . . . . . . . . . . . . . . . . 254
11.2.1 Browser requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
11.2.2 New SANnav Management Portal server platform support and Infrastructure . 254
11.2.3 Server requirements details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
11.2.4 SANnav Management Portal scalability 2.1.0 . . . . . . . . . . . . . . . . . . . . . . . . . . 255
11.2.5 Pre-installation requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
11.2.6 Downloading the SANnav Management Portal software . . . . . . . . . . . . . . . . . 256
11.2.7 SANnav installation and migration QuickStart checklists . . . . . . . . . . . . . . . . . 264
11.2.8 Installation overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
11.2.9 SANnav Management Portal deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
11.2.10 System and server requirements for SANnav Management Portal . . . . . . . . 266
11.2.11 Installation prerequisites for SANnav Management Portal . . . . . . . . . . . . . . . 267

vi IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497TOC.fm

11.2.12 Installing SANnav Management Portal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269


11.2.13 Firewall requirements for SANnav Management Portal . . . . . . . . . . . . . . . . . 270
11.2.14 SANnav Management Console commands . . . . . . . . . . . . . . . . . . . . . . . . . . 271
11.2.15 Checking the server health . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
11.2.16 Launching SANnav Management Portal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
11.2.17 Changing your password . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
11.2.18 Overview of the user interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
11.2.19 Initial setup and configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
11.3 IBM SANnav Global View 2.1 release overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
11.3.1 Browser requirements for SANnav Global View . . . . . . . . . . . . . . . . . . . . . . . . 278
11.3.2 System and Server Requirements for SANnav Global View. . . . . . . . . . . . . . . 278
11.3.3 Installation prerequisites for SANnav Global View . . . . . . . . . . . . . . . . . . . . . . 279
11.3.4 Installing SANnav Global View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
11.3.5 Logging in to SANnav Global View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
11.3.6 Quick tour of SANnav Global View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
11.3.7 Generating a license for SANnav Global View . . . . . . . . . . . . . . . . . . . . . . . . . 283
11.3.8 Global dashboard overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
11.3.9 Monitoring fabric health. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
11.3.10 Monitoring switch health across Management Portal instances . . . . . . . . . . . 287
11.4 IBM SANnav Web Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
11.4.1 Overview of the Web Tools user interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290
11.5 Reference guides . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291

Chapter 12. SAN security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293


12.1 Management Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295
12.2 Displaying IP Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295
12.3 Role-Based access controls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
12.4 Default accounts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
12.5 User accounts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
12.6 Security Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
12.7 Access Control Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
12.8 SAN Infrastructure – Securing End Points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
12.9 ISL In-Flight Encryption and Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
12.10 Policy Database Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304

Appendix A. Fabric details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305


A.1 Current fabrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306
A.2 Individual fabrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306
A.3 Device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
A.4 Metrics and effect on SAN design and performance . . . . . . . . . . . . . . . . . . . . . . . . . 307
A.5 Consolidated SAN requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
A.6 Application-specific information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309

Appendix B. FOS CLI monitoring commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311


B.1 portshow (tunnel and circuits) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
B.2 portshow (LAN Stats) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
B.3 portshow (TCP Stats) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
B.4 portStatsShow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
B.5 gePortPerfShow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
B.6 switchShow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
B.7 portPerfShow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
B.8 portErrShow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322

Contents vii
8497TOC.fm Draft Document for Review April 5, 2021 12:31 pm

viii IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497spec.fm

Notices

This information was developed for products and services offered in the US. This material might be available
from IBM in other languages. However, you may be required to own a copy of the product or product version in
that language in order to access it.

IBM may not offer the products, services, or features discussed in this document in other countries. Consult
your local IBM representative for information on the products and services currently available in your area. Any
reference to an IBM product, program, or service is not intended to state or imply that only that IBM product,
program, or service may be used. Any functionally equivalent product, program, or service that does not
infringe any IBM intellectual property right may be used instead. However, it is the user’s responsibility to
evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document. The
furnishing of this document does not grant you any license to these patents. You can send license inquiries, in
writing, to:
IBM Director of Licensing, IBM Corporation, North Castle Drive, MD-NC119, Armonk, NY 10504-1785, US

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION “AS IS”


WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A
PARTICULAR PURPOSE. Some jurisdictions do not allow disclaimer of express or implied warranties in
certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically made
to the information herein; these changes will be incorporated in new editions of the publication. IBM may make
improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time
without notice.

Any references in this information to non-IBM websites are provided for convenience only and do not in any
manner serve as an endorsement of those websites. The materials at those websites are not part of the
materials for this IBM product and use of those websites is at your own risk.

IBM may use or distribute any of the information you provide in any way it believes appropriate without
incurring any obligation to you.

The performance data and client examples cited are presented for illustrative purposes only. Actual
performance results may vary depending on specific configurations and operating conditions.

Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products and cannot confirm the
accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the
capabilities of non-IBM products should be addressed to the suppliers of those products.

Statements regarding IBM’s future direction or intent are subject to change or withdrawal without notice, and
represent goals and objectives only.

This information contains examples of data and reports used in daily business operations. To illustrate them
as completely as possible, the examples include the names of individuals, companies, brands, and products.
All of these names are fictitious and any similarity to actual people or business enterprises is entirely
coincidental.

COPYRIGHT LICENSE:

This information contains sample application programs in source language, which illustrate programming
techniques on various operating platforms. You may copy, modify, and distribute these sample programs in
any form without payment to IBM, for the purposes of developing, using, marketing or distributing application
programs conforming to the application programming interface for the operating platform for which the sample
programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore,
cannot guarantee or imply reliability, serviceability, or function of these programs. The sample programs are
provided “AS IS”, without warranty of any kind. IBM shall not be liable for any damages arising out of your use
of the sample programs.

© Copyright IBM Corp. 2021. ix


8497spec.fm Draft Document for Review April 5, 2021 12:31 pm

Trademarks
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines
Corporation, registered in many jurisdictions worldwide. Other product and service names might be
trademarks of IBM or other companies. A current list of IBM trademarks is available on the web at “Copyright
and trademark information” at http://www.ibm.com/legal/copytrade.shtml

The following terms are trademarks or registered trademarks of International Business Machines Corporation,
and might also be trademarks or registered trademarks in other countries.
AIX® IBM Spectrum® z/OS®
FICON® Redbooks® z15™
IBM® Redbooks (logo) ®
IBM FlashSystem® System Z®

The following terms are trademarks of other companies:

Intel, Intel logo, Intel Inside logo, and Intel Centrino logo are trademarks or registered trademarks of Intel
Corporation or its subsidiaries in the United States and other countries.

The registered trademark Linux® is used pursuant to a sublicense from the Linux Foundation, the exclusive
licensee of Linus Torvalds, owner of the mark on a worldwide basis.

Microsoft, Windows, and the Windows logo are trademarks of Microsoft Corporation in the United States,
other countries, or both.

Ansible, Red Hat, are trademarks or registered trademarks of Red Hat, Inc. or its subsidiaries in the United
States and other countries.

UNIX is a registered trademark of The Open Group in the United States and other countries.

VMware, and the VMware logo are registered trademarks or trademarks of VMware, Inc. or its subsidiaries in
the United States and/or other jurisdictions.

Other company, product, or service names may be trademarks or service marks of others.

x IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497pref.fm

Preface

The IBM® b-type Gen 7 Fibre Channel directors and switches provide reliable, scalable, and
secure high-performance foundations for high-density server virtualization, cloud
architectures, and next generation flash and SSD storage. They are designed to meet the
demands of highly virtualized private cloud storage and data center environments.

This IBM Redbooks® publication helps administrators understand how to implement or


migrate to an IBM b-type Gen 7 SAN. It provides an overview of the key hardware and
software products and explains how to install, monitor, tune, and troubleshoot your storage
area network (SAN).

Read this publication to learn about fabric design, managing and monitoring your SAN
environment.

Authors
This book was produced by a team of specialists from around the world working at IBM
Redbooks, Poughkeepsie Center.

Brian Larsen joined Broadcom in July 1991 and has more


than 35 years of professional experience in high-end
processing, storage, disaster recovery, Cloud, virtualization
and networking environments. Larsen is the Director of Partner
Business Development responsible for solution and business
development within all IBM divisions. Larsen, in addition to his
14 years in Business Development, has held positions in
sales/consulting, product management and solutions
marketing

Kameswara Bhaskarabhatla joined Broadcom in Oct 2019 as


a Solutions Architect and has more than 20 years of
professional experience. Before joining the Broadcom, he
worked for The Trainline and IBM as technical lead for storage.
Kamesh reviews storage environments to provide the preferred
Best Practices and architectural decisions that improves the
overall availability and reliability.

© Copyright IBM Corp. 2021. xi


8497pref.fm Draft Document for Review April 5, 2021 12:31 pm

Matt Brennan joined Broadcom in February 2000 and has


more than 35 years of professional experience in development,
design, and deployment of storage devices and networks. Matt
currently supports the qualification of Broadcom products for
IBM. Matt has previously worked as software developer, a
support engineer, and a field sales engineer for storage arrays
and networking products.

AJ Casamento s a 40+ year IT industry veteran with


experience working in a wide variety of market segments, IT
leading companies, and engineering and product development
roles. In more than 23 years as a Solutions Architect at
Brocade, a Broadcom Inc. Company, AJ spends his time
helping enterprises understand various technologies and
architectural decisions, and the impact to their business. AJ
started his career with Digital Equipment Corporation, and
worked in various engineering and product development roles
at Hewlett Packard, SUN Microsystems, IBM, and Avnet,
before joining Brocade.

Jack Consoli has more than 35 years of professional


experience in high-end processing, storage, disaster recovery,
and networking with an emphasis on mainframe environments.
Jack is a co-author of System z End-to-End Extended Distance
Guide, SG24-8047 and a contributor to FICON Planning and
Implementation Guide, SG24-6497. In addition to sales roles
as a Field Applications Engineer and Systems Engineer,
Consoli has held positions in product management,
engineering management, and development engineering.

Mark Detrick is a Principal Solutions Architect, having worked


20 years for Broadcom. Involved with many Fortune 500
large-scale IBM projects worldwide, Mark plays a consultative
role in storage network design and deployment. Mark is an
expert in many data center technologies, including mainframe
SAN design and extension; distributed systems large-scale
SAN design, FC Routing, and extension; IP routing and
Ethernet switching; and WAN technologies. Mark has over 30
years of experience in data center networking environments.
He understands Broadcom’s ASIC technology and is a
consultative resource within Brocade and externally to
customers, OEMs, and resellers. Mark applies his in-depth
technical knowledge of fabrics, protocols, flow control, TCP/IP,
and large-scale SAN/LAN architectures. Mark is adept at
next-generation data center solutions, including cloud and the
many types of virtualizations found in data centers.

xii IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497pref.fm

Johnny Fonseca has 20+ years of professional experience in


IP and optical communications networking, storage, and
disaster recovery solutions. As Field Application Engineer, he's
taken the lead in designing, implementing, and optimizing
storage networks in both Open Systems and Mainframe
environments around the world. Customers seek his expertise
and consider him a trusted advisor. Johnny began his career in
networking with the United States Marine Corps and held
senior engineering positions at NTT America, Cisco, McDATA
and Brocade before joining Broadcom (via Brocade's
acquisition in 2017).

Tim Jeka leads the technical and positioning effort of


Brocade's storage networking business with IBM, as an IBM
North America Systems Engineer of Storage Networking. Tim
is responsible for driving technical awareness and positioning
of Brocade's storage networking products to the IBM field
across the Americas. He joined Brocade in 2008 bringing over
34 plus years of experience in Networking, Storage, and SAN
technology with IBM. Prior to joining Brocade, Tim was a
Networking and Storage Area Networking RDS at IBM. Tim
also held Technical Team leadership roles in IBM's Storage and
Networking product divisions for 34 plus years with IBM.

Thanks to the following people for their contributions to this project:


򐂰 Silviano Gaona
OEM Field Application Engineer, Partner solutions.
Brocade Storage Networking (BSN), Broadcom, Inc.
򐂰 Brandon Hoff
Principal Software Architect
Brocade Storage Networking (BSN), Broadcom, Inc.
򐂰 Marcus Thordal
Principal Solution Architect
Brocade Storage Networking (BSN), Broadcom, Inc.

Now you can become a published author, too!


Here’s an opportunity to spotlight your skills, grow your career, and become a published
author—all at the same time! Join an IBM Redbooks residency project and help write a book
in your area of expertise, while honing your experience using leading-edge technologies. Your
efforts will help to increase product acceptance and customer satisfaction, as you expand
your network of technical contacts and relationships. Residencies run from two to six weeks
in length, and you can participate either in person or as a remote resident working from your
home base.

Find out more about the residency program, browse the residency index, and apply online at:
ibm.com/redbooks/residencies.html

Preface xiii
8497pref.fm Draft Document for Review April 5, 2021 12:31 pm

Comments welcome
Your comments are important to us!

We want our books to be as helpful as possible. Send us your comments about this book or
other IBM Redbooks publications in one of the following ways:
򐂰 Use the online Contact us review Redbooks form found at:
ibm.com/redbooks
򐂰 Send your comments in an email to:
[email protected]
򐂰 Mail your comments to:
IBM Corporation, IBM Redbooks
Dept. HYTD Mail Station P099
2455 South Road
Poughkeepsie, NY 12601-5400

Stay connected to IBM Redbooks


򐂰 Find us on LinkedIn:
http://www.linkedin.com/groups?home=&gid=2130806
򐂰 Explore new Redbooks publications, residencies, and workshops with the IBM Redbooks
weekly newsletter:
https://www.redbooks.ibm.com/Redbooks.nsf/subscribe?OpenForm
򐂰 Stay current on recent Redbooks publications with RSS Feeds:
http://www.redbooks.ibm.com/rss.html

xiv IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch01.fm

Chapter 1. Introduction to IBM b-type Gen 7


This chapter describes the IBM b-type Gen 7 family of products. It includes an overview of the
Gen 7 technical values that have been designed into the product(s) for this ever changing
world. In addition, an overview of naming conventions will be provided to assist and clarify the
differences between an IBM b-type Gen 7 offering versus a Brocade offering.

This chapter includes the following topics:


򐂰 1.1, “Overview of IBM b-type Gen 7 platforms” on page 2
򐂰 1.2, “IBM b-type Gen 7 platforms and naming conventions” on page 4

© Copyright IBM Corp. 2021. 1


8497ch01.fm Draft Document for Review April 5, 2021 12:31 pm

1.1 Overview of IBM b-type Gen 7 platforms


As technology evolves, IT organizations are held to a higher set of standards. Not only are IT
organizations responsible for delivering nonstop reliability, they are now judged on how
quickly they can deploy new services to the business. The only way to accelerate IT delivery
and keep pace with ever-increasing demands on existing infrastructure is to automate. The
infrastructure must be able to monitor application performance, identify network congestion,
and prioritize bandwidth. The infrastructure also needs to isolate configuration errors or
malfunctioning devices automatically, regardless of the source of the issue in the data center.

IBM b-type Gen 7 technology delivers a collection of features that combine comprehensive
data collection capabilities with powerful analytics to quickly understand the health and
performance on the environment and identify potential impact or trending problems. These
features are the foundation for realizing a self-learning, self-optimizing, and self-healing
autonomous SAN. IBM b-type Gen 7 technology provides unprecedented visibility and
actionable intelligence across the storage network. The information captured is displayed in
IBM SANnav Management Portal to quickly identify and isolate problems before they impact
application availability. The combination of these features gives IT administrators the ability to
quickly enable an autonomous infrastructure.

1.1.1 Self-learning
IBM b-type Gen 7 technology proactively monitors millions of I/O performance and behavior
data points through integrated network sensors to gain deep insight into the environment.
These data points are transformed into actionable intelligence that can monitor and alert
when there are any abnormal changes. Through these capabilities, an admin can identify
individual applications and their performance characteristics across the fabric, as well as
identify the performance of the various devices that comprise the fabric (the switches, hosts,
and targets):
򐂰 I/O Insight: Proactively monitors I/O performance and behavior through integrated network
sensors to baseline application performance and ensure operational stability. By
combining the instrumentation of IO Insight with the ability to self-learn the traffic flows, the
IBM b-type Gen 7 autonomous SAN technology can generate performance and health
metrics on each component as well as the applications to monitor for changes and alert
the administrator with MAPS.
򐂰 Monitoring and Alerting Policy Suite (MAPS): Provides an easy-to- use solution for
policy-based threshold monitoring and alerting. MAPS proactively monitors the health and
performance of any SCSI or NVMe storage infrastructure to ensure application uptime and
availability.

By leveraging prebuilt rule-based and policy-based templates, MAPS simplifies fabric-wide


threshold configuration, monitoring, and alerting.
򐂰 Automatic Flow Learning: Provides automatic learning of all traffic flows from a specific
host to storage across the SAN fabric. With this information, an admin can automatically
identify resource contention or congestion that is impacting application performance.
򐂰 Fabric Performance Impact: Leverages predefined MAPS policies to automatically detect
and alert administrators to different congestion severity levels and to identify credit-stalled
devices (for example, misbehaving slow drain devices) or oversubscribed ports that could
impact network performance. This feature pinpoints exactly which devices are causing or
are impacted by the congested port, and it quarantines the misbehaving devices.

2 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch01.fm

1.1.2 Self-optimizing
Utilizing actionable intelligence gathered from self-learning capabilities, IBM b-type Gen 7
Fibre Channel SANs automatically apply priorities for specific data traffic to help guarantee
performance levels and monitor for traffic pattern shifts. Learning traffic behavior enables the
network to make smarter decisions on traffic prioritization, congestion avoidance, and
adjustment to ensure optimal network performance for applications and storage. When
something does change, the IBM b-type Gen 7 autonomous SAN technology will isolate the
port traffic for the misbehaving device to a virtual channel in the fabric and allow all other
traffic to go around to maintain optimal performance. In addition, IBM SANnav Management
Portal has automated manual activities such as infrastructure deployment and provisioning to
expedite IT services. Self-optimizing features include the following:
򐂰 Traffic Optimizer: Automatically classifies and separates traffic with similar characteristics
to optimize performance for most common SAN configurations. It identifies and isolates
traffic flows to prevent negative impact to overall SAN performance.
򐂰 Advanced traffic shaping: Guarantees application performance by proactively monitoring
and actively shaping traffic.
򐂰 REST APIs: Eliminate human errors and performance impacts through open DevOps.
򐂰 SAN Automation and Ansible: Gain cloud-like SAN orchestration for optimizing
administration resources through IBM b-type Gen 7 REST API automation technology.

1.1.3 Self-healing
When potential disruptions are detected, the network will automatically mitigate or resolve
issues without manual intervention. This is done by proactively monitoring the network to
automatically identify abnormal or unexpected infrastructure behavior and then taking
immediate action. These actions include alerting the end devices of the issue through a
notification and signaling process within a SAN. End devices can automatically adjust traffic
or fail over to a healthy path to mitigate impact until a further infrastructure change solution
can be implemented. In addition, these capabilities detect and automatically reconfigure
fabric misconfigurations that fall outside best practices Self-healing features include:
򐂰 Fabric congestion notification: Automatically detects congestion and notifies end devices
to automatically mitigate congestion.
򐂰 Slow drain device quarantine: Automatically quarantines credit-stalled devices to prevent
the misbehaving device from impacting the rest of traffic.
򐂰 Automatic actions: Ensure data delivery with automatic failover from physical or
congestion issues such as port decommissioning, port toggling, and port fencing.
򐂰 COMPASS: Detects and automatically reconfigures out-of-compliance configurations.
򐂰 SANnav Management Portal: Reduces troubleshooting steps with built-in best practice
recommendations to quickly resolve issues.

Through the modernization process, enterprises are facing the reality that their revenue is
intertwined with the success or failure of the IT organization. To succeed, IT needs to
automate as much as possible to eliminate complexity and reduce costs. IT organizations
need to remove tedious, time-consuming, and labor-intensive tasks so they can focus on
delivering services to the business that can help deliver additional revenue. IBM b-type Gen
7’s autonomous SAN delivers automation and intelligence to admins so they do not have to
worry about the SAN and instead can focus on initiatives that are strategic to their
organization.

Chapter 1. Introduction to IBM b-type Gen 7 3


8497ch01.fm Draft Document for Review April 5, 2021 12:31 pm

1.2 IBM b-type Gen 7 platforms and naming conventions


Figure 1-1 shows the IBM b-type Gen 7 platforms: 8-slot (IBM SAN512B-7), 4-slot
(IBM SAN256B-7) directors and the mid-range (IBM SAN64B-7) switch.

Figure 1-1 IBM b-type Gen 7 platforms

Table 1-1 lists the IBM b-type Gen 7 and Brocade naming conventions for the Gen 7 platform
offerings.

Table 1-1 IBM b-type Gen 7 platform naming conventions


IBM b-type Gen 7 name IBM machine type & model Brocade name

IBM Storage Networking SAN512B-7 8961-F78 X7-8

IBM Storage Networking SAN256B-7 8961-F74 X7-4

IBM Storage Networking SAN64B-7 8960-P64 and 8960-R64 G720

4 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch02.fm

Chapter 2. IBM b-type Gen 7 product review


This chapter provides a hardware review of each Gen 7 director, supported blades and
mid-range switch. New to the Gen 7 offering is the ability to upgrade from a Gen 6 Director to
a Gen 7 supported director. An overview of the Gen 6 upgrade path to a Gen 7 supported
platform along with supported blades will be provided. The last section of this chapter will
review the supported transceiver modules for the Gen 7 platforms.

This chapter includes the following topics:


򐂰 2.1, “IBM b-type Gen 7 SAN director overview” on page 6
򐂰 2.2, “IBM b-type Gen 7 SAN512B-7 (8961-F78) director overview” on page 6
򐂰 2.3, “IBM b-type Gen 7 SAN256B-7 (8961-F74) director overview” on page 13
򐂰 2.4, “IBM b-type Gen 7 SAN64B-7 (8960-P64/R64) switch overview” on page 18
򐂰 2.5, “Upgrading Gen 6 to Gen 7 supported platform overview” on page 22
򐂰 2.6, “IBM b-type Gen7 transceiver module overview” on page 23

© Copyright IBM Corp. 2021. 5


8497ch02.fm Draft Document for Review April 5, 2021 12:31 pm

2.1 IBM b-type Gen 7 SAN director overview


Technology is evolving at an incredible pace, and businesses are demanding more from their
IT resources and infrastructure. Rapid adoption of flash storage and the ramp-up of
NVMe-based storage unleash advancements in application design that drive new levels of
performance and capacity requirements, such as advanced analytics, business intelligence,
and data-intensive workloads. With the integration of new technologies that are accelerating
the delivery of data and services, the network will need to evolve to keep pace with
innovations in storage and demands of modern applications to maximize the full value of
investments in next-generation data center infrastructure.

To meet ever-increasing demands for faster, more reliable data access, it is essential for
organizations to deploy a modernized infrastructure that reduces latency, increases
bandwidth, and ensures continuous availability. Unprecedented performance is not enough
on its own. Powerful analytics and advanced automation capabilities are required to transform
current storage networks into an autonomous SAN. This requires a network that is capable of
delivering these capabilities to maximize performance, simplify management, and reduce
operational costs. Legacy infrastructure was not designed to support the performance
requirements of evolving workloads and NVMe-based storage. In fact, an aging network will
impede the performance of the on-demand data center. By modernizing the storage network
with IBM b-type Gen 7, organizations will enable a faster, more intelligent, and more resilient
network. This will maximize the performance, productivity, and efficiency of their storage
investments and resources, even as they rapidly scale their environments.

The IBM b-type Gen 7 Director provides a modular building block, purpose-built for scalability
to accommodate growth and power large-scale storage environments. With a 50% latency
reduction compared to the previous generation, IBM b-type Gen 7 Directors maximize the
performance of NVMe storage and high-transaction workloads, eliminating I/O bottlenecks
and unleashing the full performance of next-generation storage. In addition, the IBM b-type
Gen 7 Director lays the foundation for the autonomous SAN. With autonomous SAN
technology, the director harnesses the power of analytics and the simplicity of automation to
optimize performance, ensure reliability, and simplify management. Leveraging these
capabilities enables organizations to realize a self-learning, self-optimizing, and self-healing
SAN. IBM b-type Gen 7 Directors provide up to 384 64Gb/s line rate ports or up to 512
32Gb/s line rate ports, enabling organizations to scale more devices, applications, and
workloads. With diverse deployment options, multiprotocol flexibility, and mixed blade
capability, organizations can adapt and optimize their businesses to meet next-generation
storage and server requirements. The IBM b-type Gen 7 Director supports the concurrent use
of both traditional Fibre Channel and NVMe storage traffic, allowing organizations to
seamlessly integrate IBM b-type Gen 7 Fibre Channel networks with next-generation
NVMe-based storage, without a disruptive rip-and-replace.

2.2 IBM b-type Gen 7 SAN512B-7 (8961-F78) director overview


Designed to meet continuous data growth and critical application demands, the IBM b-type
Gen 7 Director is purpose-built to power largescale storage environments that require
increased capacity, greater throughput, and higher levels of resiliency and operational
efficiency. This modular building block enables organizations to build the highest performing
data center SAN fabric that is required for all-flash and NVMe storage environments. The IBM
b-type Gen 7 Director modular design provides flexibility with two customizable chassis that
can scale on-demand for more devices, applications, and workloads. Both chassis utilize IBM
b-type Gen 7 UltraScale ICL technology to scale out modular SANs while preserving blade

6 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch02.fm

ports for device connectivity and allowing flexible SAN design that supports core-edge or
mesh topologies.

The 14U IBM b-type Gen 7 SAN512B-7 Director is built for large enterprise networks and has
eight vertical blade slots to provide up to 384 64Gb/s line rate ports or up to 512 32Gb/s line
rate ports for device connectivity. An additional 32 UltraScale InterChassis Link (ICL)
connections provide 128 ports for chassis-to-chassis interconnect.

The 8U IBM b-type Gen 7-4 Director in built for midsize networks and has four horizontal
blade slots to provide up to 192 64Gb/s line rate ports or up to 256 32Gb/s line rate ports for
device connectivity.

An additional 16 UltraScale ICL connections provide 64 ports for chassis-to-chassis


interconnect. Each blade slot within the IBM b-type Gen 7 chassis can be populated with
optional port or extension blades. For device connectivity, the following blades are available:
򐂰 IBM b-type Gen 7 FC64-48 Fibre Channel port blade provides 48 x 64Gb/s Fibre Channel
ports with backward-compatibility support for 8, 10, 16, and 32Gb/s Fibre Channel
connectivity.
򐂰 IBM b-type Gen 7 FC32-X7-48 Fibre Channel port blade provides 48 x 32Gb/s Fibre
Channel ports with backward-compatibility support for 4, 8, 10, and 16Gb/s Fibre Channel
connectivity.
򐂰 IBM b-type Gen 7 FC32-64 Fibre Channel port blade provides 64 x 32Gb/s Fibre Channel
ports with backward-compatibility support for 4, 8, and 16Gb/s Fibre Channel connectivity
as well as support for 10, 25, and 40GbE FCoE connectivity.

To support disaster recovery and data protection storage solutions over long distances, the
IBM b-type Gen 7 SX6 Extension Blade provides flexible Fibre Channel and IP storage
replication deployment options with 16 32Gb/s Fibre Channel ports, 16 1/10-GbE ports, and 2
40GbE ports. This blade allows organizations to seamlessly integrate extension capabilities
within the IBM b-type Gen 7 Director to provide replication services for large-scale, multisite
data center environments that implement block, file, and tape data protection solutions. The
IBM b-type Gen 7 SX6 Extension Blade can be deployed with the IBM b-type Gen 7 7840
Extension Switch and the IBM b-type Gen 7 7810 Extension Switch in a data-center to-edge
architecture as a cost effective option for connecting primary data centers with remote data
centers and offices.

IBM b-type Gen 7 directors build upon years of innovation and leverage the core technology
of IBM b-type Gen 7 systems to consistently deliver five-nines availability in the world’s most
demanding data centers. Delivering non-disruptive software upgrades, hot-pluggable
components, and a no-single-point-of-failure design, the IBM b-type Gen 7 offers a highly
resilient solution for today’s enterprise-class storage environments.

Product Features Key product features for this device include the following:
򐂰 Redundant and hot-swappable SFP+, SFP28, QSFP+, and QSFP56 transceivers; port,
extension, control processor (CP), and core routing (CR) blades; power supply
assemblies, fan assemblies, and WWN cards that enable a high availability platform and
allow non-disruptive software upgrades for mission-critical SAN applications.
򐂰 Support for the following features when using the FC32-64 blade:
– Up to 16 QSFP ports on each FC32-64 port blade that provide 4x32Gb/s, 4x16Gb/s,
4x8Gb/s, 4x4Gb/s, 4x25GbE, or 40GbE operation. Port speeds for 4x32Gb/s
transceivers can auto-negotiate between 4x32Gb/s and 4x16Gb/s. Port speeds for
4x16Gb/s transceivers can auto-negotiate between 4x16Gb/s, 4x8Gb/s, and 4x4Gb/s.

Chapter 2. IBM b-type Gen 7 product review 7


8497ch02.fm Draft Document for Review April 5, 2021 12:31 pm

– Flexport technology that allows you to configure each of the 16 QSFP ports on a blade
for FC operation or Ethernet operation for FCoE connections. For details on supported
transceivers and speeds, see Supported Transceivers Matrix
https://www.broadcom.com/products/fibre-channel-networking/transceiver-modul
es.
– Up to four blades and all FC transceivers installed, up to 256 32Gb/s external ports are
possible in a single chassis, enabling high-density SAN configurations with a reduced
footprint.
– Trunking technology that allows groups of up to eight ports to create high-performance
256Gb/s ISL trunks between switches using 32Gb/s ports.
򐂰 Support for the following features when using the FC64-48 blade:
– Port blade with 48 front-end 64Gb/s FC SFP+ ports with the edge switch interconnect
to the core blades.
– Supported SFPs include 10Gb/s, 32Gb/s, and 64Gb/s FC.
– Support for 48 Fibre Channel ports on each FC64-48 port blade that provide 64Gb/s,
32Gb/s, 16Gb/s, 10Gb/ s, and 8Gb/s. The 10Gb/s transceivers can be used for any
port on the FC64-48 port blades. Note that 10Gb/s transceivers on the FC64-48 and
SX6 blade are not interchangeable. For details on supported transceivers and speeds,
see Supported Transceiver Matrix
https://www.broadcom.com/products/fibre-channel-networking/transceiver-modul
es.
– Trunking technology groups up to eight ports to create high-performance 512Gb/s ISL
trunks between switches using 64Gb/s ports.
򐂰 Support for the following features when using the FC32-X7-48 blade:
– Port blade with 48 front-end 32Gb/s FC SFP+ ports.
– Supported SFPs include 32Gb/s, 16Gb/s, and 10Gb/s FC.
– Support for 48 Fibre Channel ports on each FC32-X7-48 port blade that provide
32Gb/s, 16Gb/s, 10Gb/s, 8Gb/s, and 4Gb/s. The 10Gb/s transceivers can be used for
any port on the FC32-X7-48 port blades. Note that 10Gb/s transceivers on the
FC32-X7-48 and SX6 blade are not interchangeable. For details on supported
transceivers and speeds, see Supported Transceiver Matrix
https://www.broadcom.com/products/fibre-channel-networking/transceiver-modul
es.
– Trunking technology groups up to eight ports to create high-performance 256Gb/s ISL
trunks between switches using 32Gb/s ports.
򐂰 Support for the following features when using the SX6 Blade:
– Support for 16 Fibre Channel ports supporting 4, 8, 16, and 32Gb/s.
– 16GbE ports supporting 1 or 10Gb/s.
– Two GbE ports supporting 40Gb/s on SX6 extension blades.

Note: The 10Gb/s transceivers on the FC64-48 and the FC32-X7-48 are not
interchangeable with the IBM b-type Gen 7 SX6 Blade.

For details on supported transceivers and speeds, see Supported Transceiver Matrix
https://www.broadcom.com/products/fibre-channel-networking/transceiver-modul
es.
򐂰 Considerations for the IBM b-type Gen 7 SX6 Blade:

8 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch02.fm

– The SX6 extension blades perform as extension platforms to support Fibre Channel
(FC) and IBM FICON® data flows and IP-based storage data flows over an IP WAN.
– Universal ports that self-configure as E_Ports, F_Ports, EX_Ports, M_Ports (mirror
ports), and FICON ports.

Note: The 10Gb/s ports on the FC64-48 and FC32-X7-48 port blades can function
as E_Ports only.

򐂰 ClearLink Diagnostic port (D_Port) functionality on Fibre Channel ports.


򐂰 Data compression capabilities through the port blades when ports are configured as ISLs

2.2.1 Hardware components


The device has a modular and scalable mechanical construction that allows a wide range of
flexibility in installation, fabric design, and maintenance. The device can be mounted with the
cables facing either the front or the rear of the equipment rack. It consists of the following
hardware components:
򐂰 Up to eight slots for hot-swappable port blade assemblies, providing up to 384 64Gb/s
Fibre Channel ports if FC64-48 port blades are installed or 512 32Gb/s Fibre Channel
ports if FC32-64 port blades are installed. Flexport technology allows QSFP ports on this
blade to be configured for FCoE operation at 4x10GbE, 4x25GbE, and 40GbE speeds
when using an FC32-64 port blade. For a list of supported transceivers for these blades,
see Supported Transceiver Matrix
򐂰 Two half-size slots for control processor (CP) blades:
– A single active CP blade can control all the ports in the device.
– The standby CP blade assumes control of the device if the active CP blade fails.
򐂰 Two slots for core routing (CR) blades:
– A CR blade interconnects all port blades.
– The blades provide up to 32 Gen 7 QSFP56 (ICL) ports.
– ICL ports allow interconnection with neighboring director chassis.
– Both CR blades are active and can be hot-swapped.
For a list of supported transceivers for these blades, see Supported Transceiver Matrix
https://www.broadcom.com/products/fibre-channel-networking/transceiver-modul
es
򐂰 Up to four slots for modular, hot-swappable 34-port SX6 extension blades.
– Blades provide 16 32Gb/s Fibre Channel (FC) ports supporting 8, 16, and 32Gb/s or
16 32Gb/s FC ports supporting 4, 8, 16, and 32Gb/s FC ports. Extension blades
enable long-distance communication over an existing IP infrastructure. For a list of
supported transceivers for these blades, see Supported Transceiver Matrix
https://www.broadcom.com/products/fibre-channel-networking/transceiver-modul
es.
򐂰 Modular, hot-swappable field-replaceable units (FRUs):
– Three fan assemblies, available with NPI or NPE airflow.
– Up to four power supplies, available with NPI or NPE airflow. (refer to the following url
and the Documentation tab for all specifications outlined below.
https://www.broadcom.com/products/fibre-channel-networking/directors/x7-dire
ctors

Chapter 2. IBM b-type Gen 7 product review 9


8497ch02.fm Draft Document for Review April 5, 2021 12:31 pm

See the "Power Supply Specifications (per PSU)" in the IBM b-type Gen 7 Director
Technical Specifications for maximum output power, input voltage, input line frequency,
and other specifications for your power supply model.
See the "Power Supply Requirements" section in the IBM b-type Gen 7 Director Technical
Specifications for the minimum number of power supplies required for operation and
redundancy when different input voltages are applied, such as low line and high line AC.
See the "Power Consumption" sections in the IBM b-type Gen 7 Director Technical
Specifications for power output data and the minimum number of power supplies for
supported input voltages.
Redundant primary power connections ensure high availability. Each power supply
assembly has its own connector, so the number of primary power connections is four for
optimum efficiency and redundancy.
򐂰 Two World Wide Name (WWN) cards located on the non-port side of the device behind the
WWN card bezel.
򐂰 Port blades use small form-factor pluggable (SFP+, QSFP+, and QSFP28) optical
transceivers. For details on supported transceivers per blade type, see Supported
Transceiver Matrix
https://www.broadcom.com/products/fibre-channel-networking/transceiver-modules.
򐂰 Core routing blades use QSFP28 or QSFP56 optical transceivers. For a list of supported
transceivers for these blades, see Supported Transceiver Matrix
https://www.broadcom.com/products/fibre-channel-networking/transceiver-modules.
򐂰 Chassis door. This door must be installed to meet EMI compliance certification.
򐂰 A cable management comb. These combs install on the chassis below the blades for cable
management.

Note: Device control processors and management modules contain batteries for
RTC/NVRAM backup. Do not attempt to replace these batteries. Dispose of hardware
components that contain these batteries as required by local ordinances and
regulations.

2.2.2 Port-side view of the device


The port-side view in Figure 2-1 on page 11 shows the IBM SAN512B-7 Director with
installed blades. Components are identified as follows:
1. Air Vent
2. Core Routing Blades (CR64-8)
3. Port Blades (FC64-48 or FC32-X7-48)
4. Cable Management Comb
5. Control Processor Blades (CPX7) – Slot 1 (upper), Slot 2 (lower)

Note that the SX6 extension blades is not shown in Figure 2-1 on page 11, but would be
installed in the same slots as the FC32-X7-48 or FC64-64 port blades. A maximum of four
SX6 blades are supported.

10 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch02.fm

Figure 2-1 Port-Side View of IBM SAN512B-7 Director

2.2.3 Port-side slot numbering


The IBM B-TYPE SAN512B-7 Director contains 10 full-height slots and two half-height slots,
for a total of 12 slots. Facing the port side of the device, the half-height slots are on the left,
numbered 1 (top slot) and 2 (bottom slot). The remaining full-height slots are numbered 3
through 12, counting from left to right of chassis. Slots contain guide pins and connectors
designed for specific blade types.

Only install the control processor (CP), core routing (CR), port, and extension blades into slot
numbers as follows:
򐂰 Slots 1–2 are restricted to CP blades. Note that the blade installed in slot 1 will be
designated as CP0, while the blade in slot 2 will be designated as CP1 in CLI command
and message output.
򐂰 Slots 3–6 and slots 9–12 are restricted to port and extension blades.
򐂰 Slots 7–8 are restricted to CR64-8 core blades.

2.2.4 Non-port side view of the device


Figure 2-2 on page 12 shows the non-port-side view of the IBM SAN512B-7 Director with all
fan and power supply assemblies installed. Components are identified as follows:
1. WWN Bezel (Logo Plate - WWN Cards Behind)
2. Power Supply Assembly

Chapter 2. IBM b-type Gen 7 product review 11


8497ch02.fm Draft Document for Review April 5, 2021 12:31 pm

3. Fan Assembly
4. 2AWG Panduit LCD2-14AF Lug for Building Ground Connection

Note: Depending on the fans and power supplies that are installed, airflow can be from the
port side to the non-port side of the chassis or from the non-port side to the port side of the
chassis.

Figure 2-2 Non-port side of the IBM SAN512B-7 Director (example configuration)

Although not illustrated, the chassis label that contains the serial number, SKU, and
WWN is located on the lower portion of the chassis, below the fan assemblies.

2.2.5 Additional Information

To find all documentation for the IBM b-type Gen 7 SAN512B-7 (8961-F8) refer to the
documentation tab at this url.
https://www.broadcom.com/products/fibre-channel-networking/directors/x7-directors#
documentation

12 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch02.fm

2.3 IBM b-type Gen 7 SAN256B-7 (8961-F74) director overview


Key product features for this device include the following:
򐂰 Redundant and hot-swappable SFP+, SFP28, QSFP+, and Gen 7 QSFP transceivers;
port, extension, control processor (CP), and core routing (CR) blades; power supply
assemblies, fan assemblies, and WWN cards that enable a high-availability platform and
allow non-disruptive software upgrades for mission-critical SAN applications.
򐂰 Support for the following features when using the FC32-64 blade:
– Up to 16 QSFP ports on each FC32-64 port blade that provide 4x32Gb/s, 4x16Gb/s,
4x8Gb/s, 4x4Gb/s, 4x25GbE, or 40GbE operation. Port speeds for 4x32Gb/s
transceivers can auto-negotiate between 4x32Gb/s and 4x16Gb/s. Port speeds for
4x16Gb/s transceivers can auto-negotiate between 4x16Gb/s, 4x8Gb/s, and 4x4Gb/s.
– Flexport technology that allows you to configure each of the 16 QSFP ports on a blade
for FC operation or Ethernet operation for FCoE connections. For details on supported
transceivers and speeds, see Supported Transceiver Matrix
https://www.broadcom.com/products/fibre-channel-networking/transceiver-modul
es.
– Up to four blades and all FC transceivers installed, up to 256 32Gb/s external ports are
possible in a single chassis, enabling high-density SAN configurations with a reduced
footprint.
– Trunking technology that allows groups of up to eight ports to create high-performance
256Gb/s ISL trunks between switches using 32Gb/s ports.
򐂰 Support for the following features when using the FC64-48 blade:
– Port blade with 48 front-end 64Gb/s FC SFP+ ports with the edge switch interconnect
to the core blades.
– Supported SFPs include 10Gb/s, 32Gb/s, and 64Gb/s FC.
– Support for 48 Fibre Channel ports on each FC64-48 port blade that provide 64Gb/s,
32Gb/s, 16Gb/s, 10Gb/ s, and 8Gb/s. The 10Gb/s transceivers can be used for any
port on the FC64-48 port blades. Note that 10Gb/s transceivers on the FC64-48 and
SX6 blade are not interchangeable. For details on supported transceivers and speeds,
see Supported Transceiver Matrix
https://www.broadcom.com/products/fibre-channel-networking/transceiver-modul
es.
– Trunking technology groups up to eight ports to create high-performance 512Gb/s ISL
trunks between switches using 64Gb/s ports.
򐂰 Support for the following features when using the FC32-X7-48 blade:
– Port blade with 48 front-end 32Gb/s FC SFP+ ports with edge switch interconnect to
the core blades.
– Supported SFPs include 32Gb/s, 16Gb/s, and 10Gb/s FC.
– Support for 48 Fibre Channel ports on each FC32-X7-48 port blade that provide
32Gb/s, 16Gb/s, 10Gb/s, 8Gb/s, and 4Gb/s. The 10Gb/s transceivers can be used for
any port on the FC32-X7-48 port blades.
Note that 10Gb/s transceivers on the FC32-X7-48 and SX6 blade are not
interchangeable. For details on supported transceivers and speeds, see Supported
Transceiver Matrix
https://www.broadcom.com/products/fibre-channel-networking/transceiver-modul
es.

Chapter 2. IBM b-type Gen 7 product review 13


8497ch02.fm Draft Document for Review April 5, 2021 12:31 pm

– Trunking technology groups up to eight ports to create high-performance 256Gb/s ISL


trunks between switches using 32Gb/s ports.
򐂰 Support for the following features when using the SX6 extension blade:
– 16 Fibre Channel ports supporting 4, 8, 16, and 32Gb/s.
– 16GbE ports supporting 1 or 10Gb/s.
– Two GbE ports supporting 40Gb/s on SX6 extension blades.
NOTE 10Gb/s transceivers on the FC64-48 and the FC32-X7-48 are not
interchangeable with the IBM b-type Gen 7 SX6 Blade. For details on supported
transceivers and speeds, see Supported Transceiver Matrix
https://www.broadcom.com/products/fibre-channel-networking/transceiver-modul
es.
򐂰 Considerations for the IBM b-type Gen 7 SX6 Blade:
– The SX6 extension blades perform as extension platforms to support Fibre Channel
(FC) and FICON data flows and IP-based storage data flows over an IP WAN.
– Universal ports that self-configure as E_Ports, F_Ports, EX_Ports, M_Ports (mirror
ports), and FICON ports.
Note that the 10Gb/s ports on the FC64-48 port blade can function as E_Ports only.
򐂰 ClearLink Diagnostic port (D_Port) functionality on Fibre Channel ports.
򐂰 Data compression capabilities through the port blades when ports are configured as ISLs.

2.3.1 Hardware components


The device has a modular and scalable mechanical construction that allows a wide range of
flexibility in installation, fabric design, and maintenance. The device can be mounted with the
cables facing either the front or the rear of the equipment rack. It consists of the following
hardware components:
򐂰 Up to four slots for hot-swappable port blade assemblies, providing up to 192 64Gb/s or
256 32Gb/s Fibre Channel ports for device or ISL connectivity. Flexport technology allows
QSFP ports on this blade to be configured for FCoE operation at 4x10GbE, 4x25GbE, and
40GbE speeds when using an FC32-64 port blade. For a list of supported transceivers for
these blades, see Supported Transceiver Matrix.
򐂰 Two half-size slots for control processor (CP) blades:
– A single active CP blade can control all the ports in the device.
– The standby CP blade assumes control of the device if the active CP blade fails.
򐂰 Two slots for core routing (CR) blades:
– The CR blade interconnects all port blades.
– The blades support up to 16 Gen 7 QSFP56 (ICL) ports.
– ICL ports allow interconnection with neighboring director chassis.
– Both CR blades are active and can be hot-swapped.
– For a list of supported transceivers for these blades, see Supported Transceiver Matrix
https://www.broadcom.com/products/fibre-channel-networking/transceiver-modul
es .
򐂰 Up to four slots for modular, hot-swappable 34-port SX6 extension blades. Blades provide
16 32Gb/s Fibre Channel (FC) ports supporting 4, 8, 16, and 32Gb/s FC ports. Extension

14 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch02.fm

blades enable long-distance communication over an existing IP infrastructure. For a list of


supported transceivers for these blades, see Supported Transceiver Matrix.
򐂰 Modular, hot-swappable field-replaceable units (FRUs):
– Two fan assemblies, available with NPI or NPE airflow.
– Up to two power supplies, available with the NPI or NPE airflow. For details refer to
https://www.broadcom.com/products/fibre-channel-networking/directors/x7-dire
ctors
Also refer to the following documentation:
• See the "Power Supply Specifications (per PSU)" in the IBM b-type Gen 7 Director
Technical Specifications for maximum output power, input voltage, input line
frequency, and other specifications for your power supply model.
• See the "Power Supply Requirements" section in the IBM b-type Gen 7 Director
Technical Specifications for the minimum number of power supplies required for
operation and redundancy when different input voltages are used, such as low line
and high line AC.
• See the "Power Consumption" sections in the IBM b-type Gen 7 Director Technical
Specifications for power output data and the minimum number of power supplies for
supported input voltages.
• Redundant primary power connections ensure high availability. Each power supply
assembly has its own connector, so the number of primary power connections is
two for optimum efficiency and redundancy.
– Two World Wide Name (WWN) cards located on the non-port side of the device behind
the WWN card bezel.
– Port blades use small form-factor pluggable (SFP+, QSFP+, and QSFP28) optical
transceivers. For details on supported transceivers per blade type, see Supported
Transceiver Matrix.
– Core routing blades use QSFP56 optical transceivers. For a list of supported
transceivers for these blades, see Supported Transceiver Matrix
https://www.broadcom.com/products/fibre-channel-networking/transceiver-modul
es.
򐂰 Chassis door. This door must be installed to meet EMI compliance certification.
򐂰 Two vertical cable management finger assemblies. These combs install on the equipment
rack for cable management. Refer to this url and Documentation tab for Director
Installation Guides
https://www.broadcom.com/products/fibre-channel-networking/directors/x7-directo
rs.

Note: Device control processors and management modules contain batteries for
RTC/NVRAM backup. Do not attempt to replace these batteries. Dispose of hardware
components that contain these batteries as required by local ordinances and
regulations.

2.3.2 Port-side view of the device


Figure x.x shows the port-side view of the IBM SAN256B-7 Director with installed blades.
Components are identified as follows:
1. Front Air Vent
2. Control Processor Blades (CPX7)

Chapter 2. IBM b-type Gen 7 product review 15


8497ch02.fm Draft Document for Review April 5, 2021 12:31 pm

a. Slot 1 (left),
b. Slot 2 (right)
3. Port Blades (FC64-48 or FC32-X7-48)
4. Core Routing Blades (CR64-4)
5. Side Air Vent

Note that the SX6 extension blade is not shown in the following figure, but would install in the
same slots as the FC32-X7-48 or FC64-48 port blades. A maximum of four SX6 blades is
supported.

Figure 2-3 Port-side view of the IBM SAN256B-7 Director (example configuration)

Note: When non-port-side exhaust (NPE) fan and power supply assemblies are installed,
air flows through the side vent and exhausts to the non-port side. When non-port-side
intake (NPI) fan and power supply assemblies are installed, air flows from the
non-port-side and exhausts through the side vent. When installing the chassis in a
four-post rack with the airflow diversion rack mount kit, airflow is redirected to or from the
port side of the chassis. Therefore, if NPE fan and power supply assemblies are installed,
air flows into port-side air vents and exhausts to the non-port side. If NPI fan and power
supply assemblies are installed, air flows into non-port-side air vents and exhausts through
to the port-side.

2.3.3 Port-side slot numbering


The IBM SAN256B-7 Director contains 6 full-width slots and two half-width slots, for a total of
8 slots. Facing the port side of the device, the half-width slots are on the top, numbered 1 and
2 from left to right. The remaining full-width slots are numbered 3 through 8, counting from top
to bottom of the chassis. Slots contain guide pins and connectors designed for specific blade
types. Only install the control processor (CP), core routing (CR), port, and extension blades
into slot numbers as follows:

16 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch02.fm

򐂰 Slots 1–2 are restricted to CP blades. Note that the blade installed in slot 1 will be
designated as CP0, while the blade in slot 2 will be designated as CP1 in CLI command
and message output.
򐂰 Slots 3–4 and slots 7-8 are restricted to port and extension blades.
򐂰 Slots 5–6 are restricted to CR64-4 blades.

2.3.4 Non-port-side view of the device


Figure 2-4 shows the non-port-side view of the IBM SAN256B-7 Director with all fan and
power supply assemblies installed.
1. WWN Bezel (logo plate - WWN cards behind)
2. Power Supply Assembly
3. Fan Assembly
4. 4. 2AWG Panduit LCD2-14AF Lug for Building Ground Connection

Depending on the fans and the power supplies installed, airflow can be from the port side to
the non-port side or from the non-port side to the port side of the chassis through the side air
vent.

Use the Chassis Airflow Diversion and Port Side Exhaust Kit to divert airflow so that air fully
exhausts to the port side of the chassis or is fully drawn from the port side of the chassis while
mounted in a four-post rack.

Figure 2-4 Non-port side of the IBM SAN256B-7 Director (example configuration)

Although not illustrated, the chassis label that contains the serial number, SKU, and WWN is
located on the upper portion of the chassis to the left of the fan assemblies.

Chapter 2. IBM b-type Gen 7 product review 17


8497ch02.fm Draft Document for Review April 5, 2021 12:31 pm

2.3.5 Additional Information


For more information regarding the IBM b-type Gen 7 SAN256B-7 (8961-F4) refer to:
https://www.broadcom.com/products/fibre-channel-networking/directors/x7-directors#
documentation

2.4 IBM b-type Gen 7 SAN64B-7 (8960-P64/R64) switch


overview
The IBM SAN64B-7 Switch offers the following features and capabilities:

o NOTE: The IBM SAN64B-7 switch is secure-booted.


򐂰 Enterprise class 56-port Gen 7 Fibre Channel switch supported with FOS v9.0.0 that
offers 56 x 64Gb/s SFP+ ports in a 1U form-factor.
򐂰 Each of the 56 SFP+ ports supports 8, 10, 16, 32, and 64Gb/s Fibre Channel speeds.
򐂰 64Gb/s autosensing Fibre Channel switch and router ports.
– A 64Gb/s optical transceiver can auto-negotiate to 64Gb/s, 32Gb/s, or 16Gb/s.
– A 32Gb/s optical transceiver can auto-negotiate to 32Gb/s, 16Gb/s, or 8Gb/s.
– A 10Gb/s optical transceiver can auto-negotiate to 10Gb/s.

Note: The port speed is determined by the maximum speed supported by the
optical transceiver at the other end of the link.

򐂰 10Gb/s manually set capability on FC ports.


– 10Gb/s performance is enabled by 10Gb/s SFP+ transceivers.
– Ports can be configured for 10Gb/s for metro connectivity.
򐂰 Dynamic Ports on Demand (Dynamic-POD) scaling from a base configuration of 24 ports
to 56 ports (four 8-port SFP+ PODs).
򐂰 Universal ports self-configure as a E_Ports, F_Ports, N_Ports, or D_Ports. EX_Ports can
be activated on a per-port basis with the optional Integrated Routing license.
– A Diagnostic Port (D_Port) provides diagnostics, troubleshooting, and verification
services for the physical media.
򐂰 In-flight 64Gb/s data compression and encryption provide efficient link utilization and
security. Table x.x lists the number of ports that can be enabled with compression and
encryption.

Table 2-1 IBM SAN64B-7 encryption and compression: number of supported ports
Port Speed Encryption-Only Compression-Only Compression and
Encryption

64Gb/sa 4 ports 4 ports 4 ports

32Gb/sa 4 ports 4 ports 4 ports

16Gb/sa 4 ports 4 ports 4 ports

10Gb/sa 4 ports 4 ports 4 ports

18 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch02.fm

Port Speed Encryption-Only Compression-Only Compression and


Encryption

8Gb/s 4 ports 4 ports 4 ports


a. Supported on SFP ports

򐂰 Support for port-side exhaust or non-port-side exhaust airflow for cooling.


򐂰 Hardware-enabled input and output (I/O) latency statistics collection.
򐂰 Hardware-enabled VM support.
򐂰 IBM b-type Gen 7 small form-factor pluggable plus (SFP+) optical transceivers support
any combination of Short Wavelength (SWL) and Long Wavelength (LWL) optical media
among the switch ports.
򐂰 Extended distance Fibre Channel to support long-distance native FC connectivity.
򐂰 10Gb/s Fibre Channel integration on any selected port provides DWDM metro connectivity
on the same switch.
򐂰 Port-to-port latency is minimized to 460 ns (including FEC) is minimized by using
cut-through frame switching.
򐂰 High performance T1022E processor with two cores operating at 1.2 GHz delivers high
performance, scalability, and advanced Fabric Vision functionality.
򐂰 One 1000/100Mb/s RJ-45 connector for the Ethernet management connection. In
conjunction with EZSwitchSetup, this port supports switch IP address discovery and
configuration, eliminating the need to attach a serial cable to configure the switch IP
address.
򐂰 One internal e-USB module provides 2 GB of persistent storage, increased serviceability,
and error logging functionality by facilitating easier firmware upgrades and downloads of
the system log files.
򐂰 One external USB connector.
򐂰 Two hot-swappable redundant integrated 350W AC power supplies and fan assembly
field-replaceable units.
򐂰 56 hot-pluggable SFP+ optical transceiver slots.
򐂰 48 LEDs (green/amber) for the first 48 SFP+ ports and 8 LEDs (green/amber/light-blue)
for the last 8 SFP+ ports.
򐂰 One green LED to indicate valid system power.
򐂰 One bicolor (green/amber) LED to indicate the system status.
򐂰 Two Ethernet LEDs: one green LED to indicate link speed of 1000/100/10Mb/s and one
green LED to indicate activity.
򐂰 SEEPROM for switch identification.
򐂰 Console port. One RS-232 (UART), for terminal access via mini-USB.
򐂰 Encryptable SEEPROM.
򐂰 Secure encryptable SEEPROM.

Note: The License ID is protected by the secure encryptable SEEPROM.

򐂰 Secure boot.
򐂰 Real-time power monitoring.

Chapter 2. IBM b-type Gen 7 product review 19


8497ch02.fm Draft Document for Review April 5, 2021 12:31 pm

򐂰 Real-time voltage monitoring.


򐂰 Real-time fan monitoring including airflow direction.
򐂰 Real-time digital thermometers for temperature monitoring.
򐂰 Real-time clock (RTC) with battery.

2.4.1 Software License Options


The IBM SAN64B-7 Switch uses a capacity-based Ports on Demand (POD) license method.
An Integrated Routing (IR) license is required to enable EX_Ports on this device. A license for
software release is also required. Refer to this url and Documentation tab for Director
Installation Guides:
https://www.broadcom.com/products/fibre-channel-networking/directors/x7-directors.

2.4.2 Port-side view of the device


Figure 2-5 on page 21 shows the port-side view of the IBM SAN64B-6 Fibre Channel switch.
The components are as follows:
1. System Power LED
2. System Status LED
3. Management Ethernet Port
4. SFP+ FC (upper and lower) Ports 0–7
5. SFP+ FC (upper and lower) Ports 8–15
6. SFP+ FC (upper and lower) Ports 16–23
7. SFP+ FC (upper and lower) Ports 24–31
8. SFP+ FC (upper and lower) Ports 32–39
9. SFP+ FC (upper and lower) Ports 40–47
10.SFP+ upper Port 48
11.SFP+ upper Port 50
12.SFP+ lower Port 52
13.SFP+ lower Port 54
14.SFP+ upper Port 56
15.SFP+ upper Port 58
16.SFP+ lower Port 60
17.SFP+ lower Port 62
18.SFP+ (lower) Port 14 Status LED
19.SFP+ (upper) Port 10 Status LED
20.USB Port 21. UART mini-USB Serial Console Port

20 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch02.fm

Figure 2-5 Port-Side View of the IBM SAN64B-6

Note: All the ports are connected to a single ASIC.

2.4.3 Nonport-side view of the device


Figure 2-6 shows the nonport-side view of the IBM SAN64B-7 Fibre Channel switch. The
components are as follows:
1. #6-32 for Screw Mounting of the Ground Cable
2. Ground Marking
3. Power Supply and Fan Assembly 2
4. Power Supply and Fan Assembly 1
5. Captive Screw
6. Handle
7. Power Supply and Fan Assembly Status LED
8. Power-On Switch
9. Power Supply Receptacle

Figure 2-6 Nonport-side view with AC power supply and fan assembly units

2.4.4 Additional Information


To find all documentation for the IBM b-type Gen 7 SAN64B-7 (8960-P64/R64) refer to the
documentation tab at this url:

Chapter 2. IBM b-type Gen 7 product review 21


8497ch02.fm Draft Document for Review April 5, 2021 12:31 pm

https://www.broadcom.com/products/fibre-channel-networking/switches/g720-switch#do
cumentation

2.5 Upgrading Gen 6 to Gen 7 supported platform overview


The IBM b-type Gen 7 X6 chassis has been designed to be field upgradeable from a Gen 6
director to a Gen 7 director. This section provides a high-level overview of the steps needed to
complete a conversion from Gen 6 to Gen 7 platforms.

Note: Upgrading to a Gen 7 director is permanent on the CPs. This means that you cannot
downgrade to Gen 6 after the upgrade to Gen 7 and you cannot use the same CP blades in
a Gen 6 chassis.

2.5.1 Migration steps


The steps required to migrate your Gen 6 director (X6-4 and X6-8) running either Fabric OS
8.x or 9.x to a Gen 7 director running Fabric OS 9.x. are as follows:
򐂰 Upgrade to Fabric OS 9.0.0a.
򐂰 Take a snapshot of the existing configuration.
򐂰 Convert the CP blades from Gen 6 to Gen 7.
򐂰 Swap the core blades.
򐂰 Restore the original configuration or default configuration.

Note: You should have a terminal console connection to both CPs when migrating.

Figure 2-7 lists of the functional differences between the three IBM b-type Gen 7
chassis that support Fabric OS 9.0.0a.

Figure 2-7 Functional Differences between the Chassis Supporting Fabric OS 9.0.0a

Figure 2-8 on page 23 provides a list of the Gen 6 and Gen 7 blade support.

22 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch02.fm

Figure 2-8 Blade support matrix

For documentation on Gen 6 to Gen 7 migration procedures, please contact your IBM
technical support representative.

2.6 IBM b-type Gen7 transceiver module overview


The IBM SAN512B-7, IBM SAN256B-7 and IBM SAN64B-7 platforms are based on the
Brocade Condor 5 ASIC. Brocade Condor 5 ASIC platforms (Gen 7) must use specific
transceivers that are listed in Table 2-2. Note that previous versions of SFP+ transceivers will
not be recognized by a Condor 5 based port blades or switch ports.

Table 2-2 IBM SAN512B-7, IBM SAN256B-7 and IBM SAN64B-7 supported transceivers
Description IBM SKU IBM Part IBM SAN512B-7 FC64-8 FC32-X7 SAN64B-7
Number Feature & Port Blade 48 Port
Code SAN256B-7 Blade

Gen 7 FC SWL IB-000420 03GM315 2624 X


QSFP+

10G FC SWL IB-000418 01JC462 2621 X X X


SFP+

LWL IB-000417 01JC459 2620 X X X

16G FC SWL IB-000492 01JC494 2615 X


SFP+ IB-000493 01JC496 2616

LWL IB-000498 03GM300 2617 X


IB-000499 03GM303 2618

ELWL - 25 km IB-000458 03GM306 2619 X

32G FC SWL IB-000412 01JC449 2625 X X X


SFP+ IB-000413 01JC451 2626

LWL - 10 km IB-000438 IB-000439 2627 X X X


IB-000439 01JC456 2628

Chapter 2. IBM b-type Gen 7 product review 23


8497ch02.fm Draft Document for Review April 5, 2021 12:31 pm

Description IBM SKU IBM Part IBM SAN512B-7 FC64-8 FC32-X7 SAN64B-7
Number Feature & Port Blade 48 Port
Code SAN256B-7 Blade

4x 32G FC SWL IB-000475 03GM309 2622 X


QSFP+

For a full list of supported transceivers for all IBM b-type Gen 7 platforms go to the
Supported Transceiver Matrix:
https://www.broadcom.com/products/fibre-channel-networking/transceiver-modules.

24 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch03.fm

Chapter 3. Gen 7 Fabric Operating System


(FOS) overview
FOS v9.0 introduces a broad set of new features and enhancements. It is a modern operating
system that provides solid foundation for simplify management, improve performance,
increase automation, and strengthen security.

This chapter includes the following topics:


򐂰 3.1, “Autonomous SAN” on page 26
򐂰 3.2, “Automation” on page 28
򐂰 3.3, “FOS 9.x features” on page 29
򐂰 3.4, “Management tools” on page 34
򐂰 3.5, “Security” on page 35
򐂰 3.6, “Traffic optimizer” on page 36
򐂰 3.7, “Fabric Notifications” on page 39
򐂰 3.8, “The IBM B-Type switch Fabric Notifications solution” on page 40

© Copyright IBM Corp. 2021. 25


8497ch03.fm Draft Document for Review April 5, 2021 12:31 pm

3.1 Autonomous SAN


Enterprise data centers continue to push virtualization at higher density and support a much
broader set of workloads. More virtual machines (VMs) are being deployed on the server end,
and more storage LUNs are provisioned on storage arrays. At the same time, adoption of all
flash storage has accelerated. Customers are starting to adopt next generation NVMe based
storage that offer much lower latency access to data. The use of virtualization, flash storage,
and automation tools has allowed applications and services to be deployed faster while
shattering performance barriers. The unprecedented number of application and service
interactions has also increased the complexity, risk, and instability of mission-critical
operations. For example, increasing workload virtualization leads to a higher number of
application flows but reduces the visibility of the flows. At the same time, storage virtualization
increases the number of short FC frames and the potential for congestion. As a result, IT
organizations require storage networks that deliver ultra-low latency, higher-capacity
bandwidth, and greater reliability. To meet service-level agreements (SLAs) or achieve
service-level objectives (SLOs), IT administrators also need new tools that can help ensure
nonstop operations, quickly identify potential points of congestion, and maximize application
performance, while simplifying administration.

In the past, the process of problem detection and mitigation relied on the diligence, skill, and
experience of the fabric administrator. Usually, a manual task of searching and tracing error
conditions was the regimen. The operational cost and potential disruptive impact to critical
applications are simply not acceptable in today's data center. To assist in administrative
automation and improve network uptime, Broadcom has developed Autonomous SAN
technology that offers self-learning, self-optimizing, and self-healing capabilities. Autonomous
SAN is realized through Fabric Vision technology that combines built-in capabilities in IBM
b-type Gen 6 and Gen 7 platforms, Brocade Fabric OS, and Brocade SANnav Management
Portal features. Fabric Vision provides detailed monitoring and alerting, as well as response
and mitigation, that vastly improve the fabric administrator's insight and response time.

3.1.1 Self-learning
Brocade technology proactively monitors millions of I/O performance and behaviour data
points through integrated network sensors to gain deep insight into the environment. These
data points are transformed into actionable intelligence that can monitor and alert when there
are any abnormal changes. Through these capabilities, an admin can identify individual
applications and their performance characteristics across the fabric, as well as identify the
performance of the various devices that comprise the fabric: the switches, hosts, and targets.
򐂰 IO Insight: Proactively monitors I/O performance and behaviour through integrated
network sensors to baseline application performance and ensure operational stability. By
combining the instrumentation of IO Insight with the ability to self-learn the traffic flows, the
Brocade autonomous SAN technology can generate performance and health metrics on
each component as well as the applications to monitor for changes and alert the
administrator with MAPS.
򐂰 Monitoring and Alerting Policy Suite (MAPS): Provides an easy-to use solution for
policy-based threshold monitoring and alerting. MAPS proactively monitors the health and
performance of any SCSI or NVMe storage infrastructure to ensure application uptime and
availability. By leveraging prebuilt rule-based and policy-based templates, MAPS simplifies
fabric-wide threshold configuration, monitoring, and alerting.
򐂰 Automatic Flow Learning: Provides automatic learning of all traffic flows from a specific
host to storage across the SAN fabric. With this information, an admin can automatically
identify resource contention or congestion that is impacting application performance.

26 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch03.fm

򐂰 Fabric Performance Impact: Leverages predefined MAPS policies to automatically detect


and alert administrators to different congestion severity levels and to identify credit-stalled
devices (for example, misbehaving slow drain devices) or oversubscribed ports that could
impact network performance. This feature pinpoints exactly which devices are causing or
are impacted by the congested port, and it quarantines the misbehaving devices.

3.1.2 Self-optimizing
Utilizing actionable intelligence gathered from self-learning capabilities, Brocade Fibre
Channel SANs automatically apply priorities for specific data traffic to help guarantee
performance levels and monitor for traffic pattern shifts. Learning traffic behaviour enables the
network to make smarter decisions on traffic prioritization, congestion avoidance, and
adjustment to ensure optimal network performance for applications and storage. When
something does change, the Brocade autonomous SAN technology will isolate the port traffic
for the misbehaving device to a virtual channel in the fabric and allow all other traffic to go
around to maintain optimal performance. In addition, Brocade SANnav Management Portal
has automated manual activities such as infrastructure deployment and provisioning to
expedite IT services. Self-optimizing features include the following:
򐂰 Traffic Optimizer: Automatically classifies and separates traffic with similar characteristics
to optimize performance for most common SAN configurations. It identifies and isolates
traffic flows to prevent negative impact to overall SAN performance.
򐂰 Advanced traffic shaping: Guarantees application performance by proactively monitoring
and actively shaping traffic.
򐂰 REST APIs: Eliminate human errors and performance impacts through open DevOps.
򐂰 SAN Automation and Ansible: Gain cloud-like SAN orchestration for optimizing
administration resources through Brocade REST API automation technology.

3.1.3 Self-healing
When potential disruptions are detected, the network will automatically mitigate or resolve
issues without manual intervention. This is done by proactively monitoring the network to
automatically identify abnormal or unexpected infrastructure behaviour and then taking
immediate action. These actions include alerting the end devices of the issue through a
notification and signalling process within a SAN. End devices can automatically adjust traffic
or fail over to a healthy path to mitigate impact until a further infrastructure change solution
can be implemented. In addition, these capabilities detect and automatically reconfigure
fabric misconfigurations that fall outside best practices Self-healing features include:
򐂰 Fabric congestion notification: Automatically detects congestion and notifies end devices
to automatically mitigate congestion.
򐂰 Slow drain device quarantine: Automatically quarantines credit-stalled devices to prevent
the misbehaving device from impacting the rest of traffic.
򐂰 Automatic actions: Ensure data delivery with automatic failover from physical or
congestion issues such as port decommissioning, port toggling, and port fencing.
򐂰 COMPASS: Detects and automatically reconfigures out-of-compliance configurations.
򐂰 SANnav Management Portal: Reduces troubleshooting steps with built-in best practice
recommendations to quickly resolve issues.

See https://docs.broadcom.com/docs/broadcom-autonomous-self-healing-sans for


detailed information about how self-healing SAN capabilities minimize the application
performance impact.

Chapter 3. Gen 7 Fabric Operating System (FOS) overview 27


8497ch03.fm Draft Document for Review April 5, 2021 12:31 pm

3.2 Automation
Most, if not all, IT administrators have first-hand experience in the growing complexity of
managing enterprise infrastructure, including SANs. “The cost and complexity of protecting
and storing data is increasing, and IT leaders are responding with attempts to better optimize
and automate storage—but they need better tools,” according to a report from Enterprise
Strategy Group.

IBM and Brocade are in the unique position to spot and understand the impact of automation
helping organizations get more from their SAN infrastructure.

IBM and Brocade offers a combination of SAN automation with RESTful APIs and a SAN
management platform to help organizations drive greater efficiency from their SANs. This is
accomplished through a variety of means:
򐂰 IBM and Brocade SAN automation is provided with a multilayer architecture:
򐂰 RESTful API support on switches, as well as on management tools.
򐂰 Brocade’s Ansible management framework, designed to eliminate repetitive tasks, simplify
management and orchestrate across the full SAN infrastructure.

3.2.1 Why organizations should embrace automation


Here are five reasons why organizations should embrace SAN automation:
򐂰 Reducing human error and streamlining operational processes has never been more
crucial. As organizations move to digitize and adapt to new workloads, data availability,
processing time and agility to provision applications on-demand become the life-blood of
the business. This demands a more efficient and expedient infrastructure management
approach, leaving no room for human error. As a result, storage administrators need to be
freed from repetitive manual tasks such as configuration management, reporting,
documenting inventory and troubleshooting. Instead, IT organizations need SAN
automation to help them automate and orchestrate repetitive tasks, significantly improve
efficiency, and decrease the risk of operational mistakes.
򐂰 Demand for more accurate and more frequent infrastructure reports is on the rise. It’s not
just IT managers who crave information on storage performance, utilization and
forecasting, business stakeholders are also asking for and expecting this data on demand.
No one wants to wait for a slot when storage administrators can allocate time to produce a
report. This information should be available as frequently as business demands dictate—
all at the click of a button. Automation not only provides this kind of responsiveness that
traditional manual storage management processes can’t deliver, but it can also be
customized so that each stakeholder is getting more accurate data aligned to their
responsibility.
򐂰 SAN configuration management must be streamlined. With more enterprise applications
demanding access to more data and with more virtual machines, deploying and
configuring servers, storage and the network has become more time-consuming and more
complex than ever. By streamlining SAN operations through automation, application
provisioning workflows are simplified across hypervisor, network and storage, delivering
agility and responsiveness to meet dynamic business demands.
򐂰 IT service delivery is not always as responsive as the business demands. As
organizations become increasingly reliant on having world-class IT services available to
them for proactive, agile business decision-making, it is essential that bottlenecks to IT
service delivery be identified and eliminated immediately. This has to be done without
hiring more storage administrators or boosting SAN-related Capex spending, making SAN

28 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch03.fm

automation the only viable option to drive increased IT agility and more tightly align IT
service delivery with fast-changing business needs.
򐂰 Consistent configuration validation is a must. Manual configuration changes are taking
place with greater frequency as enterprises diversify their IT architectures in general, and
their storage architectures specifically. SAN automation ensures validation of consistent
configuration parameters across the different SAN fabrics in order to facilitate
troubleshooting of frequent alerts without reliance on manual intervention.

Brocade automation solutions leverage RESTful APIs to facilitate solutions architecture,


share best practices and get to production status faster. Brocade’s SAN automation brings
automation to the switch level that helps reduce the cost and complexity of managing storage
systems.

See the Brocade Fabric OS REST API Reference Manual


https://docs.broadcom.com/docs/FOS-90x-REST-API-RM for detailed information about how
to setup and manage IBM B-Type switches using REST API.

See https://docs.broadcom.com/docs/SAN-Auto-Dummies for further information, such as


choosing your approach using RETful, Ansible or Python, and to create an automation plan.
The document will also guide you on how to clone and use the GitHub repositories.

3.3 FOS 9.x features


FOS 9.0 introduced new features and many improvements to the IBM B-Type Gen 6 and Gen
7 SAN switches, out of which Unified Addressing Mode, Fabric Zone Lock and Mandatory
FICON Logical Switch are some of the important changes.

In addition to the afore mentioned features, new features such as, New Java-Less Web Tools
based on HTML 5.0, increased zone database size to 4MB, updated SupportSave to run
parallel to save time during troubleshooting, Command reset to factory reset and support for
the Generic USB drives are few to note.

In this section, we will cover some of these features, for full list of enhancements, please refer
to Brocade Fabric OS Administration guide at
https://docs.broadcom.com/doc/FOS-90x-Admin-AG

3.3.1 Port identifiers


Port identifiers (PIDs, also called abric addresses) are used by the routing and zoning
services in Fibre Channel fabrics to identify ports in the network. All devices in a fabric must
use the same PID format. When you add new equipment to the SAN, you might need to
change the PID format on legacy equipment.

Many scenarios cause a device to receive a new PID: for example, unplugging the device
from one port and plugging it into a different port as part of fabric maintenance; or changing
the domain ID of a switch, which might be necessary when merging fabrics; or changing
compatibility mode settings.

Some device drivers use the PID to map logical disk drives to physical Fibre Channel
counterparts. Most drivers can either change PID mappings dynamically, also called dynamic
PID binding, or use the WWN of the Fibre Channel disk for mapping, also called WWN
binding.

Chapter 3. Gen 7 Fabric Operating System (FOS) overview 29


8497ch03.fm Draft Document for Review April 5, 2021 12:31 pm

Some older device drivers behave as if a PID uniquely identifies a device; they use static PID
binding. These device drivers should be updated, if possible, to use WWN binding or dynamic
PID binding instead, because static PID binding creates problems in many routine
maintenance scenarios. Fortunately, very few device drivers still behave this way. Many
current device drivers enable you to select static PID binding as well as WWN binding. You
should select static PID binding only if there is a compelling reason and only after you have
evaluated the effect of doing so.

3.3.2 Unified addressing mode


Unified addressing mode (UAM) provides a single addressing mode for an 8-bit area, 10-bit
area, or shared area, and it provides other addressing modes for different combinations of
platform and logical switch type. UAM simplifies these addressing mode combinations and
features into a single unified addressing mode. UAM is introduced in FOS 9.0.0 and is the
default address allocation mode for next generation platforms such as the IBM SAN512B-7
Director, IBM SAN256B-7 Director, and IBM SAN64B-7 Switch.

Table 3-1 describes how UAM works for FOS features:

Table 3-1 Difference between UAM and pre-UAM Mode

Feature UAM Mode Pre-UAM Mode

F-Port A port is assigned only a base area. A port is assigned only a base area.

F_Port with NPIV For F_Ports with NPIV less than 63 The 10-bit addressing mode utilizes
devices, port is assigned only a base the 8-bit area ID and the borrowed
area. If the 64th NPIV device is upper 2 bits from the AL_PA portion
received in F_Port, then a new of the PID. Areas
additional area is assigned to the 0×00 through 0×8F use only 8 bits
existing base area. Every time a port for the port address and support
exhausts an area after 64 additional upto 255 NPIV devices. Areas 0×90
devices login, another additional area through 0×FF use an additional 2
is assigned. The maximum number of bits from the AL_PA for the port
NPIVs supported is 255 devices. address. Therefore, these ports
support only 64 NPIV devices per
port.

F_Port Trunking Same as F_Port and F_Port with NPIV. F_Port trunking uses an 8-bit area
In FOS 9.0, UAM allocates additional in VF mode enabled logical switch.
areas to both F_Port trunking and Default switch in VF or non-VF
Emulex HBA trunking. mode uses existing port address.

EX_Port EX_Port in a base switch is assigned a EX_Port in a base switch uses an


10- bit area. existing port address as per the
current switch addressing mode.

EX_Port on ICL Ports When an ICL port is configured as an When an ICL is configured as an
EX_Port, a new area is assigned to the EX_Port, the switch is assigned an
ICL port from the pool. This is not 8-bit area in VF. EX_Port over ICL is
applicable for ICL ports that are not not supported in non-VF mode.
configured as an
EX_Port. From the FOS 9.0 release,
when an ICL port is configured as an
EX_Port,the new areas are assigned in
both VFmode and non-VF mode.

30 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch03.fm

Feature UAM Mode Pre-UAM Mode

EX_Port Trunking In UAM mode, an EX_Port trunk In pre-UAM mode, an EX_Port


behaves in a masterless manner both trunk behaves in a masterless
in the base switch and in non-VF mode. manner only in VF mode.

FCoE In UAM mode, FCoE address In pre-UAM mode, whenever


assignment is simple. Whenever Ethernet flexport requires an area,
Ethernet flexport requires an area, it it will be assigned an area from the
will be assigned an area from the pool pool of unused areas. For default
of unused areas. switches in VF mode or non-VF
mode, the switches are converted
to a 10-bit area mode as soon as
FCoE enodes are configured.

Port Address Binding In platforms that are enabled in UAM In pre-UAM mode, the port address
mode, the port address binding feature binding feature is supported only in
is supported in both VF and non-VF logical switches in VF enabled
mode. mode.

Persistent PID In platforms that are enabled in UAM In pre-UAM mode, the persistent
mode, the persistent PID feature is PID feature is supported only in
supported in both VF and non-VF logical switches in VF enabled
mode. mode.

See the FICON Configuration chapter in this RedBook for further details on the benefits and
restrictions with the Unified Addressing Mode.

3.3.3 Fabric zone locking


Prior to Fabric OS 9.0.0, the level of zoning transaction protection against multiple concurrent
clients only existed at the local switch level. For example, if a user on a switch starts a zone
transaction by performing a zone create or edit operation, and if another user tries to perform
a zone create or edit or commit operation on the same switch, the second user will be locked
out and prevented from performing his operation since the first user opened the transaction
first.started on all remote switches.

S w 1 -C L I

S w it c h 1 – ( 8 . 2 . x) S w it c h 2 – ( 8 . 2 . x )

D e fi n e d c o n fi g u ra ti o n :
c fg : c fg 1
zo n e 1
S w 1 -C L I zo n e : zo n e 1
H 1 ;T 1
zo n e : n e w C li U s e rZ o n e
Zo ne E d it H 2 ;T 2

E ffe c ti ve c o n fi g u ra ti o n :
c fg : c fg 1
R E JE C T! zo n e : zo n e 1
H 1 ;T 1
S w 1 -B N A

Figure 3-1 Single switch locking example

Although there is local switch protection, there is no fabric-wide locking mechanism. For
versions prior to Fabric OS 9.0.0, the fabric-level detection support (Concurrent Transaction
Detection) consists only of a warning mechanism that warns users if a transaction is open on
a remote switch, but does not prevent users from committing changes if concurrent
transactions exist across the fabric. Furthermore, this fabric-wide detection is limited in scope

Chapter 3. Gen 7 Fabric Operating System (FOS) overview 31


8497ch03.fm Draft Document for Review April 5, 2021 12:31 pm

to CLI zoning operations only. For example, if a user on one switch opens a zone transaction,
and if a user on another switch in the fabric issues a zone edit or commit command, a warning
is received that a transaction is occurring in the fabric. However, a zone commit operation can
still be issued from either switch, which overwrites any pending zone transaction on other
switches.

Switch 1 – (8.2.x) Switch 2 – (8.2.x)

Defined configuration: Defined configuration:


cfg: cfg1 cfg: cfg1
zone1 zone1
zone: zone1 zone: zone1
Sw1-CLI H1;T1 ‘cfgSave’ H1;T1 Sw1-BNA
zone: newCliUserZone zone: newBnaUserZone
H2;T2 H3;T3
zone: newBnaUserZone
H3;T3 Effective configuration:
Effective configuration: cfg: cfg1
cfg: cfg1 zone: zone1
zone: zone1 H1;T1
H1;T1

Figure 3-2 Fabric OS 8.2.x and earlier fabric-wide commit overwrite example

From Fabric OS 9.0.0 version onward, the Zone Fabric Locking feature provides zone
transaction protection for a single switch and also fabric wide. This feature is enabled by
default. If a zone edit or commit command is occurring in fabric, you cannot perform a zone
edit or commit on the same or another switch for a default timeout period of 5 minutes. A lock
request is sent at the beginning of a zone edit operation. The Fabric Lock Failsafe Timer is
configurable and it is a fabricwide setting. When a zone fabric lock is active, a failsafe timer is
started on all remote switches.

Sw1-CLI

Switch 1 – (8.2.x) Switch 2 – (8.2.x)


Zone Edit
Lock Timer: 5 min 00 secs Lock Timer: 5 min 30 secs

Defined configuration:
cfg: cfg1 REJECT!
Sw1-CLI zone1 Sw1-BNA
zone: zone1
H1;T1
zone: newCliUserZone
H2;T2
Effective configuration:
cfg: cfg1
zone: zone1
H1;T1

Figure 3-3 Fabric OS 9.0.0 zone fabric locking example

When the failsafe timer expires, the open zone transaction is not aborted. If the same user
attempts to resume the transaction by performing another edit or commit operation after the
zone fabric lock has expired, the transaction is allowed and the fabric lock is restarted. If a
different user attempts to start a new transaction after the first user's transaction timer has
expired, the transaction is allowed and the first user's transaction is aborted before the
second user's transaction starts.

32 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch03.fm

Note:
򐂰 If a fabric-lock is currently active in the fabric, the Fabric Lock Failsafe Timer timeout
value cannot be modified.
򐂰 The fabric zone lock is a fabric-wide setting. Thus, the fabric must be stable before
performing zone edit, commit, and abort operations.
򐂰 If a zone transaction is aborted, the fabric zone lock is cleared.
򐂰 When the timer expires, the lock is cleared locally and on each individual switch in the
fabric.
򐂰 When using virtual fabric functionality, the lock timeout value is independent on each
logical switch and can have different values.
򐂰 Fabric zone locks can be pre-empted by zone merges.
򐂰 The Zone Fabric Locking feature is supported on the switches with the Fabric OS
version 9.0.0 or later. Down-level switches will not participate. However, Fabric OS
9.0.0 switches owning the zone fabric lock will prevent updates from down-level
switches.
򐂰 On remote switches, an additional 30-second buffer is added to the configured lock
timeout value of the lock principal switch.

The fabric-wide locking mechanism ensures that any zone transaction started on any switch
by any interface (CLI, REST, Web Tools, SANnav, Target Driven Zone) is not overwritten or
aborted by another interface's transaction (except for zone merges) while the lock timer is
active.

3.3.4 FICON logical switch


With the FICON logical switch feature, you can create a FICON logical switch. You can also
modify an existing logical switch as a FICON logical switch. The FICON logical switch is
supported on all platforms that support Virtual Fabrics in Fabric OS 8.1.0 or later. A FICON
logical switch can be created independent of enabling FMS mode. A FICON logical switch
cannot be the default switch in the chassis nor can it be the base switch of the chassis.

FICON logical switch provides auto-configuration of the following, which simplifies the
process of establishing a logical switch in the FICON environment:
򐂰 Insistent Domain ID (IDID) mode is set to on
򐂰 Routing policy is set to device-based routing
򐂰 Area mode is set to zero-based addressing
򐂰 SCC security policy is auto-defined with only its own WWN
򐂰 The created SCC policy is activated
򐂰 The fabric-wide consistency policy is set to strict SCC policy
򐂰 High Integrity Fabric (HIF) mode is enabled
򐂰 FMS mode is enabled if FICON control unit port (CUP) license is installed on the chassis

Please refer to FICON chapter in this book for more details about creating FICON Logical
switch and the best practices.

Chapter 3. Gen 7 Fabric Operating System (FOS) overview 33


8497ch03.fm Draft Document for Review April 5, 2021 12:31 pm

3.4 Management tools

3.4.1 Web Tools


Brocade Web Tools is a graphical user interface (GUI) embedded in the Fabric OS firmware
that enables administrators to monitor and manage single or small fabrics, switches, and
ports. New Web tools are Java-Free and are architected based on modern technology
HTML-5. Web Tools is launched directly from a Web browser or from SANnav Management
Portal. This document contains the system requirements.

Web Tools system requirements


Brocade Web Tools is an embedded graphical user interface (GUI) that enables
administrators to monitor and manage single or small fabrics, switches, and ports. Before
launching Web Tools, verify that your workstation uses a supported operating system and
Web browser.

Web Tools does not require a license. It is installed on the switch when you install Fabric OS.

Supported operating systems


Web Tools supports the following operating systems:
򐂰 Red Hat 8.0 and 8.1
򐂰 Windows 10 Pro
򐂰 Windows 2019

Supported web browsers


The following browsers can be used to access Web Tools:
򐂰 Chrome
򐂰 Firefox

Launching Web Tools


Web Tools is launched directly from a Web browser or from SANnav Management Portal. You
can launch Web Tools on any workstation with a compatible operating system and Web
browser installed.

If the switch is configured with logical fabrics, you can log in to any of the logical fabrics for
which you have permission.

You can launch Web Tools directly from a browser or from SANnav Management Portal.

To launch directly from a web browser, open your browser, enter the IP address of the switch,
and press Enter. You can use HTTP or HTTPS. For example, http://10.77.77.77 or
https://10.77.77.77.

34 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch03.fm

Figure 3-4 Launching WebTools

Enter the user name, password, and logical switch name or fabric ID (FID).

For the first switch login, the default user name is admin and the default password is
password. Web Tools prompts you to change the default password.

If you are logging in to a Virtual Fabrics-enabled platform and you do not specify a logical
switch, you are logged in to the default logical switch, which uses fabric ID 128. For non-VF
platforms, the FID option is not displayed.

Note: If you launch from SANnav Management Portal, you might not need to log in,
depending on the SANnav single sign-on configuration.

For information on how to use Web Tools, see


https://docs.broadcom.com/doc/FOS-90x-WebTools-UG.

SANnav section and point it to the SANnav chapter ….

3.5 Security
Security entails authentication and encryption to restrict access and protect data from
unauthorized access. Brocade products support a wide range of authentication, encryption,
and management tools to protect fabrics and data from unauthorized access:
򐂰 Authentication: Authentication protocol support includes CHAP, DH-CHAP, FCAP, IKE,
IPsec, RADIUS, TACACS+, and P-EAP/MS-CHAP for RADIUS.
򐂰 Encryption (AES/3-DES): Brocade provides AES-128 and AES-256 encryption and
168-bit 3-DES encryption for IP links on extension products and management
connections. Brocade also supports AES and 3-DES with IPsec. These solutions provide
high-performance encryption and compression.
򐂰 In-flight encryption over ISLs: Brocade X6 with Gen 6 Fibre Channel port blades, Brocade
X7 with Gen 6 or Gen 7 Fibre Channel port blades, and Brocade G720, G630, G620
switches support in-flight encryption for traffic over ISLs to minimize the risk of
unauthorized access to data within the data center and over long distance links.
Data-at-rest and data-in-flight encryption are complementary technologies that serve
different purposes, and each may be required to achieve regulatory compliance.

Chapter 3. Gen 7 Fabric Operating System (FOS) overview 35


8497ch03.fm Draft Document for Review April 5, 2021 12:31 pm

򐂰 Secure Boot: A switch validates the integrity and authenticity of the FOS boot image to
establish a hardware based root of trust through the manufacturing supply chain.

Please refer to Chapter 12, “SAN security” on page 293 for further information.

3.6 Traffic optimizer


Traffic Optimizer is a new feature introduced in Fabric OS 9.0 on Brocade Gen 7 platforms
that optimizes SAN traffic. Traffic Optimizer automatically groups traffic flows with similar
predefined attributes, such as flow destination speed and priority, as a Performance Group.
Each Performance Group uses a dedicated resource path within the fabric. By organizing
flows by Performance Groups and preventing mixed traffic, Traffic Optimizer prevents slow
traffic from obstructing the flow of higher-speed or higher-priority traffic on that path.

Traffic flows with similar predefined attributes, such as flow destination speed and priority, are
grouped logically as a Performance Group. With Gen 7 platforms, Traffic Optimizer provides a
way to optimize shared resource allocation for each Performance Group, and it automatically
maps traffic flows to predefined Performance Group.

Fabric OS 9.0 supports the Performance Groups listed in the following table. These
predefined performance groups provide support of speed-based classification, credit-stalled
device isolation via Slow Drain Device Quarantine (SDDQ), compatibility with the Adaptive
Networking (QoS Zone and CSCTL) feature, and compatibility with previous platforms. Traffic
Optimizer is enabled by default on all Gen 7 platforms and Gen 7 blades.

Gen 6 platforms in Fabric OS 9.0 continue the legacy behaviour and maintain compatibility
with Gen 7 platforms in the same fabric. If a Gen 7 switch or blade is connected to a port on a
Gen 6 platform and the traffic flow source port is on the Gen 7 switch, the traffic flow will be
speed-based classified within the Gen 7 switch until reaching the port on the Gen 6 platform,
and then the traffic flow will be handled with legacy behaviour starting from the Gen 6 port.

Many years ago, Brocade Virtual Channels (VC) addressed this issue with great success and
significant performance gains. Resources affecting performance include bandwidth, egress
queues, ingress buffers, and QoS. The implementation has multiple distinct egress queues
establishing virtual channels across the physical link, each channel having its own set of
autonomous buffer credits. The bandwidth is shared, but flows are independent. When traffic
characteristics are uniform, VCs offer an expedited pathway. Great benefit is gained by
sorting traffic into VCs based the most pertinent characteristic, which is speed.

Normally, VCs service multiple FC flows since there’s a limited number of VCs per
classification of traffic. Users do not configure or setup VCs, they are merely a fundamental
component of the Fabric Operating System (FOS). FC wire speeds continue to increase, now
at 64 Gbps (Gen 7) and bandwidth has an extremely wide spread from 8, 16, 32, to 64
gigabits (32-56 Gbps). Wide variations in speed can impose flow-control problems within a
network.

For years, Brocade VCs operated within a relatively narrow bit rates of 1, 2, 4, 8, and 16
gigabit (8-15 Gbps) by leveraging multiple logical independent paths. VCs are assigned by a
hash called PID2VC, which results in a pseudorandom assignment of each flow to a VC. To a
degree, this statistically mitigated the interference of slower flows impeding faster flows. Plus,
optionally faster flows could be manually assigned to a high QoS VC and slower flows to a low
QoS VC to prevent interference.

Technology evolves and Brocade has optimized VC efficiency by enhancing effectiveness


through targeted flow characteristics. Demands on an enterprise SAN have never been

36 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch03.fm

greater due to other complimentary technology evolutions such as NVMe, AFA, and host
virtualization. Flows now compete and head of line blocking is not an option. Blocking must be
efficaciously dealt with… in comes Traffic Optimization.

Brocade Traffic Optimization technology takes VCs to the next level. Traffic Optimization
organizes and manages traffic flows using performance groups (PG). Fabric resources are
allocated based on performance groups. Instead, flows are assigned to a VC based on the
destination port speed and potentially via protocol (for example, NVMe). Brocade fabrics
know the destination port speed and protocol for every flow.

Brocade Gen 7 hardware added more VCs, E_Ports and EX_Ports use 16 new Traffic
Optimization VCs, which are assigned: Four VCs per speed (<16, 32, 64) for a total of 12
VCs, plus another four reserved VCs for future use. QoS VCs have not changed, there are 2
for low, 4 for medium (default), and 5 for high. QoS VCs are used when traffic has been
designated to a high or low priority. Slow Drain Device Quarantine (SDDQ) uses the QoS low
VCs.

Pre-Gen 7 and existing fabric coexistence is fully supported. E_Port connections between
Gen 7 and pre-Gen 7 platforms results in Traffic Optimized VC being remapped using the
legacy classification. QoS VCs pass-through unchanged since they remain consistent with
Traffic Optimization.

Traffic Optimization is new and unique functionality delivered in Brocade’s latest Gen 7 Fiber
Channel platforms. It is a powerful enhancement to the existing Virtual Channels feature
incumbent to past generations of Brocade switching equipment. Traffic Optimization requires
no configuration and is operational on all Gen 7 devices. Traffic is classified by speed or
NVMe Performance Groups and assigned to a specific VC. Traffic Optimization is
interoperable with FC cohort technologies (FICON, Extension, Access-Gateway, Long
Distance Links, Security, and NPV) and Brocade Gen 6 fabrics.

Table 3-2 Table 3-2 Supported Performance Groups in Fabric OS 9.0

Flow Classification Performance Group Speed Range (Gb/s) Priority Assigned


Name

Speed Based PG_64G 32 < speed <= 64 Medium

PG_32G 16 < speed <= 32 Medium

PG_16G 0 < speed <= 16 Medium

SDDQ PG_low_qos_1 0 < speed <= 64 Low

PG_low_qos_2 0 < speed <= 64 Low

QoS Zones PG_high_qos_1 0 < speed <= 64 High

PG_high_qos_2 0 < speed <= 64 High

PG_high_qos_3 0 < speed <= 64 High

PG_high_qos_4 0 < speed <= 64 High

PG_high_qos_5 0 < speed <= 64 High

3.6.1 Speed-based classification


Flows are classified into one of several Performance Groups based on the destination speed
or protocol. Automatic speed classification isolates traffic flows into different speed-based

Chapter 3. Gen 7 Fabric Operating System (FOS) overview 37


8497ch03.fm Draft Document for Review April 5, 2021 12:31 pm

Performance Groups and eliminates speed mismatches that cause performance interference
when lower speed devices collide with higher speed devices connected in the same fabric.

3.6.2 Slow drain device quarantine (SDDQ) isolation


A slow-drain device, also sometimes referred to as a “credit-stalled device”, is a misbehaving
device that promptly stops returning R_RDY signals (buffer credits) to the switch, which
causes the switch to stop sending frames to the device. A credit-stalled device that stops
returning credits to the switch for hundreds of milliseconds or more causes frame drops and
link resets. This typically happens when the end device is not working as per its specification.
An impaired flow associated with credit-stalled devices is automatically mapped to a
low-priority Performance Group: PG_low_qos_1 or PG_low_qos_2.

3.6.3 Compatibility with QoS zones and CS_CTL


Gen 7 platforms maintain compatibility if the QoS feature is enabled. If QoS Low or QoS High
priority are defined, traffic flows will be assigned to one of the QoS Performance Group
defined in the Table 3-2 Supported Performance Groups in Fabric OS 9.0. If CS_CTL is
enabled on a port, traffic flows on that port will retain the legacy CS_CTL-based classification
and will not be classified into any Performance Group.

3.6.4 Compatibility with previous platforms


Previous platforms participating in fos 9.0 fabrics continue the legacy behaviour and maintain
compatibility with Gen 7 platforms in the same fabric. If a Gen 7 switch or blade is connected
to a port on a Gen 6 platform and the traffic flow source port is on the Gen 7 switch, the traffic
flow will be speed-based classified within the Gen 7 switch until reaching the port on the Gen
6 platform, and then the traffic flow will be handled with legacy behaviour starting from the
Gen 6 port.

3.6.5 Traffic Optimizer and QoS Mode


Both the Traffic Optimizer and QoS mode are automatically enabled by default on all Gen 7
platforms. Traffic Optimizer's advance classification (e.g. map traffic flow to pre-defined
Performance Group) requires QoS mode to be enabled on all the ports of the supported
platforms. It is not recommended to disable QoS mode on any ports. If QoS mode is disabled
on a port, Traffic Optimizer’s advanced classification will not work on the port, and the port's
traffic classification will fall back to legacy PID to VC mapping.

FCR Considerations for Traffic Optimizer


Traffic Optimizer is supported only on an FCR edge-to-edge fabric topology and is not
supported on a backbone-to-edge fabric topology. For Traffic Optimizer to run over FCR, all
switches and FC routers in both edge fabrics and backbone fabrics must be running Fabric
OS 9.0 or later.

Traffic Optimization over FC routers is supported only if the Virtual Fabrics is disabled in the
Backbone fabric. Traffic Optimization over FC routers is not supported if the Virtual Fabrics is
also enabled in the Backbone fabric. For more information on the operating modes of
Backbone fabrics, please refer to Fabric OS Admin guide
https://docs.broadcom.com/docs/FOS-90x-Admin-AG. If the traffic flow in the metaSAN
encounters a switch platform that does not support Traffic Optimization, then the traffic flow

38 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch03.fm

will be mapped to the legacy behaviour and will continue to retain the legacy behaviour all the
way.

3.7 Fabric Notifications


Fabric Notifications is a facility for proactively notifying devices in a fabric of performance
impacting behaviours that affect the normal flow of I/O. Fabric impacting behaviours (FPI)
such as signal integrity degradation, frame loss, or congestion are detected by either the
Fabric or participating devices. Signalling, via Fibre Channel Primitives, and Notifications via
Extended Link Services (ELS) are then used to communicate the type and severity of the FPI
to the affected members of the Fabric.

Fabric Notifications include the following components:


򐂰 Registered capabilities
A set of ELS commands exchanged between two ports arbitrate the range of Fabric
Notification capabilities supported by each.
򐂰 Congestion detection sSignalling
Hardware generated primitive signals, supported by Gen 7 platforms only, are sent toward
the point of congestion indicate that the queue of frames to be transmitted has reached a
level where fabric performance may be impacted.
򐂰 FPI notifications (FPINs)
FPINs are software generated using ELS commands and, if a device has registered to
utilize them, administrators can be notified of events that might affect fabric performance.

These components allow members of the fabric to indicate their ability to interpret and react
to signals and notifications. Participating devices can respond by taking actions to mitigate
and reduce impacts of congestion, frame loss, and signal integrity degradation. The Fabric
OS is the primary generator of signals and notifications due to its awareness of I/O flows,
physical layer impacts, and device responses.

The relationship between the event detection, signalling, and notification operations is
depicted in the following figure.

Figure 3-5 Signaling or notification operations

When an event is detected (1), the Fabric initiates the appropriate response. In the case
where frame transmission is stalled, signal events are generated (2) toward the point of
congestion (that is credit-stalled or oversubscribed device). In the case of a transmission
failure or sustained congestion event, notification events are generated (3) and distributed to
peer devices (4). The fabric determines the distribution of the notification event from the zone

Chapter 3. Gen 7 Fabric Operating System (FOS) overview 39


8497ch03.fm Draft Document for Review April 5, 2021 12:31 pm

member information of the triggering N_Port. In the case of a transmission failure or


software-based congestion signalling, notification events are sent to the attached N_Port (5)
to engage the device in mitigating activities.

3.8 The IBM B-Type switch Fabric Notifications solution


Fabric Notifications was developed to optimize IO behaviour and avoid impaired paths by
notifying devices of current fabric conditions. The IBM B-Type Fabric Notifications solution
combines hardware (Brocade FC switching ASIC), software (Fabric Operating System), and
Management (MAPS and SANnav) to form a SAN wide solution for impairment detection,
notification, and remediation. The problem is logged and operational information concerning
affected flows (ITL), location, timestamp, and reason are provided for easy debugging and
troubleshooting.

IBM B-Type Fabric Performance Impact Notification (FPIN) is a new feature for Brocade FOS
v9.0. It is available on Brocade Gen 6 and Gen 7 switches. This feature enables the switch to
detect issues on a fabric such as congestion or physical link issues and then notify the
affected devices that have registered for these notifications. The devices that receive these
notifications can then proactively take steps such as path failover rather than to react to a
path being down. FPIN provides a means to notify devices of link and other issues with a
connection or possibly a path through the fabric. For FPIN, a device must register with fabric
services to receive these notifications. The new IBM B-Type Gen 7 hardware can send
hardware and software signal notifications. Gen 6 can only send software notifications. Both
the hardware and software notifications require FOS v9.0 on the Switches and Directors.

Hardware signals can be sent from the switch to the adapter in the device. The adapter itself
can then decide what to do about the notification. Software signals are sent higher up in the
Fibre-Channel stack, and the adapter driver would then decide how to handle the
notifications. One advantage to notifications in hardware is reaction time - the adapter can
process the notifications and react more quickly than the driver can. Another is that the
hardware-based notification is a fibre-channel primitive. This means that even if buffer credits
are depleted the signal can still be delivered to the device on the other end of the link.
Primitives are not frames so do not need buffer credits to be sent. The software layer signal is
an ELS frame, so can affected by buffer credit depletion and other link congestion. Whether
the signal sent is hardware or software, how the devices handle the notifications is up to the
vendor of the adapter. Some may log the notification, some may take action. The action that
an adapter takes is also vendor specific. FPIN can alert devices about (1) End Device
Congestion (2) Device Link Integrity (CRC) and (3) Frame Drops.

At the time of writing this book, below are some vendors that support FPIN today:
򐂰 Linux Multi-Path in RHEL 8.2
򐂰 IBM AIX® - will register for Link Integrity and Congestion notifications
򐂰 Emulex - supports Congestion and Link Integrity notifications on Linux

Support for more HBA and Storage Controller vendors for FPIN is expected in the future.

FPIN can be sent from any device that supports. Potentially the storage, the host or the switch
can share this information. Having end-to-end capability to act on the data based on the FPIN
notifications, the SAN will self-heal resulting an end-to-end autonomous SAN infrastructure.

40 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch03.fm

3.8.1 Congestion signal primitives


Congestion Signal primitives are ASIC based and supported only on IBM B-Type Gen 7
platforms. Congestion Signal primitives are signals sent within link encoding to directly
connected fibre channel devices. Congestion Signal primitives notify peer Gen 7 platforms of
degraded performance due to oversubscription or credit stall conditions. Fabric Notifications
hardware intelligently detects sudden congestion and reacts instantly by signalling the
attached physically connected port.

Before receiving a Congestion Signal primitive Gen 7 ports must let peer ports know it is
capable. For example, IBM B-Type Gen 6 platforms do not support Congestion Signal
primitives but are capable of FOS-level ELS-based FPIN Congestion notifications.

3.8.2 Extended link service notifications


Extended Link Service (ELS) is a communication protocol used between Brocade Fabric
Operating System (FOS) platforms for exchanging capabilities, service registration,
notifications, and other vital control traffic. The Fabric Notifications solution augments
hardware Congestion Signal primitives with software FPIN notifications for Congestion, Peer
Congestion, Link Integrity, and SCSI Command Delivery. Fabric Notifications are FOS-based
and supported on Gen 6 and Gen 7 platforms running FOS 9.0 or later.

Hosts, switches, and storage can generate ELS notifications upon detecting an impacting
event. A notification may be sent by any fabric member detecting a problem, be it on a
switching platform (F_Ports and E_Ports) or an end-device (N_Ports); it will be processed the
same.

Fabric Notifications and MAPS work in tandem. MAPS rules for congestion, SCSI command
discard, and link integrity will trigger corresponding FPIN ELS notifications. FPIN notifications
are sent to all in-zone end-devices that have registered to receive that particular notification
type. Notifications are not dependent on zoning type, whether traditional or peer, the
operation is the same with no advantage or disadvantage. Additionally, Fabric Notifications
sends FPIN notifications to all peer switches and the peer switches forward notifications to
their registered in-zone end-devices as well. FPIN notifications are restricted to devices:
򐂰 Supporting FPIN notifications
򐂰 Registered to receive a particular notification type
򐂰 Experiencing the notification condition
򐂰 Are an in-zone peer device

3.8.3 EDC and RDF


Exchange Diagnostic Capabilities (EDC) and Register Diagnostic Functions (RDF) are ELS
services associated with Fabric Notifications. EDC is used between devices to exchange
diagnostic capabilities. RDF is used by devices to register for particular notifications. The
fabric will not transmit any notifications (Primitive or ELS) until a device has registered to
receive them. Devices may not be capable of dealing with every type of problem, therefore,
they may not register for every available notification.

Chapter 3. Gen 7 Fabric Operating System (FOS) overview 41


8497ch03.fm Draft Document for Review April 5, 2021 12:31 pm

EDC Response Exchange Diagnostic Capabilities Request

N_Port F_Port E_Port E_Port F_Port N_Port

Exchange Diagnostic Capabilities Request EDC Response

Host Gen 6 Gen 7 Storage

RDF Response Register Diagnostic Functions Request

N_Port F_Port E_Port E_Port F_Port N_Port

Register Diagnostic Functions Request RDF Response

Host Gen 6 Gen 7 Storage

Figure 3-6 ELS: EDC and RDF

FPIN Notifications a device can register to receive:


򐂰 Congestion notification descriptors are sent to devices on congested ports. Causes are
credit loss, credit stall, and oversubscription. Hosts and targets are equal offenders.
򐂰 Peer Congestion notification descriptors are sent upstream from congested ports to
in-zone registered devices. The FPIN notification lets the receiving device know the
reason for congestion.
򐂰 Link Integrity notifications are sent to the attached device and other in-zone end-devices
when a port experiences link integrity problems. CRC and ITW MAPS alerts are translated
into a Link Integrity FPIN notification and sent to registered devices. Link Integrity
notifications should be used by end-devices to troubleshoot link problems. MPIO can
prioritize a healthier path over an impaired path while continuing operation.
򐂰 Delivery (SCSI Command frame discard), a notification is sent to the registered originator
of a dropped SCSI command or status frame. When a command frame is dropped, the
originator is notified, and immediately the SCSI exchange is failed. Rather than waiting for
an upper-level protocol timeout, which can take tens of seconds, the driver retries
immediately.

3.8.4 Congestion signal


Congestion Signals are an immediate feedback mechanism between the devices of a link
indicating transmission resources are becoming consumed. Congestion Signal primitive
signals are immediately sent upon detection, within microseconds (µs). Congestion Signals
are a link-by-link feature only. Brocade Gen 7 Fibre Channel ASICs support direct congestion
signalling between platforms. In turn, if the condition causes MAPS to trigger, the receiving
switch uses FPIN Peer Congestion notification to other registered in-zone end-devices for
potential I/O queuing optimization and remediation.

42 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch03.fm

8G 16G
Congestion impacts
16G FC Servers other workloads

8G 16G

8G 16G

16G FC
Storage
IBM B-Type SAN A

8G 16G

Max

32G FC
Storage

Over-utilized server 16G FC


Server

Figure 3-7 Congestion signal and notification for oversubscription

3.8.5 Congestion notification


FPIN Congestion notifications are valuable information for end-devices, which facilitates
optimized I/O scheduling; i.e., consider slower transfer rates or issue serial read I/Os. The
ELS Congestion Notification is the software equivalent of the Congestion Signal primitive and
sent to end-devices supporting notifications. An ELS congestion notification for
oversubscription means too much data was requested, the fabric couldn’t accommodate the
request without congestion. An ELS congestion notification for credit stall indicates to the
end-device “you’re stuck - fix the situation”. In general, congestion notifications indicate why
there may be exchange completion times.

3.8.6 Congestion peer notification


FPIN Congestion Peer Notifications are sent to registered in-zone peers of end-devices who
are experiencing congestion. There are a variety of remedies peers can leverage to relieve
congestion. Speed matching – the peer’s port may have auto-negotiated to a faster speed
than the destination port; the peer could update its speed to match. Peers could choose
alternate pathways to potentially circumvent a flow causing congestion. Paths could be
weighted based on peer congestion notifications to proportionally deliver traffic over multiple
paths in a weighted fashion.

Chapter 3. Gen 7 Fabric Operating System (FOS) overview 43


8497ch03.fm Draft Document for Review April 5, 2021 12:31 pm

Hosts Storage
Zone A
Buffer credits withheld to
prevent buffer overflow.
Over- Con
Buffered
frames Application performance
Buffered
frames
A
subscribed! gest
impeded! Possible Solutions:
Peer Congestion Notification

• Match destination port speed?

Zone A • Alternate path?


Credit Con • Weighted multi-path?
Stall! gest
B

C
Peer congestion notifications indicate No
to devices there is a fabric impact. Notification
Not in Zone

Figure 3-8 Peer Congestion Notification

Below is an example of how the Congestion Notification works in real world, in collaboration
with Emulex HBA on the IBM AIX server, with the IBM B-Type Gen 7 SAN, IBM SANnav
Management software and Emulex SAN Manager, a management tool to manage host HBAs
in the environment.

Under normal operations, as shown in Figure 3-10, IBM SANnav management tool and the
Emulex SAN Manager report minimal to no congestion.

8G 16G

16G FC Servers
8G 16G

8G 16G

16G FC
Storage
IBM B-Type SAN A

16G 32G

32G FC
Storage

16G FC
Server

Figure 3-9 Normal operations

The workloads generally grows over time and becomes larger than the hardware can support.
As shown in Figure 3-11, this will have a ripple effect on the entire infrastructure causing the
congestion resulting in major performance issues.

44 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch03.fm

Figure 3-10 Oversubscription

IBM B-Type Gen 7 SAN detects these type of F-Port congestion issues and sends FPIN
Congestion Notification to the Emulex HBA on the AIX server. The Emulex SAN Manager,
management software to manage HBAs on the AIX server, is enabled to set congestion
policy. As shown in Figure 3-12, depending on the choice of the policy between conservative,
moderate and aggressive, the action will be taken to address the end point congestion issue.
For the bully server, the workload is still too big, so additional administrative attention is
required.

Chapter 3. Gen 7 Fabric Operating System (FOS) overview 45


8497ch03.fm Draft Document for Review April 5, 2021 12:31 pm

Figure 3-11 Addressing the congestion issue

3.8.7 Link Integrity


A complete solution requires end-devices (N_Port) to notify the fabric (F_Port) of receive-side
errors that the fabric would otherwise be unaware of. A link is a two-way street and FPIN Link
Integrity notifications enable the identification of degraded paths in each direction, from
N_Port à F_Port and F_Port à N_Port.

Notification F_Port à N_Port, HBAs register to receive FPIN notifications of interest, for
example, Link Integrity. When F_Port experiences incoming errors due to faulty electronics,
marginal optic, dirty connector, damaged fiber cable… a Link Integrity notification is sent to
the HBA letting the host know of the transmission problem. In turn, the HBA forwards
notifications to the MPIO driver, which will be able to distinguish between a device failure and
physical path failure. Link Integrity notifications are used by end-devices for troubleshooting
links and by MPIO for selecting the best path.

Notification N_Port à F_Port, the HBA registers with the fabric to receive and send FPIN
notifications. If the HBA sees incoming errors, it will notify the fabric that its connection to the
end-device is erratic. An FPIN Link Integrity notification sent by N_Port to F_Port (specifically,
the Fabric Controller) will be distributed to the in-zone devices as well. The fabric logs the
notification information so that the SAN administrator can take action.

Registered in-zone end-devices must be notified of link integrity problems. MAPS F_Port
alerts for CRC (Cyclic Redundancy Check) and ITW (Invalid Transmission Word) are
translated into Link Integrity FPINs and sent to the end-devices. Notifications integrate with
the OS device manager and notifying initiators and targets of link integrity issues along a data
path enhances debugging and recovery efforts from multiple perspectives.

46 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch03.fm

MPIO handles FPIN events and manages path selection. On a new pathway, MPIO drivers
communicate with the appropriate service to manage outstanding and pending IOs.
End-devices decide their best course of remediation like immediate failover to an alternate
path within the MPIO environment. Examples of Link Integrity problems:
򐂰 Slow host application performance can be traced to CRC errors on a storage port
򐂰 Slow array performance can be traced to host port ITWs

Below is an example of how the Link Integrity works in real world, in collaboration with Emulex
HBA on the AIX server and with the IBM B-Type SAN.

Under normal operations, as shown in Figure 3-13 below, with the Fibre Channel links
working, data is balanced between the different paths to the storage system. Over time, links
may not work as well as they should.

8G 16G

IBM B-Type SAN A

16G FC 16G FC
Server Storage

8G 16G

Max
IBM B-Type SAN B

Sick but not dead link

Figure 3-12 Fibre Channel normal operations

When link(s) are sick but not dead, as shown in Figure 3-14, I/Os going over the bad link
reduce the data available to the application. Quick detection and action is essential to improve
the performance and in some cases availability of the application.

8G 16G

IBM B-Type SAN A

16G FC 16G FC
Server Storage

8G 16G

Max
IBM B-Type SAN B

Sick but not dead link

Figure 3-13 The impact of a bad link

With the Link Integrity feature, as shown in Figure 3-14, when a sick but not dead path is
detected, as shown in Figure 3-15, FPIN-LI (Fabric Performance Impact Notification – Link

Chapter 3. Gen 7 Fabric Operating System (FOS) overview 47


8497ch03.fm Draft Document for Review April 5, 2021 12:31 pm

Integrity) notification is sent to the host HBA, then the Emulex HBA passes the FPIN up the
stack to the AIX multi-pathing software to address the issue by taking corrective actions.

Figure 3-14 Fabric detected link integrity issues

As shown in Figure 3-16, after the notification is received by the host HBA, the multi-pathing
software makes intelligent decisions based on the data from the IBM B-Type Gen 7 SAN
fabric to rebalance the traffic away from the bad link, resulting in improved availability and
minimized performance problems.

Figure 3-15 Addressing Link Integrity issues by host HBA

3.8.8 SCSI Command Delivery


When the fabric discards a command/status, Fabric Notifications notifies the originator of the
failure to deliver by sending an FPIN Delivery notification. No matter if the command is
dropped by an E_Port or F_Port, the originator is notified. The notification initiates an

48 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch03.fm

immediate SCSI I/O failure without waiting for a timeout. This allows the end-device to
immediately retry the discarded command avoiding long wait times and application disruption.

Figure 3-16 FPIN SCSI CMD delivery notification

The Brocade’s Fabric Notifications solution addresses each of these issues through a
technology eco system of innovation. Various notifications with problem specific descriptors
are sent to or received from attached end-devices, and in some cases propagated to other
Directors/switches to notify peer end-devices. Notifications are limited to registered in-zone
devices and the device must register for each notification type it wishes to receive. Fabric
Notifications compliments MAPS technology and was introduced by Broadcom BSN in FOS
9.0.

Chapter 3. Gen 7 Fabric Operating System (FOS) overview 49


8497ch03.fm Draft Document for Review April 5, 2021 12:31 pm

50 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch04.fm

Chapter 4. Virtual Fabrics overview


Virtual Fabrics is an architecture to virtualize hardware boundaries. Traditionally, SAN design
and management is done at the granularity of a physical switch. Virtual Fabrics allows SAN
design and management to be done at the granularity of a port.

Virtual Fabrics is a suite of related features that can be customized based on your needs. The
Virtual Fabrics suite consists of the following specific features:
򐂰 Logical switch
򐂰 Logical fabric
򐂰 Device sharing

Terminology: Virtual Fabrics is the name of the suite of features. A logical fabric is a type
of fabric that you can create using the Virtual Fabrics suite of features.

This chapter describes a logical fabric and logical switch types, as well as device sharing with
Virtual Fabrics.

Important: SNMPv3 is required to manage Virtual Fabrics. If you use SNMPGET to poll
the switches, for example HiTrack, the SNMPv3 user must be configured based on the
logical switch that is being monitored. If SNMP monitoring is for the default switch, you do
not need the SNMP user to be configured as a user on the switch. Whereas, when SNMP
monitoring is VF specific, you must create a user with permission for the corresponding VF
and add the created user into the SNMP database as a SNMPv3 user. When the SNMPv3
user is successfully added, you can use the same user name in management applications
for SNMP queries to the corresponding logical switch.

This chapter includes the following topics:


򐂰 4.1, “Logical switch” on page 52
򐂰 4.2, “Logical fabric” on page 57
򐂰 4.3, “Device sharing” on page 67

© Copyright IBM Corp. 2021. 51


8497ch04.fm Draft Document for Review April 5, 2021 12:31 pm

4.1 Logical switch


Traditionally, each switch and all the ports in the switch act as a single Fibre Channel (FC)
switch that participates in a single fabric. The logical switch feature allows you to divide a
physical chassis into multiple fabric elements. Each of these fabric elements is referred to as
a logical switch. Each logical switch functions as an independent self-contained FC switch.

4.1.1 Default logical switch


To use the Virtual Fabrics features, you must first enable Virtual Fabrics on the switch, which
creates a single logical switch in the physical chassis. This logical switch is called the default
logical switch.
The default logical switch initially contains all of the ports in the physical chassis.

Figure 4-1 shows a switch before and after enabling Virtual Fabrics. In this example, the
switch has 10 ports, labelled P0 through P9.

Figure 4-1 Switch before and after enabling virtual fabrics

Figure 4-2 on page 53 shows a Virtual Fabrics-enabled switch before and after it is divided
into logical switches. Before you create logical switches, the chassis appears as a single
switch (default logical switch). After you create logical switches, the chassis appears as
multiple independent logical switches. All of the ports continue to belong to the default logical
switch until you explicitly move them to other logical switches.

52 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch04.fm

Figure 4-2 Switch before and after creating logical switches

The default logical switch always exists. You can add and delete other logical switches, but
you cannot delete the default logical switch unless you disable Virtual Fabrics.

Note: A FICON logical switch or an existing logical switch that has been modified to
become a FICON logical switch cannot serve as the default logical switch.

4.1.2 Logical switches and fabric IDs


When you create a logical switch, you must assign it a fabric ID (FID). The fabric ID uniquely
identifies each logical switch within a chassis and indicates to which fabric the logical switch
belongs. You cannot define multiple logical switches with the same fabric ID within the
chassis.

In the Figure 4-3 on page 54, logical switches 2, 3, 4, and 5 are assigned FIDs of 1, 15, 8, and
20, respectively. These logical switches belong to different fabrics, even though they are in the
same physical chassis. For example, you could not assign logical switch 5 a fabric ID of 15,
because logical switch 3 is already assigned FID 15 in the chassis. The default logical switch
is initially assigned FID 128. You can change this value later.

Note: Each logical switch is assigned one and only one FID. The FID identifies the logical
fabric to which the logical switch belongs.

Chapter 4. Virtual Fabrics overview 53


8497ch04.fm Draft Document for Review April 5, 2021 12:31 pm

Figure 4-3 Fabric IDs assigned to logical switches

4.1.3 Port assignment in logical switches


Initially, all ports belong to the default logical switch. When you create additional logical
switches, they are empty, and you must assign ports to those logical switches. As you assign
ports to a logical switch, the ports are moved from the default logical switch to the newly
created logical switch. A given port can be in only one logical switch.

In Figure 4-4 on page 55, the default logical switch initially has 10 ports, labeled P0 through
P9. After logical switches are created, the ports are assigned to specific logical switches.
Note that ports 0, 1, 7, and 8 have not been assigned to a logical switch and so remain
assigned to the default logical switch.

54 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch04.fm

Figure 4-4 Assigning ports to logical switches

A given port is always in one (and only one) logical switch. The following scenarios refer to the
chassis after port assignment in Figure 4-4:
򐂰 If you assign P2 to logical switch 2, you cannot assign P2 to any other logical switch.
򐂰 If you want to remove a port from a logical switch, you cannot delete it from the logical
switch, but must move it to a different logical switch. For example, if you want to remove P4
from logical switch 3, you must assign it to a different logical switch: logical switch 2,
logical switch 4, or logical switch 1 (the default logical switch).
򐂰 If you assign a port to a logical switch, it is removed automatically from the logical switch it
is currently in. If you assign P3 to Logical switch 3, P3 is automatically removed from
logical switch 2.
򐂰 If you do not assign a port to any logical switch, it remains in the default logical switch, as
is the case with ports 0, 1, 7, and 8.

A logical switch can have as many ports as are available in the chassis. In FIGURE 4-4, the
chassis has 10 ports. You could assign all 10 ports to a single logical switch, such as logical
switch 2; if you did this, however, no ports would be available for logical switches 3 and 4.

You can move only F_Ports and E_Ports from one logical switch to another. If you want to
configure a different type of port, such as a VE_Port or EX_Port, you must configure them
after you move them. Some types of ports cannot be moved from the default logical switch.

4.1.4 Logical switches and connected devices


You can connect devices to logical switches, as shown in Figure 4-5 on page 56 In logical
switch 2, P2 is an F_Port that is connected to H1. In logical switch 3, P4 is an F_Port that is
connected to D1. H1 and D1 cannot communicate with each other because they are in
different fabrics, even though they are both connected to the same physical chassis.

Chapter 4. Virtual Fabrics overview 55


8497ch04.fm Draft Document for Review April 5, 2021 12:31 pm

You can also connect other switches to logical switches. In Figure 4-5 P6 is an E_Port that
forms an inter-switch link (ISL) between logical switch 4 and the non-Virtual Fabrics switch.
Logical switch 4 is the only logical switch that can communicate with the non-Virtual Fabrics
switch and D2, because the other logical switches are in different fabrics.

Figure 4-5 Logical switches connected to devices and non-Virtual Fabrics switches

Figure 4-6 shows the logical switch view of a chassis where virtual fabrics are enabled and
the devices are connected. You can see that the logical switch with Fabric 128, default switch
when virtual fabrics enabled has no devices connected. Logical switches Fabric 1 and
Fabric 15 are connected to Host and a Storage respectively and the logical switch Fabric 8 is
connected to a non-Virtual Fabric switch and storage.

Figure 4-6 Logical switches in a Single Chassis belong to separate fabrics

56 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch04.fm

4.1.5 Management model for logical switches


The operations you can perform on a logical switch depend on the context you are in. Some
operations affect only a single logical switch, and some operations affect the entire physical
chassis.

All user operations are classified into one of the following:


򐂰 Chassis management operations
򐂰 These are operations that span logical switch boundaries, such as:
– Logical switch configuration (creating, deleting, or modifying logical switches)
– Account management (determining which accounts can access which logical switches)
– Field-replaceable unit (FRU) management (slot commands, such as slotShow)
– Firmware management (firmware upgrade, HA failover)

These are operations that are limited to the logical switch, such as displaying or changing port
states. Logical switch operations include all operations that are not covered in the chassis
management operations.

When you log in, you are assigned an active context, or active logical switch. This context
filters the view that you get, and determines which ports you can see. You can change the
active context. For example, if you are working with logical switch 1, you can change the
context to logical switch 5. When you change the context to logical switch 5, you only see the
ports that are assigned to that logical switch. You do not see any of the other ports in the
chassis.

The scope of logical switch operations is defined by the active context. When you are in the
context of a logical switch, you can perform port, switch, and fabric-level operations, subject to
Role-Based Access Control (RBAC) rules.

If you have permission to execute chassis-level commands, you can do so, regardless of
which logical switch context you are in.

4.2 Logical fabric


A logical fabric is a fabric that contains at least one logical switch.

The four fabrics shown in Figure 4-5 on page 56 and Figure 4-6 on page 56 are logical fabrics
because they each have at least one logical switch. You can connect logical switches to
non-Virtual Fabrics switches and to other logical switches. You connect logical switches to
non-Virtual Fabrics switches using an ISL, as shown in Figure 4-5 on page 56.

You connect logical switches to other logical switches in two ways:


򐂰 Using ISLs
򐂰 Using base switches and extended ISLs (XISLs)

4.2.1 Logical fabric and ISLs


Figure 4-7 on page 58 shows two physical chassis divided into logical switches. In the
following figure, ISLs are used to connect the logical switches with FID 1 and the logical
switches with FID 15. The logical switches with FID 8 are each connected to a non-Virtual

Chapter 4. Virtual Fabrics overview 57


8497ch04.fm Draft Document for Review April 5, 2021 12:31 pm

Fabrics switch. The two logical switches and the non-Virtual Fabrics switch are all in the same
fabric, with FID 8.

Figure 4-7 Logical switches connected to other logical switches through physical ISLs

The ISLs between the logical switches are dedicated ISLs because they carry traffic only for a
single logical fabric. In Figure 4-8, Fabric 128 has two switches (the default logical switches),
but they cannot communicate with each other because they have no ISLs between them and
they cannot use the ISLs between the other logical switches.

Figure 4-8 Logical switches connected to form logical fabrics

Note: Only logical switches with the same FID can form a fabric. If you connect two logical
switches with different FIDs, the link between the switches segments.

4.2.2 Base switch and extended ISLs


One method to connect logical switches is to use extended ISLs and base switches.

58 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch04.fm

When you divide a chassis into logical switches, you can designate one of the switches to be
a base switch. A base switch is a special logical switch that is used for interconnecting the
physical chassis.

A base switch has the following properties:


򐂰 ISLs connected through the base switch can be used for communication among the other
logical switches.
򐂰 Base switches do not support direct device connectivity. A base switch can have only
E_Ports, VE_Ports, or EX_Ports but no F_Ports.
򐂰 The base switch provides a common address space for communication between different
logical fabrics.
򐂰 A base switch can be configured for the preferred domain ID just like a non-Virtual Fabrics
switch.
򐂰 You can have only one base switch in a physical chassis.

A base switch can be connected to other base switches through a special ISL, called a shared
ISL or extended ISL (XISL). An extended ISL connects base switches. The XISL shares traffic
among different logical fabrics.

Note: A FICON logical switch or an existing logical switch that has been modified to
become a FICON logical switch cannot serve as the base switch.

Fabric formation across an XISL is based on the FIDs of the logical switches.

Figure 4-9 shows two physical chassis divided into logical switches. Each chassis has one
base switch. An ISL connects the two base switches. The ISL is an extended ISL (XISL)
because it connects base switches.

Figure 4-9 Base switches connected by an XISL

Chapter 4. Virtual Fabrics overview 59


8497ch04.fm Draft Document for Review April 5, 2021 12:31 pm

Traffic between the logical switches can now flow across this XISL. The traffic can flow only
between logical switches with the same fabric ID. For example, traffic can flow between logical
switch 2 in chassis 1 and logical switch 6 in chassis 2, because they both have FID 1. Traffic
cannot flow between logical switch 2 and logical switch 7, because they have different fabric
IDs (and are, therefore, in different fabrics).

It is useful to think of the logical switches as being connected with logical ISLs, as shown in
the following figure. The logical ISLs are not connected to ports, because they are not
physical cables. They are a logical representation of the switch connections that are allowed
by the XISL.

You can also connect logical switches using a combination of ISLs and XISLs, as shown in
Figure 4-10. In this diagram, traffic between the logical switches in FID 1 can travel over either
the ISL or the XISL. Traffic between the other logical switches travels only over the XISL.

Figure 4-10 Logical fabric using ISLs and XISLs

By default, the physical ISL path is favoured over the logical path (over the XISL) because the
physical path has a lower cost. This behavior can be changed by configuring the cost of the
dedicated physical ISL to match the cost of the logical ISL.

Attention: If you disable a base switch, all of the logical ISLs are disconnected and the
logical switches cannot communicate with each other unless they are connected by a
physical ISL.

Base fabric
Base switch ports on different chassis can be connected together to form a fabric, called a
base fabric. Similar to other logical switches, the base switches must have the same FID to be
connected. If the base switches have different FIDs, the link between the switches is disabled.

The base fabric follows normal routing policies. As long as physical connectivity is available,
the base fabric maintains connectivity for the logical fabrics.

60 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch04.fm

Logical ports
Logical ISLs are formed to connect logical switches. A logical port represents the ports at
each end of a logical ISL. A logical port is a software construct only and does not correspond
to any physical port.

Most port commands are not supported on logical ports. For example, you cannot change the
state or configuration of a logical port. However, state change on a logical link is permitted
using the lfcfg --lislEnable command.

The World Wide Name (WWN) for logical ports is in NAA=5 format, using the following
syntax: 5n:nn:nn:nz:zz:zz:zx:xx

The NAA=5 syntax uses the following variables:


򐂰 nnnnnn is the Organizationally Unique Identifier (OUI).
򐂰 zzzzzz is the logical fabric serial number.
򐂰 xxx is the logical port number, in the range 0 through FFF.

Logical fabric formation


Fabric formation is not based on connectivity, but on the FIDs of the logical switches. The
basic order of fabric formation is as follows:
򐂰 Base fabric forms.
򐂰 Logical fabrics form when the base fabric is stable.
򐂰 Devices begin recognizing one another.
򐂰 Traffic is initiated between the logical switches.

4.2.3 Account management and Virtual Fabrics


When user accounts are created, they are assigned a list of logical fabrics to which they can
log in and a home logical fabric (home FID). When you connect to a physical chassis, the
home FID defines the logical switch to which you are logged in by default. You can change to
a different logical switch context using setcontext command.

When you are logged in to a logical switch, the system prompt changes to display the FID of
that switch. The following are example prompts for when you are logged in to the default
logical switch (FID = 128) and a user-defined logical switch (FID = 15):
switch:FID128:admin>
switch:FID15:admin>

4.2.4 Setting up IP addresses for a logical switch


Each physical chassis has one common IP address that is shared by all of the logical
switches in the chassis. You can set up individual IPv4 addresses for each logical switch.

IPv4 addresses assigned to individual logical switches are assigned to IP over Fibre Channel
(IPFC) network interfaces. In Virtual Fabrics environments, a single chassis can be assigned
to multiple fabrics, each of which is logically distinct and separate from one another. Each
IPFC point of connection to a given chassis needs a separate IPv4 address and prefix to be
accessible to a management host.

For a management host to access logical switches, the host bus adapter (HBA) must be able
to connect with the common, shared IP address and the individual IPv4 addresses configured
for each logical switch.

Chapter 4. Virtual Fabrics overview 61


8497ch04.fm Draft Document for Review April 5, 2021 12:31 pm

Note: IPv6 is not supported when setting the IPFC interface for Virtual Fabrics.

IPFC addresses are not handled by configupload or configdownload. The IPFC address of
the default logical switch or a non-VF switch is stored on the WWN card or compact flash.
This address does not display in a configshow. Non-default logical switch IPFC addresses
display in a confgshow. The ipaddrshow command displays all switch addresses and IPFC
addresses configured in the chassis.

Use the following procedure to set up IP addresses for a logical switch:


򐂰 Connect to the switch and log in using an account with admin permissions.
򐂰 Enter the ipAddrSet -ls command.
– To add an IPv4 address, use the --add parameter. Specify the network information in
dotted-decimal notation for the Ethernet IPv4 address with a Classless Inter-Domain
Routing (CIDR) prefix.
– To delete an IPv4 address, use the --delete parameter.
򐂰 Enter the ipaddrshow command to verify the result.

Example 4-1 shows how to sets IP addresses with the CIDR prefix for logical switches with
FID 1, 2, and 128 (default logical switch).

Example 4-1 Setting IP addresses


switch:FID128:admin> ipaddrset -ls 1 --add 192.0.2.11/24
IP address is being changed...Done.
switch:FID128:admin> ipaddrset -ls 2 --add 192.0.2.22/24
IP address is being changed...Done.
switch:FID128:admin> ipaddrset -ls 128 --add 192.0.2.0/24
IP address is being changed...Done.

switch:FID128:admin> ipaddrshow

SWITCH
Ethernet IP Address: 198.51.100.0
Ethernet Subnetmask: 255.255.255.0
Gateway IP Address: 198.51.100.1
DHCP: Off
IPFC address for virtual fabric ID 128: 192.0.2.0/124
IPFC address for virtual fabric ID 1: 192.0.2.11/24
IPFC address for virtual fabric ID 2: 192.0.2.22/24

Example 4-2 show how to delete an IP address for the logical switch with FID 1.

Example 4-2 Deleting an IP address


switch:FID128:admin> ipaddrset -ls 1 –delete

4.2.5 Logical switch login context


When you log in to a logical switch with Telnet or SSH, the login shell uses the destination
IPFC address to look up and set the corresponding logical switch context. When you log in
from the serial console, the login shell always sets your default home logical switch context.
The IPFC assigned to a logical switch is managed by the ipAddrSet command. When you

62 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch04.fm

use Telnet or SSH from a remote host, the IPFC address must share the same network
(subnet mask) as the management interface. The values of FCIP addresses and subnet
masks are stored in the WWN card. For example, if the management interface IP address is
10.38.18.183 with subnet mask 255.255.240.0, you can create the IPFC address (see
Example 4-3).

Example 4-3 Adding an IP address


ipaddrset –ls 8 --add 10.38.18.183/20

Note: The logical switch login context supersedes the default FID context of a user. The
CIDR subnet mask of the IPFC address must match the CIDR of the assigned
management IP address to avoid routing issues.

To delete the IPFC address, you can use the ipaddrset –ls 8 –delete command.

If you log in to a switch with the management interface IP address, the home virtual fabric is
set for the context. You can log in to the logical switch context by using the telnet or SSH
service with the configured IPFC address. You can manage the permissions using the
userConfig command (see Example 4-4).

Example 4-4 Using the userConfig command


switch> userconfig --show admin
Account name: admin
Description: Administrator
Enabled: Yes
Password Last Change Date: Unknown (UTC)
Password Expiration Date: Not Applicable (UTC)
Locked: No
Home LF Role: admin
Role-LF List: admin: 1-128
Chassis Role: admin
Home LF: 128
Day Time Access: N/A

Role-LF is the list of logical switch contexts for which you have permission to log in over the
IPFC address. Home LF Role is the default logical switch context when you have no
permission to log in to a particular logical switch context or over management interface.
Example 4-5 shows the result of the ipAddrShow command. Only one IPFC is mapped to
FID 8.

Example 4-5 Using the ipAddrShow command


switch> ipaddrshow
SWITCH
Ethernet IP Address: 10.38.18.183
Ethernet Subnetmask: 255.255.240.0
Gateway IP Address: 10.38.16.1
DHCP: Off
IPFC address for virtual fabric ID 8: 10.38.18.183/20

Chapter 4. Virtual Fabrics overview 63


8497ch04.fm Draft Document for Review April 5, 2021 12:31 pm

The following conditions apply for default users:


򐂰 For admin and root users, the login is successful to the given logical switch with the
corresponding logical switch context set (FID 1 to 128). When you telnet to IPFC
10.38.18.183, the context is set to FID 8.

Note: If the home VF of a user is not present on the switch, then the user will be logged
in to default VF if the default VF is part of the user's VFs list. If not, the user will be
logged in to the lowest VF that is available on the switch that is part of the user's VFs
list.

The following conditions apply for non-default users:


򐂰 For non-default users configured in the switch with or without the logical switch
permission, if the user logs in with the IPFC address without the permission for the
corresponding logical switch, only the default home virtual fabric of the user is set for the
context.
򐂰 For non-default users configured in the switch with or without the logical switch
permission, if the user logs in with the IPFC address with the permission for the
corresponding logical switch, login is successful to the given logical switch with the
corresponding logical switch context set.

4.2.6 Supported platforms for Virtual Fabrics


The following platforms supports the Virtual Fabrics:
򐂰 IBM Storage Networking SAN512B-7
򐂰 IBM Storage Networking SAN256B-7
򐂰 IBM Storage Networking SAN64B-7

Some restrictions apply to the ports, depending on the port type and blade type. The following
sections explain these restrictions.

Supported port configurations in the fixed-port switch


No restrictions exist on the ports in IBM SAN64B-7 switches.

The following rules apply:


򐂰 Any port can belong to any logical switch (including the base switch and default logical
switch), with the exception that F_Ports cannot belong to the base switch.
򐂰 The default logical switch can use XISLs.
򐂰 The default logical switch can also be a base switch except on Gen 7 directors and Gen 6
directors.

Supported port configuration in directors


Some of the ports in the Gen 7 directors are not supported on all types of logical switches.
Table 4-1 on page 65 lists the blades and ports that are supported on each type of logical
switch.

64 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch04.fm

Table 4-1 Blade and port types supported on logical switches


Blade Type Default Logical User Defined Logical Base Switch
Switch Switch

FC64-48: FC Port Yes (F, E) Yes (F, E) Yes (F, E)

FC32-X7-48: FC Port Yes (F, E) Yes (F, E) Yes (F, E)

FC32-64: FC Port Yes (F, E) Yes (F, E) Yes (F, E)


FC32-64: Eth Port Yes Yes Yes

SX6: FC ports Yes (F, E) Yes (F, E) Yes (F, E)


GE ports Yes (VE) Yes (VE) Yes (VE)

ICL ports Yes (E) Yes (E) Yes (E,EX)

Virtual Fabrics interaction with other Fabric OS features


Table 4-2 lists some Fabric OS features and considerations that apply when using Virtual
Fabrics.

Table 4-2 Virtual Fabrics interaction with Fabric OS features supported logical switch creation limits
Fabric OS Features Virtual Fabrics Interaction

Access Gateway Virtual Fabrics is not supported on a switch if AG


mode is enabled.

Configuration upload and download Virtual Fabrics uses a configuration file that is
different from the configuration file used to
download system configuration parameters.

FC-FC routing service All EX_Ports must reside in a base switch. You
cannot attach EX_Ports to a logical switch that
has XISL use enabled. You must use ISLs to
connect the logical switches in an edge fabric.

FICON Up to two logical switches per chassis can run


FICON
Management Server (CUP), but the FICON
logical switch can use both ISLs and XISLs.

ISL R_RDY mode ISL R_RDY mode is supported in both the base
switch and the logical switch.

Licensing Licenses are applicable for all logical switches in


a chassis.

Network time protocol (NTP) The clock information for a chassis is maintained
by the default logical switch and not from the
principal or primary FCS switch.

QoS QoS VCs are maintained across the base fabric.

Traffic isolation Traffic Isolation with Failover-Disabled (TI FD)


zoning is not supported over LISLs; Traffic
Isolation zones with Failover-Enabled (TI FE) is
supported. TI FD zoning is supported over DISLs
(physical links). TI FD zoning is not supported in
the base fabric.

Chapter 4. Virtual Fabrics overview 65


8497ch04.fm Draft Document for Review April 5, 2021 12:31 pm

Limitations and restrictions of Virtual Fabrics


Before you use the Virtual Fabrics feature, you should be aware of the restrictions and
limitations regarding QSFP ports and the maximum number of logical switches per chassis.

In the core blades, the four ports belonging to the same QSFP can be assigned to different
logical switches. However, if one of the four ports belongs to the base switch, the remaining
three ports cannot be assigned to any other logical switch.

The maximum number of logical switches per chassis varies depending on the switch model.
The following table lists the supported platforms and the maximum number of logical switches
(including the default logical switch) supported on each.

Table 4-3 Supported logical switch creation limits


Platform Maximum Logical Switches / Chassisa

IBM Storage Networking SAN512B-7 16

IBM Storage Networking SAN256B-7 16

IBM Storage Networking SAN64B-7 4


a. Numbers include the default switch and base switch.

Note: If a blade slot is being decommissioned and has ports configured in logical switches,
the logical port assignments should be removed from that blade before removing the blade.
This action ensures a seamless transition for any new port or AP blade that might occupy
that slot in the future and does not apply if you are simply replacing a blade of the same
type.

Restrictions on using XISLs


The Allow XISL Use option of the configure command, allows a logical switch to use XISLs
in the base switch as well as any standard ISLs that are connected to that logical switch.

The following restrictions apply when using XISLs. XISL use is not permitted in any of the
following scenarios:
򐂰 The logical switch has ICL ports.
򐂰 The logical switch is the default logical switch.
򐂰 The logical switch is a base switch.
򐂰 The logical switch is an edge switch for an FC router.
In this case, if the logical switch is enabled, you cannot allow XISL use. If the logical switch
is disabled or has not yet joined the edge fabric, you can allow XISL use; however, fabric
segmentation occurs when the logical switch is enabled or connected to an edge fabric.

Note: Using XISL and fmsmode at the same time is permitted, but this combination will
only work in a one-hop topology.

Restrictions on moving ports


FC ports cannot be moved if any one of the following features is enabled:
򐂰 Long distance
򐂰 QoS
򐂰 F-Port buffers
򐂰 F_Port trunking
򐂰 E_Port credits

66 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch04.fm

򐂰 SIM_Port
򐂰 Encryption

Before moving VE_Ports, you must remove the VE_Port tunnel configuration.

VE_Ports on the SX6 blade can be moved to any logical switch independent of the location of
the physical GE port.

Refer to FOS Admin guide https://docs.broadcom.com/docs/53-1004392 for further details on


how to execute a command in a different logical switch, Changing the fabric ID of a logical
switch, changing a logical switch to base switch.

4.3 Device sharing


The behavior of device sharing in the fabric depends on whether Virtual Fabrics is enabled or
not enabled in the base switch. If the Virtual Fabrics is not enabled, FC-FC routing behavior is
unchanged. If Virtual Fabrics is enabled, then in the FC-FC routing context, a base switch is
like a backbone switch and a base fabric is like a backbone fabric.

FC-FC routing and Virtual Fabrics


If Virtual Fabrics is enabled, the following rules apply:
򐂰 EX_Ports can be configured only on the base switch. When you enable Virtual Fabrics, the
chassis is automatically rebooted. When the switch comes up, only one default logical
switch is present, with the default fabric ID (FID) of 128. All previously configured
EX_Ports are persistently disabled with the reason ExPort in non-base switch. You must
explicitly create a base switch, move the EX_Ports to the base switch, and then enable the
ports. If you move existing EX_Ports to any logical switch other than the base switch,
these ports are automatically disabled. If you want to change an EX_Port on the logical
switch to be a non-EX_Port, you must use the portCfgDefault command. You cannot use
the portCfgExPort command because that command is allowed only on the base switch.
򐂰 From FOS v9.0.x onwards, EX_Ports in a base switch receives a 10-bit dynamic area
address instead of a zero bit address. An EX_Port in the base switch is assigned any
10-bit area from 0x0000 - 0xFFC0, with the following exceptions as these areas are
reserved for internal lookups, Areas 0x8000-8300, 0x8040-8340, Areas 0x8080-8380,
Areas 0x80C0-83C0, 0xC500, 0xC540, 0xC580 and 0xC5C0.
򐂰 EX_Ports can connect to a logical switch that is in the same chassis or in a different
chassis. However, the following configuration rules apply:
– If the logical switch is on the same chassis, the EX_Port FID must be set to a different
value than the FID of the logical switch to which it is connecting.
– If the logical switch is on a different chassis, no FID for any logical switch in the FC
router backbone fabric can be the same as the FID of the logical switch to which the
EX_Port is connecting.
򐂰 EX_Ports in FC routers and those in a base switch cannot connect to any edge fabric with
logical switches configured to use XISLs. If you connect an EX_Port to an edge fabric, you
must ensure that there are no logical switches with XISL use enabled in that edge fabric. If
any logical switch in the edge fabric allows XISL use, the EX_Port is disabled. Because
XISL use is disallowed, dedicated links must be configured to route traffic across switches
in the same logical fabric.
򐂰 Backbone-to-edge routing is not supported in the base switch, see Figure 4-7 on page 58
for an example of legacy FC routers that allow backbone-to-edge routing with Virtual
Fabrics.

Chapter 4. Virtual Fabrics overview 67


8497ch04.fm Draft Document for Review April 5, 2021 12:31 pm

򐂰 All FC router commands can be executed only in the base switch context.
򐂰 The fcrconfigure --bbfid command is not allowed when Virtual Fabrics is enabled.
Instead, use the lscfg command to configure the FID.
򐂰 From FOS v9.0.x onwards, if a port is configured as an EX_Port and if the administrator
moves the port from the base switch to another logical switch, the operation fails. You must
remove the EX_Port configuration before moving it to another logical switch.
򐂰 When a Virtual Fabrics-enabled switch is part of the backbone, only the base switch with
active EX_Ports will be able to route frames between edge fabrics. If there are other base
switches that are participating in the fabric and do not have active EX_Ports, those
switches will participate as transient switches to forward frames to the next hop till they
can be routed to the intended remote edge fabric. If any other default or logical switch is
introduced as a hop between edge fabrics, the newly introduced switch which is not the
base switch will not merge with the base switch fabric. This will cause frames to be
dropped.

For further information about device sharing and FC-FC routing, refer to the Fabric OS
Administration Guide: https://docs.broadcom.com/docs/FOS-90x-Admin-AG

68 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch05.fm

Chapter 5. Fibre Channel SAN design,


implementation, and migration
This document is a high-level storage area networking (SAN) design and best-practices guide
based on IBM b-type Gen 7 products and features, focusing on Fibre Channel SAN design.
The storage landscape continues to modernize, and multiple choices must be made to design
the right Fibre Channel architecture. Covered topics include the early planning phase,
understanding possible operational challenges, and monitoring and improving an existing
SAN infrastructure.

Modern applications have an insatiable need for data. The rise of NVMe flash and
storage-class memory will drastically accelerate the performance of storage systems tasked
with holding that data. This means that having the right network technology in place is
essential. Fortunately, IBM b-type Gen 7 networking solution delivers an autonomous SAN
with next-level performance and integrated intelligence. IBM b-type Gen 7 provides the
accelerated foundation that modern data centers need to fully realize the benefits of the latest
storage technology today and possibly for the next decade.

This chapter describes a high-level guide focusing on Fibre Channel Storage Area Network
(SAN) design and best practices, covering planning, topologies, workload monitoring, and
detecting server and storage latencies—to help with decisions required for successful IBM
b-type Gen 7 SAN design.

This chapter includes the following topics:


򐂰 5.1, “Modernizing SAN infrastructures” on page 70
򐂰 5.2, “Fibre Channel SAN design” on page 71
򐂰 5.3, “Implementation planning for a Fibre Channel SAN” on page 82
򐂰 5.4, “Migration strategies of Fibre Channel SAN infrastructure” on page 90
򐂰 5.5, “References” on page 98

© Copyright IBM Corp. 2021. 69


8497ch05.fm Draft Document for Review April 5, 2021 12:31 pm

5.1 Modernizing SAN infrastructures


Technology is evolving at an incredible pace, and businesses are demanding more from their
IT resources and infrastructure. Rapid adoption of flash storage and the ramp-up of
NVMe-based storage unleash advancements in application design that drive new levels of
performance and capacity requirements, such as advanced analytics, business intelligence,
and data-intensive workloads. With the integration of new technologies that are accelerating
the delivery of data and services, the network will need to evolve to keep pace with
innovations in storage and demands of modern applications to maximize the full value of
investments in next-generation data center infrastructure. To meet ever-increasing demands
for faster, more reliable data access, it is essential for organizations to deploy a modernized
infrastructure that reduces latency, increases bandwidth, and ensures continuous availability.
Unprecedented performance is not enough on its own. Powerful analytics and advanced
automation capabilities are required to transform current storage networks into an
autonomous SAN. This requires a network that is capable of delivering these capabilities to
maximize performance, simplify management, and reduce operational costs. Legacy
infrastructure was not designed to support the performance requirements of evolving
workloads and NVMe-based storage.

In fact, an aging network will impede the performance of the on-demand data center. By
modernizing the storage network with IBM b-type Gen 7, organizations will enable a faster,
more intelligent, and more resilient network. This will maximize the performance, productivity,
and efficiency of their storage investments and resources, even as they rapidly scale their
environments.

5.1.1 IBM b-type Gen 7 platforms: the intelligent foundation of modern data centers
The products that make up IBM b-type Gen 7 portfolio are the new Brocade X7 Director and
the G720 Switch. These solutions are equipped with higher-performing hardware to unleash
NVMe technology, and they have the ability to discover and produce comprehensive
telemetry data across the fabric. They analyze and take actions based on that data to
optimize the storage network automatically.

Modernize with NVMe to Achieve Lower latency and Higher Bandwidth According to IBM
b-type Gen 7 technology substantially accelerates a data center environment, offering 50%
lower latency than the previous generation, while increasing bandwidth with 64 Gb/s links. By
leveraging NVMe-ready Fibre Channel technology, IBM b-type Gen 7 not only maximizes the
performance of NVMe storage and high-transaction workloads, but also simplifies the process
of integrating higher-performing NVMe storage into the environment without a rip-andreplace
process. IBM b-type Gen 7 has simplified IT modernization efforts. The NVMe storage
integrates seamlessly with existing ports: Brocade supports the concurrent use of SCSI and
NVMe technologies. In fact, with the release of vSphere 7 from VMware, an organization can
do a live migration of an individual virtual machine from SCSI to NVMe in the same Gen 6 or
Gen 7 fabric without stopping the application. This result constitutes one of the most risk
averse technology transitions in years.

IBM b-type Gen 7 provides investment protection by enabling organizations to upgrade their
existing Brocade Gen 6 director chassis to Gen 7. Organizations can leverage existing Gen 6
blades in Gen 7 directors, mixing Gen 6 and Gen 7 blades in the same chassis. In addition,
Brocade supports three generations of backward compatibility within the fabric. This is an
important feature, as it reduces risk and maximizes the value of the organization’s existing IT
investments.

70 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch05.fm

5.2 Fibre Channel SAN design


NVMe-ready Fibre Channel technologies shift leads to a set of demands on the design of
storage area networks. This is true regardless of the technology being used. If the
expectation is to be able to utilize the capacity and performance of these platforms, then
serious consideration has to be given to the design.

Dedicated storage network infrastructure that provides lossless, low-latency, deterministic,


scalable, and performant storage services to the applications becomes critical. Storage
admins will talk about “fan-in” or “fan-out” ratios for storage platforms, which are the number
of servers/applications in the network that are using a particular array or array port for their
storage access. Depending upon the types of applications and their performance needs, that
ratio may range anywhere from low single digits to 40 to 50 servers (hundreds of virtual
machines). As with any provisioning scenario, the storage admin is dealing with projections of
how much capacity and performance any server/application is going to use. But application
performance is variable; time of day, week, month, and seasonal or event-driven events cause
spikes or drops in application demand.

Another consideration here is that the entire application base does not refresh at the same
time. And in fact, the more common scenario is that multiple generations of performance will
exist in the environment simultaneously. This is driven by the ability to take a service window
to re-platform existing environments to new server and storage acquisitions as well as the fact
that some legacy applications may not have an environment that runs in current operating
system (OS) releases or application versions. One may have 10-year-old or older operating
systems running connected with a host bus adapter (HBA) that is two or more older
generations of technology behind. How then do you balance that still critical application
against the needs and performance of the newer machines?

The answer to this is a combination of topology, balanced provisioning, granular monitoring,


and automated mitigation.

5.2.1 SAN design basics


This section provides high-level guidelines for installing a typical SAN. The focus is on best
practices for core-edge or edge-core-edge fabrics. The discussion starts at the highest level,
the data center, and works down to the port level, providing recommendations at each point
along the way.

Topologies
A typical SAN design comprises devices on the edge of the network, switches in the core of
the network, and the cabling that connects all the devices together. Topology is usually
described in terms of how the switches are interconnected, such as collapsed core,
core-edge, and fully meshed. The recommended SAN topology to optimize performance,
management, and scalability is a tiered, core-edge topology. This approach provides good
performance without unnecessary interconnections. At a high level, the tiered topology has a
large number of edge switches used for device connectivity and a smaller number of core
switches used for routing traffic between the edge switches, as shown in Figure 5-1 on
page 72.

Chapter 5. Fibre Channel SAN design, implementation, and migration 71


8497ch05.fm Draft Document for Review April 5, 2021 12:31 pm

Figure 5-1 Tiered network topologies

The difference between these three scenarios is device placement (where devices
are attached to the network) and the associated traffic flow:
򐂰 Scenario A, collapsed core, has localized traffic, which could sit on the same ASIC, on the
same blade, or between blades. Collapsed Core can have small performance advantages
for performance-optimized workloads but does not provide ease of scalability and
depending on how many fabrics you would need, would determine the impact on
manageability.
򐂰 Scenario B, core-edge, separates the storage and servers, thus providing ease of
management and higher scalability. An core-edge topology has only one hop from server
to storage, providing similar performance benefits as full mesh while allowing higher
scalability.
򐂰 Scenario C is a full-mesh topology, and server to storage is no more than one hop.
Designing fabrics with UltraScale ICLs is an efficient way to save front-end ports, and
users can easily build a large (for example, 3456 ports or larger) fabric with minimal SAN
design considerations.

Collapsed core
The collapsed core topology places initiators (servers) and storage (targets) on the same
ASIC, line card or same chassis. This has a number of benefits depending on the size of the
environment. This is used a lot when customers are migrating from multiple switches to a
single dual core architecture where they can fit all imitators and storage into the same
chassis. If future design requirements are needed going with an core-edge could be more
beneficial in the long run when it comes to scale and ease of management.

Core-edge
The core-edge topology places initiators (servers) on the edge tier and storage (targets) on
the core tier. Since the servers and storage are on different switches, this topology provides
ease of management as well as good performance with minimal fabric latency, with most
traffic traversing only one hop from the edge to the core. (Storage-to-storage traffic is two
hops if the second storage is on another core switch, but the two cores can be connected if

72 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch05.fm

fabrics are redundant.) The disadvantage to this design is that the storage and core
connections are in contention for expansion as they scale; however, using modular platforms
allows for flexibility while allowing the use of ICL ports for intraswitch connectivity to free up
device ports.

Full mesh
A full-mesh topology allows you to place servers and storage anywhere, since communication
between source and destination is no more than one hop. Using director-class switches with
UltraScale ICL ports for interconnectivity is essential to this design in order to ensure
maximum device port availability and utilization. Design this architecture with a minimum of
two switches up to nine switches in a full mesh.

5.2.2 Capabilities you need to consider for your storage architecture


There are a considerable number of critical capabilities to contemplate when choosing the
right storage strategy, some of the more important being availability, scalability, performance,
extensibility, and manageability. Let’s discuss each in further detail.

Availability
The key to high availability and enterprise-class installation is redundancy. By eliminating a
single point of failure, business continuance can be provided through most foreseeable and
even unforeseeable events. At the highest level of fabric design, the complete network should
be redundant, with two completely separate fabrics that do not share any network equipment
(routers or switches).

Recommendations for the SAN design are to ensure application availability and resiliency via
the following:
򐂰 Redundancy built into fabrics to avoid a single point of failure
򐂰 Servers connected to storage via redundant fabrics
򐂰 MPIO-based failover from server to storage
򐂰 Redundant fabrics based on similar architectures
򐂰 Redundant ISLs/ICLs for interswitch connectivity
򐂰 Separate storage and server tiers for independent expansion
򐂰 Core switches of equal or higher performance compared to the edges
򐂰 The highest performance switch in the fabric defined to be the principal switch

Scalabilty
IBM b-type Gen 7 products are designed with scalability in mind, knowing that most
installations will continue to expand and that growth is supported with very few restrictions.
However, following the same basic principles outlined in previous sections as the network
grows will ensure that the levels of performance and availability will continue.

Evaluate the impact on topology, data flow, workload, performance, and perhaps most
importantly, redundancy and resiliency of the entire fabric any time one of the following
actions is performed:
򐂰 Adding or removing initiators:
– Changes in workload
– Changes in provisioning

Chapter 5. Fibre Channel SAN design, implementation, and migration 73


8497ch05.fm Draft Document for Review April 5, 2021 12:31 pm

򐂰 Adding or removing storage:


– Changes in provisioning
– Changes in storage media type (e.g., increased deployment of flash-based storage)
򐂰 Adding or removing switches
򐂰 Adding or removing ISLs and ICLs
򐂰 Change in virtualization (workload and storage) strategies and traffic flow pattern If these
design best practices are followed when the network is deployed, then small incremental
changes should not adversely impact the availability and performance of the network.
However, if changes are ongoing and the fabric is not properly evaluated and updated,
then performance and availability can be jeopardized.

Some key points to cover when looking at the current status of a production FC network when
reviewing redundancy and resiliency:
򐂰 Are there at least two physically independent paths between each source and destination
pair?
򐂰 Are there two redundant fabrics?
򐂰 Does each host connect to two different edge switches?
򐂰 Are edge switches connected to at least two different core switches?
򐂰 Are inter-switch connections composed of two trunks of at least two ISLs?
򐂰 Does each storage device connect to at least two different edge switches or separate port
blades?
򐂰 Are storage ports provisioned such that every host has at least two ports through which it
can access LUNs?

Performance
IBM b-type Gen 7 technology substantially accelerates a data center environment, offering
50% lower latency than the previous generation, while increasing bandwidth with 64 Gb/s
links. By leveraging NVMe-ready Fibre Channel technology, IBM b-type Gen 7 not only
maximizes the performance of NVMe storage and high-transaction workloads, but also
simplifies the process of integrating higher-performing NVMe storage into the environment
without a rip-and-replace process.

Reviewing performance requirements:


򐂰 Host-to-storage port fan-in/out ratios
򐂰 Oversubscription ratios: – Host to ISL – Edge switch to core switch – Storage to ISL
򐂰 Size of trunks • Routing policy and currently assigned routes; evaluate actual utilization for
potential imbalances
򐂰 Use of FEC for all ISLs and connections to Gen 5 (if supported) and Gen 6 devices

Watching for impacts such as:


򐂰 Poor storage performance
򐂰 Overloaded hosts or applications
򐂰 Distance issues over constrained long distance links resulting from changes in usage,
such as adding mirroring, or too many workloads)
򐂰 Deteriorating optics resulting in declining signal strength and increased error rate

74 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch05.fm

Extensibility
The IBM b-type Gen 7 Fibre Channel switch is taking advantage of newer and older
technologies within the same infrastructure. It has the ability to run multiple vendors within the
same architecture to meet evolving storage requirements. Three generations of
backwards-compatibility, future proof investments with a Gen 7 ready storage networking
platform and NVMe over Fabrics as a new protocol for solid state storage. You will want to
ensure you do not need to rip and replace.

Manageability
The IBM b-type Gen 7 Fibre Channel switches integrated to IBM SANnav Management Suite
as the SAN Management Interface. IBM SANnav 2.1.0 is a major new release of IBM ’s b-type
Gen 7 two Fibre Channel SAN management software products, IBM SANnav Management
Portal and IBM SANnav Global View. SANnav Management Portal 2.1.0 supports the
introduction of IBM ’s new Gen 7 platforms with Fabric OS 9.0.0 and adds new features and
capabilities that make managing IBM SAN environments easier than ever before.

5.2.3 Architecting a SAN


The SAN planning process is similar to any type of project planning and includes the following
phases:
򐂰 Phase I: Gathering requirements
򐂰 Phase II: Developing technical specifications
򐂰 Phase III: Estimating project costs
򐂰 Phase IV: Analyzing Return on Investment (ROI) or Total Cost of Ownership (TCO) (if
necessary)
򐂰 Phase V: Creating a detailed SAN design and implementation plan

When selecting which criteria to meet, you should engage users, server and storage subject
matter experts (SMEs), and other relevant experts to understand the role of the fabric. Since
most SANs tend to operate for a long time before they are renewed, you should take future
growth into account as SANs are difficult to re-architect. Deploying new SANs or expanding
existing ones to meet additional workloads in the fabrics requires a critical assessment of
business and technology requirements. Proper focus on planning will ensure that the SAN,
once deployed, meets all current and future business objectives, including availability,
deployment simplicity, performance, future business growth, and cost.

Design worksheets (see Appendix A, “Fabric details” on page 305) which assist in
architecting a SAN environment, are provided are provided as a reference for documenting
assets and metrics for SAN projects. A critical aspect for successful implementation that is
often overlooked is the ongoing management of the fabric. Identifying systems-level SMEs for
all components that make up the SAN, as well as adequate and up-to-date training on those
components, is critical for efficient design and operational management of the fabric. When
designing a new SAN or expanding an existing SAN, you should take into account the
following parameters.

Application virtualization
򐂰 Which applications will run under a virtual machine (VM) environment?
򐂰 How many VMs will run on a physical server?
򐂰 Under what conditions will the VMs be migrated (business and nonbusiness hours; is
additional CPU or memory needed to maintain response times)?

Chapter 5. Fibre Channel SAN design, implementation, and migration 75


8497ch05.fm Draft Document for Review April 5, 2021 12:31 pm

򐂰 Is there a need for solid-state storage media to improve read response times?

Homogeneous/heterogeneous server and storage platforms


򐂰 Is auto-tiering in place?
򐂰 Which Brocade Fabric OS (FOS) versions are supported in a multivendor storage
environment?
򐂰 What is the planned refresh cycle of servers and storage platforms (2 years/3 years)?

Sizing
򐂰 How many user ports are needed now?
򐂰 How many devices will connect through an access gateway?
򐂰 How many inter-switch links (ISLs)/Brocade UltraScale inter-chassis links (ICLs) are
required to minimize congestion in the fabric?
򐂰 What distances for ISL/ICL connections need to be supported?
򐂰 Does the fabric scale out at the edge or the core?

Backup and disaster tolerance


򐂰 Is there a centralized backup? (This will determine the number of ISLs needed to minimize
congestion at peak loads.)
򐂰 What is the impact of backup on latency-sensitive applications?
򐂰 Is the disaster solution based on a metro Fibre Channel (FC) or Fibre Channel over IP
(FCIP) solution?

Diagnostics and manageability


򐂰 What is the primary management interface to the SAN (command-line interface [CLI],
Brocade SANnav, or third-party tool)?
򐂰 How often will Brocade FOS and SANnav be updated?
򐂰 How is cable and optics integrity validated?
򐂰 Is support needed for adding Gen 7 switches into a Gen 6 fabric?
򐂰 Is support needed for storage technologies like NVMe over Fabrics?
򐂰 What device interoperability support is required?
򐂰 Is interoperability required for other technologies such as UCS?

5.2.4 High-performance workloads


Over the last few years, enterprises have come to leverage low-latency, high-throughput flash
arrays for demanding, performance-sensitive workloads. Brocade’s Gen 7 Fibre Channel is
perfectly suited to these types of workloads due to the sub-microsecond latency through the
switch and the increased bandwidth offered by 32/64-Gb throughput speeds while providing
accurate I/O latency instrumentation. Performance testing has shown that 32-Gb and even
8-Gb all-flash arrays can realize dramatic benefits by connecting to a Gen 7 SAN and host
adapter, offering gains up to 2x over Gen 5 and Gen 6 SANs. The Gen 6 and Gen 7 standard
includes the use of forward error correction (FEC) to ensure transmission reliability and a
highly deterministic data flow. FEC corrects up to 140 corrupt bits per 5280-bit frame at the
receiving end of the link, avoiding the need to retransmit frames when bit errors are detected.
For these demanding workloads, a no-hop fabric connection through a single ASIC switch like
the Brocade G720 or locally switched on a single director port blade will minimize SAN fabric

76 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch05.fm

latency to sub-microsecond speeds. Local switching is the ability to switch data traffic through
a single ASIC by using both ingress and egress switching ports in a common port group.
When using switches that contain multiple switching ASICs like the X6 /X7 director,
configuring host and target connections on the ports that share a common ASIC will minimize
latency by avoiding the need to move data across multiple ASICs/port groups or across a
director backplane to a different blade. To find details on port groups and local switching
configuration, refer to the Brocade Fabric OS Administration Guide and the hardware
installation guide for the appropriate product. Because Gen 7 FC is backwards compatible
with Gen 6 networking, a Gen 7 edge switch or director can be added to an existing Gen 6
fabric. This allows for local all-flash connectivity to a Gen 7 switch to gain the performance
advantages of Gen 7 while preserving the investment in Gen 6 networks.

For more information, refer to Brocade Fabric OS Administration Guide:


https://www.broadcom.com/products/fibre-channel-networking/directors/x7-directors#
documentation

5.2.5 Redundancy and resiliency


An important aspect of SAN topology is the resiliency and redundancy of the fabric. The main
objective is to remove any single point of failure. Resiliency is the ability of the network to
continue to function and/or recover from a failure, whereas redundancy describes duplication
of components, even an entire fabric, to eliminate a single point of failure in the network.
Brocade fabrics have resiliency built into Brocade Fabric OS (FOS), the software that runs on
all Brocade B-Series switches, that can quickly “repair” the network to overcome most
failures. For example, when a link between switches fails, FSPF quickly recalculates all traffic
flows. Of course, this assumes that there is a second route, which is when redundancy in the
fabric becomes important. The key to high availability and enterprise-class installation is
redundancy. By eliminating a single point of failure, business continuance can be provided
through most foreseeable and even unforeseeable events. At the highest level of fabric
design, the complete network should be redundant, with two completely separate fabrics that
do not share any network equipment (routers or switches). Servers and storage devices
should be connected to both networks utilizing some form of multipath I/O (MPIO) solution,
such that data can flow across both networks seamlessly in either an active/active or
active/passive mode. MPIO ensures that if one path fails, an alternative path is readily
available. Ideally, the networks would be identical, but at a minimum they should be based on
the same switch architecture to ensure consistency of performance. In some cases, these
networks are in the same location. However, in order to provide for disaster recovery (DR),
two separate locations are often used, either for each complete network or for sections of
each network. Regardless of the physical geography, there are two separate networks for
complete redundancy.

In summary, best practices for SAN design are to ensure application availability and resiliency
via the following:
򐂰 Redundancy built into fabrics to avoid a single point of failure
򐂰 Servers connected to storage via redundant fabrics
򐂰 MPIO-based failover from server to storage
򐂰 νRedundant fabrics based on similar architectures
򐂰 Redundant ISLs/ICLs for interswitch connectivity
򐂰 Separate storage and server tiers for independent expansion
򐂰 Core switches of equal or higher performance compared to the edge switches
򐂰 The highest performance switch in the fabric defined to be the principal switch

Chapter 5. Fibre Channel SAN design, implementation, and migration 77


8497ch05.fm Draft Document for Review April 5, 2021 12:31 pm

5.2.6 Switch interconnections


As mentioned previously, there should be at least two of every element in the SAN to provide
redundancy and improve resiliency. The number of available ports and device locality
(server/storage tiered design) determine the number of ISLs needed to meet performance
requirements. This means that there should be a minimum of two trunks, with at least two
ISLs per trunk. Each source switch should be connected to at least two other switches, and so
on. In Figure 5-2, each of the connection lines represents at least two physical cable
connections.

Figure 5-2 Connecting devices through redundant fabrics

In addition to redundant fabrics, redundant links should be placed on different blades,


different ASICs, or at least different port groups whenever possible, as shown in Figure 5-3 on
page 79. (See the appropriate hardware manual to determine trunk groups for the various
port blades. For more details, refer to the Brocade Fabric OS Administration Guide.)
Whatever method is used, it is important to be consistent across the fabric. For example, do
not place ISLs on lower port numbers in one chassis (as shown in the left diagram in
Figure 5-3 on page 79) and stagger them in another chassis (as shown in the right diagram in
Figure 5-3 on page 79). Doing so would be mismatched ISL placement.

78 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch05.fm

Figure 5-3 Examples of distributed ISL placement for redundancy

Note: In Figure 5-3, ISL trunks are placed on separate ASICs or port groups. It is
important to match ISL placement between devices and across fabrics to ensure simplicity
in design and assist in problem determination.

5.2.7 UltraScale ICL connectivity for IBM b-type Gen 7 platforms


The IBM b-type Gen 7 DCX 8510, X6, and X7 platforms use second-generation UltraScale
ICL technology with optical QSFPs. The IBM b-type Gen 7 DCX 8510-8, X6-8, and X7-8
support up to 32 QSFP ports per chassis (Figure 5-4 on page 80), and the IBM b-type Gen 7
DCX 8510-4, X6-4, and X7-4 support up to 16 QSFP ports to help preserve director ports for
connections to end devices. Each QSFP port actually has four independent links, each of
which terminates on a different ASIC within the core blade.

Note: Consider the following points:


򐂰 X6 ICL ports can also connect to DCX 8510 ICL ports at 16-Gb speeds with the port
speed on the X6 ICL configured to 16 Gb and use of the 16-Gb QSFP on the attached
X6 port.
򐂰 X7 ICL ports can also connect to DCX 8510 ICL ports at 16-Gb speeds with the port
speed on the X7 ICL configured to 16 Gb and use of the 16-Gb QSFP on the attached
X7 port.

Chapter 5. Fibre Channel SAN design, implementation, and migration 79


8497ch05.fm Draft Document for Review April 5, 2021 12:31 pm

Figure 5-4 12-chassis UltraScale ICL-based core-edge design

5.2.8 Best practices for IBM b-type Gen 7 UltraScale ICL connections
Each core blade in a chassis must be connected to each of the two core blades in the
destination chassis to achieve full redundancy, as shown in Figure 5-5.

Note: For redundancy, use at least one pair of links between two core blades.

Domain 1 Domain 2
IBM b-type IBM b-type
X7 X7

UltraScale ICL Trunk Boundary


Trunked UltraScale ICL
Figure 5-5 Minimum ICL connections needed between IBM b-scale X7 chassis

5.2.9 Mesh topology


A mesh design provides a single hop between source and destination. Beginning with FOS
7.3.0x, IBM supports a 9-chassis mesh design with up to 100-meter distances using select
QSFPs and OM4 fiber. In the configuration shown in Figure 5-6 on page 81, up to 1152
16Gb/s ports are supported using UltraScale ICLs with a 12:1 oversubscription. As more
UltraScale ICLs are added, oversubscription can be reduced to 3:1.

80 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch05.fm

Figure 5-6 9-chassis UltraScale ICL-based full-mesh topology

Note: For more information, see the Scale-Out Architecture with IBM (Brocade) UltraScale
Inter-Chassis Links Design Guide.

https://www.broadcom.com/products/fibre-channel-networking/directors/x7-directo
rs#documentation

UltraScale ICL connections are considered a “hop of no concern” in a FICON fabric. When
using core-edge SAN design methodologies, edge switches should connect to at least two
core switches with trunks of at least two ISLs each. Each of those trunks should be attached
to a different blade/port group. In order to be completely redundant, there would be a
completely mirrored second fabric and devices must be connected to both fabrics using
MPIO.

Recommendations for switch ISL/UltraScale ICL connectivity follow:


򐂰 There should be at least two core switches.
򐂰 Every edge switch should have at least two trunks to each core switch.
򐂰 Select small trunk groups (keep trunks to two ISLs) unless you anticipate very high traffic
volumes. This ensures that you can lose a trunk member without losing ISL connectivity.
򐂰 Place redundant links on separate blades.
򐂰 Trunks should be in a port group (ports within an ASIC boundary).
򐂰 Use the same cable length for all UltraScale ICL connections.
򐂰 Use either IS

Chapter 5. Fibre Channel SAN design, implementation, and migration 81


8497ch05.fm Draft Document for Review April 5, 2021 12:31 pm

򐂰 Use the same type of optics on both sides of the trunks: Short Wavelength (SWL), Long
Wavelength (LWL), or Extended Long Wavelength (ELWL).

5.3 Implementation planning for a Fibre Channel SAN


This guide is for technical IT architects who are directly or indirectly responsible for SAN
design based on IBM b-type Gen 6 and Gen 7 Fibre Channel SAN platforms. It describes
many of the challenges that face SAN designers in both greenfield and legacy storage
environments.

While not intended as a definitive design document, this guide introduces concepts and
guidelines to help you avoid potential issues that can result from poor design practices.

This section describes best practices guidelines.

5.3.1 Device placement and traffic locality


Device placement is a balance between traffic isolation, scalability, manageability, and
serviceability. With the growth of virtualization and multinode clustering on the UNIX platform,
frame congestion can become a serious concern in the fabric if there are interoperability
issues with the end devices.

Designing device connectivity depends a great deal on the expected data flow between
devices. For simplicity, communicating hosts and targets can be attached to the same switch
(Figure 5-7)

Figure 5-7 Hosts and targets attached to the same switch to maximize locality of data flow

However, this approach does not scale well. Given the high-speed, low-latency nature of
Fibre Channel, attaching these host-target pairs on different switches does not mean that
performance is adversely impacted for common workloads. Although traffic congestion is
possible, it can be mitigated with proper provisioning of ISLs/UltraScale ICLs. With current
generation switches, locality is not required for performance or to reduce latencies. For
mission-critical applications that depend on extremely fast response times, architects may
want to localize the traffic when using flash storage or in very exceptional cases, particularly if
the number of ISLs available is restricted or there is a concern for resiliency in a multihop
environment (Figure 5-8 on page 83).

82 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch05.fm

Figure 5-8 Hosts and targets attached to different switches for ease of management and expansion

One common scheme for scaling a core-edge topology is dividing the edge switches into a
storage tier and a host/initiator tier. This approach lends itself to ease of management as well
as ease of expansion. In addition, host and storage devices generally have different
performance requirements, cost structures, and other factors that can be readily
accommodated by placing initiators and targets in different tiers.

Planning for storage


Storage arrays have evolved dramatically over the last few years. Performance has increased
exponentially, capacities have exploded, and more LUNs are supported than ever before. The
performance and capacity of low-end arrays has also improved.

New features include the following:


򐂰 Tiers comprising all-flash or flash-enhanced storage media have become commonplace
for mid-range and enterprise-class environments.
򐂰 Some arrays time out and reset their ports if they do not receive acknowledgements from
the connected host after specific intervals.
򐂰 Use of in-band Fibre Channel for control purposes has increased, putting extra stress on
FC port buffer usage. The use of flash-enhanced storage not only reduces the response
times and increases the I/O level by up to 10x, is also increases the efficiency of the host
platforms, allowing them to run at a higher level of utilization and execute more I/O
commands rather than waiting on responses from slow disk media.

Adding a layer of flash can therefore open up bottlenecks and relieve congestion resulting
from long wait times. Still, storage array performance can degrade over time, which can be
attributed to factors that include:
򐂰 Proliferation of VMs and the "IO blender" effect, whereby IO processes become
increasingly non-sequential and increase reads and writes to disk.
򐂰 Insufficient controller cache.
򐂰 Misaligned LUN configurations where OS block sizes do not match well to those on block
storage.
򐂰 Misconfigured control values such as edge hold time and queue depths.
򐂰 Provisioning strategies can favor capacity over usage.

Chapter 5. Fibre Channel SAN design, implementation, and migration 83


8497ch05.fm Draft Document for Review April 5, 2021 12:31 pm

Design Guidelines
򐂰 Ensure that network throughput capacity matches or exceeds the capabilities of storage
devices.
򐂰 Be careful if you deploy mixed arrays with different performance characteristics.
Experience has shown that it is very easy for a Tier 3 storage array, depending on how it is
used, to impact the performance of arrays in the same fabric. Troubleshooting in these
situations is very difficult.
򐂰 Network switch and optic speeds should be limited to two generations at most, with a best
practice of uniform speed to avoid target ports from slowing to accommodate lower-speed
ports and creating a back-pressure situation.
򐂰 Enable Forward Error Correction for ISL connections as well as connectivity to Gen 5
devices that support it. FEC is on by default for Gen 6 networks and should always be
used to ensure consistency of network transit times.
򐂰 Control the number of LUNs behind each storage port based on the type of usage they will
receive.
򐂰 Check on any special short-frame traffic to avoid frame congestion at array ports. It may
be necessary to increase the number of buffers at the array port to accommodate the extra
control traffic.
򐂰 Use advance Brocade FOS threshold timers to monitor hosts and storage arrays to ensure
that array ports do not reset due to a high-latency host, and thus do not adversely impact
other connected hosts.

NVMe
The next consideration in terms of why the storage network must be rethought and
re-architected is Non-Volatile Memory Express (NVMe).

At a device level, there are some characteristics to be aware of:


򐂰 Density – Current NVMe devices have 8 to 10 times the density of DRAM.
򐂰 Latency – Current NVMe devices have a sub-20-microsecond latency.
򐂰 Bandwidth – Current NVMe devices consume 4 PCIe Gen 3 lanes (32Gb/s).
򐂰 Streamlined software – Current NVMe software has 13 required and 25 optional
commands.

For a complete review of NVMe over Fibre Channel fabrics, refer to Chapter 8 of this
Redbook.

Storage virtualization
Storage virtualization enables LUNs accessed by servers to be abstracted from the physical
storage (typically storage arrays) on which they actually reside. (These are not the same as
traditional storage array LUN allocations, which can also be viewed as a form of
virtualization.) Virtualized LUNs that are disassociated from their actual storage allow for
more flexible storage provisioning processes. Performance may also improve, as the virtual
LUNs can be striped across multiple storage arrays.

There are two general types of storage virtualization: one uses an external controller (called
in-line virtualization), and in the other, the virtualization occurs inside a storage array. In-line
solutions are slightly more flexible, because they can use physical storage from a variety of
sources and vendors.

Figure 5-9 on page 85 shows a typical implementation of an in-line virtualized storage


solution. The host or VM accesses storage via a storage controller (shown on top) through

84 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch05.fm

the storage network. The red arrows indicate data access to and from the storage controller.
The storage controller typically controls all access to the physical storage, shown on the right
(and indicated by the blue arrows). This creates a very flexible storage solution, because
logical LUNs can be striped across several physical arrays to improve performance, and
logical LUNs can be manipulated completely transparently to the host or VM.

Figure 5-9 Typical implementation of an in-line virtualization storage solution

The major benefit of this type of storage virtualization is that storage can now be provisioned
in units of capacity (500 gigabytes or a terabyte) rather than physical LUNs. This is a first step
toward viewing storage as a service instead of as physical units. VM provisioning now
becomes less complex and easier to automate. Look into products such as IBM SAN Volume
Controller.

Design Guidelines
򐂰 Each storage controller in an in-line solution serves as both an initiator and a target.
򐂰 ISL utilization increases with in-line virtualized storage. Make sure that you have enough
ISL bandwidth to handle the increased load.
򐂰 There is also the possibility that the in-line storage heads will communicate through Fibre
Channel or generate many more SCSI control frames to manage their attached storage,
which can contribute to frame congestion. You may need to increase the number of buffers
at the ports that connect to the storage controller to accommodate this behavior.
򐂰 It is much more difficult to determine initiators and targets with in-line virtualized storage.
Since they are on the same switch, be careful about deploying tools such as Port Fencing.

Monitoring
򐂰 Utilize MAPS to track thresholds of critical parameters
򐂰 FPI monitoring is very useful in determining latencies associated with virtualized storage.
򐂰 Use Flow Vision to look at high-traffic-usage flows and identify the source of bottlenecks
and congestion.
򐂰 Consider running Flow Vision constantly for the most mission-critical applications to keep
a historical record of application performance profile and intermittent irregularities.

Chapter 5. Fibre Channel SAN design, implementation, and migration 85


8497ch05.fm Draft Document for Review April 5, 2021 12:31 pm

򐂰 For "frequent caller" applications, run Flow Vision on a regular basis where time permits to
verify good health.

5.3.2 Scalability and performance


IBM b-type Gen 7 products are designed with scalability in mind, knowing that most
installations will continue to expand and that growth is supported with very few restrictions.
However, following the same basic principles outlined in previous sections as the network
grows will ensure that the levels of performance and availability will continue.

Evaluate the impact on topology, data flow, workload, performance, and perhaps most
importantly, redundancy and resiliency of the entire fabric any time one of the following
actions is performed:
򐂰 Adding or removing:
– Changes in provisioning
– Storage
– Switches
– AISLs and ICLs
򐂰 Change in virtualization (workload and storage) strategies and traffic flow pattern If these
design best practices are followed when the network is deployed, then small incremental
changes should not adversely impact the availability and performance of the network.
However, if changes are ongoing and the fabric is not properly evaluated and updated,
then performance and availability can be jeopardized.

Some key points to cover when looking at the current status of a production FC network
include:
򐂰 Reviewing redundancy and resilience:
– Are there at least two physically independent paths between each source and
destination pair?
– Are there two redundant fabrics?
– Does each host connect to two different edge switches?
– Are edge switches connected to at least two different core switches?
– Are interswitch connections composed of two trunks of at least two ISLs?
– Does each storage device connect to at least two different edge switches or separate
port blades?
– Are storage ports provisioned such that every host has at least two ports through which
it can access LUNs?
– Are redundant power supplies attached to different power sources?
– Are zoning and security policies configured to allow for patch/device failover?
򐂰 Reviewing performance requirements:
– Host-to-storage port fan
• Host to ISL
• Edge switch to core switch
– Routing policy and currently assigned routes (evaluate actual utilization for potential
imbalances)
– Use of FEC for all ISLs and connections to Gen 5 (if supported) and Gen 6 devices.
򐂰 Watching for latencies because of the following:

86 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch05.fm

– Poor storage performance


– Overloaded hosts or applications
– Distance issues over constrained long-distance links resulting from changes in usage,
such as adding mirroring or too many workloads
– Deteriorating optics resulting in declining signal strength and increased error rate.
– In Gen 6 and Gen 7 networks, storage response latency can be baselined and
monitored continuously using IO Insight in conjunction with MAPS. Deal with latencies
immediately; they can have a profound impact on the fabric.

In summary, although IBM b-type Gen 7 SANs are designed to allow for any-to-any
connectivity and they support provision-anywhere implementations, these practices can have
an adverse impact on the performance and availability of the SAN if left unchecked. As
detailed above, the network needs to be monitored for changes and routinely evaluated for
how well it meets desired redundancy and resiliency requirements.

5.3.3 SAN design for critical workloads


With all-flash arrays (AFAs) being the standard in enterprise data centers today and the
transition to FC-NVMe AFAs beginning, critical business applications are increasingly
dependent on consistent low-latency, high-throughput storage performance for demanding,
performance-sensitive workloads. When designing the SAN, it is important to consider the
placement of critical workloads (in relation to the placement of storage), the fan-in ratio to
storage ports, and the fan-in ratio to ISLs/trunks.

Whereas the IBM b-type Gen 7 SAN technology provides a suite of built-in measures to avoid
interference of workloads that expose any kind of congestion behavior, such as Traffic
Optimizer, FPIN, MAPS, and SDDQ, protecting critical workloads is crucial. Ideally the most
demanding and critical workloads have dedicated storage array target ports (or even
dedicated arrays) and the shortest possible path through the SAN. The purpose for this is to
avoid any interference of other less critical workloads that may expose behavior that results in
congestion or back pressure, which could interfere and adversely impact the performance of
critical workloads.

Placement of servers with business-critical workloads


With core-edge SAN designs, it is often advantageous to connect servers with critical
workloads directly on the core together with the storage array ports. This works well when the
number of business-critical workloads is easily defined to a specific subset of servers in the
SAN and there is an adequate number of ports available on the core to connect both storage
and business-critical servers.

In the case where the number of business-critical servers exceeds the number of available
ports on the core for server connections, it is necessary to connect the business-critical
servers on edge switches. The most common model is then to use dedicated edge switches
for business-critical servers and decrease the fan-in ratio of servers to ISLs.

An alternative model is to attempt to evenly distribute business-critical servers across edge


switches with the assumption that the workload pressure evens out with the less critical and
demanding workloads on other servers. Although this may seem to be a logical approach,
experience has shown that throughout the lifetime of applications and the SAN, it is
operationally more complex to guarantee optimal performance for business-critical workloads
with this model.

Chapter 5. Fibre Channel SAN design, implementation, and migration 87


8497ch05.fm Draft Document for Review April 5, 2021 12:31 pm

Business-critical VMs
In today’s data centers, it is not uncommon for some or all business-critical workloads to be
running on VMs. Combining this with the digital business environment, the value (business
criticality) and performance requirements for a given application often change throughout the
life cycle of the application. This means that it can be difficult (or impossible) to predict the
requirements for an application and plan placement accordingly from the beginning—luckily
hypervisors provide the ability to move VMs between hosts without downtime and to migrate
storage when necessary. To apply the same principle of server placement of (bare metal
installed) business-critical applications to VMs, deploy dedicated hypervisor clusters
connected on the core or on high-performance edge switches and use these hypervisor
clusters only for business-critical and performance-sensitive VM workloads.

Visibility into each VM’s workload on the same data store (backed by LUN or NSID) can be
achieved with VM Insight. VM Insight enables storage administrators to monitor VM-level
application performance and set baseline workload behavior. This information can be used to
quickly determine whether a storage fabric is the source of performance anomalies for
applications running on VMs. Based on VM Insight, storage administrators can provision and
plan placement in the SAN on application requirements and can fine-tune the infrastructure to
meet service-level objectives.

5.3.4 Capacity planning


One way to avoid performance issues and downtime is with proper capacity planning, which
helps you get a bird's-eye-view of your IT infrastructure's overall capacity.

Learn how to gather and use Brocade's tools to improve the performance and reliability of the
SAN infrastructure.

Gathering requirements
The SAN project team should interview all stakeholders (IT application owners, finance,
corporate facilities, IT lab administrators, storage and network administrators, and end users)
who have a vested interest in the project-and this applies equally to planning for both new and
updated SANs.

Application owners
As critical stakeholders, application owners care because everyone is measured on
application uptime. Application outages are something that users notice, and they can have
severe financial impact for a business. With a redundant or a resilient infrastructure, hardware
outages are transparent to the user, and only SAN administrators need to pay attention.

To better understand their requirements, questions to ask the application owners include:
򐂰 What is the business goal for this application? (Is it a database that multiple applications
rely on for business transactions?)
򐂰 What are the availability requirements?
򐂰 Is the application latency sensitive?
򐂰 Are there peak periods of utilization or other traffic patterns?
򐂰 What are the IOPS requirements in terms of read/writes?
򐂰 What is the worst-case response time before an outage?
򐂰 Is the application running on a cluster?
򐂰 Is there application downtime that can be used for applying patches, software upgrades,
and maintenance?

88 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch05.fm

򐂰 Can the application run on a VM? If so, how many other VMs can co-exist on the same
physical hardware?

The business criticality of the application will determine the SAN design and the DR strategy,
including backup and recovery. If the application is mission critical, the infrastructure must be
fully redundant, with no single point of failure for both mainframe or distributed open systems
architectures.

Server and storage administrators


Once the application requirements have been defined, identify physical server and storage on
which the application and data will reside to determine the overall high-level architecture of
the SAN, especially if this includes existing equipment as well as new equipment.
򐂰 Gather information about the server(s) on which the applications are running (blade or
rack, CPU, memory, HBA/embedded FC switch, OS level, OS patch level, HBA driver
version)?
򐂰 How many HBAs are in the rack servers?
򐂰 Does each server have single or multiple port HBAs?
򐂰 What is the primary?
򐂰 What is the desired storage response time (latency)? Will longer latency times resulting
from a disk hit severely impact performance?
򐂰 For hybrid storage, how much storage is allocated to SSD?
򐂰 For HDD-based storage, what is the current cache utilization? Is there enough cache to
meet required response times? What is the average drive utilization (the greater the
utilization, the longer the response times) for HDDs and SSDs? Contact your drive vendor
to identify response times based on utilization for sizing workloads.

How much storage capacity is allocated to flash (more cache allows for more consistent
response time) if using a hybrid array?
򐂰 Are storage tiers used in the environment? What is the policy used for migrating data? Are
different tiers used for online storage? What is the impact?
򐂰 What is the raid level used? This will determine available disk space and performance for
the application.
򐂰 How many FC ports are there in the array?
򐂰 Are the arrays front-ended by a storage virtualization controller? If so, what is the
additional latency?
򐂰 What are the recommended fan-in and fan-out ratios for the arrays used for this
application? What are the limits?
򐂰 Is there a Disaster Recovery (DR) site? If so, how is it connected (dark fiber, FCIP)?
򐂰 What is the available/required bandwidth between the intra-site for DR? Can the existing
storage infrastructure support DR with the additional load?
򐂰 What tools are used for mirroring and replication (host-based or array-based)? If
host-based, was the failover tested? If so, was there any impact in application uptime? If
storage-based, was the failover tested? Did the LUNs appear on the active ports?
򐂰 Was there an impact to application uptime?

SAN administrator: general


A SAN Administrator is responsible for the day-to-day operation of the network. The SAN
design must be easy to monitor, manage, and maintain.

Chapter 5. Fibre Channel SAN design, implementation, and migration 89


8497ch05.fm Draft Document for Review April 5, 2021 12:31 pm

If the current SAN is being expanded, adequate performance metrics should be collected to
ensure that the existing design can be expanded to address new workloads.
򐂰 Are there performance (bandwidth) or latency issues in the existing SAN?
򐂰 Are procedures in place to address redistribution of capacity when switch port utilization
exceeds 75 percent?
򐂰 Is the current design two-tier (core-edge) or three-tier (edge-core-edge)?
򐂰 Is the SAN centrally managed by a tool such as IBM Storage Insights or IBM Spectrum®
Control?
򐂰 If there is an existing SAN, how is it managed (CLI, IBM Network Advisor)? Is there a
separate network for SAN management?
򐂰 Are access control policies in place for change management (zoning)? Is there a zoning
policy? Are there devices in the zone database that no longer exist? What type of zoning is
used (port or WWN)?
򐂰 Is the current SAN a redundant configuration?
򐂰 Is there an identified server to capture logs from the fabric?
򐂰 Is the traffic equally distributed across the ISLs or the trunks?
򐂰 Is historical performance data available for initiators, targets, and ISLs?
򐂰 How many unused ports are available per switch?

5.4 Migration strategies of Fibre Channel SAN infrastructure


This section provides an overview of SAN migration planning and strategies. It will help you
understand how a Gen 6 platform can be upgraded to a Gen 7 platform. Migrating to the Gen
7 platform helps protect the investments you have already made but gives you the added
benefit of newer technologies.

5.4.1 IBM b-type Gen 7 provide infrastructure for today and tomorrow
The IBM b-type Gen 7 platforms breakthrough 64 Gbps performance accelerates application
response time by up to 71 percent, eliminating IO bottlenecks and unleashing the full
performance of an all-flash data center.

These systems will seamlessly integrate next-generation NVMe over Fabrics with Gen 7 Fibre
Channel networks without a disruptive rip and replace. In addition, these platforms will also
seamlessly connect older generation devices and storage networking equipment with three
generations of backward-compatibility.

Investment protection with IBM b-type Gen 7 Fibre Channel solutions


For investment protection, the IBM b-type Gen 7 family offers three generations of
backward-compatibility support for connectivity to 4, 8, and 16 Gbps Fibre Channel (and
FICON) products, allowing seamless connectivity between the older generation of devices
and storage networking equipment. This enables an older storage infrastructure to continue
to serve an organization’s needs. An organization could also strategically plan to introduce
IBM b-type Gen 7 Directors into their existing fabrics when new devices are added to their
fabrics, whether due to new requirements or just ongoing growth.

Further protecting future investments, the IBM b-type Gen 7 Director family will support future
Fibre Channel generations as a Gen 7-ready storage networking platform. The IBM b-type

90 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch05.fm

Gen 7 backplane is designed with more links, to accommodate a doubling and quadrupling of
bandwidth in the future. This will enable platform reusability for Gen 7 Fibre Channel, enabling
organizations to maximize their IBM b-type Gen 7 investments and optimize the performance
of their networking infrastructures.

IBM b-type Gen 7 platforms also enable customers to future-proof their storage networks by
accelerating new technology deployments such as NVMe over Fabrics. NVMe is a new
protocol for solid-state storage devices built with nonvolatile memory technologies. NVMe
provides dramatically lower latency for storage IO operations and significantly higher IOPS
per device. In the future, organizations will be able to transition their highperformance,
latency-sensitive workloads to flash-based storage with NVMe over Fabrics, thus realizing
significant performance gains and reaping the full benefits of flash.

NVMe over Fabrics will scale up the number of devices it can address by adopting NVMe over
Fabrics technology. NVMe over Fabrics enables end users to achieve faster application
response times from network-attached storage and to harness the performance of hundreds
of solid-state drives for better scalability across virtual data centers. Fibre Channel is one of
the fabric technologies that will be supported by NVMe over Fabrics. IBM b-type Gen 5 and
Gen 6 Fibre Channel switches already support NVMe over Fabrics, with no changes required.
Customers can seamlessly integrate IBM b-type Gen 7 Fibre Channel networks with
next-generation NVMe over Fabrics without a disruptive rip and replace for the increased
performance, application response time, and scalability that their next-generation data
centers require.

Upgrading a Gen 6 Director to a Gen 7 Director


IBM offers a simple upgrade option to extend the life of the Brocade Gen 6 chassis and gain
the extra value of Gen 7 technology. Since the IBM b-type Gen 7 leverages the same chassis,
only a simple upgrade is required to transform the Gen 6 director into a Gen 7 director.
Although the upgrade does require taking the chassis offline to replace the core routing
blades, it will be non-disruptive to the applications in a dual fabric since traffic moves to the
other fabric while the upgrade is happening. Once the Gen 6 director is upgraded,
organizations can choose to keep their existing port blades and start to leverage the new Gen
7 blades within the same chassis.

A detailed procedure is available that provides the materials and instructions required to
migrate your IBM b-type Gen 6 director (SAN512B-6 and SAN256B-6) running either Fabric
OS (FOS) 8.x or 9.x to a IBM b-type Gen 7 director (SAN512B-7 and SAN256B-7).

You will see how to do the following:


򐂰 Take a snapshot of the existing configuration.
򐂰 Prepare the configuration for conversion of the Control Processor (CP) blades.
򐂰 Swap the existing Core Routing (CR) blades.
򐂰 (Optional) Restore the original customer configuration (or default configuration).

Chapter 5. Fibre Channel SAN design, implementation, and migration 91


8497ch05.fm Draft Document for Review April 5, 2021 12:31 pm

The result is a one-way upgrade to a Brocade Gen 7 director.

Note: Upgrading to a Gen 7 director is permanent on the CPs. This means that you cannot
downgrade to Gen 6; after the upgrade to Gen 7, you cannot use the same CP blades in a
Gen 6 chassis.

The Gen 7 director supports mix-and-match blades, allowing Gen 6 and Gen 7 blades to
be installed within the same chassis. The following optional port or extension blades are
available:
򐂰 48-port Gen 6 IBM b-type Gen 7 FC32-X7-48 port blade with 32Gb/s line rate ports
– Added value over FC32-48 (reduced latency, congestion avoidance, and Traffic
Optimizer)
– Non-upgradeable to 64Gb/s speeds
򐂰 IBM b-type Gen 7 FC32-64 high-density port blade – Increases scalability and
maximizes space utilization
򐂰 IBM b-type Gen 7 SX6 Extension blade – Disaster recovery solutions over long
distances

Technical migration guides


The IBM (Brocade) X6 Field Migration Guide is available to walk through the step-by-step
process on the Broadcom Customer Support Portal (CSP)

https://www.broadcom.com/products/fibre-channel-networking/directors/x7-directors#
documentation

Gen 6 to Gen 7 upgrades - time and items required


As a guideline, the migration procedure might require from 1 hour to over 2 hours depending
on the number of upgrade tasks.

You will need to factor in the time to complete the following:


򐂰 Installing additional power supplies.
򐂰 Installing supported blades.
򐂰 The upgrade process requires you to remove the current Gen 6 core blades and replace
them with Gen 7 core blades. Except for the core blades, all other blades currently
installed in the Gen 6 chassis are still supported after the upgrade.
򐂰 The chassis must be powered off before the core blades are replaced. The power-on
process takes approximately 20 to 30 minutes.
򐂰 Cabling new blades or relocating existing cables.
򐂰 The Migrations steps are documented in the IBM (Brocade) X6 Field Migration Guide.

Technical migration guides


The IBM (Brocade) X6 Field Migration Guide is available to walk through the step-by-step
process on the Broadcom Customer Support Portal (CSP)

https://www.broadcom.com/products/fibre-channel-networking/directors/x7-directors#
documentation

92 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch05.fm

5.4.2 Tools to assist in migration planning


The following section will provide information on tools that can assist with SAN design,
planning, and migration.

Brocade SAN health


Migrations should be done using a planned process. Understanding the current state of your
infrastructure is a good place to begin. To start, SAN Health should be run to provide users
the overall health of a fabric and to capture the details of a known working environment.

Brocade SAN Health is a free tool that allows SAN administrators to securely capture,
analyze, and report comprehensive information about fabrics with switches that are running
Brocade Fabric OS and Cisco MDS fabrics that are running SANOS/NXOS.

SAN Health can perform tasks such as the following:


򐂰 Taking inventory of devices, switches, firmware versions, and SAN fabrics.
򐂰 Capturing and displaying historical performance data.
򐂰 Comparing zoning and switch configurations against best practices.
򐂰 Assessing performance statistics and error conditions.
򐂰 Alerting the administrator of relevant technical support bulletins (TSBs).
򐂰 Producing detailed reports (in Microsoft Excel) and diagrams (in Microsoft Visio).

Figure 5-10 SAN Health user interface

Download Brocade SAN Health and find details and instructions on how to use it at:
https://www.broadcom.com/products/fibre-channel-networking/software/sanhealth

SAN Health will assist in capturing information to analyze the current health of your fabric. If
issues with the health of your fabric are found, resolve them before moving forward with any
migration. In addition to the health status, information on the physical and logical layout can
be used to transition to newer SAN platforms while maintaining consistency in your design.

Brocade SAN Health data capture


SAN Health uses a data capture application and a back-end report processing engine. After
capturing switch diagnostic data, it automatically generates a Visio topology diagram and a
detailed report of SAN fabrics, switches, and individual ports. Other useful information
includes alerts, historical performance graphs, and recommended best practices.

Chapter 5. Fibre Channel SAN design, implementation, and migration 93


8497ch05.fm Draft Document for Review April 5, 2021 12:31 pm

Because it provides a point-in-time snapshot of your SAN, the SAN Health Diagnostics
Capture is invaluable to your change-tracking process.

Download Brocade SAN Health Diagnostic Capture at:


https://www.broadcom.com/support/fibre-channel-networking/tools/san-health/diagnos
tics-capture

With Brocade’s free SAN Health tool, you can generate personalized storage network
performance and inventory reports to help you prevent issues and optimize operations.

Figure 5-11 Comprehensive reporting with health and best practice checks\

Figure 5-12 Performance graphs for visualizing activity

94 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch05.fm

Figure 5-13 Useful topology diagrams to view physical layout

Figure 5-14 Search, filter and sort the audit data

5.4.3 SAN migration factors


Today’s SAN Administrators are faced with the need for more storage capacity and speed.
They require ance and redundant SAN Networks that can both meet their current demands
and scale for growth in the near future! To accommodate these new requirements, SAN
Admins often need to migrate and upgrade from their existing storage network. Migrating from
older SAN’s to newer SAN technology requires a plan that includes design, configuration, and
implementation processes along with a post migration analysis.

This section will highlight those migration factors that need to be taken into account for the
migration plan. Using SAN Health will allow you to gather information to help make the
decisions for a smooth and seamless transition.

In addition to knowing about the current infrastructure, the current SAN design should be
reviewed to help incorporate future growth, backup and recovery or business continuity
projects and server/storage upgrades.

Factors to consider include:

Chapter 5. Fibre Channel SAN design, implementation, and migration 95


8497ch05.fm Draft Document for Review April 5, 2021 12:31 pm

򐂰 Assessing the existing Fabric Topology, Inventory the network.


򐂰 Application failover considerations.
򐂰 Storage failover considerations.
򐂰 Topology Changes.
򐂰 ID critical servers, storage arrays, and applications.
򐂰 Zone configurations export/import strategy.
򐂰 Ensure correct licensing sets are included.
򐂰 Software interoperability Planning.
򐂰 Prepare/Update SAN/Storage Diagram to meet new requirements.

5.4.4 SAN migration – process overview:


This section will provide a high level overview of what steps are needed to migrate an existing
SAN infrastructure to a new SAN infrastructure. This is not a “how to guide” but provides the
concepts that a user should plan for in their migration strategy. The scope of this migration will
review replacing an existing infrastructure with a one for one replacement of SAN Directors.

Preparing for the migration


There are several steps to a migration but one of the most important steps is to capture the
information about your existing infrastructure so that a baseline of success can be
established.

The following guidelines provide a path to a new SAN infrastructure:


򐂰 Preliminary work: Run the IBM b-type Gen 7 SAN Health tool to capture a snapshot of
your environment and see if you have any current “network health” problems that need
attention prior to the migration. (See Figure 5-15 for migration example.)

Figure 5-15 SAN Director migration example (setting up new SAN)

򐂰 Install and apply power to the new IBM b-type Gen 7 SAN Directors, leaving them
disconnected from all storage or network cables.
򐂰 Gather information from the IBM b-type Gen 7 SAN Directors required for building and
configuring the infrastructure.
򐂰 Leverage SAN Health to model the new SAN infrastructure topology:

96 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch05.fm

– Modeling is done by leveraging the Visio diagram of the SAN infrastructure created by
SAN Health
– Rearrange the switches as needed, insert newly proposed server and arrays to the
SAN Health report then optimize the configuration for any-to-any connectivity.
– If there are flaws in the overall design of your data fabric, the topology diagram, helps
bring them to the forefront, making it easier for the infrastructure architect team to
improve on designs.
򐂰 Configure the IBM b-type Gen 7 SAN Directors
– Create baseline configurations for all Directors/Switches
– Import/purge zoning sets
– Create zones for new devices
򐂰 Validate the configuration
– Run SAN Health and address any errors found before moving forward with migration
򐂰 Attach all cables to server and storage as required, as shown in Figure 5-16).

Figure 5-16 SAN Director migration example (integrating new SAN Directors/Switches)

򐂰 Validate application operations


򐂰 Backup new SAN configurations

Chapter 5. Fibre Channel SAN design, implementation, and migration 97


8497ch05.fm Draft Document for Review April 5, 2021 12:31 pm

򐂰 Retire old SAN infrastructure (Figure 5-17)

Figure 5-17 SAN director migration example (retiring old SAN)

Ongoing infrastructure management using SAN health


Savvy IT leaders have learned the value of running SAN Health on a regular basis, especially
before and after any hardware or software changes in the storage infrastructure. Beyond the
insights into the architecture and performance, they find that the information gained,
increases their confidence and understanding of the SAN infrastructure, and documents the
fabric, including the attached servers and storage devices.

SAN Health improves capacity planning and productivity, and reduces the time required to
troubleshoot problems when they arise. Consider adding SAN Health to your toolkit....

It’s FREE!

5.5 References
Software and Hardware Product Documentation FOS documentation is located in two
different locations within Broadcom for IBM b-type Gen 7 products and software. The publicly
facing content is located on Broadcom.com. The nonpublic documentation is located on the
Broadcom Customer Support Portal (CSP).

For more information about your specific Fabric operating system release, see the following
resources:
򐂰 https://www.broadcom.com/products/fibre-channel-networking
򐂰 https://www.broadcom.com/products/fibre-channel-networking/directors/x7-directors
#documentation

The webpages contain all user guides, reference manuals, white papers, eBooks, product
briefs, administrative guides, compatibility guides, and case studies.
򐂰 Brocade Fabric OS Administration Guide
򐂰 Brocade Fabric OS Command Reference Manual
򐂰 Brocade Fabric OS MAPS User Guide

98 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide


Draft Document for Review April 5, 2021 12:31 pm 8497ch05.fm

򐂰 Brocade Fabric OS Flow Vision User Guide


򐂰 Brocade Fabric OS Access Gateway User Guide
򐂰 Brocade Fabric OS Extension User Guide
򐂰 Brocade X6-4 Director Hardware Installation Guide
򐂰 Brocade X6-8 Director Hardware Installation Guide
򐂰 Brocade X7-4 Director Hardware Installation Guide
򐂰 Brocade X7-8 Director Hardware Installation Guide
򐂰 SAN Fabric Resiliency and Administration Best Practices User Guide
򐂰 Brocade SANnav Flow Management User Guide
򐂰 Brocade SANnav Management Portal Installation and Migration Guide
򐂰 Brocade SANnav Management Portal REST API and Northbound Streaming Reference
Manual
򐂰 Brocade SANnav Management Portal User Guide
򐂰 Brocade SANnav Global View User Guide

Chapter 5. Fibre Channel SAN design, implementation, and migration 99


8497ch05.fm Draft Document for Review April 5, 2021 12:31 pm

100 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm

Chapter 6. FICON design, implementation


and migration
Mainframes are typically used for mission critical applications. To ensure that all components
in a mainframe SAN meet the reliability standards and consistent behavior that most mission
critical applications demand, additional testing and qualification programs have been
implemented for all devices connected to an IBM z processor. Due to test limitations, not all
fibre channel configurations are supported in IBM z environments. For more information, see
https://www-01.ibm.com/servers/resourcelink/lib03020.nsf/0/A2F5F3D820C99664852576A
2006BE95A?OpenDocument.

FICON is a protocol transported over fibre channel. In addition to meeting FICON


requirements, all fibre channel requirements must be met as well. Since supported FICON
configurations are a sub-set of supported fibre channel configurations, the only need to
consider additional fibre channel requirements is in cascaded environments. For enterprises
with cascaded environments, refer to Chapter 5 and the section that discusses Inter-Switch
Links (ISL’s) and Inter-Chassis Links (ICL’s).

Most SAN upgrades skip a generation. Most upgrades to Gen 7 therefore will be from Gen 5.
The most significant change when upgrading from Gen 5 is that a 32 port card is no longer
supported on the director class (bladed) switches. The new blades are 48 ports. Furthermore,
the control processor (CP) cards were moved resulting in new slot numbering on the bladed
(director class) switches.

Gen 5 versus Gen 7 IBM b-type SAN Directors Generation Impact:


򐂰 CHPID path mapping
򐂰 Documentation templates
򐂰 Patch room layout

IBM Network Advisor is nearing end of support and is being replaced by SANnav. IBM
Network Advisor does not support FOS 9.0 or any Gen 7 product. SANnav does support Gen
5 products on the 7.4 and 8.2 code streams. So as to minimize change activity when
upgrading to Gen 7, upgrading to SANnav first is recommended.

© Copyright IBM Corp. 2021. 101


8497ch06.fm Draft Document for Review April 5, 2021 12:31 pm

While Gen 7 offers data center managers higher performance and valuable new analytics,
neither these nor any other Gen 7 features will impact FICON planning and implementation.

This chapter will review the following topics:


򐂰 6.1, “Important support notes for Gen 7 platforms” on page 103
򐂰 6.2, “FICON SAN design” on page 106
򐂰 6.3, “FICON SAN implementation” on page 114
򐂰 6.4, “FICON SAN migration” on page 130

102 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm

6.1 Important support notes for Gen 7 platforms


Remember to plan the timing of your upgrade projects to ensure adequate time before the
end of support dates.

6.1.1 Product Support Dates


Table 6-1 FICON Support Summary

Product EOS Comments

IBM Z15™ on Gen 31 Oct 2021 Although support for these hardware platforms continues
5 through 30 April 2025, for z15 connections the last day of
support is 31 Oct 2021.

Gen 5 30 April 2025 These are the SAN768B-2 (2499-816), SAN384B-2


(2499-416) and SAN48B-5 (2498-F48) products.

IBM Network 8 Feb 2023 IBM Network Advisor FOS support stops at FOS v8.2.x
Advisor SANnav is the replacement product. Most customers plan
the management system upgrade to coincide a few months
prior to the end of their IBM Network Advisor support
contract.

6.1.2 Software
The last code stream supported on Gen 5 is FOS 8.x. Gen 6 supports both the FOS 8.x and
9.x code streams.

All SAN switches in a fabric must be no more than one major code revision apart. All switches
operating with FOS 8.x and 9.x can participate in the same fabric but switches operating with
FOS 7.x cannot join the same fabric that includes switches operating with FOS 9.x.

Deprecating the prohibit/allow matrix (aka PDCM) is planned for FOS v9.1. Customers using
the PDCM should plan on considering other methods, typically zoning, to effect the blocking
of certain paths. The PDCM is being deprecated because all IBM software products that used
the PDCM are no longer supported by IBM.

All Gen 7 switches require FOS v9.0.0 or higher.

FICON logical switch


Configuring virtual fabrics (VF) is analogous to logical partitions (LPARs) on a IBM z CEC.
Mainframe customers often disable logical fabrics when configuring switches or leave VF
enabled but use the default logical switch and configure the switch to meet the FICON
requirements. With FOS 9.0 and above, having this feature enabled will be required and you
will need to create a logical switch that is configured specifically for FICON.

The FICON logical switch was introduced in FOS 8.1.0. The difference between a logical
switch defined for FICON and configured for FICON is that the logical switch configured for
FICON can be reconfigured such that it no longer meets the FICON requirements while a
logical switch defined as a FICON logical switch cannot be reconfigured so as to remove the
FICON requirements.

A FICON logical switch provides the following advantages:


򐂰 Improves security

Chapter 6. FICON design, implementation and migration 103


8497ch06.fm Draft Document for Review April 5, 2021 12:31 pm

򐂰 Ensures properly configured FICON SAN fabrics


򐂰 Simplifies configuration

In prior versions of FOS, determining if port address was bound was challenging for
non-expert users. A “bound” address means that the fibre channel address for the port cannot
change. This is important because the IBM z channel subsystem builds fibre channel
addresses based on the link addresses defined in the Hardware Configuration Definition
(HCD) tool. Another reason for the FICON logical switch is that it’s possible to configure a
switch to meet all of the FICON requirements and then back certain changes out after the
channels have come online. Since a mainframe channel only checks the proper security
attributes at login time (when the PCHID is enabled, not when the CHPIDs are configured
online) there is the potential for a channel not to come online the next time it goes off and
back online. For most customers this is the result of a CEC IML or channel path maintenance.

In addition to addressing potential problems, there are a number of addressing mode


restrictions that are dependent on the switch type causing considerable field confusion and in
some cases limits options for customers.

Although circumstances for changing the fibre channel address assigned to a port on the
switch is unlikely, should it happen, the IODF would no longer be valid resulting in device or
channel errors. This potential problem is fixed with the defined FICON logical switch.

Control Unit Port (CUP) in a logical switch


An independent channel connected to the default switch with CUP defined is not necessary.
Chassis related issues are reported on all logical switches through the CUP.

Other than common chassis components, each logical switch behaves as independent
hardware. There is no sharing of information between one logical switch and another. When
using tools such as System Automation or Disk Magic to manage and monitor switches
through CUP, a separate channel with CUP defined must be attached to each logical switch.

If a separate logical fabric is created for disk or tape mirroring is put into its own logical fabric
for mirroring, not only will a separate channel be required to monitor mirroring ports, but since
that CHPID is a FICON channel, the logical switch must be defined as FICON logical switch.
Defining a logical switch as a FICON logical does not affect other protocols, such FCP which
is used for mirroring. The FICON logical switch only adds secure features and supports CUP.

Since TS7700 tape mirroring when done with fibre channel and Metro Mirror use FCP
protocol, this means for CUP support a dedicated FICON channel just for CUP is required. An
alternative, so as to avoid the need for dedicating a channel just for CUP, is to keep mirroring
ports in the same logical switch with FICON CHPID to control unit paths and use zoning to
isolate the traffic.

6.1.3 Link address refresher


Link addressing has not changed; however, due to the change from 32 to 48 port cards link
addresses will land on different port cards. As a result, much of the design work will be
dedicated to CHPID path mapping which requires an in-depth understanding of how link
addresses relates to fibre channel addressing

104 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm

S in g le B y t e L in k A d d r e s s 06
T w o - B y t e L in k A d d r e s s B506
F C A d d re s s B50600

D o m a in ID in H e x A LPA

P o r t A d d r e s s In H e x
Figure 6-1 Link address and fibre channel address

Domain ID & Fabric Security


For single byte addressing, when the channel logs in the domain area returned from the
switch is used for the domain area in the FC address, therefore, the channel and control unit
must be in the same logical switch. With two-byte link addressing the most significant byte of
the link address is used for the domain area of the FC address.

Once two-byte link addressing is defined for a channel, all link addresses for that channel
must use two-byte link addressing. Once two-byte link addresses are defined, whether
cascading or not, all the fabric security settings must be set on the switch. For FOS 9.0 and
above, and therefore all Gen 7 switches, this means the channel must be connected to a
FICON logical switch.

If the high integrity settings are not configured on the switch then zOS will set the channel
offline and put it in the “Invalid Attach” state. Once a channel is in the invalid attach state, the
misconfiguration on the switch must be correct and then the channel re-enabled. It is the
PCHID, not the CHPID, that is in the invalid attach state. You cannot simply configure the
CHPID off and back online. The PCHID must be toggled from the System Element (SE).

The fabric security features (HIF) are not checked by the channel when single byte
addressing and therefore not recommended. Configuring 2 byte link addresses is the most
secure method and required in some cases.

Port address
The hexadecimal port address is the single byte link address or the least significant byte of a
two byte link address.

The ALPA was originally used in fibre channel for arbitrated loop devices. Arbitrated loop is no
longer supported and was never supported for FICON. Today, the ALPA is used for an
extension of the port addressing and Node Port Identification Virtualization (NPIV).

The bits for NPIV allow multiple WWNs to log into the same switch port. These bits determine
which logical entity frames belongs to. This is how virtual servers using zLinux or zVM, which
use FCP protocol, can share the same channel. FICON does not use these bits; however, the
ALPA must be the same for all FICON attached control units and channels.

Since the 8 slot director class products can exceed 256 ports in a single domain (logical
switch), the upper 2 bits of the ALPA are used in certain addressing modes.

A FICON channel builds the ALPA portion of the FC address by using the same ALPA where
the channel logged in. The ALPA portion of the FC address of control units therefore must
have the same ALPA as the channel. This is why certain addressing modes do not work for
FICON. In a logical switch defined for FICON, the ALPA is always 00.

Chapter 6. FICON design, implementation and migration 105


8497ch06.fm Draft Document for Review April 5, 2021 12:31 pm

6.2 FICON SAN design


Before starting a design project, review the Migration section and especially the Physical
Upgrade subsection in this chapter for a clear understanding of the switch differences and to
determine how those difference will affect the SAN design. This RedBook is focused on the
SAN itself but a redesign of the cable plant is likely and possibly patch room redesign as well.

Often, storage and processor upgrades are aligned with SAN upgrades. Make sure all SAN
requirements are well known prior to commencing any design planning and remember to
account for swing ports when upgrading storage and processors.

Table 6-2 FICON design check list


Area Things to Think About

Management Although FICON SANs can be managed via the CLI, doing so is
very unusual. Nearly all FICON SANs use management
software.
Network Advisor does not support FOS 9.x. Therefore, Gen 7
switches are not supported by Network Advisor.
The SANnav Management Portal is the replacement product for
Network Advisor. The enterprise version is required whenever
managing a director class product or environments with more
than 600 ports. Otherwise, the base version is adequate.
FICON environments nearly always require the Enterprise
version of SANnav.
SANnav 2.1 or higher is required when managing Gen 7
products.
SANnav 2.2 or higher is recommended for managing FICON
fabrics.
SANnav Global View is s new software package that may be
useful when managing either multiple independent production
sites or active-active backup sites. SANnav Global View provides
a single pane of glass management for up to 20 instances of the
SANnav Management Portal.
Typically, customers plan their upgrade from Network Advisor to
SANnav to coincide with the end of their Network Advisor
support contract. It is highly recommended to start the transition
90 days in advance. There is a free fully functional 90-day trial
version of SANnav that customers can use for this purpose.

Routing method All switches in a fabric must use the same routing method so an
outage must be planned if changing the routing method.
Do all control units in your environment support FiDR?
Exchange based routing is the most efficient and resilient routing
method so if FiDR is supported on all channels and control units,
exchange based routing is recommended.

Utilizing 48 port cards and high HCD and cable plant consideration.
availability Most FICON SANs use director class products with 32 port
blades. Furthermore, the CHPID mapping tool and other tools
used to aid in channel path planning are built around 32 port
cards. CHPID mapping based on 32 port cards likely will not work
with 48 port cards.

106 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm

Area Things to Think About

Cascaded Performance HCD and cable plant consideration.


Virtual channels have always been a part of Brocade’s fibre
channel switching architecture. In cascaded environments, the
virtual channel used for a path is determined by the port address,
often referred to as the port ID or PID. The placement of where
control units are connected can effect channel I/O rate
performance.
Multiple zones with different quality of service can be deployed.
This is an especially important consideration for cascaded
channel-to-channel (CTC) traffic.

Ultra-low latency HCD and cable plant consideration.


Brocade fibre channel switches are built on ASICs which support
local switching. Local switching is when the input and output are
on the same ASIC. For most applications, the added design
considerations to take advantage of local switching is not
necessary. Applications that demand low latency are very high
transaction rate processing such as high frequency trading.

Naming Convention Prior to having a FICON logical switch, the difference between a
chassis, a switch, and a fabric were subtle and of little or no
consequence to mainframe administrators. Although the
technical differences will mean little to a mainframe
administrator, there are times when you will need to differentiate
between a chassis and a logical switch, especially when creating
multiple logical switches in the same chassis. Similarly, when
cascading you will need to be able to differentiate the logical
switches from the fabric.
Naming conventions are preferences that vary for each
organization. The only recommendation here is that you make
each name unique and easily identifiable. This can be done by
simply appending “_fab” at the end of fabric names and
“_chassis” at the end of chassis names.
Chassis, fabric, and switch names can be as long as 64
characters. Packing too much information into a name can make
it difficult to easily identify the item it is associated with and fill up
too much screen space. The SANnav database has a description
field where information such as rack and aisle location can be
added so there is no need for overly complex names. Although
there is no specific recommendation for naming conventions, the
general advice is to keep names simple.

Domain ID & Switch ID HCD and cable plant consideration.


The recommended best practice is to use the hex value of the
domain ID of the switch to be the Switch ID in HCD. This
recommendation also implies that all FICON logical switches
have a unique domain ID and that the Switch ID will always be
two characters. Valid Domain IDs are 0x01 – 0xEF

Supported Hardware & Software An inventory check to ensure all hardware and software are
currently supported and that interoperability between new
hardware and software being introduced to the fabric is
supported in the existing environment. The recommended best
practice is to complete all required changes to the existing
environment before commencing other upgrades so as to
compartmentalize changes.

Implementation Items Review the Implementation section, especially user configurable


items.

Chapter 6. FICON design, implementation and migration 107


8497ch06.fm Draft Document for Review April 5, 2021 12:31 pm

6.2.1 Typical redundant pathing


The typical configuration uses 8-way pathing to DASD spread across 4 directors. Although
two paths are connected to the same director, the paths should be planned such that the
channel and control units associated with a path do not share the same port cards. A
common design approach is to plan the first path group to use the cards to the left of the core
blades, which are in the center of the director, and the second path group to use the cards to
the right of the core blades.

Figure 6-2 Typical 8-way pathing

Despite this separation, most FICON implementations using directors with 32 port cards put
both channel paths on the same switch. Both channel paths, therefore, used the same
domain ID so the high byte of the link addresses was the same for both paths. Only the lower
byte of the link address was different.

To minimize the number of FRUs in a channel path, some organizations planned the CHPID
and control units connections to all reside on the same port card. For high-speed transaction
processing demanding the lowest latency, some organizations planned the CHPID and
control unit connections to be connected to ports sharing the same ASIC.

6.2.2 Cascaded performance and virtual channels


Virtual channels allow for multiple conversations over the same link virtually at the same time.
Even when all traffic is the same priority, it is a good practice to balance traffic across virtual
channels so that should a device become impacted, it will have a minimal effect on other
traffic. This does not have to be a scientific planning project. For most enterprises, simply
stripping CHPIDs and devices at a diagonal across the channel cards is adequate rather than
straight across the top or bottom of the card is adequate. Note that due to the card design, the
same virtual channel is used when striping across the top or bottom.

It may be desirable to use QoS zones in some cases. The two most common reasons are:
򐂰 ISL bandwidth is limited and there is a mix of tier 1 applications and backup or mirroring
applications.
򐂰 Cascaded CTC paths are defined.
– CTC traffic is typically very small block. Each block, regardless of size, will utilize at
least one fibre channel frame. Each fibre channel frame requires one buffer credit so
CTC applications can starve other applications for buffer credits even though the link
bandwidth is not fully utilized.

108 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm

– Cascaded CTC is usually put into it’s own low QoS zone

The Excel planning workbook includes the virtual channels


.

Figure 6-3 Virtual channels

6.2.3 Ultra-low latency


Planning for ultra-low latency takes additional planning and therefore should only be done
when the application demands the highest speed switching with the lowest latency.

Brocade ASICs are designed to use cut-through routing. As soon as the header of a frame is
read, the frame is forwarded to the next port. Local switching occurs when the channel and
control unit are connected to ports that share the same ASIC. The switching latency within the
Gen 7 ASIC is 460 nsec. This latency is additive when traffic is traversing multiple ASICs. All
ASIC to ASIC switching is performed via a connection through the core blade even if the
ASICS are on the same port card. This involves another ASIC and backplane traffic.

Chapter 6. FICON design, implementation and migration 109


8497ch06.fm Draft Document for Review April 5, 2021 12:31 pm

Figure 6-4 Local switching vs. switching through the backplane

6.2.4 Channel paths: 32 vs 48 port cards


Since there are 48 ports per blade instead of 32, this means channel paths that were once
planned to land on the same FRU (blade) may no longer land on the same FRU. Conversely,
channel paths designed to use different FRUs may now be on the same FRU. In
<figure>Figure 5: 32 vs 48 port card channel path example.</figure>, CHPID 10 has an entry
link address of xx00 and control unit link addresses xx0F and xx1F. Similarly, CHPID 11 has
an entry link address of xx20 and control unit link addresses xx2F and xx3F. Each is on their
own FRU when connected a 32 port card. Assuming sequential addressing, with the 48 port
cards, CHPIDs 10 and 11 are on the same FRU and the control unit at link address 3F is on a
different FRU than the CHPID.

110 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm

Figure 6-5 32 vs 48 port card channel path example

The IBM CHPID Mapping Tool has not been updated to account for 48 port blades. To aide in
link address planning, there is an Excel Workbook available from your Brocade
representative.

6.2.5 Chassis configuration options using 48 port cards


Fibre channel addresses can be assigned to port cards in any order you want. The only rule is
that the fibre channel addresses must be unique. Most customers assigned sequential
addressing as this keeps documentation simple and is less susceptible to cabling and
maintenance action errors.

With 48 port cards, an 8 slot director can contain as many as 384 ports but since link
addressing is limited to 256 addresses per switch, enterprises have to consider how to make
use of the 48 port card.

Chapter 6. FICON design, implementation and migration 111


8497ch06.fm Draft Document for Review April 5, 2021 12:31 pm

Table 6-3 Chassis configuration options with 48 port cards


Option Discussion

8 Slot Chassis, one FICON logical This is not recommended but serves as a useful discussion of
switch how 48 port cards alter the chassis configuration.
A logical switch can be created on each chassis using just the
lower 16 ports on each column of ports per blade effectively
making it a 32 port card. Most customer who do this order some
empty blades and move SFPs from the upper unused ports.
Instead of leaving the upper 16 ports vacant, they could be put
into another logical switch but few, if any, mainframe customers
would have a use for such a logical switch.
An 8 slot chassis can also be ordered with just 5 cards for a total
of 240 ports or 6 card for a total of 288 ports, 32 of which would
be left unused or used for another purposes such as mirroring.

4 Slot Director The maximum port count will be limited to 192 ports but if FRUs
were considered in the channel path planning, a new CHIPD
map likely will be required.

8 Slot Director, 2 FICON logical Creating two FICON logical switches per chassis allows for the
switches per chassis greatest density. Instead of 2 paths per switch in each chassis,
use 1 path per logical switch with two logical switches per
chassis. Instead of 254 usable ports per chassis there can be as
many as 384 usable ports per chassis resulting in greater than
50% increase in port density for the same foot print. If a fully
populated director is not necessary, this approach can still be
used. The recommended layout is to start populating the first
logical switch with port cards on the left half of the chassis and
populate the right side of the switch with port cards for the
second logical switch. This allows for symmetric growth.

6.2.6 Two logical FICON switches per chassis


Although the only thing new in logical switching is the defined FICON logical switch, for those
new to the idea of using logical switches, logical switching provides some additional options to
fully utilize all ports of a 48 port card.

For FICON environments, when using one logical switch per chassis, the maximum number
of ports per logical switch is 256 ports (254 usable ports if CUP is enabled). Since 256 is not
evenly divisible by 48, either 5 port cards can be used which yields a total of 240 ports or 6
port cards which yields a total of 288 ports. In addition to losing port count with just 5 port
cards, using an odd number of port cards creates some planning challenges with redundant
pathing. Using 6 port cards wastes 32 ports. In either case, 2 or 3 slots cannot be used for
FICON unless there is a second FICON logical switch in the chassis.

Although partitioning a chassis into two symmetrical logical FICON switches results in a
maximum logical switch size of 192 ports, by splitting redundant paths to their own logical
switch there is an additional 64 link addresses available per path group. With two logical
FICON switches per chassis this results in a total of 128 additional link addresses available
per chassis.

When CUP is enabled, link addresses xxFE and xxFF are logical addresses. If there are ports
associated with those addresses then those ports must be disabled. Since there is no need to
associate link addresses xxFE and xxFF in a 192 port switch, those physical ports are also
available resulting in a slightly better than 50% port density improvement with this approach.

112 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm

Figure 6-6 Eight Logical Switches for 8-Way Pathing Option to Fully Utilize All Ports

6.2.7 ISL B/W sharing with logical switches


There are a number of other uses for logical switches as well. <figure>Figure 7: Logical
Switch Options</figure> is intended to illustrate all of the possible options in a single drawing.
It is not a recommendation. The idea of the base switch allows multiple logical fabrics to share
the same ISLs. When using a base switch for transport, these are referred to as XISLs. XISLs
have the following limitations:
򐂰 They are not visible through CUP.
򐂰 System Automation tools can’t be used to manage or monitor the links.
򐂰 RMF or Disk Magic will not have access to the 74-7 records on these links.
򐂰 The Quality of Service (QoS) feature cannot be used.
򐂰 Traffic Isolation Zones cannot be used.
򐂰 Additional B/W planning and congestion management considerations are required.
򐂰 Over subscription of XISL B/W in one logical switch will impact available B/W for other
logical switches.

It is possible for some logical switches to use the base switch (XISLs) while other logical
switches can be configured to have their own dedicated ISLs.

Chapter 6. FICON design, implementation and migration 113


8497ch06.fm Draft Document for Review April 5, 2021 12:31 pm

Figure 6-7 Logical Switch Options

6.3 FICON SAN implementation


The SANnav Management Portal replaces Network Advisor. SANnav has additional features
for troubleshooting, monitoring, and supports larger fabrics and total port count. Although
those features are useful in FICON environments, they are not specific to FICON and beyond
the scope of this chapter.

6.3.1 Chassis level management


Whether using SANnav or Network Advisor, all chassis management is performed on the
default switch. By default, the FID for the default switch is FID 128. If you are new to using
logical fabrics for FICON, this means you will have to leave the default switch discovered for
chassis management and chassis level alarms such as faulty power supplies.

6.3.2 FICON display mode setup


Mainframe system administrators use Hex values rather than decimal and prefer to see RNID
data rather than Name Server information. Link addresses, not WWNs, are used for
managing and monitoring channel paths. Similar to the Network Advisor FICON display
setup, SANnav has a similar feature. This display mode ensures that numbers are
represented in hex and adds RNID where appropriate in the displays.

Setting up the FICON display mode is done on a per user basis and should be the first a
FICON SAN user should do after setting up their account credentials.

1. In the upper right hand corner, click on the person icon and in the drop down box, select
“User Preferences”, as shown in Figure 6-8 on page 115.

114 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm

Figure 6-8 Configure User Preferences

2. In the display that follows, scroll to the bottom of the screen and find FICON Display under
“Tables”. If the FICON Display is “Disabled”, click “Edit”, as shown in Figure 6-9.

Figure 6-9 Edit FICON Display

3. Check the “FICON Display” box. Most users select “Yes” for “Persist Last Column
Selection”. When set to “Yes”, customization of the displays will persist after logging out
and logging back in. When done, click on “Save” at the bottom of the display, as shown in
Figure 6-10.

Figure 6-10 Select user preferences

6.3.3 FICON fabric configuration


Even when configuring a single switch fabric, it is a FICON fabric that is being configured. The
individual switches are part of the fabric and will take on the properties of the fabric except for
individual switch parameters such as the switch domain ID.

Follow these steps to configure FICON fabric:


1. Navigate To the SAN configuration screen.

Chapter 6. FICON design, implementation and migration 115


8497ch06.fm Draft Document for Review April 5, 2021 12:31 pm

2. Click SANnav.
c. Click SAN Configuration.
d. Click Logical Fabric Management.

Figure 6-11 Switch configuration

3. Create A FICON fabric


a. Click on “FICON Fabrics”
b. Click on “+” to create a new FICON fabric.

Figure 6-12 Select FICON fabrics

4. Configure fabric parameters


a. Give it a fabrid name.
b. (Optional) Give it a description.
c. Give it a FID number
d. Click “Add”.

116 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm

Figure 6-13 Configure fabric parameters

Table 6-4 Fabric parameters


Option Discussion

Name This is the name of the fabric, not the logical switch.

Description This is the description of the fabric, not the logical switch. With
single switch fabrics, this is often left empty but when cascading,
the recommendation is to add a meaningful description such as
“tape backup for …” or “disk mirroring for …”

Tags Tags are a new SANnav feature that allow you to assign a tag that
can be used for generating filtered reports. This is a database
search tag in SANnav. It is unrelated to the RNID tag. Creating a
tag is optional and can always be added at a later date. Tags in
mainframe environments are typically only used by very large
environments or when the same instance of SANnav is used to
manage both mainframe and open systems environments

Fabric ID A fabric ID (FID) is an integer 1-128 that is used to identify logical


fabrics. It is required. By default, the FID for the default switch is
128. Although this can be changed, the recommended best
practice is to continue to use 128 for the default FID and create
a unique FID for other logical fabrics. It is also a common practice
to use FID 127 for the base switch for ISL sharing. Choosing a
FID number in the range of 1-126 therefore is recommended.
This is strictly a fibre channel requirement. It has no effect on
HCD or any other references to a FICON switch. The Domain ID,
not the Fabric ID, is what is used for the high order byte of the link
address in HCD.
The FID must be unique for each logical switch in a chassis and
must be the same for all logical switches in the same fabric. It
does not need to be unique for each fabric on different chassis.
Although the specific FID number does not, having a FID
numbering convention is recommended. For example:
All FICON fabrics will use FID 1
When a chassis is carved up into two FICON fabrics, slots 3, 4,
5, & 6 will use FID 1 and slots 9, 10, 11, & 12 will use FID 2
Logical fabrics for mirroring will use FID 10

5. Create the FICON Logical Switch(es) for the Fabric.


a. Select chassis to create FICON switches for this fabric.
In this screen you are selecting the chassis where the FICON logical switch will be
created. For non-cascaded environments, you will only be choosing one chassis.
Although switches can be created one at a time and later merged, if you create all the

Chapter 6. FICON design, implementation and migration 117


8497ch06.fm Draft Document for Review April 5, 2021 12:31 pm

logical switches for a single cascaded fabric at creation time, you will save some
configuration steps in the future.

Figure 6-14 Logical switch selection

b. Set the Domain IDs for each switch in the fabric. The domain ID (DID) is the highest
byte of the 2-byte link address. The recommended best practice is to make the DID
match Switch ID in HCD.

Figure 6-15 Set the switch domain ID

c. Add the ports. Most administrators will click on the “Slot/Port #” column to sort by port
number. The right hand side of this set up page, not illustrated below, allows you to
manually edit the port address (low byte of the 2-byte link address). By default, port
addresses are assigned sequentially so the recommendation is to select the ports in
the order you want the link addresses to be in. Typically, this is the same order the
slot/port is listed in.

118 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm

Figure 6-16 Select the port to add

d. Click “OK” to finish adding the ports.

Figure 6-17 Complete adding the ports

e. Configure the remaining fabric parameters. These parameters are applied to all logical
switches in the fabric.
f. Select “Fabric Parameters” to display parameter options.
g. Fill in the desired parameter options, as described in Table 6-5 on page 120, and click
“Save”.

Chapter 6. FICON design, implementation and migration 119


8497ch06.fm Draft Document for Review April 5, 2021 12:31 pm

Figure 6-18 Configure the remaining fabric parameters

Table 6-5 Fabric parameter definitions


Option Discussion

RA TOV Do not change

ED TOV Do not change

Maximum Hops Do not change

BB Credit Do not change. This is not how BB credits are changed


on a per port basis.

Data Field Size Do not change

Routing Policy As discussed in “Routing policy” on page 128

Allow XISL Should only be selected if the ISLs are in a base switch
to be shared with this logical switches in the same
chassis.

Enable Device Probing Should be un-checked for FICON fabrics

Per-frame routing priority Should be un-checked for FICON fabrics

Suppress Class F traffic Should be un-checked for FICON fabrics

120 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm

Option Discussion

Long distance fabric Checking this box adds buffer credits to the ISL ports
(E-Port) so it only makes sense to check this box in
cascaded environments. The number of buffer credits
depends on the speed of the link and fibre channel
generation. Adding buffer credits allows more frames to
be in flight which can have unintended effects on
congestion with short distance links. It also means there
are more frames to recover in the event of a network
failure. This box should only be checked when additional
buffer credits are needed.
For normal tape and DASD traffic, setting a long distance
mode is usually not necessary until the distance exceeds
1 Km so this is typically not set for cascading within a
campus environment. For distances exceeding 10 Km.,
other methods are used for providing additional buffer
credits so this is typically set for distances between 1 and
10 Km. The exceptions are channel to channel, CTC, or
extension through DWDM.
Block storage today always uses High Performance
FICON (zHPF) which means the Extended Count Key
Data (ECKD) frames are packaged with the data so the
average fibre channel frame size for FICON traffic
approaches the full 2K byte fibre channel frame size.
CTC traffic is typically used for transporting small
messages and therefore the average frame size is
typically very small. Since each fibre channel frame uses
a buffer credit, additional buffer credits are required for
CTC traffic.
Some active DWDM equipment supplies buffer credits so
there is no need to increase buffer credits. When
cascading via DWDM, it is important to discuss whether
the added buffer credits need to be configured on the
SAN switch or the DWDM interface with the DWDM
vendor.
Details of how and when to configure additional buffer
credits for extended distance links is beyond the scope of
this RedBook. Check with your IBM technical
representative before configuring any cascaded
environment.

Activate When checked, the logical switches will be enabled as


will all the ports after the creation completes. Typically,
the cabling is done before the logical switch creation so it
is common to check the “Activate” box. It is not safe to
perform cabling duties with enabled SFPs so this box
should not be checked if the cabling operations have not
been completed.

6. Discover The Logical Switches


Logical switches should be discovered by default. The system administrator may have
disabled automatic discovery or a network error may have prevented discovery. It's always
prudent to validate that the switch was in fact discovered.
7. Rename The Logical Switches
The switch names by default are “Switch_” followed by the FID number. In the example
used in this section, this means both switches will be named “Switch_10”. Go to the
“Inventory” screen to change the names of these switches.

Chapter 6. FICON design, implementation and migration 121


8497ch06.fm Draft Document for Review April 5, 2021 12:31 pm

8. Disable Unused Ports


It is not safe to have lasers enabled on SFPs that have nothing connected. Note that in
some cases the open connection is at the patch panel, not the switch. As a safety
precaution, all unused ports should be disabled.
9. Zone The FICON Ports
The usual recommendation for FICON zoning is:
– Put all ports into a single zone based on the port domain, index
– Do not put ICL ports in the zone
– Although it is ideal to not have any E-Ports in the zone when cascading, having the
E-Ports in the zone will not cause problems.
– Consider additional zoning granularity. For example, segregate mirrored ports with
zoning.
– Domain, Port Index zoning is always used for FICON.
– To simplify zoning algorithms and to accommodate various switch form factors, a port
index rather than a port number is used for zoning. This is a subtle difference few
customers notice. As long as the ports are in the zone, the internal zoning mechanism
will have no effect on how paths are configured with HCD.
– Create a zone configuration with the zone.
– Activate the zone configuration
– Disable “All Access”

a. Create a zone, as follows:


i. Select Zoning.
ii. Select Zones.
iii. Click +.

Figure 6-19 Create a zone

b. Select the fabric. Remember that zoning is applied to a fabric, not an individual switch.
Since most FICON fabrics are single switch fabrics, this distinction is subtle; however,
keep in mind that you are looking for the fabric name, not the switch name, when
zoning. In this example, “FICOn-XRC-z15” was the name of the fabric created.

122 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm

Figure 6-20 Select the fabric to zone

c. Name the zone and configure the type of zone.

Figure 6-21 Name zone and configure zone type

d. Add the ports to the zone. Typically, all but the ISL ports are added to the zone.

Figure 6-22 Add the ports to the zone

e. Complete the zone creation and save it.

Chapter 6. FICON design, implementation and migration 123


8497ch06.fm Draft Document for Review April 5, 2021 12:31 pm

Figure 6-23 Complete the zone creation

f. The zone, or zones, need to be added to a zone configuration. A zone configuration is


a collection of zones. For FICON users who have defined only one zone, this may
seem redundant.

Figure 6-24 Create a zone configuration

Figure 6-25 Define the zone configuration

g. Add the zone(s) to the zone configuration. In open systems environments, it’s common
to have multiple zone configurations so activating the zone configuration is optional.

124 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm

For most FICON environments, there is only one zone configuration which is usually
activated as soon as the zone configuration is created.

Figure 6-26 Save and activate the zone configuration

h. For added security, disable the “All Access” zone.

Figure 6-27 Disable the All Access policy

6.3.4 Managing FICON fabrics


As of this writing, the currently available version of SANnav has the following limitations:
򐂰 The equivalent of the Network Advisor connectivity display is split into multiple screens,
primarily one for channel connections and one for storage connections.
򐂰 The RNID type for the connected device was missing.

Both of these issues are scheduled to be addressed. It’s likely the corrections will be made in
SANnav before most enterprises deploy SANnav so this note is primarily to explain what is
missing in the screen captures.

Chapter 6. FICON design, implementation and migration 125


8497ch06.fm Draft Document for Review April 5, 2021 12:31 pm

Customizing the display


Optional. Setting the FICON display mode defaults all column displays to the typical displays
with RNID data that most mainframe administrators prefer to see but the displays can be
customized.

Figure 6-28 Customizing the display

Viewing connections

Figure 6-29 View channel connections

The resulting display is:

Figure 6-30 CHPID connectivity display

126 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm

Similarly, control units can be monitored by clicking on “Storage Ports” instead of “Host Ports”.

6.3.5 Converting a production switch to a FICON logical switch


This section describes how to check if an outage is required for customers considering
converting an existing switch that is not currently configured for a defined FICON logical
switch to use a defined FICON logical switch. It is expected that most customers will not need
to make this change in an existing environment.

The easiest way to check is via the CLI. Save the session output so you can email it to your
Brocade business partner or direct to your Brocade representative in the event you need
assistance.

Check HIF & address mode


Validate that the address mode is mode 1 (zero based) and that the high integrity fabric (HIF)
mode is set:
switch_30:FID30:admin> switchshow
switchName switch_30

switchType 166.0

switchState Online

switchMode Native

switchRole 1

switchDomain Principal

switchId fffc01

switchWwn 10:00:c4:f5:7c:64:5b:62

zoning OFF

switchBeacon OFF

FC Router: OFF

HIF Mode ON

Allow XISL Use OFF

LS Attributes FID: 30, Base Switch: No, Default Switch: No, Ficon
Switch: No, Address Mode 1]

If HIF Mode is not ON or the Address Mode is not 1, you likely will need an outage.

Validate address binding


switch_30:FID30:admin> portaddress --show
Partition Address Mode :1
Index Slot Port Area Mode User_bound
===========================================
72 9 8 0x0000 8 bit Y
73 9 9 0x0200 8 bit Y
74 9 10 0x0300 8 bit Y

Chapter 6. FICON design, implementation and migration 127


8497ch06.fm Draft Document for Review April 5, 2021 12:31 pm

75 9 11 0x0400 8 bit Y
76 9 12 0x0500 8 bit Y
A Y must appear in the column for all ports.

Validate FICON logical switch


In response to the switchshow command HIF mode must be “ON” and the Ficon Switch
must be “Yes”.
switch_30:FID30:admin> switchshow
switchName switch_30

switchType 162.0

switchState Online

switchMode Native

switchRole Principal

switchDomain 1

switchId fffc01

switchWwn 10:00:c4:f5:7c:16:8b:3e

zoning OFF

switchBeacon OFF

FC Router: OFF

HIF Mode ON

Allow XISL Use OFF

LS Attributes FID: 30, Base Switch: No, Default Switch: No, Ficon
Switch: No, Address Mode 1]

Routing policy
This primarily effects load balancing in cascaded environments. Most FICON environments
today support FICON Dynamic Routing (FiDR).

The routing policy must be the same for all switches in a SAN but each SAN is independent
from one another and therefore each SAN may have different routing policies. Work Load
Manager will continue to operate as expected even though paths are connected to switches
with different routing methods. This means upgrades can include a routing method change
and each SAN upgraded in separate change control window.

A route is the set of switches and ports a path takes to deliver fibre channel frames from the
channel to the control unit. As a practical matter, this is only significant in cascaded
environments.

128 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm

Table 6-6 Routing methods and descriptions


Routing Method Description

Port Based Routing (PBR) Deprecated. Many older FICON SAN deployments are
configured this way.
Routes are determined by a predefined map for each possible
route in the SAN. The channel path will always take the same
route (same ISL or ISL trunk group).
Today, the only reason to configuration a switch for PBR is in the
event a new switch must be connected to an older switch still
configured for PBR. Setting PBR is not an option in SANnav but
can be done via the Command Line Interface (CLI). Details on
how to do this are beyond the scope of this RedBook. Contact
your technical account team in the event configuring PBR is
required.

Device Based Routing (DBR) Default. Most FICON SAN deployments are configured this way.
Routes are determined by usage. A round robin algorithm is used
to select the ISL or ISL trunk group. Routes are established
dynamically as a channel to control unit path is needed but once
that path is selected, the channel path will always take the same
route. This has a better statistical probability of balancing work
load across ISLs than PBR.

Exchange Based Routing (EBR). Analogous to FICON Dynamic Routing (FiDR). Although EBR is
the preferred routing method, DBR is the default because many
FICON SANs still have older control units that do not support
FiDR. Typically, these are older tape control units. Check with
your storage vendor to be sure FiDR is supported before
selecting EBR for the routing method. Some control units may
require a driver upgrade.
EBR selects routes dynamically on a per exchange basis. A fibre
channel “exchange” is analogous to a CCW chain. The least
used ISL or ISL trunk group for a route is selected at the time a
switch receives a new CCW chain from the channel. The route
remains intact for the life of the CCW chain. This method results
in the most efficient and dynamic use of ISL bandwidth.

Exchange-based routing and FiDR background


Although this level of detail is not required knowledge, questions about FiDR and EBR are
common.

Mainframe channel protocol was developed for buffered control units that managed physical
storage devices. Channel programs were built around channel command words (CCWs).
Write commands included data. CCWs could be, and usually are, strung together in a
channel program.

Status is a bit flag in a byte returned by the control unit. The only status bits pertinent to this
discussion are Channel End (CE) and Device End (DE). CE indicates that the control unit
accepted the channel program. Device End (DE) indicates that the operation with the physical
storage completed. So that a channel can perform other work while a control unit was working
on a channel program, status could be split in that CE was returned to the channel and DE
after the control unit performed the action with the physical storage.

In a serialized bit stream across fibre channel, CCWs are broken up into fibre channel
packets and the packets sent to their destination. The equivalent to a bus connection in fibre
channel is an exchange. All CCWs, and therefore all fibre channel packets, in a channel
program are sent on the same exchange so all packets arrive in order.

Chapter 6. FICON design, implementation and migration 129


8497ch06.fm Draft Document for Review April 5, 2021 12:31 pm

When split status is returned to the channel, DE is returned on a different exchange. The
protocol hasn’t changed but devices are now often in memory so the control unit can
complete the operation much more quickly. This means DE can be available almost
immediately after returning CE. Since DE is on a different fibre channel exchange, the fibre
channel packet with DE can be routed across a shorter or less congested ISL thereby arriving
before CE. Since it’s not possible for a device to complete an operation before the control unit
completed the operation, the channel program never had to account for receiving DE prior to
CE. If this occurred prior to FiDR, a channel detected error would be reported and the job may
terminate abnormally (ABEND).

FiDR recognizes this potential race condition and waits for CE instead of returning an error.
Both the channel and control unit must support FiDR.

6.4 FICON SAN migration


Getting a SAN Health report is the best way to answer to determine migration tasks. It’s also
the best documentation for your account team to become familiar with your SAN before giving
you advice. Since SAN Health reports are confidential, please remember to authorize your
IBM and Brocade representatives to receive a copy.

It is anticipated that most migrations will be from Gen 5 to Gen 7.

6.4.1 Firmware
All supported FICON switch configurations today with any FICON supported version of FOS
are supported with switches defined as a FICON logical switch. This means that all upgrade
scenarios that include a mix of logical switches defined as FICON and FCP switches are
supported; however, all switches in a fabric cannot be more than one major FOS revision
apart. Any switches on FOS 7.x must be upgraded to 8.x before any other switch in the fabric
can be upgraded to 9.x.

6.4.2 FICON logical switch


Upgrades from FOS 8.x on Gen 6 switches to FOS 9.x will be blocked for switches that do not
meet the FOS 9.0 requirements. To avoid the need for an outage, it is highly recommended
that all new Gen 6 switch configurations have virtual fabrics enabled and a logical switch
defined as a FICON logical switch.

As of this writing, the FOS v8.x code stream was still actively supported but eventually, it will
reach end of support prior to FOS 9.x and likely prior to a hardware upgrade of the Gen 6
switch. At some future date, Gen 6 switches will need to be upgraded to FOS 9.x for CVE
patches and other changes. Enterprises with Gen 6 switches operating on FOS 8.x firmware
that were not configured for a FICON logical switch should plan on upgrading the
configuration so that a FICON logical switch is used for all FICON connections. There is no
need to change the addressing so there is no need to re-design the channel paths. Use
Table 6-7 to determine if an outage will be required to configure a FICON logical switch.

Table 6-7 Requirements for a non-disruptive reconfiguration to a FICON logical switch

130 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm

Parameter Requirement Comments


for
non-disruptive
conversion

Switch type SAN512B-6 The SAN42B-R (2498-R42) and SAN64B-6


(8961-F08) or (8960-F64/N64) will always require and outage.
SAN256B-6
(8961-F04)

Logical switch Non-default VF must be enabled and a logical switch other than the
logical switch default logical switch must be properly configured for
FICON.

Address mode Zero based Zero based address mode is the addressing mode that
(mode 1) ensures the fibre channel address assigned to a port is in
compliance with the channel path link addressing. It is not
always set today because certain configurations were
inherently in compliance so you will need to check.

High Integrity Must be enabled When HIF is set, this ensures the security policy is set and
Fabric (HIF) distributed throughout the fabric, insistent domain ID is set.
All properly configured FICON SAN fabrics should have
HIF enabled today.

Port Address Must be in effect All properly configured FICON SAN fabrics should have
Binding port address binding in effect today.

6.4.3 Inter Chassis Links (ICL)


There are no ICL considerations specific to FICON connections. The considerations for ICLs
in a FICON environment

6.4.4 Cabling
Although single-mode cable replacement is generally not necessary in mainframe
environments, it’s not uncommon to have to replace a small percentage cables. The most
common reason for this is that during cable cleaning a lens is scratched. In some cases,
degraded cables may not exhibit problems until higher speeds are attempted. Although
multi-mode cable is rare in mainframe environments, there are some exceptions. The most
common exception for using multi-mode cable is for DWDM connections. All multi-mode cable
should be at least OM-4 rated.

As with any repurposing of cabling, good cable hygiene is recommended which should
include cleaning all connection points. Even when performing a push-pull upgrade of a switch,
connector cleaning should include the connections at the patch panel and the channel or
control unit in addition to the connection at the switch port.

6.4.5 Patch room & fibre plant


When migrating from products with different form factors or changing port count, this may
affect documentation and patch room design. This is especially so when migrating from
director class products with 32 port blades to 48 port blades.

Chapter 6. FICON design, implementation and migration 131


8497ch06.fm Draft Document for Review April 5, 2021 12:31 pm

6.4.6 SFPs
When considering re-purposing SFPs, remember that Gen 7 switches and switch blades
require SFPs designed with the secure optics feature. This means that although Gen 7 can
accept a 32G optic, you cannot repurpose 32G optics from Gen 6 switches in a Gen 7 switch
or switch blade.

As with all previous fibre channel switching products from Brocade, the latest generation
product supports SFPs of the same generation and one generation below. A Gen 7 switch
port is capable of 64G and therefore can accept a 64G SFP or a 32G SFP. A Gen 6 switch
port is capable of 32G and therefore can accept a 32G or 16G SFP. Each SFP supports 3
speeds, the current generation, one generation back, and two generations back.
򐂰 A 64G optic can negotiate to 64G, 32G, and 16G.
򐂰 A 32G optic can negotiate to 32G, 16G, and 8G.
򐂰 A 16G optic can negotiate to 16G, 8G, and 4G.

If a 4G connection is required, typically for older tape or DWDM, you will need a 16G SFP and
since 16G SFPs are not supported on Gen 7 products, this means you will need either a Gen
6 switch or Gen 6 switch blade to accommodate older connections that require 4G. You can
use the Brocade branded 16G SFPs from your Gen 5 products in a Gen 6 switch or switch
blade but you should only do so if 4G support is required.

Most FICON SANs are single switch (single domain) fabrics. Channel paths can be striped
across SANs using a mix of Gen 5, Gen 6, and Gen 7 technology as long as they are
configured properly for FICON. As long as they are not in the same fabric, they can be on any
version of FOS supported for FICON, can have different addressing modes, and can have
different routing methods. This means switches in a channel path can be upgraded without
the need to be concerned with redundant channel paths.

Figure 6-31 Mixed Gen 5 & Gen 7 Single Switch Fabrics

In cascaded environments, a non-VF enabled switch can be connected to a logical switch that
has VF enabled as long as every switch in the fabric that has VF enabled has the same Fabric
ID (FID). To minimize outages, most customers will upgrade both switches in the channel path
but should the need arise, mixing switch generations in the same fabric is supported but with
the restrictions noted below. Keep in mind that a SAN42B-R (2498-R42) extension switch is a
Gen 5 switch.

Configuration requirements
򐂰 The FOS code must be at v9.0 or higher in the Gen 7 switch and a supported version of
v8.x or higher in the Gen 5 or Gen 6 switch.
򐂰 All the fabric parameters must match

132 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm

򐂰 For FICON environments, the routing policy is the most likely parameter that will be
different since most Gen 5 implementations used either port based routing or device
based routing and new installs are likely to use exchange based routing (FiDR).
򐂰 It’s unlikely that other fabric parameters will be different; however, should the need to mix
switch generations in the same fabric arise, contact your technical account team to
validate all fabric settings.

Figure 6-32 Sample Mixed Gen 5 & Gen 7 Cascaded Fabrics

FMS (aka CUP)


Enabling the FICON Management Server (FMS) allows administrators to define and configure
the Control Unit Port (CUP) device online. All chassis level alerts are reported to the CUP
device on all logical switches so there is no need to dedicate a channel to the default switch.
All other actions, such a display matrix commands and 74-7 record capturing, are limited to
the logical switch where the channel with CUP is connected. If multiple logical switches are in
use you will need to define CUP on at least one channel connected to each logical switch
where CUP management is desired.

Gen 5 to Gen 7 fixed port switches


Background

The Gen 7 SAN64B-7 (8960-P64/R64) has an additional 8 ports but otherwise has the same
physical format as the Gen 6 SAN64B-6.

Physical upgrade

Assuming slower login speeds are not an issue, a simple push-pull can be done for physical
installation when upgrading from a SAN48B-5 to a SAN64B-7.

CHPID path mapping

It is not uncommon for enterprises to make CHPID path changes as part of an upgrade but
there are no changes in the SAN64B-7 that require changes to the IODF.

Gen 6 to Gen 7 fixed port switches


Background

To take advantage of some space available on the front panel and to make full use of all ports
on the ASIC, the Gen 6 switch also has 4 QSFP ports. The QSFP ports were only qualified as
ISL ports when used in FICON environments. They could not be used with breakout cables
for FICON connectivity. They could have been used with breakout cables in a logical switch
for non-FICON connectivity.

The QSFP ports on the SAN64B-6 were replaced with dual capable SFP ports so the only
change will be for environments that used the QSFPs for inter-switch links (ISLs) or for
non-FICON applications.

Few, if any, enterprises fully utilized all QSFP. In the event that more than 56 ports were used
with the SAN64B-6, an additional switch will be required. The additional 8 ports on the

Chapter 6. FICON design, implementation and migration 133


8497ch06.fm Draft Document for Review April 5, 2021 12:31 pm

SAN64B-7 are designed to accept a dual SFP but as of this publication date, only standards
SFPs were available.

Since using QSFPs is uncommon for fibre channel, the equivalent Gen 7 switch was
designed to use the space previously used for QSFP for SFP ports that support double
density SFPs. As of this writing, double density SFPs were not available but it is anticipated
that the double density SFPs will only be supported for ISLs, not FICON connections. Since
those ports support standard SFPs as well, with three 8 port licenses the SAN64B-7 can be
configured as a 56 port switch.

Physical upgrade

For environments that used the QSFPs as ISLs with a total port count of 56 or less, the ISL
connections have to be moved off of the QSFPs to the SFP ports. Replacing the SAN64B-6
therefore may involve some re-cabling but otherwise, a simple push-pull is possible.

CHPID path mapping

It is not uncommon for enterprises to make CHPID path changes as part of an upgrade but
since the SAN64B-7 supports more FICON ports than the SAN64B-6, there are no required
changes to the IODF when upgrading Gen 6 switches to Gen 7 technology.

Gen 6 to Gen 7 directors


Background

The Gen 6 directors, SAN256B-6 and SAN512B-6, have the same physical card layout and
form factor as the Gen 7 directors, SAN256B-7 and SAN512B-7.

Physical upgrade

In most cases, a simple push-pull of the Gen 6 director for a Gen 7 director is possible.
Exceptions are when 4G connections must be supported because these connections require
a 16G SFP which is not available for Gen 7. The options to accommodate older 4G
connections is to move them to a fixed port switch or add a Gen 6 blade to the Gen 7 chassis.
Gen 6 blades are available for purchase or can be migrated from an existing Gen 6 director to
a Gen 7 director.

CHPID path mapping

It is not uncommon for enterprises to make CHPID path changes as part of an upgrade but
there are no changes in the SAN256B-7 or SAN512B-7 that require changes to the IODF.

Gen 5 to Gen 7 dDirectors


Background

Most, if not all, FICON implementations on Gen 5 directors, SAN768B-2 and SAN384B-2,
were with 32 port blades. SAN upgrades typically skip a generation so Gen 5 to Gen 7 is the
most common upgrade path to Gen 7. Since redundant channel paths are usually planned to
be on different FRUs (port cards), the same channel path mapping used with 32 port blades
likely will not work on 48 port cards. This is the primary consideration when upgrading from
Gen 5 to Gen 7 directors.

The Control Processor (CP) blades are now half height blades occupying the first physical
slot in the chassis. Reducing the total number of physical slots from 12 to 11 was done to
allow a little extra air flow to improve cooling for the port cards. This was necessary because
the higher speed optics produce more heat.

134 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm

These changes effect:


򐂰 CHPID Path Mapping.
򐂰 Cable diagrams and other documentation that include chassis slot numbering.
򐂰 Supported SAN switch addressing modes.
򐂰 Patch room design.
򐂰 Automation such as automatic service ticket generation.

Physical upgrade

The control processor (CP) cards in the 8510-4 and 8510-8 chassis occupied full slots in the
middle of the chassis. With the Gen 6 and Gen 7 chassis, the CP blades are half height
blades, both installed in the first chassis slot. The slot numbering therefore is different.

To distinguish the physical location from one CP to the other, they are referred to as slots 1 &
2 even though they both occupy the same physical space as a single slot would occupy.

Figure 6-33 Gen 5 vs. Gen 6 and Gen 7 Card Slots

Since most Gen 5 directors deployed for FICON used the 32 port card, it’s likely that several
cabling changes will need to be made. The specific cabling changes will depend on the
design approach taken. For patch room designs that were planned to match the symmetry of
the blades and channel paths, changes, at least in documentation, may be required for the
patch room as well.

Summary of physical upgrade considerations:


򐂰 All documentation templates must be updated.
򐂰 Custom scripts and software to automatically generate service tickets or other monitoring
activity may need to be updated.
򐂰 The patch room layout may need to be redesigned.
򐂰 Cabling methodologies will likely change.

CHPID path mapping

In most deployments, significant changes to the IODF will be required. Much of the design
section is dedicated to resolving channel pathing that arise as a result from architectures
build around 32 port cards to 48 port cards.

Chapter 6. FICON design, implementation and migration 135


8497ch06.fm Draft Document for Review April 5, 2021 12:31 pm

Additional resources and references:

Brocade FICON Resource Page:


https://www.broadcom.com/info/brocade/ficon-resources

IBM resources
򐂰 SANnav Introduction:
https://www.ibm.com/support/pages/introducing-ibm-sannav
򐂰 SANnav Announcement:
https://www-01.ibm.com/common/ssi/ShowDoc.wss?docURL=/common/ssi/rep_ca/1/897/E
NUS120-061/index.html&request_locale=en
򐂰 SANnav Data Sheet:
https://www.ibm.com/downloads/cas/4LX0PXRL

136 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

Chapter 7. IBM b-type Gen 7 extension


design, implementation, and
migration
Extension is the most common and least expensive form of FC/FICON storage connectivity
across a distance IP network. FCIP (Fibre Channel over Internet Protocol) is a technology
that allows block data replication over long distances by using the TCP/IP protocol. IP
Extension is a technology that accelerates, secures, and monitors IP storage replication flows
over long-distance IP networks. In most cases, Extension is implemented in Disaster
Recovery (DR) scenarios with some kind of data replication between the primary and
secondary sites.

Extension is a tunneling technology, which means FC frames (FCIP) or IP datagrams (IP


Extension) are encapsulated into TCP segments and IP datagrams (TCP/IP). As such,
encapsulation is transparent to devices connecting through via Extension. To use Extension,
a tunneling device is required on both ends of the TCP/IP link that integrates multiprotocol
connectivity. Extension platforms are either stand-alone devices (IBM SAN42B-R and IBM
SAN18B-6) or blades combined with a director-class product (IBM SX6 Extension Blade).
Storage arrays such as the IBM DS8900 and FlashSystem 9200 support IBM b-type Gen 7
Extension connectivity.
For more information, see IBM FlashSystem 9200 and 9100 Best Practices and Performance Guidelines
at http://www.redbooks.ibm.com/redbooks/pdfs/sg248448.pdf.

An essential aspect of the Extension scenario is the IP link quality. With Extension, dedicated
bandwidth is prudent if the link is shared with other traffic. Because the connection between
two sites is low-traffic or used only for e-mail, do not assume that this is always the case.
Storage is sensitive to latency and congestion; a spyware problem or a DDOS attack on the
IP network can be disruptive. Furthermore, data inflight should always be encrypted, which is
easily done on IBM b-type Gen 7 Extension platforms.

Also, when you are communicating with your organization’s networking architects, distinguish
between megabytes per second (MB/s) and megabits per second (Mbps). In the storage
world, bandwidth is often specified in MB/s, but network engineers specify bandwidth in
Mbps. If you want 1 GB/s and you fail to designate GB explicitly, you may end up with 1 Gbps,

© Copyright IBM Corp. 2021. 137


8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

which supplies approximately 100 MB/s. If you include safety margins, this link is not as fast
as you may need, so ensure the correct terminology.

This chapter describes the IBM b-type Gen 7 (Brocade) Extension family of products. It
includes an overview of the physical platforms, the different sides of an Extension platform,
features, terminology, implementation, technical insights, and architectures.

This chapter includes the following topics:


򐂰 7.1, “Extension platform overview” on page 139
򐂰 7.2, “Extension design and architectures” on page 141
򐂰 7.3, “Extension implementation” on page 153
򐂰 7.4, “Extension migration strategy” on page 194
򐂰 7.5, “Extension validation” on page 198
򐂰 7.6, “Extension Monitoring” on page 204

138 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

7.1 Extension platform overview


IBM Extension offers advanced data transport features accommodating various
environments, from small/medium businesses to the largest most demanding enterprises.
Extension is a high-speed, high-volume data transport technology for moving storage
between data centers over an IP network. Extension provides exceptional data transport
efficiency by incorporating sophisticated high-efficiency encapsulation, transport, security,
compression, and data recovery mechanisms.

Moving FC data over an IP network is referred to as FCIP. IP storage data can be transported
by IP Extension, which benefits from the advanced transport technology and encryption.

7.1.1 IBM b-type Gen 7 extension platforms


At a high level, this section familiarizes you with each physical Extension platform. Please
refer to the datasheets for more information.

IBM SAN18B-6
The IBM SAN18B-6 is a 1RU chassis of standard width, as shown in Figure 7-1. The front has
access to the serial port, Management Ethernet port, USB port, twelve 32G FC ports (Gen 6),
and six application Ethernet interfaces (GE0/GE1 and GE2/GE3 are mutually exclusive).

The IBM SAN18B-6 has two license modes, base and full. The full unit has a WAN side
maximum of 2.5 Gbps. The base unit has a WAN side maximum of 1 Gbps and is enforced by
disallowing circuits to have more than a cumulative maximum rate of 1 Gbps. There is one DP
(data processor).

The IBM SAN18B-6 does not support Virtual Fabrics or FICON.

Figure 7-1 IBM SAN18B-6/Brocade 7810 front view

The IBM SAN18B-6 combines the power supply and fans into a single hot-swappable FRU, as
shown in Figure 7-2. There are two redundant FRUs located in the rear of the chassis. An
offline power supply does not prevent power to any fan; a single online power supply can
power all fans. Airflow is unidirectional rear to front.

Figure 7-2 IBM SAN18B-6/Brocade 7810 rear view

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 139
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

IBM SAN18B-6 data sheet


For more information on IBM Storage Networking SAN18B-6, go to
https://www.ibm.com/downloads/cas/BRMYZADG

IBM SAN42B-R
The IBM SAN42B-R is a 2RU chassis of standard width, as shown in Figure 7-3below. The
front has access to the serial port, management Ethernet port, USB port, twenty-four 16G
FC/FICON ports (Gen 5), sixteen application GE/10GE Ethernet interfaces, and two 40GE
interfaces.

The IBM SAN42B-R has three license modes: 5 Gbps (base unit), 10 Gbps (Upgrade-1), and
40 Gbps. Upgrade-2 requires Upgrade-1, and Upgrade-2 is necessary to enable the 40GE
interfaces.

There are two DP (data processors), each capable of 20 Gbps. 40 Gbps is the maximum
WAN side bandwidth. 5 and 10 Gbps modes are limited by disallowing cumulative circuit
bandwidth above the licensed limit. The full 40 Gbps licensing has no restriction on the
cumulative circuit rate; nonetheless, 40 Gbps is the maximum hardware capacity.

The IBM SAN42B-R supports Virtual Fabrics and FICON.

Figure 7-3 IBM SAN42B-R/Brocade 7840 front view

Note: Additional optional hardware is not available for the empty slot.

The IBM SAN42B-R has a separate power supply and fan FRUs, as shown in Figure 7-4.
There are two redundant power supplies and three fan trays located in the rear of the chassis.
An offline power supply does not prevent power to any fan; any online power supply can
power all fans. Airflow is unidirectional rear to front.

Figure 7-4 IBM SAN42B-R/Brocade 7840 rear view

IBM SAN42B-R data sheet


For more information on IBM System Storage SAN42B-R Extension Switch, go to
https://www.ibm.com/downloads/cas/7AMMQMM2

140 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

SX6 extension blade


Up to 4 IBM SX6 Extension Blades are supported in IBM SAN256B-7/SAN512B-7 Directors,
as shown in Figure 7-5. Two classes of the SX6 Extension Blade are available:
򐂰 SX6 for Open Systems:
– Includes all license except those specific to the mainframe
– No circuit bandwidth limits
– Includes SWL MMF optics
򐂰 SX6 for Mainframe:
– Includes all licenses, including FICON Accelerator and FICON Management Server
(CUP)
– No circuit bandwidth limits
– Includes LWL SMF optics

The front has access to sixteen FC/FICON ports (Gen 6 - supported in Gen 7 Directors),
sixteen application GE/10GE Ethernet interfaces, and two 40GE interfaces. The Ethernet
SFP/SFP+ optics are not included and must be ordered as needed. Brocade branded optics
are required to come online.

There are two DP (data processors), each capable of 20 Gbps. 40 Gbps is the maximum
WAN side bandwidth. The SX6 Extension Blades have no restriction on cumulative circuit
rate; nonetheless, 40 Gbps is the maximum hardware capacity.

Figure 7-5 Brocade SX6 extension blade

IBM SX6 Extension Blade Data Sheet

For information on the IBM SX6 Extension Blade, go to


https://www.ibm.com/support/knowledgecenter/en/HW29A/san512b6.doc/extension_blade_o
verview.html

7.2 Extension design and architectures


Most commonly, Extension is used for Business Continuance via Disaster Recovery. Protect
critical data by replicating it outside the area of a potentially catastrophic event. Remote Data
Replication (RDR) and remote tape are leveraged for this purpose. Data replication systems
with Extension permits organizations to recover operations in a timelier manner.

The sections below cover the most common Extension designs and architectures.

7.2.1 FCIP design and architectures


Typically, RDR is a disk-based array to array replication. The local production site storage
array sends data to the backup site array. Often, Native FC is used when the recovery site is
within a metropolitan distance, and DWDM or dark fiber exists between the sites; however,
frequently, cost-sensitive ubiquitous IP infrastructure is available.

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 141
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

IBM Extension is high-speed and adds about 75 µs of propagation delay per pass through the
platform. Four passes round trip = 0.3 ms total added propagation delay, added to the
network latency. In some cases, the added propagation delay may be acceptable for
synchronous RDR applications. Deployment best practice connects array N_Ports to
Extension F_Ports directly and does not connect through the production fabric. Storage
replication ports are dedicated to RDR and should not pass through fabrics with host traffic,
which can cause head-of-line-blocking issues for hosts. Nevertheless, there remain valid
reasons to connect via the production fabric, such as tape applications and too many arrays
ports for the Extension platforms. If additional ports are required and connecting to the
production fabric is necessary, it is prudent to separate these ports into a logical replication
fabric using Virtual Fabrics.

An architecture best practice uses a separate autonomous replication fabric and does not
combine the replication fabric with the production fabric.

Two-box architecture
A single Extension platform can be directly connected to different storage controllers and is
referred to as a Two-Box solution, one at each DC, as shown in Figure 7-6.

A IP WAN
Extension Tunnel A
Fibre Channel Extension Tunnel
Fibre Channel
Circuit
B Circuit
Brocade Brocade B
DC LAN DC LAN
Storage Extension Extension
WAN Router WAN Router Storage

Production Site Replication Backup Site

Figure 7-6 FCIP two-box architecture

A two-box solution would be common with the IBM SAN18B-6 base unit, which supports one
circuit per VE_Port. It does support two VE_Ports (tunnels), and those tunnels can run in
parallel. However, this is not the same as two circuits that are part of a single tunnel because
there is no coordination or data recovery between different VE_Ports. Not all storage
applications support parallel FCIP connections. When using parallel Extension tunnels with
EBR (Exchange-Based Routing), exchanges may be delivered out of order. Device-Based
Routing or Port-Based Routing (flow-based) could be used to avoid issues with out of order
Exchanges.

Four-box architecture
Alternatively, an Extension platform is dedicated to fabric and controller “A.” A physically
different Extension platform is dedicated to fabric and controller “B.” Two Extension platforms
in each data center is referred to as a Four-Box solution; refer to Figure 7-7 on page 143
below. A single WAN link for both tunnels, all circuits, may be used. Alternatively, two WAN
links from different service providers may be used, with the circuits symmetrically spread
across the WAN connections. The number of WAN connections depends on the availability
requirements, the cost, and recovery tolerance. With two WAN connections, Cir0 from each
tunnel would be pinned to WAN-A, and Cir1 from each tunnel would be pinned to WAN-B.

142 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

BET BET
A A

IBM IBM
Extension Circuits Circuits Extension

B B

IP WAN IP WAN
Router Router

Production Replication Backup


Site Site

Figure 7-7 FCIP four -box architecture

Extension + FCR architecture


A production fabric can be extended using Extension with FC Routing (FCR), as shown in
Figure 7-8, although not recommended unless there is a compelling reason. The most
common reason is distributed systems tape to connect too many end devices then pipelining
the tape traffic back to a DR data center.

FC FC
BET BET

Fabric A Fabric A

Circuits Circuits
Fabric B Fabric B

FC
FC

IBM IP WAN IP WAN IBM


Edge Core Extension Router Router Extension Core Edge
Production Site Backup Site
Replication

Figure 7-8 FCR+FCIP production edge fabrics extension architecture

Common device - not recommended architecture


In environments requiring Extension attached to a production SAN, it is not best practice to
interconnect the same Extension platform to both the “A” and “B” fabrics. A best practice is to
have two separate and redundant fabrics in a production environment, especially if the
organization could suffer financial losses during a SAN outage. Even a momentary SAN
outage can “blue screen” or hang servers, forcing a reboot and database reconciliation, which
in most situations takes significant time.

For maximum availability, it is best practice to divide a SAN into “A” and “B” fabrics, which
implies there is an “air gap” between the two autonomous fabrics from the server to the
storage. Intentionally, there are no physical links between the two fabrics. Servers, storage,
and VMs are equipped with drivers that monitor pathways and send data accordingly. When a
path is detected as down, the driver fails traffic over to a remaining path.

Connecting Extension via FCR to both production edge fabrics, as shown in Figure 7-9 on
page 144, is not recommended and IBM does not recommend this architecture. Without FCR,
the fabrics would merge into one big fabric, which destroys any notion of redundant
autonomous fabrics. Nonetheless, if FCR is used, the fabrics don’t merge; however, there still
is a common Linux kernel on a device attached to both fabrics. If maximum availability is the
goal in a critical environment, this architecture is not acceptable and considered high risk. An
architecture with a common platform to both fabrics is susceptible to an operating system

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 143
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

defect, driver defects, unexpected hardware failures, environmental conditions, and human
error, which can bring down the entire SAN.

Circuits (BET) Circuits (BET)


Brocade
Brocade
Extension
Extension

Edge WAN Router WAN Router


Core Edge
Core

Not Recommended! – Poor Practice

Figure 7-9 Common extension to production edge fabrics - not recommended architecture

When connecting Extension to production fabrics, each production fabric should be designed
using best practice core-edge concepts. Since storage attaches directly to the core in a
core-edge design, Extension switches connect directly to the core, or an Extension blade is
placed in the Core Directors. A standalone Extension platform should be connected to a
fabric with at least two ISLs (Inter-Switch Links) for the port, optic, and cable redundancy.

In a four-box architecture, it is inappropriate to make ISL cross-connections between the two


Extension platforms and the “A” and “B” fabrics because of the same reasons discussed
above, a common Linux kernel, defects, failures, human error….

Cross-connecting circuits from a tunnel to Ethernet DC LAN switches or network devices is


encouraged. Circuits traversing the IP network are point-to-point and can take alternate
resilient and redundant paths without merging fabrics.

7.2.2 IP Extension design and architectures


Below are various architectures supported with IP Extension. IBM Extension plays a large
role in IBM TS7700 Grid and TCT architectures and is specifically described here. The other
architectures are generic and can be used in mixed or managed environments.

IBM TS7700 grid and TCT


IP Extension is often used in IBM TS7700 Grid environments. This section addresses design
considerations specific to these environments.

A consideration in deploying IP Extension with IBM TS7700 Grid is how to direct grid traffic to
the IP Extension gateway; there are two methods. The first method uses the default gateway
on the EN (Ethernet Network) interfaces to send traffic directly to the IP Extension gateway.
The second method sends traffic to the traditional gateway and routes on the gateway router
forwards traffic to the IP Extension gateway.

Four subnets should be used, one for each EN# interface on the TS7700 system. EN
(Ethernet Network) interfaces of the same number must be able to communicate with each
other. EN interfaces of different numbers do not communicate with each other; therefore,
each subnet below should be in its own VLAN and have its own IP Extension gateway.
Example configuration on EN# interfaces:
򐂰 EN0
– IP addresses:10.10.0.100-199
– Gateway (IPEX):10.10.0.254
– Subnet Mask:255.255.255.0 (/24)
򐂰 EN1
– IP addresses:10.10.1.100-199

144 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

– Gateway (IPEX):10.10.1.254
– Subnet Mask:255.255.255.0 (/24)

Example subnets and IPEX_GWs:


򐂰 Subnet used for all EN0 in the data center: 10.10.0.0/24 IPEX-GW 10.10.0.254 VLAN 100
򐂰 Subnet used for all EN1 in the data center: 10.10.1.0/24 IPEX-GW 10.10.1.254 VLAN 101
򐂰 Subnet used for all EN2 in the data center: 10.10.2.0/24 IPEX-GW 10.10.2.254 VLAN 102
򐂰 Subnet used for all EN3 in the data center: 10.10.3.0/24 IPEX-GW 10.10.3.254 VLAN 103

Depending on the architecture being built, IP Extension gateways can be consolidated or


distributed:
򐂰 All IP Extension gateways on the same DP: highest risk
򐂰 IP Extension gateways divided across two DPs on the same blade: high risk
򐂰 IP Extension gateways divided across two DPs on different blades - same chassis:
medium risk
򐂰 IP Extension gateways divided across four DPs all different blades - two chassis: medium
risk
򐂰 IP Extension gateways divided across four DPs all different blades - four chassis: lowest
risk

The second DP may be kept in reserve for eHCL moving circuits into or consumed by different
applications such as RDR.

It is imperative to understand TCL rules only live on the DP unto which the target VE_Port
exists. VE_Ports exist only in one DP, either DP0 or DP1. Refer to the TCL section for IP
Extension gateways are created on a specific DP, either DP0 or DP1. Ingress traffic matching
a TCL rule is directed to a particular target (VE_Port). If the VE_Port target designated in a
TCL rule is not congruent with the IP Extension gateway’s DP, the rule is not evaluated. In
other words, both the IP Extension gateway and target VE_Port used must be on the same
DP for the rule to be evaluated. If the rule is expected to be assessed but is not, check if traffic
is hitting an IP Extension gateway configured on the other DP.

Method 1
The IBM TS7700 Grid EN (Ethernet Network) interfaces are configured with their default
gateway set to the IP Extension gateway, as shown in Figure 7-10 on page 146. The IP
Extension gateway becomes the default gateway for the EN interfaces. IBM TS7700 Grid EN
interfaces do not support independent routing tables; each interface cannot specify an IP
route for destination subnet specific gateways. The IP Extension gateway is required for
transporting grid traffic to the remote data center via IP Extension. The remote grid subnet is
the only destination outside of the local grid subnet. In Method 1, the EN interfaces cannot be
configured with the traditional router gateway.

A LAN switch that switches traffic without a gateway router requires EN interfaces to be on the
same subnet and VLAN; this is Layer-2 switching (L2 switching). The Extension platforms are
not Ethernet switches and cannot facilitate communication between different clusters; the
data center LAN switch must do this. Communications to other grid nodes or clusters must be
done at layer 2 (the Ethernet layer) through their common subnet, thereby not involving a
gateway router. Traffic on the same subnet is sent directly to the other device via Ethernet.

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 145
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

Figure 7-10 Remote grid connectivity via IP extension gateway

As long as all the local grid EN<#> interfaces live on the same subnet and VLAN, they can
communicate via L2 switching, as shown in Figure 7-11. Grid traffic destined for a remote
data center via IP Extension is sent to the IP Extension gateway using the gateway configured
on the EN<#> interfaces.

Figure 7-11 IBM TS7700 Grid IP Extension Architecture - EN Interfaces with IP Extension Gateway

Method 2:
In method 2, the IBM TS7700 EN interfaces are configured to use the traditional router as the
default gateway. Grid traffic destined for various subnets is sent to a conventional gateway
router. The router forwards the traffic towards the destination, which may be another
node/cluster within the local data center or a remote cluster via the IP Extension gateway, as
shown in Figure 7-12 on page 147.

146 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

DC LAN Switch
10.10.10.0/24
GW: 10.10.10.1
Grid
Cluster A

.1 Router
IPEX Destination
Traditional Gateway IP Extension
Tunnel WAN 172.16.10.0/24
IP route 172.16.10.0/24 via 192.168.10.2 IP Extension GW subnet .2
.1 .1
(192.168.10.0/30) iproute lan.dp# 10.10.10.0/24 192.168.10.1
iproute lan.dp# 10.10.20.0/24 192.168.10.1

Grid
Cluster B 10.10.20.0/24
GW: 10.10.20.1
DC LAN Switch

Figure 7-12 Local and remote grid connectivity via a traditional gateway

Method 2 enables communications between multiple clusters using different subnets;


however, it does require the IP network to route traffic originating from the local grid subnet
and destine to the remote grid subnet via the IP Extension gateway. The traditional router can
be configured to route to the IP Extension gateway in various ways, such as connected
interface routing protocol participation, static routes, or PBR (Policy Based Routing). IBM
b-type Gen 7 Extension platforms are not IP routers and do not support routing protocols.
Different grid subnets between nodes and clusters use the default gateway designated on the
local data center's EN interfaces.

IBM TS7700 grid three DC architecture


The three data center IBM TS7700 Grid architecture spans three data centers, and IP
Extension is used between two metro data centers and the remote DR site. This three data
center Grid architecture uses the SAN42B-R and/or SX6 Blades. Both DPs are used on each
Extension platform (DP0 and DP1), though not necessarily both for each EN interface. Each
DP has a capacity of 20 Gbps when licensed. The DR data center has a pair of Extension
platforms for each incoming site. If Sx6 Extension Blades are being used, this requires a
second blade in each chassis. If the SAN42B-R is being used, this requires an additional box.
The diagram and Extension icons are generic; there are either two Directors with two SX6
blades in each or four SAN42B-R boxes.

IBM TS7700 Grid traffic is already compressed; therefore, compression is disabled. With no
compression, the ingress and egress data rates are equal at 20 Gbps for IP Extension.
Encryption is enabled and causes no degradation in throughput or performance.

A different subnet is used for each EN (Ethernet Network) interface. Each cluster has four EN
interfaces (EN0, EN1, EN2, and EN3). DC-A and DC-B are metro data centers connected by
DWDM at L2, the Ethernet layer. The respective EN interfaces from clusters (4, 6, 5, and 7)
are on the same subnets across these two data centers. All EN0 interfaces can communicate
without routing (L3), same with EN1, EN2, and EN3. In DC-A and DC-B, all like numbered EN
interfaces can talk to each other without Extension. DC-C is further away and requires a
routed L3 IP network and Extension.

Figure 7-13 on page 148 shows the schema for addressing the EN interfaces, creating
gateways, and connecting the various DPs. Keep in mind the gateways and VE_Ports are
associated with one specific DP. EN interface IP addresses have been selected to readily
designate the data center (3rd byte), EN interface (3rd byte), and cluster (4th byte) within and
across desired broadcast (L2) domains. The subnets are /24 (255.255.255.0). The layout
depends on the number of DPs in each data center and how best to disperse EN interfaces
across the DPs to gain bandwidth and high availability.

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 147
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

On each switch in DC-C (sw0, sw1, sw2, and sw3), there are two IP Extension gateways on
each subnet, one for each DP (.250 and .251). There are four IP Extension gateways on each
subnet in DC-A and DC-B, one for each DP in each data center (.250, .251, .252, and .253).
You do not want to traverse the DWDM network to reach an IP Extension gateway. In DC-A
and DC-B, EN0 and EN2 use DP0, and EN1 and EN3 use DP1. In DC-A and DC-B, EN0 and
EN1 use different switches, as well; EN2 and EN3 use separate switches, for example,
DCA-sw0 and DCA-sw1. IN DC-C, because there are four platforms, each EN interface uses
a different switch. In DC-C, cluster 1 uses DP0, and cluster 3 uses DP1.

Min-Rate Max-Rate
CL/EN Source IP Destination IP IPEX GW DP Target SW SW Target DP IPEX GW Source IP Destination IP CL/EN # of Cirs Each (Gbps) Each (Gbps)
CL4 CL1
0 10.0.0.40 10.0.10.10 10.0.0.250/24-dp0 DP0 12/16 DCA-s w0 DCC-s w0 12/16 DP0 10.0.10.250/24-dp0 10.0.10.10 10.0.0.40 0 2 5 5
1 10.0.1.40 10.0.11.10 10.0.1.250/24-dp0 DP0 12/16 DCA-s w1 DCC-s w1 12/16 DP0 10.0.11.250/24-dp0 10.0.11.10 10.0.1.40 1 2 5 5
2 10.0.2.40 10.0.12.10 10.0.2.251/24-dp1 DP1 12/26 DCA-s w0 DCC-s w2 12/16 DP0 10.0.12.250/24-dp0 10.0.12.10 10.0.2.40 2 2 5 5
3 10.0.3.40 10.0.13.10 10.0.3.251/24-dp1 DP1 12/26 DCA-s w1 DCC-s w3 12/16 DP0 10.0.13.250/24-dp0 10.0.13.10 10.0.3.40 3 2 5 5
CL5 CL1
0 10.0.0.50 10.0.10.10 10.0.0.252/24-dp0 DP0 12/16 DCB-s w0 DCC-s w0 12/17 DP0 10.0.10.250/24-dp0 10.0.10.10 10.0.0.50 0 2 5 5
1 10.0.1.50 10.0.11.10 10.0.1.252/24-dp0 DP0 12/16 DCB-s w1 DCC-s w1 12/17 DP0 10.0.11.250/24-dp0 10.0.11.10 10.0.1.50 1 2 5 5
2 10.0.2.50 10.0.12.10 10.0.2.253/24-dp1 DP1 12/26 DCB-s w0 DCC-s w2 12/17 DP0 10.0.12.250/24-dp0 10.0.12.10 10.0.2.50 2 2 5 5
3 10.0.3.50 10.0.13.10 10.0.3.253/24-dp1 DP1 12/26 DCB-s w1 DCC-s w3 12/17 DP0 10.0.13.250/24-dp0 10.0.13.10 10.0.3.50 3 2 5 5
CL6 CL1
0 10.0.0.60 10.0.10.10 10.0.0.250/24-dp0 DP0 12/16 DCA-s w0 DCC-s w0 12/16 DP0 10.0.10.250/24-dp0 10.0.10.10 10.0.0.60 0 2 5 5
1 10.0.1.60 10.0.11.10 10.0.1.250/24-dp0 DP0 12/16 DCA-s w1 DCC-s w1 12/16 DP0 10.0.11.250/24-dp0 10.0.11.10 10.0.1.60 1 2 5 5
2 10.0.2.60 10.0.12.10 10.0.2.251/24-dp1 DP1 12/26 DCA-s w0 DCC-s w2 12/16 DP0 10.0.12.250/24-dp0 10.0.12.10 10.0.2.60 2 2 5 5
3 10.0.3.60 10.0.13.10 10.0.3.251/24-dp1 DP1 12/26 DCA-s w1 DCC-s w3 12/16 DP0 10.0.13.250/24-dp0 10.0.13.10 10.0.3.60 3 2 5 5
CL7 CL1
0 10.0.0.70 10.0.10.10 10.0.0.252/24-dp0 DP0 12/16 DCB-s w0 DCC-s w0 12/17 DP0 10.0.10.250/24-dp0 10.0.10.10 10.0.0.70 0 2 5 5
1 10.0.1.70 10.0.11.10 10.0.1.252/24-dp0 DP0 12/16 DCB-s w1 DCC-s w1 12/17 DP0 10.0.11.250/24-dp0 10.0.11.10 10.0.1.70 1 2 5 5
2 10.0.2.70 10.0.12.10 10.0.2.253/24-dp1 DP1 12/26 DCB-s w0 DCC-s w2 12/17 DP0 10.0.12.250/24-dp0 10.0.12.10 10.0.2.70 2 2 5 5
3 10.0.3.70 10.0.13.10 10.0.3.253/24-dp1 DP1 12/26 DCB-s w1 DCC-s w3 12/17 DP0 10.0.13.250/24-dp0 10.0.13.10 10.0.3.70 3 2 5 5
CL4 CL3
0 10.0.0.40 10.0.10.30 10.0.0.250/24-dp0 DP0 12/17 DCA-s w0 DCC-s w0 12/26 DP1 10.0.10.251/24-dp1 10.0.10.30 10.0.0.40 0 2 5 5
1 10.0.1.40 10.0.11.30 10.0.1.250/24-dp0 DP0 12/17 DCA-s w1 DCC-s w1 12/26 DP1 10.0.11.251/24-dp1 10.0.11.30 10.0.1.40 1 2 5 5
2 10.0.2.40 10.0.12.30 10.0.2.251/24-dp1 DP1 12/27 DCA-s w0 DCC-s w2 12/26 DP1 10.0.12.251/24-dp1 10.0.12.30 10.0.2.40 2 2 5 5
3 10.0.3.40 10.0.13.30 10.0.3.251/24-dp1 DP1 12/27 DCA-s w1 DCC-s w3 12/26 DP1 10.0.13.251/24-dp1 10.0.13.30 10.0.3.40 3 2 5 5
CL5 CL3
0 10.0.0.50 10.0.10.30 10.0.0.252/24-dp0 DP0 12/17 DCB-s w0 DCC-s w0 12/27 DP1 10.0.10.251/24-dp1 10.0.10.30 10.0.0.50 0 2 5 5
1 10.0.1.50 10.0.11.30 10.0.1.252/24-dp0 DP0 12/17 DCB-s w1 DCC-s w1 12/27 DP1 10.0.11.251/24-dp1 10.0.11.30 10.0.1.50 1 2 5 5
2 10.0.2.50 10.0.12.30 10.0.2.253/24-dp1 DP1 12/27 DCB-s w0 DCC-s w2 12/27 DP1 10.0.12.251/24-dp1 10.0.12.30 10.0.2.50 2 2 5 5
3 10.0.3.50 10.0.13.30 10.0.3.253/24-dp1 DP1 12/27 DCB-s w1 DCC-s w3 12/27 DP1 10.0.13.251/24-dp1 10.0.13.30 10.0.3.50 3 2 5 5
CL6 CL3
0 10.0.0.60 10.0.10.30 10.0.0.250/24-dp0 DP0 12/17 DCA-s w0 DCC-s w0 12/26 DP1 10.0.10.251/24-dp1 10.0.10.30 10.0.0.60 0 2 5 5
1 10.0.1.60 10.0.11.30 10.0.1.250/24-dp0 DP0 12/17 DCA-s w1 DCC-s w1 12/26 DP1 10.0.11.251/24-dp1 10.0.11.30 10.0.1.60 1 2 5 5
2 10.0.2.60 10.0.12.30 10.0.2.251/24-dp1 DP1 12/27 DCA-s w0 DCC-s w2 12/26 DP1 10.0.12.251/24-dp1 10.0.12.30 10.0.2.60 2 2 5 5
3 10.0.3.60 10.0.13.30 10.0.3.251/24-dp1 DP1 12/27 DCA-s w1 DCC-s w3 12/26 DP1 10.0.13.251/24-dp1 10.0.13.30 10.0.3.60 3 2 5 5
CL7 CL3
0 10.0.0.70 10.0.10.30 10.0.0.252/24-dp0 DP0 12/17 DCB-s w0 DCC-s w0 12/27 DP1 10.0.10.251/24-dp1 10.0.10.30 10.0.0.70 0 2 5 5
1 10.0.1.70 10.0.11.30 10.0.1.252/24-dp0 DP0 12/17 DCB-s w1 DCC-s w1 12/27 DP1 10.0.11.251/24-dp1 10.0.11.30 10.0.1.70 1 2 5 5
2 10.0.2.70 10.0.12.30 10.0.2.253/24-dp1 DP1 12/27 DCB-s w0 DCC-s w2 12/27 DP1 10.0.12.251/24-dp1 10.0.12.30 10.0.2.70 2 2 5 5
3 10.0.3.70 10.0.13.30 10.0.3.253/24-dp1 DP1 12/27 DCB-s w1 DCC-s w3 12/27 DP1 10.0.13.251/24-dp1 10.0.13.30 10.0.3.70 3 2 5 5

Figure 7-13 Cluster IP addresses, IP Extension Gateways, Tunnels, and DP Assignments

There are two tunnels per DP. Each tunnel has two circuits, one circuit for each data center
LAN switch. If one of the data center LAN switches goes offline, the remaining circuit
continues to operate without disrupting the Grid. As you can see, there are four circuits per
DP. The maximum capacity of a DP is about 20 Gbps; therefore, each circuit should be
configured at 5 Gbps, which is 10 Gbps per tunnel. Both the --min-comm-rate and the
--max-comm-rate should be set equal at 5 Gbps, called CIR (committed information rate). When
min and max are different, it is referred to as ARL. In a Grid environment with well-balanced
traffic across all paths, disabling ARL and using CIR can improve performance. Moreover, if
metric 1 backup circuits and failover groups are being used, the best practice is CIR over
ARL.

Figure 7-14 on page 149 shows a high-level connectivity diagram with ports.

148 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

Figure 7-14 TS7700 Grid Three DC Architecture

Consider tunnels as connecting a local IP Extension gateway to a remote IP Extension


gateway. IP Extension gateways belong to a specific DP, either DP0 or DP1. Think of the local
and remote IP Extension gateways as endpoints of a tunnel. Considering the TCL directs
traffic into a specific tunnel, an IP Extension gateway can be the endpoint of more than one
tunnel within the same DP. Traffic must traverse the same path outbound and return;
therefore, the same gateway pair must be deterministic. The TCL ensures a deterministic
path (gateways and tunnel) is used when outbound and return traffic consistently arrives at
the lan.dpx ipif.

It helps make a diagram of how the tunnels connect from end to end - IP Extension gateway
to IP Extension gateway; see Figure 7-15. The Extension tunnels between DC-A and DC-B to
DC-C complete a full mesh architecture in which every EN interface of the same number can
communicate. Remember, Clusters 4, 6, 5, and 7 can communicate through Layer2 since
their EN interfaces are on the same subnet and connected by DWDM. The figure below
shows tunnel connectivity. Notice the VE_Port range for DP0 and DP1 are different. On the
SX6 Blade, DP0 is VE16 to VE20, and DP1 is VE26 to VE30.

Figure 7-15 S7700 grid three DC tunnel schema

The same gateway can direct traffic into multiple tunnels via the TCL by matching the
source/destination IP addresses. For example, on the Extension platform connected to
Cluster1-EN0, if the rule matches src:10.0.10.10 and dst:10.0.0.40 and designates target 16,
that specific traffic uses tunnel VE16 to Cluster4-EN0. A mirrored TCL must be configured on
the Extension platform connected to Cluster4 to return traffic.

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 149
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

Multiple gateways can direct traffic into the same tunnel via the TCL. For example, an
Extension platform has the gateway 10.0.0.250 for EN0 and directs traffic into VE16. A
different gateway for EN2, 10.0.2.250, can direct traffic into the same tunnel, VE16. If
Upgrade1 + Upgrade2 licenses have been installed, it is possible to create parallel tunnels for
each EN interface; however, this assumes enough VE_Ports are available. The DP maximum
capacity is no more than 20 Gbps on the WAN side, no matter how much circuit bandwidth is
configured. With WAN Upgrade 1+2 licenses, there is no bandwidth restriction on creating
circuits. The best practice is to use the same tunnels for various gateways when a tunnel is
between the same domains and DPs.

DC-C dedicates a DP to each EN interface. DC-C has four switches (sw0, sw1, sw2, sw3)
because it consolidates incoming connections from two remote data centers. In DC-A and
DC-B, a DP is dedicated to EN0 and EN2, and a DP is dedicated to EN1 and EN3. There is a
balance between the two metro sites going into the one DR site. Adding SX6 Blades or
SAN42B-R boxes increases the number of DPs available to EN interfaces, for example, if
more bandwidth was needed. One DP can accommodate 20 Gbps, which is twice the
bandwidth of a single EN interface. Of course, more than one DP per EN interface would not
be beneficial.

Two IBM SAN18B-6 (base unit) architecture


The IBM SAN18B-6 base unit supports one circuit per VE_Port (tunnel). The base unit does
not support BET. Two VE_Ports are supported; therefore, two parallel tunnels between the
same domains is an option. Two parallel tunnels are different than one tunnel with two
circuits. Two autonomous tunnels have no coordination, bandwidth management between
them.

The TCL cannot match IP Extension traffic and direct it into more than one target (tunnel). If
two VE_Ports were used, the traffic across each would require a rule match from different
TCL rules.

As shown in Figure 7-16, two IP Extension fabrics are shown. Each fabric has one tunnel with
one circuit. The maximum bandwidth supported by the IBM SAN18B-6 base unit is 1 Gbps on
the WAN side. Two WAN connections are shown; however, a single WAN connection would
work as well. The GE interfaces can be either GE or 10GE. Two different IP Extension
gateways are required, and the end devices must be capable of configuring the different
gateways per interface or per set of interfaces.

LAN Side LAN Side


GE/ 10GE GW: lan.dp0 GW: lan.dp0 GE/ 10GE

cir0
IP Storage

IP Storage

DC LAN IP WAN WAN IP DC LAN


WAN
Switch Extension Router Router Extension Switch

cir0

GW: lan.dp0 GW: lan.dp0

Figure 7-16 IBM SAN18B-6 base unit architecture

Two IBM SAN18B-6 (full unit) architecture


The IBM SAN18B-6 full unit supports four VE_Ports, 2.5 Gbps maximum WAN side
bandwidth, and BET. In Figure 7-17 on page 151 below, two circuits are shown (red and blue).
Each circuit takes a different path through the IP network. Each circuit has its settings for
bandwidth, failover, QoS, and KATOV.

150 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

cir0 cir0

IP Storage

IP Storage
LAN Side LAN Side
GE/ 10GE GW: lan.dp0 GE/ 10GE
IP WAN IP WAN
DC LAN IBM Service IBM DC LAN
Switch IP Extension Provider(s) IP Extension Switch

cir1 cir1

WAN WAN
Router Router

Figure 7-17 Two IBM18B-6 full unit architecture

Two IBM IP extension with two non-virtualized DC LAN switches


Not all data center LAN switches are configured to form a single logical switch. The
architecture in Figure 7-18 shows IP storage that is dual connected to each data center LAN
switch; there are two LANs. There is a single IP Extension connection to each LAN
(broadcast domain), which may be a LAG. Each LAN must have its own subnet and IP
Extension gateway.

Two circuits (red and blue) make up a single tunnel between Extension platforms. Each circuit
takes a different path through the IP network.

cir0 cir0
IP Storage

IP Storage
lan.dp0 GW0 lan.dp0 GW0

LAN Side IP WAN IP WAN LAN Side


GE/ 10GE GE/ 10GE
lan.dp0 GW1 Service lan.dp0 GW1
IBM Provider(s) IBM
IP Extension IP Extension
cir1 cir1

DC LAN WAN WAN DC LAN


Switch Router Router Switch

Figure 7-18 Two IP extension platforms with non-virtualized DC LAN switches

Two IBM Extension Platforms with Virtualized DC LAN Switches


The data center LAN switches in this architecture are virtualized, meaning they form a single
logical switch (see Figure 7-19 on page 152). Two switches creating one virtual switch is often
referred to as vPC or VSS in the Cisco nomenclature. There are connections from the IP
Extension platform to both switches because the switches are logically one. It is perceived as
a single Link Aggregated (LAG) connection, indicated by the circles in the diagram. The links
can be tagged or untagged and must be consistent on each end. There is a single IP
Extension gateway for the IP storage VLAN. The end devices can connect to either one of the
data center LAN switches or dual connect to both.

On the WAN side, there are two circuits (red and blue) comprising one tunnel between the
Extension platforms. The circuits take different paths for redundancy.

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 151
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

WAN WAN
Router Router

DC LAN cir0 cir0 DC LAN


Switch Switch

IP Storage
IP Storage
LAN Side lan.dp0 IP WAN IP WAN lan.dp0 LAN Side

vPC

vPC
GE/10GE GW0 GW0 GE/10GE
IBM Service IBM
IP Extension Provider(s) IP Extension

cir1 cir1

WAN WAN
Router Router

Figure 7-19 Two IP Extension Platforms with Virtualized DC LAN Switches

Four IBM Extension Platforms with Non-Virtualized DC LAN Switches


For a higher degree of availability, resiliency, and bandwidth capacity; a four Extension
platform architecture could be deployed (see Figure 7-20). The end devices are dual
connected to separate LAN switches; each is a LAN (broadcast domain). Each connection
(purple and green) requires its own unique subnet and IP Extension gateway. The end
devices must be capable of independently designating a gateway per interface or per set of
interfaces.

The Extension platforms are LAG connected to a corresponding data center LAN switch. The
links can be tagged or untagged and must be consistent on both ends.

There is a single tunnel between the top two Extension platforms (red tunnel) and one
between the bottom two (blue tunnel). Each tunnel has two circuits (cir0 and cir1), each taking
a different path.

cir0 cir0

lan.dp0 GW0 cir1 cir1 lan.dp0 GW0

IP Storage
IP Storage

10Gb/s
LAN Side IP WAN IP WAN LAN Side
10GE 10GE
GE/ 10GE GE/ 10GE
Service
lan.dp0 GW1 cir1
Provider(s) lan.dp0 GW1
cir1

cir0 cir0
Ethernet IBM WAN WAN IBM Ethernet
Switch IP Extension Router Router IP Extension Switch

Figure 7-20 Four IP extension platforms with non-virtualized DC LAN switches

Four IBM Extension Platforms with Virtualized DC LAN Switches


For a higher degree of availability, resiliency, and bandwidth capacity; a four Extension
platform architecture could be deployed (see Figure 7-21 on page 153). Each end device is
dual connected to a single logically virtualized LAN switch, which has two VLANs (two
broadcast domains). Each connection (purple and green) requires its own unique subnet and
IP Extension gateway. The same IP Extension gateway cannot be put on two Extension
platforms, and VRRP (Virtual Router Redundancy Protocol) is not supported. The end
devices must be capable of independently designating a gateway per interface or per set of
interfaces.

The Extension platforms are LAG connected across the two virtualized data center LAN
switches. The links can be tagged or untagged and must be consistent on both ends.

152 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

There is a single tunnel between the top two Extension platforms (red tunnel) and a tunnel
between the bottom two Extension platforms (blue tunnel). Each tunnel has two circuits (cir0
and cir1), each taking a different path.

10GE 10GE
cir0 cir0
cir1 cir1
lan.dp0 GW0 lan.dp0 GW0
IP Storage

IP Storage
Logically 1 Switch

Logically 1 Switch
LAN Side IP WAN IP WAN LAN Side

vPC

vPC
GE/ 10GE GE/ 10GE
Service
lan.dp0 GW1 lan.dp0 GW1
Provider(s) cir1
cir1
cir0 cir0
10GE 10GE
DC LAN IBM WAN WAN IBM DC LAN
Switch IP Extension Router Router IP Extension Switch

Figure 7-21 Four IP extension platforms with vrtualized DC LAN switches

7.3 Extension implementation


Extension platforms are considered to have three sides: the WAN side, the FC/FICON side,
and the LAN side. This chapter covers details about the various extension platforms, best
practices, features/functions, monitoring, and architectures.

This section covers extension implementation details, which can be segregated into three
sections:
򐂰 The WAN side
򐂰 The FC and FICON side
򐂰 The LAN side

7.3.1 The WAN Side


Consider IBM Extension platforms as having three “sides”: the WAN side, the LAN side, and
the FC/FICON side. The FC/FICON side and LAN side may be used; however, the WAN side
must be used when doing Extension. The WAN side connects to an IP network and forms an
Extension tunnel between data centers. The WAN side has features, functions, and
configurations specific to creating the tunnel, routing traffic, encryption, compression, and
protocol optimization. Additionally, the WAN side has to deal with the IP network and WAN.

IP Network
The IP network must transport Extension datagrams with adequate performance and
reliability to meet the service level goal. The service level desired is entirely dependent on the
policy set forth by the organization. Acceptable performance depends on the amount of data
that must be sent within an interim period, the WAN's capacity to accommodate the data
being sent, and the IP network's ability to deliver the data without loss. Reliability depends on
the stability of the IP network, its overall architecture, redundancy and resiliency, and its ability
to manage and deliver data across the WAN.

IBM does not support IP network devices with oversubscribed or blocking switch ports. Some
Ethernet switches have oversubscribed or blocking host access ports, which may cause
excessive drop packets and degrade storage throughput.

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 153
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

The WAN
Any type and size of WAN connection can be used for Extension as long as it meets the
application and extension platform requirements. The storage application may or may not
have more stringent requirements than the Extension platforms.

Most commonly, Extension is deployed between data centers, separated by a distance of 2


ms RTT or more. The supported propagation delay across an IP network is up to 200 ms RTT,
which in most cases, reaches anywhere in the world.

IBM supports no more than 0.1% packet loss between the Extension Ethernet ports. By
modern standards, 0.1% is considered significant packet loss. Packet loss beyond this level
indicates an evaluation and potential remediation of the IP WAN.

It’s important to understand that the effective bandwidth of a WAN link is highly dependent on
latency and link quality in terms of packet loss. Even a small amount of packet loss with
resulting retransmits significantly degrades overall link effective bandwidth.

All IBM Extension platforms have a minimum bandwidth of 20 Mbps per circuit.

The IBM SAN18B-6 can have a maximum circuit size of 2.5 Gbps if licensed as a full unit;
otherwise, 1 Gbps as a base unit. VE_Ports can accommodate up to 2.5 Gbps of cumulative
circuits as a full unit; otherwise, 1 Gbps as a base unit. On the full unit, a tunnel can be
comprised of multiple circuits of varying sizes up to 6 circuits; otherwise, only a single circuit
per VE_Port as a base unit.

The IBM SAN42B-R has a maximum circuit bandwidth of 10 Gbps if the Upgrade 1 license is
applied; otherwise, 5 Gbps as a base unit. VE_Ports can accommodate up to 20 Gbps of
cumulative circuits if the Upgrade1 + Upgrade2 licenses are applied; otherwise, 10 Gbps for
Upgrade1 and 5 Gbps as a base unit. A tunnel is comprised of multiple circuits of varying
sizes, up to 10 circuits.

IBM SX6 Extension Blade has a maximum circuit bandwidth of 10 Gbps. VE_Ports can
accommodate up to 20 Gbps of cumulative circuits. Upgrade1 and Upgrade2 licenses do not
apply to this blade. A tunnel can be comprised of multiple circuits of varying sizes, up to 10
circuits.

A best practice is to connect Extension directly to the WAN routers or switches, avoiding
intermediate devices such as firewalls. If this is not possible, every effort should be made to
connect the Extension Ethernet ports as close to the WAN as possible to prevent upstream
oversubscription and performance degradation. Extension traffic drives large amounts of
data, and uplink oversubscription often bodes poorly.

Consider the following steps when you are planning for your Extension links:
򐂰 For redundancy, use at least two WAN links between sites; however, in critical
environments deploying two or more replication fabrics, consider a WAN connection for
each fabric. Generally, WAN connections are the weakest and least reliable part of
replication fabrics.
򐂰 Consider dedicated WAN connections for storage interconnections. Separate storage and
non-storage traffic to better manage bandwidth, latency, and prevent congestion.
򐂰 Ensure that you have a service level agreement (SLA) with the WAN service provider that
meets the enterprise's needs and expectations.
򐂰 If not using Global Mirror with Change Volumes (GMCV), make sure to size the WAN
connection to sustain peak workloads. For replication over long distances, the best
practice is to use GMCV.

154 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

Bandwidth
For critical RDR (Remote Data Replication), the best practice is to use a separate and
dedicated IP connection between the production data center and the backup site.
Nonetheless, often a dedicated IP connection between data centers is not practical. In this
case, bandwidth must be, at a minimum, logically dedicated to Extension, which can be done
by:
1. Use QoS giving Extension a higher priority, which logically gives shared bandwidth to
Extension over other competing traffic.
2. Use Committed Access Rate (CAR) to identify and rate-limit certain traffic types. Use CAR
on non-Extension traffic to apportion and limit that traffic to a maximum bandwidth, leaving
the remaining bandwidth to Extension. Set the aggregate of the circuit min-comm-rates to
use the remaining dedicated bandwidth. Restricting non-Extension traffic results in
logically allocating bandwidth for Extension.
3. Massive overprovisioning to provide bandwidth for all traffic to coexist without congestion.
This method is the easiest, most costly, and most common practice seen in Extension
deployments.

Brocade Extension uses an aggressive TCP stack called WAN Optimized TCP (WO-TCP),
which dominates other TCP flows on the path causing them to dramatically back-off. UDP
based flows may result in considerable congestion and excessive packet drops for all traffic.

The best practice is to rate-limit Extension traffic on the platforms at the source and
destination and not rate limit Extension in the IP network. Rate limiting Extension in the IP
network often leads to performance problems and difficult troubleshooting issues. IBM
Extension rate limiting is advanced, accurate, and consistent; there is no need to double
rate-limit.

To determine the amount of network bandwidth needed, record the number of bytes written
over a month (or more). A granular recording with the ability to go back and calculate rolling
averages of varying lengths is most helpful. It is essential to understand the number of bytes
written to volumes during interims of the day and night. Write profiles show the quantities of
data needing to be replicated to maintain RPO (Recovery Point Objective). Expediting data
transmission is essential.

If replicating asynchronous RDR (RDR/A) across a tunnel, calculate the average value over a
finite number of minutes throughout the day and night to determine the maximum average.
Keep in mind, business transactions may increase, and replication may require significantly
more bandwidth during quarter-end, fiscal-end, and certain holidays. RDR/A needs only
enough bandwidth to accommodate the high averages discovered over a finite period. RDR/A
essentially performs traffic shaping, which moves peaks into troughs—averaging over
interims that are too long causes a backlog of writes when the troughs do not adequately
relieve the peaks. A replication backlog, which is data journaling, may result in extended
periods of recovery.

For RDR/S (synchronous replication), the best practice is to dedicate the network without
sharing bandwidth. A 10 Gbps DWDM lambda is an example of dedicated bandwidth often
used with synchronous replication. Dedicated bandwidth eliminates packet drops and timely
TCP retransmits. If planning to replicate RDR/S across an Extension tunnel, record the peak
traffic rates. RDR/S must have enough bandwidth to send writes immediately,
accommodating the entire demand at any time.

Plot the recorded values into a histogram. Suppose you have calculated 5-minute rolling
averages from your recorded data. The number of bytes written in each period is an integer
value. The x-axis contains the number of bytes written during each 5-minute average. The

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 155
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

x-axis starts at zero bytes on the left and goes up to the largest number of bytes written
during an interim—many interims have the exact number of bytes forming a Gaussian curve.

The y-axis is the number of times a particular interim occurred having the exact value. The
smallest size occurred relatively rarely, and the largest size occurred relatively infrequently.
68% of the averages fell into the middle section, as shown in Figure 7-22, the largest number
of occurrences.

Figure 7-22 Gaussian standard normal distribution

Based on cost, plan your WAN bandwidth to accommodate at least the first standard
deviation (68% µ+?), and if possible, include the second standard deviation (95% µ+2?). In
most cases, beyond the second deviation, occurrences are rarer and potentially disregarded
in bandwidth planning.

You can plan for a certain amount of compression, such as 2:1 if your data is compressible;
however, data compressibility is a moving target. The best practice is to use compression as a
safety margin to address unusual and unforeseen situations. As well, achieving some amount
of compression can provide headroom for potential future growth. From the start, pushing the
limit may not be fortuitous, mainly if no margin accommodates periodic demand increases.

For remote tape, Extension connects VTLs located in various data centers. It is vital to
measure tape volume in MB/h and the number of simultaneous drives in use. If an adequate
number of drives are available, bandwidth may become the limiting factor for backup
windows.

GE Interfaces
The IBM Extension platforms have various gigabit Ethernet (GE) interface connectivity to the
IP network for transporting application data to/from the LAN and WAN sides. GE interfaces
are used by application data to communicate either to the LAN side or from the WAN side to
the long-distance IP network.

All Extension Ethernet interfaces are referred to as GE (gigabit Ethernet) interfaces.

WAN-side Ethernet interfaces do not support LAG; instead, Brocade Extension Trunking
(BET) is used. BET accomplishes the same as LAG, plus it has the benefit of not requiring the
destination switches to be virtually connected, lossless link loss, and guaranteed in-order
delivery.

IBM Extension GE interfaces cannot enable loops within a data center LAN. Extension GE
interfaces are akin to server NIC interfaces; neither are they Ethernet switch ports, nor are
they router ports. GE interfaces do not participate in routing protocols and do not run
spanning tree protocol because there is no need. Ethernet frames are not permitted to
“hairpin,” meaning to come back out of an incoming interface to prevent loops. Ingress traffic

156 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

cannot go out of any other GE interface of the same type; it must go to a different side of the
Extension platform, or it is dropped.

Table 7-1 on page 158 below shows the various Ethernet interfaces and VE_Ports available
on IBM b-type Gen 7 Extension platforms.

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 157
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

Table 7-1 Ethernet interfaces on IBM b-type Gen 7 extension platforms

Brocade 7810 Brocade 7840 SX6 Extension Blade

Data Processor (DP) DP0 DP0 DP1 DP0 DP1

GE 6 GE/10GE 16 GE/10GE shared by both 16 GE/10GE shared by both


(2 can be GE RJ-45) DPs DPs
10GE

40GE n/a 2 shared by both DPs 2 shared by both DPs

Max VE Bandwidth 2.5 Gbps 20 Gbps 20 Gbps 20 Gbps 20 Gbps

Number of VEs

Platform max WAN 2.5 Gbps 40 Gbps 40 Gbps


side Bandwidth

Note: ll Extension platform Ethernet interfaces are referred to as GE (gigabit Ethernet),


regardless of their type and speed. The management Ethernet interface is not referred to
as a GE port.

All commands have --help


help portcfgge or portcfgge --help

Set GE Interface Mode


On the IBM SAN18B-6 Extension Switch, there are two RJ-45 copper interfaces (GE0 &
GE1). They are 1 Gbps only. These interfaces and the first two optical bays (GE2 & GE3) are
mutually exclusive. They enable/disable as a set. Either GE0 & GE1 or GE2 & GE3 are
enabled, but not both. No platform reboot is required to make the change; however, no
configuration can be tied to the ports. Refer to the CLI configuration below:

The IBM SAN18B-6 enables either the copper (GE0 & GE1) or the optical (GE2 & GE3).
extncfg --help
extncfg --ge-mode optical

Set GE Interface Type


When deploying IP Extension, at least one GE interface must be configured in LAN mode,
preferably more than one for redundancy. The LAN side GE interfaces are associated with IP
storage communicating to the IP Extension gateway. The IP Extension gateway is not
assigned to specific LAN GE interfaces; instead, the IP Extension gateway is accessed from
any LAN interface. If planning to use a PortChannel (LAG), multiple LAN interfaces are
required.

There are two types of GE interfaces: WAN and LAN. The default type is WAN, shown as
“FCIP” in switchshow.

Setting a GE interface to be of type LAN:


7840-Side-A:FID128:admin> portcfgge ge6 --set -lan

Showing the configuration of the GE interfaces.


Local7810:admin> portcfgge --show
Port Speed Flags Channel Lag Name
---------------------------------------------------------------------------
ge0 1G AC-- N/A -

158 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

ge1 1G AC-- N/A -


ge2 10G --LG N/A MYLAG1
ge3 10G --LG N/A MYLAG1
ge4 10G ---- N/A -
ge5 10G ---- N/A -
ge6 10G ---- N/A -
ge7 10G ---- N/A -
---------------------------------------------------------------------------

Flags: A:Auto-Negotiation Enabled C:Copper Media Type


L:LAN Port G: LAG Member

Set a GE interface to LAN


portcfgge ge2 --set -lan

Set GE Interface Speed


Configure the speed to match both the data center LAN switch ports and the SFP (1 Gbps) or
SFP+ (10 Gbps) populated in the Extension platform. SFP and SFP+ optics are not capable
of changing speed. There is no speed auto-negotiation in GE/10GE ports; this is a manual
configuration setting.

Set the GE interface speed.


portcfgge ge2 --set -speed 10G

Showing the GE interface types:


7840-Side-A:FID128:admin> portcfgge --show
Port Speed Flags Channel Lag Name
---------------------------------------------------------------------------
ge0 40G ---- N/A -
ge1 40G ---- N/A -
ge2 10G ---- N/A -
ge3 10G ---- N/A -
ge4 10G ---- N/A -
ge5 10G ---- N/A -
ge6 10G --L- N/A -
ge7 10G --L- N/A -
ge8 10G --L- N/A -
ge9 10G --L- N/A -
ge10 10G ---- N/A -
ge11 10G ---- N/A -
ge12 10G ---- N/A -
ge13 10G ---- N/A -
ge14 10G ---- N/A -
ge15 10G ---- N/A -
ge16 10G ---- N/A -
ge17 10G ---- N/A -
---------------------------------------------------------------------------
Flags: A:Auto-Negotiation Enabled C:Copper Media Type
L:LAN Port G: LAG Member

switchshow is useful for showing the GE interface settings as well.

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 159
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

On the IBM SAN42B-R and IBM SX6 Extension Blade, there are two 40GE interfaces (GE0 &
GE1). They are 40 Gbps only. The IBM SAN42B-R requires Upgrade1 + Upgrade2 license to
enable the 40GE interfaces. The IBM SX6 Extension Blade does not require an optional
license for the 40GE interfaces; it comes with the 40GE interfaces enabled. The 40GE optics
are an optional purchase.

GE Interface Speeds
Depending on the platform model, various interface speeds and types are available: GE
(RJ-45), GE (SFP), 10GE, and 40GE.

The IBM SAN42B-R requires Upgrade1 + Upgrade2 to enable the 40GE interfaces.

The SX6 Extension Blade includes Upgrade1 and Upgrade2, and the 40GE interfaces are
enabled.

All GE/10GE interfaces are enabled on the base IBM SAN42B-R and SX6 Blade. The
GE/10GE interfaces are in Ethernet switching ASIC port-groups, and within a group, the
speed must be consistent; otherwise, blocking occurs. Below is a list of ports that block each
other when their speed is not set the same. The best practice is first to use ports 2, 3, 4, 5, 10,
11, 12, and 13, then move on doubling up within a port group; refer to Table 7-2 below.

Table 7-2 GE Interface Groups on IBM SAN42B-R and SX6 Blade


Ethernet Interfaces in the same group

2&6

3&7

4&8

5&9

10 & 14

11 & 15

12 & 16

13 & 17

For detailed information, see Brocade Fabric OS Extension User Guide at


https://docs.broadcom.com/doc/FOS-90x-Ext-UG

The IBM SAN18B-6 has two RJ-45 interfaces, which are 1 Gbps only. When they are
enabled, SFP bays GE2 and GE3 are disabled, and vice-versa. SFP bays GE2 and GE3 are
capable of GE/10GE. No reboot of the IBM SAN18B-6 is required to change the active
interfaces; however, no configuration (ipif or iproute) can be attached to the currently active
ports.

GE SFP / 10GE SFP+


GE interfaces do not support auto-negotiate between 1 Gbps and 10 Gbps port speeds. The
speed of a GE interface must is manually configured. Use switchshow to check the current
configuration.

SFP (GE) is not capable of 10 Gbps, and SFP+ (10GE) is not capable of 1 Gbps. You must
have the correct optic for the desired port speed.

160 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

LLDP
A best practice is to use LLDP as a validator of Ethernet connectivity between the Extension
application Ethernet interfaces and the directly connected DC switch. LLDP is beneficial to
both Extension Administrators and Network Administrators. Network Administrators see what
is connected to their switches. Storage Administrators see what they have connected to. At
both ends of an Ethernet link, LLDP logs connectivity information. For troubleshooting, adds,
moves, and changes; LLDP is a handy and convenient tool for referencing Ethernet interfaces
that should be investigated.

Example LLDP CLI output:


lldp --show -nbr
Local7810:admin> lldp --show -nbr
Local-Int DeadInt Remain Remote-Int ChassisID Tx Rx System
ge2 120 106 Ethernet1/16 0062.ecbb.f14f 88928 88922 torsw9
ge3 120 102 Ethernet1/16 0062.ecbb.eea5 88927 100 torsw8
ge6 120 106 Ethernet1/17 0062.ecbb.f150 86675 812 torsw9
ge7 120 102 Ethernet1/17 0062.ecbb.eea5 88901 109 torsw8

LLDP is enabled by default and operates on all application data GE interfaces. It does not
operate on the management interface. LLDP is supported and enabled by default on most
data center LAN switches.

There are certain TLVs (type, length, value) that should be enabled/disabled because they
are used/not used on IBM Extension platforms, as shown in Table 7-3.

Table 7-3 LLDP TLVs


LLDP TLVs

Enable (used) Disable (unused)

chassis ID dcbx

mgmt-addr dot1

port ID dot3

port-desc fcoe-app

sys-capabilities fcoe-lls

sys-desc

sys-name

Example LLDP before (defaults):


7840-Side-A:FID128:admin> lldp --show
LLDP Global Information
system-name: 7840-Side-A
system-description: Brocade 7840, Fabric OS Version 8.2.2
State: Enabled
Mode: Receive/Transmit
Advertise Transmitted: 30 seconds
Hold time for advertise: 120 seconds
Tx Delay Timer: 1 seconds
Transmit TLVs: Chassis ID Port ID
TTL Port Description
System Name System Description
System Capabilities Management Address

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 161
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

IEEE 8021 IEEE 8023


IEEE DCBx DCBx FCoE App
DCBx FCoE Logical Link
DCBx FCoE Priority Values: 3

Example LLDP after:


7840-Side-A:FID128:admin> lldp --disable -tlv dcbx
7840-Side-A:FID128:admin> lldp --disable -tlv dot1
7840-Side-A:FID128:admin> lldp --disable -tlv dot3
7840-Side-A:FID128:admin> lldp --disable -tlv fcoe-app
7840-Side-A:FID128:admin> lldp --disable -tlv fcoe-lls
7840-Side-A:FID128:admin> lldp --show
LLDP Global Information
system-name: 7840-Side-A
system-description: Brocade 7840, Fabric OS Version 8.2.2
State: Enabled
Mode: Receive/Transmit
Advertise Transmitted: 30 seconds
Hold time for advertise: 120 seconds
Tx Delay Timer: 1 seconds
Transmit TLVs: Chassis ID Port ID
TTL Port Description
System Name System Description
System Capabilities Management Address
DCBx FCoE Priority Values: 3

Tunnels
Tunnels are logical entities, each represented by a VE_Port. A tunnel is made up of one or
more circuits. Circuits are made up of multiple TCP sessions. Within the scope of a tunnel is
encryption (IPsec), compression, protocol optimization (FCIP-FastWrite, FCIP-OSTP,
FICON-Accelerator), and enabling IP Extension; these settings apply to the entire tunnel, not
per circuit.

VE_Ports and tunnels are synonymous. Below are the number of tunnels per platform:
򐂰 IBM SAN18B-6 Extension Switch2 or 4 (base or full)
򐂰 IBM SAN42B-R Extension Switch 10 or 20 (two modes of operation)
򐂰 SX6 Extension Blade10 or 20 (two modes of operation)

The IBM SAN256B-7 and SAN512B-7 support up to four SX6 Extension Blades per chassis.

A tunnel configuration mismatch is indicated when the OpStatus is not Up. In this case, one
side of the tunnel is Degrade, and the other side is UpWarn. Some mismatched settings
prevent the tunnel from coming up, for example, IPsec policy or compression mismatches.
Some mismatched settings give warnings, for instance, IP Extension enabled/disabled and
inconsistent local/remote min/max-comm-rates.

162 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

Tunnel Circuit OpStatus Flags Uptime TxMBps RxMBps ConnCnt CommRt Met/G
-------------------------------------------------------------------------------
24 - Degrade --i----DI 1m13s 0.00 0.00 2 - -
24 0 ge2 Degrade ----a--i4 1m13s 0.00 0.00 2 100/150 0/-
24 1 ge4 Degrade ----a--i4 1m13s 0.00 0.00 2 100/150 0/-
-------------------------------------------------------------------------------
Flags (tunnel): l=Legacy QOS Mode
i=IPSec f=Fastwrite T=TapePipelining F=FICON r=ReservedBW
a=FastDeflate d=Deflate D=AggrDeflate P=Protocol
I=IP-Ext
(circuit): h=HA-Configured v=VLAN-Tagged p=PMTU i=IPSec 4=IPv4 6=IPv6
ARL a=Auto r=Reset s=StepDown t=TimedStepDown S=SLA
7840-Side-B:FID128:admin> portshow fciptunnel -c --summary
Tunnel Circuit OpStatus Flags Uptime TxMBps RxMBps ConnCnt CommRt Met/G
-------------------------------------------------------------------------------
24 - UpWarn --i----D- 7s 0.00 0.00 2 - -
24 0 ge3 UpWarn ----a--i4 7s 0.00 0.00 2 100/150 0/-
24 1 ge5 UpWarn ----a--i4 7s 0.00 0.00 2 100/150 0/-
-------------------------------------------------------------------------------
Flags (tunnel): l=Legacy QOS Mode
i=IPSec f=Fastwrite T=TapePipelining F=FICON r=ReservedBW
a=FastDeflate d=Deflate D=AggrDeflate P=Protocol
I=IP-Ext
(circuit): h=HA-Configured v=VLAN-Tagged p=PMTU i=IPSec 4=IPv4 6=IPv6
ARL a=Auto r=Reset s=StepDown t=TimedStepDown S=SLA

Below is the portshow fciptunnel --detail output from both Extension switches. In this
case, IP Extension has been enabled on one side and not the other. Configuration must
match on both ends.
7840-Side-A:FID128:admin> portshow fciptunnel --detail
Tunnel: VE-Port:24 (idx:0, DP0) Broadcom Extension
====================================================
Oper State : Degraded
TID : 24
Flags : 0x00000000
IP-Extension : Enabled
Compression : Aggressive Deflate
FC-Compression : Aggressive Deflate (Inherited)
IP-Compression : Aggressive Deflate (Inherited)
QoS Distribution : Protocol (FC:50% / IP:50%)
FC QoS BW Ratio : 50% / 30% / 20%
IP QoS BW Ratio : 50% / 30% / 20%
Fastwrite : Disabled
Tape Pipelining : Disabled
IPSec : Enabled
IPSec-Policy : MyTunnel
Legacy QOS Mode : Disabled
Load-Level (Cfg/Peer): Failover (Failover / Failover)
Local WWN : 10:00:c4:f5:7c:3b:24:86
Peer WWN : 00:00:00:00:00:00:00:00
RemWWN (config) : 00:00:00:00:00:00:00:00
cfgmask : 0x0000001f 0x4001034e
Uncomp/Comp Bytes : 11048 / 11048 / 1.00 : 1
Uncomp/Comp Byte(30s): 0 / 0 / 1.00 : 1
Configuration Warnings:

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 163
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

Tunnel Type (FCIP|HYBIRD)


Flow Status : 0
ConCount/Duration : 2 / 2m31s
Uptime : 1m30s
Stats Duration : 1m30s
Receiver Stats : 41214 bytes / 212 pkts / 55.00 Bps Avg
Sender Stats : 31216 bytes / 160 pkts / 55.00 Bps Avg
TCP Bytes In/Out : 532370 / 638344
ReTx/OOO/SloSt/DupAck: 8 / 6 / 6 / 0
RTT (min/avg/max) : 0 / 11 / 21 ms
Wan Util : 0.7%
TxQ Util : 0.0%
7840-Side-B:FID128:admin> portshow fciptunnel --detail
Tunnel: VE-Port:24 (idx:0, DP0) Broadcom Extension
====================================================
Oper State : Online Warning
TID : 24
Flags : 0x00000000
IP-Extension : Disabled
Compression : Aggressive Deflate
QoS BW Ratio : 50% / 30% / 20%
Fastwrite : Disabled
Tape Pipelining : Disabled
IPSec : Enabled
IPSec-Policy : MyTunnel
Legacy QOS Mode : Disabled
Load-Level (Cfg/Peer): Failover (Failover / Failover)
Local WWN : 10:00:c4:f5:7c:3b:25:06
Peer WWN : 10:00:c4:f5:7c:3b:24:86
RemWWN (config) : 00:00:00:00:00:00:00:00
cfgmask : 0x0000001f 0x4000030e
Uncomp/Comp Bytes : 22514 / 22514 / 1.00 : 1
Uncomp/Comp Byte(30s): 0 / 0 / 1.00 : 1
Configuration Warnings:
Tunnel Type (FCIP|HYBIRD)
Flow Status : 0
ConCount/Duration : 2 / 2m35s
Uptime : 1m51s
Stats Duration : 1m51s
Receiver Stats : 33176 bytes / 167 pkts / 65.00 Bps Avg
Sender Stats : 42970 bytes / 219 pkts / 58.00 Bps Avg
TCP Bytes In/Out : 639720 / 800940
ReTx/OOO/SloSt/DupAck: 9 / 4 / 6 / 0
RTT (min/avg/max) : 20 / 20 / 21 ms
Wan Util : 0.5%
TxQ Util : 0.0%

VE_Ports
VE_Ports are Virtual Expansion Ports. They are a type of E_Port (Expansion Port) commonly
used in FC fabrics. VE_Ports connect to VE_Ports and are used to connect a domain to a
domain (a switch to a switch) the same as E_Ports; the difference is VE_Ports are WAN side
facing.

VE_Ports are tied to a DP (data processor). All Extension platforms have a DP0; additionally,
the IBM SAN42B-R and IBM SX6 Extension Blade have a DP1. The best practice is to start at

164 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

the first VE_Port on DP0. When a second tunnel is required on the SAN42B-R and SX6
Blade, use the first VE_Port on DP1. On the SAN18B-6, move to the next VE_Port on DP0.
򐂰 IBM SAN42B-R
– VE_Port max bandwidth 20 Gbps (applicable license required)
– DP0: VE 24 to 28
– DP1: VE 34 to 38
򐂰 IBM SX6 Extension Blade
– VE_Port max bandwidth 20 Gbps (applicable license required)
– DP0: VE 16 to 20
– DP1: VE 26 to 30
򐂰 IBM SAN18B-6
– VE_Port max bandwidth 2.5 Gbps (applicable license required)
– DP0: VE 12 to 15

It is imperative to understand TCL rules only live on the DP unto which the target VE_Port
exists. VE_Ports exist only in one DP, either DP0 or DP1. Refer to the TCL section for details.
IP Extension gateways are created on a specific DP, either DP0 or DP1. Ingress traffic
matching a TCL rule is directed to a particular target (VE_Port). If the VE_Port target
designated in a TCL rule is not congruent with the IP Extension gateway’s DP, the rule is not
evaluated. In other words, both the IP Extension gateway and target VE_Port used must be
on the same DP for the rule to be evaluated. If the rule is expected to be assessed but is not,
check if traffic is hitting an IP Extension gateway configured on the other DP.

A VE_Port (tunnel) cannot be shared across different VF LS; the VE_Port must live within a
single LS. GE interfaces should stay in the Default Switch, where they can be used by circuits
originating from any LS. See the section on Ethernet Interface Sharing.

The best practice is to persistently disable unused ports.

The maximum cumulative circuit bandwidth of a VE_Port cannot exceed (license permitting):
򐂰 2.5 Gbps on the IBM SAN18B-6
򐂰 20 Gbps on the IBM SAN42B-R
򐂰 20 Gbps on the IBM SX6 Extension Blade

Interoperability
From the perspective of tunnel connectivity and features (VE_Port to VE_Port), the IBM
SAN18B-6, SAN42B-R, and SX6 Extension Blade interoperate together.

The SAN42B-R and SX6 Blade have equal bandwidth capabilities when licensed
appropriately.

Connecting a SAN42B-R or SX6 Blade to a SAN18B-6, the maximum bandwidth to a


SAN18B-6 base unit is 1 Gbps and 2.5 Gbps to a full unit.

The IBM SAN42B-R and SX6 Blade support performing eHCL; the IBM SAN18B-6 does not;
however, the IBM SAN18B-6 does support the required eHCL connectivity needed by the IBM
SAN42B-R and SX6 Blade.

Virtual Fabrics
Virtual Fabrics (VF) are comprised of one or more Logical Switches (LS) within the physical
chassis. Connected Logical Switches form Logical Fabrics within the overall physical fabric.
VF plays a primary role in achieving deterministic paths for Extension protocol optimization.
Additionally, in mainframe environments, the implementation of FICON and FC Extension

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 165
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

often requires unique configurations and management, which is only achievable within the
same chassis when using different LS.

Protocol optimization requires that an exchange’s sequences and frames pass through the
same VE_Ports for both outbound and return. This determinant path means that only a single
VE_Port can exist within a VF LS. Putting a single VE_Port in an LS defines a single path
between the two LS. An LS sufficiently scales FC port connectivity, are operationally
simplistic, and creates a stable environment. The IBM SAN42B-R and IBM SX6 Extension
Blade support VF. The IBM SAN18B-6 does not support VF.

IP Extension
IP Extension is enabled within the tunnel scope (fciptunnel) using --ipext enable. The IBM
SAN18B-6 Extension platform does not have different app-modes for FCIP Only and Hybrid;
essentially, it is always in Hybrid mode. The IBM SAN42B-R and IBM SX6 Extension Blade
must be put into the Hybrid app-mode before IP Extension can be enabled on a tunnel.
Changing the app-mode requires a reboot of the platform or slot. Use the extncfg command
to check the current status or perform this app-mode task. Only change the app-mode to
Hybrid if, at some point, IP Extension will be used.

Compression
Compression is recommended with RDR (Remote Data Replication) applications because it
improves the efficient use of WAN bandwidth. Fast-Deflate compression is ultra-fast and
ultra-low latency. It can be used with RDR/S (sync) to improve response time by reducing
data serialization time across the wire, essentially more data quicker.

Commonly, tape traffic is already compressed, and compressing data again is not helpful;
therefore, if the traffic is already compressed, leaving compression disabled may be prudent.

Note: IBM makes no guarantees, warranties, or claims about the actual compression ratio
achieved with customer-specific data.

There are three modes of compression, not including disabled:


򐂰 Fast-Deflate
Fast-Delate typically gets about a 2:1 compression ratio. Fast-Deflate is a
hardware-implemented (FPGA = Field Programmable Gate Array) compression algorithm
suitable for synchronous applications adding a mere 10 µs of added propagation delay
and accommodating large bandwidth rates. In hybrid mode on the IBM SAN42B-R and
IBM SX6 Extension Blade at 2:1 compression, fast-deflate can accommodate 20 Gbps of
FC traffic per DP. In FCIP mode, the IBM SAN42B-R and IBM SX6 Extension Blade can
accommodate 40 Gbps of FC traffic per DP.
򐂰 Deflate
Delate typically gets about a 3:1 compression ratio. Deflate accommodates up to 16 Gbps
ingress from the FC side per DP. Deflate has been designed to work efficiently with circuits
up to 5 Gbps per DP. Deflate is a software processed algorithm with a hardware assist.
Software-based algorithms take longer to process and may not be suitable for
synchronous applications.
򐂰 Aggressive Deflate
Aggressive-Deflate takes further the tradeoff between the compression ratio and the
compression rate. Aggressive-Deflate typically gets about a 4:1 compression ratio. The
maximum rate per DP is 10 Gbps ingress from the FC side. Aggressive-Deflate has been
designed to work efficiently with circuits up to 2.5 Gbps per DP. Aggressive-Deflate is a
software-based algorithm and may not be suitable for synchronous applications.

166 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

IPsec/Encryption
The best practice is to leverage encryption to protect data end-to-end. It is prudent to enable
IPsec when implementing IBM Extension. All data leaving the secure confines of the data
center into public infrastructure guarantees no security, plus service providers do not
guarantee unencrypted data while in flight. Links must be authenticated and data encrypted
to prevent attacks and eavesdropping. IBM IPsec is easy and practical to deploy. Would your
company operate WiFi with no encryption? Of course not, and Extension is no different!

Encryption is a hardware implementation that operates at line-rate. IPsec is included in all


base Extension platforms, and there are no licenses to enable IPsec. Encryption adds a
minor propagation delay (?5 µs). Preshared Key (PSK) configuration is simple. CA-signed or
self-signed certificates are supported as well.

IPsec is Suite B compliant and implements the latest encryption technologies such as AES
256, SHA-512 HMAC, IKEv2, and Diffie-Hellman. Rekeying occurs in the background
approximately every 2 billion frames or every 4 hours, and the process is non-disruptive.

Firewalls are not considered best practice because:


򐂰 IPsec operates up to 20 Gbps per DP, which is the WAN side line-rate. 40 Gbps if
implementing both DPs. Most firewalls cannot meet this throughput requirement.
򐂰 Storage traffic requires minimal propagation delay.
򐂰 IPsec encrypts data closer to the data source and destination, which is considered best
practice.
򐂰 Firewalls that proxy TCP sessions, like some WAN optimization devices, result in remote
TCP segments not being identical, which is not supported. Such WAN optimization
devices are also not supported for this same reason.

From end to end, the IP network must allow specific protocols to pass. When not using IPsec,
Brocade Extension uses TCP destination ports 3225 and 3226. The Extension TCP
implementation selects a random source port between 49152 and 65535 as the ephemeral
(or initiating) port to open to destination ports 3225 and 3226. TCP URG flags must not be
dropped or modified. When using IPsec, IKEv2 uses UDP port 500. Management connectivity
protocols (ssh, snmp, https...) are not covered here.

Before creating the tunnel, first, create the IPsec policy:


portcfg ipsec-policy <name> <option> [<args>]
portcfg ipsec-policy MYPOLICY create –-preshared-key "mykeymykeymykeymykey"

Protocol Acceleration
Protocol acceleration is used to expedite data without the time needed to traverse the WAN to
request a transfer ready. Protocol optimization effectively mitigates the latency effects on
SCSI/FCP disk writes and tape read/writes. Additionally, there is protocol optimization for
FICON tape and IBM z/OS® GM (XRC).

FastWrite, Open Systems Tape Pipelining (OSTP), and FICON Acceleration are features of
IBM Extension platforms. IBM protocol acceleration requires no additional hardware or
hardware licenses. The same Data Processor (DP) that forms tunnels/trunks also performs
protocol acceleration for a higher level of integration, efficiency, resource utilization, and
reduced assets.

Flows through Extension are monitored for protocol optimization viability. Not including
FICON, IBM storage products do not benefit from or engage FCIP-FastWrite. FastWrite
should not be enabled if Extension is being used for Global Mirror or Metro Mirror alone. If
Extension is being deployed with another application known to benefit from and engage

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 167
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

FastWrite, then FastWrite should be enabled. When FastWrite is enabled, it will work with the
other application while ignoring Global Mirror and Metro Mirror. IBM FICON tape, Teradata,
and z/OS GM (XRC) can be optimized with IBM Advanced FICON Accelerator, which is
included in the mainframe version of the IBM SX6 Extension Blade and an optional license on
the IBM SAN42B-R.

BET (Brocade Extension Trunking) is essential for protocol optimization techniques like
FastWrite, Open Systems Tape Pipelining (OSTP), and FICON Acceleration. BET uses a
single VE_Port and is essentially an ISL. Taking a different VE_Port outbound versus inbound
breaks protocol optimization by creating an inconsistent state machine. Individual tunnels,
each with their own VE_Port, leads to indeterminate VE_Ports for sequences. It is not
possible to perform protocol acceleration with multiple parallel VE_Ports between the same
domains. Regardless of the routing mode (EBR, DBR, and PBR), multiple VE_Ports prevents
a bidirectional deterministic path. For instance, with PBR, the outgoing sequence may
consistently take the same VE_Port; however, the return sequence may always take the same
wrong VE_Port.

When using protocol acceleration, it is necessary to confine traffic to a single bidirectional


path, which can be accomplished in a few ways:
򐂰 Use only a single pair of VE_Ports between the domains.
򐂰 If more than one VE_Port is required, use VF-LS (Virtual Fabrics Logical Switch) to
separate each VE_Port pair.
.

Note: Protocol optimization techniques prevent using load balancing and failover between
VE_Ports

Protocol acceleration uses state machines to perform optimization and requires traffic to be
confined to a single VE_Ports pair. State machines need to know every sequence that has
passed during the exchange. State machines facilitate proper handling of an exchange until it
is complete, after which the state machine is merely discarded. State machines are created
and destroyed with every exchange. IBM Extension has the capacity for tens of thousands of
simultaneous state machines and an equivalent number of flows. Ingress flows are verified if
they can be optimized; if not, the flow does not engage protocol acceleration. If a flow cannot
be optimized, it merely passes without optimization. A state machine cannot exist across
multiple VE_Ports, and there is no communication between different state machines.

If an exchange starts on VE24 and returns on VE25, the exchange passes through different
state machines, as shown in Figure 7-23. One is knowing nothing about the other. This
situation causes protocol acceleration to break and an error condition.

Protocol Acceleration uses state machines, which live in the VE_Ports.


Each VE_Port keeps their own state machines
Outbound Path 1 (VE24) – WRT_CMD
State machine initiated for this exchange

? ?

FC switch may use an inconsistent path


Return Path 2 (VE25) – XFER_RDY
State machine doesn’t exist for exchange in
progress!
Error!
Figure 7-23 No Determinate Path for Protocol Optimization

168 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

One advantage of using BET is to have a single tunnel regardless of how many circuits make
up the tunnel. The state machines exist at the TCP Supervisor endpoints and remain
consistent no matter which circuit is used (see Figure 7-24). BET permits protocol
optimization to function across load-balanced links with non-disruptive failover/failback.

An Extension Trunk uses a single pair of VE_Ports.


There is no path ambiguity.
Outbound circuit0 (VE24) – WRT_CMD
State machine initiated for this exchange

BET
Return circuit1 – XFER_RDY
State machine consistency
Multiple circuits, one tunnel

Figure 7-24 Deterministic Path for Protocol Optimization

In situations where protocol acceleration is not enabled, data flows can be routed across
multiple VE_Ports between domains. The routing mode is based on the APTpolicy. One of
three methods is used internally to route FC flows between equal-cost VE_Ports from the FC
switching ASIC:
򐂰 Exchange Based Routing (EBR), the default): Originator Exchange ID/Source
ID/Destination ID (OXID/SID/DID)
򐂰 Device-Based Routing (DBR): SID/DID
򐂰 Port-Based Routing (PBR): Source Port

FastWrite (FCIP-FW)
FCIP-FastWrite is a protocol optimization technique that intercepts an FCP exchange and
spoofs a local response back to the initiator to expedite data out to the target. There is no
need for the initiator’s request to first traverse the WAN, be responded to by the target, and
travel the WAN again before data starts being sent.

IBM GM and MM do not benefit from or engage FastWrite on IBM b-type Gen 7 Extension
platforms. Optimization is already built into these replication applications. Nonetheless,
FastWrite may still be needed and can be used with cohort replication applications without
detriment to GM or MM.

OSTP (Open Systems Tape Pipelining)


FCIP-OSTP is a protocol optimization technique that intercepts FCP exchanges and spoofs a
local response. OSTP supports read and write tape optimization. Read optimization involves
spoofed responses at the target side to expedite sequential block reads. Write optimization
involves spoofed responses at the initiator side to expedite sequential block data out. There is
no need for sequential block tape requests to traverse the WAN, target response, and
traverse the WAN again.

FICON Accelerator
Advanced FICON Accelerator is an FCIP protocol optimization technique that works with
FICON tape, Teradata, and z/OS GM (XRC) over distances 2 ms RTT or greater.

There are two versions of the IBM SX6 Extension Blade available; there are a distributed
systems and a mainframe version. The IBM SX6 Extension Blade mainframe version includes
the FICON Accelerator License (FAL). FAL is an optional license on the SAN42B-R. The
SAN18B-6 does not support FICON features and functions.

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 169
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

Circuits
Circuits make up a tunnel. A tunnel consists of one or more circuits. A tunnel consisting of
multiple circuits does not have to rely on a single port, optic, cable, switch, router, WAN
connection, or service provider; in essence, a single point of failure in the IP network can be
avoided.

The tunnel (VE_Port) manages circuits. Circuits contain several WAN Optimized TCP
sessions. Different TCP sessions are created for each QoS priority (H/M/L) within each
protocol plus control traffic. There are two Extension protocols, IP Extension and FCIP.

Within the scope of circuits is ARL (min/max committed rate), local/remote IP, local/remote
HA IP (not on the IBM SAN18B-6), metrics/group, KATOV, QoS (DSCP and L2CoS).

IP Interfaces (ipif)
A WAN side GE interface can have multiple IP interfaces (ipif) assigned to it; each is a circuit
endpoint. A circuit is assigned to one Ethernet interface, for example, ge2.dp0. You cannot
split a circuit across multiple Ethernet interfaces. An IP interface (ipif) is configured with an IP
address and GE interface. A circuit is created with a local IP address and bound to a GE
interface; the ipif the IP address was configured with.

IP Routes (iproute)
IP routes are used to forward traffic to the next-hop gateway. IP routes apply to both the LAN
side and the WAN side. For the LAN side, an IP route is only used to send IP Extension traffic
out to an end device, traffic that has come in from the WAN side. IP routes are not used to
route ingress LAN traffic into an egress tunnel; only TCL is used for this.

IP routes are specific to a GE interface and specified when created. Every GE interface has
its own routing table. Therefore, if two circuits, each using the same source subnet, going to
the same destination subnet, and assigned to different GE interfaces would require the
following ipif and iproute creation:
򐂰 portcfg ipif ge2.dp0 create 10.10.10.10/24
򐂰 portcfg ipif ge3.dp0 create 10.10.10.11/24
򐂰 portcfg iproute ge2.dp0 create 10.20.20.0/24 10.10.10.1
򐂰 portcfg iproute ge3.dp0 create 10.20.20.0/24 10.10.10.1

IP routes are used on the WAN side to specify the gateway when the destination subnet is not
the same as the source subnet (Layer3 network). For example, the local ipif subnet is
10.10.10.0/24, and the remote ipif subnet is 10.20.20.0/24. Because these subnets are
different, an IP route is required. IP routes are not required if the local and remote IP
addresses are on the same subnet (Layer2 network). Continuing with this example, assume
the local router gateway was 10.10.10.1. The IP route would be: to get to subnet
10.20.20.0/24 use gateway 10.10.10.1. The gateway is an IP address on the same subnet as
the IP address on the ipif.

Diverse Pathways
Each circuit is intended to take a different pathway to gain resiliency and redundancy across
the IP network. Therefore, circuits should connect via various physical Ethernet links to the
different physical data center LAN switches or WAN routers. When there is more than one
WAN connection, the circuits should be statically assigned to those WAN connections. BET
failover/failback is automatic, lossless, and guarantees in-order delivery; therefore, it is best to
have the circuit go offline, stopping the keepalives, and forcing traffic to failover at the
FC/FICON level. In the case of two WAN links, typically, four circuits are used; here is an
example:
򐂰 Cir0: DC LAN Switch A - WAN 1

170 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

򐂰 Cir1: DC LAN Switch A - WAN 2


򐂰 Cir2: DC LAN Switch B - WAN 1
򐂰 Cir3: DC LAN Switch B - WAN 2

The effective RTT for all circuits in a trunk is the longest RTT experienced by a circuit. By
effectively simulating the same RTT for all circuits, data can be driven at line-rate without
significant data arriving out of order. Wait times climb with excessive out of order data.

Bandwidth scheduling is weighted per circuit based on the comm-rate. A typical example is a
tunnel deployed across a ring architecture, in which one span is a relatively short distance
and the other span is much longer. It remains practical to trunk two circuits across each span
of the ring. Asynchronous RDR (Remote Data Replication) is not affected when both paths
experience longer latency, yet the data rate delivery rate is optimized.

Brocade Extension Trunking (BET)


Brocade Extension Trunking is available on all Extension platforms. The base units of the IBM
SAN42B-R and SX6 Extension Blade include BET. The IBM SAN18B-6 requires the full unit
license for this feature. BET is an advanced extension feature providing the following benefits:
򐂰 Single logical tunnel comprised of one or more individual circuits
򐂰 Efficient use of VE_Ports
򐂰 Aggregation of circuit bandwidth
򐂰 Failover/failback
򐂰 Failover groups and metrics
򐂰 Use of disparate WAN paths with differing characteristics
򐂰 Lossless Link Loss (LLL)
򐂰 In-Order Delivery
򐂰 Non-disruptive link loss
򐂰 A deterministic path for protocol acceleration

Brocade Extension Trunking (BET), in essence, provides a single logical tunnel comprised of
multiple circuits. A tunnel with multiple circuits is referred to as a trunk. Brocade Extension
uses a VE_Port to form an Extension ISL consisting of associated circuits. If each circuit were
a tunnel, a VE_Port would be required for each; however, each circuit makes up a single
tunnel with BET. A tunnel is an Inter-Switch Link (ISL) and treated as such in fabric design.
Circuits are individual connections within the tunnel, each with a unique source and
destination IP address and other settings.

Figure 7-25 on page 172 shows an example of a tunnel trunking four circuits. Each circuit
endpoint is assigned a unique IP address by way of IP interfaces (portcfg ipif). An IP
interface is bound to an Ethernet interface when the ipif is configured. In this case, each IP
interface has been assigned to a different Ethernet interface; however, this is not required.
Depending on the environment’s needs, Ethernet interface assignments are flexible and can
be made as needed. For instance, multiple IP interfaces can be assigned to a single Ethernet
interface. There are no subnet restrictions. Circuits connect from the local IP interface to the
remote IP interface via the bound Ethernet interfaces.

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 171
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

Figure 7-25 Brocade extension trunking

VE_Ports are the demarcation point between the FC world and the TCP/IP world. Brocade
Extension uses FC switching ASICs (Application Specific Integrated Circuits), which knows
only FC protocol. VE_Ports are not an included part of these ASICs and are logical
representations of FC ports. Internally, multiple backend FC ports feed a VE_Port to gain high
data rates into the Extension data processor (DP).

Since one VE_Port represents the endpoint of multiple trunked circuits, this comes with
benefits:
򐂰 Fewer VE_Ports are needed, and the remaining VE_Ports are available for other
destinations. Typically, only one VE_Port is required per remote location.
򐂰 Bandwidth aggregation is achieved by adding circuits to a trunk. Circuits do not have to be
configured identically and do not have to take the same or similar paths. The RTT (Round
Trip Time) does not have to be the same.
򐂰 Link failover is fully automated with BET. Using circuit metrics and failover groups enables
active and passive circuits to coexist within the same trunk.

When a connection is lost, data inflight is lost as well. Any frames being transmitted at the
instance of the outage are lost due to partial transmission. Consider multiple circuits,
Out-Of-Order-Segments (OOOS) occurs when some segments arrive, one or more are lost
inflight, and then segments continue to arrive. OOOS are problematic for some devices,
particularly mainframes, which results in an IFCC (Interface Control Check). For example,
frames 1 through 4 are sent. Frames 1 and 2 are received. Frame 3 is lost in transit, and
frame 4 is received. The receiving side detects an OOOS because it was expecting frame 3
but received frame 4. BET solves this problem with LLL (Lossless Link Loss).

For BET, the DP (Data Processor) that owns the VE_Port controls all circuits. There is no
distributed processing, load sharing, or LLL across DPs. Failover of FC traffic from one DP to
another (for example, a VE on DP0 gets rerouted to a VE on DP1) is done at the FC level
provided the configuration permits it. The only shared components may be various circuits
bound to the Ethernet interfaces.

When a link is lost, the lost frames are retransmitted by LLL (see Figure 7-26 on page 173).
Usually, when a TCP segment is lost due to a dirty link, bit error, congestion, or whatever,
TCP is aware and retransmits the segment. In the case of an offline circuit, TCP has no way
to retransmit lost segments because TCP is no longer operating on that connection.
Recovery must occur at a higher level in the stack.

172 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

Figure 7-26 Lossless link loss process

TCP sessions are managed by a “TCP Supervisor,” which feeds each circuit’s TCP sessions
through a load balancer and a sophisticated egress queuing/scheduling mechanism that
compensates for different bandwidths, latencies, and congestion events.

The TCP Supervisor is a Brocade proprietary technology, purpose-designed, and a special


function algorithm. It operates at the presentation level (Level 6) of the Open Systems
Interconnection (OSI) model and does not affect any network devices monitoring or providing
security at the TCP level (L4).

Suppose a link goes offline, and data can continue over remaining connections. In that case,
missing frames indicated by noncontiguous sequence numbers trigger a duplicate
acknowledgment back to the TCP Supervisor to retransmit the missing segment, even though
that segment was initially sent by a different, now defunct, TCP session. Data lost in flight
requires all segments to be held in memory for possible retransmission until acknowledged at
the supervisor level.

Failover/Failback/Spillover
Circuits can be active or passive within a trunk and configured with metrics. Within a failover
group, all online circuits with the lowest metric value are active, for example, a value of 0.
Circuits with a value of 1 are not active and only become active after ALL metric 0 circuits
have gone offline within the group. An example of two groups, one-to-one pairings of a metric
1 circuit with a metric 0 circuit is shown in Figure 7-27 on page 174. Within a group, the metric
1 circuits will losslessly resume the metric 0 traffic upon the last metric 0 going offline. Metric
1 circuits should take different routes, considering network outages are the most likely cause
the offline metric 0 circuits.

Metrics with failover groups permit passive circuits to become active over a less desirable
path when the regular production path becomes unavailable. In a three-site triangular
architecture, the primary circuit has a metric of 0 in which regular production traffic takes a
single hop, shortest distance, lowest latency path. Sending traffic through the secondary path,

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 173
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

which has two hops and is typically longer in distance and latency, is prevented unless the
primary path is interrupted. Setting a circuit to metric 1 makes it passive until all metric 0
circuits within the group have gone offline. Nonetheless, dormant circuits are members of the
trunk. Convergence over to the secondary path is lossless, and there will be no out-of-order
segments. No IFCC is generated on a mainframe. Extension Trunking is required to assign
metrics to circuits.

Figure 7-27 Failover Metrics and Groups

ARL
ARL is an integral part of most Extension designs. When there is more than one circuit
feeding into the same WAN link or when the WAN is shared with other traffic, ARL is
essential. ARL is used to manage data sent into the IP network based on min and max rate
settings and the unconsumed bandwidth between min and max (see Figure 7-28). There may
be one or multiple WAN connections. The cumulative circuit bandwidth within a particular
WAN connection is what matters. Each WAN connection is evaluated independently.

In a shared WAN, consider bandwidth as separated into three distinct sections:


򐂰 Extension guaranteed bandwidth (0 to min-comm-rate)
򐂰 Extension adaptive bandwidth (min-comm-rate to max-comm-rate, used when available)
򐂰 Non-Extension bandwidth (max-comm-rate to WAN BW)

ARL manages the bandwidth into these sections. The Extension guaranteed section is the
minimum bandwidth available for Extension on the WAN. The Extension guaranteed section
is the amount of bandwidth reserved exclusively for Extension, and ARL aggressively pushes
at least this amount. It is important to note; the minimum bandwidth is the aggregate of ALL
circuit minimums on the WAN connection.

Non-Extension Bandwidth
Bandwidth greater than the cumulative max-comm-rate
of all circuits on this WAN link

Adaptive Bandwidth
Max-comm-rate > available WAN BW > min-comm-rate
Extension bandwidth used when available

Guaranteed Bandwidth
Cumulative min-rate will consume at least this much
bandwidth

Figure 7-28 Guaranteed, adaptive, and non-extension bandwidth using ARL

Adaptive is the area between the top of the guaranteed section and the bottom of the
Non-Extension section. The bottom of the non-Extension section is the aggregate of the

174 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

max-comm-rates for all WAN connection circuits. Extension circuits may use the adaptive
bandwidth when other non-Extension applications are not using it.

The non-Extension bandwidth section is reserved exclusively for applications sharing the
WAN. Extension traffic does not exceed the aggregate of the max-comm-rates for all circuits
on the WAN connection.

Assume a 1 Gbps WAN as an example. The ARL max-comm-rate is set to either the line-rate
GE interface or the maximum available WAN BW, whichever is lowest. In this case, the
max-comm-rate is set to 1 Gbps. Each circuit is configured with a floor (--min-comm-rate) and
ceiling (--max-comm-rate) bandwidth value in kbps (kilobits per second). Assume adequate
bandwidth demand from the storage application; the minimum circuit bandwidth is never less
than the min-comm-rate and is never more than the max-comm-rate. The circuit bandwidth
adjusts automatically between the min and max based on WAN conditions at that moment. A
congestion event causes the rate-limit to readjust towards the minimum. An absence of
congestion events causes it to rise towards the maximum. If the current rate is not maximum,
ARL attempts to adjust upward; however, the rate remains stable if repeated congestion
events are detected. When there is constant congestion, the minimum comm-rate becomes
the prevailing rate.

The ARL min-comm-rate is set to the max-comm-rate ÷ the number of circuits feeding the
WAN connection. In this example, 1 Gbps ÷ 4 = 250 Mbps. The min-comm-rate is set to 250
Mbps. When all the circuits are up, each runs at 250 Mbps. In an extreme case in which three
circuits have gone offline, the remaining circuit runs at 1 Gbps. At 1 Gbps, all the WAN
bandwidth continues to be consumed, and the replication application remains satisfied.

When more than one circuit with the same ARL settings feeds a WAN link, the two circuits
equally utilize the available bandwidth. When an interface or the entire platform goes offline,
ARL readjusts to use available bandwidth that is no longer being used by offline circuits.
Adaptivity and readjustment of bandwidth maintain optimal utilization during periods of
maintenance and failures.

Note: Changing min/max comm-rates on circuits is a non-disruptive operation.

There are many ways in which ARL can be used.

Extension Hot Code Load (eHCL)


eHCL is unique to IBM b-type Gen 7 Extension platforms. It enables Extension platforms to
non-disruptively update firmware without the tunnels going offline. Moreover, the process
maintains all data inflight and ensures in-order delivery. No lost data and no out of order data.

eHCL works by moving the circuits from their native DP to the opposite DP. Firmware
upgrades occur sequentially one DP at a time. When a DP upgrade is complete, the circuits
move back to their original DP. Firmware updates can only be initiated on one tunnel endpoint
at a time. A stable fabric with no device adds, moves, or changes occurring is strongly
suggested.

To enable eHCL, an additional set of HA (high availability) ipif, iproute, and IP addresses
needs to be added to each tunnel on both ends. Assume a tunnel and circuits are on
DP0-VE24:
򐂰 The HA ipifs are created on DP1 (the opposite DP), for example, ipif ge2.dp0
10.10.10.10/24, the complimentary HA ipif ge2.dp1 10.10.10.11/24. Typically, the same
subnet is used for convenience but not required.

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 175
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

򐂰 The HA iproutes are created on DP1, for example, iproute ge2.dp0 192.168.10.0/24
10.10.10.1, the complimentary HA iproute ge2.dp1 192.168.10.0/24 10.10.10.1.
򐂰 Add to fcipcircuit --local-ha-ip 10.10.10.11 (as shown above), and the --remote-ha-ip
192.168.10.11 (not shown above).

There is nothing more to configure or enable. The eHCL process occurs automatically with
each firmware update provided the HA status is “HA Online,” refer to the example CLI output
below.

To check the status of HA configuration:


ASH-7840-A:FID128:admin> portshow fciptunnel -c --hcl-status
Checking FCIP Tunnel HA Status.
Current Status : Ready
CP Version : v8.2.1c1
DP0 Status:
State : Online - Inactive
Version : v8.2.1c1
Current FC HA Stage : IDLE
Current IP HA Stage : IDLE
IP SVI Swapped : NO
DP COMM Status : UP
DP1 Status:
State : Online - Inactive
Version : v8.2.1c1
Current FC HA Stage : IDLE
Current IP HA Stage : IDLE
IP SVI Swapped : NO
DP COMM Status : UP
Tunnel 24 (FID:128) FC:HA Online IP:HA Online - Traffic will not be disrupted.
Tunnel 25 (FID:128) FC:HA Online IP:HA Online - Traffic will not be disrupted.
Tunnel 26 (FID:128) FC:HA Online IP:HA Online - Traffic will not be disrupted.
Tunnel 27 (FID:128) FC:HA Online IP:HA Online - Traffic will not be disrupted.
Tunnel 28 (FID:128) FC:HA Online IP:HA Online - Traffic will not be disrupted.
Tunnel 34 (FID:128) FC:HA Ready IP:HA Ready - FC and IP traffic will be
disrupted.
Tunnel 35 (FID:128) FC:HA Online IP:HA Online - Traffic will not be disrupted.
Tunnel 36 (FID:128) FC:HA Ready IP:HA Ready - FC and IP traffic will be
disrupted.
Tunnel 37 (FID:128) FC:HA Ready IP:HA Ready - FC and IP traffic will be
disrupted.
Tunnel 38 (FID:128) FC:HA Online IP:HA Online - Traffic will not be disrupted.

KATOV
Each circuit uses keepalives to reset the keepalive timer. The time interval between
keepalives is a configurable setting. If the keepalive timer expires, the circuit is deemed to be
down and is removed from the trunk. Also, when an Ethernet interface loses light, those
circuits are immediately considered down. When a circuit goes offline, the egress queue for
that circuit is removed from the load-balancing algorithm. Traffic continues across the
remaining circuits, albeit at the reduced bandwidth due to removing the link.
1. How does the keepalive mechanism work on a circuit?
A Keepalive Timeout Value (KATOV) is configured for each circuit in the tunnel. BET uses
the value to determine how frequently a keepalive is sent. The math for that is not entirely
straightforward and is beyond the scope of this document. The algorithm ensures that

176 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

multiple keepalives are sent within the timeout period. If the receiving side does not
receive any keepalives within the timeout period, the circuit is deemed down.
2. How are keepalives queued and handled through the network layers?
Keepalives are treated as normal data to WAN-Optimized TCP (WO-TCP). They are not
sent on any unique path through the network stack; therefore, they intermix with the other
data passing through TCP, guaranteeing keepalive transmission. Keepalives cannot be
lost unless the circuit is down. Moreover, the keepalive process determines actual delay
through an IP network. The mechanism takes latency, reorder, retransmits, and any other
network condition into account. TCP is the preferred transport for keepalives because it
keeps tight control of the time data is permitted on the WAN. If getting through the network
exceeds the KATOV, the circuit is torn down, and pending data is re-queued into the
remaining circuits.
It is recommended to configure the KATOV based on the storage application’s timeout
values. The exception is circuits configured with the FICON option, which have a 1-second
KATOV and should not be altered. The default value for non-FICON tunnels is 6 seconds
and can often be reduced depending on the network characteristics. This reduction is
evaluated on a case-by-case basis. When a tunnel contains multiple circuits, the KATOV
should be less than the storage application timeout; this facilitates Lossless Link Loss
(LLL) before the application times out.
3. Can too much data through a circuit bring it down?
Yes and no; the answer is no if there are no issues in the IP network, and the tunnel is just
not big enough to handle the driven workload. No network issues mean no congestion or
packet loss, and ARL is appropriately configured; the bandwidth merely cannot
accommodate the application’s needs. When all the available bandwidth is used, buffer
credit flow control applies backpressure to the incoming flows to prevent buffer overflow.
No circuits go down due to keepalive timeout.
Keepalives are not negatively affected by large amounts of queued data. The amount of
data that can be queued but not sent to the network is relatively small, no more than a few
milliseconds. A few milliseconds is not enough time to cause a keepalive timeout due to
saturation.
On the other hand; misconfigured ARL bandwidth, network issues, congestion, and packet
loss are conditions that may lead to keepalive timer expiration, and subsequently, a
dropped circuit. Improper rate limiting allows more data to enter the network than
bandwidth is available, which results in network errors. Severe buffer overruns, excessive
packet loss, and time exhausting retransmits can result in enough lost keepalives to bring
a circuit down.

QoS
IBM Extension has a comprehensive Quality of Service suite.
򐂰 Protocol Distribution (enforcement)
򐂰 Priority -High/Medium/Low (enforcement)
򐂰 DSCP (marking)
򐂰 802.1P (marking)
򐂰 PTQ (isolation)

Note: The best practice is NOT to alter class-F (control traffic) QoS markings unless
required to differentiate and expedite control traffic across the IP network. Control traffic
failing to arrive promptly causes fabric instability.

Below is an example CLI output showing a tunnel (VE24) with one circuit (0 on GE2) and its
individual QoS components (control, high, medium, and low). The CommRt shows bandwidth

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 177
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

distribution to each priority out of 100%; these are not rate-limits. Control is a strict priority - if
there is something to send, the control traffic is sent before any other.
LVN-7840-B_CA:FID1:admin> portshow fciptunnel -c --qos
Tunnel Circuit OpStatus Flags Uptime TxMBps RxMBps ConnCnt CommRt Met/G
-------------------------------------------------------------------------------
24 - Up c----F-D- 28d22h 0.00 0.00 1 - -
24 0 ge2 Up ----r---4 28d22h 0.00 0.00 1 0/100 0/-
24 - Up h----F-D- 28d22h 0.00 0.00 1 - -
24 0 ge2 Up ----r---4 28d22h 0.00 0.00 1 50/100 0/-
24 - Up m----F-D- 28d22h 0.00 0.00 1 - -
24 0 ge2 Up ----r---4 28d22h 0.00 0.00 1 30/100 0/-
24 - Up l----F-D- 28d22h 0.00 0.00 1 - -
24 0 ge2 Up ----r---4 28d22h 0.00 0.00 1 20/100 0/-
-------------------------------------------------------------------------------
Flags (tunnel): c=Control h=HighPri m=MedPri l=LowPri
i=IPSec f=Fastwrite T=TapePipelining F=FICON r=ReservedBW
a=FastDeflate d=Deflate D=AggrDeflate P=Protocol
I=IP-Ext
(circuit): h=HA-Configured v=VLAN-Tagged p=PMTU i=IPSec 4=IPv4 6=IPv6
ARL a=Auto r=Reset s=StepDown t=TimedStepDown S=SLA

Protocol Distribution

Protocol distribution is the highest tier of QoS enforcement in Brocade Extension.


Enforcement only occurs when there is contention for WAN side bandwidth. There are two
protocols, FCIP and IP Extension.

Which one gets the bandwidth? Protocol distribution answers this question, and it is
user-configurable. The default is 50/50, meaning that each protocol receives 50% of the
tunnel’s cumulative circuit bandwidth during contention. When there is no egress contention,
there is no limit imposed by protocol distribution. For example, FCIP would get 100% of the
bandwidth if there was no IP Extension traffic at that moment. Protocol distribution does not
need to be configured when implementing only one of the two protocols because there is no
contention between protocols.

A frequently used case is a tape grid (IP Extension) with disk replication (FCIP). The best
practice is to manage the overall bandwidth using the same VE_Port (tunnel). Often, disk
replication is more critical, and the desire is to permit disk replication over the tape. Assign
disk replication the bandwidth it needs, and the tape grid gets what disk does not use. Setting
protocol distribution to 90/10 FCIP/IP, respectively, the disk gets 90% of the bandwidth if and
when it needs that much. Worst case scenario, the tape grid gets 10% of the bandwidth. If the
WAN is sized correctly for disk plus tape, rarely will disk replication require 90% of the
bandwidth. Unused bandwidth is utilized by IP Extension for the tape grid. During periods of
low disk bandwidth usage, the tape grid gets most of the bandwidth. Protocol Distribution is a
safety mechanism for disk replication on a shared WAN connection.

Priority (High, Medium, Low)

IBM Extension supports three levels of priority (H/M/L). The default amount of BW that the
scheduler apportions during times of contention is 50/30/20 percent. QoS apportioning of BW
occurs only during contention; otherwise, the BW is shared equally across all priorities. It is
possible to change the default portions to any quantity needed, as long as the amounts are
10% or greater and add up to 100%.

There are seven outbound priorities per circuit. The default percentages are shown:
򐂰 Class-F (strict)

178 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

򐂰 FCIP (50%)
– High (50%)
– Medium (30%)
– Low (20%)
򐂰 IP Extension (50%)
– High (50%)
– Medium (30%)
– Low (20%)

Class-F (control) uses strict queuing, meaning if there is any data to send, it is transmitted
first. There is very little control traffic, and it does not interfere with data. Each QoS priority
has an autonomous TCP session and no reliance on other sessions. Each priority TCP
session can be configured with its own DSCP, VLAN tagging, and 802.1P values. The ability
to set each circuit uniquely permits sessions to be treated independently based on priority
SLA between sites.

There is QoS within the SAN fabrics and across FC ISLs via Virtual Channels (VCs). There
are different VCs for H/M/L and Class-F. Each VC has its own Buffer-to-Buffer Credits and
flow control. Devices are assigned to QoS VCs by enabling QoS in the fabric and putting
QOSH_ or QOSL_ as the prefix to a zone name. The default is QOSM_; therefore, there is no
need to explicitly designate medium zones; although, the prefix is supported.

Devices use VCs throughout the fabric, including Extension. If data ingresses via a “QOSH_”
prefixed zone, the data uses High VCs within the fabric and is automatically assigned to a
High TCP session in the Extension circuits. Traversing the fabric is not required to set traffic to
Extension TCP priorities. Any device directly connected to an Extension platform is assigned
a priority TCP session based on the zone name prefix. The default TCP session is medium.

DSCP

Differentiated Services Code Point (DSCP) is an IP based (L3) Quality of Service (QoS)
marking; therefore, it is an end-to-end QoS marking protocol. DSCP has 64 values; however,
the values 0 through 63 do not denote lowest to highest priority; the marking schema is
different.

All odd values are for private use and can be used in any way an enterprise sees fit. These
odd numbers are for private use, similar to how RFC 1918 IP addresses are for private use.
For example, 192.168.10.1 and DSCP 3 are private values used in any way an enterprise
desires.

Value 46, non-private DSCP value, is referred to as “Expedited Forwarding” and is the highest
DSCP priority. Zero is the default and the lowest priority. There are four groups of
High/Medium/Low values referred to as “Assured Forwarding.” Another group of numbers is
used for backward compatibility with legacy ToS (Type of Service).

Using DSCP in an IP network is the network administrators' responsibility. Without their buy-in
and configuration of the Per-Hop Behavior (PHB) associated with QoS, no QoS can happen.
Ethernet switches' default behavior is to ignore QoS values replacing any ingress value with
zero; unless data arriving on that interface is configured as trusted. Explicitly configuring
specific ingress data as trusted prevents end-users from setting QoS values unannounced to
their Network Administrators.

DSCP marking is configured per-circuit + per-protocol + per-priority (H/M/L).

802.1P

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 179
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

802.1P is a data link-based (L2) QoS marking; therefore, the scope extends from the
Extension platform's GE interface to the Ethernet interface of the directly attached switch.
Devices enforcing 802.1P provide QoS across the data link. The 802.1P header resides
within the 802.1Q VLAN header; therefore, VLAN tagging is required to get 802.1P QoS
marking. Brocade Fabric Operating System refers to 802.1P as L2CoS. (Layer 2 Class of
Service). There are only eight 802.1P values from 0 to 7. Zero is the default and lowest
priority. Seven is the highest priority. The IP network must be configured to acknowledge QoS
markings before being useful; otherwise, the Extension markings are ignored.

L2CoS is configured per-circuit + per-protocol + per-priority (H/M/L).

PTQ

PTQ (Priority TCP QoS) is not configurable; nonetheless, it is a crucial QoS concept to
understand when extending QoS across Extension. Any FC zone with a prefix indicating QoS
uses priority virtual circuits within the fabric and automatically maps traffic to a corresponding
TCP priority in the Extension circuits.

One TCP hallmark is ensuring in-order delivery. QoS, by nature, indicates flows requiring
expediency and designates such handling by the network. TCP and QoS would work against
each other if QoS expedited a flow, and TCP returned it to the original sent order. Therefore,
PTQ applies autonomous TCP sessions for each QoS priority. With PTQ, expediting flows
across different priorities is not an issue because all traffic within each TCP flow has the same
priority (H/M/L).

Virtual Fabrics, Logical Switches, and Ethernet Interface Sharing


Moving a VE_Port into a VF-LS has no functional effect on the LAN side; however, for
administration and role-based access, putting an IP Extension VE_Port into a logical switch
could be done to reach those objectives.

Ethernet Interface Sharing applies only to circuits on the WAN side. The IP Extension
gateways sit behind the LAN interfaces, and LAN interfaces must remain in the Default LS.

Note: IP Extension GE interfaces must exist in the Default LS and cannot be moved to a
different context.

PortChannels
LAN side interfaces support IEEE 802.1AX Link Aggregation (LAG or PortChannel) to the
data center LAN switches. Only a single “connection” per VLAN from the Extension platform
to the data center LAN is supported; however, multiple links can be considered one logical
connection (see ). If the data center switches are a single virtual switch, often referred to as
“vPC” in the Cisco nomenclature. A LAN side PortChannel spanning the switches is
considered a single connection.

In Figure 7-29 on page 181, four links form one LAG. From the viewpoint of the Extension
platform, this is a single connection.

180 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

Figure 7-29 IP extension LAN side connectivity to two switches

Figure 7-30 shows a PortChannel consisting of two links to one data center LAN switch.
Forming a LAG to a single data center switch is an alternative means of connectivity when the
data center switches are not joined together into a single virtual switch.

Figure 7-30 P Extension LAN Side Connectivity to a Switch

There are two PortChannel modes, static and dynamic. The dynamic mode uses LACP (Link
Aggregation Control Protocol) and requires a unique system priority (sysprio) configured on
the data center LAN switches. A DC LAN switch may have multiple LAG (Link Aggregation)
connections from various other systems; therefore, it is important to assign a unique sysprio
(aka, key) to your dynamic LAG. Assigning a sysprio is not required when configuring a static
LAG.
lacp --config -sysprio <priority>
lacp –-sysprio 781
portchannel --create <po_name> -type <static|dynamic> [-key
<po-number>]
portchannel –-create MYLAG1 -type dynamic –key 781
portchannel --add <po_name> -port <port-num|port-range>
portchannel –-add MYLAG1 –port ge2,ge3

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 181
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

IP Extension Compression
Compression operates within the scope of a tunnel. Compression cannot be configured per
circuit. Compression must be configured identically at both tunnel ends; asymmetrical
compression is not supported.

Fast-Deflate is not available for IP Extension on the IBM SAN42B-R and IBM SX6 Extension
Blade; IP Extension compression is limited to Deflate and Aggressive-Deflate. The IBM
SAN18B-6 does not have Fast-Deflate because the platform doesn’t support the data rates
requiring Fast-Deflate.

Compression can be configured specifically for each protocol (FCIP and IP Extension). For
example, on the IBM SAN42B-R or SX6 Blade, Fast-Deflate for FCIP and Deflate for IP
Extension can be configured together. Configuring fast-deflate for FCIP is considered best
practice in a hybrid environment because the Fast-Deflate and Deflate compression engines
are entirely different resources. Fast-Deflate uses an HW engine, and Deflate uses the
processor. 20 Gbps of FC ingress to Fast-Deflate does not consume any IP Extension
capacity available for Deflate or Aggressive-Deflate.

WAN side egress rate... Which compression algorithm to use for IP Extension?

Table 7-4 Compression algorithm in hybrid modes


Hybrid Mode Brocade 7810 Brocade 7840 Brocade SX6 Blade

Disabled (1:1) 2.5 Gbps(VE max 20 Gbps (VE max 20 Gbps (VE max
egress) egress) egress)

Fast Deflate(2:1) N/A(not on the N/A(No IP Extension) N/A(No IP Extension)


platform)

Deflate (3:1) 1.5 Gbps to 2.5 Gbps 10 Gbps to 15 Gbps 10 Gbps to 15 Gbps

Aggressive Deflate 20 Mbps to 1.5 Gbps 20 Mbps to 10 Gbps 20 Mbps to 10 Gbps


(4:1)

Note: Compression ratios are approximate. IBM makes no warranties, guarantees, or claims about
the actual compression ratio achieved with customer-specific data.

Traffic Control List (TCL)


TCL is used to match LAN side ingress traffic to a specific outgoing tunnel. Each TCL rule is
given a name, give TCL rules meaningful human-readable names. IP Extension traffic will not
flow until there is an active (--admin enable) TCL priority that matches the incoming traffic,
directing that traffic into a specified target (tunnel/VE_Port). Multiple tunnels may exist, and
specific traffic can be directed into a particular tunnel by matching the rule with the desired
target.

The priority number order of TCL rules matters because rule processing stops upon the first
match. Rules are processed from lowest to highest priority number. The best practice is to
pick priority numbers by leaving adequate space between, in the event, another rule must be
added. For example, start at 10 and count by 10s when creating multiple rules. Traffic cannot
be matched to more than one tunnel because only a single tunnel can be specified in a rule
matching a flow. Traffic not matching any configured rule eventually reaches the bottom. The
last rule has the highest priority number (65535); it is a “deny all” rule and is not modifiable or
able to be deleted. Any traffic reaching the bottom is dropped.

Note: For TCL rules, the default admin mode is --admin disabled; therefore, ensure the rule’s
priority number has an “*” next to it, indicating that it is active; see the TCL example below.

182 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

The hit counter for each TCL rule identifies matching incoming LAN traffic. The hit counter
increases by 1 for every new TCP session formed (3-way handshake). Additionally, if the rule
matches a UDP, ICMP, or BUM (broadcast, unknown, multicast) packet, the hit counter counts
by 1 for every datagram. Besides some BUM traffic, for example, broadcasts, no traffic should
unintentionally arrive at the IP Extension gateway.

The TCL rules below are named PINGS and MYTCL; use human-readable nomenclature
meaningful to your environment. Priority 10 gives some space before (1-9) to add rules if
needed. For example, a temporary rule can be added for counting ICMP pings arriving at the
IP Extension gateway. Such a rule is useful for troubleshooting communications between
end-devices and the IP Extension gateway.

The example below shows production rules (priority 10) and a temporary rule for
troubleshooting by capturing pings (proto 1). The TCL hit counter registers any matched
pings. You can ping local LAN side IP addresses from the LAN, but you cannot ping local
WAN side IP addresses from the LAN. You can ping local WAN side IP addresses from the
WAN, but you cannot ping the WAN's local LAN side IP address. There is no need to delete
the temporary PINGS rule; however, it should be --admin disable when not being used.
portcfg tcl <name> <option> [<args>]
Example: Captures ICMP pings to see on hit counter
portcfg tcl PINGS create --priority 8 --target 12 --src-addr
10.10.10.0/24 --dst-addr 192.168.10.0/24 --admin enable --l4proto
icmp
Example: Local-SAN18B-6
portcfg tcl MYTCL create --priority 10 --target 12 --src-addr
10.10.10.0/24 --dst-addr 192.168.10.0/24 --admin enable
Example: Remote-SAN18B-6
portcfg tcl MYTCL create --priority 10 --target 12 --src-addr
192.168.10.0/24 --dst-addr 10.10.10.0/24 --admin enable
Local7810:admin> portshow tcl
Pri Name Flgs Target L2COS VLAN DSCP Proto Port Hit
Src-Addr Dst-Addr
--------------------------------------------------------------------------
*8 PINGS AI--- 12-Med ANY ANY ANY 1 ANY 29
10.10.10.0/24 192.168.10.0/24
*10 MYTCL AI--- 12-Med ANY ANY ANY ANY ANY 248223
10.10.10.0/24 192.168.10.0/24
*65535 default D---- - ANY ANY ANY ANY ANY 2949
ANY ANY
--------------------------------------------------------------------------
Flags: *=Enabled ..=Name Truncated (see --detail for full name)
A=Allow D=Deny I=IP-Ext P=Segment Preservation
R=End-to-End RST Propagation N=Non Terminated.
Active TCL Limits: Cur / Max
--------------------------------------------
DP0 3 / 32
--------------------------------------------
Configured Total: 3 / 256

Do not create an open TCL in which everything can communicate with everything, including
broadcast and multicasts. Sending broadcasts across the WAN is not considered best
practice. On the other hand, it is not required to close communications to only the devices'
exact IP addresses. Typically, specifying the entire end-device source and destination
subnets (i.e., -S 192.168.10.0/24 -D 10.10.10.0/24) works well.

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 183
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

It is imperative to understand TCL rules only live on the DP unto which the target VE_Port
exists. VE_Ports exist only in one DP, either DP0 or DP1. Refer to the TCL section for details.
IP Extension gateways are created on a specific DP, either DP0 or DP1. Ingress traffic
matching a TCL rule is directed to a particular target (VE_Port). If the VE_Port target
designated in a TCL rule is not congruent with the IP Extension gateway’s DP, the rule is not
evaluated. In other words, both the IP Extension gateway and target VE_Port used must be
on the same DP for the rule to be evaluated. If the rule is expected to be assessed but is not,
check if traffic is hitting an IP Extension gateway configured on the other DP.

TCL rules are evaluated once when the TCP session first connects during the TCP 3-way
handshake. After removing, adding, or making changes to TCL rules; the VE_Port for the
tunnel must be portdisable and portenable to cycle the tunnel. Cycling the VE_Port forces the
proxied TCP sessions to reform their connections and be evaluated by the updated TCL. If
bringing down the entire tunnel is too disruptive, for example, FCIP replication coexists within
the same tunnel; it is possible to disable all the LAN side GE interfaces. Wait until all the TCP
sessions have timed-out, then re-enable the GE interfaces. Another alternative is to disable
the IP storage replication ports on the end-device. The ultimate goal is to get the TCP
sessions to initiate new connections to engage the updated TCL.

The TCL is specific to IP Extension and has no bearing on WAN side IP routing or FC/FICON
side functions. WAN-side IP interfaces (ipif) starting with ge#.dp# use iproute to direct
outgoing traffic toward a specific router gateway. IP interfaces starting with lan.dp# use TCL to
match incoming LAN side traffic to one particular tunnel for IP Extension.

FICON
Native FICON, such as FICON tape and z/OS GM (XRC), is supported by the IBM SAN42B-R
and the SX6 Extension Blade. The Extension of FICON tape alongside TS7700 Grid is
referred to as the high availability architecture. The host on either side can access the
opposite side VTL if some portion of the infrastructure has failed.

When creating a tunnel for carrying FICON traffic, the FICON attribute must be specified
within the tunnel scope. For example, portcfg fciptunnel 24 create -F creates a FICON
tunnel on VE_Port 24.

It is essential to understand that array to array replication of mainframe volumes, for example,
Global Mirror between two DS8900, is not FICON. Even though the volumes are FICON, the
GM replication of those volumes is FCP. Tunnels do not need to be specified as FICON when
the replication protocol is FCP.

FICON connectivity requires the creation of a FICON LS (logical switch). For example, lscfg
--create 1 -ficon. This CLI command creates a FICON logical switch with the Fabric ID
(FID) of 1. The VE_Port must be moved into this context. The GE interfaces should stay in the
Default LS.

Note:
򐂰 KATOV for FICON tunnels is automatically set to 1 second.
򐂰 FICON is not supported on the IBM SAN18B-R.

For further information concerning FICON Administration, see the Brocade Fabric OS FICON
Configuration Guide at https://docs.broadcom.com/doc/53-1004394-02.

Virtual Fabrics
Virtual Fabrics (VF) are comprised of one or more Logical Switches (LS) within the physical
chassis. Connected Logical Switches form Logical Fabrics within the overall physical fabric.

184 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

VF plays a primary role in achieving deterministic paths for Extension protocol optimization.
Additionally, in mainframe environments, the implementation of FICON and FC Extension
often requires unique configurations and management, which is only achievable within the
same chassis when using different LS.

Protocol optimization requires all of an exchange’s sequences, and frames pass through the
same VE_Port pair both outbound and return. This deterministic path means that only a
single VE_Port can exist within a VF LS when enabling protocol optimization. Putting a single
VE_Port in an LS defines a single path between the two LS. An LS can sufficiently scale FC
port connectivity; they are operationally simplistic and create a stable environment. The IBM
SAN42B-R and SX6 Extension Blade support VF; the SAN18B-6 does not.

Extension + FCR
An Extension tunnel nearly always traverses an IP WAN, which has very different
characteristics than an FC network. An Extension tunnel crossing a WAN is essentially an FC
ISL over a WAN. An FCIP link is considered an ISL. A WAN link experiencing flapping can
disrupt a connected fabric if permitted to merge across the WAN. With each flap, disruption
from fabric services attempting to converge, converge again and converge yet again across a
WAN. The more devices logged-in to the fabric, the more difficult convergence is.
Convergence requires CPU and excessive convergence leads to an overload. If the CPU can
no longer process tasks promptly, instability may occur.

Limiting fabric services within a local edge-fabric and not permitting services to span the
WAN prevents large scale and taxing convergence. FCR is not required unless a production
SAN needs to be isolated from potential WAN instability. The best practice is to construct
completely separate production and replication SANs. FCR provides no benefit when only
array replication ports are directly connected to a dedicated replication SAN; there are no
production end devices to disrupt.

Isolated fabrics connected to FCR EX_Ports are referred to as “edge-fabrics.” EX_Ports are
the demarcation points in which fabric services do not extend. Fabric services do not pass
beyond an EX_Port, which forms an “edge.” FCR constrains fabric services to within either an
edge-fabric or the backbone.

Basic architectures with and without FCR:


򐂰 The simplest is no FCR. One fabric, this is the most common Extension implementation for
both distributed systems and mainframe. Directly connecting storage array replication
ports to the Extension boxes is highly reliable and easy to implement, manage and
troubleshoot.
򐂰 Edge-backbone-Edge using FCR and Extension in which edge-fabrics bookend a transit
Extension backbone fabric, as shown in Figure 7-31 on page 186. The backbone fabric
has EX_Ports outward facing to the edge-fabrics. Extension is located within the backbone
segment using VE_Ports.

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 185
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

IBM IBM
Extension Extension
with FCR with FCR
IP WAN
E_Port EX_Port Extension Tunnel
Extension Tunnel EX_Port E_Port

WAN Router WAN Router


Edge Fabric Edge Fabric

Backbone Fabric

Figure 7-31 Edge-backbone-edge extension architecture

Note: Consider the following points:


򐂰 Mainframe FICON environments do not support FCR.
򐂰 When a mainframe host writes FICON to a DASD (Direct Access Storage Device)
volume and subsequently performs RDR to a remote DASD, the array to array
replication (Global Mirror - PPRC) does not use FICON. IBM Global Mirror replication is
FC based.

Compression
There are three compression algorithms; refer above for more information on compression.
Specific to this section on FC/FICON is the Fast-Deflate algorithm. Fast-Deflate is only on the
IBM SAN42B-R and SX6 Extension Blade and only available to FC/FICON. Fast-Deflate is
not available for IP Extension or on the IBM SAN18B-6.

Fast-Deflate is an ultra-high-rate, ultra-low-latency hardware (FPGA) implemented


compression algorithm. It can accommodate 40 Gbps FC/FICON ingress from the application
and 20 Gbps egress to the WAN. Of course, these numbers may vary based on the actual
compression ratio your data may achieve. The algorithm design is for an average of 2:1.

FC/FICON can use the Deflate (16 Gbps max ingress) and Aggressive-Deflate (10 Gbps max
ingress) algorithms. The compression ingress data rates must be accommodating to the
application and the desired WAN side bandwidth. If Deflate gets 3:1 compression, the WAN
side egress needs about 5 Gbps. If Aggressive-Deflate gets 4:1 compression, the WAN side
egress needs about 2.5 Gbps. The improper assignment of a compression algorithm can
result in a bottleneck.

7.3.2 The FC/FICON side


Another side of Extension is the FC/FICON side, and it is associated with FCIP. Not all sides
must be used; for example, IP Extension can be used without FC/FICON; this is supported.
The FC/FICON switch in the Extension platforms is a full-fledged switch with the same Fabric
OS, features, and functions of other IBM b-type Gen 7 SAN switches.

FCIP Only and Hybrid Modes


Extension app-mode (extncfg --show) on the IBM SAN42B-R and SX6 Extension Blade can
be set to either FCIP Only or Hybrid (FCIP + IP Extension). If the implementation of IP
Extension is intended, these platforms should be put into Hybrid mode, which requires a
reboot of the platform or slot.

If there is no intention of using IP Extension, the default is FCIP Only mode, which should be
maintained. Internal resources are partitioned between FCIP and IP Extension when in

186 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

Hybrid mode; therefore, it is best to remain in FCIP Only mode when not intending to use IP
Extension.
LVN-7840-B_CA:FID1:admin> extncfg --show
APP Mode is FCIP
VE-Mode: configured for 10VE mode.
GE-Mode: Not Applicable.

Ethernet Interface Sharing


Any circuit from any VE_Port in any Logical Switch can use GE interfaces in the Default
Switch.

It is possible to assign multiple IP addresses and circuits to the same Ethernet interface by
assigning multiple ipif to the same interface, each with its unique IP address. The same IP
address cannot be configured more than once in the same box. An IP address, ipif, and circuit
can be associated only with one Ethernet interface. When more than one Ethernet interface is
needed, multiple circuits must be created.

Brocade Extension Trunk (BET) uses a single VE_Port with multiple circuits. Frequently,
circuits are assigned to different GE interfaces for the port, optic, and cable redundancy.
Ethernet interfaces do not need to be dedicated to one circuit.

Sharing an Ethernet interface with multiple circuits belonging to different VF LSs is possible.
The VE_Port is moved into the LS context using the lscfg CLI command. The tunnel and
circuits also reside in the LS context and are configured there. The Ethernet interfaces, ipif (IP
interface), and iproute (IP routes) must remain and be configured in the Default Switch
(context 128). An ipif is configured by designating the IP address, subnet mask, VLAN, MTU,
and GE interface. The creation of an ipif assigns the IP address to an Ethernet interface.
When the circuit is configured, the --local-ip binds the circuit to the Ethernet interface.
Ethernet Interface Sharing facilitates efficient use of the 40GE interfaces by allowing multiple
VE_Ports living in multiple Logical Switches to use the large bandwidth connection.

Figure 7-32 shows an example of two VF LS (10 and 20). GE0 and GE1 are 40 Gbps
interfaces and in the Default Switch. There are two circuits per BET, the red BET, and the blue
BET. VE24 (lives in DP0) has one circuit that goes to GE0 and one circuit that goes to GE1.
VE34 (lives in DP1) has one circuit that goes to GE0 and one circuit that goes to GE1. There
are two 10 Gbps circuits within each 40GE interface, each from a different LS. The 40GE
interfaces are physically connected to respective A & B DC LAN switches. Enough 40GE
bandwidth remains to create additional backup circuits (metric 1 circuits) if a 40GE link goes
offline.

IBM Extension Platform


Default LS GE0
LS-10 Physical Ethernet Link
DC LAN Switch A
VE24
circuits

LS-20 DC LAN Switch B


Physical Ethernet Link
VE34 GE1

Figure 7-32 Virtual fabric logical switches ethernet interface sharing

In Figure 7-33 on page 188, Ethernet Interface Sharing is used in an architecture where the
40GE interfaces are used for physical connectivity to the data center LAN. Circuits from 3

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 187
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

tunnels pass through the physical Ethernet links. The traffic on each circuit is VLAN tagged,
so the data center LAN switches can forward the traffic into specific DWDM circuits.

Figure 7-33 Ethernet interface sharing used in practice

7.3.3 The LAN Side (IP Extension)


The LAN side is one of the three sides intrinsic to IBM Extension. The LAN side is
synonymous with IP Extension and is only used to connect IP storage. By leveraging IBM
Extension technology, IP Extension accelerates, secures, and manages the bandwidth of
supported IP storage applications. FCIP and IP Extension protocols use the same tunnel
technology, and sharing a common tunnel is considered best practice. Latency and packet
loss are enemies to TCP/IP distance replication. IP Extension overcomes throughput
degradation caused by these inherent characteristics, plus provides encryption.

This section covers points unique to IP Extension design, best practice, and architectures.

Use Cases/Purpose
IP Extension use cases include acceleration, encryption, bandwidth management, data
migration, network visibility, troubleshooting tools, and congestion avoidance. Figure 7-34 on
page 189 shows a high availability architecture in an IBM System Z® with TS7700
environment. Another IP Extension solution is IBM Transparent Cloud Tiering (TCT). Data is
transferred between VTLs in different data centers. Data is replicated quickly and securely to
maintain a coherent RPO (Recovery Point Objective).

188 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

FICON
IBM z Systems with TS7700 Grid IBM TS7700 Grid
BET Top
BET Bottom

IBM IBM
TS7700 TS7700

IBM z
10Gb/s IBM z

10GE
IP WAN IP WAN 10GE

DS8900 DS8900
Service
Provider(s)
IBM IBM
TS7700 TS7700

Ethernet IBM WAN WAN IBM Ethernet


Switch IP Extension Router Router IP Extension Switch

Global Mirror (FC) or z/Global Mirror (FICON)

Figure 7-34 IP extension high availability architecture for TS7700

FCIP Only and Hybrid Modes


Extension app-mode (extncfg --show) on the IBM SAN42B-R and SX6 Extension Blade can
be set to either FCIP Only or Hybrid (FCIP + IP Extension). If the implementation of IP
Extension is intended, these platforms should be put into Hybrid mode, which requires a
reboot of the platform or slot. There is no IP Extension Only mode.

If there is no intention of using IP Extension, the default is FCIP Only mode, which should be
maintained. Internal resources are partitioned between FCIP and IP Extension when in
Hybrid mode; therefore, it is best to remain in FCIP Only mode when not intending to use IP
Extension.
7840-Side-A:FID128:admin> extncfg --show
APP Mode is HYBRID (FCIP with IPEXT)
VE-Mode: configured for 10VE mode.
GE-Mode: Not Applicable.

VLANs
Ethernet frames can be tagged (IEEE 802.1Q) or untagged. Ethernet ISLs with tagging
enabled are called trunks, which incorporates traffic from multiple VLANs. Respective VLAN
traffic is differentiated via the tags. If IP Extension traffic lives across multiple VLANs, only a
single set of trunked links needs to be connected. All the VLANs can communicate across a
single PortChannel of trunks. An IP Extension gateway is configured explicitly for each subnet
in each VLAN; these are the same subnets that the end-device replication ports are on. There
may be many IP Extension gateways depending on how many subnets are being used. Eight
is the maximum number of IP Extension gateways per DP.

If tagging is configured on the data center LAN switch, it must be configured on the Extension
platform. A tagging mismatch causes Ethernet links not to come online. VLAN ID mismatches
cause no communication; however, the links come online. On the Extension platform,
configuring VLAN tagging is accomplished by adding the VLAN ID when creating the ipif
(portcfg ipif). In the example below, an IP Extension Gateway (10.0.0.2/24) is created and
receives incoming and sends outgoing traffic only on VLAN 10. When creating the ipif, the
MTU can be specified as well; the default MTU is 1500 bytes.

Create an IP Extension Gateway on VLAN 10:


portcfg ipif [<slot>/]<port> <option> [<args>]
portcfg ipif lan.dp0 create 10.0.0.2/24 vlan 10

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 189
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

Show configured IP Extension gateways:


Local7810:admin> portshow ipif lan
Port IP Address / Pfx MTU VLAN Flags
--------------------------------------------------------------------------
lan.dp0 10.0.0.2 / 24 1500 10 U R M
--------------------------------------------------------------------------
Flags: U=Up B=Broadcast D=Debug L=Loopback P=Point2Point R=Running I=InUse
N=NoArp PR=Promisc M=Multicast S=StaticArp LU=LinkUp X=Crossport

If the IP storage traffic is on a single VLAN, there is no reason to configure VLAN tagging. The
data center LAN switch ports should be members of the VLAN without tagging. The Ethernet
links should not be trunks. A LAG of untagged Ethernet links can be established between the
LAN switches and the Extension platform.

Create an IP Extension Gateway without VLAN tagging:


portcfg ipif [<slot>/]<port> <option> [<args>]
portcfg ipif lan.dp0 create 10.0.0.2/24

Show configured IP Extension gateways:


Local7810:admin> portshow ipif lan
Port IP Address / Pfx MTU VLAN Flags
--------------------------------------------------------------------------
lan.dp0 10.0.0.2 / 24 1500 0 U R M
--------------------------------------------------------------------------
Flags: U=Up B=Broadcast D=Debug L=Loopback P=Point2Point R=Running I=InUse
N=NoArp PR=Promisc M=Multicast S=StaticArp LU=LinkUp X=Crossport

IP Extension Gateway
IP Extension is the gateway for data crossing the WAN between data centers. If IP storage
traffic is not forwarded to the IP Extension gateway, it will not utilize IP Extension.

Figure 7-35 shows an example of traditional data center routing. Traffic comes from the IP
storage cluster to the data center LAN switch. The LAN switch forwards it to the traditional
gateway. The router could be an integral part of the LAN switch, or not. The router forwards
the traffic onto the WAN towards the destination.

Traditional
Gateway
WAN Router

WAN

IP Storage Cluster Ethernet

DC LAN Switch

Figure 7-35 Data center traditional gateway implementation

IP Extension requires the end-device to direct traffic to its gateway, usually by static route or a
gateway setting. Static routes are more specific than a default route; therefore, traffic for that
particular destination subnet is forwarded to the IP Extension gateway instead of the default
gateway. The destination subnet and IP Extension gateway are specified in the static route.

190 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

The default route stays pointed to the traditional router gateway and is used when a more
specific route does not exist.

Keep in mind, putting IP Extension in the path and removing it from the path merely involves
activating or deactivating these end-device static routes. With static routes, the traffic goes to
the IP Extension gateway (IP Extension in the path). Without static routes, traffic would go to
the traditional router (IP Extension out of the path).

IP Extension supports the connectivity of multiple data center LANs using VLAN tagging
(802.1Q). Tagging frames on Ethernet links connecting the Extension platform and the data
center LAN switches is called trunking. An IP Extension gateway (ipif lan.dp# with vlan
<ID>) must be configured for each subnet trunked to the Extension platform.

In the example below, two ipif are created, one for each DP. The same subnet can be used;
however, the gateways are different. The Ethernet trunks connected to the LAN interfaces
must be tagging frames for VLAN 100 and VLAN 200. IP Extension gateways 10.10.10.2 and
10.10.10.3 are listening only to VLAN 100 traffic, and 192.168.10.2 and 192.168.10.3 are
listening only to VLAN 200. Of course, in your environment, the VLANs will be different.

Create IP Extension Gateways:


7840-Side-A:FID128:admin> portcfg ipif lan.dp0 create 10.10.10.2/24 vlan 100
Operation Succeeded.
7840-Side-A:FID128:admin> portcfg ipif lan.dp1 create 10.10.10.3/24 vlan 100
Operation Succeeded.
7840-Side-A:FID128:admin> portcfg ipif lan.dp1 create 192.168.10.3/24 vlan 200
Operation Succeeded.
7840-Side-A:FID128:admin> portcfg ipif lan.dp0 create 192.168.10.2/24 vlan 200
Operation Succeeded.

Show the IP Extension Gateways:


7840-Side-A:FID128:admin> portshow ipif lan
Port IP Address / Pfx MTU VLAN Flags
-------------------------------------------------------------------------------
lan.dp0 10.10.10.2 / 24 1500 100 U R M
lan.dp0 192.168.10.2 / 24 1500 200 U R M
lan.dp1 10.10.10.3 / 24 1500 100 U R M
lan.dp1 192.168.10.3 / 24 1500 200 U R M
-------------------------------------------------------------------------------
Flags: U=Up B=Broadcast D=Debug L=Loopback P=Point2Point R=Running I=InUse
N=NoArp PR=Promisc M=Multicast S=StaticArp LU=LinkUp X=Crossport

On the IBM SAN42B-R and SX6 Extension Blade, there are two DPs (data processors). The
IP Extension gateway on each DP must be unique; the same gateway cannot be used on both
DPs. VRRP (Virtual Router Redundancy Protocol) is not supported on the IP Extension
platforms. Only one IP Extension gateway can be configured per subnet on each DP. If
planning to use both DPs, the storage application end devices must either be segregated into
groups that use each IP Extension gateway, or replication ports on the end devices must be
capable of independent gateway settings.

It is imperative to understand TCL rules only live on the DP unto which the target VE_Port
exists. VE_Ports exist only in one DP, either DP0 or DP1. Refer to the TCL section for details.
IP Extension gateways are created on a specific DP, either DP0 or DP1. Ingress traffic
matching a TCL rule is directed to a particular target (VE_Port). If the VE_Port target
designated in a TCL rule is not congruent with the IP Extension gateway’s DP, the rule is not
evaluated. In other words, both the IP Extension gateway and target VE_Port used must be

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 191
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

on the same DP for the rule to be evaluated. If the rule is expected to be assessed but is not,
check if traffic is hitting an IP Extension gateway configured on the other DP.

IP storage traffic forwarded to the IP Extension gateway is first evaluated by the TCL (Traffic
Control List), as shown in Figure 7-36. If a flow matches an allow rule, the traffic is forwarded
to the specified tunnel (target). The data is handled by the WAN side of the Extension
platform for delivery to the remote data center.

IP Storage DC LAN Switch


Cluster
Same
subnet

IP Extension Gateway
Flows to remote data center go here IP Extension
Tunnel WAN

Router

Figure 7-36 IP extension gateway

Note: The LAN side IP subnet(s) used on the IP storage end-devices cannot be the same
in both the local and remote data centers. Different subnets for the end devices in each
data center is required.

TCP Flows/Sessions/Streams
IP Extension operates entirely at layer 4 using TCP. Flows, sessions, connections, and
streams are synonymous TCP terms; each represents a connection established by a
successful TCP 3-way handshake. IP Extension regulates ingress data from storage
applications using TCP receive window flow control. By controlling ingress data, IP Extension
expedites data needed by the compression engine and manages data transmission alongside
FCIP flows and flows within a shared WAN. IP Extension maintains separate virtual TCP
sessions within WAN Optimized TCP to manage flows and prevent head of line blocking
individually. IP Extension accommodates a limited number of TCP flows from IP storage end
devices; refer below to the maximum number of supported flows per platform:
򐂰 IBM SAN18B-6256 TCP sessions
򐂰 IBM SAN42B-R512 TCP sessions per DP (two DPs)
򐂰 IBM SX6 Extension Blade512 TCP sessions per DP (two DPs)

IP Extension uses proxy TCP sessions to locally terminate and acknowledge traffic; this is
part of TCP Acceleration. These proxied TCP sessions are local to the data center and can
operate as fast as the end device can send data. WAN Optimized TCP, an aggressive
purpose-built Extension TCP stack, is leveraged to transport large storage quantities across
long-distance WAN connections. Data is repackaged, optimized, and overhead lowered by
IBM b-type Gen 7 High-Efficiency Encapsulation for WAN side delivery.

Not all flows need to be terminated and optimized. For example, the control traffic is not
benefitted by optimization, yet, in many cases, it consumes a valuable TCP session. Most of
the time, control traffic sits idle or produces negligible amounts of traffic. The IBM TS7700 grid
consumes many IP Extension TCP sessions and may exhaust all of them. IBM TS7700 tape
grid generates a few TCP sessions for transporting tape data, but in a cluster generates
hundreds of control TCP sessions. Tape flows generate large data quantities, are
time-sensitive, are rarely idle, and benefit from IP Extension. IBM TS7700 control traffic (TCP
ports 1415 and 1416) uses a different TCP port than data traffic (TCP port 350). To
accommodate this, IP Extension has TCL options for matching particular traffic, enabling

192 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

control traffic to be handled differently by not terminating it and allowing it to pass without
consuming an IP Extension TCP session. Refer to the example TCL below.
7840-Side-A:FID128:admin> portcfg tcl MyTS7700_CTL create --priority 100 --target
24 --src-addr 10.10.10.0/24 --dst-addr 192.168.10.0/24 --proto-port 1415-1416
--non-terminated enabled --admin enable
Operation Succeeded.
7840-Side-A:FID128:admin> portcfg tcl MyTS7700_TAPE create --priority 200 --target
24 --src-addr 10.10.10.0/24 --dst-addr 192.168.10.0/24 --admin enable
Operation Succeeded.
7840-Side-A:FID128:admin> portshow tcl
Pri Name Flgs Target L2COS VLAN DSCP Proto Port
Hit
Src-Addr Dst-Addr
----------------------------------------------------------------------------------
-
*100 MyTS7700_CTL AI--N 24-Med ANY ANY ANY ANY 1415-1416 0
10.10.10.0/24 192.168.10.0/24
*200 MyTS7700_TAPE AI--- 24-Med ANY ANY ANY ANY ANY 0
10.10.10.0/24 192.168.10.0/24
*65535 default D---- - ANY ANY ANY ANY ANY 0
ANY ANY
----------------------------------------------------------------------------------
-
Flags: *=Enabled ..=Name Truncated (see --detail for full name)
A=Allow D=Deny I=IP-Ext P=Segment Preservation
R=End-to-End RST Propagation N=Non Terminated.
Active TCL Limits: Cur / Max
--------------------------------------------
DP0 3 / 128
DP1 1 / 128
--------------------------------------------
Configured Total: 3 / 1024

Note: Consider the following points:


򐂰 Beyond the supported number of flows, when exceeded, newly attempted TCP
sessions are denied - they are not merely passed through un-optimized. End-devices
must either not produce an excessive number of flows, be configured to limit the
maximum number of flows, or operate without fault when flows are not permitted to
connect.
򐂰 IP storage applications must reconcile remote data delivery at the application layer and
not rely on TCP acknowledgments. IP Extension operates by using TCP proxies, which
provide local ACKs. Local TCP acknowledgments are not indicators of ultimate data
arrival at the remote end device. An Extension platform or network failure occurring
after the local ACK and before the remote ACK results in data loss that the sending end
device has already seen an ACK for. At this point, it is up to the storage application to
reconcile the data lost inflight. IBM supported IP storage applications for IP Extension
have been tested to ensure data reconciliation at the application layer in the event of
inflight data loss.
򐂰 UDP and BUM (Broadcast, Unknown, Multicast) traffic is supported in small quantities.
No optimization of these traffic types is performed. The handling of these traffic types is
not done in hardware and requires processor intervention for every datagram received.
It is not recommended to purposely send these traffic types to the IP Extension gateway
or permit these traffic types in the TCL to cross the tunnel.

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 193
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

VE_Ports
VE_Ports for IP Extension are merely representative of the tunnel ID. Unlike FCIP, no IP
Extension data flows through a VE_Port. Although, disabling a VE_Port will disable the entire
tunnel, even if only configured for IP Extension. The VE_Port must be up for the tunnel to be
up.

The best practice is to use the same VE_Port for FCIP and IP Extension between the same
two domains; tunnels are point-to-point. The desire is to have the VE_Port manage the
bandwidth of both protocols across the tunnel. Bandwidth management cannot be
accomplished when using different VE_Ports, one for each protocol.

It is imperative to understand TCL rules only live on the DP unto which the target VE_Port
exists. VE_Ports exist only in one DP, either DP0 or DP1. Refer to the TCL section for details.
IP Extension gateways are created on a specific DP, either DP0 or DP1. Ingress traffic
matching a TCL rule is directed to a particular target (VE_Port). If the VE_Port target
designated in a TCL rule is not congruent with the IP Extension gateway’s DP, the rule is not
evaluated. In other words, both the IP Extension gateway and target VE_Port used must be
on the same DP for the rule to be evaluated. If the rule is expected to be assessed but is not,
check if traffic is hitting an IP Extension gateway configured on the other DP.

7.4 Extension migration strategy


The primary technology used to deliver performance and reliable data replication have been
Fibre Channel over IP (FCIP). FCIP uses high-efficiency encapsulation to form TCP/IP
datagrams from FC frames. WAN Optimized TCP transports data over long-distance
connections from anywhere to anywhere—across the country to around the globe. IBM
continues to deliver the industry’s most advanced Extension products achieving the most
predictable, stable, and performant replication networks, giving clients the ability to protect
their most critical asset reliably: data.

The most common data protection scheme used by enterprise customers is disk replication
or mirroring. There are and continues to be other methods for protecting data, but
array-based mirroring is one of the most commonly used methods in the enterprise market
today. The IBM Extension platforms are designed to excel in this environment, delivering the
most outstanding data protection and performance regardless of distance.

With the ever-increasing amount of data that needs to be protected and the increase in WAN
speed from GE to 10GE to 100GE, many older IBM Extension platforms are reaching the end
of their product support lifecycle. For those charged with business continuance/disaster
recovery for their enterprise data, it would be derelict not to review the current BC/DR
architecture to avoid an unsupported platform situation. This section describes the general
principles of migrating and refreshing a current environment to a new, modernized Extension
platform.

7.4.1 IBM Extension Platforms


Today, IBM offers three platforms to help clients meet their long-distance replication
requirements. To assist our largest enterprise clients with director-class SAN switching
platforms, IBM offers a “blade” (SX6 blade) that integrates directly into those director
platforms. The SX6 Extension Blade transports FC frames through the IP WAN to remote
locations where secondary copies are kept. The SX6 Extension Blade is compatible with the
IBM SAN256B-6, SAN512B-6, SAN256B-7, and SAN512B-7 Director platforms to help

194 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

design flexibility and more cost-effective solutions. SX6 Extension tunnels are interoperable
with the IBM SAN18B-6 and SAN42B-R.

For clients requiring high port density for both FC connectivity and Extension tunnel
bandwidth but may not be deploying directors, IBM offers a 2RU platform called the IBM
SAN42B-R Extension Switch. The IBM SAN42B-R and SX6 Extension Blade are considered
enterprise-class systems and support all the solutions for both open systems and mainframe
(FICON) environments.

The third offered Extension platform is the IBM SAN18B-6 Extension Switch; its predecessor
was the IBM SAN06B-R. The IBM SAN18B-6 is a 1RU system that incorporates many of the
advanced functions found in the SX6 and SAN42B-R. The SAN18B-6 can be integrated into
enterprise-class solutions, but it also supports clients who do not need nor require high port
densities and large Extension tunnel bandwidths. The IBM SAN18B-6 supports all disk
replication solutions but does not support specific mainframe solutions implementing FICON
or emulation techniques.

The IBM SAN06B-R platform was introduced over a decade ago and has now entered End of
Life (EOL). There are thousands of production units today, and those should be refreshed
before the End of Service (EOS) date on October 31, 2024. The remainder of this section
focuses on the methods and strategies for refreshing the IBM SAN06B-R infrastructure.

7.4.2 IBM Extension Solution Designs


IBM Extension platforms give users the flexibility to create a highly effective infrastructure to
meet their current and future data replication needs. Extension technology and its ability to
intermix with an FC SAN allows for endless possibilities in infrastructure design. To narrow
this section's scope, we focus on the most common network designs for business continuity
and disaster recovery (BC/DR), emphasizing migration from EOL Extension platforms.

The most common Extension deployments contain redundant fabrics, dual paths connecting
the storage array through an IP network to a remote site. Having two paths between data
centers offsets the impact if one path fails. Redundancy and data path protection are intrinsic
to building the most resilient infrastructure.

Users should have different IP WAN service providers when redundant paths are needed.
Different WAN service providers create a tremendous amount of additional protection by
having wholly separate and disparate routes between sites. History has shown that users who
do not confirm that the paths are entirely separate and disparate likely will be impacted during
an outage event such as a backhoe digging for a new sewer line. Figure 7-37 shows the “dual
fabric” configuration.

A Brocade BET BET Brocade A


Extension Extension
IP WAN IP WAN
Circuits Routers Routers Circuits

B B

BET BET
Brocade Brocade
Extension Extension

Production Backup
Site Site

Figure 7-37 Dual Replication SAN

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 195
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

The example dual fabric replication SAN shows how a storage array on the left has two
separate and isolated paths. Each IBM SAN18B-6 makes up replication Fabric A and Fabric
B. Those paths are “paired” with the remote storage array on the right side. Note, the paths
are paired A-to-A and B-to-B. Each Extension switch has two Ethernet connections to the IP
network. The IP network has redundant switches, routers, and WAN connections. By
designing a network like this, you can protect your replication scheme from single points of
failure. This example of a replication architecture is the most commonly deployed.

7.4.3 Migration methodology and techniques


Replacing any part of your BC/DR infrastructure can be stressful, especially if you already
have a working infrastructure and process in place. Fortunately, changing from an existing
Brocade Extension platform to a newer model limits migration-related anxiety. You can
achieve greater performance and enhanced functionality when working with a familiar
platform built upon proven generations of IBM Extension technology.

One of the most stressful parts of migrating to a new Brocade Extension platform is the
question of “will I need to take an outage, and if so, how long will that last?” The short answer
is “maybe.” Users can minimize replication downtime when following proper preparation and
methods.

To remove nearly all the risk of taking a full outage, you should plan to stage and configure
your new platforms to mimic as close as possible the same configurations being used today.
Of course, suppose you are going through a consolidation effort, increasing the number of
WAN connections, or upgrading GE to 10GE Ethernet. In that case, there will be additional
steps needed to integrate those changes. For details on the changes and upgrades, see
Brocade Fabric OS Extension User Guide at
https://docs.broadcom.com/doc/FOS-90x-Ext-UG

In general, the following migration methodology and techniques can be applied to any
platform refresh strategy. However, this section focuses on migrating from the IBM SAN06B-R
to the SAN18B-6 platform. This migration section is not intended to be a step-by-step
technical guide but should be used as a planning guideline. For a detailed technical “how-to”
guide on migrating from the IBM SAN16B-R to the IBM SAN18B-6, see Brocade 7800
Extension to Brocade 7810 Extension Migration User Guide at
https://docs.broadcom.com/doc/7800-7810-Migrate-UG

This migration example assumes a one-for-one replacement of the IBM SAN06B-R to


SAN18B-6 with no FC connection changes or Extension circuit changes. Also, it assumes the
same subnets and bandwidth settings are used for the Extension circuits. However, the IP
addresses on the subnet can be the same or different. Using Figure 7-37 on page 195 as our
example network, we walk through some guidelines for migrating platforms.

A benefit of having a “dual fabric” architecture is to provide a seamless transition from one
platform to another. By leveraging one of the fabrics (or paths), replication can be maintained
without taking a full outage.

The following are guidelines for migrating to a new IBM Extension platform:
򐂰 Preliminary work: Run Brocade SAN Health to capture a snapshot of your environment to
see if there are any current “network health” problems requiring attention before the
migration. For more information, see Brocade SAN Health at
https://docs.broadcom.com/doc/SAN-Health-FAQ
򐂰 Install and apply power to the IBM SAN18B-6 at both sites, leave the storage and network
cables disconnected.

196 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

򐂰 Gather required information from the SAN06B-R for building and configuring the IBM
SAN18B-6. Record the output from these CLI commands.
– Local IP addresses and subnet masks (portshow ipif)
– Gateways and remote subnets (
– portshow iproutes)
– Min and Max bandwidth rates (
– portshow fciptunnel -c)
򐂰 Configure the IBM SAN18B-6 at both sites with the configuration information gathered
from the IBM SAN18B-6. There are two ways to configure the Brocade 7810s.
– IBM SANnav Management Tool
– CLI
򐂰 At this point, the Extension tunnel on the IBM SAN18B-R should have a status of “Up.” If
not, do not proceed. The tunnel must be operational before proceeding. Refer to the
validation section of the Extension section to troubleshoot the tunnel and circuits.
򐂰 Select which fabric you intend to migrate first, and, if required by the storage vendor’s
process, suspend the path on the storage array’s replication interface(s).
– Review your storage vendor’s administration guide for the proper process to take down
one or more replication paths, etc.
– With a dual fabric design, the other fabric/path continues replication; however, there
may be some backlog in replication with half the bandwidth if there is a heavy
workload.
򐂰 For the first path to be migrated, disable power to the IBM SAN06B-R at both sites.
򐂰 Remove storage and network cables and reattach the cables to the IBM SAN18B-R.
򐂰 Check for Extension tunnel status and error messages.
– Note: The new path should be online and able to replicate
– You
– can skip the WAN Test Tool validation if you feel confident that the IP network's load
testing is unnecessary.
򐂰 If you wish to confirm that the IP WAN is capable of supporting your bandwidth SLA,
disable the circuit(s), create Wtool sessions, and run Wtool at the bit rate you expecting on
the WAN. Wtool should be run for at least 10 minutes. Review the results.
򐂰 Once the path has been verified, disable the Wtool sessions, and enable the circuit(s) that
were disabled.
򐂰 Once the Extension tunnel has been made operational, enable storage replication
interfaces.
򐂰 Monitor replication and the network for errors, and confirm everything is operating as
expected.
򐂰 Once the initial fabric/path is working as expected, repeat these steps on the other fabric.

As you can see, by taking one path down at a time, the user can reduce or eliminate the need
for a complete outage.

Closing out the guidelines for migrating from an IBM SAN06B-R to a SAN18B-6, it should be
noted that a newer SAN Management tool is available. The IBM Network Advisor, the legacy
SAN Management tool for managing the IBM SAN06-R, is no longer on the market. IBM
SANnav SAN Management tool has a new look and feel, and it has been wholly

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 197
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

re-architected from scratch for the next generation of SAN infrastructure. IBM SANnav
comprises two management portals for local and global views to monitor and manage small
to large environments. For more information on IBM SANnav, see Brocade SANnav
Management Portal User Guide at https://docs.broadcom.com/doc/SANnav-21x-MP-UG.

7.4.4 Migration Summary


As the digital world explodes with new workloads and the risk of losing your most critical data
becomes increasingly intense, having the assurances of securing your data with the most
current and flexible infrastructure becomes a necessity to ensure business sustainability. IBM
Extension switching platforms are relevant with the increased use of Flash and new
technologies like NVMe over Fabrics (Fibre Channel). Keeping the network one step ahead of
servers and storage is critical to ensuring any investment in those new endpoints gives you
the expected performance.

It is also essential to understand the need for product life cycle management and the eventual
migration to newer supported technologies. As you continue to deploy new workloads and
enhance protection requirements with solutions like Remote Disk Replication (RDR), avoid
explaining sudden budget requirements for refreshing replication infrastructure when it’s
imminent—or beyond—the ended support date.

When migrating from the IBM SAN16B-R (Brocade 7800) to the IBM SAN18B-6 (Brocade
7810), there are a few considerations. Extension technology has evolved; features, functions,
and the CLI have changed. Notable changes include IPsec and encryption, IP Extension,
licensing, WAN side bandwidth, and various CLI syntax and references.

7.5 Extension validation


After setting up new circuits or troubleshooting existing ones, validating connectivity is the
next logical step. Below are useful tools for validating WAN side connectivity:

portcmd --wtool (WAN Test Tool)


The Wtool is a base feature included with all IBM Extension platforms. WAN Test Tool (called
W-tool) is the most comprehensive and thorough of the validation tools. The Wtool utility can
generate line-rate traffic between the local and remote Extension platforms via one or more of
the tunnel’s circuits. Wtool simulates actual circuits using the same IP addresses, rate
limiting, IPsec, QoS, TCP sessions, and TCP ports to ensure realistic measurements over
actual pathways. Wtool can be run for any length of time. The following Wtool output shows
measurements taken during the test interval:
Local7810:admin> portcmd --wtool show --peer
WTool Session (0) (Local) (Remote)
--------------------------------------------------------------------------
Admin / Oper State : Up : Up
Up Time : 5m57s : 5m57s
Run Time : 15m0s : 0s
Time Remaining : - : 15m0s
Port : ge2 : -
IP Addr : 172.16.1.3 : 172.16.2.3
IP-Sec Policy : Enabled : Enabled
Configured Comm Rate : 1000000 kbps : 1000000 kbps
Actual Comm Rate : 1000000 kbps : 1000000 kbps
PMTU Discovery (MTU) : disabled (1500) : disabled (1500)

198 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

L2cos /DSCP : (none) / (none) : (none) / (none)


Tx Rate : 963893.01 Kbps : 2073.04 Kbps
Rx Rate : 2013.07 Kbps : 963112.38 Kbps
Tx Utilization : 98.03% : 4.15%
Rx Utilization : 4.03% : 98.47%
RTT (Min/Max) : 19 ms/204 ms : 20 ms/249 ms
RTT VAR (Min/Max) : 1 ms/337 ms : 1 ms/372 ms
Tx pkts : 12348259 : 0
Rx pkts : 0 : 12348245
Ooo pkts : 0 : 0
Drop pkts : 0 : 14
Drop% (Overall / 5s) : 0.00% / 0.00% : 0.00% / -

Best practice is to perform a comprehensive evaluation and baseline of the IP network before
placing an Extension tunnel into production. The evaluation can run for a few minutes, days,
or longer depending on the important time period of interest for testing the IP network, for
example during end of quarter processing. Often the IP network is not what is expected.

An existing circuit must be used with Wtool; however, the circuit must be taken offline before
the associated Wtool session can be enabled. Other circuits can remain in production. Only
disable the circuits you wish to test at one time. Wtool is configured on both ends, as with any
circuit. The Wtool session bit-rate must be configured; the circuit ARL settings are not used.
After completing the Wtool evaluation, the sessions can merely be disabled, and the circuits
enabled again. Sessions can be used again later; there is no need to delete them. Upon
re-enabling the circuits, they will be back in production.

Wtool reports the amount of local data sent and remote data received to determine the IP
network's characteristics and reliability, including packet drops, latency (RTT), jitter (RTT Var),
and out-of-order segments plus more. Typically, one end is the data source, and the other end
is the data sink; however, bidirectional traffic is supported. The process continues for the
user-specified duration or until keyboard terminated (Ctrl+C).

For guidance on setting up and using Wtool, see the Brocade 7810 IP Extension Deployment
Guide at https://docs.broadcom.com/doc/7810-IPEX-UG

portcmd --ping
The portcmd --ping command tests simple connectivity between a local IP address (ipif) on a
particular GE interface and a destination IP address. ICMP ECHO protocol is used. The
portcmd --ping requires designating which GE interface the ping will be sent from, this can be
lan.dpx or ge.dpx interfaces. The source IP address for the ping must already exist on the
specified lan or ge interface.

A LAN side ping to an end device must be on the same subnet as the IP Extension gateway.
Pinging a different subnet requires a configured iproute on the Extension platform, and the
destination device may also require a complimenting return route.

In this example, a ping was done from the ipif lan.dp0 IP Extension gateway to the traditional
router gateway on the same VLAN. Ping verifies connectivity and communications from the IP
Extension gateway through the data center LAN switch.
LVN-7840-A:FID128:admin> portcmd --ping lan.dp0 -s 10.175.1.248 -d 10.175.1.1
PING 10.175.1.1 (10.175.1.248) with 64 bytes of data.
64 bytes from 10.175.1.1: icmp_seq=1 ttl=255 time=37 ms
64 bytes from 10.175.1.1: icmp_seq=2 ttl=255 time=7 ms
64 bytes from 10.175.1.1: icmp_seq=3 ttl=255 time=1 ms
64 bytes from 10.175.1.1: icmp_seq=4 ttl=255 time=100 ms

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 199
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

-- 10.175.1.1 ping statistics ---


4 packets transmitted, 4 received, 0% packet loss, time 887 ms
rtt min/avg/max = 1/36/100 ms

If no local or return IP route exists, ping a destination within the subnet, such as the local
gateway, typically .1. A native subnet ping demonstrates if L2 (Ethernet level) connectivity is
operational through the connected data center switch. If local and return IP routes exist, you
can ping to a foreign subnet destination; however, the remote device must have a
complimenting IP return route to reply. A foreign subnet ping demonstrates if L3 (IP level)
connectivity is operational.
Portcmd --ping [<sx6 slot>/]<port.dp#> <options>

Example ping:
Local7810:admin> portcmd --ping ge6.dp0 -s 172.16.1.3 -d 172.16.2.3
1. PING 172.16.2.3 (172.16.1.3) with 64 bytes of data.
64 bytes from 172.16.2.3: icmp_seq=1 ttl=250 time=19 ms
64 bytes from 172.16.2.3: icmp_seq=2 ttl=250 time=19 ms
64 bytes from 172.16.2.3: icmp_seq=3 ttl=250 time=19 ms
64 bytes from 172.16.2.3: icmp_seq=4 ttl=250 time=19 ms
-- 172.16.2.3 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 789 ms
rtt min/avg/max = 19/19/19 ms

Note: The ping command is used for the mgmt interface only. portcmd --ping is used for the GE
interfaces.

Traceroute should be used on L3 routed networks with 2 or more hops. The example below
has two hops, first to the router, second to the Extension platform. The intermediate and
remote devices must have a complementing IP route to return the traceroute response. It is
not unusual for intermediate devices owned by service providers to not return traceroute
responses.
7840-A:FID128:admin> portcmd --traceroute ge3.dp0 -s 10.2.0.10 -d 10.1.0.10
traceroute to 10.1.0.10, 64 hops max, 64 byte packets
1 10.2.0.1 59 ms 59 ms 59 ms
2 10.1.0.10 59 ms 59 ms 59 ms

nsShow
When connecting end devices to an FC fabric, they log into the fabric. nsshow CLI command
shows the devices that have logged into the fabric and information about those devices. If the
device has successfully logged into the fabric, it is connected and communicating with the
switch.
7840-Side-A:FID128:admin> nsshow
{
Type Pid COS PortName NodeName SCR
N 0a0000; 3;10:00:00:10:9b:34:6e:e0;20:00:00:10:9b:34:6e:e0; 0x00000003
SCR: Fabric-Detected Nx-Port-Detected
FC4s: FCP
PortSymb: [34] "Emulex PPN-10:00:00:10:9b:34:6e:e0"
NodeSymb: [104] "Emulex LPe32002-M2 FV11.4.204.20 DV11.4.33.1
HN:Lenx3650M5-212999.dhcp.broadcom.net OS:VMware ESXi 6.7.0"
Fabric Port Name: 20:00:c4:f5:7c:3b:24:86
Permanent Port Name: 10:00:00:10:9b:34:6e:e0

200 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

Device type: Physical Initiator


Port Index: 0
Share Area: No
Redirect: No
Partial: No
LSAN: No
Slow Drain Device: No
Device link speed: 16G
Connected through AG: No
Real device behind AG: No
FCoE: No
FC4 Features [FCP]: Initiator
N 0a0100; 3;10:00:00:10:9b:34:6e:e1;20:00:00:10:9b:34:6e:e1; 0x00000003
SCR: Fabric-Detected Nx-Port-Detected
FC4s: FCP
PortSymb: [34] "Emulex PPN-10:00:00:10:9b:34:6e:e1"
NodeSymb: [104] "Emulex LPe32002-M2 FV11.4.204.20 DV11.4.33.1
HN:Lenx3650M5-212999.dhcp.broadcom.net OS:VMware ESXi 6.7.0"
Fabric Port Name: 20:01:c4:f5:7c:3b:24:86
Permanent Port Name: 10:00:00:10:9b:34:6e:e1
Device type: Physical Initiator
Port Index: 1
Share Area: No
Redirect: No
Partial: No
LSAN: No
Slow Drain Device: No
Device link speed: 16G
Connected through AG: No
Real device behind AG: No
FCoE: No
FC4 Features [FCP]: Initiator
The Local Name Server has 2 entries }

switchShow
Switchshow is useful for an overall summary of the fabric, domain, FC/FICON ports, state,
connected devices, VE_Ports, GE interfaces, and logical switch context.
7840-Side-A:FID128:admin> switchshow
switchName: 7840-Side-A
switchType: 148.0
switchState: Online
switchMode: Native
switchRole: Principal
switchDomain: 10
switchId: fffc0a
switchWwn: 10:00:c4:f5:7c:3b:24:86
zoning: OFF
switchBeacon: OFF
FC Router: OFF
Fabric Name: REPLICATION
HIF Mode: OFF
Allow XISL Use: OFF
LS Attributes: [FID: 128, Base Switch: No, Default Switch: Yes, Ficon Switch: No,
Address Mode 0]

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 201
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

Index Port Address Media Speed State Proto


==================================================
0 0 0a0000 id N16 Online FC F-Port 10:00:00:10:9b:34:6e:e0
1 1 0a0100 id N16 Online FC F-Port 10:00:00:10:9b:34:6e:e1
2 2 0a0200 id N8 No_Light FC
3 3 0a0300 id N16 No_Light FC
4 4 0a0400 id N16 No_Light FC
5 5 0a0500 id N16 No_Light FC
6 6 0a0600 id N16 No_Light FC
7 7 0a0700 id N16 No_Light FC
8 8 0a0800 id N16 No_Light FC
9 9 0a0900 id N16 No_Light FC
10 10 0a0a00 id N16 No_Light FC
11 11 0a0b00 id N16 No_Light FC
12 12 0a0c00 id N16 No_Light FC
13 13 0a0d00 id N16 No_Light FC
14 14 0a0e00 id N16 No_Light FC
15 15 0a0f00 id N16 No_Light FC
16 16 0a1000 id N16 No_Light FC
17 17 0a1100 id N16 No_Light FC
18 18 0a1200 id N16 No_Light FC
19 19 0a1300 -- N16 No_Module FC
20 20 0a1400 id N16 No_Light FC
21 21 0a1500 id N16 No_Light FC
22 22 0a1600 id N16 No_Light FC
23 23 0a1700 -- N16 No_Module FC
24 24 0a1800 -- -- Online VE VE-Port
10:00:c4:f5:7c:3b:25:06 "7840-Side-B" (downstream)
25 25 0a1900 -- -- Offline VE
26 26 0a1a00 -- -- Offline VE
27 27 0a1b00 -- -- Offline VE
28 28 0a1c00 -- -- Offline VE
29 29 0a1d00 -- -- Offline VE Disabled (10VE Mode)
30 30 0a1e00 -- -- Offline VE Disabled (10VE Mode)
31 31 0a1f00 -- -- Offline VE Disabled (10VE Mode)
32 32 0a2000 -- -- Offline VE Disabled (10VE Mode)
33 33 0a2100 -- -- Offline VE Disabled (10VE Mode)
34 34 0a2200 -- -- Offline VE
35 35 0a2300 -- -- Offline VE
36 36 0a2400 -- -- Offline VE
37 37 0a2500 -- -- Offline VE
38 38 0a2600 -- -- Offline VE
39 39 0a2700 -- -- Offline VE Disabled (10VE Mode)
40 40 0a2800 -- -- Offline VE Disabled (10VE Mode)
41 41 0a2900 -- -- Offline VE Disabled (10VE Mode)
42 42 0a2a00 -- -- Offline VE Disabled (10VE Mode)
43 43 0a2b00 -- -- Offline VE Disabled (10VE Mode)
ge0 -- 40G No_Module FCIP
ge1 -- 40G No_Module FCIP
ge2 id 10G Online FCIP
ge3 -- 10G No_Module FCIP
ge4 id 10G Online FCIP
ge5 -- 10G No_Module FCIP
ge6 -- 10G No_Module LAN
ge7 -- 10G No_Module LAN

202 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

ge8 -- 10G No_Module LAN


ge9 -- 10G No_Module LAN
ge10 -- 10G No_Module FCIP
ge11 -- 10G No_Module FCIP
ge12 -- 10G No_Module FCIP
ge13 -- 10G No_Module FCIP
ge14 id 10G No_Light FCIP
ge15 id 10G No_Light FCIP
ge16 -- 10G No_Module FCIP
ge17 -- 10G No_Module FCIP

Example of Switchshow with a FICON context used for Extension.


LVN-7840-B_CA:FID1:admin> switchshow
switchName: LVN-7840-B_CA
switchType: 148.0
switchState: Online
switchMode: Native
switchRole: Principal
switchDomain: 16
switchId: fffc10
switchWwn: 10:00:50:eb:1a:69:ee:93
zoning: OFF
switchBeacon: OFF
FC Router: OFF
HIF Mode: ON
Allow XISL Use: ON
LS Attributes: [FID: 1, Base Switch: No, Default Switch: No, Ficon Switch: Yes,
Address Mode 1]
Index Port Address Media Speed State Proto
==================================================
0 0 100000 id N16 Online FC F-Port c0:50:76:cf:ed:08:1c:d1
1 1 100100 id N16 Online FC F-Port c0:50:76:cf:ed:08:18:11
2 2 100200 id N16 No_Light FC Disabled (Persistent)
3 3 100300 id N16 No_Light FC Disabled (Persistent)
4 4 100400 -- N16 No_Module FC Disabled (Persistent)
5 5 100500 -- N16 No_Module FC Disabled (Persistent)
24 24 101800 -- -- Online VE VE-Port
10:00:50:eb:1a:fc:e0:da "BRM-7840_CA" (downstream)

fabricShow
When making connections between domains, such as FCIP connections, fabricshow
displays the merged switches that have joined the fabric. In this example, two switches with
domains 10 and 20 have joined fabric REPLICATION. The mgmt IP address of each switch
are shown. Before FCIP traffic will flow between connected end devices (end-to-end), the two
domains connected by FCIP must first form a fabric.
7840-Side-A:FID128:admin> fabricshow
Switch ID Worldwide Name Enet IP Addr FC IP Addr Name
------------------------------------------------------------------------
10: fffc0a 10:00:c4:f5:7c:3b:24:86 10.155.2.221 0.0.0.0 >"7840-Side-A"
20: fffc14 10:00:c4:f5:7c:3b:25:06 10.155.2.220 0.0.0.0 "7840-Side-B"
The Fabric has 2 switches
Fabric Name: REPLICATION

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 203
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

portShow
Portshow displays detailed information about FC/FICON ports. In the example below, FC port
0 is shown.
7840-Side-A:FID128:admin> portshow 0
portIndex: 0
portName: port0
portHealth: HEALTHY
Authentication: None
portDisableReason: None
portCFlags: 0x1
portFlags: 0x20b03 PRESENT ACTIVE F_PORT G_PORT U_PORT LOGICAL_ONLINE LOGIN
NOELP ACCEPT FLOGI
LocalSwcFlags: 0x0
portType: 24.0
portState: 1 Online
Protocol: FC
portPhys: 6 In_Sync portScn: 32 F_Port
port generation number: 0
state transition count: 1
portId: 0a0000
portIfId: 43020028
portWwn: 20:00:c4:f5:7c:3b:24:86
portWwn of device(s) connected:
10:00:00:10:9b:34:6e:e0
16b Area list:
Distance: normal
portSpeed: N16Gbps
FEC: Inactive
Credit Recovery: Inactive
Aoq: Inactive
FAA: Inactive
F_Trunk: Inactive
LE domain: 0
Peer beacon: Off
FC Fastwrite: OFF
Interrupts: 22 Link_failure: 0 Frjt: 0
Unknown: 2 Loss_of_sync: 0 Fbsy: 0
Lli: 22 Loss_of_sig: 2
Proc_rqrd: 33 Protocol_err: 0
Timed_out: 0 Invalid_word: 0
Tx_unavail: 0 Invalid_crc: 0
Delim_err: 0 Address_err: 0
Lr_in: 2 Ols_in: 0
Lr_out: 0 Ols_out: 2

7.6 Extension Monitoring


Monitoring of Extension tunnels and circuits can be done in various ways: IBM SANnav SAN
Management tool, CLI, Rest API automation, and SNMP.

204 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

7.6.1 IBM SANnav SAN Management Tool


IBM SANnav SAN Management Tool is a comprehensive tool for configuring, troubleshooting,
monitoring, and performing various operational tasks. For more details concerning the usage
of IBM SANnav, see Chapter 11 - IBM SANnav Management Suite Overview.

Various dashboards can be constructed to fit your environment’s needs and interests. Top
port traffic, top port utilization, top buffer-buffer credit at zero, REPLICATION fabric health,
configuration drifts, and other top fabric events are monitored, as shown in Figure 7-38. The
widgets have a pulldown menu, and a wide variety of other statistics and errors can be
monitored. Additionally, more rows and widgets can be added.

Figure 7-38 SANnav FC replication dashboard

SANnav Health Summary is shown in Figure 7-39 on page 206. The overall status of the
fabrics, switches, hosts, and storage is indicated. In this case, there is one replication fabric;
typically, there would be two or more replication fabrics (for example, A & B). If SANnav were
to monitor the production SAN as well, those fabrics and devices would be included.

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 205
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

Figure 7-39 SANnav health summary

In Figure 7-40, a simple two-site replication SAN is shown. In many practical instances, the
replication SAN is autonomous from the production SAN. The topology view can narrow down
to a particular fabric for simplicity and investigation. The FOS fabricname has been
configured as “REPLICATION.” In SANnav, this topology has been saved as REPLICATION,
as well. There is a single tunnel using VE24.

Figure 7-40 SANnav Topology of Replication Environment

Clicking on the GE ports displays information specific to the interfaces, as shown in


Figure 7-41 on page 207.

206 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

Figure 7-41 SANnav extension GE interfaces in topology

In Figure 7-42, IBM SANnav is monitoring the traffic going through physical GE interfaces.
The GE interfaces show what quantity of traffic is being sent to the directly connected data
center switches. It does not explicitly show tunnel or circuit traffic statistics. The GE interfaces
are merely abstracted portals through which circuits pass; there may be zero to many circuits
per GE interface.

Figure 7-42 SANnav extension GE interface monitoring

Figure 7-43 on page 208 shows a customizable Dashboard, although a pre-constructed


Extension Dashboard is available. MyDashboard shows tunnel and circuit Rx and Tx activity,
dropped packets, RTT, jitter, and out of order packets. There are other available network
characteristics not shown.

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 207
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

Figure 7-43 SANnav extension dashboard

IBM SANnav is monitoring the Extension Tunnel with selected characteristics, as shown in
Figure 7-44. Compression for this particular data is shown to be approximately 2.3:1. Latency
is being shown at 20 ms RTT. The Extension platform is detecting out-of-order segments,
which is indicative of packet loss on the network. Transmit utilization and MB/s are monitored.
Traffic is being sent from 7840-Side-A (switch1) to 7840-Side-B (switch2). The Receive traffic
consists of mere acknowledgments and has been chosen not to be monitored here.

Figure 7-44 SANnav investigative mode for an extension tunnel

IBM SANnav is monitoring the Extension Circuits with selected characteristics, as shown in
Figure 7-45 on page 209. Transmission in MB/s is shown. Rx MB/s was not selected because
it shows very little traffic consisting of acknowledgments; data is replicated from switch1 to
switch2. Four measurements can be enabled at one time, while two circuits are monitored.
Mouse-over the charts displays the legend, as shown.

208 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

Figure 7-45 SANnav investigative mode for extension circuits

A customized Extension report (see Figure 7-46) can be constructed from the most
interesting widgets for your environment. MAPS (Monitoring Alerting Policy Suite) monitors
every aspect of the Extension platform, and it is prudent to be aware of MAPS violations. Over
long operational periods, a WAN often experiences interims of low quality. By monitoring
packet drop rates, RTT stability, increases in jitter, and reduced tunnel utilization; operations
can drive inquiries about observed changes with the Network Administrators and potentially
preempt further degradation or an entire outage.

Figure 7-46 SANnav extension report

Your customized Extension Report with selected statistics can be generated at regular
intervals, as shown in Figure 7-47 on page 210. For example, if you wish to have the report

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 209
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

generated daily at midnight, waiting for you in the morning, this can be configured. The
formats that can be delivered are HTML, PDF, and CSV.

Figure 7-47 SANnav schedule report

7.6.2 Fabric OS CLI


Many CLI commands show statistical output based on traffic, interfaces, circuits, tunnels, TCP
sessions; in the Appendix are some of the most commonly used commands with their output
for monitoring Extension.
򐂰 portshow (tunnels and circuits)
򐂰 portshow (LAN stats)
򐂰 portshow (TCP stats)
򐂰 portstatsshow
򐂰 geportperfshow
򐂰 switchshow
򐂰 portperfshow
򐂰 porterrshow

7.6.3 REST API


IBM Extension platforms include a fully capable REST API in which configuration and
monitoring can be performed. See Chapter 10 - IBM b-type Gen 7 SAN Automation. Typically,
Ansible Playbooks are used in environments leveraging this type of automation.

7.6.4 SNMP
SNMPv1 and SNMPv3 can be configured to send traps or have statistics read by other
management tools. Below is the output of snmpconfig for snmpv1 traps.
BCM-7840-A:FID128:admin> snmpconfig --show snmpv1
SNMPv1 community and trap recipient configuration:
Community 1: Secret C0de (rw)
No trap recipient configured yet
Community 2: OrigEquipMfr (rw)
No trap recipient configured yet
Community 3: private (rw)
No trap recipient configured yet
Community 4: mySecret1 (ro)
Trap recipient: 10.10.10.10
Trap port: 162
Trap recipient Severity level: 4
Community 5: mySecret2 (ro)

210 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm

Trap recipient: 192.168.10.10


Trap port: 162
Trap recipient Severity level: 4
Community 6: mySecret3 (ro)
Trap recipient: 172.16.10.10
Trap port: 162
Trap recipient Severity level: 4
SNMPv1:Enabled

Below is a list of the trap MIBs available.


7840-Side-A:FID128:admin> snmptraps --show
# |Mib Name |Supported Traps
--|----------------|--------------------------------
001|SW-MIB |sw-fc-port-scn
| |sw-ip-v6-change-trap
| |sw-pmgr-event-trap
| |sw-event-trap
| |sw-fabric-reconfig-trap
| |sw-fabric-segment-trap
| |sw-state-change-trap
| |sw-zone-config-change-trap
| |sw-port-move-trap
| |sw-brcd-generic-trap
| |sw-device-status-trap
002|FICON-MIB |link-rnid-device-registration
| |link-rnid-device-deregistration
| |link-lirr-listener-added
| |link-lirr-listener-removed
| |link-rlir-failure-incident
003|FA-MIB |conn-unit-status-change
| |conn-unit-port-status-change
| |conn-unit-event-trap
004|MIB-2 |cold-restart-trap
| |warm-restart-trap
005|IF-MIB |if-link-up-trap
| |if-link-down-trap
006|RFC1157 |snmp-authetication-trap
007|HA-MIB |fru-status-change-trap
| |fru-history-trap
| |cp-status-change-trap
008|MAPS-MIB |maps-trap
| |maps-quiet-time-trap
009|T11-FC-ZONE-SERVER-MIB|t11ZsRequestRejectNotify
| |t11ZsMergeSuccessNotify
| |t11ZsMergeFailureNotify
| |t11ZsDefZoneChangeNotify
| |t11ZsActivateNotify

For more details, see the Brocade Fabric OS MIB Reference Manual at
https://docs.broadcom.com/doc/FOS-90x-MIB-RM.

Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 211
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm

212 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch08.fm

Chapter 8. NVMe over FC design,


implementation and migration
strategies
The combination of Gen 7 and FOS 9.0 bring a new level of support for NVM Express (NVMe)
in the product set. This includes not only the capability to carry the traffic but the specific
visibility to monitor the traffic as well. Previous generations while capable of carrying the
traffic had little or no ability to monitor it.

This chapter includes the following topics:


򐂰 8.1, “NVMe overview” on page 214
򐂰 8.2, “NVMe over Fibre Channel – SAN design considerations” on page 220
򐂰 8.3, “NVMe over Fibre Channel –implementation overview” on page 221
򐂰 8.4, “NVMe over Fibre Channel - migration example” on page 222
򐂰 8.5, “NVMe over FC - summary” on page 226

© Copyright IBM Corp. 2021. 213


8497ch08.fm Draft Document for Review April 5, 2021 12:31 pm

8.1 NVMe overview


At base, it is important to understand that NVMe is an industry standard that deals with how
software accesses non-volatile memory devices using a PCI Express (PCIeâ) bus interface to
the device. The organization driving the industry standard is the NVM Express consortium
(https://nvmexpress.org/) and is responsible for the definition, management and promotion
of the protocol in the industry. The organization is also responsible for the specification
concerning Fabrics (NVMe-oFä) which addresses the use of NVMe commands over
networked fabrics such as Fibre Channel, InfiniBand or Ethernet (this specification can be
found on the NVM Express website as well). The reason that it is important to understand that
NVMe is a protocol developed for accessing memory devices is that the vast majority of
implementations for storage to date have dealt with protocols (e.g., SCSI, SAS, SATA) that
expected rotational media or even non-storage related devices such as printers or scanners.
As such, NVMe is a much more streamlined and less “chatty” protocol than say SCSI (Small
Computer System Interconnect) which was written more than 30 years ago and did have to
accommodate such a broad range of devices. In fact, one of the earliest significant benefits of
running NVMe-oF is the improved performance on the server stack. As we will see later on,
protocol comparisons show between a 30-60% reduction in CPU overhead for the IO. There
is a lot of excitement for this change because application owners are expecting performance
gains even on existing hardware. For more information about the software stack, see
Performance Benchmarking for PCIe and NVMe Enterprise Solid-State Drives, found at
https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/performa
nce-pcie-nvme-enterprise-ssds-white-paper.pdf.

Figure 8-1 CPU cycles for IOPS using ACHI/SCSI

Figure 8-2 CPU cycles for IOPS using NVMe

In addition to the fact that the NVMe software stack is designed for memory devices and so
avoids all of the legacy software necessary for dealing with rotational media, disk drive
geometries and so on, the definition also includes a level of parallelism that is very different
than the traditional SCSI, Serial Attach SCSI (SAS) or Serial ATA (SATA) environments for
storage. Those older protocols traditionally have a single command queue with a more limited
queue depth. In the case of SCSI a maximum of 64 commands. In the NVMe standard there

214 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch08.fm

is support for 64K (65,535) I/O queues each with 64K (65,535) commands which is commonly
referred to a queue depth or the number of outstanding commands. It is important to keep in
mind that use of this parallelism will continue to be developed by operating systems and
applications. The following diagram provides a view of how the parallelism might be viewed.

Figure 8-3 SCSI and NVMe queues

NVMe queues are mapped to CPU cores which allows a very scalable performance model.
Over time this parallelism may enable changes where virtualization platforms can assign an
NVMe queue to each virtual machine.

When NVMe was first brought to market it was almost exclusively implemented as an
embedded device directly on the PCIe bus internal to the server. Basically, it was intended as
an expansion of system memory that traded off the higher latency that NVMe has compared
to DRAM for a lower cost versus DRAM. At a device level common NVMe products use 4 x
Gen 3 PCIe lanes (4 x 8Gb/s = 32Gb) at a device latency of sub 20 microseconds. Eventually
the market determined that the following characteristics made the need for NVMe over Fabric
a desirable goal.
򐂰 The ability to scale memory capacity past that which could be achieved on a single card
򐂰 The desire not to burden every single server with the cost of an NVMe device
򐂰 The create a shared resource pool of NVMe that could be accessed by multiple servers
򐂰 The desire for platform reliability where single server failure/reboot would not affect the
application

These requirements fed a migration, as shown in Figure 8-4.

Figure 8-4 Migration to NVMe over Fabric

The NVMe Consortium, as previously mentioned, is also responsible for the efforts around
NVMe over Fabrics. A part of that specification is the NVMe Transport binding specification.

Chapter 8. NVMe over FC design, implementation and migration strategies 215


8497ch08.fm Draft Document for Review April 5, 2021 12:31 pm

The NVMe Transport binding specification for Fibre Channel is defined in INCITS 540 Fibre
Channel – Non-Volatile Memory Express (FC-NVMe) at http://www.incits.org/.

NVMe, as previously mentioned, is a protocol specifically designed to speak to memory


devices over a PCI interface. As a result, it has a much lighter weight stack than SCSI. In
NVMe over FC, the FC acts as the transport protocol for the NVMe protocol. NVMe frames
are encapsulated in FC and transmitted over the SAN. Table 8-1 shows the mappings
between the NVMe protocol and the FC protocol.

8.1.1 Comparing mapping of NVMe versus FCP protocols


Table 8-1 Mappings bewtween the NVMe and FCP protocols
NVMe Entity FCP Mapping

Namespace/ NSID LUN

I/Os Exchanges

Command Command routing-control (RCTL) = 6

Status Status RCTL = 7

Extended/ intermediate status E_Status RCTL = 7

NVMe subsystem reset Unsupported

No Recovery, only retries Sequence retransmission request (SRR)


/ read exchange concise (REC)

Association creation Extended link service (ELS)

IO Q creation ELS

Controller Target

Namespace LUN

Rd Read

Wr Write

When architecting a networked environment for NVMe it is also important to remember that
the technology is still expanding in terms of functionality and support. Many of these
capabilities will become available over time in the market and some of those will be made
available through software updates to the existing releases. Some of these have already been
released such as Asymmetric Namespace Access (ANA) for multipathing (the equivalent of
the ALUA in SCSI) and Sequence Level Error Recovery (SLER) while others will take some
time to be implemented.

At the time this is being written, NVMe has 13 required commands and 25 optional. While it is
certainly to be expected that augmentation of functionality over time will increase the number
of commands it provides an exceedingly streamlined interface as compared to the SCSI
software stack which over 30+ years has accommodated not only storage devices but all
sorts of other peripherals (e.g., printers, scanners).

Figure 8-5 on page 217 shows a simplified view of the comparison between the SCSI
software stack and the NVMe software stack. In addition to the fact that NVMe is specifically
written to access memory devices instead of traditional SCSI disk drive elements the software
layer has been streamlined to address the block layer directly yielding a significant increase in
performance.

216 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch08.fm

Figure 8-5 SCSI software stack versus NVMe software stack

It is in part this simplified software stack that produces the reduction in IO overhead observed
in the Intel whitepaper referenced previously. It is expected that over time the NVMe interface
will become the preferred software interface for enterprise storage. While this won’t happen
overnight given the millions and millions of lines of code and product implementations for
SCSI that are already in use, the benefits of the NVMe software performance will eventually
become dominant. This will be especially true for the All Flash Array environments in the
enterprise as organizations seek to get a full utilization of the performance of the assets. This
is also one of the considerations in choosing the type of fabric to use for the storage. It is easy
in the planning stages to assume that “all bandwidth is the same”. That is to say, 100Gb is
100Gb in performance regardless of the software stack or protocol being used. This is a
flawed view. The overhead and services that are encompassed in every protocol always
matter. It is in fact that point that caused the decision to use User Datagram Protocol (UDP) in
the Remote Direct Memory over Converged Ethernet version 2 (RoCE v2) standard for
NVMe. The choice was made because the latency of the Transmission Control Protocol
(TCP) was considered to be too high to allow maximum NVMe performance.

Fibre Channel already uses a Zero Copy mechanism, that is to say that when an application
requests data from a Fibre Channel storage platform it provides a logical address range in the
application buffer for the data to be written to. Effectively the data returned from the request is
written directly into the application. Fibre Channel has been providing this function and its
associated performance gain for more than 20 years. As the storage landscape moves to
utilize more and more flash storage the desire to achieve a maximum Fan-in or Fan-out ratio
(these terms are used to describe multiple server connections sharing access to a single port
on a storage array) for those environments has made Fibre Channel an optimal choice. Part
of the discussion here is the shift in the technology. While Solid State Drives (SSD) have been
packaged for years to be the same form factor, same connectors and same software protocol
for access as actual Hard Disk Drives (HDD) there is, in fact, no disk drive involved. SSD is
comprised most frequently of Flash Memory. It is persistent or non-volatile memory, yet
memory all the same. The term non-volatile is used to distinguish this type of memory from
the Double Data Rate Synchronous Dynamic-Access Memory or DDR SDRAM that is used
for server memory. While SDRAM only retains data while power is applied to the device, flash
memory is persistent in that it only requires power to change state, not to maintain state.
Virtually all USB and SSD devices today make use of NAND style memory chips which are
non-volatile memory. And as the name suggests the same applies to drives comprised of
NVMe devices. What makes this interesting is that Moore's Law (or equivalent variations) now
applies to enterprise storage for the first time in the history of enterprise storage. Density and

Chapter 8. NVMe over FC design, implementation and migration strategies 217


8497ch08.fm Draft Document for Review April 5, 2021 12:31 pm

latency of 3D NAND is continuing to improve with several 3D NAND suppliers moving to 176
layer technology. This drives additional considerations on the types of SAN fabrics that will be
suitable to carry such performance as well as to be able to accommodate multiple iterations of
new technologies within a normal (for example, 5 year) depreciation schedule of capital
assets.

In the current data center infrastructure, the need for NVMe and SCSI to co-exist for a
significant period of time is readily apparent.

8.1.2 NVMe over Fibre Channel


It’s important to remember that Fibre Channel, although long associated predominately with
SCSI, has always been a protocol that was meant from the beginning to carry multiple upper
layer protocols. For example:

Figure 8-6 Fibre Channel protocol

However, the predominance of uses cases for Fibre Channel today would fall into the
scenario depicted in Figure 8-7.

Figure 8-7 Fibre Channel protocol stack

As a market dynamic, each time a new Fibre Channel transmission rate is enabled the oldest
speed is decremented from the product sets. This would normally have meant that in Gen 7
the 4Gb transmission rate would no longer be supported. However, in the instance of the Gen
7 platforms the IBM b-type Gen 7 SAN does retain the 4Gb transmission rate in the Brocade

218 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch08.fm

FC32-X7-48 blade which can be incorporated into either the IBM Storage Networking
SAN512B-7 or the IBM Storage Networking SAN256B-7 chassis-based platforms.

NVMe is the first new Upper Layer Protocol (ULP) mapping in several years. But what makes
this interesting is that the infrastructure that IBM has been selling for years has the potential
to “carry” NVMe traffic. It should be noted that Transmission Rate is only a concern in regard
to performance. All of the current shipping transceivers/optics are able to carry the NVMe
traffic if they are in a compatible Fibre Channel product. A quick summary:
򐂰 NVMe capable adapter: In the case of Fibre Channel this would be a Host Bus Adapter
(HBA). Any Gen 6 or newer HBA with the correct driver can already carry NVMe traffic.
Operating System (OS) support varies for drivers, so it is important to verify supported
versions with your OS vendor.
򐂰 NVMe capable fabric: In the case of Fibre Channel, any IBM B-Type SAN platform that is
Gen 6 or Gen 7 is able to carry NVMe traffic. The required OS level Gen 6 is FOS v8.1 or
higher. For Gen 7 it is FOS v9.0 or higher.

Note: The ability to carry the NVMe traffic is not the same as visibility of the NVMe
traffic. Gen 5 has no visibility to NVMe traffic in IO Insight. In the Gen 6 platforms only
the SAN128B-6 and the 64-port blade in the SAN512B-6 or SAN256B-6 have IO Insight
visibility to NVMe traffic. However, ALL Gen 7 platforms have IO Insight visibility to
NVMe traffic.

򐂰 NVMe capable array: In the case of Fibre Channel in the IBM storage product line any of
the FlashSystem family of FlashSystem 5000, FlashSystem 7200 or the FlashSystem
9200 are capable of supporting NVMe over Fibre Channel fabric.

It is historically true that over time as various elements in the IT infrastructure go through
generational development (e.g., 16Gb FC to 32 Gb FC) bottlenecks move around in the
environment. One of the common scenarios in design consideration is balance. Extremely
fast server platforms communicating with slow storage platforms will potentially experience
congestion. Similarly, the mix of legacy platforms can be problematic especially in the
interaction in the network. Why is this a discussion point as regards NVMe? Because the
latency of NVMe is such that the “inter IO-Gap” is shrinking. This can be thought of as the
“white space” between IOs on the cable, as shown in Figure 8-8.

Figure 8-8 Inter IO-gap reduction as technology improves

A frequent misunderstanding in SAN design is to consider only the bandwidth requirements of


a particular device or devices. The reason that this causes challenges is that it is easy to
forget that while bandwidth, IOs per second (IOPs) and latency are related characteristics,
they are not the same characteristic. Think of bandwidth as the number of lanes on the

Chapter 8. NVMe over FC design, implementation and migration strategies 219


8497ch08.fm Draft Document for Review April 5, 2021 12:31 pm

highway. IOPs is how many vehicles are going past a given point on the highway per second
and latency is how long it takes to get from home to work. Performance applications are
becoming more and more latency and IOP driven. The environment may not drive a full traffic
load on the highway, but it may frequently peak for brief periods of time (for example, during
database analytics) or may have an application set that routinely requires very low latency for
say payment card data. It is always important to have a grasp of the range of application
workloads that are going to be supported by any SAN design. It is an understanding of the
types of workloads to be supported which will provide the “fit for purpose” decision data about
which network technology will meet those needs. The IBM FlashSystem® platforms are
already capable of achieving a less than 100 microsecond latency off the front end of the
array controller using FC-NVMe as the fabric. The fact that the SCSI driver stack is frequently
in the millisecond to seconds range for latency is what is driving a lot of the interest in NVMe
over Fibre Channel.

As previously mentioned, the NVMe over fabric standards continue to develop. One of the
most recent updates include the function of Sequence Level Error Recovery (SLER). The
SLER mechanism allows a much faster recovery of errors. Instead of waiting for an
application timeout the SLER implementation uses a new set of commands (FLUSH and
Responder Error Detected – RED) between the initiator and target ports to recover from any
sequence level errors. Fibre channel utilizes Frames. Multiple Frames make up a Sequence
and multiple Sequences make up an Exchange. An easy way to think of it is Word, Sentence
and Paragraph. Rather than wait for the Exchange to be retried recovery is possible at the
Sequence level via these new signalling commands. For the current version of the
FC-NVMe-2 standard, see FC-NVMe-2 at
https://standards.incits.org/apps/group_public/project/details.php?project_id=174.

8.2 NVMe over Fibre Channel – SAN design considerations


For most customers the implementation of NVMe over Fibre Channel will not be an “all or
nothing” event. In fact, part of the SAN design will need to take into account the fact that there
will be a mix of technologies involved. The use of SCSI based infrastructure will not go away
overnight and for most implementations that will mean the need to allow for concurrent
operations of both the SCSI over Fibre Channel and NVMe over Fibre Channel traffic without
being disruptive to either traffic or hampering the desired performance of the NVMe systems.

Fortunately, as previously discussed, the two protocols can run concurrently in the same
fabric. The most common scenario will encounter some portion of the environment already
being equipment that is capable of participating in an NVMe over Fibre Channel deployment,
such as the following:
򐂰 Gen 6 HBAs (32Gb) with a currently supported NVMe driver stack
򐂰 Gen 6 or Gen 7 IBM b-type Gen 7 SAN platforms with a minimum of FOS v8.1
򐂰 An IBM Storage array that supports NVMe over Fabric

Both the HBAs and the SAN switches/directors have been shipping for more than six years so
there is a good likelihood that some portion of most SAN infrastructure is already in place to
support NVMe over Fibre Channel. Further, the most common SAN topology of Core-Edge is
very well suited to the “in place” integration of NVMe technologies into those existing SANs.
Additional server elements or storage elements that do support NVMe over Fibre Channel
may readily be added into such a topology in a non-disruptive manner. And if there is a need
to refresh the either the core or the edge SAN switches/directors that too is readily
accomplished with little to no risk of downtime due to the dual redundant hardware
infrastructure design of SAN implementations.

220 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch08.fm

Because of the intermix of the technologies the ability of the Gen 7 IBM b-type Gen 7 SAN
platforms to provide functions such as Traffic Optimizer and IO Insight will be of critical help to
the SAN administrator. And, as we will discuss later in this chapter, the migration of
applications/virtual machines from SCSI based storage to NVMe based storage has proven to
be an extremely risk averse technology migration.

As you will see, the combination of the proper topology, along with balanced provisioning,
granular monitoring and automated mitigation provide today’s SAN administrator a very
straightforward window into implementing NVMe over Fibre Channel fabrics.

For help and best practices regarding this type of implementation, see Brocade SAN Design
Guide & Best Practice Guide at https://docs.broadcom.com/doc/53-1004781.

8.3 NVMe over Fibre Channel –implementation overview


Actually, one of the really interesting things about NVMe over Fibre Channel is how much of
the environment is the same as the existing SAN environment for SCSI or FICON. The
enterprise services expected of a Fibre Channel SAN are readily available. But so too are
some of the newer NVMe functions so that the NVMe services become a superset of the
normal SAN services including:
򐂰 Centralized discovery
򐂰 Sharing of discovery data across the fabric
򐂰 Zoning and Security
򐂰 Registered State Change Notification
򐂰 Sequence Level Error Recovery

The normal functions of Zone creation, inclusive of Peer Zoning or Target Driven Zoning, are
the same for NVMe devices as it is for SCSI. For detailed information on the complete
environment please, see FOS 9 Admin Guide at
https://docs.broadcom.com/doc/FOS-90x-Admin-AG. Further, all of the Fabric Vision
functions of IO Insight, Fabric Performance Impact Monitoring and so on are supported as
well.

Some things, however, do change. In an NVMe subsystem the naming conventions are a bit
different. In the SCSI world a Logical Unit Number (LUN) is how individual volumes behind
the SCSI target device are configured. NVMe uses the terminology of a Namespace ID for
that same function. It should be noted that a Namespace is actually comprised of a set of
Logical Block Addresses (LBA) and not a set of names.

It is true that since Fibre Channel is a standard link level network transport that, as previously
mentioned, supports multiple protocols it only needs the new upper layer protocols (ULPs) to
support NVMe over FC. So, while no changes are necessary to the actual FC switching logic
another element to understand is that while the data is being carried as standard Fibre
Channel Protocol (FCP) traffic the NVMe commands are being carried as a new frame type
FC-NVMe. So, when an NVMe device logs in on the fabric and the generic Name Server
entry is created, it also updates the additional information which includes but is not limited to:
򐂰 FC4 type: NVMe device (0x28)
򐂰 Device type: None, host, target, or both host and target
򐂰 Discovery services: Absent or present (specific to NVMe devices only)
򐂰 Port symbolic name
򐂰 Node symbolic name

Figure 8-9 on page 222 shows the namespace and namespace IDs of an NVMe subsystem.
The logical equivalent of a SCSI target is the NVMe controller in this diagram. The logical

Chapter 8. NVMe over FC design, implementation and migration strategies 221


8497ch08.fm Draft Document for Review April 5, 2021 12:31 pm

equivalent of the SCSI LUN (address range) is the namespace ID behind the controller. The
physical namespace or range of Logical Block Addresses is then presented via the
namespace ID. An individual namespace may be presented back to the application
environment via multiple controllers for purposes of either performance or design
redundancy/application resilience via Multi Path IO (MPIO) driver support. In the diagram the
Namespace-B is presented to two separate controllers via Namespace ID-X and Namespace
ID-Y. Consequently, although addressing the same range of LBA the Namespace ID (NSID) is
different depending upon whether it is being addressed through Controller-1 or Controller-2.

Figure 8-9 NVMe subsystem

In Figure 8-9, the MPIO driver would have access to Namespace-B through either controller.

8.4 NVMe over Fibre Channel - migration example


As previously discussed, there is a good chance that much of the infrastructure running today
is already capable of supporting NVMe over Fibre Channel with a simple software update. To
run through a quick example, let’s use VMWARE vSphere 7.1 for a migration from SCSI to
NVMe. For more information, see vSphere ESXi 7.1 Release notes at
https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-esxi-701-release-notes.ht
m. The configuration is Red Had Linux v8.2 running as a guest on top of ESXi 7.1 in the
server.

Figure 8-10 depicts a topology view that shows the host and the storage.

Figure 8-10 Host and storage topology view

Here using IBM SANnav Management Portal one can see the ESXi server platform
connecting across a SAN64B-7 switch to an IBM FlashSystem 9100. In Figure 8-11 on

222 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch08.fm

page 223, the FS9100 (shown in Figure 8-11) is expanded to show the Port World Wide
Names (PWWN) of each of the multi-ported controllers.

Figure 8-11 FS9100 multi-ported controllers

Now by selecting the top controller port and doing and drilling into it we can see the World
Wide Names of the Virtual Ports behind the controller port, as shown in Figure 8-12.

Figure 8-12 Virtual ports

In this fashion, the Zone alias designations for both the SCSI LUN target and the NVMe target
can be seen. And now by selecting the ESXi server we can see the detail of the two VMware
HBA ports, as shown in Figure 8-13.

Figure 8-13 BRCM-ESX VMware HBA ports

Chapter 8. NVMe over FC design, implementation and migration strategies 223


8497ch08.fm Draft Document for Review April 5, 2021 12:31 pm

In Figure 8-14, we can see that the NVMe device is now assigned as a storage element to the
ESXi platform. You will also see that the transport if Fibre Channel and the owner is
something that VMware refers to as HPP. The VMware High Performance Plug-In (HPP)
replaces the VMware Native Multipathing Plug-In for high speed devices such as NVMe. In
fact in NVMe-oF environments HPP is the default plug-in that will help eliminate the higher
latency of the SCSI implementations. Within ESXi, the NVMe-oF targets will be emulated and
presented to users as SCSI targets for simplicity. For more information on the HPP
environments, see VMware High Performance Plug-In and Path Selection Schemes at
https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.storage.doc/GUID-
F7B60A5A-D077-4E37-8CA7-8CB912173D24.html.

Figure 8-14 ESX HPP ownership

Notice that the Fibre Channel SCSI device in this case is still owned by the NMP driver, as
shown in Figure 8-15.

Figure 8-15 ESX NMP ownership

The next step is to select the storage element that is to be mirrored. In this case the SCSI
target in the FS9100 we are going to select the RH8 instance and perform a storage only

224 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch08.fm

migration. First identifying the SCSI target that is the source of the migration. The datastore
source to be migrated is SCSI_Migration.

And now selecting the NVMe device that will be the target of the migration where the name of
the target datastore is NVME_Migration.

Figure 8-16 Migration source and destination selection

The next step will be to initiate the per VM migration of the datastore named SCSI_Migration

After the migration is completed, we can see that the NVMe device is now assigned as a
storage element to the ESXi platform, as shown in Figure 8-17. Now both disks (vDisks) are
on NSID (NameSpace ID) backed data stores.

Figure 8-17 NSID-backed data stores

And finally by selecting the datastore source and looking at “Recent Tasks” at the bottom of
the image, as shown in Figure 8-18 on page 226, it is possible to see that the VM migration to
the new datastore was successfully completed.

Chapter 8. NVMe over FC design, implementation and migration strategies 225


8497ch08.fm Draft Document for Review April 5, 2021 12:31 pm

Figure 8-18 Recent Tasks

Part of the discussion here is that this migration can be accomplished with a live Storage
vMotion of the application in question with no downtime. From the same Fibre Channel HBA
across the same IBM b-type Gen 7 SAN switch to the same IBM FS9100 array. Arguably one
of the most risk averse technology migrations possible today. In this migration the actual
protocol being used to address the storage is changed. Benchmark comparisons for SCSI vs
NVMe with ESXi v7.1 are showing as much as an 80% increase in IOPS providing an
enormous benefit to the application performance.

8.5 NVMe over FC - summary


As a quick review of what we covered:
򐂰 NVMe is more than just a “new drive hardware”
򐂰 NVMe is a new language for talking to memory based storage
򐂰 NVMe is an incredibly efficient protocol in terms of overhead as compared to SCSI
򐂰 Much of the IBM B-Type SAN infrastructure can already carry NVMe traffic
򐂰 The implementation of NVMe is supported in standard SAN topologies whether we are
discussing “core-edge” or “full mesh” designs
򐂰 The ability to drop NVMe capable systems into an existing IBM B-Type SAN without
disruption is key
򐂰 Migration of VMware workloads from SCSI based datastores to NVMe based datastores
without application downtime makes Fibre Channel SAN a clear choice for NVMe
implementations
򐂰 Implementing NVMe is an extremely straightforward and risk averse operation in an IBM
Fibre Channel SAN.

Given the benefits of NVMe based storage platforms the ability of Gen 7 platforms to so
readily support NVMe technology makes IBM b-type Gen 7 SAN the logical evolutionary
choice for the adoption of the technology. It provides:
򐂰 A stable infrastructure that is well understood
򐂰 Lossless, low latency, deterministic network characteristics which are critical for storage
򐂰 The ability to provide both scale and performance simultaneously
򐂰 IO Insight auto learning up to 20,000 SID/DID flows per switch or director (including
NVMe)
򐂰 The ability to manage NVMe as part of a standard Fibre Channel environment
򐂰 At 64Gb line rate the IBM SAN64B-7 provides a 460-nanosecond latency for NVMe
򐂰 The same SAN management (e.g., Zoning) that is already well understood by IT

For more information on NVMe, see IBM NVMe YouTube at


https://community.ibm.com/community/user/storage/communities/community-hom
e/digestviewer

226 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch09.fm

Chapter 9. Fabric OS administrative tools


and diagnostics
This chapter provides a review of the embedded management tools available in Fabric OS
9.0.x. The external SAN management tool SANnav is covered in Chapter 11, “IBM SANnav
Management Suite overview” on page 251.

This chapter includes the following topics:


򐂰 9.1, “Command-line interface (CLI) overview” on page 228
򐂰 9.2, “9.2 Web tools overview” on page 229
򐂰 9.3, “Fabric Vision overview” on page 231

© Copyright IBM Corp. 2021. 227


8497ch09.fm Draft Document for Review April 5, 2021 12:31 pm

9.1 Command-line interface (CLI) overview


The command-line interface (CLI) enables an administrator to monitor and manage individual
switches, ports, and entire fabrics from a standard workstation. It is accessed through the
Ethernet network through either Telnet or SSH connections or physically through the serial
console on the switch.

Fabric OS uses RBAC to control access to all Fabric OS operations. Each feature is
associated with an RBAC role. You need to know which role is allowed to run a command,
make modifications to the switch, or view the output of the command. Consider these
requirements when you create users.

To determine which RBAC role you need to run a command, review the “Role-Based Access
Control” section in the Fabric OS Administrator’s 9.0.0 Guide at the following link:
https://docs.broadcom.com/doc/FOS-90x-Admin-AG

When you use the CLI, remember that both commands and options are case-sensitive. See
the Fabric OS command reference for a description of all of the available CLI commands, their
options, and their uses.

You can connect to the Fabric OS through a SSH or telnet connection or by using a console
session on the serial port.

9.1.1 Connecting to fabric OS through the serial port


Be aware of the following behaviors for serial connections:
򐂰 Some procedures require that you connect through the serial port; for example, setting the
IP address or setting the boot PROM password.
򐂰 Brocade X7 and X6 Director families: You can connect to CP0 or CP1 using either of the
two serial ports.

Use the following procedure to connect to the Fabric OS using the serial port.

Note: If you connect the serial console port to a terminal server, ensure that flow control is
disabled.

1. Connect the serial cable to the serial port on the switch and to an RS-232 serial port on
the workstation.If the serial port on the workstation is an RJ-45 port, instead of RS-232,
remove the adapter on the end of the serial cable and iwnsert the exposed RJ-45
connector into the RJ-45 serial port on the workstation.
2. Open a terminal emulator application (such as HyperTerminal in a Windows environment,
or TERM, TIP, or Kermit in aUNIX environment), and configure the application as follows
– In a UNIX environment, enter the following string at the prompt:
tip /dev/ttyb -9600
If ttyb is already in use, you may use ttya at the prompt.
tip /dev/ttya -9600

In a Windows environment, enter the parameters from Table 9-1 on page 229:

228 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch09.fm

Table 9-1 Parameters to Configure a Windows Terminal Emulator Application


Parameter Value

Bits per second 9600

Data bits 8

Parity None

Stop bits 1

Flow control None

9.1.2 Connecting to Fabric operating system through SSH or telnet


connection
The switch must be physically connected to the network. If the switch network interface is not
configured or the switch has been disconnected from the network, use a console session on
the serial port to set up the IP address.

Use the following procedure to connect to the Fabric OS using the management interface.
1. From a management station, open a SSH or telnet connection using the IP address of the
switch to which you want to connect. The login prompt is displayed when the SSH or telnet
connection finds the switch in the network.
2. Enter the account ID at the login prompt.
3. Enter the password.
If you have not changed the default system password, you are prompted to change it upon
login.
4. Verify that the login was successful.
The prompt displays the switch name and user ID to which you are connected.
login: admin
password: xxxxxxx

Beginning in Fabric OS 9.0.0 several CLI commands were consolidated and the redundant
command were deprecated. The complete list of deprecated commands is listed in the Fabric
OS Command Reference Manual, 9.0.0 at
https://docs.broadcom.com/doc/FOS-90x-Command-RM

9.2 9.2 Web tools overview


Brocade Web Tools is a graphical user interface (GUI) embedded in the Fabric OS firmware
that enables administrators to monitor and manage single or small fabrics, switches, and
ports. Web Tools is launched directly from a Web browser or from SANnav Management
Portal.

Web Tools supports the following operating systems:


򐂰 Red Hat 8.0 and 8.1
򐂰 Windows 10 Pro
򐂰 Windows 2019

Chapter 9. Fabric OS administrative tools and diagnostics 229


8497ch09.fm Draft Document for Review April 5, 2021 12:31 pm

The following browsers can be used to access Web Tools:


򐂰 Chrome
򐂰 Firefox

If the switch is configured with logical fabrics, you can log in to any of the logical fabrics for
which you have permission.

Web Tools does not require a license. It is installed on the switch when you install Fabric OS.

Web Tools can be launched from a browser or through SANnav Management Portal.
򐂰 To launch directly from a Web browser, open your browser, enter the IP address of the
switch, and press Enter. You can use HTTP or HTTPS, for example: http://10.77.77.77
or https://10.77.77.77.
򐂰 To launch from SANnav Management Portal, locate the switch on the SANnav Inventory
page, click the down arrow to the right of the switch, and select View in WebTools from
the action menu. See Figure 9-1.
.

Figure 9-1 Launching Web Tools from SANnav Management Portal

When the Web Tools Element Manager is launched you will see a log in page. See Figure 9-2
on page 231.
򐂰 Enter the user name, password, and logical switch name or fabric ID (FID).
򐂰 For the first switch login, the default user name is admin and the default password is
password. Web Tools prompts you to change the default password.
򐂰 If you are logging in to a Virtual Fabrics-enabled platform and you do not specify a logical
switch, you are logged in to the default logical switch, which uses fabric ID 128. For
non-VF platforms, the FID option is not displayed.
򐂰 If you launch from SANnav Management Portal, you might not need to log in, depending
on the SANnav single sign-on configuration.

230 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch09.fm

Figure 9-2 Web Tools Element Log in screen

For complete instructions on working with Web Tools, download the Fabric OS WebTools
User Guide, 9.0.0 at : https://docs.broadcom.com/doc/FOS-90x-WebTools-UG

9.3 Fabric Vision overview


Fabric Vision technology is an advanced hardware and software architecture that combines
capabilities from Fabric Operating System (FOS), b-type Gen 7 devices, and SANnav. This
suite of tools can be used to help administrators address problems before operations are
affected and accelerate new application deployments.

Fabric Vision technology includes the following features:

9.3.1 Monitoring, alerting, and performance suite (MAPS)


The Monitoring and Alerting Policy Suite (MAPS) is a storage area network (SAN) health
monitoring and alerting application with the capability of early fault detection and isolation.
MAPS helps the SAN administrators to perform the following:
򐂰 Proactively monitor the health and performance of the SAN infrastructure to ensure
application uptime and availability by leveraging predefined and custom policies and rules.
򐂰 Automate the policy-based real-time monitoring and alert thresholds on a port or group
basis.
򐂰 Configure each MAPS-enabled switch to constantly monitor itself for potential faults and
automatically alert you to problems before they become failures.
򐂰 Configure an entire fabric at one time using common rules and policies, or custom policies
for specific ports or switch elements with the simplified fabric-wide threshold configuration,
monitoring, and alerting.
򐂰 Simplify the configuration of a fabric with a common policy and remediate the conditions
that require attention.
򐂰 Automatically fence, toggle, or quarantine ports without manual intervention.

Monitor fabric-wide events, ports, FRUs, environmental parameters, security, traffic flows, and
performance impacts.

Chapter 9. Fabric OS administrative tools and diagnostics 231


8497ch09.fm Draft Document for Review April 5, 2021 12:31 pm

MAPS is a licensed feature and requires a Fabric Vision license. This guide assumes that the
administrator has activated (enabled) the Fabric Vision license is activated (enabled) in the
Fabric OS software to use the full set of MAPS options. For more information on licensing,
refer to the Brocade Fabric OS Software Licensing Guide.

The Brocade Fabric OS Software Licensing Guide can be found at


https://docs.broadcom.com/doc/FOS-90x-Licensing-UG

MAPS is shipped with predefined rules, groups, and policies. However, you can create
custom rules, groups, and policies. MAPS enables the cloning of predefined rules, groups,
and policies to facilitate the monitoring. Predefined rules and groups are system-specific, and
each system has its own rules and groups. All MAPS configuration is persistent, is retained
across reboots or HA failovers, and can be uploaded and downloaded.

MAPS predefined policies are:


򐂰 dflt_conservative_policy – Contains rules with more lenient thresholds that allow a
buffer and do not immediately trigger actions. Use this policy in environments where the
elements are resilient and can accommodate errors.
򐂰 dflt_moderate_policy – Contains rules with threshold values between aggressive and
conservative policies.
򐂰 dflt_aggressive_policy – Contains rules with stringent thresholds.
򐂰 dflt_base_policy – Contains rules based on the features that can be monitored without a
Fabric Vision license.

MAPS can be configured and managed using the Command Line Interface (CLI).

For complete instructions on Configuring MAPS, download the Fabric OS MAPS User Guide,
9.0.x at: https://docs.broadcom.com/doc/FOS-90x-MAPS-UG

MAPS can also be configured and managed through SANnav MAPS policy management.
This allows an administrator to create new MAPS policies, rules, and custom groups; and
apply and activate MAPS policies across one or more switches of a fabric. You can import
and modify existing MAPS policies. You can also view MAPS-enabled switches, policies and
rules, fabrics and groups to which the switches belong, and switch configuration actions.

Follow these steps to manage MAPS through SANnav.

232 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch09.fm

1. Click SANnav in the navigation bar, and then select SAN Monitoring > MAPS Policy
Management. The Switches window is displayed in Figure 9-3.

Figure 9-3 Web tools element log in screen

2. Click the Switches tab to view the list of MAPS-enabled switches with an active policy
name, policy type, and the fabric to which they belong, and switch configuration actions.
3. Click the Policies tab to view the policies applied to the switch.
4. Click the Groups tab to view the preconfigured groups available on the switch. You can
create user-defined custom groups only for ports, SFPs, and circuits.

For complete instructions on working with SANnav to manage MAPS, download the Brocade
SANnav Management Portal User Guide, 2.1.0x at
https://docs.broadcom.com/doc/SANnav-21x-MP-UG and refer to the MAPS management
section.

9.3.2 Flow Vision


Flow Vision provides a comprehensive vision of and deep insight into fabric traffic flows, along
with the ability to nondisruptively create and capture copies of traffic flows for analysis of
traffic flows, bottlenecks, bandwidth utilization, and similar fabric connectivity functionality.
Flow Vision also provides a test flow generation capability that you can use to pretest a SAN
infrastructure for robustness. This test flow generation capability is also useful for testing the
internal connections on a switch before deploying the switch into a production environment. In
addition, Flow Vision allows you to test for fabric connectivity issues, such as slow drain,
bandwidth utilization, and similar issues.

Automatic monitoring (introduced with FOS 9.0.0) provides I/O monitoring to the SANnav
administrator. With these IO Insight capabilities, Flow Vision provides proactive, non-intrusive,
real-time monitoring and alerting of storage I/O health and performance. The additional
visibility delivers deep insights into problems and concerns affecting the ability to maintain
service levels.

Chapter 9. Fabric OS administrative tools and diagnostics 233


8497ch09.fm Draft Document for Review April 5, 2021 12:31 pm

Integration with MAPS allows the automatically detected flows to be monitored using the
predefined rules. Hardware realtime violations are flagged at the I/O (ECT and FRT) and FC
level (frame latency). For details on MAPS, see Brocade Fabric OS MAPS User Guide, 9.0.x
at https://docs.broadcom.com/doc/FOS-90x-WebTools-UG.

Automated monitoring is supported by SANnav. For details on SANnav, see Brocade SANnav
Management Portal User Guide, 2.1.x f.

The SANnav Management Portal User Guide, 9.0.x can be found at


https://docs.broadcom.com/doc/SANnav-21x-MP-UG

Flow Vision includes:


򐂰 Flow Monitor
Flow Monitor running FOS 9.0.0 provides automated monitoring, which enables you to
learn SCSI and NVMe traffic and obtain Read/Write Frame Count and Read/Write Data
statistics. A cohesive view of metrics provides you all the necessary information in one
output.
To provide automated non-intrusive monitoring in FOS 9.0.0, a new predefined flow
command has been introduced. When you upgrade to FOS 9.0.0, the flow
sys_flow_monitor is activated.
Once activated, the flow enables automatic learning for Gen 6 and 7 switches/blades
running FOS 9.0.0 and automatically clears the associated flow statistics and force
deletes all user-defined flows. The predefined flow automatically discovers application
flows based on the direction and port parameters. With direction either ingress or egress,
this flow scan gathers DID, SID, SFID, and Protocol details.
򐂰 Flow Generator
Flow Generator is a test traffic generator that allows you to pretest a SAN infrastructure
(including internal connections) for robustness before deploying it.
Flow Generator lets you perform the following:
– Configure a 32Gb/s or 64Gb/s Fibre Channel-capable port as a simulated device that
can transmit frames at a full 16Gb/s, 32Gb/s, or 64Gb/s line rate.
– Emulate a 32Gb/s or 64Gb/s SAN without actually having any 32Gb/s or 64Gb/s hosts
or targets or SAN testers.
– Pretest the entire SAN fabric at the full line rate, including optics and cables on ISLs as
well as internal connections within a switch.
Flow Generator uses simulation mode (SIM) ports. SIM ports behave like normal F_Ports,
but are used only for testing. By using SIM ports, Flow Generator traffic is terminated at
the destination port and does not leave the fabric.
Flow Generator can generate standard frames or create custom frames with sizes and
patterns that you specify. A sample use case would be to create a traffic flow from a
Source ID (SID) to a Destination ID (DID) to validate routing and throughput. Creating a
Flow From a Specific Source ID to a Specific Destination ID provides an example of the
command and the results for this use case.
Flow Generator supports predefined flows to generate traffic between all configured SIM
ports.
After you activate a Flow Generator flow, the flow stays active until you deactivate the flow.
The flow stays active even though the SID and DID SIM-ports are offline. As soon as SID
and DID SIM-ports are online, traffic starts.

234 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch09.fm

Flow Generator also supports generating simulated traffic to a routed fabric. To generate
traffic, you specify the imported proxy ID or the real PWWN of the device. Flow Generator
support is similar to Flow Monitor support, except Flow Generator supports only flows
from edge to edge but not from backbone to edge or from edge to backbone.
The Brocade Monitoring and Alerting Policy Suite (MAPS) can be used to monitor SIM
port traffic thresholds while Flow Generator is running. MAPS treats SIM ports as F_Ports,
so MAPS can issue warnings on these ports if threshold values are triggered. If you do not
want to see MAPS warnings for SIM ports, you must disable MAPS monitoring for those
ports.
򐂰 Flow Mirror

Important: Do not use Flow Generator in an active production environment. Traffic


created by Flow Generator can saturate the links, which might have a negative impact
on any other traffic sharing the same links.

As storage networks get larger and more complicated, non-intrusive diagnostic tools are
becoming increasingly important to help identify problems without disturbing the operating
fabric. Flow mirroring is a diagnostic feature within Flow Vision that addresses this need.
With Flow Mirror, you can do the following:
– Non-disruptively create copies of application flows that can optionally be captured for
deeper analysis.
– Define a traffic pattern and create a real-time copy of this traffic, allowing you to
analyze a live system without disturbing existing connections. You can also use this
feature to view traffic passing through a port.
– Use a MAPS logical port group for either an ingress or egress port. Note that logical
port groups are not supported on CPU flow mirroring (CFM) or local flow mirroring
(LFM).
– Use a predefined flow with the flow --show command to mirror SCSI command, first
response, status, and ABTS frames from all Gen 6 F_Ports on a switch. For example:
flow --show sys_analytics_vtap .
Flow Mirror duplicates the specified frames in a user-defined flow and sends them to a
sink. This sink could be either:
– The local switch control central processor unit (CPU); this form is called “CPU flow
mirroring” (CFM), and has a limit of 256 frames per second.
– An analyzer/packet sniffer connected through a port in the metaSAN. The bandwidth
limit for a flow using this type of link is the bandwidth of the mirror destination port. This
form is called “Local flow mirroring” (LFM), and mirrors the flow to a port on the same
physical switch. This requires that a loopback SFP be plugged in at the other end of the
analyzer or on the port configured as a mirror port, which must be in the same domain.
All forms of mirroring are mutually exclusive. Only one form of mirroring (LFM, CFM, or
RFM) can be active at a time.
Flow Mirror flows can be in an active or inactive state. If the mirror flow is “active”,
mirroring starts immediately; if the flow is “inactive”, the flow must be activated (by using
the flow --activate command) for mirroring to start. Mirrored flows can be unidirectional or
bidirectional. A sample use case would be to mirror the traffic flow from a slow-draining
F_Port to see what is causing this condition.
For complete instructions using SANnav to create and manage Flow Vision Flows,
download the Brocade SANnav Flow Management User Guide, 2.1.0x. at:
https://docs.broadcom.com/doc/SANnav-21x-Flow-UG

Chapter 9. Fabric OS administrative tools and diagnostics 235


8497ch09.fm Draft Document for Review April 5, 2021 12:31 pm

9.3.3 ClearLink Diagnostics


Brocade ClearLink Diagnostic Port (D_Port) mode allows you to convert a Fibre Channel port
into a diagnostic port for testing traffic and running electrical loopback and optical loopback
tests. The test results can be very useful in diagnosing a variety of port and link problems.

ClearLink D_Port testing can be used to exercise the hardware before deploying ports into the
fabric. For example, you might install new SFPs to link two switches or to link a host/target to
a switch, and you want to exercise the hardware before adding them to the fabric. If the test
passes, you can go ahead and add the ports to the fabric. If the test fails, you can fix the
problem accordingly.

When D_Port testing is in progress, SANnav Management Portal changes the port type for all
selected ports and associated attached ports to D-Port for the duration of the test. A port in
D_Port mode does not carry any user traffic and is designed to run only specific diagnostics
tests for identifying link-level faults or failures. When the test is complete, SANnav changes
the port type back to the original port type.

The following tests are performed:


򐂰 Electrical loopback test
򐂰 Optical loopback test
򐂰 Link traffic test
򐂰 Link latency and distance measurement

Test results are presented after each test completes.

D_Port testing is supported on the following port types:


򐂰 E_Ports
򐂰 F_Ports
򐂰 N_Ports

Starting D_Port Testing


D_Port testing is useful if you want to diagnose port and link problems or if you want to
exercise hardware before deploying ports into the fabric.

Requirements:
򐂰 You must have Troubleshooting privilege with read-write permission.
򐂰 Both the source and destination ports must be managed by SANnav Management Portal.

Once you start a D_Port test, you cannot stop it.

When you run a D_Port test, SANnav Management Portal changes the port type for all
selected ports and associated attached ports to D_Port for the duration of the test. This may
cause the fabric to segment. When the test is complete, SANnav changes the port type back
to the original port type.
1. Click Inventory in the navigation bar, and then select Switch Ports from the drop-down
list.
2. Select the ports on which you want to perform the diagnostic tests.
– To select a single port, click the down arrow in the rightmost column for the switch port,
and select Run Diagnostics.
– Alternatively, to select multiple ports, use the bulk select option.

236 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch09.fm

i. Click the more button ( ), and select Bulk Select.


A column of checkboxes displays on the leftmost side of the table.
ii. Select the checkboxes for the ports on which you want to run diagnostic tests, and
then click Actions > Run Diagnostics in the upper-right corner of the window.
If Run Diagnostics is grayed out, one or more of the selected ports are not supported for
running diagnostic tests.
3. Click OK in the confirmation dialog.
4. Configure the test parameters, or accept the default values.

Figure 9-4 Run Diagnostics - test parameters

a. Specify either the number of frames to send (in millions) or the time duration for which
the frame traffic test will run. The default is to send one million test frames.
b. In the Frame Size field, enter the size of the test frames that are generated to run the
test. The default value is 1024.
Note that for Fabric OS 9.0.x and higher, the test uses a test frame size that is a
multiple of four. If you specify a size that is not a multiple of four, when the test is run,
the nearest higher multiple of four is used as the test frame size. For example, if you
specify a test frame size of 37, the tests are run with a test frame size of 40.
c. Select a predefined pattern to be used in the payload, or enter a user-defined pattern in
decimal. The default is to use the pattern jCRPAT.
You can optionally select the checkboxes at the bottom of the dialog. Note that FEC is
not supported on D_Ports that are configured with dense wavelength division
multiplexing (DWDM). For 32Gbps ports, FEC is automatically enabled and cannot be
disabled.
5. Click Next.
The diagnostic test starts running. A dialog displays the status of the test.

Chapter 9. Fabric OS administrative tools and diagnostics 237


8497ch09.fm Draft Document for Review April 5, 2021 12:31 pm

Figure 9-5 Run Diagnostics - test status

Testing might take a few minutes to several hours, depending on the selected parameters.
6. Click Close. You can view this status on the Outputs page.
7. Click Done in the confirmation dialog to return to the Inventory page. Note that the port
type for the selected ports has been changed to D-Port.

To view the test results, click Outputs in the subnavigation bar, and select Diagnostics from
the drop-down list.

Scheduling D_Port Testing


You can schedule D_Port testing to run at some time in the future. Since D_Port testing may
cause the fabric to segment, you might want to schedule the test at a time when traffic is
lighter, such as in the middle of the night.

Requirements:
򐂰 You must have Troubleshooting privilege with read-write permission.
򐂰 Both the source and destination ports must be managed by SANnav Management Portal.

Procedure:
1. Click Inventory in the navigation bar, and then select Switch Ports from the drop-down
list.
2. Select the ports on which you want to perform the diagnostic tests.
– To select a single port, click the down arrow in the rightmost column for the switch port,
and select Schedule.
– Alternatively, to select multiple ports, use the bulk select option.
i. Click the more button ( ), and select Bulk Select. A column of checkboxes
displays on the leftmost side of the table.
ii. Select the checkboxes for the ports on which you want to run diagnostic tests, and
then click Actions > Schedule in the upper-right corner of the window.
If Schedule is grayed out, one or more of the selected ports are not supported for running
diagnostic tests.
3. Select Diagnostics from the drop-down list, select the date and time for the test to start,
and click OK.

238 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch09.fm

Note that if Diagnostics is grayed out, one or more of the selected ports are not
supported for running diagnostics tests.

Figure 9-6 Schedule - Diagnostics

4. Click OK in the confirmation dialog.


5. Configure the test parameters or accept the default values, and click Next.
6. Click Done in the confirmation dialog to return to the Inventory page.
You can view the scheduled test by clicking Outputs in the subnavigation bar and then
selecting Diagnostics from the drop-down in the upper-right corner of the page.

Figure 9-7 Dashboard & Reports - Inventory Output

From the action menu of the scheduled test, you can display the ports selected for the test,
edit the schedule, and delete the test.

Viewing D_Port Test Results


While D_Port tests are running and after the tests are complete, you can view the results from
the Diagnostics section of the Outputs tab.

Only the user who ran or scheduled the test can view, delete, or re-run the test.
1. Click Inventory in the navigation bar, and then click Outputs.

Chapter 9. Fabric OS administrative tools and diagnostics 239


8497ch09.fm Draft Document for Review April 5, 2021 12:31 pm

2. Select Diagnostics from the drop-down list in the upper-right corner.


3. Locate the test that you want to view, and click View from the action menu in the rightmost
column.
Note that the View option is available only for tests that have completed or are in progress.
It is not available for scheduled tests.
If you used the Bulk Select option to run diagnostics on several ports at the same time,
only one entry exists in the Outputs page for those tests.

Figure 9-8 Dashboard & Reports - Inventory Output with Bulk Select option

4. Click Details to see the detailed results of the test.


The following screen capture shows the test results when the Bulk Select option is used
to run diagnostics on multiple ports.
Note that you can re-run a test on the same port by selecting Re-run from the action menu
for a completed test in the Outputs page.

Figure 9-9 Dashboard & Reports - Inventory Output Details

For complete instructions using SANnav to manage ClearLink diagnostics download the
Brocade SANnav Management Portal User Guide, 2.1.0x and refer to the Troubleshooting
and Diagnostics section.

240 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch09.fm

The Brocade SANnav Management Portal User Guide, 2.1.0x can be found at
https://docs.broadcom.com/doc/SANnav-21x-MP-UG

9.3.4 Fabric Vision SANnav Managed Features

Dashboards
Customizable health and performance dashboard that provides at-a-glance views of switch
status and various conditions that are contributing to performance issues, enabling
administrators to get instant visibility into any hot spots at a switch level and take corrective
actions. Dashboard views include:
򐂰 Overall status of the switch health and the status of each monitoring category, including
any out-of-range conditions and the rules that were triggered.
򐂰 Historical information on the switch status for up to the last seven days; automatically
provides raw counter information for a variety of error counters. This integrated dashboard
view also provides a single collection point for all dashboard data from a fabric for a
specific application flow

Configuration and Operational Monitoring Policy Automation Services


Suite (COMPASS)
Simplifies deployment, safeguards consistency, and increases operational efficiencies of
larger environments with automated switch and fabric configuration services. Administrators
can configure a template or adopt an existing configuration as a template and seamlessly
deploy the configuration across the fabric. In addition, they can ensure that settings do not
drift over time with COMPASS configuration and policy violation monitoring within SANnav
dashboards.

Fabric Performance Impact (FPI)


Leverages predefined MAPS policies to automatically detect and alert administrators to
different congestion severity levels and to identify creditstalled devices (also known as
slow-drain devices) or oversubscribed ports that could impact network performance. This
feature pinpoints exactly which devices are causing or are impacted by congested ports, and
it automatically quarantines credit-stalled devices to prevent buffer credit starvation.

Allows you to monitor the congestion-related issues on all physical E_Ports and F_Ports at all
times. Specifically, FPI monitors abnormal device latency and oversubscription issues.

9.3.5 IO Insight
Proactively monitors I/O performance and behavior through integrated network sensors to
gain deep insight into problems and ensure service levels. This capability automatically learns
the traffic flows to a fabric non-disruptively and non-intrusively gathers I/O statistics for both
SCSI and NVMe traffic from any device port on a Gen 6 and Gen 7 Fibre Channel platform;
and then it applies this information within an intuitive, policy-based monitoring and alerting
suite to configure thresholds and alarms. Integrated application-level and device-level I/O
latency and IOPS monitoring provide the ability to baseline application performance and
detect degraded performance. This enables administrators to proactively control performance
and availability to ensure operational stability.

Key capabilities include:


򐂰 Monitors individual host or storage devices to gain deeper insight into the performance of
the network to maintain SLA compliance.

Chapter 9. Fabric OS administrative tools and diagnostics 241


8497ch09.fm Draft Document for Review April 5, 2021 12:31 pm

򐂰 Obtains total I/Os, first response time max/average, I/O latency (Exchange Completion
Time, or ECT) max/ average, and outstanding I/Os total/ average performance metrics for
a specific host or storage for a LUN or NSID in order to diagnose I/O operational issues.
򐂰 Enables tuning of device configurations with integrated I/O metrics to optimize storage
performance.

9.3.6 VM Insight
Seamlessly monitors VM performance throughout a storage fabric with standards based,
end-to-end VM tagging. Administrators can quickly determine the source of VM/application
performance anomalies, as well as provision and fine-tune the infrastructure based on
VM/application requirements to meet service level objectives.

9.3.7 Fabric Vision Diagnostics


FOS 9.0.x has the following diagnostics that don’t require configuration or management.

Forward Error Correction (FEC)


Provides a data transmission error control method by including redundant data (error
correcting code) to ensure error-free transmission on a specified port or port range. This
enables recovery from bit errors in Fibre Channel links, negating the need for retransmission.
򐂰 FEC is always active on ports operating at 64Gb/s and 32Gb/s speed based on the
standard, regardless of the FEC settings on the switch.
򐂰 FEC is supported on E_Ports on 16Gb/s-capable switches at 10G/16G speeds.
򐂰 FEC is supported on the N_Ports and F_Ports of an Access Gateway using RDY, Normal
(R_RDY), or Virtual Channel (VC_RDY) flow control modes.
򐂰 FEC is supported on F_Ports if the device attached to it supports FEC.
򐂰 FEC is enabled by default.
򐂰 FEC enables automatically when negotiation with a switch detects FEC capability.
򐂰 FEC persists after driver reloads and system reboots.
򐂰 FEC functions with features, such as QoS, trunking, and BB_Credit recovery.

Buffer Credit Recovery


Buffer credit recovery allows links to recover after buffer credits are lost when the buffer credit
recovery logic is enabled. The buffer credit recovery feature also maintains performance. If a
credit is lost, a recovery attempt is initiated. During link reset, the frame and credit loss
counters are reset without performance degradation.
򐂰 Credit recovery is supported on E_Ports, F_Ports, and EX_Ports. Buffer credit recovery is
enabled automatically across any long-distance connection for which the E_Port, F_Port,
or EX_Port buffer credit recovery mechanism is supported
򐂰 For 64Gb/s (Gen 7) and 32Gb/s (Gen 6) FC switches and blades, use the
portCfgCreditRecovery command to disable or enable buffer credit recovery on a port.

For additional information on Forward Error Correction (FEC) and Buffer Credit Recovery,
download the Fabric OS Administrator’s 9.0.0 Guide at
https://docs.broadcom.com/doc/FOS-90x-Admin-AG

242 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch10.fm

10

Chapter 10. IBM b-type Gen 7 SAN


automation
Most, if not all, IT administrators have first-hand experience in the growing complexity of
managing enterprise infrastructure, including SANs. “The cost and complexity of protecting
and storing data is increasing, and IT leaders are responding with attempts to better optimize
and automate storage—but they need better tools,” according to a report from Enterprise
Strategy Group.

IBM b-type Gen 7 is in the unique position to spot and understand the impact of automation
helping organizations get more from their SAN infrastructure.

IBM b-type Gen 7 offers a combination of SAN automation with RESTful APIs and a SAN
management platform to help organizations drive greater efficiency from their SANs.

This is accomplished through a variety of means:


򐂰 IBM b-type Gen 7 SAN automation is provided with a multilayer architecture.
򐂰 RESTful API support on directors and switches, as well as on management tools.
򐂰 IBM b-type Gen 7’s Ansible management framework, designed to eliminate repetitive
tasks, simplify management and orchestrate across the full SAN infrastructure.

This chapter includes the following topics:


򐂰 10.1, “Motivation to automate” on page 244
򐂰 10.2, “Overview of the REST API” on page 244
򐂰 10.3, “Simple Automation Example” on page 247
򐂰 10.4, “Ansible as an alternative” on page 249
򐂰 10.5, “SANnav’s REST API” on page 250
򐂰 10.6, “Conclusion” on page 250

© Copyright IBM Corp. 2021. 243


8497ch10.fm Draft Document for Review April 5, 2021 12:31 pm

10.1 Motivation to automate


Organizations should embrace SAN automation for the following five reasons:
򐂰 Reducing human error and streamlining operational processes has never been more
crucial. As organizations move to digitize and adapt to new workloads, data availability,
processing time and agility to provision applications on-demand become the life-blood of
the business. This demands a more efficient and expedient infrastructure management
approach, leaving no room for human error. As a result, storage administrators need to be
freed from repetitive manual tasks such as configuration management, reporting,
documenting inventory and troubleshooting. Instead, IT organizations need SAN
automation to help them automate and orchestrate repetitive tasks, significantly improve
efficiency, and decrease the risk of operational mistakes.
򐂰 Demand for more accurate and more frequent infrastructure reports is on the rise. It’s not
just IT managers who crave information on storage performance, utilization and
forecasting, business stakeholders are also asking for and expecting this data on demand.
No one wants to wait for a slot when storage administrators can allocate time to produce a
report. This information should be available as frequently as business demands dictate—
all at the click of a button. Automation not only provides this kind of responsiveness that
traditional manual storage management processes can’t deliver, but it can also be
customized so that each stakeholder is getting more accurate data aligned to their
responsibility.
򐂰 SAN configuration management must be streamlined. With more enterprise applications
demanding access to more data and with more virtual machines, deploying and
configuring servers, storage and the network has become more time-consuming and more
complex than ever. By streamlining SAN operations through automation, application
provisioning workflows are simplified across hypervisor, network and storage, delivering
agility and responsiveness to meet dynamic business demands.
򐂰 IT service delivery is not always as responsive as the business demands. As
organizations become increasingly reliant on having world-class IT services available to
them for proactive, agile business decision-making, it is essential that bottlenecks to IT
service delivery be identified and eliminated immediately. This has to be done without
hiring more storage administrators or boosting SAN-related Capex spending, making SAN
automation the only viable option to drive increased IT agility and more tightly align IT
service delivery with fast-changing business needs.
򐂰 Consistent configuration validation is a must. Manual configuration changes are taking
place with greater frequency as enterprises diversify their IT architectures in general, and
their storage architectures specifically. SAN automation ensures validation of consistent
configuration parameters across the different SAN fabrics in order to facilitate
troubleshooting of frequent alerts without reliance on manual intervention.

IBM b-type Gen 7 automation solutions leverage RESTful APIs to facilitate solutions
architecture, share best practices and get to production status faster.

10.2 Overview of the REST API


The FOS REST API is a programmable web service interface for IBM b-type Gen 7 Fabric OS
that can manage IBM b-type Gen 7 SAN switches across a fabric. This API uses standard
HTTP methods to perform Create, Read, Update, and Delete (CRUD) operations on the fabric
configuration data, and provides an interface for provisioning, status, and validation
operations using the YANG data model described in the YANG 1.1 RFC, but not the datastore

244 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch10.fm

managed with NETCONF. An Apache web server embedded in Fabric OS is used to serve
the API.

The RESTful API approach lets you think of a network device as a web server. By using
standard web-based tools, an automation can send and receive transactions to or from a
network device just as it would send transactions to and from a website. This means that
transactions take place over a secure socket using HTTP rules to handle the exchange. The
data appears in the form of XML or JSON depending on the RESTful API services
implemented on the networking device.

To interact with a SAN (or other) device, you need to consult its RESTful API reference to
learn, among other things, what “Uniform Resource Identifiers” (URIs) you need to use.
(Simply put, URIs are identifiers that can be used as part of a web address.) According to the
documentation, the URL for accessing a listing of the zones in the active configuration is as
follows:

GET <base_URI>/running/zoning/defined-configuration/

The model that is used to represent state and configuration information is expressed in a
modeling language called Yang. Yang describes the structure of the different elements inside
the model and is used to describe whether each element is read-only or read-write. It
describes the type of data that the element can hold, such as string or integer, and it shows
the relationship among various elements, the other nested elements they contain, their peer
elements, and the parent elements that contain them. Here is a segment of the description of
a zone in Yang:
list zone {
key "zone-name";
description
"List of the members in the zone. The members can only be identified as a
WWN, domain, index, or a zone alias.";
leaf zone-name {
type zoning-name-type;
description
"The zone name.";
}
leaf zone-type {
type zone-type-type;
description
"The zone type. Not that target zone types cannot be created or modified
(only deleted).";
}
container member-entry {
description
"The zone member.";
leaf-list entry-name {
type zone-member-type;
min-elements 1;
description
"List of the members in the zone. The members can only be identified as a
WWN, domain, index, or zone alias.";
}
leaf-list principal-entry-name {
when "../..zone-type=1 or ../../zone-type=2";
type zone-member-type;
min-elements 1;

Chapter 10. IBM b-type Gen 7 SAN automation 245


8497ch10.fm Draft Document for Review April 5, 2021 12:31 pm

description
"List of the principal members in the peer zone. The members can only be
identified as a WWN, domain,index, or zone alias.";
}
}
}

Ordinarily more information goes into a Yang module such as revisioning and governance
information; this listing omits them for brevity. Thus, the Yang description is complete, but it’s
also wordy. Although this precision is necessary when interacting with the model
programmatically, it’s sometimes useful to get a global view of the abstraction provided by the
model to see how the data is structured.

An open source tool called pyang can parse the Yang model and produce a tree that
represents the elements in the model. The listing includes information about each element,
such as whether it’s read-only or read write, a list, optional, or nested. Here is the
representation of the zoning model in tree form:
module: IBM b-type-zone
+--rw IBM b-type-zone
+--rw defined-configuration
| +--rw cfg* [cfg-name]
| | +--rw cfg-name zoning-name-type
| | +--rw member-zone
| | +--rw zone-name* zoning-name-type
| +--rw zone* [zone-name]
| | +--rw zone-name zoning-name-type
| | +--rw zone-type? zone-type-type
| | +--rw member-entry
| | +--rw entry-name* zone-member-type
| | +--rw principal-entry-name* zone-member-type
| +--rw alias* [alias-name]
| +--rw alias-name zoning-name-type
| +--rw member-entry
| +--rw alias-entry-name* union
+--rw effective-configuration
+--rw cfg-name? zoning-name-type
+--rw checksum? string
+--rw cfg-action? uint8
+--rw default-zone-access? uint8
+--ro db-max? uint32
+--ro db-avail? uint32
+--ro db-committed? uint32
+--ro db-transaction? uint32
+--ro transaction-token? uint32
+--ro db-chassis-wide-committed? uint32
+--ro enabled-zone* [zone-name]
+--ro zone-name zoning-name-type
+--ro zone-type? zone-type-type
+--ro member-entry
+--ro entry-name* union
+--ro principal-entry-name* union

246 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch10.fm

10.3 Simple Automation Example


In this example, the <base_URI> is http://<our device IP address>/rest. Begin by creating a
login session with a switch in the fabric by executing the following command (which you’d type
as a single line):
curl -X POST -v -u admin:password http://10.18.254.37/rest/login

curl is the name of the command.


򐂰 X POST specifies the POST HTTP method (instead of GET).
򐂰 v specifies verbose output to access the authorization string in the header of the response
used in the next step.
򐂰 u admin:password specifies the credentials to use.

The last parameter is the Uniform Resource Identifier (URI) for curl to use to login. (URI value
is described in the RESTful API reference.)

This command establishes the session used for the following commands. The following is a
trace of its execution:
* Trying 10.18.254.37...
* Connected to 10.18.254.37 (10.18.254.37) port 80
(#0)
* Server auth using Basic with user 'admin'
> POST /rest/login HTTP/1.1
> Host: 10.18.254.37
> Authorization: Basic YWRtaW46cGFzc3dvcmQ=
> User-Agent: curl/7.47.0
> Accept: */* >
< HTTP/1.1 200 OK
< Date: Wed, 31 Jan 2018 16:01:24 GMT
< Server: Apache
< Authorization: Custom_Basic
YWRtaW46eHh4OjNkYTllZmM3NzMxYjk4OGU2ODg1YzZkMGRjNWJlM
zMyNjBhZDYxZThkOWQ2MWMxNzNiMGVlMjU3YmM2OTcyYjA=
< Cache-Control: no-cache
< X-Frame-Options: DENY
< Content-Secure-Policy: default-src 'self'
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: application/yang-data+xml

Next, you perform a GET of the URI to return the current configuration using the
Custom Basic value returned from the login for authentication:
curl -v -H "Authorization: Custom_Basic
YWRtaW46eHh4OjNkYTllZmM3NzMxYjk4OGU2ODg1YzZkMGRjNWJlM
zMyNjBhZDYxZThkOWQ2MWMxNzNiMGVlMjU3YmM2OTcyYjA="
http://10.18.254.37/rest/running/zoning/defined-configuration

By default, curl uses the GET method, so you don’t need to specify it. -H
“Authorization: Custom_Basic YWR...jA=” is the authentication and session
identifying string returned in the previous command. -H places the string into the
GET request header as seen in the following trace:

Chapter 10. IBM b-type Gen 7 SAN automation 247


8497ch10.fm Draft Document for Review April 5, 2021 12:31 pm

* Trying 10.18.254.37...
* Connected to 10.18.254.37 (10.18.254.37) port 80
(#0)
> GET /rest/running/zoning/defined-configuration
HTTP/1.1
> Host: 10.18.254.37
> User-Agent: curl/7.47.0
> Accept: */*
> Authorization: Custom_Basic
YWRtaW46eHh4OjNkYTllZmM3NzMxYjk4OGU2ODg1YzZkMGRjNWJlM
zMyNjBhZDYxZThkOWQ2MWMxNzNiMGVlMjU3YmM2OTcyYjA=
>
< HTTP/1.1 200 OK
< Date: Wed, 31 Jan 2018 16:09:39 GMT
< Server: Apache
< Cache-Control: no-cache
< X-Frame-Options: DENY
< Content-Secure-Policy: default-src 'self'
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: application/yang-data+xml <
<?xml version="1.0"?>
<Response>
<defined-configuration>
<cfg>
<cfg-name>CFG_FABRIC_A</cfg-name> <member-zone> <zone-name>CLUSTER1</zone-name>
<zone-name>Z_AIXHOST_FCS2_VMAX01_SN1234_9F0 </zone-name>
...
<alias> <alias-name>esx66_5d3d00</alias-name> <member-entry>
<alias-entry-name>10:00:8c:7c:ff:5d:3d:00 </alias-entry-name>
</member-entry>
</alias>
</defined-configuration>
</Response>
* Closing connection 0

The results appear as an XML data segment structured according to the description in the
Yang model, and so it’s important to have access to that model along with the RESTful API
manual. The models can be found on GitHub as a repository among those in IBM b-type’s
repositories at http://github.com/IBM b-type/yang. The RESTful API manual can be
retrieved from the Broadcom web site. Having retrieved the zoning information from the
fabric, you should close the session using the CLI command (the results are omitted to save
space):
curl -v -H "Authorization: Custom_Basic
YWRtaW46eHh4OjNkYTllZmM3NzMxYjk4OGU2ODg1YzZkMGRjNWJlM
zMyNjBhZDYxZThkOWQ2MWMxNzNiMGVlMjU3YmM2OTcyYjA=" http://10.18.254.37/rest/logout

In version 8.2.2 or later of FOS, the REST API session-less operation allows you to provide
authentication credentials directly for each GET request. Essentially, FOS REST API
session-less operation completes the login, GET operation, and logout as one complete
request. You can only use basic authentication formats for REST API session-less operation,
which includes plain text or Base64.

248 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch10.fm

The following example shows a GET request using plain text authentication:
curlysw -u admin:password http://10.155.2.190/rest/running/IBM
b-type-media/media-rdp>"

10.4 Ansible as an alternative


The previous section shows an example of an approach that uses a procedural methodology.
The workflow starts at the beginning, executes a series of steps and then terminates. Most
traditional programs work this way.

Ansible takes a declarative approach. Rather than provide sequential steps, Ansible
describes each of the hosts in an inventory. The description appears in a document called a
playbook. For example, rather than provide steps to install a particular application, Ansible
describes a host state where the application is already installed. When you run the playbook,
Ansible takes no action if the application is already installed. If the application isn’t installed,
Ansible calls installation routines so the host is brought into the desired state without requiring
the administrator to write any specific steps.

In the realm of storage networks, the use of a declarative language means you can describe
switches and fabrics where, for example, a zone is already configured with the proper hosts
and storage arrays. When you run the Ansible playbook, those zones are defined as needed,
and the hosts and storage arrays are added to them if necessary.

With some other declarative automation utilities, it’s necessary to install an agent on each
host that the utility manages. This agent retrieves the commands from a command center and
runs them on the local host. Ansible is unique in that it doesn’t require agents. In order to
make switch state changes, Ansible establishes a secure shell session to a proxy and sends it
a small Python script. The script carries out the necessary operations using the switch API
and removes itself from the host.

You need two different skill sets to successfully implement an Ansible solution. First, you must
understand the most common playbook operations. These operations are coded and installed
for use by the playbooks. As vendors announce support for Ansible, they also provide script
libraries for the most common tasks. If there is a task that is required but isn’t available in the
official Ansible distribution, the open source community may provide code for that task in
publicly available repositories.

Second, you must understand your business needs to provide ongoing playbook
development. The person maintaining the playbooks doesn’t need to be a programmer and
doesn’t need to know how remote system operations occur. That person only needs to know
the desired outcomes and should be able to construct play- books in YAML — the markup
language used by Ansible. This is an example of an Ansible playbook:
---
- hosts: edgeSwitches
vars_files:
../fos_passwords.yml
gather_facts: False
tasks:
- name: run fos commands
IBM b-type_fos_command:
switch_login: “{{switch_admin_account}}”
switch_password: “{{switch_password}}”
switch_address: “{{switch_ip_address}}”

Chapter 10. IBM b-type Gen 7 SAN automation 249


8497ch10.fm Draft Document for Review April 5, 2021 12:31 pm

command_set:
- command: alicreate "SampleAlias1", "10:23:45:67:76:54:32:10"

The three dashes at the beginning are part of the YAML specification. The hosts section
identifies automation target switches. You can keep sensitive information in a separate file, as
demonstrated by the fos_passwords.yml line. The name of a variable in double braces such
as {{switch_password}} indicates variable substitution. The variable file specified in vars_files
tells where to find external variables.

10.5 SANnav’s REST API


SANnav Management Portal and SANnav Global View provide end-to-end visibility into
enterprise SANs. These tools detect, analyze and take action based on SAN behavior and
performance, helping administrators get to the root of problems faster and remediate them
fully. Storage administrators can troubleshoot across the storage fabric in as little as 30
seconds. This is unprecedented with any other shared storage infrastructure architectures.

The SANnav REST API provides functionality that complements that found in the Fabric OS
REST APIs. The SANnav REST API includes support for the following SANnav features:
򐂰 List the fabrics managed by the SANnav server.
򐂰 List the seed and principal switch data for each fabric.
򐂰 List the switches (members) in the fabric.
򐂰 Display the FCR topology information for routing topologies such as edge-to-edge,
backbone-to-edge, and edge-to- backbone.
򐂰 Configure event-forwarding.
򐂰 Acknowledge or unacknowledge events.
򐂰 Display a list of filtered events.
򐂰 Perform an inventory search.

10.6 Conclusion
SAN automation is a critical element in IT modernization and digital transformation because it
helps organizations handle storage-related processes more efficiently without hiring more
administrators or adding to the storage infrastructure Capex budget. SAN automation is a
high-leverage approach to turning network storage into a strategic asset.

IBM b-type Gen 7’s commitment to SAN automation, combined with its long-standing
leadership in storage fabrics and technical innovation, makes it an ideal candidate for your IT
infrastructure automation strategy.

For detailed information on Automation, see Programming Guides at


https://www.broadcom.com/products/fibre-channel-networking/software/fabric-operati
ng-system#documentation.

250 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm

11

Chapter 11. IBM SANnav Management Suite


overview
This chapter describes how to deploy IBM SANnav Management Suite 2.1, in your SAN
Infrastructure.

This guide contains detailed steps for installing the IBM SANnav Management Portal and
SANnav Global View 2.1.

This guide describes all the system and server requirements before you start the IBM
SANnav Management suite installation. The following document lists the system and server
requirements for deployment of both SANav Management Portal and SANnav Global View.

Within this document, SANnav Management Portal might also be referred to simply as
"SANnav".

This chapter includes the following topics:


򐂰 11.1, “IBM SANnav Management Suite 2.1 release overview” on page 252
򐂰 11.2, “IBM SANnav Management Portal 2.1 release overview” on page 254
򐂰 11.3, “IBM SANnav Global View 2.1 release overview” on page 278
򐂰 11.4, “IBM SANnav Web Tools” on page 289
򐂰 11.5, “Reference guides” on page 291

© Copyright IBM Corp. 2021. 251


8497ch11.fm Draft Document for Review April 5, 2021 12:31 pm

11.1 IBM SANnav Management Suite 2.1 release overview


SANnav Overview SANnav is the next generation SAN management application suite for IBM
b-type Gen 7 SAN environments. SANnav allows you to efficiently manage your SAN
infrastructure through various easy-to-use functions. SANnav implements a highly scalable
client-server architecture for SAN management. With a modern browser-based UI, SANnav
eliminates the need for a Java-based thick client. The user interface of SANnav is designed
based on real-world use cases and user workflows, providing a highly intuitive user
experience. SANnav uses a micro-services-based architecture based on Docker container
technology that allows it to scale to meet the management needs of both small and large SAN
environments and those that may change over time. This scalable architecture also allows
SANnav to support new functionality in the future without causing degradation to the
performance of the application. To address the management needs of very large-scale SAN
environments or those that are distributed by function or location, SANnav supports a
hierarchical management model, where a higher-level “global” application, SANnav Global
View, provides comprehensive visibility, summarization, and seamless navigation across
multiple instances of the SANnav Management Portal application.

There are two distinct SANnav product offerings:


򐂰 IBM SANnav Management Portal
򐂰 IBM SANnav Global View

11.1.1 IBM SANnav Management Portal


SANnav allows you to efficiently manage your SAN infrastructure through various easy-to-use
functions. SANnav implements a highly scalable client-server architecture for SAN
management. With a modern browser-based UI, SANnav eliminates the need for a
Java-based thick client. The SANnav user interface is designed based on real-world use
cases and user workflows, providing a highly intuitive user experience. SANnav uses a
micro-services architecture based on Docker container technology that allows it to scale to
meet the management needs of both small and large SAN environments and those that may
change over time. This scalable architecture also allows SANnav to support new functionality
in the future without causing degradation to the performance of the application.

SANnav Management Portal allows management of one or more SAN fabrics that are in the
same or different geographical locations, and it supports a maximum of 15,000 physical SAN
ports. For environments that are larger than15,000 ports, you can deploy multiple SANnav
Management Portal instances. SANnav Management Portal does not replace Brocade Web
Tools or the Fabric OS command line interface.

For more information, see Brocade SANnav Management Portal User Guide, found at:
https://www.broadcom.com/products/fibre-channel-networking/software/sannav-managem
ent-portal.

11.1.2 IBM SANnav Global View


You can get a comprehensive view of a SAN environment that spans multiple SANnav
Management Portal instances through the SANnav Global View product. SANnav Global
View allows you to navigate seamlessly across multiple SANnav Management Portal
instances and drill down to any individual instance to perform detailed monitoring,
investigation, and troubleshooting.

252 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm

For more information, see Brocade SANnav Global View User Guide, found at:
https://www.broadcom.com/products/fibre-channel-networking/software/sannav-managem
ent-portal.

11.1.3 New Hardware Platforms supported in SANnav 2.1


SANnav 2.1.0 is a major new release of IBM b-type Gen 7 two Fibre Channel SAN
management software products, IBM SANnav Management Portal and IBM SANnav Global
View. SANnav Management Portal 2.1.x supports the introduction of IBM b-type Gen 7
platforms with Fabric OS 9.0.0, and adds new features and capabilities that make managing
IBM b-type Gen 7 SAN environments easier than ever before.

Supported hardware and software


SANnav Management Portal 2.1.x supports the following Fabric OS software versions and
hardware platforms.

Fabric OS software support


The following Fabric OS software versions are supported by this release of SANnav.
򐂰 Fabric OS 9.0 or later
򐂰 Fabric OS 8.0 or later
򐂰 Fabric OS 7.4 or later

Supported SAN switches


򐂰 IBM b-type Gen 7 (64Gb/s) Fixed-Port Switches
򐂰 IBM SAN64B-7 Gen 7 56-port Switch
򐂰 IBM b-type Gen 7 (64Gb/s) Directors
– IBM SAN256B-7 Gen 7 4-slot Director•
– IBM SAN512B-7 Gen 7 8-slot Director
򐂰 IBM b-type Gen 6 (32Gb/s) Fixed-Port Switches
– IBM SAN24B-6 Switch
– IBM SAN64B-6 Switch
– IBM SAN128B-6 Switch
– IBM SAN18B-6 Extension Switch
򐂰 IBM b-type Gen 6 (32Gb/s) Directors
– IBM SAN256B-6 Director
– IBM SAN512B-6 Director
򐂰 IBM b-type Gen 5 (16Gb/s) Fixed-Port Switches
– IBM 24B-5 Switch
– IBM SAN48B-5 Switch
– IBM SAN96B-5 Switch
– IBM SAN42B-R Extension Switch
򐂰 IBM b-type Gen 5 (16Gb/s) Directors
– IBM SAN384B-2 Director
– IBMSAN768B-2 Director
򐂰 IBM b-type Gen 4 (8Gb/s) Fixed-Port Switches
– IBM SAN24B-4 Switch•
– IBM SAN06B-R Extension Switch
򐂰 IBM b-type Gen 4 (8Gb/s) Directors
– • IBM SAN768B Director
– • IBM SAN384B Director

Switches That Support Data Streaming SANnav uses Kafka to provide real-time data
streaming from switches that are running Fabric OS 8.2.1a or higher. Data streaming provides
collection of high-frequency performance statistics, FCIP performance and error statistics,

Chapter 11. IBM SANnav Management Suite overview 253


8497ch11.fm Draft Document for Review April 5, 2021 12:31 pm

and flow statistics and violations. The following Gen 7 platforms support data streaming: •
IBM SAN64B-7 Switch • IBM SAN256B-7 and SAN512B-7 Directors

The following IBM b-type Gen 6 platforms support data streaming: • IBM SAN24B-6,
SAN64B-6, and SAN128B-6 Switches • IBM SAN18B-6 Switch • IBM SAN256B-6 and
SAN512B-6 Directors

11.2 IBM SANnav Management Portal 2.1 release overview


SANnav Management Portal allows management of one or more SAN fabrics that are in the
same or different geographical locations, and it supports managing a maximum of 15,000
physical SAN ports. For environments that are larger than 15,000 ports, you can deploy
multiple SANnav Management Portal instances, which in turn can be managed by SANnav
Global View. Use SANnav Management Portal to monitor and manage fabrics, switches,
switch ports, and other elements of your SAN. Dashboards provide summary status and
performance information, from which you can drill down to get detailed views. Using filters
and tags, you can sort and search your inventory to find exactly the information that you want.
A highly flexible reporting infrastructure allows you to generate custom graphical or tabular
reports. SANnav Management Portal does not replace Brocade Web Tools or the Fabric OS
command line interface.

11.2.1 Browser requirements


Any laptop or machine that launches web applications can be used to launch SANnav
Management Portal. For optimal performance, have at least 16 GB of memory. The following
browsers can be used to access the SANnav server:
򐂰 Chrome
򐂰 Firefox

Note that if you access the client from Remote Desktop, the user interface may have
degraded performance. Launching Brocade Web Tools 9.0.0 and later from a SANnav client
is supported on Chrome and Firefox. Launching Brocade Web Tools versions earlier than
9.0.0 is supported only on Firefox.

11.2.2 New SANnav Management Portal server platform support and


Infrastructure

SANnav deployment: requirements and scalability


SANnav Management Portal 2.1.0 or 2.1.0a can be deployed either on a single bare-metal
host or virtual machine (VM) or on a cluster of bare-metal servers/VMs. The following tables
provide details of server requirements.

For specific details on installing SANnav Management Portal on a virtual machine (VM)
please refer to the Brocade SANnav User Guide for detailed information.

Documentation is located at the following link:


https://www.broadcom.com/products/fibre-channel-networking/software/sannav-managem
ent-portal

254 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm

11.2.3 Server requirements details

Figure 11-1 VM or bare metal installation

Figure 11-2 Requirements for SANnav installation on a bare metal server or VM

11.2.4 SANnav Management Portal scalability 2.1.0

Figure 11-3 Scalability limits

11.2.5 Pre-installation requirements


Before you install IBM SANnav Management Suite, ensure that you meet the following
requirements:
򐂰 Downloading the SANnav Management Portal Software,
򐂰 SANnav Installation Pre-Reqs and migration Quicklist Checklists

Chapter 11. IBM SANnav Management Suite overview 255


8497ch11.fm Draft Document for Review April 5, 2021 12:31 pm

11.2.6 Downloading the SANnav Management Portal software


Complete the following steps to download the software and documentation from the Brocade
IBM assist website:
򐂰 IBM b-type Gen 7 Software/Firmware Download Instructions (IBM Assist Site for IBM
Brocade Software downloads)
򐂰 Step-by-Step instructions for access to Fabric OS firmware (FOS), SANnav and b-type
Gen 7 IBM Network Advisor (INA)

Important: You must have active IBM Support in order to access Brocade Software for
any device.

Brocade FOS and SAN Management software may be accessed through the IBM Support Fix
Central, found at: https://www.ibm.com/support/fixcentral/options
1. Navigate to https://www.ibm.com/support/fixcentral/options
2. Select “Select product” tab
3. Under “Product Group” dropdown, select “System Storage”
4. Under “Select from System Storage”, select “Storage area network (SAN)”
5. Under “Storage area network (SAN)”, select “SAN b-type” for Brocade switches or “SAN
management software” for IBM Network Advisor and SANnav (once released).
6. If “SAN b-type” is selected:
a. Under “Select from SAN b-type”, select the proper model.
b. Under “Installed Version”, select the proper FOS version.
c. Click the Continue box.
7. If “SAN management software” is selected:
a. Under “Select from SAN management software”, select the proper management
software.
b. Under “Installed Version”, select the proper version.
c. Click the Continue box.

256 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm

Figure 11-4 Fix Central

8. On the page “Select fixes”, select the appropriate fix-pack (FOS version) for your product
and select the download method.

Chapter 11. IBM SANnav Management Suite overview 257


8497ch11.fm Draft Document for Review April 5, 2021 12:31 pm

Figure 11-5 Fix Central - Select fixes

9. Select Download Options.

258 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm

Figure 11-6 Download options

10.Log in to IBM with your IBMid (if requested) and create an IBMid account (if necessary)

Chapter 11. IBM SANnav Management Suite overview 259


8497ch11.fm Draft Document for Review April 5, 2021 12:31 pm

Figure 11-7 Log in window

11.This will take you to the IBM “Fix Central” page where you must provide the proper
information for the FOS entitlement check.

260 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm

Figure 11-8 Fix Central - Quick order

12.Once the Entitlement is successfully confirmed, select the “LinkToBroadcomSWPortal”


link.
13.On the “Leaving IBM Web site” page, select “Continue” to be redirected to the BrocadeIBM
Assist Portal to complete the download.
14.When a user accesses the site, they will be asked to enter a business email address, as
shown in Figure 11-9.

Figure 11-9 Welcome to Assist Portal - enter email address

15.Business email addresses are required. Use of a non-business address will produce an
error message, as shown in Figure 11-10 on page 262.

Chapter 11. IBM SANnav Management Suite overview 261


8497ch11.fm Draft Document for Review April 5, 2021 12:31 pm

Figure 11-10 Welcome to Assist Portal - email address error message

16.Once logged into the Assist Portal, the user will be reminded of the Brocade Enterprise
User License Agreement (EULA), as shown in Figure 11-11.

Figure 11-11 Assist Portal - EULA

17.The Assist Portal is divided into two sections:


– Software Downloads
– Documentation
Navigation between Software Downloads and Documentation can be accomplished via
the appropriate link on the upper right of the page, as shown in Figure 11-12.

Figure 11-12 Assist Portal - Software Downloads and Documentation tabs

262 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm

18.The Software Downloads tab allows the user to download relevant FOS versions and
software including SANnav and EZSwitchSetup.
a. Click Browse and select the appropriate category, as shown in Figure 11-13.

Figure 11-13 Assist Portal - Browse

b. The following products are available depending on the option chosen from the Select
list, as shown in Figure 11-14.

Figure 11-14 Available products

19.Navigation/Downloading of Software can be accomplished by selecting All Software


Products and then relevant product from the lower listing, as shown in Figure 11-15.

Figure 11-15 Assist Portal - Software Downloads

Chapter 11. IBM SANnav Management Suite overview 263


8497ch11.fm Draft Document for Review April 5, 2021 12:31 pm

20.For Documentation on Software, select All Software Products from the select box and
utilize the same search method to download documentation, as shown in Figure 11-16.

Figure 11-16 Assist Portal - Documentation

11.2.7 SANnav installation and migration QuickStart checklists


These checklists are provided for experienced users who are familiar with SANnav
installation. For all other users, start with Installation Overview.

Documentation is located at:


https://www.broadcom.com/products/fibre-channel-networking/software/sannav-managem
ent-portal

Installation checklist
The following table provides a quick checklist for installing SANnav.

Figure 11-17 Installation checklist

264 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm

Migration checklist
The following table provides a quick checklist for migrating from an earlier version of SANnav.

Figure 11-18 Migration checklist

11.2.8 Installation overview


The SANnav application uses a script-based installation. You must run the scripts that are
provided in the <install_home> directory to install the application. All the scripts for the
SANnav application must be executed in the bash shell.

Note:

SANnav Management Portal and SANnav Global View are two different software products.
You cannot install both software products on the same physical host or virtual machine
(VM). You can, however, install Management Portal and Global View on different VMs in
the same host, if the host has enough resources.

For switches that are running Fabric OS versions lower than 8.2.2, port 22 is required for
SANnav Management Portal to use the internal firmware repository and SCP and SFTP
servers. See Installation Prerequisites for additional details. If there is a firewall between
the client and the server or between the server and the SAN, you must open a set of ports
for SANnav to function properly. The list of ports is provided in Firewall Requirements for
SANnav Management Portal. If the installation script detects that an earlier version of
SANnav is running, you are prompted whether you want to migrate your data to the new
version.

After installation, if you choose to move SANnav from one server or VM to another, you must
first release the current license. This process is called rehosting a license. Refer to the
Brocade SANnav Management Portal User Guide for details.

Documentation is located at:


https://www.broadcom.com/products/fibre-channel-networking/software/sannav-managem
ent-portal

Chapter 11. IBM SANnav Management Suite overview 265


8497ch11.fm Draft Document for Review April 5, 2021 12:31 pm

11.2.9 SANnav Management Portal deployment


During SANnav Management Portal installation, you are prompted several times to accept
default values or provide customized values for various settings. If you are migrating from an
earlier version of SANnav, you are not prompted for these customizations, and the settings
from the previous installation remain in effect. The following table lists the installation
customization options.

Figure 11-19 SNAnav installation customization

11.2.10 System and server requirements for SANnav Management Portal


You must meet all the system and server requirements before you start the SANnav
Management Portal installation. The following table lists the system and server requirements
for deployment of SANnav Management Portal.

266 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm

Note: The disk space requirement that is listed in the table is for SANnav only. Be sure to
account for additional space required by the operating system, for saving files, and for
SANnav TAR files and extracted files.

The disk space can be from a direct-attached disk or through a network-mounted disk.
򐂰 The default home directory for installing Docker is /var/lib/docker, but you can choose
another location during installation. Docker must be installed on a local disk.
򐂰 The default swap space directory is the "/" directory. If the directory does not have enough
space, you can choose a different location during installation. The required number of
CPU cores should be equally distributed over the sockets.

Figure 11-20 System and server requirements for SANnav management portal installation

11.2.11 Installation prerequisites for SANnav Management Portal


Review all SANnav Management Portal installation prerequisites before you unzip the
installation file.

Note: Use the latest generation processors for better SANnav performance.

Chapter 11. IBM SANnav Management Suite overview 267


8497ch11.fm Draft Document for Review April 5, 2021 12:31 pm

268 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm

Figure 11-21 . Installation prerequisites

11.2.12 Installing SANnav Management Portal


Complete these steps to download and install SANnav Management Portal on the server.

Before you unzip the installation file, be sure to review and comply with the system and server
requirements and the installation prerequisites that are listed in the following sections:
򐂰 System and Server Requirements for SANnav Management Portal
򐂰 Installation Prerequisites NOTE If the scripts fail during the installation or startup, uninstall
SANnav, reboot the server, and then reinstall SANnav. Do not try to fix the issue and
re-run the installation script without first uninstalling the application.

Download and copy the SANnav Management Portal software package to the server. The
package contains the SANnav Management Portal tarball.
1. Download the SANnav Management Portal tarball (for example,
Portal_<version>-distribution.tar.gz) to the folder where you want to install the
application. NOTE Do not create the SANnav Management Portal installation folder with a
space in the name; otherwise, installation fails.
2. Untar the .gz file to extract the file to the current location. tar -xvzf
Portal_<version>-distribution.tar.gz This step creates a directory with a name similar
to Portal_<version>_bldxx. This directory is referred to as the <install_home> directory
in this document.
3. Go to the <install_home>/bin directory. [root@RHEL7-10-100 home]# cd
Portal_<version>_bldxx/bin
4. Run the install-sannav.sh script to install SANnav Management Portal.
[root@RHEL7-10-100 bin]# ./install-sannav.sh If an earlier instance of SANnav
Management Portal is installed, the installation script prompts whether you want to
continue with migration or exit the installation.
5. If you are prompted about migrating SANnav, enter one of the following options.
– To proceed with migration, press Enter. You are prompted to enter the location of the
existing SANnav installation.
– To exit the installation, press Ctrl-C. The script ends. At this point, you can back up the
current SANnav instance and restart the installation script. Or you can uninstall the
current SANnav instance, and restart the installation script without migrating.

Chapter 11. IBM SANnav Management Suite overview 269


8497ch11.fm Draft Document for Review April 5, 2021 12:31 pm

As the installation proceeds, the script runs a pre-install requirements test. If any test fails, the
installation exits with error messages. You must fix the reported issues, uninstall the
application, and repeat from Step 1. After the diagnostics pass, installation of SANnav
Management Portal software continues. On successful installation of the software, the
SANnav Management Portal server starts up.

The startup may take up to 20 minutes.

11.2.13 Firewall requirements for SANnav Management Portal


If your network utilizes a firewall between the SANnav Management Portal client and the
server or between the server and the SAN, a set of ports must be open in the firewall to
ensure proper communication. The following table lists the ports that must be open in the
firewall. These ports are added to the IP tables by default when the SANnav server is running,
so you do not need to open them in the firewalld if it is enabled and running on the SANnav
server.

Figure 11-22 Ports that must be open in the firewall

If firewalld is enabled, you must add the SSH service to the trusted zone in the firewalld for
the firmware download feature to work.

See Configuring a Firewall for SANnav for instructions on how to configure firewalld. If you
configure an external authentication server (LDAP, RADIUS, or TACACS+) or an email server
(SMTP), ensure that the SANnav Management Portal server has access to the ports listed in
the following table. The default ports are listed in the table, but you can change the default.

270 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm

Figure 11-23 Ports that the server must be able to access

11.2.14 SANnav Management Console commands


The sannav-management-console.sh script allows you to perform several actions on the
SANnav server without having to run individual scripts.

Using this one script, you can perform the following actions:
򐂰 Check SANnav status
򐂰 Restart SANnav
򐂰 Stop SANnav
򐂰 Start SANanv
򐂰 Uninstall SANnav
򐂰 Show the SANnav port and server configuration. Go to the <install_home>/bin folder and
run the following script: ./sannav-management-console.sh You are presented with a list of
options from which to choose.

11.2.15 Checking the server health


After the installation is complete, you can check the health of the SANnav server using the
check-sannav-status.sh script. If any of the services is down, it will be listed in the script
output.

To check the health of the server, go to the <install_home>/bin folder and run the following
script: ./check-sannav-status.sh

Note: If any service is found down while checking the server health status, it will be
automatically started by systemmonitor within 20 minutes.

The following is sample output of a healthy server.


-bash-4.2# sh ./check-sannav-status.sh
SANnav server is healthy.
All the services are currently in running state.

Chapter 11. IBM SANnav Management Suite overview 271


8497ch11.fm Draft Document for Review April 5, 2021 12:31 pm

The following is sample output of an unhealthy server.


-bash-4.2# sh ./check-sannav-status.sh
Following services are currently down or starting filters-middleware
topology-middleware SANnav Disk Usage Alert monitors the disk space used for every
ten minutes.

There are three threshold levels: 70%, 80%, and 90%. An event is sent when the threshold
levels exceed or drop below the defined level. The following is the list of event severity:
threshold:
򐂰 Warning — 70%
򐂰 Error — 80%
򐂰 Critical — 90%

Changing the self-signed certificates for client and server


communication
You can replace the SSL self-signed certificates with third-party signed certificates. Make
sure that the SSL certificate and key files are copied to this host or VM.

Go to the <install_home>/bin folder and run the following script:


./replace-server-certificates.sh

When you run this script, SANnav is automatically restarted for the new certificates to take
effect. After the server is back up, you must rediscover or unmonitor and then monitor all
switches that are registered for telemetry data; otherwise, the new certificates do not take
effect.

Configuring a firewall for SANnav


Perform the following steps to set up a firewall using firewalld. This example uses Red Hat
Enterprise Linux (RHEL).
1. Start the firewall using the following command. sudo systemctl start firewalld
2. Check that the firewall is running. sudo systemctl status firewalld
3. Enable the firewall automatically after a system reboot. sudo systemctl enable firewalld
4. Add the SSH service to the trusted zone.
sudo firewall-cmd --zone=public --permanent --add-service=ssh
If any other default ports are customized, add the services for those ports as well. For
example, if you are using HTTPS port 443, enter the following command:
sudo firewall-cmd --zone=public --permanent --add-service=https
5. Add ports using the following commands. Note that in the following commands, public is
the default zone. If your default zone is different, use your default zone for the ports.
sudo firewall-cmd --zone=public --add-port=2377/tcp --permanent
sudo firewall-cmd --zone=public --add-port=7946/tcp --permanent
sudo firewall-cmd --zone=public --add-port=7946/udp --permanent
sudo firewall-cmd --zone=public --add-port=4789/udp --permanent
6. If you are installing on CentOS or RHEL, version 8.0 or higher, turn on masquerading on
the firewall. sudo firewall-cmd --zone=public --add-masquerade --permanent.
7. Associate the interface (if this is not done already) with the default profile.
sudo firewall-cmd --permanent --zone=public --change-interface=<interface_name>

272 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm

8. After the ports are added, use the following command to reload the firewall configuration.
sudo firewall-cmd --reload
9. Verify whether the configuration is correct. sudo firewall-cmd --list-all

11.2.16 Launching SANnav Management Portal


Launch SANnav Management Portal to monitor and manage your fabrics.
1. Open your browser and enter the IP address or fully qualified domain name (FQDN) of the
SANnav Management Portal server. You can use http or https, for example:
http://192.0.2.0 or https://192.0.2.0
The SANnav Management Portal login window appears.

Figure 11-24 SANnav Management Portal - Login

If the login window does not appear, try the following:


򐂰 Ping the SANnav server to ensure that it is up
򐂰 Check if SANnav services are running. See "Checking the Server Health" in Additional
Scripts.
򐂰 Check the firewall settings.
2. Enter your SANnav user name and password, and click Login.

For the first SANnav login, the default user name is "Administrator" and the default password
is "password".

SANnav launches with the default dashboard displayed. If, instead of launching, SANnav
displays the message "Login Failed. Service is not available at this time.",

SANnav is in the process of starting up. Wait a few minutes and try to log in again. The first
time you log in, you should change the default password.

Chapter 11. IBM SANnav Management Suite overview 273


8497ch11.fm Draft Document for Review April 5, 2021 12:31 pm

11.2.17 Changing your password


The first time you log in to SANnav, you should change your password. After you change your
password, you are automatically logged out, and you must log in again using the new
password.
1. Click the user icon in the top-right corner of the window, and then click User Preferences.

Figure 11-25 User Preferences

2. Click the Edit button next to Logging in.

Figure 11-26 Logging in

3. Click Change on the Logging in page.


4. Fill out the Change Password fields, and click OK. You are automatically logged out of
SANnav.
5. Log in to SANnav again using your new password.

11.2.18 Overview of the user interface


SANnav Management Portal allows you to manage one or more SAN fabrics in multiple
locations. Once familiar with the basic components of SANnav, you can quickly start
monitoring and managing your fabrics. The following screen capture shows the basic layout of
the SANnav user interface.

274 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm

Figure 11-27 SANnav Management Portal User Interface

1. Navigation bar. Contains links to feature pages. The SANnav link displays the
Configurations and Settings page.
2. Search. Click the icon to display the search box, select a context on which to search from
the list, and enter the search term in the field.
3. Profile menu. Click the icon to display additional links for changing user preferences,
displaying the SANnav version, and logging out.
4. Subnavigation bar. Provides the page title and optional item count within parentheses.
Also includes buttons and menus to take actions within the page. The subnavigation bar is
the main way to navigate within a feature.
5. Filter bar. Allows you to filter the display based on columns, fabrics, network scope, and
customized filters.
6. Expandable sidebar. Provides an area where you can save selected inventory items for
investigation later.

Detail Pages Clicking a fabric name, switch name, or port name in a table opens a detail page
for that object. The detail page displays additional information about the object and may
contain additional actions that you can perform.

The detail page is different depending on the context. For example, the detail page for a fabric
on the Inventory page is different from the detail page for the same fabric on the Discovery
page.

Chapter 11. IBM SANnav Management Suite overview 275


8497ch11.fm Draft Document for Review April 5, 2021 12:31 pm

Figure 11-28 Detail page for a switch

Tables In tables, if an entry is truncated (indicated by an ellipsis at the end of the entry), hover
the mouse over the entry to see the full text.

Figure 11-29 Hovering the mouse over truncated table text

Many tables have an action menu that you can access by clicking the down arrow in the
rightmost column. Click this arrow to display additional actions that you can perform on the
associated object.

276 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm

Figure 11-30 Using the down arrow to display additional actions

On the right side of the subnavigation bar, the More button ( ) often contains a Bulk Select
option, which allows you to select and perform operations on one or more items at the same
time. For example, you can turn monitoring on for multiple fabrics, or you can create a tag and
apply it to multiple switches.

Figure 11-31 Bulk Select

11.2.19 Initial setup and configuration


If you are familiar with SAN network management, the following lists a number of tasks that
you might want to perform directly after SANnav installation, with a link to subsequent
sections of this guide for detailed information on each.
򐂰 Discover fabrics. SANnav must connect to a seed switch per fabric in order to discover the
items in the fabric. See Discovering a Fabric.
򐂰 Configure service notifications. Configure the SMTP mail server and Call Home, register
the SANnav server as a syslog and SNMP trap recipient, and forward SNMP traps and
syslog messages.

See Call Home and ESRS and Event Management.


򐂰 Set up user accounts. Add users, roles, and areas of responsibility (AORs). Configure a
password policy. Set up external authentication. See Security.
򐂰 Configure MAPS policies. You can create a new policy or import policies from a JSON or
XML file. For example, you can import MAPS policies that were exported from Brocade
Network Advisor. See Monitoring and Alerting Policy Suite.

Chapter 11. IBM SANnav Management Suite overview 277


8497ch11.fm Draft Document for Review April 5, 2021 12:31 pm

򐂰 Create custom dashboards. SANnav provides several default dashboards, but you can
create custom dashboards to fit your specific needs. See Dashboards.

For detailed information for Brocade SANnav setup, configuration and navigation, refer to the
Brocade SANnav User Guide.

Documentation is located at:


https://www.broadcom.com/products/fibre-channel-networking/software/sannav-managem
ent-portal

11.3 IBM SANnav Global View 2.1 release overview


SANnav Global View is a higher-level "global" management application that provides a
comprehensive view of a SAN environment that spans multiple SANnav Management Portal
instances. Using SANnav Global View, you can navigate seamlessly across multiple SANnav
Management Portal instances and drill down to any individual instance to perform detailed
monitoring, investigation, and troubleshooting.

Note:

SANnav Global View is a separate product from SANnav Management Portal, and it
requires separate installation and licensing. You log in to the SANnav Global View
application and add portals to SANnav Global View, which then uses information in the
portals to build a global view of the dashboard and inventory.

SANnav Global View is compatible only with SANnav Management Portal instances that
are running the same version as Global View. Older versions of SANnav Management
Portal are not supported.

11.3.1 Browser requirements for SANnav Global View


Any laptop or machine that launches web applications can be used to launch SANnav Global
View. For optimal performance, have at least 16GB memory.

The following browsers can be used to access SANnav Global View:


򐂰 Chrome (Google)
򐂰 Firefox (Mozilla) Note that if you access the client from the Remote Desktop, the user
interface may have degraded performance.

11.3.2 System and Server Requirements for SANnav Global View


It is mandatory to meet all the system and server requirements before you start SANnav
Global View installation.

The following table lists the system and server requirements for SANnav Global View.

Note: The disk space requirement listed in the table is for SANnav Global View only. Be
sure to account for additional space required by the operating system and for saving files.

278 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm

Figure 11-32 SANnav Global View single-node deployment

11.3.3 Installation prerequisites for SANnav Global View


Review all SANnav Global View installation prerequisites before you unzip the installation
files.

Note: Use the latest generation processors for better SANnav performance.

Figure 11-33 SANnav Global View - Installation Prerequisites

Chapter 11. IBM SANnav Management Suite overview 279


8497ch11.fm Draft Document for Review April 5, 2021 12:31 pm

11.3.4 Installing SANnav Global View


Complete these steps to install SANnav Global View on the server.

Before unzipping the installation file, be sure to review and comply with the system and server
requirements and the installation prerequisites listed in the following sections:
򐂰 System and Server Requirements for SANnav Global View
򐂰 Installation Prerequisites for SANnav Global View

Download and copy the SANnav Global View software package to the server. The package
contains the SANnav Global View tarball.
1. 1. Download the SANnav Global View tarball ( for example,
Global_2.1.0_rc_bldxx-distribution.tar.gz) to the folder where you want to install the
application. NOTE Do not create the SANnav Global View installation folder with spaces in
the name; otherwise, installation will fail.
2. Untar the .gz file to extract the file to the current location. tar -xvzf
Global_2.1.0_rc_bldxx-distribution.tar.gz This step creates a directory with a name
similar to Global_2.1.0_rc_bldxx. This directory is referred to as the <install_home>
directory in this document.
3. Go to the <install_home>/bin directory.
4. Run the install-sannav.sh script to install SANnav Global View. The
./install-sannav.sh installation script checks whether an earlier instance of SANnav
Global View is installed, and if so, it prompts if you want to exit installation and take a
backup, or continue installation without backing up the server:
5. If you are prompted about migrating SANnav, enter one of the following options:
– Press Ctrl-C to exit the installation procedure. Either back up the server or uninstall the
existing SANnav instance. Then restart the installation script.
– Press Enter to proceed with migration. You are prompted to enter the location of the
existing SANnav installation.

As the installation proceeds, the script runs a pre-install requirements test. If any test fails, the
installation exits with error messages. You must fix the reported issues and re-run the install
script. After the diagnostics pass, installation of SANnav Global View software continues. On
successful installation of the software, the SANnav Global View server starts up. The startup
may take up to 10 minutes.

11.3.5 Logging in to SANnav Global View


To log in to SANnav Global View, perform the following:
1. 1. Open your browser and enter the IP address or fully qualified domain name (FQDN) of
the SANnav Global View server.
You can use HTTP or HTTPS, for example: http://192.0.2.0 or https://192.0.2.0. The
SANnav Global View login window appears.

280 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm

Figure 11-34 SANnav Global View - Login

If the login window does not appear, try the following:


– Ping the SANnav server to ensure that it is up.
– Check if SANnav services are running. See "Checking the Server Health" in Additional
Scripts for Managing SANnav.
– Check the firewall settings.
2. Enter your SANnav user name and password, and click Login. For the first SANnav login,
the default user name is "Administrator" and the default password is "password". SANnav
launches with the default dashboard displayed. If, instead of launching, SANnav displays
the message "Login Failed. Service is not available at this time. ", SANnav is in the
process of starting up. Wait a few minutes and try to log in again.

11.3.6 Quick tour of SANnav Global View


Once familiar with the basic components of SANnav Global View, you can quickly start
monitoring SANnav Management Portal instances. When you first log in to SANnav Global
View, you see the Summary dashboard, the single dashboard provided with Global View. The
following screen capture shows the basic layout of the SANnav Global View user interface.

Chapter 11. IBM SANnav Management Suite overview 281


8497ch11.fm Draft Document for Review April 5, 2021 12:31 pm

Figure 11-35 Summary

1. Navigation bar. Contains links to feature pages. The SANnav link displays the page where
various settings and configurations can be performed.
2. Profile menu. Displays links for changing user preferences, displaying the SANnav
version, and logging out.
3. Subnavigation bar. Provides the page title and optional item count within parentheses.
Also includes buttons and menus to take actions within the page.
4. Filter bar. Allows you to filter the display based on columns, portal scope, and customized
filters.
Click Inventory to display inventory information about fabrics, switches, switch ports,
chassis, host and storage devices, and host and storage ports across all Management
Portal instances.

Figure 11-36 Inventory

Click Events to show all application events that stem from a user or system action. A system
action is triggered under certain situations like when a portal is disconnected. You can filter
the events list by date range. Events from SANnav Management Portal instances are not
listed here.

282 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm

Figure 11-37 Events

Click SANnav to discover Management Portals, manage filters, and perform various
configuration settings across several feature categories. The profile menu is where you can
add your phone number and change your password. It is also where you can enable Persist
Last Column Selection, wherein the system remembers your column customization, and
FICON Display. Column customization is the ability to choose what columns you want to see
in a tabular view. For example, by default, only a few of the many available columns in
inventory are shown, but you can customize the table by adding or removing columns.

When FICON Display is enabled, certain columns in the switch and switch port inventory
pages are rearranged for FICON mode.

Figure 11-38 Administrator User Preferences

11.3.7 Generating a license for SANnav Global View


Refer to Section 1.5.1 Downloading the software for step-to-step instructions.

Complete the following steps to download the software and documentation from the Brocade
IBM assist website:
򐂰 IBM b-type Gen 7 Software/Firmware Download Instructions (IBM Assist Site for IBM
Brocade Software downloads)
򐂰 Step-by-Step instructions for access to Fabric OS firmware (FOS), SANnav and b-type
Gen 7 IBM Network Advisor (INA).

Chapter 11. IBM SANnav Management Suite overview 283


8497ch11.fm Draft Document for Review April 5, 2021 12:31 pm

Important: You must have active IBM Support in order to access Brocade Software for
any device.

򐂰 Brocade FOS and SAN Management software may be accessed through the IBM Support
Fix Central, found at: https://www.ibm.com/support/fixcentral/options

11.3.8 Global dashboard overview


The SANnav Global View dashboard provides comprehensive "global" visibility across
multiple SANnav Management Portal instances. The Global View dashboard is
auto-refreshed every 15 minutes. The Global View dashboard provides the following widgets:
Health summary, Switch Health, Port Usage Summary, and Alerts. These widgets indicate the
status of the fabric, switch, host, and storage entities managed by the Portals that are
currently added in SANnav Global View. The health summary widgets are categorized by
health state: Healthy, Degraded, or Poor. Listed below each widget are the exact number of
entities in each health state. The health state for each type of object is determined by its
health score, which is determined by a set of predefined factors. For example, a health score
for fabrics is influenced by whether a link is down or redundant paths are missing. For more
details on how health score is computed, see Factors Contributing to the Overall Health
Score.

Figure 11-39 Health states and scores

SANnav dashboards allow you to gather additional information by drilling down into the
dashboard widgets. By default, the scope of SANnav Global View is set to All Portals (that is,
all Management Portal instances). You can choose one or more Management Portal
instances to change the scope.

284 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm

Figure 11-40 Summary

11.3.9 Monitoring fabric health


The Fabric Health widget of SANnav Global View shows the number of fabrics in Healthy,
Degraded, and Poor state. The least overall score across all objects in this group of fabrics is
shown in the center of the circular widget.

The following procedure shows


1. Click Dashboard & Reports in the navigation bar.
2. Hover over a colored portion of the Fabric Health widget to display details of the fabrics
with that state. For example, hover over the red portion to display a list of portals and the
number of fabrics in each portal having a poor state.

Chapter 11. IBM SANnav Management Suite overview 285


8497ch11.fm Draft Document for Review April 5, 2021 12:31 pm

Figure 11-41 Fabric Health: 20 Poor

3. To view further details on the fabrics, click the colored segment of the widget. For example,
clicking the red segment displays a table showing the health scores for the fabrics in poor
health.
4. Click Show Details on the action menu of the fabric list to see details associated with that
health score. The Show Details option is available only if the score is less than 100.

Figure 11-42 Fabric Health: 20 Poor - Show Details

A popup window displays the factors that cause deductions in the health score for this
fabric.
5. Click Back to return to the list of portals.
6. Select Show in SANnav Portal from the action menu to launch the Management Portal
instance responsible for the data you are viewing. You might need to make sure that

286 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm

popups are not blocked in your browser. If you are not logged in to this portal, credentials
(for this portal) are required.

11.3.10 Monitoring switch health across Management Portal instances


The Switch Health widget displays details of switch health across Management Portal
instances. You can get a summarized view as well as details of switches based on firmware
version, Management Portal instance, switch model, and product category.

The following procedure shows how to use the Global View dashboard to display switch
health across multiple Management Portal instances.
1. Click Dashboard & Reports in the navigation bar. On the bottom row of the dashboard, the
Switch Health widget displays. In this screen you see switches grouped by SANnav Portal
instances. You can select a different grouping from the drop-down list.

Figure 11-43 Switch Health - SANnav Portal

2. Hover over the graph to display details. Note that for the Firmware Version and Model
categories, more bars may be displayed in the graph than are listed in the row names on
the left. In this case, hover over an intermediate bar in the graph to display the details.

Chapter 11. IBM SANnav Management Suite overview 287


8497ch11.fm Draft Document for Review April 5, 2021 12:31 pm

Figure 11-44 Switch Health - Model

3. To see the details on all switches in a particular category, click the bar chart of the widget,
and then select Show Details.

Figure 11-45 Product Health: Poor

4. Click Close to return to the dashboard.


5. Select SANnav Portal from the drop-down list in the widget, click the bar graph, and select
Show in Portal to display the switch inventory in the selected SANnav Management Portal
instance. If you are not logged in to this portal, credentials (for this portal) are required. A
separate window opens for the Management Portal instance.

288 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm

11.4 IBM SANnav Web Tools


Web Tools is launched directly from a Web browser or from SANnav Management Portal. You
can launch Web Tools on any workstation with a compatible operating system and Web
browser installed.

If the switch is configured with logical fabrics, you can log in to any of the logical fabrics for
which you have permission.
1. Launch Web Tools directly from a browser or from SANnav Management Portal.
– To launch directly from a Web browser, open your browser, enter the IP address of the
switch, and press Enter.
You can use HTTP or HTTPS, for example: http://10.77.77.77 or
https://10.77.77.77.
– To launch from SANnav Management Portal, locate the switch on the SANnav
Inventory page, click the down arrow to the right of the switch, and select View in
WebTools from the action menu.

Figure 11-46 Launching Web Tools from SANnav Management Portal

2. Enter the user name, password, and logical switch name or fabric ID (FID). For the first
switch login, the default user name is admin and the default password is password. Web
Tools prompts you to change the default password. If you are logging in to a Virtual
Fabrics-enabled platform and you do not specify a logical switch, you are logged in to the
default logical switch, which uses fabric ID 128. For non-VF platforms, the FID option is not
displayed.

Chapter 11. IBM SANnav Management Suite overview 289


8497ch11.fm Draft Document for Review April 5, 2021 12:31 pm

Figure 11-47 WebTools Element Manager

If you launch from SANnav Management Portal, you might not need to log in, depending
on the SANnav single sign-on configuration.
3. Click Login

11.4.1 Overview of the Web Tools user interface


Once familiar with the basic components of Web Tools, you can quickly start monitoring and
managing your switch. The following screenshot shows the basic layout of the Web Tools user
interface.

Figure 11-48 Web Tools user interface

1. Navigation bar. Contains links to feature pages.


2. Profile menu. Displays the link for logging out.
3. Subnavigation bar. Provides the page title and includes buttons and menus to take actions
within the page.

290 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm

For more information about the SANnav Web Tools product, see Web Tools section of the
Brocade Fabric OS Web Tools User Guide, 9.0.x., found at:
https://www.broadcom.com/products/fibre-channel-networking/software/fabric-operati
ng-system

11.5 Reference guides


For more information about the hardware and software that is supported for SANnav
Management Portal V2.1, see the Documentation section at:
https://www.broadcom.com/products/fibre-channel-networking/software/sannav-managem
ent-portal

Refer to the following guides for additional information:


򐂰 Brocade SANnav Management Portal User Guide describes how to monitor and manage
your storage area network (SAN) using Brocade SANnav Management Portal.
򐂰 Brocade SANnav Flow Management User Guide explains how to configure and manage
flows using SANnav Management Portal.
򐂰 Brocade SANnav Management Portal REST API and Northbound Streaming Reference
Manual contains definitions of REST APIs you can use to access SANnav Management
Portal, including streaming performance and flow metrics to an external server.
򐂰 Brocade SANnav Global View User Guide describes how to use SANnav Global View to
monitor and manage multiple Management Portal instances. SANnav Global View is a
separate product.
򐂰 Brocade SANnav Management Portal Release Notes includes a summary of the new,
unsupported, and deprecated features for this release.

Documentation is located at:


https://www.broadcom.com/products/fibre-channel-networking/software/sannav-managem
ent-portal

Chapter 11. IBM SANnav Management Suite overview 291


8497ch11.fm Draft Document for Review April 5, 2021 12:31 pm

292 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch12.fm

12

Chapter 12. SAN security


Security has always been an advantage with Fibre Channel based SANs. From its
air-gapped deployment (often times these networks are physically separated from the rest of
the enterprise network) to the much more difficult medium (fiber) used for eaves dropping to
the much reduced compromise domain where an impacted server can only affect the LUNs
zoned and masked to it and no others. However, this does not mean Fibre Channel SANs
can altogether ignore security. After all, mission critical applications run on SANs and this
critical network and its data must be protected.

When we discuss FC SAN Security, there are several challenges, and a layered security
model should be deployed to ensure it continues to be secure for years to come.

We will look at each of these layers independently, working from the outermost layers dealing
with Management Security to the innermost layers dealing with the data itself.

This chapter includes the following topics:


򐂰 12.1, “Management Interfaces” on page 295
򐂰 12.2, “Displaying IP Filter” on page 295
򐂰 12.3, “Role-Based access controls” on page 297
򐂰 12.4, “Default accounts” on page 297
򐂰 12.5, “User accounts” on page 297
򐂰 12.6, “Security Protocols” on page 297
򐂰 12.7, “Access Control Lists” on page 298
򐂰 12.8, “SAN Infrastructure – Securing End Points” on page 299
򐂰 12.9, “ISL In-Flight Encryption and Compression” on page 300
򐂰 12.10, “Policy Database Distribution” on page 304

© Copyright IBM Corp. 2021. 293


8497ch12.fm Draft Document for Review April 5, 2021 12:31 pm

Figure 12-1 shows the layers of FC SAN Security.

Figure 12-1 Figure 1 - A Layered Security Overview

294 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch12.fm

12.1 Management Interfaces


In this section we’ll discuss the Management Interface, which is the first and most vulnerable
area we’ll be looking into. We have several methods available to us in FOS for protection. Of
course, we want to limit what IP Addresses have access to the Management VLAN where our
devices are managed from. We restrict those IP Addresses using separate VLANs, while
locking down which protocols are used (disabling the least secure protocols, and closing their
ports).

We use IP Filter in FOS to allow or deny the entry of traffic across these interfaces. We also
want to make sure, if necessary, we equally limit both IPv4 and IPv6 traffic. The IP Filter
policy is a set of rules that are applied to the IP management interfaces and acts as a packet
filtering firewall. This firewall permits or denies the traffic to go through the IP management
interfaces according to the IP Filter rules.

As a preferred practice, non-secure IP protocols that are used for switch management such
as Telnet and HTTP should be blocked. SSH is enabled on the default IP Filter policy and SSL
should be configured to use HTTPS for web access.

The IP Filter policy should be used in environments where you need strict control of fabric
access. As with other ACL policies, it is important to regularly review and properly maintain
the IP Filter policy.

12.2 Displaying IP Filter

To display the current IP Filter in use, use the ipfilter --show command as show below:
switch:admin> ipfilter --show
Name: default_ipv4, Type: ipv4, State: active
Rule Source IP Protocol Dest Port Action

1 any tcp 22 permit

2 any tcp 23 permit

3 any tcp 80 permit

4 any tcp 443 permit

5 any udp 161 permit

6 any udp 123 permit

7 any tcp 600 - 1023 permit

8 any udp 600 - 1023 permit

Chapter 12. SAN security 295


8497ch12.fm Draft Document for Review April 5, 2021 12:31 pm

Name: default_ipv4, Type: ipv4, State: active


Rule Source IP Protocol Dest Port Action

1 any tcp 22 permit

2 any tcp 23 permit

3 any tcp 80 permit

4 any tcp 443 permit

5 any udp 161 permit

6 any udp 123 permit

7 any tcp 600 - 1023 permit

8 any udp 600 - 1023 permit

Below is an example a newly created IP Filter to deny an unsecure protocol such as Telnet,
which uses TCP Port 23. Note the IPv6 filter remains unchanged.
ipfilter --clone BlockTelnet -from default_ipv4
ipfilter --save BlockTelnet
ipfilter --addrule BlockTelnet -rule 1 -sip any -dp 23 -proto tcp -act deny
ipfilter --save
ipfilter --activate BlockTelnet

switch:admin> ipfilter --show


Name: default_ipv4, Type: ipv4, State: active
Rule Source IP Protocol Dest Port Action

1 any tcp 22 permit

2 any tcp 23 deny

3 any tcp 80 permit

4 any tcp 443 permit

5 any udp 161 permit

6 any udp 123 permit

7 any tcp 600 - 1023 permit

8 any udp 600 - 1023 permit

Name: default_ipv6, Type: ipv6, State: active


Rule Source IP Protocol Dest Port Action

1 any tcp 22 permit

2 any tcp 23 permit

3 any tcp 80 permit

4 any tcp 443 permit

5 any udp 161 permit

296 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch12.fm

6 any udp 123 permit

7 any tcp 600 - 1023 permit

8 any udp 600 - 1023 permit

12.3 Role-Based access controls


One way to provide limited accessibility to the fabric is through user roles. FOS has
predefined user roles, each of which has access to a subset of the CLI commands. These
roles are known as role-based access controls (RBACs), and they are associated with the
user’s login credentials. When you log in to a switch, your user account is associated with a
predefined role or a user-defined role. The role that your account is associated with
determines the level of access you have on that switch and in the fabric.

12.4 Default accounts


FOS offers four predefined accounts: Admin, factory, root, and user. Although the root and
factory accounts are reserved for development and manufacturing, the password for all
default accounts should be changed and secured during the initial installation and
configuration of each switch. Recovering lost passwords might require significant effort and
fabric downtime as well as assistance from Support resources.

12.5 User accounts


In addition to the default permissions assigned to the roles of root, factory, admin, and user,
Fabric OS supports up to 252 additional user accounts on the chassis. These accounts
expand the ability to track account access and audit administrative activities.

Each user account is associated with the following attributes:


򐂰 Virtual Fabric list: Specifies the Virtual Fabric that a user account is allowed to log in to.
򐂰 Home Virtual Fabric: Specifies the Virtual Fabric that the user is logged in to, if available.
The home Virtual Fabric must be a member of the user’s Virtual Fabric list. If the fabric ID
is not available, the next-lower valid fabric ID is used.
򐂰 LF Permission List: Determines functional access levels within the bounds of the user’s
Virtual Fabrics.
򐂰 Chassis role. Similar to switch-level roles, but applies to a different subset of commands.

12.6 Security Protocols


Security protocols provide endpoint authentication and communications privacy by using
cryptography. Typically, the user is authenticated to the switch while the switch remains
unauthenticated. This authentication means that you can be sure that you know with whom
you are communicating. The next level of security, in which both ends of the conversation are
sure with whom they are communicating, is known as two-factor authentication. Two-factor
authentication requires public key infrastructure (PKI) deployment to clients.

Chapter 12. SAN security 297


8497ch12.fm Draft Document for Review April 5, 2021 12:31 pm

Protocol Description: 12.7, “Access Control Lists” on page 298 shows the security protocols
that are supported by Fabric OS.

Table 12-1 Security protocols that are supported by Fabric OS


CHAP Challenge Handshake Authentication Protocol (CHAP) uses shared secrets to
authenticate switches.

HTTPS Hyper Text Transfer Protocol over SSL (HTTPS) is a Uniform Resource Identifier
scheme that is used to indicate a secure HTTP connection. Web Tools supports
the use of HTTPS.

IPSec Internet Protocol Security (IPSec) is a framework of open standards for providing
confidentiality, authentication, and integrity for IP data transmitted over untrusted
links or networks.

LDAP Lightweight Directory Access Protocol (LDAP) with Transport Layer Security (TLS)
uses a certificate authority (CA). By default, LDAP traffic is transmitted unsecured.
With imported signed certificates, you can make LDAP traffic confidential and
secure by using Secure Sockets Layer (SSL) / TLS technology with LDAP.

SCP Secure Copy Protocol (SCP) is a means of securely transferring computer files
between a local and a remote host or between two remote hosts that use the
Secure Shell (SSH) protocol. Configuration upload and download support the use
of SCP.

Secure Syslog Secure syslog requires importing syslog CA certificates by using the
secCertMgmt command.

SFTP Secure File Transfer Protocol (SFTP) is a network protocol for securely
transferring files on a network. Configuration upload and download support the
use of SFTP.

SNMP Simple Network Management Protocol (SNMP) is used in network management


systems to monitor network-attached devices for conditions that warrant
administrative attention. Supports SNMPv1 and v3.

SSH Secure Shell (SSH) is a network protocol that allows data to be exchanged over a
secure channel between two computers. Encryption provides confidentiality and
integrity of data. SSH uses public-key cryptography to authenticate the remote
computer and allow the remote computer to authenticate the user, if necessary.

SSL Secure Sockets Layer (SSL) is used by Fabric OS to support HTTPS. A certificate
must be generated and installed on each switch to enable SSL. This configuration
supports SSLv3, 128-bit encryption by default. It also supports TLSv1.0, TLSv1.1,
and TLSv1.2.

12.7 Access Control Lists


Access control lists (ACLs) are used to provide network security through policy sets. FOS
provides several ACL policies, including a Switch Connection Control (SCC) policy, a Device
Connection Control (DCC) policy, a Fabric Configuration Server (FCS) policy, an IP Filter, and
others. The following sections briefly describe each policy and provide basic guidelines.
򐂰 SCC policy
– Use the SCC policy to restrict FC switches that can join the fabric. Only switches that
are specified in the policy are allowed to join the fabric. All other switches fail

298 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch12.fm

authentication if they attempt to connect to the fabric, resulting in the respective


E_Ports being segmented because of the security violation.
– Use the SCC policy in environments where you need strict control of fabric switch
members.
– The SCC policy can prevent switches from participating in a fabric, so it is very
important to regularly review and properly maintain the SCC policy.
򐂰 DCC policy
– Use the DCC policy to restrict devices that can attach to a single Fibre Channel (FC)
port. The policy specifies the FC port and one or more worldwide names (WWNs) that
are allowed to connect to the port. The DCC policy comprises all of the DCC policies
that are defined for individual FC ports. Not every FC port must have a DCC policy, and
only ports with a DCC policy in the active policy set enforce access controls. A port that
is present in the active DCC policy allows only WWNs in its respective DCC policy to
connect and join the fabric. All other devices fail authentication when they attempt to
connect to the fabric, resulting in the respective F_Ports being disabled because of the
security violation.
– Use the DCC policy in environments where you need strict control of fabric device
members.
– Because the DCC policy can prevent devices from participating in a fabric, it is very
important to regularly review and properly maintain the DCC policy.
򐂰 FCS policy
– Use the FCS policy to restrict the source of fabric-wide settings to one FC switch. The
policy contains the WWN of one or more switches, and the first WWN (that is online) in
the list is the primary FCS. If the FCS policy is active, then only the primary FCS is
allowed to make or propagate fabric-wide parameters. These parameters include
zoning, security (ACL) policies databases, and other settings.

12.8 SAN Infrastructure – Securing End Points


The SAN infrastructure consists of HBAs, Switches, and Storage Devices. Each of these
components offer security features that should be enabled to maximize security.

On the HBA front, we have several options available to us. We can use Single Initiator Zones
to limit the number of LUNs that could potentially be compromised should the server be
compromised. On the storage device LUN Masking should be implemented to ensure only
those LUNs zoned to the HBA are accessible to the HBA through its zoned storage port. We
also have the option of enabling DH-CHAP to authenticate these devices when they join the
fabric.

On the Storage device front, we have similar options available. We should implement Single
Initiator Zones along with LUN Masking to limit exposure should a device be compromised.
We also have secure protocols such as DH-CHAP available to authenticate these devices
when they join the fabric.

Finally on the FC Switch front, we have even more options available. We can disable
persistently ports that are not used in the switch. We can also disable the ability of the online
ports from even becoming E-Ports (ISLs) by locking down their port configuration to disallow
that type of login. Leveraging a DCC Policy we can create an ACL of sorts whereby we can
control what device is allowed to log into specific ports by its WWN. Leveraging a SCC Policy
we can also use this same ACL concept to lock down what switches are allowed to participate
in a fabric by the switch’s WWN. These features are leveraged heavily when creating High

Chapter 12. SAN security 299


8497ch12.fm Draft Document for Review April 5, 2021 12:31 pm

Integrity Fabrics required for most FICON environments. We can also use either DH-CHAP
or FCAP to authenticate switches, this authentication type is required when establishing
Encrypted E-Ports (ISLs) between switches, usually across long distances or connected via
DWDM devices. It is a best practice to enable encryption on ISLs or FCIP links extending
beyond the data center walls. DH-CHAP is a more simple deployment where shared secrets
are exchanged between switches, whereas FCAP used more secure signed digital
certificates for authentication.

12.9 ISL In-Flight Encryption and Compression


IBM b-type Gen 7 Fibre Channel platforms support both in-flight compression and encryption
at a port level for both local and long-distance inter-switch links (ISLs). In-flight data
compression is a useful tool for saving money when either bandwidth caps or bandwidth
usage charges are in place for transferring data between fabrics. Similarly, in-flight encryption
enables a further layer of security with no key management impact when transferring data
between local and long-distance data centers besides the initial setup.

Enabling in-flight ISL data compression or encryption slightly increases the latency as the
ASIC processes the frames for compression or encryption or both.

Here is an example of how to implement DH-CHAP for ISL Encryption.


򐂰 Step 1 - Verify if DH-CHAP Secrets exist in the switch database
switch:admin> secauthsecret --show
Chap secret key database does not exist.
򐂰 Step 2 - Run secauthsecret --set to set DH-CHAP authentication and create secrets
on each switch
switch:admin> secauthsecret --set
This command is used to set up secret keys for the DH-CHAP authentication.
The minimum length of a secret key is 8 characters and maximum 40
characters.
Setting up secret keys does not initiate DH-CHAP authentication.
If switch is configured to do DH-CHAP, it is performed whenever a port or a
switch is enabled.
Warning: Please use a secure channel for setting secrets.
Using an insecure channel is not safe and may compromise secrets.
Following inputs should be specified for each entry.
WWN for which secret is being set up.
Peer secret: The secret of the peer that authenticates to peer.
Local secret: The local secret that authenticates peer.
Press Enter to start setting up shared secrets >
Enter WWN, Domain, or switch name (Leave blank when done):
10:00:8j:94:7f:40:90:20
Enter peer secret:
Re-enter peer secret:
Enter local secret:
Re-enter local secret:
Warning: Keeping same peer and local secrets can compromise the security of
your network. To change, reenter WWN and secrets.
Enter peer WWN, Domain, or switch name (Leave blank when done):
Are you done? (yes, y, no, n): [no] y
Saving data to key store... Done.
򐂰 Step 3 - Validate switch secrets are now stored in the switch database.

300 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch12.fm

switch:admin> secauthsecret --show


WWN DId Name
-----------------------------------------------------------
10:00:8j:94:7f:40:90:20 60 switchAname_DH-CHAP_example
򐂰 Step 4 - Enable DH-CHAP as the authentication method.
switch:admin> authutil --set -a dhchap
Authentication is set to dhchap.
򐂰 Step 5 - Set the DH Group to 4.
switch:admin> authutil --set -g “4”
switch:admin> authutil --show
AUTH TYPE HASH TYPE GROUP TYPE
-------------------------------------
fcap,dhchap sha256,sha1,md5 4
򐂰 Step 6 - Activate the configuration.
switch:admin> authutil --policy -sw active
Warning: Activating the authentication policy requires either DH-CHAP
secrets or PKI certificates depending on the protocol selected. Otherwise,
ISLs will be segmented during next E-port bring-up.
ARE YOU SURE (yes, y, no, n): [no] yes
Auth Policy is set to active.
򐂰 Step 7 - Validate the authentication configuration.
switch:admin> authutil --show

AUTH TYPE HASH TYPE GROUP TYPE


-------------------------------------
fcap,dhchap sha256,sha1,md5 4
Switch Authentication Policy: ACTIVE
Device Authentication Policy: OFF
򐂰 Step 8 - Configure ports to use encryption for ISLs on local and remote switches.
switch:admin> portdisable 1/23
switch:admin> portcfgencrypt --enable 1/23
switch:admin> portenable 1/23
򐂰 Step 9 - Validate the port’s configuration.
switch:admin> portcfgshow 1/23
Area Number: 23
Octet Speed Combo: 1(32G|16G|8G|4G)
Speed Level: 8G
AL_PA Offset 13: OFF
Trunk Port ON
Long Distance OFF
VC Link Init OFF
Locked L_Port OFF
Locked G_Port OFF
Disabled E_Port OFF
Locked E_Port OFF
ISL R_RDY Mode OFF
RSCN Suppressed OFF
Persistent Disable OFF
LOS TOV mode 0(OFF)
NPIV capability ON

Chapter 12. SAN security 301


8497ch12.fm Draft Document for Review April 5, 2021 12:31 pm

QOS Port AE
Port Auto Disable: OFF
EX Port OFF
Mirror Port OFF
SIM Port OFF
Credit Recovery ON
F_Port Buffers OFF
E_Port Credits OFF
Fault Delay: 0(R_A_TOV)
NPIV PP Limit: 126
NPIV FLOGI Logout: OFF
CSCTL mode: OFF
TDZ mode: OFF
D-Port mode: OFF
D-Port over DWDM: OFF
Compression: OFF
Encryption: ON
10G/16G FEC: ON
16G FEC via TTS: OFF

The following steps describe an alternative way to enable FCAP for ISL Encryption:
򐂰 Step 1 - Generate Switch CSR Files
switch:admin> seccertmgmt generate -csr fcap
򐂰 Step 2 - Export Switch CSR Files to CA for signing
switch:admin> seccertmgmt export -csr fcap
Exact Sample (using complete and customer supplied input)
seccertmgmt export -csr fcap -protocol scp -ipaddr 10.1.1.1 -login
hostsusername -password hostspassword -remotedir /home/fcap/fcap_files
򐂰 Step 3 - Import CA Cert to each Switch
switch:admin> seccertmgmt import -ca -client fcap
The following code is an exact sample (using complete and customer supplied input).
seccertmgmt import -ca -client fcap -protocol scp -ipaddr 10.1.1.1
-remotedir /home/fcap/fcap_files -certname ca.pem -login hostsusername
-password hostspassword
򐂰 Step 4 (On Host) - Verify Switch CSRs
openssl req -text -noout -verify -in FcapSw.csr
򐂰 Step 5 (On Host) - CA signs Switch CSRs
openssl ca -in FcapSw.csr -out FcapSw.pem
򐂰 Step 6 - Verify CA signed Switch CSRs
openssl x509 -in FcapSw.pem -text -noout
򐂰 Step 7 - Validate filename permission on host
chmod 777 FcapSw.pem
򐂰 Step 8 - Import CA signed Switch CSRs
switch:admin> seccertmgmt import -cert fcap
The following code is an exact sample (using complete and customer supplied input).

302 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch12.fm

seccertmgmt import -cert fcap -protocol scp -ipaddr 10.1.1.1 -remotedir


/home/fcap/fcap_files -certname FcapSw.pem -login hostsusername -password
hostspassword
򐂰 Step 9 - Validate switch database
switch:admin> seccertmgmt show -all
򐂰 Step 10 - Enable FCAP Authentication
switch:admin> authutil --set -a fcap
Authentication is set to fcap.
switch:admin> authutil --set -g "4"
DH Group was set to 4.
switch:admin> authutil --set -h sha256
Hash is set to sha256.
򐂰 Step 11 - Configure ports to use ISL Encryption on local and remote switches.
switch:admin> portdisable 1/23
switch:admin> portcfgencrypt --enable 1/23
switch:admin> portenable 1/23
򐂰 Step 12 - Validate the port’s configuration.
switch:admin> portcfgshow 1/23
Area Number: 23
Octet Speed Combo: 1(32G|16G|8G|4G)
Speed Level: 8G
AL_PA Offset 13: OFF
Trunk Port ON
Long Distance OFF
VC Link Init OFF
Locked L_Port OFF
Locked G_Port OFF
Disabled E_Port OFF
Locked E_Port OFF
ISL R_RDY Mode OFF
RSCN Suppressed OFF
Persistent Disable OFF
LOS TOV mode 0(OFF)
NPIV capability ON
QOS Port AE
Port Auto Disable: OFF
EX Port OFF
Mirror Port OFF
SIM Port OFF
Credit Recovery ON
F_Port Buffers OFF
E_Port Credits OFF
Fault Delay: 0(R_A_TOV)
NPIV PP Limit: 126
NPIV FLOGI Logout: OFF
CSCTL mode: OFF
TDZ mode: OFF
D-Port mode: OFF
D-Port over DWDM: OFF
Compression: OFF
Encryption: ON
10G/16G FEC: ON

Chapter 12. SAN security 303


8497ch12.fm Draft Document for Review April 5, 2021 12:31 pm

16G FEC via TTS: OFF

Additional In-flight encryption and compression considerations:


򐂰 Encryption and compression are supported on E_Ports and EX_Ports.
򐂰 ISL ports must be set to Long-Distance (LD) mode when compression is used.
򐂰 Twice the number of buffers should be allocated if compression is enabled for long
distance because frame sizes might be half the size.
򐂰 If both compression and encryption are used, enable compression first.

Implementing ISL encryption using multiple ISLs between the same switch pairs requires that
all ISLs be configured for encryption, or none at all. No more than 4 ports on one ASIC can
be configured with encryption, compression, or both when running at 64 Gbps speed.
Additional ports can be used for data encryption, data compression, or both if the ports are
running lower than 64 Gbp speeds.

Virtual Fabric considerations (encryption and compression): The E_Ports in the


user-created Logical Switch, Base Switch, or default switch can support encryption and
compression. Both encryption and compression are supported on XISL ports, but not on LISL
ports. If encryption or compression is enabled and ports are being moved from one LS to
another, it must be disabled before the ports are moved.

12.10 Policy Database Distribution


Security Policy Database Distribution provides a mechanism for controlling the distribution of
each policy on a per-switch basis. Switches can individually configure policies to either accept
or reject a policy distribution from another switch in the fabric. In addition, a fabric-wide
distribution policy can be defined for the SCC and DCC policies with support for strict,
tolerant, and absent modes. This setting can be used to enforce whether the SCC or DCC
policy must be consistent throughout the fabric.

The Policy Database Distribution has three modes:


򐂰 Strict mode: All updated and new policies of the type specified (SCC, DCC, or both) must
be distributed to all switches in the fabric, and all switches must accept the policy
distribution.
򐂰 Tolerant mode: All updated and new policies of the type specified (SCC, DCC, or both)
are distributed to all switches (FOS V6.2.0 or later) in the fabric. However, the policy does
not need to be accepted.
򐂰 Absent mode: Updated and new policies of the type specified (SCC, DCC, or both) are
not automatically distributed to other switches in the fabric. Policies can still be manually
distributed.

Together, the policy distribution and fabric-wide consistency settings provide a range of
control on the security policies from little or no control to strict control. Strict mode is required
for FICON environments that require High Integrity Fabrics be enabled.

There are other topics as well in this arena that are out of the scope of this document, such as
Encryption of Data-at-Rest and Encryption of Data-in-Flight. These are covered in more
detail in the Brocade Fabric OS Administration Guide, 9.0x
https://docs.broadcom.com/docs/FOS-90x-Admin-AG.

304 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ax01.fm

Appendix A. Fabric details


This appendix provides example tables that you can use to identify dominant factors,
including facilities that can affect the SAN design.

This appendix includes the following topics:


򐂰 A.1, “Current fabrics” on page 306
򐂰 A.2, “Individual fabrics” on page 306
򐂰 A.3, “Device” on page 307
򐂰 A.4, “Metrics and effect on SAN design and performance” on page 307
򐂰 A.5, “Consolidated SAN requirements” on page 308
򐂰 A.6, “Application-specific information” on page 309

© Copyright IBM Corp. 2021. 305


8497ax01.fm Draft Document for Review April 5, 2021 12:31 pm

A.1 Current fabrics


Use the table that is shown in Table A-1 to record the fabrics that are used.

Table A-1 Current fabrics


SAN/ Number Type of Total Domains Number of Number Location Notes
fabric of switches ports servers of storage
switches devices

Fabric 1

Fabric 2

Fabric 3

Fabric 4

Fabric 5

A.2 Individual fabrics


Use the table that is shown in Table A-2 to record the individual fabrics that are used.

Table A-2 Individual fabrics


SAN/ Domain Serial Model Speed WWN IP Brocade Notes
fabric number number addresses FOS/M-EOS
Version

Switch 1

Switch 2

Switch 3

Switch 4

Switch 5

306 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ax01.fm

A.3 Device
Use the table that is shown in Table A-3 to record information about the device that is used.

Table A-3 Devices


Servers Vendor Model WWN Alias Zone Version Application Switches Notes
and
Storage

Server 1

Server 2

Server 3

Storage 1

Storage 2

Storage 3

A.4 Metrics and effect on SAN design and performance


Use the table that is shown in Table A-4 to record the metrics and effects on SAN design.

Table A-4 Metrics and effect on SAN design and performance


Metric Source Effect

Servers in the SAN Estimate/Brocade SAN Health Normal operations

Host Level Mirroring2 Estimate Distance, ISL, traffice levels

Clusters (MSFT, HACMP, NetApp) Estimate In-band heartbeat, frame congestion,


Aaverage number of nodes Estimate: High/Med/Low host fan-in, traffic isolation
Workload level

Virtualization:VIO Server Estimate Frame congestion, edge traffic


Number of servers Estimate increase per port, server fan-in on
Consolidation ratio Estimate target ports, device latencies

Virtualization: VMware Estimate Frame congestion, device latencies,


Number of VMware servers Estimate and SCS12 reser
Consolidated ratio? Yes/No
Shared VMFS? Yes (%)/No
DRS? Yes (%)/No
RDM? High/Med/Low
I/O intensive? Yes/No

Appendix A. Fabric details 307


8497ax01.fm Draft Document for Review April 5, 2021 12:31 pm

A.5 Consolidated SAN requirements


Use the table that is shown in Table A-5 to record the consolidated SAN requirements.

Table A-5 Consolidated SAN requirements


SAN requirements data (complete for each SAN)

Fabric information

Target number of user ports per fabric

Target number of total ports per fabric

Target number of switches per fabric (number of


switches/switch type, total switches)

Number of fabrics

Number of sites in environment

Topology (core-edge, ring, mesh, other)

Maximum hop count

Fabric licenses

SAN device information

Number/types of hosts and operating system platforms

Number/types of storage devices

Number/types of tapes

Number/types of HBAs

Other devices (VTL/deduplication applicance)

Total number of SAN devices per fabric

Customer requirement for failover/redundancy, reliability of


SAN (multipathing software utilized)

Application details

SAN Application (Storage Consolidation, Backup and


Restore, Business Continuance)
Fabric management application(s)

Fabric management application(s)

Performance

Maximum latency (ms)

Targeted ISL oversubscription ratio (3:1, 7:1, 15:1, other)

308 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ax01.fm

A.6 Application-specific information


Use the table that is shown in Table A-6 to record application-specific information.

Table A-6 Application-specific information


Backup/restore infrastructure

Servers

System Operating System Version, Patch Level HBA Driver Version

Server 1/HBA

Server 2/HBA

Server 3/HBA

Backup software

Vendor Version Patch

FC switch

Vendor Model Firmware

Brocade

Storage

Vendor Model Firmware

Array 1

Array 2

Tape library

Vendor Model Firmware

Library

Note: Keep a similar table for each application.

Appendix A. Fabric details 309


8497ax01.fm Draft Document for Review April 5, 2021 12:31 pm

310 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ax02.fm

Appendix B. FOS CLI monitoring commands


Many CLI commands show statistical output based on traffic, interfaces, circuits, tunnels, and
TCP sessions. In this chapter, we described some of the most commonly used commands.

For more information about fabric operating system commands, see the Brocade Fabric OS
Command Reference Manual.

This appendix includes the folliwing topics:


򐂰 B.1, “portshow (tunnel and circuits)” on page 312
򐂰 B.3, “portshow (TCP Stats)” on page 313
򐂰 B.4, “portStatsShow” on page 318
򐂰 B.5, “gePortPerfShow” on page 320
򐂰 B.6, “switchShow” on page 320
򐂰 B.7, “portPerfShow” on page 321
򐂰 B.8, “portErrShow” on page 322

© Copyright IBM Corp. 2021. 311


8497ax02.fm Draft Document for Review April 5, 2021 12:31 pm

B.1 portshow (tunnel and circuits)


Fabric Operating System (FOS) provides for the CLI monitoring of tunnels and circuits. Take
note:
򐂰 State: Up
򐂰 Tunnel: VE24
򐂰 Circuits: 2 (0 and 1)
򐂰 Tx rate: 18.4 MB/s (sending data)
򐂰 Rx rate: 0.4 MB/s (receiving ACK)
򐂰 Load Balance: 9.2 MB/s
򐂰 Compression ratio: 4.4:1, indicating 4 times the WAN rate is coming from the application
򐂰 RTT: 20 ms
򐂰 ReTx (retransmits): IP network has packet loss
򐂰 ARL BW:
򐂰 circuit minimum comm-rate is 100 Mbps
򐂰 circuit maximum comm-rate is 150 Mbps
򐂰 TxWAN%: 73% (transmitting into the minimum comm-rate)
򐂰 Tx Q%: 0 (Transmit Queue % is empty)
򐂰 Met: 0 (failover metric can be 0 or 1, 0 is the default)
򐂰 G: - (group number - no group is the default)

7840-Side-A:FID128:admin> portshow fciptunnel -c --perf --summary


Tunnel Circuit St Flg TxMBps RxMBps CmpRtio RTTms ReTx TxWAN% TxQ%/BW Met/G
-------------------------------------------------------------------------------
24 - Up --- 18.4 0.4 4.4:1 - 19367 73 0 -
24 0 ge2 Up --- 9.2 0.2 - 20 9799 71 100/150 0/-
24 1 ge4 Up --- 9.2 0.2 - 20 9568 75 100/150 0/-
-------------------------------------------------------------------------------
Flg (tunnel): I=IP-Ext, s=Spillover
St: High level state, Up or Dn
TxWAN (tunnel): Tx WAN utilization high of primary circuits (--qos for range)
(circuit): Tx WAN utilization high (--qos for range)
TxQ (tunnel): Tx data buffering utilization high (--qos for range)

B.2 portshow (LAN Stats)


LAN stats per-flow shows active flows from the end devices to the IP Extension gateways. If
there are flows listed, the end device is directing traffic to the IP Extension gateway. The flows
are matching a TCL rule and being directed into a tunnel. The amount of Tx/Rx traffic per flow
is shown.
LVN-7840-A:FID128:admin> portshow lan-stats --per-flow -all
*** Displaying 142 connections ***
DP Idx Src-Address Dst-Address Sport Dport Pro Tx(B/s) Rx(B/s)
-------------------------------------------------------------------------------
DP0 1592 10.13.30.171 10.75.62.74 49630 6690 TCP 0 0
DP1 5599 10.75.52.130 10.89.122.129 62447 5667 TCP 10 0
DP1 8600 10.75.52.133 10.89.122.126 37638 5667 TCP 8 0
… <cut for brevity> …
DP1 6567 10.75.52.130 10.89.122.126 62461 5667 TCP 8 0
DP1 7554 10.75.52.133 10.89.122.127 51029 5667 TCP 8 0
-------------------------------------------------------------------------------

312 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ax02.fm

Sport=Source-Port Dport=Destination-Port Pro=Protocol


DP ActTCP ExdTCP TCLDeny TCLFail
---------------------------------------------------------
DP0 159 0 38928 0
DP1 254 0 53038 0
---------------------------------------------------------
ActTCP=Active TCP Conns ExdTCP=Exceeded TCP Conn Cnt

B.3 portshow (TCP Stats)


Monitoring of TCP statistics shows all the sessions carried by a single circuit (VE24-Circuit0):
LVN-7840-B_CA:FID1:admin> portshow fciptunnel 24 0 --tcp
Tunnel: VE-Port:24 (idx:0, DP0)
====================================================
Oper State : Online
TID : 24
Flags : 0x00000000
IP-Extension : Disabled
Compression : Aggressive Deflate
QoS BW Ratio : 50% / 30% / 20%
Fastwrite : Disabled
Tape Pipelining : Disabled
IPSec : Disabled
Legacy QOS Mode : Disabled
Load-Level (Cfg/Peer): Failover (Failover / Failover)
Local WWN : 10:00:50:eb:1a:69:ee:93
Peer WWN : 10:00:50:eb:1a:fc:e0:da
RemWWN (config) : 00:00:00:00:00:00:00:00
FICON : Enabled
XRC : Enabled
Tape Pipelining : Disabled
Read Block id : Disabled
Tin Tir : Enabled
DevAck : Enabled
Tera Read : Disabled
Tera Write : Disabled
Print : Disabled
FICON Max Read Pipe : 32
FICON Max Write Pipe : 32
FICON Max Read Devs : 16
FICON Max Write Devs : 16
FICON Write Timer : 300
FICON Write Chain : 3200000
FICON OXID Base : 0x9000
FICON Debug Flags : 0x77c90000
cfgmask : 0x003fc33f 0x40000208
Uncomp/Comp Bytes : 5836840800 / 5836828155 / 1.00 : 1
Uncomp/Comp Byte(30s): 24169 / 24169 / 1.00 : 1
Flow Status : 0
ConCount/Duration : 1 / 28d22h
Uptime : 28d22h
Stats Duration : 28d22h
Receiver Stats : 9253712699 bytes / 37116200 pkts / 2.73 KBps Avg

Appendix B. FOS CLI monitoring commands 313


8497ax02.fm Draft Document for Review April 5, 2021 12:31 pm

Sender Stats : 5997624195 bytes / 29887256 pkts / 870.00 Bps Avg


TCP Bytes In/Out : 22185797130 / 19158427516
ReTx/OOO/SloSt/DupAck: 2555 / 1790 / 1519 / 1476
RTT (min/avg/max) : 21 / 21 / 53 ms
Wan Util : 0.1%
TxQ Util : 0.0%
TCP Connection 24.0 HA-Type:Main Pri:Low Conn:0x0a8a941c
===================================================
Local / Remote Port : 57308 / 3225
Duration : 28d22h
MSS : 1460 bytes
ARL Min / Cur / Max : 10000 / 10000 / 50000
ARL Reset Algo : Reset
Send Window
Size / Scale : 1249280 / 9
Slow Start Threshold : 16777216
Congestion Window : 16778676
Pkts InFlight : 0
Recv Window
Size / Scale : 1249792 (Max:1249792) / 9
SendQ Nxt / Min / Max : 0x3a459b50 / 0x3a459b50 / 0x3a459b50
RecvQ Nxt / Min / Max : 0xf20a52e9 / 0xf20a52e9 / 0xf21d6449
RecvQ Pkts : 0
Sender Stats
Sent Bytes / Pkts : 1105903340 / 10103550
Unacked Data : 0
Retransmits Slow / Fast : 177 / 0 (High:0)
SlowStart : 123
Receiver Stats
Recv Bytes / Pkts : 1105864704 / 10103344
Out-of-Order : 7 (High:18)
Duplicate ACKs : 0
RTT / Variance (High) : 20 ms (52 ms) / 0 ms (32 ms)
TCP Connection 24.0 HA-Type:Main Pri:Low Conn:0x0a8a941e
===================================================
Local / Remote Port : 51162 / 3226
Duration : 28d22h
MSS : 1460 bytes
ARL Min / Cur / Max : 10000 / 10000 / 50000
ARL Reset Algo : Reset
Send Window
Size / Scale : 1249280 / 9
Slow Start Threshold : 16777216
Congestion Window : 16778676
Pkts InFlight : 0
Recv Window
Size / Scale : 1249792 (Max:1249792) / 9
SendQ Nxt / Min / Max : 0x3a4555be / 0x3a4555be / 0x3a4555be
RecvQ Nxt / Min / Max : 0xd5999547 / 0xd5999547 / 0xd5aca747
RecvQ Pkts : 0
Sender Stats
Sent Bytes / Pkts : 1105897920 / 10103599
Unacked Data : 0
Retransmits Slow / Fast : 182 / 0 (High:0)
SlowStart : 128

314 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ax02.fm

Receiver Stats
Recv Bytes / Pkts : 1105888500 / 10103382
Out-of-Order : 7 (High:23)
Duplicate ACKs : 0
RTT / Variance (High) : 20 ms (54 ms) / 0 ms (48 ms)
TCP Connection 24.0 HA-Type:Main Pri:Medium Conn:0x0a8a9526
===================================================
Local / Remote Port : 57309 / 3225
Duration : 28d22h
MSS : 1460 bytes
ARL Min / Cur / Max : 15000 / 15000 / 50000
ARL Reset Algo : Reset
Send Window
Size / Scale : 1249280 / 9
Slow Start Threshold : 16777216
Congestion Window : 16778676
Pkts InFlight : 0
Recv Window
Size / Scale : 1249792 (Max:1249792) / 9
SendQ Nxt / Min / Max : 0xfa9fa897 / 0xfa9fa897 / 0xfa9fa897
RecvQ Nxt / Min / Max : 0x22bad4e3 / 0x22bad4e3 / 0x22cde643
RecvQ Pkts : 0
Sender Stats
Sent Bytes / Pkts : 5975313510 / 51406132
Unacked Data : 0
Retransmits Slow / Fast : 462 / 207 (High:0)
SlowStart : 327
Receiver Stats
Recv Bytes / Pkts : 7321106136 / 51436908
Out-of-Order : 0 (High:28)
Duplicate ACKs : 750
RTT / Variance (High) : 20 ms (55 ms) / 0 ms (40 ms)
TCP Connection 24.0 HA-Type:Main Pri:Medium Conn:0x0a8a9527
===================================================
Local / Remote Port : 51163 / 3226
Duration : 28d22h
MSS : 1460 bytes
ARL Min / Cur / Max : 15000 / 15000 / 50000
ARL Reset Algo : Reset
Send Window
Size / Scale : 1249280 / 9
Slow Start Threshold : 16777216
Congestion Window : 16778676
Pkts InFlight : 0
Recv Window
Size / Scale : 1249792 (Max:1249792) / 9
SendQ Nxt / Min / Max : 0xfa9d726d / 0xfa9d71cd / 0xfa9d726d
RecvQ Nxt / Min / Max : 0xdd8bb7a2 / 0xdd8bb7a2 / 0xdd9ec9a2
RecvQ Pkts : 0
Sender Stats
Sent Bytes / Pkts : 5975138146 / 51404866
Unacked Data : 160
Retransmits Slow / Fast : 472 / 203 (High:0)
SlowStart : 342
Receiver Stats

Appendix B. FOS CLI monitoring commands 315


8497ax02.fm Draft Document for Review April 5, 2021 12:31 pm

Recv Bytes / Pkts : 7320332818 / 51434434


Out-of-Order : 0 (High:31)
Duplicate ACKs : 701
RTT / Variance (High) : 20 ms (52 ms) / 0 ms (35 ms)
TCP Connection 24.0 HA-Type:Main Pri:High Conn:0x0a8a962f
===================================================
Local / Remote Port : 57310 / 3225
Duration : 28d22h
MSS : 1460 bytes
ARL Min / Cur / Max : 25000 / 25000 / 50000
ARL Reset Algo : Reset
Send Window
Size / Scale : 1249280 / 9
Slow Start Threshold : 16777216
Congestion Window : 16778676
Pkts InFlight : 0
Recv Window
Size / Scale : 1249792 (Max:1249792) / 9
SendQ Nxt / Min / Max : 0x3a455ef4 / 0x3a455ef4 / 0x3a455ef4
RecvQ Nxt / Min / Max : 0x181ac577 / 0x181ac577 / 0x182dd6d7
RecvQ Pkts : 0
Sender Stats
Sent Bytes / Pkts : 1105865240 / 10103399
Unacked Data : 0
Retransmits Slow / Fast : 192 / 0 (High:0)
SlowStart : 133
Receiver Stats
Recv Bytes / Pkts : 1105894888 / 10103203
Out-of-Order : 20 (High:23)
Duplicate ACKs : 0
RTT / Variance (High) : 20 ms (58 ms) / 0 ms (43 ms)
TCP Connection 24.0 HA-Type:Main Pri:High Conn:0x0a8a9631
===================================================
Local / Remote Port : 51164 / 3226
Duration : 28d22h
MSS : 1460 bytes
ARL Min / Cur / Max : 25000 / 25000 / 50000
ARL Reset Algo : Reset
Send Window
Size / Scale : 1249280 / 9
Slow Start Threshold : 16777216
Congestion Window : 16778676
Pkts InFlight : 0
Recv Window
Size / Scale : 1249792 (Max:1249792) / 9
SendQ Nxt / Min / Max : 0x3a459e77 / 0x3a459e77 / 0x3a459e77
RecvQ Nxt / Min / Max : 0x46b0fc20 / 0x46b0fc20 / 0x46c40e20
RecvQ Pkts : 0
Sender Stats
Sent Bytes / Pkts : 1105886288 / 10103344
Unacked Data : 0
Retransmits Slow / Fast : 171 / 0 (High:0)
SlowStart : 122
Receiver Stats
Recv Bytes / Pkts : 1105864848 / 10103140

316 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ax02.fm

Out-of-Order : 5 (High:29)
Duplicate ACKs : 0
RTT / Variance (High) : 20 ms (54 ms) / 0 ms (43 ms)
TCP Connection 24.0 HA-Type:Main Pri:Control Conn:0x0a8a9739
===================================================
Local / Remote Port : 57311 / 3225
Duration : 28d22h
MSS : 1460 bytes
ARL Min / Cur / Max : 0 / 0 / 50000
ARL Reset Algo : Reset
Send Window
Size / Scale : 1249280 / 9
Slow Start Threshold : 16777216
Congestion Window : 16778676
Pkts InFlight : 0
Recv Window
Size / Scale : 1249792 (Max:1249792) / 9
SendQ Nxt / Min / Max : 0x447a03c6 / 0x447a03c6 / 0x447a03c6
RecvQ Nxt / Min / Max : 0x0f08db12 / 0x0f08db12 / 0x0f1bed12
RecvQ Pkts : 0
Sender Stats
Sent Bytes / Pkts : 1392172536 / 12980231
Unacked Data : 0
Retransmits Slow / Fast : 241 / 4 (High:0)
SlowStart : 168
Receiver Stats
Recv Bytes / Pkts : 1560151020 / 13039059
Out-of-Order : 2 (High:11)
Duplicate ACKs : 12
RTT / Variance (High) : 20 ms (63 ms) / 0 ms (43 ms)
TCP Connection 24.0 HA-Type:Main Pri:Control Conn:0x0a8a973a
===================================================
Local / Remote Port : 51165 / 3226
Duration : 28d22h
MSS : 1460 bytes
ARL Min / Cur / Max : 0 / 0 / 50000
ARL Reset Algo : Reset
Send Window
Size / Scale : 1249280 / 9
Slow Start Threshold : 16777216
Congestion Window : 16778676
Pkts InFlight : 0
Recv Window
Size / Scale : 1249792 (Max:1249792) / 9
SendQ Nxt / Min / Max : 0x447b4df6 / 0x447b4df6 / 0x447b4df6
RecvQ Nxt / Min / Max : 0x8e1cf9f2 / 0x8e1cf9f2 / 0x8e300bf2
RecvQ Pkts : 0
Sender Stats
Sent Bytes / Pkts : 1392250536 / 12980470
Unacked Data : 0
Retransmits Slow / Fast : 240 / 4 (High:0)
SlowStart : 176
Receiver Stats
Recv Bytes / Pkts : 1560694216 / 13039703
Out-of-Order : 0 (High:14)

Appendix B. FOS CLI monitoring commands 317


8497ax02.fm Draft Document for Review April 5, 2021 12:31 pm

Duplicate ACKs : 13
RTT / Variance (High) : 20 ms (52 ms) / 0 ms (42 ms)

B.4 portStatsShow
Many port statistics are maintained. To display port traffic and error statistics, use the portstatsshow
CLI command. To get FC, VE, and GE detailed statistics on the CLI issue portstatsshow <port#>.
FC and VE ports use just the number alone <#>. GE interfaces use ge<#>. Below is an example of the
output for FC port 0.
<< Note to Editor: Table needs to be formatted. Compare it with Word doc. >>
7840-Side-A:FID128:admin> portstatsshow
Usage:
portstatsshow [--detail] PortNumber
OR
portstatsshow [--detail] -i <port_index | portindex_range> [-f]
OR
portstatsshow [--detail] <-slot | -s > <slot# | slotrange>
-i: Confirms port swap has been disabled and to give port index as operand
-x: Confirms port swap has been disabled and to give port index
in HEX format as operand
-f: ignores non-existing indexes
portindex_range:Specifies the range of port index <portindex1-portindex2>
(example: 12-14)
OR
portstatsshow [<SlotNumber>/]ge<PortNumber>
7840-Side-A:FID128:admin> portstatsshow ge2
ge_stat_tx_frms 542382084 GE transmitted frames
ge_stat_tx_octets 636962488472 GE transmitted octets
ge_stat_tx_ucast_frms 542237792 GE transmitted unicast frames
ge_stat_tx_mcast_frms 0 GE transmitted multicast frames
ge_stat_tx_bcast_frms 144292 GE transmitted broadcast frames
ge_stat_tx_vlan_frms 0 GE transmitted vlan frames
ge_stat_tx_pause_frms 0 GE transmitted pause frames
ge_stat_rx_frms 258254545 GE received frames
ge_stat_rx_octets 40648834562 GE received octets
ge_stat_rx_ucast_frms 258110264 GE received unicast frames
ge_stat_rx_mcast_frms 0 GE received multicast frames
ge_stat_rx_bcast_frms 144281 GE received broadcast frames
ge_stat_rx_vlan_frms 0 GE received vlan frames
ge_stat_rx_pause_frms 0 GE received pause frames
ge_err_crc 0 GE CRC Errors
ge_err_carrier 1 GE lost carrier sense
ge_err_jabber 0 GE jabbers
ge_stat_tx_octets 636962488472 GE transmitted
octets
ge_stat_tx_pkts64octets 347 GE transmitted
64byte octets
ge_stat_tx_pkts65to127octets 62670368 GE transmitted
65to127byte octets
ge_stat_tx_pkts128to255octets 21247343 GE transmitted
128to255byte octets
ge_stat_tx_pkts256to511octets 34952514 GE transmitted
256to511byte octets

318 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ax02.fm

ge_stat_tx_pkts512to1023octets 1495311 GE transmitted


512to1023byte octets
ge_stat_tx_pkts1024to1518octets 422016201 GE transmitted
1024to1518byte octets
ge_stat_tx_pkts1519to2047octets 0 GE transmitted
1519to2047byte octets
ge_stat_tx_pkts2048to4095octets 0 GE transmitted
2048to4095byte octets
ge_stat_tx_pkts4096to9216octets 0 GE transmitted
4096to9216byte octets
ge_stat_rx_octets 40648834562 GE received
octets
ge_stat_rx_pkts64octets 399 GE received
64byte octets octets
ge_stat_rx_pkts65to127octets 143348534 GE received
65to127byte octets
ge_stat_rx_pkts128to255octets 43617129 GE received
128to255byte octets
ge_stat_rx_pkts256to511octets 70946937 GE received
256to511byte octets
ge_stat_rx_pkts512to1023octets 140918 GE received
512to1023byte octets
ge_stat_rx_pkts1024to1518octets 200628 GE received
1024to1518byte octets
ge_stat_rx_pkts1519to2047octets 0 GE received
1519to2047byte octets
ge_stat_rx_pkts2048to4095octets 0 GE received
2048to4095byte octets
ge_stat_rx_pkts4096to9216octets 0 GE received
4096to9216byte octets
ge_stat_rx_pfc_control_frame 0 GE Rx PFC control
frame
ge_stat_tx_pfc_control_frame 0 GE Tx PFC control
frame
ge_stat_rx_dvlan_tag_frame 0 GE Rx Double VLAN
tag frame
ge_stat_tx_dvlan_tag_frame 0 GE Tx Double VLAN
tag frame
7840-Side-A:FID128:admin> portstatsshow 0
stat_wtx 217334 4-byte words transmitted
stat_wrx 77324 4-byte words received
stat_ftx 8841 Frames transmitted
stat_frx 4537 Frames received
stat_c2_frx 0 Class 2 frames received
stat_c3_frx 4537 Class 3 frames received
stat_lc_rx 0 Link control frames received
stat_mc_rx 0 Multicast frames received
stat_mc_to 0 Multicast timeouts
stat_mc_tx 0 Multicast frames transmitted
tim_rdy_pri 0 Time R_RDY high priority
tim_txcrd_z 0 Time TX Credit Zero (2.5Us ticks)
tim_txcrd_z_vc 0- 3: 0 0 0 0
tim_txcrd_z_vc 4- 7: 0 0 0 0
tim_txcrd_z_vc 8-11: 0 0 0 0
tim_txcrd_z_vc 12-15: 0 0 0 0

Appendix B. FOS CLI monitoring commands 319


8497ax02.fm Draft Document for Review April 5, 2021 12:31 pm

tim_latency_vc 0- 3: 1 1 1 1
tim_latency_vc 4- 7: 1 1 1 1
tim_latency_vc 8-11: 1 1 1 1
tim_latency_vc 12-15: 1 1 1 1
fec_cor_detected 0 Count of blocks that were corrected by
FEC
fec_uncor_detected 0 Count of blocks that were left
uncorrected by FEC
er_enc_in 0 Encoding errors inside of frames
er_crc 0 Frames with CRC errors
er_trunc 0 Frames shorter than minimum
er_toolong 0 Frames longer than maximum
er_bad_eof 0 Frames with bad end-of-frame
er_enc_out 0 Encoding error outside of frames
er_bad_os 0 Invalid ordered set
er_pcs_blk 0 PCS block errors
er_rx_c3_timeout 0 Class 3 receive frames discarded due
to timeout
er_tx_c3_timeout 0 Class 3 transmit frames discarded due
to timeout
er_unroutable 0 Frames that are unroutable
er_unreachable 0 Frames with unreachable destination
er_other_discard 0 Other discards
er_type1_miss 0 frames with FTB type 1 miss
er_type2_miss 0 frames with FTB type 2 miss
er_type6_miss 0 frames with FTB type 6 miss
er_zone_miss 0 frames with hard zoning miss
er_lun_zone_miss 0 frames with LUN zoning miss
er_crc_good_eof 0 Crc error with good eof
er_inv_arb 0 Invalid ARB
er_single_credit_loss 0 Single vcrdy/frame loss on link
er_multi_credit_loss 0 Multiple vcrdy/frame loss on link
other_credit_loss 0 Link timeout/complete credit loss
phy_stats_clear_ts 0 Timestamp of phy_port stats clear
lgc_stats_clear_ts 0 Timestamp of lgc_port stats clear

B.5 gePortPerfShow
To monitor the data rates through the GE interfaces, use the geportperfshow CLI command.
7840-Side-A:FID128:admin> geportperfshow
Throughput of GE port
ge0 ge1 ge2 ge3 ge4 ge5 ge6 ge7 ge8
========================================================================
0 0 11.3m 0 11.1m 0 0 0 0
ge9 ge10 ge11 ge12 ge13 ge14 ge15 ge16 ge17 Total
================================================================================
0 0 0 0 0 0 0 0 0 22.5m

B.6 switchShow
The best way to show the status of a VE_Port is using the switchshow command. In the
example below, VE_Port 12 shows the port type (VE), port status (online), and the connected

320 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ax02.fm

device “Remote7810”. Be aware that for the GE interfaces, the notation FCIP indicates WAN.
FCIP terminology remains due to legacy output prior to IP Extension.
switchshow

Example:
Local7810:admin> switchshow
switchName: Local7810
switchType: 178.0
switchState: Online
switchMode: Native
switchRole: Principal
switchDomain: 1
switchId: fffc01
switchWwn: 10:00:88:94:71:5c:f7:e2
zoning: OFF
switchBeacon: OFF
FC Router: OFF
HIF Mode: OFF
Index Port Address Media Speed State Proto
==================================================
0 0 010000 -- N32 No_Module FC
1 1 010100 -- N32 No_Module FC
2 2 010200 -- N32 No_Module FC
3 3 010300 -- N32 No_Module FC
4 4 010400 -- N32 No_Module FC
5 5 010500 -- N32 No_Module FC
6 6 010600 -- N32 No_Module FC
7 7 010700 -- N32 No_Module FC
8 8 010800 -- N32 No_Module FC
9 9 010900 -- N32 No_Module FC
10 10 010a00 -- N32 No_Module FC
11 11 010b00 -- N32 No_Module FC
12 12 010c00 -- -- Online VE VE-Port
10:00:88:94:71:60:98:5e "Remote7810" (downstream)
13 13 010d00 -- -- Offline VE Disabled (Persistent)
14 14 010e00 -- -- Offline VE Disabled (Persistent)
15 15 010f00 -- -- Offline VE Disabled (Persistent)
ge0 cu 1G Offline FCIP Copper Disabled
ge1 cu 1G Offline FCIP Copper Disabled
ge2 id 10G Online LAN
ge3 id 10G Online LAN
ge4 id 10G No_Sync LAN Disabled (Persistent)
ge5 id 10G No_Sync LAN Disabled (Persistent)
ge6 id 10G Online FCIP
ge7 id 10G Online FCIP

B.7 portPerfShow
<< Note to Editor: For the code sample, snipit the code sample taken from the Word doc >>

PortPerfShow is a real-time iterative CLI output showing the current rate of traffic on the port.
Only FC/FICON and VE ports are shown with PortPerfShow, use geportperfshow for GE

Appendix B. FOS CLI monitoring commands 321


8497ax02.fm Draft Document for Review April 5, 2021 12:31 pm

interfaces. In this case, the two FC ports (0 & 1) are feeding into VE_Port 24 for transmission
across FCIP.
7840-Side-A:FID128:admin> portperfshow

Figure B-1 portPerfShow command

B.8 portErrShow
<< Note to Editor: For the code sample, snipit the code sample taken from the Word doc >>

To view the number of transmitted/received frames and errors on F_Port (FC fabric ports) and
VE_Port (FCIP Virtual E_Ports), use porterrshow.
7840-Side-A:FID128:admin> porterrshow

Figure B-2 portErrShow command

322 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
To determine the spine width of a book, you divide the paper PPI into the number of pages in the book. An example is a 250 page book using Plainfield opaque 50# smooth which has a PPI of 526. Divided
250 by 526 which equals a spine width of .4752". In this case, you would use the .5” spine. Now select the Spine width for the book and hide the others: Special>Conditional
Text>Show/Hide>SpineSize(-->Hide:)>Set . Move the changed Conditional text settings to all files in your book by opening the book file with the spine.fm still open and File>Import>Formats the
Conditional Text Settings (ONLY!) to the book files.
Draft Document for Review April 5, 2021 12:31 pm 8497spine.fm 323
IBM b-type Gen 7 SG24-8497-00
Installation, Migration, and Best ISBN 0738459437
(1.5” spine)
1.5”<-> 1.998”
789 <->1051 pages
IBM b-type Gen 7 SG24-8497-00
Installation, Migration, and Best ISBN 0738459437
(1.0” spine)
0.875”<->1.498”
460 <-> 788 pages
SG24-8497-00
IBM b-type Gen 7 Installation, Migration, and Best Practices GuideISBN 0738459437
(0.5” spine)
0.475”<->0.873”
250 <-> 459 pages
IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
(0.2”spine)
0.17”<->0.473”
90<->249 pages
(0.1”spine)
0.1”<->0.169”
53<->89 pages
To determine the spine width of a book, you divide the paper PPI into the number of pages in the book. An example is a 250 page book using Plainfield opaque 50# smooth which has a PPI of 526. Divided
250 by 526 which equals a spine width of .4752". In this case, you would use the .5” spine. Now select the Spine width for the book and hide the others: Special>Conditional
Text>Show/Hide>SpineSize(-->Hide:)>Set . Move the changed Conditional text settings to all files in your book by opening the book file with the spine.fm still open and File>Import>Formats the
Conditional Text Settings (ONLY!) to the book files.
Draft Document for Review April 5, 2021 12:31 pm 8497spine.fm 324
IBM b-type Gen 7 SG24-8497-00
Installation, Migration, ISBN 0738459437
(2.5” spine)
2.5”<->nnn.n”
1315<-> nnnn pages
IBM b-type Gen 7 SG24-8497-00
Installation, Migration, and Best ISBN 0738459437
Practices Guide
(2.0” spine)
2.0” <-> 2.498”
1052 <-> 1314 pages
Back cover
Draft Document for Review April 5, 2021 12:31 pm

SG24-8497-00

ISBN 0738459437

Printed in U.S.A.

®
ibm.com/redbooks

You might also like