IBM B Type Gen 7 Installation Migration and Best Practices Guide
IBM B Type Gen 7 Installation Migration and Best Practices Guide
Redbooks
Draft Document for Review April 5, 2021 12:31 pm 8497edno.fm
IBM Redbooks
April 2021
SG24-8497-00
8497edno.fm Draft Document for Review April 5, 2021 12:31 pm
Note: Before using this information and the product it supports, read the information in “Notices” on
page ix.
Contents
Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .x
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
Authors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
Now you can become a published author, too! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv
Stay connected to IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration . . . 137
7.1 Extension platform overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
7.1.1 IBM b-type Gen 7 extension platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
7.2 Extension design and architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
7.2.1 FCIP design and architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
7.2.2 IP Extension design and architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
7.3 Extension implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
7.3.1 The WAN Side . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
7.3.2 The FC/FICON side . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
7.3.3 The LAN Side (IP Extension) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
7.4 Extension migration strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
7.4.1 IBM Extension Platforms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
7.4.2 IBM Extension Solution Designs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
7.4.3 Migration methodology and techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
7.4.4 Migration Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
7.5 Extension validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
Contents v
8497TOC.fm Draft Document for Review April 5, 2021 12:31 pm
Contents vii
8497TOC.fm Draft Document for Review April 5, 2021 12:31 pm
viii IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497spec.fm
Notices
This information was developed for products and services offered in the US. This material might be available
from IBM in other languages. However, you may be required to own a copy of the product or product version in
that language in order to access it.
IBM may not offer the products, services, or features discussed in this document in other countries. Consult
your local IBM representative for information on the products and services currently available in your area. Any
reference to an IBM product, program, or service is not intended to state or imply that only that IBM product,
program, or service may be used. Any functionally equivalent product, program, or service that does not
infringe any IBM intellectual property right may be used instead. However, it is the user’s responsibility to
evaluate and verify the operation of any non-IBM product, program, or service.
IBM may have patents or pending patent applications covering subject matter described in this document. The
furnishing of this document does not grant you any license to these patents. You can send license inquiries, in
writing, to:
IBM Director of Licensing, IBM Corporation, North Castle Drive, MD-NC119, Armonk, NY 10504-1785, US
This information could include technical inaccuracies or typographical errors. Changes are periodically made
to the information herein; these changes will be incorporated in new editions of the publication. IBM may make
improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time
without notice.
Any references in this information to non-IBM websites are provided for convenience only and do not in any
manner serve as an endorsement of those websites. The materials at those websites are not part of the
materials for this IBM product and use of those websites is at your own risk.
IBM may use or distribute any of the information you provide in any way it believes appropriate without
incurring any obligation to you.
The performance data and client examples cited are presented for illustrative purposes only. Actual
performance results may vary depending on specific configurations and operating conditions.
Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products and cannot confirm the
accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the
capabilities of non-IBM products should be addressed to the suppliers of those products.
Statements regarding IBM’s future direction or intent are subject to change or withdrawal without notice, and
represent goals and objectives only.
This information contains examples of data and reports used in daily business operations. To illustrate them
as completely as possible, the examples include the names of individuals, companies, brands, and products.
All of these names are fictitious and any similarity to actual people or business enterprises is entirely
coincidental.
COPYRIGHT LICENSE:
This information contains sample application programs in source language, which illustrate programming
techniques on various operating platforms. You may copy, modify, and distribute these sample programs in
any form without payment to IBM, for the purposes of developing, using, marketing or distributing application
programs conforming to the application programming interface for the operating platform for which the sample
programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore,
cannot guarantee or imply reliability, serviceability, or function of these programs. The sample programs are
provided “AS IS”, without warranty of any kind. IBM shall not be liable for any damages arising out of your use
of the sample programs.
Trademarks
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines
Corporation, registered in many jurisdictions worldwide. Other product and service names might be
trademarks of IBM or other companies. A current list of IBM trademarks is available on the web at “Copyright
and trademark information” at http://www.ibm.com/legal/copytrade.shtml
The following terms are trademarks or registered trademarks of International Business Machines Corporation,
and might also be trademarks or registered trademarks in other countries.
AIX® IBM Spectrum® z/OS®
FICON® Redbooks® z15™
IBM® Redbooks (logo) ®
IBM FlashSystem® System Z®
Intel, Intel logo, Intel Inside logo, and Intel Centrino logo are trademarks or registered trademarks of Intel
Corporation or its subsidiaries in the United States and other countries.
The registered trademark Linux® is used pursuant to a sublicense from the Linux Foundation, the exclusive
licensee of Linus Torvalds, owner of the mark on a worldwide basis.
Microsoft, Windows, and the Windows logo are trademarks of Microsoft Corporation in the United States,
other countries, or both.
Ansible, Red Hat, are trademarks or registered trademarks of Red Hat, Inc. or its subsidiaries in the United
States and other countries.
UNIX is a registered trademark of The Open Group in the United States and other countries.
VMware, and the VMware logo are registered trademarks or trademarks of VMware, Inc. or its subsidiaries in
the United States and/or other jurisdictions.
Other company, product, or service names may be trademarks or service marks of others.
Preface
The IBM® b-type Gen 7 Fibre Channel directors and switches provide reliable, scalable, and
secure high-performance foundations for high-density server virtualization, cloud
architectures, and next generation flash and SSD storage. They are designed to meet the
demands of highly virtualized private cloud storage and data center environments.
Read this publication to learn about fabric design, managing and monitoring your SAN
environment.
Authors
This book was produced by a team of specialists from around the world working at IBM
Redbooks, Poughkeepsie Center.
xii IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497pref.fm
Find out more about the residency program, browse the residency index, and apply online at:
ibm.com/redbooks/residencies.html
Preface xiii
8497pref.fm Draft Document for Review April 5, 2021 12:31 pm
Comments welcome
Your comments are important to us!
We want our books to be as helpful as possible. Send us your comments about this book or
other IBM Redbooks publications in one of the following ways:
Use the online Contact us review Redbooks form found at:
ibm.com/redbooks
Send your comments in an email to:
[email protected]
Mail your comments to:
IBM Corporation, IBM Redbooks
Dept. HYTD Mail Station P099
2455 South Road
Poughkeepsie, NY 12601-5400
xiv IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch01.fm
IBM b-type Gen 7 technology delivers a collection of features that combine comprehensive
data collection capabilities with powerful analytics to quickly understand the health and
performance on the environment and identify potential impact or trending problems. These
features are the foundation for realizing a self-learning, self-optimizing, and self-healing
autonomous SAN. IBM b-type Gen 7 technology provides unprecedented visibility and
actionable intelligence across the storage network. The information captured is displayed in
IBM SANnav Management Portal to quickly identify and isolate problems before they impact
application availability. The combination of these features gives IT administrators the ability to
quickly enable an autonomous infrastructure.
1.1.1 Self-learning
IBM b-type Gen 7 technology proactively monitors millions of I/O performance and behavior
data points through integrated network sensors to gain deep insight into the environment.
These data points are transformed into actionable intelligence that can monitor and alert
when there are any abnormal changes. Through these capabilities, an admin can identify
individual applications and their performance characteristics across the fabric, as well as
identify the performance of the various devices that comprise the fabric (the switches, hosts,
and targets):
I/O Insight: Proactively monitors I/O performance and behavior through integrated network
sensors to baseline application performance and ensure operational stability. By
combining the instrumentation of IO Insight with the ability to self-learn the traffic flows, the
IBM b-type Gen 7 autonomous SAN technology can generate performance and health
metrics on each component as well as the applications to monitor for changes and alert
the administrator with MAPS.
Monitoring and Alerting Policy Suite (MAPS): Provides an easy-to- use solution for
policy-based threshold monitoring and alerting. MAPS proactively monitors the health and
performance of any SCSI or NVMe storage infrastructure to ensure application uptime and
availability.
1.1.2 Self-optimizing
Utilizing actionable intelligence gathered from self-learning capabilities, IBM b-type Gen 7
Fibre Channel SANs automatically apply priorities for specific data traffic to help guarantee
performance levels and monitor for traffic pattern shifts. Learning traffic behavior enables the
network to make smarter decisions on traffic prioritization, congestion avoidance, and
adjustment to ensure optimal network performance for applications and storage. When
something does change, the IBM b-type Gen 7 autonomous SAN technology will isolate the
port traffic for the misbehaving device to a virtual channel in the fabric and allow all other
traffic to go around to maintain optimal performance. In addition, IBM SANnav Management
Portal has automated manual activities such as infrastructure deployment and provisioning to
expedite IT services. Self-optimizing features include the following:
Traffic Optimizer: Automatically classifies and separates traffic with similar characteristics
to optimize performance for most common SAN configurations. It identifies and isolates
traffic flows to prevent negative impact to overall SAN performance.
Advanced traffic shaping: Guarantees application performance by proactively monitoring
and actively shaping traffic.
REST APIs: Eliminate human errors and performance impacts through open DevOps.
SAN Automation and Ansible: Gain cloud-like SAN orchestration for optimizing
administration resources through IBM b-type Gen 7 REST API automation technology.
1.1.3 Self-healing
When potential disruptions are detected, the network will automatically mitigate or resolve
issues without manual intervention. This is done by proactively monitoring the network to
automatically identify abnormal or unexpected infrastructure behavior and then taking
immediate action. These actions include alerting the end devices of the issue through a
notification and signaling process within a SAN. End devices can automatically adjust traffic
or fail over to a healthy path to mitigate impact until a further infrastructure change solution
can be implemented. In addition, these capabilities detect and automatically reconfigure
fabric misconfigurations that fall outside best practices Self-healing features include:
Fabric congestion notification: Automatically detects congestion and notifies end devices
to automatically mitigate congestion.
Slow drain device quarantine: Automatically quarantines credit-stalled devices to prevent
the misbehaving device from impacting the rest of traffic.
Automatic actions: Ensure data delivery with automatic failover from physical or
congestion issues such as port decommissioning, port toggling, and port fencing.
COMPASS: Detects and automatically reconfigures out-of-compliance configurations.
SANnav Management Portal: Reduces troubleshooting steps with built-in best practice
recommendations to quickly resolve issues.
Through the modernization process, enterprises are facing the reality that their revenue is
intertwined with the success or failure of the IT organization. To succeed, IT needs to
automate as much as possible to eliminate complexity and reduce costs. IT organizations
need to remove tedious, time-consuming, and labor-intensive tasks so they can focus on
delivering services to the business that can help deliver additional revenue. IBM b-type Gen
7’s autonomous SAN delivers automation and intelligence to admins so they do not have to
worry about the SAN and instead can focus on initiatives that are strategic to their
organization.
Table 1-1 lists the IBM b-type Gen 7 and Brocade naming conventions for the Gen 7 platform
offerings.
To meet ever-increasing demands for faster, more reliable data access, it is essential for
organizations to deploy a modernized infrastructure that reduces latency, increases
bandwidth, and ensures continuous availability. Unprecedented performance is not enough
on its own. Powerful analytics and advanced automation capabilities are required to transform
current storage networks into an autonomous SAN. This requires a network that is capable of
delivering these capabilities to maximize performance, simplify management, and reduce
operational costs. Legacy infrastructure was not designed to support the performance
requirements of evolving workloads and NVMe-based storage. In fact, an aging network will
impede the performance of the on-demand data center. By modernizing the storage network
with IBM b-type Gen 7, organizations will enable a faster, more intelligent, and more resilient
network. This will maximize the performance, productivity, and efficiency of their storage
investments and resources, even as they rapidly scale their environments.
The IBM b-type Gen 7 Director provides a modular building block, purpose-built for scalability
to accommodate growth and power large-scale storage environments. With a 50% latency
reduction compared to the previous generation, IBM b-type Gen 7 Directors maximize the
performance of NVMe storage and high-transaction workloads, eliminating I/O bottlenecks
and unleashing the full performance of next-generation storage. In addition, the IBM b-type
Gen 7 Director lays the foundation for the autonomous SAN. With autonomous SAN
technology, the director harnesses the power of analytics and the simplicity of automation to
optimize performance, ensure reliability, and simplify management. Leveraging these
capabilities enables organizations to realize a self-learning, self-optimizing, and self-healing
SAN. IBM b-type Gen 7 Directors provide up to 384 64Gb/s line rate ports or up to 512
32Gb/s line rate ports, enabling organizations to scale more devices, applications, and
workloads. With diverse deployment options, multiprotocol flexibility, and mixed blade
capability, organizations can adapt and optimize their businesses to meet next-generation
storage and server requirements. The IBM b-type Gen 7 Director supports the concurrent use
of both traditional Fibre Channel and NVMe storage traffic, allowing organizations to
seamlessly integrate IBM b-type Gen 7 Fibre Channel networks with next-generation
NVMe-based storage, without a disruptive rip-and-replace.
ports for device connectivity and allowing flexible SAN design that supports core-edge or
mesh topologies.
The 14U IBM b-type Gen 7 SAN512B-7 Director is built for large enterprise networks and has
eight vertical blade slots to provide up to 384 64Gb/s line rate ports or up to 512 32Gb/s line
rate ports for device connectivity. An additional 32 UltraScale InterChassis Link (ICL)
connections provide 128 ports for chassis-to-chassis interconnect.
The 8U IBM b-type Gen 7-4 Director in built for midsize networks and has four horizontal
blade slots to provide up to 192 64Gb/s line rate ports or up to 256 32Gb/s line rate ports for
device connectivity.
To support disaster recovery and data protection storage solutions over long distances, the
IBM b-type Gen 7 SX6 Extension Blade provides flexible Fibre Channel and IP storage
replication deployment options with 16 32Gb/s Fibre Channel ports, 16 1/10-GbE ports, and 2
40GbE ports. This blade allows organizations to seamlessly integrate extension capabilities
within the IBM b-type Gen 7 Director to provide replication services for large-scale, multisite
data center environments that implement block, file, and tape data protection solutions. The
IBM b-type Gen 7 SX6 Extension Blade can be deployed with the IBM b-type Gen 7 7840
Extension Switch and the IBM b-type Gen 7 7810 Extension Switch in a data-center to-edge
architecture as a cost effective option for connecting primary data centers with remote data
centers and offices.
IBM b-type Gen 7 directors build upon years of innovation and leverage the core technology
of IBM b-type Gen 7 systems to consistently deliver five-nines availability in the world’s most
demanding data centers. Delivering non-disruptive software upgrades, hot-pluggable
components, and a no-single-point-of-failure design, the IBM b-type Gen 7 offers a highly
resilient solution for today’s enterprise-class storage environments.
Product Features Key product features for this device include the following:
Redundant and hot-swappable SFP+, SFP28, QSFP+, and QSFP56 transceivers; port,
extension, control processor (CP), and core routing (CR) blades; power supply
assemblies, fan assemblies, and WWN cards that enable a high availability platform and
allow non-disruptive software upgrades for mission-critical SAN applications.
Support for the following features when using the FC32-64 blade:
– Up to 16 QSFP ports on each FC32-64 port blade that provide 4x32Gb/s, 4x16Gb/s,
4x8Gb/s, 4x4Gb/s, 4x25GbE, or 40GbE operation. Port speeds for 4x32Gb/s
transceivers can auto-negotiate between 4x32Gb/s and 4x16Gb/s. Port speeds for
4x16Gb/s transceivers can auto-negotiate between 4x16Gb/s, 4x8Gb/s, and 4x4Gb/s.
– Flexport technology that allows you to configure each of the 16 QSFP ports on a blade
for FC operation or Ethernet operation for FCoE connections. For details on supported
transceivers and speeds, see Supported Transceivers Matrix
https://www.broadcom.com/products/fibre-channel-networking/transceiver-modul
es.
– Up to four blades and all FC transceivers installed, up to 256 32Gb/s external ports are
possible in a single chassis, enabling high-density SAN configurations with a reduced
footprint.
– Trunking technology that allows groups of up to eight ports to create high-performance
256Gb/s ISL trunks between switches using 32Gb/s ports.
Support for the following features when using the FC64-48 blade:
– Port blade with 48 front-end 64Gb/s FC SFP+ ports with the edge switch interconnect
to the core blades.
– Supported SFPs include 10Gb/s, 32Gb/s, and 64Gb/s FC.
– Support for 48 Fibre Channel ports on each FC64-48 port blade that provide 64Gb/s,
32Gb/s, 16Gb/s, 10Gb/ s, and 8Gb/s. The 10Gb/s transceivers can be used for any
port on the FC64-48 port blades. Note that 10Gb/s transceivers on the FC64-48 and
SX6 blade are not interchangeable. For details on supported transceivers and speeds,
see Supported Transceiver Matrix
https://www.broadcom.com/products/fibre-channel-networking/transceiver-modul
es.
– Trunking technology groups up to eight ports to create high-performance 512Gb/s ISL
trunks between switches using 64Gb/s ports.
Support for the following features when using the FC32-X7-48 blade:
– Port blade with 48 front-end 32Gb/s FC SFP+ ports.
– Supported SFPs include 32Gb/s, 16Gb/s, and 10Gb/s FC.
– Support for 48 Fibre Channel ports on each FC32-X7-48 port blade that provide
32Gb/s, 16Gb/s, 10Gb/s, 8Gb/s, and 4Gb/s. The 10Gb/s transceivers can be used for
any port on the FC32-X7-48 port blades. Note that 10Gb/s transceivers on the
FC32-X7-48 and SX6 blade are not interchangeable. For details on supported
transceivers and speeds, see Supported Transceiver Matrix
https://www.broadcom.com/products/fibre-channel-networking/transceiver-modul
es.
– Trunking technology groups up to eight ports to create high-performance 256Gb/s ISL
trunks between switches using 32Gb/s ports.
Support for the following features when using the SX6 Blade:
– Support for 16 Fibre Channel ports supporting 4, 8, 16, and 32Gb/s.
– 16GbE ports supporting 1 or 10Gb/s.
– Two GbE ports supporting 40Gb/s on SX6 extension blades.
Note: The 10Gb/s transceivers on the FC64-48 and the FC32-X7-48 are not
interchangeable with the IBM b-type Gen 7 SX6 Blade.
For details on supported transceivers and speeds, see Supported Transceiver Matrix
https://www.broadcom.com/products/fibre-channel-networking/transceiver-modul
es.
Considerations for the IBM b-type Gen 7 SX6 Blade:
– The SX6 extension blades perform as extension platforms to support Fibre Channel
(FC) and IBM FICON® data flows and IP-based storage data flows over an IP WAN.
– Universal ports that self-configure as E_Ports, F_Ports, EX_Ports, M_Ports (mirror
ports), and FICON ports.
Note: The 10Gb/s ports on the FC64-48 and FC32-X7-48 port blades can function
as E_Ports only.
See the "Power Supply Specifications (per PSU)" in the IBM b-type Gen 7 Director
Technical Specifications for maximum output power, input voltage, input line frequency,
and other specifications for your power supply model.
See the "Power Supply Requirements" section in the IBM b-type Gen 7 Director Technical
Specifications for the minimum number of power supplies required for operation and
redundancy when different input voltages are applied, such as low line and high line AC.
See the "Power Consumption" sections in the IBM b-type Gen 7 Director Technical
Specifications for power output data and the minimum number of power supplies for
supported input voltages.
Redundant primary power connections ensure high availability. Each power supply
assembly has its own connector, so the number of primary power connections is four for
optimum efficiency and redundancy.
Two World Wide Name (WWN) cards located on the non-port side of the device behind the
WWN card bezel.
Port blades use small form-factor pluggable (SFP+, QSFP+, and QSFP28) optical
transceivers. For details on supported transceivers per blade type, see Supported
Transceiver Matrix
https://www.broadcom.com/products/fibre-channel-networking/transceiver-modules.
Core routing blades use QSFP28 or QSFP56 optical transceivers. For a list of supported
transceivers for these blades, see Supported Transceiver Matrix
https://www.broadcom.com/products/fibre-channel-networking/transceiver-modules.
Chassis door. This door must be installed to meet EMI compliance certification.
A cable management comb. These combs install on the chassis below the blades for cable
management.
Note: Device control processors and management modules contain batteries for
RTC/NVRAM backup. Do not attempt to replace these batteries. Dispose of hardware
components that contain these batteries as required by local ordinances and
regulations.
Note that the SX6 extension blades is not shown in Figure 2-1 on page 11, but would be
installed in the same slots as the FC32-X7-48 or FC64-64 port blades. A maximum of four
SX6 blades are supported.
Only install the control processor (CP), core routing (CR), port, and extension blades into slot
numbers as follows:
Slots 1–2 are restricted to CP blades. Note that the blade installed in slot 1 will be
designated as CP0, while the blade in slot 2 will be designated as CP1 in CLI command
and message output.
Slots 3–6 and slots 9–12 are restricted to port and extension blades.
Slots 7–8 are restricted to CR64-8 core blades.
3. Fan Assembly
4. 2AWG Panduit LCD2-14AF Lug for Building Ground Connection
Note: Depending on the fans and power supplies that are installed, airflow can be from the
port side to the non-port side of the chassis or from the non-port side to the port side of the
chassis.
Figure 2-2 Non-port side of the IBM SAN512B-7 Director (example configuration)
Although not illustrated, the chassis label that contains the serial number, SKU, and
WWN is located on the lower portion of the chassis, below the fan assemblies.
To find all documentation for the IBM b-type Gen 7 SAN512B-7 (8961-F8) refer to the
documentation tab at this url.
https://www.broadcom.com/products/fibre-channel-networking/directors/x7-directors#
documentation
Note: Device control processors and management modules contain batteries for
RTC/NVRAM backup. Do not attempt to replace these batteries. Dispose of hardware
components that contain these batteries as required by local ordinances and
regulations.
a. Slot 1 (left),
b. Slot 2 (right)
3. Port Blades (FC64-48 or FC32-X7-48)
4. Core Routing Blades (CR64-4)
5. Side Air Vent
Note that the SX6 extension blade is not shown in the following figure, but would install in the
same slots as the FC32-X7-48 or FC64-48 port blades. A maximum of four SX6 blades is
supported.
Figure 2-3 Port-side view of the IBM SAN256B-7 Director (example configuration)
Note: When non-port-side exhaust (NPE) fan and power supply assemblies are installed,
air flows through the side vent and exhausts to the non-port side. When non-port-side
intake (NPI) fan and power supply assemblies are installed, air flows from the
non-port-side and exhausts through the side vent. When installing the chassis in a
four-post rack with the airflow diversion rack mount kit, airflow is redirected to or from the
port side of the chassis. Therefore, if NPE fan and power supply assemblies are installed,
air flows into port-side air vents and exhausts to the non-port side. If NPI fan and power
supply assemblies are installed, air flows into non-port-side air vents and exhausts through
to the port-side.
Slots 1–2 are restricted to CP blades. Note that the blade installed in slot 1 will be
designated as CP0, while the blade in slot 2 will be designated as CP1 in CLI command
and message output.
Slots 3–4 and slots 7-8 are restricted to port and extension blades.
Slots 5–6 are restricted to CR64-4 blades.
Depending on the fans and the power supplies installed, airflow can be from the port side to
the non-port side or from the non-port side to the port side of the chassis through the side air
vent.
Use the Chassis Airflow Diversion and Port Side Exhaust Kit to divert airflow so that air fully
exhausts to the port side of the chassis or is fully drawn from the port side of the chassis while
mounted in a four-post rack.
Figure 2-4 Non-port side of the IBM SAN256B-7 Director (example configuration)
Although not illustrated, the chassis label that contains the serial number, SKU, and WWN is
located on the upper portion of the chassis to the left of the fan assemblies.
Note: The port speed is determined by the maximum speed supported by the
optical transceiver at the other end of the link.
Table 2-1 IBM SAN64B-7 encryption and compression: number of supported ports
Port Speed Encryption-Only Compression-Only Compression and
Encryption
Secure boot.
Real-time power monitoring.
Figure 2-6 Nonport-side view with AC power supply and fan assembly units
https://www.broadcom.com/products/fibre-channel-networking/switches/g720-switch#do
cumentation
Note: Upgrading to a Gen 7 director is permanent on the CPs. This means that you cannot
downgrade to Gen 6 after the upgrade to Gen 7 and you cannot use the same CP blades in
a Gen 6 chassis.
Note: You should have a terminal console connection to both CPs when migrating.
Figure 2-7 lists of the functional differences between the three IBM b-type Gen 7
chassis that support Fabric OS 9.0.0a.
Figure 2-7 Functional Differences between the Chassis Supporting Fabric OS 9.0.0a
Figure 2-8 on page 23 provides a list of the Gen 6 and Gen 7 blade support.
For documentation on Gen 6 to Gen 7 migration procedures, please contact your IBM
technical support representative.
Table 2-2 IBM SAN512B-7, IBM SAN256B-7 and IBM SAN64B-7 supported transceivers
Description IBM SKU IBM Part IBM SAN512B-7 FC64-8 FC32-X7 SAN64B-7
Number Feature & Port Blade 48 Port
Code SAN256B-7 Blade
Description IBM SKU IBM Part IBM SAN512B-7 FC64-8 FC32-X7 SAN64B-7
Number Feature & Port Blade 48 Port
Code SAN256B-7 Blade
For a full list of supported transceivers for all IBM b-type Gen 7 platforms go to the
Supported Transceiver Matrix:
https://www.broadcom.com/products/fibre-channel-networking/transceiver-modules.
In the past, the process of problem detection and mitigation relied on the diligence, skill, and
experience of the fabric administrator. Usually, a manual task of searching and tracing error
conditions was the regimen. The operational cost and potential disruptive impact to critical
applications are simply not acceptable in today's data center. To assist in administrative
automation and improve network uptime, Broadcom has developed Autonomous SAN
technology that offers self-learning, self-optimizing, and self-healing capabilities. Autonomous
SAN is realized through Fabric Vision technology that combines built-in capabilities in IBM
b-type Gen 6 and Gen 7 platforms, Brocade Fabric OS, and Brocade SANnav Management
Portal features. Fabric Vision provides detailed monitoring and alerting, as well as response
and mitigation, that vastly improve the fabric administrator's insight and response time.
3.1.1 Self-learning
Brocade technology proactively monitors millions of I/O performance and behaviour data
points through integrated network sensors to gain deep insight into the environment. These
data points are transformed into actionable intelligence that can monitor and alert when there
are any abnormal changes. Through these capabilities, an admin can identify individual
applications and their performance characteristics across the fabric, as well as identify the
performance of the various devices that comprise the fabric: the switches, hosts, and targets.
IO Insight: Proactively monitors I/O performance and behaviour through integrated
network sensors to baseline application performance and ensure operational stability. By
combining the instrumentation of IO Insight with the ability to self-learn the traffic flows, the
Brocade autonomous SAN technology can generate performance and health metrics on
each component as well as the applications to monitor for changes and alert the
administrator with MAPS.
Monitoring and Alerting Policy Suite (MAPS): Provides an easy-to use solution for
policy-based threshold monitoring and alerting. MAPS proactively monitors the health and
performance of any SCSI or NVMe storage infrastructure to ensure application uptime and
availability. By leveraging prebuilt rule-based and policy-based templates, MAPS simplifies
fabric-wide threshold configuration, monitoring, and alerting.
Automatic Flow Learning: Provides automatic learning of all traffic flows from a specific
host to storage across the SAN fabric. With this information, an admin can automatically
identify resource contention or congestion that is impacting application performance.
3.1.2 Self-optimizing
Utilizing actionable intelligence gathered from self-learning capabilities, Brocade Fibre
Channel SANs automatically apply priorities for specific data traffic to help guarantee
performance levels and monitor for traffic pattern shifts. Learning traffic behaviour enables the
network to make smarter decisions on traffic prioritization, congestion avoidance, and
adjustment to ensure optimal network performance for applications and storage. When
something does change, the Brocade autonomous SAN technology will isolate the port traffic
for the misbehaving device to a virtual channel in the fabric and allow all other traffic to go
around to maintain optimal performance. In addition, Brocade SANnav Management Portal
has automated manual activities such as infrastructure deployment and provisioning to
expedite IT services. Self-optimizing features include the following:
Traffic Optimizer: Automatically classifies and separates traffic with similar characteristics
to optimize performance for most common SAN configurations. It identifies and isolates
traffic flows to prevent negative impact to overall SAN performance.
Advanced traffic shaping: Guarantees application performance by proactively monitoring
and actively shaping traffic.
REST APIs: Eliminate human errors and performance impacts through open DevOps.
SAN Automation and Ansible: Gain cloud-like SAN orchestration for optimizing
administration resources through Brocade REST API automation technology.
3.1.3 Self-healing
When potential disruptions are detected, the network will automatically mitigate or resolve
issues without manual intervention. This is done by proactively monitoring the network to
automatically identify abnormal or unexpected infrastructure behaviour and then taking
immediate action. These actions include alerting the end devices of the issue through a
notification and signalling process within a SAN. End devices can automatically adjust traffic
or fail over to a healthy path to mitigate impact until a further infrastructure change solution
can be implemented. In addition, these capabilities detect and automatically reconfigure
fabric misconfigurations that fall outside best practices Self-healing features include:
Fabric congestion notification: Automatically detects congestion and notifies end devices
to automatically mitigate congestion.
Slow drain device quarantine: Automatically quarantines credit-stalled devices to prevent
the misbehaving device from impacting the rest of traffic.
Automatic actions: Ensure data delivery with automatic failover from physical or
congestion issues such as port decommissioning, port toggling, and port fencing.
COMPASS: Detects and automatically reconfigures out-of-compliance configurations.
SANnav Management Portal: Reduces troubleshooting steps with built-in best practice
recommendations to quickly resolve issues.
3.2 Automation
Most, if not all, IT administrators have first-hand experience in the growing complexity of
managing enterprise infrastructure, including SANs. “The cost and complexity of protecting
and storing data is increasing, and IT leaders are responding with attempts to better optimize
and automate storage—but they need better tools,” according to a report from Enterprise
Strategy Group.
IBM and Brocade are in the unique position to spot and understand the impact of automation
helping organizations get more from their SAN infrastructure.
IBM and Brocade offers a combination of SAN automation with RESTful APIs and a SAN
management platform to help organizations drive greater efficiency from their SANs. This is
accomplished through a variety of means:
IBM and Brocade SAN automation is provided with a multilayer architecture:
RESTful API support on switches, as well as on management tools.
Brocade’s Ansible management framework, designed to eliminate repetitive tasks, simplify
management and orchestrate across the full SAN infrastructure.
automation the only viable option to drive increased IT agility and more tightly align IT
service delivery with fast-changing business needs.
Consistent configuration validation is a must. Manual configuration changes are taking
place with greater frequency as enterprises diversify their IT architectures in general, and
their storage architectures specifically. SAN automation ensures validation of consistent
configuration parameters across the different SAN fabrics in order to facilitate
troubleshooting of frequent alerts without reliance on manual intervention.
In addition to the afore mentioned features, new features such as, New Java-Less Web Tools
based on HTML 5.0, increased zone database size to 4MB, updated SupportSave to run
parallel to save time during troubleshooting, Command reset to factory reset and support for
the Generic USB drives are few to note.
In this section, we will cover some of these features, for full list of enhancements, please refer
to Brocade Fabric OS Administration guide at
https://docs.broadcom.com/doc/FOS-90x-Admin-AG
Many scenarios cause a device to receive a new PID: for example, unplugging the device
from one port and plugging it into a different port as part of fabric maintenance; or changing
the domain ID of a switch, which might be necessary when merging fabrics; or changing
compatibility mode settings.
Some device drivers use the PID to map logical disk drives to physical Fibre Channel
counterparts. Most drivers can either change PID mappings dynamically, also called dynamic
PID binding, or use the WWN of the Fibre Channel disk for mapping, also called WWN
binding.
Some older device drivers behave as if a PID uniquely identifies a device; they use static PID
binding. These device drivers should be updated, if possible, to use WWN binding or dynamic
PID binding instead, because static PID binding creates problems in many routine
maintenance scenarios. Fortunately, very few device drivers still behave this way. Many
current device drivers enable you to select static PID binding as well as WWN binding. You
should select static PID binding only if there is a compelling reason and only after you have
evaluated the effect of doing so.
F-Port A port is assigned only a base area. A port is assigned only a base area.
F_Port with NPIV For F_Ports with NPIV less than 63 The 10-bit addressing mode utilizes
devices, port is assigned only a base the 8-bit area ID and the borrowed
area. If the 64th NPIV device is upper 2 bits from the AL_PA portion
received in F_Port, then a new of the PID. Areas
additional area is assigned to the 0×00 through 0×8F use only 8 bits
existing base area. Every time a port for the port address and support
exhausts an area after 64 additional upto 255 NPIV devices. Areas 0×90
devices login, another additional area through 0×FF use an additional 2
is assigned. The maximum number of bits from the AL_PA for the port
NPIVs supported is 255 devices. address. Therefore, these ports
support only 64 NPIV devices per
port.
F_Port Trunking Same as F_Port and F_Port with NPIV. F_Port trunking uses an 8-bit area
In FOS 9.0, UAM allocates additional in VF mode enabled logical switch.
areas to both F_Port trunking and Default switch in VF or non-VF
Emulex HBA trunking. mode uses existing port address.
EX_Port on ICL Ports When an ICL port is configured as an When an ICL is configured as an
EX_Port, a new area is assigned to the EX_Port, the switch is assigned an
ICL port from the pool. This is not 8-bit area in VF. EX_Port over ICL is
applicable for ICL ports that are not not supported in non-VF mode.
configured as an
EX_Port. From the FOS 9.0 release,
when an ICL port is configured as an
EX_Port,the new areas are assigned in
both VFmode and non-VF mode.
Port Address Binding In platforms that are enabled in UAM In pre-UAM mode, the port address
mode, the port address binding feature binding feature is supported only in
is supported in both VF and non-VF logical switches in VF enabled
mode. mode.
Persistent PID In platforms that are enabled in UAM In pre-UAM mode, the persistent
mode, the persistent PID feature is PID feature is supported only in
supported in both VF and non-VF logical switches in VF enabled
mode. mode.
See the FICON Configuration chapter in this RedBook for further details on the benefits and
restrictions with the Unified Addressing Mode.
S w 1 -C L I
S w it c h 1 – ( 8 . 2 . x) S w it c h 2 – ( 8 . 2 . x )
D e fi n e d c o n fi g u ra ti o n :
c fg : c fg 1
zo n e 1
S w 1 -C L I zo n e : zo n e 1
H 1 ;T 1
zo n e : n e w C li U s e rZ o n e
Zo ne E d it H 2 ;T 2
E ffe c ti ve c o n fi g u ra ti o n :
c fg : c fg 1
R E JE C T! zo n e : zo n e 1
H 1 ;T 1
S w 1 -B N A
Although there is local switch protection, there is no fabric-wide locking mechanism. For
versions prior to Fabric OS 9.0.0, the fabric-level detection support (Concurrent Transaction
Detection) consists only of a warning mechanism that warns users if a transaction is open on
a remote switch, but does not prevent users from committing changes if concurrent
transactions exist across the fabric. Furthermore, this fabric-wide detection is limited in scope
to CLI zoning operations only. For example, if a user on one switch opens a zone transaction,
and if a user on another switch in the fabric issues a zone edit or commit command, a warning
is received that a transaction is occurring in the fabric. However, a zone commit operation can
still be issued from either switch, which overwrites any pending zone transaction on other
switches.
Figure 3-2 Fabric OS 8.2.x and earlier fabric-wide commit overwrite example
From Fabric OS 9.0.0 version onward, the Zone Fabric Locking feature provides zone
transaction protection for a single switch and also fabric wide. This feature is enabled by
default. If a zone edit or commit command is occurring in fabric, you cannot perform a zone
edit or commit on the same or another switch for a default timeout period of 5 minutes. A lock
request is sent at the beginning of a zone edit operation. The Fabric Lock Failsafe Timer is
configurable and it is a fabricwide setting. When a zone fabric lock is active, a failsafe timer is
started on all remote switches.
Sw1-CLI
Defined configuration:
cfg: cfg1 REJECT!
Sw1-CLI zone1 Sw1-BNA
zone: zone1
H1;T1
zone: newCliUserZone
H2;T2
Effective configuration:
cfg: cfg1
zone: zone1
H1;T1
When the failsafe timer expires, the open zone transaction is not aborted. If the same user
attempts to resume the transaction by performing another edit or commit operation after the
zone fabric lock has expired, the transaction is allowed and the fabric lock is restarted. If a
different user attempts to start a new transaction after the first user's transaction timer has
expired, the transaction is allowed and the first user's transaction is aborted before the
second user's transaction starts.
Note:
If a fabric-lock is currently active in the fabric, the Fabric Lock Failsafe Timer timeout
value cannot be modified.
The fabric zone lock is a fabric-wide setting. Thus, the fabric must be stable before
performing zone edit, commit, and abort operations.
If a zone transaction is aborted, the fabric zone lock is cleared.
When the timer expires, the lock is cleared locally and on each individual switch in the
fabric.
When using virtual fabric functionality, the lock timeout value is independent on each
logical switch and can have different values.
Fabric zone locks can be pre-empted by zone merges.
The Zone Fabric Locking feature is supported on the switches with the Fabric OS
version 9.0.0 or later. Down-level switches will not participate. However, Fabric OS
9.0.0 switches owning the zone fabric lock will prevent updates from down-level
switches.
On remote switches, an additional 30-second buffer is added to the configured lock
timeout value of the lock principal switch.
The fabric-wide locking mechanism ensures that any zone transaction started on any switch
by any interface (CLI, REST, Web Tools, SANnav, Target Driven Zone) is not overwritten or
aborted by another interface's transaction (except for zone merges) while the lock timer is
active.
FICON logical switch provides auto-configuration of the following, which simplifies the
process of establishing a logical switch in the FICON environment:
Insistent Domain ID (IDID) mode is set to on
Routing policy is set to device-based routing
Area mode is set to zero-based addressing
SCC security policy is auto-defined with only its own WWN
The created SCC policy is activated
The fabric-wide consistency policy is set to strict SCC policy
High Integrity Fabric (HIF) mode is enabled
FMS mode is enabled if FICON control unit port (CUP) license is installed on the chassis
Please refer to FICON chapter in this book for more details about creating FICON Logical
switch and the best practices.
Web Tools does not require a license. It is installed on the switch when you install Fabric OS.
If the switch is configured with logical fabrics, you can log in to any of the logical fabrics for
which you have permission.
You can launch Web Tools directly from a browser or from SANnav Management Portal.
To launch directly from a web browser, open your browser, enter the IP address of the switch,
and press Enter. You can use HTTP or HTTPS. For example, http://10.77.77.77 or
https://10.77.77.77.
Enter the user name, password, and logical switch name or fabric ID (FID).
For the first switch login, the default user name is admin and the default password is
password. Web Tools prompts you to change the default password.
If you are logging in to a Virtual Fabrics-enabled platform and you do not specify a logical
switch, you are logged in to the default logical switch, which uses fabric ID 128. For non-VF
platforms, the FID option is not displayed.
Note: If you launch from SANnav Management Portal, you might not need to log in,
depending on the SANnav single sign-on configuration.
3.5 Security
Security entails authentication and encryption to restrict access and protect data from
unauthorized access. Brocade products support a wide range of authentication, encryption,
and management tools to protect fabrics and data from unauthorized access:
Authentication: Authentication protocol support includes CHAP, DH-CHAP, FCAP, IKE,
IPsec, RADIUS, TACACS+, and P-EAP/MS-CHAP for RADIUS.
Encryption (AES/3-DES): Brocade provides AES-128 and AES-256 encryption and
168-bit 3-DES encryption for IP links on extension products and management
connections. Brocade also supports AES and 3-DES with IPsec. These solutions provide
high-performance encryption and compression.
In-flight encryption over ISLs: Brocade X6 with Gen 6 Fibre Channel port blades, Brocade
X7 with Gen 6 or Gen 7 Fibre Channel port blades, and Brocade G720, G630, G620
switches support in-flight encryption for traffic over ISLs to minimize the risk of
unauthorized access to data within the data center and over long distance links.
Data-at-rest and data-in-flight encryption are complementary technologies that serve
different purposes, and each may be required to achieve regulatory compliance.
Secure Boot: A switch validates the integrity and authenticity of the FOS boot image to
establish a hardware based root of trust through the manufacturing supply chain.
Please refer to Chapter 12, “SAN security” on page 293 for further information.
Traffic flows with similar predefined attributes, such as flow destination speed and priority, are
grouped logically as a Performance Group. With Gen 7 platforms, Traffic Optimizer provides a
way to optimize shared resource allocation for each Performance Group, and it automatically
maps traffic flows to predefined Performance Group.
Fabric OS 9.0 supports the Performance Groups listed in the following table. These
predefined performance groups provide support of speed-based classification, credit-stalled
device isolation via Slow Drain Device Quarantine (SDDQ), compatibility with the Adaptive
Networking (QoS Zone and CSCTL) feature, and compatibility with previous platforms. Traffic
Optimizer is enabled by default on all Gen 7 platforms and Gen 7 blades.
Gen 6 platforms in Fabric OS 9.0 continue the legacy behaviour and maintain compatibility
with Gen 7 platforms in the same fabric. If a Gen 7 switch or blade is connected to a port on a
Gen 6 platform and the traffic flow source port is on the Gen 7 switch, the traffic flow will be
speed-based classified within the Gen 7 switch until reaching the port on the Gen 6 platform,
and then the traffic flow will be handled with legacy behaviour starting from the Gen 6 port.
Many years ago, Brocade Virtual Channels (VC) addressed this issue with great success and
significant performance gains. Resources affecting performance include bandwidth, egress
queues, ingress buffers, and QoS. The implementation has multiple distinct egress queues
establishing virtual channels across the physical link, each channel having its own set of
autonomous buffer credits. The bandwidth is shared, but flows are independent. When traffic
characteristics are uniform, VCs offer an expedited pathway. Great benefit is gained by
sorting traffic into VCs based the most pertinent characteristic, which is speed.
Normally, VCs service multiple FC flows since there’s a limited number of VCs per
classification of traffic. Users do not configure or setup VCs, they are merely a fundamental
component of the Fabric Operating System (FOS). FC wire speeds continue to increase, now
at 64 Gbps (Gen 7) and bandwidth has an extremely wide spread from 8, 16, 32, to 64
gigabits (32-56 Gbps). Wide variations in speed can impose flow-control problems within a
network.
For years, Brocade VCs operated within a relatively narrow bit rates of 1, 2, 4, 8, and 16
gigabit (8-15 Gbps) by leveraging multiple logical independent paths. VCs are assigned by a
hash called PID2VC, which results in a pseudorandom assignment of each flow to a VC. To a
degree, this statistically mitigated the interference of slower flows impeding faster flows. Plus,
optionally faster flows could be manually assigned to a high QoS VC and slower flows to a low
QoS VC to prevent interference.
greater due to other complimentary technology evolutions such as NVMe, AFA, and host
virtualization. Flows now compete and head of line blocking is not an option. Blocking must be
efficaciously dealt with… in comes Traffic Optimization.
Brocade Traffic Optimization technology takes VCs to the next level. Traffic Optimization
organizes and manages traffic flows using performance groups (PG). Fabric resources are
allocated based on performance groups. Instead, flows are assigned to a VC based on the
destination port speed and potentially via protocol (for example, NVMe). Brocade fabrics
know the destination port speed and protocol for every flow.
Brocade Gen 7 hardware added more VCs, E_Ports and EX_Ports use 16 new Traffic
Optimization VCs, which are assigned: Four VCs per speed (<16, 32, 64) for a total of 12
VCs, plus another four reserved VCs for future use. QoS VCs have not changed, there are 2
for low, 4 for medium (default), and 5 for high. QoS VCs are used when traffic has been
designated to a high or low priority. Slow Drain Device Quarantine (SDDQ) uses the QoS low
VCs.
Pre-Gen 7 and existing fabric coexistence is fully supported. E_Port connections between
Gen 7 and pre-Gen 7 platforms results in Traffic Optimized VC being remapped using the
legacy classification. QoS VCs pass-through unchanged since they remain consistent with
Traffic Optimization.
Traffic Optimization is new and unique functionality delivered in Brocade’s latest Gen 7 Fiber
Channel platforms. It is a powerful enhancement to the existing Virtual Channels feature
incumbent to past generations of Brocade switching equipment. Traffic Optimization requires
no configuration and is operational on all Gen 7 devices. Traffic is classified by speed or
NVMe Performance Groups and assigned to a specific VC. Traffic Optimization is
interoperable with FC cohort technologies (FICON, Extension, Access-Gateway, Long
Distance Links, Security, and NPV) and Brocade Gen 6 fabrics.
Performance Groups and eliminates speed mismatches that cause performance interference
when lower speed devices collide with higher speed devices connected in the same fabric.
Traffic Optimization over FC routers is supported only if the Virtual Fabrics is disabled in the
Backbone fabric. Traffic Optimization over FC routers is not supported if the Virtual Fabrics is
also enabled in the Backbone fabric. For more information on the operating modes of
Backbone fabrics, please refer to Fabric OS Admin guide
https://docs.broadcom.com/docs/FOS-90x-Admin-AG. If the traffic flow in the metaSAN
encounters a switch platform that does not support Traffic Optimization, then the traffic flow
will be mapped to the legacy behaviour and will continue to retain the legacy behaviour all the
way.
These components allow members of the fabric to indicate their ability to interpret and react
to signals and notifications. Participating devices can respond by taking actions to mitigate
and reduce impacts of congestion, frame loss, and signal integrity degradation. The Fabric
OS is the primary generator of signals and notifications due to its awareness of I/O flows,
physical layer impacts, and device responses.
The relationship between the event detection, signalling, and notification operations is
depicted in the following figure.
When an event is detected (1), the Fabric initiates the appropriate response. In the case
where frame transmission is stalled, signal events are generated (2) toward the point of
congestion (that is credit-stalled or oversubscribed device). In the case of a transmission
failure or sustained congestion event, notification events are generated (3) and distributed to
peer devices (4). The fabric determines the distribution of the notification event from the zone
IBM B-Type Fabric Performance Impact Notification (FPIN) is a new feature for Brocade FOS
v9.0. It is available on Brocade Gen 6 and Gen 7 switches. This feature enables the switch to
detect issues on a fabric such as congestion or physical link issues and then notify the
affected devices that have registered for these notifications. The devices that receive these
notifications can then proactively take steps such as path failover rather than to react to a
path being down. FPIN provides a means to notify devices of link and other issues with a
connection or possibly a path through the fabric. For FPIN, a device must register with fabric
services to receive these notifications. The new IBM B-Type Gen 7 hardware can send
hardware and software signal notifications. Gen 6 can only send software notifications. Both
the hardware and software notifications require FOS v9.0 on the Switches and Directors.
Hardware signals can be sent from the switch to the adapter in the device. The adapter itself
can then decide what to do about the notification. Software signals are sent higher up in the
Fibre-Channel stack, and the adapter driver would then decide how to handle the
notifications. One advantage to notifications in hardware is reaction time - the adapter can
process the notifications and react more quickly than the driver can. Another is that the
hardware-based notification is a fibre-channel primitive. This means that even if buffer credits
are depleted the signal can still be delivered to the device on the other end of the link.
Primitives are not frames so do not need buffer credits to be sent. The software layer signal is
an ELS frame, so can affected by buffer credit depletion and other link congestion. Whether
the signal sent is hardware or software, how the devices handle the notifications is up to the
vendor of the adapter. Some may log the notification, some may take action. The action that
an adapter takes is also vendor specific. FPIN can alert devices about (1) End Device
Congestion (2) Device Link Integrity (CRC) and (3) Frame Drops.
At the time of writing this book, below are some vendors that support FPIN today:
Linux Multi-Path in RHEL 8.2
IBM AIX® - will register for Link Integrity and Congestion notifications
Emulex - supports Congestion and Link Integrity notifications on Linux
Support for more HBA and Storage Controller vendors for FPIN is expected in the future.
FPIN can be sent from any device that supports. Potentially the storage, the host or the switch
can share this information. Having end-to-end capability to act on the data based on the FPIN
notifications, the SAN will self-heal resulting an end-to-end autonomous SAN infrastructure.
Before receiving a Congestion Signal primitive Gen 7 ports must let peer ports know it is
capable. For example, IBM B-Type Gen 6 platforms do not support Congestion Signal
primitives but are capable of FOS-level ELS-based FPIN Congestion notifications.
Hosts, switches, and storage can generate ELS notifications upon detecting an impacting
event. A notification may be sent by any fabric member detecting a problem, be it on a
switching platform (F_Ports and E_Ports) or an end-device (N_Ports); it will be processed the
same.
Fabric Notifications and MAPS work in tandem. MAPS rules for congestion, SCSI command
discard, and link integrity will trigger corresponding FPIN ELS notifications. FPIN notifications
are sent to all in-zone end-devices that have registered to receive that particular notification
type. Notifications are not dependent on zoning type, whether traditional or peer, the
operation is the same with no advantage or disadvantage. Additionally, Fabric Notifications
sends FPIN notifications to all peer switches and the peer switches forward notifications to
their registered in-zone end-devices as well. FPIN notifications are restricted to devices:
Supporting FPIN notifications
Registered to receive a particular notification type
Experiencing the notification condition
Are an in-zone peer device
8G 16G
Congestion impacts
16G FC Servers other workloads
8G 16G
8G 16G
16G FC
Storage
IBM B-Type SAN A
8G 16G
Max
32G FC
Storage
Hosts Storage
Zone A
Buffer credits withheld to
prevent buffer overflow.
Over- Con
Buffered
frames Application performance
Buffered
frames
A
subscribed! gest
impeded! Possible Solutions:
Peer Congestion Notification
C
Peer congestion notifications indicate No
to devices there is a fabric impact. Notification
Not in Zone
Below is an example of how the Congestion Notification works in real world, in collaboration
with Emulex HBA on the IBM AIX server, with the IBM B-Type Gen 7 SAN, IBM SANnav
Management software and Emulex SAN Manager, a management tool to manage host HBAs
in the environment.
Under normal operations, as shown in Figure 3-10, IBM SANnav management tool and the
Emulex SAN Manager report minimal to no congestion.
8G 16G
16G FC Servers
8G 16G
8G 16G
16G FC
Storage
IBM B-Type SAN A
16G 32G
32G FC
Storage
16G FC
Server
The workloads generally grows over time and becomes larger than the hardware can support.
As shown in Figure 3-11, this will have a ripple effect on the entire infrastructure causing the
congestion resulting in major performance issues.
IBM B-Type Gen 7 SAN detects these type of F-Port congestion issues and sends FPIN
Congestion Notification to the Emulex HBA on the AIX server. The Emulex SAN Manager,
management software to manage HBAs on the AIX server, is enabled to set congestion
policy. As shown in Figure 3-12, depending on the choice of the policy between conservative,
moderate and aggressive, the action will be taken to address the end point congestion issue.
For the bully server, the workload is still too big, so additional administrative attention is
required.
Notification F_Port à N_Port, HBAs register to receive FPIN notifications of interest, for
example, Link Integrity. When F_Port experiences incoming errors due to faulty electronics,
marginal optic, dirty connector, damaged fiber cable… a Link Integrity notification is sent to
the HBA letting the host know of the transmission problem. In turn, the HBA forwards
notifications to the MPIO driver, which will be able to distinguish between a device failure and
physical path failure. Link Integrity notifications are used by end-devices for troubleshooting
links and by MPIO for selecting the best path.
Notification N_Port à F_Port, the HBA registers with the fabric to receive and send FPIN
notifications. If the HBA sees incoming errors, it will notify the fabric that its connection to the
end-device is erratic. An FPIN Link Integrity notification sent by N_Port to F_Port (specifically,
the Fabric Controller) will be distributed to the in-zone devices as well. The fabric logs the
notification information so that the SAN administrator can take action.
Registered in-zone end-devices must be notified of link integrity problems. MAPS F_Port
alerts for CRC (Cyclic Redundancy Check) and ITW (Invalid Transmission Word) are
translated into Link Integrity FPINs and sent to the end-devices. Notifications integrate with
the OS device manager and notifying initiators and targets of link integrity issues along a data
path enhances debugging and recovery efforts from multiple perspectives.
MPIO handles FPIN events and manages path selection. On a new pathway, MPIO drivers
communicate with the appropriate service to manage outstanding and pending IOs.
End-devices decide their best course of remediation like immediate failover to an alternate
path within the MPIO environment. Examples of Link Integrity problems:
Slow host application performance can be traced to CRC errors on a storage port
Slow array performance can be traced to host port ITWs
Below is an example of how the Link Integrity works in real world, in collaboration with Emulex
HBA on the AIX server and with the IBM B-Type SAN.
Under normal operations, as shown in Figure 3-13 below, with the Fibre Channel links
working, data is balanced between the different paths to the storage system. Over time, links
may not work as well as they should.
8G 16G
16G FC 16G FC
Server Storage
8G 16G
Max
IBM B-Type SAN B
When link(s) are sick but not dead, as shown in Figure 3-14, I/Os going over the bad link
reduce the data available to the application. Quick detection and action is essential to improve
the performance and in some cases availability of the application.
8G 16G
16G FC 16G FC
Server Storage
8G 16G
Max
IBM B-Type SAN B
With the Link Integrity feature, as shown in Figure 3-14, when a sick but not dead path is
detected, as shown in Figure 3-15, FPIN-LI (Fabric Performance Impact Notification – Link
Integrity) notification is sent to the host HBA, then the Emulex HBA passes the FPIN up the
stack to the AIX multi-pathing software to address the issue by taking corrective actions.
As shown in Figure 3-16, after the notification is received by the host HBA, the multi-pathing
software makes intelligent decisions based on the data from the IBM B-Type Gen 7 SAN
fabric to rebalance the traffic away from the bad link, resulting in improved availability and
minimized performance problems.
immediate SCSI I/O failure without waiting for a timeout. This allows the end-device to
immediately retry the discarded command avoiding long wait times and application disruption.
The Brocade’s Fabric Notifications solution addresses each of these issues through a
technology eco system of innovation. Various notifications with problem specific descriptors
are sent to or received from attached end-devices, and in some cases propagated to other
Directors/switches to notify peer end-devices. Notifications are limited to registered in-zone
devices and the device must register for each notification type it wishes to receive. Fabric
Notifications compliments MAPS technology and was introduced by Broadcom BSN in FOS
9.0.
Virtual Fabrics is a suite of related features that can be customized based on your needs. The
Virtual Fabrics suite consists of the following specific features:
Logical switch
Logical fabric
Device sharing
Terminology: Virtual Fabrics is the name of the suite of features. A logical fabric is a type
of fabric that you can create using the Virtual Fabrics suite of features.
This chapter describes a logical fabric and logical switch types, as well as device sharing with
Virtual Fabrics.
Important: SNMPv3 is required to manage Virtual Fabrics. If you use SNMPGET to poll
the switches, for example HiTrack, the SNMPv3 user must be configured based on the
logical switch that is being monitored. If SNMP monitoring is for the default switch, you do
not need the SNMP user to be configured as a user on the switch. Whereas, when SNMP
monitoring is VF specific, you must create a user with permission for the corresponding VF
and add the created user into the SNMP database as a SNMPv3 user. When the SNMPv3
user is successfully added, you can use the same user name in management applications
for SNMP queries to the corresponding logical switch.
Figure 4-1 shows a switch before and after enabling Virtual Fabrics. In this example, the
switch has 10 ports, labelled P0 through P9.
Figure 4-2 on page 53 shows a Virtual Fabrics-enabled switch before and after it is divided
into logical switches. Before you create logical switches, the chassis appears as a single
switch (default logical switch). After you create logical switches, the chassis appears as
multiple independent logical switches. All of the ports continue to belong to the default logical
switch until you explicitly move them to other logical switches.
The default logical switch always exists. You can add and delete other logical switches, but
you cannot delete the default logical switch unless you disable Virtual Fabrics.
Note: A FICON logical switch or an existing logical switch that has been modified to
become a FICON logical switch cannot serve as the default logical switch.
In the Figure 4-3 on page 54, logical switches 2, 3, 4, and 5 are assigned FIDs of 1, 15, 8, and
20, respectively. These logical switches belong to different fabrics, even though they are in the
same physical chassis. For example, you could not assign logical switch 5 a fabric ID of 15,
because logical switch 3 is already assigned FID 15 in the chassis. The default logical switch
is initially assigned FID 128. You can change this value later.
Note: Each logical switch is assigned one and only one FID. The FID identifies the logical
fabric to which the logical switch belongs.
In Figure 4-4 on page 55, the default logical switch initially has 10 ports, labeled P0 through
P9. After logical switches are created, the ports are assigned to specific logical switches.
Note that ports 0, 1, 7, and 8 have not been assigned to a logical switch and so remain
assigned to the default logical switch.
A given port is always in one (and only one) logical switch. The following scenarios refer to the
chassis after port assignment in Figure 4-4:
If you assign P2 to logical switch 2, you cannot assign P2 to any other logical switch.
If you want to remove a port from a logical switch, you cannot delete it from the logical
switch, but must move it to a different logical switch. For example, if you want to remove P4
from logical switch 3, you must assign it to a different logical switch: logical switch 2,
logical switch 4, or logical switch 1 (the default logical switch).
If you assign a port to a logical switch, it is removed automatically from the logical switch it
is currently in. If you assign P3 to Logical switch 3, P3 is automatically removed from
logical switch 2.
If you do not assign a port to any logical switch, it remains in the default logical switch, as
is the case with ports 0, 1, 7, and 8.
A logical switch can have as many ports as are available in the chassis. In FIGURE 4-4, the
chassis has 10 ports. You could assign all 10 ports to a single logical switch, such as logical
switch 2; if you did this, however, no ports would be available for logical switches 3 and 4.
You can move only F_Ports and E_Ports from one logical switch to another. If you want to
configure a different type of port, such as a VE_Port or EX_Port, you must configure them
after you move them. Some types of ports cannot be moved from the default logical switch.
You can also connect other switches to logical switches. In Figure 4-5 P6 is an E_Port that
forms an inter-switch link (ISL) between logical switch 4 and the non-Virtual Fabrics switch.
Logical switch 4 is the only logical switch that can communicate with the non-Virtual Fabrics
switch and D2, because the other logical switches are in different fabrics.
Figure 4-5 Logical switches connected to devices and non-Virtual Fabrics switches
Figure 4-6 shows the logical switch view of a chassis where virtual fabrics are enabled and
the devices are connected. You can see that the logical switch with Fabric 128, default switch
when virtual fabrics enabled has no devices connected. Logical switches Fabric 1 and
Fabric 15 are connected to Host and a Storage respectively and the logical switch Fabric 8 is
connected to a non-Virtual Fabric switch and storage.
These are operations that are limited to the logical switch, such as displaying or changing port
states. Logical switch operations include all operations that are not covered in the chassis
management operations.
When you log in, you are assigned an active context, or active logical switch. This context
filters the view that you get, and determines which ports you can see. You can change the
active context. For example, if you are working with logical switch 1, you can change the
context to logical switch 5. When you change the context to logical switch 5, you only see the
ports that are assigned to that logical switch. You do not see any of the other ports in the
chassis.
The scope of logical switch operations is defined by the active context. When you are in the
context of a logical switch, you can perform port, switch, and fabric-level operations, subject to
Role-Based Access Control (RBAC) rules.
If you have permission to execute chassis-level commands, you can do so, regardless of
which logical switch context you are in.
The four fabrics shown in Figure 4-5 on page 56 and Figure 4-6 on page 56 are logical fabrics
because they each have at least one logical switch. You can connect logical switches to
non-Virtual Fabrics switches and to other logical switches. You connect logical switches to
non-Virtual Fabrics switches using an ISL, as shown in Figure 4-5 on page 56.
Fabrics switch. The two logical switches and the non-Virtual Fabrics switch are all in the same
fabric, with FID 8.
Figure 4-7 Logical switches connected to other logical switches through physical ISLs
The ISLs between the logical switches are dedicated ISLs because they carry traffic only for a
single logical fabric. In Figure 4-8, Fabric 128 has two switches (the default logical switches),
but they cannot communicate with each other because they have no ISLs between them and
they cannot use the ISLs between the other logical switches.
Note: Only logical switches with the same FID can form a fabric. If you connect two logical
switches with different FIDs, the link between the switches segments.
When you divide a chassis into logical switches, you can designate one of the switches to be
a base switch. A base switch is a special logical switch that is used for interconnecting the
physical chassis.
A base switch can be connected to other base switches through a special ISL, called a shared
ISL or extended ISL (XISL). An extended ISL connects base switches. The XISL shares traffic
among different logical fabrics.
Note: A FICON logical switch or an existing logical switch that has been modified to
become a FICON logical switch cannot serve as the base switch.
Fabric formation across an XISL is based on the FIDs of the logical switches.
Figure 4-9 shows two physical chassis divided into logical switches. Each chassis has one
base switch. An ISL connects the two base switches. The ISL is an extended ISL (XISL)
because it connects base switches.
Traffic between the logical switches can now flow across this XISL. The traffic can flow only
between logical switches with the same fabric ID. For example, traffic can flow between logical
switch 2 in chassis 1 and logical switch 6 in chassis 2, because they both have FID 1. Traffic
cannot flow between logical switch 2 and logical switch 7, because they have different fabric
IDs (and are, therefore, in different fabrics).
It is useful to think of the logical switches as being connected with logical ISLs, as shown in
the following figure. The logical ISLs are not connected to ports, because they are not
physical cables. They are a logical representation of the switch connections that are allowed
by the XISL.
You can also connect logical switches using a combination of ISLs and XISLs, as shown in
Figure 4-10. In this diagram, traffic between the logical switches in FID 1 can travel over either
the ISL or the XISL. Traffic between the other logical switches travels only over the XISL.
By default, the physical ISL path is favoured over the logical path (over the XISL) because the
physical path has a lower cost. This behavior can be changed by configuring the cost of the
dedicated physical ISL to match the cost of the logical ISL.
Attention: If you disable a base switch, all of the logical ISLs are disconnected and the
logical switches cannot communicate with each other unless they are connected by a
physical ISL.
Base fabric
Base switch ports on different chassis can be connected together to form a fabric, called a
base fabric. Similar to other logical switches, the base switches must have the same FID to be
connected. If the base switches have different FIDs, the link between the switches is disabled.
The base fabric follows normal routing policies. As long as physical connectivity is available,
the base fabric maintains connectivity for the logical fabrics.
Logical ports
Logical ISLs are formed to connect logical switches. A logical port represents the ports at
each end of a logical ISL. A logical port is a software construct only and does not correspond
to any physical port.
Most port commands are not supported on logical ports. For example, you cannot change the
state or configuration of a logical port. However, state change on a logical link is permitted
using the lfcfg --lislEnable command.
The World Wide Name (WWN) for logical ports is in NAA=5 format, using the following
syntax: 5n:nn:nn:nz:zz:zz:zx:xx
When you are logged in to a logical switch, the system prompt changes to display the FID of
that switch. The following are example prompts for when you are logged in to the default
logical switch (FID = 128) and a user-defined logical switch (FID = 15):
switch:FID128:admin>
switch:FID15:admin>
IPv4 addresses assigned to individual logical switches are assigned to IP over Fibre Channel
(IPFC) network interfaces. In Virtual Fabrics environments, a single chassis can be assigned
to multiple fabrics, each of which is logically distinct and separate from one another. Each
IPFC point of connection to a given chassis needs a separate IPv4 address and prefix to be
accessible to a management host.
For a management host to access logical switches, the host bus adapter (HBA) must be able
to connect with the common, shared IP address and the individual IPv4 addresses configured
for each logical switch.
Note: IPv6 is not supported when setting the IPFC interface for Virtual Fabrics.
IPFC addresses are not handled by configupload or configdownload. The IPFC address of
the default logical switch or a non-VF switch is stored on the WWN card or compact flash.
This address does not display in a configshow. Non-default logical switch IPFC addresses
display in a confgshow. The ipaddrshow command displays all switch addresses and IPFC
addresses configured in the chassis.
Example 4-1 shows how to sets IP addresses with the CIDR prefix for logical switches with
FID 1, 2, and 128 (default logical switch).
switch:FID128:admin> ipaddrshow
SWITCH
Ethernet IP Address: 198.51.100.0
Ethernet Subnetmask: 255.255.255.0
Gateway IP Address: 198.51.100.1
DHCP: Off
IPFC address for virtual fabric ID 128: 192.0.2.0/124
IPFC address for virtual fabric ID 1: 192.0.2.11/24
IPFC address for virtual fabric ID 2: 192.0.2.22/24
Example 4-2 show how to delete an IP address for the logical switch with FID 1.
use Telnet or SSH from a remote host, the IPFC address must share the same network
(subnet mask) as the management interface. The values of FCIP addresses and subnet
masks are stored in the WWN card. For example, if the management interface IP address is
10.38.18.183 with subnet mask 255.255.240.0, you can create the IPFC address (see
Example 4-3).
Note: The logical switch login context supersedes the default FID context of a user. The
CIDR subnet mask of the IPFC address must match the CIDR of the assigned
management IP address to avoid routing issues.
To delete the IPFC address, you can use the ipaddrset –ls 8 –delete command.
If you log in to a switch with the management interface IP address, the home virtual fabric is
set for the context. You can log in to the logical switch context by using the telnet or SSH
service with the configured IPFC address. You can manage the permissions using the
userConfig command (see Example 4-4).
Role-LF is the list of logical switch contexts for which you have permission to log in over the
IPFC address. Home LF Role is the default logical switch context when you have no
permission to log in to a particular logical switch context or over management interface.
Example 4-5 shows the result of the ipAddrShow command. Only one IPFC is mapped to
FID 8.
Note: If the home VF of a user is not present on the switch, then the user will be logged
in to default VF if the default VF is part of the user's VFs list. If not, the user will be
logged in to the lowest VF that is available on the switch that is part of the user's VFs
list.
Some restrictions apply to the ports, depending on the port type and blade type. The following
sections explain these restrictions.
Table 4-2 Virtual Fabrics interaction with Fabric OS features supported logical switch creation limits
Fabric OS Features Virtual Fabrics Interaction
Configuration upload and download Virtual Fabrics uses a configuration file that is
different from the configuration file used to
download system configuration parameters.
FC-FC routing service All EX_Ports must reside in a base switch. You
cannot attach EX_Ports to a logical switch that
has XISL use enabled. You must use ISLs to
connect the logical switches in an edge fabric.
ISL R_RDY mode ISL R_RDY mode is supported in both the base
switch and the logical switch.
Network time protocol (NTP) The clock information for a chassis is maintained
by the default logical switch and not from the
principal or primary FCS switch.
In the core blades, the four ports belonging to the same QSFP can be assigned to different
logical switches. However, if one of the four ports belongs to the base switch, the remaining
three ports cannot be assigned to any other logical switch.
The maximum number of logical switches per chassis varies depending on the switch model.
The following table lists the supported platforms and the maximum number of logical switches
(including the default logical switch) supported on each.
Note: If a blade slot is being decommissioned and has ports configured in logical switches,
the logical port assignments should be removed from that blade before removing the blade.
This action ensures a seamless transition for any new port or AP blade that might occupy
that slot in the future and does not apply if you are simply replacing a blade of the same
type.
The following restrictions apply when using XISLs. XISL use is not permitted in any of the
following scenarios:
The logical switch has ICL ports.
The logical switch is the default logical switch.
The logical switch is a base switch.
The logical switch is an edge switch for an FC router.
In this case, if the logical switch is enabled, you cannot allow XISL use. If the logical switch
is disabled or has not yet joined the edge fabric, you can allow XISL use; however, fabric
segmentation occurs when the logical switch is enabled or connected to an edge fabric.
Note: Using XISL and fmsmode at the same time is permitted, but this combination will
only work in a one-hop topology.
SIM_Port
Encryption
Before moving VE_Ports, you must remove the VE_Port tunnel configuration.
VE_Ports on the SX6 blade can be moved to any logical switch independent of the location of
the physical GE port.
All FC router commands can be executed only in the base switch context.
The fcrconfigure --bbfid command is not allowed when Virtual Fabrics is enabled.
Instead, use the lscfg command to configure the FID.
From FOS v9.0.x onwards, if a port is configured as an EX_Port and if the administrator
moves the port from the base switch to another logical switch, the operation fails. You must
remove the EX_Port configuration before moving it to another logical switch.
When a Virtual Fabrics-enabled switch is part of the backbone, only the base switch with
active EX_Ports will be able to route frames between edge fabrics. If there are other base
switches that are participating in the fabric and do not have active EX_Ports, those
switches will participate as transient switches to forward frames to the next hop till they
can be routed to the intended remote edge fabric. If any other default or logical switch is
introduced as a hop between edge fabrics, the newly introduced switch which is not the
base switch will not merge with the base switch fabric. This will cause frames to be
dropped.
For further information about device sharing and FC-FC routing, refer to the Fabric OS
Administration Guide: https://docs.broadcom.com/docs/FOS-90x-Admin-AG
Modern applications have an insatiable need for data. The rise of NVMe flash and
storage-class memory will drastically accelerate the performance of storage systems tasked
with holding that data. This means that having the right network technology in place is
essential. Fortunately, IBM b-type Gen 7 networking solution delivers an autonomous SAN
with next-level performance and integrated intelligence. IBM b-type Gen 7 provides the
accelerated foundation that modern data centers need to fully realize the benefits of the latest
storage technology today and possibly for the next decade.
This chapter describes a high-level guide focusing on Fibre Channel Storage Area Network
(SAN) design and best practices, covering planning, topologies, workload monitoring, and
detecting server and storage latencies—to help with decisions required for successful IBM
b-type Gen 7 SAN design.
In fact, an aging network will impede the performance of the on-demand data center. By
modernizing the storage network with IBM b-type Gen 7, organizations will enable a faster,
more intelligent, and more resilient network. This will maximize the performance, productivity,
and efficiency of their storage investments and resources, even as they rapidly scale their
environments.
5.1.1 IBM b-type Gen 7 platforms: the intelligent foundation of modern data centers
The products that make up IBM b-type Gen 7 portfolio are the new Brocade X7 Director and
the G720 Switch. These solutions are equipped with higher-performing hardware to unleash
NVMe technology, and they have the ability to discover and produce comprehensive
telemetry data across the fabric. They analyze and take actions based on that data to
optimize the storage network automatically.
Modernize with NVMe to Achieve Lower latency and Higher Bandwidth According to IBM
b-type Gen 7 technology substantially accelerates a data center environment, offering 50%
lower latency than the previous generation, while increasing bandwidth with 64 Gb/s links. By
leveraging NVMe-ready Fibre Channel technology, IBM b-type Gen 7 not only maximizes the
performance of NVMe storage and high-transaction workloads, but also simplifies the process
of integrating higher-performing NVMe storage into the environment without a rip-andreplace
process. IBM b-type Gen 7 has simplified IT modernization efforts. The NVMe storage
integrates seamlessly with existing ports: Brocade supports the concurrent use of SCSI and
NVMe technologies. In fact, with the release of vSphere 7 from VMware, an organization can
do a live migration of an individual virtual machine from SCSI to NVMe in the same Gen 6 or
Gen 7 fabric without stopping the application. This result constitutes one of the most risk
averse technology transitions in years.
IBM b-type Gen 7 provides investment protection by enabling organizations to upgrade their
existing Brocade Gen 6 director chassis to Gen 7. Organizations can leverage existing Gen 6
blades in Gen 7 directors, mixing Gen 6 and Gen 7 blades in the same chassis. In addition,
Brocade supports three generations of backward compatibility within the fabric. This is an
important feature, as it reduces risk and maximizes the value of the organization’s existing IT
investments.
Another consideration here is that the entire application base does not refresh at the same
time. And in fact, the more common scenario is that multiple generations of performance will
exist in the environment simultaneously. This is driven by the ability to take a service window
to re-platform existing environments to new server and storage acquisitions as well as the fact
that some legacy applications may not have an environment that runs in current operating
system (OS) releases or application versions. One may have 10-year-old or older operating
systems running connected with a host bus adapter (HBA) that is two or more older
generations of technology behind. How then do you balance that still critical application
against the needs and performance of the newer machines?
Topologies
A typical SAN design comprises devices on the edge of the network, switches in the core of
the network, and the cabling that connects all the devices together. Topology is usually
described in terms of how the switches are interconnected, such as collapsed core,
core-edge, and fully meshed. The recommended SAN topology to optimize performance,
management, and scalability is a tiered, core-edge topology. This approach provides good
performance without unnecessary interconnections. At a high level, the tiered topology has a
large number of edge switches used for device connectivity and a smaller number of core
switches used for routing traffic between the edge switches, as shown in Figure 5-1 on
page 72.
The difference between these three scenarios is device placement (where devices
are attached to the network) and the associated traffic flow:
Scenario A, collapsed core, has localized traffic, which could sit on the same ASIC, on the
same blade, or between blades. Collapsed Core can have small performance advantages
for performance-optimized workloads but does not provide ease of scalability and
depending on how many fabrics you would need, would determine the impact on
manageability.
Scenario B, core-edge, separates the storage and servers, thus providing ease of
management and higher scalability. An core-edge topology has only one hop from server
to storage, providing similar performance benefits as full mesh while allowing higher
scalability.
Scenario C is a full-mesh topology, and server to storage is no more than one hop.
Designing fabrics with UltraScale ICLs is an efficient way to save front-end ports, and
users can easily build a large (for example, 3456 ports or larger) fabric with minimal SAN
design considerations.
Collapsed core
The collapsed core topology places initiators (servers) and storage (targets) on the same
ASIC, line card or same chassis. This has a number of benefits depending on the size of the
environment. This is used a lot when customers are migrating from multiple switches to a
single dual core architecture where they can fit all imitators and storage into the same
chassis. If future design requirements are needed going with an core-edge could be more
beneficial in the long run when it comes to scale and ease of management.
Core-edge
The core-edge topology places initiators (servers) on the edge tier and storage (targets) on
the core tier. Since the servers and storage are on different switches, this topology provides
ease of management as well as good performance with minimal fabric latency, with most
traffic traversing only one hop from the edge to the core. (Storage-to-storage traffic is two
hops if the second storage is on another core switch, but the two cores can be connected if
fabrics are redundant.) The disadvantage to this design is that the storage and core
connections are in contention for expansion as they scale; however, using modular platforms
allows for flexibility while allowing the use of ICL ports for intraswitch connectivity to free up
device ports.
Full mesh
A full-mesh topology allows you to place servers and storage anywhere, since communication
between source and destination is no more than one hop. Using director-class switches with
UltraScale ICL ports for interconnectivity is essential to this design in order to ensure
maximum device port availability and utilization. Design this architecture with a minimum of
two switches up to nine switches in a full mesh.
Availability
The key to high availability and enterprise-class installation is redundancy. By eliminating a
single point of failure, business continuance can be provided through most foreseeable and
even unforeseeable events. At the highest level of fabric design, the complete network should
be redundant, with two completely separate fabrics that do not share any network equipment
(routers or switches).
Recommendations for the SAN design are to ensure application availability and resiliency via
the following:
Redundancy built into fabrics to avoid a single point of failure
Servers connected to storage via redundant fabrics
MPIO-based failover from server to storage
Redundant fabrics based on similar architectures
Redundant ISLs/ICLs for interswitch connectivity
Separate storage and server tiers for independent expansion
Core switches of equal or higher performance compared to the edges
The highest performance switch in the fabric defined to be the principal switch
Scalabilty
IBM b-type Gen 7 products are designed with scalability in mind, knowing that most
installations will continue to expand and that growth is supported with very few restrictions.
However, following the same basic principles outlined in previous sections as the network
grows will ensure that the levels of performance and availability will continue.
Evaluate the impact on topology, data flow, workload, performance, and perhaps most
importantly, redundancy and resiliency of the entire fabric any time one of the following
actions is performed:
Adding or removing initiators:
– Changes in workload
– Changes in provisioning
Some key points to cover when looking at the current status of a production FC network when
reviewing redundancy and resiliency:
Are there at least two physically independent paths between each source and destination
pair?
Are there two redundant fabrics?
Does each host connect to two different edge switches?
Are edge switches connected to at least two different core switches?
Are inter-switch connections composed of two trunks of at least two ISLs?
Does each storage device connect to at least two different edge switches or separate port
blades?
Are storage ports provisioned such that every host has at least two ports through which it
can access LUNs?
Performance
IBM b-type Gen 7 technology substantially accelerates a data center environment, offering
50% lower latency than the previous generation, while increasing bandwidth with 64 Gb/s
links. By leveraging NVMe-ready Fibre Channel technology, IBM b-type Gen 7 not only
maximizes the performance of NVMe storage and high-transaction workloads, but also
simplifies the process of integrating higher-performing NVMe storage into the environment
without a rip-and-replace process.
Extensibility
The IBM b-type Gen 7 Fibre Channel switch is taking advantage of newer and older
technologies within the same infrastructure. It has the ability to run multiple vendors within the
same architecture to meet evolving storage requirements. Three generations of
backwards-compatibility, future proof investments with a Gen 7 ready storage networking
platform and NVMe over Fabrics as a new protocol for solid state storage. You will want to
ensure you do not need to rip and replace.
Manageability
The IBM b-type Gen 7 Fibre Channel switches integrated to IBM SANnav Management Suite
as the SAN Management Interface. IBM SANnav 2.1.0 is a major new release of IBM ’s b-type
Gen 7 two Fibre Channel SAN management software products, IBM SANnav Management
Portal and IBM SANnav Global View. SANnav Management Portal 2.1.0 supports the
introduction of IBM ’s new Gen 7 platforms with Fabric OS 9.0.0 and adds new features and
capabilities that make managing IBM SAN environments easier than ever before.
When selecting which criteria to meet, you should engage users, server and storage subject
matter experts (SMEs), and other relevant experts to understand the role of the fabric. Since
most SANs tend to operate for a long time before they are renewed, you should take future
growth into account as SANs are difficult to re-architect. Deploying new SANs or expanding
existing ones to meet additional workloads in the fabrics requires a critical assessment of
business and technology requirements. Proper focus on planning will ensure that the SAN,
once deployed, meets all current and future business objectives, including availability,
deployment simplicity, performance, future business growth, and cost.
Design worksheets (see Appendix A, “Fabric details” on page 305) which assist in
architecting a SAN environment, are provided are provided as a reference for documenting
assets and metrics for SAN projects. A critical aspect for successful implementation that is
often overlooked is the ongoing management of the fabric. Identifying systems-level SMEs for
all components that make up the SAN, as well as adequate and up-to-date training on those
components, is critical for efficient design and operational management of the fabric. When
designing a new SAN or expanding an existing SAN, you should take into account the
following parameters.
Application virtualization
Which applications will run under a virtual machine (VM) environment?
How many VMs will run on a physical server?
Under what conditions will the VMs be migrated (business and nonbusiness hours; is
additional CPU or memory needed to maintain response times)?
Is there a need for solid-state storage media to improve read response times?
Sizing
How many user ports are needed now?
How many devices will connect through an access gateway?
How many inter-switch links (ISLs)/Brocade UltraScale inter-chassis links (ICLs) are
required to minimize congestion in the fabric?
What distances for ISL/ICL connections need to be supported?
Does the fabric scale out at the edge or the core?
latency to sub-microsecond speeds. Local switching is the ability to switch data traffic through
a single ASIC by using both ingress and egress switching ports in a common port group.
When using switches that contain multiple switching ASICs like the X6 /X7 director,
configuring host and target connections on the ports that share a common ASIC will minimize
latency by avoiding the need to move data across multiple ASICs/port groups or across a
director backplane to a different blade. To find details on port groups and local switching
configuration, refer to the Brocade Fabric OS Administration Guide and the hardware
installation guide for the appropriate product. Because Gen 7 FC is backwards compatible
with Gen 6 networking, a Gen 7 edge switch or director can be added to an existing Gen 6
fabric. This allows for local all-flash connectivity to a Gen 7 switch to gain the performance
advantages of Gen 7 while preserving the investment in Gen 6 networks.
In summary, best practices for SAN design are to ensure application availability and resiliency
via the following:
Redundancy built into fabrics to avoid a single point of failure
Servers connected to storage via redundant fabrics
MPIO-based failover from server to storage
νRedundant fabrics based on similar architectures
Redundant ISLs/ICLs for interswitch connectivity
Separate storage and server tiers for independent expansion
Core switches of equal or higher performance compared to the edge switches
The highest performance switch in the fabric defined to be the principal switch
Note: In Figure 5-3, ISL trunks are placed on separate ASICs or port groups. It is
important to match ISL placement between devices and across fabrics to ensure simplicity
in design and assist in problem determination.
5.2.8 Best practices for IBM b-type Gen 7 UltraScale ICL connections
Each core blade in a chassis must be connected to each of the two core blades in the
destination chassis to achieve full redundancy, as shown in Figure 5-5.
Note: For redundancy, use at least one pair of links between two core blades.
Domain 1 Domain 2
IBM b-type IBM b-type
X7 X7
Note: For more information, see the Scale-Out Architecture with IBM (Brocade) UltraScale
Inter-Chassis Links Design Guide.
https://www.broadcom.com/products/fibre-channel-networking/directors/x7-directo
rs#documentation
UltraScale ICL connections are considered a “hop of no concern” in a FICON fabric. When
using core-edge SAN design methodologies, edge switches should connect to at least two
core switches with trunks of at least two ISLs each. Each of those trunks should be attached
to a different blade/port group. In order to be completely redundant, there would be a
completely mirrored second fabric and devices must be connected to both fabrics using
MPIO.
Use the same type of optics on both sides of the trunks: Short Wavelength (SWL), Long
Wavelength (LWL), or Extended Long Wavelength (ELWL).
While not intended as a definitive design document, this guide introduces concepts and
guidelines to help you avoid potential issues that can result from poor design practices.
Designing device connectivity depends a great deal on the expected data flow between
devices. For simplicity, communicating hosts and targets can be attached to the same switch
(Figure 5-7)
Figure 5-7 Hosts and targets attached to the same switch to maximize locality of data flow
However, this approach does not scale well. Given the high-speed, low-latency nature of
Fibre Channel, attaching these host-target pairs on different switches does not mean that
performance is adversely impacted for common workloads. Although traffic congestion is
possible, it can be mitigated with proper provisioning of ISLs/UltraScale ICLs. With current
generation switches, locality is not required for performance or to reduce latencies. For
mission-critical applications that depend on extremely fast response times, architects may
want to localize the traffic when using flash storage or in very exceptional cases, particularly if
the number of ISLs available is restricted or there is a concern for resiliency in a multihop
environment (Figure 5-8 on page 83).
Figure 5-8 Hosts and targets attached to different switches for ease of management and expansion
One common scheme for scaling a core-edge topology is dividing the edge switches into a
storage tier and a host/initiator tier. This approach lends itself to ease of management as well
as ease of expansion. In addition, host and storage devices generally have different
performance requirements, cost structures, and other factors that can be readily
accommodated by placing initiators and targets in different tiers.
Adding a layer of flash can therefore open up bottlenecks and relieve congestion resulting
from long wait times. Still, storage array performance can degrade over time, which can be
attributed to factors that include:
Proliferation of VMs and the "IO blender" effect, whereby IO processes become
increasingly non-sequential and increase reads and writes to disk.
Insufficient controller cache.
Misaligned LUN configurations where OS block sizes do not match well to those on block
storage.
Misconfigured control values such as edge hold time and queue depths.
Provisioning strategies can favor capacity over usage.
Design Guidelines
Ensure that network throughput capacity matches or exceeds the capabilities of storage
devices.
Be careful if you deploy mixed arrays with different performance characteristics.
Experience has shown that it is very easy for a Tier 3 storage array, depending on how it is
used, to impact the performance of arrays in the same fabric. Troubleshooting in these
situations is very difficult.
Network switch and optic speeds should be limited to two generations at most, with a best
practice of uniform speed to avoid target ports from slowing to accommodate lower-speed
ports and creating a back-pressure situation.
Enable Forward Error Correction for ISL connections as well as connectivity to Gen 5
devices that support it. FEC is on by default for Gen 6 networks and should always be
used to ensure consistency of network transit times.
Control the number of LUNs behind each storage port based on the type of usage they will
receive.
Check on any special short-frame traffic to avoid frame congestion at array ports. It may
be necessary to increase the number of buffers at the array port to accommodate the extra
control traffic.
Use advance Brocade FOS threshold timers to monitor hosts and storage arrays to ensure
that array ports do not reset due to a high-latency host, and thus do not adversely impact
other connected hosts.
NVMe
The next consideration in terms of why the storage network must be rethought and
re-architected is Non-Volatile Memory Express (NVMe).
For a complete review of NVMe over Fibre Channel fabrics, refer to Chapter 8 of this
Redbook.
Storage virtualization
Storage virtualization enables LUNs accessed by servers to be abstracted from the physical
storage (typically storage arrays) on which they actually reside. (These are not the same as
traditional storage array LUN allocations, which can also be viewed as a form of
virtualization.) Virtualized LUNs that are disassociated from their actual storage allow for
more flexible storage provisioning processes. Performance may also improve, as the virtual
LUNs can be striped across multiple storage arrays.
There are two general types of storage virtualization: one uses an external controller (called
in-line virtualization), and in the other, the virtualization occurs inside a storage array. In-line
solutions are slightly more flexible, because they can use physical storage from a variety of
sources and vendors.
the storage network. The red arrows indicate data access to and from the storage controller.
The storage controller typically controls all access to the physical storage, shown on the right
(and indicated by the blue arrows). This creates a very flexible storage solution, because
logical LUNs can be striped across several physical arrays to improve performance, and
logical LUNs can be manipulated completely transparently to the host or VM.
The major benefit of this type of storage virtualization is that storage can now be provisioned
in units of capacity (500 gigabytes or a terabyte) rather than physical LUNs. This is a first step
toward viewing storage as a service instead of as physical units. VM provisioning now
becomes less complex and easier to automate. Look into products such as IBM SAN Volume
Controller.
Design Guidelines
Each storage controller in an in-line solution serves as both an initiator and a target.
ISL utilization increases with in-line virtualized storage. Make sure that you have enough
ISL bandwidth to handle the increased load.
There is also the possibility that the in-line storage heads will communicate through Fibre
Channel or generate many more SCSI control frames to manage their attached storage,
which can contribute to frame congestion. You may need to increase the number of buffers
at the ports that connect to the storage controller to accommodate this behavior.
It is much more difficult to determine initiators and targets with in-line virtualized storage.
Since they are on the same switch, be careful about deploying tools such as Port Fencing.
Monitoring
Utilize MAPS to track thresholds of critical parameters
FPI monitoring is very useful in determining latencies associated with virtualized storage.
Use Flow Vision to look at high-traffic-usage flows and identify the source of bottlenecks
and congestion.
Consider running Flow Vision constantly for the most mission-critical applications to keep
a historical record of application performance profile and intermittent irregularities.
For "frequent caller" applications, run Flow Vision on a regular basis where time permits to
verify good health.
Evaluate the impact on topology, data flow, workload, performance, and perhaps most
importantly, redundancy and resiliency of the entire fabric any time one of the following
actions is performed:
Adding or removing:
– Changes in provisioning
– Storage
– Switches
– AISLs and ICLs
Change in virtualization (workload and storage) strategies and traffic flow pattern If these
design best practices are followed when the network is deployed, then small incremental
changes should not adversely impact the availability and performance of the network.
However, if changes are ongoing and the fabric is not properly evaluated and updated,
then performance and availability can be jeopardized.
Some key points to cover when looking at the current status of a production FC network
include:
Reviewing redundancy and resilience:
– Are there at least two physically independent paths between each source and
destination pair?
– Are there two redundant fabrics?
– Does each host connect to two different edge switches?
– Are edge switches connected to at least two different core switches?
– Are interswitch connections composed of two trunks of at least two ISLs?
– Does each storage device connect to at least two different edge switches or separate
port blades?
– Are storage ports provisioned such that every host has at least two ports through which
it can access LUNs?
– Are redundant power supplies attached to different power sources?
– Are zoning and security policies configured to allow for patch/device failover?
Reviewing performance requirements:
– Host-to-storage port fan
• Host to ISL
• Edge switch to core switch
– Routing policy and currently assigned routes (evaluate actual utilization for potential
imbalances)
– Use of FEC for all ISLs and connections to Gen 5 (if supported) and Gen 6 devices.
Watching for latencies because of the following:
In summary, although IBM b-type Gen 7 SANs are designed to allow for any-to-any
connectivity and they support provision-anywhere implementations, these practices can have
an adverse impact on the performance and availability of the SAN if left unchecked. As
detailed above, the network needs to be monitored for changes and routinely evaluated for
how well it meets desired redundancy and resiliency requirements.
Whereas the IBM b-type Gen 7 SAN technology provides a suite of built-in measures to avoid
interference of workloads that expose any kind of congestion behavior, such as Traffic
Optimizer, FPIN, MAPS, and SDDQ, protecting critical workloads is crucial. Ideally the most
demanding and critical workloads have dedicated storage array target ports (or even
dedicated arrays) and the shortest possible path through the SAN. The purpose for this is to
avoid any interference of other less critical workloads that may expose behavior that results in
congestion or back pressure, which could interfere and adversely impact the performance of
critical workloads.
In the case where the number of business-critical servers exceeds the number of available
ports on the core for server connections, it is necessary to connect the business-critical
servers on edge switches. The most common model is then to use dedicated edge switches
for business-critical servers and decrease the fan-in ratio of servers to ISLs.
Business-critical VMs
In today’s data centers, it is not uncommon for some or all business-critical workloads to be
running on VMs. Combining this with the digital business environment, the value (business
criticality) and performance requirements for a given application often change throughout the
life cycle of the application. This means that it can be difficult (or impossible) to predict the
requirements for an application and plan placement accordingly from the beginning—luckily
hypervisors provide the ability to move VMs between hosts without downtime and to migrate
storage when necessary. To apply the same principle of server placement of (bare metal
installed) business-critical applications to VMs, deploy dedicated hypervisor clusters
connected on the core or on high-performance edge switches and use these hypervisor
clusters only for business-critical and performance-sensitive VM workloads.
Visibility into each VM’s workload on the same data store (backed by LUN or NSID) can be
achieved with VM Insight. VM Insight enables storage administrators to monitor VM-level
application performance and set baseline workload behavior. This information can be used to
quickly determine whether a storage fabric is the source of performance anomalies for
applications running on VMs. Based on VM Insight, storage administrators can provision and
plan placement in the SAN on application requirements and can fine-tune the infrastructure to
meet service-level objectives.
Learn how to gather and use Brocade's tools to improve the performance and reliability of the
SAN infrastructure.
Gathering requirements
The SAN project team should interview all stakeholders (IT application owners, finance,
corporate facilities, IT lab administrators, storage and network administrators, and end users)
who have a vested interest in the project-and this applies equally to planning for both new and
updated SANs.
Application owners
As critical stakeholders, application owners care because everyone is measured on
application uptime. Application outages are something that users notice, and they can have
severe financial impact for a business. With a redundant or a resilient infrastructure, hardware
outages are transparent to the user, and only SAN administrators need to pay attention.
To better understand their requirements, questions to ask the application owners include:
What is the business goal for this application? (Is it a database that multiple applications
rely on for business transactions?)
What are the availability requirements?
Is the application latency sensitive?
Are there peak periods of utilization or other traffic patterns?
What are the IOPS requirements in terms of read/writes?
What is the worst-case response time before an outage?
Is the application running on a cluster?
Is there application downtime that can be used for applying patches, software upgrades,
and maintenance?
Can the application run on a VM? If so, how many other VMs can co-exist on the same
physical hardware?
The business criticality of the application will determine the SAN design and the DR strategy,
including backup and recovery. If the application is mission critical, the infrastructure must be
fully redundant, with no single point of failure for both mainframe or distributed open systems
architectures.
How much storage capacity is allocated to flash (more cache allows for more consistent
response time) if using a hybrid array?
Are storage tiers used in the environment? What is the policy used for migrating data? Are
different tiers used for online storage? What is the impact?
What is the raid level used? This will determine available disk space and performance for
the application.
How many FC ports are there in the array?
Are the arrays front-ended by a storage virtualization controller? If so, what is the
additional latency?
What are the recommended fan-in and fan-out ratios for the arrays used for this
application? What are the limits?
Is there a Disaster Recovery (DR) site? If so, how is it connected (dark fiber, FCIP)?
What is the available/required bandwidth between the intra-site for DR? Can the existing
storage infrastructure support DR with the additional load?
What tools are used for mirroring and replication (host-based or array-based)? If
host-based, was the failover tested? If so, was there any impact in application uptime? If
storage-based, was the failover tested? Did the LUNs appear on the active ports?
Was there an impact to application uptime?
If the current SAN is being expanded, adequate performance metrics should be collected to
ensure that the existing design can be expanded to address new workloads.
Are there performance (bandwidth) or latency issues in the existing SAN?
Are procedures in place to address redistribution of capacity when switch port utilization
exceeds 75 percent?
Is the current design two-tier (core-edge) or three-tier (edge-core-edge)?
Is the SAN centrally managed by a tool such as IBM Storage Insights or IBM Spectrum®
Control?
If there is an existing SAN, how is it managed (CLI, IBM Network Advisor)? Is there a
separate network for SAN management?
Are access control policies in place for change management (zoning)? Is there a zoning
policy? Are there devices in the zone database that no longer exist? What type of zoning is
used (port or WWN)?
Is the current SAN a redundant configuration?
Is there an identified server to capture logs from the fabric?
Is the traffic equally distributed across the ISLs or the trunks?
Is historical performance data available for initiators, targets, and ISLs?
How many unused ports are available per switch?
5.4.1 IBM b-type Gen 7 provide infrastructure for today and tomorrow
The IBM b-type Gen 7 platforms breakthrough 64 Gbps performance accelerates application
response time by up to 71 percent, eliminating IO bottlenecks and unleashing the full
performance of an all-flash data center.
These systems will seamlessly integrate next-generation NVMe over Fabrics with Gen 7 Fibre
Channel networks without a disruptive rip and replace. In addition, these platforms will also
seamlessly connect older generation devices and storage networking equipment with three
generations of backward-compatibility.
Further protecting future investments, the IBM b-type Gen 7 Director family will support future
Fibre Channel generations as a Gen 7-ready storage networking platform. The IBM b-type
Gen 7 backplane is designed with more links, to accommodate a doubling and quadrupling of
bandwidth in the future. This will enable platform reusability for Gen 7 Fibre Channel, enabling
organizations to maximize their IBM b-type Gen 7 investments and optimize the performance
of their networking infrastructures.
IBM b-type Gen 7 platforms also enable customers to future-proof their storage networks by
accelerating new technology deployments such as NVMe over Fabrics. NVMe is a new
protocol for solid-state storage devices built with nonvolatile memory technologies. NVMe
provides dramatically lower latency for storage IO operations and significantly higher IOPS
per device. In the future, organizations will be able to transition their highperformance,
latency-sensitive workloads to flash-based storage with NVMe over Fabrics, thus realizing
significant performance gains and reaping the full benefits of flash.
NVMe over Fabrics will scale up the number of devices it can address by adopting NVMe over
Fabrics technology. NVMe over Fabrics enables end users to achieve faster application
response times from network-attached storage and to harness the performance of hundreds
of solid-state drives for better scalability across virtual data centers. Fibre Channel is one of
the fabric technologies that will be supported by NVMe over Fabrics. IBM b-type Gen 5 and
Gen 6 Fibre Channel switches already support NVMe over Fabrics, with no changes required.
Customers can seamlessly integrate IBM b-type Gen 7 Fibre Channel networks with
next-generation NVMe over Fabrics without a disruptive rip and replace for the increased
performance, application response time, and scalability that their next-generation data
centers require.
A detailed procedure is available that provides the materials and instructions required to
migrate your IBM b-type Gen 6 director (SAN512B-6 and SAN256B-6) running either Fabric
OS (FOS) 8.x or 9.x to a IBM b-type Gen 7 director (SAN512B-7 and SAN256B-7).
Note: Upgrading to a Gen 7 director is permanent on the CPs. This means that you cannot
downgrade to Gen 6; after the upgrade to Gen 7, you cannot use the same CP blades in a
Gen 6 chassis.
The Gen 7 director supports mix-and-match blades, allowing Gen 6 and Gen 7 blades to
be installed within the same chassis. The following optional port or extension blades are
available:
48-port Gen 6 IBM b-type Gen 7 FC32-X7-48 port blade with 32Gb/s line rate ports
– Added value over FC32-48 (reduced latency, congestion avoidance, and Traffic
Optimizer)
– Non-upgradeable to 64Gb/s speeds
IBM b-type Gen 7 FC32-64 high-density port blade – Increases scalability and
maximizes space utilization
IBM b-type Gen 7 SX6 Extension blade – Disaster recovery solutions over long
distances
https://www.broadcom.com/products/fibre-channel-networking/directors/x7-directors#
documentation
https://www.broadcom.com/products/fibre-channel-networking/directors/x7-directors#
documentation
Brocade SAN Health is a free tool that allows SAN administrators to securely capture,
analyze, and report comprehensive information about fabrics with switches that are running
Brocade Fabric OS and Cisco MDS fabrics that are running SANOS/NXOS.
Download Brocade SAN Health and find details and instructions on how to use it at:
https://www.broadcom.com/products/fibre-channel-networking/software/sanhealth
SAN Health will assist in capturing information to analyze the current health of your fabric. If
issues with the health of your fabric are found, resolve them before moving forward with any
migration. In addition to the health status, information on the physical and logical layout can
be used to transition to newer SAN platforms while maintaining consistency in your design.
Because it provides a point-in-time snapshot of your SAN, the SAN Health Diagnostics
Capture is invaluable to your change-tracking process.
With Brocade’s free SAN Health tool, you can generate personalized storage network
performance and inventory reports to help you prevent issues and optimize operations.
Figure 5-11 Comprehensive reporting with health and best practice checks\
This section will highlight those migration factors that need to be taken into account for the
migration plan. Using SAN Health will allow you to gather information to help make the
decisions for a smooth and seamless transition.
In addition to knowing about the current infrastructure, the current SAN design should be
reviewed to help incorporate future growth, backup and recovery or business continuity
projects and server/storage upgrades.
Install and apply power to the new IBM b-type Gen 7 SAN Directors, leaving them
disconnected from all storage or network cables.
Gather information from the IBM b-type Gen 7 SAN Directors required for building and
configuring the infrastructure.
Leverage SAN Health to model the new SAN infrastructure topology:
– Modeling is done by leveraging the Visio diagram of the SAN infrastructure created by
SAN Health
– Rearrange the switches as needed, insert newly proposed server and arrays to the
SAN Health report then optimize the configuration for any-to-any connectivity.
– If there are flaws in the overall design of your data fabric, the topology diagram, helps
bring them to the forefront, making it easier for the infrastructure architect team to
improve on designs.
Configure the IBM b-type Gen 7 SAN Directors
– Create baseline configurations for all Directors/Switches
– Import/purge zoning sets
– Create zones for new devices
Validate the configuration
– Run SAN Health and address any errors found before moving forward with migration
Attach all cables to server and storage as required, as shown in Figure 5-16).
Figure 5-16 SAN Director migration example (integrating new SAN Directors/Switches)
SAN Health improves capacity planning and productivity, and reduces the time required to
troubleshoot problems when they arise. Consider adding SAN Health to your toolkit....
It’s FREE!
5.5 References
Software and Hardware Product Documentation FOS documentation is located in two
different locations within Broadcom for IBM b-type Gen 7 products and software. The publicly
facing content is located on Broadcom.com. The nonpublic documentation is located on the
Broadcom Customer Support Portal (CSP).
For more information about your specific Fabric operating system release, see the following
resources:
https://www.broadcom.com/products/fibre-channel-networking
https://www.broadcom.com/products/fibre-channel-networking/directors/x7-directors
#documentation
The webpages contain all user guides, reference manuals, white papers, eBooks, product
briefs, administrative guides, compatibility guides, and case studies.
Brocade Fabric OS Administration Guide
Brocade Fabric OS Command Reference Manual
Brocade Fabric OS MAPS User Guide
100 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm
Most SAN upgrades skip a generation. Most upgrades to Gen 7 therefore will be from Gen 5.
The most significant change when upgrading from Gen 5 is that a 32 port card is no longer
supported on the director class (bladed) switches. The new blades are 48 ports. Furthermore,
the control processor (CP) cards were moved resulting in new slot numbering on the bladed
(director class) switches.
IBM Network Advisor is nearing end of support and is being replaced by SANnav. IBM
Network Advisor does not support FOS 9.0 or any Gen 7 product. SANnav does support Gen
5 products on the 7.4 and 8.2 code streams. So as to minimize change activity when
upgrading to Gen 7, upgrading to SANnav first is recommended.
While Gen 7 offers data center managers higher performance and valuable new analytics,
neither these nor any other Gen 7 features will impact FICON planning and implementation.
102 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm
IBM Z15™ on Gen 31 Oct 2021 Although support for these hardware platforms continues
5 through 30 April 2025, for z15 connections the last day of
support is 31 Oct 2021.
IBM Network 8 Feb 2023 IBM Network Advisor FOS support stops at FOS v8.2.x
Advisor SANnav is the replacement product. Most customers plan
the management system upgrade to coincide a few months
prior to the end of their IBM Network Advisor support
contract.
6.1.2 Software
The last code stream supported on Gen 5 is FOS 8.x. Gen 6 supports both the FOS 8.x and
9.x code streams.
All SAN switches in a fabric must be no more than one major code revision apart. All switches
operating with FOS 8.x and 9.x can participate in the same fabric but switches operating with
FOS 7.x cannot join the same fabric that includes switches operating with FOS 9.x.
Deprecating the prohibit/allow matrix (aka PDCM) is planned for FOS v9.1. Customers using
the PDCM should plan on considering other methods, typically zoning, to effect the blocking
of certain paths. The PDCM is being deprecated because all IBM software products that used
the PDCM are no longer supported by IBM.
The FICON logical switch was introduced in FOS 8.1.0. The difference between a logical
switch defined for FICON and configured for FICON is that the logical switch configured for
FICON can be reconfigured such that it no longer meets the FICON requirements while a
logical switch defined as a FICON logical switch cannot be reconfigured so as to remove the
FICON requirements.
In prior versions of FOS, determining if port address was bound was challenging for
non-expert users. A “bound” address means that the fibre channel address for the port cannot
change. This is important because the IBM z channel subsystem builds fibre channel
addresses based on the link addresses defined in the Hardware Configuration Definition
(HCD) tool. Another reason for the FICON logical switch is that it’s possible to configure a
switch to meet all of the FICON requirements and then back certain changes out after the
channels have come online. Since a mainframe channel only checks the proper security
attributes at login time (when the PCHID is enabled, not when the CHPIDs are configured
online) there is the potential for a channel not to come online the next time it goes off and
back online. For most customers this is the result of a CEC IML or channel path maintenance.
Although circumstances for changing the fibre channel address assigned to a port on the
switch is unlikely, should it happen, the IODF would no longer be valid resulting in device or
channel errors. This potential problem is fixed with the defined FICON logical switch.
Other than common chassis components, each logical switch behaves as independent
hardware. There is no sharing of information between one logical switch and another. When
using tools such as System Automation or Disk Magic to manage and monitor switches
through CUP, a separate channel with CUP defined must be attached to each logical switch.
If a separate logical fabric is created for disk or tape mirroring is put into its own logical fabric
for mirroring, not only will a separate channel be required to monitor mirroring ports, but since
that CHPID is a FICON channel, the logical switch must be defined as FICON logical switch.
Defining a logical switch as a FICON logical does not affect other protocols, such FCP which
is used for mirroring. The FICON logical switch only adds secure features and supports CUP.
Since TS7700 tape mirroring when done with fibre channel and Metro Mirror use FCP
protocol, this means for CUP support a dedicated FICON channel just for CUP is required. An
alternative, so as to avoid the need for dedicating a channel just for CUP, is to keep mirroring
ports in the same logical switch with FICON CHPID to control unit paths and use zoning to
isolate the traffic.
104 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm
S in g le B y t e L in k A d d r e s s 06
T w o - B y t e L in k A d d r e s s B506
F C A d d re s s B50600
D o m a in ID in H e x A LPA
P o r t A d d r e s s In H e x
Figure 6-1 Link address and fibre channel address
Once two-byte link addressing is defined for a channel, all link addresses for that channel
must use two-byte link addressing. Once two-byte link addresses are defined, whether
cascading or not, all the fabric security settings must be set on the switch. For FOS 9.0 and
above, and therefore all Gen 7 switches, this means the channel must be connected to a
FICON logical switch.
If the high integrity settings are not configured on the switch then zOS will set the channel
offline and put it in the “Invalid Attach” state. Once a channel is in the invalid attach state, the
misconfiguration on the switch must be correct and then the channel re-enabled. It is the
PCHID, not the CHPID, that is in the invalid attach state. You cannot simply configure the
CHPID off and back online. The PCHID must be toggled from the System Element (SE).
The fabric security features (HIF) are not checked by the channel when single byte
addressing and therefore not recommended. Configuring 2 byte link addresses is the most
secure method and required in some cases.
Port address
The hexadecimal port address is the single byte link address or the least significant byte of a
two byte link address.
The ALPA was originally used in fibre channel for arbitrated loop devices. Arbitrated loop is no
longer supported and was never supported for FICON. Today, the ALPA is used for an
extension of the port addressing and Node Port Identification Virtualization (NPIV).
The bits for NPIV allow multiple WWNs to log into the same switch port. These bits determine
which logical entity frames belongs to. This is how virtual servers using zLinux or zVM, which
use FCP protocol, can share the same channel. FICON does not use these bits; however, the
ALPA must be the same for all FICON attached control units and channels.
Since the 8 slot director class products can exceed 256 ports in a single domain (logical
switch), the upper 2 bits of the ALPA are used in certain addressing modes.
A FICON channel builds the ALPA portion of the FC address by using the same ALPA where
the channel logged in. The ALPA portion of the FC address of control units therefore must
have the same ALPA as the channel. This is why certain addressing modes do not work for
FICON. In a logical switch defined for FICON, the ALPA is always 00.
Often, storage and processor upgrades are aligned with SAN upgrades. Make sure all SAN
requirements are well known prior to commencing any design planning and remember to
account for swing ports when upgrading storage and processors.
Management Although FICON SANs can be managed via the CLI, doing so is
very unusual. Nearly all FICON SANs use management
software.
Network Advisor does not support FOS 9.x. Therefore, Gen 7
switches are not supported by Network Advisor.
The SANnav Management Portal is the replacement product for
Network Advisor. The enterprise version is required whenever
managing a director class product or environments with more
than 600 ports. Otherwise, the base version is adequate.
FICON environments nearly always require the Enterprise
version of SANnav.
SANnav 2.1 or higher is required when managing Gen 7
products.
SANnav 2.2 or higher is recommended for managing FICON
fabrics.
SANnav Global View is s new software package that may be
useful when managing either multiple independent production
sites or active-active backup sites. SANnav Global View provides
a single pane of glass management for up to 20 instances of the
SANnav Management Portal.
Typically, customers plan their upgrade from Network Advisor to
SANnav to coincide with the end of their Network Advisor
support contract. It is highly recommended to start the transition
90 days in advance. There is a free fully functional 90-day trial
version of SANnav that customers can use for this purpose.
Routing method All switches in a fabric must use the same routing method so an
outage must be planned if changing the routing method.
Do all control units in your environment support FiDR?
Exchange based routing is the most efficient and resilient routing
method so if FiDR is supported on all channels and control units,
exchange based routing is recommended.
Utilizing 48 port cards and high HCD and cable plant consideration.
availability Most FICON SANs use director class products with 32 port
blades. Furthermore, the CHPID mapping tool and other tools
used to aid in channel path planning are built around 32 port
cards. CHPID mapping based on 32 port cards likely will not work
with 48 port cards.
106 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm
Naming Convention Prior to having a FICON logical switch, the difference between a
chassis, a switch, and a fabric were subtle and of little or no
consequence to mainframe administrators. Although the
technical differences will mean little to a mainframe
administrator, there are times when you will need to differentiate
between a chassis and a logical switch, especially when creating
multiple logical switches in the same chassis. Similarly, when
cascading you will need to be able to differentiate the logical
switches from the fabric.
Naming conventions are preferences that vary for each
organization. The only recommendation here is that you make
each name unique and easily identifiable. This can be done by
simply appending “_fab” at the end of fabric names and
“_chassis” at the end of chassis names.
Chassis, fabric, and switch names can be as long as 64
characters. Packing too much information into a name can make
it difficult to easily identify the item it is associated with and fill up
too much screen space. The SANnav database has a description
field where information such as rack and aisle location can be
added so there is no need for overly complex names. Although
there is no specific recommendation for naming conventions, the
general advice is to keep names simple.
Supported Hardware & Software An inventory check to ensure all hardware and software are
currently supported and that interoperability between new
hardware and software being introduced to the fabric is
supported in the existing environment. The recommended best
practice is to complete all required changes to the existing
environment before commencing other upgrades so as to
compartmentalize changes.
Despite this separation, most FICON implementations using directors with 32 port cards put
both channel paths on the same switch. Both channel paths, therefore, used the same
domain ID so the high byte of the link addresses was the same for both paths. Only the lower
byte of the link address was different.
To minimize the number of FRUs in a channel path, some organizations planned the CHPID
and control units connections to all reside on the same port card. For high-speed transaction
processing demanding the lowest latency, some organizations planned the CHPID and
control unit connections to be connected to ports sharing the same ASIC.
It may be desirable to use QoS zones in some cases. The two most common reasons are:
ISL bandwidth is limited and there is a mix of tier 1 applications and backup or mirroring
applications.
Cascaded CTC paths are defined.
– CTC traffic is typically very small block. Each block, regardless of size, will utilize at
least one fibre channel frame. Each fibre channel frame requires one buffer credit so
CTC applications can starve other applications for buffer credits even though the link
bandwidth is not fully utilized.
108 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm
– Cascaded CTC is usually put into it’s own low QoS zone
Brocade ASICs are designed to use cut-through routing. As soon as the header of a frame is
read, the frame is forwarded to the next port. Local switching occurs when the channel and
control unit are connected to ports that share the same ASIC. The switching latency within the
Gen 7 ASIC is 460 nsec. This latency is additive when traffic is traversing multiple ASICs. All
ASIC to ASIC switching is performed via a connection through the core blade even if the
ASICS are on the same port card. This involves another ASIC and backplane traffic.
110 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm
The IBM CHPID Mapping Tool has not been updated to account for 48 port blades. To aide in
link address planning, there is an Excel Workbook available from your Brocade
representative.
With 48 port cards, an 8 slot director can contain as many as 384 ports but since link
addressing is limited to 256 addresses per switch, enterprises have to consider how to make
use of the 48 port card.
8 Slot Chassis, one FICON logical This is not recommended but serves as a useful discussion of
switch how 48 port cards alter the chassis configuration.
A logical switch can be created on each chassis using just the
lower 16 ports on each column of ports per blade effectively
making it a 32 port card. Most customer who do this order some
empty blades and move SFPs from the upper unused ports.
Instead of leaving the upper 16 ports vacant, they could be put
into another logical switch but few, if any, mainframe customers
would have a use for such a logical switch.
An 8 slot chassis can also be ordered with just 5 cards for a total
of 240 ports or 6 card for a total of 288 ports, 32 of which would
be left unused or used for another purposes such as mirroring.
4 Slot Director The maximum port count will be limited to 192 ports but if FRUs
were considered in the channel path planning, a new CHIPD
map likely will be required.
8 Slot Director, 2 FICON logical Creating two FICON logical switches per chassis allows for the
switches per chassis greatest density. Instead of 2 paths per switch in each chassis,
use 1 path per logical switch with two logical switches per
chassis. Instead of 254 usable ports per chassis there can be as
many as 384 usable ports per chassis resulting in greater than
50% increase in port density for the same foot print. If a fully
populated director is not necessary, this approach can still be
used. The recommended layout is to start populating the first
logical switch with port cards on the left half of the chassis and
populate the right side of the switch with port cards for the
second logical switch. This allows for symmetric growth.
For FICON environments, when using one logical switch per chassis, the maximum number
of ports per logical switch is 256 ports (254 usable ports if CUP is enabled). Since 256 is not
evenly divisible by 48, either 5 port cards can be used which yields a total of 240 ports or 6
port cards which yields a total of 288 ports. In addition to losing port count with just 5 port
cards, using an odd number of port cards creates some planning challenges with redundant
pathing. Using 6 port cards wastes 32 ports. In either case, 2 or 3 slots cannot be used for
FICON unless there is a second FICON logical switch in the chassis.
Although partitioning a chassis into two symmetrical logical FICON switches results in a
maximum logical switch size of 192 ports, by splitting redundant paths to their own logical
switch there is an additional 64 link addresses available per path group. With two logical
FICON switches per chassis this results in a total of 128 additional link addresses available
per chassis.
When CUP is enabled, link addresses xxFE and xxFF are logical addresses. If there are ports
associated with those addresses then those ports must be disabled. Since there is no need to
associate link addresses xxFE and xxFF in a 192 port switch, those physical ports are also
available resulting in a slightly better than 50% port density improvement with this approach.
112 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm
Figure 6-6 Eight Logical Switches for 8-Way Pathing Option to Fully Utilize All Ports
It is possible for some logical switches to use the base switch (XISLs) while other logical
switches can be configured to have their own dedicated ISLs.
Setting up the FICON display mode is done on a per user basis and should be the first a
FICON SAN user should do after setting up their account credentials.
1. In the upper right hand corner, click on the person icon and in the drop down box, select
“User Preferences”, as shown in Figure 6-8 on page 115.
114 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm
2. In the display that follows, scroll to the bottom of the screen and find FICON Display under
“Tables”. If the FICON Display is “Disabled”, click “Edit”, as shown in Figure 6-9.
3. Check the “FICON Display” box. Most users select “Yes” for “Persist Last Column
Selection”. When set to “Yes”, customization of the displays will persist after logging out
and logging back in. When done, click on “Save” at the bottom of the display, as shown in
Figure 6-10.
2. Click SANnav.
c. Click SAN Configuration.
d. Click Logical Fabric Management.
116 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm
Name This is the name of the fabric, not the logical switch.
Description This is the description of the fabric, not the logical switch. With
single switch fabrics, this is often left empty but when cascading,
the recommendation is to add a meaningful description such as
“tape backup for …” or “disk mirroring for …”
Tags Tags are a new SANnav feature that allow you to assign a tag that
can be used for generating filtered reports. This is a database
search tag in SANnav. It is unrelated to the RNID tag. Creating a
tag is optional and can always be added at a later date. Tags in
mainframe environments are typically only used by very large
environments or when the same instance of SANnav is used to
manage both mainframe and open systems environments
logical switches for a single cascaded fabric at creation time, you will save some
configuration steps in the future.
b. Set the Domain IDs for each switch in the fabric. The domain ID (DID) is the highest
byte of the 2-byte link address. The recommended best practice is to make the DID
match Switch ID in HCD.
c. Add the ports. Most administrators will click on the “Slot/Port #” column to sort by port
number. The right hand side of this set up page, not illustrated below, allows you to
manually edit the port address (low byte of the 2-byte link address). By default, port
addresses are assigned sequentially so the recommendation is to select the ports in
the order you want the link addresses to be in. Typically, this is the same order the
slot/port is listed in.
118 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm
e. Configure the remaining fabric parameters. These parameters are applied to all logical
switches in the fabric.
f. Select “Fabric Parameters” to display parameter options.
g. Fill in the desired parameter options, as described in Table 6-5 on page 120, and click
“Save”.
Allow XISL Should only be selected if the ISLs are in a base switch
to be shared with this logical switches in the same
chassis.
120 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm
Option Discussion
Long distance fabric Checking this box adds buffer credits to the ISL ports
(E-Port) so it only makes sense to check this box in
cascaded environments. The number of buffer credits
depends on the speed of the link and fibre channel
generation. Adding buffer credits allows more frames to
be in flight which can have unintended effects on
congestion with short distance links. It also means there
are more frames to recover in the event of a network
failure. This box should only be checked when additional
buffer credits are needed.
For normal tape and DASD traffic, setting a long distance
mode is usually not necessary until the distance exceeds
1 Km so this is typically not set for cascading within a
campus environment. For distances exceeding 10 Km.,
other methods are used for providing additional buffer
credits so this is typically set for distances between 1 and
10 Km. The exceptions are channel to channel, CTC, or
extension through DWDM.
Block storage today always uses High Performance
FICON (zHPF) which means the Extended Count Key
Data (ECKD) frames are packaged with the data so the
average fibre channel frame size for FICON traffic
approaches the full 2K byte fibre channel frame size.
CTC traffic is typically used for transporting small
messages and therefore the average frame size is
typically very small. Since each fibre channel frame uses
a buffer credit, additional buffer credits are required for
CTC traffic.
Some active DWDM equipment supplies buffer credits so
there is no need to increase buffer credits. When
cascading via DWDM, it is important to discuss whether
the added buffer credits need to be configured on the
SAN switch or the DWDM interface with the DWDM
vendor.
Details of how and when to configure additional buffer
credits for extended distance links is beyond the scope of
this RedBook. Check with your IBM technical
representative before configuring any cascaded
environment.
b. Select the fabric. Remember that zoning is applied to a fabric, not an individual switch.
Since most FICON fabrics are single switch fabrics, this distinction is subtle; however,
keep in mind that you are looking for the fabric name, not the switch name, when
zoning. In this example, “FICOn-XRC-z15” was the name of the fabric created.
122 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm
d. Add the ports to the zone. Typically, all but the ISL ports are added to the zone.
g. Add the zone(s) to the zone configuration. In open systems environments, it’s common
to have multiple zone configurations so activating the zone configuration is optional.
124 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm
For most FICON environments, there is only one zone configuration which is usually
activated as soon as the zone configuration is created.
Both of these issues are scheduled to be addressed. It’s likely the corrections will be made in
SANnav before most enterprises deploy SANnav so this note is primarily to explain what is
missing in the screen captures.
Viewing connections
126 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm
Similarly, control units can be monitored by clicking on “Storage Ports” instead of “Host Ports”.
The easiest way to check is via the CLI. Save the session output so you can email it to your
Brocade business partner or direct to your Brocade representative in the event you need
assistance.
switchType 166.0
switchState Online
switchMode Native
switchRole 1
switchDomain Principal
switchId fffc01
switchWwn 10:00:c4:f5:7c:64:5b:62
zoning OFF
switchBeacon OFF
FC Router: OFF
HIF Mode ON
LS Attributes FID: 30, Base Switch: No, Default Switch: No, Ficon
Switch: No, Address Mode 1]
If HIF Mode is not ON or the Address Mode is not 1, you likely will need an outage.
75 9 11 0x0400 8 bit Y
76 9 12 0x0500 8 bit Y
A Y must appear in the column for all ports.
switchType 162.0
switchState Online
switchMode Native
switchRole Principal
switchDomain 1
switchId fffc01
switchWwn 10:00:c4:f5:7c:16:8b:3e
zoning OFF
switchBeacon OFF
FC Router: OFF
HIF Mode ON
LS Attributes FID: 30, Base Switch: No, Default Switch: No, Ficon
Switch: No, Address Mode 1]
Routing policy
This primarily effects load balancing in cascaded environments. Most FICON environments
today support FICON Dynamic Routing (FiDR).
The routing policy must be the same for all switches in a SAN but each SAN is independent
from one another and therefore each SAN may have different routing policies. Work Load
Manager will continue to operate as expected even though paths are connected to switches
with different routing methods. This means upgrades can include a routing method change
and each SAN upgraded in separate change control window.
A route is the set of switches and ports a path takes to deliver fibre channel frames from the
channel to the control unit. As a practical matter, this is only significant in cascaded
environments.
128 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm
Port Based Routing (PBR) Deprecated. Many older FICON SAN deployments are
configured this way.
Routes are determined by a predefined map for each possible
route in the SAN. The channel path will always take the same
route (same ISL or ISL trunk group).
Today, the only reason to configuration a switch for PBR is in the
event a new switch must be connected to an older switch still
configured for PBR. Setting PBR is not an option in SANnav but
can be done via the Command Line Interface (CLI). Details on
how to do this are beyond the scope of this RedBook. Contact
your technical account team in the event configuring PBR is
required.
Device Based Routing (DBR) Default. Most FICON SAN deployments are configured this way.
Routes are determined by usage. A round robin algorithm is used
to select the ISL or ISL trunk group. Routes are established
dynamically as a channel to control unit path is needed but once
that path is selected, the channel path will always take the same
route. This has a better statistical probability of balancing work
load across ISLs than PBR.
Exchange Based Routing (EBR). Analogous to FICON Dynamic Routing (FiDR). Although EBR is
the preferred routing method, DBR is the default because many
FICON SANs still have older control units that do not support
FiDR. Typically, these are older tape control units. Check with
your storage vendor to be sure FiDR is supported before
selecting EBR for the routing method. Some control units may
require a driver upgrade.
EBR selects routes dynamically on a per exchange basis. A fibre
channel “exchange” is analogous to a CCW chain. The least
used ISL or ISL trunk group for a route is selected at the time a
switch receives a new CCW chain from the channel. The route
remains intact for the life of the CCW chain. This method results
in the most efficient and dynamic use of ISL bandwidth.
Mainframe channel protocol was developed for buffered control units that managed physical
storage devices. Channel programs were built around channel command words (CCWs).
Write commands included data. CCWs could be, and usually are, strung together in a
channel program.
Status is a bit flag in a byte returned by the control unit. The only status bits pertinent to this
discussion are Channel End (CE) and Device End (DE). CE indicates that the control unit
accepted the channel program. Device End (DE) indicates that the operation with the physical
storage completed. So that a channel can perform other work while a control unit was working
on a channel program, status could be split in that CE was returned to the channel and DE
after the control unit performed the action with the physical storage.
In a serialized bit stream across fibre channel, CCWs are broken up into fibre channel
packets and the packets sent to their destination. The equivalent to a bus connection in fibre
channel is an exchange. All CCWs, and therefore all fibre channel packets, in a channel
program are sent on the same exchange so all packets arrive in order.
When split status is returned to the channel, DE is returned on a different exchange. The
protocol hasn’t changed but devices are now often in memory so the control unit can
complete the operation much more quickly. This means DE can be available almost
immediately after returning CE. Since DE is on a different fibre channel exchange, the fibre
channel packet with DE can be routed across a shorter or less congested ISL thereby arriving
before CE. Since it’s not possible for a device to complete an operation before the control unit
completed the operation, the channel program never had to account for receiving DE prior to
CE. If this occurred prior to FiDR, a channel detected error would be reported and the job may
terminate abnormally (ABEND).
FiDR recognizes this potential race condition and waits for CE instead of returning an error.
Both the channel and control unit must support FiDR.
6.4.1 Firmware
All supported FICON switch configurations today with any FICON supported version of FOS
are supported with switches defined as a FICON logical switch. This means that all upgrade
scenarios that include a mix of logical switches defined as FICON and FCP switches are
supported; however, all switches in a fabric cannot be more than one major FOS revision
apart. Any switches on FOS 7.x must be upgraded to 8.x before any other switch in the fabric
can be upgraded to 9.x.
As of this writing, the FOS v8.x code stream was still actively supported but eventually, it will
reach end of support prior to FOS 9.x and likely prior to a hardware upgrade of the Gen 6
switch. At some future date, Gen 6 switches will need to be upgraded to FOS 9.x for CVE
patches and other changes. Enterprises with Gen 6 switches operating on FOS 8.x firmware
that were not configured for a FICON logical switch should plan on upgrading the
configuration so that a FICON logical switch is used for all FICON connections. There is no
need to change the addressing so there is no need to re-design the channel paths. Use
Table 6-7 to determine if an outage will be required to configure a FICON logical switch.
130 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm
Logical switch Non-default VF must be enabled and a logical switch other than the
logical switch default logical switch must be properly configured for
FICON.
Address mode Zero based Zero based address mode is the addressing mode that
(mode 1) ensures the fibre channel address assigned to a port is in
compliance with the channel path link addressing. It is not
always set today because certain configurations were
inherently in compliance so you will need to check.
High Integrity Must be enabled When HIF is set, this ensures the security policy is set and
Fabric (HIF) distributed throughout the fabric, insistent domain ID is set.
All properly configured FICON SAN fabrics should have
HIF enabled today.
Port Address Must be in effect All properly configured FICON SAN fabrics should have
Binding port address binding in effect today.
6.4.4 Cabling
Although single-mode cable replacement is generally not necessary in mainframe
environments, it’s not uncommon to have to replace a small percentage cables. The most
common reason for this is that during cable cleaning a lens is scratched. In some cases,
degraded cables may not exhibit problems until higher speeds are attempted. Although
multi-mode cable is rare in mainframe environments, there are some exceptions. The most
common exception for using multi-mode cable is for DWDM connections. All multi-mode cable
should be at least OM-4 rated.
As with any repurposing of cabling, good cable hygiene is recommended which should
include cleaning all connection points. Even when performing a push-pull upgrade of a switch,
connector cleaning should include the connections at the patch panel and the channel or
control unit in addition to the connection at the switch port.
6.4.6 SFPs
When considering re-purposing SFPs, remember that Gen 7 switches and switch blades
require SFPs designed with the secure optics feature. This means that although Gen 7 can
accept a 32G optic, you cannot repurpose 32G optics from Gen 6 switches in a Gen 7 switch
or switch blade.
As with all previous fibre channel switching products from Brocade, the latest generation
product supports SFPs of the same generation and one generation below. A Gen 7 switch
port is capable of 64G and therefore can accept a 64G SFP or a 32G SFP. A Gen 6 switch
port is capable of 32G and therefore can accept a 32G or 16G SFP. Each SFP supports 3
speeds, the current generation, one generation back, and two generations back.
A 64G optic can negotiate to 64G, 32G, and 16G.
A 32G optic can negotiate to 32G, 16G, and 8G.
A 16G optic can negotiate to 16G, 8G, and 4G.
If a 4G connection is required, typically for older tape or DWDM, you will need a 16G SFP and
since 16G SFPs are not supported on Gen 7 products, this means you will need either a Gen
6 switch or Gen 6 switch blade to accommodate older connections that require 4G. You can
use the Brocade branded 16G SFPs from your Gen 5 products in a Gen 6 switch or switch
blade but you should only do so if 4G support is required.
Most FICON SANs are single switch (single domain) fabrics. Channel paths can be striped
across SANs using a mix of Gen 5, Gen 6, and Gen 7 technology as long as they are
configured properly for FICON. As long as they are not in the same fabric, they can be on any
version of FOS supported for FICON, can have different addressing modes, and can have
different routing methods. This means switches in a channel path can be upgraded without
the need to be concerned with redundant channel paths.
In cascaded environments, a non-VF enabled switch can be connected to a logical switch that
has VF enabled as long as every switch in the fabric that has VF enabled has the same Fabric
ID (FID). To minimize outages, most customers will upgrade both switches in the channel path
but should the need arise, mixing switch generations in the same fabric is supported but with
the restrictions noted below. Keep in mind that a SAN42B-R (2498-R42) extension switch is a
Gen 5 switch.
Configuration requirements
The FOS code must be at v9.0 or higher in the Gen 7 switch and a supported version of
v8.x or higher in the Gen 5 or Gen 6 switch.
All the fabric parameters must match
132 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm
For FICON environments, the routing policy is the most likely parameter that will be
different since most Gen 5 implementations used either port based routing or device
based routing and new installs are likely to use exchange based routing (FiDR).
It’s unlikely that other fabric parameters will be different; however, should the need to mix
switch generations in the same fabric arise, contact your technical account team to
validate all fabric settings.
The Gen 7 SAN64B-7 (8960-P64/R64) has an additional 8 ports but otherwise has the same
physical format as the Gen 6 SAN64B-6.
Physical upgrade
Assuming slower login speeds are not an issue, a simple push-pull can be done for physical
installation when upgrading from a SAN48B-5 to a SAN64B-7.
It is not uncommon for enterprises to make CHPID path changes as part of an upgrade but
there are no changes in the SAN64B-7 that require changes to the IODF.
To take advantage of some space available on the front panel and to make full use of all ports
on the ASIC, the Gen 6 switch also has 4 QSFP ports. The QSFP ports were only qualified as
ISL ports when used in FICON environments. They could not be used with breakout cables
for FICON connectivity. They could have been used with breakout cables in a logical switch
for non-FICON connectivity.
The QSFP ports on the SAN64B-6 were replaced with dual capable SFP ports so the only
change will be for environments that used the QSFPs for inter-switch links (ISLs) or for
non-FICON applications.
Few, if any, enterprises fully utilized all QSFP. In the event that more than 56 ports were used
with the SAN64B-6, an additional switch will be required. The additional 8 ports on the
SAN64B-7 are designed to accept a dual SFP but as of this publication date, only standards
SFPs were available.
Since using QSFPs is uncommon for fibre channel, the equivalent Gen 7 switch was
designed to use the space previously used for QSFP for SFP ports that support double
density SFPs. As of this writing, double density SFPs were not available but it is anticipated
that the double density SFPs will only be supported for ISLs, not FICON connections. Since
those ports support standard SFPs as well, with three 8 port licenses the SAN64B-7 can be
configured as a 56 port switch.
Physical upgrade
For environments that used the QSFPs as ISLs with a total port count of 56 or less, the ISL
connections have to be moved off of the QSFPs to the SFP ports. Replacing the SAN64B-6
therefore may involve some re-cabling but otherwise, a simple push-pull is possible.
It is not uncommon for enterprises to make CHPID path changes as part of an upgrade but
since the SAN64B-7 supports more FICON ports than the SAN64B-6, there are no required
changes to the IODF when upgrading Gen 6 switches to Gen 7 technology.
The Gen 6 directors, SAN256B-6 and SAN512B-6, have the same physical card layout and
form factor as the Gen 7 directors, SAN256B-7 and SAN512B-7.
Physical upgrade
In most cases, a simple push-pull of the Gen 6 director for a Gen 7 director is possible.
Exceptions are when 4G connections must be supported because these connections require
a 16G SFP which is not available for Gen 7. The options to accommodate older 4G
connections is to move them to a fixed port switch or add a Gen 6 blade to the Gen 7 chassis.
Gen 6 blades are available for purchase or can be migrated from an existing Gen 6 director to
a Gen 7 director.
It is not uncommon for enterprises to make CHPID path changes as part of an upgrade but
there are no changes in the SAN256B-7 or SAN512B-7 that require changes to the IODF.
Most, if not all, FICON implementations on Gen 5 directors, SAN768B-2 and SAN384B-2,
were with 32 port blades. SAN upgrades typically skip a generation so Gen 5 to Gen 7 is the
most common upgrade path to Gen 7. Since redundant channel paths are usually planned to
be on different FRUs (port cards), the same channel path mapping used with 32 port blades
likely will not work on 48 port cards. This is the primary consideration when upgrading from
Gen 5 to Gen 7 directors.
The Control Processor (CP) blades are now half height blades occupying the first physical
slot in the chassis. Reducing the total number of physical slots from 12 to 11 was done to
allow a little extra air flow to improve cooling for the port cards. This was necessary because
the higher speed optics produce more heat.
134 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch06.fm
Physical upgrade
The control processor (CP) cards in the 8510-4 and 8510-8 chassis occupied full slots in the
middle of the chassis. With the Gen 6 and Gen 7 chassis, the CP blades are half height
blades, both installed in the first chassis slot. The slot numbering therefore is different.
To distinguish the physical location from one CP to the other, they are referred to as slots 1 &
2 even though they both occupy the same physical space as a single slot would occupy.
Since most Gen 5 directors deployed for FICON used the 32 port card, it’s likely that several
cabling changes will need to be made. The specific cabling changes will depend on the
design approach taken. For patch room designs that were planned to match the symmetry of
the blades and channel paths, changes, at least in documentation, may be required for the
patch room as well.
In most deployments, significant changes to the IODF will be required. Much of the design
section is dedicated to resolving channel pathing that arise as a result from architectures
build around 32 port cards to 48 port cards.
IBM resources
SANnav Introduction:
https://www.ibm.com/support/pages/introducing-ibm-sannav
SANnav Announcement:
https://www-01.ibm.com/common/ssi/ShowDoc.wss?docURL=/common/ssi/rep_ca/1/897/E
NUS120-061/index.html&request_locale=en
SANnav Data Sheet:
https://www.ibm.com/downloads/cas/4LX0PXRL
136 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
An essential aspect of the Extension scenario is the IP link quality. With Extension, dedicated
bandwidth is prudent if the link is shared with other traffic. Because the connection between
two sites is low-traffic or used only for e-mail, do not assume that this is always the case.
Storage is sensitive to latency and congestion; a spyware problem or a DDOS attack on the
IP network can be disruptive. Furthermore, data inflight should always be encrypted, which is
easily done on IBM b-type Gen 7 Extension platforms.
Also, when you are communicating with your organization’s networking architects, distinguish
between megabytes per second (MB/s) and megabits per second (Mbps). In the storage
world, bandwidth is often specified in MB/s, but network engineers specify bandwidth in
Mbps. If you want 1 GB/s and you fail to designate GB explicitly, you may end up with 1 Gbps,
which supplies approximately 100 MB/s. If you include safety margins, this link is not as fast
as you may need, so ensure the correct terminology.
This chapter describes the IBM b-type Gen 7 (Brocade) Extension family of products. It
includes an overview of the physical platforms, the different sides of an Extension platform,
features, terminology, implementation, technical insights, and architectures.
138 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
Moving FC data over an IP network is referred to as FCIP. IP storage data can be transported
by IP Extension, which benefits from the advanced transport technology and encryption.
IBM SAN18B-6
The IBM SAN18B-6 is a 1RU chassis of standard width, as shown in Figure 7-1. The front has
access to the serial port, Management Ethernet port, USB port, twelve 32G FC ports (Gen 6),
and six application Ethernet interfaces (GE0/GE1 and GE2/GE3 are mutually exclusive).
The IBM SAN18B-6 has two license modes, base and full. The full unit has a WAN side
maximum of 2.5 Gbps. The base unit has a WAN side maximum of 1 Gbps and is enforced by
disallowing circuits to have more than a cumulative maximum rate of 1 Gbps. There is one DP
(data processor).
The IBM SAN18B-6 combines the power supply and fans into a single hot-swappable FRU, as
shown in Figure 7-2. There are two redundant FRUs located in the rear of the chassis. An
offline power supply does not prevent power to any fan; a single online power supply can
power all fans. Airflow is unidirectional rear to front.
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 139
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
IBM SAN42B-R
The IBM SAN42B-R is a 2RU chassis of standard width, as shown in Figure 7-3below. The
front has access to the serial port, management Ethernet port, USB port, twenty-four 16G
FC/FICON ports (Gen 5), sixteen application GE/10GE Ethernet interfaces, and two 40GE
interfaces.
The IBM SAN42B-R has three license modes: 5 Gbps (base unit), 10 Gbps (Upgrade-1), and
40 Gbps. Upgrade-2 requires Upgrade-1, and Upgrade-2 is necessary to enable the 40GE
interfaces.
There are two DP (data processors), each capable of 20 Gbps. 40 Gbps is the maximum
WAN side bandwidth. 5 and 10 Gbps modes are limited by disallowing cumulative circuit
bandwidth above the licensed limit. The full 40 Gbps licensing has no restriction on the
cumulative circuit rate; nonetheless, 40 Gbps is the maximum hardware capacity.
Note: Additional optional hardware is not available for the empty slot.
The IBM SAN42B-R has a separate power supply and fan FRUs, as shown in Figure 7-4.
There are two redundant power supplies and three fan trays located in the rear of the chassis.
An offline power supply does not prevent power to any fan; any online power supply can
power all fans. Airflow is unidirectional rear to front.
140 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
The front has access to sixteen FC/FICON ports (Gen 6 - supported in Gen 7 Directors),
sixteen application GE/10GE Ethernet interfaces, and two 40GE interfaces. The Ethernet
SFP/SFP+ optics are not included and must be ordered as needed. Brocade branded optics
are required to come online.
There are two DP (data processors), each capable of 20 Gbps. 40 Gbps is the maximum
WAN side bandwidth. The SX6 Extension Blades have no restriction on cumulative circuit
rate; nonetheless, 40 Gbps is the maximum hardware capacity.
The sections below cover the most common Extension designs and architectures.
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 141
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
IBM Extension is high-speed and adds about 75 µs of propagation delay per pass through the
platform. Four passes round trip = 0.3 ms total added propagation delay, added to the
network latency. In some cases, the added propagation delay may be acceptable for
synchronous RDR applications. Deployment best practice connects array N_Ports to
Extension F_Ports directly and does not connect through the production fabric. Storage
replication ports are dedicated to RDR and should not pass through fabrics with host traffic,
which can cause head-of-line-blocking issues for hosts. Nevertheless, there remain valid
reasons to connect via the production fabric, such as tape applications and too many arrays
ports for the Extension platforms. If additional ports are required and connecting to the
production fabric is necessary, it is prudent to separate these ports into a logical replication
fabric using Virtual Fabrics.
An architecture best practice uses a separate autonomous replication fabric and does not
combine the replication fabric with the production fabric.
Two-box architecture
A single Extension platform can be directly connected to different storage controllers and is
referred to as a Two-Box solution, one at each DC, as shown in Figure 7-6.
A IP WAN
Extension Tunnel A
Fibre Channel Extension Tunnel
Fibre Channel
Circuit
B Circuit
Brocade Brocade B
DC LAN DC LAN
Storage Extension Extension
WAN Router WAN Router Storage
A two-box solution would be common with the IBM SAN18B-6 base unit, which supports one
circuit per VE_Port. It does support two VE_Ports (tunnels), and those tunnels can run in
parallel. However, this is not the same as two circuits that are part of a single tunnel because
there is no coordination or data recovery between different VE_Ports. Not all storage
applications support parallel FCIP connections. When using parallel Extension tunnels with
EBR (Exchange-Based Routing), exchanges may be delivered out of order. Device-Based
Routing or Port-Based Routing (flow-based) could be used to avoid issues with out of order
Exchanges.
Four-box architecture
Alternatively, an Extension platform is dedicated to fabric and controller “A.” A physically
different Extension platform is dedicated to fabric and controller “B.” Two Extension platforms
in each data center is referred to as a Four-Box solution; refer to Figure 7-7 on page 143
below. A single WAN link for both tunnels, all circuits, may be used. Alternatively, two WAN
links from different service providers may be used, with the circuits symmetrically spread
across the WAN connections. The number of WAN connections depends on the availability
requirements, the cost, and recovery tolerance. With two WAN connections, Cir0 from each
tunnel would be pinned to WAN-A, and Cir1 from each tunnel would be pinned to WAN-B.
142 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
BET BET
A A
IBM IBM
Extension Circuits Circuits Extension
B B
IP WAN IP WAN
Router Router
FC FC
BET BET
Fabric A Fabric A
Circuits Circuits
Fabric B Fabric B
FC
FC
For maximum availability, it is best practice to divide a SAN into “A” and “B” fabrics, which
implies there is an “air gap” between the two autonomous fabrics from the server to the
storage. Intentionally, there are no physical links between the two fabrics. Servers, storage,
and VMs are equipped with drivers that monitor pathways and send data accordingly. When a
path is detected as down, the driver fails traffic over to a remaining path.
Connecting Extension via FCR to both production edge fabrics, as shown in Figure 7-9 on
page 144, is not recommended and IBM does not recommend this architecture. Without FCR,
the fabrics would merge into one big fabric, which destroys any notion of redundant
autonomous fabrics. Nonetheless, if FCR is used, the fabrics don’t merge; however, there still
is a common Linux kernel on a device attached to both fabrics. If maximum availability is the
goal in a critical environment, this architecture is not acceptable and considered high risk. An
architecture with a common platform to both fabrics is susceptible to an operating system
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 143
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
defect, driver defects, unexpected hardware failures, environmental conditions, and human
error, which can bring down the entire SAN.
Figure 7-9 Common extension to production edge fabrics - not recommended architecture
When connecting Extension to production fabrics, each production fabric should be designed
using best practice core-edge concepts. Since storage attaches directly to the core in a
core-edge design, Extension switches connect directly to the core, or an Extension blade is
placed in the Core Directors. A standalone Extension platform should be connected to a
fabric with at least two ISLs (Inter-Switch Links) for the port, optic, and cable redundancy.
A consideration in deploying IP Extension with IBM TS7700 Grid is how to direct grid traffic to
the IP Extension gateway; there are two methods. The first method uses the default gateway
on the EN (Ethernet Network) interfaces to send traffic directly to the IP Extension gateway.
The second method sends traffic to the traditional gateway and routes on the gateway router
forwards traffic to the IP Extension gateway.
Four subnets should be used, one for each EN# interface on the TS7700 system. EN
(Ethernet Network) interfaces of the same number must be able to communicate with each
other. EN interfaces of different numbers do not communicate with each other; therefore,
each subnet below should be in its own VLAN and have its own IP Extension gateway.
Example configuration on EN# interfaces:
EN0
– IP addresses:10.10.0.100-199
– Gateway (IPEX):10.10.0.254
– Subnet Mask:255.255.255.0 (/24)
EN1
– IP addresses:10.10.1.100-199
144 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
– Gateway (IPEX):10.10.1.254
– Subnet Mask:255.255.255.0 (/24)
The second DP may be kept in reserve for eHCL moving circuits into or consumed by different
applications such as RDR.
It is imperative to understand TCL rules only live on the DP unto which the target VE_Port
exists. VE_Ports exist only in one DP, either DP0 or DP1. Refer to the TCL section for IP
Extension gateways are created on a specific DP, either DP0 or DP1. Ingress traffic matching
a TCL rule is directed to a particular target (VE_Port). If the VE_Port target designated in a
TCL rule is not congruent with the IP Extension gateway’s DP, the rule is not evaluated. In
other words, both the IP Extension gateway and target VE_Port used must be on the same
DP for the rule to be evaluated. If the rule is expected to be assessed but is not, check if traffic
is hitting an IP Extension gateway configured on the other DP.
Method 1
The IBM TS7700 Grid EN (Ethernet Network) interfaces are configured with their default
gateway set to the IP Extension gateway, as shown in Figure 7-10 on page 146. The IP
Extension gateway becomes the default gateway for the EN interfaces. IBM TS7700 Grid EN
interfaces do not support independent routing tables; each interface cannot specify an IP
route for destination subnet specific gateways. The IP Extension gateway is required for
transporting grid traffic to the remote data center via IP Extension. The remote grid subnet is
the only destination outside of the local grid subnet. In Method 1, the EN interfaces cannot be
configured with the traditional router gateway.
A LAN switch that switches traffic without a gateway router requires EN interfaces to be on the
same subnet and VLAN; this is Layer-2 switching (L2 switching). The Extension platforms are
not Ethernet switches and cannot facilitate communication between different clusters; the
data center LAN switch must do this. Communications to other grid nodes or clusters must be
done at layer 2 (the Ethernet layer) through their common subnet, thereby not involving a
gateway router. Traffic on the same subnet is sent directly to the other device via Ethernet.
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 145
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
As long as all the local grid EN<#> interfaces live on the same subnet and VLAN, they can
communicate via L2 switching, as shown in Figure 7-11. Grid traffic destined for a remote
data center via IP Extension is sent to the IP Extension gateway using the gateway configured
on the EN<#> interfaces.
Figure 7-11 IBM TS7700 Grid IP Extension Architecture - EN Interfaces with IP Extension Gateway
Method 2:
In method 2, the IBM TS7700 EN interfaces are configured to use the traditional router as the
default gateway. Grid traffic destined for various subnets is sent to a conventional gateway
router. The router forwards the traffic towards the destination, which may be another
node/cluster within the local data center or a remote cluster via the IP Extension gateway, as
shown in Figure 7-12 on page 147.
146 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
DC LAN Switch
10.10.10.0/24
GW: 10.10.10.1
Grid
Cluster A
.1 Router
IPEX Destination
Traditional Gateway IP Extension
Tunnel WAN 172.16.10.0/24
IP route 172.16.10.0/24 via 192.168.10.2 IP Extension GW subnet .2
.1 .1
(192.168.10.0/30) iproute lan.dp# 10.10.10.0/24 192.168.10.1
iproute lan.dp# 10.10.20.0/24 192.168.10.1
Grid
Cluster B 10.10.20.0/24
GW: 10.10.20.1
DC LAN Switch
Figure 7-12 Local and remote grid connectivity via a traditional gateway
IBM TS7700 Grid traffic is already compressed; therefore, compression is disabled. With no
compression, the ingress and egress data rates are equal at 20 Gbps for IP Extension.
Encryption is enabled and causes no degradation in throughput or performance.
A different subnet is used for each EN (Ethernet Network) interface. Each cluster has four EN
interfaces (EN0, EN1, EN2, and EN3). DC-A and DC-B are metro data centers connected by
DWDM at L2, the Ethernet layer. The respective EN interfaces from clusters (4, 6, 5, and 7)
are on the same subnets across these two data centers. All EN0 interfaces can communicate
without routing (L3), same with EN1, EN2, and EN3. In DC-A and DC-B, all like numbered EN
interfaces can talk to each other without Extension. DC-C is further away and requires a
routed L3 IP network and Extension.
Figure 7-13 on page 148 shows the schema for addressing the EN interfaces, creating
gateways, and connecting the various DPs. Keep in mind the gateways and VE_Ports are
associated with one specific DP. EN interface IP addresses have been selected to readily
designate the data center (3rd byte), EN interface (3rd byte), and cluster (4th byte) within and
across desired broadcast (L2) domains. The subnets are /24 (255.255.255.0). The layout
depends on the number of DPs in each data center and how best to disperse EN interfaces
across the DPs to gain bandwidth and high availability.
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 147
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
On each switch in DC-C (sw0, sw1, sw2, and sw3), there are two IP Extension gateways on
each subnet, one for each DP (.250 and .251). There are four IP Extension gateways on each
subnet in DC-A and DC-B, one for each DP in each data center (.250, .251, .252, and .253).
You do not want to traverse the DWDM network to reach an IP Extension gateway. In DC-A
and DC-B, EN0 and EN2 use DP0, and EN1 and EN3 use DP1. In DC-A and DC-B, EN0 and
EN1 use different switches, as well; EN2 and EN3 use separate switches, for example,
DCA-sw0 and DCA-sw1. IN DC-C, because there are four platforms, each EN interface uses
a different switch. In DC-C, cluster 1 uses DP0, and cluster 3 uses DP1.
Min-Rate Max-Rate
CL/EN Source IP Destination IP IPEX GW DP Target SW SW Target DP IPEX GW Source IP Destination IP CL/EN # of Cirs Each (Gbps) Each (Gbps)
CL4 CL1
0 10.0.0.40 10.0.10.10 10.0.0.250/24-dp0 DP0 12/16 DCA-s w0 DCC-s w0 12/16 DP0 10.0.10.250/24-dp0 10.0.10.10 10.0.0.40 0 2 5 5
1 10.0.1.40 10.0.11.10 10.0.1.250/24-dp0 DP0 12/16 DCA-s w1 DCC-s w1 12/16 DP0 10.0.11.250/24-dp0 10.0.11.10 10.0.1.40 1 2 5 5
2 10.0.2.40 10.0.12.10 10.0.2.251/24-dp1 DP1 12/26 DCA-s w0 DCC-s w2 12/16 DP0 10.0.12.250/24-dp0 10.0.12.10 10.0.2.40 2 2 5 5
3 10.0.3.40 10.0.13.10 10.0.3.251/24-dp1 DP1 12/26 DCA-s w1 DCC-s w3 12/16 DP0 10.0.13.250/24-dp0 10.0.13.10 10.0.3.40 3 2 5 5
CL5 CL1
0 10.0.0.50 10.0.10.10 10.0.0.252/24-dp0 DP0 12/16 DCB-s w0 DCC-s w0 12/17 DP0 10.0.10.250/24-dp0 10.0.10.10 10.0.0.50 0 2 5 5
1 10.0.1.50 10.0.11.10 10.0.1.252/24-dp0 DP0 12/16 DCB-s w1 DCC-s w1 12/17 DP0 10.0.11.250/24-dp0 10.0.11.10 10.0.1.50 1 2 5 5
2 10.0.2.50 10.0.12.10 10.0.2.253/24-dp1 DP1 12/26 DCB-s w0 DCC-s w2 12/17 DP0 10.0.12.250/24-dp0 10.0.12.10 10.0.2.50 2 2 5 5
3 10.0.3.50 10.0.13.10 10.0.3.253/24-dp1 DP1 12/26 DCB-s w1 DCC-s w3 12/17 DP0 10.0.13.250/24-dp0 10.0.13.10 10.0.3.50 3 2 5 5
CL6 CL1
0 10.0.0.60 10.0.10.10 10.0.0.250/24-dp0 DP0 12/16 DCA-s w0 DCC-s w0 12/16 DP0 10.0.10.250/24-dp0 10.0.10.10 10.0.0.60 0 2 5 5
1 10.0.1.60 10.0.11.10 10.0.1.250/24-dp0 DP0 12/16 DCA-s w1 DCC-s w1 12/16 DP0 10.0.11.250/24-dp0 10.0.11.10 10.0.1.60 1 2 5 5
2 10.0.2.60 10.0.12.10 10.0.2.251/24-dp1 DP1 12/26 DCA-s w0 DCC-s w2 12/16 DP0 10.0.12.250/24-dp0 10.0.12.10 10.0.2.60 2 2 5 5
3 10.0.3.60 10.0.13.10 10.0.3.251/24-dp1 DP1 12/26 DCA-s w1 DCC-s w3 12/16 DP0 10.0.13.250/24-dp0 10.0.13.10 10.0.3.60 3 2 5 5
CL7 CL1
0 10.0.0.70 10.0.10.10 10.0.0.252/24-dp0 DP0 12/16 DCB-s w0 DCC-s w0 12/17 DP0 10.0.10.250/24-dp0 10.0.10.10 10.0.0.70 0 2 5 5
1 10.0.1.70 10.0.11.10 10.0.1.252/24-dp0 DP0 12/16 DCB-s w1 DCC-s w1 12/17 DP0 10.0.11.250/24-dp0 10.0.11.10 10.0.1.70 1 2 5 5
2 10.0.2.70 10.0.12.10 10.0.2.253/24-dp1 DP1 12/26 DCB-s w0 DCC-s w2 12/17 DP0 10.0.12.250/24-dp0 10.0.12.10 10.0.2.70 2 2 5 5
3 10.0.3.70 10.0.13.10 10.0.3.253/24-dp1 DP1 12/26 DCB-s w1 DCC-s w3 12/17 DP0 10.0.13.250/24-dp0 10.0.13.10 10.0.3.70 3 2 5 5
CL4 CL3
0 10.0.0.40 10.0.10.30 10.0.0.250/24-dp0 DP0 12/17 DCA-s w0 DCC-s w0 12/26 DP1 10.0.10.251/24-dp1 10.0.10.30 10.0.0.40 0 2 5 5
1 10.0.1.40 10.0.11.30 10.0.1.250/24-dp0 DP0 12/17 DCA-s w1 DCC-s w1 12/26 DP1 10.0.11.251/24-dp1 10.0.11.30 10.0.1.40 1 2 5 5
2 10.0.2.40 10.0.12.30 10.0.2.251/24-dp1 DP1 12/27 DCA-s w0 DCC-s w2 12/26 DP1 10.0.12.251/24-dp1 10.0.12.30 10.0.2.40 2 2 5 5
3 10.0.3.40 10.0.13.30 10.0.3.251/24-dp1 DP1 12/27 DCA-s w1 DCC-s w3 12/26 DP1 10.0.13.251/24-dp1 10.0.13.30 10.0.3.40 3 2 5 5
CL5 CL3
0 10.0.0.50 10.0.10.30 10.0.0.252/24-dp0 DP0 12/17 DCB-s w0 DCC-s w0 12/27 DP1 10.0.10.251/24-dp1 10.0.10.30 10.0.0.50 0 2 5 5
1 10.0.1.50 10.0.11.30 10.0.1.252/24-dp0 DP0 12/17 DCB-s w1 DCC-s w1 12/27 DP1 10.0.11.251/24-dp1 10.0.11.30 10.0.1.50 1 2 5 5
2 10.0.2.50 10.0.12.30 10.0.2.253/24-dp1 DP1 12/27 DCB-s w0 DCC-s w2 12/27 DP1 10.0.12.251/24-dp1 10.0.12.30 10.0.2.50 2 2 5 5
3 10.0.3.50 10.0.13.30 10.0.3.253/24-dp1 DP1 12/27 DCB-s w1 DCC-s w3 12/27 DP1 10.0.13.251/24-dp1 10.0.13.30 10.0.3.50 3 2 5 5
CL6 CL3
0 10.0.0.60 10.0.10.30 10.0.0.250/24-dp0 DP0 12/17 DCA-s w0 DCC-s w0 12/26 DP1 10.0.10.251/24-dp1 10.0.10.30 10.0.0.60 0 2 5 5
1 10.0.1.60 10.0.11.30 10.0.1.250/24-dp0 DP0 12/17 DCA-s w1 DCC-s w1 12/26 DP1 10.0.11.251/24-dp1 10.0.11.30 10.0.1.60 1 2 5 5
2 10.0.2.60 10.0.12.30 10.0.2.251/24-dp1 DP1 12/27 DCA-s w0 DCC-s w2 12/26 DP1 10.0.12.251/24-dp1 10.0.12.30 10.0.2.60 2 2 5 5
3 10.0.3.60 10.0.13.30 10.0.3.251/24-dp1 DP1 12/27 DCA-s w1 DCC-s w3 12/26 DP1 10.0.13.251/24-dp1 10.0.13.30 10.0.3.60 3 2 5 5
CL7 CL3
0 10.0.0.70 10.0.10.30 10.0.0.252/24-dp0 DP0 12/17 DCB-s w0 DCC-s w0 12/27 DP1 10.0.10.251/24-dp1 10.0.10.30 10.0.0.70 0 2 5 5
1 10.0.1.70 10.0.11.30 10.0.1.252/24-dp0 DP0 12/17 DCB-s w1 DCC-s w1 12/27 DP1 10.0.11.251/24-dp1 10.0.11.30 10.0.1.70 1 2 5 5
2 10.0.2.70 10.0.12.30 10.0.2.253/24-dp1 DP1 12/27 DCB-s w0 DCC-s w2 12/27 DP1 10.0.12.251/24-dp1 10.0.12.30 10.0.2.70 2 2 5 5
3 10.0.3.70 10.0.13.30 10.0.3.253/24-dp1 DP1 12/27 DCB-s w1 DCC-s w3 12/27 DP1 10.0.13.251/24-dp1 10.0.13.30 10.0.3.70 3 2 5 5
There are two tunnels per DP. Each tunnel has two circuits, one circuit for each data center
LAN switch. If one of the data center LAN switches goes offline, the remaining circuit
continues to operate without disrupting the Grid. As you can see, there are four circuits per
DP. The maximum capacity of a DP is about 20 Gbps; therefore, each circuit should be
configured at 5 Gbps, which is 10 Gbps per tunnel. Both the --min-comm-rate and the
--max-comm-rate should be set equal at 5 Gbps, called CIR (committed information rate). When
min and max are different, it is referred to as ARL. In a Grid environment with well-balanced
traffic across all paths, disabling ARL and using CIR can improve performance. Moreover, if
metric 1 backup circuits and failover groups are being used, the best practice is CIR over
ARL.
Figure 7-14 on page 149 shows a high-level connectivity diagram with ports.
148 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
It helps make a diagram of how the tunnels connect from end to end - IP Extension gateway
to IP Extension gateway; see Figure 7-15. The Extension tunnels between DC-A and DC-B to
DC-C complete a full mesh architecture in which every EN interface of the same number can
communicate. Remember, Clusters 4, 6, 5, and 7 can communicate through Layer2 since
their EN interfaces are on the same subnet and connected by DWDM. The figure below
shows tunnel connectivity. Notice the VE_Port range for DP0 and DP1 are different. On the
SX6 Blade, DP0 is VE16 to VE20, and DP1 is VE26 to VE30.
The same gateway can direct traffic into multiple tunnels via the TCL by matching the
source/destination IP addresses. For example, on the Extension platform connected to
Cluster1-EN0, if the rule matches src:10.0.10.10 and dst:10.0.0.40 and designates target 16,
that specific traffic uses tunnel VE16 to Cluster4-EN0. A mirrored TCL must be configured on
the Extension platform connected to Cluster4 to return traffic.
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 149
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
Multiple gateways can direct traffic into the same tunnel via the TCL. For example, an
Extension platform has the gateway 10.0.0.250 for EN0 and directs traffic into VE16. A
different gateway for EN2, 10.0.2.250, can direct traffic into the same tunnel, VE16. If
Upgrade1 + Upgrade2 licenses have been installed, it is possible to create parallel tunnels for
each EN interface; however, this assumes enough VE_Ports are available. The DP maximum
capacity is no more than 20 Gbps on the WAN side, no matter how much circuit bandwidth is
configured. With WAN Upgrade 1+2 licenses, there is no bandwidth restriction on creating
circuits. The best practice is to use the same tunnels for various gateways when a tunnel is
between the same domains and DPs.
DC-C dedicates a DP to each EN interface. DC-C has four switches (sw0, sw1, sw2, sw3)
because it consolidates incoming connections from two remote data centers. In DC-A and
DC-B, a DP is dedicated to EN0 and EN2, and a DP is dedicated to EN1 and EN3. There is a
balance between the two metro sites going into the one DR site. Adding SX6 Blades or
SAN42B-R boxes increases the number of DPs available to EN interfaces, for example, if
more bandwidth was needed. One DP can accommodate 20 Gbps, which is twice the
bandwidth of a single EN interface. Of course, more than one DP per EN interface would not
be beneficial.
The TCL cannot match IP Extension traffic and direct it into more than one target (tunnel). If
two VE_Ports were used, the traffic across each would require a rule match from different
TCL rules.
As shown in Figure 7-16, two IP Extension fabrics are shown. Each fabric has one tunnel with
one circuit. The maximum bandwidth supported by the IBM SAN18B-6 base unit is 1 Gbps on
the WAN side. Two WAN connections are shown; however, a single WAN connection would
work as well. The GE interfaces can be either GE or 10GE. Two different IP Extension
gateways are required, and the end devices must be capable of configuring the different
gateways per interface or per set of interfaces.
cir0
IP Storage
IP Storage
cir0
150 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
cir0 cir0
IP Storage
IP Storage
LAN Side LAN Side
GE/ 10GE GW: lan.dp0 GE/ 10GE
IP WAN IP WAN
DC LAN IBM Service IBM DC LAN
Switch IP Extension Provider(s) IP Extension Switch
cir1 cir1
WAN WAN
Router Router
Two circuits (red and blue) make up a single tunnel between Extension platforms. Each circuit
takes a different path through the IP network.
cir0 cir0
IP Storage
IP Storage
lan.dp0 GW0 lan.dp0 GW0
On the WAN side, there are two circuits (red and blue) comprising one tunnel between the
Extension platforms. The circuits take different paths for redundancy.
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 151
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
WAN WAN
Router Router
IP Storage
IP Storage
LAN Side lan.dp0 IP WAN IP WAN lan.dp0 LAN Side
vPC
vPC
GE/10GE GW0 GW0 GE/10GE
IBM Service IBM
IP Extension Provider(s) IP Extension
cir1 cir1
WAN WAN
Router Router
The Extension platforms are LAG connected to a corresponding data center LAN switch. The
links can be tagged or untagged and must be consistent on both ends.
There is a single tunnel between the top two Extension platforms (red tunnel) and one
between the bottom two (blue tunnel). Each tunnel has two circuits (cir0 and cir1), each taking
a different path.
cir0 cir0
IP Storage
IP Storage
10Gb/s
LAN Side IP WAN IP WAN LAN Side
10GE 10GE
GE/ 10GE GE/ 10GE
Service
lan.dp0 GW1 cir1
Provider(s) lan.dp0 GW1
cir1
cir0 cir0
Ethernet IBM WAN WAN IBM Ethernet
Switch IP Extension Router Router IP Extension Switch
The Extension platforms are LAG connected across the two virtualized data center LAN
switches. The links can be tagged or untagged and must be consistent on both ends.
152 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
There is a single tunnel between the top two Extension platforms (red tunnel) and a tunnel
between the bottom two Extension platforms (blue tunnel). Each tunnel has two circuits (cir0
and cir1), each taking a different path.
10GE 10GE
cir0 cir0
cir1 cir1
lan.dp0 GW0 lan.dp0 GW0
IP Storage
IP Storage
Logically 1 Switch
Logically 1 Switch
LAN Side IP WAN IP WAN LAN Side
vPC
vPC
GE/ 10GE GE/ 10GE
Service
lan.dp0 GW1 lan.dp0 GW1
Provider(s) cir1
cir1
cir0 cir0
10GE 10GE
DC LAN IBM WAN WAN IBM DC LAN
Switch IP Extension Router Router IP Extension Switch
This section covers extension implementation details, which can be segregated into three
sections:
The WAN side
The FC and FICON side
The LAN side
IP Network
The IP network must transport Extension datagrams with adequate performance and
reliability to meet the service level goal. The service level desired is entirely dependent on the
policy set forth by the organization. Acceptable performance depends on the amount of data
that must be sent within an interim period, the WAN's capacity to accommodate the data
being sent, and the IP network's ability to deliver the data without loss. Reliability depends on
the stability of the IP network, its overall architecture, redundancy and resiliency, and its ability
to manage and deliver data across the WAN.
IBM does not support IP network devices with oversubscribed or blocking switch ports. Some
Ethernet switches have oversubscribed or blocking host access ports, which may cause
excessive drop packets and degrade storage throughput.
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 153
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
The WAN
Any type and size of WAN connection can be used for Extension as long as it meets the
application and extension platform requirements. The storage application may or may not
have more stringent requirements than the Extension platforms.
IBM supports no more than 0.1% packet loss between the Extension Ethernet ports. By
modern standards, 0.1% is considered significant packet loss. Packet loss beyond this level
indicates an evaluation and potential remediation of the IP WAN.
It’s important to understand that the effective bandwidth of a WAN link is highly dependent on
latency and link quality in terms of packet loss. Even a small amount of packet loss with
resulting retransmits significantly degrades overall link effective bandwidth.
All IBM Extension platforms have a minimum bandwidth of 20 Mbps per circuit.
The IBM SAN18B-6 can have a maximum circuit size of 2.5 Gbps if licensed as a full unit;
otherwise, 1 Gbps as a base unit. VE_Ports can accommodate up to 2.5 Gbps of cumulative
circuits as a full unit; otherwise, 1 Gbps as a base unit. On the full unit, a tunnel can be
comprised of multiple circuits of varying sizes up to 6 circuits; otherwise, only a single circuit
per VE_Port as a base unit.
The IBM SAN42B-R has a maximum circuit bandwidth of 10 Gbps if the Upgrade 1 license is
applied; otherwise, 5 Gbps as a base unit. VE_Ports can accommodate up to 20 Gbps of
cumulative circuits if the Upgrade1 + Upgrade2 licenses are applied; otherwise, 10 Gbps for
Upgrade1 and 5 Gbps as a base unit. A tunnel is comprised of multiple circuits of varying
sizes, up to 10 circuits.
IBM SX6 Extension Blade has a maximum circuit bandwidth of 10 Gbps. VE_Ports can
accommodate up to 20 Gbps of cumulative circuits. Upgrade1 and Upgrade2 licenses do not
apply to this blade. A tunnel can be comprised of multiple circuits of varying sizes, up to 10
circuits.
A best practice is to connect Extension directly to the WAN routers or switches, avoiding
intermediate devices such as firewalls. If this is not possible, every effort should be made to
connect the Extension Ethernet ports as close to the WAN as possible to prevent upstream
oversubscription and performance degradation. Extension traffic drives large amounts of
data, and uplink oversubscription often bodes poorly.
Consider the following steps when you are planning for your Extension links:
For redundancy, use at least two WAN links between sites; however, in critical
environments deploying two or more replication fabrics, consider a WAN connection for
each fabric. Generally, WAN connections are the weakest and least reliable part of
replication fabrics.
Consider dedicated WAN connections for storage interconnections. Separate storage and
non-storage traffic to better manage bandwidth, latency, and prevent congestion.
Ensure that you have a service level agreement (SLA) with the WAN service provider that
meets the enterprise's needs and expectations.
If not using Global Mirror with Change Volumes (GMCV), make sure to size the WAN
connection to sustain peak workloads. For replication over long distances, the best
practice is to use GMCV.
154 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
Bandwidth
For critical RDR (Remote Data Replication), the best practice is to use a separate and
dedicated IP connection between the production data center and the backup site.
Nonetheless, often a dedicated IP connection between data centers is not practical. In this
case, bandwidth must be, at a minimum, logically dedicated to Extension, which can be done
by:
1. Use QoS giving Extension a higher priority, which logically gives shared bandwidth to
Extension over other competing traffic.
2. Use Committed Access Rate (CAR) to identify and rate-limit certain traffic types. Use CAR
on non-Extension traffic to apportion and limit that traffic to a maximum bandwidth, leaving
the remaining bandwidth to Extension. Set the aggregate of the circuit min-comm-rates to
use the remaining dedicated bandwidth. Restricting non-Extension traffic results in
logically allocating bandwidth for Extension.
3. Massive overprovisioning to provide bandwidth for all traffic to coexist without congestion.
This method is the easiest, most costly, and most common practice seen in Extension
deployments.
Brocade Extension uses an aggressive TCP stack called WAN Optimized TCP (WO-TCP),
which dominates other TCP flows on the path causing them to dramatically back-off. UDP
based flows may result in considerable congestion and excessive packet drops for all traffic.
The best practice is to rate-limit Extension traffic on the platforms at the source and
destination and not rate limit Extension in the IP network. Rate limiting Extension in the IP
network often leads to performance problems and difficult troubleshooting issues. IBM
Extension rate limiting is advanced, accurate, and consistent; there is no need to double
rate-limit.
To determine the amount of network bandwidth needed, record the number of bytes written
over a month (or more). A granular recording with the ability to go back and calculate rolling
averages of varying lengths is most helpful. It is essential to understand the number of bytes
written to volumes during interims of the day and night. Write profiles show the quantities of
data needing to be replicated to maintain RPO (Recovery Point Objective). Expediting data
transmission is essential.
If replicating asynchronous RDR (RDR/A) across a tunnel, calculate the average value over a
finite number of minutes throughout the day and night to determine the maximum average.
Keep in mind, business transactions may increase, and replication may require significantly
more bandwidth during quarter-end, fiscal-end, and certain holidays. RDR/A needs only
enough bandwidth to accommodate the high averages discovered over a finite period. RDR/A
essentially performs traffic shaping, which moves peaks into troughs—averaging over
interims that are too long causes a backlog of writes when the troughs do not adequately
relieve the peaks. A replication backlog, which is data journaling, may result in extended
periods of recovery.
For RDR/S (synchronous replication), the best practice is to dedicate the network without
sharing bandwidth. A 10 Gbps DWDM lambda is an example of dedicated bandwidth often
used with synchronous replication. Dedicated bandwidth eliminates packet drops and timely
TCP retransmits. If planning to replicate RDR/S across an Extension tunnel, record the peak
traffic rates. RDR/S must have enough bandwidth to send writes immediately,
accommodating the entire demand at any time.
Plot the recorded values into a histogram. Suppose you have calculated 5-minute rolling
averages from your recorded data. The number of bytes written in each period is an integer
value. The x-axis contains the number of bytes written during each 5-minute average. The
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 155
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
x-axis starts at zero bytes on the left and goes up to the largest number of bytes written
during an interim—many interims have the exact number of bytes forming a Gaussian curve.
The y-axis is the number of times a particular interim occurred having the exact value. The
smallest size occurred relatively rarely, and the largest size occurred relatively infrequently.
68% of the averages fell into the middle section, as shown in Figure 7-22, the largest number
of occurrences.
Based on cost, plan your WAN bandwidth to accommodate at least the first standard
deviation (68% µ+?), and if possible, include the second standard deviation (95% µ+2?). In
most cases, beyond the second deviation, occurrences are rarer and potentially disregarded
in bandwidth planning.
You can plan for a certain amount of compression, such as 2:1 if your data is compressible;
however, data compressibility is a moving target. The best practice is to use compression as a
safety margin to address unusual and unforeseen situations. As well, achieving some amount
of compression can provide headroom for potential future growth. From the start, pushing the
limit may not be fortuitous, mainly if no margin accommodates periodic demand increases.
For remote tape, Extension connects VTLs located in various data centers. It is vital to
measure tape volume in MB/h and the number of simultaneous drives in use. If an adequate
number of drives are available, bandwidth may become the limiting factor for backup
windows.
GE Interfaces
The IBM Extension platforms have various gigabit Ethernet (GE) interface connectivity to the
IP network for transporting application data to/from the LAN and WAN sides. GE interfaces
are used by application data to communicate either to the LAN side or from the WAN side to
the long-distance IP network.
WAN-side Ethernet interfaces do not support LAG; instead, Brocade Extension Trunking
(BET) is used. BET accomplishes the same as LAG, plus it has the benefit of not requiring the
destination switches to be virtually connected, lossless link loss, and guaranteed in-order
delivery.
IBM Extension GE interfaces cannot enable loops within a data center LAN. Extension GE
interfaces are akin to server NIC interfaces; neither are they Ethernet switch ports, nor are
they router ports. GE interfaces do not participate in routing protocols and do not run
spanning tree protocol because there is no need. Ethernet frames are not permitted to
“hairpin,” meaning to come back out of an incoming interface to prevent loops. Ingress traffic
156 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
cannot go out of any other GE interface of the same type; it must go to a different side of the
Extension platform, or it is dropped.
Table 7-1 on page 158 below shows the various Ethernet interfaces and VE_Ports available
on IBM b-type Gen 7 Extension platforms.
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 157
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
Number of VEs
The IBM SAN18B-6 enables either the copper (GE0 & GE1) or the optical (GE2 & GE3).
extncfg --help
extncfg --ge-mode optical
There are two types of GE interfaces: WAN and LAN. The default type is WAN, shown as
“FCIP” in switchshow.
158 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 159
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
On the IBM SAN42B-R and IBM SX6 Extension Blade, there are two 40GE interfaces (GE0 &
GE1). They are 40 Gbps only. The IBM SAN42B-R requires Upgrade1 + Upgrade2 license to
enable the 40GE interfaces. The IBM SX6 Extension Blade does not require an optional
license for the 40GE interfaces; it comes with the 40GE interfaces enabled. The 40GE optics
are an optional purchase.
GE Interface Speeds
Depending on the platform model, various interface speeds and types are available: GE
(RJ-45), GE (SFP), 10GE, and 40GE.
The IBM SAN42B-R requires Upgrade1 + Upgrade2 to enable the 40GE interfaces.
The SX6 Extension Blade includes Upgrade1 and Upgrade2, and the 40GE interfaces are
enabled.
All GE/10GE interfaces are enabled on the base IBM SAN42B-R and SX6 Blade. The
GE/10GE interfaces are in Ethernet switching ASIC port-groups, and within a group, the
speed must be consistent; otherwise, blocking occurs. Below is a list of ports that block each
other when their speed is not set the same. The best practice is first to use ports 2, 3, 4, 5, 10,
11, 12, and 13, then move on doubling up within a port group; refer to Table 7-2 below.
2&6
3&7
4&8
5&9
10 & 14
11 & 15
12 & 16
13 & 17
The IBM SAN18B-6 has two RJ-45 interfaces, which are 1 Gbps only. When they are
enabled, SFP bays GE2 and GE3 are disabled, and vice-versa. SFP bays GE2 and GE3 are
capable of GE/10GE. No reboot of the IBM SAN18B-6 is required to change the active
interfaces; however, no configuration (ipif or iproute) can be attached to the currently active
ports.
SFP (GE) is not capable of 10 Gbps, and SFP+ (10GE) is not capable of 1 Gbps. You must
have the correct optic for the desired port speed.
160 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
LLDP
A best practice is to use LLDP as a validator of Ethernet connectivity between the Extension
application Ethernet interfaces and the directly connected DC switch. LLDP is beneficial to
both Extension Administrators and Network Administrators. Network Administrators see what
is connected to their switches. Storage Administrators see what they have connected to. At
both ends of an Ethernet link, LLDP logs connectivity information. For troubleshooting, adds,
moves, and changes; LLDP is a handy and convenient tool for referencing Ethernet interfaces
that should be investigated.
LLDP is enabled by default and operates on all application data GE interfaces. It does not
operate on the management interface. LLDP is supported and enabled by default on most
data center LAN switches.
There are certain TLVs (type, length, value) that should be enabled/disabled because they
are used/not used on IBM Extension platforms, as shown in Table 7-3.
chassis ID dcbx
mgmt-addr dot1
port ID dot3
port-desc fcoe-app
sys-capabilities fcoe-lls
sys-desc
sys-name
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 161
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
Tunnels
Tunnels are logical entities, each represented by a VE_Port. A tunnel is made up of one or
more circuits. Circuits are made up of multiple TCP sessions. Within the scope of a tunnel is
encryption (IPsec), compression, protocol optimization (FCIP-FastWrite, FCIP-OSTP,
FICON-Accelerator), and enabling IP Extension; these settings apply to the entire tunnel, not
per circuit.
VE_Ports and tunnels are synonymous. Below are the number of tunnels per platform:
IBM SAN18B-6 Extension Switch2 or 4 (base or full)
IBM SAN42B-R Extension Switch 10 or 20 (two modes of operation)
SX6 Extension Blade10 or 20 (two modes of operation)
The IBM SAN256B-7 and SAN512B-7 support up to four SX6 Extension Blades per chassis.
A tunnel configuration mismatch is indicated when the OpStatus is not Up. In this case, one
side of the tunnel is Degrade, and the other side is UpWarn. Some mismatched settings
prevent the tunnel from coming up, for example, IPsec policy or compression mismatches.
Some mismatched settings give warnings, for instance, IP Extension enabled/disabled and
inconsistent local/remote min/max-comm-rates.
162 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
Tunnel Circuit OpStatus Flags Uptime TxMBps RxMBps ConnCnt CommRt Met/G
-------------------------------------------------------------------------------
24 - Degrade --i----DI 1m13s 0.00 0.00 2 - -
24 0 ge2 Degrade ----a--i4 1m13s 0.00 0.00 2 100/150 0/-
24 1 ge4 Degrade ----a--i4 1m13s 0.00 0.00 2 100/150 0/-
-------------------------------------------------------------------------------
Flags (tunnel): l=Legacy QOS Mode
i=IPSec f=Fastwrite T=TapePipelining F=FICON r=ReservedBW
a=FastDeflate d=Deflate D=AggrDeflate P=Protocol
I=IP-Ext
(circuit): h=HA-Configured v=VLAN-Tagged p=PMTU i=IPSec 4=IPv4 6=IPv6
ARL a=Auto r=Reset s=StepDown t=TimedStepDown S=SLA
7840-Side-B:FID128:admin> portshow fciptunnel -c --summary
Tunnel Circuit OpStatus Flags Uptime TxMBps RxMBps ConnCnt CommRt Met/G
-------------------------------------------------------------------------------
24 - UpWarn --i----D- 7s 0.00 0.00 2 - -
24 0 ge3 UpWarn ----a--i4 7s 0.00 0.00 2 100/150 0/-
24 1 ge5 UpWarn ----a--i4 7s 0.00 0.00 2 100/150 0/-
-------------------------------------------------------------------------------
Flags (tunnel): l=Legacy QOS Mode
i=IPSec f=Fastwrite T=TapePipelining F=FICON r=ReservedBW
a=FastDeflate d=Deflate D=AggrDeflate P=Protocol
I=IP-Ext
(circuit): h=HA-Configured v=VLAN-Tagged p=PMTU i=IPSec 4=IPv4 6=IPv6
ARL a=Auto r=Reset s=StepDown t=TimedStepDown S=SLA
Below is the portshow fciptunnel --detail output from both Extension switches. In this
case, IP Extension has been enabled on one side and not the other. Configuration must
match on both ends.
7840-Side-A:FID128:admin> portshow fciptunnel --detail
Tunnel: VE-Port:24 (idx:0, DP0) Broadcom Extension
====================================================
Oper State : Degraded
TID : 24
Flags : 0x00000000
IP-Extension : Enabled
Compression : Aggressive Deflate
FC-Compression : Aggressive Deflate (Inherited)
IP-Compression : Aggressive Deflate (Inherited)
QoS Distribution : Protocol (FC:50% / IP:50%)
FC QoS BW Ratio : 50% / 30% / 20%
IP QoS BW Ratio : 50% / 30% / 20%
Fastwrite : Disabled
Tape Pipelining : Disabled
IPSec : Enabled
IPSec-Policy : MyTunnel
Legacy QOS Mode : Disabled
Load-Level (Cfg/Peer): Failover (Failover / Failover)
Local WWN : 10:00:c4:f5:7c:3b:24:86
Peer WWN : 00:00:00:00:00:00:00:00
RemWWN (config) : 00:00:00:00:00:00:00:00
cfgmask : 0x0000001f 0x4001034e
Uncomp/Comp Bytes : 11048 / 11048 / 1.00 : 1
Uncomp/Comp Byte(30s): 0 / 0 / 1.00 : 1
Configuration Warnings:
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 163
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
VE_Ports
VE_Ports are Virtual Expansion Ports. They are a type of E_Port (Expansion Port) commonly
used in FC fabrics. VE_Ports connect to VE_Ports and are used to connect a domain to a
domain (a switch to a switch) the same as E_Ports; the difference is VE_Ports are WAN side
facing.
VE_Ports are tied to a DP (data processor). All Extension platforms have a DP0; additionally,
the IBM SAN42B-R and IBM SX6 Extension Blade have a DP1. The best practice is to start at
164 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
the first VE_Port on DP0. When a second tunnel is required on the SAN42B-R and SX6
Blade, use the first VE_Port on DP1. On the SAN18B-6, move to the next VE_Port on DP0.
IBM SAN42B-R
– VE_Port max bandwidth 20 Gbps (applicable license required)
– DP0: VE 24 to 28
– DP1: VE 34 to 38
IBM SX6 Extension Blade
– VE_Port max bandwidth 20 Gbps (applicable license required)
– DP0: VE 16 to 20
– DP1: VE 26 to 30
IBM SAN18B-6
– VE_Port max bandwidth 2.5 Gbps (applicable license required)
– DP0: VE 12 to 15
It is imperative to understand TCL rules only live on the DP unto which the target VE_Port
exists. VE_Ports exist only in one DP, either DP0 or DP1. Refer to the TCL section for details.
IP Extension gateways are created on a specific DP, either DP0 or DP1. Ingress traffic
matching a TCL rule is directed to a particular target (VE_Port). If the VE_Port target
designated in a TCL rule is not congruent with the IP Extension gateway’s DP, the rule is not
evaluated. In other words, both the IP Extension gateway and target VE_Port used must be
on the same DP for the rule to be evaluated. If the rule is expected to be assessed but is not,
check if traffic is hitting an IP Extension gateway configured on the other DP.
A VE_Port (tunnel) cannot be shared across different VF LS; the VE_Port must live within a
single LS. GE interfaces should stay in the Default Switch, where they can be used by circuits
originating from any LS. See the section on Ethernet Interface Sharing.
The maximum cumulative circuit bandwidth of a VE_Port cannot exceed (license permitting):
2.5 Gbps on the IBM SAN18B-6
20 Gbps on the IBM SAN42B-R
20 Gbps on the IBM SX6 Extension Blade
Interoperability
From the perspective of tunnel connectivity and features (VE_Port to VE_Port), the IBM
SAN18B-6, SAN42B-R, and SX6 Extension Blade interoperate together.
The SAN42B-R and SX6 Blade have equal bandwidth capabilities when licensed
appropriately.
The IBM SAN42B-R and SX6 Blade support performing eHCL; the IBM SAN18B-6 does not;
however, the IBM SAN18B-6 does support the required eHCL connectivity needed by the IBM
SAN42B-R and SX6 Blade.
Virtual Fabrics
Virtual Fabrics (VF) are comprised of one or more Logical Switches (LS) within the physical
chassis. Connected Logical Switches form Logical Fabrics within the overall physical fabric.
VF plays a primary role in achieving deterministic paths for Extension protocol optimization.
Additionally, in mainframe environments, the implementation of FICON and FC Extension
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 165
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
often requires unique configurations and management, which is only achievable within the
same chassis when using different LS.
Protocol optimization requires that an exchange’s sequences and frames pass through the
same VE_Ports for both outbound and return. This determinant path means that only a single
VE_Port can exist within a VF LS. Putting a single VE_Port in an LS defines a single path
between the two LS. An LS sufficiently scales FC port connectivity, are operationally
simplistic, and creates a stable environment. The IBM SAN42B-R and IBM SX6 Extension
Blade support VF. The IBM SAN18B-6 does not support VF.
IP Extension
IP Extension is enabled within the tunnel scope (fciptunnel) using --ipext enable. The IBM
SAN18B-6 Extension platform does not have different app-modes for FCIP Only and Hybrid;
essentially, it is always in Hybrid mode. The IBM SAN42B-R and IBM SX6 Extension Blade
must be put into the Hybrid app-mode before IP Extension can be enabled on a tunnel.
Changing the app-mode requires a reboot of the platform or slot. Use the extncfg command
to check the current status or perform this app-mode task. Only change the app-mode to
Hybrid if, at some point, IP Extension will be used.
Compression
Compression is recommended with RDR (Remote Data Replication) applications because it
improves the efficient use of WAN bandwidth. Fast-Deflate compression is ultra-fast and
ultra-low latency. It can be used with RDR/S (sync) to improve response time by reducing
data serialization time across the wire, essentially more data quicker.
Commonly, tape traffic is already compressed, and compressing data again is not helpful;
therefore, if the traffic is already compressed, leaving compression disabled may be prudent.
Note: IBM makes no guarantees, warranties, or claims about the actual compression ratio
achieved with customer-specific data.
166 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
IPsec/Encryption
The best practice is to leverage encryption to protect data end-to-end. It is prudent to enable
IPsec when implementing IBM Extension. All data leaving the secure confines of the data
center into public infrastructure guarantees no security, plus service providers do not
guarantee unencrypted data while in flight. Links must be authenticated and data encrypted
to prevent attacks and eavesdropping. IBM IPsec is easy and practical to deploy. Would your
company operate WiFi with no encryption? Of course not, and Extension is no different!
IPsec is Suite B compliant and implements the latest encryption technologies such as AES
256, SHA-512 HMAC, IKEv2, and Diffie-Hellman. Rekeying occurs in the background
approximately every 2 billion frames or every 4 hours, and the process is non-disruptive.
From end to end, the IP network must allow specific protocols to pass. When not using IPsec,
Brocade Extension uses TCP destination ports 3225 and 3226. The Extension TCP
implementation selects a random source port between 49152 and 65535 as the ephemeral
(or initiating) port to open to destination ports 3225 and 3226. TCP URG flags must not be
dropped or modified. When using IPsec, IKEv2 uses UDP port 500. Management connectivity
protocols (ssh, snmp, https...) are not covered here.
Protocol Acceleration
Protocol acceleration is used to expedite data without the time needed to traverse the WAN to
request a transfer ready. Protocol optimization effectively mitigates the latency effects on
SCSI/FCP disk writes and tape read/writes. Additionally, there is protocol optimization for
FICON tape and IBM z/OS® GM (XRC).
FastWrite, Open Systems Tape Pipelining (OSTP), and FICON Acceleration are features of
IBM Extension platforms. IBM protocol acceleration requires no additional hardware or
hardware licenses. The same Data Processor (DP) that forms tunnels/trunks also performs
protocol acceleration for a higher level of integration, efficiency, resource utilization, and
reduced assets.
Flows through Extension are monitored for protocol optimization viability. Not including
FICON, IBM storage products do not benefit from or engage FCIP-FastWrite. FastWrite
should not be enabled if Extension is being used for Global Mirror or Metro Mirror alone. If
Extension is being deployed with another application known to benefit from and engage
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 167
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
FastWrite, then FastWrite should be enabled. When FastWrite is enabled, it will work with the
other application while ignoring Global Mirror and Metro Mirror. IBM FICON tape, Teradata,
and z/OS GM (XRC) can be optimized with IBM Advanced FICON Accelerator, which is
included in the mainframe version of the IBM SX6 Extension Blade and an optional license on
the IBM SAN42B-R.
BET (Brocade Extension Trunking) is essential for protocol optimization techniques like
FastWrite, Open Systems Tape Pipelining (OSTP), and FICON Acceleration. BET uses a
single VE_Port and is essentially an ISL. Taking a different VE_Port outbound versus inbound
breaks protocol optimization by creating an inconsistent state machine. Individual tunnels,
each with their own VE_Port, leads to indeterminate VE_Ports for sequences. It is not
possible to perform protocol acceleration with multiple parallel VE_Ports between the same
domains. Regardless of the routing mode (EBR, DBR, and PBR), multiple VE_Ports prevents
a bidirectional deterministic path. For instance, with PBR, the outgoing sequence may
consistently take the same VE_Port; however, the return sequence may always take the same
wrong VE_Port.
Note: Protocol optimization techniques prevent using load balancing and failover between
VE_Ports
Protocol acceleration uses state machines to perform optimization and requires traffic to be
confined to a single VE_Ports pair. State machines need to know every sequence that has
passed during the exchange. State machines facilitate proper handling of an exchange until it
is complete, after which the state machine is merely discarded. State machines are created
and destroyed with every exchange. IBM Extension has the capacity for tens of thousands of
simultaneous state machines and an equivalent number of flows. Ingress flows are verified if
they can be optimized; if not, the flow does not engage protocol acceleration. If a flow cannot
be optimized, it merely passes without optimization. A state machine cannot exist across
multiple VE_Ports, and there is no communication between different state machines.
If an exchange starts on VE24 and returns on VE25, the exchange passes through different
state machines, as shown in Figure 7-23. One is knowing nothing about the other. This
situation causes protocol acceleration to break and an error condition.
? ?
168 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
One advantage of using BET is to have a single tunnel regardless of how many circuits make
up the tunnel. The state machines exist at the TCP Supervisor endpoints and remain
consistent no matter which circuit is used (see Figure 7-24). BET permits protocol
optimization to function across load-balanced links with non-disruptive failover/failback.
BET
Return circuit1 – XFER_RDY
State machine consistency
Multiple circuits, one tunnel
In situations where protocol acceleration is not enabled, data flows can be routed across
multiple VE_Ports between domains. The routing mode is based on the APTpolicy. One of
three methods is used internally to route FC flows between equal-cost VE_Ports from the FC
switching ASIC:
Exchange Based Routing (EBR), the default): Originator Exchange ID/Source
ID/Destination ID (OXID/SID/DID)
Device-Based Routing (DBR): SID/DID
Port-Based Routing (PBR): Source Port
FastWrite (FCIP-FW)
FCIP-FastWrite is a protocol optimization technique that intercepts an FCP exchange and
spoofs a local response back to the initiator to expedite data out to the target. There is no
need for the initiator’s request to first traverse the WAN, be responded to by the target, and
travel the WAN again before data starts being sent.
IBM GM and MM do not benefit from or engage FastWrite on IBM b-type Gen 7 Extension
platforms. Optimization is already built into these replication applications. Nonetheless,
FastWrite may still be needed and can be used with cohort replication applications without
detriment to GM or MM.
FICON Accelerator
Advanced FICON Accelerator is an FCIP protocol optimization technique that works with
FICON tape, Teradata, and z/OS GM (XRC) over distances 2 ms RTT or greater.
There are two versions of the IBM SX6 Extension Blade available; there are a distributed
systems and a mainframe version. The IBM SX6 Extension Blade mainframe version includes
the FICON Accelerator License (FAL). FAL is an optional license on the SAN42B-R. The
SAN18B-6 does not support FICON features and functions.
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 169
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
Circuits
Circuits make up a tunnel. A tunnel consists of one or more circuits. A tunnel consisting of
multiple circuits does not have to rely on a single port, optic, cable, switch, router, WAN
connection, or service provider; in essence, a single point of failure in the IP network can be
avoided.
The tunnel (VE_Port) manages circuits. Circuits contain several WAN Optimized TCP
sessions. Different TCP sessions are created for each QoS priority (H/M/L) within each
protocol plus control traffic. There are two Extension protocols, IP Extension and FCIP.
Within the scope of circuits is ARL (min/max committed rate), local/remote IP, local/remote
HA IP (not on the IBM SAN18B-6), metrics/group, KATOV, QoS (DSCP and L2CoS).
IP Interfaces (ipif)
A WAN side GE interface can have multiple IP interfaces (ipif) assigned to it; each is a circuit
endpoint. A circuit is assigned to one Ethernet interface, for example, ge2.dp0. You cannot
split a circuit across multiple Ethernet interfaces. An IP interface (ipif) is configured with an IP
address and GE interface. A circuit is created with a local IP address and bound to a GE
interface; the ipif the IP address was configured with.
IP Routes (iproute)
IP routes are used to forward traffic to the next-hop gateway. IP routes apply to both the LAN
side and the WAN side. For the LAN side, an IP route is only used to send IP Extension traffic
out to an end device, traffic that has come in from the WAN side. IP routes are not used to
route ingress LAN traffic into an egress tunnel; only TCL is used for this.
IP routes are specific to a GE interface and specified when created. Every GE interface has
its own routing table. Therefore, if two circuits, each using the same source subnet, going to
the same destination subnet, and assigned to different GE interfaces would require the
following ipif and iproute creation:
portcfg ipif ge2.dp0 create 10.10.10.10/24
portcfg ipif ge3.dp0 create 10.10.10.11/24
portcfg iproute ge2.dp0 create 10.20.20.0/24 10.10.10.1
portcfg iproute ge3.dp0 create 10.20.20.0/24 10.10.10.1
IP routes are used on the WAN side to specify the gateway when the destination subnet is not
the same as the source subnet (Layer3 network). For example, the local ipif subnet is
10.10.10.0/24, and the remote ipif subnet is 10.20.20.0/24. Because these subnets are
different, an IP route is required. IP routes are not required if the local and remote IP
addresses are on the same subnet (Layer2 network). Continuing with this example, assume
the local router gateway was 10.10.10.1. The IP route would be: to get to subnet
10.20.20.0/24 use gateway 10.10.10.1. The gateway is an IP address on the same subnet as
the IP address on the ipif.
Diverse Pathways
Each circuit is intended to take a different pathway to gain resiliency and redundancy across
the IP network. Therefore, circuits should connect via various physical Ethernet links to the
different physical data center LAN switches or WAN routers. When there is more than one
WAN connection, the circuits should be statically assigned to those WAN connections. BET
failover/failback is automatic, lossless, and guarantees in-order delivery; therefore, it is best to
have the circuit go offline, stopping the keepalives, and forcing traffic to failover at the
FC/FICON level. In the case of two WAN links, typically, four circuits are used; here is an
example:
Cir0: DC LAN Switch A - WAN 1
170 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
The effective RTT for all circuits in a trunk is the longest RTT experienced by a circuit. By
effectively simulating the same RTT for all circuits, data can be driven at line-rate without
significant data arriving out of order. Wait times climb with excessive out of order data.
Bandwidth scheduling is weighted per circuit based on the comm-rate. A typical example is a
tunnel deployed across a ring architecture, in which one span is a relatively short distance
and the other span is much longer. It remains practical to trunk two circuits across each span
of the ring. Asynchronous RDR (Remote Data Replication) is not affected when both paths
experience longer latency, yet the data rate delivery rate is optimized.
Brocade Extension Trunking (BET), in essence, provides a single logical tunnel comprised of
multiple circuits. A tunnel with multiple circuits is referred to as a trunk. Brocade Extension
uses a VE_Port to form an Extension ISL consisting of associated circuits. If each circuit were
a tunnel, a VE_Port would be required for each; however, each circuit makes up a single
tunnel with BET. A tunnel is an Inter-Switch Link (ISL) and treated as such in fabric design.
Circuits are individual connections within the tunnel, each with a unique source and
destination IP address and other settings.
Figure 7-25 on page 172 shows an example of a tunnel trunking four circuits. Each circuit
endpoint is assigned a unique IP address by way of IP interfaces (portcfg ipif). An IP
interface is bound to an Ethernet interface when the ipif is configured. In this case, each IP
interface has been assigned to a different Ethernet interface; however, this is not required.
Depending on the environment’s needs, Ethernet interface assignments are flexible and can
be made as needed. For instance, multiple IP interfaces can be assigned to a single Ethernet
interface. There are no subnet restrictions. Circuits connect from the local IP interface to the
remote IP interface via the bound Ethernet interfaces.
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 171
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
VE_Ports are the demarcation point between the FC world and the TCP/IP world. Brocade
Extension uses FC switching ASICs (Application Specific Integrated Circuits), which knows
only FC protocol. VE_Ports are not an included part of these ASICs and are logical
representations of FC ports. Internally, multiple backend FC ports feed a VE_Port to gain high
data rates into the Extension data processor (DP).
Since one VE_Port represents the endpoint of multiple trunked circuits, this comes with
benefits:
Fewer VE_Ports are needed, and the remaining VE_Ports are available for other
destinations. Typically, only one VE_Port is required per remote location.
Bandwidth aggregation is achieved by adding circuits to a trunk. Circuits do not have to be
configured identically and do not have to take the same or similar paths. The RTT (Round
Trip Time) does not have to be the same.
Link failover is fully automated with BET. Using circuit metrics and failover groups enables
active and passive circuits to coexist within the same trunk.
When a connection is lost, data inflight is lost as well. Any frames being transmitted at the
instance of the outage are lost due to partial transmission. Consider multiple circuits,
Out-Of-Order-Segments (OOOS) occurs when some segments arrive, one or more are lost
inflight, and then segments continue to arrive. OOOS are problematic for some devices,
particularly mainframes, which results in an IFCC (Interface Control Check). For example,
frames 1 through 4 are sent. Frames 1 and 2 are received. Frame 3 is lost in transit, and
frame 4 is received. The receiving side detects an OOOS because it was expecting frame 3
but received frame 4. BET solves this problem with LLL (Lossless Link Loss).
For BET, the DP (Data Processor) that owns the VE_Port controls all circuits. There is no
distributed processing, load sharing, or LLL across DPs. Failover of FC traffic from one DP to
another (for example, a VE on DP0 gets rerouted to a VE on DP1) is done at the FC level
provided the configuration permits it. The only shared components may be various circuits
bound to the Ethernet interfaces.
When a link is lost, the lost frames are retransmitted by LLL (see Figure 7-26 on page 173).
Usually, when a TCP segment is lost due to a dirty link, bit error, congestion, or whatever,
TCP is aware and retransmits the segment. In the case of an offline circuit, TCP has no way
to retransmit lost segments because TCP is no longer operating on that connection.
Recovery must occur at a higher level in the stack.
172 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
TCP sessions are managed by a “TCP Supervisor,” which feeds each circuit’s TCP sessions
through a load balancer and a sophisticated egress queuing/scheduling mechanism that
compensates for different bandwidths, latencies, and congestion events.
Suppose a link goes offline, and data can continue over remaining connections. In that case,
missing frames indicated by noncontiguous sequence numbers trigger a duplicate
acknowledgment back to the TCP Supervisor to retransmit the missing segment, even though
that segment was initially sent by a different, now defunct, TCP session. Data lost in flight
requires all segments to be held in memory for possible retransmission until acknowledged at
the supervisor level.
Failover/Failback/Spillover
Circuits can be active or passive within a trunk and configured with metrics. Within a failover
group, all online circuits with the lowest metric value are active, for example, a value of 0.
Circuits with a value of 1 are not active and only become active after ALL metric 0 circuits
have gone offline within the group. An example of two groups, one-to-one pairings of a metric
1 circuit with a metric 0 circuit is shown in Figure 7-27 on page 174. Within a group, the metric
1 circuits will losslessly resume the metric 0 traffic upon the last metric 0 going offline. Metric
1 circuits should take different routes, considering network outages are the most likely cause
the offline metric 0 circuits.
Metrics with failover groups permit passive circuits to become active over a less desirable
path when the regular production path becomes unavailable. In a three-site triangular
architecture, the primary circuit has a metric of 0 in which regular production traffic takes a
single hop, shortest distance, lowest latency path. Sending traffic through the secondary path,
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 173
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
which has two hops and is typically longer in distance and latency, is prevented unless the
primary path is interrupted. Setting a circuit to metric 1 makes it passive until all metric 0
circuits within the group have gone offline. Nonetheless, dormant circuits are members of the
trunk. Convergence over to the secondary path is lossless, and there will be no out-of-order
segments. No IFCC is generated on a mainframe. Extension Trunking is required to assign
metrics to circuits.
ARL
ARL is an integral part of most Extension designs. When there is more than one circuit
feeding into the same WAN link or when the WAN is shared with other traffic, ARL is
essential. ARL is used to manage data sent into the IP network based on min and max rate
settings and the unconsumed bandwidth between min and max (see Figure 7-28). There may
be one or multiple WAN connections. The cumulative circuit bandwidth within a particular
WAN connection is what matters. Each WAN connection is evaluated independently.
ARL manages the bandwidth into these sections. The Extension guaranteed section is the
minimum bandwidth available for Extension on the WAN. The Extension guaranteed section
is the amount of bandwidth reserved exclusively for Extension, and ARL aggressively pushes
at least this amount. It is important to note; the minimum bandwidth is the aggregate of ALL
circuit minimums on the WAN connection.
Non-Extension Bandwidth
Bandwidth greater than the cumulative max-comm-rate
of all circuits on this WAN link
Adaptive Bandwidth
Max-comm-rate > available WAN BW > min-comm-rate
Extension bandwidth used when available
Guaranteed Bandwidth
Cumulative min-rate will consume at least this much
bandwidth
Adaptive is the area between the top of the guaranteed section and the bottom of the
Non-Extension section. The bottom of the non-Extension section is the aggregate of the
174 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
max-comm-rates for all WAN connection circuits. Extension circuits may use the adaptive
bandwidth when other non-Extension applications are not using it.
The non-Extension bandwidth section is reserved exclusively for applications sharing the
WAN. Extension traffic does not exceed the aggregate of the max-comm-rates for all circuits
on the WAN connection.
Assume a 1 Gbps WAN as an example. The ARL max-comm-rate is set to either the line-rate
GE interface or the maximum available WAN BW, whichever is lowest. In this case, the
max-comm-rate is set to 1 Gbps. Each circuit is configured with a floor (--min-comm-rate) and
ceiling (--max-comm-rate) bandwidth value in kbps (kilobits per second). Assume adequate
bandwidth demand from the storage application; the minimum circuit bandwidth is never less
than the min-comm-rate and is never more than the max-comm-rate. The circuit bandwidth
adjusts automatically between the min and max based on WAN conditions at that moment. A
congestion event causes the rate-limit to readjust towards the minimum. An absence of
congestion events causes it to rise towards the maximum. If the current rate is not maximum,
ARL attempts to adjust upward; however, the rate remains stable if repeated congestion
events are detected. When there is constant congestion, the minimum comm-rate becomes
the prevailing rate.
The ARL min-comm-rate is set to the max-comm-rate ÷ the number of circuits feeding the
WAN connection. In this example, 1 Gbps ÷ 4 = 250 Mbps. The min-comm-rate is set to 250
Mbps. When all the circuits are up, each runs at 250 Mbps. In an extreme case in which three
circuits have gone offline, the remaining circuit runs at 1 Gbps. At 1 Gbps, all the WAN
bandwidth continues to be consumed, and the replication application remains satisfied.
When more than one circuit with the same ARL settings feeds a WAN link, the two circuits
equally utilize the available bandwidth. When an interface or the entire platform goes offline,
ARL readjusts to use available bandwidth that is no longer being used by offline circuits.
Adaptivity and readjustment of bandwidth maintain optimal utilization during periods of
maintenance and failures.
eHCL works by moving the circuits from their native DP to the opposite DP. Firmware
upgrades occur sequentially one DP at a time. When a DP upgrade is complete, the circuits
move back to their original DP. Firmware updates can only be initiated on one tunnel endpoint
at a time. A stable fabric with no device adds, moves, or changes occurring is strongly
suggested.
To enable eHCL, an additional set of HA (high availability) ipif, iproute, and IP addresses
needs to be added to each tunnel on both ends. Assume a tunnel and circuits are on
DP0-VE24:
The HA ipifs are created on DP1 (the opposite DP), for example, ipif ge2.dp0
10.10.10.10/24, the complimentary HA ipif ge2.dp1 10.10.10.11/24. Typically, the same
subnet is used for convenience but not required.
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 175
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
The HA iproutes are created on DP1, for example, iproute ge2.dp0 192.168.10.0/24
10.10.10.1, the complimentary HA iproute ge2.dp1 192.168.10.0/24 10.10.10.1.
Add to fcipcircuit --local-ha-ip 10.10.10.11 (as shown above), and the --remote-ha-ip
192.168.10.11 (not shown above).
There is nothing more to configure or enable. The eHCL process occurs automatically with
each firmware update provided the HA status is “HA Online,” refer to the example CLI output
below.
KATOV
Each circuit uses keepalives to reset the keepalive timer. The time interval between
keepalives is a configurable setting. If the keepalive timer expires, the circuit is deemed to be
down and is removed from the trunk. Also, when an Ethernet interface loses light, those
circuits are immediately considered down. When a circuit goes offline, the egress queue for
that circuit is removed from the load-balancing algorithm. Traffic continues across the
remaining circuits, albeit at the reduced bandwidth due to removing the link.
1. How does the keepalive mechanism work on a circuit?
A Keepalive Timeout Value (KATOV) is configured for each circuit in the tunnel. BET uses
the value to determine how frequently a keepalive is sent. The math for that is not entirely
straightforward and is beyond the scope of this document. The algorithm ensures that
176 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
multiple keepalives are sent within the timeout period. If the receiving side does not
receive any keepalives within the timeout period, the circuit is deemed down.
2. How are keepalives queued and handled through the network layers?
Keepalives are treated as normal data to WAN-Optimized TCP (WO-TCP). They are not
sent on any unique path through the network stack; therefore, they intermix with the other
data passing through TCP, guaranteeing keepalive transmission. Keepalives cannot be
lost unless the circuit is down. Moreover, the keepalive process determines actual delay
through an IP network. The mechanism takes latency, reorder, retransmits, and any other
network condition into account. TCP is the preferred transport for keepalives because it
keeps tight control of the time data is permitted on the WAN. If getting through the network
exceeds the KATOV, the circuit is torn down, and pending data is re-queued into the
remaining circuits.
It is recommended to configure the KATOV based on the storage application’s timeout
values. The exception is circuits configured with the FICON option, which have a 1-second
KATOV and should not be altered. The default value for non-FICON tunnels is 6 seconds
and can often be reduced depending on the network characteristics. This reduction is
evaluated on a case-by-case basis. When a tunnel contains multiple circuits, the KATOV
should be less than the storage application timeout; this facilitates Lossless Link Loss
(LLL) before the application times out.
3. Can too much data through a circuit bring it down?
Yes and no; the answer is no if there are no issues in the IP network, and the tunnel is just
not big enough to handle the driven workload. No network issues mean no congestion or
packet loss, and ARL is appropriately configured; the bandwidth merely cannot
accommodate the application’s needs. When all the available bandwidth is used, buffer
credit flow control applies backpressure to the incoming flows to prevent buffer overflow.
No circuits go down due to keepalive timeout.
Keepalives are not negatively affected by large amounts of queued data. The amount of
data that can be queued but not sent to the network is relatively small, no more than a few
milliseconds. A few milliseconds is not enough time to cause a keepalive timeout due to
saturation.
On the other hand; misconfigured ARL bandwidth, network issues, congestion, and packet
loss are conditions that may lead to keepalive timer expiration, and subsequently, a
dropped circuit. Improper rate limiting allows more data to enter the network than
bandwidth is available, which results in network errors. Severe buffer overruns, excessive
packet loss, and time exhausting retransmits can result in enough lost keepalives to bring
a circuit down.
QoS
IBM Extension has a comprehensive Quality of Service suite.
Protocol Distribution (enforcement)
Priority -High/Medium/Low (enforcement)
DSCP (marking)
802.1P (marking)
PTQ (isolation)
Note: The best practice is NOT to alter class-F (control traffic) QoS markings unless
required to differentiate and expedite control traffic across the IP network. Control traffic
failing to arrive promptly causes fabric instability.
Below is an example CLI output showing a tunnel (VE24) with one circuit (0 on GE2) and its
individual QoS components (control, high, medium, and low). The CommRt shows bandwidth
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 177
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
distribution to each priority out of 100%; these are not rate-limits. Control is a strict priority - if
there is something to send, the control traffic is sent before any other.
LVN-7840-B_CA:FID1:admin> portshow fciptunnel -c --qos
Tunnel Circuit OpStatus Flags Uptime TxMBps RxMBps ConnCnt CommRt Met/G
-------------------------------------------------------------------------------
24 - Up c----F-D- 28d22h 0.00 0.00 1 - -
24 0 ge2 Up ----r---4 28d22h 0.00 0.00 1 0/100 0/-
24 - Up h----F-D- 28d22h 0.00 0.00 1 - -
24 0 ge2 Up ----r---4 28d22h 0.00 0.00 1 50/100 0/-
24 - Up m----F-D- 28d22h 0.00 0.00 1 - -
24 0 ge2 Up ----r---4 28d22h 0.00 0.00 1 30/100 0/-
24 - Up l----F-D- 28d22h 0.00 0.00 1 - -
24 0 ge2 Up ----r---4 28d22h 0.00 0.00 1 20/100 0/-
-------------------------------------------------------------------------------
Flags (tunnel): c=Control h=HighPri m=MedPri l=LowPri
i=IPSec f=Fastwrite T=TapePipelining F=FICON r=ReservedBW
a=FastDeflate d=Deflate D=AggrDeflate P=Protocol
I=IP-Ext
(circuit): h=HA-Configured v=VLAN-Tagged p=PMTU i=IPSec 4=IPv4 6=IPv6
ARL a=Auto r=Reset s=StepDown t=TimedStepDown S=SLA
Protocol Distribution
Which one gets the bandwidth? Protocol distribution answers this question, and it is
user-configurable. The default is 50/50, meaning that each protocol receives 50% of the
tunnel’s cumulative circuit bandwidth during contention. When there is no egress contention,
there is no limit imposed by protocol distribution. For example, FCIP would get 100% of the
bandwidth if there was no IP Extension traffic at that moment. Protocol distribution does not
need to be configured when implementing only one of the two protocols because there is no
contention between protocols.
A frequently used case is a tape grid (IP Extension) with disk replication (FCIP). The best
practice is to manage the overall bandwidth using the same VE_Port (tunnel). Often, disk
replication is more critical, and the desire is to permit disk replication over the tape. Assign
disk replication the bandwidth it needs, and the tape grid gets what disk does not use. Setting
protocol distribution to 90/10 FCIP/IP, respectively, the disk gets 90% of the bandwidth if and
when it needs that much. Worst case scenario, the tape grid gets 10% of the bandwidth. If the
WAN is sized correctly for disk plus tape, rarely will disk replication require 90% of the
bandwidth. Unused bandwidth is utilized by IP Extension for the tape grid. During periods of
low disk bandwidth usage, the tape grid gets most of the bandwidth. Protocol Distribution is a
safety mechanism for disk replication on a shared WAN connection.
IBM Extension supports three levels of priority (H/M/L). The default amount of BW that the
scheduler apportions during times of contention is 50/30/20 percent. QoS apportioning of BW
occurs only during contention; otherwise, the BW is shared equally across all priorities. It is
possible to change the default portions to any quantity needed, as long as the amounts are
10% or greater and add up to 100%.
There are seven outbound priorities per circuit. The default percentages are shown:
Class-F (strict)
178 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
FCIP (50%)
– High (50%)
– Medium (30%)
– Low (20%)
IP Extension (50%)
– High (50%)
– Medium (30%)
– Low (20%)
Class-F (control) uses strict queuing, meaning if there is any data to send, it is transmitted
first. There is very little control traffic, and it does not interfere with data. Each QoS priority
has an autonomous TCP session and no reliance on other sessions. Each priority TCP
session can be configured with its own DSCP, VLAN tagging, and 802.1P values. The ability
to set each circuit uniquely permits sessions to be treated independently based on priority
SLA between sites.
There is QoS within the SAN fabrics and across FC ISLs via Virtual Channels (VCs). There
are different VCs for H/M/L and Class-F. Each VC has its own Buffer-to-Buffer Credits and
flow control. Devices are assigned to QoS VCs by enabling QoS in the fabric and putting
QOSH_ or QOSL_ as the prefix to a zone name. The default is QOSM_; therefore, there is no
need to explicitly designate medium zones; although, the prefix is supported.
Devices use VCs throughout the fabric, including Extension. If data ingresses via a “QOSH_”
prefixed zone, the data uses High VCs within the fabric and is automatically assigned to a
High TCP session in the Extension circuits. Traversing the fabric is not required to set traffic to
Extension TCP priorities. Any device directly connected to an Extension platform is assigned
a priority TCP session based on the zone name prefix. The default TCP session is medium.
DSCP
Differentiated Services Code Point (DSCP) is an IP based (L3) Quality of Service (QoS)
marking; therefore, it is an end-to-end QoS marking protocol. DSCP has 64 values; however,
the values 0 through 63 do not denote lowest to highest priority; the marking schema is
different.
All odd values are for private use and can be used in any way an enterprise sees fit. These
odd numbers are for private use, similar to how RFC 1918 IP addresses are for private use.
For example, 192.168.10.1 and DSCP 3 are private values used in any way an enterprise
desires.
Value 46, non-private DSCP value, is referred to as “Expedited Forwarding” and is the highest
DSCP priority. Zero is the default and the lowest priority. There are four groups of
High/Medium/Low values referred to as “Assured Forwarding.” Another group of numbers is
used for backward compatibility with legacy ToS (Type of Service).
Using DSCP in an IP network is the network administrators' responsibility. Without their buy-in
and configuration of the Per-Hop Behavior (PHB) associated with QoS, no QoS can happen.
Ethernet switches' default behavior is to ignore QoS values replacing any ingress value with
zero; unless data arriving on that interface is configured as trusted. Explicitly configuring
specific ingress data as trusted prevents end-users from setting QoS values unannounced to
their Network Administrators.
802.1P
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 179
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
802.1P is a data link-based (L2) QoS marking; therefore, the scope extends from the
Extension platform's GE interface to the Ethernet interface of the directly attached switch.
Devices enforcing 802.1P provide QoS across the data link. The 802.1P header resides
within the 802.1Q VLAN header; therefore, VLAN tagging is required to get 802.1P QoS
marking. Brocade Fabric Operating System refers to 802.1P as L2CoS. (Layer 2 Class of
Service). There are only eight 802.1P values from 0 to 7. Zero is the default and lowest
priority. Seven is the highest priority. The IP network must be configured to acknowledge QoS
markings before being useful; otherwise, the Extension markings are ignored.
PTQ
PTQ (Priority TCP QoS) is not configurable; nonetheless, it is a crucial QoS concept to
understand when extending QoS across Extension. Any FC zone with a prefix indicating QoS
uses priority virtual circuits within the fabric and automatically maps traffic to a corresponding
TCP priority in the Extension circuits.
One TCP hallmark is ensuring in-order delivery. QoS, by nature, indicates flows requiring
expediency and designates such handling by the network. TCP and QoS would work against
each other if QoS expedited a flow, and TCP returned it to the original sent order. Therefore,
PTQ applies autonomous TCP sessions for each QoS priority. With PTQ, expediting flows
across different priorities is not an issue because all traffic within each TCP flow has the same
priority (H/M/L).
Ethernet Interface Sharing applies only to circuits on the WAN side. The IP Extension
gateways sit behind the LAN interfaces, and LAN interfaces must remain in the Default LS.
Note: IP Extension GE interfaces must exist in the Default LS and cannot be moved to a
different context.
PortChannels
LAN side interfaces support IEEE 802.1AX Link Aggregation (LAG or PortChannel) to the
data center LAN switches. Only a single “connection” per VLAN from the Extension platform
to the data center LAN is supported; however, multiple links can be considered one logical
connection (see ). If the data center switches are a single virtual switch, often referred to as
“vPC” in the Cisco nomenclature. A LAN side PortChannel spanning the switches is
considered a single connection.
In Figure 7-29 on page 181, four links form one LAG. From the viewpoint of the Extension
platform, this is a single connection.
180 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
Figure 7-30 shows a PortChannel consisting of two links to one data center LAN switch.
Forming a LAG to a single data center switch is an alternative means of connectivity when the
data center switches are not joined together into a single virtual switch.
There are two PortChannel modes, static and dynamic. The dynamic mode uses LACP (Link
Aggregation Control Protocol) and requires a unique system priority (sysprio) configured on
the data center LAN switches. A DC LAN switch may have multiple LAG (Link Aggregation)
connections from various other systems; therefore, it is important to assign a unique sysprio
(aka, key) to your dynamic LAG. Assigning a sysprio is not required when configuring a static
LAG.
lacp --config -sysprio <priority>
lacp –-sysprio 781
portchannel --create <po_name> -type <static|dynamic> [-key
<po-number>]
portchannel –-create MYLAG1 -type dynamic –key 781
portchannel --add <po_name> -port <port-num|port-range>
portchannel –-add MYLAG1 –port ge2,ge3
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 181
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
IP Extension Compression
Compression operates within the scope of a tunnel. Compression cannot be configured per
circuit. Compression must be configured identically at both tunnel ends; asymmetrical
compression is not supported.
Fast-Deflate is not available for IP Extension on the IBM SAN42B-R and IBM SX6 Extension
Blade; IP Extension compression is limited to Deflate and Aggressive-Deflate. The IBM
SAN18B-6 does not have Fast-Deflate because the platform doesn’t support the data rates
requiring Fast-Deflate.
Compression can be configured specifically for each protocol (FCIP and IP Extension). For
example, on the IBM SAN42B-R or SX6 Blade, Fast-Deflate for FCIP and Deflate for IP
Extension can be configured together. Configuring fast-deflate for FCIP is considered best
practice in a hybrid environment because the Fast-Deflate and Deflate compression engines
are entirely different resources. Fast-Deflate uses an HW engine, and Deflate uses the
processor. 20 Gbps of FC ingress to Fast-Deflate does not consume any IP Extension
capacity available for Deflate or Aggressive-Deflate.
WAN side egress rate... Which compression algorithm to use for IP Extension?
Disabled (1:1) 2.5 Gbps(VE max 20 Gbps (VE max 20 Gbps (VE max
egress) egress) egress)
Deflate (3:1) 1.5 Gbps to 2.5 Gbps 10 Gbps to 15 Gbps 10 Gbps to 15 Gbps
Note: Compression ratios are approximate. IBM makes no warranties, guarantees, or claims about
the actual compression ratio achieved with customer-specific data.
The priority number order of TCL rules matters because rule processing stops upon the first
match. Rules are processed from lowest to highest priority number. The best practice is to
pick priority numbers by leaving adequate space between, in the event, another rule must be
added. For example, start at 10 and count by 10s when creating multiple rules. Traffic cannot
be matched to more than one tunnel because only a single tunnel can be specified in a rule
matching a flow. Traffic not matching any configured rule eventually reaches the bottom. The
last rule has the highest priority number (65535); it is a “deny all” rule and is not modifiable or
able to be deleted. Any traffic reaching the bottom is dropped.
Note: For TCL rules, the default admin mode is --admin disabled; therefore, ensure the rule’s
priority number has an “*” next to it, indicating that it is active; see the TCL example below.
182 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
The hit counter for each TCL rule identifies matching incoming LAN traffic. The hit counter
increases by 1 for every new TCP session formed (3-way handshake). Additionally, if the rule
matches a UDP, ICMP, or BUM (broadcast, unknown, multicast) packet, the hit counter counts
by 1 for every datagram. Besides some BUM traffic, for example, broadcasts, no traffic should
unintentionally arrive at the IP Extension gateway.
The TCL rules below are named PINGS and MYTCL; use human-readable nomenclature
meaningful to your environment. Priority 10 gives some space before (1-9) to add rules if
needed. For example, a temporary rule can be added for counting ICMP pings arriving at the
IP Extension gateway. Such a rule is useful for troubleshooting communications between
end-devices and the IP Extension gateway.
The example below shows production rules (priority 10) and a temporary rule for
troubleshooting by capturing pings (proto 1). The TCL hit counter registers any matched
pings. You can ping local LAN side IP addresses from the LAN, but you cannot ping local
WAN side IP addresses from the LAN. You can ping local WAN side IP addresses from the
WAN, but you cannot ping the WAN's local LAN side IP address. There is no need to delete
the temporary PINGS rule; however, it should be --admin disable when not being used.
portcfg tcl <name> <option> [<args>]
Example: Captures ICMP pings to see on hit counter
portcfg tcl PINGS create --priority 8 --target 12 --src-addr
10.10.10.0/24 --dst-addr 192.168.10.0/24 --admin enable --l4proto
icmp
Example: Local-SAN18B-6
portcfg tcl MYTCL create --priority 10 --target 12 --src-addr
10.10.10.0/24 --dst-addr 192.168.10.0/24 --admin enable
Example: Remote-SAN18B-6
portcfg tcl MYTCL create --priority 10 --target 12 --src-addr
192.168.10.0/24 --dst-addr 10.10.10.0/24 --admin enable
Local7810:admin> portshow tcl
Pri Name Flgs Target L2COS VLAN DSCP Proto Port Hit
Src-Addr Dst-Addr
--------------------------------------------------------------------------
*8 PINGS AI--- 12-Med ANY ANY ANY 1 ANY 29
10.10.10.0/24 192.168.10.0/24
*10 MYTCL AI--- 12-Med ANY ANY ANY ANY ANY 248223
10.10.10.0/24 192.168.10.0/24
*65535 default D---- - ANY ANY ANY ANY ANY 2949
ANY ANY
--------------------------------------------------------------------------
Flags: *=Enabled ..=Name Truncated (see --detail for full name)
A=Allow D=Deny I=IP-Ext P=Segment Preservation
R=End-to-End RST Propagation N=Non Terminated.
Active TCL Limits: Cur / Max
--------------------------------------------
DP0 3 / 32
--------------------------------------------
Configured Total: 3 / 256
Do not create an open TCL in which everything can communicate with everything, including
broadcast and multicasts. Sending broadcasts across the WAN is not considered best
practice. On the other hand, it is not required to close communications to only the devices'
exact IP addresses. Typically, specifying the entire end-device source and destination
subnets (i.e., -S 192.168.10.0/24 -D 10.10.10.0/24) works well.
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 183
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
It is imperative to understand TCL rules only live on the DP unto which the target VE_Port
exists. VE_Ports exist only in one DP, either DP0 or DP1. Refer to the TCL section for details.
IP Extension gateways are created on a specific DP, either DP0 or DP1. Ingress traffic
matching a TCL rule is directed to a particular target (VE_Port). If the VE_Port target
designated in a TCL rule is not congruent with the IP Extension gateway’s DP, the rule is not
evaluated. In other words, both the IP Extension gateway and target VE_Port used must be
on the same DP for the rule to be evaluated. If the rule is expected to be assessed but is not,
check if traffic is hitting an IP Extension gateway configured on the other DP.
TCL rules are evaluated once when the TCP session first connects during the TCP 3-way
handshake. After removing, adding, or making changes to TCL rules; the VE_Port for the
tunnel must be portdisable and portenable to cycle the tunnel. Cycling the VE_Port forces the
proxied TCP sessions to reform their connections and be evaluated by the updated TCL. If
bringing down the entire tunnel is too disruptive, for example, FCIP replication coexists within
the same tunnel; it is possible to disable all the LAN side GE interfaces. Wait until all the TCP
sessions have timed-out, then re-enable the GE interfaces. Another alternative is to disable
the IP storage replication ports on the end-device. The ultimate goal is to get the TCP
sessions to initiate new connections to engage the updated TCL.
The TCL is specific to IP Extension and has no bearing on WAN side IP routing or FC/FICON
side functions. WAN-side IP interfaces (ipif) starting with ge#.dp# use iproute to direct
outgoing traffic toward a specific router gateway. IP interfaces starting with lan.dp# use TCL to
match incoming LAN side traffic to one particular tunnel for IP Extension.
FICON
Native FICON, such as FICON tape and z/OS GM (XRC), is supported by the IBM SAN42B-R
and the SX6 Extension Blade. The Extension of FICON tape alongside TS7700 Grid is
referred to as the high availability architecture. The host on either side can access the
opposite side VTL if some portion of the infrastructure has failed.
When creating a tunnel for carrying FICON traffic, the FICON attribute must be specified
within the tunnel scope. For example, portcfg fciptunnel 24 create -F creates a FICON
tunnel on VE_Port 24.
It is essential to understand that array to array replication of mainframe volumes, for example,
Global Mirror between two DS8900, is not FICON. Even though the volumes are FICON, the
GM replication of those volumes is FCP. Tunnels do not need to be specified as FICON when
the replication protocol is FCP.
FICON connectivity requires the creation of a FICON LS (logical switch). For example, lscfg
--create 1 -ficon. This CLI command creates a FICON logical switch with the Fabric ID
(FID) of 1. The VE_Port must be moved into this context. The GE interfaces should stay in the
Default LS.
Note:
KATOV for FICON tunnels is automatically set to 1 second.
FICON is not supported on the IBM SAN18B-R.
For further information concerning FICON Administration, see the Brocade Fabric OS FICON
Configuration Guide at https://docs.broadcom.com/doc/53-1004394-02.
Virtual Fabrics
Virtual Fabrics (VF) are comprised of one or more Logical Switches (LS) within the physical
chassis. Connected Logical Switches form Logical Fabrics within the overall physical fabric.
184 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
VF plays a primary role in achieving deterministic paths for Extension protocol optimization.
Additionally, in mainframe environments, the implementation of FICON and FC Extension
often requires unique configurations and management, which is only achievable within the
same chassis when using different LS.
Protocol optimization requires all of an exchange’s sequences, and frames pass through the
same VE_Port pair both outbound and return. This deterministic path means that only a
single VE_Port can exist within a VF LS when enabling protocol optimization. Putting a single
VE_Port in an LS defines a single path between the two LS. An LS can sufficiently scale FC
port connectivity; they are operationally simplistic and create a stable environment. The IBM
SAN42B-R and SX6 Extension Blade support VF; the SAN18B-6 does not.
Extension + FCR
An Extension tunnel nearly always traverses an IP WAN, which has very different
characteristics than an FC network. An Extension tunnel crossing a WAN is essentially an FC
ISL over a WAN. An FCIP link is considered an ISL. A WAN link experiencing flapping can
disrupt a connected fabric if permitted to merge across the WAN. With each flap, disruption
from fabric services attempting to converge, converge again and converge yet again across a
WAN. The more devices logged-in to the fabric, the more difficult convergence is.
Convergence requires CPU and excessive convergence leads to an overload. If the CPU can
no longer process tasks promptly, instability may occur.
Limiting fabric services within a local edge-fabric and not permitting services to span the
WAN prevents large scale and taxing convergence. FCR is not required unless a production
SAN needs to be isolated from potential WAN instability. The best practice is to construct
completely separate production and replication SANs. FCR provides no benefit when only
array replication ports are directly connected to a dedicated replication SAN; there are no
production end devices to disrupt.
Isolated fabrics connected to FCR EX_Ports are referred to as “edge-fabrics.” EX_Ports are
the demarcation points in which fabric services do not extend. Fabric services do not pass
beyond an EX_Port, which forms an “edge.” FCR constrains fabric services to within either an
edge-fabric or the backbone.
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 185
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
IBM IBM
Extension Extension
with FCR with FCR
IP WAN
E_Port EX_Port Extension Tunnel
Extension Tunnel EX_Port E_Port
Backbone Fabric
Compression
There are three compression algorithms; refer above for more information on compression.
Specific to this section on FC/FICON is the Fast-Deflate algorithm. Fast-Deflate is only on the
IBM SAN42B-R and SX6 Extension Blade and only available to FC/FICON. Fast-Deflate is
not available for IP Extension or on the IBM SAN18B-6.
FC/FICON can use the Deflate (16 Gbps max ingress) and Aggressive-Deflate (10 Gbps max
ingress) algorithms. The compression ingress data rates must be accommodating to the
application and the desired WAN side bandwidth. If Deflate gets 3:1 compression, the WAN
side egress needs about 5 Gbps. If Aggressive-Deflate gets 4:1 compression, the WAN side
egress needs about 2.5 Gbps. The improper assignment of a compression algorithm can
result in a bottleneck.
If there is no intention of using IP Extension, the default is FCIP Only mode, which should be
maintained. Internal resources are partitioned between FCIP and IP Extension when in
186 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
Hybrid mode; therefore, it is best to remain in FCIP Only mode when not intending to use IP
Extension.
LVN-7840-B_CA:FID1:admin> extncfg --show
APP Mode is FCIP
VE-Mode: configured for 10VE mode.
GE-Mode: Not Applicable.
It is possible to assign multiple IP addresses and circuits to the same Ethernet interface by
assigning multiple ipif to the same interface, each with its unique IP address. The same IP
address cannot be configured more than once in the same box. An IP address, ipif, and circuit
can be associated only with one Ethernet interface. When more than one Ethernet interface is
needed, multiple circuits must be created.
Brocade Extension Trunk (BET) uses a single VE_Port with multiple circuits. Frequently,
circuits are assigned to different GE interfaces for the port, optic, and cable redundancy.
Ethernet interfaces do not need to be dedicated to one circuit.
Sharing an Ethernet interface with multiple circuits belonging to different VF LSs is possible.
The VE_Port is moved into the LS context using the lscfg CLI command. The tunnel and
circuits also reside in the LS context and are configured there. The Ethernet interfaces, ipif (IP
interface), and iproute (IP routes) must remain and be configured in the Default Switch
(context 128). An ipif is configured by designating the IP address, subnet mask, VLAN, MTU,
and GE interface. The creation of an ipif assigns the IP address to an Ethernet interface.
When the circuit is configured, the --local-ip binds the circuit to the Ethernet interface.
Ethernet Interface Sharing facilitates efficient use of the 40GE interfaces by allowing multiple
VE_Ports living in multiple Logical Switches to use the large bandwidth connection.
Figure 7-32 shows an example of two VF LS (10 and 20). GE0 and GE1 are 40 Gbps
interfaces and in the Default Switch. There are two circuits per BET, the red BET, and the blue
BET. VE24 (lives in DP0) has one circuit that goes to GE0 and one circuit that goes to GE1.
VE34 (lives in DP1) has one circuit that goes to GE0 and one circuit that goes to GE1. There
are two 10 Gbps circuits within each 40GE interface, each from a different LS. The 40GE
interfaces are physically connected to respective A & B DC LAN switches. Enough 40GE
bandwidth remains to create additional backup circuits (metric 1 circuits) if a 40GE link goes
offline.
In Figure 7-33 on page 188, Ethernet Interface Sharing is used in an architecture where the
40GE interfaces are used for physical connectivity to the data center LAN. Circuits from 3
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 187
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
tunnels pass through the physical Ethernet links. The traffic on each circuit is VLAN tagged,
so the data center LAN switches can forward the traffic into specific DWDM circuits.
This section covers points unique to IP Extension design, best practice, and architectures.
Use Cases/Purpose
IP Extension use cases include acceleration, encryption, bandwidth management, data
migration, network visibility, troubleshooting tools, and congestion avoidance. Figure 7-34 on
page 189 shows a high availability architecture in an IBM System Z® with TS7700
environment. Another IP Extension solution is IBM Transparent Cloud Tiering (TCT). Data is
transferred between VTLs in different data centers. Data is replicated quickly and securely to
maintain a coherent RPO (Recovery Point Objective).
188 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
FICON
IBM z Systems with TS7700 Grid IBM TS7700 Grid
BET Top
BET Bottom
IBM IBM
TS7700 TS7700
IBM z
10Gb/s IBM z
10GE
IP WAN IP WAN 10GE
DS8900 DS8900
Service
Provider(s)
IBM IBM
TS7700 TS7700
If there is no intention of using IP Extension, the default is FCIP Only mode, which should be
maintained. Internal resources are partitioned between FCIP and IP Extension when in
Hybrid mode; therefore, it is best to remain in FCIP Only mode when not intending to use IP
Extension.
7840-Side-A:FID128:admin> extncfg --show
APP Mode is HYBRID (FCIP with IPEXT)
VE-Mode: configured for 10VE mode.
GE-Mode: Not Applicable.
VLANs
Ethernet frames can be tagged (IEEE 802.1Q) or untagged. Ethernet ISLs with tagging
enabled are called trunks, which incorporates traffic from multiple VLANs. Respective VLAN
traffic is differentiated via the tags. If IP Extension traffic lives across multiple VLANs, only a
single set of trunked links needs to be connected. All the VLANs can communicate across a
single PortChannel of trunks. An IP Extension gateway is configured explicitly for each subnet
in each VLAN; these are the same subnets that the end-device replication ports are on. There
may be many IP Extension gateways depending on how many subnets are being used. Eight
is the maximum number of IP Extension gateways per DP.
If tagging is configured on the data center LAN switch, it must be configured on the Extension
platform. A tagging mismatch causes Ethernet links not to come online. VLAN ID mismatches
cause no communication; however, the links come online. On the Extension platform,
configuring VLAN tagging is accomplished by adding the VLAN ID when creating the ipif
(portcfg ipif). In the example below, an IP Extension Gateway (10.0.0.2/24) is created and
receives incoming and sends outgoing traffic only on VLAN 10. When creating the ipif, the
MTU can be specified as well; the default MTU is 1500 bytes.
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 189
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
If the IP storage traffic is on a single VLAN, there is no reason to configure VLAN tagging. The
data center LAN switch ports should be members of the VLAN without tagging. The Ethernet
links should not be trunks. A LAG of untagged Ethernet links can be established between the
LAN switches and the Extension platform.
IP Extension Gateway
IP Extension is the gateway for data crossing the WAN between data centers. If IP storage
traffic is not forwarded to the IP Extension gateway, it will not utilize IP Extension.
Figure 7-35 shows an example of traditional data center routing. Traffic comes from the IP
storage cluster to the data center LAN switch. The LAN switch forwards it to the traditional
gateway. The router could be an integral part of the LAN switch, or not. The router forwards
the traffic onto the WAN towards the destination.
Traditional
Gateway
WAN Router
WAN
DC LAN Switch
IP Extension requires the end-device to direct traffic to its gateway, usually by static route or a
gateway setting. Static routes are more specific than a default route; therefore, traffic for that
particular destination subnet is forwarded to the IP Extension gateway instead of the default
gateway. The destination subnet and IP Extension gateway are specified in the static route.
190 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
The default route stays pointed to the traditional router gateway and is used when a more
specific route does not exist.
Keep in mind, putting IP Extension in the path and removing it from the path merely involves
activating or deactivating these end-device static routes. With static routes, the traffic goes to
the IP Extension gateway (IP Extension in the path). Without static routes, traffic would go to
the traditional router (IP Extension out of the path).
IP Extension supports the connectivity of multiple data center LANs using VLAN tagging
(802.1Q). Tagging frames on Ethernet links connecting the Extension platform and the data
center LAN switches is called trunking. An IP Extension gateway (ipif lan.dp# with vlan
<ID>) must be configured for each subnet trunked to the Extension platform.
In the example below, two ipif are created, one for each DP. The same subnet can be used;
however, the gateways are different. The Ethernet trunks connected to the LAN interfaces
must be tagging frames for VLAN 100 and VLAN 200. IP Extension gateways 10.10.10.2 and
10.10.10.3 are listening only to VLAN 100 traffic, and 192.168.10.2 and 192.168.10.3 are
listening only to VLAN 200. Of course, in your environment, the VLANs will be different.
On the IBM SAN42B-R and SX6 Extension Blade, there are two DPs (data processors). The
IP Extension gateway on each DP must be unique; the same gateway cannot be used on both
DPs. VRRP (Virtual Router Redundancy Protocol) is not supported on the IP Extension
platforms. Only one IP Extension gateway can be configured per subnet on each DP. If
planning to use both DPs, the storage application end devices must either be segregated into
groups that use each IP Extension gateway, or replication ports on the end devices must be
capable of independent gateway settings.
It is imperative to understand TCL rules only live on the DP unto which the target VE_Port
exists. VE_Ports exist only in one DP, either DP0 or DP1. Refer to the TCL section for details.
IP Extension gateways are created on a specific DP, either DP0 or DP1. Ingress traffic
matching a TCL rule is directed to a particular target (VE_Port). If the VE_Port target
designated in a TCL rule is not congruent with the IP Extension gateway’s DP, the rule is not
evaluated. In other words, both the IP Extension gateway and target VE_Port used must be
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 191
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
on the same DP for the rule to be evaluated. If the rule is expected to be assessed but is not,
check if traffic is hitting an IP Extension gateway configured on the other DP.
IP storage traffic forwarded to the IP Extension gateway is first evaluated by the TCL (Traffic
Control List), as shown in Figure 7-36. If a flow matches an allow rule, the traffic is forwarded
to the specified tunnel (target). The data is handled by the WAN side of the Extension
platform for delivery to the remote data center.
IP Extension Gateway
Flows to remote data center go here IP Extension
Tunnel WAN
Router
Note: The LAN side IP subnet(s) used on the IP storage end-devices cannot be the same
in both the local and remote data centers. Different subnets for the end devices in each
data center is required.
TCP Flows/Sessions/Streams
IP Extension operates entirely at layer 4 using TCP. Flows, sessions, connections, and
streams are synonymous TCP terms; each represents a connection established by a
successful TCP 3-way handshake. IP Extension regulates ingress data from storage
applications using TCP receive window flow control. By controlling ingress data, IP Extension
expedites data needed by the compression engine and manages data transmission alongside
FCIP flows and flows within a shared WAN. IP Extension maintains separate virtual TCP
sessions within WAN Optimized TCP to manage flows and prevent head of line blocking
individually. IP Extension accommodates a limited number of TCP flows from IP storage end
devices; refer below to the maximum number of supported flows per platform:
IBM SAN18B-6256 TCP sessions
IBM SAN42B-R512 TCP sessions per DP (two DPs)
IBM SX6 Extension Blade512 TCP sessions per DP (two DPs)
IP Extension uses proxy TCP sessions to locally terminate and acknowledge traffic; this is
part of TCP Acceleration. These proxied TCP sessions are local to the data center and can
operate as fast as the end device can send data. WAN Optimized TCP, an aggressive
purpose-built Extension TCP stack, is leveraged to transport large storage quantities across
long-distance WAN connections. Data is repackaged, optimized, and overhead lowered by
IBM b-type Gen 7 High-Efficiency Encapsulation for WAN side delivery.
Not all flows need to be terminated and optimized. For example, the control traffic is not
benefitted by optimization, yet, in many cases, it consumes a valuable TCP session. Most of
the time, control traffic sits idle or produces negligible amounts of traffic. The IBM TS7700 grid
consumes many IP Extension TCP sessions and may exhaust all of them. IBM TS7700 tape
grid generates a few TCP sessions for transporting tape data, but in a cluster generates
hundreds of control TCP sessions. Tape flows generate large data quantities, are
time-sensitive, are rarely idle, and benefit from IP Extension. IBM TS7700 control traffic (TCP
ports 1415 and 1416) uses a different TCP port than data traffic (TCP port 350). To
accommodate this, IP Extension has TCL options for matching particular traffic, enabling
192 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
control traffic to be handled differently by not terminating it and allowing it to pass without
consuming an IP Extension TCP session. Refer to the example TCL below.
7840-Side-A:FID128:admin> portcfg tcl MyTS7700_CTL create --priority 100 --target
24 --src-addr 10.10.10.0/24 --dst-addr 192.168.10.0/24 --proto-port 1415-1416
--non-terminated enabled --admin enable
Operation Succeeded.
7840-Side-A:FID128:admin> portcfg tcl MyTS7700_TAPE create --priority 200 --target
24 --src-addr 10.10.10.0/24 --dst-addr 192.168.10.0/24 --admin enable
Operation Succeeded.
7840-Side-A:FID128:admin> portshow tcl
Pri Name Flgs Target L2COS VLAN DSCP Proto Port
Hit
Src-Addr Dst-Addr
----------------------------------------------------------------------------------
-
*100 MyTS7700_CTL AI--N 24-Med ANY ANY ANY ANY 1415-1416 0
10.10.10.0/24 192.168.10.0/24
*200 MyTS7700_TAPE AI--- 24-Med ANY ANY ANY ANY ANY 0
10.10.10.0/24 192.168.10.0/24
*65535 default D---- - ANY ANY ANY ANY ANY 0
ANY ANY
----------------------------------------------------------------------------------
-
Flags: *=Enabled ..=Name Truncated (see --detail for full name)
A=Allow D=Deny I=IP-Ext P=Segment Preservation
R=End-to-End RST Propagation N=Non Terminated.
Active TCL Limits: Cur / Max
--------------------------------------------
DP0 3 / 128
DP1 1 / 128
--------------------------------------------
Configured Total: 3 / 1024
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 193
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
VE_Ports
VE_Ports for IP Extension are merely representative of the tunnel ID. Unlike FCIP, no IP
Extension data flows through a VE_Port. Although, disabling a VE_Port will disable the entire
tunnel, even if only configured for IP Extension. The VE_Port must be up for the tunnel to be
up.
The best practice is to use the same VE_Port for FCIP and IP Extension between the same
two domains; tunnels are point-to-point. The desire is to have the VE_Port manage the
bandwidth of both protocols across the tunnel. Bandwidth management cannot be
accomplished when using different VE_Ports, one for each protocol.
It is imperative to understand TCL rules only live on the DP unto which the target VE_Port
exists. VE_Ports exist only in one DP, either DP0 or DP1. Refer to the TCL section for details.
IP Extension gateways are created on a specific DP, either DP0 or DP1. Ingress traffic
matching a TCL rule is directed to a particular target (VE_Port). If the VE_Port target
designated in a TCL rule is not congruent with the IP Extension gateway’s DP, the rule is not
evaluated. In other words, both the IP Extension gateway and target VE_Port used must be
on the same DP for the rule to be evaluated. If the rule is expected to be assessed but is not,
check if traffic is hitting an IP Extension gateway configured on the other DP.
The most common data protection scheme used by enterprise customers is disk replication
or mirroring. There are and continues to be other methods for protecting data, but
array-based mirroring is one of the most commonly used methods in the enterprise market
today. The IBM Extension platforms are designed to excel in this environment, delivering the
most outstanding data protection and performance regardless of distance.
With the ever-increasing amount of data that needs to be protected and the increase in WAN
speed from GE to 10GE to 100GE, many older IBM Extension platforms are reaching the end
of their product support lifecycle. For those charged with business continuance/disaster
recovery for their enterprise data, it would be derelict not to review the current BC/DR
architecture to avoid an unsupported platform situation. This section describes the general
principles of migrating and refreshing a current environment to a new, modernized Extension
platform.
194 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
design flexibility and more cost-effective solutions. SX6 Extension tunnels are interoperable
with the IBM SAN18B-6 and SAN42B-R.
For clients requiring high port density for both FC connectivity and Extension tunnel
bandwidth but may not be deploying directors, IBM offers a 2RU platform called the IBM
SAN42B-R Extension Switch. The IBM SAN42B-R and SX6 Extension Blade are considered
enterprise-class systems and support all the solutions for both open systems and mainframe
(FICON) environments.
The third offered Extension platform is the IBM SAN18B-6 Extension Switch; its predecessor
was the IBM SAN06B-R. The IBM SAN18B-6 is a 1RU system that incorporates many of the
advanced functions found in the SX6 and SAN42B-R. The SAN18B-6 can be integrated into
enterprise-class solutions, but it also supports clients who do not need nor require high port
densities and large Extension tunnel bandwidths. The IBM SAN18B-6 supports all disk
replication solutions but does not support specific mainframe solutions implementing FICON
or emulation techniques.
The IBM SAN06B-R platform was introduced over a decade ago and has now entered End of
Life (EOL). There are thousands of production units today, and those should be refreshed
before the End of Service (EOS) date on October 31, 2024. The remainder of this section
focuses on the methods and strategies for refreshing the IBM SAN06B-R infrastructure.
The most common Extension deployments contain redundant fabrics, dual paths connecting
the storage array through an IP network to a remote site. Having two paths between data
centers offsets the impact if one path fails. Redundancy and data path protection are intrinsic
to building the most resilient infrastructure.
Users should have different IP WAN service providers when redundant paths are needed.
Different WAN service providers create a tremendous amount of additional protection by
having wholly separate and disparate routes between sites. History has shown that users who
do not confirm that the paths are entirely separate and disparate likely will be impacted during
an outage event such as a backhoe digging for a new sewer line. Figure 7-37 shows the “dual
fabric” configuration.
B B
BET BET
Brocade Brocade
Extension Extension
Production Backup
Site Site
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 195
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
The example dual fabric replication SAN shows how a storage array on the left has two
separate and isolated paths. Each IBM SAN18B-6 makes up replication Fabric A and Fabric
B. Those paths are “paired” with the remote storage array on the right side. Note, the paths
are paired A-to-A and B-to-B. Each Extension switch has two Ethernet connections to the IP
network. The IP network has redundant switches, routers, and WAN connections. By
designing a network like this, you can protect your replication scheme from single points of
failure. This example of a replication architecture is the most commonly deployed.
One of the most stressful parts of migrating to a new Brocade Extension platform is the
question of “will I need to take an outage, and if so, how long will that last?” The short answer
is “maybe.” Users can minimize replication downtime when following proper preparation and
methods.
To remove nearly all the risk of taking a full outage, you should plan to stage and configure
your new platforms to mimic as close as possible the same configurations being used today.
Of course, suppose you are going through a consolidation effort, increasing the number of
WAN connections, or upgrading GE to 10GE Ethernet. In that case, there will be additional
steps needed to integrate those changes. For details on the changes and upgrades, see
Brocade Fabric OS Extension User Guide at
https://docs.broadcom.com/doc/FOS-90x-Ext-UG
In general, the following migration methodology and techniques can be applied to any
platform refresh strategy. However, this section focuses on migrating from the IBM SAN06B-R
to the SAN18B-6 platform. This migration section is not intended to be a step-by-step
technical guide but should be used as a planning guideline. For a detailed technical “how-to”
guide on migrating from the IBM SAN16B-R to the IBM SAN18B-6, see Brocade 7800
Extension to Brocade 7810 Extension Migration User Guide at
https://docs.broadcom.com/doc/7800-7810-Migrate-UG
A benefit of having a “dual fabric” architecture is to provide a seamless transition from one
platform to another. By leveraging one of the fabrics (or paths), replication can be maintained
without taking a full outage.
The following are guidelines for migrating to a new IBM Extension platform:
Preliminary work: Run Brocade SAN Health to capture a snapshot of your environment to
see if there are any current “network health” problems requiring attention before the
migration. For more information, see Brocade SAN Health at
https://docs.broadcom.com/doc/SAN-Health-FAQ
Install and apply power to the IBM SAN18B-6 at both sites, leave the storage and network
cables disconnected.
196 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
Gather required information from the SAN06B-R for building and configuring the IBM
SAN18B-6. Record the output from these CLI commands.
– Local IP addresses and subnet masks (portshow ipif)
– Gateways and remote subnets (
– portshow iproutes)
– Min and Max bandwidth rates (
– portshow fciptunnel -c)
Configure the IBM SAN18B-6 at both sites with the configuration information gathered
from the IBM SAN18B-6. There are two ways to configure the Brocade 7810s.
– IBM SANnav Management Tool
– CLI
At this point, the Extension tunnel on the IBM SAN18B-R should have a status of “Up.” If
not, do not proceed. The tunnel must be operational before proceeding. Refer to the
validation section of the Extension section to troubleshoot the tunnel and circuits.
Select which fabric you intend to migrate first, and, if required by the storage vendor’s
process, suspend the path on the storage array’s replication interface(s).
– Review your storage vendor’s administration guide for the proper process to take down
one or more replication paths, etc.
– With a dual fabric design, the other fabric/path continues replication; however, there
may be some backlog in replication with half the bandwidth if there is a heavy
workload.
For the first path to be migrated, disable power to the IBM SAN06B-R at both sites.
Remove storage and network cables and reattach the cables to the IBM SAN18B-R.
Check for Extension tunnel status and error messages.
– Note: The new path should be online and able to replicate
– You
– can skip the WAN Test Tool validation if you feel confident that the IP network's load
testing is unnecessary.
If you wish to confirm that the IP WAN is capable of supporting your bandwidth SLA,
disable the circuit(s), create Wtool sessions, and run Wtool at the bit rate you expecting on
the WAN. Wtool should be run for at least 10 minutes. Review the results.
Once the path has been verified, disable the Wtool sessions, and enable the circuit(s) that
were disabled.
Once the Extension tunnel has been made operational, enable storage replication
interfaces.
Monitor replication and the network for errors, and confirm everything is operating as
expected.
Once the initial fabric/path is working as expected, repeat these steps on the other fabric.
As you can see, by taking one path down at a time, the user can reduce or eliminate the need
for a complete outage.
Closing out the guidelines for migrating from an IBM SAN06B-R to a SAN18B-6, it should be
noted that a newer SAN Management tool is available. The IBM Network Advisor, the legacy
SAN Management tool for managing the IBM SAN06-R, is no longer on the market. IBM
SANnav SAN Management tool has a new look and feel, and it has been wholly
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 197
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
re-architected from scratch for the next generation of SAN infrastructure. IBM SANnav
comprises two management portals for local and global views to monitor and manage small
to large environments. For more information on IBM SANnav, see Brocade SANnav
Management Portal User Guide at https://docs.broadcom.com/doc/SANnav-21x-MP-UG.
It is also essential to understand the need for product life cycle management and the eventual
migration to newer supported technologies. As you continue to deploy new workloads and
enhance protection requirements with solutions like Remote Disk Replication (RDR), avoid
explaining sudden budget requirements for refreshing replication infrastructure when it’s
imminent—or beyond—the ended support date.
When migrating from the IBM SAN16B-R (Brocade 7800) to the IBM SAN18B-6 (Brocade
7810), there are a few considerations. Extension technology has evolved; features, functions,
and the CLI have changed. Notable changes include IPsec and encryption, IP Extension,
licensing, WAN side bandwidth, and various CLI syntax and references.
198 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
Best practice is to perform a comprehensive evaluation and baseline of the IP network before
placing an Extension tunnel into production. The evaluation can run for a few minutes, days,
or longer depending on the important time period of interest for testing the IP network, for
example during end of quarter processing. Often the IP network is not what is expected.
An existing circuit must be used with Wtool; however, the circuit must be taken offline before
the associated Wtool session can be enabled. Other circuits can remain in production. Only
disable the circuits you wish to test at one time. Wtool is configured on both ends, as with any
circuit. The Wtool session bit-rate must be configured; the circuit ARL settings are not used.
After completing the Wtool evaluation, the sessions can merely be disabled, and the circuits
enabled again. Sessions can be used again later; there is no need to delete them. Upon
re-enabling the circuits, they will be back in production.
Wtool reports the amount of local data sent and remote data received to determine the IP
network's characteristics and reliability, including packet drops, latency (RTT), jitter (RTT Var),
and out-of-order segments plus more. Typically, one end is the data source, and the other end
is the data sink; however, bidirectional traffic is supported. The process continues for the
user-specified duration or until keyboard terminated (Ctrl+C).
For guidance on setting up and using Wtool, see the Brocade 7810 IP Extension Deployment
Guide at https://docs.broadcom.com/doc/7810-IPEX-UG
portcmd --ping
The portcmd --ping command tests simple connectivity between a local IP address (ipif) on a
particular GE interface and a destination IP address. ICMP ECHO protocol is used. The
portcmd --ping requires designating which GE interface the ping will be sent from, this can be
lan.dpx or ge.dpx interfaces. The source IP address for the ping must already exist on the
specified lan or ge interface.
A LAN side ping to an end device must be on the same subnet as the IP Extension gateway.
Pinging a different subnet requires a configured iproute on the Extension platform, and the
destination device may also require a complimenting return route.
In this example, a ping was done from the ipif lan.dp0 IP Extension gateway to the traditional
router gateway on the same VLAN. Ping verifies connectivity and communications from the IP
Extension gateway through the data center LAN switch.
LVN-7840-A:FID128:admin> portcmd --ping lan.dp0 -s 10.175.1.248 -d 10.175.1.1
PING 10.175.1.1 (10.175.1.248) with 64 bytes of data.
64 bytes from 10.175.1.1: icmp_seq=1 ttl=255 time=37 ms
64 bytes from 10.175.1.1: icmp_seq=2 ttl=255 time=7 ms
64 bytes from 10.175.1.1: icmp_seq=3 ttl=255 time=1 ms
64 bytes from 10.175.1.1: icmp_seq=4 ttl=255 time=100 ms
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 199
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
If no local or return IP route exists, ping a destination within the subnet, such as the local
gateway, typically .1. A native subnet ping demonstrates if L2 (Ethernet level) connectivity is
operational through the connected data center switch. If local and return IP routes exist, you
can ping to a foreign subnet destination; however, the remote device must have a
complimenting IP return route to reply. A foreign subnet ping demonstrates if L3 (IP level)
connectivity is operational.
Portcmd --ping [<sx6 slot>/]<port.dp#> <options>
Example ping:
Local7810:admin> portcmd --ping ge6.dp0 -s 172.16.1.3 -d 172.16.2.3
1. PING 172.16.2.3 (172.16.1.3) with 64 bytes of data.
64 bytes from 172.16.2.3: icmp_seq=1 ttl=250 time=19 ms
64 bytes from 172.16.2.3: icmp_seq=2 ttl=250 time=19 ms
64 bytes from 172.16.2.3: icmp_seq=3 ttl=250 time=19 ms
64 bytes from 172.16.2.3: icmp_seq=4 ttl=250 time=19 ms
-- 172.16.2.3 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 789 ms
rtt min/avg/max = 19/19/19 ms
Note: The ping command is used for the mgmt interface only. portcmd --ping is used for the GE
interfaces.
Traceroute should be used on L3 routed networks with 2 or more hops. The example below
has two hops, first to the router, second to the Extension platform. The intermediate and
remote devices must have a complementing IP route to return the traceroute response. It is
not unusual for intermediate devices owned by service providers to not return traceroute
responses.
7840-A:FID128:admin> portcmd --traceroute ge3.dp0 -s 10.2.0.10 -d 10.1.0.10
traceroute to 10.1.0.10, 64 hops max, 64 byte packets
1 10.2.0.1 59 ms 59 ms 59 ms
2 10.1.0.10 59 ms 59 ms 59 ms
nsShow
When connecting end devices to an FC fabric, they log into the fabric. nsshow CLI command
shows the devices that have logged into the fabric and information about those devices. If the
device has successfully logged into the fabric, it is connected and communicating with the
switch.
7840-Side-A:FID128:admin> nsshow
{
Type Pid COS PortName NodeName SCR
N 0a0000; 3;10:00:00:10:9b:34:6e:e0;20:00:00:10:9b:34:6e:e0; 0x00000003
SCR: Fabric-Detected Nx-Port-Detected
FC4s: FCP
PortSymb: [34] "Emulex PPN-10:00:00:10:9b:34:6e:e0"
NodeSymb: [104] "Emulex LPe32002-M2 FV11.4.204.20 DV11.4.33.1
HN:Lenx3650M5-212999.dhcp.broadcom.net OS:VMware ESXi 6.7.0"
Fabric Port Name: 20:00:c4:f5:7c:3b:24:86
Permanent Port Name: 10:00:00:10:9b:34:6e:e0
200 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
switchShow
Switchshow is useful for an overall summary of the fabric, domain, FC/FICON ports, state,
connected devices, VE_Ports, GE interfaces, and logical switch context.
7840-Side-A:FID128:admin> switchshow
switchName: 7840-Side-A
switchType: 148.0
switchState: Online
switchMode: Native
switchRole: Principal
switchDomain: 10
switchId: fffc0a
switchWwn: 10:00:c4:f5:7c:3b:24:86
zoning: OFF
switchBeacon: OFF
FC Router: OFF
Fabric Name: REPLICATION
HIF Mode: OFF
Allow XISL Use: OFF
LS Attributes: [FID: 128, Base Switch: No, Default Switch: Yes, Ficon Switch: No,
Address Mode 0]
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 201
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
202 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
fabricShow
When making connections between domains, such as FCIP connections, fabricshow
displays the merged switches that have joined the fabric. In this example, two switches with
domains 10 and 20 have joined fabric REPLICATION. The mgmt IP address of each switch
are shown. Before FCIP traffic will flow between connected end devices (end-to-end), the two
domains connected by FCIP must first form a fabric.
7840-Side-A:FID128:admin> fabricshow
Switch ID Worldwide Name Enet IP Addr FC IP Addr Name
------------------------------------------------------------------------
10: fffc0a 10:00:c4:f5:7c:3b:24:86 10.155.2.221 0.0.0.0 >"7840-Side-A"
20: fffc14 10:00:c4:f5:7c:3b:25:06 10.155.2.220 0.0.0.0 "7840-Side-B"
The Fabric has 2 switches
Fabric Name: REPLICATION
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 203
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
portShow
Portshow displays detailed information about FC/FICON ports. In the example below, FC port
0 is shown.
7840-Side-A:FID128:admin> portshow 0
portIndex: 0
portName: port0
portHealth: HEALTHY
Authentication: None
portDisableReason: None
portCFlags: 0x1
portFlags: 0x20b03 PRESENT ACTIVE F_PORT G_PORT U_PORT LOGICAL_ONLINE LOGIN
NOELP ACCEPT FLOGI
LocalSwcFlags: 0x0
portType: 24.0
portState: 1 Online
Protocol: FC
portPhys: 6 In_Sync portScn: 32 F_Port
port generation number: 0
state transition count: 1
portId: 0a0000
portIfId: 43020028
portWwn: 20:00:c4:f5:7c:3b:24:86
portWwn of device(s) connected:
10:00:00:10:9b:34:6e:e0
16b Area list:
Distance: normal
portSpeed: N16Gbps
FEC: Inactive
Credit Recovery: Inactive
Aoq: Inactive
FAA: Inactive
F_Trunk: Inactive
LE domain: 0
Peer beacon: Off
FC Fastwrite: OFF
Interrupts: 22 Link_failure: 0 Frjt: 0
Unknown: 2 Loss_of_sync: 0 Fbsy: 0
Lli: 22 Loss_of_sig: 2
Proc_rqrd: 33 Protocol_err: 0
Timed_out: 0 Invalid_word: 0
Tx_unavail: 0 Invalid_crc: 0
Delim_err: 0 Address_err: 0
Lr_in: 2 Ols_in: 0
Lr_out: 0 Ols_out: 2
204 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
Various dashboards can be constructed to fit your environment’s needs and interests. Top
port traffic, top port utilization, top buffer-buffer credit at zero, REPLICATION fabric health,
configuration drifts, and other top fabric events are monitored, as shown in Figure 7-38. The
widgets have a pulldown menu, and a wide variety of other statistics and errors can be
monitored. Additionally, more rows and widgets can be added.
SANnav Health Summary is shown in Figure 7-39 on page 206. The overall status of the
fabrics, switches, hosts, and storage is indicated. In this case, there is one replication fabric;
typically, there would be two or more replication fabrics (for example, A & B). If SANnav were
to monitor the production SAN as well, those fabrics and devices would be included.
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 205
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
In Figure 7-40, a simple two-site replication SAN is shown. In many practical instances, the
replication SAN is autonomous from the production SAN. The topology view can narrow down
to a particular fabric for simplicity and investigation. The FOS fabricname has been
configured as “REPLICATION.” In SANnav, this topology has been saved as REPLICATION,
as well. There is a single tunnel using VE24.
206 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
In Figure 7-42, IBM SANnav is monitoring the traffic going through physical GE interfaces.
The GE interfaces show what quantity of traffic is being sent to the directly connected data
center switches. It does not explicitly show tunnel or circuit traffic statistics. The GE interfaces
are merely abstracted portals through which circuits pass; there may be zero to many circuits
per GE interface.
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 207
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
IBM SANnav is monitoring the Extension Tunnel with selected characteristics, as shown in
Figure 7-44. Compression for this particular data is shown to be approximately 2.3:1. Latency
is being shown at 20 ms RTT. The Extension platform is detecting out-of-order segments,
which is indicative of packet loss on the network. Transmit utilization and MB/s are monitored.
Traffic is being sent from 7840-Side-A (switch1) to 7840-Side-B (switch2). The Receive traffic
consists of mere acknowledgments and has been chosen not to be monitored here.
IBM SANnav is monitoring the Extension Circuits with selected characteristics, as shown in
Figure 7-45 on page 209. Transmission in MB/s is shown. Rx MB/s was not selected because
it shows very little traffic consisting of acknowledgments; data is replicated from switch1 to
switch2. Four measurements can be enabled at one time, while two circuits are monitored.
Mouse-over the charts displays the legend, as shown.
208 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
A customized Extension report (see Figure 7-46) can be constructed from the most
interesting widgets for your environment. MAPS (Monitoring Alerting Policy Suite) monitors
every aspect of the Extension platform, and it is prudent to be aware of MAPS violations. Over
long operational periods, a WAN often experiences interims of low quality. By monitoring
packet drop rates, RTT stability, increases in jitter, and reduced tunnel utilization; operations
can drive inquiries about observed changes with the Network Administrators and potentially
preempt further degradation or an entire outage.
Your customized Extension Report with selected statistics can be generated at regular
intervals, as shown in Figure 7-47 on page 210. For example, if you wish to have the report
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 209
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
generated daily at midnight, waiting for you in the morning, this can be configured. The
formats that can be delivered are HTML, PDF, and CSV.
7.6.4 SNMP
SNMPv1 and SNMPv3 can be configured to send traps or have statistics read by other
management tools. Below is the output of snmpconfig for snmpv1 traps.
BCM-7840-A:FID128:admin> snmpconfig --show snmpv1
SNMPv1 community and trap recipient configuration:
Community 1: Secret C0de (rw)
No trap recipient configured yet
Community 2: OrigEquipMfr (rw)
No trap recipient configured yet
Community 3: private (rw)
No trap recipient configured yet
Community 4: mySecret1 (ro)
Trap recipient: 10.10.10.10
Trap port: 162
Trap recipient Severity level: 4
Community 5: mySecret2 (ro)
210 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch07.fm
For more details, see the Brocade Fabric OS MIB Reference Manual at
https://docs.broadcom.com/doc/FOS-90x-MIB-RM.
Chapter 7. IBM b-type Gen 7 extension design, implementation, and migration 211
8497ch07.fm Draft Document for Review April 5, 2021 12:31 pm
212 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch08.fm
In addition to the fact that the NVMe software stack is designed for memory devices and so
avoids all of the legacy software necessary for dealing with rotational media, disk drive
geometries and so on, the definition also includes a level of parallelism that is very different
than the traditional SCSI, Serial Attach SCSI (SAS) or Serial ATA (SATA) environments for
storage. Those older protocols traditionally have a single command queue with a more limited
queue depth. In the case of SCSI a maximum of 64 commands. In the NVMe standard there
214 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch08.fm
is support for 64K (65,535) I/O queues each with 64K (65,535) commands which is commonly
referred to a queue depth or the number of outstanding commands. It is important to keep in
mind that use of this parallelism will continue to be developed by operating systems and
applications. The following diagram provides a view of how the parallelism might be viewed.
NVMe queues are mapped to CPU cores which allows a very scalable performance model.
Over time this parallelism may enable changes where virtualization platforms can assign an
NVMe queue to each virtual machine.
When NVMe was first brought to market it was almost exclusively implemented as an
embedded device directly on the PCIe bus internal to the server. Basically, it was intended as
an expansion of system memory that traded off the higher latency that NVMe has compared
to DRAM for a lower cost versus DRAM. At a device level common NVMe products use 4 x
Gen 3 PCIe lanes (4 x 8Gb/s = 32Gb) at a device latency of sub 20 microseconds. Eventually
the market determined that the following characteristics made the need for NVMe over Fabric
a desirable goal.
The ability to scale memory capacity past that which could be achieved on a single card
The desire not to burden every single server with the cost of an NVMe device
The create a shared resource pool of NVMe that could be accessed by multiple servers
The desire for platform reliability where single server failure/reboot would not affect the
application
The NVMe Consortium, as previously mentioned, is also responsible for the efforts around
NVMe over Fabrics. A part of that specification is the NVMe Transport binding specification.
The NVMe Transport binding specification for Fibre Channel is defined in INCITS 540 Fibre
Channel – Non-Volatile Memory Express (FC-NVMe) at http://www.incits.org/.
I/Os Exchanges
IO Q creation ELS
Controller Target
Namespace LUN
Rd Read
Wr Write
When architecting a networked environment for NVMe it is also important to remember that
the technology is still expanding in terms of functionality and support. Many of these
capabilities will become available over time in the market and some of those will be made
available through software updates to the existing releases. Some of these have already been
released such as Asymmetric Namespace Access (ANA) for multipathing (the equivalent of
the ALUA in SCSI) and Sequence Level Error Recovery (SLER) while others will take some
time to be implemented.
At the time this is being written, NVMe has 13 required commands and 25 optional. While it is
certainly to be expected that augmentation of functionality over time will increase the number
of commands it provides an exceedingly streamlined interface as compared to the SCSI
software stack which over 30+ years has accommodated not only storage devices but all
sorts of other peripherals (e.g., printers, scanners).
Figure 8-5 on page 217 shows a simplified view of the comparison between the SCSI
software stack and the NVMe software stack. In addition to the fact that NVMe is specifically
written to access memory devices instead of traditional SCSI disk drive elements the software
layer has been streamlined to address the block layer directly yielding a significant increase in
performance.
216 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch08.fm
It is in part this simplified software stack that produces the reduction in IO overhead observed
in the Intel whitepaper referenced previously. It is expected that over time the NVMe interface
will become the preferred software interface for enterprise storage. While this won’t happen
overnight given the millions and millions of lines of code and product implementations for
SCSI that are already in use, the benefits of the NVMe software performance will eventually
become dominant. This will be especially true for the All Flash Array environments in the
enterprise as organizations seek to get a full utilization of the performance of the assets. This
is also one of the considerations in choosing the type of fabric to use for the storage. It is easy
in the planning stages to assume that “all bandwidth is the same”. That is to say, 100Gb is
100Gb in performance regardless of the software stack or protocol being used. This is a
flawed view. The overhead and services that are encompassed in every protocol always
matter. It is in fact that point that caused the decision to use User Datagram Protocol (UDP) in
the Remote Direct Memory over Converged Ethernet version 2 (RoCE v2) standard for
NVMe. The choice was made because the latency of the Transmission Control Protocol
(TCP) was considered to be too high to allow maximum NVMe performance.
Fibre Channel already uses a Zero Copy mechanism, that is to say that when an application
requests data from a Fibre Channel storage platform it provides a logical address range in the
application buffer for the data to be written to. Effectively the data returned from the request is
written directly into the application. Fibre Channel has been providing this function and its
associated performance gain for more than 20 years. As the storage landscape moves to
utilize more and more flash storage the desire to achieve a maximum Fan-in or Fan-out ratio
(these terms are used to describe multiple server connections sharing access to a single port
on a storage array) for those environments has made Fibre Channel an optimal choice. Part
of the discussion here is the shift in the technology. While Solid State Drives (SSD) have been
packaged for years to be the same form factor, same connectors and same software protocol
for access as actual Hard Disk Drives (HDD) there is, in fact, no disk drive involved. SSD is
comprised most frequently of Flash Memory. It is persistent or non-volatile memory, yet
memory all the same. The term non-volatile is used to distinguish this type of memory from
the Double Data Rate Synchronous Dynamic-Access Memory or DDR SDRAM that is used
for server memory. While SDRAM only retains data while power is applied to the device, flash
memory is persistent in that it only requires power to change state, not to maintain state.
Virtually all USB and SSD devices today make use of NAND style memory chips which are
non-volatile memory. And as the name suggests the same applies to drives comprised of
NVMe devices. What makes this interesting is that Moore's Law (or equivalent variations) now
applies to enterprise storage for the first time in the history of enterprise storage. Density and
latency of 3D NAND is continuing to improve with several 3D NAND suppliers moving to 176
layer technology. This drives additional considerations on the types of SAN fabrics that will be
suitable to carry such performance as well as to be able to accommodate multiple iterations of
new technologies within a normal (for example, 5 year) depreciation schedule of capital
assets.
In the current data center infrastructure, the need for NVMe and SCSI to co-exist for a
significant period of time is readily apparent.
However, the predominance of uses cases for Fibre Channel today would fall into the
scenario depicted in Figure 8-7.
As a market dynamic, each time a new Fibre Channel transmission rate is enabled the oldest
speed is decremented from the product sets. This would normally have meant that in Gen 7
the 4Gb transmission rate would no longer be supported. However, in the instance of the Gen
7 platforms the IBM b-type Gen 7 SAN does retain the 4Gb transmission rate in the Brocade
218 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch08.fm
FC32-X7-48 blade which can be incorporated into either the IBM Storage Networking
SAN512B-7 or the IBM Storage Networking SAN256B-7 chassis-based platforms.
NVMe is the first new Upper Layer Protocol (ULP) mapping in several years. But what makes
this interesting is that the infrastructure that IBM has been selling for years has the potential
to “carry” NVMe traffic. It should be noted that Transmission Rate is only a concern in regard
to performance. All of the current shipping transceivers/optics are able to carry the NVMe
traffic if they are in a compatible Fibre Channel product. A quick summary:
NVMe capable adapter: In the case of Fibre Channel this would be a Host Bus Adapter
(HBA). Any Gen 6 or newer HBA with the correct driver can already carry NVMe traffic.
Operating System (OS) support varies for drivers, so it is important to verify supported
versions with your OS vendor.
NVMe capable fabric: In the case of Fibre Channel, any IBM B-Type SAN platform that is
Gen 6 or Gen 7 is able to carry NVMe traffic. The required OS level Gen 6 is FOS v8.1 or
higher. For Gen 7 it is FOS v9.0 or higher.
Note: The ability to carry the NVMe traffic is not the same as visibility of the NVMe
traffic. Gen 5 has no visibility to NVMe traffic in IO Insight. In the Gen 6 platforms only
the SAN128B-6 and the 64-port blade in the SAN512B-6 or SAN256B-6 have IO Insight
visibility to NVMe traffic. However, ALL Gen 7 platforms have IO Insight visibility to
NVMe traffic.
NVMe capable array: In the case of Fibre Channel in the IBM storage product line any of
the FlashSystem family of FlashSystem 5000, FlashSystem 7200 or the FlashSystem
9200 are capable of supporting NVMe over Fibre Channel fabric.
It is historically true that over time as various elements in the IT infrastructure go through
generational development (e.g., 16Gb FC to 32 Gb FC) bottlenecks move around in the
environment. One of the common scenarios in design consideration is balance. Extremely
fast server platforms communicating with slow storage platforms will potentially experience
congestion. Similarly, the mix of legacy platforms can be problematic especially in the
interaction in the network. Why is this a discussion point as regards NVMe? Because the
latency of NVMe is such that the “inter IO-Gap” is shrinking. This can be thought of as the
“white space” between IOs on the cable, as shown in Figure 8-8.
highway. IOPs is how many vehicles are going past a given point on the highway per second
and latency is how long it takes to get from home to work. Performance applications are
becoming more and more latency and IOP driven. The environment may not drive a full traffic
load on the highway, but it may frequently peak for brief periods of time (for example, during
database analytics) or may have an application set that routinely requires very low latency for
say payment card data. It is always important to have a grasp of the range of application
workloads that are going to be supported by any SAN design. It is an understanding of the
types of workloads to be supported which will provide the “fit for purpose” decision data about
which network technology will meet those needs. The IBM FlashSystem® platforms are
already capable of achieving a less than 100 microsecond latency off the front end of the
array controller using FC-NVMe as the fabric. The fact that the SCSI driver stack is frequently
in the millisecond to seconds range for latency is what is driving a lot of the interest in NVMe
over Fibre Channel.
As previously mentioned, the NVMe over fabric standards continue to develop. One of the
most recent updates include the function of Sequence Level Error Recovery (SLER). The
SLER mechanism allows a much faster recovery of errors. Instead of waiting for an
application timeout the SLER implementation uses a new set of commands (FLUSH and
Responder Error Detected – RED) between the initiator and target ports to recover from any
sequence level errors. Fibre channel utilizes Frames. Multiple Frames make up a Sequence
and multiple Sequences make up an Exchange. An easy way to think of it is Word, Sentence
and Paragraph. Rather than wait for the Exchange to be retried recovery is possible at the
Sequence level via these new signalling commands. For the current version of the
FC-NVMe-2 standard, see FC-NVMe-2 at
https://standards.incits.org/apps/group_public/project/details.php?project_id=174.
Fortunately, as previously discussed, the two protocols can run concurrently in the same
fabric. The most common scenario will encounter some portion of the environment already
being equipment that is capable of participating in an NVMe over Fibre Channel deployment,
such as the following:
Gen 6 HBAs (32Gb) with a currently supported NVMe driver stack
Gen 6 or Gen 7 IBM b-type Gen 7 SAN platforms with a minimum of FOS v8.1
An IBM Storage array that supports NVMe over Fabric
Both the HBAs and the SAN switches/directors have been shipping for more than six years so
there is a good likelihood that some portion of most SAN infrastructure is already in place to
support NVMe over Fibre Channel. Further, the most common SAN topology of Core-Edge is
very well suited to the “in place” integration of NVMe technologies into those existing SANs.
Additional server elements or storage elements that do support NVMe over Fibre Channel
may readily be added into such a topology in a non-disruptive manner. And if there is a need
to refresh the either the core or the edge SAN switches/directors that too is readily
accomplished with little to no risk of downtime due to the dual redundant hardware
infrastructure design of SAN implementations.
220 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch08.fm
Because of the intermix of the technologies the ability of the Gen 7 IBM b-type Gen 7 SAN
platforms to provide functions such as Traffic Optimizer and IO Insight will be of critical help to
the SAN administrator. And, as we will discuss later in this chapter, the migration of
applications/virtual machines from SCSI based storage to NVMe based storage has proven to
be an extremely risk averse technology migration.
As you will see, the combination of the proper topology, along with balanced provisioning,
granular monitoring and automated mitigation provide today’s SAN administrator a very
straightforward window into implementing NVMe over Fibre Channel fabrics.
For help and best practices regarding this type of implementation, see Brocade SAN Design
Guide & Best Practice Guide at https://docs.broadcom.com/doc/53-1004781.
The normal functions of Zone creation, inclusive of Peer Zoning or Target Driven Zoning, are
the same for NVMe devices as it is for SCSI. For detailed information on the complete
environment please, see FOS 9 Admin Guide at
https://docs.broadcom.com/doc/FOS-90x-Admin-AG. Further, all of the Fabric Vision
functions of IO Insight, Fabric Performance Impact Monitoring and so on are supported as
well.
Some things, however, do change. In an NVMe subsystem the naming conventions are a bit
different. In the SCSI world a Logical Unit Number (LUN) is how individual volumes behind
the SCSI target device are configured. NVMe uses the terminology of a Namespace ID for
that same function. It should be noted that a Namespace is actually comprised of a set of
Logical Block Addresses (LBA) and not a set of names.
It is true that since Fibre Channel is a standard link level network transport that, as previously
mentioned, supports multiple protocols it only needs the new upper layer protocols (ULPs) to
support NVMe over FC. So, while no changes are necessary to the actual FC switching logic
another element to understand is that while the data is being carried as standard Fibre
Channel Protocol (FCP) traffic the NVMe commands are being carried as a new frame type
FC-NVMe. So, when an NVMe device logs in on the fabric and the generic Name Server
entry is created, it also updates the additional information which includes but is not limited to:
FC4 type: NVMe device (0x28)
Device type: None, host, target, or both host and target
Discovery services: Absent or present (specific to NVMe devices only)
Port symbolic name
Node symbolic name
Figure 8-9 on page 222 shows the namespace and namespace IDs of an NVMe subsystem.
The logical equivalent of a SCSI target is the NVMe controller in this diagram. The logical
equivalent of the SCSI LUN (address range) is the namespace ID behind the controller. The
physical namespace or range of Logical Block Addresses is then presented via the
namespace ID. An individual namespace may be presented back to the application
environment via multiple controllers for purposes of either performance or design
redundancy/application resilience via Multi Path IO (MPIO) driver support. In the diagram the
Namespace-B is presented to two separate controllers via Namespace ID-X and Namespace
ID-Y. Consequently, although addressing the same range of LBA the Namespace ID (NSID) is
different depending upon whether it is being addressed through Controller-1 or Controller-2.
In Figure 8-9, the MPIO driver would have access to Namespace-B through either controller.
Figure 8-10 depicts a topology view that shows the host and the storage.
Here using IBM SANnav Management Portal one can see the ESXi server platform
connecting across a SAN64B-7 switch to an IBM FlashSystem 9100. In Figure 8-11 on
222 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch08.fm
page 223, the FS9100 (shown in Figure 8-11) is expanded to show the Port World Wide
Names (PWWN) of each of the multi-ported controllers.
Now by selecting the top controller port and doing and drilling into it we can see the World
Wide Names of the Virtual Ports behind the controller port, as shown in Figure 8-12.
In this fashion, the Zone alias designations for both the SCSI LUN target and the NVMe target
can be seen. And now by selecting the ESXi server we can see the detail of the two VMware
HBA ports, as shown in Figure 8-13.
In Figure 8-14, we can see that the NVMe device is now assigned as a storage element to the
ESXi platform. You will also see that the transport if Fibre Channel and the owner is
something that VMware refers to as HPP. The VMware High Performance Plug-In (HPP)
replaces the VMware Native Multipathing Plug-In for high speed devices such as NVMe. In
fact in NVMe-oF environments HPP is the default plug-in that will help eliminate the higher
latency of the SCSI implementations. Within ESXi, the NVMe-oF targets will be emulated and
presented to users as SCSI targets for simplicity. For more information on the HPP
environments, see VMware High Performance Plug-In and Path Selection Schemes at
https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.storage.doc/GUID-
F7B60A5A-D077-4E37-8CA7-8CB912173D24.html.
Notice that the Fibre Channel SCSI device in this case is still owned by the NMP driver, as
shown in Figure 8-15.
The next step is to select the storage element that is to be mirrored. In this case the SCSI
target in the FS9100 we are going to select the RH8 instance and perform a storage only
224 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch08.fm
migration. First identifying the SCSI target that is the source of the migration. The datastore
source to be migrated is SCSI_Migration.
And now selecting the NVMe device that will be the target of the migration where the name of
the target datastore is NVME_Migration.
The next step will be to initiate the per VM migration of the datastore named SCSI_Migration
After the migration is completed, we can see that the NVMe device is now assigned as a
storage element to the ESXi platform, as shown in Figure 8-17. Now both disks (vDisks) are
on NSID (NameSpace ID) backed data stores.
And finally by selecting the datastore source and looking at “Recent Tasks” at the bottom of
the image, as shown in Figure 8-18 on page 226, it is possible to see that the VM migration to
the new datastore was successfully completed.
Part of the discussion here is that this migration can be accomplished with a live Storage
vMotion of the application in question with no downtime. From the same Fibre Channel HBA
across the same IBM b-type Gen 7 SAN switch to the same IBM FS9100 array. Arguably one
of the most risk averse technology migrations possible today. In this migration the actual
protocol being used to address the storage is changed. Benchmark comparisons for SCSI vs
NVMe with ESXi v7.1 are showing as much as an 80% increase in IOPS providing an
enormous benefit to the application performance.
Given the benefits of NVMe based storage platforms the ability of Gen 7 platforms to so
readily support NVMe technology makes IBM b-type Gen 7 SAN the logical evolutionary
choice for the adoption of the technology. It provides:
A stable infrastructure that is well understood
Lossless, low latency, deterministic network characteristics which are critical for storage
The ability to provide both scale and performance simultaneously
IO Insight auto learning up to 20,000 SID/DID flows per switch or director (including
NVMe)
The ability to manage NVMe as part of a standard Fibre Channel environment
At 64Gb line rate the IBM SAN64B-7 provides a 460-nanosecond latency for NVMe
The same SAN management (e.g., Zoning) that is already well understood by IT
226 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch09.fm
Fabric OS uses RBAC to control access to all Fabric OS operations. Each feature is
associated with an RBAC role. You need to know which role is allowed to run a command,
make modifications to the switch, or view the output of the command. Consider these
requirements when you create users.
To determine which RBAC role you need to run a command, review the “Role-Based Access
Control” section in the Fabric OS Administrator’s 9.0.0 Guide at the following link:
https://docs.broadcom.com/doc/FOS-90x-Admin-AG
When you use the CLI, remember that both commands and options are case-sensitive. See
the Fabric OS command reference for a description of all of the available CLI commands, their
options, and their uses.
You can connect to the Fabric OS through a SSH or telnet connection or by using a console
session on the serial port.
Use the following procedure to connect to the Fabric OS using the serial port.
Note: If you connect the serial console port to a terminal server, ensure that flow control is
disabled.
1. Connect the serial cable to the serial port on the switch and to an RS-232 serial port on
the workstation.If the serial port on the workstation is an RJ-45 port, instead of RS-232,
remove the adapter on the end of the serial cable and iwnsert the exposed RJ-45
connector into the RJ-45 serial port on the workstation.
2. Open a terminal emulator application (such as HyperTerminal in a Windows environment,
or TERM, TIP, or Kermit in aUNIX environment), and configure the application as follows
– In a UNIX environment, enter the following string at the prompt:
tip /dev/ttyb -9600
If ttyb is already in use, you may use ttya at the prompt.
tip /dev/ttya -9600
In a Windows environment, enter the parameters from Table 9-1 on page 229:
228 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch09.fm
Data bits 8
Parity None
Stop bits 1
Use the following procedure to connect to the Fabric OS using the management interface.
1. From a management station, open a SSH or telnet connection using the IP address of the
switch to which you want to connect. The login prompt is displayed when the SSH or telnet
connection finds the switch in the network.
2. Enter the account ID at the login prompt.
3. Enter the password.
If you have not changed the default system password, you are prompted to change it upon
login.
4. Verify that the login was successful.
The prompt displays the switch name and user ID to which you are connected.
login: admin
password: xxxxxxx
Beginning in Fabric OS 9.0.0 several CLI commands were consolidated and the redundant
command were deprecated. The complete list of deprecated commands is listed in the Fabric
OS Command Reference Manual, 9.0.0 at
https://docs.broadcom.com/doc/FOS-90x-Command-RM
If the switch is configured with logical fabrics, you can log in to any of the logical fabrics for
which you have permission.
Web Tools does not require a license. It is installed on the switch when you install Fabric OS.
Web Tools can be launched from a browser or through SANnav Management Portal.
To launch directly from a Web browser, open your browser, enter the IP address of the
switch, and press Enter. You can use HTTP or HTTPS, for example: http://10.77.77.77
or https://10.77.77.77.
To launch from SANnav Management Portal, locate the switch on the SANnav Inventory
page, click the down arrow to the right of the switch, and select View in WebTools from
the action menu. See Figure 9-1.
.
When the Web Tools Element Manager is launched you will see a log in page. See Figure 9-2
on page 231.
Enter the user name, password, and logical switch name or fabric ID (FID).
For the first switch login, the default user name is admin and the default password is
password. Web Tools prompts you to change the default password.
If you are logging in to a Virtual Fabrics-enabled platform and you do not specify a logical
switch, you are logged in to the default logical switch, which uses fabric ID 128. For
non-VF platforms, the FID option is not displayed.
If you launch from SANnav Management Portal, you might not need to log in, depending
on the SANnav single sign-on configuration.
230 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch09.fm
For complete instructions on working with Web Tools, download the Fabric OS WebTools
User Guide, 9.0.0 at : https://docs.broadcom.com/doc/FOS-90x-WebTools-UG
Monitor fabric-wide events, ports, FRUs, environmental parameters, security, traffic flows, and
performance impacts.
MAPS is a licensed feature and requires a Fabric Vision license. This guide assumes that the
administrator has activated (enabled) the Fabric Vision license is activated (enabled) in the
Fabric OS software to use the full set of MAPS options. For more information on licensing,
refer to the Brocade Fabric OS Software Licensing Guide.
MAPS is shipped with predefined rules, groups, and policies. However, you can create
custom rules, groups, and policies. MAPS enables the cloning of predefined rules, groups,
and policies to facilitate the monitoring. Predefined rules and groups are system-specific, and
each system has its own rules and groups. All MAPS configuration is persistent, is retained
across reboots or HA failovers, and can be uploaded and downloaded.
MAPS can be configured and managed using the Command Line Interface (CLI).
For complete instructions on Configuring MAPS, download the Fabric OS MAPS User Guide,
9.0.x at: https://docs.broadcom.com/doc/FOS-90x-MAPS-UG
MAPS can also be configured and managed through SANnav MAPS policy management.
This allows an administrator to create new MAPS policies, rules, and custom groups; and
apply and activate MAPS policies across one or more switches of a fabric. You can import
and modify existing MAPS policies. You can also view MAPS-enabled switches, policies and
rules, fabrics and groups to which the switches belong, and switch configuration actions.
232 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch09.fm
1. Click SANnav in the navigation bar, and then select SAN Monitoring > MAPS Policy
Management. The Switches window is displayed in Figure 9-3.
2. Click the Switches tab to view the list of MAPS-enabled switches with an active policy
name, policy type, and the fabric to which they belong, and switch configuration actions.
3. Click the Policies tab to view the policies applied to the switch.
4. Click the Groups tab to view the preconfigured groups available on the switch. You can
create user-defined custom groups only for ports, SFPs, and circuits.
For complete instructions on working with SANnav to manage MAPS, download the Brocade
SANnav Management Portal User Guide, 2.1.0x at
https://docs.broadcom.com/doc/SANnav-21x-MP-UG and refer to the MAPS management
section.
Automatic monitoring (introduced with FOS 9.0.0) provides I/O monitoring to the SANnav
administrator. With these IO Insight capabilities, Flow Vision provides proactive, non-intrusive,
real-time monitoring and alerting of storage I/O health and performance. The additional
visibility delivers deep insights into problems and concerns affecting the ability to maintain
service levels.
Integration with MAPS allows the automatically detected flows to be monitored using the
predefined rules. Hardware realtime violations are flagged at the I/O (ECT and FRT) and FC
level (frame latency). For details on MAPS, see Brocade Fabric OS MAPS User Guide, 9.0.x
at https://docs.broadcom.com/doc/FOS-90x-WebTools-UG.
Automated monitoring is supported by SANnav. For details on SANnav, see Brocade SANnav
Management Portal User Guide, 2.1.x f.
234 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch09.fm
Flow Generator also supports generating simulated traffic to a routed fabric. To generate
traffic, you specify the imported proxy ID or the real PWWN of the device. Flow Generator
support is similar to Flow Monitor support, except Flow Generator supports only flows
from edge to edge but not from backbone to edge or from edge to backbone.
The Brocade Monitoring and Alerting Policy Suite (MAPS) can be used to monitor SIM
port traffic thresholds while Flow Generator is running. MAPS treats SIM ports as F_Ports,
so MAPS can issue warnings on these ports if threshold values are triggered. If you do not
want to see MAPS warnings for SIM ports, you must disable MAPS monitoring for those
ports.
Flow Mirror
As storage networks get larger and more complicated, non-intrusive diagnostic tools are
becoming increasingly important to help identify problems without disturbing the operating
fabric. Flow mirroring is a diagnostic feature within Flow Vision that addresses this need.
With Flow Mirror, you can do the following:
– Non-disruptively create copies of application flows that can optionally be captured for
deeper analysis.
– Define a traffic pattern and create a real-time copy of this traffic, allowing you to
analyze a live system without disturbing existing connections. You can also use this
feature to view traffic passing through a port.
– Use a MAPS logical port group for either an ingress or egress port. Note that logical
port groups are not supported on CPU flow mirroring (CFM) or local flow mirroring
(LFM).
– Use a predefined flow with the flow --show command to mirror SCSI command, first
response, status, and ABTS frames from all Gen 6 F_Ports on a switch. For example:
flow --show sys_analytics_vtap .
Flow Mirror duplicates the specified frames in a user-defined flow and sends them to a
sink. This sink could be either:
– The local switch control central processor unit (CPU); this form is called “CPU flow
mirroring” (CFM), and has a limit of 256 frames per second.
– An analyzer/packet sniffer connected through a port in the metaSAN. The bandwidth
limit for a flow using this type of link is the bandwidth of the mirror destination port. This
form is called “Local flow mirroring” (LFM), and mirrors the flow to a port on the same
physical switch. This requires that a loopback SFP be plugged in at the other end of the
analyzer or on the port configured as a mirror port, which must be in the same domain.
All forms of mirroring are mutually exclusive. Only one form of mirroring (LFM, CFM, or
RFM) can be active at a time.
Flow Mirror flows can be in an active or inactive state. If the mirror flow is “active”,
mirroring starts immediately; if the flow is “inactive”, the flow must be activated (by using
the flow --activate command) for mirroring to start. Mirrored flows can be unidirectional or
bidirectional. A sample use case would be to mirror the traffic flow from a slow-draining
F_Port to see what is causing this condition.
For complete instructions using SANnav to create and manage Flow Vision Flows,
download the Brocade SANnav Flow Management User Guide, 2.1.0x. at:
https://docs.broadcom.com/doc/SANnav-21x-Flow-UG
ClearLink D_Port testing can be used to exercise the hardware before deploying ports into the
fabric. For example, you might install new SFPs to link two switches or to link a host/target to
a switch, and you want to exercise the hardware before adding them to the fabric. If the test
passes, you can go ahead and add the ports to the fabric. If the test fails, you can fix the
problem accordingly.
When D_Port testing is in progress, SANnav Management Portal changes the port type for all
selected ports and associated attached ports to D-Port for the duration of the test. A port in
D_Port mode does not carry any user traffic and is designed to run only specific diagnostics
tests for identifying link-level faults or failures. When the test is complete, SANnav changes
the port type back to the original port type.
Requirements:
You must have Troubleshooting privilege with read-write permission.
Both the source and destination ports must be managed by SANnav Management Portal.
When you run a D_Port test, SANnav Management Portal changes the port type for all
selected ports and associated attached ports to D_Port for the duration of the test. This may
cause the fabric to segment. When the test is complete, SANnav changes the port type back
to the original port type.
1. Click Inventory in the navigation bar, and then select Switch Ports from the drop-down
list.
2. Select the ports on which you want to perform the diagnostic tests.
– To select a single port, click the down arrow in the rightmost column for the switch port,
and select Run Diagnostics.
– Alternatively, to select multiple ports, use the bulk select option.
236 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch09.fm
a. Specify either the number of frames to send (in millions) or the time duration for which
the frame traffic test will run. The default is to send one million test frames.
b. In the Frame Size field, enter the size of the test frames that are generated to run the
test. The default value is 1024.
Note that for Fabric OS 9.0.x and higher, the test uses a test frame size that is a
multiple of four. If you specify a size that is not a multiple of four, when the test is run,
the nearest higher multiple of four is used as the test frame size. For example, if you
specify a test frame size of 37, the tests are run with a test frame size of 40.
c. Select a predefined pattern to be used in the payload, or enter a user-defined pattern in
decimal. The default is to use the pattern jCRPAT.
You can optionally select the checkboxes at the bottom of the dialog. Note that FEC is
not supported on D_Ports that are configured with dense wavelength division
multiplexing (DWDM). For 32Gbps ports, FEC is automatically enabled and cannot be
disabled.
5. Click Next.
The diagnostic test starts running. A dialog displays the status of the test.
Testing might take a few minutes to several hours, depending on the selected parameters.
6. Click Close. You can view this status on the Outputs page.
7. Click Done in the confirmation dialog to return to the Inventory page. Note that the port
type for the selected ports has been changed to D-Port.
To view the test results, click Outputs in the subnavigation bar, and select Diagnostics from
the drop-down list.
Requirements:
You must have Troubleshooting privilege with read-write permission.
Both the source and destination ports must be managed by SANnav Management Portal.
Procedure:
1. Click Inventory in the navigation bar, and then select Switch Ports from the drop-down
list.
2. Select the ports on which you want to perform the diagnostic tests.
– To select a single port, click the down arrow in the rightmost column for the switch port,
and select Schedule.
– Alternatively, to select multiple ports, use the bulk select option.
i. Click the more button ( ), and select Bulk Select. A column of checkboxes
displays on the leftmost side of the table.
ii. Select the checkboxes for the ports on which you want to run diagnostic tests, and
then click Actions > Schedule in the upper-right corner of the window.
If Schedule is grayed out, one or more of the selected ports are not supported for running
diagnostic tests.
3. Select Diagnostics from the drop-down list, select the date and time for the test to start,
and click OK.
238 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch09.fm
Note that if Diagnostics is grayed out, one or more of the selected ports are not
supported for running diagnostics tests.
From the action menu of the scheduled test, you can display the ports selected for the test,
edit the schedule, and delete the test.
Only the user who ran or scheduled the test can view, delete, or re-run the test.
1. Click Inventory in the navigation bar, and then click Outputs.
Figure 9-8 Dashboard & Reports - Inventory Output with Bulk Select option
For complete instructions using SANnav to manage ClearLink diagnostics download the
Brocade SANnav Management Portal User Guide, 2.1.0x and refer to the Troubleshooting
and Diagnostics section.
240 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch09.fm
The Brocade SANnav Management Portal User Guide, 2.1.0x can be found at
https://docs.broadcom.com/doc/SANnav-21x-MP-UG
Dashboards
Customizable health and performance dashboard that provides at-a-glance views of switch
status and various conditions that are contributing to performance issues, enabling
administrators to get instant visibility into any hot spots at a switch level and take corrective
actions. Dashboard views include:
Overall status of the switch health and the status of each monitoring category, including
any out-of-range conditions and the rules that were triggered.
Historical information on the switch status for up to the last seven days; automatically
provides raw counter information for a variety of error counters. This integrated dashboard
view also provides a single collection point for all dashboard data from a fabric for a
specific application flow
Allows you to monitor the congestion-related issues on all physical E_Ports and F_Ports at all
times. Specifically, FPI monitors abnormal device latency and oversubscription issues.
9.3.5 IO Insight
Proactively monitors I/O performance and behavior through integrated network sensors to
gain deep insight into problems and ensure service levels. This capability automatically learns
the traffic flows to a fabric non-disruptively and non-intrusively gathers I/O statistics for both
SCSI and NVMe traffic from any device port on a Gen 6 and Gen 7 Fibre Channel platform;
and then it applies this information within an intuitive, policy-based monitoring and alerting
suite to configure thresholds and alarms. Integrated application-level and device-level I/O
latency and IOPS monitoring provide the ability to baseline application performance and
detect degraded performance. This enables administrators to proactively control performance
and availability to ensure operational stability.
Obtains total I/Os, first response time max/average, I/O latency (Exchange Completion
Time, or ECT) max/ average, and outstanding I/Os total/ average performance metrics for
a specific host or storage for a LUN or NSID in order to diagnose I/O operational issues.
Enables tuning of device configurations with integrated I/O metrics to optimize storage
performance.
9.3.6 VM Insight
Seamlessly monitors VM performance throughout a storage fabric with standards based,
end-to-end VM tagging. Administrators can quickly determine the source of VM/application
performance anomalies, as well as provision and fine-tune the infrastructure based on
VM/application requirements to meet service level objectives.
For additional information on Forward Error Correction (FEC) and Buffer Credit Recovery,
download the Fabric OS Administrator’s 9.0.0 Guide at
https://docs.broadcom.com/doc/FOS-90x-Admin-AG
242 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch10.fm
10
IBM b-type Gen 7 is in the unique position to spot and understand the impact of automation
helping organizations get more from their SAN infrastructure.
IBM b-type Gen 7 offers a combination of SAN automation with RESTful APIs and a SAN
management platform to help organizations drive greater efficiency from their SANs.
IBM b-type Gen 7 automation solutions leverage RESTful APIs to facilitate solutions
architecture, share best practices and get to production status faster.
244 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch10.fm
managed with NETCONF. An Apache web server embedded in Fabric OS is used to serve
the API.
The RESTful API approach lets you think of a network device as a web server. By using
standard web-based tools, an automation can send and receive transactions to or from a
network device just as it would send transactions to and from a website. This means that
transactions take place over a secure socket using HTTP rules to handle the exchange. The
data appears in the form of XML or JSON depending on the RESTful API services
implemented on the networking device.
To interact with a SAN (or other) device, you need to consult its RESTful API reference to
learn, among other things, what “Uniform Resource Identifiers” (URIs) you need to use.
(Simply put, URIs are identifiers that can be used as part of a web address.) According to the
documentation, the URL for accessing a listing of the zones in the active configuration is as
follows:
GET <base_URI>/running/zoning/defined-configuration/
The model that is used to represent state and configuration information is expressed in a
modeling language called Yang. Yang describes the structure of the different elements inside
the model and is used to describe whether each element is read-only or read-write. It
describes the type of data that the element can hold, such as string or integer, and it shows
the relationship among various elements, the other nested elements they contain, their peer
elements, and the parent elements that contain them. Here is a segment of the description of
a zone in Yang:
list zone {
key "zone-name";
description
"List of the members in the zone. The members can only be identified as a
WWN, domain, index, or a zone alias.";
leaf zone-name {
type zoning-name-type;
description
"The zone name.";
}
leaf zone-type {
type zone-type-type;
description
"The zone type. Not that target zone types cannot be created or modified
(only deleted).";
}
container member-entry {
description
"The zone member.";
leaf-list entry-name {
type zone-member-type;
min-elements 1;
description
"List of the members in the zone. The members can only be identified as a
WWN, domain, index, or zone alias.";
}
leaf-list principal-entry-name {
when "../..zone-type=1 or ../../zone-type=2";
type zone-member-type;
min-elements 1;
description
"List of the principal members in the peer zone. The members can only be
identified as a WWN, domain,index, or zone alias.";
}
}
}
Ordinarily more information goes into a Yang module such as revisioning and governance
information; this listing omits them for brevity. Thus, the Yang description is complete, but it’s
also wordy. Although this precision is necessary when interacting with the model
programmatically, it’s sometimes useful to get a global view of the abstraction provided by the
model to see how the data is structured.
An open source tool called pyang can parse the Yang model and produce a tree that
represents the elements in the model. The listing includes information about each element,
such as whether it’s read-only or read write, a list, optional, or nested. Here is the
representation of the zoning model in tree form:
module: IBM b-type-zone
+--rw IBM b-type-zone
+--rw defined-configuration
| +--rw cfg* [cfg-name]
| | +--rw cfg-name zoning-name-type
| | +--rw member-zone
| | +--rw zone-name* zoning-name-type
| +--rw zone* [zone-name]
| | +--rw zone-name zoning-name-type
| | +--rw zone-type? zone-type-type
| | +--rw member-entry
| | +--rw entry-name* zone-member-type
| | +--rw principal-entry-name* zone-member-type
| +--rw alias* [alias-name]
| +--rw alias-name zoning-name-type
| +--rw member-entry
| +--rw alias-entry-name* union
+--rw effective-configuration
+--rw cfg-name? zoning-name-type
+--rw checksum? string
+--rw cfg-action? uint8
+--rw default-zone-access? uint8
+--ro db-max? uint32
+--ro db-avail? uint32
+--ro db-committed? uint32
+--ro db-transaction? uint32
+--ro transaction-token? uint32
+--ro db-chassis-wide-committed? uint32
+--ro enabled-zone* [zone-name]
+--ro zone-name zoning-name-type
+--ro zone-type? zone-type-type
+--ro member-entry
+--ro entry-name* union
+--ro principal-entry-name* union
246 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch10.fm
The last parameter is the Uniform Resource Identifier (URI) for curl to use to login. (URI value
is described in the RESTful API reference.)
This command establishes the session used for the following commands. The following is a
trace of its execution:
* Trying 10.18.254.37...
* Connected to 10.18.254.37 (10.18.254.37) port 80
(#0)
* Server auth using Basic with user 'admin'
> POST /rest/login HTTP/1.1
> Host: 10.18.254.37
> Authorization: Basic YWRtaW46cGFzc3dvcmQ=
> User-Agent: curl/7.47.0
> Accept: */* >
< HTTP/1.1 200 OK
< Date: Wed, 31 Jan 2018 16:01:24 GMT
< Server: Apache
< Authorization: Custom_Basic
YWRtaW46eHh4OjNkYTllZmM3NzMxYjk4OGU2ODg1YzZkMGRjNWJlM
zMyNjBhZDYxZThkOWQ2MWMxNzNiMGVlMjU3YmM2OTcyYjA=
< Cache-Control: no-cache
< X-Frame-Options: DENY
< Content-Secure-Policy: default-src 'self'
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: application/yang-data+xml
Next, you perform a GET of the URI to return the current configuration using the
Custom Basic value returned from the login for authentication:
curl -v -H "Authorization: Custom_Basic
YWRtaW46eHh4OjNkYTllZmM3NzMxYjk4OGU2ODg1YzZkMGRjNWJlM
zMyNjBhZDYxZThkOWQ2MWMxNzNiMGVlMjU3YmM2OTcyYjA="
http://10.18.254.37/rest/running/zoning/defined-configuration
By default, curl uses the GET method, so you don’t need to specify it. -H
“Authorization: Custom_Basic YWR...jA=” is the authentication and session
identifying string returned in the previous command. -H places the string into the
GET request header as seen in the following trace:
* Trying 10.18.254.37...
* Connected to 10.18.254.37 (10.18.254.37) port 80
(#0)
> GET /rest/running/zoning/defined-configuration
HTTP/1.1
> Host: 10.18.254.37
> User-Agent: curl/7.47.0
> Accept: */*
> Authorization: Custom_Basic
YWRtaW46eHh4OjNkYTllZmM3NzMxYjk4OGU2ODg1YzZkMGRjNWJlM
zMyNjBhZDYxZThkOWQ2MWMxNzNiMGVlMjU3YmM2OTcyYjA=
>
< HTTP/1.1 200 OK
< Date: Wed, 31 Jan 2018 16:09:39 GMT
< Server: Apache
< Cache-Control: no-cache
< X-Frame-Options: DENY
< Content-Secure-Policy: default-src 'self'
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: application/yang-data+xml <
<?xml version="1.0"?>
<Response>
<defined-configuration>
<cfg>
<cfg-name>CFG_FABRIC_A</cfg-name> <member-zone> <zone-name>CLUSTER1</zone-name>
<zone-name>Z_AIXHOST_FCS2_VMAX01_SN1234_9F0 </zone-name>
...
<alias> <alias-name>esx66_5d3d00</alias-name> <member-entry>
<alias-entry-name>10:00:8c:7c:ff:5d:3d:00 </alias-entry-name>
</member-entry>
</alias>
</defined-configuration>
</Response>
* Closing connection 0
The results appear as an XML data segment structured according to the description in the
Yang model, and so it’s important to have access to that model along with the RESTful API
manual. The models can be found on GitHub as a repository among those in IBM b-type’s
repositories at http://github.com/IBM b-type/yang. The RESTful API manual can be
retrieved from the Broadcom web site. Having retrieved the zoning information from the
fabric, you should close the session using the CLI command (the results are omitted to save
space):
curl -v -H "Authorization: Custom_Basic
YWRtaW46eHh4OjNkYTllZmM3NzMxYjk4OGU2ODg1YzZkMGRjNWJlM
zMyNjBhZDYxZThkOWQ2MWMxNzNiMGVlMjU3YmM2OTcyYjA=" http://10.18.254.37/rest/logout
In version 8.2.2 or later of FOS, the REST API session-less operation allows you to provide
authentication credentials directly for each GET request. Essentially, FOS REST API
session-less operation completes the login, GET operation, and logout as one complete
request. You can only use basic authentication formats for REST API session-less operation,
which includes plain text or Base64.
248 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch10.fm
The following example shows a GET request using plain text authentication:
curlysw -u admin:password http://10.155.2.190/rest/running/IBM
b-type-media/media-rdp>"
Ansible takes a declarative approach. Rather than provide sequential steps, Ansible
describes each of the hosts in an inventory. The description appears in a document called a
playbook. For example, rather than provide steps to install a particular application, Ansible
describes a host state where the application is already installed. When you run the playbook,
Ansible takes no action if the application is already installed. If the application isn’t installed,
Ansible calls installation routines so the host is brought into the desired state without requiring
the administrator to write any specific steps.
In the realm of storage networks, the use of a declarative language means you can describe
switches and fabrics where, for example, a zone is already configured with the proper hosts
and storage arrays. When you run the Ansible playbook, those zones are defined as needed,
and the hosts and storage arrays are added to them if necessary.
With some other declarative automation utilities, it’s necessary to install an agent on each
host that the utility manages. This agent retrieves the commands from a command center and
runs them on the local host. Ansible is unique in that it doesn’t require agents. In order to
make switch state changes, Ansible establishes a secure shell session to a proxy and sends it
a small Python script. The script carries out the necessary operations using the switch API
and removes itself from the host.
You need two different skill sets to successfully implement an Ansible solution. First, you must
understand the most common playbook operations. These operations are coded and installed
for use by the playbooks. As vendors announce support for Ansible, they also provide script
libraries for the most common tasks. If there is a task that is required but isn’t available in the
official Ansible distribution, the open source community may provide code for that task in
publicly available repositories.
Second, you must understand your business needs to provide ongoing playbook
development. The person maintaining the playbooks doesn’t need to be a programmer and
doesn’t need to know how remote system operations occur. That person only needs to know
the desired outcomes and should be able to construct play- books in YAML — the markup
language used by Ansible. This is an example of an Ansible playbook:
---
- hosts: edgeSwitches
vars_files:
../fos_passwords.yml
gather_facts: False
tasks:
- name: run fos commands
IBM b-type_fos_command:
switch_login: “{{switch_admin_account}}”
switch_password: “{{switch_password}}”
switch_address: “{{switch_ip_address}}”
command_set:
- command: alicreate "SampleAlias1", "10:23:45:67:76:54:32:10"
The three dashes at the beginning are part of the YAML specification. The hosts section
identifies automation target switches. You can keep sensitive information in a separate file, as
demonstrated by the fos_passwords.yml line. The name of a variable in double braces such
as {{switch_password}} indicates variable substitution. The variable file specified in vars_files
tells where to find external variables.
The SANnav REST API provides functionality that complements that found in the Fabric OS
REST APIs. The SANnav REST API includes support for the following SANnav features:
List the fabrics managed by the SANnav server.
List the seed and principal switch data for each fabric.
List the switches (members) in the fabric.
Display the FCR topology information for routing topologies such as edge-to-edge,
backbone-to-edge, and edge-to- backbone.
Configure event-forwarding.
Acknowledge or unacknowledge events.
Display a list of filtered events.
Perform an inventory search.
10.6 Conclusion
SAN automation is a critical element in IT modernization and digital transformation because it
helps organizations handle storage-related processes more efficiently without hiring more
administrators or adding to the storage infrastructure Capex budget. SAN automation is a
high-leverage approach to turning network storage into a strategic asset.
IBM b-type Gen 7’s commitment to SAN automation, combined with its long-standing
leadership in storage fabrics and technical innovation, makes it an ideal candidate for your IT
infrastructure automation strategy.
250 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm
11
This guide contains detailed steps for installing the IBM SANnav Management Portal and
SANnav Global View 2.1.
This guide describes all the system and server requirements before you start the IBM
SANnav Management suite installation. The following document lists the system and server
requirements for deployment of both SANav Management Portal and SANnav Global View.
Within this document, SANnav Management Portal might also be referred to simply as
"SANnav".
SANnav Management Portal allows management of one or more SAN fabrics that are in the
same or different geographical locations, and it supports a maximum of 15,000 physical SAN
ports. For environments that are larger than15,000 ports, you can deploy multiple SANnav
Management Portal instances. SANnav Management Portal does not replace Brocade Web
Tools or the Fabric OS command line interface.
For more information, see Brocade SANnav Management Portal User Guide, found at:
https://www.broadcom.com/products/fibre-channel-networking/software/sannav-managem
ent-portal.
252 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm
For more information, see Brocade SANnav Global View User Guide, found at:
https://www.broadcom.com/products/fibre-channel-networking/software/sannav-managem
ent-portal.
Switches That Support Data Streaming SANnav uses Kafka to provide real-time data
streaming from switches that are running Fabric OS 8.2.1a or higher. Data streaming provides
collection of high-frequency performance statistics, FCIP performance and error statistics,
and flow statistics and violations. The following Gen 7 platforms support data streaming: •
IBM SAN64B-7 Switch • IBM SAN256B-7 and SAN512B-7 Directors
The following IBM b-type Gen 6 platforms support data streaming: • IBM SAN24B-6,
SAN64B-6, and SAN128B-6 Switches • IBM SAN18B-6 Switch • IBM SAN256B-6 and
SAN512B-6 Directors
Note that if you access the client from Remote Desktop, the user interface may have
degraded performance. Launching Brocade Web Tools 9.0.0 and later from a SANnav client
is supported on Chrome and Firefox. Launching Brocade Web Tools versions earlier than
9.0.0 is supported only on Firefox.
For specific details on installing SANnav Management Portal on a virtual machine (VM)
please refer to the Brocade SANnav User Guide for detailed information.
254 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm
Important: You must have active IBM Support in order to access Brocade Software for
any device.
Brocade FOS and SAN Management software may be accessed through the IBM Support Fix
Central, found at: https://www.ibm.com/support/fixcentral/options
1. Navigate to https://www.ibm.com/support/fixcentral/options
2. Select “Select product” tab
3. Under “Product Group” dropdown, select “System Storage”
4. Under “Select from System Storage”, select “Storage area network (SAN)”
5. Under “Storage area network (SAN)”, select “SAN b-type” for Brocade switches or “SAN
management software” for IBM Network Advisor and SANnav (once released).
6. If “SAN b-type” is selected:
a. Under “Select from SAN b-type”, select the proper model.
b. Under “Installed Version”, select the proper FOS version.
c. Click the Continue box.
7. If “SAN management software” is selected:
a. Under “Select from SAN management software”, select the proper management
software.
b. Under “Installed Version”, select the proper version.
c. Click the Continue box.
256 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm
8. On the page “Select fixes”, select the appropriate fix-pack (FOS version) for your product
and select the download method.
258 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm
10.Log in to IBM with your IBMid (if requested) and create an IBMid account (if necessary)
11.This will take you to the IBM “Fix Central” page where you must provide the proper
information for the FOS entitlement check.
260 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm
15.Business email addresses are required. Use of a non-business address will produce an
error message, as shown in Figure 11-10 on page 262.
16.Once logged into the Assist Portal, the user will be reminded of the Brocade Enterprise
User License Agreement (EULA), as shown in Figure 11-11.
262 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm
18.The Software Downloads tab allows the user to download relevant FOS versions and
software including SANnav and EZSwitchSetup.
a. Click Browse and select the appropriate category, as shown in Figure 11-13.
b. The following products are available depending on the option chosen from the Select
list, as shown in Figure 11-14.
20.For Documentation on Software, select All Software Products from the select box and
utilize the same search method to download documentation, as shown in Figure 11-16.
Installation checklist
The following table provides a quick checklist for installing SANnav.
264 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm
Migration checklist
The following table provides a quick checklist for migrating from an earlier version of SANnav.
Note:
SANnav Management Portal and SANnav Global View are two different software products.
You cannot install both software products on the same physical host or virtual machine
(VM). You can, however, install Management Portal and Global View on different VMs in
the same host, if the host has enough resources.
For switches that are running Fabric OS versions lower than 8.2.2, port 22 is required for
SANnav Management Portal to use the internal firmware repository and SCP and SFTP
servers. See Installation Prerequisites for additional details. If there is a firewall between
the client and the server or between the server and the SAN, you must open a set of ports
for SANnav to function properly. The list of ports is provided in Firewall Requirements for
SANnav Management Portal. If the installation script detects that an earlier version of
SANnav is running, you are prompted whether you want to migrate your data to the new
version.
After installation, if you choose to move SANnav from one server or VM to another, you must
first release the current license. This process is called rehosting a license. Refer to the
Brocade SANnav Management Portal User Guide for details.
266 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm
Note: The disk space requirement that is listed in the table is for SANnav only. Be sure to
account for additional space required by the operating system, for saving files, and for
SANnav TAR files and extracted files.
The disk space can be from a direct-attached disk or through a network-mounted disk.
The default home directory for installing Docker is /var/lib/docker, but you can choose
another location during installation. Docker must be installed on a local disk.
The default swap space directory is the "/" directory. If the directory does not have enough
space, you can choose a different location during installation. The required number of
CPU cores should be equally distributed over the sockets.
Figure 11-20 System and server requirements for SANnav management portal installation
Note: Use the latest generation processors for better SANnav performance.
268 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm
Before you unzip the installation file, be sure to review and comply with the system and server
requirements and the installation prerequisites that are listed in the following sections:
System and Server Requirements for SANnav Management Portal
Installation Prerequisites NOTE If the scripts fail during the installation or startup, uninstall
SANnav, reboot the server, and then reinstall SANnav. Do not try to fix the issue and
re-run the installation script without first uninstalling the application.
Download and copy the SANnav Management Portal software package to the server. The
package contains the SANnav Management Portal tarball.
1. Download the SANnav Management Portal tarball (for example,
Portal_<version>-distribution.tar.gz) to the folder where you want to install the
application. NOTE Do not create the SANnav Management Portal installation folder with a
space in the name; otherwise, installation fails.
2. Untar the .gz file to extract the file to the current location. tar -xvzf
Portal_<version>-distribution.tar.gz This step creates a directory with a name similar
to Portal_<version>_bldxx. This directory is referred to as the <install_home> directory
in this document.
3. Go to the <install_home>/bin directory. [root@RHEL7-10-100 home]# cd
Portal_<version>_bldxx/bin
4. Run the install-sannav.sh script to install SANnav Management Portal.
[root@RHEL7-10-100 bin]# ./install-sannav.sh If an earlier instance of SANnav
Management Portal is installed, the installation script prompts whether you want to
continue with migration or exit the installation.
5. If you are prompted about migrating SANnav, enter one of the following options.
– To proceed with migration, press Enter. You are prompted to enter the location of the
existing SANnav installation.
– To exit the installation, press Ctrl-C. The script ends. At this point, you can back up the
current SANnav instance and restart the installation script. Or you can uninstall the
current SANnav instance, and restart the installation script without migrating.
As the installation proceeds, the script runs a pre-install requirements test. If any test fails, the
installation exits with error messages. You must fix the reported issues, uninstall the
application, and repeat from Step 1. After the diagnostics pass, installation of SANnav
Management Portal software continues. On successful installation of the software, the
SANnav Management Portal server starts up.
If firewalld is enabled, you must add the SSH service to the trusted zone in the firewalld for
the firmware download feature to work.
See Configuring a Firewall for SANnav for instructions on how to configure firewalld. If you
configure an external authentication server (LDAP, RADIUS, or TACACS+) or an email server
(SMTP), ensure that the SANnav Management Portal server has access to the ports listed in
the following table. The default ports are listed in the table, but you can change the default.
270 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm
Using this one script, you can perform the following actions:
Check SANnav status
Restart SANnav
Stop SANnav
Start SANanv
Uninstall SANnav
Show the SANnav port and server configuration. Go to the <install_home>/bin folder and
run the following script: ./sannav-management-console.sh You are presented with a list of
options from which to choose.
To check the health of the server, go to the <install_home>/bin folder and run the following
script: ./check-sannav-status.sh
Note: If any service is found down while checking the server health status, it will be
automatically started by systemmonitor within 20 minutes.
There are three threshold levels: 70%, 80%, and 90%. An event is sent when the threshold
levels exceed or drop below the defined level. The following is the list of event severity:
threshold:
Warning — 70%
Error — 80%
Critical — 90%
When you run this script, SANnav is automatically restarted for the new certificates to take
effect. After the server is back up, you must rediscover or unmonitor and then monitor all
switches that are registered for telemetry data; otherwise, the new certificates do not take
effect.
272 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm
8. After the ports are added, use the following command to reload the firewall configuration.
sudo firewall-cmd --reload
9. Verify whether the configuration is correct. sudo firewall-cmd --list-all
For the first SANnav login, the default user name is "Administrator" and the default password
is "password".
SANnav launches with the default dashboard displayed. If, instead of launching, SANnav
displays the message "Login Failed. Service is not available at this time.",
SANnav is in the process of starting up. Wait a few minutes and try to log in again. The first
time you log in, you should change the default password.
274 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm
1. Navigation bar. Contains links to feature pages. The SANnav link displays the
Configurations and Settings page.
2. Search. Click the icon to display the search box, select a context on which to search from
the list, and enter the search term in the field.
3. Profile menu. Click the icon to display additional links for changing user preferences,
displaying the SANnav version, and logging out.
4. Subnavigation bar. Provides the page title and optional item count within parentheses.
Also includes buttons and menus to take actions within the page. The subnavigation bar is
the main way to navigate within a feature.
5. Filter bar. Allows you to filter the display based on columns, fabrics, network scope, and
customized filters.
6. Expandable sidebar. Provides an area where you can save selected inventory items for
investigation later.
Detail Pages Clicking a fabric name, switch name, or port name in a table opens a detail page
for that object. The detail page displays additional information about the object and may
contain additional actions that you can perform.
The detail page is different depending on the context. For example, the detail page for a fabric
on the Inventory page is different from the detail page for the same fabric on the Discovery
page.
Tables In tables, if an entry is truncated (indicated by an ellipsis at the end of the entry), hover
the mouse over the entry to see the full text.
Many tables have an action menu that you can access by clicking the down arrow in the
rightmost column. Click this arrow to display additional actions that you can perform on the
associated object.
276 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm
On the right side of the subnavigation bar, the More button ( ) often contains a Bulk Select
option, which allows you to select and perform operations on one or more items at the same
time. For example, you can turn monitoring on for multiple fabrics, or you can create a tag and
apply it to multiple switches.
Create custom dashboards. SANnav provides several default dashboards, but you can
create custom dashboards to fit your specific needs. See Dashboards.
For detailed information for Brocade SANnav setup, configuration and navigation, refer to the
Brocade SANnav User Guide.
Note:
SANnav Global View is a separate product from SANnav Management Portal, and it
requires separate installation and licensing. You log in to the SANnav Global View
application and add portals to SANnav Global View, which then uses information in the
portals to build a global view of the dashboard and inventory.
SANnav Global View is compatible only with SANnav Management Portal instances that
are running the same version as Global View. Older versions of SANnav Management
Portal are not supported.
The following table lists the system and server requirements for SANnav Global View.
Note: The disk space requirement listed in the table is for SANnav Global View only. Be
sure to account for additional space required by the operating system and for saving files.
278 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm
Note: Use the latest generation processors for better SANnav performance.
Before unzipping the installation file, be sure to review and comply with the system and server
requirements and the installation prerequisites listed in the following sections:
System and Server Requirements for SANnav Global View
Installation Prerequisites for SANnav Global View
Download and copy the SANnav Global View software package to the server. The package
contains the SANnav Global View tarball.
1. 1. Download the SANnav Global View tarball ( for example,
Global_2.1.0_rc_bldxx-distribution.tar.gz) to the folder where you want to install the
application. NOTE Do not create the SANnav Global View installation folder with spaces in
the name; otherwise, installation will fail.
2. Untar the .gz file to extract the file to the current location. tar -xvzf
Global_2.1.0_rc_bldxx-distribution.tar.gz This step creates a directory with a name
similar to Global_2.1.0_rc_bldxx. This directory is referred to as the <install_home>
directory in this document.
3. Go to the <install_home>/bin directory.
4. Run the install-sannav.sh script to install SANnav Global View. The
./install-sannav.sh installation script checks whether an earlier instance of SANnav
Global View is installed, and if so, it prompts if you want to exit installation and take a
backup, or continue installation without backing up the server:
5. If you are prompted about migrating SANnav, enter one of the following options:
– Press Ctrl-C to exit the installation procedure. Either back up the server or uninstall the
existing SANnav instance. Then restart the installation script.
– Press Enter to proceed with migration. You are prompted to enter the location of the
existing SANnav installation.
As the installation proceeds, the script runs a pre-install requirements test. If any test fails, the
installation exits with error messages. You must fix the reported issues and re-run the install
script. After the diagnostics pass, installation of SANnav Global View software continues. On
successful installation of the software, the SANnav Global View server starts up. The startup
may take up to 10 minutes.
280 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm
1. Navigation bar. Contains links to feature pages. The SANnav link displays the page where
various settings and configurations can be performed.
2. Profile menu. Displays links for changing user preferences, displaying the SANnav
version, and logging out.
3. Subnavigation bar. Provides the page title and optional item count within parentheses.
Also includes buttons and menus to take actions within the page.
4. Filter bar. Allows you to filter the display based on columns, portal scope, and customized
filters.
Click Inventory to display inventory information about fabrics, switches, switch ports,
chassis, host and storage devices, and host and storage ports across all Management
Portal instances.
Click Events to show all application events that stem from a user or system action. A system
action is triggered under certain situations like when a portal is disconnected. You can filter
the events list by date range. Events from SANnav Management Portal instances are not
listed here.
282 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm
Click SANnav to discover Management Portals, manage filters, and perform various
configuration settings across several feature categories. The profile menu is where you can
add your phone number and change your password. It is also where you can enable Persist
Last Column Selection, wherein the system remembers your column customization, and
FICON Display. Column customization is the ability to choose what columns you want to see
in a tabular view. For example, by default, only a few of the many available columns in
inventory are shown, but you can customize the table by adding or removing columns.
When FICON Display is enabled, certain columns in the switch and switch port inventory
pages are rearranged for FICON mode.
Complete the following steps to download the software and documentation from the Brocade
IBM assist website:
IBM b-type Gen 7 Software/Firmware Download Instructions (IBM Assist Site for IBM
Brocade Software downloads)
Step-by-Step instructions for access to Fabric OS firmware (FOS), SANnav and b-type
Gen 7 IBM Network Advisor (INA).
Important: You must have active IBM Support in order to access Brocade Software for
any device.
Brocade FOS and SAN Management software may be accessed through the IBM Support
Fix Central, found at: https://www.ibm.com/support/fixcentral/options
SANnav dashboards allow you to gather additional information by drilling down into the
dashboard widgets. By default, the scope of SANnav Global View is set to All Portals (that is,
all Management Portal instances). You can choose one or more Management Portal
instances to change the scope.
284 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm
3. To view further details on the fabrics, click the colored segment of the widget. For example,
clicking the red segment displays a table showing the health scores for the fabrics in poor
health.
4. Click Show Details on the action menu of the fabric list to see details associated with that
health score. The Show Details option is available only if the score is less than 100.
A popup window displays the factors that cause deductions in the health score for this
fabric.
5. Click Back to return to the list of portals.
6. Select Show in SANnav Portal from the action menu to launch the Management Portal
instance responsible for the data you are viewing. You might need to make sure that
286 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm
popups are not blocked in your browser. If you are not logged in to this portal, credentials
(for this portal) are required.
The following procedure shows how to use the Global View dashboard to display switch
health across multiple Management Portal instances.
1. Click Dashboard & Reports in the navigation bar. On the bottom row of the dashboard, the
Switch Health widget displays. In this screen you see switches grouped by SANnav Portal
instances. You can select a different grouping from the drop-down list.
2. Hover over the graph to display details. Note that for the Firmware Version and Model
categories, more bars may be displayed in the graph than are listed in the row names on
the left. In this case, hover over an intermediate bar in the graph to display the details.
3. To see the details on all switches in a particular category, click the bar chart of the widget,
and then select Show Details.
288 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm
If the switch is configured with logical fabrics, you can log in to any of the logical fabrics for
which you have permission.
1. Launch Web Tools directly from a browser or from SANnav Management Portal.
– To launch directly from a Web browser, open your browser, enter the IP address of the
switch, and press Enter.
You can use HTTP or HTTPS, for example: http://10.77.77.77 or
https://10.77.77.77.
– To launch from SANnav Management Portal, locate the switch on the SANnav
Inventory page, click the down arrow to the right of the switch, and select View in
WebTools from the action menu.
2. Enter the user name, password, and logical switch name or fabric ID (FID). For the first
switch login, the default user name is admin and the default password is password. Web
Tools prompts you to change the default password. If you are logging in to a Virtual
Fabrics-enabled platform and you do not specify a logical switch, you are logged in to the
default logical switch, which uses fabric ID 128. For non-VF platforms, the FID option is not
displayed.
If you launch from SANnav Management Portal, you might not need to log in, depending
on the SANnav single sign-on configuration.
3. Click Login
290 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch11.fm
For more information about the SANnav Web Tools product, see Web Tools section of the
Brocade Fabric OS Web Tools User Guide, 9.0.x., found at:
https://www.broadcom.com/products/fibre-channel-networking/software/fabric-operati
ng-system
292 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch12.fm
12
When we discuss FC SAN Security, there are several challenges, and a layered security
model should be deployed to ensure it continues to be secure for years to come.
We will look at each of these layers independently, working from the outermost layers dealing
with Management Security to the innermost layers dealing with the data itself.
294 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch12.fm
We use IP Filter in FOS to allow or deny the entry of traffic across these interfaces. We also
want to make sure, if necessary, we equally limit both IPv4 and IPv6 traffic. The IP Filter
policy is a set of rules that are applied to the IP management interfaces and acts as a packet
filtering firewall. This firewall permits or denies the traffic to go through the IP management
interfaces according to the IP Filter rules.
As a preferred practice, non-secure IP protocols that are used for switch management such
as Telnet and HTTP should be blocked. SSH is enabled on the default IP Filter policy and SSL
should be configured to use HTTPS for web access.
The IP Filter policy should be used in environments where you need strict control of fabric
access. As with other ACL policies, it is important to regularly review and properly maintain
the IP Filter policy.
To display the current IP Filter in use, use the ipfilter --show command as show below:
switch:admin> ipfilter --show
Name: default_ipv4, Type: ipv4, State: active
Rule Source IP Protocol Dest Port Action
Below is an example a newly created IP Filter to deny an unsecure protocol such as Telnet,
which uses TCP Port 23. Note the IPv6 filter remains unchanged.
ipfilter --clone BlockTelnet -from default_ipv4
ipfilter --save BlockTelnet
ipfilter --addrule BlockTelnet -rule 1 -sip any -dp 23 -proto tcp -act deny
ipfilter --save
ipfilter --activate BlockTelnet
296 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch12.fm
Protocol Description: 12.7, “Access Control Lists” on page 298 shows the security protocols
that are supported by Fabric OS.
HTTPS Hyper Text Transfer Protocol over SSL (HTTPS) is a Uniform Resource Identifier
scheme that is used to indicate a secure HTTP connection. Web Tools supports
the use of HTTPS.
IPSec Internet Protocol Security (IPSec) is a framework of open standards for providing
confidentiality, authentication, and integrity for IP data transmitted over untrusted
links or networks.
LDAP Lightweight Directory Access Protocol (LDAP) with Transport Layer Security (TLS)
uses a certificate authority (CA). By default, LDAP traffic is transmitted unsecured.
With imported signed certificates, you can make LDAP traffic confidential and
secure by using Secure Sockets Layer (SSL) / TLS technology with LDAP.
SCP Secure Copy Protocol (SCP) is a means of securely transferring computer files
between a local and a remote host or between two remote hosts that use the
Secure Shell (SSH) protocol. Configuration upload and download support the use
of SCP.
Secure Syslog Secure syslog requires importing syslog CA certificates by using the
secCertMgmt command.
SFTP Secure File Transfer Protocol (SFTP) is a network protocol for securely
transferring files on a network. Configuration upload and download support the
use of SFTP.
SSH Secure Shell (SSH) is a network protocol that allows data to be exchanged over a
secure channel between two computers. Encryption provides confidentiality and
integrity of data. SSH uses public-key cryptography to authenticate the remote
computer and allow the remote computer to authenticate the user, if necessary.
SSL Secure Sockets Layer (SSL) is used by Fabric OS to support HTTPS. A certificate
must be generated and installed on each switch to enable SSL. This configuration
supports SSLv3, 128-bit encryption by default. It also supports TLSv1.0, TLSv1.1,
and TLSv1.2.
298 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch12.fm
On the HBA front, we have several options available to us. We can use Single Initiator Zones
to limit the number of LUNs that could potentially be compromised should the server be
compromised. On the storage device LUN Masking should be implemented to ensure only
those LUNs zoned to the HBA are accessible to the HBA through its zoned storage port. We
also have the option of enabling DH-CHAP to authenticate these devices when they join the
fabric.
On the Storage device front, we have similar options available. We should implement Single
Initiator Zones along with LUN Masking to limit exposure should a device be compromised.
We also have secure protocols such as DH-CHAP available to authenticate these devices
when they join the fabric.
Finally on the FC Switch front, we have even more options available. We can disable
persistently ports that are not used in the switch. We can also disable the ability of the online
ports from even becoming E-Ports (ISLs) by locking down their port configuration to disallow
that type of login. Leveraging a DCC Policy we can create an ACL of sorts whereby we can
control what device is allowed to log into specific ports by its WWN. Leveraging a SCC Policy
we can also use this same ACL concept to lock down what switches are allowed to participate
in a fabric by the switch’s WWN. These features are leveraged heavily when creating High
Integrity Fabrics required for most FICON environments. We can also use either DH-CHAP
or FCAP to authenticate switches, this authentication type is required when establishing
Encrypted E-Ports (ISLs) between switches, usually across long distances or connected via
DWDM devices. It is a best practice to enable encryption on ISLs or FCIP links extending
beyond the data center walls. DH-CHAP is a more simple deployment where shared secrets
are exchanged between switches, whereas FCAP used more secure signed digital
certificates for authentication.
Enabling in-flight ISL data compression or encryption slightly increases the latency as the
ASIC processes the frames for compression or encryption or both.
300 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch12.fm
QOS Port AE
Port Auto Disable: OFF
EX Port OFF
Mirror Port OFF
SIM Port OFF
Credit Recovery ON
F_Port Buffers OFF
E_Port Credits OFF
Fault Delay: 0(R_A_TOV)
NPIV PP Limit: 126
NPIV FLOGI Logout: OFF
CSCTL mode: OFF
TDZ mode: OFF
D-Port mode: OFF
D-Port over DWDM: OFF
Compression: OFF
Encryption: ON
10G/16G FEC: ON
16G FEC via TTS: OFF
The following steps describe an alternative way to enable FCAP for ISL Encryption:
Step 1 - Generate Switch CSR Files
switch:admin> seccertmgmt generate -csr fcap
Step 2 - Export Switch CSR Files to CA for signing
switch:admin> seccertmgmt export -csr fcap
Exact Sample (using complete and customer supplied input)
seccertmgmt export -csr fcap -protocol scp -ipaddr 10.1.1.1 -login
hostsusername -password hostspassword -remotedir /home/fcap/fcap_files
Step 3 - Import CA Cert to each Switch
switch:admin> seccertmgmt import -ca -client fcap
The following code is an exact sample (using complete and customer supplied input).
seccertmgmt import -ca -client fcap -protocol scp -ipaddr 10.1.1.1
-remotedir /home/fcap/fcap_files -certname ca.pem -login hostsusername
-password hostspassword
Step 4 (On Host) - Verify Switch CSRs
openssl req -text -noout -verify -in FcapSw.csr
Step 5 (On Host) - CA signs Switch CSRs
openssl ca -in FcapSw.csr -out FcapSw.pem
Step 6 - Verify CA signed Switch CSRs
openssl x509 -in FcapSw.pem -text -noout
Step 7 - Validate filename permission on host
chmod 777 FcapSw.pem
Step 8 - Import CA signed Switch CSRs
switch:admin> seccertmgmt import -cert fcap
The following code is an exact sample (using complete and customer supplied input).
302 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ch12.fm
Implementing ISL encryption using multiple ISLs between the same switch pairs requires that
all ISLs be configured for encryption, or none at all. No more than 4 ports on one ASIC can
be configured with encryption, compression, or both when running at 64 Gbps speed.
Additional ports can be used for data encryption, data compression, or both if the ports are
running lower than 64 Gbp speeds.
Together, the policy distribution and fabric-wide consistency settings provide a range of
control on the security policies from little or no control to strict control. Strict mode is required
for FICON environments that require High Integrity Fabrics be enabled.
There are other topics as well in this arena that are out of the scope of this document, such as
Encryption of Data-at-Rest and Encryption of Data-in-Flight. These are covered in more
detail in the Brocade Fabric OS Administration Guide, 9.0x
https://docs.broadcom.com/docs/FOS-90x-Admin-AG.
304 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ax01.fm
Fabric 1
Fabric 2
Fabric 3
Fabric 4
Fabric 5
Switch 1
Switch 2
Switch 3
Switch 4
Switch 5
306 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ax01.fm
A.3 Device
Use the table that is shown in Table A-3 to record information about the device that is used.
Server 1
Server 2
Server 3
Storage 1
Storage 2
Storage 3
Fabric information
Number of fabrics
Fabric licenses
Number/types of tapes
Number/types of HBAs
Application details
Performance
308 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ax01.fm
Servers
Server 1/HBA
Server 2/HBA
Server 3/HBA
Backup software
FC switch
Brocade
Storage
Array 1
Array 2
Tape library
Library
310 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ax02.fm
For more information about fabric operating system commands, see the Brocade Fabric OS
Command Reference Manual.
312 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ax02.fm
314 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ax02.fm
Receiver Stats
Recv Bytes / Pkts : 1105888500 / 10103382
Out-of-Order : 7 (High:23)
Duplicate ACKs : 0
RTT / Variance (High) : 20 ms (54 ms) / 0 ms (48 ms)
TCP Connection 24.0 HA-Type:Main Pri:Medium Conn:0x0a8a9526
===================================================
Local / Remote Port : 57309 / 3225
Duration : 28d22h
MSS : 1460 bytes
ARL Min / Cur / Max : 15000 / 15000 / 50000
ARL Reset Algo : Reset
Send Window
Size / Scale : 1249280 / 9
Slow Start Threshold : 16777216
Congestion Window : 16778676
Pkts InFlight : 0
Recv Window
Size / Scale : 1249792 (Max:1249792) / 9
SendQ Nxt / Min / Max : 0xfa9fa897 / 0xfa9fa897 / 0xfa9fa897
RecvQ Nxt / Min / Max : 0x22bad4e3 / 0x22bad4e3 / 0x22cde643
RecvQ Pkts : 0
Sender Stats
Sent Bytes / Pkts : 5975313510 / 51406132
Unacked Data : 0
Retransmits Slow / Fast : 462 / 207 (High:0)
SlowStart : 327
Receiver Stats
Recv Bytes / Pkts : 7321106136 / 51436908
Out-of-Order : 0 (High:28)
Duplicate ACKs : 750
RTT / Variance (High) : 20 ms (55 ms) / 0 ms (40 ms)
TCP Connection 24.0 HA-Type:Main Pri:Medium Conn:0x0a8a9527
===================================================
Local / Remote Port : 51163 / 3226
Duration : 28d22h
MSS : 1460 bytes
ARL Min / Cur / Max : 15000 / 15000 / 50000
ARL Reset Algo : Reset
Send Window
Size / Scale : 1249280 / 9
Slow Start Threshold : 16777216
Congestion Window : 16778676
Pkts InFlight : 0
Recv Window
Size / Scale : 1249792 (Max:1249792) / 9
SendQ Nxt / Min / Max : 0xfa9d726d / 0xfa9d71cd / 0xfa9d726d
RecvQ Nxt / Min / Max : 0xdd8bb7a2 / 0xdd8bb7a2 / 0xdd9ec9a2
RecvQ Pkts : 0
Sender Stats
Sent Bytes / Pkts : 5975138146 / 51404866
Unacked Data : 160
Retransmits Slow / Fast : 472 / 203 (High:0)
SlowStart : 342
Receiver Stats
316 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ax02.fm
Out-of-Order : 5 (High:29)
Duplicate ACKs : 0
RTT / Variance (High) : 20 ms (54 ms) / 0 ms (43 ms)
TCP Connection 24.0 HA-Type:Main Pri:Control Conn:0x0a8a9739
===================================================
Local / Remote Port : 57311 / 3225
Duration : 28d22h
MSS : 1460 bytes
ARL Min / Cur / Max : 0 / 0 / 50000
ARL Reset Algo : Reset
Send Window
Size / Scale : 1249280 / 9
Slow Start Threshold : 16777216
Congestion Window : 16778676
Pkts InFlight : 0
Recv Window
Size / Scale : 1249792 (Max:1249792) / 9
SendQ Nxt / Min / Max : 0x447a03c6 / 0x447a03c6 / 0x447a03c6
RecvQ Nxt / Min / Max : 0x0f08db12 / 0x0f08db12 / 0x0f1bed12
RecvQ Pkts : 0
Sender Stats
Sent Bytes / Pkts : 1392172536 / 12980231
Unacked Data : 0
Retransmits Slow / Fast : 241 / 4 (High:0)
SlowStart : 168
Receiver Stats
Recv Bytes / Pkts : 1560151020 / 13039059
Out-of-Order : 2 (High:11)
Duplicate ACKs : 12
RTT / Variance (High) : 20 ms (63 ms) / 0 ms (43 ms)
TCP Connection 24.0 HA-Type:Main Pri:Control Conn:0x0a8a973a
===================================================
Local / Remote Port : 51165 / 3226
Duration : 28d22h
MSS : 1460 bytes
ARL Min / Cur / Max : 0 / 0 / 50000
ARL Reset Algo : Reset
Send Window
Size / Scale : 1249280 / 9
Slow Start Threshold : 16777216
Congestion Window : 16778676
Pkts InFlight : 0
Recv Window
Size / Scale : 1249792 (Max:1249792) / 9
SendQ Nxt / Min / Max : 0x447b4df6 / 0x447b4df6 / 0x447b4df6
RecvQ Nxt / Min / Max : 0x8e1cf9f2 / 0x8e1cf9f2 / 0x8e300bf2
RecvQ Pkts : 0
Sender Stats
Sent Bytes / Pkts : 1392250536 / 12980470
Unacked Data : 0
Retransmits Slow / Fast : 240 / 4 (High:0)
SlowStart : 176
Receiver Stats
Recv Bytes / Pkts : 1560694216 / 13039703
Out-of-Order : 0 (High:14)
Duplicate ACKs : 13
RTT / Variance (High) : 20 ms (52 ms) / 0 ms (42 ms)
B.4 portStatsShow
Many port statistics are maintained. To display port traffic and error statistics, use the portstatsshow
CLI command. To get FC, VE, and GE detailed statistics on the CLI issue portstatsshow <port#>.
FC and VE ports use just the number alone <#>. GE interfaces use ge<#>. Below is an example of the
output for FC port 0.
<< Note to Editor: Table needs to be formatted. Compare it with Word doc. >>
7840-Side-A:FID128:admin> portstatsshow
Usage:
portstatsshow [--detail] PortNumber
OR
portstatsshow [--detail] -i <port_index | portindex_range> [-f]
OR
portstatsshow [--detail] <-slot | -s > <slot# | slotrange>
-i: Confirms port swap has been disabled and to give port index as operand
-x: Confirms port swap has been disabled and to give port index
in HEX format as operand
-f: ignores non-existing indexes
portindex_range:Specifies the range of port index <portindex1-portindex2>
(example: 12-14)
OR
portstatsshow [<SlotNumber>/]ge<PortNumber>
7840-Side-A:FID128:admin> portstatsshow ge2
ge_stat_tx_frms 542382084 GE transmitted frames
ge_stat_tx_octets 636962488472 GE transmitted octets
ge_stat_tx_ucast_frms 542237792 GE transmitted unicast frames
ge_stat_tx_mcast_frms 0 GE transmitted multicast frames
ge_stat_tx_bcast_frms 144292 GE transmitted broadcast frames
ge_stat_tx_vlan_frms 0 GE transmitted vlan frames
ge_stat_tx_pause_frms 0 GE transmitted pause frames
ge_stat_rx_frms 258254545 GE received frames
ge_stat_rx_octets 40648834562 GE received octets
ge_stat_rx_ucast_frms 258110264 GE received unicast frames
ge_stat_rx_mcast_frms 0 GE received multicast frames
ge_stat_rx_bcast_frms 144281 GE received broadcast frames
ge_stat_rx_vlan_frms 0 GE received vlan frames
ge_stat_rx_pause_frms 0 GE received pause frames
ge_err_crc 0 GE CRC Errors
ge_err_carrier 1 GE lost carrier sense
ge_err_jabber 0 GE jabbers
ge_stat_tx_octets 636962488472 GE transmitted
octets
ge_stat_tx_pkts64octets 347 GE transmitted
64byte octets
ge_stat_tx_pkts65to127octets 62670368 GE transmitted
65to127byte octets
ge_stat_tx_pkts128to255octets 21247343 GE transmitted
128to255byte octets
ge_stat_tx_pkts256to511octets 34952514 GE transmitted
256to511byte octets
318 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ax02.fm
tim_latency_vc 0- 3: 1 1 1 1
tim_latency_vc 4- 7: 1 1 1 1
tim_latency_vc 8-11: 1 1 1 1
tim_latency_vc 12-15: 1 1 1 1
fec_cor_detected 0 Count of blocks that were corrected by
FEC
fec_uncor_detected 0 Count of blocks that were left
uncorrected by FEC
er_enc_in 0 Encoding errors inside of frames
er_crc 0 Frames with CRC errors
er_trunc 0 Frames shorter than minimum
er_toolong 0 Frames longer than maximum
er_bad_eof 0 Frames with bad end-of-frame
er_enc_out 0 Encoding error outside of frames
er_bad_os 0 Invalid ordered set
er_pcs_blk 0 PCS block errors
er_rx_c3_timeout 0 Class 3 receive frames discarded due
to timeout
er_tx_c3_timeout 0 Class 3 transmit frames discarded due
to timeout
er_unroutable 0 Frames that are unroutable
er_unreachable 0 Frames with unreachable destination
er_other_discard 0 Other discards
er_type1_miss 0 frames with FTB type 1 miss
er_type2_miss 0 frames with FTB type 2 miss
er_type6_miss 0 frames with FTB type 6 miss
er_zone_miss 0 frames with hard zoning miss
er_lun_zone_miss 0 frames with LUN zoning miss
er_crc_good_eof 0 Crc error with good eof
er_inv_arb 0 Invalid ARB
er_single_credit_loss 0 Single vcrdy/frame loss on link
er_multi_credit_loss 0 Multiple vcrdy/frame loss on link
other_credit_loss 0 Link timeout/complete credit loss
phy_stats_clear_ts 0 Timestamp of phy_port stats clear
lgc_stats_clear_ts 0 Timestamp of lgc_port stats clear
B.5 gePortPerfShow
To monitor the data rates through the GE interfaces, use the geportperfshow CLI command.
7840-Side-A:FID128:admin> geportperfshow
Throughput of GE port
ge0 ge1 ge2 ge3 ge4 ge5 ge6 ge7 ge8
========================================================================
0 0 11.3m 0 11.1m 0 0 0 0
ge9 ge10 ge11 ge12 ge13 ge14 ge15 ge16 ge17 Total
================================================================================
0 0 0 0 0 0 0 0 0 22.5m
B.6 switchShow
The best way to show the status of a VE_Port is using the switchshow command. In the
example below, VE_Port 12 shows the port type (VE), port status (online), and the connected
320 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
Draft Document for Review April 5, 2021 12:31 pm 8497ax02.fm
device “Remote7810”. Be aware that for the GE interfaces, the notation FCIP indicates WAN.
FCIP terminology remains due to legacy output prior to IP Extension.
switchshow
Example:
Local7810:admin> switchshow
switchName: Local7810
switchType: 178.0
switchState: Online
switchMode: Native
switchRole: Principal
switchDomain: 1
switchId: fffc01
switchWwn: 10:00:88:94:71:5c:f7:e2
zoning: OFF
switchBeacon: OFF
FC Router: OFF
HIF Mode: OFF
Index Port Address Media Speed State Proto
==================================================
0 0 010000 -- N32 No_Module FC
1 1 010100 -- N32 No_Module FC
2 2 010200 -- N32 No_Module FC
3 3 010300 -- N32 No_Module FC
4 4 010400 -- N32 No_Module FC
5 5 010500 -- N32 No_Module FC
6 6 010600 -- N32 No_Module FC
7 7 010700 -- N32 No_Module FC
8 8 010800 -- N32 No_Module FC
9 9 010900 -- N32 No_Module FC
10 10 010a00 -- N32 No_Module FC
11 11 010b00 -- N32 No_Module FC
12 12 010c00 -- -- Online VE VE-Port
10:00:88:94:71:60:98:5e "Remote7810" (downstream)
13 13 010d00 -- -- Offline VE Disabled (Persistent)
14 14 010e00 -- -- Offline VE Disabled (Persistent)
15 15 010f00 -- -- Offline VE Disabled (Persistent)
ge0 cu 1G Offline FCIP Copper Disabled
ge1 cu 1G Offline FCIP Copper Disabled
ge2 id 10G Online LAN
ge3 id 10G Online LAN
ge4 id 10G No_Sync LAN Disabled (Persistent)
ge5 id 10G No_Sync LAN Disabled (Persistent)
ge6 id 10G Online FCIP
ge7 id 10G Online FCIP
B.7 portPerfShow
<< Note to Editor: For the code sample, snipit the code sample taken from the Word doc >>
PortPerfShow is a real-time iterative CLI output showing the current rate of traffic on the port.
Only FC/FICON and VE ports are shown with PortPerfShow, use geportperfshow for GE
interfaces. In this case, the two FC ports (0 & 1) are feeding into VE_Port 24 for transmission
across FCIP.
7840-Side-A:FID128:admin> portperfshow
B.8 portErrShow
<< Note to Editor: For the code sample, snipit the code sample taken from the Word doc >>
To view the number of transmitted/received frames and errors on F_Port (FC fabric ports) and
VE_Port (FCIP Virtual E_Ports), use porterrshow.
7840-Side-A:FID128:admin> porterrshow
322 IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
To determine the spine width of a book, you divide the paper PPI into the number of pages in the book. An example is a 250 page book using Plainfield opaque 50# smooth which has a PPI of 526. Divided
250 by 526 which equals a spine width of .4752". In this case, you would use the .5” spine. Now select the Spine width for the book and hide the others: Special>Conditional
Text>Show/Hide>SpineSize(-->Hide:)>Set . Move the changed Conditional text settings to all files in your book by opening the book file with the spine.fm still open and File>Import>Formats the
Conditional Text Settings (ONLY!) to the book files.
Draft Document for Review April 5, 2021 12:31 pm 8497spine.fm 323
IBM b-type Gen 7 SG24-8497-00
Installation, Migration, and Best ISBN 0738459437
(1.5” spine)
1.5”<-> 1.998”
789 <->1051 pages
IBM b-type Gen 7 SG24-8497-00
Installation, Migration, and Best ISBN 0738459437
(1.0” spine)
0.875”<->1.498”
460 <-> 788 pages
SG24-8497-00
IBM b-type Gen 7 Installation, Migration, and Best Practices GuideISBN 0738459437
(0.5” spine)
0.475”<->0.873”
250 <-> 459 pages
IBM b-type Gen 7 Installation, Migration, and Best Practices Guide
(0.2”spine)
0.17”<->0.473”
90<->249 pages
(0.1”spine)
0.1”<->0.169”
53<->89 pages
To determine the spine width of a book, you divide the paper PPI into the number of pages in the book. An example is a 250 page book using Plainfield opaque 50# smooth which has a PPI of 526. Divided
250 by 526 which equals a spine width of .4752". In this case, you would use the .5” spine. Now select the Spine width for the book and hide the others: Special>Conditional
Text>Show/Hide>SpineSize(-->Hide:)>Set . Move the changed Conditional text settings to all files in your book by opening the book file with the spine.fm still open and File>Import>Formats the
Conditional Text Settings (ONLY!) to the book files.
Draft Document for Review April 5, 2021 12:31 pm 8497spine.fm 324
IBM b-type Gen 7 SG24-8497-00
Installation, Migration, ISBN 0738459437
(2.5” spine)
2.5”<->nnn.n”
1315<-> nnnn pages
IBM b-type Gen 7 SG24-8497-00
Installation, Migration, and Best ISBN 0738459437
Practices Guide
(2.0” spine)
2.0” <-> 2.498”
1052 <-> 1314 pages
Back cover
Draft Document for Review April 5, 2021 12:31 pm
SG24-8497-00
ISBN 0738459437
Printed in U.S.A.
®
ibm.com/redbooks