Page 1 of 41

Cisco MDS NX-OS Installation/Implementation Guide

Table of Contents
Introduction...................................................................................................................................................... 2
Change Summary............................................................................................................................................. 2
Start a Switch................................................................................................................................................... 2
Connect to the Console Port......................................................................................................................... 2
Power up the New Switch............................................................................................................................. 3
Prerequisites................................................................................................................................................. 3
Initial Setup Routine...................................................................................................................................... 4
License Key...................................................................................................................................................... 8
Advanced Feature Licensing......................................................................................................................... 8
Obtain the License Key File.......................................................................................................................... 8
Install the License Key File............................................................................................................................ 9
Cisco MDS NX-OS Software Image Upgrade............................................................................................... 10
General Upgrade Guidelines....................................................................................................................... 10
Prepare for Software Image Upgrade on a Switch with Storage Services Module (SSM) Installed............10
Upgrade Software Image via Terminal........................................................................................................ 11
Cisco MDS NX-OS Software Image Downgrade..........................................................................................15
Consideration.............................................................................................................................................. 15
General Downgrading Guidelines............................................................................................................... 15
Downgrade Software Image via Terminal................................................................................................... 16
Inter-Switch Link (ISL)................................................................................................................................... 16
Trunking...................................................................................................................................................... 16
Trunking E Ports....................................................................................................................................................... 16
Trunking F Ports........................................................................................................................................................ 16
Trunking Misconfiguration........................................................................................................................... 16
Enable Trunking.......................................................................................................................................... 17
PortChannel.................................................................................................................................................... 17
Configuration Guidelines and Restrictions.................................................................................................. 18
Create a PortChannel................................................................................................................................. 18
Verification.................................................................................................................................................. 18
Fibre Channel over IP (FCIP)......................................................................................................................... 19
FCIP High-Availability Solutions.................................................................................................................. 19
Fibre Channel PortChannels..................................................................................................................................... 19
FSPF......................................................................................................................................................................... 19
Prerequisites............................................................................................................................................... 20
Basic FCIP Configuration Outline................................................................................................................ 20
Configure FCIP........................................................................................................................................... 21
Verify the FCIP Configuration...................................................................................................................... 22
Configure FCIP High-Availability................................................................................................................. 23
PortChannel Configuration with FCIP Tunnels.........................................................................................................23
Tune FCIP Performance............................................................................................................................. 23
Tune TCP Parameters.............................................................................................................................................. 23
Inter-VSAN Routing (IVR).............................................................................................................................. 24

retain original - ACT+3

 Page 2 of 41

Autonomous Fabric ID................................................................................................................................ 25

IVR Configuration Task List......................................................................................................................... 25
Enable IVR.................................................................................................................................................. 25
Distribute the IVR Configuration using CFS................................................................................................ 25
Enable Configuration Distribution.............................................................................................................................. 25
IVR NAT...................................................................................................................................................... 25
Enable IVR NAT........................................................................................................................................................ 25
Transit VSAN Guidelines............................................................................................................................. 26
Configure IVR Topology.............................................................................................................................. 26
Lock the Fabric............................................................................................................................................ 26
Commit the Changes.................................................................................................................................. 27
Discard the Changes................................................................................................................................... 27
Clear a Locked Session.............................................................................................................................. 27
Verify the IVR Topology.............................................................................................................................. 27
Connecting QFX3500 acting FCoE gateway switch to Cisco's native FC switch (MDS or NEXUS)........27
Fixing the NPIV-UNSUPPORTED issue if the Cisco native FC switch is used...........................................28
Cisco MDS Golden Settings.......................................................................................................................... 30

Introduction
This document details the procedure to configure and implement new switches and fabrics with Cisco MDS
NX-OS on Cisco MDS 9000 Family of Fibre Channel Switch and Director Class products.

Change Summary
Versio Date Change / Summary
n Updated by
01 09/28/2010 Yew Oon Sian Initial document
02 04/19/2011 Yew Oon Sian Removed Even Fabric Configuration in the example to
avoid confusion
03 06/09/2011 Yew Oon Sian Added default parameters to configuration template
04 08/17/2011 Yew Oon Sian Added Compatibility Check and Troubleshooting section
05 29/Nov/2011 Michael E Smith Removed references to SAN-OS 3.x
06 6/Nov/2012 Sebastian Luque Corrected configuration backup. Included a comment to
reference an example
07 14/Jun/2013 Cancilla, Bret A.  Changed to OASIS format

 Added support for Fibre Channel over Ethernet

(FCoE)
08 17/Jun/2016 Cancilla, Bret A. Updated to reflect Golden Settings
09 28/Jun/2016 Cancilla, Bret A. Added more color to Golden Settings and corrected
spelling errors.
10 06/Feb/2017 Cancilla, Bret A. Added detail on switch naming convention.

retain original - ACT+3

 Page 3 of 41

Versio Date Change / Summary

n Updated by
11 06/Sep/2017 Haris, Nur Hazura Added the SNMP configuration for alerts monitoring.
Replaced the email group address to Storage Ops
mailbox.
Updated the Golden Settings to reflect the new changes
above.
12 20/Nov/2017 Dale G Brooks Updated license portal hyperlink

Start a Switch

Connect to the Console Port

The console port, labeled “Console,” is an RS-232 port with RJ-45 interface. It is an asynchronous
(async) serial port; any device connected to this port must be capable of asynchronous transmission.

Use this port to create a local management connection to set the IP address and other initial
configuration settings before connecting the switch to the network for the first time.

To connect the console port to a computer terminal, the computer must support VT100 terminal
emulation. The terminal emulation software – frequently an application such as HyperTerminal or
PuTTY – makes communication between the switch and computer possible during setup and
configuration.

To connect the console port to a computer terminal, follow these steps:

1. Configure the terminal emulator program to match the following default port characteristics:
 9,600 baud
 8 data bits
 1 stop bit
 No parity
2. Connect the supplied console cable (normally blue color) to the console port and to the
computer serial port.

Power up the New Switch

A setup utility automatically launches the first time a switch being powered up and guides through
the basic configuration.

The setup utility allows building an initial configuration file using the System Configuration Dialog.
The setup starts automatically when a device has no configuration file in NVRAM.

Press [Ctrl+C] at any prompt to skip the remaining configuration options and proceed with what is
configured up to that point, except for the administrator password.

Press [Enter] to skip answers to any questions. If a default answer is not available (ex:, the device
hostname), the device uses what was previously configured and skips to the next question.

Prerequisites
Before setting up a switch, the tasks below should be completed.

1. Obtain general information:

 Switch Name

retain original - ACT+3

 Page 4 of 41

Switches names should follow three letter site code, three letter function code, sequential
number, i.e. sssFCfnn (HCFFCX01).
Where the function code is one of the following registered functions (FCH, FCX, FCS,
and FCE are targeted to sites with more than 2500 host potential):
FCH – FC Host Edge switches (Hosts/initiators only)
FCX – FC Core switches (ISLs/PortChannel between FCH/FCS/FCE only. No
initiators/targets)
FCS – FC Storage Edge switches (storage/targets only)
FCE – FC Edge switches (for the Edge – Core Model. Host/Storage initiators/targets)
FCF – FC Fabric (for site with less than 2500 hosts)
If in doubt contact storage engineering.
 Domain Name
 Switch IP Address and Subnet Mask
 Gateway
 DNS
2. Obtain the License Key file.
3. Create a touch file on the target TFTP server in tftpboot directory named <switchname>_cfg.txt
with rwxrwxrwx permissions (chmod 777).
4. Switch’s IP address must be registered with SMTP services in order to make callhome email
feature work. Refer to SMTP External Messaging Service site on
http://intrattd.na.xom.com/EMGS/ITHELP/solutions/email/smtpservices/smtp_service.html or
contact [email protected] to verify registration.
5. Pick an NTP server in the same region as the switch with “w32tm /monitor /domain:<domain>”
command.
6. The MDS Active Directory groups (EMIT.MDS.ADM.GBL.RWXD.GG and
EMIT.MDS.OPER.GBL.RX.GG) control access to the switches.
NOTE: The owner of the group will grant approval/disapproval of requests for users joining that
Active Directory group. Any user that is part of those Active Directory groups will gain the level of
access predefined for that group (such as network-admin or network-operator).

Initial Setup Routine

Configure out-of-band management on the mgmt 0 interface through console port.

1. After powering-up the switch, it will boot automatically until it prompts to enter password for
administrator. Enter the new password.
Enter the password for “admin:” <admin-password>
Confirm the password for “admin:” <admin-password>

NOTE: If the password is weak, the following message will be prompted.

Password should contain characters from at least three of the following classes: lower case
letters, upper case letters, digits and special characters.

2. When prompted to enter basic configuration dialog, enter no to skip the dialog.
Would you like to enter the basic configuration dialog (yes/no): no
3. Configure the out-of-band (mgmt 0) management. In this example, we are configuring the
management port with IP address 172.19.26.39/27.
switch# configure terminal

retain original - ACT+3

 Page 5 of 41

Enter configuration commands, one per line. End with CNTL/Z.

switch(config)# interface mgmt 0
switch(config-if)# ip address 172.19.26.39 255.255.255.224
switch(config-if)# no shutdown
switch(config-if)# exit
switch(config)#
4. Configure the default gateway. Enter the default gateway IP address, e.g. 172.19.26.62.
switch(config)# ip default-gateway 172.19.26.62
5. Enable telnet service.
switch(config)# feature telnet
switch(config)# end
switch#
6. By using another computer, verify the switch is accessible via telnet.
# telnet 172.19.26.39

Verify the software image version. Refer to NetApp FAS-V Series Firmware and Software Versions,
SAN-CLARiiON: Flarecode, Firmware, and Software Versions, SAN-Symmetrix: Microcode, Drivers,
and Software Versions, and Fibre Channel over Ethernet (FCoE) Requirements Document for
current certified version.

switch# show version

. . .
Software
BIOS: version 1.0.16
kickstart: version 3.3(2)
system: version 3.3(2)
7. If the version shown is not the same as the current certified version, proceed to the Cisco MDS
NX-OS Software Image Upgrade section.
8. Follow the instructions in the License Key section to install all required licenses.
9. If the switch is accessible via telnet, copy and configure the template below to Notepad. This
configuration template is only applicable to the first switch in a physical fabric.
! General Configuration
switchname XXXXXXXX
system default switchport shutdown
system default zone mode enhanced
system default zone distribute full
feature scheduler
ip domain-name XXXXX.com
ip name-server YYYYY 144.201.7.31
ntp distribute
ntp server ZZZZZ prefer
ntp commit
ssh key rsa 1024 force
feature ssh

! TACACS+ Configuration
feature tacacs+

retain original - ACT+3

 Page 6 of 41

tacacs-server key 7 AAAAAAAA

tacacs-server host 158.35.215.36
tacacs-server host 172.19.17.21
tacacs-server host 192.168.13.202
aaa group server tacacs+ tacacs+
server 158.35.215.36
server 172.19.17.21
server 192.168.13.202
aaa authentication login default group tacacs+
aaa authentication login console group tacacs+
aaa accounting default group tacacs+

! Callhome Configuration
snmp-server contact Enterprise Storage Operations @ +1-713-656-4357
(option 4)
snmp-server location SiteX, CountryX
snmp-server community ECCSANADMINRO group network-operator
snmp-server community ECCSANADMINRW group network-admin
callhome
contract-id 1234567
customer-id 1234567
switch-priority 4
site-id 1234567
email-contact [email protected]
phone-contact +1-800-446-3272
streetaddress SiteAddress
distribute
destination-profile full_txt message-level 3
destination-profile short_txt message-level 4
destination-profile xml message-level 3
destination-profile short_txt email-addr GSC-EMIT-STORAGE-
[email protected]
transport email smtp-server smtp.na.xom.com port 25
transport email from [email protected]
transport email reply-to [email protected]
enable
periodic-inventory notification interval 30
periodic-inventory notification timeofday YY:ZZ
commit

! VSAN Configuration
vsan database
vsan 1 suspend
vsan 101 name “MV_Odd”
vsan 801 name “Virtualization_801”
vsan 3800 name “Links”
device-alias mode enhanced
fcdomain distribute
fcdomain priority 64 vsan 101
fcdomain domain nn static vsan 101
fcdomain priority 64 vsan 801
fcdomain domain nn static vsan 801
fcdomain priority 64 vsan 3800
fcdomain domain nn static vsan 3800
fcdomain commit vsan 101

retain original - ACT+3

 Page 7 of 41

fcdomain commit vsan 801

fcdomain commit vsan 3800

! SNMP Configuration for Alert Monitoring

snmp-server community SCOMRO group network-operator
snmp-server host 10.128.5.73 traps version 2c SCOMRO udp-port 162
snmp-server host 10.128.5.77 traps version 2c SCOMRO udp-port 162
snmp-server host 10.128.5.53 traps version 2c SCOMRO udp-port 162
snmp-server enable traps entity
snmp-server enable traps link
snmp-server enable traps callhome

! Scheduler Configuration
scheduler job name daily_run_backup
copy running-config tftp://144.201.242.109/(hostname)_cfg.txt
exit
scheduler schedule name daily_backup
job name daily_run_backup
time daily YY:ZZ
end

10. General Configuration:

switchname XXXXXXXX  switchname KULMDS01

ip domain-name XXXXX.com  ip domain-name upstreamaccts.xom.com

ip name-server YYYYY 144.201.7.31  ip name-server 10.194.1.1

144.201.7.31
where YYYYY is the DNS server at the site where the equipments
being installed. To display local DNS information, logon to any
Windows server which is native to the installation site, and issue
“ipconfig /all” command at the DOS prompt.

ntp server ZZZZZ prefer  ntp server malmklvn3.upstreamaccts.xom.com

prefer
where ZZZZZ is the NTP server closest to the site where the
equipments being installed. To display a list of NTP servers, logon to
any Windows server which is native to the installation site, and issue
“w32tm /monitor /domain:upstreamaccts.xom.com” command at the DOS
prompt. Pick an NTP server in the same region as the switch.

TACACS+ Configuration:

tacacs-server key 7 AAAAAAAA  tacacs-server key 7 12345678

where AAAAAAAA is the security key obtainable from the LAN
share.

Callhome Configuration:

snmp-server location SiteX, CountryX  snmp-server location

Houston , TX 77060
streetaddress SiteAddress  streetaddress EXXONMOBIL CORPORATION ,
233 BENMAR , HOUSTON , US , 77060

retain original - ACT+3

 Page 8 of 41

periodic-inventory notification timeofday YY:ZZ  periodic-inventory

notification timeofday 00:00

VSAN Configuration:
The template is to be used for Odd Fabric, change the following for Even Fabric:
vsan 102 name “MV_Even”
vsan 802 name “Virtualization_802”

WARNING: If this is not the first switch in the fabric, remove all “fcdomain priority” commands.
fcdomain priority 64 vsan 801

Refer to Domain ID spreadsheet for Domain ID reserved for the new switch. Use the same
Domain ID for all VSANs within the switch.
fcdomain domain nn static vsan 101  fcdomain domain 33 static
vsan 101
fcdomain domain nn static vsan 801  fcdomain domain 33 static
vsan 801
fcdomain domain nn static vsan 3800  fcdomain domain 33 static
vsan 3800

The template is to be used for Odd Fabric, change the above for Even Fabric.

SNMP Configuration for Alert Monitoring:

At least 3 IP addresses of the SCOM Production Z are configured. The
community string used to send the SNMP trap from Cisco MDS to SCOM
Production Z is SCOMRO. The SNMP port number is 162.

Scheduler Configuration:

time daily YY:ZZ  time daily 16:00

where YY:ZZ is time to run the backup based on UTC

(Coordinated Universal Time).
Choose the time based on the region:

Region Backup Time

EA: Europe & Africa 00:00
SA: South America 04:00
NA: North America 06:00
AP: Asia Pacific 16:00
ME: Middle East 21:00

For more information on Backup Schedule configuration, refer to OASIS Document Cisco MDS 9000

retain original - ACT+3

 Page 9 of 41

Series Initialization & SAN-OS Upgrade Guide page 5.

11. Copy and paste the configuration section by section to the console under global configuration
mode.
switch# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
switch(config)#

License Key

Advanced Feature Licensing

The following features should be enabled on each switch and fully licensed before production
implementation:

 Enterprise Package (ENTERPRISE_PKG)

 Fabric Manager Server Package (FM_SERVER_PKG)

Obtain the License Key File

EMC will provide a license sheet (on paper) that contains a Product Authorization Key (PAK) and a
blank for the switch serial number. Obtain switch serial number and its WWN beforehand:

switch# show license host-id

License hostid: VDH=FOX0923023Q
switch# show wwn switch
Switch WWN is 20:00:00:0d:ec:21:98:00
1. Navigate to https://fusion.emc.com/assetMgmt/ and log on with the EMC Powerlink ID.
2. Follow the on-screen instructions. The goal is to convert the PAK to a license key file which
binds the switch serial number with the authorization key.

After all the PAKs are entered, a display should appear, similar to the following example:

3. To download the license file, click on the download button and save the file to the local

retain original - ACT+3

 Page 10 of 41

computer.

NOTE: The license key file is digitally signed to only authorize use on the requested switch. The
requested features are also enabled once the Cisco NX-OS software on the specified switch
accesses the license key file. Install the license key in the specified MDS switch without making any
modifications.

Install the License Key File

Follow the steps below to install the license key:
1. Use the PING command to ensure the TFTP/FTP server can be contacted.
switch# ping 172.19.26.35
64 bytes from 172.19.26.35: icmp_seq=1 ttl=128 time=0.158 ms
. . .
2. Copy the license key file from the TFTP/FTP server to bootflash: and verify the results.
switch# copy tftp://172.19.26.35/ MDS20080424185329067.lic bootflash:
MDS20080424185329067.lic
Trying to connect to tftp server......
/
TFTP get operation was successful
switch# dir bootflash:
. . .
317 Apr 24 19:39:39 2008 MDS20080424185329067.lic
. . .
3. Perform the installation by issuing the install license command on the active supervisor module
from the switch console.
switch# install license bootflash: MDS20080424185329067.lic
Installing license ..done
4. Issue the show license brief command to display a list of license files installed on the switch.
switch# show license brief
MDS20080424185329067.lic
MDS20080424185726061.lic
MDS20080424190235381.lic
MDS20080424190434051.lic
5. To display information about a specific license file installed on the switch, issue command
below:
switch# show license file MDS20080424185329067.lic

retain original - ACT+3

 Page 11 of 41

NOTE: It is recommended to save a backup of the installed license files. The backup file should
always be .tar extension.
CETMDS81# copy licenses bootflash:CETMDS81_Licenses.tar
Backing up license done

Cisco MDS NX-OS Software Image Upgrade

General Upgrade Guidelines

Use the following guidelines when upgrading the software image:

 Scheduling – Schedule the upgrade when the fabric is stable and steady. Ensure that everyone
who has access to the switch or the network is not configuring the switch or the network during
this upgrade, because all configurations will be disallowed at this time.
 Hardware – Avoid power interruptions to the hardware during any installation procedures;
power interruption problems can corrupt the software image.
 Connectivity – Connectivity to remote servers to retrieve software images requires a
configured IP address for the management Ethernet port on the switch (interface mgmt0) and
make sure that the switch has a route to the remote server.
 Software Images – Ensure that the specified system and kickstart images are compatible with
each other.
 Save Current Configuration – Save running-config to startup-config (NVRAM).
 Backup Configuration – Keep a copy of the configuration outside of switch (FTP, TFTP, etc.)
 Save Tech-Support – Keep a copy of the tech-support detail file.
 Redundancy – Perform software image upgrade, “One Switch at a Time.”

Prepare for Software Image Upgrade on a Switch with Storage Services Module
(SSM) Installed
If the switch has SSM installed, ensure that the admin mode on the SSM interfaces is correctly
configured.

1. Verify the operational mode for each interface on the SSM using the command below.
CETMDS81# show interface fc9/1 – 32 brief
----------------------------------------------------------------------
Interface Vsan Admin Admin Status SFP Oper Oper Port
Mode Trunk Mode Speed Channel
Mode (Gbps)
----------------------------------------------------------------------
fc9/1 3 auto -- up -- F 2 --
. . .
2. Change the configuration of the first interface of the port groups when the admin mode is auto.
These are the interfaces: 1, 5, 9, 13, 17, 21, 25, and 29. Do not leave the port mode for these
interfaces set to auto.
 If the current operating port mode is F or FL, set the admin mode to FX.
CETMDS81(config)# interface fc9/1
CETMDS81(config-if)# switchport mode fx
 If the current operating port mode is E or TE, set the admin mode to E.

retain original - ACT+3

 Page 12 of 41

CETMDS81(config)# interface fc9/29

CETMDS81(config-if)# switchport mode e
3. Change the configuration for the remaining three ports in each port group (2-4, 6-8, 10-12, 14-
16, 18-20, 22-24, 26-28, and 30-32). Do not leave the port mode for these interfaces set to auto
or E.
 If the admin mode of these ports is auto or E and the first interface in the port group does
not have admin mode of E, change the admin port mode to FX.
CETMDS81(config)# interface fc9/2
CETMDS81(config-if)# switchport mode fx

 If the first interface in the port group has admin mode E or is operating in E port mode,
change the admin state of remaining ports to shutdown.
CETMDS81(config)# interface fc9/30 - 32
CETMDS81(config-if)# shutdown

NOTE: Only the first port in a port group is available for traffic in E port mode. Shutting down
the other three interfaces in the port group disrupts the traffic on those interfaces.
4. Save the configuration.

Upgrade Software Image via Terminal

The software image can be upgraded through CLI via terminal to all modules in any supported Cisco
MDS 9000 Family switches.

1. Prepare the new image files and check its file sizes. Calculate the total size of both files.
For example:
m9100-s2ek9-kickstart-mz.4.2.3.bin (kickstart image) file size: 18,678,784 bytes
m9100-s2ek9-mz.4.2.3.bin (system image) file size: 52,766,038 bytes

18,678,784 + 52,766,038 = 71,444,822 bytes

2. Display the details of the currently installed images, by entering the following command.
KLLMDS01# show version
. . .
Software
BIOS: version 1.0.16
kickstart: version 3.3(2)
system: version 3.3(2)

BIOS compile time: 10/23/08

kickstart image file is: bootflash:/m9100-s2ek9-kickstart-
mz.3.3.2.bin
kickstart compile time: 10/3/2008 11:00:00 [11/07/2008 09:17:47]
system image file is: bootflash:/m9100-s2ek9-mz.3.3.2.bin
system compile time: 10/3/2008 11:00:00 [11/07/2008 09:36:21]
. . .

3. Display the current boot variables by entering the following command.

KLLMDS01# show boot

kickstart variable = bootflash:/m9100-s2ek9-kickstart-mz.3.3.2.bin

system variable = bootflash:/m9100-s2ek9-mz.3.3.2.bin

retain original - ACT+3

 Page 13 of 41

No module boot variable set

If you have an SSM module installed, the output will be similar as below. Take note on the
location of SSI image, e.g. modflash://9-1/

CETMDS81# show boot

sup-1
kickstart variable = bootflash:/m9500-sf2ek9-kickstart-mz.3.0.2a.bin
system variable = bootflash:/m9500-sf2ek9-mz.3.0.2a.bin
sup-2
kickstart variable = bootflash:/m9500-sf2ek9-kickstart-mz.3.0.2a.bin
system variable = bootflash:/m9500-sf2ek9-mz.3.0.2a.bin
Module 9
ssi variable = modflash://9-1/m9000-ek9-ssi-mz.3.0.2m.bin
4. Display the contents of the bootflash: filesystem. Be sure to include the colon (:) after bootflash.
Look at the total free size to confirm the two new images can fit in the bootflash: directory.
KLLMDS01# dir bootflash:
. . .
Usage for bootflash://sup-local
82991104 bytes used
77186048 bytes free
160177152 bytes total

In this example, 77,186,048 bytes are free, so the directory can accommodate the two new
images. If the total size of new images is bigger than free space, delete unwanted files by using
the command:

KLLMDS01# delete bootflash:unwantedfile.bin

CAUTION: Do not delete any license files.

NOTE: Do not delete current images. If there is insufficient space for both the current images
and new images, copy the current images to the TFTP server. Only after the current images
are copied onto the TFTP server, they may be deleted.
5. From the CLI, determine the software version for both system and kickstart images. Use the
show version command to determine that the system and kickstart image versions match and
are compatible.
KLLMDS01# show version image bootflash:m9100-s2ek9-kickstart-
mz.3.3.2.bin
image name: m9100-s2ek9-kickstart-mz.3.3.2.bin
kickstart: version 3.3(2)
compiled: 10/3/2008 11:00:00 [11/07/2008 09:17:47]

KLLMDS01# show version image bootflash:m9100-s2ek9-mz.3.3.2.bin

image name: m9100-s2ek9-mz.3.3.2.bin
bios: v1.0.16(10/23/08)
system: version 3.3(2)
compiled: 10/3/2008 11:00:00 [11/07/2008 09:36:21]

If you have SSM module installed, determine the SSI image version.

KLLMDS01# show version image modflash://9-1/m9000-ek9-ssi-mz.3.3.2.bin

retain original - ACT+3

 Page 14 of 41

NOTE: Do not assume the image versions by its filenames.

6. Use the PING command to ensure the TFTP/FTP server can be contacted. Copy the new
system and kickstart images from the TFTP/FTP server to bootflash: and verify the results.
KLLMDS01# ping 172.19.26.35
64 bytes from 172.19.26.35: icmp_seq=1 ttl=128 time=0.158 ms
. . .

KLLMDS01# copy tftp://172.19.26.35/m9100-s2ek9-kickstart-mz.4.2.3.bin

bootflash:m9100-s2ek9-kickstart-mz.4.2.3.bin
Trying to connect to tftp server......
/
TFTP get operation was successful

KLLMDS01# copy tftp://172.19.26.35/m9100-s2ek9-mz.4.2.3.bin

bootflash:m9100-s2ek9-mz.4.2.3.bin
Trying to connect to tftp server......
/
TFTP get operation was successful

If you have an SSM module installed, copy the SSI image to modflash:, e.g. modflash://9-1/

CETMDS81# copy tftp://169.254.34.10/m9000-ek9-ssi-mz.4.2.3.bin

modflash://9-1/m9000-ek9-ssi-mz.4.2.3.bin
Trying to connect to tftp server......
/
TFTP get operation was successful

Verify the images have been copied.

KLLMDS01# dir bootflash:

. . .
14581760 Oct 13 23:02:02 2009 m9100-s2ek9-kickstart-
mz.3.3.2.bin
18678784 Jun 27 23:11:16 2010 m9100-s2ek9-kickstart-
mz.4.2.3.bin
55596324 Oct 13 23:03:59 2009 m9100-s2ek9-mz.3.3.2.bin
52766038 Jun 27 23:12:11 2010 m9100-s2ek9-mz.4.2.3.bin
. . .

If you have an SSM module installed, verified the SSI image has been copied.
CETMDS81# dir modflash://9-1
. . .
15277810 Jul 05 11:23:56 2010 m9000-ek9-ssi-mz.3.3.2.bin
19194301 Jul 05 11:27:20 2010 m9000-ek9-ssi-mz.4.2.3.bin
. . .
7. Verify the MD5 checksum of the copied files to ensure its integrity. Refer the MD5 hash from
GES-L-2395.
KLLMDS01# show file bootflash:m9100-s2ek9-kickstart-mz.4.2.3.bin
md5sum
5af1849ef6e650b34b91f92576cf72bb

retain original - ACT+3

 Page 15 of 41

KLLMDS01# show file bootflash:m9100-s2ek9-mz.4.2.3.bin md5sum

e4b71de5be11d0ac92b2103d7bedb719
8. Determine the new software version for both system and kickstart images. Enter the commands
below:
KLLMDS01# show version image bootflash:m9100-s2ek9-kickstart-
mz.4.2.3.bin

KLLMDS01# show version image bootflash:m9100-s2ek9-mz.4.2.3.bin

If you have SSM module installed, determine the new SSI image version.

KLLMDS01# show version image modflash://9-1/m9000-ek9-ssi-mz.4.2.3.bin

9. Use the show install all impact command to view the effect of updating the system from the
running image to a specified software image version. Enter this command:
KLLMDS01# show install all impact kickstart bootflash:m9100-s2ek9-
kickstart-mz.4.2.3.bin system bootflash:m9100-s2ek9-mz.4.2.3.bin
. . .
Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes non-disruptive reset
. . .

If you have SSM module installed, include SSI image in the command as below.

CETMDS81# show install all impact kickstart bootflash:m9500-sf2ek9-

kickstart-mz.4.2.3.bin system bootflash:m9500-sf2ek9-mz.4.2.3.bin ssi
modflash://9-1/m9000-ek9-ssi-mz.4.2.3.bin

NOTE: If the new images have incompatibilities with features that are either used or configured
on the installed release, the system response will provide the reason in the particular module.
For example, “running-config is not supported by new image”, “Incompatible image”, etc.

10. Use the show incompatibility system command to obtain further information on what needs to be
disabled in order to perform a non-disruptive installation. You can use this command to verify
once more to make sure no compatibility issue will occurred during installation.
KLLMDS01# show incompatibility system bootflash:m9100-s2ek9-
mz.4.2.3.bin
No incompatible configurations
Upgrade software using the install all command. When asked to continue, type “Y”.
KLLMDS01# install all kickstart bootflash:m9100-s2ek9-kickstart-
mz.4.2.3.bin system bootflash:m9100-s2ek9-mz.4.2.3.bin
. . .
Do you want to continue with the installation (y/n)? [n] y

Install is in progress, please wait.

. . .
Rebooting the switch to proceed with the upgrade.
Telnet will now be disabled.

If you have SSM module installed, include SSI image in the command as below.

retain original - ACT+3

 Page 16 of 41

CETMDS81# install all kickstart bootflash:m9500-sf2ek9-kickstart-

mz.4.2.3.bin system bootflash:m9500-sf2ek9-mz.4.2.3.bin ssi
modflash://9-1/m9000-ek9-ssi-mz.4.2.3.bin

NOTE: On dual supervisor systems, open console windows for both active and standby
supervisors and observe the system switchover between supervisors during the install all
process. On MDS switches with a single supervisor, this will be a disruptive upgrade.

11. On single supervisor systems, verify the system version after reboot. On dual supervisors
systems, you need to access to the previous standby supervisor, because it is now an active
supervisor after system switchover.

NOTE: It takes a little bit longer for MDS 9500 Series to upgrade because all the modules will
be upgraded in sequence. At this time, configuration will be locked as upgrade is in progress.

KLLMDS01# show version

. . .
Software
BIOS: version 1.0.16
loader: version N/A
kickstart: version 4.2(3)
system: version 4.2(3)
. . .
12. Display the new boot variables.
KLLMDS01# show boot
Current Boot Variables:
kickstart variable = bootflash:/m9100-s2ek9-kickstart-mz.4.2.3.bin
system variable = bootflash:/m9100-s2ek9-mz.4.2.3.bin. . .
13. Verify if all modules have been upgraded correctly.
CETMDS83# show module
Mod Ports Module-Type Model Status
--- ----- -------------------------------- ----------------
-----------
1 16 2x1GE IPS, 14x1/2Gbps FC Module DS-X9302-14K9 ok
3 24 1/2/4 Gbps FC Module DS-X9124 ok
5 0 Supervisor/Fabric-2 DS-X9530-SF2-K9 ha-standby
6 0 Supervisor/Fabric-2 DS-X9530-SF2-K9 active *
9 32 Storage Services Module DS-X9032-SSM ok

Mod Sw Hw World-Wide-Name(s) (WWN)

--- ------- ------ --------------------------------------------------
1 4.2(3) 3.0 20:01:00:0d:ec:20:81:40 to 20:0e:00:0d:ec:20:81:40
3 4.2(3) 1.4 20:81:00:0d:ec:20:81:40 to 20:98:00:0d:ec:20:81:40
5 4.2(3) 1.6 --
6 4.2(3) 1.7 --
9 4.2(3) 5.1 22:01:00:0d:ec:20:81:40 to 22:20:00:0d:ec:20:81:40

Mod Application Image Description Application Image Version

-------- ----------------------------- -------------------------

retain original - ACT+3

 Page 17 of 41

9 SSI linecard image 4.2(3)

. . .
14. If the SSM module not running correctly, identify the module and reload it by issuing command
below.
CETMDS81# reload module 9
15. Delete the kickstart and system image files that are not reflected in the output of the current boot
variables from bootflash: file system.
KLLMDS01# dir bootflash:
. . .
14581760 Oct 13 23:02:02 2009 m9100-s2ek9-kickstart-
mz.3.3.2.bin
18678784 Jun 27 23:11:16 2010 m9100-s2ek9-kickstart-
mz.4.2.3.bin
55596324 Oct 13 23:03:59 2009 m9100-s2ek9-mz.3.3.2.bin
52766038 Jun 27 23:12:11 2010 m9100-s2ek9-mz.4.2.3.bin
. . .

KLLMDS01# delete bootflash:m9100-s2ek9-kickstart-mz.3.3.2.bin

KLLMDS01# delete bootflash:m9100-s2ek9-mz.3.3.2.bin

Cisco MDS NX-OS Software Image Downgrade

Consideration
When downgrading software image, consider the following:

 New features need to be disabled for a non-disruptive downgrade.

 Features in the running-config that are not supported by the image version being installed,
can put switch in an inconsistent state.
 At a minimum, it is needed to disable the default device alias distribution feature using the
no device-alias distribute command in global configuration mode.
 The install all compatibility check shows the impact as disruptive and the reason why the
current running-config is not supported by new image.
NOTE: Use the install all command to gracefully reload the switch and handle configuration conversions.
When downgrading any switch in the Cisco MDS 9000 series, avoid using the reload command.

General Downgrading Guidelines

Use the following guidelines when upgrading the software image:

 Scheduling – Schedule the upgrade when the fabric is stable and steady. Ensure that
everyone who has access to the switch or the network is not configuring the switch or the
network during this upgrade, because all configurations will be disallowed at this time.
 Hardware – Avoid power interruptions to the hardware during any installation procedures;
power interruption problems can corrupt the software image.
 Connectivity – Connectivity to remote servers to retrieve software images requires a
configured IP address for the management Ethernet port on the switch (interface mgmt0)
and make sure the switch has a route to the remote server.
 Software Images – Ensure that the specified system and kickstart images are compatible
with each other.

retain original - ACT+3

 Page 18 of 41

 Save Current Configuration – Save running-config to startup-config (NVRAM).

 Backup Configuration – Keep a copy of the configuration outside of switch (FTP, TFTP,
etc.)
 Save Tech-Support – Keep a copy of the tech-support detail file.
 Redundancy – Perform software image upgrade, “One Switch at a Time.”

Downgrade Software Image via Terminal

The software image can be downgraded through CLI via terminal to all modules in any supported
Cisco MDS 9000 Family switches.

1. Configure the terminal emulator program to match the following default port characteristics:
2. Follow Steps 1 – 8 from Upgrading Software Image via Terminal section.
3. Issue the show incompatibility system <image-filename> command to determine if you need to
disable any features not supported by the older release.
KLLMDS01# show incompatibility system bootflash: m9100-s2ek9-mz.3.3.2.bin

The following configurations on active are incompatibility...

4. If it returns any incompatible messages, disable the impacted features before proceed.
5. Save the configuration using the copy running-config startup-config command.
6. Follow Steps 10 – 13 from Upgrading Software Image via Terminal section.

Inter-Switch Link (ISL)

Inter-Switch Link (ISL) maintains VSAN information as traffic flows between switches. It allows multiple
VSANs to share a common interface.

Trunking
Trunking, also known as VSAN trunking, is a feature specific to switches in the Cisco MDS 9000
Family. Trunking enables interconnect ports to transmit and receive frames in more than one VSAN,
over the same physical link. Trunking is supported on E ports and F ports.

Trunking E Ports
Trunking E ports enable interconnected ports to transmit and receive frames in more than one
VSAN, over the same physical link, using enhanced ISL (EISL) frame format.

Trunking F Ports
Trunking F ports allow interconnected ports to transmit and receive tagged frames in more than one
VSAN, over the same physical link. The most possible trunking scenarios will be between MDS core
switches and NPV switches.

Trunking Misconfiguration
If the VSANs are not configured correctly, issues with the connection may occur. It is important to
ensure that both ends of a trunking link belong to the same port VSAN. On certain switches or fabric
switches where the port VSANs is different, one end returns an error and the other end is not
connected.

WARNING: Ensure that Domain IDs of VSANs on both switches (and/or all switches) that going to
be connected are not the same. Static Domain ID should be assigned to each switch before enabling
trunking. Reconfigure Domain ID as necessary if found overlapped.

retain original - ACT+3

 Page 19 of 41

Refer to Device List spreadsheet to obtain the target Domain ID.

Enable Trunking
To enable trunking, follow these steps:

1. Refer to Cisco MDS Best Practices for standard VSAN number assignment on connection type.
2. Assign ports that are going to be used for ISL to the specified VSAN on both switches. In this
example, we will be assigning two ports to VSAN 3800.
KLLMDS01# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
KLLMDS01(config)# vsan database
KLLMDS01(config-vsan-db)# vsan 3800 interface fc1/23
KLLMDS01(config-vsan-db)# vsan 3800 interface fc1/24
KLLMDS01(config-vsan-db)# exit
KLLMDS01(config)#
3. Configure the ports to be TE_port. Configure this on both switches and change the port
description accordingly. Identify the VSANs that allowed getting through the trunk. In this
example, we will allow VSAN 801 and VSAN 3800 to get through.
KLLMDS01(config)# interface fc1/23
KLLMDS01(config-if)# switchport description ISL_to_KLLMDS03_fc1/23
KLLMDS01(config-if)# switchport mode E
KLLMDS01(config-if)# switchport trunk allowed vsan 801
KLLMDS01(config-if)# switchport trunk allowed vsan add 3800
KLLMDS01(config-if)# switchport trunk mode on
KLLMDS01(config-if)# no shutdown
KLLMDS01(config-if)# exit
KLLMDS01(config)# interface fc1/24
KLLMDS01(config-if)# switchport description ISL_to_KLLMDS03_fc1/24
KLLMDS01(config-if)# switchport mode E
KLLMDS01(config-if)# switchport trunk allowed vsan 801
KLLMDS01(config-if)# switchport trunk allowed vsan add 3800
KLLMDS01(config-if)# switchport trunk mode on
KLLMDS01(config-if)# no shutdown
KLLMDS01(config-if)# end
KLLMDS01#

NOTE: To allow additional VSANs in the trunk, use command below:

(config-if)# switchport trunk allowed vsan add <VSAN>

PortChannel
PortChannel refers to the aggregation of multiple physical fibre channel interfaces into one logical interface
to provide higher aggregated bandwidth, load balancing, and link redundancy. PortChannel can connect to
interfaces across switching modules, so a failure of a switching module cannot bring down the PortChannel
link.

Recommendation: Establish a PortChannel that aggregates two links, with each link connected to a
different switching module. If the switching module goes down, traffic is not affected.

An E PortChannel has the following features and restrictions:

 Provides a point-to-point connection over ISL (E ports) or EISL (TE ports). Multiple links
can be combined into a PortChannel.
 Increases the aggregate bandwidth on an ISL by distributing traffic among all functional

retain original - ACT+3

 Page 20 of 41

links in the channel.

 Load balances across multiple links and maintains optimum bandwidth utilization. Load
balancing is based on the source ID, destination ID, and exchanged ID (OX ID).
 Provides high availability on an ISL. If one link fails, traffic previously carried on this link is
switched to the remaining links. If a link goes down in a PortChannel, the upper protocol is
not aware of it. To the upper protocol, the link is still there, although the bandwidth is
diminished. The routing tables are not affected by link failure. PortChannels may contain up
to 16 physical links and may span multiple modules for added high availability.

Configuration Guidelines and Restrictions

Cisco MDS 9000 Family switches support the following number of PortChannels per switch:

 Switches with Generation 1 switching modules, or a combination of Generation 1 and

Generation 2 switching modules, support a maximum of 128 PortChannels. Only
Generation 2 ports can be included in the PortChannels.
 Switches with only Generation 2 switching modules or Generation 2 and Generation 3
modules support a maximum of 256 PortChannels with 16 interfaces per PortChannel.
 A PortChannel number refers to the unique identifier for each channel group. This number
ranges from 1 to 256.

Create a PortChannel
To create a PortChannel, follow these steps:

1. Enter configuration mode and configure the specified PortChannel. The configuration process
must be repeated on both switches. Change the switch port description accordingly. Remember
to use the same PortChannel number on both switches. Do not repeat the PortChannel number
within the physical fabric.
KLLMDS01# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
KLLMDS01(config)# interface port-channel 1
KLLMDS01(config-if)# channel mode active
KLLMDS01(config-if)# switchport mode E
KLLMDS01(config-if)# switchport trunk allowed vsan 801
KLLMDS01(config-if)# switchport trunk allowed vsan add 3800
KLLMDS01(config-if)# switchport trunk mode on
KLLMDS01(config-if)# switchport description PortChannel_to_KLLMDS03
KLLMDS01(config-if)# exit
KLLMDS01(config)#
2. Assign PortChannel interface to the specified VSAN on both switches. In this example, we will
be assigning two ports to VSAN 3800. Refer to Cisco MDS Best Practices for standard VSAN
number assignment on connection type.
KLLMDS01(config)# vsan database
KLLMDS01(config-vsan-db)# vsan 3800 interface port-channel 1
KLLMDS01(config-vsan-db)# exit
KLLMDS01(config)#
3. Add members to a PortChannel and enable it. The configuration process must be repeated on
both switches. In this example, we are adding the two existing ISL connections to the
PortChannel.
KLLMDS01(config)# interface fc1/23-24
KLLMDS01(config-if)# channel-group 1 force
KLLMDS01(config-if)# exit

retain original - ACT+3

 Page 21 of 41

KLLMDS01(config)# interface port-channel 1

KLLMDS01(config-if)# no shutdown
KLLMDS01(config-if)# end
KLLMDS01#

Verification
To verify if PortChannel is up and running, issue command below:

KLLMDS01# show port-channel database

port-channel 1
Administrative channel mode is active
Operational channel mode is active
Last membership update succeeded
First operational port is fc1/24
2 ports in total, 2 ports up
Ports: fc1/23 [up]
fc1/24 [up] *

Fibre Channel over IP (FCIP)

Fibre Channel over IP (FCIP or FC/IP, also known as Fibre Channel tunneling or storage tunneling), enable
the transmission of Fibre Channel (FC) information by tunneling data between storage area network (SAN)
facilities over IP networks; this capacity facilitates data sharing over a geographically distributed sites. An
FCIP Entity functions to forward Fibre Channel frames after encapsulating them and viewed from an IP
Network perspective, these entities are peers that communicate using TCP/IP.
Cisco MDS 9000 Family IP storage (IPS) services extend the reach of Fibre Channel SANs by using open-
standard, IP-based technology. The switch can connected separated SAN islands using FCIP.

NOTE: FCIP is supported on the MDS 9222i switch, MSM-18+4 module, MDS 9216i switch, MPS-14/2
module, and IPS modules on MDS 9200 switches and 9500 Directors.

FCIP High-Availability Solutions

Fibre Channel PortChannels

PortChannel-based load balancing configuration provides the solution to address link failures. In the
diagram below, you need two IP addresses on each SAN island.

retain original - ACT+3

 Page 22 of 41

The following characteristics set Fibre Channel PortChannel solutions apart from other solutions:

 The entire bundle is one logical (E)ISL link.

 All FCIP links in the PortChannel should be across the same two switches.
 The Fibre Channel traffic is load balanced across the FCIP links in the PortChannel.

FSPF
FPSF-based load balancing configuration also provides high availability. This configuration requires
two IP addresses on each SAN island, and addresses IP and FCIP link failures.

The following characteristics set FSPF solutions apart from other solutions:

 Each FCIP link is a separate (E) ISL.

 The FCIP links can connect to different switches across two SAN islands.
 The Fibre Channel traffic is load balanced across the FCIP link.

Prerequisites
Before setting up an FCIP link, obtain all the IP addresses for both local and remote switches from
Network. Interface “B” and “C” should be configured as part of prerequisites.

retain original - ACT+3

 Page 23 of 41

Basic FCIP Configuration Outline

To create a basic FCIP configuration, follow these steps on both switches:

1. Configure the Gigabit Ethernet interface:

 Specify IP address and mask
 Other parameters include Maximum Transmission Unit (MTU) size; if the entire data path
supports jumbo frames, specify an MTU=3000
2. Configure static IP routes to the remote peer’s IP subnet.
3. Enable the FCIP feature:
 Required before configuring the FCIP components
 Requires the SAN Extension License
4. Create an FCIP profile, including these configurable properties:
 Assign the IP address configured on the Gigabit Ethernet interface. This binds the profile to
the Gigabit Ethernet interface
 TCP port
 Windowing parameters
 Timeouts
 Retransmissions
5. Create an FCIP interface (tunnel):
 Configure the FCIP and E_port parameters in the context of the FCIP interface
 Assign the FCIP profile to the interface. This indirectly binds the tunnel to the Gigabit
Ethernet interface
 Specify the peer IP address
 Enable the FCIP interface.
NOTE: Each profile can have up to three FCIP interfaces, effectively creating three tunnels off one
Gigabit Ethernet interface.

Other configurable parameters include:

 The number of TCP connections for the FCIP link.

 E_port parameters, including trunk mode and trunk allowed VSAN list.

Configure FCIP
To create a FCIP tunnel, follow the steps below. In this example, we are using following information:

Local “A-B” subnet 169.254.209.0/30 Remote “D-C” subnet 169.254.209.64/30

IP address “A” IP address “D”
169.254.209.1/30 169.254.209.66/30
GE 1/1 GE 2/1
IP address “B” 169.254.209.2/30 IP address “C” 169.254.209.65/30

1. Configure Gigabit Ethernet interface that going to be used as FCIP on both switches. In this
example, interface “A”.
CETMDS81# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
CETMDS81(config)# interface GigabitEthernet1/1

retain original - ACT+3

 Page 24 of 41

CETMDS81(config-if)# ip address 169.254.209.1 255.255.255.252

CETMDS81(config-if)# switchport description
FCIP_Link_to_CETMDS82_gi2/1
CETMDS81(config-if)# switchport mtu 3000
CETMDS81(config-if)# no shutdown
CETMDS81(config-if)# exit
CETMDS81(config)#
2. The IPS module does not support dynamic routing protocols and the default-gateway configured
in the initial setup applies to the management interface (mgmt0). Without a route entry for the
peer connection, the FCIP tunnel does not come up. To configure a static route for remote
subnet, in this example, 169.254.209.64/30 connecting through router 169.254.209.2/30 for
Gigabit Ethernet interface 1/1, invoke the command below on both switches.
CETMDS81(config)# ip route 169.254.209.64 255.255.255.252
169.254.209.2 interface GigabitEthernet1/1
3. PING the remote interface to ensure end-to-end connectivity is up. In this example, interface
“D”.
CETMDS81(config)# do ping 169.254.209.66
PING 169.254.209.66 (169.254.209.66) 56(84) bytes of data.
64 bytes from 169.254.209.66: icmp_seq=1 ttl=253 time=6.18 ms
. . .
4. Enable the FCIP feature before performing any FCIP commands. Invoke command below on
both switches.
CETMDS81(config)# feature fcip
5. Configure the FCIP profile on both switches. Remember to use the same FCIP profile ID on
both switches. Associate the profile with the local interface. In this example, interface “A”.
CETMDS81(config)# fcip profile 1
CETMDS81(config-profile)# ip address 169.254.209.1
CETMDS81(config-profile)# exit
CETMDS81(config)#
6. Create the FCIP link between two FCIP link endpoints. Perform the following command
sequence on both switches. Remember to use the same FCIP interface number on both
switches. DO NOT repeat the FCIP interface number within the physical fabric.
7. Use this command to create FCIP interface.
CETMDS81(config)# interface fcip 1
CETMDS81(config-if)#

Use this command to assign the profile to the FCIP interface.

CETMDS81(config-if)# use-profile 1

Use this command to assign the peer IP address information to the FCIP interface. In this
example, interface “D”.

CETMDS81(config-if)# peer-info ipaddr 169.254.209.66

Use this command to enable the FCIP interface.

CETMDS81(config-if)# no shutdown
CETMDS81(config-if)# end
CETMDS81#

retain original - ACT+3

 Page 25 of 41

Verify the FCIP Configuration

There are multiple commands and tools to display the FCIP configuration and verify that the tunnel is
operational.

To display all configured FCIP profiles, invoke the command below. The output shows the Profile
IDs, their associated IP address, and TCP port.

CETMDS81# show fcip profile

-------------------------------------------------------------------
ProfileId Ipaddr TcpPort
-------------------------------------------------------------------
1 169.254.209.1 3225

To drill down and display additional TCP parameters for each FCIP profile, invoke the command
below. The output shows the current values for the TCP parameters of that profile.

CETMDS81# show fcip profile 1

FCIP Profile 1
Internet Address is 169.254.209.1 (interface GigabitEthernet1/1)
Tunnels Using this Profile: fcip1
Listen Port is 3225
TCP parameters
SACK is enabled
PMTU discovery is enabled, reset timeout is 3600 sec
Keep alive is 60 sec
Minimum retransmission timeout is 200 ms
Maximum number of re-transmissions is 4
Send buffer size is 0 KB
Maximum allowed bandwidth is 1000000 kbps
Minimum available bandwidth is 500000 kbps
Configured round trip time is 1000 usec
Congestion window monitoring is enabled, burst size is 50 KB
Auto jitter detection is enabled

Assuming the peer switch has completed and enabled its FCIP tunnel, you can verify that the FCIP
interface is operational using the command below.

CETMDS81# show interface brief

To display active link properties, invoke command below. The output displays typical FC ISL
information plus FCIP specific parameters related to the TCP/IP connection.

CETMDS81# show interface fcip 1

fcip1 is trunking
Hardware is GigabitEthernet
Port WWN is 20:14:00:0d:ec:21:98:00
Peer port WWN is 20:10:00:0d:ec:14:13:00
Admin port mode is auto, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 1
Speed is 1 Gbps
Belongs to port-channel 12
Trunk vsans (admin allowed and active) (101,801,3800)
Trunk vsans (up) (101,801,3800)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()

retain original - ACT+3

 Page 26 of 41

Interface last changed at Wed Nov 28 19:32:04 2007

Using Profile id 1 (interface GigabitEthernet1/1)

Peer Information
Peer Internet address is 169.254.209.66 and port is 3225
. . .

Configure FCIP High-Availability

High availability is important in any storage environment, and is even more important in wide area
network environments, where outages are more frequent than in the data center.

PortChannel Configuration with FCIP Tunnels

To configure a PortChannel with FCIP links as members, create two or more FCIP tunnels, and then
add the selected tunnels to a Channel Group.

1. Perform Steps 1 – 2 in Create a PortChannel section. DO NOT repeat the PortChannel number
within the physical fabric.
2. Add FCIP interfaces to a PortChannel. Invoke the following command sequence.
CETMDS81# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
CETMDS81(config)# interface fcip 1-2
CETMDS81(config-if)# channel-group 2 force
CETMDS81(config-if)# end
CETMDS81#

Tune FCIP Performance

Tune TCP Parameters

Configure the FCIP profile TCP parameters to optimize the FCIP tunnel:

1. Measure the round trip time from source to target by issuing commands below.
CETMDS81# show fcip summary
----------------------------------------------------------------------
Tun prof Eth-if peer-ip Status T W T Enc Comp Bandwidth rtt
E A A max/min (us)
----------------------------------------------------------------------
1 1 GE1/1 169.254.209.66 TRNK Y N N N M1 1000M/500M 1000

CETMDS81# ips measure-rtt 169.254.209.66 interface GigabitEthernet1/1

Round trip time is 5820 micro seconds (5.82 milli seconds)
CETMDS81#
2. From the information obtained above, we will configure TCP parameters below on both
switches.
Max Bandwidth: 700 Mb
Min Bandwidth: 350 Mb
Estimated RTT: 5820 us

CETMDS81# configure terminal

Enter configuration commands, one per line. End with CNTL/Z.
CETMDS81(config)# fcip profile 1
CETMDS81(config-profile)# tcp max-bandwidth-mbps 700 min-available-
bandwidth-mbps 350 round-trip-time-us 5820

retain original - ACT+3

 Page 27 of 41

CETMDS81(config-profile)# tcp max-jitter 1

CETMDS81(config-profile)# exit
CETMDS81(config)#
3. On both switches, simultaneously disable, and then enable the fcip 1 interface.
CETMDS81# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
CETMDS81(config)# interface fcip 1
CETMDS81(config-if)# shutdown
CETMDS81(config-if)# no shutdown

Inter-VSAN Routing (IVR)

VSANs are like virtual switches. VSANs improve SAN scalability, availability, and security by allowing
multiple SANs to share a common physical infrastructure of switches and inter-switch links (ISLs). These
benefits are derived from the separation of FC services in each VSAN and isolation of traffic between
VSANs. Data traffic isolation between the VSANs also inherently prevents sharing of resources attached to
a VSAN, for example robotic tape libraries. Using IVR, resources across VSANs are accessed without
compromising other VSAN benefits.
It is important to understand that SANs have always had the potential to, and in many cases do, hold
targets captive. Arrays can mitigate this problem, but not always. The limited number of ports on a tape
array and the inconsistent device usage drives the desire to share this device across SANs. VSANs just
make this problem more visible.
When implementing IVR, data traffic is transported between specific initiators and targets from different
VSANs without merging them into a single logical fabric. FC control traffic does not flow between VSANs,
and initiators can access only designated resources across VSANs. With IVR, valuable resources like tape
libraries can be shared across VSANs and in conjunction with Fibre Channel over IP (FCIP) provides more
efficient business continuity or disaster recovery solutions.
The IVR process is stateful. If the IVR process fails, it can be restarted without losing state information.
Virtually all FC control traffic is blocked, thus maintaining one of the key benefits of VSANs. A small subset
of control frames are still needed to allow communication between devices involved in IVR.

retain original - ACT+3

 Page 28 of 41

Autonomous Fabric ID
The autonomous fabric ID (AFID) distinguishes segmented VSANs (that is, two VSANs that are
logically and physically separate but have the same VSAN number). Cisco MDS NX-OS supports
AFIDs from 1 through 64.

IVR Configuration Task List

To configure IVR, follow these steps:

1. DO NOT use IVR Network Address Translation (NAT).

2. Verify that unique domain IDs are configured in all switches and VSANs participating in IVR.
3. Enable IVR in the border switches.
4. Configure fabric distribution.
5. Configure the IVR topology manually.
6. Create and activate ICR zone sets in all of the IVR-enabled border switches using fabric
distribution.
7. Verify the IVR configuration.

Enable IVR
The IVR feature must be enabled in all border switches in the fabric that participate in the IVR.

To enable IVR on any participating switch, follow these steps:

CETMDS81# configure terminal

Enter configuration commands, one per line. End with CNTL/Z.
CETMDS81(config)# feature ivr

Distribute the IVR Configuration using CFS

The IVR feature uses the Cisco Fabric Services (CFS) infrastructure to enable efficient configuration
management and to provide a single point of configuration for the entire fabric in the VSAN.

The following configurations are distributed:

 IVR zones
 IVR zone sets
 IVR VSAN topology
 IVR active topology and zone set (activating these features in one switch propagates the
configuration to all other distribution-enabled switches in the fabric)
 AFID database

Enable Configuration Distribution

To enable IVR configuration distribution, follow this step:

CETMDS81(config)# ivr distribute

IVR NAT
Without Network Address Translation (NAT), IVR requires unique domain IDs for all switches in the
fabric; this include unique domain IDs for all VSANs within the same switch.

IVR NAT allows setting up IVR in a fabric without needing unique domain IDs on every switch in the
IVR path. IVR utilizes the switches in other VSANs by using local VSAN for the destination IDs in the

retain original - ACT+3

 Page 29 of 41

Fibre Channel headers.

Enable IVR NAT

To configure IVR NAT, follow this step:

CETMDS81(config)# ivr nat

Transit VSAN Guidelines

Consider the following guidelines for transit VSANs:

 Besides defining the IVR zone membership, specify a set of transit VSANs to provide
connectivity between two edge VSANs:
 If two edge VSANs in an IVR zone overlap, then a transit VSAN is not required to provide
connectivity.
 If two edge VSANs in an IVR zone do not overlap, one or more transit VSANs to provide
connectivity. Two edge VSANs in an IVR zone will not overlap if IVR is not enabled on a
switch that is a member of both the source and destination edge VSANs.
 Traffic between the edge VSANs only traverses through the shortest IVR path.
 Transit VSAN information is common to all IVR zone sets. Sometimes, a transit VSAN can
also act as an edge VSAN in another IVR zone.

Configure IVR Topology

To configure IVR topology, follow these steps:

1. Use command below to obtain the switch WWNs of the IVR-enabled switches. Invoke the
command on both switches.
CETMDS81# show wwn switch
Switch WWN is 20:00:00:0d:ec:20:07:00

CETMDS82# show wwn switch

Switch WWN is 20:00:00:0d:ec:1f:86:00
2. Once obtained the switch WWNs, configure the IVR topology with commands below on both
switches.
CETMDS81# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
CETMDS81(config)# ivr vsan-topology database
CETMDS81(config-ivr-topology-db)# autonomous-fabric-id 1 switch-wwn
20:00:00:0d:ec:20:07:00 vsan-ranges 101, 3800
CETMDS81(config-ivr-topology-db)# autonomous-fabric-id 1 switch-wwn
20:00:00:0d:ec:1f:86:00 vsan-ranges 801, 3800

retain original - ACT+3

 Page 30 of 41

CETMDS81(config-ivr-topology-db)# exit
CETMDS81(config)#
3. Activate VSAN topology. Invoke the command on both switches.
CETMDS81(config)# ivr vsan-topology activate

Lock the Fabric

The first action that modifies the database creates the pending database and locks the feature in the
VSAN. Once the fabric locked, the following situations apply:

 No other user can make any configuration changes to this feature.

 A copy of the configuration database becomes the pending database along with the first
active change.

Commit the Changes

If the changes made to the active database committed, the configuration is committed to all the
switches in the fabric. On a successful commit, the configuration change is applied throughout the
fabric and the lock is released.

To commit IVR configuration changes, issue the command below in the configuration mode.

CETMDS81(config)# ivr commit

Discard the Changes

If you discard (abort) the changes made to the pending database, the configuration database
remains unaffected and the lock is released.

To discard IVR configuration changes, issue the command below in the configuration mode.

CETMDS81(config)# ivr abort

Clear a Locked Session

If someone performed an IVR task and has forgotten to release the lock by either committing or
discarding the changes, an administrator can release the lock from any switch in the fabric. If the
administrator performs this task, the changes to the pending database are discarded and the fabric
lock is released.

To use administrative privileges and release a locked session, issue command below.

CETMDS81# clear ivr session

Verify the IVR Topology

To verify the VSAN topology, issue command below.

CETMDS81# show ivr vsan-topology

Connecting QFX3500 acting FCoE gateway switch to Cisco's native FC

switch (MDS or NEXUS)
This section describes the configuration setting required on a Cisco switch, acting as Native Fibre-channel switch
(FC switch), for Node Port Virtualization (NPIV).
When the QFX3500 switch acts as the FCoE gateway and connects to Cisco's native FC switch
(Nexus or MDS) on the FC port, the NPIV feature on the Cisco switch has to be enabled.

retain original - ACT+3

 Page 31 of 41

If the Cisco switch is not configured for NPIV mode, the FCF function on the QFX3500 may not come
up and complete the Fabric login process.

Verify whether the FC port is up and has completed the fabric login:

From the Juniper QFX3500 Gateway Console

> show fibre-channel proxy np-port detail

Fabric: ftfc, Fabric-id: 10

NP-Port: fc-0/0/0.0

State: online

Sessions: 15

LB state: ON

LB weight: 8

Ref-count: 16

Flags: UP LB C3

NP-Port: fc-0/0/1.0

State: online

Sessions: 0

LB state: OFF

LB weight: 0

Ref-count: 1

Flags: UP C3 NPIV-UNSUPPORTED

 Online – Link is up & FC port logged in still may be problems – see flags
 Down – Physical link is down
 Isolated – something occurred during the port’s login process preventing the link from
establishing correctly.
Examples:

 FC ports in same fabric are attached to two autonomous SANs.

 FC State machine not completed properly for FC port.
When you see the NPIV-UNSUPPORTED flag, it means that fabric login is not completed by the FC
port.

Fixing the NPIV-UNSUPPORTED issue if the Cisco native FC switch is used

Configure the NPIV feature on the Cisco switch, for it to be compatible and work with FCoE NPIV
mode on QFX FCoE gateway:

To check the Cisco MDS configuration from the Command Line:

<switch name># conf t

retain original - ACT+3

 Page 32 of 41

Enter configuration commands, one per line. End with CNTL/Z.

<switch name>(config)# feature ?

cimserver Enable/Disable cimserver

fabric-binding Enable/Disable Fabric Binding

fc-port-security Enable/Disable port-security

fcoe Enable/Disable FCoE/FC feature

fcsp Enable/Disable FC-SP

fex Enable/Disable FEX

interface-vlan Enable/Disable interface vlan

lacp Enable/Disable LACP

npiv Enable/Disable Nx port Id Virtualization (NPIV)

npv Enable/Disable FC N_port Virtualizer

port-track Enable/Disable port track feature

private-vlan Enable/Disable private-vlan

ssh Enable/Disable ssh

tacacs+ Enable/Disable tacacs+

telnet Enable/Disable telnet

udld Enable/Disable UDLD

vpc Enable/Disable VPC (Virtual Port Channel)

To enable NPIV on Cisco switch

<switch name>(config)# feature npiv

retain original - ACT+3

 Page 33 of 41

Cisco MDS Golden Settings

To validate settings login into each switch (via SSH [using PuTTY or similar]) using credentials with network-admin role (twID) and execute verification
command at the command prompt then comparing output to sample output, until automated method is developed.

To resolve variations, visit OASIS document listed under Setting Description.

Golden Setting Description Verification Sample output

Command
Red text is the key value
VSAN 1 inactive show vsan 1 name:VSAN0001 state:suspended
| grep state operational state:down
state:suspended, operational state:down is normal
mode of operation, but state:active, operational or

state:down is needed for functionality with interconnected name:VSAN0001 state:active

switches to pass VSAN/Zoning information. operational state:down

Documented in Cisco MDS 9000 Series Initialization & SAN-OS

Upgrade Guide
IT Controls Manager (ITCM) Controls Listing ID CL-387077
VSANs 2-3839, except 3800 and Symmetrix (in transition) show vsan | vsan 1 information
should not exist on Fabric A/B. grep vsan 801 information
information
One fabric should contain odd numbers (usually A) and the other and
even numbers (usually B). vsan 1 information

Documented in Cisco MDS Best Practices vsan 802 information

retain original - ACT+3

 Page 34 of 41

Backup show Schedule Name : daily_backup

scheduler ----------------------------------
Region Backup Time (UTC)
schedule
AP: Asia Pacific 16:00 User Name : admin

EA: Europe & Africa 0:00 Schedule Type : Run every day at 6 Hrs 0 Mins

ME: Middle East 21:00 Last Execution Time : Thu Jan 1 6:00:00 2007

Last Completion Time: Thu Jan 1 6:00:00 2007

NA: North America 6:00
Execution count : 1
SA: South America 4:00
-----------------------------------------------
Documented in Cisco MDS 9000 Series Initialization & SAN-OS
Upgrade Guide Job Name Last Execution Status

IT Controls Manager (ITCM) Controls Listing ID CL-387079 -----------------------------------------------

daily_run_backup Success (0)

Login Banner show banner ExxonMobil owns and maintains this computer system. Your use of this
motd and other ExxonMobil computer systems signifies your agreement that you
Line returns prior to 72nd column so that formatting is correct.
will use the systems for business purposes only in accordance with
Documented in Cisco MDS 9000 Series Initialization & SAN-OS
ExxonMobil policies and standards. Except as otherwise provided by
Upgrade Guide
applicable law, your use of those systems is not covered by any right of

privacy. ExxonMobil from time to time monitors the use of its systems

to detect security problems, improper uses of the systems and for other

purposes, all as permitted by law. Your use of ExxonMobil systems,

including your e-mail messages, may become public outside of ExxonMobil

for unanticipated reasons such as litigation involving ExxonMobil.

Validate NTP show ntp Distribution : Enabled

status |
Network Time Protocol
grep
Documented in Cisco MDS 9000 Series Initialization & SAN-OS Distribution
Upgrade Guide

retain original - ACT+3

 Page 35 of 41

Validate TACACS+ for Authentication show aaa radius

groups tacacs+
Documented in Cisco MDS 9000 Series Initialization & SAN-OS
Upgrade Guide
IT Controls Manager (ITCM) Controls Listing ID CL-387074, CL-
387163, and CL-387242

SSH enabled show ssh ssh version 2 is enabled

server
Documented in Cisco MDS 9000 Series Initialization & SAN-OS
Upgrade Guide
IT Controls Manager (ITCM) Controls Listing ID CL-387177
Telnet disabled show telnet telnet service not enabled
server
Documented in Cisco MDS 9000 Series Initialization & SAN-OS
Upgrade Guide
IT Controls Manager (ITCM) Controls Listing ID CL-387178 and
CL-387179

Default switch port settings show system System default port state is down
default System default trunk mode is on
IT Controls Manager (ITCM) Controls Listing ID CL-386995 and
switchport
CL-387076 System default link failure syslog logging level is 5

System default tx credit queue type is 0

Default zone settings show system system default zone default-zone deny
default zone system default zone distribute full
Documented in Cisco MDS NX-OS Installation/Implementation
Guide and Cisco MDS NX-OS Operations Guide system default zone mode enhanced

system default zone gs read-write

IT Controls Manager (ITCM) Controls Listing ID CL-387162
system default zone smart-zone disabled

Ports that are down set to VSAN 1 show --------------------------------------------------------------------------

interface Interface Vsan Admin Admin Status SFP Oper Oper Port
Documented in Cisco MDS 9000 Series Initialization & SAN-OS
brief | grep
Upgrade Guide Mode Trunk Mode Speed Chann
down
Mode (Gbps)
IT Controls Manager (ITCM) Controls Listing ID CL-418836
fc1/13 1 auto on down swl -- --

retain original - ACT+3

 Page 36 of 41

Alerts turned on show Distribution : Enabled

callhome
Documented in Cisco MDS 9000 Series Initialization & SAN-OS
status
Upgrade Guide and Cisco MDS NX-OS
Installation/Implementation Guide
Alerts are emailed to support team show from email addr: [email protected]
callhome reply to email addr: [email protected]
Documented in Cisco MDS 9000 Series Initialization & SAN-OS
transport |
Upgrade Guide and Cisco MDS NX-OS
grep addr
Installation/Implementation Guide
Alerts are passed to the SMTP relay show smtp server:SMTP.NA.XOM.COM
callhome smtp server port:25
Documented in Cisco MDS 9000 Series Initialization & SAN-OS
transport |
Upgrade Guide and Cisco MDS NX-OS smtp server priority:0
grep smtp
Installation/Implementation Guide
(Does not mean the switch is registered with relay)

retain original - ACT+3

 Page 37 of 41

Destination for call home show xml destination profile information

callhome maximum message size:500000
Documented in Cisco MDS 9000 Series Initialization & SAN-OS
destination-
Upgrade Guide and Cisco MDS NX-OS message format:XML
profile
Installation/Implementation Guide message-level:3

transport-method:email

email addresses configured:

[email protected]

url addresses configured:

alert groups configured:

cisco-tac

full_txt destination profile information

maximum message size:500000

message format:full-txt

message-level:3

transport-method:email

email addresses configured:

[email protected]

url addresses configured:

alert groups configured:

all

short_txt destination profile information

maximum message size:4000

message format:short-txt

message-level:4

transport-method:email

retain original - ACT+3

 Page 38 of 41

email addresses configured:

[email protected]

url addresses configured:

alert groups configured:

all

SNMP community string show snmp Community Group / Access context

community _________ ______________ _______
Documented in Cisco MDS NX-OS Installation/Implementation
Guide SCOMRO network-operator

public network-operator

private network-admin

retain original - ACT+3

 Page 39 of 41

Default Roles show snmp Role: network-admin

group | grep Role: network-operator
Documented in Cisco MDS/ Data Center Network Manager
Role
Security Matrix Role: server-admin

Role: priv-15
IT Controls Manager (ITCM) Controls Listing ID CL-387180, CL-
387183, and CL-387242 Role: priv-14

Role: priv-13

Role: priv-12

Role: priv-11

Role: priv-10

Role: priv-9

Role: priv-8

Role: priv-7

Role: priv-6

Role: priv-5

Role: priv-4

Role: priv-3

Role: priv-2

Role: priv-1

Role: priv-0

Role: default-role

retain original - ACT+3

 Page 40 of 41

IP address of relevant FMS, OCI and SCOM servers show snmp -------------------------------------------------------------------
host Host Port Version Level Type SecName
OCI Server FMS Servers SCOM ProdZ
-------------------------------------------------------------------
Symmetrix HOEFMS09 hoeomm07.na.xom.com
10.216.181.70 1164 v2c noauth trap public
DALOCI02 (10.128.2.85) 10.128.5.73 10.215.156.122 2162 v2c noauth trap public
(10.216.181.70) HOEFMS10 hoeomm08.na.xom.com 10.215.156.157 2162 v2c noauth trap public

Houston DC (10.128.3.50) 10.128.5.77 10.250.16.1 1163 v2c noauth trap public

10.250.160.135 1164 v2c noauth trap public

HOEOCI01 HOEFMS11 hoeomm09.na.xom.com
10.128.5.73 162 v2c noauth trap SCOMRO
(10.215.187.35) (10.128.5.179) 10.128.5.74
10.128.5.77 162 v2c noauth trap SCOMRO
Dallas DC HOEFMS12 hoeomm11.na.xom.com
10.128.5.53 162 v2c noauth trap SCOMRO
DALOCI01 (10.128.5.165) 10.128.5.53

(10.216.180.111) HOEFMS13

HOEOCI02 (10.128.17.77)

(10.215.187.121) HOEFMS14

Rest of World (10.128.16.138)

EA/AF/ME/AP HOEFMS15

HOEOCI03 (10.128.5.167)

(10.215.187.22) HOEFMS16
(10.128.5.184)
HOEFMS17
(10.128.5.164)
Documented in NetApp OnCommand Insight (OCI) Operations
Guide

retain original - ACT+3

 Page 41 of 41

SNMP Users show snmp ______________________________________________________________

user SNMP USERS
admin
(switch embedded account, cannot be renamed) ______________________________________________________________

xsconfig
(switch local account ) User Auth Priv(enforce) Groups

____ ____ _____________ ______

xsOnCommandInsight
(switch local account ) admin md5 des(no) network-admin

Documented in NetApp OnCommand Insight (OCI) Operations

Guide xsconfig md5 des(no) network-admin

IT Controls Manager (ITCM) Controls Listing ID CL-387180, CL-

387183, and CL-387242 xsOnCommandInsight md5 no network-operator

______________________________________________________________

NOTIFICATION TARGET USERS (configured for sending V3 Inform)

______________________________________________________________

User Auth Priv

____ ____ ____

FC Domains are distributed. show fcdomain distribution is enabled

fcdomain
Documented in Cisco MDS NX-OS Installation/Implementation
status
Guide
Feature Sets Enabled show feature http-server 1 enabled
| grep ivr 1 enabled
npiv is required for FCoE (Houston Campus and Calgary) and
enabled
NetApp Clustered Data OnTap. npiv 1 enabled

scheduler 1 enabled
Documented in Cisco MDS 9000 Series Initialization & SAN-OS
Upgrade Guide sshServer 1 enabled

tacacs 1 enabled

retain original - ACT+3