R - 04 - MR-1CP-NSSANTS - Cisco FC Troubleshooting

Module 4: Cisco Fibre Channel Fabric
Troubleshooting
Upon completion of this module, you should be able to:

• Describe end to end connectivity troubleshooting
• Analyze and correct ISL issues
• Analyze and correct fabric merge failures
• Show how to find and resolve virtualized SAN issues
• List the steps to find and resolve routing and trunking issues
• Use logs and debugging tools to troubleshoot a SAN
• Analyze and correct VSAN issues
• Resolve security issues in the FC SAN
• Use FC tools to trace errors for root cause determination
Copyright © 2011 EMC Corporation. All Rights Reserved. Cisco Fibre Channel Fabric Troubleshooting 1
Module 4: Cisco Fibre Channel
Fabric Troubleshooting
Lesson 1: Cisco Switch Configuration

Upon completion of this lesson, you should be able to:
• Define a correctly configured switch
• List switch commands used to gather data
• List the commonly used switch log files
• Show how to collect switch configuration data
• Analyze the collected switch data
Cisco Switch Backups
• CLI
• Fabric Manager
• Must have a FTP or TFTP server available to save the
configuration
MDS# copy running-config startup-config

MDS# copy running-config ftp:
Enter hostname for the ftp server: 10.127.118.13
Enter username: root
Password: *******
Viewing Logs with the CLI
MDS# show logging
Logging console: enabled (Severity: critical)
Logging monitor: enabled (Severity: notifications)
Logging linecard: enabled (Severity: notifications)
Logging timestamp: Seconds
Logging server: disabled
Logging logfile: enabled
Name - messages: Severity - notifications Size - 4194304
Facility Default Severity Current Session Severity
-------- ---------------- ------------------------
aaa 2 2
acl 2 2
auth 0 0
authpriv 3 3
bootvar 5 5
Viewing NVRAM Logs
MDS# show logging nvram
2010 Apr 15 19:33:54 MDS %PLATFORM-2-PS_OK: Power supply 1 ok (Serial
number QCS0918103Y)
2010 Apr 15 19:33:54 MDS %PLATFORM-2-PS_FANOK: Fan in Power supply 1
ok
2010 Apr 15 19:33:54 MDS %PLATFORM-2-PS_OK: Power supply 2 ok (Serial
number QCS0916107B)
2010 Apr 15 19:33:54 MDS %PLATFORM-2-PS_FANOK: Fan in Power supply 2
ok
2010 Apr 15 19:33:54 MDS %PLATFORM-2-FAN_OK: Fan module ok
2010 Apr 15 19:33:54 MDS %PLATFORM-2-CHASSIS_CLKMODOK: Chassis clock
module A ok
2010 Apr 15 19:33:54 MDS %PLATFORM-2-CHASSIS_CLKMODOK: Chassis clock
module B ok
2010 Apr 15 19:33:54 MDS %PLATFORM-2-CHASSIS_CLKSRC: Current chassis
clock source is clock-A
2010 Apr 15 19:34:50 MDS %IMAGE_DNLD-SLOT2-2-IMG_DNLD_STARTED:
Module image download process. Please wait until completion...
show cores
MDS# show cores
Module-num Process-name PID Core-create-time
---------- ------------ --- ----------------
5 fspf 1524 Apr 15 02:10
6 fcc 919 Apr 15 02:09
8 acltcam 285 Apr 15 02:09
8 fib 283 Apr 15 02:08
Lab Exercise 11: Cisco Data Collection
Lesson 2: Cisco Data Collection

• List the key logs to capture for root cause analysis
• Show how to decipher common log files
• Use commands to determine root cause and capture data for
analysis
• Show how to use Device and Fabric Manager to read and
collect logs
Cisco Syslog
• In Fabric Manager, choose Switches > Events > Syslog in the Physical pane,
and then click the Servers tab in the Information pane. In Device Manager,
choose Logs > Syslog > Setup and click the Servers tab in the Syslog dialog
box.
• Click the Create Row icon in Fabric Manager or Create in Device Manager to
add a new Syslog server.
• Enter the name or IP address in dotted decimal notation of the Syslog server
in the Name or IP Address field.
• Set the message severity threshold by clicking the MsgSeverity radio button,
and set the facility by clicking the Facility radio button.
• Click Apply Changes in Fabric Manager or click Create in Device Manager to
save and apply your changes.
• If CFS is enabled in Fabric Manager for the Syslog feature, click CFS to commit
these changes to propagate the configuration through the fabric.
CLI Debug
MDS# debug flogi event interface fc1/13
2010 Apr 16 15:51:48.701662 flogi: fu_priority_select: - setting fd[5] for select
call - setting fd[6] for select call - setting fd[7] for select call - setting
fd[8] for select call
2010 Apr 16 15:51:48.701750 flogi: fu_priority_select_select_queue: round credit(6)
2010 Apr 16 15:51:48.701784 flogi: curr_q - FU_PSEL_Q_CAT_FD, usr_q_info(1),
fd(7), priority(2), credit(0), empty
2010 Apr 16 15:51:48.701817 flogi: fu_priority_select: returning FU_PSEL_Q_CAT_MTS
queue, fd(6), usr_q_info(3)
2010 Apr 16 15:51:48.701882 flogi: fu_fsm_engine: line[2253]
2010 Apr 16 15:51:48.702515 flogi: fu_fsm_execute_all: match_msg_id(0),
log_already_open(0)
2010 Apr 16 15:51:48.702558 flogi: fu_fsm_execute_all: null fsm_event_list MDS# 2010
Apr 16 15:51:48.702631 flogi: fu_fsm_engine_post_event_processing: mts msg
MTS_OPC_DEBUG_WRAP_MSG(msg_id 46472) dropped
Running Processes
show process
MDS# show processes
PID State PC Start_cnt TTY Process
----- ----- -------- ----------- ---- -------------
1 S 2aba5fb2 1 - init
2 S 0 1 - keventd
3 S 0 1 - ksoftirqd_CPU0
4 S 0 1 - kswapd
5 S 0 1 - bdflush
6 S 0 1 - kupdated
211 S 0 1 - kjournald
708 S 2abc1566 1 - portmap
717 S 2ad7afb2 1 - httpd
show processes log
MDS# show processes log
Process PID Normal-exit Stack Core Log-create-time
---------------- ------ ----------- ----- ----- ---------------
fcs 2850 N Y N Mon Jun 19 19:44:55 2006
ilc_helper 1679 N Y N Thu Sep 13 17:49:38 2007
ilc_helper 2035 N Y N Wed Oct 17 18:32:40 2007
ilc_helper 3477 N Y N Thu Sep 13 18:01:04 2007
installer 1858 N N N Tue Feb 16 09:36:05 2010
installer 20769 N N N Wed Feb 17 12:13:41 2010
installer 26811 N N N Tue Feb 16 09:12:04 2010
show process log details
MDS# show processes log pid 2850
======================================================
Service: fcs
Description: Fabric Config Server
Started at Wed Jun 7 18:34:28 2006 (694067 us)

Stopped at Mon Jun 19 19:44:55 2006 (279788 us)
Uptime: 12 days 1 hours 10 minutes 27 seconds
Start type: SRV_OPTION_RESTART_STATELESS (23)

Death reason: SYSMGR_DEATH_REASON_FAILURE_SIGNAL (2)
System image version: 3.0(0.350) S15
PID: 2850
Exit code: signal 11 (core dumped)
Using On-Board Failure Logging
• The data stored by the OBFL facility includes the following:
 Time of initial power-on
 Slot number of the card in the chassis
 Initial temperature of the card
 Firmware, BIOS, FPGA, and ASIC versions
 Serial number of the card
 Stack trace for crashes
 CPU hog information
 Memory leak information
 Software error messages
 Hardware exception logs
 Environmental history
 OBFL specific history information
 ASIC interrupt and error statistics history
 ASIC register dumps
Configuring OBFL for the Switch
MDS# conf
Enter configuration commands, one per line. End with CNTL/Z.
MDS(config)# hw-module logging onboard
MDS(config)# hw-module logging onboard cpu-hog
MDS(config)# hw-module logging onboard environmental-history
MDS(config)# hw-module logging onboard error-stats
MDS(config)# hw-module logging onboard interrupt-stats
MDS(config)# hw-module logging onboard mem-leak
MDS(config)# hw-module logging onboard miscellaneous-error
MDS(config)# hw-module logging onboard obfl-log
MDS(config)# exit
MDS# sho logging onboard stat
Switch OBFL Log: Enabled
Displaying OBFL Logs
MDS# show logging onboard status
Switch OBFL Log: Enabled
Module: 6 OBFL Log: Enabled
error-stats Enabled
miscellaneous-error Enabled
obfl-log (boot-time/device-version/obfl-history) Enabled
system-health Enabled
stack-trace Enabled
Fabric Manager Tools
Viewing Logs with Fabric Manager
show system uptime
MDS# show system uptime
System start time: Sun Feb 14 00:04:02 2010
System uptime: 67 days, 13 hours, 37 minutes, 30 seconds
Kernel uptime: 6 days, 20 hours, 11 minutes, 5 seconds
Active supervisor uptime: 6 days, 20 hours, 9 minutes, 23 seconds
show system reset-reason
MDS# show system reset-reason module 4
*************** module reset reason (4) *************
Time stamp : At 272075 usecs after Thu Apr 15 17:41:24 2010
Service name :
Reset reason : Disruptive upgrade => Disruptive upgrade
Serial number: JAB0813052L
Error code : NA
Time stamp : At 25693 usecs after Wed Apr 14 11:53:35 2010

Service name : IPS manager
Reset reason : Disruptive upgrade => Disruptive upgrade
Serial number: JAB0813052L
Error code : NA
Time stamp : At 534306 usecs after Fri Mar 12 07:31:39 2010

Service name :
LED Status
• Verify Power and fans
• Verify Supervisor mode
• Verify port modes
Power Supply Checks
MDS# show environment power
-----------------------------------------------------
PS Model Power Power Status
(Watts) (Amp @42V)
-----------------------------------------------------
1 DS-CAC-845W 800.10 19.05 ok
2 DS-CAC-845W 800.10 19.05 ok
Mod Model Power Power Power Power Status
Requested Requested Allocated Allocated
(Watts) (Amp @42V) (Watts) (Amp @42V)
--- ------------------- ------- ---------- --------- ---------- ----------
1 DS-X9216-K9-SUP 209.16 4.98 209.16 4.98 powered-up
2 DS-X9032-SSM 280.98 6.69 280.98 6.69 powered-up
Power Usage Summary:
--------------------
Power Supply redundancy mode: redundant
Total Power Capacity 800.10 W
Fan Check
MDS# show environment fan
------------------------------------------------------
Fan Model Hw Status
------------------------------------------------------
Chassis DS-9SLOT-FAN 1.2 ok
PS-1 -- -- ok
PS-2 -- -- ok
Temperature Check
MDS# show environment temperature
---------------------------------------------------------------
Module Sensor MajorThresh MinorThres CurTemp Status
(Celsius) (Celsius) (Celsius)
---------------------------------------------------------------
1 Outlet 75 60 42 ok
1 Intake 65 50 35 ok
Troubleshooting Clock Module Issues
MDS# show environment clock
----------------------------------------------------------
Clock Model Hw Status
----------------------------------------------------------
A DS-C9509-CL 1.0 ok/active
B DS-C9509-CL 1.0 ok/standby
Exception Log
MDS# show module internal exceptionlog
********* Exception info for module 5 ********
exception information --- exception instance 1 ----
Module Slot Number: 5
Device Id : 84
Device Name : PowerSupply
Device Errorcode : 0xc5402208
Device ID : 84 (0x54)
Device Instance : 02 (0x02)
Dev Type (HW/SW) : 02 (0x02)
ErrNum (devInfo) : 08 (0x08)
System Errorcode : 0x40380014 PS_FAIL: Powersupply Failed or shutdown
Error Type : Warning
PhyPortLayer : Unknown
Port(s) Affected : none
Error Description :
DSAP : 0 (0x0)
Supervisor Reset Reasons
MDS# show system reset-reason
--- reset reason for Supervisor-module 6 (from Supervisor in slot 6) ---
1) No time
Reason: Unknown
Service:
Version: 3.2(2c)
2) At 961035 usecs after Wed Apr 14 16:08:55 2010
Reason: Reset triggered due to Switchover Request by User
Service: SAP(93): Swover due to install
Version: 3.3(4)
3) No time
Reason: Unknown
Service:
Version: 3.2(2c)
4) At 62170 usecs after Fri Mar 5 19:08:37 2010
Reason: Reset triggered due to Switchover Request by User
Service: SAP(93): Swover due to install
Version: 3.3(4)
Active Supervisor Reboot – Process Crash
Switch# show logging
2010 Feb 21 19:08:27 MDS %PORT-5-IF_UP: %$VSAN 200%$ Interface fc2/6 is
up in mode F
2010 Feb 21 19:08:27 MDS %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configuring
console from pts/0 10.4.131.255
2010 Feb 21 19:08:44 MDS last message repeated 4 times
2010 Feb 21 19:08:44 MDS %BOOTVAR-3-VERIFY_FAILED: Image
bootflash:/m9000-ek9-ssi-mz.3.2.3i.bin.S10 for bootvar ssi verification
failed, reason=imghdr library error
Active Supervisor Reboot – Diagnostics Fail
MDS# show module internal exceptionlog module 7
********* Exception info for module 7 ********
exception information --- exception instance 1 ----
Module Slot Number: 7
Device Id : 6
Device Name : D-frontier
Device Errorcode : 0x30502322
Device ID : 773 (0x305)
Device Instance : 02 (0x02)
Dev Type (HW/SW) : 03 (0x03)
ErrNum (devInfo) : 34 (0x22)
System Errorcode : 0x40420009 X-bar Interface ASIC Error
Error Type : Minor error
PhyPortLayer : Fibre Channel
Port(s) Affected :
1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,
28,29,30,31,32
Error Description :
Standby Supervisor Not Recognized
MDS# show mod
Mod Ports Module-Type Model Status
--- ----- -------------------------------- ------------------ ------------
1 16 1/2 Gbps FC Module DS-X9016 ok
2 32 1/2 Gbps FC Module DS-X9032 ok
3 32 Storage Services Module DS-X9032-SSM ok
4 8 IP Storage Services Module DS-X9308-SMIP ok
5 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 active *
6 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 NA
Mod Sw Hw World-Wide-Name(s) (WWN)
--- -------------- ------ --------------------------------------------------
1 3.3(4) 5.2 20:01:00:0d:ec:0d:8a:00 to 20:10:00:0d:ec:0d:8a:00
2 3.3(4) 3.2 20:41:00:0d:ec:0d:8a:00 to 20:60:00:0d:ec:0d:8a:00
3 3.3(4) 5.1 20:81:00:0d:ec:0d:8a:00 to 20:a0:00:0d:ec:0d:8a:00
4 3.3(4) 5.0 --
5 3.3(4) 5.2 --
6 3.3(4) 5.2 --
Mod Application Image Description Application Image Version
-------- ----------------------------- -------------------------
3 SSI linecard image 3.2(3v)
Redundancy Status
MDS# show system redundancy status
Redundancy mode
---------------
administrative: HA
operational: None
This supervisor (sup-2)
-----------------------
Redundancy state: Active
Supervisor state: Active
Internal state: Active with HA standby
Other supervisor (sup-1)
------------------------
Redundancy state: Standby
Supervisor state: HA standby
Internal state: HA synchronization in progress
Internal Errors
MDS# show system internal sysmgr gsyncstats
Name Gsync done Gsync time(sec)
---------------- ---------- -------------
aaa 1 0
cert_enroll 1 0
ExceptionLog 1 0
platform 1 1
radius 1 0
securityd 1 0
SystemHealth 1 0
tacacs 0 N/A
acl 1 0
ascii-cfg 1 1
bios_daemon 0 N/A
bootvar 1 0
callhome 1 0
capability 1 0
cdp 1 0
Analyzing Log Files
• show logging
• show module internal exception-log module
• show module internal event-history module
• show hardware internal errors
Internal Services
MDS# show system internal sysmgr service all
Name UUID PID SAP state Start count
---------------- ---------- ------ ----- ----- -----------
aaa 0x000000B5 1657 111 s0009 1
cert_enroll 0x0000012B 1656 169 s0009 1
ExceptionLog 0x00000050 1706 92 s0009 1
platform 0x00000018 1317 39 s0009 1
radius 0x000000B7 1655 113 s0009 1
securityd 0x0000002A 1654 55 s0009 1
SystemHealth 0x00000086 1653 1011 s0009 1
tacacs 0x000000B6 [NA] [NA] s0075 None
acl 0x00000026 1651 22 s0009 1
ascii-cfg 0x00000051 1650 76 s0009 1
bios_daemon 0x00000081 1649 [NA] s0007 1
bootvar 0x00000052 1707 80 s0009 1
callhome 0x0000003F 1713 66 s0009 1
. . .
xbar 0x0000007C 1699 946 s0009 23
Show the Module Status
MDS# show module 4
Mod Ports Module-Type Model Status
--- ----- -------------------------------- ------------------ --------
4 8 IP Storage Services Module DS-X9308-SMIP ok
Mod Sw Hw World-Wide-Name(s) (WWN)
--- -------------- ------ -------------------------------------------
4 3.3(4) 5.1 --
Mod MAC-Address(es) Serial-Num

--- -------------------------------------- ----------
4 00-12-00-3c-44-32 to 00-12-00-3c-44-3e JAB08530101
Review Event History Errors
MDS# show platform internal event-history errors
1) Event:E_DEBUG, length:77, at 646120 usecs after Fri Apr 16 14:32:01 2010
[103] lc-status for module 3 is 0x00f8 card_id 19 reg5 = 0x00f8 reg3 =
0x0077
0x0077
0x0077
[103] pfm_read_fan_sprom: zeroing out fan sprom as it is not valid
[103] pfm_read_fan_sprom: Error fan sprom 0 PSS outdated
[103] send_pfm_snmp_fru_mod_sts_chg:1046: SNMP ERR PhysIndex is 0, Invalid
Image Download
MDS# show logging
MODULE-2-MOD_DNLD_FAIL: Image download failed for module [dec].
Explanation The module failed to download a new image from the supervisor module.
Recommended Action Collect module information using show module internal all
module
IMAGE_DNLD-SLOT#-2-ADDON_IMG_DNLD_FAILED: Module image download
. process failed. [chars].
Explanation The add-on image download to the module failed. This module is not
operational until an add-on image has been successfully installed.
Recommended Action Verify the location and version of your module image.
Enter install module command or similar Fabric Manager/Device Manager
command to download a new module image
.
Runtime Diagnostics
MDS# show logging
MODULE-2-MOD_DIAG_FAIL: Module [dec] reported failure on ports
[dec]/[dec]-[dec]/[dec] ([chars]) due to [chars] in device [dec] (device error
[hex]).
Explanation The module reported a failure in the runtime diagnostic. Module manager
is going to power cycle the module.
Recommended Action Collect information about the module by entering the
show module internal all module command.
State Transition Log
MDS# show module internal event-history module
7) FSM:<ID(2): Slot 8, node 0x0800> Transition at 14258 usecs after Mon Sep 26
17:50:56 2005
Previous state: [LCM_ST_LC_POWERED_UP]
Triggered event: [LCM_EV_PFM_LC_STATUS_POWERED_DOWN]
Next state: [LCM_ST_LC_NOT_PRESENT]
Power On or Switch Reboot Hangs
• The bootflash is corrupted
• The loader is corrupted
• The BIOS is corrupted
• The kickstart image is corrupted
• Boot parameters are incorrect
• The system image is corrupted
Normal Boot Sequence
BIOS • Loads the loader
Loader • Loads the kickstart image
• Loads the kernel, basic drivers,

Kickstart Image
and system image
System Image • Login prompt
Recovery from the Loader Prompt
• Enter the local IP address for the switch at the loader> prompt and press Enter.
loader> net --ip=172.16.1.2
• Specify the subnet mask.
loader> net --nm= 255.255.255.0
• Specify the IP address of the default gateway.
loader> net --gw=172.16.1.1
• Boot the kickstart image file from the required server.
loader> boot tftp://172.16.10.100/m9500-kickstart-3.3.bin
In this example, 172.16.10.100 is the IP address of the TFTP server, and m9500-
kickstart-3.3.bin is the name of the kickstart image file that exists on that server.
• Issue the init system command at the switch(boot)# prompt.
switch(boot)# init system
The switch(boot)# prompt indicates that you have a usable Kickstart image.
Recovering a Supervisor With Corrupted
Bootflash
loader> boot tftp://172.16.10.100/kickstart-latest
Address: 172.16.1.2
Netmask: 255.255.255.0
Server: 172.16.10.100
Gateway: 172.16.1.1
Booting: /kick-282 console=ttyS0,9600n8nn quiet loader_ver= “2.1(2)”....
............................................Image verification OK
Starting kernel...
INIT: version 2.78 booting
Checking all filesystems..... done.
Loading system software
INIT: Sending processes the TERM signal
Sending all processes the TERM signal... done.
Sending all processes the KILL signal... done.
Entering single-user mode...
INIT: Going single user
INIT: Sending processes the TERM signal
switch(boot)#
Recoverable System Restarts
MDS# show log logfile | include error
2010 Feb 26 14:29:29 MDS %SYSMGR-3-SERVICE_TERMINATED: Service
"installer" (PID 28897) has finished with error code
SYSMGR_EXITCODE_FAILURE_NOCALLHOME (20).
2010 Mar 2 10:12:05 MDS %SYSMGR-3-SERVICE_TERMINATED: Service
2010 Feb 23 11:12:24 MDS %DAEMON-3-SYSTEM_MSG: error: PAM:
Authentication failure for admin from 10.4.16.107
2010 Apr 13 17:21:58 MDS %DAEMON-3-SYSTEM_MSG: error: PAM:
Authentication failure for admin from 10.12.243.57
2010 Mar 5 19:02:02 MDS %SYSMGR-3-SERVICE_TERMINATED: Service
Lab Exercise 12: Cisco Data Analysis
Parts 1 through 3
Lesson 3: Cisco Data Analysis

• Describe end to end connectivity troubleshooting
• List common port issues and steps to resolve
• List the key logs to capture for root cause analysis
• Follow the login process with standard tools to resolve issues
• Use logs and debugging tools to troubleshoot end to end
connectivity
Port Negotiation
OLS LR LR LR LR
NOS LRR LRR OLS
Port Error Checklist
Port Checklist
Check the physical media to ensure there are no damaged parts. 
Verify that the SFP (small form-factor pluggable) devices in use are those
authorized by Cisco and that they are not faulty. 
Verify that you have enabled the port by right-clicking the port in Device
Manager and selecting enable or by using the no shut command. 
Right-click the port in Device Manager or use the show interface command to
verify the state of the interface. 
Verify that you if you have one host-optimized port configured as an ISL, you
have not connected to the other three ports in the port group. 
Verify that no ports on a Generation 2 module are out of service. 
Reason Codes – All Port Types
P1-MDS9222i# sho int br
-------------------------------------------------------------------------------
Interface Vsan Admin Admin Status SFP Oper Oper Port
Mode Trunk Mode Speed Channel
Mode (Gbps)
-------------------------------------------------------------------------------
fc1/1 10 E on down swl -- --
fc1/2 10 FX on up swl F 2 --
fc1/3 1 FX on down swl -- --
Port Manager
module-1# show hardware internal fc-mac port 9 port-info
Hw Access OK? : yes
Enable State : PORT_ENABLE_STATE_UP
Operating Mode : PORT_MODE_E
Operating Speed : PORT_SPEED_2G
Transmit Stream : Not Enabled
Sync State : SYNC_ACQ(82)
OSM/FRM Mode : FRM
Receive Credit : 16
Transmit Credit : 7
Link Status : LINK_STATUS_SUCCESS
Link Status Reason : LINK_STATUS_REASON_LINK_INIT_SUCCESS
Traffic Paused? : no
Port TS enable at : 0x0
Port TS disable at : 0x0
Trunk Enabled? : no
shutdown flag set? : no
credit set? : yes
pm set oper mode : PORT_MODE_F
timestamp check : Enabled
int link state : RIV_LINK_UP
Troubleshooting Port States from the CLI
MDS# show interface fc 1/1
fc1/1 is up
Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
Port WWN is 20:01:00:0d:ec:0c:b4:40
Admin port mode is auto, trunk mode is on
snmp link state traps are enabled
Port mode is F, FCID is 0x0a0000
Port vsan is 100
Speed is 2 Gbps
Transmit B2B Credit is 3
Receive B2B Credit is 16
Receive data field Size is 2112
Beacon is turned off
5 minutes input rate 24248 bits/sec, 3031 bytes/sec, 10 frames/sec
5 minutes output rate 7040 bits/sec, 880 bytes/sec, 10 frames/sec
6599329 frames input, 5145463232 bytes
0 discards, 0 errors
0 CRC, 0 unknown class
0 too long, 0 too short
12640689 frames output, 15362142764 bytes
0 input OLS, 0 LRR, 0 NOS, 26 loop inits
5 output OLS, 4 LRR, 0 NOS, 4 loop inits
16 receive B2B credit remaining
3 transmit B2B credit remaining
3 low priority transmit B2B credit remaining
Interface last changed at Fri Apr 16 19:21:51 2010
Port Debug
command Description
show hardware internal fc-mac Performs a series of checks to isolate the
port port link-status problem
show hardware internal fc-mac Provides the current state and configuration
port port port-info of the port
show hardware internal fc-mac Gives all non-zero statistics for the port
port port statistics
show hardware internal fc-mac Displays the current state of the SFP.
port port gbic-info
show hardware internal error Collects interrupt statistics, error statistics,
and exception log information for the entire
module.
show hardware internal debug- Represents an aggregation of a number of
info interface fc-interface debug commands from all ASICs.
Isolating Port Issues Using the CLI
MDS# show flogi database
---------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
---------------------------------------------------------------------------
fc1/9 100 0x0a0007 50:06:04:8a:cc:d2:3a:61 50:06:04:8a:cc:d2:3a:61
fc1/10 100 0x0a0009 50:06:04:8a:cc:d2:3a:6e 50:06:04:8a:cc:d2:3a:6e
fc1/11 100 0x0a000f 10:00:00:00:c9:44:2b:ce 20:00:00:00:c9:44:2b:ce
fc1/12 100 0x0a000e 10:00:00:00:c9:44:2b:8e 20:00:00:00:c9:44:2b:8e
fc1/13 200 0x0a0300 50:06:0b:00:00:22:7e:f0 50:06:0b:00:00:22:7e:f1
fc1/14 200 0x0a0400 50:06:0b:00:00:25:ca:f2 50:06:0b:00:00:25:ca:f3
fc1/15 100 0x0a0300 21:00:00:e0:8b:18:eb:48 20:00:00:e0:8b:18:eb:48
fc1/16 100 0x0a0400 21:00:00:e0:8b:18:89:4c 20:00:00:e0:8b:18:89:4c
MDS# show fcns database
VSAN 100:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x0a0007 N 50:06:04:8a:cc:d2:3a:61 (EMC) scsi-fcp:both 253
0x0a0009 N 50:06:04:8a:cc:d2:3a:6e (EMC) scsi-fcp:both 253
Bad Port Connection
module-2# show hardware int fc-mac port 29 gbic-info
--------- GBIC Info for user port 29 -----------
Present: Yes Tx Fault: No Rx Loss: No Tx Enable: Yes
Read State: not read
Present: Yes Tx Fault: No Rx Loss: Yes Tx Enable: No
Present: Yes Tx Fault: No Rx Loss: No Tx Enable: Yes
Hardware Internal Errors
MDS# show hardware internal errors all
----------------------------------------
Hardware errors as reported in module 1
----------------------------------------
Fc-Mac version: 00000003
==================== FC-MAC ERROR STATISTICS ====================
Non-Zero Port Counters for User Port 01
fcIfLinkFailures : 4
fcIfSyncLosses : 3
fcIfLinkResetIns : 4
fcIfLinkResetOuts : 1
fcIfOlsOuts : 5
fcIfLRROut : 4
fcIfLipF8In : 1a
fcIfNonLipF8Out : 4
MacInvalidTransmissonWords : c
MacLossofSync : 3
Faulty SFP
lab1# show interface brief
-------------------------------------------------------------------------------
Mode (Gbps)
-------------------------------------------------------------------------------
fc1/13 1 FX on sfpAbsent -- -- --
Link Stuck in Init State
2010 Feb 20 00:11:20 MDS %MODULE-5-MOD_OK: Module 3 is online (serial:
JAE1048GBNH)
2010 Feb 21 19:00:04 MDS %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configuring
console from pts/0 10.4.131.255
2010 Feb 21 19:05:42 MDS %PORT-5-IF_DOWN_INITIALIZING: %$VSAN 200%$
Interface fc2/5 is down (Initializing)
2010 Feb 21 19:05:43 MDS %PORT-5-IF_DOWN_INITIALIZING: %$VSAN 200%$
Interface fc2/6 is down (Initializing)
2010 Feb 21 19:05:44 MDS %PORT-5-IF_DOWN_FCOT_NOT_PRESENT: %$VSAN 100%$
Interface fc3/1 is down (FCOT not present)
Port Registration Issues
MDS# show interface fc1/8
fc1/8 is up
Port WWN is 20:08:00:0d:ec:0c:b4:40
. . . .
Flogi Event History
MDS# show flogi internal event-history interface fc1/8
>>>>FSM: <[100]50:01:24:82:01:ad:d0:cf> has 17 logged transitions<<<<<
1) FSM:<[100]50:01:24:82:01:ad:d0:cf> Transition at 928301 usecs after Fri Apr 16
19:55:15 2010
Previous state: [FLOGI_ST_FLOGI_RECEIVED]
Triggered event: [FLOGI_EV_FLOGI_RECEIVED]
Next state: [FLOGI_ST_FLOGI_RECEIVED]
19:55:15 2010
Previous state: [FLOGI_ST_FLOGI_RECEIVED]
Triggered event: [FLOGI_EV_VALID_FLOGI]
Next state: [FLOGI_ST_DPVM_CHECK]
19:55:15 2010
Previous state: [FLOGI_ST_DPVM_CHECK]
Triggered event: [FLOGI_EV_DPVM_IGNORE]
Next state: [FLOGI_ST_FETCH_PRECFG_INFO]
Configure Interfaces
MDS# config
MDS(config)# interface fc1/8
MDS(config-if)# no shut
MDS(config-if)# 2010 Apr 16 20:25:39.703198 fcns: vsan 100: reg timer started for
port-id 0x0a0008
2010 Apr 16 20:25:39.703386 fcns: vsan 100: Created entry for port-id 0x0a0008
2010 Apr 16 20:25:39.703420 fcns: vsan 100: Got Entry for port-id 0x0a0008
2010 Apr 16 20:25:39.703497 fcns: vsan 100: Registered port-name
50:01:24:82:01:ad:d0:cf for port-id 0x0a0008
2010 Apr 16 20:25:39.703649 fcns: vsan 100: Registered node-name
50:01:24:82:01:ad:d0:ce for port-id 0x0a0008
2010 Apr 16 20:25:39.703683 fcns: vsan 100: Registered cos 8 for port-id 0x0a0008
2010 Apr 16 20:25:39.703774 fcns: vsan 100: Registered port-type 1 for port-id
0x0a0008
2010 Apr 16 20:25:39.713245 fcns: vsan 100: Reading configuration for entry with
port-name 50:01:24:82:01:ad:d0:cf, node
-name 50:01:24:82:01:ad:d0:ce
2010 Apr 16 20:25:39.713351 fcns: vsan 100: No configuration present for this
portname
2010 Apr 16 20:25:39.713383 fcns: vsan 100: No configuration present for this
nodename
2010 Apr 16 20:25:39.713424 fcns: vsan 100: Saving new entry into pss
2010 Apr 16 20:25:39.713586 fcns: vsan 100: Sending sync message to the standby
FCNS Details
MDS# show fcns database detail vsan 200
------------------------
VSAN:200 FCID:0x0a0100
------------------------
port-wwn (vendor) :21:01:00:e0:8b:aa:2a:50 (Qlogic)
node-wwn :20:01:00:e0:8b:aa:2a:50
class :3
node-ip-addr :0.0.0.0
ipa :ff ff ff ff ff ff ff ff
fc4-types:fc4_features :scsi-fcp:init
symbolic-port-name :
symbolic-node-name :QLA2342 FW:v3.03.19 DVR:v7.08-vm66
port-type :N
port-ip-addr :0.0.0.0
fabric-port-wwn :20:45:00:0d:ec:0c:b4:40
hard-addr :0x000000
permanent-port-wwn (vendor) :21:01:00:e0:8b:aa:2a:50 (Qlogic)
Troubleshooting Port States with Device
Manager
Port Performance
Port Monitoring
Lab Exercise 12: Cisco Data Analysis
Parts 4 through 6
Lesson 4: Cisco Login

• Define the logins performed by an N_Port
• Use tools to collect and analyze data from login issues
• Analyze the login process using show tech-support
• Decipher the show tech-support file
• Perform root cause analysis of login issues
FC Login Process Review
dNS
Link Negotiation Link Negotiation

FLOGI FLOGI
dNS PLOGI dNS PLOGI
PLOGI
PRLI
Fx_Port Failure
Host and No
Troubleshoot
Begin storage completed
Physical Port
flogi?
Yes Troubleshoot
Node port
Host and
No
storage completed Check VSAN
dNS Membership
registration?
Yes
Check Zoning
Verify data path and LUN Masking
Show Link Negotiation
MDS# show interface brief
-------------------------------------------------------------------------------
Mode (Gbps)
-------------------------------------------------------------------------------
fc1/13 1 FX on sfpAbsent -- -- --
Show Interface
MDS# show interface
fc1/10 is up
Port WWN is 20:0a:00:0d:ec:82:66:00
Admin port mode is FX, trunk mode is on
Port mode is F, FCID is 0x290000
Port vsan is 11
Speed is 2 Gbps
Rate mode is shared
Port Errors
MDS# show interface
fc1/10 is up
. . .
Interface last changed at Mon Apr 26 20:13:42 2010
Port FLOGI Statistics
MDS# show flogi internal info
Interface fc1/10: mode[F] [1009000] Mode: F State: UP Vsan: 11
VF State: INVALID, admin_tagging_mode 0, vf_tagging_capable 0, log_if_index 0x0
Port wwn: 20:0a:00:0d:ec:82:66:00
B2B credit: 16 Buf size: 2112 Multiplier: 0
Perf buffers: 0 bb-scn not-supported cfg_vsan[11]
phy_port(TRUE) flogi - recv informed(TRUE) acc sent(TRUE)
flogi_allowed(TRUE) Invalid flogis 0
non_volatile - pbuf_reqd[0] reinit_sent[0] prev_vsan[11]
Num flogis: 1 FCIDs allocated: 1
Reinit reqd: FALSE
Permanent Port Name: [50:06:0b:00:00:1c:d6:4a]
NPIV was accepted: FALSE
Stats:
timestamp: Mon Apr 26 20:13:41 2010
MSG_FLOGI: 1
MSG_FC2_ACCEPT_OUT: 1
MSG_LCP_INIT_DONE: 1
Vsan list:
11
Vsan no: 11 Max flogi key: 0x1(1) num_fl[0x1] logged[0x1] flogi bitset: 0
FLOGI Database
MDS# show flogi database
--------------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
--------------------------------------------------------------------------------
fc1/2 10 0x150000 10:00:00:00:c9:6d:f1:2c 20:00:00:00:c9:6d:f1:2c
fc1/4 10 0x150001 50:06:04:8a:cc:d2:3c:b1 50:06:04:8a:cc:d2:3c:b1
fc1/5 10 0x150002 50:06:04:8a:cc:d2:3c:be 50:06:04:8a:cc:d2:3c:be
fc1/7 10 0x150003 10:00:00:00:c9:4e:a9:82 20:00:00:00:c9:4e:a9:82
fc1/8 10 0x1500ef 50:06:01:68:44:60:1e:39 50:06:01:60:c4:60:1e:39
fc1/10 11 0x290000 50:06:0b:00:00:1c:d6:4a 50:06:0b:00:00:1c:d6:4b
fc1/16 10 0x150004 50:06:04:8a:cc:d2:3c:8f 50:06:04:8a:cc:d2:3c:8f
fc1/17 10 0x150005 10:00:00:00:c9:6d:ec:e0 20:00:00:00:c9:6d:ec:e0
Total number of flogi = 8.
Name Server Database
MDS# show fcns database vsan 1-4093
. . .
VSAN 10:
--------------------------------------------------------------------------
--------------------------------------------------------------------------
0x150000 N 10:00:00:00:c9:6d:f1:2c (Emulex) scsi-fcp:init
0x150001 N 50:06:04:8a:cc:d2:3c:b1 (EMC) scsi-fcp:both 253
0x150002 N 50:06:04:8a:cc:d2:3c:be (EMC) scsi-fcp:both 253
0x150003 N 10:00:00:00:c9:4e:a9:82 (Emulex) ipfc scsi-fcp:init
0x150004 N 50:06:04:8a:cc:d2:3c:8f (EMC) scsi-fcp:both 253
0x150005 N 10:00:00:00:c9:6d:ec:e0 (Emulex) ipfc scsi-fcp:init
0x1500ef N 50:06:01:68:44:60:1e:39 (Clariion) scsi-fcp:target
0x6db560 N 50:06:01:68:44:60:20:8f (Clariion) scsi-fcp:target
Total number of entries = 8
VSAN 11:
--------------------------------------------------------------------------
--------------------------------------------------------------------------
0x290000 N 50:06:0b:00:00:1c:d6:4a (HP) scsi-fcp:init fc-gs
Total number of entries = 1
Ping and Traceroute
MDS# ping 10.127.118.30
PING 10.127.118.30 (10.127.118.30) 56(84) bytes of data.
64 bytes from 10.127.118.30: icmp_seq=1 ttl=64 time=0.238 ms
--- 10.127.118.30 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4000ms
rtt min/avg/max/mdev = 0.190/0.216/0.238/0.021 ms
MDS# traceroute 10.127.118.30
traceroute to 10.127.118.30 (10.127.118.30), 30 hops max, 38 byte
packets
1 10.127.118.30 (10.127.118.30) 0.300 ms 0.187 ms 0.172 ms
fcping
MDS# fcping fcid 0x0a0600 vsan 100
28 bytes from 0x0a0600 time = 830 usec
5 frames sent, 5 frames received, 0 timeouts
Round-trip min/avg/max = 687/812/879 usec
fctrace
MDS# show fcns database vsan 100
VSAN 100:
--------------------------------------------------------------------------
--------------------------------------------------------------------------
0x0a000e N 10:00:00:00:c9:44:2b:8e (Emulex)
0x0a000f N 10:00:00:00:c9:44:2b:ce (Emulex)
0x98000e N 10:00:00:00:c9:44:2c:81 (Emulex)
0x98000f N 10:00:00:00:c9:42:b1:7a (Emulex)
0x980400 N 21:00:00:e0:8b:18:32:49 (Qlogic) scsi-fcp:init
. . .
MDS# fctrace fcid 0x0a000f vsan 100
Route present for : 0x0a000f
20:00:00:0d:ec:0d:8a:00(0xfffc98) MDS Originating the trace
20:00:00:0d:ec:0c:b4:40(0xfffc0a) First hop to destination
20:00:00:0d:ec:0c:b4:40(0xfffc0a) Destination MDS
Zoning
MDS# show zoneset active vsan 1-4093
zoneset name POD1_MDSSERIES vsan 10
zone name POD1_HP1_HBA1_CX1_SPB_P0 vsan 10
* fcid 0x1500ef [pwwn 50:06:01:68:44:60:1e:39]
pwwn 50:06:0b:00:00:1c:d6:4a
zone name POD1_HP1_HBA1_DMX_FA16C_PA vsan 10
* fcid 0x150004 [pwwn 50:06:04:8a:cc:d2:3c:8f]
pwwn 50:06:0b:00:00:1c:d6:4a
zone name POD1_SUN1_HBA1_DMX_FA16C_PA vsan 10
* fcid 0x150005 [pwwn 10:00:00:00:c9:6d:ec:e0]
zone name POD1_ESX1_HBA1_DMX_FA16C_PA vsan 10
* fcid 0x150000 [pwwn 10:00:00:00:c9:6d:f1:2c]
zone name POD1_W2K8_HBA1_DMX_FA16C_PA vsan 10
* fcid 0x150003 [pwwn 10:00:00:00:c9:4e:a9:82]
IVR
MDS# show tech-support ivr
Show IVR
=============================================
Inter-VSAN Routing is enabled
Inter-VSAN enabled switches
---------------------------
AFID VSAN DOMAIN CAPABILITY SWITCH WWN
-------------------------------------------------------------------
1 1 0xd6(214) 0000000f 20:00:00:0d:ec:82:66:00 *
1 1 0xeb(235) 0000000f 20:00:00:0d:ec:72:46:c0
1 10 0x15( 21) 0000000f 20:00:00:0d:ec:82:66:00 *
1 10 0x29( 41) 0000000f 20:00:00:0d:ec:72:46:c0
1 11 0x29( 41) 0000000f 20:00:00:0d:ec:82:66:00 *
1 100 0x64(100) 0000000f 20:00:00:0d:ec:82:66:00 *
1 100 0x66(102) 0000000f 20:00:00:0d:ec:72:46:c0
1 200 0x67(103) 0000000f 20:00:00:0d:ec:72:46:c0
1 200 0x85(133) 0000000f 20:00:00:0d:ec:82:66:00 *
Total: 9 IVR-enabled VSAN-Domain pairs
Link Flapping
fc1/5 is up
. . .
Link Initialization Flow
LF1 Link Failure
NOS LF2
OL1Link Initialize
OL2 OLS
LR1
LR Link Reset
LR2
LRR
LR3
IDLE
AC Active
AC IDLE
Viewing Port Counters
MDS# show interface counters brief
------------------------------------------------------------------------------
Interface Input (rate is 5 min avg) Output (rate is 5 min avg)
----------------------------- -----------------------------
Rate Total Rate Total
MB/s Frames MB/s Frames
-------------------------------------------------------------------------------
fc1/1 0 9113364 0 28838284
fc1/2 0 5890942 0 11800555
fc1/3 0 41513505 0 11637678
fc1/4 0 43152985 0 12092181
fc1/5 0 3328245 0 4166022
fc1/6 0 2584477 0 3686663
fc1/7 0 692767 0 1089194
fc1/8 0 792114 0 10354643
fc1/9 0 65989332 0 104568378
fc1/10 0 25975167 0 49400927
fc1/11 0 350724 0 562256
fc1/12 0 291059 0 444236
fc1/13 0 4 0 4
Port Bouncing
fc2/25 is down (Isolation due to ELP failure: class F param error)
Port WWN is 20:59:00:0d:ec:0d:8a:00
Admin port mode is E, trunk mode is on
Port vsan is 300
Show Port Internal Errors
MDS# show port internal event-history interface fc2/25
197) FSM:<fc2/25> Transition at 981158 usecs after Thu Apr 22 14:33:55 2010
Previous state: [PI_FSM_ST_ELP_SENT]
Triggered event: [PI_FSM_EV_FC_RCVD_SW_RJT]
Next state: [FSM_ST_NO_CHANGE]
Triggered event: [PI_FSM_EV_MY_ELP_REJECTED]
Next state: [PI_FSM_ST_MY_ELP_REJECTED_WAIT_FOR_OTHER_ELP]
Previous state: [PI_FSM_ST_MY_ELP_REJECTED_WAIT_FOR_OTHER_ELP]
Triggered event: [PI_FSM_EV_FC_RCVD_ELP]
Triggered event: [PI_FSM_EV_ELP_RCVD_RJT_RESULT]
Next state: [PI_FSM_ST_E_PORT_ISOLATED_ELP_ESC]
Curr state: [PI_FSM_ST_E_PORT_ISOLATED_ELP_ESC]
Port Interface
MDS# show port internal info interface fc2/25
fc2/25 - if_index: 0x01098000, phy_port_index: 0x38
local_index: 0x0
Admin Config - state(up), mode(TE), speed(auto), trunk(on)
beacon(off), snmp trap(on), tem(false)
rx bb_credit(default), rx bb_credit multiplier(default)
rx bb_credit performance buffers(default)
bb scn config(on)
ignore flags (ignore:none), service state(in service)
rxbufsize(2112), encap(default), user_cfg_flag(0x5)
description()
admin port-down link-failure(disabled)
Operational Info - state(down), mode(E), speed(2 Gbps), trunk(off)
state reason(Isolation due to ELP failure: class F param error)
phy port enable (1), phy layer (FC), port_osm_mode(OSM)
participating(1), fcid(0x000000), bb_scn(0)
config port vsan (300), oper port vsan (300)
rx bb_credit(12), rx bb_credit multiplier(400)
rx bb_credit performance buffers(0), tx bb_credit(0)
not first time port bringup is 1
Port Is in ErrDisabled State
fc2/10 is down (errDisabled)
show port internal event-history interface
MDS# show port internal event-history interface fc2/25
Triggered event: [PI_FSM_EV_FC_RCVD_SW_RJT]
Triggered event: [PI_FSM_EV_MY_ELP_REJECTED]
Next state: [PI_FSM_ST_MY_ELP_REJECTED_WAIT_FOR_OTHER_ELP]
Triggered event: [PI_FSM_EV_FC_RCVD_ELP]
Triggered event: [PI_FSM_EV_ELP_RCVD_RJT_RESULT]
Next state: [PI_FSM_ST_E_PORT_ISOLATED_ELP_ESC]
Curr state: [PI_FSM_ST_E_PORT_ISOLATED_ELP_ESC]
show logging logfile
MDS# show logging logfile
2010 Apr 22 14:32:59 MDS %PORT-5-IF_TRUNK_DOWN: %$VSAN 300%$ Interface
fc2/25, vsan 300 is down (Gracefully shutdown)
2010 Mar 5 14:29:31 MDS %PORT-5-IF_UP: %$VSAN 100%$ Interface fc1/7 is
up in mode F
2010 Apr 22 14:32:59 MDS %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 300%$
Interfacefc2/25 is down (OLS received)
2010 Mar 5 14:29:31 MDS %PORT-5-IF_UP: %$VSAN 100%$ Interface fc1/8 is
up in mode F
2010 Apr 22 14:33:55 MDS %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 300%$
Interfacefc2/25 is down (OLS received)
2010 Mar 5 14:30:44 MDS %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 100%$
Interface fc1/4 is down (Link failure)
2010 Apr 22 14:33:55 MDS %PORT-5-
IF_DOWN_ELP_FAILURE_ISOLATION_CLASSF_PARAM_ERR: %$VSAN 300%$ Interface
fc2/25 is down (Isolation due to ELP failure: class F param error)
Lab Exercise 13: Cisco Login
Lesson 5: Fibre Channel Fabrics

• List the steps required to create an ISL
• List common issues and misconfiguration steps prior to a merge
• Analyze and correct fabric merge failures
• Describe port aggregation principles
• List the steps to find and resolve routing issues
• Use tools to collect and analyze data
Fabric Configuration Overview
Word Determine Port
Synchronization Operating Mode
Acquired (FL, F, E)
E_Port
Switch Port
YES Mode? NO
Initialization
Exchange Link Exchange Switch
Parameters Capabilities Wait for Fabric
(ELP) (ESC) (Optional) Login
Select Principal Switch Exchange Fabric

Principal Selected Parameters
Switch (DIA) (EFP)
Assign
Domain Request Domain
ID Identifiers (RDI)
Build
Compute Least
Routing Tables Assigns
Domain ID Build Topology Cost Paths
and Domain
NO Available? YES Database (FSPF) Build Routing
Select Table ID
Table
Principal Switch Selection
Downstream Switch Principal Switch
The notation “X#n” refers to The Switch with the lowest
the unique Exchange ID of DIA Switch_Priority/ Switch_Name
each SW_ILS request and declares itself the Principal Switch,
response in the process. assigns itself a Domain_ID, and
informs all downstream Switches
SW_ACC that it has an address identifier.
Once the Principal Switch
has been chosen and has
assigned itself a Domain_ID, The SW_ACC completes the DIA
the downstream Switch may RDI Exchange.
request a Domain_ID.
The Principal Switch grants a
Domain_ID and sends the
SW_ACC
SW_ACC.
The downstream Switch has
received a Domain_ID and After processing each RDI, the
may send DIA to any Principal Switch originates an EFP
Switches further EFP request to all Principal ISLs to
downstream. announce the change to the
Domain_ID_List.
SW_ACC
The EFP is propagated
downstream by all other NOTE: ACK_1 frames and
Switches. R_RDYs have been omitted for
clarity.
Fabric Shortest Path First (FSPF)
A B
Speed: 4 Gbps
Cost: 250
Speed: 10 Gbps Speed: 1 Gbps

Cost: 100 Cost: 1000
Speed: 2 Gbps
Cost: 500
C D
FSPF Issues
Domain ID 1
Domain ID 4
Port 1
Port 1
Index 0x00010001
Index 0x00010000
Metric 1000
Domain ID 1 Domain ID 4
Port 1 Port 2
Index 0x00010001 Index 0x00010001
Metric 1000 Metric 1000
Domain ID 2 Domain ID 3
Port 2 Port 2
Index 0x00010001 Index 0x00010003
Domain ID 2
Domain ID 3
Port 4
Port 3
Index 0x00010003
Index 0x00010002
Metric 1000
View the FSPF Database
MDS# show fspf database vsan 300

FSPF Link State Database for VSAN 300 Domain 0x01(1)
LSR Type = 1
Advertising domain ID 
= 0x01(1)
LSR Age = 1151 
LSR Incarnation number = 0x800000c3 
LSR Checksum = 0x6ee6
Number of links = 2
NbrDomainId IfIndex NbrIfIndex Link Type Cost
-----------------------------------------------------------------------------
0xef(239) 0x00010118 0x0001011c 1 500 
0x02(2) 0x00040000 0x00040000 1 250

LSR Type = 1

FSPF Link State Database for VSAN 300 Domain 0x02(2)
Advertising domain ID
LSR Age

= 0x02(2)
= 685
LSR Incarnation number = 0x800000c6
LSR Checksum = 0x2468
Number of links = 2
NbrDomainId IfIndex NbrIfIndex Link Type Cost
Verify FSPF is Active
MDS# show fspf vsan 300 interface fc2/25
FSPF interface fc2/25 in VSAN 300

Interface cost is 500

FSPF routing administrative state is active
Timer intervals configured, Hello 20 s, Dead 80 s, Retransmit 5s

FSPF State is FULL

Neighbor Domain Id is 0xb4(180)
Neighbor Interface is fc2/29 (0x0001011c)
Statistics counters :
Number of packets received : LSU 187 LSA 554 Hello 16514 Error packets 0
Number of packets transmitted : LSU 554 LSA 187 Hello 16513 Retransmitted
LSU 0
Number of times inactivity timer expired for the interface = 0
Show FSPF Routes
MDS# show fspf internal route vsan 300
FSPF Unicast Routes
---------------------------
VSAN Number Dest Domain Route Cost Next hops
-----------------------------------------------------------
300 0x01(1) 250 port-channel 1
300 0xb4(180) 500 fc2/25
300 0xef(239) 750 port-channel 1
Resolving a Wrong Hello Interval
MDS# show fspf internal route vsan 300
FSPF Unicast Routes
---------------------------
VSAN Number Dest Domain Route Cost Next hops
----------------------------------------------------------
300 0x01(1) 250 port-channel 1
300 0xb4(180) 500 fc2/25
300 0xef(239) 750 port-channel 1
MDS# debug fspf all
MDS# 2010 Apr 26 10:47:51.074041 fspf: FC2 packet Receive for VSAN 300 intf fc2/25
2010 Apr 26 10:47:51.074126 fspf: 14 00 00 00 02 00 00 00 00 00 00 B4 00 00 00 00
2010 Apr 26 10:47:51.074168 fspf: 00 00 00 00 00 00 00 00 00 00 00 14 00 00 00 50
2010 Apr 26 10:47:51.074204 fspf: 00 00 00 02 00 01 01 1C
2010 Apr 26 10:47:51.074234 fspf:
2010 Apr 26 10:47:51.074278 fspf: Wrong hello interval for packet on interface 1098000 in VSAN 300
2010 Apr 26 10:47:51.074651 fspf: Error in processing hello packet , error = Bad packet received
2010 Apr 26 10:47:51.832566 fspf: FC2 packet Receive for VSAN 100 intf port-channel 1
2010 Apr 26 10:47:51.832641 fspf: 14 00 00 00 02 00 00 00 00 00 00 0A 00 00 00 00
2010 Apr 26 10:47:51.832683 fspf: 00 00 00 00 00 00 00 00 00 00 00 14 00 00 00 50
2010 Apr 26 10:47:51.832719 fspf: 00 00 00 98 00 04 00 00
2010 Apr 26 10:47:51.832750 fspf:
FSPF Operating Incorrectly
Timer intervals configured, Hello 20 s, Dead 80 s, Retransmit 5 s
FSPF State is INIT
LSU 0
Resolving a Mismatched Retransmit Interval
FSPF State is INIT
LSU 0
FSPF Transmit State Resolved
FSPF State is FULL
Neighbor Domain Id is 0xb4(180)
Neighbor Interface is fc2/29 (0x0001011c)
LSU 0
ISL Behavior
Switch Upstream: none

A Downstream: AB, AC
Upstream: CA
Downstream: none
Upstream: BA Switch Switch
Downstream: BD, BE B C
Switch Switch
Upstream: DB Upstream: EB
Downstream: none
D E Downstream: EF
Upstream: FE Switch
Downstream: none Principal ISL
F
ISL
Trunk Aggregation
3.5 Gb 4 Gb 3 Gb 2 Gb 4 Gb 1 Gb 1.5 Gb 3.5 Gb 4 Gb 3 Gb 2 Gb 4 Gb 1 Gb 1.5 Gb
Oversubscribed ISL 6 ISLs 6 ISL Aggregate

4 Gbps Total: 24 Gbps
3.5 Gb 4 Gb 3 Gb 2 Gb 4 Gb 1 Gb 1.5 Gb 3.5 Gb 4 Gb 3 Gb 2 Gb 4 Gb 1 Gb 1.5 Gb
No Aggregation Aggregate ISLs
Lesson 6: Cisco Fabrics

• List the commands used to troubleshooting
 ISL
 VSANs
 PortChannels
 Fabric errors
• Define the Cisco Fabric merge process
• Show how to find and resolve VSAN issues
• Use tools to collect and analyze data
Zone Merge Failure
• A zone merge request may fail because of the following
configuration issues:
 Too many zone sets
 Too many aliases
 Too many attribute groups
 Too many zones
 Too many LUN members
 Too many zone members
Failed Zone Merge Using Fabric Manager
Zoning Issues Using CLI
fc2/25 is down (Isolation is due to zone merge failure)
Port vsan is 300
ISL Isolation Using Fabric Manager
FC Domain ID Overlap
fc2/25 is down (Isolation is due to domain overlap)
Port vsan is 300
Fabric Reconfiguration for Domain ID
Assignments
MDS# show fcdomain vsan 300
The local switch is a Subordinated Switch.
Local switch run time information:
State: Stable
Local switch WWN: 21:2c:00:0d:ec:0d:8a:01
Running fabric name: 21:2c:00:0d:ec:0c:b4:41
Running priority: 128
Current domain ID: 0x02(2)
Local switch configuration information:
State: Enabled
FCID persistence: Enabled
Auto-reconfiguration: Disabled
Contiguous-allocation: Disabled
Configured fabric name: 20:01:00:05:30:00:28:df
Optimize Mode: Disabled
Configured priority: 128
Configured domain ID: 0x02(2) (static)
Principal switch run time information:
Running priority: 2
Interface Role RCF-reject
---------------- ------------- ------------
fc2/25 Downstream Disabled
port-channel 1 Upstream Disabled
Allowed Domain ID List
MDS# show fcdomain allowed
VSAN 1
Already assigned or reserved domain IDs: none.
[User] configured allowed domain IDs: 1-239.
VSAN 300
Already assigned or reserved domain IDs: 1-2,180,239.
[User] configured allowed domain IDs: 1-239.
MDS# show fcdomain domain-list
VSAN 1
Number of domains: 1
Domain ID WWN
--------- -----------------------
0x7c(124) 20:01:00:0d:ec:0d:8a:01 [Local] [Principal]
VSAN 300
Number of domains: 4
Domain ID WWN
--------- -----------------------
0x01(1) 21:2c:00:0d:ec:0c:b4:41 [Principal]
0x02(2) 21:2c:00:0d:ec:0d:8a:01 [Local]
0xef(239) 21:2c:00:0d:ec:18:d4:c1
0xb4(180) 21:2c:00:0d:ec:18:cb:41
VSAN Attributes
• Traffic isolation:
 Control over each incoming and VSAN 10 VSAN 20 VSAN 30 VSAN 1
outgoing port (default)
• 256 VSANs max per switch:
 239 switches per VSAN
 Number of supported VSANs is code
dependent
• Each frame in the fabric is uniquely MDS 9506
Chassis
identified:
 Labeled with a VSAN_ID header on the
ingress port
 VSAN_ID stripped away across E_Ports
 Maintained across TE_Ports
 SAN and priority in header for QoS
• FCID can be reused across multiple
VSANs:
 Switch granularity
• Ease of management:
 Port-level configs on moves Fabric 10 Fabric 20 Fabric 30 Fabric 1
Domain ID Domain ID Domain ID Domain ID
0x61 0x94 0x33 0x12
44 Ports 24 Ports 12 Ports 8 Ports
(Logically within the MDS 9506 single chassis)
Configuring a TE_Port
(config-if)# switchport mode e
(config-if)# switchport trunk mode on
(config-if)# switchport speed 2000
(config-if)# no shutdown
(config-if)# end Trunk mode is set globally
# show interface fc1/2 brief during the switch setup.
-------------------------------------------------------------
Interface Vsan Admin Admin Status FCOT Oper Oper Port
Mode (Gbps)
-------------------------------------------------------------
fc1/2 1 E on up swl TE 2 --
Configuring a TE_Port continued
1 ISL (E) 2
EISL (TE)
Administrative Trunk Administrative Trunk Operational Trunk

Mode (1) Mode (2) Mode
ON ON TE_Port
ON OFF E_Port
ON AUTO TE_Port
AUTO AUTO TE_Port
Configure Trunking Mode and Allowed VSANs
(config-if)# switchport trunk ?
allowed Configure allowed list for interface(s)
mode Configure trunking mode
(config-if)# switchport trunk allowed ?
vsan Configure allowed list for interface(s)
(config-if)# switchport trunk allowed vsan 1
(config-if)# switchport trunk allowed vsan add 20
updated trunking membership
(config-if)# switchport trunk mode ?
auto Autosense trunking for an interface
off Disable trunking for an interface
on Enable trunking for an interface
(config-if)# switchport trunk mode on
VSAN Checklist
VSAN Checklist
Verify the domain parameters for switches in the VSAN

Verify the physical connectivity for any problem ports or VSANs


Verify that you have both devices in the name server


Verify that you have both end devices in the same VSAN

Verify that you have both end devices in the same zone

Verify that the zone is part of the active zone set


Verify the FSPF parameters for switches in the VSAN


Common Troubleshooting Commands
• Use the following commands to display VSAN, FC domain, and FSPF information:
 show vsan
 show vsan vsan-id
 show vsan membership
 show interface fc slot/port trunk vsan-id
 show vsan-id membership
 show vsan membership interface fc slot/port
 show fcdomain
 show fspf
 show fspf internal route vsan vsan-id
 show fcns database vsan vsan-id
• Use the following zone commands to validate your configuration:
 show zoneset name zoneset-name vsan vsan-id
 show zoneset active vsan vsan-id
 show zone vsan vsan-id
 show zone status show vsan vsan-range
Verify VSAN Membership
MDS# show vsan membership
vsan 1 interfaces:
fc2/7 fc2/8 fc2/9 fc2/10
fc2/11 fc2/12 fc2/13 fc2/14
fc2/15 fc2/16 fc2/17 fc2/18
fc2/19 fc2/20 fc2/22 fc2/23
fc2/24 fc3/1 fc3/2 fc3/3
fc3/4 fc3/5 fc3/6 fc3/7
fc3/8 fc3/9 fc3/10 fc3/11
fc3/12 fc3/13 fc3/14 fc3/15
fc3/16 fc3/17 fc3/18 fc3/19
fc3/20 fc3/21 fc3/22 fc3/23
fc3/24 fc3/25 fc3/26 fc3/27
fc3/28 fc3/29 fc3/30 fc3/31
fc3/32
vsan 100 interfaces:
fc1/1 fc1/2 fc1/3 fc1/4
fc1/5 fc1/6 fc1/7 fc1/8
fc1/9 fc1/10 fc1/11 fc1/12
fc1/15 fc1/16 fc2/1 fc2/2
Verify the Port Configuration
fc2/29 is trunking
Port WWN is 20:5d:00:0d:ec:0d:8a:00
Peer port WWN is 20:5d:00:0d:ec:0c:b4:40
Port mode is TE
Port vsan is 300
Speed is 2 Gbps
Belongs to port-channel 1
Trunk vsans (admin allowed and active) (100,300)
Trunk vsans (up) (100,300)
Trunk vsans (isolated) ()
Verify Allowed List
fc2/25 is trunking
Peer port WWN is 20:5d:00:0d:ec:18:cb:40
Port mode is TE
Port vsan is 300
Speed is 2 Gbps
Trunk vsans (admin allowed and active) (1,300)
Trunk vsans (up) (300)
Trunk vsans (isolated) (1)
Trunk vsans (initializing) ()
VSAN Misconfiguration
MDS# show interface fc2/25 trunk vsan 300
fc2/25 is trunking
Vsan 300 is up (None)
MDS# show interface fc2/25 trunk vsan 1
fc2/25 is trunking
Vsan 1 is down (Isolation due to vsan not configured on peer)
Root Cause for VSAN Isolation
MDS# show port internal info interface fc2/25
fc2/25 - if_index: 0x01098000, phy_port_index: 0x38
local_index: 0x0
Admin Config - state(up), mode(TE), speed(auto), trunk(on)
beacon(off), snmp trap(on), tem(false)
rx bb_credit(default), rx bb_credit multiplier(default)
rx bb_credit performance buffers(default)
bb scn config(on)
ignore flags (ignore:none), service state(in service)
rxbufsize(2112), encap(default), user_cfg_flag(0x5)
description()
admin port-down link-failure(disabled)
Operational Info - state(trunking), mode(TE), speed(2 Gbps), trunk(on)
. . .
TE_Port per vsan information
fc2/25, Vsan 1 - state(down), state reason(Isolation due to vsan not
configured on peer), fcid(0x000000)
port init flag(0x0), Lock Info: resource [fc2/25, vsan 1]
PortChannel Overview
• A PortChannel has the following functionality:
 Increases the aggregate bandwidth
 Balances loads across multiple links and maintains optimum
bandwidth utilization
 Provides fault tolerance on an ISL
 A PortChannel can include up to 16 ISLs
Single PortChannel Between Multiple PortChannels Between

Two MDS Switches Two MDS Switches
PortChannel Configuration Guidelines
• Configure the PortChannel across switching modules:
 Provides redundancy on switching module reboots or upgrades.
• Ensure that the PortChannel is not connected across different
sets of switches:
• PortChannel member interfaces must be able to configure into E-
ports
VSAN Trunking
ACL
VSAN 10
VSAN 30
VSAN 10 VSAN 10
VSAN 20 VSAN 20
VSAN 30 VSAN 30
PortChannel Checklist
PortChannel and Trunking Checklist
Use the show port-channel compatibility-parameters command to determine

PortChannel requirements 
Ensure that one PortChannel is not connected to different sets of switches.

PortChannels require point-to-point connections between the same set of 
switches
Verify that either side of a PortChannel is connected to the same number of
interfaces 
Verify that each interface is connected to the same type of interface on the
other side 
Verify that all required VSANS on a TE_Port are in the allowed-active VSAN list

Common Troubleshooting Commands in the
CLI
• The following commands may be useful in troubleshooting
PortChannel and trunking:
 show port-channel compatibility-parameters
 show port-channel summary
 show port-channel database
 show port-channel consistency detail
 show port-channel usage
 show interface
 show interface trunk
 show trunk protocol
Common PortChannel Issues
MDS# show port-channel database
port-channel 1
Administrative channel mode is on
Operational channel mode is on
Last membership update succeeded
First operational port is fc2/21
2 ports in total, 2 ports up
Ports: fc2/21 [up] *
fc2/29 [up]
Common Trunking Issues
MDS# show trunk protocol
Trunk Protocol is enabled
MDS# show interface port-channel 1
port-channel 1 is trunking
Hardware is Fibre Channel
Port mode is TE
Port vsan is 300
Speed is 4 Gbps
Trunk vsans (admin allowed and active) (300)
Trunk vsans (up) (300)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
Lab Exercise 14: Cisco Fabrics
Lesson 7: Cisco FC Security

• List security enhancements within the FC space
• List common issues and misconfiguration steps when creating
security configurations
• Resolve security issues in the FC SAN
• Use tools to collect and analyze security data
Security Checklist
Security Checklist
Verify licensing requirements 

Verify that your installed HBAs support FC-SP 
Verify that you have configured MD5 for the hash algorithm if you are
authenticating through a RADIUS or TACACS+ server. 
Verify that your AAA server is functioning properly 

Verify that port security is activated and that the end devices are present in
the port security active database 
Verify that no unauthorized devices (host or switch) are connected to a port. 
Verify that you have activated fabric binding 

Verify that all switches in the fabric have the same fabric binding database
settings. 
Common Troubleshooting Commands
• commands to troubleshoot FC-SP issues:
 show fcsp interface
 show fcsp internal event-history errors
 show fcsp dhchap
 show fcsp dhchap database
• commands to troubleshoot port security issues:
 show port-security status
 show port-security database vsan
 show port-security database active vsan
 show port-security violations
 show port-security internal global
 show port-security internal info vsan
 show port-security internal state-history vsan
 show port-security internal commit-history vsan
 show port-security internal merge-history vsan
• commands to troubleshoot fabric binding issues:
 show fabric-binding status
 show fabric-binding database vsan
 show fabric-binding database active vsan
 show fabric-binding violations
 show fabric-binding internal global
 show fabric-binding internal info
 show fabric-binding internal event-history
 show fabric-binding internal efmd event-history
Enable FC-SP
MDS# conf
MDS(config)# fcsp enable
ENTERPRISE_PKG license not installed. FCSP feature will be shutdown
after grace period of approximately 115 day(s).
MDS(config)# exit
MDS# show fcsp
fc-sp authentication TOV:30
Modify the DHCHAP Database
MDS# show fcsp dhchap database
DHCHAP Local Password:
Non-device specific password:
Other Devices' Passwords:
MDS# show wwn switch
Switch WWN is 20:00:00:0d:ec:0d:8a:00
MDS# conf
MDS(config)# fcsp dhchap devicename 20:00:00:0d:ec:0d:8a:00 password SW-
pass1
MDS(config)# exit
MDS# show fcsp dhchap database
DHCHAP Local Password:
Non-device specific password:
Other Devices' Passwords:
Password for device with WWN:20:00:00:0d:ec:0d:8a:00 is ********
View Port Security Database
MDS# show port-security database
--------------------------------------------------------------------------------
VSAN Logging-in Entity Logging-in Point (Interface)
--------------------------------------------------------------------------------
100 50:06:04:8a:cc:d2:3a:41(pwwn) 20:0a:00:0d:ec:0d:8a:00(fc1/10)*
200 21:00:00:e0:8b:8a:09:4b(pwwn) 20:46:00:0d:ec:0d:8a:00(fc2/6)*
300 20:00:00:0d:ec:18:cb:40(swwn) 20:59:00:0d:ec:0d:8a:00(fc2/25)*
[Total 3 entries]
Verify Port Binding
MDS# show interface brief
-------------------------------------------------------------------------------
Mode (Gbps)
-------------------------------------------------------------------------------
. . .
fc1/8 100 auto on up swl F 2 --
fc1/13 200 auto on invalidBinding swl -- --
fc1/14 200 auto on invalidBinding swl -- --
fc2/1 100 FX -- up swl F 2 --
fc2/2 100 FX -- up swl F 2 --
View Port Security Violations
MDS# show port-security violations
-------------------------------------------------------------------------------
VSAN Interface Logging-in Entity Last-Time [Repeat count]
-------------------------------------------------------------------------------
200 fc1/13 50:06:0b:00:00:23:60:d2(pwwn) Apr 26 12:57:55 2010 [1]
50:06:0b:00:00:23:60:d3(nwwn)
200 fc1/14 50:06:0b:00:00:25:28:fe(pwwn) Apr 26 12:57:55 2010 [1]
50:06:0b:00:00:25:28:ff(nwwn)
200 fc2/5 21:01:00:e0:8b:aa:09:4b(pwwn) Apr 26 12:57:55 2010 [1]
20:01:00:e0:8b:aa:09:4b(nwwn)
[Total 3 entries]
Fabric Binding
MDS# show fabric-binding database
--------------------------------------------------
Vsan Logging-in Switch WWN Domain-id
--------------------------------------------------
300 20:00:00:0d:ec:0d:8a:00 0x2(2) [Local]
300 20:00:00:0d:ec:18:cb:40 Any
300 20:00:00:0d:ec:18:cb:4e Any
[Total 3 entries]
MDS# show fabric-binding violations
-------------------------------------------------------------------------------
VSAN Switch WWN [domain] Last-Time [Repeat count] Reason
-------------------------------------------------------------------------------
300 20:00:00:0d:ec:18:cb:4e Apr 26 12:57:55 2010 [3] sWWN not found
Fabric Binding in an MDS-Series Environment
Switch# config
(config)# feature fabric-binding
(config)# fabric-binding database vsan 10
(config-fabric-binding)# swwn 21:00:05:30:23:11:11:11
(config-fabric-binding)# swwn 21:00:05:30:23:1a:11:03 domain 25
(config-fabric-binding)# exit
(config) fabric-binding activate vsan 10
Switch# fabric-binding database copy vsan 10
Switch# copy running-config startup-config
Switch# show fabric-binding database
--------------------------------------------------
Vsan Logging-in Switch WWN Domain-id
--------------------------------------------------
10 21:00:05:30:23:11:11:11
10 21:00:05:30:23:1a:11:03 0x19(25)
10 20:00:00:05:30:00:2a:1e 0xea(234) [Local]
[Total 3 entries]
Lesson 8: Cisco FC Optimization

• Define key performance points and bottlenecks
• Locate and resolve Credit Starvation issues
• Use tools to collect and analyze performance data
Performance Issues
Congestion
Slow
Credit Starvation SAN Slow Draining Device
Performance
Marginal Link
Oversubscription
• Allows the sharing of a few expensive array ports among many
host ports
• Fan-out ratios of storage subsystem ports typically range from
4:1 to 15:1
• Good oversubscription designs maximize performance while
minimizing cost
Port Bandwidth Reservation
• Allows for greater flexibility in
deploying oversubscribed modules.
• 2nd and 3rd-generation modules
only. 12-Gbps
Shared Bandwidth
• Ports within a port group can be Shared Unused
allocated 1, 2,4 or 8 Gbps of Bandwidth 6 Gbps
Port 1: Dedicated 4 Gbps
Port 6: Dedicated 2 Gbps

guaranteed bandwidth:
 Interface port mode becomes
dedicated
• Other ports share unused
bandwidth:
Port 2
Port 3
Port 5
Port 4
 Shared ports have 16 BB credits
Example of 24-port FC Module:

• Dedicating one port to 4 Gbps
• Dedicating another port to 2 Gbps
Credit Starvation
• Performance can drop when there are insufficient buffers to
keep the link full
 Often associated with long distance configurations
 Higher number of credits are required for long distance
• Can occur with end nodes and ISLs
 If R_RDY primitives are dropped the port will have unavailable
credit and show “slow performance”
 The port will stay in this state unit it is reset or it reaches zero
buffer credits for ED_TOV (2 seconds)
Overview of BB_Credit
Sequence Local Remote Sequence
Initiator F_Port F_Port Recipient
BB_Credit_CNT BB_Credit_CNT BB_Credit_CNT BB_Credit_CNT
16 64 7 16
Data Frame Data Frame
-1 -1
+1 +1
R_RDY R_RDY
-1 -1
-1 -1
-1 -1
+1 +1
+1 +1
+1 +1
Resolving Credit Starvation
• For long distance, configure the port with the correct amount of
long distance buffers
• Check cables, SFPs and patch panels
 Each patch panel can lead to a .5dB loss in signal
 Lost R_RDYs are often physical layer faults
 Resetting the port will temporarily resolve the problem
Monitor Credit Starvation
• Monitor performance while congestion is occurring
• Look for class 3 frame drops on the F_Port from the initiator
• Look for lower than expected utilization and buffer starvation on ISLs
• If the problem is a N_Port check for lower utilization and frame drop on the
ISL
Low I/O Low I/O

Credit Starvation Credit Starvation
Frame Drop
Frame Drop
Insufficient
BB_Credit
Slow Draining Devices
• The Slow Drain device does not send R_RDYs back to the transmitting port
• This prevents the transmitting switch port to run out of buffer credits
• Frames bound for the device will have to be held in switch buffers
• Backup will occur throughout the fabric until the switch hold time is exceeded and are
then dropped
• As switch buffers are filled other nodes will also be affected
Frames Backup Frames Backup Slow Drain
ISLs
Monitor Slow Draining Devices
• Monitor performance while congestion is occurring
• Look for class 3 frame drops on the F_Port from the initiator
• Look for lower than expected utilization on ISL egress ports
• Look for buffer starvation on ISL ingress ports
Low I/O Low I/O

Credit Starvation Credit Starvation
Frame Drop
Frame Drop
Slow Drain
Device
Lab Exercise 15: Cisco FC Optimization
Module Summary
Key points covered in this module:
• How to resolve end to end connectivity issues
• How to analyze and correct ISL issues
• How to analyze and correct fabric merge failures
• How to find and resolve virtualized SAN issues
• How to find and resolve routing and trunking issues
• How to use logs and debugging tools to troubleshoot a SAN
• How to analyze and correct VSAN issues
• How to resolve security issues in the FC SAN
• How to use FC tools to trace errors for root cause determination

R - 04 - MR-1CP-NSSANTS - Cisco FC Troubleshooting

Uploaded by

Copyright:

Available Formats

R - 04 - MR-1CP-NSSANTS - Cisco FC Troubleshooting

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

R - 04 - MR-1CP-NSSANTS - Cisco FC Troubleshooting

Uploaded by

Copyright:

Available Formats

Module 4: Cisco Fibre Channel Fabric

Upon completion of this module, you should be able to:

Lesson 1: Cisco Switch Configuration

MDS# copy running-config startup-config

Lesson 2: Cisco Data Collection

Started at Wed Jun 7 18:34:28 2006 (694067 us)

Start type: SRV_OPTION_RESTART_STATELESS (23)

Time stamp : At 25693 usecs after Wed Apr 14 11:53:35 2010

Time stamp : At 534306 usecs after Fri Mar 12 07:31:39 2010

Mod MAC-Address(es) Serial-Num

BIOS • Loads the loader

Loader • Loads the kickstart image

• Loads the kernel, basic drivers,

System Image • Login prompt

Lesson 3: Cisco Data Analysis

Verify that no ports on a Generation 2 module are out of service. 

Lesson 4: Cisco Login

Link Negotiation Link Negotiation

LF1 Link Failure

Lesson 5: Fibre Channel Fabrics

Select Principal Switch Exchange Fabric

Speed: 10 Gbps Speed: 1 Gbps

Metric 1000 Metric 1000

Switch Upstream: none

3.5 Gb 4 Gb 3 Gb 2 Gb 4 Gb 1 Gb 1.5 Gb 3.5 Gb 4 Gb 3 Gb 2 Gb 4 Gb 1 Gb 1.5 Gb

Oversubscribed ISL 6 ISLs 6 ISL Aggregate

3.5 Gb 4 Gb 3 Gb 2 Gb 4 Gb 1 Gb 1.5 Gb 3.5 Gb 4 Gb 3 Gb 2 Gb 4 Gb 1 Gb 1.5 Gb

No Aggregation Aggregate ISLs

Lesson 6: Cisco Fabrics

(Logically within the MDS 9506 single chassis)

Administrative Trunk Administrative Trunk Operational Trunk

AUTO AUTO TE_Port

Verify the physical connectivity for any problem ports or VSANs

Verify that you have both devices in the name server

Verify that the zone is part of the active zone set

Verify the FSPF parameters for switches in the VSAN

Single PortChannel Between Multiple PortChannels Between

Use the show port-channel compatibility-parameters command to determine

Ensure that one PortChannel is not connected to different sets of switches.

Lesson 7: Cisco FC Security

Verify licensing requirements 

Verify that your AAA server is functioning properly 

Verify that no unauthorized devices (host or switch) are connected to a port. 

Verify that you have activated fabric binding 

Lesson 8: Cisco FC Optimization

Port 1: Dedicated 4 Gbps

Port 6: Dedicated 2 Gbps

Example of 24-port FC Module:

Low I/O Low I/O

Frames Backup Frames Backup Slow Drain

Low I/O Low I/O

You might also like