Appendix: Troubleshooting FCoE

Appendix
A. Call Flow Diagram
B. Troubleshooting Fibre Channel problems
C. Additional Spanning Tree Slides
D. Troubleshooting NEX-5020 Hardware
E. Troubleshooting NEX-5020 Upgrades
F. NEX-5020 Trace Tools

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 1

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 1

Appendix: Troubleshooting FCoE

Appendix A

Call Flow Diagram

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 2

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 2

 Appendix: Troubleshooting FCoE

Connectrix NEX-5020 Call Flow

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 3

This call flow diagram comes from the Global Services Product Support Bulletin NEX-5020
Fibre Channel over Ethernet Switch.
Customer Service Rules of Engagement…
In all customer support cases, EMC will take the first call and transfer the call to EMC Level 1
support. EMC will maintain primary control of the customer support relationship. EMC and
Cisco will work together to ensure customer issues are resolved as quickly as possible. It is
agreed that at times that will mean resolving the call with EMC resources, and at other times
that will result in rapid escalation to Cisco. The decision to rapidly escalate to Cisco will be
based on criteria such as customer requests, Service Level Agreements, the nature of the error
condition and call severity.
In all customer support cases, if EMC determines that the problem can not be resolved using
EMC resources, or if, at any time, the customer requests to be transferred to Cisco for support,
EMC will engage Cisco and work jointly through product resolution.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 3

Appendix: Troubleshooting FCoE

Appendix B

Troubleshooting Fibre Channel problems

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 4

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 4

 Appendix: Troubleshooting FCoE

Identifying the Optics – Fibre Channel

N5K# sh interface fc 3/1 transceiver details

fc3/1 sfp is present
name is CISCO-FINISAR
part number is FTLF8524P2BNL-C2
revision is 0000
serial number is FNS0928K161
fc-transmitter type is short wave laser w/o OFC (SN)
fc-transmitter supports intermediate distance link length
media type is multi-mode, 62.5m (M6)
Supported speed is 400 MBytes/sec
Nominal bit rate is 4300 MBits/sec
Link length supported for 50/125mm fiber is 150 m(s)
Link length supported for 62.5/125mm fiber is 70 m(s)
cisco extended id is unknown (0x0)

no tx fault, no rx loss, in sync state, Diag mon type 104

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 5

E Port - In expansion port (E port) mode, an interface functions as a fabric expansion port. This
port may be connected to another E port to create an Inter-Switch Link (ISL) between two
switches.
TE Port - In trunking E port (TE port) mode, an interface functions as a trunking expansion
port. It may be connected to another TE port to create an Enhanced ISL (EISL) between two
switches. TE ports are specific to Cisco MDS 9000 and Nexus 5020 Family switches
F Port - In fabric port (F port) mode, an interface functions as a fabric port. This port may be
connected to a peripheral device (host or disk) operating as an N port. An F port can be attached
to only one N port. F ports support class 2 and class 3 service.
Auto Mode - Interfaces configured in auto mode can operate in one of the following modes: F
port, E port, or TE port. The port mode is determined during interface initialization.
SD Port- In SPAN destination port (SD port) mode, an interface functions as a switched port
analyzer (SPAN).
NP Port - An NP port is a port on a device that is in NPV mode and connected to the core NPV
switch through an F port. NP ports operate like N ports that function as proxies for multiple
physical N ports.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 5

Appendix: Troubleshooting FCoE

Identifying the Optics – Fibre Channel (continued)

N5K# sh interface fc 3/1 transceiver details

SFP Detail Diagnostics Information

----------------------------------------------------------------------------
Alarms Warnings
High Low High Low
----------------------------------------------------------------------------
Temperature 41.50 C 95.00 C -25.00 C 90.00 C -20.00 C
Voltage 3.45 V 3.90 V 2.70 V 3.70 V 2.90 V
Current 7.18 mA 17.00 mA 1.00 mA 14.00 mA 2.00 mA
Tx Power -4.41 dBm -2.00 dBm -11.74 dBm -2.00 dBm -11.02 dBm
Rx Power -4.40 dBm 1.00 dBm -20.00 dBm -1.00 dBm -18.24 dBm
Transmit Fault Count = 0
----------------------------------------------------------------------------
Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 6

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 6

Appendix: Troubleshooting FCoE

Identifying the Optics – Ethernet

N5K# sh interface ethernet 1/19 transceiver details

Ethernet1/19
sfp is present
name is Molex Inc.
part number is 74752-1301
revision is E
serial number is 733010037
nominal bitrate is 0 MBits/sec
Link length supported for 50/125mm fiber is 0 m(s)
Link length supported for 62.5/125mm fiber is 0 m(s)
cisco id is --
cisco extended id number is 4

Invalid calibration

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 7

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 7

 Appendix: Troubleshooting FCoE

Ethernet Interface Counters N5K# sh interface ethernet 1/19 counters

Ethernet1/19
detailed all

64 bit counters:
0. rxHCTotalPkts = 152894790
switch# show interface eth1/21
1. txHCTotalPks = 330867280
2. rxHCUnicastPkts = 0
Ethernet1/21 is up 3. txHCUnicastPkts = 0
Hardware is 10000 Ethernet, address is 000d.ec6d.84dc 4. rxHCMulticastPkts = 152894790
(bia 000d.ec6d.84dc) 5. txHCMulticastPkts = 330867231
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec, 6. rxHCBroadcastPkts = 0
7. txHCBroadcastPkts = 49
reliability 255/255, txload 1/255, rxload 1/255
8. rxHCOctets = 9785619336
Encapsulation ARPA 9. txHCOctets = 21176709656
Port mode is access 10. rxTxHCPkts64Octets = 483725278
full-duplex, 10000 Mb/s 11. rxTxHCpkts65to127Octets = 29422
12. rxTxHCpkts128to255Octets = 7358
Input flow-control is off, output flow-control is off 13. rxTxHCpkts256to511Octets = 12
5 minute input rate 22203 bytes/sec, 346 packets/sec 14. rxTxHCpkts512to1023Octets = 0
5 minute output rate 640597 bytes/sec, 10000 15. rxTxHCpkts1024to1518Octets = 0
packets/sec 16. rxTxHCpkts1519to1548Octets = 0
Rx 17. rxHCTrunkFrames = 0
18. txHCTrunkFrames = 0
16501327 Input Packets 9 Unicast Packets 16500923 19. rxHCDropEvents = 0
Multicast Packets
395 Broadcast Packets 0 Jumbo Packets 0 Storm All Port Counters:
Suppression Packets 0. InPackets = 152894790
1056159080 Bytes 1. InOctets = 9785619336
0 No buffer 0 runt 0 crc 0 ecc 2. InUcastPkts = 0
3. InMcastPkts = 152894790
0 Overrun 0 Underrun 0 Ignored 0 Bad etype drop
4. InBcastPkts = 0
0 Bad proto drop 0 If down drop 0 Collision 5. InJumboPkts = 0
0 Late collision 0 Lost carrier 0 No carrier 6. StormSuppressPkts = 0
0 Babble 7. OutPackets = 330867280
8. OutOctets = 21176709656
Tx 9. OutUcastPkts = 0
433943286 Output Packets 26171 Multicast Packets 10. OutMcastPkts = 330867231
0 Broadcast Packets 0 Jumbo Packets 11. OutBcastPkts = 49
12. OutJumboPkts = 0
27772499094 Bytes 13. rxHCPkts64Octets = 152880091
0 Ouput errors 14. rxHCPkts65to127Octets = 14699
16499333 Rx pause 0 Tx pause 0 Reset 15. rxHCPkts128to255Octets = 0
16. rxHCPkts256to511Octets = 0
[snip]

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 8

Virtual interface is a concept in Forwarding (LU and AC). MACs and Packet Editing only
understand physical interface.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 8

8
Appendix: Troubleshooting FCoE

Troubleshooting 10Gbe MAC

switch# show hardware internal gatos port ethernet 1/1 | begin MAC
MAC statistics:
MAC Statistic | Value
-------------------------------+-----------------------------------
TX_PKT_SIZE_LT_64 | 0
TX_PKT_SIZE_IS_64 | 0xd33
TX_PKT_SIZE_IS_65_TO_127 | 0x1b78
TX_PKT_SIZE_IS_128_TO_255 | 0x9c
TX_PKT_SIZE_IS_256_TO_511 | 0xe
TX_PKT_SIZE_IS_512_TO_1023 | 0
TX_PKT_SIZE_IS_1024_TO_1518 | 0
TX_PKT_SIZE_IS_1519_TO_2047 | 0
TX_PKT_SIZE_IS_2048_TO_4095 | 0
TX_PKT_SIZE_IS_4095_TO_8191 | 0
TX_PKT_SIZE_IS_8192_TO_9216 | 0
TX_PKT_SIZE_GT_9216 | 0
TX_PKT_TOTAL | 0x2955
TX_PKT_OCTETS | 0xb6f87
TX_PKT_GOOD | 0x2955
TX_PKT_UCAST | 0
TX_PKT_MCAST | 0x28c9
TX_PKT_BCAST | 0x8c
TX_PKT_VLAN | 0xd1f
TX_PKT_802.3x_PAUSE | 0
TX_PKT_PER_PRIORITY_PAUSE | 0

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 9

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 9

Appendix: Troubleshooting FCoE

Troubleshooting FC MAC

N5K# show hardware internal fc-mac 3 port 1 statistics

ADDRESS STAT COUNT
__________ ________ __________________
0x0000003c FCP_CNTR_MAC_RX_LOSS_OF_SYNC 0x8
0x0000003d FCP_CNTR_MAC_RX_BAD_WORDS_FROM_DECODER 0xc0
0x00000042 FCP_CNTR_MAC_CREDIT_IG_XG_MUX_SEND_RRDY_REQ 0x752c
0x00000043 FCP_CNTR_MAC_CREDIT_EG_DEC_RRDY 0x752e
0x00000061 FCP_CNTR_MAC_DATA_RX_CLASS3_FRAMES 0x1c
0x00000062 FCP_CNTR_MAC_DATA_RX_CLASSF_FRAMES 0x7510
0x00000069 FCP_CNTR_MAC_DATA_RX_CLASS3_WORDS 0x850
0x0000006a FCP_CNTR_MAC_DATA_RX_CLASSF_WORDS 0x221090
0x00000065 FCP_CNTR_MAC_DATA_TX_CLASS3_FRAMES 0x1c
0x00000066 FCP_CNTR_MAC_DATA_TX_CLASSF_FRAMES 0x7512
0x0000006d FCP_CNTR_MAC_DATA_TX_CLASS3_WORDS 0x8a4
0x0000006e FCP_CNTR_MAC_DATA_TX_CLASSF_WORDS 0x18e9d4
0xffffffff FCP_CNTR_LINK_RESET_IN 0x1
0xffffffff FCP_CNTR_OLS_IN 0x1
0xffffffff FCP_CNTR_NOS_IN 0x1
0xffffffff FCP_CNTR_LRR_IN 0x1
0xffffffff FCP_CNTR_OLS_OUT 0x1
0xffffffff FCP_CNTR_LRR_OUT

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 10

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 10

Appendix: Troubleshooting FCoE

Troubleshooting FC MAC

N5K# show hardware internal fc-mac 3 port 1 interrupt-counts

INTERRUPT COUNT THRESH
_________ ________ ______
IP_FCMAC_INTR_PRIM_RX_SEQ_NOS 2 0
IP_FCMAC_INTR_PRIM_RX_SEQ_OLS 2 0
IP_FCMAC_INTR_PRIM_RX_SEQ_LR 4 0
IP_FCMAC_INTR_PRIM_RX_SEQ_LRR 2 0
IP_FCMAC_INTR_PRIM_RX_SIG_IDLE 2 0
IP_FCMAC_INTR_CMD_TX_RUN_QUEUE_DONE 7 0

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 11

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 11

Appendix: Troubleshooting FCoE

Identifying FC Port State

N5K# sh hardware internal fc-mac 3 port 1 link-event
MMDDYY HHMMSS usecs Event Current State
-------------------- ----- -------------
- - LINK_ACTIVE
081108 203640 716832 (0001) E_LINK_IDLE LINK_LR_RX
081108 203640 713733 (004B) E_LINK_LR LINK_ACTIVE
081108 203639 954150 (0001) E_LINK_IDLE LINK_LRR_RX
081108 203639 951715 (0000) E_LINK_LRR LINK_LR_RX
081108 203639 949686 (0001) E_LINK_LR LINK_OLS_RX
081108 203639 943264 (0000) E_LINK_OLS LINK_NOS_RX
081108 203639 934058 (0000) E_LINK_NOS LINK_OLS_TX
081108 203639 933758 (0001) E_LINK_MIN_OLS LINK_INIT
081108 203639 923797 (0000) E_LINK_LINK_INIT LINK_DIS

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 12

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 12

 Appendix: Troubleshooting FCoE

HBAs with unique Area requirements

Some HBA vendors require that the target they are logging into be in a different area, for example: Qlogic

N5K# show fcid-allocation ?

area Show information about fcid-allocation list
company-id-from-wwn Company id (or OUI).

N5K# show fcid-allocation area

Fcid area allocation company id info:

00: 2:6B
00: 6:2B
00: 9:6B
00:11:25
00:14:5E
00:50:2E
00:50:8B
00:60:B0
00:A0:B8
00:D0:60
00:D0:B2
00:E0:69
00:E0:8B

Total company ids: 13

+ - Additional user configured company ids.
* - Explicitly deleted company ids from default list.

N5K(config)# fcid-allocation area company-id ?

<0x0-0xffffff> Enter company id(or OUI) (0xhhhhhh)

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 13

NEX-5020 use of Area and Port bits:

The NEX-5020 platform conservatively assigns area ID's.

Every VSAN starts with the area ID of 00 and we increment the Port ID's, based on order of
login, until we run out at which time we increase the Area ID from 00 to 01 and start the process
over.

We also maintain a database mapping of HBA's and FCID so we try to reassign the same FCID
to an HBA even if someone plugs the HBA into another port on the same switch in the same
VSAN.

This database is maintained until some many HBA's have logged in the system needs to reclaim
the FCID or the switch is rebooted. If you assign a persistent FCID then the HBA will get the
same FCID no matter what.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 13

Appendix: Troubleshooting FCoE

World Wide Names

 The NEX-5020 has multiple WWNs. There is a switch WWN:
N5K# sh wwn switch
Switch WWN is 20:00:00:0d:ec:6b:cd:40

 There is also a SWWN per VSAN:

N5K# show fcdomain
VSAN 1
The local switch is a Subordinated Switch.
Local switch run time information:
State: Stable
Local switch WWN: 20:01:00:0d:ec:6b:cd:41
Running fabric name: 20:01:00:0d:ec:3f:a5:c1
[snip]
VSAN 5
The local switch is a Subordinated Switch.
Local switch run time information:
State: Stable
Local switch WWN: 20:05:00:0d:ec:6b:cd:41
Running fabric name: 20:05:00:0d:ec:3f:a5:c1

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 14

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 14

Appendix: Troubleshooting FCoE

World Wide Names

 The VSAN WWN is derived as follows:
– Byte 1 = 20 (second nibble used for VSAN if required)
– Byte 2 = VSAN # in hex (using a nibble from byte 1 if required)
– Bytes 3-5 = Cisco OUI

 Bytes 6-8 = derived from MAC in backplane SPROM

– For example:
– VSAN 4093 = Local switch WWN: 2f:fd:00:05:30:01:97:43
– FFD = 1111 1111 1101
= 2048 + 1024 + 512 + 256 + 128 + 64 + 32 + 16 + 8 + 4 + 1 = 4093

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 15

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 15

 Appendix: Troubleshooting FCoE

FLOGI – Fabric Login

Fibre Channel FC ELS
R_CTL: 0x22 Cmd Code: FLOGI (0x04)
Notice Common Svc Parameters
Dest Addr: ff.ff.fe
SID B2B Credit: 4
CS_CTL: 0x00 Common Svc Parameters: 0x0 (Alt B2B Credit Mgmt)
Src Addr: 00.00.00 0000 .... = BB_SC Number: 0
.... 1000 0000 0000 = Receive Size: 2048
Type: Ext Link Svc (0x01)
Max Concurrent Seq: 18
F_CTL: 0x290000 (Exchange Originator, Relative Offset By Info Cat: 20352
Seq Initiator, Exchg First, Seq Last, E_D_TOV: 0
N_Port Port_Name: 10:00:00:01:73:00:94:84 (00:01:73)
Transfer Seq Initiative,
Fabric/Node Name: 10:00:00:01:73:00:94:84 (00:01:73)
Last Data Frame - No Info, ABTS - Abort/MS, ) Class 1 Svc Parameters
SEQ_ID: 0x00 F_CTL Service Options: 0x0(Class Not Supported)

DF_CTL: 0x00 bits Class 2 Svc Parameters

Service Options: 0x0(Class Not Supported)
SEQ_CNT: 0 Class 3 Svc Parameters
OX_ID: 0x008e Service Options: 0x8800(Seq Delivery Requested)
Initiator Control: 0x0(Seq Delivery Requested)
RX_ID: 0xffff
Recipient Control: 0x0(Seq Delivery Requested)
Parameter: 0x00000000 Class Recv Size: 0
Total Concurrent Seq: 0
End2End Credit: 0
Open Seq Per Exchg: 0
Class 4 Svc Parameters
Service Options: 0x0(Class Not Supported)
Class 3 Vendor Version: 00000000000000000000000000000000
only
© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 16

Fabric Login Server

The destination address for fabric login (FLOGI) requests
 Located at address 0x’FF FF FE’
Can also be used for the Discover Fabric Service Parameters (FDISC), Read Link Error Status
Block (LESB), and Read Timeout Values (RTV) commands.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 16

 Appendix: Troubleshooting FCoE

FLOGI - ACC FC ELS

Cmd Code: ACC (0x02)
Common Svc Parameters
B2B Credit: 12
Fibre Channel Common Svc Parameters: 0x1300 (Alt B2B Credit Mgmt)
0000 .... = BB_SC Number: 0
R_CTL: 0x23
Notice .... 1000 0100 0000 = Receive Size: 2112
Dest Addr: 7e.00.00 Max Concurrent Seq: 0
DID Relative Offset By Info Cat: 10000
CS_CTL: 0x00 E_D_TOV: 2000
Src Addr: ff.ff.fe N_Port Port_Name: 20:41:00:05:30:00:51:1e (00:05:30)
Fabric/Node Name: 20:01:00:05:30:00:51:1f (00:05:30)
Type: Ext Link Svc (0x01) Class 1 Svc Parameters
F_CTL: 0x980000 (Exchange Responder, Seq Service Options: 0x0(Class Not Supported)
Initiator, Exchg Last, Seq Last, CS_CTL, Last Class 2 Svc Parameters
Data Frame - No Info, ABTS - Abort/MS, ) Service Options: 0x8800(Seq Delivery Requested)
Initiator Control: 0x0(Seq Delivery Requested)
SEQ_ID: 0x00 Recipient Control: 0x0(Seq Delivery Requested)
DF_CTL: 0x00 Class Recv Size: 0
Total Concurrent Seq: 0
SEQ_CNT: 0 F_CTL End2End Credit: 0
OX_ID: 0x008e bits Open Seq Per Exchg: 0
Class 3 Svc Parameters
RX_ID: 0x3d32 Service Options: 0x8800(Seq Delivery Requested)
Parameter: 0x00000000 Initiator Control: 0x0(Seq Delivery Requested)
Recipient Control: 0x0(Seq Delivery Requested)
Class Recv Size: 0
Total Concurrent Seq: 0
Class End2End Credit: 0
2&3 Open Seq Per Exchg: 0
Class 4 Svc Parameters
Service Options: 0x0(Class Not Supported)
Vendor Version: 00000000000000000000000000000000

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 17

FLOGI Accept

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 17

Appendix: Troubleshooting FCoE

Sample NEX-5020 FLOGI Action Debug

N5K# debug flogi action interface vfc23/1
Interface is “Shut” here
2008 Aug 12 17:26:41.706850 flogi: fs_add_del_flogi_in_vsan_fcid_hash: Fcid 0xea0000 delete from vsan 5
.. Successful. Has 0 more fcids
2008 Aug 12 17:26:41.706954 flogi: fs_should_free_fcid: if[0x1a000b00] vsan[0x5] fcid[ea00] fcidrcvd[1]
2008 Aug 12 17:26:41.706977 flogi: fs_should_free_fcid: if[0x1a000b00] vsan[0x5] num_alloc[1]
2008 Aug 12 17:26:41.706998 flogi: fs_should_free_fcid: if[0x1a000b00] vsan[0x5] num_alloc[0] decrement
2008 Aug 12 17:26:41.707025 flogi: FCID free req to DM mask: SINGLE.
2008 Aug 12 17:26:41.707249 flogi: FCID free to DM for flogi(0x812d79c)
2008 Aug 12 17:26:41.707285 flogi: fs_set_nx_port_login_status: Fcid 0xea0000 logged out of vsan 5.
2008 Aug 12 17:26:41.707396 flogi: All ports logged out of port 0x1a000b00.
2008 Aug 12 17:26:41.707444 flogi: fs_update_port_vsan_flogi_info: Update vsan[0x5] flogi info(1)
2008 Aug 12 17:26:41.708436 flogi: Cleared fcid 0x090000 info in fc2.
2008 Aug 12 17:26:41.708471 flogi: flogi_ac_group_cleanup_flogi_done: if[0x5] vsan[0x1a000b00]
2008 Aug 12 17:26:41.708530 flogi: Group cleanup for 1 flogis
2008 Aug 12 17:26:41.708554 flogi: Entry count 0 with ifindex 0x1a000b00(0x1a000b00) and vsan 5(5).
2008 Aug 12 17:26:41.708590 flogi: Bulk cleanup for 1 entries in vsan 5 ifindex vfc23/1
2008 Aug 12 17:26:41.708704 flogi: fs_update_fcredirect: fcredirect conf not needed if[1a000b00]
2008 Aug 12 17:26:41.708859 flogi: fs_flogi_send_sync_flogi_message: if[0x1a000b00] vsan[0x5]
fc2_req[1] fcid[0]
2008 Aug 12 17:26:41.726197 flogi: Dropping MTS wrap MTS_OPC_PM_CLEANUP_PORT_STATE(333) id 26029498
2008 Aug 12 17:26:41.726295 flogi: Last ref of MTS_OPC_PM_CLEANUP_PORT_STATE(333) id 26029498

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 18

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 18

Appendix: Troubleshooting FCoE

Sample NEX-5020 FLOGI Action Debug continued

switch# debug flogi action int fc1/1
Interface is “no shut” here:
2008 Aug 12 17:29:25.709915 flogi: fs_set_flogi_for_wwn_port: table not pres if[1a000b00] vsan[5]
2008 Aug 12 17:29:25.709966 flogi: fs_set_flogi_for_wwn_port: table not pres if[1a000b00] vsan[5]
2008 Aug 12 17:29:25.710000 flogi: fs_set_port_vsan_ht: if[1a000b00] vsan[5] type[0] key[3F 0 0 0 0 0 0 0 5F1F0 01A
0 B 0]
2008 Aug 12 17:29:25.710446 flogi: Flogi allowed for port. FLOGI_EV_VALID_FLOGI
2008 Aug 12 17:29:25.710480 flogi: flogi_ac_copy_flogi_and_generate_event: new_event[0xf] if[0x1a000b00] vsan[0x5]
2008 Aug 12 17:29:25.711159 flogi: (Error) Port-Security service not running sending port binding request
2008 Aug 12 17:29:25.711374 flogi: setting first_flogi_timer_started to TRUE
2008 Aug 12 17:29:25.711400 flogi: flogi_ac_pre_config_check_seq_start: timer_rcode[0]
2008 Aug 12 17:29:25.711516 flogi: Notify PM of first flogi.
2008 Aug 12 17:29:25.711570 flogi: FCID 0x000000 allocate req to DM mask: SINGLE.
2008 Aug 12 17:29:25.712020 flogi: fs_flogi_ac_request_fcid: FCID req to DM for flogi(0x812fb54)
2008 Aug 12 17:29:25.712420 flogi: fs_assign_fcid_for_flogi: if[0x1a000b00] vsan[0x5] num_alloc[1] increment
2008 Aug 12 17:29:25.712624 flogi: fs_flogi_send_sync_flogi_message: if[0x1a000b00] vsan[0x5] fc2_req[1] fcid[0]
2008 Aug 12 17:29:25.712716 flogi: fs_add_del_flogi_in_vsan_fcid_hash: Fcid 0xea0000 add from vsan 5 .. Successful.
Has 1 more fcids
2008 Aug 12 17:29:25.713110 flogi: TLV to LCP sent successfully.
2008 Aug 12 17:29:25.713480 flogi: fs_flogi_perform_ns_register: sym_name[] if[1a000b00] ns_flags[8a0f] class[c]
2008 Aug 12 17:29:25.713538 flogi: Init RIB entry successful.
2008 Aug 12 17:29:25.715592 flogi: fs_update_fcredirect: fcredirect conf not needed if[1a000b00]
2008 Aug 12 17:29:25.728156 flogi: Added fcid 0xea0000 info in fc2.
2008 Aug 12 17:29:25.728302 flogi: flogi_ac_flogi_done: timer - rid(0x40000000), event_id(82), timer_id(1),
2008 Aug 12 17:29:25.728336 flogi: Sending flogi acc for 0xea0000
2008 Aug 12 17:29:25.728366 flogi: FC2 flogi resp for vsan 5 ifindex 0x1a000b00 for exchange 0x32a1e66 fcid
0xea0000.
2008 Aug 12 17:29:25.728388 flogi: Sending fc2 accept.
[snip]

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 19

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 19

Appendix: Troubleshooting FCoE

NEX-5020 FLOGI Debugs - Summary

N5K# debug flogi ?
action Configure debugging of flogi actions
all Configure all debug flags of flogii
demux Configure debugging of flogi message demux
error Configure debugging of flogi error
event Configure debugging of flogi FSM and Events
ha Configure debugging of flogi HA
init Configure debugging of flogi adds, deletes and inits
timers Configure debugging of flogi message timers
trace Configure debugging of flogi trace
warning Configure debugging of flogi warning

 Not enough to just run ethanalyzer

 Need to run debugs as well to verify FLOGI-ACC exchange

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 20

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 20

Appendix: Troubleshooting FCoE

Viewing Fabric Login Service Parameters on NEX-5020

FLOGI Service
Parameters

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 21

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 21

Appendix: Troubleshooting FCoE

RFT_ID – Register FC-4 Types by Port ID

Fibre Channel Fibre Channel
R_CTL: 0x02 R_CTL: 0x03
Dest Addr: ff.ff.fc Dest Addr: 7e.00.00
CS_CTL: 0x00 CS_CTL: 0x00
Src Addr: 7e.00.00 Src Addr: ff.ff.fc

[snip] [snip]

FC_CT FC_CT
Revision: 1 Revision: 1
IN_ID: 00.00.00 IN_ID: 00.00.00
GS Type: Directory Service (0xfc) GS Type: Directory Service (0xfc)
GS Subtype: 0x02 GS Subtype: 0x02
Server: dNS (0x01) Server: dNS (0x01)
Options: 0x00 Options: 0x00
dNS dNS
Opcode: RFT_ID (0x0217) Opcode: MSG_ACC (0x8002)
Maximum/Residual Size: 4 Maximum/Residual Size: 0
Port Identifier: 7e.00.00
FC-4 TYPEs Supported: FCP,

Request Response

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 22

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 22

 Appendix: Troubleshooting FCoE

GID_FT – Get Port ID by FC-4 Types

Fibre Channel Fibre Channel
R_CTL: 0x02 R_CTL: 0x03
Dest Addr: ff.ff.fc
Dest Addr: 7e.00.00
CS_CTL: 0x00
CS_CTL: 0x00
Src Addr: 7e.00.00
Src Addr: ff.ff.fc
Type: FC_CT (0x20)
Type: FC_CT (0x20)
[snip]
[snip]
FC_CT
Revision: 1 FC_CT
IN_ID: 00.00.00 Revision: 1
GS Type: Directory Service (0xfc) IN_ID: 00.00.00
GS Subtype: 0x02 GS Type: Directory Service (0xfc)
Server: dNS (0x01) GS Subtype: 0x02
Options: 0x00
Server: dNS (0x01)
dNS
Options: 0x00
Opcode: GID_FT (0x0171)
dNS
Maximum/Residual Size: 256
Opcode: MSG_ACC (0x8002)
Domain ID Scope: 0x00
Area ID Scope: 0x00 Maximum/Residual Size: 0
FC-4 Type: FCP (0x08) Port Identifier: 7e.00.00
Request Response

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 23

Directory Server Query Commands:

These are some of the more commonly used commands used by Fibre Channel devices to query
the Directory Server on a switch
Get objects
 GFT_ID – Get FC-4 Types by Port ID
Register Objects
 RFT_ID - Register FC-4 Types

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 23

Appendix: Troubleshooting FCoE

Sample MDS FCNS Event Query Debug

N5K# debug fcns events query vsan 5

N5K# 2008 Aug 12 17:41:50.741393 fcns: vsan 5: fc_ct_parse_frame() succeeded - request_id = 26, flags = 0
2008 Aug 12 17:41:50.741445 fcns: vsan 5: received sw query NS_CMD_GE_ID from switch 0xfffca4 to switch
0xfffcea
2008 Aug 12 17:41:50.741472 fcns: vsan 5: Got entry for port-id 0xea0000 matching the query
2008 Aug 12 17:41:50.741500 fcns: vsan 5: sending accept response to port-id 0xfffca4
2008 Aug 12 17:41:51.038089 fcns: vsan 5: fc_ct_parse_frame() succeeded - request_id = 27, flags = 0
2008 Aug 12 17:41:51.038139 fcns: vsan 5: received sw query NS_CMD_GE_ID from switch 0xfffca4 to switch
0xfffcea
2008 Aug 12 17:41:51.038168 fcns: vsan 5: Got entry for port-id 0xea0000 matching the query
2008 Aug 12 17:41:51.038196 fcns: vsan 5: sending accept response to port-id 0xfffca4
2008 Aug 12 17:41:52.720708 fcns: vsan 5: fc_ct_parse_frame() succeeded - request_id = 28, flags = 0
2008 Aug 12 17:41:52.721216 fcns: vsan 5: fc_ct_parse_frame() succeeded - request_id = 29, flags = 0
2008 Aug 12 17:41:52.721421 fcns: vsan 5: fc_ct_parse_frame() succeeded - request_id = 30, flags = 0
2008 Aug 12 17:41:52.721846 fcns: vsan 5: fc_ct_parse_frame() succeeded - request_id = 31, flags = 0
2008 Aug 12 17:41:52.722478 fcns: vsan 5: fc_ct_parse_frame() succeeded - request_id = 32, flags = 0
2008 Aug 12 17:41:52.722512 fcns: vsan 5: received query NS_CMD_GID_FT
2008 Aug 12 17:41:52.722547 fcns: vsan 5: Applying zoning constraints on this query; limiting search space
to the zone(s) containing port-id 0xea0000
2008 Aug 12 17:41:52.722723 fcns: vsan 5: Got entry matching the object in the query; entry belongs to
port-id 0xa401cb
2008 Aug 12 17:41:52.722752 fcns: vsan 5: port-id 0xa401cb is not in the same zone; discarding this entry
2008 Aug 12 17:41:52.722775 fcns: vsan 5: Got entry matching the object in the query; entry belongs to
port-id 0xa401cc
2008 Aug 12 17:41:52.722797 fcns: vsan 5: port-id 0xa401cc is not in the same zone; discarding this entry

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 24

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 24

Appendix: Troubleshooting FCoE

Sample MDS FCNS Event Query Debug continued

2008 Aug 12 17:41:52.722819 fcns: vsan 5: Got entry matching the object in the query; entry belongs to
port-id 0xa401cd
2008 Aug 12 17:41:52.722841 fcns: vsan 5: port-id 0xa401cd is not in the same zone; discarding this
entry
2008 Aug 12 17:41:52.722862 fcns: vsan 5: Got entry matching the object in the query; entry belongs to
port-id 0xa401d2
2008 Aug 12 17:41:52.722884 fcns: vsan 5: port-id 0xa401d2 is not in the same zone; discarding this
entry
2008 Aug 12 17:41:52.722906 fcns: vsan 5: Got entry matching the object in the query; entry belongs to
port-id 0xa401d3
2008 Aug 12 17:41:52.722933 fcns: vsan 5: port-id 0xa401d3 is not in the same zone; discarding this
entry
2008 Aug 12 17:41:52.722955 fcns: vsan 5: Got entry matching the object in the query; entry belongs to
port-id 0xa401d4
2008 Aug 12 17:41:52.722977 fcns: vsan 5: port-id 0xa401d4 is not in the same zone; discarding this
entry
2008 Aug 12 17:41:52.722999 fcns: vsan 5: Got entry matching the object in the query; entry belongs to
port-id 0xa401d5
2008 Aug 12 17:41:52.723021 fcns: vsan 5: port-id 0xa401d5 is not in the same zone; discarding this
entry
2008 Aug 12 17:41:52.723043 fcns: vsan 5: Got entry matching the object in the query; entry belongs to
port-id 0xea0000
2008 Aug 12 17:41:52.723065 fcns: vsan 5: port-id 0xea0000 is in the same zone
2008 Aug 12 17:41:52.723088 fcns: vsan 5: sending accept response to port-id 0xea0000
2008 Aug 12 17:41:53.018175 fcns: vsan 5: fc_ct_parse_frame() succeeded - request_id = 33, flags = 0
2008 Aug 12 17:41:53.018228 fcns: vsan 5: received sw query NS_CMD_GE_ID from switch 0xfffca4 to switch
0xfffcea
2008 Aug 12 17:41:53.018266 fcns: vsan 5: Got entry for port-id 0xea0000 matching the query
2008 Aug 12 17:41:53.018294 fcns: vsan 5: sending accept response to port-id 0xfffca

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 25

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 25

Appendix: Troubleshooting FCoE

NEX-5020 FCNS Debugs - Summary

N5K# debug fcns ?

all Debug name server
errors Debug name server errors
events Debug name server events

N5K2# debug fcns events ?

mts Debug name server mts events
query Debug name server query events
register Debug name server registration events

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 26

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 26

Appendix: Troubleshooting FCoE

Fabric Controller
 Each Switch has a Fabric Controller
 Assigned address ‘FF FF FD’
– Every Fabric Controller in the fabric has the same address
– Responsible for managing fabric addressing, initialization, routing...
– Node ports can register with Fabric Controller to receive port or
fabric state changes
– A required service

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 27

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 27

Appendix: Troubleshooting FCoE

ELS - SCR
SCR – State Change Registration
– Used when ports want to receive notification when the login state of other ports
in the fabric changes.
– SCR sent to Fabric Controller
– Can register for fabric events, port events, or both (Full)
– Login state may change due to:
 Loss of signal (NOS, LOS, FLOGI)
 LIP has occurred
 New port logins

N5K# sh rscn scr-table

SCR table for VSAN: 5
---------------------------------------------
FC-ID REGISTERED FOR
---------------------------------------------
0xea0000 fabric and nport detected rscns

Total number of entries = 1

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 28

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 28

Appendix: Troubleshooting FCoE

SCR – State Change Registration

Fibre Channel
R_CTL: 0x22 Fibre Channel
Dest Addr: ff.ff.fd R_CTL: 0x23
CS_CTL: 0x00 Dest Addr: 7e.00.00
Src Addr: 7e.00.00 CS_CTL: 0x00
Type: Ext Link Svc (0x01) Src Addr: ff.ff.fd
F_CTL: 0x290000 (Exchange Originator, Seq Type: Ext Link Svc (0x01)
Initiator, Exchg First, Seq Last, CS_CTL, F_CTL: 0x980000 (Exchange Responder, Seq
Transfer Seq Initiative, Last Data Frame - No Initiator, Exchg Last, Seq Last, CS_CTL, Last
Info, ABTS - Abort/MS, ) Data Frame - No Info, ABTS - Abort/MS, )
SEQ_ID: 0x69 SEQ_ID: 0x00
DF_CTL: 0x00 DF_CTL: 0x00
SEQ_CNT: 0 SEQ_CNT: 0
OX_ID: 0x0034 OX_ID: 0x0034
RX_ID: 0xffff RX_ID: 0x0073
Parameter: 0x00000000 Parameter: 0x00000000
FC ELS FC ELS
Cmd Code: SCR (0x62) Cmd Code: ACC (0x02)
Registration Function: Full Regn (0x03)

Response
Request

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 29

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 29

Appendix: Troubleshooting FCoE

RSCN – Registered State Change Notification

 Indicates a new device is available, or an existing device
is no longer available
 Device receiving RSCN can initiate discovery of the new
device, or verify status of an existing device. (Usually
results in a PLOGI and subsequent queries of the Name
Server)

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 30

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 30

Appendix: Troubleshooting FCoE

NEX-5020 specific RSCN information

 There is no priority difference for SW-ILS RSCNs -vs.- local ELS RSCNs.
Depending on the order in which they are generated the frames will be sent out. If
both RSCNs and SW-RSCNs get generated at the same time from the NEX-5020
switch then they will be sent out in the same order they have been generated.
 The NEX-5020 will intentionally delay SENDING some types of RSCNs in order to
allow for coalescing of data. For example, if a device disappears then we'll send the
RSCN indicating this immediately. In the case of a device appearing (FLOGI), we
delay sending the RSCN associated with that given its likely that it'll register
something additional about itself in the name server, and if we delay sending that
RSCN, it means that we can typically just send one RSCN.
 There is also a "rscn multi-pid" option. This allows the packaging of multiple
fcids/portids into a single RSCN. This may be useful in multi-initiator zones. For
example, if there is a zone of 50 devices, and 10 disappear, instead of sending 400
RSCNs, 40 will be sent which will include the fc.ids of the 10 devices that have
gone away.
– N5K(config)# rscn multi-pid vsan 10

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 31

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 31

Appendix: Troubleshooting FCoE

Fabric Configuration Stages

Operation Starting Process Ending

Condition Condition
Establish Link Switch Port has The Switch Port attempts to Switch Port mode is known. If
Parameters and achieved word discover whether it is an FL, F, or a Port is an E port, link
Switch Port synchronization E port. (ELP, ESC, & EPP) parameters have been
Operating Mode exchanged & Credit has been
initialized.
Select Principle BF or RCF SW_ILS Switch_Names are exchanged The Principle Switch is
Switch transmitted or received over all ISLs to select a Principle selected
Switch, which becomes the
Domain Address Manager (EFP)
Domain ID Domain Address Switch requests a Domain_ID Switch has a Domain_ID
Acquisition Manager had been from the Domain Address
selected Manager (DIA & RDI)

Path Selection Switch has a Path selection (FSPF) is defined Switch is operational with
Domain_ID in the next section (HLO, LSU and routes established
LSA)

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 32

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 32

Appendix: Troubleshooting FCoE

ELP – Exchange Link Parameters

 A point to point frame sent from the fabric controller of one switch
(FF.FF.FD) to the fabric controller of the other switch.
 This frame is used to setup and govern the communication
between the two ports. Supported classes of service, fibre
channel timers, buffer to buffer credits, and several other things
are exchanged using the ELP frame and the subsequent SW_ACC
(switch accept) to the ELP.
 This is also where isolation can occur due to a mismatch in fibre
channel timers, domain ID conflicts…

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 33

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 33

 Appendix: Troubleshooting FCoE

ELP – Exchange Link Parameters

Fibre Channel Class F Svc Parameters: (Class F Valid | No X_ID
R_CTL: 0x02 Interlk)
Dest Addr: ff.ff.fd Max Class F Frame Size: 2112
CS_CTL: 0x00 Class F Max Concurrent Seq: 1
Src Addr: ff.ff.fd Class F E2E Credit: 1
Type: SW_ILS (0x22) Class F Max Open Seq: 1
F_CTL: 0x290000 (Exchange Originator, Seq
Initiator, Exchg First, Seq Last, CS_CTL,
Class 1 Svc Parameters: (Class 1 Invalid)
Transfer Seq Initiative, ast Data Frame - No Class 2 Svc Parameters: (Class 2 Valid | Seq
Info, ABTS - Abort/MS, ) Delivery)
SEQ_ID: 0x00 Class 2 Frame Size: 2112
DF_CTL: 0x00 Class 3 Svc Parameters: (Class 3 Valid | Seq
SEQ_CNT: 0 Delivery)
OX_ID: 0xc27d Class 3 Frame Size: 2112
RX_ID: 0xffff ISL Flow Ctrl Mode: R_RDY Flow Ctrl
Parameter: 0x00000000 Flow Ctrl Param Len: 20
SW_ILS B2B Credit: 255
Cmd Code: ELP (0x10)
Compatability Param 1: 2112
Revision: 2
Compatability Param 2: 10000
Flag: 0000
Compatability Param 3: 2000
R_A_TOV: 10000 msecs
E_D_TOV: 2000 msecs Compatability Param 4: 0

Req Eport Name: 20:01:00:0c:30:e8:a6:00

Req Switch Name: 20:01:00:0c:30:e8:a6:01

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 34

Need to investigate ELP revisions

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 34

Appendix: Troubleshooting FCoE

SW_ACC to the ELP

Fibre Channel Class F Svc Parameters: (Class F Valid | No X_ID
R_CTL: 0x03 Interlk)
Dest Addr: ff.ff.fd Max Class F Frame Size: 2112
CS_CTL: 0x00
Class F Max Concurrent Seq: 1
Src Addr: ff.ff.fd
Class F E2E Credit: 1
Type: SW_ILS (0x22)
F_CTL: 0x980000 (Exchange Responder, Class F Max Open Seq: 1
Seq
Class 1 Svc Parameters: (Class 1 Invalid)
Initiator, Exchg Last, Seq Last,
CS_CTL, Last Data Frame - No Info, Class 2 Svc Parameters: (Class 2 Valid | Seq
ABTS - Abort/MS, ) Delivery)
SEQ_ID: 0x00
Class 2 Frame Size: 2112
DF_CTL: 0x00
Class 3 Svc Parameters: (Class 3 Valid | Seq
SEQ_CNT: 0
Delivery)
OX_ID: 0xc27d
Class 3 Frame Size: 2112
RX_ID: 0xdd20
Parameter: 0x00000000 ISL Flow Ctrl Mode: R_RDY Flow Ctrl
SW_ILS Flow Ctrl Param Len: 20
Cmd Code: SW_ACC (0x02) B2B Credit: 12
Revision: 2 Compatability Param 1: 2112
Flag: 0000
Compatability Param 2: 10000
R_A_TOV: 10000 msecs
E_D_TOV: 2000 msecs Compatability Param 3: 2000
Req Eport Name: 20:41:00:05:30:00:51:1e Compatability Param 4: 0
Req Switch Name:
20:01:00:05:30:00:51:1f

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 35

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 35

 Appendix: Troubleshooting FCoE

ESC – Exchange Switch Capabilities

Fibre Channel
R_CTL: 0x02
Dest Addr: ff.ff.fd
CS_CTL: 0x00
Src Addr: ff.ff.fd
Type: SW_ILS (0x22)
F_CTL: 0x290000 (Exchange Originator, Seq Initiator, Exchg First, Seq Last, CS_CTL, Transfer
Seq Initiative, Last Data Frame - No Info, ABTS - Abort/MS, )
SEQ_ID: 0x01
DF_CTL: 0x00
SEQ_CNT: 0 NEX-5020
OX_ID: 0xc27e
RX_ID: 0xffff Switch
Parameter: 0x00000000
SW_ILS
Cmd Code: ESC (0x30)
Payload Length: 36
Switch Vendor ID: andiamo Use FSPF
Protocol Descriptor 0
Vendor ID:
Protocol
Protocol ID: FSPF Protocol (0x0002)
Protocol Descriptor 1
Vendor ID: andiamo
Protocol ID: Unknown (0x8000)

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 36

The ESC (Exchange Switch Capabilities) frame allows switches to exchange information about
the routing protocols supported and agree on a common routing protocol.
The vendor ID field is used by the NEX-5020 to determine that the switch on the other end is
another MDS or NEX-5020 switch. This is indication that Trunking E_Ports (TE) and VSANs
are supported.
Only the ELP winner, the port that receives the SW_ACC to it’s ELP, sends the ESC.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 36

Appendix: Troubleshooting FCoE

SW_ACC to the ESC

Fibre Channel
R_CTL: 0x03
Dest Addr: ff.ff.fd
CS_CTL: 0x00
Src Addr: ff.ff.fd
Type: SW_ILS (0x22)
F_CTL: 0x980000 (Exchange Responder, Seq Initiator, Exchg Last, Seq Last,
CS_CTL, Last Data Frame - No Info, ABTS - Abort/MS, )
SEQ_ID: 0x01
DF_CTL: 0x00
SEQ_CNT: 0
NEX-5020
OX_ID: 0xc27e switch
RX_ID: 0xdd21
Parameter: 0x00000000
SW_ILS
Cmd Code: SW_ACC (0x02)
Switch Vendor ID: andiamo
Accepted Protocol Descriptor
Vendor ID: andiamo
Protocol ID: Unknown (0x8000)

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 37

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 37

 Appendix: Troubleshooting FCoE

EPP – Exchange Peer Parameters

ff.ff.fd -> ff.ff.fd SW_ILS 1 0xc27f 0xffff 0xff -> 0x0 0x71
ff.ff.fd -> ff.ff.fd FC 1 0xc27f 0xdd22 0x0 -> 0xf Link Ctl, ACK1
SYNC
ff.ff.fd -> ff.ff.fd SW_ILS 1 0xc27f 0xdd22 0x0 -> 0xf SW_ACC (0x71)
ff.ff.fd -> ff.ff.fd FC 1 0xc27f 0xdd22 0xff -> 0x0 Link Ctl, ACK1
ff.ff.fd -> ff.ff.fd SW_ILS 1 0xdd23 0xffff 0x0 -> 0xf 0x71
ff.ff.fd -> ff.ff.fd FC 1 0xdd23 0xc280 0xff -> 0x0 Link Ctl, ACK1
Commit
ff.ff.fd -> ff.ff.fd SW_ILS 1 0xdd23 0xc280 0xff -> 0x0 SW_ACC (0x71)
ff.ff.fd -> ff.ff.fd FC 1 0xdd23 0xc280 0x0 -> 0xf Link Ctl, ACK1

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 38

Exchange Peer Parameters (EPP) uses command (0x71). This is in the standard as a vendor
unique command code.
We use this frame for trunk negotiation, VSAN list exchange, and port VSAN consistency
check.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 38

 Appendix: Troubleshooting FCoE

Exchange Fabric Parameters

ff.ff.fd -> ff.ff.fd SW_ILS 1 0xdd24 0xffff 0x0 -> 0xf EFP
ff.ff.fd -> ff.ff.fd FC 1 0xdd24 0xc281 0xff -> 0x0 Link Ctl, ACK1
ff.ff.fd -> ff.ff.fd SW_ILS 1 0xc282 0xffff 0xff -> 0x0 EFP
ff.ff.fd -> ff.ff.fd FC 1 0xc282 0xdd25 0x0 -> 0xf Link Ctl, ACK1
ff.ff.fd -> ff.ff.fd SW_ILS 1 0xdd24 0xc281 0xff -> 0x0 SW_ACC (EFP)
ff.ff.fd -> ff.ff.fd SW_ILS 1 0xc282 0xdd25 0x0 -> 0xf SW_ACC (EFP)

Cmd Code: EFP (0x11) Cmd Code: EFP (0x11)

Payload Len: 16 Payload Len: 16
Principal Switch Priority: 128 Principal Switch Priority: 128
Principal Switch Name: Principal Switch Name:
20:01:00:05:30:00:51:1f 20:01:00:0c:30:e8:a6:01

Cmd Code: SW_ACC (0x02) Cmd Code: SW_ACC (0x02)

Payload Len: 16 Payload Len: 16
Principal Switch Priority: 128 Principal Switch Priority: 128
Principal Switch Name: Principal Switch Name:
20:01:00:05:30:00:51:1f 20:01:00:05:30:00:51:1f

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 39

Exchanging Fabric Parameters:

If both switches have non-empty Domain ID lists, and there are no overlapping domains, a
merge will occur and a non-disruptive Build Fabric (BF) will be sent to restart PS selection and
Domain ID assignment. Since there were no overlaps the switches should get their same Domain
IDs.
If there are overlaps the switch will isolate the E_Port indicating there is a Domain ID conflict.
Note: An RCF could be sent to start a disruptive merge. An NEX-5020 switch will NOT do this
automatically. More about this later.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 39

 Appendix: Troubleshooting FCoE

Domain ID Assignment

ff.ff.fd -> ff.ff.fd SW_ILS 1 0xdd26 0xffff 0x0 -> 0xf DIA
ff.ff.fd -> ff.ff.fd FC 1 0xdd26 0xc283 0xff -> 0x0 Link Ctl, ACK1
ff.ff.fd -> ff.ff.fd SW_ILS 1 0xdd26 0xc283 0xff -> 0x0 SW_ACC (DIA)
ff.ff.fd -> ff.ff.fd FC 1 0xdd26 0xc283 0x0 -> 0xf Link Ctl, ACK1
ff.ff.fd -> ff.ff.fd SW_ILS 1 0xc284 0xffff 0xff -> 0x0 RDI
ff.ff.fd -> ff.ff.fd FC 1 0xc284 0xdd27 0x0 -> 0xf Link Ctl, ACK1
ff.ff.fd -> ff.ff.fd SW_ILS 1 0xc284 0xdd27 0x0 -> 0xf SW_ACC (RDI)

Cmd Code: DIA (0x12) Cmd Code: RDI (0x13)

Switch Name: 20:01:00:05:30:00:51:1f Payload Len: 16
Req Switch Name: 20:01:00:0c:30:e8:a6:01
Requested Domain ID: 1

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 40

Request Domain ID (RDI)

Domain ID Assigned (DIA)

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 40

Appendix: Troubleshooting FCoE

Final EFP

ff.ff.fd -> ff.ff.fd SW_ILS 1 0xdd28 0xffff 0x0 -> 0xf EFP
ff.ff.fd -> ff.ff.fd FC 1 0xdd28 0xc285 0xff -> 0x0 Link Ctl, ACK1
ff.ff.fd -> ff.ff.fd SW_ILS 1 0xdd28 0xc285 0xff -> 0x0 SW_ACC (EFP)
ff.ff.fd -> ff.ff.fd FC 1 0xdd28 0xc285 0x0 -> 0xf Link Ctl, ACK1

Cmd Code: EFP (0x11) Cmd Code: SW_ACC (0x02)

Payload Len: 48 Payload Len: 48
Principal Switch Priority: 2 Principal Switch Priority: 2
Principal Switch Name: Principal Switch Name:
20:01:00:05:30:00:51:1f 20:01:00:05:30:00:51:1f
Domain ID Record Domain ID Record
Domain ID: 0x01 Domain ID: 0x01
Switch Name: 20:01:00:05:30:00:51:1f Switch Name: 20:01:00:05:30:00:51:1f
Domain ID Record Domain ID Record
Domain ID: 0xee Domain ID: 0xee
Switch Name: 20:01:00:0c:30:e8:a6:01 Switch Name: 20:01:00:0c:30:e8:a6:01

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 41

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 41

 Appendix: Troubleshooting FCoE

Fabric Configuration – NEX-5020

The NEX-5020 switch will never automatically send a RCF
N5K# show fcdomain vsan 1
[snip] We wont send
Local switch configuration information: RCF
State: Enabled
FCID persistence: Enabled
Auto-reconfiguration: Disabled
[snip]
Can configure
switch to RJT
Principal switch run time information:
a received RCF
Running priority: 2
Interface Role RCF-reject
---------------- ------------- ------------
fc3/1 Upstream Disabled
---------------- ------------- ------------

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 42

RCF = Reconfigure Fabric (this is a disruptive operation)

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 42

 Appendix: Troubleshooting FCoE

Fabric Configuration – NEX-5020

 Manually send an RCF:

– N5K(config)# fcdomain restart disruptive vsan 5

 Configure the NEX-5020 to SW_RJT a received RCF

– N5K(config-if)# fcdomain rcf-reject vsan 5

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 43

FSPF Offers:
 - Fast Computation of new paths in case of Failure
 - Routing loop free topology
 - Load sharing over equivalent paths
 - Guaranteed in order delivery during topology changes

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 43

Appendix: Troubleshooting FCoE

NEX-5020 FSPF interface settings

N5K(config-if)# sh fspf interface

FSPF interface fc3/1 in VSAN 1
FSPF routing administrative state is active
Interface cost is 250
Timer intervals configured, Hello 20 s, Dead 80 s, Retransmit 5 s
FSPF State is FULL
Neighbor Domain Id is 0x67(103)
Neighbor Interface is fc2/13 (0x0001010c)

Statistics counters :
Number of packets received : LSU 4 LSA 4 Hello 58 Error packets 0
Number of packets transmitted : LSU 4 LSA 4 Hello 58 Retransmitted LSU 0
Number of times inactivity timer expired for the interface = 0

[snip]

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 44

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 44

Appendix: Troubleshooting FCoE

NEX-5020 FSPF displays

N5K(config-if)# show fspf database vsan 5

FSPF Link State Database for VSAN 5 Domain 0xee(238)

LSR Type = 1
Advertising domain ID = 0xee(238)
LSR Age = 806
LSR Incarnation number = 0x80000002
LSR Checksum = 0x452f
Number of links = 1
NbrDomainId IfIndex NbrIfIndex Link Type Cost
-----------------------------------------------------------------------------
0xef(239) 0x0001010c 0x00010200 1 250

FSPF Link State Database for VSAN 5 Domain 0xef(239)

LSR Type = 1
Advertising domain ID = 0xef(239)
LSR Age = 805
LSR Incarnation number = 0x80000002
LSR Checksum = 0x373c
Number of links = 1
NbrDomainId IfIndex NbrIfIndex Link Type Cost
-----------------------------------------------------------------------------
0xee(238) 0x00010200 0x0001010c 1 250

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 45

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 45

Appendix: Troubleshooting FCoE

Other useful FSPF displays

5K(config-if)# show fspf internal route vsan 5

FSPF Unicast Routes

---------------------------
VSAN Number Dest Domain Route Cost Next hops
-----------------------------------------------
5 0xee(238) 250 fc3/1

N5K(config-if)# show topology vsan 5

FC Topology for VSAN 5 :

--------------------------------------------------------------------------------
Interface Peer Domain Peer Interface Peer IP Address
--------------------------------------------------------------------------------
fc3/1 0xee(238) fc2/13 172.18.217.70

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 46

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 46

Appendix: Troubleshooting FCoE

Timers

 Five Main Timers (defaults)

– Receiver-Transmitter Time Out (R_T_TOV – 100ms)

– Error Detect Time Out (E_D_TOV – 2 seconds)

– Resource Allocation Time Out (R_A_TOV – 10 seconds)

– Fabric Stability Time Out (F_S_TOV – 5 seconds)

– Distributed Services Time Out (D_S_TOV - 5 seconds)

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 47

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 47

Appendix: Troubleshooting FCoE

NEX-5020 FC Timers

N5K(config-if)# sh fctimer

F_S_TOV D_S_TOV E_D_TOV R_A_TOV

----------------------------------------
5000 ms 5000 ms 2000 ms 10000 ms

N5K(config-if)# fctimer ?
No matches in current mode, matching in (config) mode
D_S_TOV D_S_TOV in milliseconds(5000-100000)
E_D_TOV E_D_TOV in milliseconds(1000-100000)
R_A_TOV R_A_TOV in milliseconds(5000-100000)
abort abort the fctimer configuration commands
commit commit the fctimer configuration commands
distribute Enable distribution of fctimer configuration using CFS

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 48

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 48

Appendix: Troubleshooting FCoE

Databases used for zoning

 Active zoneset

This database contains only the currently active zone and zoneset
information. FC aliases are not included in this database.

 Full zoneset
This database contains the currently active zone and zoneset information
as well as the other configured zones and zonesets that are not
currently active. FC aliases are included in this database.

A switch may maintain a copy of either of these databases

for management and configuration activity, but the copy is
for local use only.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 49

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 49

 Appendix: Troubleshooting FCoE

FCalias and Device-alias

 FCalias tied to zoning services only.
 FCalias is associated to the configured VSAN.
 FCalias can contain multiple members.
 FCalias is distributed with full-zoneset.
 FCalias can be interoperable with OEM switches.
 Device-alias can be used for zoning, QoS, IVR and other applications.
 Device alias is distributed via CFS, not via zoning database
 Device-alias is mapped to 1 PWWN only.
 Device-alias is not tied to any VSAN, and can be used in multiple VSANs.
 Device-alias can not be distributed to OEM switches or 9020.

N5K# device-alias mode ?

enhanced Configure device-alias enhanced mode

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 50

Device-alias is used to map 1 PWWN to a user friendly naming convention. It can not be sent
to, or used on, non Cisco switches. It can be used in place of the PWWN for application
configuration like zoning, QoS, IVR, fcping, fctrace.
FCalias is only used in conjunction with a single vsan zoning. It is not available for other
applications. It can contain multiple members of different types.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 50

 Appendix: Troubleshooting FCoE

A zone member can be?

N5K(config-if)# zone name test vsan 1

N5K(config-zone)# member ?
device-alias Add device-alias member to zone
domain-id Add member based on domain-id,port-number
fcalias Add fcalias to zone
fcid Add FCID member to zone
fwwn Add Fabric Port WWN member to zone
interface Add member based on interface
pwwn Add Port WWN member to zone
symbolic-nodename Add Symbolic Node Name member to zone

Domain-id is for interop 2 - 4 modes only.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 51

These are the possible ways to define members in a zone. You can mix member types within a
zone.
If the zone is part of an interop VSAN or fabric, you should only use PWWN members.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 51

Appendix: Troubleshooting FCoE

Standards for zoning

 FC-SW4 is the current switch function standard from the
T11 group.
 FC-GS4 is the current generic services standard from the
T11 group.
 Section 10 of T11’s FC-SW4 standard details how the
zoning operations are expected to function on and
between fibre channel switches.
 Link for current standards http://www.t11.org/index.htm

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 52

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 52

 Appendix: Troubleshooting FCoE

Flows for basic zoning (Activate zoneset)

Management Station

0x76 0x65
Changes initiated from seed switch
Seed switch Subordinate switch

Upon zoneset activate, 4 command frames

are sent from seed to each subordinate switch

• ACA (Acquire Chance Authorization) locks the zone application so no other 0xEF
changes can be attempted
Subordinate switch
• SFC (Stage Fabric Configuration) sends the zone data in the frame payload
• UFC (Update Fabric Configuration) triggers all switches to make the new data
active

• RCA (Release Change Authorization) release the zone applications so future

changes can be made

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 53

There is a timer that is started by the seed switch for each command.
If the seed switch does not receive and ACC (accept) from each subordinate switch for the
current command, it will abort the zoning activation.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 53

Appendix: Troubleshooting FCoE

ACA frame from FCanalyzer detail

Fibre Channel
R_CTL: 0x02
Dest Addr: ff.fc.65 FCID’s are the Domain name servers
CS_CTL: 0x00 of the sender/receiver
Src Addr: ff.fc.76
Type: SW_ILS (0x22)
F_CTL: 0x290000 (Exchange Originator, Seq Initiator,
Exchg First, Seq Last, CS_CTL, Transfer SeqInitiative,
Last Data Frame - No Info, ABTS - Abort/MS, )
SEQ_ID: 0x0c
DF_CTL: 0x00
SEQ_CNT: 0 ACA
OX_ID: 0x0159 Command
RX_ID: 0xffff code
Parameter: 0x00000000
SW_ILS
Cmd Code: ACA (0x23)
Domain ID List Length: 8
Domain ID 0: 101
Domains (shown in decimal) in the fabric
Domain ID 1: 118 that will receive the zoning change,
Domain ID 2: 239 including the initiating switch

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 54

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 54

Appendix: Troubleshooting FCoE

SFC frame from FCanalyzer detail

Fibre Channel
R_CTL: 0x02
Dest Addr: ff.fc.65
CS_CTL: 0x00
Src Addr: ff.fc.76
Type: SW_ILS (0x22)
F_CTL: 0x290000 (Exchange Originator, Seq Initiator, Exchg First, Seq Last, CS_CTL, Transfer Seq
Initiative, Last Data Frame - No Info, ABTS - Abort/MS, )
SEQ_ID: 0x0d
DF_CTL: 0x00 Operation Request. 03 or 04 are
SEQ_CNT: 0 SFC for Basic Mode Zoning
OX_ID: 0x015a command
RX_ID: 0xffff code
Parameter: 0x00000000
SW_ILS
Cmd Code: SFC (0x25)
Operation Request: Activate Zone Set (0x03)
ZoneSet Length: 56
Zone Set Name: zs3 Zoneset
Zone Set name
Number of zoning objects: 1
Zone Object 0
Zone Object Type: Zone (0x02)
Zone Protocol: 0x00 Zone name
Zone Object Name: zone3
Number of Zone Members: 2
Zone Member 0
Zone Member Type: WWN (0x01)
Flags: 0x0
Identifier Length: 8
Member Identifier: 10:00:00:01:73:00:94:84 (00:01:73)
Zone Member 1
Zone Member Type: WWN (0x01)
Flags: 0x0
Identifier Length: 8
Member Identifier: 22:00:00:04:cf:6e:4b:dd (00:04:cf)
Full Zone Set Length: 0
© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 55

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 55

Appendix: Troubleshooting FCoE

UFC frame from FCanalyzer detail

Fibre Channel
R_CTL: 0x02
Dest Addr: ff.fc.65
CS_CTL: 0x00
Src Addr: ff.fc.76
Type: SW_ILS (0x22)
F_CTL: 0x290000 (Exchange Originator, Seq Initiator,
Exchg First, Seq Last, CS_CTL, Transfer Seq
Initiative, Last Data Frame - No Info, ABTS - Abort/MS, )
SEQ_ID: 0x0e
DF_CTL: 0x00
SEQ_CNT: 0
OX_ID: 0x015b
RX_ID: 0xffff
Parameter: 0x00000000
SW_ILS UFC
Cmd Code: UFC (0x26) command
code

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 56

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 56

Appendix: Troubleshooting FCoE

RCA from FCanalyzer detail

Fibre Channel
R_CTL: 0x02
Dest Addr: ff.fc.65
CS_CTL: 0x00
Src Addr: ff.fc.76
Type: SW_ILS (0x22)
F_CTL: 0x290000 (Exchange Originator, Seq Initiator, Exchg First,
Seq Last, CS_CTL, Transfer Seq
Initiative, Last Data Frame - No Info, ABTS - Abort/MS, )
SEQ_ID: 0x0f
DF_CTL: 0x00
SEQ_CNT: 0
OX_ID: 0x015c
RX_ID: 0xffff RCA
Parameter: 0x00000000 command
SW_ILS code
Cmd Code: RCA (0x24)

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 57

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 57

 Appendix: Troubleshooting FCoE

Frames as seen in a Finisar Trace

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 58

Here we see the ACA, the SFC, the UFC and the RCA.
Notice the FCID’s used for the frames. The last byte is the domain ID of the switch sending the
frame or the ACC.
Also, notice that each exchange (OX_ID) is 4 frames. Class F frames require an ACK1.
The Zoning standard requires an ACC (Accept) to each command frame.
In this example, Domain 0x76 is the seed switch, while Domain 0x65 is the subordinate switch.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 58

Appendix: Troubleshooting FCoE

Zone server modes

 Zone server supports 2 different modes

– Basic mode – represents the zone server behavior of
FC-GS3/FC-SW2 standard. All SAN-OS support basic
mode.
– Enhanced mode – represents the zone server
behavior of FC-GS4/FC-SW3 standard. SAN-OS 2.0
and later are required for enhanced mode.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 59

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 59

4
 Appendix: Troubleshooting FCoE

What is Enhanced Zoning

 Enhanced zoning provides the following

advantages
– Disallow parallel configuration attempts
– Standardized generation of RSCN
– Reduced payload size of the SFC frame
– Fabric-wide policy enforcement (default zone, merge control)
– Enhanced error reporting
– Distributing zonesets without activation
– Unique Vendor Types
– FWWN based member type standardized
– Enhanced interop thru ESS

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 60

Disallow parallel configuration: GS4 enforces all configurations to be done within a session. A session begins with the
reception of session-begin command and ends on reception of session-end command. When a session-begin command is
received, the entry switch should lock the entire fabric before it accepts any more commands within the session. The locking
ensures that there exists a single management session which is effecting a configuration change in the fabric. This feature ensures
consistency in the fabric.
GS4 mandates the usage of session for any Name server interaction/registration by a device. Earlier, NS registration could be
done at any time. Normally, as soon as devices login, they would do multiple back-to-back registrations with few msecs apart.
Whenever registrations are done, RSCN needs to be generated. However, to reduce the no. of RSCNs, NS used to run a timer to
wait for some time hoping to consume some more registrations so that a single RSCN can be generated for multiple registrations.
The timeout values are fixed based on heuristics.
This problem has been addressed in GS4. The new standard states that the device should start a session, do multiple registrations
and complete it by a session-end command. The switch can generate the RSCN on reception of session-end command. This
mechanism exactly defines when the RSCN should be generated and clearly better than the earlier timer-based approach. It also
saves timer resources.
GS4/SW3 have defined a new zoning object called `Zone Reference Object'. Zone Reference object is nothing but pointers (in
Coding terminology). If an object (ex: Zone) is part of multiple other higher-level objects (ex: Zonesets), instead of creating an
instance of the object for each higher-level object, you can define the object once and higher-level objects can use references to
the object. This feature certainly reduces the payload size as Reference object consume lesser bytes than the actual object.This
advantage is more pronounced with bigger databases with lot of zonesets most of which reuse the same set of zones.
SW3/GS4 defines standard way of enforcing/exchanging default-zone setting throughout the fabric. Also, when fabrics are
merged, default-zone settings are exchanged to validate the same setting on either side of the fabric.
GS4 defines a new command (GAR) to retrieve the activation results from each of the remote switches. This would also include
the nature of the problem at each switch. This feature would help in easier trouble-shooting of any activation problems.
GS4/SW3 has defined new command codes to distribute the zoning database without any reactivations. This avoids any
hardware changes (TCAM programming) on the remote switches.
SW3/GS4 and has mandated the use of Vendor-ID along with Vendor-specific type value. This would uniquely identify a
member type.
MDS switch supports FWWN members as part of zones. FWWN based zone membership is an easier way of configuring
JBODs/FL devices.
ESS (Exchange Switch Support) defined in SW-3 also enhances interop. Currently, Name Server decides the object in its
response to GE-XX queries based on the interop mode. With ESS, vendors can specify what type of entry objects they
support(Large, small, ..). This will enhance interoperability between the switches of different vendors.
Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 60
Appendix: Troubleshooting FCoE

Enhanced vs. Basic Zoning

Enhanced
Basic Zoning Enhanced Zoning
Advantages

All configuration changes are One configuration

Administrators can make
made within a single session. session for entire fabric
simultaneous
Switch locks entire fabric to to ensure consistency
configuration changes
implement change within fabric

Reduced payload size as

If a zone is a member of
References to the zone are used the zone is referenced.
multiple zonesets , an
by the zonesets as required once The size is more
instance is created per
you define the zone. pronounced with bigger
zoneset.
database

Enforces and exchanges default Fabric-wide policy

Default zone policy is
zone setting throughout the enforcement reduces
defined per switch.
fabric troubleshooting time.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 61

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 61

 Appendix: Troubleshooting FCoE

Enhanced vs. Basic Zoning

Enhanced
Basic Zoning Enhanced Zoning
Advantages

Managing switch provides Retrieves the activation Enhanced error

combined status about results and the nature of the reporting reduces
activation. Will not identify a problem from each remote troubleshooting
failure switch. switch. process.

Implements changes to the

This avoids hardware
To distribute zoneset must re- zoning database and
changes for hard
activate the same zoneset. distributes it without
zoning in the switches.
activation.

During a merge NEX-5020 Provides a vendor ID along

specific types can be with a vendor-specific type
Unique Vendor type
misunderstood by non-cisco value to uniquely identify a
switches. member type

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 62

To distribute the zoning database, you must reactivate the same zone set. The reactivation may
affect hardware changes for hard zoning on the local switch and on remote switches.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 62

 Appendix: Troubleshooting FCoE

Moving from basic to enhanced mode

 When the mode changes from basic to enhanced the

following operations happen
– The active as well as the full database is distributed to all switches
in enhanced mode payload format
– The zoning policies (eg: default-zone permit) are distributed to all
switches in enhanced mode payload format.
– If both SFCs are acceptable to all switches, then the mode
changes to enhanced mode.
– If either of the SFCs are rejected, then mode change fails.
 Moving from basic to enhanced mode is done using the
following command.
– N5k (config)# zone mode enhanced vsan <vsan-id>

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 63

There are 2 types of SFC, basic and enhanced. Both are sent when making the change. If both
are accepted, then the change to enhanced mode is successful. If the enhanced SFC is rejected,
then the mode stays in basic.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 63

4
Appendix: Troubleshooting FCoE

Zone mode enhanced vsan xxx

1.197674 ff.fc.ef -> ff.fc.13 0x76c 0xffff SW_ILS ACA
1.197800 ff.fc.13 -> ff.fc.ef 0x76c 0x6ba FC Link Ctl, ACK1
1.199916 ff.fc.13 -> ff.fc.ef 0x76c 0x6ba SW_ILS SW_ACC (ACA)
1.200305 ff.fc.ef -> ff.fc.13 0x76c 0x6ba FC Link Ctl, ACK1
1.202355 ff.fc.ef -> ff.fc.13 0x76d 0xffff SW_ILS SFC
1.202437 ff.fc.13 -> ff.fc.ef 0x76d 0x6bb FC Link Ctl, ACK1
1.204349 ff.fc.13 -> ff.fc.ef 0x76d 0x6bb SW_ILS SW_ACC (SFC)
1.204698 ff.fc.ef -> ff.fc.13 0x76d 0x6bb FC Link Ctl, ACK1
1.206307 ff.fc.ef -> ff.fc.13 0x76e 0xffff SW_ILS SFC
1.206382 ff.fc.13 -> ff.fc.ef 0x76e 0x6bc FC Link Ctl, ACK1
1.208764 ff.fc.13 -> ff.fc.ef 0x76e 0x6bc SW_ILS SW_ACC (SFC)
1.209088 ff.fc.ef -> ff.fc.13 0x76e 0x6bc FC Link Ctl, ACK1
1.218453 ff.fc.ef -> ff.fc.13 0x76f 0xffff SW_ILS UFC
1.218538 ff.fc.13 -> ff.fc.ef 0x76f 0x6bd FC Link Ctl, ACK1
1.297823 ff.fc.13 -> ff.fc.ef 0x76f 0x6bd SW_ILS SW_ACC (UFC)
1.298171 ff.fc.ef -> ff.fc.13 0x76f 0x6bd FC Link Ctl, ACK1
1.308122 ff.fc.ef -> ff.fc.13 0x770 0xffff SW_ILS ESS
1.308291 ff.fc.13 -> ff.fc.ef 0x770 0x6be FC Link Ctl, ACK1
1.311366 ff.fc.13 -> ff.fc.ef 0x770 0x6be SW_ILS SW_ACC (ESS)
1.311631 ff.fc.ef -> ff.fc.13 0x770 0x6be FC Link Ctl, ACK1
1.394716 ff.fc.ef -> ff.fc.13 0x771 0xffff SW_ILS RCA
1.394833 ff.fc.13 -> ff.fc.ef 0x771 0x6bf FC Link Ctl, ACK1
1.396708 ff.fc.13 -> ff.fc.ef 0x771 0x6bf SW_ILS SW_ACC (RCA)
1.397056 ff.fc.ef -> ff.fc.13 0x771 0x6bf FC Link Ctl, ACK1

Two SFCs are sent as a result of moving to Enhanced Mode Zoning

One is a zoneset activate in enhanced mode zoning
Two is a set zoning policies

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 64

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 64

Appendix: Troubleshooting FCoE

The two SFCs, both must be ACCepted

Fibre Channel Fibre Channel

Exchange Last In: 0 Exchange Last In: 0
R_CTL: 0x2(Device_Data/Unsolicited Control) R_CTL: 0x2(Device_Data/Unsolicited Control)
Dest Addr: ff.fc.13 Dest Addr: ff.fc.13
CS_CTL: 0x00 CS_CTL: 0x00
Src Addr: ff.fc.ef Src Addr: ff.fc.ef
Type: SW_ILS (0x22) Type: SW_ILS (0x22)
F_CTL: 0x290000 Exchange Originator, Seq Initiator, Exchg First, F_CTL: 0x290000 Exchange Originator, Seq Initiator, Exchg First,
Seq Last, CS_CTL, Transfer Seq Seq Last, CS_CTL, Transfer Seq
Initiative, Last Data Frame - No Info, ABTS - Abort/MS, Initiative, Last Data Frame - No Info, ABTS - Abort/MS,
0... .... .... .... .... .... = ExgRpd: Exchange Originator 0... .... .... .... .... .... = ExgRpd: Exchange Originator
.0.. .... .... .... .... .... = SeqRec: Seq Initiator .0.. .... .... .... .... .... = SeqRec: Seq Initiator
..1. .... .... .... .... .... = ExgFst: Exchg First ..1. .... .... .... .... .... = ExgFst: Exchg First
...0 .... .... .... .... .... = ExgLst: NOT exchg last ...0 .... .... .... .... .... = ExgLst: NOT exchg last
.... 1... .... .... .... .... = SeqLst: Seq Last .... 1... .... .... .... .... = SeqLst: Seq Last
.... ..0. .... .... .... .... = Pri: CS_CTL .... ..0. .... .... .... .... = Pri: CS_CTL
.... ...1 .... .... .... .... = TSI: Transfer Seq Initiative .... ...1 .... .... .... .... = TSI: Transfer Seq Initiative
.... .... 00.. .... .... .... = LDF: Last Data Frame - No Info .... .... 00.. .... .... .... = LDF: Last Data Frame - No Info
(0x000000) (0x000000)
.... .... ..00 .... .... .... = A01: no ack required (0x000000) .... .... ..00 .... .... .... = A01: no ack required (0x000000)
.... .... .... ..0. .... .... = RetSeq: NOT retransmitted .... .... .... ..0. .... .... = RetSeq: NOT retransmitted
sequence sequence
.... .... .... .... ..00 .... = AA: ABTS - Cont (0x000000) .... .... .... .... ..00 .... = AA: ABTS - Cont (0x000000)
.... .... .... .... .... 0... = RelOff: rel offset NOT set .... .... .... .... .... 0... = RelOff: rel offset NOT set
SEQ_ID: 0xea SEQ_ID: 0xeb
DF_CTL: 0x00 DF_CTL: 0x00
SEQ_CNT: 0 SEQ_CNT: 0
OX_ID: 0x0888 OX_ID: 0x0889
RX_ID: 0xffff RX_ID: 0xffff
Parameter: 0x00000000 Parameter: 0x00000000
SW_ILS SW_ILS
Cmd Code: SFC (0x25000000) Cmd Code: SFC (0x25000000)
Operation Request: Unknown (0x08) Operation Request: Unknown (0x0c)
ZoneSet Length: 0 ZoneSet Length: 0
Full Zone Set Length: 0 Full Zone Set Length: 8
Full Zone Set
Number of zoning objects: 50386056

Next Slide details the Operation Request

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 65

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 65

Appendix: Troubleshooting FCoE

SFC operation request values

Basic mode

Enhanced mode

Default zone permit/deny

or zone merge rules can be
exchanged with this type
of SFC.

We use F0 from this range

for moving from Enhanced
to Basic

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 66

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 66

Appendix: Troubleshooting FCoE

Moving from enhanced mode to basic mode

 This is only supported in NEX-5020. Standard does
not support this.
 Can lose attributes like broadcast if in basic zoning
mode broadcast is not enabled.
 When the mode changes from enhanced to basic the
following operations happen
– The active as well as the full database is distributed to all switches
in proprietary payload format.
– If SFC is accepted by all switches, then the mode changes to
basic mode.
– If SFC is not acceptable then mode change fails.
– The SFC contains a vendor specific operation request of F0

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 67

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 67

4
 Appendix: Troubleshooting FCoE

Zoning policies

 Zoning policies are distributed fabric wide.

 Zoning policies supported are
– Default-zone policy - permit/deny
– Merge Ctrl policy – allow/restrict
 Merge ctrl policy defines if merge rules should be
applied during a merge or should the databases be an
exact match prior to the merge, for the merge to
succeed.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 68

Merge control prevents a change from being introduced by accident if a change was made while
the fabrics were isolated.
If a fabric were isolated, and a change made on one side, without merge control restrict if there
were no conflicts, the change is propagated once the isolation issue is resolved.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 68

4
Appendix: Troubleshooting FCoE

Zone policy bits in the SFC

At this point in time,

Ethereal can not decode
the Zone Policy bits in
the ‘SET ZONING POLICY’
SFC

0000 1000 Bit 3 is on

7654 3210 BIT numbers
This shows the setting of Default Zone to deny for the fabric.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 69

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 69

 Appendix: Troubleshooting FCoE

Enhanced mode operation

 The first configuration command on the zoning database, would
acquire a fabric wide lock preventing other changes on the
VSAN.
 The first configuration command on the zoning database, would
also create a local copy of that VSANs zoning database.
 Changes done to the zoning database are done on the copy.
 ‘Commit’ has to be issued to apply the changes.
 Commit would destroy the copy of the zoning database after
activation.
 Commit would also release the fabric wide lock
 If the changes are to be discarded prior to the commit, issue “no
zone commit” command, which would release lock and throw
away the changes.
© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 70

The intent of the lock is prevent conflicting changes. In basic mode, the last user to activate a
zoneset will have their changes moved to the fabric. With enhanced zoning, the intent is to lock
the fabric once a change is started. Additional attempts to start a zoning change will be blocked,
and the user will not be able to make a zoning change.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 70

4
 Appendix: Troubleshooting FCoE

Example configuration
ACA sent
Sample configuration to lock
fabric

N5K(config-zoneset)# zone name fcoe-zone vsan 5

N5K(config-zone)# mem pwwn 10:00:00:00:c9:3c:f4:82
N5K(config-zone)# mem pwwn 21:00:00:04:cf:e1:75:f9
N5K(config-zone)# exit
N5K(config)# zoneset activate name fcoe-zoneset vsan 5
Zone not present
SFC and UFC sent to
N5K(config)# zoneset name fcoe-zoneset vsan 5 propagate the changed
N5K(config-zoneset)# member fcoe-zone zoning information.
RCA is then sent to
N5K(config-zoneset)# exit unlock the fabric.
N5K(config)# zoneset activate name fcoe-zoneset vsan 5
N5K(config)# zone commit vsan 5
Commit operation initiated. Check zone status

N5K(config)# zone commit vsan 5 RCA sent to unlock

fabric and local
changes are discarded.

Once changes are made, you must commit or no commit!!!!

There is no ‘abort’ like there is in CFS.
© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 71

In enhanced mode, as soon a zone change is started on any switch in the fabric (VSAN), the
ACA is sent to the other switches to lock the fabric.
The SFC, UFC, and RCA are sent on the commit, not the activate command.
No commit will sent RCA to discard changes. Like if you got distracted during the zone
change.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 71

Appendix: Troubleshooting FCoE

Zoning lock and lock owners

 The zoning lock is issued to a single user on a single

switch.
 Only one user is allowed to hold the lock.
 If for some reason the lock is held by another user, and
the lock has to be cleared forcefully, then issue “no
commit” with the force flag.
 Eg: switch (config)# no zone commit vsan 1 force

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 72

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 72

4
Appendix: Troubleshooting FCoE

Who has the lock?

 Lock can be held by another user on the same

switch.
 Lock can be held by same user on another switch.
 Lock can be held by another user on another
switch
 If you’re locked out, follow these steps.
1. Determine which switch (domain) has the lock
2. Determine which user has the lock on that switch
3. Clear the lock for that user on that switch

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 73

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 73

 Appendix: Troubleshooting FCoE

Which switch/domain has the lock?

N5K(config)# zone name is_it_locked vsan 5
Zoning database update in progress, command rejected Error message when attempting
zone change while fabric is
locked

N5K# show zone internal vsan 5

VSAN: 5 default-zone: deny(rw) distribute: active only
E_D_TOV: 2000 R_A_TOV: 10000 D_S_TOV: 5000 F_S_TOV: 5000 F_D_TOV: 2000
Interop: default IOD: disable bcast: enable dflt-bcast: disable dflt-qos: 0
DBLock:C(T count:1) Ifindex Table Size: 8 Transit Frame Index: 0
Total Transit Frame Count: 0 Transit Discard Count: 0
[snip] Domain that currently
Change protocol info : has the lock is 0xee
local domain id = 239, ACA by 0xee
State = ACA Rcvd, reply_cnt = 1, req_sent_cnt = 1, req_pending = 0
Remote domains :
238(0xee) [20:05:00:0d:ec:3f:a5:c1]
ACA was received, this
is why the fabric is
locked

Local domain (0xef) is

N5K# sh fcdomain domain-list vsan 5 different, so another
switch has the lock
Number of domains: 2
Domain ID WWN
--------- -----------------------
0xef(239) 20:05:00:0d:ec:6b:cd:41 [Local] [Principal]
0xee(238) 20:05:00:0d:ec:3f:a5:c1

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 74

When the lock is present, the fabric is locked, and the user attempting the change will see an
error message.
To determine which switch/domain has the lock, issue show zone internal command.
Look for the ACA by field.
View all domains with the show fcdomain domain-list command.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 74

 Appendix: Troubleshooting FCoE

Which switch has the lock? Local Domain matches

the “ACA by” display
from the other switch.
N5K# sh fcs ie vsan 5

IE List for VSAN: 5

---------------------------------------------------------------------------
----
IE-WWN IE Mgmt-Id Mgmt-Addr
---------------------------------------------------------------------------
----
20:05:00:0d:ec:3f:a5:c1 S(Adj) 0xfffcee 172.18.217.70
20:05:00:0d:ec:6b:cd:41 S(Loc) 0xfffcef 172.18.217.80
[Total 2 IEs in Fabric]]

mds9216i# show zone internal v 5

VSAN: 5 default-zone: deny(rw) distribute: active only
E_D_TOV: 2000 R_A_TOV: 10000 D_S_TOV: 5000 F_S_TOV: 5000 F_D_TOV: 2000
Interop: default IOD: disable bcast: enable dflt-bcast: disable dflt-qos: 0
DBLock:G(T count:1) Ifindex Table Size: 8 Transit Frame Index: 0
Total Transit Frame Count: 0 Transit Discard Count: 0
This switch has the
[snip] lock, so ACA by 0xff
is displayed.
Change protocol info :
local domain id = 238, ACA by 0xff
State = ACA Complete, reply_cnt = 1, req_sent_cnt = 1, req_pending = 0
Remote domains :
239(0xef) [20:05:00:0d:ec:6b:cd:41] ACA complete means that
Status : all other domains have
responded with an ACC.

Note: These displays are not from the same switch as

the previous slide. We have changed to the switch where the lock is held.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 75

On the switch that issued the ACA and acquired the lock, the show zone internal will look a bit
different.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 75

 Appendix: Troubleshooting FCoE

Which user has the lock on this switch?

mds9216i# show zone status v 5
VSAN: 5 default-zone: deny distribute: active only Interop: default
mode: enhanced merge-control: allow
session: cli [admin] User admin has
hard-zoning: enabled broadcast: enabled the lock on this
Default zone: switch.
qos: none broadcast: disabled ronly: disabled
Full Zoning Database :
DB size: 156 bytes
Zonesets:1 Zones:1 Aliases: 0 Attribute-groups: 1
Active Zoning Database :
DB size: 84 bytes
Name: fcoe-zoneset Zonesets:1 Zones:1
Status: Activation completed at 23:04:13 UTC Aug 12 2008

Display is seen from

mds9216i# show users user admin since it
admin pts/0 Aug 12 23:01 (64.102.80.101)* has the * next to it.

mds9216i# clear zone lock v 5

mds9216i# sh zone status vsan 5
VSAN: 5 default-zone: deny distribute: active only Interop: default
mode: enhanced merge-control: allow
session: none Session
hard-zoning: enabled broadcast: enabled cleared for
Default zone: user admin
qos: none broadcast: disabled ronly: disabled
Full Zoning Database :
DB size: 156 bytes
Zonesets:1 Zones:1 Aliases: 0 Attribute-groups: 1
Active Zoning Database :
DB size: 84 bytes
Name: fcoe-zoneset Zonesets:1 Zones:1
Status: Activation completed at 23:04:13 UTC Aug 12 2008
© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 76

On the switch with the lock, the show zone status will show which user has the lock.
Once the lock is cleared, the show zone status display shows none for the session.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 76

 Appendix: Troubleshooting FCoE

Internal displays show that lock has been cleared

mds9216i# show zone internal

[snip] ACA by 0xff indicates that

no lock is in effect
Change protocol info :
local domain id = 103, ACA by 0xff
State = Idle, reply_cnt = 0, req_sent_cnt = 0, req_pending = 0
Remote domains :
228(0xe4) [20:01:00:0d:ec:6b:cd:41]
Different
Switches!
N5K# show zone internal

[snip]

Change protocol info :

local domain id = 228, ACA by 0xff
State = Idle, reply_cnt = 0, req_sent_cnt = 0, req_pending = 0
Remote domains :
103(0x67) [20:01:00:0d:ec:3f:a5:c1]

State = is now blank

since there is no lock

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 77

The show zone internal command on both switches look the same for state and ACA by now
that there is no lock in effect any more.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 77

Appendix: Troubleshooting FCoE

How does FM show the lock?

Which user has the

lock and at what
switch the lock is
held from.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 78

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 78

Appendix: Troubleshooting FCoE

Using FM to clear the lock

Under the Enhanced tab

Click here and then

Which user has the lock click on the Green
and at what switch Arrow to apply the
Under VSAN and change.
Default Zone

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 79

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 79

Appendix: Troubleshooting FCoE

Lock is gone!

Zone lock is cleared.

Un-committed changes are discarded and

user that held the lock is not notified
© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 80

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 80

Appendix: Troubleshooting FCoE

Enhanced zoning commands

show zone ess vsan <vsan-id>
• Shows the ESS information in vsan

Bits of
Interest
N5K# sh zone ess vsan 5
ESS info on VSAN 5 :
Domain : 239, SWWN : 20:05:00:0d:ec:6b:cd:41, Cap1 : 0xfb, Cap2 : 0x0
Domain : 238, SWWN : 20:05:00:0d:ec:3f:a5:c1, Cap1 : 0xfb, Cap2 : 0x0

Bit 0 = Enhanced zoning supported? 1 = yes 0 = no

Bit 1 = Enhanced zoning enabled? 1 = yes 0 = no
Bit 2 = Merge Control? 1 = restrict 0 = allow
Bit 3 = Default zone setting? 1 = deny 0= permit

0xfb = Enhanced mode supported, enhanced mode enabled, and default zone set to
deny

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 81

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 81

 Appendix: Troubleshooting FCoE

Zoneset activate failure

 Look at the messages on the seed switch to determine
what caused the activate to fail. ‘show logging log’ (seed
is the switch where the change was initiated from)
 For multi switch fabrics, check that the ISL or TE-ISL is
operational.
 Show interface fc x/y or show interface port-channel x
 Show fcdomain domain-list vsan x
 Show zone internal

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 82

The syslog should contain messages indicating why the zoneset activate failed. Check the seed
switch first, and then the remote switches.
Check that the ISL is operational.
Collect zone displays or debugs to assist with problem determination.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 82

 Appendix: Troubleshooting FCoE

Change event history

N5K# sh zone internal change event-history v 5

Change Protocol Event Log For VSAN: 5
>>>>FSM has 41 logged transitions<<<<

[snip]

6) Transition at Tue Aug 12 19:23:48 2008

Prev State: [Idle]
Trig event: [SEND_ACA]
Next State: [ACA Sent]
One event.
7) Transition at Tue Aug 12 19:23:48 2008
Prev State: [ACA Sent]
Time of event is noted
Trig event: [RCVD_ACC] (Dom:238)
previous state, event,
Next State: [ACA Sent]
and next state are
8) Transition at Tue Aug 12 19:23:48 2008
shown.
Prev State: [ACA Sent]
Trig event: [RCVD_ALL_ACC]
Domain that initiated
Next State: [ACA Complete]
the event is also seen
9) Transition at Tue Aug 12 19:23:48 2008
Prev State: [ACA Complete]
Trig event: [SEND_SFC] (Activate Enh)
Next State: [Tz Req Sent]
10) Transition at Tue Aug 12 19:23:48 2008
Prev State: [Tz Req Sent]
Trig event: [RCVD_ALL_ACC]
Next State: [SFC Sent]

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 83

Here we see the 4 frames of a normal activation. This a state machine trace.
The current state, the triggering event, and the resulting stat are shown.
Here we see that the ACA, SFC, UFC, and RCA are all received from domain ID 210.
Timer expiration event will be displayed here.
This trace has the most recent events at the bottom. Here we see there are 50 events. Event 50
is the most recent.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 83

 Appendix: Troubleshooting FCoE

Zone merge failure options

 Confirm that there is a discrepancy in the zonesets on
opposite sides of the ISL or E-ISL that is isolated.
 Determine which of the 2 fabrics contain the desired
active zoneset, then use the zoneset import/export
command. This command only works if the ISL/E-ISL is
isolated.
 Prune the VSAN from the TE port, and add it back .
 Edit one or the other zonesets and then shut/no shut the
ISL or E-ISL. This action will impact all VSANs on the E-
ISL even those that are not isolated.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 84

Do not deactivate the zoneset on either end, because that will cause local traffic contained
within that fabric or switch…in the affected VSAN to be impacted.
Manually correct one or the other side
Or use the import/export command to push/pull the desired zoneset.
This command only applies to an E or TE that is in the isolated state.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 84

 Appendix: Troubleshooting FCoE

Summary – Not sure what to collect?

TM1# sh tech-support ?
<CR>
> Redirect it to a file
aaa Display aaa information
aclmgr ACL commands
adjmgr Display Adjmgr information
arp Display ARP information
bootvar Gather detailed information for bootvar troubleshooting
brief Display the switch summary
cdp Gather information for CDP trouble shooting
commands Show commands executed as part of show tech-support commands
details Gather detailed information for troubleshooting
ethpm Gather detailed information for ETHPM troubleshooting
fc Get fibre channel related information TM1# sh tech-support details ?
fcdomain Gather detailed information for fcdomain troubleshooting <CR>
flogi Gather detailed information for flogi troubleshooting > Redirect it to a file
im Gather detailed information for IM troubleshooting | Pipe command output to filter
interface-vlan Gather detailed information for interface-vlan troubleshooting
internal Gather internal info for troubleshooting
ip Display IP information
l3vm Display VRF information
license Gather detailed information for license troubleshooting
logging Show information on logging for technical support staff
module Gather info related to a module
monitor Gather detailed information for monitor troubleshooting
npv Show information for NPV technical support staff
platform Get platform related information
port-channel Gather detailed information for port channel troubleshooting
private-vlan Gather detailed information for private-vlan troubleshooting
radius Display radius information
routing Display routing information
security show tech support information for security
snmp Gather info related to snmp
stp Gather detailed information for STP troubleshooting
sysmgr Gather detailed information for sysmgr troubleshooting
vdc Gather detailed information for VDC troubleshooting
vlan Gather detailed information for VLAN troubleshooting
vsan Gather info related to an vsan
| Pipe command output to filter

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 85

The user has to decide which end of the isolate link contains the zoneset that is desired to be
active on both ends.
You can export (force a push of the current active zoneset) or import (force a pull of the remote
side’s current active zoneset)
This example is of an import..so we can track the change on the same switch. There will be no
changes to the remote side zoneset.
The only difference on the remote side, will be that the isolated vsan is no longer isolated.
The only zoneset change occurs on the switch that did the import, or was opposite the export.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 85

 Appendix: Troubleshooting FCoE

Zoneset import/export command

N5K# show int fc 3/1
Fc3/1 is trunking (Not all VSANs UP on the trunk)
Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
Port WWN is 20:c5:00:05:30:00:49:1e
Peer port WWN is 20:81:00:0d:ec:0f:b4:c0
Admin port mode is E, trunk mode is on
snmp traps are enabled
Port mode is TE
Port vsan is 1
Speed is 2 Gbps
Transmit B2B Credit is 255
Receive B2B Credit is 12
Receive data field Size is 2112 VSAN 5 is
Beacon is turned off isolated
Trunk vsans (admin allowed and active) (1,5)
Trunk vsans (up) (1)
Trunk vsans (isolated) (5)

Command to bypass merge

N5K# zoneset import int fc 3/1 v 5 checking and force import

You can force an export also

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 86

The user has to decide which end of the isolate link contains the zoneset that is desired to be
active on both ends.
You can export (force a push of the current active zoneset) or import (force a pull of the remote
side’s current active zoneset)
This example is of an import..so we can track the change on the same switch. There will be no
changes to the remote side zoneset.
The only difference on the remote side, will be that the isolated vsan is no longer isolated.
The only zoneset change occurs on the switch that did the import, or was opposite the export.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 86

Appendix: Troubleshooting FCoE

Appendix C

Additional Spanning Tree Slides

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 87

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 87

Appendix: Troubleshooting FCoE

Issues in a Poorly Designed Network

 Unbounded failure
domains
 Large broadcast
domains
 Large amount of
unknown MAC unicast
traffic
 Unbounded multicast
traffic
 Management and
support challenges
 Possible security
vulnerabilities

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 88

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 88

 Appendix: Troubleshooting FCoE

VLAN Overview

 Segmentation
 Flexibility
 Security

VLAN = Broadcast Domain = Logical Network (Subnet)

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 89

VLAN design must take into consideration the implementation of a hierarchical network
addressing scheme. The benefits of hierarchical addressing are:
 Ease of management and troubleshooting
 Minimization of errors
 Reduced number of routing table entries

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 89

Appendix: Troubleshooting FCoE

Network Traffic Types

Traffic type
must be
considered.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 90

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 90

Appendix: Troubleshooting FCoE

VLAN Operation

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 91

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 91

Appendix: Troubleshooting FCoE

802.1Q Trunking

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 92

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 92

Appendix: Troubleshooting FCoE

802.1Q Frame

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 93

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 93

Appendix: Troubleshooting FCoE

Interconnection Technologies

Technology Use
Fast Ethernet Connects end-user devices
to the access layer switch

Gigabit Connects an access switch

Ethernet to the distribution switch
and high-use servers
switches together
10 Gigabit Provides high-speed switch
Ethernet to switch links, backbones

EtherChannel Provides high-speed

switches to switch the link
and backbone with
redundancy

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 94

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 94

Appendix: Troubleshooting FCoE

Advantages of EtherChannel

 Logical aggregation of similar

links between switches
 Load-shares across links
 Viewed as one logical port
to STP
 Redundancy

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 95

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 95

Appendix: Troubleshooting FCoE

Redundant Topology

 Redundant topology eliminates single points of failure.

 Redundant topology causes broadcast storms, multiple
frame copies, and MAC address table instability problems.
© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 96

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 96

Appendix: Troubleshooting FCoE

Loop Resolution with STP

 Provides a loop-free redundant network topology

by placing certain ports in the blocking state
 Published in the IEEE 802.1D specification
 Enhanced with the PVST+ implementation

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 97

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 97

Appendix: Troubleshooting FCoE

Spanning-Tree Operation
 One root bridge per broadcast domain.
 One root port per nonroot bridge.
 One designated port per segment.
 Nondesignated ports are unused.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 98

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 98

Appendix: Troubleshooting FCoE

STP Root Bridge Selection

 BPDU (default = sent every 2 seconds)

 Root bridge = bridge with the lowest bridge ID

Bridge MAC
 Bridge ID = Priority Address

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 99

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 99

Appendix: Troubleshooting FCoE

Spanning-Tree Port States

Spanning tree transits each port through several different states:

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 100

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 100
Appendix: Troubleshooting FCoE

Describing PortFast

PortFast is configured on access ports, not trunk ports.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 101

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 101
Appendix: Troubleshooting FCoE

Spanning-Tree Operation Example

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 102

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 102
Appendix: Troubleshooting FCoE

Spanning-Tree Path Cost

Cost (Revised IEEE Cost (Previous IEEE

Link Speed
Specification) Specification)
10 Gb/s 2 1
1 Gb/s 4 1
100 Mb/s 19 10
10 Mb/s 100 100

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 103

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 103
Appendix: Troubleshooting FCoE

Spanning-Tree Recalculation

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 104

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 104
Appendix: Troubleshooting FCoE

Per VLAN Spanning Tree Plus

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 105

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 105
Appendix: Troubleshooting FCoE

PVST+ Extended Bridge ID

Bridge ID Without the

Extended System ID

Extended Bridge ID
with System ID

System ID = VLAN

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 106

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 106
Appendix: Troubleshooting FCoE

Rapid Spanning Tree Protocol

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 107

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 107
Appendix: Troubleshooting FCoE

NEX-5020 Example

 Ethernet Port-channel 3
2/2

1/2

 VLAN 213
1 /4
1/3

 Veth 1/28 interface accessing VLAN 213

TM3
1/28

 VLAN 213 allowed across port-chan

Server3

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 108

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 108
Appendix: Troubleshooting FCoE

NEX-5020 Configuration Defaults

loopguard Spanning tree loopguard options
mode Spanning Tree operating mode
mst Multiple spanning tree configuration
pathcost Spanning tree pathcost options
port Spanning tree port options
vlan VLAN Switch Spanning Trees

TM3# show spanning-tree summary

Switch is in rapid-pvst mode
Root bridge for: none
Port Type Default is disable
Edge Port [PortFast] BPDU Guard Default is disabled
Edge Port [PortFast] BPDU Filter Default is disabled
Bridge Assurance is disabled
Loopguard Default is disabled
Pathcost method used is short

Name Blocking Listening Learning Forwarding STP Active

--------------------------------------------- -------- --------- -------- ---------- ----------
VLAN0001 0 0 0 2 2
VLAN0213 0 0 0 3 3
--------------------------------------------- -------- --------- -------- ---------- ----------
2 vlans 0 0 0 5 5

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 109

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 109
Appendix: Troubleshooting FCoE

Spanning-tree outputs

POD2# show spanning-tree ?

<CR>
> Redirect it to a file
active Report on active interfaces only
blockedports Show blocked ports
bridge Status and configuration of this bridge
brief Brief summary of interface information
detail Detailed information
inconsistentports Show inconsistent ports
interface Spanning Tree interface status and configuration
internal Show internal STP information
mst Multiple spanning trees
pathcost Show Spanning pathcost options
root Status and configuration of the root bridge
summary Summary of port states
vlan VLAN Switch Spanning Trees
| Pipe command output to filter

POD2#

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 110

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 110
 Appendix: Troubleshooting FCoE

Show Spanning Tree

Spanning tree
protocol RSTP
TM2# sh spanning-tree vlan 212

VLAN0212
Spanning tree enabled protocol rstp
Root ID Priority 32768
Address 0011.bc52.20d4
Cost 1
Port 4097 (port-channel2)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32980 (priority 32768 sys-id-ext 212)

Address 000d.ec6d.97c1
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type

---------------- ---- --- --------- -------- --------------------------------
Port-channel 2
Po2 Root FWD 1 128.4097 P2p (root)
veth1/1 Desg FWD 2 128.129 Edge P2p

Access
members

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 111

Also: show tech-support stp

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 111
Appendix: Troubleshooting FCoE

Ethernet Interface Counters

switch# show interface eth1/21
Ethernet1/21 is up switch# show interface veth2/1
Hardware is 10000 Ethernet, address is vethernet2/1 is up
000d.ec6d.84dc (bia 000d.ec6d.84dc)
Hardware is VEthernet
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
MTU 1500 bytes, BW 0 Kbit, DLY 0 usec,
reliability 255/255, txload 1/255, rxload 1/255
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
Encapsulation ARPA
Port mode is access
Port mode is access
full-duplex, 10000 Mb/s
full-duplex, 10000 Mb/s
Input flow-control is off, output flow-control is
off Input flow-control is off, output flow-control is off
5 minute input rate 22203 bytes/sec, 346 packets/sec 5 minute input rate 640761 bytes/sec, 10000 packets/sec
5 minute output rate 640597 bytes/sec, 10000 5 minute output rate 34 bytes/sec, 0 packets/sec
packets/sec Rx
Rx 436231536 Input Packets 27918818304 Bytes
16501327 Input Packets 9 Unicast Packets 16500923 Tx
Multicast Packets
395 Broadcast Packets 0 Jumbo Packets 0 Storm 12844278 Output Packets 1079015571 Bytes
Suppression Packets
1056159080 Bytes switch# show int ethernet 1/21 counters detailed all
0 No buffer 0 runt 0 crc 0 ecc
0 Overrun 0 Underrun 0 Ignored 0 Bad etype drop
0 Bad proto drop 0 If down drop 0 Collision
0 Late collision 0 Lost carrier 0 No carrier
0 Babble
Tx
433943286 Output Packets 26171 Multicast Packets
0 Broadcast Packets 0 Jumbo Packets
27772499094 Bytes
0 Ouput errors
16499333 Rx pause 0 Tx pause 0 Reset

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 112

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 112
Appendix: Troubleshooting FCoE

Monitoring PAUSE frame counters

switch# show int ethernet 1/5 priority-flow-control
--------------------------------------------------------------------------------
Port Mode Oper RxPPP TxPPP
--------------------------------------------------------------------------------
Eth1/5 auto on 2967222 0
PFC was negotiated to
switch# show interface ethernet 1/6 flowcontrol enabled (on)

--------------------------------------------------------------------------------
Port Send FlowControl Receive FlowControl RxPause TxPause
admin oper admin oper
--------------------------------------------------------------------------------
Eth1/5 off off off off 3127212 0

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 113

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 113
Appendix: Troubleshooting FCoE

Debugging Multicast Forwarding

 Configuration
 Default behavior enables IGMP snooping on all VLANs
 Static (SG) or layer2 multicast entries can be configured

 Showing multicast entries in software and hardware

– show ip igmp snooping <options>: shows groups, router-ports, etc.
– show mac-address-table multicast: shows SG entries as programmed in
the station table; ‘fib’ option to the command shows hardware entries
switch# show mac-address-table multicast fib
VLAN MAC Address Type Ports
---------+-------------------------+---------+------------------------------
1 1111.0000.0000 user Eth1/31 Eth3/1
FIB Ports Eth1/31 Eth3/1  read from hw
Total Multicast MAC Addresses: 1

– System installs some default multicast entries to forward IGMP multicast

entries only to router-ports (as opposed to flood them). Multicast index
for these entries should never change.
© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 114

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 114
Appendix: Troubleshooting FCoE

Debugging Multicast Forwarding (cont..)

 show platform fwm info sg, oifl: dumps SG entry or an output
interface list (oifl) corresponding to it. VLAN flood lists can be
dumped using oifl routines.

switch# show plat fwm in sg 0.0.0.0 0.0.0.0 1  router group entry on VLAN 1 has S, G as
(0.0.0.0, 0.0.0.0)s
sg vlan 1.1 sgp sip 0.0.0.0 gip 0.0.0.0: vlan 1.1 pss_flags 2
sg vlan 1.1 sgp sip 0.0.0.0 gip 0.0.0.0: oifl idx 1
sg vlan 1.1 sgp sip 0.0.0.0 gip 0.0.0.0: iod list -
sg vlan 1.1 sgp sip 0.0.0.0 gip 0.0.0.0 oifl 1: vdc 1 gpinif 0, iods
sg vlan 1.1 sgp sip 0.0.0.0 gip 0.0.0.0 oifl 1: max_iod 4, ref count 58 num_oifs 0
sg vlan 1.1 sgp sip 0.0.0.0 gip 0.0.0.0 oifl 1: if_indexs -
sg vlan 1.1 sgp sip 0.0.0.0 gip 0.0.0.0 oifl 1 pd: mcast Idx 1 ifmap idx 1 (ref 1)

Other useful forwarding multicast commands

switch# show platform fwm info vlan 1 all_sgs: lists all SG entries in a given VLAN
switch# show platform fwm info vlan 1 all_macgs: lists all MAC group entries in a given VLAN
switch# show plat fwm info vdc 1: lists all oifls in a given vdc (Ethernet or FC)
switch# show plat fwm info oifl <oifl_index>: lists all interfaces in an oifl from an oifl
index (from vlan, vdc, SG, or macg dump)

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 115

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 115
Appendix: Troubleshooting FCoE

Other Diagnostic CLIs

 debug platform hardware peek: dumps the content of a specific hardware table entry
 debug hardware internal poke: modifies the content of a specific hardware table entry [Be careful!]
 show platform fwm info <options>: various options to dump software structures indicating the intended
hardware programming for Ethernet and FC forwarding.

 show platform fcfib <options>: various options to dump software structures and fib database for FC
forwarding.

 show interface <if> counters storm-control: storm control related drops

 show system internal mts: interprocess communication details (help find out leaking MTS buffers, or
outstanding responses, etc.)

 ‘show platform software <options>’ and ‘show system internal <options>’: software
internal information of various software processes (help find out internal finite state machine details, event
history, memory allocation statistics, structures dump, global and other debug information)

 ‘debug <options>’, ‘debug platform <options>’, ‘debug hardware internal

<options>’: commands to turn on process debugs, specific backdoor options/overrides, or hardware/asic
details.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 116

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 116
Appendix: Troubleshooting FCoE

Appendix D

Troubleshooting NEX-5020 Hardware

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 117

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 117
 Appendix: Troubleshooting FCoE

Hardware Architecture

Gatos ASIC
is the “Unified
Port Controller”

Altos ASIC
is the “Unified
Crossbar Fabric”

Eth 3 & 4
Inband lo-hi
© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 118

Let’s look at the internal hardware architecture of the system, It is composed of

 A Supervisor subsystem
 A Unified Crossbar Fabric ASIC
 A Set of Unified Port Controller ASICs
The supervisor subsystems is composed of Intel Sossamap processor with Intel South bridge,
memory components (NVRAM, FLASH, DRAM), dual management ports for out of band
management, console port. The supervisor is responsible for control plane operations and run
NXOS on Intel CPU.
Each Unified Port Controller manages 4 ports at 10GE, and makes forwarding decisions for the
packets received on those ports. Once a forwarding decision is made, packets are queued in
Virtual Output Queues, waiting to be granted access to the Unified Crossbar Fabric.
The Unified Crossbar Fabric is a single-stage, high performance 58x58 non-blocking crossbar
with integrated scheduler. The Crossbar provides the interconnectivity between input ports and
output ports with a total switching capacity of 1.4 Tbps. The Unified Crossbar Fabric internally
connects each 10GE interface via fabric interfaces running at 12Gbps. This allow 20% over
speed.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 118
 Appendix: Troubleshooting FCoE

Troubleshooting/viewing “sfpInvalid” Status

switch# show logging | grep 1/7

2005 Jul 1 16:07:41 switch %ETHPORT-3-IF_UNSUPPORTED_TRANSCEIVER: Transceiver
for interface Ethernet1/7 is not supported
2005 Jul 1 16:07:41 switch %ETHPORT-3-IF_UNSUPPORTED_TRANSCEIVER_VENDOR:
Transceiver vendor for interface Ethernet1/7 is not supported

switch# show system internal ethpm event-history errors | grep 1/7

[102] Ifindex (Ethernet1/7)0x2006000, SFP security check: CRC failed, rcvd
CRC 0x0 calculated CRC 0xe9777080

Other method:
switch# show hardware internal gatos port ethernet 1/7 xcvr regs | begin "SFP"

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 119

Troubleshooting port not coming up issues

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 119
 Appendix: Troubleshooting FCoE

Troubleshooting 10GBE link not coming up

Check PHY driver software link state:
POD2# sh hardware internal gatos port ethernet 1/13 xcvr info |grep State
State: DN
POD2# sh hardware internal gatos port ethernet 1/14 xcvr info |grep State
State: UP

Check MAC local/remote fault:

POD2# sh hardware internal gatos port ethernet 1/13

Gatos port xgb1/13 card-config info:

if_index : 0x200c000
logical_port : 12
front_port : 12
gatos instance : 8  ASIC #
mac port : 1
state : va swi pos hw1 hw2 hwf
gatos eport : 1
gatos iport : 1
fabric_port : 4
fab_voq_range : [0-7]
port_type : 10gbe(5)
fw_instance : 1
xcvr_ctrl_gatos : 8
xcvr_ctrl_port : 0
serdes lane 0 : rx_opp_lane:0(pol 0) tx_opp_lane:0(pol 0)
serdes lane 1 : rx_opp_lane:2(pol 0) tx_opp_lane:2(pol 1)
serdes lane 2 : rx_opp_lane:3(pol 0) tx_opp_lane:1(pol 0)
serdes lane 3 : rx_opp_lane:1(pol 1) tx_opp_lane:3(pol 1)
ln:0, rx_eq: 0x1, tx_swing: 0x2, tx_de: 0x8
ln:1, rx_eq: 0x1, tx_swing: 0x2, tx_de: 0x8
ln:2, rx_eq: 0x1, tx_swing: 0x2, tx_de: 0x8
--More--
© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 120

Troubleshooting port not coming up issues

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 120
 Appendix: Troubleshooting FCoE

Troubleshooting 10GBE link not coming up (cont.,)

Check MAC local/remote fault (cont)

switch# sh hardware internal gatos asic 3 registers match .*mac_status

Gatos 3 register contents:
Register Name | Offset | Value
-------------------------------------------------------+----------+-----------
mm_xgbe_wrapper_STA_mac_status0 | 0x3c804c | 0x1
mm_xgbe_wrapper_STA_mac_status1 | 0x3c904c | 0
mm_xgbe_wrapper_STA_mac_status2 | 0x3ca04c | 0x1
mm_xgbe_wrapper_STA_mac_status3 | 0x3cb04c | 0
Done.

Value 0x1 = LOCAL fault

Value 0x2 = REMOTE fault
Value 0x3 = BOTH faults

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 121

Troubleshooting port not coming up issues

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 121
 Appendix: Troubleshooting FCoE

Ethernet Interface Counters

switch# show interface eth1/21 switch# show interface veth2/1
Ethernet1/21 is up vethernet2/1 is up
Hardware is 10000 Ethernet, address is Hardware is VEthernet
000d.ec6d.84dc (bia 000d.ec6d.84dc)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec, MTU 1500 bytes, BW 0 Kbit, DLY 0 usec,
reliability 255/255, txload 1/255, rxload 1/255 reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA Encapsulation ARPA
Port mode is access Port mode is access
full-duplex, 10000 Mb/s full-duplex, 10000 Mb/s
Input flow-control is off, output flow-control is Input flow-control is off, output flow-control is
off off
5 minute input rate 22203 bytes/sec, 346 packets/sec 5 minute input rate 640761 bytes/sec, 10000
packets/sec
5 minute output rate 640597 bytes/sec, 10000 5 minute output rate 34 bytes/sec, 0 packets/sec
packets/sec
Rx Rx
16501327 Input Packets 9 Unicast Packets 16500923 436231536 Input Packets 27918818304 Bytes
Multicast Packets Tx
395 Broadcast Packets 0 Jumbo Packets 0 Storm 12844278 Output Packets 1079015571 Bytes
Suppression Packets
1056159080 Bytes
0 No buffer 0 runt 0 crc 0 ecc switch# show int ethernet 1/21 counters detailed all
0 Overrun 0 Underrun 0 Ignored 0 Bad etype drop
0 Bad proto drop 0 If down drop 0 Collision
0 Late collision 0 Lost carrier 0 No carrier
0 Babble
Tx No Buf credits on vfc because of FCoE
433943286 Output Packets 26171 Multicast Packets
0 Broadcast Packets 0 Jumbo Packets
27772499094 Bytes
0 Ouput errors
16499333 Rx pause 0 Tx pause 0 Reset

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 122

Troubleshooting Port MAC

Virtual interface is a concept in Forwarding (LU and AC). MACs and Packet Editing only
understand physical interface.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 122
 Appendix: Troubleshooting FCoE

Troubleshooting 10Gbe MAC

switch# show hardware internal gatos port ethernet 1/1 | begin MAC
MAC statistics:
MAC Statistic | Value
-------------------------------+-----------------------------------
TX_PKT_SIZE_LT_64 | 0
TX_PKT_SIZE_IS_64 | 0xd33
TX_PKT_SIZE_IS_65_TO_127 | 0x1b78
TX_PKT_SIZE_IS_128_TO_255 | 0x9c
TX_PKT_SIZE_IS_256_TO_511 | 0xe
TX_PKT_SIZE_IS_512_TO_1023 | 0
TX_PKT_SIZE_IS_1024_TO_1518 | 0
TX_PKT_SIZE_IS_1519_TO_2047 | 0
TX_PKT_SIZE_IS_2048_TO_4095 | 0
TX_PKT_SIZE_IS_4095_TO_8191 | 0
TX_PKT_SIZE_IS_8192_TO_9216 | 0
TX_PKT_SIZE_GT_9216 | 0
TX_PKT_TOTAL | 0x2955
TX_PKT_OCTETS | 0xb6f87
TX_PKT_GOOD | 0x2955
TX_PKT_UCAST | 0
TX_PKT_MCAST | 0x28c9
TX_PKT_BCAST | 0x8c
TX_PKT_VLAN | 0xd1f
TX_PKT_802.3x_PAUSE | 0
TX_PKT_PER_PRIORITY_PAUSE | 0

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 123

Troubleshooting Port MAC

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 123
Appendix: Troubleshooting FCoE

Troubleshooting Gatos Fatal Errors

 Slow draining (< 5 Mbps for 10s straight) ports or stuck ports are
error disabled. shut / no shut is required to bring the port back up.

 Other fatal error interrupts also cause the corresponding ports to be

brought down (hwFailure status)

 The following command helps identify the error

switch# show hardware internal gatos event-history errors | grep error_interrupt

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 124

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 124
Appendix: Troubleshooting FCoE

Gatos Debugging
switch# show hardware internal gatos

Gatos ASIC Info:

num(u)|slo|pts|state |kaddr |uaddr |bus:fn|irq(hex)|MMmode[0:3]|diag
------+---+---+------+--------+--------+------+--------+-----------+----
0(0) |0x1|4 |ff |f8c00000|b6101000|9:0 |32 |xg:xg:xg:xg|pass
1(1) |0x1|4 |ff |f8800000|b5d01000|10:0 |3a |xg:xg:xg:xg|pass
2(2) |0x1|4 |ff |f8400000|b5901000|11:0 |42 |xg:xg:xg:xg|pass
3(3) |0x1|4 |ff |f8000000|b5501000|12:0 |4a |xg:xg:xg:xg|pass
4(4) |0x5|4 |ff |f7800000|b5101000|15:0 |52 |xg:xg:xg:xg|pass
5(5) |0x5|5 |ff |f7400000|b4d01000|16:0 |5a |sp:xg:xg:xg|pass
6(6) |0x5|3 |ff |f7000000|b4901000|17:0 |62 |xg:xg:xg:na|pass
7(7) |0x1|4 |ff |f6400000|b4501000|21:0 |6a |xg:xg:xg:xg|pass
8(8) |0x1|4 |ff |f6000000|b4101000|22:0 |72 |xg:xg:xg:xg|pass
9(9) |0x1|4 |ff |f5c00000|b3d01000|23:0 |7a |xg:xg:xg:xg|pass
10(10)|0x1|4 |ff |f5800000|b3901000|24:0 |82 |xg:xg:xg:xg|pass
11(11)|0x3|4 |ff |f9c00000|b3501000|27:0 |8a |xg:xg:xg:xg|pass
12(12)|0x3|3 |ff |f9800000|b3101000|28:0 |92 |na:xg:xg:xg|pass
13(13)|0x3|5 |ff |f9400000|b2d01000|29:0 |9a |mf:mf:xg:na|pass

switch# show hardware internal gatos asic 0 registers match .*^pc.*

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 125

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 125
Appendix: Troubleshooting FCoE

Dumping Gatos Registers

switch# show hardware internal gatos asic 0 registers match <reg-exp>

STA: Status Registers

CNT: Counters
INT: Interrupt Registers

e.g. Dump QS counters on Gatos 0

d14-switch-1# show hardware internal gatos asic 13 registers match .*qs.*CNT.*

Gatos 13 register contents:

Register Name | Offset | Value
-------------------------------------------------------+----------+-----------
qsCNT_bm0_qs0_ig_eqsof0 | 0x3e86dc | 0
qsCNT_bm0_qs0_ig_eqsof1 | 0x3ea6dc | 0x198f386d
qsCNT_bm0_qs0_ig_eqsof_m0 | 0x3e86e0 | 0
qsCNT_bm0_qs0_ig_eqsof_m1 | 0x3ea6e0 | 0xaa3aff
qsCNT_bm0_qs0_ig_eqeof0 | 0x3e86e4 | 0
qsCNT_bm0_qs0_ig_eqeof1 | 0x3ea6e4 | 0x198f38ad
qsCNT_qs_fi0_ig_request0 | 0x3e86e8 | 0
qsCNT_qs_fi0_ig_request1 | 0x3ea6e8 | 0x19855343

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 126

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 126
Appendix: Troubleshooting FCoE

Monitoring Altos Interrupt Counters

switch# show hardware internal altos counters interrupts all

Altos interrupt statistics:

Interrupt name |Count |ThresRch|ThresCnt|Ivls
-----------------------------------------------+--------+--------+--------+----
ALT_FIC2_INT_3_FI_tx_pkt_err |6 |2 |0 |0
ALT_FIC2_INT_3_FI_tx_pkt_eot_err |3 |1 |0 |0
ALT_FIC13_INT_0_FI_tx_pkt_err |6 |2 |0 |0
ALT_FIC13_INT_0_FI_tx_pkt_eot_err |3 |1 |0 |0
ALT_FIC13_INT_1_FI_rx_pkt_crc_err |6 |2 |0 |0
ALT_FIC13_INT_1_FI_rx_pkt_err |3 |1 |0 |0
Done.

Note: shows only non-zero interrupt counters. Includes fatal and non fatal error interrupts

switch# show hardware internal altos event-history errors

Note: Two altos ports are unused and hence always report errors

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 127

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 127
 Appendix: Troubleshooting FCoE

LU Packet Drops - Basics

Virtual interface is a concept in Forwarding (LU and AC). MACs and
Packet Editing only understand physical interface.
 Drops: commonly occurring situations can be attributed to a configuration error
or a software bug
– usually leaves a clue in drop counters.
 Discards: often a problem with the frame or an error condition asserted via an
interrupt
– requires observing interrupts.
 Drops/Discards are counted for each interface (physical Ethernet and FC, or in
IOC mode for vethernet and vfc)
 The reason for drops/discards are not specific to a port (ASIC wide)
 A true indication of a drop is the drop counter incrementing on an interface i.e.
a frame may satisfy a drop condition, but may not be dropped eventually
– for example an LACP frame directed towards CPU might not satisfy VLAN
membership check and the drop condition is ignored.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 128

Virtual interface is a concept in Forwarding (LU and AC). MACs and Packet Editing only
understand physical interface.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 128
 Appendix: Troubleshooting FCoE

LU Packet Drops – Sample output

switch# show platform fwm info pif ether1/22  FC or ethernet interfaces
dump pif info: ifindex 0x2015000 dump_all 0 verbose 1
Eth1/22: slot 0 port 21 state 0x0 fwimpd ctx 0x85a4744
Eth1/22: iftype 0x1 encap 0x1 #lifs 1 fwimpd ctx 0x85cb9a4
Eth1/22 pd: lif_entries 2 if_map_idx 26 lif_blk 10-11 if_lid 7 if_fcoe_lid 10
Eth1/22 pd: vif if alloc bmp - 0-1
Eth1/22 pd: lif_tree - 0 1 Config:
Eth1/22 pd: slot 0 logical port num 21 gatos_num 9 fwm_inst 3 fc 0 interface vig 1
bind interface Ethernet1/22
Eth1/22 pd: pif_type 'data eth'(1) hw_present 1 port map idx 26
interface vethernet1/1
Eth1/22 pd: fabric a info: voq 0-7 port_id 15 connected 1 up 1
interface vfc1/1
Eth1/22 pd: fabric b info: voq 0-7 port_id 15 connected 1 up 1 no shutdown
Eth1/22 pd: subported 0 primary 1
Eth1/22 pd: sup_src_dst_if 7 lif_blk 0-1
Eth1/22 pd: policer info: uc (sel 11) mc (sel 10) bc (sel 9)
Look for drops and discards;
Eth1/22 pd: tx stats: bytes 566251 frames 6647 discard 0 drop 0
packets may be dropped in QS/BM
Eth1/22 pd: rx stats: bytes 433136 frames 2354 discard 0 drop 0
if there are no drops here. Note
Eth1/22 pd fcoe: tx stats: bytes 12400 frames 63 discard 0 drop 0
that the drop and discard counters
Eth1/22 pd fcoe: rx stats: bytes 10856 frames 59 discard 0 drop 0
are separate for vethernet and vfc
switch(config-if)# show platform fwm info gatos-errors 11
Printing non zero Gatos error registers:
DROP_SRC_VLAN_MBR: res0 = 39 res1 = 0 This command may indicate reason
DROP_SRC_MASK_TO_NULL: res0 = 63 res1 = 0 for packet drop (must be correlated with
DROP_INGRESS_ACL: res0 = 1305 res1 = 0 above command’s output

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 129

N5k# show platform fwm info pif ?

<0-4294967295> Interface Index
all all instances
ethernet Ethernet IEEE 802.3z
fc Fibre Channel interface
port-channel Port Channel interface
san-port-channel SAN Port Channel interface
vethernet Virtual ethernet interface
vfc Virtual FC interface
vig Virtual interface group

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 129
 Appendix: Troubleshooting FCoE

LU Packet Drops: common reasons

• Only commonly occurred LU drops are mentioned in this section (from among 45 of them)
• A lot of time supervisor directed frames registers these drop conditions however ACL decision
overrides the behavior and the frame is not dropped eventually
Reason Explanation What to check
DROP_INGRESS_VLAN_XLATE Unable to translate external VLAN from VLAN ID in incoming frame; configuration; internal
incoming frame to an internal VLAN structures; fib information
DROP_EGRESS_VLAN_XLATE Unable to translate internal VLAN on an Configuration; internal structures; supervisor inband
outgoing frame to an external VLAN database
DROP_INGRESS_FW_PARSING_ERROR Could not parse an incoming frame Incoming frame; configuration
DROP_EGRESS_FW_PARSING_ERROR Could not parse an outgoing frame Ethanalyzer output; supervisor inband database
DROP_INGRESS_ACL Hit drop ACL on ingress ACL configuration; hardware ACL entries; ACL
statistics; software ACL information; LIF/VLAN labels
DROP_EGRESS_ACL Hit drop ACL on egress ACL configuration; hardware ACL entries; ACL
statistics; software ACL information
DROP_SRC_VLAN_MBR Source interface is not a member of Interface, VLAN configuration; VLAN, LIF, and PIF
incoming/default VLAN; this error also software and hardware structures
indicate STP ‘blocking state’ drops
DROP_VLAN_MASK_TO_NULL Destination address(s) not part of the VLAN Software and hardware FIB; VLAN, LIF, and PIF
(very likely due to software bug or the frame softwrae and hardware structures
was destined to the supervisor)
DROP_SRC_MASK_TO_NULL Frame tried to go out on the interface it came Shared ethernet network (connected via hub);
in on. Misconfigured Ixia streams; Software and hardware
FIB; VLAN, LIF, and PIF softwrae and hardware
structures
DROP_SUP_SRC_MASK_TO_NULL Supervisor originated frame was dropped: Supervisor inband database; ethanalyzer output; peek
source interface encoding prevents it from sup_source_mask table in hardware
being forwarded to any valid port

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 130

Virtual interface is a concept in Forwarding (LU and AC). MACs and Packet Editing only
understand physical interface.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 130
 Appendix: Troubleshooting FCoE

LU Packet Drops: common reasons (cont)

• FC forwarding related drops or Fabric Selection related drops

Reason Explanation What to check

DROP_INVALID_FCF_BYPASS Frame is required to have FCF MAC as Menlo or FCoE stack’s understanding of FCF MAC
destination however it did not; can happen for address; global FCF MAC from supervisor inband
control frames. database; hardware FCF mac address register

DROP_L2_BIND_CHK_FAILURE Frame was received with an unexpected source CNA logs, tcpdump on CNA to know the frame’s source
MAC address (FC OUI | SID isn’t correct) MAC address; software and hardware LIF dump; software
FLOGI database; FCFIB MAC addresses (show mac-
address-table internal)

DROP_FCF_SW_VSAN_IDX_MISS VSAN index table lookup resulted null VSAN configuration; softwrae and hardware vsan index
tables

DROP_FCF_SW_DOMAIN_IDX_MISS Domain index table lookup resulted null Domain configuration; software and hardware domain
index tables

DROP_FCF_SW_TBL_MISS Unable to find a FC path-select table for Software and hardware vsan index table, domain index
(domain, vsan) tables, and fc path-select tables

DROP_FCF_LOCAL_DOMAIN_MISS STM lookup failed for traffic destined to a VFC Software and hardware mac-address-table
DROP_NO_FABRIC_SELECTED Frame could not be forwarded to the fabric due Software and hardware PIF, LIF, Local Id Table, Multicast
"DROP_FABRIC_SELECT_TO_NULL to improper fabric selection (in PIF, LIF, Local Index Table.
Index Table, Multicast Index Table, etc.) or if the
"DROP_NO_FABRIC_CONNECTIVITY
destination was a reserved ‘drop index’.
DROP_HIT_DROP_PORT_MAP_IDX

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 131

Virtual interface is a concept in Forwarding (LU and AC). MACs and Packet Editing only
understand physical interface.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 131
 Appendix: Troubleshooting FCoE

LU Packet Drops: common reasons (cont)

• End Host Mode (Ethernet and FC) related drops in forwarding

Reason Explanation What to check

DROP_RPF_UNICAST_FAILURE End host mode: unicast frame for a destination If an error is suspected, software and hardware PIF, LIF
received on a non pinned interface; may be a settings for pbp_index; pinning information.
normal situation
DROP_RPF_MULTICAST_FAILURE End host mode: multicast frame for a destination If an error is suspected, software and hardware PIF, LIF
received on a non designated interface; may be settings for pbp_index; pinning information.
a normal situation

DROP_PINNED_DEJA_VU_FAILURE End host mode: a frame with a source MAC of a If an error is suspected, software and hardware PIF, LIF
server interface received on a border interface settings for pbp_index; pinning information.
DROP_ST_MISS_NPIV_BORDER Station table entry not found when forwarding Software and hardware mac-address-table
traffic towards VFC in NPV mode.
DROP_PINNED_BORDER_TO_NULL There is no pinned interface; typically this would Software and hardware PIF, LIF settings for pbp_index;
not happen because a server interface is pinning information.
brought down if it can not be pinned to a border
interface

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 132

Virtual interface is a concept in Forwarding (LU and AC). MACs and Packet Editing only
understand physical interface.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 132
Appendix: Troubleshooting FCoE

LU, FW, and AC Discards

Interrupt counters: Interrupts are asserted by Gatos to indicate an event that require
software attention. They could represent an error (discard or drop condition) or a normal
condition (dma completion). Looking for asserted interrupts and correlating the
incrementing counters for various interrupts with pattern of occurrence of error can help
narrow down the cause of an error.

switch# show hardware internal gatos asic 13 interrupt match lu  ‘lu’ can be substituted with
Interrupt name |Status ‘fw’ or ‘ac’
----------------------------------------------------------------+------
gat_lu_lkup1_INT_VlanXlateKeyTable_tcam0_full_f |Set
gat_lu_lkup1_INT_VlanXlateKeyTable_tcam0_write_on_full_f |Set
gat_lu_lkup1_INT_VlanXlateKeyTable_tcam1_full_f |Set
gat_lu_lkup1_INT_VlanXlateKeyTable_tcam1_write_on_full_f |Set
Done.

switch# show hardware internal gatos asic 13 counters interrupt match lu

Interrupt name |Count |ThresRch|ThresCnt|Ivls
-----------------------------------------------+--------+--------+--------+----
gat_lu_lkup1_INT_VlanXlateKeyTable_tcam0_full_f|887 |0 |8 |0
gat_lu_lkup1_INT_VlanXlateKeyTable_tcam0_write_|887 ll_f|0 |8 |0  watch for
gat_lu_lkup1_INT_VlanXlateKeyTable_tcam1_full_f|887 |0 |8 |0 incrementing counts;
gat_lu_lkup1_INT_VlanXlateKeyTable_tcam1_write_|887 ll_f|0 |8 |0 If an internal
gat_lu_lkup1_INT_VlanXlateKeyTable_tcam0_mhit_i|5cf |0 |8 |0 threshold is met the
gat_lu_lkup1_INT_VlanXlateKeyTable_tcam1_mhit_i|814 |0 |9 |0 interface or an ASIC
gat_lu_lkup1_INT_VlanXlateKeyTable_mhit_r |886 |0 |7 |0 may be disabled.
gat_lu_lkup3_INT_func_drop_fcf_sw_vsan_idx_miss|1 |0 |1 |0
gat_lu_lkup3_INT_func_drop_fcf_sw_domain_idx_mi|886 |0 |7 |0
gat_lu_lkup3_INT_func_drop_fcf_sw_tbl_miss_i |886 |0 |7 |0
gat_lu_lkup4_INT_func_lo_drop_no_fabric_selecte|886 |0 |7 |0
gat_lu_lkup4_INT_func_lo_drop_vlan_mask_to_null|886 |0 |7 |0
gat_lu_stm_INT_st_scan_complete |10ff |0 |1 |0
Done.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 133

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 133
Appendix: Troubleshooting FCoE

Learning and Aging - Basics

 Ingress and Egress ASIC learning

– Line rate on for all frames
– Facilitates distributed table population

 Ingress notifies Supervisor to develop database

 Supervisor pushes new addresses to all Gatos
– Adds entries if missed (due to hw learning speeds, or port-
channel)
– Re-enforces existing entries

 Supervisor queries tables to check for consistency

– Maintains aging state

 CPU removes entries that are obsolete

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 134

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 134
Appendix: Troubleshooting FCoE

Interface, VLANs, and STP - Basics

 VSANs are represented no different from VLANs

 except they are identified to belong to a different logical VLAN space

 Interface VLAN membership

 Ingress and Egress membership checks: both done on ingress Gatos
 Ethernet VLAN flood lists: unknown unicast, multicast, and broadcast
 FC flood memberships lists are NULL: all traffic is expected to match a
forwarding entry

 STP state is enforced using VLAN membership checks

 For FC interfaces: STP state logic abstracts enabling or
disabling forwarding-state on different (interface, VSAN) on a TE
port based on exchange with remote end.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 135

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 135
Appendix: Troubleshooting FCoE

FC Forwarding - Basics

 VSAN and Interfaces are treated similar to Ethernet

– Ingress and Egress membership checks

 Incoming FC traffic is converted/forwarded like FCoE; SA is set

from interface settings or FC_OUI | SID; DA is set as FC_OUI | DID or
FCF_MAC.
 SA Lookup: SA lookup is done to do binding checks
 DA Lookup: Traffic towards FCF MAC is looked up in FC multipath
table for FC RIB based forwarding.
 FC Path Selection: derived from multipath index (from domain, vsan)
and target domain
 NPV Mode: SA or DA is set to pinned border ports MAC based.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 136

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 136
Appendix: Troubleshooting FCoE

Interface, VLANs, and STP - Debugging

Configuration
– vlan 2-10
– interface Ethernet1/22
– switchport mode trunk

Checking VLAN/VSAN’s forwarding information: following commands displays forwarding

information, including interface membership, hardware indexes, etc. Note that term vdc in the
debugs has no consequence of what VDC in nexus 7k is used for. It merely represents a
demultiplexer for FC VSANs and Ethernet VLANs. Ethernet VDC: 1, FC VDC: 14.

switch(config-if)# show platform fwm info vlan 7  this command has more options to
look at entities that are per VLAN
vlan 1.7: dump_all 0 verbose 1
vlan 1.7: state 0x0 num_lifs 1 fwm_ptr 0x85da32c pd_ptr 0x85daef4
vlan 1.7: lif list - Eth1/22
vlan 1.7 fwm: age 300, cfg age 300
vlan 1.7 fwm: mac count 0, ipg count 0, macg count 0 vlanif count 1
vlan 1.7 fwm: dyn_macs 0 fwd_vlanifs 1 igmp_sgs 1
vlan 1.7 fwm: pss_flags 0 igmp snoop: 1
vlan 1.7 pd: int-vlan 10 state table idx 10 vacl_label 128
vlan 1.7 pd: flood refs - unicast 7 broadcast 7 multicast 7
vlan 1.7 pd: gatos 9 lif_members: 0
vlan 1.7 pd: if members (VLAN egress masks): 1-2,26

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 137

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 137
Appendix: Troubleshooting FCoE

Interface, VLANs, and STP – Debugging (cont)

Checking Interface’s forwarding information: following commands displays forwarding

information, including VLAN membership, hardware indexes, etc. for an interface. An interface
is represented heirarchicaly as physical interface (pif) and logical interface (lif). Here is a sample
output of pif.

switch(config-if)# show plat fwm in pif ether 1/22

dump pif info: ifindex 0x2015000 dump_all 0 verbose 1
Eth1/22: slot 0 port 21 state 0x0 fwimpd ctx 0x85a48e4
Eth1/22: iftype 0x1 encap 0x1 #lifs 1 fwimpd ctx 0x85cb30c
Eth1/22 pd: lif_entries 1 if_map_idx 26 lif_blk 7-7 if_lid 7 if_fcoe_lid 0
Eth1/22 pd: vif if alloc bmp - 0
Eth1/22 pd: lif_tree - 0
Eth1/22 pd: slot 0 logical port num 21 gatos_num 9 fwm_inst 3 fc 0
Eth1/22 pd: pif_type 'data eth'(1) hw_present 1 port map idx 26
Eth1/22 pd: fabric a info: voq 0-7 port_id 15 connected 1 up 1
Eth1/22 pd: fabric b info: voq 0-7 port_id 15 connected 1 up 1
Eth1/22 pd: subported 0 primary 1
Eth1/22 pd: sup_src_dst_if 7 lif_blk 0-0
Eth1/22 pd: policer info: uc (sel 11) mc (sel 10) bc (sel 9)
Eth1/22 pd: tx stats: bytes 1579358 frames 18388 discard 0 drop 0
Eth1/22 pd: rx stats: bytes 876944 frames 4766 discard 0 drop 0

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 138

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 138
Appendix: Troubleshooting FCoE

Interface, VLANs, and STP – Debugging (cont)

Checking Interface’s forwarding information: checking lif information

switch(config-if)# show plat fwm in lif ether 1/22

dump lif info: if_index 0x2015000 dump_all 0 verbose 1
Eth1/22: state 0x2 if_ord 59 if 'pif Eth1/22' pinned-if 'none'
Eth1/22: fwmpi ctx 0x85a5e9c fwimpd ctx 0x85cb384
Eth1/22: misc cfg - trustFrameVlan 1 untagDefaultVlan 1 isl2Trunk 1 vdc_id 1
Eth1/22: misc cfg - default_vlan 1 fc_port_mode: 0x0
Eth1/22: misc cfg - learn override disable: 0 prio tag fcoe: 0
Eth1/22: num_sids (npiv) 0
Eth1/22: vlan list - 1 2 3 4 5 6 7 8 9 10
Eth1/22 fwm: if_index 2015000, num_macs 0, num_oifl_refs 2
Eth1/22 fwm: num_macs 0 block-unicast 0, block-multicast 0 “Ether1/22 pd” prefix
Eth1/22 fwm: referring oifls - 6 7 indicates platform
Eth1/22 pd: gatos 9 pif_idx(Eth1/22) 6 lif_idx 0 dependent information
Eth1/22 pd: vif id 0 egress_mcast_lif 0 vif_id_alloced 1 vlan_xlate_index 17
Eth1/22 pd: pbp_idx 0 (alloced 0) gatos_lif_cfg (nil)
Eth1/22 pd: ingress bind check - l2-en 0 l3-en 0
Eth1/22 pd: fc zone check: ingress enabled 1 egress enabled 0
Eth1/22 pd: cos settings - trust 1 internal rewrite 0 external rewrite 0
Eth1/22 pd: default_ext_cos_de (cos, de)=0,0
Eth1/22 pd: int_to_ext_cos_map (cos, de): [0]=0,0 [1]=1,0 [2]=2,0 [3]=3,0 <snip>
Eth1/22 pd: ext_to_int_cos_map (cos, de): [0]=0,0 [1]=0,0 [2]=0,0 [3]=0,0 <snip>
Eth1/22 pd: ingress vlan_xlate location (vlan 1) - type 1, bucket 855, entry 0
Eth1/22 pd: ingress vlan_xlate location (vlan 1) -type 1, bucket 306, entry 0 <snip>
Eth1/22 pd: policer info: uc (en 0 100.00 pct) mc (en 0 100.00 pct) bc (en 0 100.00 pct)

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 139

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 139
Appendix: Troubleshooting FCoE

Debugging Supervisor Path

 Supervisor PIF/LIF information
– show platform fwm pif all | grep sup-eth (shows rx/tx statistics towards supervisor hi and low ports)
– show platform fwm lif all | grep sup-eth (shows lif, pif index values that can be used to dump hardware
information)

Packet Manager Client statistics: shows per interface/protocol statistics

of supervisor frames transmitted or received.
switch(config-if)# show system internal pktmgr client stp
Client: stp, uuid: 303, 2 filters
Filter 0: EthType 0x4242, Dmac 0180.c200.0000
Filter 0: EthType 0x010b, Snap 267, Dmac 0100.0ccc.cccd

Options: TO 0, Flags 0x1, AppId 0, Epid 0

Ctrl SAP: 171, Data SAP 177 (1)
Rx: 162217, Drop: 0, Tx: 485077, Drop: 0

show ip traffic: shows drops in IP stack if any

ethanalyzer: tool to capture supervisor frames as they are sent out or received (also dumps internal
headers)

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 140

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 140
Appendix: Troubleshooting FCoE

Debugging Multicast Forwarding

 Configuration
– Default behavior enables IGMP snooping on all VLANs
– Static (SG) or layer2 multicast entries can be configured
 Showing multicast entries in software and hardware
– show ip igmp snooping <options>: shows groups, router-ports, etc.
– show mac-address-table multicast: shows SG entries as programmed in
the station table; ‘fib’ option to the command shows hardware entries

switch# show mac-address-table multicast fib

VLAN MAC Address Type Ports
---------+-----------------+-------+------------------------------
1 1111.0000.0000 user Eth1/31 Eth3/1
FIB Ports Eth1/31 Eth3/1  read from hw
Total Multicast MAC Addresses: 1

– System installs some default multicast entries to forward IGMP multicast

entries only to router-ports (as opposed to flood them). Multicast index for
these entries should never change.
© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 141

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 141
Appendix: Troubleshooting FCoE

Debugging Multicast Forwarding (cont..)

show platform fwm info sg, oifl: dumps SG entry or an output interface list (oifl) corresponding to it. VLAN flood lists
can be dumped using oifl routines.

switch# show plat fwm in sg 0.0.0.0 0.0.0.0 1  router group entry on VLAN 1 has
S, G as (0.0.0.0, 0.0.0.0)s
sg vlan 1.1 sgp sip 0.0.0.0 gip 0.0.0.0: vlan 1.1 pss_flags 2
sg vlan 1.1 sgp sip 0.0.0.0 gip 0.0.0.0: oifl idx 1
sg vlan 1.1 sgp sip 0.0.0.0 gip 0.0.0.0: iod list -
sg vlan 1.1 sgp sip 0.0.0.0 gip 0.0.0.0 oifl 1: vdc 1 gpinif 0, iods
sg vlan 1.1 sgp sip 0.0.0.0 gip 0.0.0.0 oifl 1: max_iod 4, ref count 58 num_oifs 0
sg vlan 1.1 sgp sip 0.0.0.0 gip 0.0.0.0 oifl 1: if_indexs -
sg vlan 1.1 sgp sip 0.0.0.0 gip 0.0.0.0 oifl 1 pd: mcast Idx 1 ifmap idx 1 (ref 1)

Other useful forwarding multicast commands

switch# show platform fwm info vlan 1 all_sgs: lists all SG entries in a given VLAN

switch# show platform fwm info vlan 1 all_macgs: lists all MAC group entries in a given VLAN

switch# show plat fwm info vdc 1: lists all oifls in a given vdc (Ethernet or FC)

switch# show plat fwm info oifl <oifl_index> : lists all interfaces in an oifl from an oifl index (from vlan,
vdc, SG, or macg dump)

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 142

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 142
Appendix: Troubleshooting FCoE

Debugging Port Channel Forwarding

show platform fwm info pc

switch(config-if)# show port-channel summary

Flags: D - down U - up in port-channel
I - Individual S - suspended
H - Hot-standby (LACP only)
R - Module-removed
-----------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
-----------------------------------------------------------------------------
11 Po11(U) Eth NONE Eth1/31(U) Eth3/1(U)

switch(config-if)# show plat fwm in pc po11

dump pc info: if_index 369098762 dump_all 0 verbose 1
Po11: state 0x0 #pifs 2 fwimpd ctx 0x85a7a6c  indicates number of physical interfaces in the port-channel
Po11: hash params - l2_da 1 l2_sa 1 l3_da 1 l3_sa 1
Po11: hash params - l4_da 0 l4_sa 0 xor_sa_da 1 hash_elect 1
Po11: pif list - Eth1/31
Po11: pif list - Eth3/1
Po11: iftype 0x2 encap 0x1 #lifs 1 fwimpd ctx 0x858b7a4
Po11 pd: lif_entries 1 if_map_idx 112 lif_blk 6-6 if_lid 6 if_fcoe_lid 0
Po11 pd: vif if alloc bmp - 0
Po11 pd: lif_tree - 0
Po11 pd: bundle map table index 0 alloced = 1  bundle map_index identifies hardware table index for
the port-channel

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 143

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 143
Appendix: Troubleshooting FCoE

Debugging Port Channel Forwarding (cont)

show platform fwm info lif for port-channels
switch(config-if)# show plat fwm in pc po11
dump lif info: if_index 0x1600000a dump_all 0 verbose 1
Po11: state 0x2 if_ord 86 if 'pc Po11' pinned-if 'none'
Po11: fwmpi ctx 0x85a97c4 fwimpd ctx 0x858b894
Po11: misc cfg - trustFrameVlan 0 untagDefaultVlan 1 isl2Trunk 0 vdc_id 1
Po11: misc cfg - default_vlan 1 fc_port_mode: 0x0
Po11: misc cfg - learn override disable: 0 prio tag fcoe: 0
Po11: num_sids (npiv) 0
Po11: vlan list - 1
Po11 fwm: if_index 1600000a, num_macs 0, num_oifl_refs 1
Po11 fwm: num_macs 0 block-unicast 0, block-multicast 0
Po11 fwm: referring oifls - 12
Po11 pd: gatos 4 pif_idx(Eth1/31) 2 lif_idx 0  shows more than one lifs and pifs corresponding to the lif
Po11 pd: gatos 4 pif_idx(Eth3/1) 4 lif_idx 1
Po11 pd: vif id 0 egress_mcast_lif 0 vif_id_alloced 1 vlan_xlate_index 17
Po11 pd: pbp_idx 0 (alloced 0) gatos_lif_cfg (nil)
Po11 pd: ingress bind check - l2-en 0 l3-en 0
Po11 pd: fc zone check: ingress enabled 1 egress enabled 0
Po11 pd: cos settings - trust 1 internal rewrite 0 external rewrite 0
Po11 pd: default_ext_cos_de (cos, de)=0,0
Po11 pd: int_to_ext_cos_map (cos, de): [0]=0,0 [1]=1,0 [2]=2,0 [3]=3,0 <snip>
Po11 pd: ext_to_int_cos_map (cos, de): [0]=0,0 [1]=0,0 [2]=0,0 [3]=0,0 <snip>
Po11 pd: ingress vlan_xlate location (vlan 1) - type 1, bucket 855, entry 0
Po11 pd: ingress vlan_xlate location (vlan 1) -type 1, bucket 306, entry 0
Po11 pd: policer info: uc (en 0 100.00 pct) mc (en 0 100.00 pct) bc (en 0 100.00 pct)

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 144

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 144
Appendix: Troubleshooting FCoE

Other Diagnostic CLIs

 debug platform hardware peek: dumps the content of a specific hardware table entry
 debug hardware internal poke: modifies the content of a spefici hardware table entry [Be
careful!]
 show platform fwm info <options>: various options to dump software structures indicating
the intended hardware programming for Ethernet and FC forwarding.
 show platform fcfib <options>: various options to dump software structures and fib database
for FC forwarding.
 show interface <if> counters storm-control: storm control related drops

 other non forwarding related diagnostic CLIs:

– show system internal mts: interprocess communication details (help find out leaking MTS buffers, or
outstanding responses, etc.)
– ‘show platform software <options>’ and ‘show system internal <options>’: software internal
information of various software processes (help find out internal finite state machine details, event history,
memory allocation statistics, structures dump, global and other debug information)
– ‘debug <options>’, ‘debug platform <options>’, ‘debug hardware internal <options>’: commands to
turn on process debugs, specific backdoor options/overrides, or hardware/asic details.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 145

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 145
Appendix: Troubleshooting FCoE

Cisco Discover Protocol v2

CDP is applied to the physical Ethernet interface and only works
when enabled at both ends of link. LLDP standard is based off of
CDP

Used to verify proper connectivity to correct Network devices,

very useful at switch deployment.

POD2# sh cdp ?
all Show interfaces that are CDP enabled
entry Show CDP entries in database
global Show CDP global parameters
interface Show CDP parameters for an interface
neighbors Show CDP neighbors
traffic Show CDP traffic statistics

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 146

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 146
Appendix: Troubleshooting FCoE

CDP details TM2# sh cdp global

Global CDP information:
CDP enabled globally
Sending CDP packets every 60 seconds
Sending a holdtime value of 180 seconds
Sending CDPv2 advertisements is enabled
Sending DeviceID TLV in Default Format
----------------------------------------
Device ID:TM-6506-1
System Name:
Interface address(es):
IPv4 Address: 11.1.1.1
Platform: cisco WS-C6506, Capabilities: Router Switch IGMP Filtering
Interface: Ethernet1/4, Port ID (outgoing port): TenGigabitEthernet1/2
<Verifies proper port connections>
Holdtime: 133 sec

Version:
Cisco Internetwork Operating System Software
IOS (tm) s72033_rp Software (s72033_rp-IPSERVICES_WAN-VM), Version
12.2(18)SXF11, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2007 by cisco Systems, Inc.
Compiled Fri 14-Sep-07 23:09 by kellythw

Advertisement Version: 2
Native VLAN: 1  Sent on Native VLAN
Duplex: full

----------------------------------------
© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 147

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 147
 Appendix: Troubleshooting FCoE

DCBX Overview
DCBX is used to:
 Auto-negotiation of capability and configuration
– Priority Flow Control capability and all associated CoS values
enabled with PFC
– FCoE capability
 Allows one link peer to push config to other link peer
– FCoE CoS value
 Logical Link Up/down signaling of Ethernet and FC
 DCBX negotiation failures will result in
 vfc not coming up
 Per-priority-pause not enabled on CoS values with PFC config

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 148

http://download.intel.com/technology/eedc/dcb_cep_spec.pdf

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 148
Appendix: Troubleshooting FCoE

DCBX Troubleshooting (cont)

Steps for debugging DCBX negotiation failures
Step 1: Check your config
vig/vfc/veth should be created. Typical config is shown below
switch# conf t
switch(config)# int vig 10
switch(config)# int veth10/1
switch(config)# int vfc 10/1
switch(config)# no shut  Default for vfc is shut unless you have changed it in the first setup
switch(config)# int vig 10
switch(config)# bind interface eth2/3 Also check that the underlying physical Ethernet interface is up

LDP Transmit and Receive should be enabled on the interface for DCBX negotiation to be successful
switch# show lldp interface ethernet 2/1 | grep tx_en
tx_enabled: TRUE
switch# show lldp interface ethernet 2/1 | grep rx_en
rx_enabled: TRUE

If not enabled please enable lldp transmit & receive using the following commands
switch# conf t
switch(config)# int eth2/3 If lldp disabled vfc will not come up
switch(config-if)# lldp transmit
switch(config-if)# lldp receive

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 149

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 149
Appendix: Troubleshooting FCoE

DCBX Troubleshooting (cont)

Step 2: Check peer supports LLDP

switch# show lldp interface ethernet 2/2 | grep "No remote peers exist"
No remote peers exist  The remote adapter/switch does not support LLDP

Note that this case may also happen because of the peer sending bad LLDP frames. The user can find out
using
show platform software dcbx internal errors | grep "Received bad LLDP Packet"

Step 3: Check peer supports DCBX

show platform software dcbx internal info interface ethernet 2/2 | grep "DCX TLV”
If DCBX TLV’s are not shown then the peer does not support DCBX TLV

As of FCS Menlo based CNA will support the following DCBX TLV’s
switch# show platform software dcbx internal info interface ethernet 2/1 | grep "DCX TLV"
DCX TLV's
DCX TLV type: 1 DCX TLV Length: 10 DCX TLV Value DCBX Control TLV
DCX TLV type: 6 DCX TLV Length: 5 DCX TLV Value DCBX Ethernet Logical Link TLV (Sub type 1)
DCX TLV type: 6 DCX TLV Length: 5 DCX TLV Value DCBX FC Logical Link TLV (Sub type 0)
DCX TLV type: 5 DCX TLV Length: 5 DCX TLV Value DCBX FCoE Cos TLV
DCX TLV type: 3 DCX TLV Length: 5 DCX TLV Value DCBX PFC TLV

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 150

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 150
Appendix: Troubleshooting FCoE

DCBX Troubleshooting (cont)

Step 3: Check peer supports DCBX (cont)

Oplin based CNA will support the following DCBX TLV’s today:
DCX TLV's
DCX TLV type: 1 DCX TLV Length: 10 DCX TLV Value --> DCBX Control TLV
DCX TLV type: 2 DCX TLV Length: 28 DCX TLV Value --> DCBX Priority Groups TLV
DCX TLV type: 3 DCX TLV Length: 5 DCX TLV Value --> DCBX PFC TLV
DCX TLV type: 5 DCX TLV Length: 5 DCX TLV Value --> DCBX FCoE Cos TLV

NEX-5020 currently does not support Priority groups TLV at FCS

Feature will not be operational.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 151

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 151
 Appendix: Troubleshooting FCoE

DCBX Troubleshooting (cont)

 Step 4: Check for DCBX negotiation results
In the dump of show platform software dcbx internal info interface ethernet 2/1
look for every feature negotiation result as shown below

feature type 3 sub_type 0

feature state variables: oper_version 0 error 0 oper_mode 1 feature_seq_no 0
remote_feature_tlv_present 1
remote_tlv_not_present_notification_sent 0 remote_tlv_aged_out 0
feature register params max_version 0, enable 1, willing 0 advertise 1, disruptive_error 0
mts_addr_node
0x101mts_addr_sap 0x1e5
Desired config cfg length: 1 data bytes:08
Operating config cfg length: 1 data bytes:08

 Error
– Indicates negotiation error.
– Never expected to happen when connected to CNA
– When two NEX-5020’s are connected back-to-back
 If PFC is enabled on different CoS values negotiation error can happen
– Operating Config
– Indicates negotiation result
– Absence of operating config indicates that the peer does not support this DCBX TLV or negotiation error
– “remote_feature_tlv_present” indicates whether the remote peer supports this feature TLV or not

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 152

N5k# show platform software dcbx internal info interface

ethernet 1/4
Interface info for if_index: 0x2003000
tx_enabled: TRUE
rx_enabled: TRUE
dcbx_enabled: TRUE

Timers
TX Delay While timer info: Timer type 0, Timer ID 10726,eventID
DCX_TX_EV_TXDELAYWHILE_EXPIRED Is timer active 0time remaining
-1817611244
.
.
.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 152
Appendix: Troubleshooting FCoE

Handling DCBX negotiation errors

 DCBX Feature may not be working because of:
– Peer does not support LLDP Protocol
– Peer does not support DCBX Protocol
– Peer does not support some DCBX TLV’s
– Unexpected DCBX negotiation result
 User has an option to ‘force” a feature using the “on” option for FCoE mode and
PFC on a interface
switch(config)# int eth1/21
switch(config-if)# fcoe mode ?
auto Advertise fcoe capability auto means use DCBX negotiation result
on Turn on fcoe on means ignore DCBX and enable the feature

switch(config-if)# priority-flow-control mode ?

auto Advertise priority-flow-control capability
on Turn on priority-flow-control

 auto is the default

– “no” option for the above commands return the mode to auto

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 153

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 153
Appendix: Troubleshooting FCoE

Troubleshooting other errors

 After successful negotiation the vfc can still be in “init” state
not “up” state
– In this case the switch is waiting for the adapter to do FLOGI
– In the Oplin case
 tcpdump will help in figuring out whether the adapter is sending FLOGI or not
– In the MENLO case
 adapter logs (FC ACL Egress log) will help to see if the FLOGI is sent
 Adapter logs can be seen from the adapter GUI/CLI

 DCBX Logical Link TLV’s

– Oplin does not support Logical Link TLV’s
– Physical interface brought down when vfc or veth should be admin shut
or brought down

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 154

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 154
Appendix: Troubleshooting FCoE

Other useful DCBX debug commands

Can use ethanalyzer to see packets going from/to CNA
Typical Menlo based CNA trace
switch(config)# ethanalyzer local interface inbound-hi brief display-filter eth.addr==00:c0:dd:0a:ad:1d

Capturing on eth4
2008-04-18 18:34:00.659286 00:c0:dd:0a:ad:1d -> 01:80:c2:00:00:0e 1 0 00:c0:dd:0a:ad:1d -> 01:80:c2:00:00:0e 0x0
0x0 LLDP Chassis Id = 00:c0:dd:0a:ad:1c Port Id = 00:c0:dd:0a:ad:1d TTL = 120
2008-04-18 18:34:05.105217 00:c0:dd:0a:ad:1d -> 01:80:c2:00:00:0e 1 0 00:c0:dd:0a:ad:1d -> 01:80:c2:00:00:0e 0x0
0x0 LLDP Chassis Id = 00:c0:dd:0a:ad:1c Port Id = 00:c0:dd:0a:ad:1d TTL = 120 ---> DCX Initial Negotiation
2008-04-18 18:34:05.106611 00:c0:dd:0a:ad:1d -> fc:fc:fc:ff:ff:fe 4 0 00.00.00 -> ff.ff.fe 0x33e 0xffff FC
ELS FLOGI
2008-04-18 18:34:05.124385 00:0d:ec:6d:83:c0 -> 00:c0:dd:0a:ad:1d 4 0 ff.ff.fe -> 1b.00.00 0x33e 0x4b5 FC
ELS ACC (FLOGI) ---> FLOGI/LSACC for VFC bringup
2008-04-18 18:34:06.105306 00:c0:dd:0a:ad:1d -> 01:80:c2:00:00:0e 1 0 00:c0:dd:0a:ad:1d -> 01:80:c2:00:00:0e 0x0
0x0 LLDP Chassis Id = 00:c0:dd:0a:ad:1c Port Id = 00:c0:dd:0a:ad:1d TTL = 120
2008-04-18 18:34:07.105351 00:c0:dd:0a:ad:1d -> 01:80:c2:00:00:0e 1 0 00:c0:dd:0a:ad:1d -> 01:80:c2:00:00:0e 0x0
0x0 LLDP Chassis Id = 00:c0:dd:0a:ad:1c Port Id = 00:c0:dd:0a:ad:1d TTL = 120
2008-04-18 18:34:08.105441 00:c0:dd:0a:ad:1d -> 01:80:c2:00:00:0e 1 0 00:c0:dd:0a:ad:1d -> 01:80:c2:00:00:0e 0x0
0x0 LLDP Chassis Id = 00:c0:dd:0a:ad:1c Port Id = 00:c0:dd:0a:ad:1d TTL = 120
2008-04-18 18:34:09.105537 00:c0:dd:0a:ad:1d -> 01:80:c2:00:00:0e 1 0 00:c0:dd:0a:ad:1d -> 01:80:c2:00:00:0e 0x0
0x0 LLDP Chassis Id = 00:c0:dd:0a:ad:1c Port Id = 00:c0:dd:0a:ad:1d TTL = 120
2008-04-18 18:34:39.007974 00:c0:dd:0a:ad:1d -> 01:80:c2:00:00:0e 1 0 00:c0:dd:0a:ad:1d -> 01:80:c2:00:00:0e 0x0
0x0 LLDP Chassis Id = 00:c0:dd:0a:ad:1c Port Id = 00:c0:dd:0a:ad:1d TTL = 120
2008-04-18 18:35:09.010407 00:c0:dd:0a:ad:1d -> 01:80:c2:00:00:0e 1 0 00:c0:dd:0a:ad:1d -> 01:80:c2:00:00:0e 0x0
0x0 LLDP Chassis Id = 00:c0:dd:0a:ad:1c Port Id = 00:c0:dd:0a:ad:1d TTL = 120 ---> DCX KeepAlives

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 155

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 155
Appendix: Troubleshooting FCoE

Other useful DCBX debug commands

Typical Oplin based CNA trace
switch# ethanalyzer local interface inbound-hi brief display-filter
eth.addr==00:1b:21:06:d5:1f
Capturing on eth4
2008-04-18 18:43:50.571013 00:1b:21:06:d5:1f -> 01:80:c2:00:00:0e 1 0 00:1b:21:06:d5:1f ->
01:80:c2:00:00:0e 0x0 0x0 LLDP Chassis Id = eth2 Port Id = 00:1b:21:06:d5:1f TTL = 120
2008-04-18 18:43:51.572053 00:1b:21:06:d5:1f -> 01:80:c2:00:00:0e 1 0 00:1b:21:06:d5:1f ->
01:80:c2:00:00:0e 0x0 0x0 LLDP Chassis Id = eth2 Port Id = 00:1b:21:06:d5:1f TTL = 120
2008-04-18 18:43:52.573196 00:1b:21:06:d5:1f -> 01:80:c2:00:00:0e 1 0 00:1b:21:06:d5:1f ->
01:80:c2:00:00:0e 0x0 0x0 LLDP Chassis Id = eth2 Port Id = 00:1b:21:06:d5:1f TTL = 120
2008-04-18 18:43:53.133123 00:1b:21:06:d5:1f -> fc:fc:fc:ff:ff:fe 4 0 00.00.00 ->
ff.ff.fe 0x413 0xffff FC ELS FLOGI
2008-04-18 18:43:53.143511 00:0d:ec:6d:83:c0 -> 00:1b:21:06:d5:1f 4 0 ff.ff.fe ->
1b.00.05 0x413 0x7c2 FC ELS ACC (FLOGI)
2008-04-18 18:43:53.574335 00:1b:21:06:d5:1f -> 01:80:c2:00:00:0e 1 0 00:1b:21:06:d5:1f ->
01:80:c2:00:00:0e 0x0 0x0 LLDP Chassis Id = eth2 Port Id = 00:1b:21:06:d5:1f TTL = 120
2008-04-18 18:43:54.575379 00:1b:21:06:d5:1f -> 01:80:c2:00:00:0e 1 0 00:1b:21:06:d5:1f ->
01:80:c2:00:00:0e 0x0 0x0 LLDP Chassis Id = eth2 Port Id = 00:1b:21:06:d5:1f TTL = 120

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 156

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 156
Appendix: Troubleshooting FCoE

POST and OHMS

Types Of Errors Types of Reaction

Failures causing NXOS not Console continuous print error messages every 30 seconds.
be able to come up System LED sets to Flashing Amber. Example of such failure:
properly DRAM, backplane SPROM checksum failure, PCIe enumeration
failure

Failures not fatal and System comes all the way up. Syslog, OBFL and callhome initiated
NXOS can boot up to indicate failure. Example of such failure: OBFL flash, CTS
keystore.

Failure causing port System comes all the way up. Syslog, OBFL and callhome initiated
failures to indicate failure. Example of such failure: ASIC ECC error found
during POST or OHMS

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 157

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 157
Appendix: Troubleshooting FCoE

Environmental Monitoring
 switch# show environment
 Displays following status:
– Fan
– Temperature
– Power Supply
– Power Usage Summary

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 158

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 158
Appendix: Troubleshooting FCoE

Showing Diagnostic Result

switch# show diagnostic result module 1
Current bootup diagnostic level: complete
Module 1: 40x10GE/Supervisor SerialNo : JAB1208005T
Overall Diagnostic Result for Module 1 : PASS
Diagnostic level at card bootup: complete
Test results: (. = Pass, F = Fail, I = Incomplete,
U = Untested, A = Abort)
1) TestUSBFlash ------------------------> .
2) TestSPROM ---------------------------> .
3) TestPCIe ----------------------------> .
4) TestLED -----------------------------> .
5) TestOBFL ----------------------------> .
6) TestNVRAM ---------------------------> .
7) TestPowerSupply ---------------------> F
8) TestTemperatureSensor ---------------> .
9) TestFan -----------------------------> .
10) TestVoltage -------------------------> .
11) TestGPIO ----------------------------> .
12) TestSupervisorPort ------------------> .
13) TestMemory --------------------------> .
14) TestFabricEngine :
Eth 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Port ------------------------------------------------------------
. . . . . . . . . . . . . . . . . . . .
Eth 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
Port ------------------------------------------------------------
. . . . . . . . . . . . . . . . . . . .

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 159

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 159
Appendix: Troubleshooting FCoE

Showing Diagnostic Result (cont)

switch# show diagnostic result module 2
Current bootup diagnostic level: complete
Module 2: 4x10GE + 4x1/2/4G FC Module SerialNo : JAB120600DT
Overall Diagnostic Result for Module 2 : PASS
Diagnostic level at card bootup: complete
Test results: (. = Pass, F = Fail, I = Incomplete,
U = Untested, A = Abort)
1) TestSPROM ---------------------------> .
2) TestLED -----------------------------> .
3) TestTemperatureSensor ---------------> .
4) TestFabricEngine :
Eth 1 2 3 4
Port ------------
. . . .
Fc 1 2 3 4
Port ------------
. . . .
5) TestFabricPort :
Eth 1 2 3 4
Port ------------
. . . .
Fc 1 2 3 4
Port ------------
. . . .
6) TestForwardingEngine :
<snip>
© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 160

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 160
Appendix: Troubleshooting FCoE

Showing Diagnostic Result (cont)

switch# show diagnostic result module 3

Current bootup diagnostic level: complete

6) TestForwardingEngine
Module 3: 6x10GE Ethernet Module SerialNo : :
JAB120800VF
Eth 1 2 3 4 5 6
Overall Diagnostic Result for Module 3 : PASS Port ------------------
Diagnostic level at card bootup: complete . . . . . .

Test results: (. = Pass, F = Fail, I = Incomplete,

U = Untested, A = Abort) 7)
TestForwardingEnginePort :
1) TestSPROM ---------------------------> .
2) TestLED -----------------------------> . Eth 1 2 3 4 5 6
3) TestTemperatureSensor ---------------> . Port ------------------
4) TestFabricEngine : . . . . . .

Eth 1 2 3 4 5 6
Port ------------------ 8) TestFrontPort :
. . . . . .
Eth 1 2 3 4 5 6
Port ------------------
5) TestFabricPort : . . . . . .

Eth 1 2 3 4 5 6
Port ------------------
. . . . . .

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 161

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 161
Appendix: Troubleshooting FCoE

NOHMs logging
switch# show logging |grep NOHMS

2008 Apr 18 23:00:01 switch %NOHMS-2-NOHMS_DIAG_ERROR: Module 1: Runtime diag

detected major event: Port failure: Ethernet1/1
2008 Apr 18 23:00:01 switch %NOHMS-2-NOHMS_DIAG_ERROR: Module 1: Runtime diag
detected major event: Port failure: Ethernet1/2
2008 Apr 18 23:00:01 switch %NOHMS-2-NOHMS_DIAG_ERROR: Module 1: Runtime diag
detected major event: Port failure: Ethernet1/5
2008 Apr 18 23:00:01 switch %NOHMS-2-NOHMS_DIAG_ERROR: Module 1: Runtime diag
detected major event: Port failure: Ethernet1/6
2008 Apr 19 01:45:25 swor35p %NOHMS-2-NOHMS_ENV_ERROR: Module 1 temperature
sensor 1 failed.
2008 Apr 19 01:45:25 swor35p %NOHMS-2-NOHMS_ENV_ERROR: Module 1 temperature
sensor 2 failed.
2008 Apr 19 01:45:25 swor35p %NOHMS-2-NOHMS_ENV_ERROR: System major
temperature alarm on Module 1. Sensor 9 Temperature 42 Major Threshold 0

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 162

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 162
Appendix: Troubleshooting FCoE

Show tech
Capture to terminal emulator buffer or log file:
switch# terminal length 0
switch# show tech-support details
`show switchname`
switch
`show system uptime`
System start time: Mon Aug 11 15:33:17 2008
System uptime: 2 days, 0 hours, 46 minutes, 4 seconds
.
.
.

Capture to file in volatile:

switch# tac-pac
switch# dir volatile:
66860 Aug 13 16:23:03 2008 show_tech_out.gz
switch# copy volatile:show_tech_out.gz
sftp://[email protected]/ vrf management

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 163

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 163
Appendix: Troubleshooting FCoE

What is captured in show tech details or tac-pac

show switchname show monitor session show switchname
show system uptime show accounting log show system uptime
show interface mgmt0 show process show interface mgmt0
show system resources show process cpu show system resources
show version
show version show process log
show inventory
show inventory show process memory
show diagnostic result module all
show diagnostic result module show processes log details
show logging log
all show license host-id show module
show logging log show license show environment
show module show license usage show sprom backplane
show environment show system reset-reason show clock
show sprom backplane show logging nvram show callhome
show clock show install all status show cfs application
show callhome show install all failure-reason show cfs lock
show cfs application show system internal log install show snmp
show cfs lock show system internal log install show interface brief
show snmp details show interface
show running-config
show interface brief show cores
show startup-config
show interface show topology
show ip route
show running-config show kernel internal aipc show arp
show startup-config show tech-support acl show monitor session
show ip route show aclmgr status
show arp show aclmgr internal dictionaries
show aclmgr internal log

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 164

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 164
Appendix: Troubleshooting FCoE

What is captured in show tech details or tac-pac?

show accounting log show aclmgr internal log
show process show aclmgr internal ppf
show process cpu show aclmgr internal state-cache
show process log show access-lists
show process memory show vlan access-map
show processes log details show vlan
show license host-id show mac-address-table
show license show spanning-tree summary
show license usage show spanning-tree active
show system reset-reason show interface trunk
show logging nvram show system internal ethpm info all
show install all status show logging onboard obfl-logs
show install all failure-reason show hardware internal gatos counters interrupt
show system internal log install show hardware internal gatos interrupt
show system internal log install details show hardware internal cpu-mac mgmt counters
show cores show hardware internal cpu-mac mgmt stats
show topology show hardware internal cpu-mac inband counters
show kernel internal aipc show hardware internal cpu-mac inband stats
show tech-support acl
show aclmgr status
show aclmgr internal dictionaries

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 165

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 165
Appendix: Troubleshooting FCoE

Core Files
Show cores
switch# show cores
Module-num Process-name PID Core-create-time
---------- ------------ --- ----------------
1 fwm 2834 Aug 13 16:3

Copy to a remote server:

switch# copy core:?
core: Enter URL "core://<module-number>/<process-id>"

switch# copy core://1/2834 ftp://128.107.65.217/ vrf management

Enter username: anonymous
Password:
***** Transfer of file Completed Successfully *****

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 166

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 166
Appendix: Troubleshooting FCoE

What’s in a Core File?

Rename it with a .tar.gz extension and extract it with WinZip or gunzip/tar

0x101_fwm_core.2834.gz is a compress memory dump

0x101_fwm_log.2834 is a text file

Service: fwm
Description: FWM Daemon

Started at Sat Jan 29 22:40:44 2005 (252593 us)

Stopped at Mon Jan 31 02:18:30 2005 (550547 us)
Uptime: 1 days 3 hours 37 minutes 46 seconds
Start type: SRV_OPTION_RESTART_STATELESS (23)
Death reason: SYSMGR_DEATH_REASON_FAILURE_SIGNAL (2)
System image version: 4.0(0)N1(2) S0

PID: 2834
Exit code: signal 3 (core dumped)

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 167

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 167
Appendix: Troubleshooting FCoE

Appendix E

Troubleshooting NEX-5020 Upgrades

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 168

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 168
Appendix: Troubleshooting FCoE

Normal Upgrade Procedure

 Same Procedure as MDS
 Uses the Install All command:

install all system scp://user@system/code/n5000-uk9.4.0.0.N1.2.bin kickstart

scp://user@system/code/n5000-uk9-kickstart.4.0.0.N1.2.bin
For scp://user@system, please enter password:
Copying images…
Verifying images…
Extracting "system" version from image…
Extracting "kickstart" version from image…

See Nexus 5000 Series Switch CLI Software Configuration Guide

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 169

169
Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 169
Appendix: Troubleshooting FCoE

Normal Upgrade Procedure continued . . .

Extracting "bios" version from image…

Collecting "running" plugin(s) information.
Collecting plugin(s) information from "new" image
Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes disruptive reset Reset due to single supervisor
Images will be upgraded according to following table:
Module Image Running-Version New-Version Upg-Required
------ ---------- ---------------------- ---------------------- ------------
1 system 4.0(0)N1(1a) 4.0(0)N1(2) yes
1 kickstart 4.0(0)N1(1a) 4.0(0)N1(2) yes
1 bios v1.1.0(05/20/08) v1.2.0(06/19/08) yes

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 170

170
Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 170
Appendix: Troubleshooting FCoE

Normal Upgrade Procedure continued . . .

Do you want to continue with the installation (y/n)? [n]y

Install is in progress, please wait.
Setting boot variables.
Performing configuration copy.
Module 1: Upgrading Bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
Rebooting the switch to finish the upgrade.

 Switch reboots and comes up on new code and with boot variables set
© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 171

171
Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 171
Appendix: Troubleshooting FCoE

Normal Upgrade Procedure

 Common problems
Communication issues: If using ftp, sftp, tftp, or scp verify
network paths, login credentials, and file names and locations.
Space issues: There must be sufficient space on the bootflash for
the new binary files.
Compatibility issues: New files are either corrupt or not
compatible with this system.

 Helpful commands
Ping, show file filename cksum/md5sum, show install all
impact/status/failure-reason

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 172

172
Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 172
Appendix: Troubleshooting FCoE

Recovery Methods
Boot Sequence

 A clear understanding of the Nexus 5000 boot sequence

is necessary to be able to recover from forgotten
passwords, corrupt file systems, unknown console
settings, corrupt images and/or configs, etc.
 Loader Prompt is used to set console settings, reset
CMOS, set ip and boot images from network or
bootflash
 Switch Boot Prompt is used to recover password,
initialize system, format bootflash, etc.

See Nexus 5000 Series Switch CLI Software Configuration Guide page 3-2
© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 173

173
Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 173
Appendix: Troubleshooting FCoE

Boot Sequence (1)

Within 2 sec
Of power on

 Golden BIOS can not be modified

 When you press Ctrl-Shift-6, the console settings must be set to
their defaults: 9600 baud, 8 data bits, no parity, and 1 stop bit.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 174

174
Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 174
 Appendix: Troubleshooting FCoE

Boot Sequence (2)

 Before the boot sequence starts, the BIOS performs internal tests
on the switch. If the tests fail, then the loader does not gain control.
Instead, the BIOS image retains control and prints a message to
the console at 9600 baud every 30 seconds that indicates a failure.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 175

Console Settings
BIOS only operates at 9600 baud, 8 data bits, no parity, and 1 stop bit. The user cannot change the console settings
for the BIOS. This is normally not a problem as the BIOS only interacts with the console when checking for if the
user wants to break into the golden BIOS, as well as output in the case of critical errors such as failed DRAM test.
The console settings for loader, kickstart and system are all stored in CMOS. All three images follow the settings
in the CMOS. There are two places in the system that allow the users to change the console settings stored in the
CMOS
 Loader via the CLI command serial
 System image via the CLI command line
The factory default for the console settings for the loader, kickstart and system images is 9600, 8 data bits, no
parity, and 1 stop bit.
Recovering from corrupted/forgotten console settings
If the user cannot figure out or forgets the console settings stored in CMOS, or if the CMOS gets corrupted, users
can go back to 9600-8-N-1 on the terminal side and then reboot the switch. When the switch is booting, user can
force the loader to run at the factory default console settings of 9600-8-N-1 by hitting Ctrl-Shift-R repeatedly until
the loader> prompt appears. Note that Ctrl-Shift-R must be entered when the console server settings are at 9600-8-
N-1. The loader gives the user a one second window where it will detect this, and it will not give any sort of
indication that it has started/stopped checking for this keystroke. This is consistent with MDS ideology regarding
loader behavior. The user can then reset the CMOS to factory default settings, or write their own.

175
Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 175
 Appendix: Troubleshooting FCoE

Boot Sequence (3)

 The BIOS launches the loader which then launches

kickstart. There are two ways to break into the loader
 Cntr-Shift-R at 9600 baud
 Cntr –Shift-L at CMOS baud rate

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 176

Breaking into loader

Sometimes the user may want to break into the loader. The most common reason for this is
because the bootflash is corrupted to the point where there is no viable kickstart image to load
on it anymore. Another valid reason is if the user wants to insert a kernel command-line
argument (probably only useful for developers). Breaking into the loader can be done by hitting
Ctrl-Shift-L during boot. Note that Ctrl-Shift-L must be entered when the console server
settings are at whatever is set in the CMOS. The loader gives the user a one second window
where it will detect this, and it will not give any sort of indication that it has started/stopped
checking for this keystroke. This is consistent with MDS ideology regarding loader behavior

176
Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 176
Appendix: Troubleshooting FCoE

Boot Sequence (4)

 The Kickstart CLI can be used to recover from lost password and to
recover form corrupt configs etc.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 177

177
Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 177
Appendix: Troubleshooting FCoE

Recovering From Bad Startup Configuration

 If the startup configuration causes system image to
crash or do other bad things, users may not be allowed
a chance to change the configuration via the normal
method supported by system image.
 Break into the kickstart image during booting by hitting
Ctrl-Shift-B. Note that Ctrl-Shift-B must be entered
when the console server settings are at whatever is set
in the CMOS.
 Kickstart CLI supports write erase. Once configuration
file is deleted, system image can once again startup with
factory default..

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 178

178
Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 178
Appendix: Troubleshooting FCoE

Recovering From Lost Password

 The “admin” command in kickstart configuration mode
allows the administrator to change the admin password.
Local console access is required to break into the
kickstart image, so this is as secure as local console
access
switch(boot)# config terminal
switch(boot)(config)# admin-password newpassword

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 179

179
Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 179
Appendix: Troubleshooting FCoE

Recovering From Bad Boot Files

 If kickstart image is valid and functional, users can break
into the switch(boot)# prompt during boot process.
Kickstart supports IP connectivity and can download
system images via the supported protocols in the copy
command.
 If both kickstart and system images are bad, users can
break into the loader> prompt during boot process.
Loader supports IP connectivity and can netboot
kickstart image via tftp. After that, kickstart can then
download both kickstart and system images onto the
bootflash:.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 180

180
Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 180
Appendix: Troubleshooting FCoE

Network Boot example

 Power on switch and from console enter Ctrl-Shift-L
until:
User break into bootloader
Loader Version pr-1.0
loader>

 Set IP address
loader> set ip 172.22.42.157 255.255.254.0
Correct - ip addr is 172.22.42.157, mask is 255.255.254.0
Found Intel 82571EB at 0x8020, ROM address 0x9980
Probing...[Intel 82571EB]Ethernet addr: 00:0D:EC:A3:38:40
Address: 172.22.42.157
Netmask: 255.255.254.0
Server: 0.0.0.0
Gateway: 0.0.0.0

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 181

181
Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 181
Appendix: Troubleshooting FCoE

Network Boot example (continued)

 Set Gateway
loader> set gw 172.22.42.1
Correct gateway addr 172.22.42.1
Address: 172.22.42.157
Netmask: 255.255.254.0
Server: 0.0.0.0
Gateway: 172.22.42.1
loader>

 Boot kickstart file from tftp server

loader> boot tftp://171.71.55.139/n5000-uk9-
kickstart.4.0.0.N1.2.476.bin
Address: 172.22.42.157
Netmask: 255.255.254.0
Server: 171.71.55.139
Gateway: 172.22.42.1
Booting: /n5000-uk9-kickstart.4.0.0.N1.2.476.bin

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 182

182
Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 182
Appendix: Troubleshooting FCoE

Network Boot example (continued)

Configure mgmt0 port
switch(boot)# config t
switch(boot)(config)# interface mgmt 0
switch(boot)(config-if)# ip address 172.22.42.157 255.255.254.0
switch(boot)(config-if)# no shutdown
switch(boot)(config-if)# exit
switch(boot)(config)# ip default-gateway 172.22.42.1
switch(boot)(config)# exit
switch(boot)#

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 183

183
Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 183
Appendix: Troubleshooting FCoE

Network Boot example (continued)

 Copy System Files to bootflash
switch(boot)# copy tftp://171.71.55.139/n5000-uk9-
kickstart.4.0.0.N1.2.476.bin bootflash:
switch(boot)# copy tftp://171.71.55.139/n5000-uk9.4.0.0.N1.2.476.bin
bootflash:

 Load System file

switch(boot)# load bootflash:n5000-uk9.4.0.0.N1.2.476.bin

 These procedures are very much like the MDS

model

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 184

184
Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 184
Appendix: Troubleshooting FCoE

Appendix F

NEX-5020 Trace Tools

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 185

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 185
Appendix: Troubleshooting FCoE

What tool captures what?

 Ethanalyzer is a tool that will collect frames that are
destined to, or originate from the NEX-5020 control
plane. Node to switch, or switch to switch traffic can be
seen with this tool.
 SPAN is a feature whereby frames that are transient to
the switch, will be copied to a second port for
analysis. Node to switch or node to node traffic can be
seen via this method.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 186

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 186
Appendix: Troubleshooting FCoE

Ethanalyzer
 Ethanalyzer is a Cisco NX-OS protocol analyzer tool based on the Wireshark
(formerly Ethereal) open source code. Ethanalyzer is a command-line version of
Wireshark that captures and decodes packets. You can use Ethanalyzer to
troubleshoot your network and analyze the control-plane traffic.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 187

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 187
Appendix: Troubleshooting FCoE

Ethanalyzer Configuration Examples

N5K(config)# ethanalyzer local interface ?

No matches in current mode, matching in (exec) mode
inbound-hi Inbound(high priority) interface
inbound-low Inbound(low priority) interface
mgmt Management interface

N5K(config)# ethanalyzer local interface mgmt brief

Capturing on eth0
2008-08-13 01:34:23.776519 10.116.167.244 -> 172.18.217.80 TCP 1106 > telnet
[ACK] Seq=0 Ack=0 Win=64040 Len=0
2008-08-13 01:34:23.777752 172.18.217.80 -> 10.116.167.244 TELNET Telnet Data ...
2008-08-13 01:34:23.966262 00:04:dd:2f:75:10 -> 01:80:c2:00:00:00 STP Conf. Root
= 32768/00:04:c1:0f:6e:c0 Cost = 57 Port = 0x801d
[snip]

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 188

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 188
Appendix: Troubleshooting FCoE

Ethanalyzer Configuration Example

N5K(config)# ethanalyzer local interface inbound-hi brief limit-captured-frames 0
Capturing on eth4
2008-08-13 01:37:16.639896 00:0d:ec:6b:cd:41 -> 01:80:c2:00:00:00 1 0
00:0d:ec:6b:cd:41 -> 01:80:c2:00:00:00 0x0 0x0 STP RST. Root =
32769/00:0d:ec:6b:cd:41 Cost = 0 Port = 0x8093
2008-08-13 01:37:18.639992 00:0d:ec:6b:cd:41 -> 01:80:c2:00:00:00 1 0
00:0d:ec:6b:cd:41 -> 01:80:c2:00:00:00 0x0 0x0 STP RST. Root =
32769/00:0d:ec:6b:cd:41 Cost = 0 Port = 0x8093
[snip]
2008-08-13 01:37:23.220253 00:0d:ec:6b:cd:40 -> fc:fc:fc:ff:ff:fd 4 0 ff.ff.fd ->
ff.ff.fd 0x5384 0xffff SW_ILS ELP
2008-08-13 01:37:23.220615 00:0d:ec:6b:cd:40 -> aa:bb:cc:dd:01:04 4 0 ff.ff.fd ->
ff.ff.fd 0x5384 0x2b3f FC Link Ctl, ACK1
2008-08-13 01:37:23.227202 00:0d:ec:6b:cd:40 -> aa:bb:cc:dd:01:04 4 0 ff.ff.fd ->
ff.ff.fd 0x5384 0x2b3f SW_ILS SW_ACC (ELP)
2008-08-13 01:37:23.229927 00:0d:ec:6b:cd:40 -> fc:fc:fc:ff:ff:fd 4 0 ff.ff.fd ->
ff.ff.fd 0x5384 0x2b3f FC Link Ctl, ACK1

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 189

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 189
Appendix: Troubleshooting FCoE

Ethanalyzer Configuration Example

N5K(config)# ethanalyzer local interface inbound-hi limit-captured-frames 0
Capturing on eth4
Frame 1 (57 bytes on wire, 57 bytes captured)
Arrival Time: Aug 13, 2008 01:41:32.631969000
[Time delta from previous captured frame: 1218591692.631969000 seconds]
[Time delta from previous displayed frame: 1218591692.631969000 seconds]
[Time since reference or first frame: 1218591692.631969000 seconds]
Frame Number: 1
Frame Length: 57 bytes
Capture Length: 57 bytes
[Frame is marked: False]
[Protocols in frame: eth:vlan:llc:stp]
[snip]
DSAP: Spanning Tree BPDU (0x42)
IG Bit: Individual
SSAP: Spanning Tree BPDU (0x42)
CR Bit: Command
Control field: U, func=UI (0x03)
000. 00.. = Command: Unnumbered Information (0x00)
.... ..11 = Frame type: Unnumbered frame (0x03)
[snip]

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 190

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 190
Appendix: Troubleshooting FCoE

SPAN Sources
 The Switched Port Analyzer (SPAN) feature (sometimes called port
mirroring or port monitoring) selects network traffic for analysis by a
network analyzer. The network analyzer can be a Cisco SwitchProbe, a
Fibre Channel Analyzer, or other Remote Monitoring (RMON) probes.
 SPAN sources refer to the interfaces from which traffic can be monitored.
The Cisco Nexus 5000 Series switch supports Ethernet, virtual Ethernet,
Fibre Channel, virtual Fibre Channel, port channels, SAN port channels,
VLANs, and VSANs as SPAN sources. With VLANs or VSANs, all
supported interfaces in the specified VLAN or VSAN are included as SPAN
sources. You can choose the SPAN traffic in the ingress direction, the
egress direction, or both directions for Ethernet, virtual Ethernet, Fibre
Channel, and virtual Fibre Channel source interfaces:
– Ingress source (Rx)—Traffic entering the switch through this source port is copied
to the SPAN destination port.
– Egress source (Tx)—Traffic exiting the switch through this source port is copied to
the SPAN destination port.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 191

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 191
Appendix: Troubleshooting FCoE

Source Ports
 A source port, also called a monitored port, is a switched interface that you monitor for
network traffic analysis. The switch supports any number of ingress source ports (up to
the maximum number of available ports on the switch) and any number of source
VLANs or VSANs.
 A source port has these characteristics:
– Can be of any port type: Ethernet, virtual Ethernet, Fibre Channel, virtual Fibre
Channel, port channel, SAN port channel, VLAN, and VSAN.
– Cannot be monitored in multiple SPAN sessions.
– Cannot be a destination port.
– Each source port can be configured with a direction (ingress, egress, or both) to
monitor. For VLAN, VSAN, port channel, and SAN port channel sources, the
monitored direction can only be ingress and applies to all physical ports in the group.
The rx/tx option is not available for VLAN or VSAN SPAN sessions.
– Source ports can be in the same or different VLANs or VSANs.
– For VLAN or VSAN SPAN sources, all active ports in the source VLAN or VSAN are
included as source ports.
– The switch supports a maximum of two egress SPAN source ports.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 192

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 192
Appendix: Troubleshooting FCoE

SPAN Destinations
 SPAN destinations refer to the interfaces that monitors source
ports. The Cisco Nexus 5000 Series switch supports Ethernet
and Fibre Channel interfaces as SPAN destinations.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 193

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 193
Appendix: Troubleshooting FCoE

Characteristics of Destination Ports

 Each local SPAN session must have a destination port (also called a
monitoring port) that receives a copy of traffic from the source ports,
VLANs, or VSANs. A destination port has these characteristics:
 Can be any physical port, Ethernet, Ethernet (FCoE), or Fibre Channel.
Virtual Ethernet and virtual Fibre Channel ports cannot be destination
ports.
 Cannot be a source port.
 Cannot be a port channel or SAN port channel group.
 Does not participate in spanning tree while the SPAN session is active.
 Is excluded from the source list and is not monitored if it belongs to a
source VLAN of any SPAN session.
 Receives copies of sent and received traffic for all monitored source ports.
If a destination port is oversubscribed, it can become congested. This
congestion can affect traffic forwarding on one or more of the source ports.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 194

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 194
Appendix: Troubleshooting FCoE

Monitor (SPAN) Caveats

Idiosyncrasies of NEX-5020 SPAN:
 COS values will not be preserved at monitor (span) destination
 Packets coming in on monitor source with unknown vlan tag will be
spanned out with 0 vlan tag (priority tag)
 For ethernet destination, monitor session will be up only if the
destination port is configured as switchport monitor
 Out of 18 configurable sessions, only two will be active (up state). The
rest will be in down state (hardware resource unavailable).

Configuration limitations:
 Vlan or port-channel cannot be configured as egress source
 Vlan or port-channel cannot be a monitor destination
 Only two egress sources supported
 Only one destination port can be configured for a session

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 195

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 195
Appendix: Troubleshooting FCoE

SPAN Configuration Example

N5K(config)# int fc3/2
N5K(config-if)# switchport mode sd
N5K(config-if)# switchport speed 1000
N5K(config-if)# no shut
N5K(config-if)# exit
N5K(config)# monitor session 1
N5K(config-monitor)# source interface fc3/1 tx
N5K(config-monitor)# source interface fc3/1 rx
N5K(config-monitor)# destination interface fc3/2

Wireshark

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 196

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 196
Appendix: Troubleshooting FCoE

Verifying the SPAN Session

N5K# sh monitor session
SESSION STATE REASON DESCRIPTION
------- ----------- ---------------------- --------------------------------
1 up The session is up

N5K# show monitor session 1

session 1
---------------
type : local
state : up
source intf :
rx : fc3/1
tx : fc3/1
both : fc3/1
source VLANs :
rx :
source VSANs :
rx :
destination ports : fc3/2

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 197

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 197
Appendix: Troubleshooting FCoE

Suspending a SPAN Session

N5K(config)# monitor session 1 suspend

N5K(config)# show monitor session 1

session 1
---------------
type : local
state : down (Session suspended)
source intf :
rx : fc3/1
tx : fc3/1
both : fc3/1
source VLANs :
rx :
source VSANs :
rx :
destination ports : fc3/2

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 198

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 198
Appendix: Troubleshooting FCoE

DS-PAA-2 – FC Trace Tool

 Switch Settings
– 0001 1G NTM
– 1001 1G ETM
– 0101 1G STM
– 0011 1G DTM
– 0000 2G NTM
– 1000 2G ETM
– 0100 2G STM
– 0010 2G DTM
– 1111 1G MNM (Used for diagnostics only)

 MODES
– NTM (No Truncate Mode) - FC Frames are passed without any modifications.
– ETM (Ethernet Truncate Mode) - Truncates FC frame to max 1496B by reducing payload size from
528 lines to 368 lines.
– STM (Shallow Truncate Mode) - Truncates FC frame to max 256B by reducing payload size from 528
lines to 58 lines.
– DTM (Deep Truncate Mode) - Truncates FC frame to max 64B by reducing payload size from 528
lines to 10 lines.

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 199

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 199
Appendix: Troubleshooting FCoE

Intentionally Blank Page

© 2008 EMC Corporation. All rights reserved. Appendix: Troubleshooting FCoE - 200

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 200