Configure Fibre Channel

Note: Be sure to review the Objectives and

Job Aids links above for required
information. Password Information and
Command Lists for Detailed Lab Steps are
in the Job Aids link.

Task 1: Configure Fibre Channel Connectivity Between MDS and N9K-A

In this procedure, you will configure Fibre Channel connectivity for SAN A—from the storage to the N9K-A.

Show Steps

• Step 1:

Click the Play icon in the middle of the screen or the Play button on the player below the simulation screen to begin the simulation.

Note

If stuck, click the Forward button in the Playbar to continue to the next slide.

Tip:
The following are the Playbar buttons and their functions:
◦ Rewind: Resets the interactive activity.
◦ Play: Plays the interactive activity. In the middle of the interactive activity, it can also be used to go to the next slide.
◦ Back: Go to the previous slide.
◦ Forward: Go to the next slide.

Note

When clicking off the interactive activity, the hover function will stop responding. To fix this, click anywhere in the interactive activity. If the step
requires text input, you must also click on the input field needed for this step (the cursor will change to the text pointer). When using the CLI-
based slide, this will usually be just behind the number (#) sign.

• Step 2:

Use PuTTY and choose N9K-A from the preconfigured device list to connect to your N9K-A.

Tip:
Open PuTTY, under Saved Sessions choose the N9K-A option, and click the Open button.
• Step 3:

In the CLI enter the username and password provided in the Job Aids and note down the Pod, VSAN, and NetApp PWWNs values.

Tip:
Login using the admin username and the 1234QWer password.

Note
 The password characters in the activity are rendered as dots, in the actual CLI the password characters are not rendered.

login as: admin

========== Info ==========

Device: N9K-A
Pod: 1
VSAN: 1011
NetApp PWWNs:
20:0e:00:a0:98:25:60:36 (NetApp1)
20:0f:00:a0:98:25:60:36 (NetApp2)
==========================
Using keyboard-interactive authentication.
Password:

N9K-A#

Note the highlighted values from the output above.

• Step 4:

On N9K-A, investigate where Fibre Channel SFPs are inserted.

Tip:
In the CLI enter the show interface transceiver command.

Note

This is a lab simulation so only the Spacebar key is supported to show more of the output in the CLI.

N9K-A# show interface transceiver

<... output omitted ...>
Ethernet1/47
transceiver is present
type is Unknown Type-(unknown)
name is CISCO-AVAGO
part number is SFBR-5780AMZ-CS1
revision is G4.1
serial number is AVJ1806B14F
nominal bitrate is 8500 MBit/sec
Link length supported for 50/125um OM2 fiber is 50 m
Link length supported for 62.5/125um fiber is 21 m
Link length supported for 50/125um OM3 fiber is 150 m
cisco id is 3
cisco extended id number is 4
cisco part number is 10-2418-01
cisco product id is DS-SFP-FC8G-SW
cisco version id is V01

Ethernet1/48
transceiver is present
type is Unknown Type-(unknown)
name is CISCO-AVAGO
part number is SFBR-5780AMZ-CS1
revision is G4.1
serial number is AVJ1806B172
nominal bitrate is 8500 MBit/sec
Link length supported for 50/125um OM2 fiber is 50 m
Link length supported for 62.5/125um fiber is 21 m
Link length supported for 50/125um OM3 fiber is 150 m
cisco id is 3
cisco extended id number is 4
cisco part number is 10-2418-01
cisco product id is DS-SFP-FC8G-SW
cisco version id is V01
<... output omitted ...>

You should find the Fibre Channel transceivers in slots 47 and 48. You should also see that transceivers have the nominal rate of 8500 Mbps,
which effectively means that these transceivers are capable of 8 Gbps.

At this point, interfaces are in the Ethernet mode. If you want to use the Fibre Channel SFP modules, you need to convert these two interfaces to
Fibre Channel.

• Step 5:

On N9K-A, install and enable the Fibre Channel over Ethernet (FCoE) feature set.

Tip:
In the CLI enter the following commands.
N9K-A# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
N9K-A(config)# install feature-set fcoe
 N9K-A(config)# feature-set fcoe
Please configure the following for fcoe to be fully functional:
- hardware access-list tcam region ing-racl <TCAM size>
- hardware access-list tcam region ing-ifacl <TCAM size>
- hardware access-list tcam region ing-redirect <TCAM size>

Note

The switch informs you to configure TCAM carving. You will configure it in the following next steps.

• Step 6:

Reserve the required TCAM resources for the FC features. When asked, do not reload the system yet.

TCAM region

Required TCAM reservation size

ing-racl

1536

ing-ifacl

256

ing-redirect

256

Tip:
In the CLI enter the following commands.
N9K-A-PodX(config)# hardware access-list tcam region ing-racl 1536
Please save config and reload the system for the configuration to take effect

N9K-A-PodX(config)# hardware access-list tcam region ing-ifacl 256

Please save config and reload the system for the configuration to take effect

N9K-A-PodX(config)# hardware access-list tcam region ing-redirect 256

Warning: Please save config and reload the system for the configuration to take effect

• Step 7:

On N9K-A, convert ports 47 and 48 from Ethernet to the Fibre Channel mode. Use the slot number command to enter the slot configuration
mode. Use the port first_in_range-last_in_range type fc command to change ports to the Fibre Channel type.

Tip:
In the CLI enter the following commands.
N9K-A(config)# slot 1
N9K-A(config-slot)# port 47-48 type fc
All ports in a port-group will be converted together.
Ports 45-48 will be converted to FC
Do you want to continue (y/n): [n] y
Port type is changed. ACTION REQUIRED: Please save configurations and reload the switch

You will need to count the slots from the back of the module in the Cisco Nexus switch. Slot 1 is the last group of ports in the module. You would
normally perform this allocation only once because it is a disruptive process and your data center is usually a part of your network that you
cannot simply reboot at will. You could leave one half of the ports as Ethernet ports and convert the other half to Fibre Channel. However, this
ratio depends on your needs. Connect to the term server and observe your switch boot.

• Step 8:

Save the running configuration and reload the switch.

Tip:
In the CLI enter the following commands.
N9K-A(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete.
N9K-A(config)# reload
This command will reboot the system. (y/n)? [n] y

Because you are connected to the switch via SSH, you will need to re-establish the session to the N9K-A after it comes back online. In the pop-
up window click the OK button.
• Step 9:

Enter the username and password.

Tip:
Login using the admin username and the 1234QWer password.
login as: admin

========== Info ==========

Device: N9K-A
Pod: 1
VSAN: 1011
NetApp PWWNs:
20:0e:00:a0:98:25:60:36 (NetApp1)
20:0f:00:a0:98:25:60:36 (NetApp2)
==========================
Using keyboard-interactive authentication.
Password:

N9K-A#

• Step 10:

On N9K-A, re-investigate the status of ports fc1/47 and fc1/48.

Tip:
In the CLI enter the show interface brief command.
N9K-A# show interface brief
<... output omitted ...>
-------------------------------------------------------------------------------
Interface Vsan Admin Admin Status SFP Oper Oper Port
Mode Trunk Mode Speed Channel
Mode (Gbps)
-------------------------------------------------------------------------------
fc1/45 1 auto on sfpAbsent -- -- -- --
fc1/46 1 auto on sfpAbsent -- -- -- --
fc1/47 1 auto on down swl -- -- --
fc1/48 1 auto on down swl -- -- --
<... output omitted ...>

You should see that ports fc1/47 and fc1/48 are now configured as Fibre Channel interfaces and that their status is down.

• Step 11:

On N9K-A, enable interfaces fc1/47 and fc1/48, configure them as E Ports, set the port speed to 8000 and enable of port license activation. To
configure interface as E Ports, use the switchport mode E command from the interface configuration mode, set the speed with switchport
speed 8000 and license activation with port-license acquire.
 Tip:
In the CLI enter the following commands.
N9K-A# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
N9K-A(config)# interface fc 1/47-48
N9K-A(config-if-range)# switchport mode E
N9K-A(config-if-range)# switchport speed 8000
N9K-A(config-if-range)# port-license acquire
N9K-A(config-if-range)# no shutdown

• Step 12:

On N9K-A, re-investigate the status of ports fc1/47 and fc1/48.

Tip:
In the CLI enter the show interface brief command.
N9K-A(config-if-range)# show interface brief
<... output omitted ...>
-------------------------------------------------------------------------------
Interface Vsan Admin Admin Status SFP Oper Oper Port
Mode Trunk Mode Speed Channel
Mode (Gbps)
-------------------------------------------------------------------------------
fc1/45 1 auto on sfpAbsent -- -- -- --
fc1/46 1 auto on sfpAbsent -- -- -- --
fc1/47 1 E on notConnected swl -- -- --
fc1/48 1 E on notConnected swl -- -- --
<... output omitted ...>

The port status notConnected indicates that there is no physical connectivity to MDS; the port on MDS is currently disabled. By default, the port
has the admin mode of auto, which is the default mode. It means that the port will try to negotiate the port type with the other side.

• Step 13:

Use PuTTY to connect to your MDS.

Tip:
Open another PuTTY instance. Under Saved Sessions choose the MDS option and click the Open button.
• Step 14:

Verify the running configuration on ports fc1/1 and fc1/2. Enable and statically configure ports fc1/1 and fc1/2 as E Ports.

Tip:
In the CLI enter the following commands.
MDS# show run interface fc1/1-2
<... output omitted ...>
interface fc1/1
switchport speed 8000
switchport description Connects to N9K-A fc 1/47
port-license acquire
switchport fill-pattern IDLE speed 8000

interface fc1/2
switchport speed 8000
switchport description Connects to N9K-A fc 1/48
port-license acquire
switchport fill-pattern IDLE speed 8000

At this point, interfaces on MDS that connect to your N9K-A should be shut down, have a Fibre Channel license activated and set to port speed
8000.

MDS# configure terminal

Enter configuration commands, one per line. End with CNTL/Z.
MDS(config)# interface fc1/1-2
MDS(config-if)# switchport mode E
MDS(config-if)# no shutdown

If you issue the switchport mode ? command from the interface configuration mode on both Cisco MDS and Cisco Nexus 9000, you will see
many more options on the Cisco MDS switch. Cisco Nexus 9000 does not have a full Fibre Channel fabric implementation. Cisco has designed it
to provide basic Fibre Channel connectivity.

• Step 15:

On N9K-A, re-check the status of ports fc1/47 and fc1/48.

Tip:
Switch to the N9K-A PuTTY instance.
 In the CLI enter the show interface brief command.

N9K-A# show interface brief

<... output omitted ...>
-------------------------------------------------------------------------------
Interface Vsan Admin Admin Status SFP Oper Oper Port
Mode Trunk Mode Speed Channel
Mode (Gbps)
-------------------------------------------------------------------------------
fc1/45 1 auto on sfpAbsent -- -- -- --
fc1/46 1 auto on sfpAbsent -- -- -- --
fc1/47 1 E on trunking swl TE 8 --
fc1/48 1 E on trunking swl TE 8 --
<... output omitted ...>

Interfaces fc1/47 and fc1/48 should now have operational mode of TE and operational speed of 8 Gbps. You will not be able to set the port to
mode to TE. You configured the port mode E and then Cisco switches negotiated the operational state of TE. If one of the two switches was not
a Cisco switch, the operational mode would be E.

• Step 16:

On N9K-A, examine the B2B credits on the FC ports. Later you will compare the values with the MDS side.

Tip:
In the CLI enter the show interface fc1/47-48 command.
N9K-A# show interface fc1/47-48
<... output omitted ...>
Port mode is TE
Port vsan is 1
Speed is 8 Gbps
Transmit B2B Credit is 500
Receive B2B Credit is 64
Receive data field Size is 2112
Beacon is turned off
Trunk vsans (admin allowed and active) (1)
Trunk vsans (up) (1)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
<... output omitted ...>

Buffer-to-buffer credits (BB_Credits) are an inherent part of the Fibre Channel standards and are used as flow control mechanism. BB_Credits
represent the number of frames a port can store. Therefore, buffer credits can limit the overall throughput on a link if insufficient BB_Credits are
available to support the link distance.

When the transmitting port sends a port login request, the receiver responds with accept (ACC) frame that includes information on the size and
number of frame buffers the receiver has (BB_Credit). The transmitting port stores the BB_Credit value in a table.

• Step 17:
 On MDS, examine the B2B credits of the respective connecting port. The B2B credits should be symmetric. VSAN 1011 should be isolated.

Tip:
Switch to the MDS PuTTY instance.

In the CLI enter the show interface fc1/1-2 command.

MDS# show interface fc1/1-2

fc1/1 is trunking
Port description is Connects to N9K-A fc 1/47
<... output omitted ...>
Port mode is TE
Port vsan is 1
Admin Speed is auto
Operating Speed is 8 Gbps
Rate mode is dedicated
Port flow-control is R_RDY

Transmit B2B Credit is 64

Receive B2B Credit is 500
B2B State Change: Admin(on), Oper(down)
Receive data field Size is 2112
Beacon is turned off
Logical type is core
Trunk vsans (admin allowed and active) (1,1011)
Trunk vsans (up) (1)
Trunk vsans (isolated) (1011)
Trunk vsans (initializing) ()
<... output omitted ...>

VSAN 1011 is isolated because the interfaces have not yet been assigned to VSAN 1011. You will do that later.

• Step 18:

On MDS, assign interfaces fc1/1 and fc1/2 to VSAN 1011. Interface fc1/3 on MDS, which connects to the core MDS, is already in VSAN 1011.
Interfaces between MDS and N9K-A should currently reside in VSAN 1, which is the default VSAN. Use the vsan database command to enter
the VSAN database configuration mode. Use the vsan vsan interface interface slot/number command to assign an interface to a VSAN.

Tip:
In the CLI enter the following commands.
MDS(config-if)# vsan database
MDS(config-vsan-db)# vsan 1011 interface fc1/1, fc1/2

• Step 19:

On N9K-A, assign interfaces fc1/47 and fc1/48 to VSAN 1011. You will not be allowed to assign interfaces to VSAN 1011 until you create VSAN
1011. Use the vsan vsan command from the VSAN database configuration mode.
 Tip:
Switch to the N9K-A PuTTY instance.

In the CLI enter the following commands.

N9K-A(config)# vsan database

N9K-A(config-vsan-db)# vsan 1011 interface fc 1/47-48
vsan 1011 is not configured on the switch
N9K-A(config-vsan-db)# vsan 1011
N9K-A(config-vsan-db)# vsan 1011 interface fc 1/47-48

• Step 20:

The SAN configuration on the lab infrastructure depends on zoning for SAN separation on multiple pods. On N9K-A enable the zone enhanced
mode for VSAN 1011.

Tip:
In the CLI enter the zone mode enhanced vsan 1011 command.
N9K-A(config)# zone mode enhanced vsan 1011
WARNING: This command would distribute the zoning database of this switch throughout the fabric. Do you want to continue? (y/n) [n] y
Current zoning mode same as specified zoning mode

Note

This command is required in order to bring up the 1011. You will learn about zoning in another Discovery.

• Step 21:

On N9K-A shut down an bring back up interfaces fc1/47 and fc1/48.

Tip:
In the CLI enter the following commands.
N9K-A(config)# interface fc1/47-48
N9K-A(config-if-range)# shutdown
N9K-A(config-if-range)# no shutdown

• Step 22:

On N9K-A, investigate VSAN membership.

Tip:
In the CLI enter the show vsan membership command.
 N9K-A# show vsan membership
vsan 1 interfaces:
fc1/45 fc1/46

vsan 1011 interfaces:

fc1/47 fc1/48

vsan 4079(evfp_isolated_vsan) interfaces:

vsan 4094(isolated_vsan) interfaces:

You should see that fc1/47 and fc1/48 are members of VSAN 1011. Similarly, if you investigated VSAN membership on MDS, you would see that
fc1/1, fc1/2 and fc1/3 are members of VSAN 1011.

You would see similar results on the Cisco MDS switch.

• Step 23:

On N9K-A, investigate the VSAN database.

Tip:
In the CLI enter the show vsan command.
N9K-A# show vsan
vsan 1 information
name:VSAN0001 state:active
interoperability mode:default
loadbalancing:src-id/dst-id/oxid
operational state:up

vsan 1011 information

name:VSAN1011 state:active
interoperability mode:default
loadbalancing:src-id/dst-id/oxid
operational state:up

vsan 4079:evfp_isolated_vsan

vsan 4094:isolated_vsan

You should see four VSANs:

◦ VSAN 1: The default VSAN. You will not be able to delete it, like you cannot delete VLAN 1.
◦ VSAN 1011: The VSAN that you just created. Notice that its load balancing is "src-id/dst-it/oxid," which means that per-exchange load
balancing is enabled by default. Operational state should be "up."
◦ VSANs 4079 and 4094: Dedicated isolated VSANs where a switch places VSANs that do not exist but have assigned interfaces. Similar
to the inactive VLAN.

You would see similar results on the Cisco MDS switch.

Notice that load balancing on non-isolated VSANs is configured as src-id/dst-id/oxid. This configuration is default and it means that the switch
performs per-exchange load-balancing. You could change the load balancing to per-flow, where traffic between two given end devices would
always flow through a given link. Usually you would leave this setting at default. To change load balancing for a given VSAN, use the vsan
number loadbalancing {src-dst-id | src-dst-ox-id} command from the VSAN database configuration mode.

• Step 24:

On MDS bundle interconnecting interfaces into a port channel. Use 10 as the port channel number. Use the channel-group group command
from the interface configuration mode.

Tip:
Switch to the MDS PuTTY instance.
 In the CLI enter the following commands.

MDS(config)# interface fc1/1-2

MDS(config-if)# channel-group 10
fc1/1 fc1/2 added to port-channel 10 and disabled
please do the same operation on the switch at the other end of the port-channel, then do "no shutdown" at both ends to bring it up
MDS(config-if)# no shutdown

• Step 25:

On N9K-A bundle interconnecting interfaces into a port channel. Use 10 as the port channel number. Use the channel-group group command
from the interface configuration mode.

Tip:
Switch to the N9K-A PuTTY instance.
 In the CLI enter the following commands.

N9K-Aconfig)# interface fc1/47-48

N9K-A(config-if-range)# channel-group 10
fc1/47 fc1/48 added to port-channel 10 and disabled
please do the same operation on the switch at the other end of the port-channel, then do "no shutdown" at both ends to bring it up.
N9K-A(config-if-range)# no shutdown

You should configure the port channel identically on both devices. You will be prompted to enable the port channel using the no shutdown
command.

• Step 26:

On N9K-A and MDS, examine the port channel database. The channel mode will be ON on N9K and ACTIVE on MDS. One of the ports,
identified by the asterisk (*), is designated as the first operational port (FOP). In your case, the FOP may be different from the output.

Tip:
In the CLI enter the show san-port-channel database command.
N9K-A# show san-port-channel database
san-port-channel10
Administrative channel mode is on
Operational channel mode is on
Last membership update succeeded
First operational port is fc1/47
2 ports in total, 2 ports up
Ports: fc1/47 [up] *
fc1/48 [up]

• Step 27:

On MDS examine the port channel database. The channel mode will be ACTIVE. One of the ports, identified by the asterisk (*), is designated as
the first operational port (FOP). In your case, the FOP may be different from the output.

Tip:
Switch to the MDS PuTTY instance.
 In the CLI enter the show san-port-channel database command.

MDS(config-if)# show port-channel database

port-channel10
Administrative channel mode is active
Operational channel mode is active
Last membership update succeeded
First operational port is fc1/1
2 ports in total, 2 ports up
Ports: fc1/1 [up] *
fc1/2 [up]

The first operational port (FOP) is the primary operational interface selected in the PortChannel to carry control-plane traffic (no load-balancing).
The FOP is the first port that comes up in a PortChannel and can change if the port goes down.

The port channel is in mode ON, which is the default mode. In this mode, the PortChannel protocol is not initiated. The member ports only
operate as part of a PortChannel or remain inactive. If a PortChannel protocol frame is received from a peer port, the software indicates its
nonnegotiable status.

In the ACTIVE mode, the member ports would initiate PortChannel protocol negotiation with the peer port(s) regardless of the channel group
mode of the peer port. If the peer port, while configured in a channel group, does not support the PortChannel protocol, or responds with a
nonnegotiable status, it will default to the ON mode behavior. The ACTIVE PortChannel mode allows automatic recovery without explicitly
enabling and disabling the PortChannel member ports at either end.

• Step 28:

On MDS allow only for VSAN 1011 to traverse the interconnecting trunk links.

To limit which VSANs are allowed on the trunk link, use the switchport trunk allowed vsan vsan command from the interface configuration
mode.

If you are adding VSANs to the allowed list, you should use the add keyword before the VSAN number. Cisco Nexus Operating System (NX-OS)
does not allow you to specify a list of allowed VSANs. If you specify only one VSAN, this action will overwrite the existing configuration.

Tip:
In the CLI enter the interface port-channel 10 command.
MDS(config)# interface port-channel 10
MDS(config-if)# switchport trunk allowed vsan 1011
Warning: This command will remove all VSANs currently being trunked and trunk only the specified VSANs.
Do you want to continue? (y/n) [n] y

• Step 29:

On N9K-A allow only for VSAN 1011 to traverse the interconnecting trunk links.

Tip:
Switch to the N9K-A PuTTY instance.
 In the CLI enter the interface san-port-channel 10 command.

N9K-A(config)# interface san-port-channel 10

N9K-A(config-if)# switchport trunk allowed vsan 1011
Warning: This command will remove all VSANs currently being trunked and trunk only the specified VSANs.
Do you want to continue? (y/n) [n] y

When you apply a configuration to a port channel, the port channel will in turn apply it to all member interfaces. After you bundle interfaces into a
port channel, Cisco NX-OS will not allow you to make configuration changes on a per-interface basis.

In this interface configuration, you could also configure the channel mode, if needed. To change the port channel mode from on to auto mode,
you would use the channel mode active command. To change the port channel from auto to on mode, you would use the no channel mode
auto command.

• Step 30:

On N9K-A, investigate trunking on the SAN port channel 10 and its member interfaces.

Tip:
In the CLI enter the following commands.
N9K-A# show interface san-port-channel 10
san-port-channel 10 is trunking
<... output omitted ...>
Trunk vsans (admin allowed and active) (1011)
Trunk vsans (up) (1011)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
<... output omitted ...>
N9K-A# show interface fc1/47
fc1/47 is trunking
<... output omitted ...>
Belongs to san-port-channel 10
Trunk vsans (admin allowed and active) (1011)
Trunk vsans (up) (1011)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
<... output omitted ...>
N9K-A# show interface fc1/48
fc1/48 is trunking
<... output omitted ...>
Belongs to san-port-channel 10
Trunk vsans (admin allowed and active) (1011)
Trunk vsans (up) (1011)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
<... output omitted ...>
 You should see that the SAN port channel and its member interfaces have the same allowed VSANs (only VSAN 1011).

You could perform a similar verification on MDS.

If you are not seeing VSAN 1011 as up and active, perform the shutdown command and then no shutdown command on the SAN port channel
on both MDS and your N9K-A.

• Step 31:

On N9K-A, delete VSAN 1011 and observe what happens to VSAN membership of interfaces fc1/47 and fc1/48. Lastly, restore the original state.
Use the no vsan vsan command from the VSAN database configuration mode to delete the VSAN. Use the vsan vsan interface interface slot/
number command from the VSAN database mode to assign membership of the interfaces.

Tip:
In the CLI enter the following commands.
N9K-A(config)# vsan database
N9K-A(config-vsan-db)# no vsan 1011
Do you want to continue? (y/n) y
N9K-A(config-vsan-db)# show vsan membership
vsan 1 interfaces:
fc1/45 fc1/46

vsan 4079(evfp_isolated_vsan) interfaces:

vsan 4094(isolated_vsan) interfaces:

fc1/47 fc1/48 san-port-channel10

In the Ethernet world, if you had an interface in a particular VLAN and you deleted that VLAN, the VLAN would be moved to the "inactive" state.
The interface would lose connectivity because the previously configured VLAN was no longer present on the switch. However, if you added the
exact same VLAN back to the switch, connectivity would be restored right away.

In the Fibre Channel world, if you delete a VSAN that has member interfaces, the switch will move those interfaces into an isolated VLAN; VLAN
4094. The major difference in Fibre Channel is that only re-adding the VSAN back to the switch will not restore connectivity; you will also have to
assign interfaces back to the VSAN.

N9K-A(config-vsan-db)# vsan 1011

N9K-A(config-vsan-db)# vsan 1011 interface san-port-channel 10
N9K-A(config-vsan-db)# show vsan membership
vsan 1 interfaces:
fc1/45 fc1/46

vsan 1011 interfaces:

fc1/47 fc1/48 san-port-channel10

vsan 4079(evfp_isolated_vsan) interfaces:

vsan 4094(isolated_vsan) interfaces:

• Step 32:

The SAN configuration on the lab infrastructure depends on zoning for SAN separation on multiple pods. On N9K-A use the command zone
mode enhanced vsan vsan for enabling the zone enhanced mode for VSAN 1011.

Tip:
In the CLI enter the zone mode enhanced vsan 1011 command.
N9K-A(config)# zone mode enhanced vsan 1011
WARNING: This command would distribute the zoning database of this switch throughout the fabric. Do you want to continue? (y/n) [n] y
Current zoning mode same as specified zoning mode

Note

This command is required in order to bring up the 1011. You will learn about zoning in another Discovery.

• Step 33:

On N9K-A shut down an bring back up interface san-port-channel 10.

Tip:
In the CLI enter the following commands.
N9K-A(config)# interface san-port-channel 10
N9K-A(config-if)# shutdown
N9K-A(config-if)# no shutdown

• Step 34:

On N9K-A, verify that the VSAN 1011 is up in the SAN port channel interface.
 Tip:
In the CLI enter the show interface san-port-channel 10 command.
N9K-A(config)# show interface san-port-channel 10
san-port-channel 10 is trunking
Hardware is Fibre Channel
Port WWN is 24:02:8c:60:4f:72:06:80
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 1011
Speed is 16 Gbps
Trunk vsans (admin allowed and active) (1011)
Trunk vsans (up) (1011)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()

Task 2: Investigate FLOGI and FCNS Databases

Show Steps

• Step 1:

On MDS investigate the FLOGI database.

Tip:
Switch to the MDS PuTTY instance.

In the CLI enter the show flogi database command.

MDS(config-if)# show flogi database

No flogi sessions found.

• Step 2:

On N9K-A investigate the FLOGI database.

Tip:
Switch to the N9K-A PuTTY instance.
 In the CLI enter the show flogi database command.

N9K-A(config-if)# show flogi database

No flogi sessions found.

The FLOGI in SAN is similar to the MAC address table in Ethernet. You should not see any FLOGI database entries because no initiators
(servers) or targets (storage) are connected to the switches.

• Step 3:

On MDS investigate the FCNS database.

Tip:
Switch to the MDS PuTTY instance.
 After sending a FLOGI, each device will register with the name server in the switch to which the device is connected. Switches then distribute
information to all switches within the same fabric or VSAN, so that the FCNS database contains the same information on every fabric switch.

In the CLI enter the show fcns database command.

MDS(config-if)# show fcns database

VSAN 1011:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x0b49aa N 20:0e:00:a0:98:25:60:36 (NetApp) scsi-fcp:target
[target1]
0x0b4aaa N 20:0f:00:a0:98:25:60:36 (NetApp) scsi-fcp:target
[target2]
Total number of entries = 2

The FCNS database contains the pWWN that you identified in the FLOGI table. It belongs to the NetApp storage (notice that the type is target. If
you had a server that was logged in to the fabric, the type would be initiator.)

The FCNS is analogous to the ARP database; it translates between Layer 2 (WWNs) and Layer 3 (FCIDs) addresses. FCNS is end-to-end for a
given VSAN. If you do not see all entries, the fabric is broken. Possible reasons are VSAN isolation, EISL not allowing VSANs to traverse, and
so on.

• Step 4:

On N9K-A investigate the FCNS database.

Tip:
Switch to the N9K-A PuTTY instance.
 In the CLI enter the show fcns database command.

N9K-A(config-if)# show fcns database

VSAN 1011:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x0b49aa N 20:0e:00:a0:98:25:60:36 (NetApp) scsi-fcp:target
0x0b4aaa N 20:0f:00:a0:98:25:60:36 (NetApp) scsi-fcp:target
Total number of entries = 2

Since N9K-A is now connected to MDS using a trunk link where only VSAN 1011 is allowed to traverse, you should see the same FCNS entries
on N9K-A as you do on MDS.

Task 3: Configure FCoE

Configure FCoE for the SAN-A side of your storage network. The server has already been configured. You will configure FCoE from the N9K-A switch to
the server.

Show Steps

• Step 1:

On N9K-A, enable Link Level Discovery Protocol (LLDP) using the feature lldp command. Data Center Bridging Capability Exchange (DCBX)
protocol is an extension of the LLDP and is used by the FCoE.

Tip:
In the CLI enter the feature lldp commands.
N9K-A(config-if)# feature lldp

• Step 2:

N9K-A is connected to the UCS Server over the Ethernet 1/5 interface. On this interface, you are configuring FCoE. Since FCoE requires
specific Quality of Service (QoS) settings, examine the preconfigured QoS policy map using the show policy-map default-fcoe-in-policy
command. Then apply that policy in the inbound direction on the Ethernet 1/5 interface.

Tip:
In the CLI enter the following commands.
N9K-A(config)# show policy-map default-fcoe-in-policy

Type qos policy-maps

====================
 policy-map type qos default-fcoe-in-policy
class default-fcoe-in-qos
set cos 3
set qos-group 1
N9K-A(config)# interface ethernet 1/5
N9K-A(config-if)# service-policy type qos input default-fcoe-in-policy

• Step 3:

Since Fibre Channel frames have a payload of 2112 bytes, enable jumbo frames on the interface. On N9K-A Configure the maximum
transmission unit (MTU) to 9216 bytes on the eth1/5 interface. If you examine the default QoS policy for FCoE on N9K-A, you can see that qos-
group 1, which you have configured in a previous step, requires the MTU of 9216.

Tip:
In the CLI enter the following commands.
N9K-A(config-if)# mtu 9216
N9K-A(config-if)# show policy type network-qos default-fcoe-nq-policy

Type network-qos policy-maps

============================
policy-map type network-qos default-fcoe-nq-policy
class type network-qos c-nq3
match qos-group 3
mtu 1500
class type network-qos c-nq2
match qos-group 2
mtu 1500
class type network-qos c-nq1
match qos-group 1
pause pfc-cos 3
mtu 9216
class type network-qos c-nq-default
match qos-group 0
mtu 1500

• Step 4:

You have to allocate one VLAN to carry the FCoE traffic. Configure VLAN 1000 to carry VSAN traffic. Under VLAN configuration, associate the
VLAN with the POD_1011.

Tip:
In the CLI enter the following commands.
N9K-A(config-if)# vlan 1000
N9K-A(config-vlan)# fcoe vsan 1011

• Step 5:

Make sure that the interface eth1/5 on N9K-A is configured as a trunk and that VLAN 1000 is allowed on it.

Tip:
In the CLI enter the show interface ethernet 1/5 trunk command.
N9K-A(config-vlan)# show interface ethernet 1/5 trunk

--------------------------------------------------------------------------------
Port Native Status Port
Vlan Channel
--------------------------------------------------------------------------------
Eth1/5 1 trunking --

--------------------------------------------------------------------------------
Port Vlans Allowed on Trunk
--------------------------------------------------------------------------------
Eth1/5 1-4094

--------------------------------------------------------------------------------
Port Vlans Err-disabled on Trunk
--------------------------------------------------------------------------------
Eth1/5 none

--------------------------------------------------------------------------------
Port STP Forwarding
--------------------------------------------------------------------------------
Eth1/5 1,100,1000

--------------------------------------------------------------------------------
Port Vlans in spanning tree forwarding state and not pruned
--------------------------------------------------------------------------------
Eth1/5 Feature VTP is not enabled
1,100

• Step 6:

On N9K-A, create the vFC interface 1. The vFC interface is the virtual Fibre Channel (FCoE) interface that the switch will use to communicate
with the FCoE interface on the Cisco UCS server. Bind the vFC with the physical Ethernet 1/5 interface.
 Tip:
In the CLI use the following commands.
N9K-A(config-vlan)# interface vfc1
N9K-A(config-if)# bind interface ethernet 1/5

• Step 7:

Configure the vFC1 interface in fabric mode (F Port). This port may be connected to a peripheral device (host or disk) operating as a node port
(N port). Also, enable your 1011 on the interface and enable the interface.

Tip:
In the CLI use the following commands.
N9K-A(config-if)# switchport mode F
N9K-A(config-if)# switchport trunk allowed vsan 1011
Warning: This command will remove all VSANs currently being trunked and trunk only the specified VSANs.
Do you want to continue? (y/n) [n] y

• Step 8:

Verify the running configuration of interface vFC 1.

Tip:
In the CLI enter the show running-config interface vfc1 command.
N9K-A(config-if)# show running-config interface vfc1

!Command: show running-config interface vfc1

!Running configuration last done at: Wed Apr 26 10:30:54 2021
!Time: Wed Apr 26 10:31:06 2021

version 9.3(5) Bios:version 05.39

interface vfc1
bind interface Ethernet1/5
switchport trunk allowed vsan 1011

• Step 9:

The fabric mode is the default mode on vFC interfaces. View interface configuration together with default settings.

Tip:
In the CLI enter the show running-config interface vfc1 all command.
N9K-A(config-if)# show running-config interface vfc1 all

!Command: show running-config interface vfc1 all

!Running configuration last done at: Wed Apr 26 10:35:56 2021
!Time: Wed Apr 26 10:36:08 2021

version 9.3(5) Bios:version 05.39

interface vfc1
bind interface Ethernet1/5
switchport mode F
switchport trunk allowed vsan 1011
shutdown
switchport speed auto
switchport trunk mode on
link-state-trap

• Step 10:

Enable the vfc1 interface.

Tip:
In the CLI enter the no shutdown command
N9K-A(config-if)# no shutdown

• Step 11:

On the N9K-A switch, assign the interface vFC1 to 1011. Enter the VSAN database configuration mode using the vsan database command.
Then assign the vFC interface to a VSAN as you would if it was a native Fibre Channel interface. Use the vsan 1011 interface vfc1 command to
assign the interface.

Tip:
In the CLI enter the following commands.
N9K-A(config-if)# vsan database
N9K-A(config-vsan-db)# vsan 1011 interface vfc1

• Step 12:

On the N9K-A, verify that vFC1 is a member of 1011.

 Tip:
In the CLI enter the show vsan 1011 membership command.
N9K-A(config-vsan-db)# show vsan 1011 membership
vsan 1011 interfaces:
fc1/47 fc1/48 san-port-channel10 vfc1

You have four interfaces as members of 1011. Those four ports are:

◦ The vFC 1 interface serves the FCoE part (downstream) of the topology.
◦ Interfaces FC1/47 and FC1/48 toward the MDS switch.
◦ Interface SAN port-channel 10, representing bundle created from interfaces FC1/47 and FC1/48.

Note

Interface Ethernet 1/5 is not part of the VSAN. It is only a physical carrier interface for the vfc1 interface.

• Step 13:

Investigate the status of the vFC1 interface

Tip:
In the CLI enter the show interface vfc1 command.
N9K-A# show interface vfc1
vfc1 is trunking
Bound interface is Ethernet1/5
Hardware is Ethernet
Port WWN is 20:00:c0:14:fe:f9:93:5f
Admin port mode is F, trunk mode is on
snmp link state traps are enabled
Port mode is TF
Port vsan is 1011
Operating Speed is 10 Gbps
Admin Speed is auto
Trunk vsans (admin allowed and active) (1011)
Trunk vsans (up) (1011)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
91 fcoe in packets
9240 fcoe in octets
0 fcoe out packets
0 fcoe out octets
Interface last changed at Fri Nov 24 09:03:03 2023

The vFC1 interface is trunking and 1011 is "admin allowed and active." FCoE is up between the N9K-A switch and the Cisco UCS server. FCoE
is up because the configuration of the FCoE on the Cisco UCS server is preconfigured.

• Step 14:

On the N9K-A switch, verify that the Cisco UCS server managed to perform fabric login (FLOGI). Successful FLOGI is a proof that FCoE works
and that the Cisco UCS server can reach the rest of the SAN using the FCoE protocol. Fibre Channel ID (FCID) value is allocated by the N9K-A
switch during fabric login of the Cisco UCS server.

Tip:
In the CLI enter the show flogi database command.
N9K-A# show flogi database
------------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME

------------------------------------------------------------------------------
vfc1 1011 0xe10000 20:00:00:25:b5:00:08:01 20:00:00:25:b5:00:08:01

Total number of flogi = 1.

• Step 15:

On the N9K-A switch, investigate the FCNS database.

Tip:
In the CLI enter the show fcns database command.
N9K-A# show fcns database

VSAN 1011:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x133ea6 N 20:0e:00:a0:98:25:60:36 (NetApp) scsi-fcp:target
0x1340a6 N 20:0f:00:a0:98:25:60:36 (NetApp) scsi-fcp:target
0x910000 N 20:00:00:25:b5:00:01:01 (Cisco)
 Total number of entries = 1

You can see the Node (N) Port with the FCID that you saw in the previous step. Notice that the server (initiator) is marked as scsi-fcp:init fc-gs.

• Step 16:

Examine the FCNS database on the Cisco MDS switch.

Tip:
Switch to the MDS PuTTY instance.

In the CLI enter the show fcns database command.

MDS# show fcns database

VSAN 1011:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x133ea6 N 20:0e:00:a0:98:25:60:36 (NetApp) scsi-fcp:target
0x1340a6 N 20:0f:00:a0:98:25:60:36 (NetApp) scsi-fcp:target
0x910000 N 20:00:00:25:b5:00:01:01 (Cisco)

Total number of entries = 1

• Step 17:

Examine the FLOGI database on the Cisco MDS switch.

Tip:
In the CLI enter the show flogi database command.
MDS# show flogi database
No flogi sessions found.

The FLOGI database on the Cisco MDS switch is empty since the Cisco UCS server has logged into SAN by using the N9K-A switch.

• Step 18:

You have completed the lab activity. Click the Finish button to go to the Summary slide.

Tip:
Click the Finish button.
Lab Completion Instructions
You have now completed this lab exercise.
Please click 'End Session'.

Choose 'Exit'.