Virtual Connect CLI Network Management

Guide with Nexus vPC Configuration

Feedback: [email protected]

Page 1
 Revision History

Revision Number Date Changes

1.1 1/11/2012 First Edition

Page 2
 Table of Contents

• Introduction
• Network Hardware Topology and Virtual Connect Port Numbering
• Virtual Connect Terminology and CLI Summary Table
• Virtual Connect CLI Snapshots
• Troubleshooting Scenario 1: Track Down MAC Address
• Troubleshooting Scenario 2: Display VLAN MAC table
• Appendix #1: vPC Configuration: Nexus5000-1
• Appendix #2: vPC Configuration: Nexus5000-2
• Appendix #3: Configuration: Virtual Connect
• Reference Links

Page 3
 Introduction

Page 4
 Introduction
• This guide is targeted to network administrators who want to manage Virtual Connect network part
of configuration by CLI. It also demos how to configure Nexus vPC with Virtual Connect.

• Most of Virtual Connect management tasks can be categorized into two parts: Network and Server
Profile. Some customers network team will handle VC network configuration while server team will
handle VC Server Profile configuration. Others may choose to have server admins to own the
configuration for the whole Virtual Connect module.

• Virtual Connect can be managed by GUI or CLI mode. Each mode offers full configuration
functionality.

• Many network admins like to manage VC by CLI and this guide is to help them on this task.
Network configuration can be done by GUI mode as well. Please refer to VC GUI user guide and
other links at the end of this presentation for more GUI configuration information.

• This guide does not cover server profile part of CLI management as many server admins prefer
VC GUI for server configuration. Please refer to VC CLI user guide link at the end of this
presentation on CLI syntax to manage server profile.

Page 5
 Network Hardware Topology and
Virtual Connect Port Numbering

Page 6
 Network Hardware Topology
VPC
N5K-1 po1000 N5K-2

Eth1/3 Eth1/4 vpc peer-link Eth1/4 Eth1/3

po1 on N5Ks po2 on N5Ks

X1 X2 X2 X1

d1 d1
VC-1 VC-2
Enclosure Internal Connection

LOM:1 LOM:2

Blade Server in Device Bay 1

Page 7
 Virtual Connect FlexFabric Module Port Numbering

X1 X2 X3 X4 X5 X6 X7 X8

d1 d2 d3 d4 d5 d6 d7 d8 d9 d10 d11 d12 d13 d14 d15 d16

Enclosure Internal Connection

Bay Bay Bay Bay Bay Bay Bay Bay Bay Bay Bay Bay Bay Bay Bay Bay
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Note:
• X1-X4: Besides 10G ethernet, these 4 ports can be configured as native Fibre Channel(Note: it’s FC, not
FCOE) on port-by-port basis to connect with SAN switches. Default port mode is ethernet.
• X5-X8: Support 1G or 10G ethernet on port-by-port basis
Page 8
 Virtual Connect Flex-10 Module Port Numbering

X1 X1 X2 X3 X4 X5 X6 X7 X8

d1 d2 d3 d4 d5 d6 d7 d8 d9 d10 d11 d12 d13 d14 d15 d16

Enclosure Internal Connection

Bay Bay Bay Bay Bay Bay Bay Bay Bay Bay Bay Bay Bay Bay Bay Bay
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Note:
• X1 is a shared port between SFP+ and CX4 media. You can only use one media at a time.
• X1-X8: These SFP+ 10G ports can also support 1G on port-by-port basis with supported SFPs plugged in.

Page 9
 VC Port Numbering Syntax in the VC Domain

Enc0:1:X1 Enc0:1:X2 Enc0:2:X1 Enc0:2:X2

VC-1 VC-2

Enc0:1:d1 Enc0:1:d8 Enc0:1:d16 Enc0:2:d1 Enc0:2:d8 Enc0:2:d16

d1
Enclosure Internal Connection

LOM:1 LOM:2 LOM:1 LOM:2 LOM:1 LOM:2

Bay Bay Bay

Page 10
1 8 16
 Virtual Connect Terminology and CLI
Summary Table

Page 11
 Virtual Connect/Cisco Terminology Translation

Cisco Virtual Connect

VLAN Ethernet Network or Virtual Connect Network
(Vnet)
Spanning Tree N/A (VC does not use STP to communicate with network switches)
Access port Ethernet Network port
Trunk port Shared Uplink Set(SUS) Port
Port Channel No need to define a static port channel interface. Ports under one vnet
or SUS will try to negotiate LACP bundle with network switch and
generate a link aggregation group (LAG) ID automatically.

Page 12
 Virtual Connect/Cisco CLI Comparison
Cisco Virtual Connect
Show version Show version
Show module Show interconnect
Show device-bay
Show cdp neighbor N/A (VC doesn’t support CDP)
Show lldp neighbor Show lldp
Show interface brief Show uplinkport
Show server-port
System Show spanning-tree N/A (VC doesn’t use STP to communicate with network switches)
Information Show vlan Show network
• If ports are trunk ports • If ports are trunk ports
Show int switchport Show uplinkset <uplinkset name>
Show int trunk
• If ports are access ports
• If ports are access ports Show network <ethernet network name>
Show int switchport
• If ports are in LACP bundle
• If ports are in LACP bundle Show network <ethernet network name>
Show port-channel summary Show uplinkset
Show port-channel database Show uplinkset <uplinkset name>
Show uplinkport <port number>
• Configuring access port • Configuring upstream access port
Vlan <number> Add network <name>
Switchport mode access Add uplinkport <port> network=<name>
Switchport access vlan <number>
• Configuring upstream trunk port
• Configuring trunk port Add uplinkset <name>
Configuration Vlan <number> Add uplinkport <number> uplinkset=<name>
Switchport mode trunk Add network-range
Switchport trunk allowed vlan <number>
• Configuring upstream port-channel
• Configuring port-channel No need to configure manually. Automatically negotiated
Channel group <number> mode
• Note: Downlink server ports mode are configured in Server Profile
Page 13 Int port-channel <number>
 Virtual Connect/Cisco CLI Comparison (Continue)

Cisco Virtual Connect

Show interface Show statistics
Show interface counter Show statistics-throughput
Clear counters Reset statistics
Show mac address-table Show interconnect-mac-table
Monitoring & Show process cpu/mem Show interconnect <Module>
Troubleshooting Monitor session/Show monitor Add/show port-monitor
Show diagnostics Show status
Show logging Show systemlog
Show tech Show all *
Save supportinfo
Show running-config Show config
Image and
Configuration Wr mem or copy start run N/A (VC automatically saves configuration after each config line)

Management Dir bootflash: or flash: N/A (VC manage flash space internally)
Copy tftp flash or install Using HP SUM or VCSU
Copy run ftp/tftp Save configbackup

Page 14
 How to Use CLI Online Help
• Virtual Connect CLI offers detailed online help.
• Virtual Connect CLI configuration is one level architecture instead of 3 levels (Exec, Global Config and Interface Config) like
Cisco CLI.
• Configuration is saved automatically after each CLI line.

Virtual Connect CLI syntax consists of two

major components.
1) Subcommand, which is an action like
add, delete, set.

Virtual Connect CLI syntax consists of two

major components.
2) Managed Elements, which is a property
you can configure, like network (VLAN) or
uplinkset (trunk).
15
Page 15
 How to Use CLI Online Help
• For example, if you want to find out anything around VLAN config, first do “help network”
• And then display a specific subcommand help syntax on this element.
• There is “help –all” option to dump all help info.

Page 16 16
 CLI Filter Function
• Often in show commands, you want to filter information to something you focus on. VC show commands will have column
name at the top and you can filter by one column or several columns at the same time.

Show
network

Show
network,
Filter by
VLAN ID

Show network,
Filter by VLAN
ID and SUS
name at the
same time

Page 17
 Virtual Connect CLI Snapshots

Page 18
 Show Version
(Cisco: show version)

VC 3.30 release

Page 19
 Show Interconnect
(Cisco: show module)

VC Model Name

Enc0:1  VC in I/O bay 1 Virtual Connect Manager (VCM) status. VCM is VC control
Enc0:2  VC in I/O bay 2 process. For control plane management, only one VC module
is active in a VC domain. A typical VC domain is a pair of VC
Enc0 means local enclosure. In the case of modules inside a HP blade enclosure.
VC stacking, you may see enc1, enc2 or
enc3 for remote enclosures. Both modules can forward user traffic at the same time from
data plane perspective.

Page 20
 Show devicebay
(Cisco: show module)

Server in device bay 7. Server model name Server Profile Name

Page 21
 Show lldp
(Cisco: show lldp neighbor)

VC FlexFabric or Flex-10 Module

Enc0:1:X1 means local

enclosure, VC in I/O Bay 1, X1 X2 X8
the first uplink port - X1

LLDP neighbor information

Page 22
 Show Uplinkport
(Cisco: show int brief for uplinks)

Which Ethernet
Network or SUS owns
the port. In this case,
it’s SUS “n5k-trunk-a”.

Enc0:1:X1  local The port is in forwarding

enclosure, VC in I/O Bay 1, state
the first uplink port - X1

Link Aggregation Group (LAG) ID,

dynamically generated and used by VC
to bundle LACP individual physical
links. It’s like a dynamic int port-channel
<int number> on a Cisco switch.

LLDP neighbor info of Chassis

ID and Port

Page 23
 Show server-port
(Cisco: show int brief for downlinks)

Midplane wired to Server in device bay 1,

VC module in I/O the first 10G LAN-On-
bay 1 Motherboard port

Server in
device bay 1

VC module port
number for this
server 10G port
First 10G port Enc0:1:d1 means
on this blade local enclosure, VC
server module in I/O bay 1,
the first downlink
port

Page 24
 Show network
(Cisco: show vlan)
The network (VLAN) is defined
under the SUS (trunk) “n5k-trunk-
a”

Ethernet
network name

Show all VC networks with VLAN id 400. “VLAN

id” is one of columns’ header which can be
used by filter function

Page 25
 Show Uplinkset
(Cisco: sh int trunk, show int switchport, show port-channel summary)
“n5k-trunk-a” is the SUS(trunk) defined for VC module in I/O
Shows the list bay 1 and “n5k-trunk-b” is the SUS(trunk) defined for VC
of all Shared module in I/O bay 2. Both links under each SUS are active.
Uplink Set (SUS) This shows successful LACP bundle negotiation with
defined. upstream switch. Two LAG ID(one for each LACP bundle)
will be generated.

To see the bundle LACP ID generated by VC, use “show

uplinkport <port number>” command.

vnets (VLANs)
Page 26 defined under the
trunk
 Show network <ethernet network name>
(Cisco: show int switchport for an access port)

This uplink port is solely owned by

the network so it’s an access port.
Only untagged packets will be sent
to and received from the upstream
switch

Page 27
 Add network <name>
Configuring an access port to upstream switch

1. Define a vnet (VLAN)

2. Assign upstream ports to this vnet
3. Enable smartlink feature for this vnet.

Note: Smartlink feature is like link state tracking in cisco switch

so if the uplink port is down physically, Virtual Connect will
disable all server downlinks associated with this network in
order to trigger server side NIC teaming failover. In most
cases, you should enable this feature.

Page 28
 Add uplinkset <name>
Configuring a trunk port to upstream switch

New in VC3.30

Note: In 3.30, “show config” will show the defined network line by line
instead of network-range like what we did here in “add network-range”.
“show config” display will be enhanced in future version.

1. Define a SUS (trunk)

2. Assign upstream ports to this SUS
3. Add vnets (VLANs) under this SUS and enable
smartlink for vnets.

Note: Smartlink feature is similar to Link State Tracking in

Cisco switch so if the uplink port is down physically, Virtual
Connect will disable all server downlinks associated with
this network in order to trigger server side NIC teaming
failover. In most cases, you should enable this feature.

The Shared Uplink Set just

created

Page 29
 Show statistics
(Cisco: show interface counter)

Detailed historical interface

counters. Please see VC
CLI user guide reference
link at the end of
presentation for detailed
explanation of each field.

Page 30
 Show statistics-throughput New in VC3.30
(Cisco: show interface traffic throughput counters)

Traffic throughput statistics in the last one

hour for VC module 1, first 10G downlink
port, which is connected to the first 10G
motherboard NIC of blade server in device
bay 1.

The traffic was 500pps

stream from server blade.
“Packets Received”
column shows what VC
received from the server,
“Packets Transmitted”
Column shows what VC
sent to the server

Some other interfaces numbers examples:

Enc0:1:X1 Local Enclosure, VC module 1, first uplink port

Enc0:2:d1 Local Enclosure, VC module 2, first 10G downlink port, which is connected
to the second 10G motherboard NIC of blade server in device bay 1.

Page 31
 Reset statistics
(Cisco: clear counter)

Clear counter for local enclosure, VC

module in I/O bay 1, the first uplink port

Page 32
 Show interconnect-mac-table New in VC3.30
(Cisco: show mac address-table)

Display VC-1 MAC table, enc0:1 means local

enclosure VC module in I/O bay 1.

Also filter by a mac address in “mac

address” column

Display VC-2 MAC table

For detailed MAC address troubleshooting, please check section Troubleshooting Scenario 1: Track a MAC
address inside Virtual Connect

Page 33
 Show interconnect <module>
(Cisco: show proc cpu/mem)

New in VC3.30

The CPU load number here is not a % value like show proc cpu on Cisco platforms. It’s Linux “Load Averages”. A good link
to explain this number is at:
http://blog.scoutapp.com/articles/2009/07/31/understanding-load-averages

For memory utilization, in general as long as free memory is above 10%, users shouldn’t be concerned.

Page 34
 Show config New in VC3.30
(Cisco: show run)

Scroll down and the part of Shared

Uplink Set (trunk) configuration

Scroll down and the part of vnet

(VLAN) configuration
Page 35
 Show/add port-monitor
(Cisco: monitor session)

Source Port: It’s “Monitor Port” in VC term, enc0:1:d6 means to capture traffic on local enclosure, VC module in I/O bay 1,
10G downlink port 6, which is connected to the first 10G port of server in device bay 6.

Destination Port: It’s “Analyzer Port” in VC term, enc0:1:4 means to send captured server traffic to uplink port 4 on local
enclosure, VC module in I/O bay 2. You can connect a traffic analyzer on this port.

Note: Currently VC only supports server link traffic capture. VC uplink traffic capture needs to be done at network switch side.

Page 36
 Show status
(Cisco: Show diagnostics)

VC Domain alert status

Page 37
 Show systemlog
(Cisco: show logging)

Note: Syslog server is set by “set log-target”

Page 38
 Show all *
(Cisco: Show tech)

Show all or show all * shows general configuration and

information on the system. For deep-dive troubleshooting
information, “save supportinfo” (next slide) needs to be
used.

Page 39
 Save Supportinfo
(Cisco: show tech)

The output file can be copied to management server through

FTP or TFTP. You have the option to specify filename.

The file is in binary format. Only HP support has the ability to

decode the support dump.

Page 40
Virtual Connect Upgrade Using Virtual Connect Support Utility (VCSU)
10.1.63.2: OA address
-u: OA username
-p: OA password
-vcu: VC username
-vcp: VC password

VC can be upgraded by HP
Smart Update Manager
(SUM) or VCSU.

VCSU is a small windows

utility. It is not part of VC CLI.

This upgrade mode is

different than Cisco
router/switches where you do
‘upgrade” or “copy tftp flash”
inside NX-OS or IOS.

By using HP SUM or VCSU,

admins can upgrade multiple
VC domains at the same time
by pointing to different OA
addresses.

VCSU can be downloaded at

http://h20000.www2.hp.com/bizsupport/Te
chSupport/SoftwareIndex.jsp?lang=en&cc
=us&prodNameId=4144085&prodTypeId=
3709945&prodSeriesId=4144084&swLang
=8&taskId=135&swEnvOID=1093

41
Page 41
 Save configbackup

The exported config file is a binary file which you can load it
back to the same enclosure to restore domain config. To
see config in text format, please use “show config”.

Please don’t load this binary file to other enclosures as a

way to replicate configuration as the config binary file
contains specific internal domain information.

Page 42
 Troubleshooting Scenario 1:

Objective: Track down the path of MAC address 00:50:56:12:12:12 (a VM NIC

address) and understand the logical topology related with it.

Page 43
 Step 1: Show Interconnect-mac-table New in VC3.30

4
This entry shows
the VC-1 learned 1
the MAC in another First show VC-1 mac table
VLAN (internal matching this address
ID=235) from
internal stack link
(LAG ID=26). For
the server data
forwarding, this
entry won’t be used
by VC-1 as “d1” port
5
is not associated But How about these
with this VLAN. LAG IDs and Internal
IDs? We’ll find out in
the next 2 slides for
full topology
3 Now we know this MAC is learned from d1 port on VC-1 module, 2
this port is mapped to the server motherboard 10G NIC1 (LOM:1)
Then show VC-2 mac table
in bay 1. Vswitch uses the FlexNIC inside LOM:1 as its uplink and
hashed vm1 MAC address to the FlexNIC on server side. That’s
matching this address
the reason the incoming traffic from this MAC comes in d1 port

Page 44
 Step 2: Show Network * for VLAN Info
Which VLAN does
Which VLAN does “Internal ID” 235 refer to?
“Internal ID” 233 refer to?

It’s VLAN 400 mapped to vnet (VLAN) It’s VLAN 400 mapped to vnet (VLAN)
vnet-400-a under SUS (trunk) n5k-trunk- vnet-400-b under SUS (trunk) n5k-trunk-
a. We know this SUS is defined in VC-1 b. We know this SUS is defined in VC-2
using X1 and X2 ports. Or you can use using X1 and X2 ports. Or you can use
“show uplinkset <name> to find out this “show uplinkset <name> to find out this
info info

Page 45
 Step 3: Show Uplinkport * for Port-Channel Info
Which Port-channel is using
LAG ID 26 in VC-1
Which Port-channel is using
LAG ID 25 in VC-2

Which Port-channel is using

LAG ID 26 in VC-2

It’s VC-1 internal stacking

link bundle of X7 and X8 to
VC-2 It’s VC-2 internal stacking
link bundle of X7 and X8 to
VC-1

It’s VC-2 SUS (trunk) n5k-

trunk-b using X1 and X2
port.
Page 46
 Network Logical Diagram Discovered
(VC is using Active/Active Design)

N5K vPC

po1 po2

VLAN 400 VLAN 400

LAG25 LAG26

VC-1 VC-2
LAG26

LAG25
vnet-400-a Internal Stacking Link vnet-400-b
VLAN ID=400, Internal ID=233 Only for VC internal VLAN ID=400, Internal ID=235
heartbeat and config
synch. No user traffic
d1 traversed d1

VLAN 400 VLAN 400

LOM:1 LOM:2

MAC A vSwitch
00:50:56:12:12:12

vm1 Blade 1

HP BladeSystem Enclosure
Page 47
 Network Logical Diagram with CLI Captures

N5K vPC
po1 po2

VLAN 400 VLAN 400

LAG25 LAG26

VC-1 LAG26 VC-2

LAG25
vnet-400-a Internal Stacking Link vnet-400-b
VLAN ID=400, Internal ID=233 VLAN ID=400, Internal ID=235
Only for VC internal
heartbeat and config
synch. No user traffic
d1 traversed d1

VLAN 400 VLAN 400

vswitch

vm1 Blade 1

Page 48
HP BladeSystem Enclosure
 Troubleshooting Scenario 2:

Objective: Display VLAN 400 MAC address table in both VC modules

Page 49
 Step 1: Show Network * for Internal ID

First do “show network *” filter by

“VLAN id” to find out which vnets are
mapped to this VLAN and what are
internal IDs for vnets

vnet (VLAN) vnet-400-a under SUS (trunk) n5k-

trunk-a is mapped to this VLAN. Internal ID is 233.

If you know vnet-400-a or n5k-trunk-a is

configured in VC-1, you can skip Step 2 and go to
Step 3 directly for show mac

vnet (VLAN) vnet-400-b under SUS (trunk) n5k-

trunk-b is mapped to this VLAN. Internal ID is 235.

If you know vnet-400-b or n5k-trunk-b is

configured in VC-2, you can skip Step 2 and go to
Step 3 directly for show mac

Page 50
 Step 2: Show Uplinkset to See Where Is the SUS/vnet

SUS n5k-trunk-a is in VC-1 (enc0:1)

so we know internal ID 233 is inside
VC-1

SUS n5k-trunk-b is in VC-2 (enc0:2)

so we know internal ID 235 is inside
VC-2

Page 51
 Step 3: Show Interconnect-mac-table for VLAN MAC
Table

In VC-1, do “show
interconnect-mac-table” for
VLAN 400 (internal id=233)

In VC-2, do “show
interconnect-mac-table” for
VLAN 400 (internal id=235)

Page 52
 Nexus5000-1 vPC Configuration and
Show Information

Page 53
 N5K-1 vPC Configuration

Best Practice:
Configure STP edge port and enable BPDU Guard feature on
ports connecting with Virtual Connect. VC doesn’t talk STP with
network switch, treat VC uplink ports as server ports.

Some customers choose to use Nexus 5K+2K to connect with

Virtual Connect. This also demos that VC is a host device to
switch layer because N2K ports are host ports with BPDU Filter
and Guard turned on.

Page 54
 N5K-1 Show vPC

Page 55
 N5K-1 Show LLDP and LLCP

Page 56
 N5K-1 Show Spanning-tree

You should never receive any BPDU

packets from Virtual Connect
Page 57
 Nexus5000-2 vPC Configuration and
Show Information

Page 58
 N5K-2 vPC Configuration

Best Practice:
Configure STP edge port and enable BPDU Guard feature on
ports connecting with Virtual Connect. VC doesn’t talk STP with
network switch, treat VC uplink ports as server ports.

Some customers choose to use Nexus 5K+2K to connect with

Virtual Connect. This also demos that VC is a host device to
switch layer because N2K ports are host ports with BPDU Filter
and Guard turned on.

Page 59
 N5K-2 Show vPC

Page 60
 N5K-2 Show LLDP and LLCP

Page 61
 Topology N5K-2

You should never receive any BPDU

packets from Virtual Connect
Page 62
 Virtual Connect Configuration

Page 63
 Configuration Virtual Connect

Page 64
 References

Virtual Connect Release 3.30 CLI User Guide.

http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c02996642/c02996642.pdf

Virtual Connect Release 3.30 GUI User Guide

http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c02996013/c02996013.pdf

Virtual Connect for Cisco Network Administrators

http://h20000.www2.hp.com/bc/docs/support/SupportManual/c01386629/c01386629.pdf

Virtual Connect FlexFabric Cookbook

http://h20000.www2.hp.com/bc/docs/support/SupportManual/c02616817/c02616817.pdf
Note: Virtual Connect configuration examples in GUI mode

Virtual Connect Ethernet Cookbook

http://h20000.www2.hp.com/bc/docs/support/SupportManual/c01990371/c01990371.pdf

Virtual Connect and HP Networking Switch IRF Integration Guide

http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c02843088/c02843088.pdf
Note: The same VC design and GUI configuration snapshots also apply to vPC/VSS design.

Page 65