White Paper

Cisco HyperFlex Systems

Converting to Distributed Virtual Switches for Cisco HyperFlex VM Guest and
vMotion Networks

Author: Brian Everitt

August 2016

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 1 of 20
 Contents
Solution Overview
Document Purpose
Solution Purpose
Business Challenge
The Solution
Solution Benefits
Solution Requirements
Known Constraints
Conversion Procedure
Create the Distributed Switches
Configure the Virtual Machine Guest Distributed Switch
Configure the VMotion Distributed Switch
Migrate to the Virtual Machine Guest Distributed Switch
Migrate to the VMware vMotion Distributed Switch
Test and Clean Up
Conclusion
For More Information
Appendix A: Configuring Additional Virtual Machine Guest VLANs
Appendix B: Alternative Configuration for Tagged VLANs

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 2 of 20
 Executive Summary
VMware offers the vSphere Distributed Switch (vDS or dvSwitch) feature for centralized management of virtual
switches in VMware ESX clusters. Since the introduction of vDS in ESX 4.0, this feature has gained widespread
™
adoption in many virtualized landscapes. With the initial launch of Cisco HyperFlex Systems, the networking
configuration of our hyperconverged product is installed and configured automatically using standard ESX virtual
switches (vSS or vSwitch). To better align with the common practice of using vDS when customers are licensed to
do so, Cisco allows the conversion of networks that do not support Cisco HyperFlex infrastructure, such as virtual
machine guest networks and VMware vMotion networks, to use distributed virtual switches instead of standard
virtual switches.

Solution Overview
This section defines current IT Infrastructure challenges and presents a high-level view of the solution and its
benefits.

Document Purpose
This document describes how to convert guest networks and port groups in a Cisco HyperFlex cluster to use vDS
instead of the standard vSwitches.

Solution Purpose
™ ®
Cisco HyperFlex solutions are built on the Cisco Unified Computing System (Cisco UCS ) platform. They offer
faster deployment, greater flexibility and efficiency at a competitive price while lowering risk for the customer.
Proven components from Cisco are integrated to form a software-defined storage (SDS) platform. The approach
eliminates or reduces the number of decisions that need to be made related to planning and configuration, while
allowing customization to meet customer workload needs. The platform and management model adopted is an
extension of the established Cisco UCS data center strategy in which familiar components are managed in a
consistent manner through a policy-based framework with Cisco UCS Manager.

Business Challenge
Use of standard vSwitches in any ESX virtualized infrastructure presents several challenges:

● Inefficiency: Standard switches must be configured one by one, on each individual host of a vSphere
cluster. Any configuration change must be repeated on each host across the cluster to remain consistent.
● Discrepancy: Configurations across multiple standard switches can be set differently, which can lead to
configuration faults and to failure of virtual machines to migrate across hosts.
● Lack of features: Standard vSwitches lack features needed in large-scale and next-generation
environments.

The Solution
Cisco HyperFlex Systems can be converted to use vDS instead of standard vSwitches for networks that are not
part of the Cisco HyperFlex core infrastructure.

Solution Benefits
This solution provides the following benefits to customers:

● Simplicity: vDS is configured once in VMware vCenter. Then all the hosts in the vSphere cluster are
attached to it, and they inherit the configuration.

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 3 of 20
 ● Centralized management: From vCenter, the configuration can be altered, and all hosts will change
simultaneously, eliminating configuration discrepancies.
®
● Features: vDS adds support for several needed features, such as private VLANs (PVLANs), Cisco
NetFlow monitoring, and enhanced single-root I/O virtualization (SR-IOV). vDS also adds support for Link
Layer Discovery Protocol (LLDP), which can be used in conjunction with the Cisco Application Policy
Infrastructure Controller (APIC) to define network endpoints and policies as part of a Cisco Application
™
Centric Infrastructure (Cisco ACI ) environment in the future.

Customers already invested in Cisco products and technologies have the opportunity to mitigate their risk further
by deploying familiar and tested Cisco UCS technology.

Solution Requirements
To configure a Cisco HyperFlex cluster to use vDS requires:

● A functional and healthy running Cisco HyperFlex cluster

● VMware vCenter Server appliance or Microsoft Windows–based VMware vCenter Server
● VMware vSphere Enterprise Plus licensing for the VMware ESXi hosts in the Cisco HyperFlex cluster

Known Constraints
Network components in a Cisco HyperFlex cluster that are related to the Cisco HyperFlex infrastructure cannot be
converted to vDS. The following network port groups, VMkernel ports, standard vSwitches, and uplinks must
remain in the default Cisco HyperFlex configuration and cannot be modified to use distributed switches:

● Standard vSwitch named vswitch-hx-inband-mgmt

● Standard vSwitch named vswitch-hx-storage-data
● Port groups named Storage Controller Management Network and Storage Controller Data Network
● VMkernel ports named Management Network and Storage Hypervisor Data Network
● Network adapter uplinks vmnic0, vmnic1, vmnic2, and vmnic3

The Cisco HyperFlex storage controller virtual machines must not have their guest virtual networking connections
migrated to the distributed switches. Their connections to the Storage Controller Management Network and
Storage Controller Data Network port groups must remain unaltered.

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 4 of 20
 Conversion Procedure
The conversion procedure consists of four basic tasks: creation of the distributed switches, configuration of the
switch settings, migration to the new switches, and testing and cleanup activities.

Create the Distributed Switches

The first step in the conversion process is to create the distributed switches to which the standard vSwitches will be
migrated. In this document, two vDS are created: one to replace the default standard vSwitch named vswitch-hx-
vm-network, which carries guest virtual machine network traffic, and one to replace the default standard vSwitch
named vmotion, which carries vMotion traffic. To create the two vDSs, complete the following steps:

1. Open and log in to the vCenter web client of the vCenter server that is managing the Cisco HyperFlex cluster.
2. Choose vCenter Inventory Lists > Distributed Switches.
3. Click the Create a New Distributed Switch icon (Figure 1).
4. In the wizard that appears, enter the desired name of the new distributed switch, click the data center object
on which the Cisco HyperFlex cluster resides, and then click Next (Figure 2). For example, name the two
vDSs DSwitch-VMNetworks and DSwitch-VMotion.
5. Select the version of the new switch to match the version of VMware ESXi running on the Cisco HyperFlex
hosts. Then click Next.
6. Set the number of uplinks to 2. Network I/O Control can be enabled or disabled as desired. Leave the box
checked to create a default port group and enter the name of the port group. Then click Next (Figure 3). For
example, name the port group DPortGroup-VMNetwork or DPortGroup-VMotion.
7. Click Finish.
8. Repeat steps 3 through 7 to create the second distributed switch.

Figure 1. Add Distributed Switch

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 5 of 20
 Figure 2. Name the Distributed Switch

Figure 3. Distributed Switch Settings

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 6 of 20
 Configure the Virtual Machine Guest Distributed Switch
The basic settings from the distributed switch wizard will often be insufficient to meet all the network needs of the
guest virtual machines. If only a single untagged port group is used for all guest virtual machine traffic, then no
additional configuration is needed, and the steps in this section can be ignored.

However, if guest virtual machine traffic is spread across multiple VLANs, then multiple port groups must be
created, each configured with the VLAN ID used by the port group. Prior to the creation of the additional port
groups, you must perform configuration steps in Cisco UCS Manager to create the necessary VLAN IDs and add
them to the virtual network interface card (vNIC) templates. These steps are described in Appendix A and
Appendix B.

To configure the distributed switch, complete the following steps:

1. In the vCenter web client, choose vCenter Inventory Lists > Distributed Switches.
2. Click the distributed vSwitch created for virtual machine traffic.
3. In the right pane, click the Manage tab, click Settings, and choose the Topology menu item.
4. To create an additional port group, click the Create a New Distributed Port Group icon. If additional port
groups are not needed, continue to step 9 (Figure 4).
5. Enter the name of the port group and click Next. As a best practice, you should include the VLAN ID number
in the name (Figure 5).
6. Change the VLAN type and number as required. Then click Next (Figure 6). For example, for a standard
tagged port group on VLAN 100, change the VLAN type to VLAN and enter 100 as the VLAN ID.
7. Click Finish.
8. Repeat steps 4 through 7 for each additional port group required.
9. To modify the settings of an existing port group, click the name of the port group and then click the Edit
Distributed Port Group Settings icon (Figure 7).
10. Edit the name and VLAN settings as needed. Then click OK (Figure 8). For example, for a standard tagged
port group on VLAN 101, change the VLAN type to VLAN and enter 101 as the VLAN ID. Modify the name of
the port group to include the VLAN ID number.
11. Repeat steps 9 and 10 for each distributed port group that requires modification.

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 7 of 20
 Figure 4. Create Additional Port Groups

Figure 5. Port Group Name

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 8 of 20
 Figure 6. Port Group Settings

Figure 7. Edit Port Group

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 9 of 20
 Figure 8. Port Group VLAN Settings

Configure the VMotion Distributed Switch

The basic settings from the distributed switch wizard will be insufficient to service all the network needs of vMotion.
You will need to make changes to allow jumbo frames to pass on the vMotion network and to modify the uplink
failover order. To configure the distributed switch, complete the following steps:

1. In the vCenter web client, choose vCenter Inventory Lists > Distributed Switches.
2. Click the distributed vSwitch created for vMotion traffic.
3. In the right pane, click the Manage tab, click Settings, choose the Properties menu item, and click the Edit
button.
4. Choose the Advanced menu item, change the maximum transmission unit (MTU) value to 9000, and then
click OK (Figure 9).
5. Choose the Topology menu item, click the name of the vMotion port group, and click the Edit Distributed
Port Group Settings icon.
6. Choose the Teaming and Failover menu item, highlight uplink 2, and click the downward-pointing blue arrow
to move Uplink 2 underneath Standby Uplinks. Then click OK (Figure 10).

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 10 of 20
 Figure 9. Edit vMotion MTU Value

Figure 10. Edit vMotion Failover Order

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 11 of 20
 Migrate to the Virtual Machine Guest Distributed Switch
To migrate networking to the new distributed switch, complete the following steps:

1. In the vCenter web client, choose vCenter Inventory Lists > Distributed Switches.
2. Click the distributed vSwitch created for guest virtual machine network traffic.
3. In the right pane, click the Manage tab, click Settings, and choose the Topology menu item.
4. Click the Add Hosts to This Distributed Switch and Migrate Physical or Virtual Network Adapters icon
(Figure 11).
5. Select Add Hosts and click Next.
6. Click New Hosts, select the hosts in the Cisco HyperFlex cluster that you want to migrate, and click OK
(Figure 12).
7. Select the box for Configure Identical Network Settings on Multiple Hosts and click Next.
8. Select one host in the Cisco HyperFlex cluster to serve as the template for changes made to all the previously
selected hosts. Then click Next.
9. Select the boxes for Manage Physical Adapters and Migrate Virtual Machine Networking. Then click Next
(Figure 13).
10. In the upper pane, scroll down until you can see the uplinks vmnic4 and vmnic5 on the template host. Click
vmnic4 and click the Assign Uplink button.
11. Choose Uplink 1 for vmnic4 and click OK.
12. Repeat steps 10 and 11 but this time choose vmnic5 and Uplink 2.
13. Verify that vmnic4 and vmnic5 are properly reassigned on the template host. Click the Apply to All button and
then click Next twice (Figure 14).
14. Click each virtual machine that will have its network connections migrated and click the Assign Port Group
button. To select multiple virtual machines, you can press Ctrl and simultaneously click the mouse button and
then click the Assign Port Group button.

Note: Do not migrate the networking of any of the Cisco HyperFlex controller virtual machines.

15. Click the name of the distributed port group to be used by virtual machines in the new switch and then click
OK (Figure 15).
16. Repeat steps 14 and 15 until all the virtual machines have been assigned to their new port groups on the
distributed switch. Then click Next (Figure 16).
17. Click Finish.

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 12 of 20
 Figure 11. Add Hosts to vDS

Figure 12. Select Hosts

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 13 of 20
 Figure 13. Migrate Adapters and Virtual Machine Networking

Figure 14. Manage Physical Adapters

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 14 of 20
 Figure 15. Select Virtual Machine Network

Figure 16. Migrate Virtual Machine Networking

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 15 of 20
 Migrate to the VMware vMotion Distributed Switch
To migrate networking to the new distributed switch, complete the following steps:

1. In the vCenter web client, choose vCenter Inventory Lists > Distributed Switches.
2. Click the distributed vSwitch created for vMotion traffic.
3. In the right pane, click the Manage tab, click Settings, and choose the Topology menu item.
4. Click the Add Hosts to This Distributed Switch and Migrate Physical or Virtual Network Adapters icon.
5. Select Add Hosts and click Next.
6. Click New Hosts, select the hosts in the Cisco HyperFlex cluster that you want to migrate, and click OK.
7. Select the box for Configure Identical Network Settings on Multiple Hosts and click Next.
8. Select one host in the Cisco HyperFlex cluster to serve as the template for changes made to all the previously
selected hosts. Then click Next.
9. Select the boxes for Manage Physical Adapters and Manage VMkernel Adapters. Then click Next
(Figure 17).
10. In the upper pane, scroll down until you see the uplinks vmnic6 and vmnic7 on the template host. Click
vmnic6 and click the Assign Uplink button.
11. Choose Uplink 1 for vmnic6 and click OK (Figure 18).
12. Repeat steps 10 and 11, but this time choose vmnic7 and Uplink 2.
13. Verify that vmnic6 and vmnic7 are properly reassigned on the template host. Click the Apply to All button and
then click Next (Figure 19).
14. Click the VMkernel port assigned to vMotion traffic and click the Assign Port Group button.
15. Click the name of the distributed port group to be used by vMotion in the new switch and click OK (Figure 20).
16. Verify that the VMkernel port is properly reassigned on the template host. Then click the Apply to All button.
17. Enter the IP addresses that will be used by the vMotion VMkernel ports for the additional hosts, using a
comma-separated list. Then click OK.
18. Click Next twice. Then click Finish.

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 16 of 20
 Figure 17. Migrate Adapters and VMkernel Adapters

Figure 18. Uplink Settings

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 17 of 20
 Figure 19. Assign Uplinks

Figure 20. Assign Port Group

Test and Clean Up

After the migration, test vMotion migration of virtual machines across the hosts to verify that everything works
properly. Verify network connectivity to the guest virtual machines on the proper network subnets and VLANs.
Typically, continuous pings to the virtual machines will show a loss of a single ping during the migration process.

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 18 of 20
 After the migration is complete, you can delete the original standard vSwitches (named vswitch-hx-vm-network
and vmotion) for the guest virtual machine traffic and vMotion from the Cisco HyperFlex hosts. To delete the
unused standard vSwitches, complete the following steps:

1. In the vCenter web client, select Hosts and Clusters.

2. Click the first Cisco HyperFlex platform ESXi host in the navigation tree.
3. In the right pane, click the Manage tab, click Networking, and choose the Virtual Switches menu item.
4. Click the standard vSwitch named vswitch-hx-vm-network in the list. Then click the Remove Selected
Standard Switch icon.
5. Click Yes to remove the switch.
6. Repeat steps 3 through 5 for the vSwitch named vmotion.
7. Repeat steps 2 through 6 for the remaining hosts in the Cisco HyperFlex cluster.

vCenter will display numerous alarms that were generated during the virtual machine guest switch migration
process. You can safely clear these alarms.

Conclusion
This Cisco solution addresses the needs of many enterprise data centers by allowing the use of VMware
Distributed Virtual Switches in Cisco HyperFlex clusters, staying aligned with established practices. By employing
this solution, you can enable your Cisco HyperFlex system to take advantage of the features offered by vDS and
remain fully supported by the Cisco Technical Assistance Center (TAC).

For More Information

The following documentation, available at Cisco Online Support, provides additional relevant information. If you do
not have access to the documentation, please contact your Cisco representative.

Cisco HyperFlex technical support documentation:

http://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-data-platform-software/tsd-products-
support-series-home.html

Appendix A: Configuring Additional Virtual Machine Guest VLANs

In many cases, the Cisco HyperFlex System requires configuration of multiple VLAN IDs for the guest virtual
machines to use for communication. If additional VLANs are required, you must define the VLANs in Cisco UCS
Manager before you can added them to vNIC templates, and they must be configured as part of the ESXi host
networking port groups. To configure additional VLANs for guest virtual machine traffic, complete the following
steps:

1. In Cisco UCS Manager, click the LAN tab in the navigation pane.
2. Choose LAN > LAN Cloud > VLANs.
3. Right-click VLANs and choose Create VLANs.
4. Enter the name of the VLAN.
5. Select the multicast policy named HyperFlex.
6. Leave the setting for Common/Global.
7. Enter the VLAN ID number.

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 19 of 20
 8. Leave the Sharing Type set to None.
9. Click OK twice.
10. Repeat steps 3 through 9 for each additional VLAN required.

Appendix B: Alternative Configuration for Tagged VLANs

The default Cisco HyperFlex installer defines only a single VLAN for virtual machine traffic in the Cisco UCS
configuration. In many cases, multiple VLANs are used to carry guest virtual machine traffic, and the standards set
at the customer data center require that all virtual machine traffic be tagged with the appropriate IEEE 802.1Q
VLAN ID. To accommodate that requirement, changes must first be made in the Cisco UCS configuration. To
modify the Cisco UCS settings for carrying multiple tagged VLANs, complete the following steps:

1. In Cisco UCS Manager, click the LAN tab in the navigation pane.
2. Choose LAN > Policies > root > Sub-Organizations > hx-cluster > vNIC-Templates.
3. Click the vNIC template named vm-network-a.
4. In the configuration pane, click Modify VLANs.
5. In the Modify VLANs window, verify that the radio buttons in the Native VLAN column are all cleared.
6. In the Modify VLANs window, click the Select box next to each VLAN that will be tagged and carry guest
virtual machine traffic to the vDS.
7. Click OK.
8. Repeat steps 3 through 7 for the vNIC template named vm-network-b, which will also carry tagged guest
virtual machine traffic.

Printed in USA C11-737724-00 08/16

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 20 of 20