Cisco Catalyst 9800-CL Wireless Controller For Cloud Deployment Guide
Cisco Catalyst 9800-CL Wireless Controller For Cloud Deployment Guide
Cisco public
The table below shows an example of the Cisco IOS XE 17.6.x release train.
Table 1. Supported hypervisor versions for the 9800-CL private cloud running 17.6.x.
Microsoft Hyper-V1 Microsoft Windows Server: 2016 or 2019 (Standard, Enterprise, and Datacenter)
Hyper-V Manager: 10.0.14393
1
Supported only on Cisco IOS XE Amsterdam 17.1.1 or later
Table 2. Minimum virtual resource requirements for small, medium, and large configurations
Existing supported templates pre Cisco IOS XE Templates added as part of Cisco IOS XE
Release 17.3 Release 17.3
Minimum number 4 6 10 7 9 13
of vCPUs1
(hyperthreading is
not supported)
Minimum memory 8 16 32 8 16 32
(GB)
Required storage2 16 16 16 16 16 16
(GB)
Hyper-V vNIC NETVSC NETVSC NETVSC Not supported Not supported Not supported
Hyper-V NIC VMBus VMBus VMBus Not supported Not supported Not supported
Virtualization
VMware tools Not supported Not supported Not supported Not supported Not supported Not supported
1
To avoid stability and performance issues, it’s advisable to fully reserve the vCPU resources needed for the 9800-CL and never
oversubscribe them. Hyperthreading is not supported and will need to be disabled on the host machine.
2
Starting from Cisco IOS XE Amsterdam 17.3.1, the required storage has increased from 8 GB to 16 GB. If upgrading to Cisco IOS XE
Amsterdam 17.3.x from a previous release, the existing storage can be kept at 8 GB. For all new installations, it is required to go to
16 GB.
3
The VM operations are supported with some design considerations. Please see the section Design considerations with VMware VM
operations.
4
Support for Layer 2 LAG starts in Cisco IOS XE Bengaluru 17.5.1 and supports deployments with single-root I/O virtualization (SR-IOV).
Table 3. 9800-CL vCPU allocation for small, medium, and large configurations
CPU allocation
OVA template size Total number of vCPUs Control plane Data plane
Small 4 2 2
(Low throughput)
Small 7 2 5
(High throughput)
Medium 6 4 2
(Low throughput)
Medium 9 4 5
(High throughput)
Large 10 8 2
(Low throughput)
Large 13 8 5
(High throughput)
High Availability
High Availability (HA) is supported on the 9800-CL VM hosts using virtual redundant ports, in a stateful
switchover (SSO) configuration as well as in an N+1 configuration.
When the Catalyst 9800-CL is booted for the first time, the router interfaces are mapped to the logical vNIC
interfaces that were added when the VM was created. The figure below shows the relationship between the
vNICs and the Catalyst 9800-CL interfaces.
By default, the 9800-CL comes with three network interfaces. Below is an example of interface mapping:
● GigabitEthernet1 > Device management interface: Map it to the out-of-band management network. This
is the equivalent of the service port on the physical appliance.
● GigabitEthernet2 > Wireless management interface: Map it to the network to reach APs and services.
Usually, this interface is a trunk to carry multiple VLANs.
● GigabitEthernet3 > High Availability interface: Map it to a separate network for peer-to-peer
communication for HA SSO. This is the equivalent of the RP port. This port is not needed if HA SSO is
not going to be configured.
Note: Do not connect two interfaces to a single network, as that may cause network loops. When a trunk
port is used, you must either prune VLANs from vCenter or have the GigabitEthernet interfaces in a
different vSwitch.
The figure below shows an example of mapping the hypervisor physical port (vmnic2, connected to a switch
trunk) to vSwitch0, as intended for the 9800-CL VM management interface, in ESXi. An optional interface
intended for use in the redundant HA configuration (vmnic3) is named RP and mapped to vSwitch3.
Note: When testing two 9800-CL controllers in the same Cisco UCS® server and using RP ports for HA, it
is not necessary to connect the physical RP mapped physical adapters at all. However, if active and
standby 9800-CL controllers are on separate hypervisors, the RP mapped physical ports need to be
connected to the network and must be Layer 2 adjacent and reachable by each other.
Figure 2.
Mapping the hypervisor to the VM management interface in ESXi
By default, a hypervisor vSwitch is configured to reject promiscuous mode. If the 9800-CL is using tagged
traffic (for a management VLAN, AP VLAN, etc.) via the management port, promiscuous mode needs to be set
to accept in order for the vSwitch to carry tagged traffic.
Secure Boot
Starting with Cisco IOS XE 17.6.1, Secure Boot deployments of the 9800-CL VM hosts are supported.
vMotion
When deploying vMotion on the 9800-CL in standalone mode, vMotion will work without caveats.
However, when the 9800-CL is deployed in HA SSO, there are a few considerations to take into account.
● Do not run vMotion on both the active and standby VMs at the same time. In the time it takes for the
active and standby to move to the new hardware resource, the 9800-CL may be seen as going down.
● When using vMotion with 9800-CL in HA SSO, there will be an extended data outage if no packets
originate from WLC. This is due to a limitation in ESXi for Virtual Guest Tagging (VGT mode). As a
workaround, a continuous ping will need to be initiated from the 9800-CL to update the MAC address in
the right port on the physical switch. For more details, see
https://kb.vmware.com/s/article/2113783?lang=en_US.
The following considerations apply when deploying the OVA package to the VM:
● The single OVA package creates a VM with options for three types of virtual wireless controllers, small,
medium, and large. Selecting a profile specifies the required virtual CPU and memory. The hard disk
requirement will be the same—16 GB—for any wireless controller type. We do not recommend changing
the virtual CPU and memory configuration after deployment.
● When deploying using the OVA template, the VM will bootstrap with three interfaces: one is for out-of-
band management, one is for wireless management (usually mapped to a trunk interface on the switch
side), and the third is for HA to connect to the SSO peer.
● The installation process can be monitored using the virtual VGA console or the console on the virtual
serial port. A virtual serial port is optional and can be added after deploying the OVA. At first customer
shipment (FCS) the serial console port is not supported for large-scale deployments. If the serial port is
required, please see Appendix B: Adding a virtual serial port in ESXi.
Step 4. Enter a name for the 9800-CL VM and select the OVA file that will be deployed. Click Next.
Step 6. In the Network mappings section, allocate one port group for each of the required network
interfaces.
Note: Please refer to Appendix A: Creating a port group in ESXi for steps to configure the port groups
for the VM.
Note: SR-IOV interfaces are high-performance interfaces possible in certain Intel® NIC cards. Refer to
Appendix C: Enabling and using the SR-IOV NIC in ESXi to see how to enable and attach them to the
9800-CL.
Step 8. The rest of the settings in the Disk provisioning and Power on automatically sections can be left
at their defaults. Click Next.
Note: If using the virtual serial port, uncheck the box for Power on automatically, and refer to Appendix B:
Adding a virtual serial port in ESXi (optional).
Step 10. Once the VM is finished deploying, select the 9800-CL VM and open the console.
Step 4. Enter a name and select a location for the VM. Click Next.
Step 10. As was mentioned earlier, vCenter deployment provides an option to customize or bootstrap
the Catalyst 9800-CL wireless controller with a hostname, network configuration, and login. Go
through the steps to provide any necessary information using the provided template, and click
Next.
Note: When specifying the device management interface, choose the interface mapped to the out-of-
band management network. It will be configured as a Layer 3 routed interface with the IP address provided
in this step. The network entered in step 2.4 is the remote network from which the 9800-CL will be
managed. This will create a static route to that specified network.
Step 11. Finally, review the configuration data. Click Finish to deploy the 9800-CL in vCenter.
Note: If you are removing the configuration bootstrap, please do not power on the 9800-CL after
deploying and refer to the steps in Appendix E: Resetting the 9800-CL to the factory default.
Step 5. Add the required network adapters and select the port groups needed. Ensure that the network
adapter type is VMXNET 3.
Step 7. Review the configuration and click Finish. Start the VM.
Step 8. After ISO installation, walk through the initial setup wizard and configure SVIs, wireless
interface, trustpoint, etc. (Review the previous sections, as these will be the same.)
Step 9. Go to the Configuring the 9800-CL section.
This guide will not cover every aspect of Linux or KVM components, only the general set of instructions needed
to deploy a virtual wireless controller on KVM.
Table 4. Minimum virtual resource requirements for small, medium, and large configurations
Existing supported templates pre Cisco Templates added as part of Cisco IOS XE
IOS XE Release 17.3 Release 17.3
1
At FCS, the large image will support the full scale with Cisco FlexConnect® local switching and fabric deployment mode. For local and
Cisco Flex® central switching deployment modes, the scale is 3000 APs and 32,000 clients, the same as for the medium configuration.
To see if the processor supports one of these extensions, issue the following command and review the output:
egrep -c '(vmx|svm)' /proc/cpuinfo
If it is 1 or more, it does support hardware virtualization, but virtualization needs to be enabled in the BIOS.
KVM networking
Networking options vary within Linux. Effectively, KVM supports the following:
Linux bridge
OVS switch
auto br0
iface br0 inet static
address 10.104.170.99
netmask 255.255.255.0
network 10.104.170.0
broadcast 10.104.170.255
#gateway 10.104.170.1
#up route add default gw 10.104.170.1
# dns-* options are implemented by the resolvconf package, if installed
bridge_ports eth0
bridge_stp off
bridge_fd 0
bridge_maxwait 0
dns-nameservers 72.163.128.140
auto br1
iface br1 inet static
address 9.11.124.44
network 9.11.124.0
netmask 255.255.255.0
bridge_ports eth1
bridge_stp off
bridge_fd 0
bridge_maxwait 0
For how to configure the SR-IOV interfaces for KVM, see Appendix D: Enabling and using the SR-IOV NIC in
KVM.
C9800> en
C9800#show platform software vnic-if interface-mapping
-------------------------------------------------------------
Interface Name Driver Name Mac Addr
-------------------------------------------------------------
GigabitEthernet2 net_i40e_vf 3cfd.fede.ccbd
GigabitEthernet1 net_i40e_vf 3cfd.fede.ccbc
-------------------------------------------------------------
causes the address domain, bus, slot, and function to be derived from the PCI BDF by the script in step 1.
Step 4. Create a disk of 8 GB (standard for all deployment sizes). Click Forward.
The default is a single interface at the time of VM creation. This can be used as any of the functional virtual
9800-CL interfaces, for example, the wireless management interface. However, if an additional interface
(or serial port) is needed, use the Add New Virtual Hardware tool.
Step 6. Go to Add New Virtual Hardware > Network interface. Map each of the vNICs to the target
bridge interface defined in the Linux network configuration.
Step 8. A virtual serial console also exists for KVM. Simply add the virtual hardware, select Serial, Host
= 127.0.0.1 (local host), and the port number (user-defined), and check Use Telnet.
Click Finish.
Step 9. Next, click Begin Installation. The VM will boot and progress through the installation process.
Table 6. Minimum virtual resource requirements for small, medium, and large configurations on Hyper-V
C9800-CL-universalk9.***.iso
C9800-CL-universalk9.***.bin
Bring up Hyper-V
If Hyper-V is not already installed, follow the steps here to install it: https://docs.microsoft.com/en-
us/windows-server/virtualization/hyper-v/hyper-v-on-windows-server
Hyper-V networking
Hyper-V supports mapping the physical network interfaces to a virtual switch. The virtual switch is used by the
VM network adapter to send traffic to the rest of the network.
The easiest method to create the virtual switch is done using the Hyper-V Manager GUI tool.
Step 1. Open Hyper-V Manager by going to Control Panel > System & Security > Administrative Tools.
Step 3. In the New virtual network switch section, select an External virtual switch. Click Create Virtual
Switch.
Creating the Catalyst 9800-CL VM using the Hyper-V Manager GUI tool
Download the ISO from Cisco for use with the Hyper-V Manager. This GUI tool is the easiest method for
deploying the virtual 9800-CL wireless controller.
Step 1. Open Hyper-V Manager by going to Control Panel > System & Security > Administrative
Tools.
Step 3. Provide a name for the VM and specify a location to store it. Click Next.
Step 4. Specify either generation of the VM. This can be either Generation 1 or Generation 2.
Click Next.
Note: The generation cannot be changed after the VM has been created.
Step 6. Specify the network connection for the included network adapter, or this can be done later.
This network adapter will correspond to GigabitEthernet1 within the 9800-CL VM.
Step 8. To boot from the ISO file for the 9800-CL, select Install an operating system from a bootable
CD/DVD-ROM and enter the location for the required ISO file of the 9800-CL image.
Step 10. Navigate to the settings page for the VM. Right-click the 9800-CL VM and select settings.
Step 12. The default is a single network adapter at the time of VM creation. This can be used as any of
the functional virtual 9800-CL interfaces, such as the wireless management interface.
However, if additional adapters are required, you will need to use either PowerShell or the Hyper-V GUI,
depending on whether the traffic for the network adapters will be tagged or untagged or trunked. These options
are described below.
The network adapter will have traffic that is untagged or tagged with a single VLAN ID
(GigabitEthernet1 and GigabitEthernet3):
For each network adapter, go to the required Network Adapter section. Select the required Virtual switch. If
VLAN tagging is required, check Enable virtual LAN identification and enter the required VLAN ID. Click OK.
The network adapter needs to be configured as a trunk port (normally for GigabitEthernet2):
Open the PowerShell application as administrator.
Add-VMNetworkAdapter -VMName <VM Name> -SwitchName <Virtual Switch Name> Name <Network
Adapter Name>
Set-VMNetworkAdapterVlan -VMName <VM Name> -VMNetworkAdapterName <Network Adapter Name> -
Trunk -AllowedVlanIdList <VLAN Range> -NativeVlanId <VLAN ID>
Example: Creating the network adapter for GigabitEthernet2 on the 9800-CL. It is configured as a trunk port
with a native VLAN ID of 0 that allows traffic with VLAN IDs between 1 and 4000.
To verify the VLAN settings for the VM adapters, type the command:
Example:
Step 13. Start the 9800-CL VM by right clicking the VM and selecting Start.
Creating the Catalyst 9800-CL VM in Hyper-V with the CLI using PowerShell
In addition to the Hyper-V Manager GUI, the 9800-CL can be deployed using PowerShell. This can help in
automating the deployment process of 9800-CL VMs.
Step 1. Open a PowerShell window as Administrator.
Step 2. Create the VM using the following command:
New-VM -Name <VMName> -Path <VMPath> -MemoryStartupBytes <Startup Memory> -NewVHDPath <VHD
Path> -NewVHDSizeBytes <VHD(X) size> -SwitchName <VM Switch Name> -Generation <VM Gen (1 or
2)>
Example:
Example:
Example:
If the interface needs to be tagged with a single VLAN ID, enter the following command to set the
interface as an access port with traffic tagged with the configured VLAN.
Set-VMNetworkAdapterVlan -VMName <VM Name> -Access -VlanId <VLAN ID>
Example:
If the interface needs to be configured as a trunk port, enter the following command:
Set-VMNetworkAdapterVlan -VMName <VM Name> -VMNetworkAdapterName <Network Adapter Name> -
Trunk -AllowedVlanIdList <VLAN Range> -NativeVlanId <VLAN ID>
Example:
Example:
Example:
The Cisco Enterprise NFVIS solution helps convert critical network functions into software, making it possible to
deploy network services in minutes across dispersed locations. It provides a fully integrated platform that can
run on top of a diverse network of both virtual and physical devices.
The Cisco 5400 Enterprise Network Compute System (ENCS) combines routing, switching, storage, processing,
and a host of other computing and networking activities into a compact 1-Rack-Unit (1RU) box. This high-
performance unit achieves this goal by providing the infrastructure to deploy virtualized network functions and
acting as a server that addresses processing, workload, and storage challenges.
The virtual Catalyst 9800-CL Wireless Controller for Cloud can be deployed on a Cisco ENCS NFVIS platform
using an ISO file or tar.gz (download from the Cisco website).
The NFVIS software version should be higher than 3.8. If the NFVIS software version is lower than 3.8, it should
be upgraded to any version higher than 3.8. To upgrade the NFVIS software, refer to the “Upgrading Cisco
NFVIS” section in the following document:
https://www.cisco.com/c/en/us/td/docs/routers/nfvis/config/3-10-1/nfvis-config-guide-3-10-1.html
It needs four CPUs, 8 GB of RAM, 8 GB of storage space, and three vNICs. (The third vNIC is for HA/SSO.)
Installation procedure
Log in to the WebUI of NFVIS with the username (admin) and the password that was set up.
After the image is uploaded, NFVIS creates respective profiles and registers the image. The file can be found
listed in the Images section on the same page.
Creating a network
Follow the procedure below to create a network.
Step 1. Select VM Life Cycle > Networking.
Note: Create separate network interfaces for the wireless management network, service interface, and
HA, and map them to separate bridge interfaces. Each bridge interface maps to a physical interface.
Example:
● A wireless management network named mgmt-intf with the Mode set to Trunk, carrying multiple VLANs,
mapped to a bridge interface named mgmt-br tied to physical interface GE0-0.
● An HA network named ha-intf with the Mode set to access, mapped to a bridge interface named ha-br
tied to physical interface GE0-1.
Step 5. Click the console symbol next to the VM name to open the console to the 9800-CL virtual
controller that is deployed.
nfvis# vmConsole ?
Possible completions:
VM name; "show system deployments" command shows list of VM names.
This opens up the VM CPU Allocation tab, which displays the overall CPU allocations.
Viewing VM statistics
Follow the procedure below to view the VM resource utilization.
Step 1. From NFVIS, select VM Life Cycle > VM Monitoring.
Note: If the DAY 0 WebUI wizard is preferred, please see the Configure the basic 9800-CL settings
section first.
Would you like to enter the initial configuration dialog? [yes/no]: yes
At any point you may enter a question mark ‘?’ for help.
Use ctrl-c to abort configuration dialog at any prompt.
Default settings are in square brackets ‘[]’.
Building configuration...
[OK]
Before 17.4.1, DAY 0 assumes that the box has two separate virtual interfaces (one for device management
and one for wireless management and client traffic) and that the first login happens on the device management
(out-of-band) interface. The wireless management interface is configured via the DAY 0 guided flow. If using a
different setup (for example, if you are using only a single interface), see the previous section or go to
Configuring the 9800-CL via the CLI: Skipping the DAY 0 guided flow section.
Connect to the CLI via the VGA console and follow these steps for the basic configuration:
Step 1. Terminate the configuration wizard (this is the general Cisco IOS CLI wizard, and it’s not specific
for wireless).
Step 4. Add an IP address on the device management interface. The example assumes
GigabitEthernet1 is mapped to the out-of-band/device management network during VM
bootstrap:
Step 5. Add the route to the remote network from which the 9800-CL will be managed.
Note: With an ESXi direct host, no default bootstrap configuration is passed to the instance. If one is
desired, you must enter the following configurations manually (these are automatically configured if using
vCenter):
netconf-yang
ip http server
ip http secure-server
line vty 0 4
transport input telnet ssh
login local
From a computer, verify that the computer can ping the 9800-CL.
To simplify the bootstrap process of the Catalyst 9800-CL wireless controller, the DAY 0 wizard will appear
after a virtual instance is deployed, with network connectivity but without any other wireless configuration.
To log in, use the username and password credentials given during the 9800-CL instance creation described in
the previous sections.
Step 2. Once you are logged in, the 9800-CL presents a simplified configuration flow to set the basic
parameters and have the controller fully operational. On the first page, enter the required
information.
These settings are Deployment Mode, Country, Date, Time, NTP Servers (optional), and AAA Servers (optional).
Note: For Deployment Mode, the available options are Standalone or Active/Standby if configuring HA
SSO.
Note: Only an interface that is different from the one used to access the GUI can be selected. In the
example above, only GigabitEthernet2 or GigabitEthernet3 can be selected, as GigabitEthernet1 is used to
access the GUI.
Step 4. Configure the interface by choosing the VLAN, the IP address, and the default gateway. This
will automatically configure the:
● Interface as a trunk port
● Switch Virtual Interface (SVI) for wireless management
● Default gateway.
Click Next.
Step 5. On the next page, add a WLAN (optional) so that clients can connect. In this example, the PSK
dialog is shown.
A trustpoint is essentially a certificate authority that is trusted implicitly. A trustpoint certificate is a self-signed
certificate, hence the name “trustpoint,” since it does not rely on the trust of anyone else or another party. A
trustpoint is needed for an AP to join the 9800-CL. It can be automatically generated during DAY 0. Otherwise,
if the Generate Certificate is toggled to No, a certificate authority will have to be configured at DAY 1 for APs to
join.
Click Summary to review the configuration, and then click Finish. The configuration and trustpoint will be
pushed to the device and the user will be logged out. The 9800-CL controller will not reboot, but it will take
about 60 seconds to prompt you to log in again. Enter the same credentials.
This time it will skip the DAY 0 page, since the box already has an initial configuration, and redirect to the main
dashboard for the DAY 1 configuration.
Configuring the 9800-CL via the CLI: Skipping the DAY 0 guided flow
If two separate virtual interfaces for device management and wireless management are not needed, create the
DAY 0 configuration via the CLI and then access the GUI for the DAY 1 configuration. However, if separate
interfaces are necessary, please follow the steps in the “Configure the basic 9800-CL settings” section first.
Follow the steps below to configure the 9800-CL with a wireless management interface and skip the DAY 0
flow.
Step 4. Enter the config mode and add login credentials using the following command:
Step 6. Configure the SVI for the wireless management interface; for example:
Step 8. Configure a default route (or a more specific route) to reach the box:
Step 10. A certificate is needed for the AP to join the virtual 9800-CL. This can be created automatically
via the DAY 0 flow or manually using the following commands.
C9800# wireless config vwlc-ssc key-size 2048 signature-algo sha256 password 0 <pwd>
Configuring vWLC-SSC…
Script is completed
Note: Ensure that the gateway for the WMI is reachable from the 9800-CL.
Note: The certificate/trustpoint configuration can be skipped, but APs will not be able to join. This can be
configured via the GUI by importing the desired certificate.
Step 12. Verify that the WMI can be pinged, and enter https://<Management IP>. Use the
credentials entered earlier. Since the box has a country code configured, the GUI will skip the
DAY 0 page and go directly to the main dashboard for the DAY 1 configuration.
◦ The VLAN ID should be the same VLAN ID used for rest of the network. This would be the case for the
management and redundancy interfaces, which would be on their own respective VLANs.
◦ For the data port, the port group should be set to be a trunk port. To do this, set the VLAN ID to 4095.
● Virtual switch
● Security
◦ Both Promiscuous mode and Forged Transmits need to be set to Accept on the port group where the
9800-CL is connected. This is needed for both trunk and nontrunk connections. These security
settings can be restricted to the single port group where the 9800-CL is connected, and as long as
the VLANs are available only on this port group, the settings will not affect other VMs connected to
other port groups. This is recommended, as setting Promiscuous mode to Accept will result in
flooding traffic to all the other VMs on the same VLAN.
vCenter
To create a port group, a virtual switch needs to be created. Follow the steps outlined here to create a virtual
switch: https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.networking.doc/GUID-
DAF824CD-104D-4ED7-8BA3-D769DF688CEB.html
Step 1. Go to the vSphere Web Client.
Step 2. Select the required ESXi host.
Step 5. Select the required vSwitch for the port group. Click Next.
Note: The VLAN ID should be the same VLAN ID used for rest of the network. This would be the case for
the management and redundancy interfaces, which would be on their own respective VLANs. For the data
port, the port group should be set to be a trunk port. To do this, set the VLAN ID to 4095.
Step 8. To change the security settings of the port group, click the … icon for the newly created port
group. Choose Edit Settings.
Note: Both Promiscuous mode and Forged Transmits need to be set to Accept on the port group where
the 9800-CL is connected. This is needed both for trunk and nontrunk connections. These security settings
can be restricted to the single port group where the 9800-CL is connected, and as long as the VLANs are
available only on this port group, the settings will not affect other VMs connected to other port groups. This
is recommended, as setting Promiscuous mode to Accept will result in flooding traffic to all the other VMs
on the same VLAN.
a. Select Use network, as Telnet will be used to connect to the ESXi network address and custom port
assignment.
telnet://<ESXi IP address>:<port>
Note: The ESXi IP address can be omitted (for example, telnet://:<port>). In this case, the IP address
of the ESXi host the device is hosted on will used.
Step 3. To allow for computers to access the serial port of the VM, go to Networking > Firewall rules.
Select VM serial port connected over network.
Step 5. Click the Play icon to power on the VM and bring up the VGA console simultaneously.
Note: If this step is missed, the console output can be changed to the serial port via the 9800-CL CLI.
Please see step 6.
Step 7. If vWLC Virtual (VGA) Console was selected, enter the followings commands in the 9800-CL
CLI.
C9800> enable
C9800# configure terminal
C9800(config)# platform console serial
C9800> enable
C9800# configure terminal
C9800(config)# no platform console serial
C9800(config)# platform console virtual
● Each virtual machine is directly assigned and given access to the physical resources (VFs) by the
hypervisor (VMM).
● VMs load up specific drivers to support SR-IOV.
● The VM boots up and probes its PCIe config space to see what devices it has.
● VMM tells that it has a VF attached and indicates the HW registers for VFs to the NIC driver in the VM.
Figure 3.
Reference: Intel SR-IOV architecture
Note: Ethernet Server Adapter X710 supports up to 32 VFs per port. Creating one VF per port gives the
maximum performance. Each VF would represent a NIC.
Step 3. Reboot the ESXi host and the SR-IOV configurations will take effect.
Step 4. Create a new virtual switch without any physical NICs attached to it.
Step 6. Edit the VM and reserve all the guest memory for the VM, which is necessary for SR-IOV.
For the 9800-CL, this will be 8192 MB.
Step 7. Edit the VM, remove the network port already attached, and add a new network adapter with
the following settings:
● Port group: Set to the newly created port group
● Adapter Type: SR-IOV passthrough
● Physical function: Required Physical NIC with SR-IOV enabled
● Guest OS MTU Change: Allow
https://kb.vmware.com/s/varticle/74909
vSwitch security policy is not persistent when SR-IOV is enabled. To work around this, SSH to ESXi.
Step 10. Use the following command to verify that the NIC is trusted and spoof check is disabled.
Example:
Step 2. Use the CLI to set the trust parameters using the following command:
For firmware and driver versions prior to and including firmware version 7.0 and driver version 1.8.6, the above
two steps are always needed on VM reboot.
VMware Version 6.5 Ciscoized x710 I40en 1.8.6 7.0 7.0 firmware and 1.8.6 driver have a trust
mode persistence issue across VM
Plugin version 1.4.1 reload. Issue will be fixed in subsequent
firmware and driver versions.
The Ethernet and driver versions can be verified using the following command:
Example output:
The script below can print all the Ethernet information followed by the driver versions and the SR-IOV VF
names.
#!/bin/bash
echo --------------------------------------------------------
NIC_DIR="/sys/class/net"
for i in $( ls $NIC_DIR) ;
do
declare -a VF_PCI_BDF
declare -a VF_INTERFACE
k=0
for j in $( ls "${NIC_DIR}/$i/device" ) ;
do
VF_PCI_BDF[$k]=$VF_PCI
do
VF_INTERFACE[$k]=$iface
fi
done
((k++))
fi
done
NUM_VFs=${#VF_PCI_BDF[@]}
echo +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
echo "Driver Versions"
ethtool -i $i
echo --------------------------------------------------------------------
echo -e "=======\t\t========="
do
echo -e "${VF_PCI_BDF[$l]}\t${VF_INTERFACE[$l]}"
done
unset VF_PCI_BDF
unset VF_INTERFACE
fi
done
References for the firmware downloads can be found in the links below.
https://downloadcenter.intel.com/product/82947/Intel-Ethernet-Controller-X710-Series
https://downloadcenter.intel.com/download/24411/Intel-Network-Adapter-Driver-for-PCIe-40-Gigabit-
Ethernet-Network-Connections-Under-Linux-?product=82947
https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/c/sw/fwp/user/guide/Firmware_Upgrade_Utili
ty/Using.html
Step 2. Verify that the Intel VT-D support is enabled on the Linux Kernel.
The output should show that the IOMMU is enabled, as shown in the example below
grub2-mkconfig -o /etc/grub2.cfg
3. Reboot the system to enable the changes. The system is now capable of PCI device
assignment.
Step 3. Configure SR-IOV VFs on the NIC.
In step 1, if the VF is not seen in the output of the script, enable it using the following commands:
Configure spoofcheck, trust mode, and MAC using the commands below:
Example:
SR-IOV configurations configured in the above way are not persistent across the reboots. To fix this, the above
configuration can be run as a service that is auto enabled on host reboots.
1. Create a bash script with the commands to be persisted, for example, in /usr/bin/sriov-
config. Write the script in /usr/bin/sriov-config :
#!/bin/sh
echo 1 > /sys/class/net/enp129s0f0/device/sriov_numvfs
ip link set dev enp129s0f0 vf 0 trust on
ip link set enp129s0f0 vf 0 spoofchk off
ip link set enp129s0f0 vf 0 mac 3c:fd:fe:de:cc:bc
4. Create the system service: Define a new systemd service to be executed at the end of boot.
This service executes the bash script which has the required SR-IOV commands, as shown
in step 1.
Note: "ExecStart=/usr/bin/sriov-config" given below executes the script.
To create the system service, make a new file "sriov.service" in /usr/lib/systemd/system with the
following content:
[Unit]
Description=SR-IOV configuration
After=rc-local.service
Before=getty.target
[Service]
Type=oneshot
ExecStart=/usr/bin/sriov-config
[Install]
WantedBy=multi-user.target
This command will start the service immediately and also ensures that the service runs every time the host
reboots.
https://www.intel.com/content/www/us/en/embedded/products/networking/xl710-sr-iov-config-guide-gbe-
linux-brief.html
Step 6. Attach the SR-IOV to the C9800-CL:
Add the PCI VF devices using the host device command. Using the script from step 1, note the PCI BDF number
and use that to attach the devices.
C9800-CL# wr erase
Erasing the nvram filesystem will remove all configuration files! Continue? [confirm]
[OK]
Erase of nvram: complete
Step 2. Reload the box and enter no at the Save configuration prompt.
C9800-CL# reload
System configuration has been modified. Save? [yes/no]: no
Reload command is being issued on Active unit, this will reload the whole stack
Proceed with reload? [confirm]
Step 3. Under CD/DVD drive 2, click the X icon to delete. This stores the bootstrap configuration
configured during deployment. Deleting it enables the 9800-CL to reset to the factory defaults.