Cisco ACI

Virtual Machine Management

www.lumoscloud.com
[email protected]
Agenda
 ACI - Hypervisor Integration Overview
 ACI - Hypervisor Integration with VMware – Traditional Method
 Integration with VMware VDS
 ACI & VMware Configuration
 ACI Hypervisor Integration with VMware – ACI Plugin Method
 Application Virtual Edge
 OpFlex
 Design Considerations
ACI Hypervisor Integration
Overview
 Supported ACI Integration
Hypervisor Platform Integration Container Platform Integration
• VMware vSphere • Kubernetes ACI 3.0
• Hyper-V • OpenShift ACI 3.1
– SCVMM
– Azure Pack
• OpenStack
• Red Hat Virtualization ACI 3.1
 Policy Coordination with VM Managers Hypervisor
Management
Virtual
Integration
 Network policy coordination
with virtualization managers
 Automatic virtual end point
NetworkPolicy
detection and policy placement Coordination
 Policies consistently Web App DB PortGroups VM networks

implemented in virtual and Application Profile Web App DB

physical
 Network policy stays sticky with
VM VM mobility
notification
VM
Attach/ Detachnotification
PortGroup
Hypervisor Integration with ACI – VMM Domains

 Multiple Virtual Machine

Managers (VMMs) likely on a
single Fabric
 Each VMM and associated Virtual
hosts are grouped within APIC
vCenter SCVMM KVM
 Called VMM Domain

VMM Domain 1 VMM Domain 2 VMM Domain 3

Hypervisor Integration with ACI – VMM Domains & VLANs
16M Virtual Networks
 VLAN ID only gives 4K EPGs
(12 bits)
 Scale by creating “pockets”
of 4K EPGs
 Map to scope of live
migration
EP
 Place VM anywhere
EP EP EP
EP
EP EP
EP EP EP EP
EP EP
EP
 Live migrate within VMM
VMM Domain 1 VMM Domain 2 domain
4K EPGs 4K EPGs
EPG Spanning across VMM Domains

VMM Domain 1 VMM Domain 2

EPGs can take different vCenter vCenter
network identities across vShield vShield

VMM Domain Hosts Hosts

 Applications can be deployed

across VMM Domains
 VM Mobility is not allowed
between VMM Domain due to VMM Domain 1 VMM Domain 2
vCenter/SCVMM limitation 4k EPGs
Web App DB App
EPG EPG EPG EPG
VM VM VM VM VM VM VM VM VM
Recommended Practice for VLAN Networks

 VLAN name space can now be overlapping

 Use Per Port VLAN (PPV) name space when
VMM domains share TOR

Best Practice for VMM

Domain definition

VMM Definition to
avoid
vCenter 1 vCenter 1 vCenter 1 vCenter 2
VLAN range VLAN range
2-4000 2-4000
Overlapping VLAN name space
on the same ToR – use PPV
Per Port VLAN (PPV) Configuration

An interface on a switch
can be set for port local,
meaning VLAN 10 on e1/1
could be different than
VLAN 10 on e1/2
Integration with VMware VDS
ACI Fabric and VMWare VDS Integration
 How does ACI Fabric implement
APIC
policy?
- Assigning EPs to EPGs
 What are EPs in a virtual environment?
- VM vNICs
 How does VMware apply network
Application Network Profile
EPG EPG EPG configuration?
F/W WEB L/B APP DB
- Port Groups
WEB PORT GROUP APP PORT GROUP DB PORT GROUP
 How are EPGs exposed to VMware?
- Map EPGs to Port Groups
VM VM VM
 End-Point identification ACI Leaf

 Physical end-point
Baremetal server Defined in APIC
 Switch interface
 VLAN
 Virtual end-point
APIC Admin VMware Admin

VMware vShield for VXLAN

vCenter/vShield only

Port Group
End Point Group VLAN and VXLAN
Cisco ACI Hypervisor Integration – VMWare VDS
Application Network Profile
5 EPG EPGAP
EPG DB
WEB P
APIC Create Application
F/W L/B

Policy

APIC Admin

9 ACI
Fabric
Push Policy
1 (On Demand)
6
Cisco APIC and VMware 4 Learn location of ESX
vCenter Initial Handshake Automatically Map Host through CDP/LLDP
EPG To Port Groups or Pre-Provision

APIC Creates VIRTUAL DISTRIBUTED SWITCH

2
VDS
WEB PORT GROUP APP PORT GROUP DB PORT GROUP

vCenter 7 APIC Creates

Server Port Groups
Instantiate VMs, Web App DB Web Web DB
8 Attach
VI/Server Admin Assign to Port Hypervisor to
3 VDS
Groups
HYPERVISOR HYPERVISOR
 Staging VMware Integration
VMware
vCenter
Create ACI Virtual Distributed
VM Management Switch Created by APIC
Domain
APIC
Admin

Add Hosts to
ACI VDS
VI/Server
Admin
 VMware Integration – App Instantiation

vCenter

Create Creation of
Application PortGroups
Profile
Web Web
APIC Admin
App App Map to
DB PortGroups
DB

Policy
Download
Instantiate VMs
Fabric
VI/Server
Admin

2013
 Endpoint Discovery
Virtual Endpoints are discovered for
reachability & policy purposes via 2 APIC
methods:
 Control Plane Learning:
- Out-of-Band Handshake: vCenter APIs
- Inband Handshake: OpFlex-enabled Host
Control
(N1KV, Windows Server vNext, etc..) or via (vCenter API)
LLDP/CDP, or now via Pre-Provision
 Data Path Learning: Distributed switch
VMM
learning Control Data Path Data Path
(OpFlex)
 LLDP used to resolve Virtual host ID to
attached port on leaf node (non-OpFlex DVS Host

Hosts)
OpFlex Host
Endpoint Status
 Deployment/Discovery with Hosts Directly Attached
Policy
Attachable Entity Profile must be APIC
resolution

mapped to the ports

AEP must be mapped to the ports
 AEP points to the VMM domain
 The ESX host discovers the neighbors Leaf 101
10 10
Leaf 102

via LLDP LLDP LLDP

 Communicates this information to

vCenter which in turn provides it to
APIC “I am connected to leaf
101 port 10 & leaf
102 port 10”
ESX Discovery with UCS B-Series
• ACI uses LLDP for host discovery
• UCS-B
 LLDP supported northbound from FI
 LLDP not supported between FI and ESXi
host – only CDP CDP
• No path for LLDP between leaf and UCS
ESXi host
Solution: IOM IOM
CDP
 Leaf policy to enable LLDP and CDP
VIC/vNICs
 Configure CDP from UCS ESXi host
Configuration with UCS
Fabric Interconnect
• Configure UCSM to send CDP from the vNICs
• Configure the Policy-group on ACI to Disable
LLDP and enable CDP
• Override the LACP policy within the AEP
vSwitch policy
• APIC configures the vDS with CDP to detect
the UCS Fabric Interconnect
Endpoint Discovery with UCS Fabric Interconnect
As a result:
• Leaf detects the Fabric Interconnect (CDP/)
• ESX hosts detects the Fabric Interconnect (CDP)
• APIC correlates the two information to understand that
which Fabric Interconnect port are providing
connectivity to the ESX host. Also, Pre-provision
specifies that a policy is downloaded to a leaf switch
even before a hypervisor is attached to the VDS, thereby CDP
pre-provisioning the configuration on the switch.

IOM IOM
CDP

VIC/vNICs

vCenter
Endpoint Discovery with UCS Fabric Interconnect
As a result:
• Leaf detects the Fabric Interconnect (CDP/)
• ESX hosts detects the Fabric Interconnect (CDP)
• APIC correlates the two information to understand that
which Fabric Interconnect port are providing
connectivity to the ESX host. Also, Pre-provision
specifies that a policy is downloaded to a leaf switch
even before a hypervisor is attached to the VDS, thereby CDP
pre-provisioning the configuration on the switch.

IOM IOM
CDP

VIC/vNICs

vCenter
ACI & VMware Configuration
ACI VMware Integration – Create VMM Domain
Create vCenter Domain
ACI VMware Integration – Create VMM Domain

Name of VMM Domain

Type of vSwitch (DVS, AVS or AVE)
Associated Attachable Entity Profile (AEP)

VLAN Pool

Security Domain(s)

vCenter Administrator Credentials

30
ACI VMWare Integration – Create VMM Domain
Create vCenter Credentials
Required vCenter Credentials
 Use vCenter administrator
credentials
 Create a vCenter custom role
with the following Privileges:
 Alarms
 Datacenter
 Distributed switch
 dvPort Group
 Folder
 Host
 Network
ACI VMWare Integration – Create VMM Domain
Create vCenter Controller Association
ACI VMWare Integration – Create VMM Domain
Display VMM Domain Association
ACI VMWare Integration – Create VMM Domain
Display new VDS on vCenter
ACI VMWare Integration – Associate EPG to VMM
Add VMM Domain Association
ACI VMWare Integration – Associate EPG to VMM
Add VMM Domain Association
ACI Hypervisor Integration
with VMware using the
ACI Plugin Method
VMware Web Client – ACI Plugin
Access ACI Plugin: https://x.x.x.x/vcplugin

39
VMware Web Client – ACI Plugin

40
Connect vCenter to ACI Fabric

41
Provide IP and Credentials

42
Create a Tenant

43
Name Tenant and Provide Subnet(s)

44
Tenant Created in APIC

45
ANP, BD, PN, Security - All Automatically Created

46
Create a VMware vDS Port Group

47
Name the Port Group and Assign to Tenant/ANP

48
Assign VM to New vDS Port Group

49
Assign VM to New vDS Port Group

50
APIC Reflects all Changes and Shows Endpoint VM

51
View ACI Plugin Nav Tree with Tenant Health

53
View Port Groups, Endpoints and Learned Info

54
View and Edit VMs Directly From vDS / ACI Plugin

55
Application Virtual Edge (AVE)
Cisco ACI Virtual Edge
Decoupled From Hypervisor Kernel API Dependencies
Cisco ACI Virtual Edge
Cisco AVE Architecture

59
Cisco AVE Architecture
Cisco AVE – Deployment Steps
Cisco AVE – Recommendations
 Cisco
AVE AVE – Software
Software from Cisco.com
vSphere Web Client: Content Libraries
• Upload AVE OVA to vSphere
 Create
Create or Modify
or modify existing
existing VLAN pool withinVLAN
the APICPool forforAVE
controller AVE usage.

VLAN Encap: On the wire

VXLAN: Internal
 Create
Create Newdomain
new VMM VMM Domain
within for AVE
the APIC controller for AVE usage.
Switching enforcement
preference: This determines
whether switching can be
done within the virtual switch
(Local Switching) or whether
all switched traffic must go
through the fabric (No Local
Switching). The preference can
be:

No Local Switching

Local Switching
Add
Add Hosthosts
selected Physical NICs
physical NICs toDVSACI
to ACI DVS
InstallAVE
Install AVE on select
on select host,
host, once onceyou
installed provisioned
will see statusyou
of
will status
online of online
and a new andprovisioned.
VM has been a newly provisioned VM
Verify
VerifyAVE
AVEisisinstalled
installedononhypervisor
hypervisorwith
withinside/outside
inside/outsideinterfaces
AVE–
AVEAssociate EPG toEPG
– Associate VMMto VMM
Add VMM Domain Association
AVE–
AVE Associate VM toVM
– Associate Portto
Group
Port Group
Validate
Validate Hosts – VLAN/VXLAN
Hosts Association
– VLAN/VXLAN Association