BRKACI-3101

ACI Under the Hood

How Your Configuration is Deployed

Joseph Ristaino, Technical Leader – DCBU ACI Escalation

Carlo Schmidt, Technical Leader – ACI Solution Support
 Cisco Webex Teams

Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session

How
1 Find this session in the Cisco Events Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space

cs.co/ciscolivebot#BRKACI-3101

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda
• Introduction
• Building the Overlay
• Access Policies
• Configuration Deployment and Validation
• Loop Prevention
• Traversing the Overlay
• Learning, Forwarding, and Policy
Enforcement
• Shared Services and Route Leaking
• L3outs and Routing Protocols

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
 Acronyms/Definitions
Acronyms Definitions Acronyms Definitions
ACI Application Centric Infrastructure LPM Longest Prefix Match
ACL Access Control List MDT Multicast Distribution Tree

APIC/IFC Application Policy Infrastructure Controller/ MST Multiple Spanning Tree

Insieme Fabric Controller
BD Bridge Domain pcTag Policy Control Tag
COOP Council of Oracle Protocol PL Physical Local
ECMP Equal Cost Multipath SVI Switch Virtual Interface

EP Endpoint TC Topology Change

EPG Endpoint Group VL Virtual Local
FTEP/VTEP Fabric/Virtual or VXLAN Tunnel Endpoint VNID Virtual Network Identifier
GIPo Outer Group IP Address VXLAN/iVXLAN Virtual Extensible LAN / Insieme VXLAN
ISIS Intermediate System to Intermediate XR VXLAN Remote
System

 Reference
Slide
BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Introduction
 Introduction
What are our basic network requirements?
1) Provide paths for endpoints to communicate at 4) Communication to external L2 networks (DCI)
Layer2(MAC) and Layer3(IP)
5) Communication to external L3 networks (WAN)
2) Provide separation of endpoint into Layer2
forwarding domains (vlan or BD)
3) Routing between IP/IPv6 subnets and allow
separation of these into multiple VRFs

VLAN L2 L3
EP1 EP2 EP3 VLAN EP4
1 2 External External
EP3
VRF-1

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Introduction
What are our basic network requirements?
6) Allow security policies in order to limit communication to between endpoints to allowed protocols

ip access-list web-in
VLAN 1 VRF1 VLAN 2 permit tcp Subnet1 Subnet2 eq 80
Subnet1 Subnet2 ip access-list web-out
permit tcp Subnet2 eq 80 Subnet1

EP1
80 EP3

ip access-group web1 in

ip access-group web2 out

EP2
22 EP4

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
What physical topology is required?
Physical topology must support our endpoint communication (layer-2 /
layer-3), and the location of endpoints within the physical network will affect
the supporting design/configuration.

L2 L3
EP1 VLAN EP2 EP3 VLAN EP4
1 2 External External

VRF-1

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Traditional Topology – Routing at Core/Spine
STP results in unused links / limits scale / slower convergence

Layer2 – STP forwarding

Layer2 – STP blocked
Layer3 – ECMP

L2 L3
EP1 VLAN EP2 EP3 VLAN
1 2 External External

VRF-1

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
Traditional Topology – Routing at Access
Restricts L2 endpoint locations / requires separate links for L2 / segmented
STP
Layer2 – STP forwarding
Layer2 – STP blocked
Layer3 – ECMP

L2 L3
EP1 VLAN EP2 EP3 VLAN
1 2 External External

VRF-1

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
ACI Infrastructure
ISIS is run on links between spines / leaves

Physical links

ISIS / MDT

L2 L3
EP1 EP2 EP3
External External

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
ACI Infrastructure
APICs communicate to fabric over infra vlan

Physical links

ISIS / MDT

L2 L3
EP1 EP2 EP3 APIC
External External

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
ACI Infrastructure Physical links
Leaves/spines advertise TEP via ISIS
ISIS / MDT

T Tunnel Endpoint (TEP)

L2 v4 v6 L2 v4 v6
L2 v4 v6 Anycast Spine Proxy TEPs

T T T T T

T
L2 L3
EP1 EP2 EP3 APIC
External External

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
ACI Infrastructure Physical links
Leaves advertise learned EP to spines via COOP
ISIS / MDT
COOP Oracles T Tunnel Endpoint (TEP)
L2 v4 v6 L2 v4 v6
L2 v4 v6 Anycast Spine Proxy TEPs
10.1.1.57

COOP Citizens
T T T T T
10.1.1.57

L2 L3
EP1 EP2 EP3 APIC
External External
10.1.1.57

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
ACI Infrastructure Physical links
BL advertises external routes to fabric through MP-BGP
ISIS / MDT
MP-BGP RRs T Tunnel Endpoint (TEP)
L2 v4 v6 L2 v4 v6
0.0.0.0/0
0.0.0.0/0
0.0.0.0/0 L2 v4 v6 Anycast Spine Proxy TEPs

RR-Clients
T T T T T
0.0.0.0/0
0.0.0.0/0

L2 L3
EP1 EP2 EP3 APIC
External External

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
ACI Infrastructure
APIC provisions BD/VRF VXLAN overlays based on EPG attachments

EPG1 EPG1 EPG2 EPG-L2Ext l3extSubnet L2Out

EPG2
EPG1 104/1/8
103/1/3
101/1/5
l3extInstP
102/1/1105/1/10
L2 L3
EP1 EP2 EP3 APIC
BD-1 BD-2 External External
VRF-1 VRF-1 VRF-1

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
 VXLAN
VXLAN differentiates tunneled traffic based on VNID field.

OUTER INNER
MAC Header IPv4 Header UDP Header VXLAN Header MAC Header IPv4 Header UDP Header PAYLOAD FCS
802.1Q

Flags
Reserved
I
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Virtual Network Identifier (VNID) Reserved

32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
 iVXLAN
In addition to differentiating traffic based on VNID, iVXLAN allows the source EPG of traffic to be identified
by the Source Group (PCTAG) bits and to determine if policy was applied by source (SP) / destination (DP).
Endpoint Learning can be enabled/disabled via the Don’t Learn (DL) Bit.
Exception (E) bit ensures packet cannot be sent back into the fabric for certain flows. Blocks Loops.
Example is Proxy Flow. Packet was proxied and should not be re-directed anywhere else.

OUTER INNER
MAC Header IPv4 Header UDP Header VXLAN Header
iVXLAN Header MAC Header IPv4 Header UDP Header PAYLOAD FCS
802.1Q

Flags D S D
E Reserved Source Group
I L P P
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Virtual Network Identifier (VNID) Reserved

32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
ACI Infrastructure
Policy is implemented through contracts / filters specifying allowed traffic

EPGs have a consumer / provider

relationship to a contract.
cons prov
EPG1 EPG2

HTTP (80)

EPG1 EPG2 EPG-L2Ext l3extSubnet

L2 L3
EP1 EP2 EP3 APIC
BD-1 BD-2 External External

VRF-1

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Access Policies
 Access Policies
What is the goal? What are we trying to accomplish?
1) Provide consistent configurations across the whole 3) Define what policies are allowed to be deployed on
fabric. leafs/ports
2) A simplified and well organized configuration, where 4) Restrict Resource deployment in a multi-tenant
policy is defined once and re-used. environment.

Bare Metal

Hypervisors

Pool 1 Pool 2
L2 L3
External External

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Access Policies
Access policies refer to the configuration that is applied for physical and virtual
(hypervisors/VMs) devices attached to the fabric.

Broken into a few major areas:

Global Policy
Switch Policy Interface Policy • Pools
• Policies • Policies • Domains
• Policy • Policy • Attachable Access Entity
Groups Groups Profiles
• Profiles • Profiles

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
 Access Policies SWITCH POLICY

Policies define protocol / feature configurations

Policy Groups select which policies should be applied

Profiles associate policy groups to switches or interfaces,

through the use of selectors

Switch Policy Types: Interface Policy Types:

VPC Domain Link-level Storm Control
Spanning-tree (MST) CDP Data plane policing
BFD LLDP MCP
Fibre-channel SAN/Node Port-channel / LAG L2 (Vlan local / global) INTERFACE POLICY
Port-channel member Firewall
Spanning-tree

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
vPC Protection Group Policy

vPC Domain 1
vPC Domain 1 vPC Domain 2
Classical vPC Domain configuration ACI vPC Domain configuration
Required configuration of domain, peer-link, and Specify the Domain ID and the two Leaf switch IDs
peer-keepalive link on both devices in domain that form the domain pair
vpc domain 1
peer-keepalive destination 172.168.1.2 / VPC Protection Group
source 172.168.1.1 vrf vpc-keepalive
peer-gateway
ip arp synchronize
Name: vPC-Domain100
ID: 100
interface port-channel 20 Switch1: 101
vpc peer-link
Switch2: 102

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
Interface Policies
Used to define a particular policy for a given interface level function. The intention of
Interface Policies is that they are defined once and re-used among interfaces that need
like policies.

Examples:
VPC Domain 1
• LLDP On/Off
• CDP On/Off
• Port-Channel
• LACP
• Mode On
EP1 EP2 EP3
• Storm Control
• MACsec

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Interface Policy Groups
Used to specify which interface policies to be applied to a particular interface type.
It also associates an AEP (which defines which domains are allowed on the interface).

Types:
VPC Domain 1
Access port (EP1)
Access Bundle Groups
• Virtual Port-channel (EP2)
• Port-channel (EP3)

EP1 EP2 EP3

Note: Separate policy groups should be created for each port-channel (standard or VPC) that you
need to configure. All interfaces on leaf that are associated with a particular access bundle group
reside in same channel.

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
 Global Policy
Pools (Vlan / VXLAN) Pool1 Pool2
A resource pool of encapsulations that can
be allocated within the fabric.

Domains (Physical / VMM / External Bridged / External

Routed)
DomPhy1 DomL2Ext1
Administrative domain which selects a vlan/vxlan pool for
allocation of encaps within the domain

Attachable Access Entity Profiles (AEP)

AEP
Selects one or more domains and is referenced/applied by TenantA
interface policy groups.

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Access Policy Example
General Configuration (reused for many interfaces): AEP Pool1
1) Configure a physical domain and vlan pool CiscoLive
DomPhy1
2) Create an AEP and associate physical domain
3) Create switch/interfaces profiles for leaf (LEAF101) Switch Profile
• very easy to apply configurations if you create a
switch/interface profile for each leaf and one for each LEAF101
VPC domain pair
Leaf_101
4) Configure Interface policies (LACP / LLDP)
Interface Profile
LACP Active LEAF101
Policies

LLDP Rx / Tx enabled

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Access Policy Example
Interface specific (each time you add a new interface): AEP Pool1
1) Create policy group for device (VPC / PC / Access) CiscoLive
DomPhy1
2) Within the policy group, select the desired policies / AEP
3) Associate interfaces to policy group via desired leaf Switch Profile
profile
• use specific leaf profile if access or PC LEAF101
• use VPC leaf profile if policy group is VPC Leaf_101

Interface Profile
LACP Active PC_Server_1 LEAF101
Policies Policy Groups blk_1/1-2

LLDP Rx / Tx enabled Access_Servers blk_1/47-48

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
Configuration Deployment
and Validation
VRF/BD/EPG Logical Configuration
VRF-CiscoLive
BD-WOS BD-Breakouts
WISP TSC Breakouts
EP1 EP2 EP3 EP1 EP2

Classical configuration steps ACI Logical configuration

• Create VRF • Create Tenant
• Create Vlans • Create VRF
• Create Vlan interfaces • Create BDs
• Associate to VRF • Associate to VRF
• Assign Subnets / configure gateway • Define a Subnet (optional)
redundancy • Create App Profile
• Assign encapsulation to interfaces • Create EPGs
• Associate to Domain
• Define a Subnet (optional)
BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
 Each node must be individually configured with the
Classical VRF/BD config VRF, associated vlans/BDs, and an SVI with unique IP.
For gateway redundancy, HSRP must also be
configured.
VRF-CiscoLive vrf context CiscoLive
vlan 100
BD-WOS BD-Breakouts name WOS
vlan 200
WISP TSC Breakouts name Breakouts
feature interface-vlan
IM1 IM2 IM3 EX1 EX2 feature hsrp
interface Vlan100
vrf member CiscoLive
ip address 10.10.0.2/24
ip address 10.20.0.2/24 secondary
vlan-100 vlan-100 vlan-100 vlan-200 vlan-200 hsrp 100
ip 10.10.0.1
interface Vlan200
1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 vrf member CiscoLive
ip address 10.30.0.2/24
hsrp 200
ip 10.30.0.1
5 6 6 5 7 5 6 2
interface Ethernet1/1
switchport trunk vlan allowed 100
vlan-100 interface Port-channel1
vPC Domain vlan-100 vlan-200 switchport access vlan 200

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
55
 ACI Logical Configuration

Tenant: CiscoLive • Create Tenant

• Create VRF
Networking App Profile: Operations • Create BDs
VRF: CiscoLive EPG: WISP • Associate to VRF
Subnet: • Define a Subnet (optional)
10.20.0.1/24 • Create an App Profile
BD: WOS • Create EPGs
EPG: TSC • Associate to Domain
Subnet: 10.10.0.1/24
• Define a Subnet (optional)
EPG: Breakouts
BD: Breakouts Subnet:
10.30.0.1/24 What have we accomplished?
Specified the logical configuration that should
be deployed on each leaf where EPG is
deployed. We also restricted which interfaces
Domain: DomPhy1 can deploy the EPG through Domain
associations.

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
 ACI Logical Configuration Deployment
NGINX Receives REST API
Call and Parses Request
NGINX (Web Server)
• Create Tenant (fvTenant)
APIC APIC APIC • Create VRF (fvCtx)
• Create BDs (fvBD)
PolicyDistributor Validates the • Associate to VRF
Configuration is Deployable PolicyDistributor (Validation) • Define a Subnet (optional)
PolicyManager Writes the PolicyManager (DataReplication) • Create an App Profile
Config to DB and Distributes • Create EPGs (fvAEPg)
Data to other Cluster Members • Associate to Domain
• Define a Subnet (optional)

NOTE: No Policy is
Pushed to Switches
Yet…

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
 Overlay Fabric Allocations
VRF-VNID – allocated per VRF
Tenant: CiscoLive • (unique within fabric)

Networking App Profile: Operations BD-VNID – allocated per BD

VRF: CiscoLive EPG: WISP • (unique within fabric)
Subnet:
10.20.0.1/24 EPG-VNID – allocated from vlan pool
BD: WOS (domain specific) and is unique within
EPG: TSC fabric
Subnet: 10.10.0.1/24
• Used for STP BPDU flooding
EPG: Breakouts and flood in encap for
BD: Breakouts Subnet: unknown unicast traffic
10.30.0.1/24
PCTAG – allocated per EPG
• FABRIC-global if shared
service provider
Domain: DomPhy1
• VRF-local otherwise

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
 EPG Deployment to Leaf
EPG are deployed through:
VRF-CiscoLive • Static binding to port/PC/VPC
BD-WOS BD-Breakouts • Static binding to node
• Static binding to AEP
WISP TSC Breakouts • VM attachment
EP1 EP2 EP3 EP1 EP2
To successfully deploy an EPG
configuration on a leaf:
1. AEP of target interface must allow
same domain as assigned to EPG
2. encapsulation/vlan must be
vPC Domain allowed in the target domain
1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4

AEP
VPC1 102/1/2 103/1/1 PC1 104/1/3 Pool1
Statics
DomPhy1
EP1 EP2 EP3 EP1 EP2
vlan 100-200
BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
ACI EPG Configuration Deployment

NGINX (Web Server)

EPG are deployed through:
• Static binding to port/PC/VPC
APIC APIC APIC • Static binding to node
• Static binding to AEP
PolicyDistributor (Validation) • VM attachment

PolicyManager (Deployment)
PolicyManager sends policy to appropriate
nodes where EPG was deployed.

PolicyElem translates the Logical Config to

PolicyElement (Deployment) Concrete Config independent of Hardware
Platform. Also Validates Config against
Hardware Dependencies.
NXOS picks up the config and
NXOS (SW and HW Programming) programs the SW/HW.

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
 ACI EPG Configuration Deployment
• Why is this Useful?

FAULT! NGINX (Web Server)

APIC APIC APIC Logical Configuration Errors

are Detected by APIC.
PolicyDistributor (Validation) Faults are raised.

FAULT! PolicyManager (DataReplication)

PolicyElement (Deployment) Platform Configuration Errors

are Detected by Switch.
Faults are raised.

NXOS (SW and HW Programming)

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
 EPG Static Path Deployment Leaf101 BD-WOS vlan-101

VRF-CiscoLive VRF-CiscoLive 10.10.0.1/24

BD-WOS 10.20.0.1/24
BD-Breakouts
Leaf102 BD-WOS vlan-101
WISP TSC Breakouts
vlan-102
EP1 EP2 EP3 EP1 EP2
VRF-CiscoLive 10.10.0.1/24
10.20.0.1/24

Leaf103 BD-Breakouts vlan-102

vlan-101 vlan-102 vlan-500 vlan-102 vlan-200

vPC Domain
VRF-CiscoLive 10.10.0.1/24
1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4

Leaf104 BD-Breakouts vlan-200

AEP
VPC1 102/1/2 103/1/1 PC1 104/1/3 Pool1
Statics
VRF-CiscoLive DomPhy1
10.30.0.1/24
vlan 100-200 EP1 EP2 EP3 EP1 EP2
vlan 100-200
BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
 EPG Static Path Deployment Leaf101 BD-WOS vlan-101

VRF-CiscoLive VRF-CiscoLive 10.10.0.1/24

BD-WOS 10.20.0.1/24
BD-Breakouts
Leaf102 BD-WOS vlan-101
WISP TSC Breakouts
vlan-102
EP1 EP2 EP3 EP1 EP2
VRF-CiscoLive 10.10.0.1/24
10.20.0.1/24

Leaf103 BD-Breakouts vlan-102

vlan-101 vlan-102 vlan-110 vlan-102 vlan-200

vPC Domain VRF-CiscoLive 10.30.0.1/24

10.10.0.1/24
1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 BD-WOS vlan-110

Leaf104 BD-Breakouts vlan-200

AEP
VPC1 102/1/2 103/1/1 PC1 104/1/3 Pool1
Statics
VRF-CiscoLive DomPhy1
10.30.0.1/24
vlan 100-200 EP1 EP2 EP3 EP1 EP2
vlan 100-200
BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
L2Outs and Loop
Prevention
 Role Description
Spanning Tree R Root port
Classical behavior D Designated port
• STP BPDUs (PVST or MST) are generated
by each switch in the topology. B (Blk) Blocking port
• STP root is elected and interface Root Bridge
forwarding is calculated to prevent loops
by blocking some interfaces.
• All interfaces with best-path (highest D D
bandwidth) towards root bridge will be
forwarding.
• Backup paths will be put in a blocking
state by the switch with worst path
towards root on the affected path R R
(usually based on either the bridge
identifier or port priority)
• Topology changes (TC) trigger MAC D B
addresses to be flushed in received vlan,
allowing traffic reconvergence based on
new topology

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
Spanning Tree
ACI floods BPDUs in the fabric encap
• ACI leaves don’t participate in spanning
tree (generate BPDUs or block any ports)
• STP BPDUs (PVST or MST) are flooded
within the fabric/EPG encap (allocated
per vlan encap in a domain)
• Leaves flush endpoints in the EPG if a TC
BPDU is received. EPG - Web
• Spanning Tree Domain policy
determines which EPGs to flush for
MST domain TCs
BPDU BPDU
NOTE: MST BPDUs are untagged and D D
require an untagged/native EPG to be
deployed on all interfaces connected to Root Bridge
MST domain (this includes L3outs using
SVIs)

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
 Spanning Tree Domain Policy
ACI MST Configuration
Configuration is fabric-wide and supports multiple
regions for use within different tenants/domains.

Any ports connecting to MST switches within the

same region MUST have untagged static-path.

Each MST region should have it’s own EPG for BPDU
flooding.

Fabric -> Access Policies -> Policies -> Switch ->

Spanning Tree -> default
• Add a Region Policy
• Add a Domain Policy for each MST instance
within the region (instance 0 is implicit)
• Add vlan blocks

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
Common mistakes that cause loops
Missing untagged/native EPG in MST region

MST BPDUs are sent untagged by switches

and will only be accepted by leaf if an EPG
is deployed with an untagged/native EPG
path binding.

EPG - Web
All interfaces connected to a common MST
region should have the same EPG deployed vlan-100 vlan-100
(this is to ensure BPDU is flooded to all of D D
the MST switches connected to fabric).
LOOP!!
BPDUD R BPDU
BPDU BPDU
Root
Bridge

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
Common mistakes that cause loops
Multiple fabric encaps used for same EPG

BPDUs are flooded within the fabric encap

of an EPG (allocated based on domain/vlan
pool).
Domain Domain
A B
In order for BPDUs to be flooded properly,
all interfaces within the EPG that are EPG - Web
connected to external bridges MUST reside
in the same physical or L2 external domain vlan-100 vlan-100
and vlan encapsulation. D LOOP!! D
BPDUD R BPDU
BPDU BPDU
Root
Bridge

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
Common STP Misconfiguration
STP Link Type Must Be Shared
BPDU BPDU
Since BPDU’s are flooded, ACI acts as a
HUB from an STP Perspective.

Full Duplex Links default to Spanning-Tree

Link-Type PTP.
SW1 SW3
If multiple switches connect to ACI on BPDU BPDU
separate links, Link-Type must be set to
Shared to allow processing of multiple
BPDU’s on the same interface.

Root(config-if)#spanning-tree link-type shared

Legacy
Network

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
 Loop Prevention - MCP
Mis-Cabling Protocol

Mis-Cabling Protocol can be used to detect loops.

With MCP, a special frame is sent out with a multicast
destination MAC so that the downstream devices will
MCP
flood it. Domain LOOP!! Domain
MCP Can be sent on a per VLAN basis. A
X B
If that frame is received back on a leaf in the fabric, it
will err-disable the interface if ONE of the following EPG - Web
conditions are met: vlan-100 vlan-100
1. MD5 Digest is the same
2. Send time is within ~2s of receive time
D D
D R

SNAP
Fabric ID/Digest/Time
OUI: C
LLC 802.1Q SMAC 0100.0ccd.cdce Root
Bridge

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
 • Introduction
Agenda • Building the Overlay
• Access Policies
• VRFs, Bridge Domains, and
Endpoint Groups
• L2Outs and Loop Prevention

• Traversing the Overlay

• Learning, Forwarding, and Policy
Enforcement
• Shared Services and Route
Leaking
• L3outs and Routing Protocols
• MultiPod

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
Learning, Forwarding,
and Policy Enforcement
 ACI Learning and Forwarding (MAC and IP)

Packet MAC IP
flow
Switched Learned X
Encap + Interface => EPG
Routed Learned Learned
Forwarding lookup

L3Out Learned X
L4/Payload Proto DIP SIP 802.1Q SMAC DMAC

192.168.1.10
BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
 ACI Learning and Forwarding (ARP)

Packet MAC IP
flow
Switched learning X
Encap + Interface => EPG
Routed Learned Learned
Forwarding lookup ARP Learned Learned

L3Out Learned X
Target Target Sender Sender Hdr/ ethtype
802.1Q SMAC DMAC
IP MAC IP MAC Opcode ARP

192.168.1.10
BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
 ACI Learning (Remote - XR) Dst Leaf VTEP
Src Leaf VTEP
Inner Header iVXLAN Outer Header

flags Proto
L4/Payload Proto DIP SIP ethtype SMAC DMAC VNID DIP SIP 802.1Q SMAC DMAC
EPG UDP

EPG (pcTag)
BD or VRF VNID (based on routed or switched)

L2 Learning for (BD, SMAC) => (EPG, Tunnel)

Endpoint database L3 Learning for (VRF, SIP) => (EPG, Tunnel)
VLAN/ Domain Encap MAC/IP Info Interface EPG
Address
BD Name BD VNID SMAC Tunnel oSIP VXLAN Flags
VRF Name VRF VNID SIP Tunnel oSIP VXLAN Flags

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
 ACI Forwarding and QoS
Inner Header iVXLAN Outer Header
Fabric QoS

L4/Payload Proto DIP SIP ethtype SMAC DMAC Used

EPGfor UDP
flags
VNID tracing
Proto
DIP flows
SIP 802.1Q SMAC DMAC

within the fabric.

COS Function
Reserved
Notes
for CPU
generated traffic
3, 4, 5 APIC, SPAN, ContrPln SPAN = low Pri

6 iTraceroute Punted on Leaf

0 Level 1 User Traffic

1 Level 2 1 Priority
2 Level 3 (Default)
2 + DEI Level 4 New in 4.0!
3 + DEI Level 5 User Traffic
5 + DEI Level 6 5 Priority

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 94
 ACI Forwarding and QoS – Preserve COS
Layer 2 COS encoded into 3 bits of DSCP

flags
L4/Payload Proto DIP SIP 802.1Q SMAC DMAC VNID DSCP DIP SIP 802.1Q SMAC DMAC
EPG

Note: COS and DSCP is

Outer COS Value
not used unless custom
QoS policy is configured matches the Level
(Contract/EPG)

Configure Dot1p Preserve! The egress leaf will

look at the 3 MSB bits of the DSCP value to know
which COS value to use for packet rewrite

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 95
Broken traffic flow example Fix? Configure “DSCP class-cos
translation policy for L3 traffic”
Last hop IPN router The spine will map the outer COS
writes COS based on value to a new DSCP class on
DSCP egress and map DSCP to oCOS in
…DSCP 48 = COS6 4 ingress
Datacenter interconnect
(IPN, ISN)
DC1 treats 3
IP packet
packet as with DCSP 48
iTraceroute
5

Data Data
Center 1 Center 2
2
Leaf forwards frame
towards DC1 with
1 COS 0 and an outer
Frame with DSCP of 48
COS 6 set 0b110 000

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 97
Broken Traffic Flow Example
• A Layer3 gateway device (GW) is connected to the fabric via
a normal BD/EPG. Host H3 is using GW as its gateway for a
L3Out subset of traffic.
• The initial EP database show the IP’s and MACs learned in
the correct locations.
Subnet E1 E2 Subnet
int-S1 int-S2 MAC EP Database
BD-B1 BD-B2
BD MAC EPG Port
1/1 1/2 1/3
BD-B1 mac:G1 E1 1/1
BD-B2 mac:G2 E2 1/2
GW H3
IP:G1 IP:G2 BD-B2 mac:H3 E2 1/3
mac:G1 mac:G2 IP:H3
IP EP Database
H3 gateway mac:H3
FW, LB, Router, etc. Vrf IP MAC EPG Port
v1 IP:G1 mac:G1 E1 1/1
v1 IP:G2 mac:G2 E2 1/2
v1 IP:H3 mac:H3 E2 1/3 109
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Broken Traffic Flow Example
• H3 sends a frame to GW on BD-B2 (L2 switched through the
fabric). GW routes the frame and sends it toward the fabric
L3Out to be routed out.
• Fabric performs IP learning on routed traffic, IP:H3 moves to
mac:G1 on EGP E1, port 1/1
Subnet E1 E2 Subnet
int-S1 int-S2 MAC EP Database
BD-B1 BD-B2
BD MAC EPG Port
1/1 1/2 1/3
BD-B1 mac:G1 E1 1/1
BD-B2 mac:G2 E2 1/2
GW H3
IP:G1 IP:G2 BD-B2 mac:H3 E2 1/3
mac:G1 mac:G2 IP:H3
IP EP Database
H3 gateway mac:H3
FW, LB, Router, etc. Vrf IP MAC EPG Port
v1 IP:G1 mac:G1 E1 1/1
v1 IP:G2 mac:G2 E2 1/2
v1 IP:H3 mac:H3
mac:G1 E2
E1 1/3
1/1 110
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Broken Traffic Flow Example
ARP for IP:H3 What’s Broken?
sent out EPG-E1
L3Out • ARP to IP:H3 may fail since the IP is pointing to the wrong
port
• Routed traffic to IP:H3 may be policy dropped since it’s
Subnet E1 E2 Subnet classified in EPG-E1 instead of EPG-E2
int-S1 int-S2
BD-B1 BD-B2 • IP:H3 may rapidly move within the fabric.
1/1 1/2 1/3 IP EP Database
ARP for Vrf IP MAC EPG Port
IP:H3H3
GW v1 IP:G1 mac:G1 E1 1/1
IP:G1 IP:G2
mac:G1 mac:G2 v1 IP:G2 mac:G2 E2 1/2
IP:H3
H3 gateway mac:H3 v1 IP:H3 mac:H3
mac:G1 E2
E1 1/3
1/1
FW, LB, Router, etc.

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 111
Broken Traffic Flow Example
Solutions

L3Out 1. Connect devices that perform routing functionality to

L3Outs.
2. Disable unicast routing on BD-B2 and enable ARP
Subnet E1 E2 Subnet flooding so only MAC is examined when forwarding
int-S1 int-S2 ARP instead of performing (VRF,IP) lookup on ARP
BD-B1 BD-B2 target-IP
1/1 1/2 1/3
3. Enable NAT on routed device connected to internal BD.
In this way, source IP address will be translated
preventing fabric from learning IP address in wrong
GW H3
location
IP:G1 IP:G2
mac:G1 mac:G2 IP:H3 4. Disable IP data-plane learning for VRF
H3 gateway mac:H3
5. Enable IP subnet prefix check on BD-B1 or enable
FW, LB, Router, etc. global subnet check. This will prevent learning of IP’s
outside of the subnets configured under the BD.

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 112
Classical Policy Enforcement
Ingress Egress Type Access Control Entry (ACE) Format
Pipeline Pipeline MAC action src/mask dst/mask ethertype [PD filters]
ARP action opcode srcIp/mask dstIp/mask srcMac/mask
1 2 3 4 5 dstMac/mask [PD filters]
IP/IPv6 action protocol srcIp/mask srcPort/mask
dstIp/mask dstPort/mask [PD filters]

• Multiple logical locations where ACLs can

be applied depending on what type of
Egress VLAN ACL traffic and what type of filters are needed
Egress Routed ACL (very flexible)
Ingress Routed ACL • ACE primarily based on src and dst values
within frame (may be hard to maintain)
Ingress VLAN ACL
Ingress Port ACL • ACLs often need to be configured and
maintained on multiple devices in the
network

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 117
 Policy is created based on contract between
ACI Policy Enforcement
•
EPGs with support for L2/L3/L4 filters similar to
traditional ACLs.
• Leaf derives source EPG pcTag based on:
Scope Access Control Entry (ACE) Format • match in EP database
VRF action src-EPG dst-EPG [filters] src MAC for L2 traffic or src IP for L3 traffic
VRF permit any any (unenforced mode) • longest-prefix match against src IP
(IP-based EPG or L3Out external EPG)
• ingress port + encap
1
• Leaf derives destination EPG pcTag based on:
• match in EP database
dst MAC for L2 traffic or dst IP for L3 traffic
Apply Policy • longest-prefix match against dst IP
(L3Out external EPG or shared-services)
Derive destination EPG pcTag
EP lookup, IP Prefix • Rules are programmed with scope of VRF.
Policy lookup is always (VRF, src-EPG, dst-EPG,
Derive source EPG pcTag filter).
local EP, IP Prefix, or Encap
• Allow traffic between all EPGs without a contract
by setting the VRF to unenforced mode

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 118
 ACI Policy Enforcement SYN
Web Server
(S1)
Reference TCP Packet SYN+ACK
H1
Data
Seq#, Ack# Dst Src Proto
DIP SIP ethtype SMAC DMAC
ACK
flags, etc.. Port Port TCP
port x data… port 80

Classical Switch ACL ACI Contract

Generally applied at one or more L3 boundaries
assuming H1 and S1 are in different subnets
H1
ip access-list web
permit tcp host H1 host S1 eq 80
permit tcp host S1 eq 80 host H1 EPG-Client EPG-Web
BD-X VRF-V1 BD-Y
ACI Desired Behavior
EPG-Web is Providing
Scope Access Control Entry
a service on port 80
VRF-V1 permit tcp EPG-Client EPG-Web eq 80
VRF-V1 permit tcp EPG-Web eq 80 EPG-Client
How do we get here?

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 119
 Option 1 – Unidirectional filters
ACI Policy Enforcement Apply both flt-1 and flt-2 to subject
 Identify Provider (P) EPG and Consumer (C) EPG
flt-1 (C to P) and flt-2 (P to C)
src-port dst-port
permit tcp Consumer Provider eq 80
H1 C P
permit tcp Provider eq 80 Consumer

EPG-Client EPG-Web
Option 2 – Bidirectional filters with reverse ports
BD-X VRF-V1 BD-Y
flt-1 (C to P implied)

• With a bidirectional contract, the ‘provider’ will be permit tcp Consumer Provider eq 80
the dst-port filters and the ‘consumer’ will be the
src-port filters (opposite of contract arrows)
flt-1 + apply both directions
 Create Filters permit tcp Consumer Provider eq 80
Name EthType Proto Src Port Dst Port permit tcp Provider Consumer eq 80 Only flt-1
flt-1 IP TCP Any 80 needed!
flt-2 IP TCP 80 Any flt-1 + apply both directions + reverse ports

 Create a contract, subject, and filter(s). Apply to EPGs permit tcp Consumer Provider eq 80
EGP-Web as provider and EPG-Client as consumer permit tcp Provider eq 80 Consumer

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 120
High Policy CAM Utilization Example
E0 • 100 EPGs all providing a basic management
E1 E1 E1 E1 contract to a single consumer EPG.
E2 E2 E2 E2
E3 E3 E3 E3 mgmt- mgmt- • TCAM Utilization Calculation (Approximate)
E2 E2 E2 E2 contract EPG ~= (entries in contract)(# of Cons)(# of
E1 E2 E3 E4 Providers)(2)
100 EPGs ~= 2 * 1 * 100 * 2
~= 400 entries in hardware
Name EthType Proto Src Port Dst Port
flt-ssh IP TCP 1-65535 22
• Policy CAM utilization increases by over 6400
flt-snmp IP UDP 1-65535 161 Why?

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 127
 High Policy CAM Utilization Example
Name EthType Proto Src Port Dst Port permit tcp E1 eq 1 E0 eq 22

flt-ssh IP TCP 1-65535 22 permit tcp E1 2-3 E0 eq 22

flt-snmp IP UDP 1-65535 161 permit tcp E1 4-7 E0 eq 22

Expanded permit tcp E1 8-15 E0 eq 22

• Port Ranges permit tcp E1 16-31 E0 eq 22
Policy CAM, as with any TCAM, uses a value and
permit tcp E1 32-63 E0 eq 22
mask to perform matching.
• Matching a single port utilizes only one entry in TCAM. permit tcp E1 64-127 E0 eq 22
• Using a range of ports may need to be expanded to permit tcp E1 128-255 E0 eq 22
multiple entries in hardware depending on the start and permit tcp E1 256-511 E0 eq 22
end values.
permit tcp E1 512-1023 E0 eq 22

How to fix this issue? permit tcp E1 1024-2047 E0 eq 22

• Use port 0-65535 or ‘unspecified’ source port permit tcp E1 2048-4095 E0 eq 22

=> utilization down from 6400 to 400 entries permit tcp E1 4096-8191 E0 eq 22
• Consider using VzAny if all EPGs in the VRF need it permit tcp E1 8192-16383 E0 eq 22
=> utilization down from 400 to 4 entries
permit tcp E1 16384-32767 E0 eq 22
E0 permit tcp E1 32768-65535 E0 eq 22
Any
mgmt- mgmt-
contract EPG BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 128
ACI Preferred Group
Allow any any for a subset of EPGs • EPGs that are part of the preferred group do
not require contracts to communicate with
each other
• EPGs and External EPGs can be configured to
S1 included or excluded from the preferred
S2 group
• EPGs which are excluded, have hardware
rules programmed to prevent
communication to EPGs which are included
L1 L2 L3 L4 L5 L6 Contract VRF Action Src Dst Filter
C1 V1 permit E2 E3 all
VRF-V1 V1 permit E3 E2 all

H2 L3Out V1 permit E2 ext1 all

H1 H3 H3
V1 permit E3 ext1 all
EPG-E1 EPG-E2 EPG-E3 ext
V1 permit ext1 E2 all
1
implicit V1 deny any any all

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 130
ACI Preferred Groups
• Only recommended if the majority of EPGs
Allow any any for a subset of EPGs require unenforced policy
• Deny rules are installed for EPGs outside of
the preferred groups
• Contracts can still be used to enable
S1 S2 communication between excluded and
included EPGs

L1 L2 L3 L4 L5 L6 Contract VRF Action Src Dst Filter

implicit V1 deny any E1 all
VRF-V1 V1 deny E1 any all

H2 L3Out implicit V1 permit any any all

H1 H3 H3

EPG-E1 EPG-E2 EPG-E3 ext

1

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 131
 ACI Contracts and Resource Utilization
Contract created between E2 and E3 • BD-B1 and BD-B2 each have a subnet
defined. Subnet int-S1 on BD-B1 exists on
E2 E3 L1 and L3, while subnet int-S2 for BD-B2
exists on L6

S1 When creating the contract between E2 and E3:

Add contract and S2 Add contract and
• Program contract rule between E2 and E3
route to int-S2 route to int-S1
in TCAM. Add Static route for int-S1
created on L6 pointing to spine proxy.

L1 L2 L4 L5 L6 • Program contract rule between E2 and E3

L3
in TCAM. Add Static route for int-S2
created on L3 pointing to spine proxy.
• Contracts are only programmed on leafs that
have provider/consumer EPGs. BD routes are
H1 H2 H3 only programmed on leafs that need them!
BD-B1 VRF-V1 BD-B2
Subnet Subnet Contracts contribute to both
EPG-E1 int-S1 EPG-E2 int-S2 EPG-E3
policy AND routing entries on
leafs!

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 132
 ACI Policy Enforcement ARP has resolved on hosts for ACI GW
L1 has not learned H3 from L6
Unknown Layer3 Unicast 1. H1 sends layer3 unicast frame to H3
(destination MAC of BD-B1).
2. L1 performs layer3 lookup on H3
Policy Applied destination IP and pervasive route
S1 S2 on egress L6 pointing to the Spine Proxy.
L1 does not set policy applied bits -
2 3 frame is sent to Proxy TEP with EPG-E1
(PCTag) and VRF-V1 set in VXLAN
header.
4
L1 L2 L3 L4 L5 L6 3. Spine receives frame and preforms
proxy lookup. Frame is sent to L6.
5
1 4. L6 does layer3 lookup on H3 destination
IP in VRF-V1. Hit in local EP database
H1 H2 H3 and derives destination EPG-E3
BD-B1 VRF-V1 BD-B2 (PCTag). Policy check is enforced
EPG-E1 EPG-E2 EPG-E3
5. L6 forwards traffic to H3 with
appropriate encap if permitted by
contract
BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 133
 ACI Policy Based Redirect (PBR)
UDP 1) Create L4-L7 Device
H1 C P Define Interface, VLAN, etc.
PBR 2) Create redirect policy
HTTP Contains the MAC & IP of service
EPG-Client EPG-Web
Device
BD-X BD-Y 3) Create Graph Template & check
MAC A.A.A Redirect
4) Apply Graph template between
• Contract can now redirect traffic to service device two EPGs
(FW, LB etc) for inspection prior to allowing Creates redirect contract
Name EthType Proto Src Port Dst Port Action Can be reused with different EPGs

flt-1 IP TCP Any 80 Redirect

(Grp 1)
flt-2 IP UDP Any Any Permit

Name Dest MAC Dest BD Tunnel Int

Redir-Grp1 A.A.A ServiceBD Mac Proxy

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 134
Shared Services and
Route Leaking
 export to T2
ACI Shared Services Consume
Interface
Provide
C1 C1-export
• What is a shared service? scope: global
• Shared Service (Route Leaking) enables traffic
between endpoints in different VRFs. E1 E2 E3 E4
• A shared service EPG provider is an EPG that
BD-B1 BD-B2
provides a contract consumed by an EPG in a
different VRF
VRF-V1 VRF-V2
Restrictions
Tenant-T1 Tenant-T2
• Provider Subnet must be defined under the
provider EPG EPG-E1 Subnet: S1 BD-B2 Subnet: S2
• Both provider and consumer subnets scope: shared scope: shared
must have scope set to shared VRF Route pcTag Flags VRF EPG pcTag
• contract needs correct scope
V1 S1 1 proxy V1 E1 49155
• VzAny not supported as provider
V2 S2 1 proxy V1 E2 49156
Scope: V2 E3 16387
 Private to VRF
V2 E4 49155
 Advertise Externally
 Share Between VRFs BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 136
 export to T2
ACI Shared Services Consume
Interface
Provide
• What happens in the fabric? C1 C1-export
scope: global
• EPG-E1 is now a shared service
provider. It is reallocated a fabric unique E1 E2 E3 E4
pcTag (<16384)
BD-B1 BD-B2
• All subnets on consumer BD
programmed in provider VRF VRF-V1 VRF-V2
• Provider subnet programmed in Tenant-T1 Tenant-T2
consumer VRF with pcTag of provider
EPG-E1 Subnet: S1 BD-B2 Subnet: S2
EPG
scope: shared scope: shared
VRF Route pcTag Flags VRF EPG pcTag
V1 S1 1 proxy V1 E1 49155
17
V1 S2 1 proxy, rewrite V1 E2 49156
VNID(V2)
V2 E3 16387
V2 S2 1 proxy
V2 E4 49155
V2 S1 1717 proxy, rewrite
BRKACI-3101
VNID(V1)
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 137
 export to T2
ACI Shared Services Consume
Interface
Provide
• What happens in the fabric? C1 C1-export
scope: global
• EPG-E1 is now a shared service
provider. It is reallocated a fabric unique E1 E2 E3 E4
pcTag (<16384)
BD-B1 BD-B2
• All subnets on consumer BD
programmed in provider VRF VRF-V1 VRF-V2
• Provider subnet programmed in Tenant-T1 Tenant-T2
consumer VRF with pcTag of provider
EPG-E1 Subnet: S1 BD-B2 Subnet: S2
EPG
scope: shared scope: shared
• Policy enforcement always performed in
consumer VRF. Therefore, contracts are Contract VRF Action Src Dst Filter
always programmed in consumer VRF. C1 V2 permit E4 E1 flt1
V2 permit E1 E4 *flt1
V1 - - - -

No Rule added in provider VRF

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 138
 Shared Service Forwarding 1. H1 sends packet toward gateway in
EPG-E1 with destination IP of H3
2. L1 performs layer3 lookup for H3 in
• From Provider E1 to Consumer E4 VRF-V1 and hits LPM entry for H3
subnet. LPM entry points to proxy with
VNID rewrite info for VRF-V2.
Policy Applied on Packet is sent to Spine Anycast IPv4
S1 S2 egress L6 Proxy VTEP with VRF-V2 VNID and
(consumer VRF) EPG-E1 set in VXLAN header.
2 3 No policy applied in provider VRF

4 3. Spine performs proxy lookup for H3 IP in

L1 L2 L6 VRF-V2. Normal Proxy behavior to
L3 L4 L5
forward packet to VTEP of L6
5
4. L6 performs layer3 lookup on H3
1 destination IP in VRF-V2. Hit in local EP
database and derives destination EPG-
H1 H3 E4 L6 applies policy between EPG-E1
BD-B1 BD-B2
and EPG-E6
EPG-E1 EPG-E4
5. If permitted, traffic forwarded to H3 with
appropriate encap
VRF-V1 VRF-V2
BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 143
Shared Service Forwarding 1. H3 sends packet toward gateway in
EPG-E4 with destination IP of H1
2. L6 performs layer3 lookup for H1 in
• From Consumer E4 to Provider E1 VRF-V2 and hits LPM entry for H1
subnet. LPM entry points to proxy with
VNID rewrite info for VRF-V1 and pcTag
Policy Applied on of EPG-E1.
S1 S2 ingress L6 L6 applies policy between EPG-E4 and
(consumer VRF) EPG-E1 in consumer VRF-V2.
3 If permitted, packet is sent to Spine
2 Anycast IPv4 Proxy VTEP with
4 VRF-V1 VNID and EPG-E4 set in VXLAN
L1 L2 L3 L4 L5 L6
3. Spine performs proxy lookup for H1 IP in
VRF-V1. If unknown drops the packet.
5 1 Else forward to VTEP of L1
4. L1 performs layer3 lookup on H1
H1 H2 H3 destination IP in VRF-V1. Hit in local EP
BD-B1 BD-B2
database and derives destination EPG-
EPG-E1 EPG-E4 E1 Policy already applied by L6
5. Traffic is forwarded to H1 with
VRF-V1 VRF-V2
appropriate encap
BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 144
 Policy TCAM
Contract Review Contract VRF Action Src Dst Filter
C1 V2 permit E2 E1 flt1
• Shared Service EPGs V2 permit E1 E2 *flt1
EPGs that provide contract consumed by C2 V2 permit E4 E3 flt2
EPG in a different VRF: E1, E2*
V2 permit E3 E4 *flt2
• Application EPGs V2 permit ext2 E3 flt2
E1, E2, E3, E4 V2 permit E3 ext2 *flt2
C3 V2 permit ext1 any flt3
• External EPGs
configured on L3Out and classified based V2 permit any ext1 *flt3
on IP prefix: ext1, ext2

• VzAny E1 E2 E3 E4 Any L3Out

Represents all EPGs in a single VRF: Any subnet
int-S1
ext ext
Contract Assumptions for this Example : 1 2
• All contract subjects have both directions C3
subnet subnet
C1 C2 ext-S1 ext-S2
and reverse filters enabled.
VRF-V1 VRF-V2
BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 148
L3outs and Routing
Protocols
 Basic Connectivity node-103
RID: #
node-104
RID: #
IP: A IP: B

Layer3 Out: L3Out-1

VRF: VRF-V1
Layer-3 Domain: DomL3 vlan-x
Logical Node Profile: node-103-104

node: node-103 node: node-104 L3Out-1

Router-ID: # Router-ID: #
VRF-V1
Logical Interface Profile: ipv4-lif
Create the L3Out
• Associate VRF and L3 Domain
path: topology/pod1/…vpcX • Create Logical Node Profile and associate fabric
type: ext-svi, encap: vlan-x nodes to the L3Out.
IP-A, IP-B, MTU, MAC
• Create Logical Interface Profile
• Specify Path attributes containing physical interface,
encapsulation, and IPs

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 150
 Configuring Routing Protocols
Layer3 Out: L3Out-1 Enable Protocol
VRF: VRF-V1 BGP (fabric ASN configured in fabric pod policy)
Layer-3 Domain: DomL3 OSPF - area, area-type, area-configuration
EIGRP - ASN
Logical Node Profile: node-103-104

node: node-103 node: node-104 Static Routes configured per node

Router-ID: # Router-ID: # BGP Peer Profiles tied to Loopback

Logical Interface Profile: ipv4-lif

BGP Peer Profiles tied to Interface
path: topology/pod1/…vpcX OSPF Interface Policy and Authentication
type: ext-svi, encap: vlan-x EIGRP Interface Policy
IP-A, IP-B, MTU, MAC BFD Interface Policy
Majority of configuration under the interface/peer policies…

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 158
 Ensure BGP RR
Types of Fabric Routes is configured to
enable MP-BGP

MP-BGP
overlay-1

L3Out-1 L3Out-2

E1 E2 ext
ext
1 BD-B1 BD-B2 2
subnet subnet subnet subnet
ext-S1 int-S1 int-S2 ext-S2

• Internal Routes: Subnets defined under BD are internal routes and create static pervasive routes within
the fabric.
• External Routes: Routes learned via a routing protocol or static routes configured under an L3Out.
These routes are redistributed into MP-BGP and advertise to all leafs that contain the VRF
• Transit Routes – Routes advertised between L3Outs.

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 161
 Types of Fabric Routes – Internal Routes
Subnet: int-S2 3
Subnet int-S2 installed on border leaf
Scope: MP-BGP
when creating contract between EPG
 Private to VRF E2 and external overlay-1
EPG ext2
 Advertise Externally
 Share Between VRFs
L3Out-1 L3Out-2
1

E1 E2 ext
ext
1 BD-B1 BD-B2 2 2
subnet subnet subnet C1 subnet
ext-S1 int-S1 int-S2 ext-S2
There are three requirements to advertise Internal Routes out an L3Out:
1. The BD must be associated with the L3Out*
The association adds prefix entry to route map controlling advertised routes
2. A contract must exists between an EPG within the BD and an external EPG on the L3Out.
The contract creates internal BD route on border leaf (cannot advertise route until it exists locally)
3. The subnet must have a public scope (Advertise Externally)

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 163
 Types of Fabric Routes – External Routes
ext-S1
ospf, eigrp, static route RD: L1:V1
redistributed into bgp and RT: ASN:V1 MP-BGP
exported into mp-bgp
overlay-1
ext-S1

L3Out-1 L3Out-2

VRF-V1 VRF-V1 VRF-V1

ext import RT from mp- ext
ext-S1 ext-S1
1 bgp and install route 2
subnet via:L1(bgp) into vrf as bgp learn via:L1(bgp) subnet
ext-S1 ext-S2

• External Routes from ospf, eigrp, or static are redistributed on the border leaf into the local bgp process.
• The bgp route is exported into MP-BGP with a route-target (RT) of the corresponding VRF. Each leaf in the fabric with
the VRF present will import the RT and install the route. External routes on the non-originating border leaf will be seen
as bgp learned routes.
• External Routes are controlled via Import Route Control flag

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 165
 Types of Fabric Routes – Transit Routes

MP-BGP
overlay-1
ext-S1
ext-S1

L3Out-1 L3Out-2

VRF-V1 VRF-V1 VRF-V1

ext ext
ext-S1 ext-S1
1 2
subnet via:L1(bgp) via:L1(bgp) subnet
ext-S1 ext-S2

• In this example, external route ext-S1 is a Transit Route when advertised out L3Out-2.
• If OSPF or EIGRP on L3Out-2, ext-S1 is redistributed from BGP into the IGP and advertised.
• Transit Routes are controlled via Export Route Control flag

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 166
 Configure L3Out External Network node-103
RID: #
node-104
RID: #
IP: A IP: B
Define an External Network, ext1 in this example
• Note: At least one external network required to
bring up L3Out interfaces on border leaf
vlan-x
• Add Subnet to External Network

Prefix-based EPG for Contracts: Scope:

• External Subnets for the External  Export Route Control Subnet L3Out-1
EPG  Import Route Control Subnet ext
• Shared Security Import  External Subnets for the External 1 VRF-V1
EPG
Route Control  Shared Route Control Subnet subnet
• Export Route Control  Shared Security Import Subnet ext-S1
• Import Route Control
• Shared Route Control Categorize Aggregate: Subnet
• Aggregate Export options  Aggregate Export
options
• Aggregate Import  Aggregate Import
• Aggregate Shared Routes  Aggregate Shared Routes

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 167
 External Subnets for the External EPG
Previously: Import-Security Subnet: ext-S2/mask
Scope:
External Subnet for the External EPG is used to classify  External Subnets for the External EPG
dataplane packets into external EPG for policy enforcement.
EPG to pcTag
• An IP prefix is installed into leaf TCAM to classify traffic VRF EPG pcTag
to/from the external network and assign correct pcTag for
policy enforcement V1 E1 49156
V1 E2 16387
Host Table LPM Table
V1 ext2 49155
VRF EP PcTag Dst VRF Subnet PcTag Dst
V1 Host1 49156 Leaf1 V1 int-S1 1 Proxy
E1 L3Out
V1 Host2 16387 Leaf2 V1 ext-S2
ext-S2 49155 L3Out

L4/Payload Proto DIP SIP 802.1Q SMAC DMAC ext

2
C1 subnet
• Apply policy between src E1(49156) and dst ext2(49155) ext-S2
BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 169
 Import Route Control Subnet: ext-S1/mask
*Import Route Control supported only for OSPF & BGP Scope:
Import Route Control is used to filter External Routes  Import Route Control Subnet
received on an L3Out

• A route-map is created per BGP neighbor to filter

incoming routes. Subnets defined with the import
flag will be added to corresponding prefix list to
allow in remote routes.
• The import flag must be enabled on the L3Out to
set import flag per external subnet.
• By default, import is disabled on the L3Out

neighbor neighbor-1
Inbound route-map imp-l3out-vrf
L3Out-1
Outbound route-map exp-l3out-vrf Allow
Advertisement:
route-map imp-l3out-peer-vrf permit
- ext-S1/mask
match: prefix-list IPv4-network-vrf-exc-ext-inferred-import-dst
- ext-S2/mask
ip prefix-list IPv4-network-vrf-exc-ext-inferred-import-dst BGP - ext-S3/mask
permit ext-S1/mask Neighbor-1 Ignore

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 171
 Aggregate Import Subnet: 0.0.0.0/0
Scope:
*Aggregate Import supported only for 0.0.0.0/0 or ::/0  Import Route Control Subnet
Aggregate:
Import Route Control allows fabric to permit a  Aggregate Import
specific prefix. Instead of creating each prefix
advertised by a neighbor, multiple prefixes can be
aggregated together by using the Aggregate Import
flag.

neighbor neighbor-1
Inbound route-map imp-l3out-vrf
Outbound route-map exp-l3out-vrf

route-map imp-l3out-peer-vrf permit

match prefix-list IPv4-network-vrf-exc-ext-inferred-import-dst L3Out-1 Allow All

ip prefix-list IPv4-network-vrf-exc-ext-inferred-import-dst Advertisement:

permit 0.0.0.0/0 le 32 - ext-S1/mask
- ext-S2/mask
- ext-S3/mask
BGP
Neighbor-1

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 172
 Export Route Control & Aggregate Export
Subnet: ext-S1/mask
Export Route Control allows Transit Routes to be Scope:
advertised out of the fabric.  Export Route Control Subnet
• Export control does NOT affect pervasive BD SVIs,
they are only advertised when the BD is
associated with the L3Out.
• Similar to import route control subnet, a prefix list
with corresponding exported subnets is created to
allow routes to be advertised out

Aggregate Export is identical concept to aggregate

import, allowing prefixes to be aggregated together
in export direction. L3Out-1 L3Out-2

Subnet: 0.0.0.0/0
Scope: Export all Transit Advertisement: Export:
 Export Route Control Subnet Routes within - ext-S1/mask - ext-S1/mask
Aggregate: VRF - ext-S2/mask
- ext-S3/mask
 Aggregate Export

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 173
Shared L3Out
Similar to Shared Services, a Shared L3Out uses
contracts to leak routes between VRFs. The leaked
routes can be:
int-S1 subnet from VRF-V1 to VRF-V2
ext-S2 subnet from VRF-V2 into VRF-V1
Similar Restrictions as Shared Services
If the application EPG is providing the contract for E1 L3Out-1
shared L3Out, the internal subnet must be defined
under the EPG. BD-B1
If the external EPG is providing the contract for ext
shared L3Out, then internal subnet can be defined VRF-V1 2
either under the EPG or the BD EPG-subnet
C1
subnet
Internal subnet must have shared and Advertise int-S1 ext-S2
Externally(public) scope.
Contract must be appropriately scoped. Scope: VRF-V2
For shared L3Out, shared subnet must be globally  Private to VRF
unique within the entire ACI fabric.  Advertise Externally
 Share Between VRFs

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 174
 Shared L3Out E1 L3Out-1
What happens in the fabric when contract is added?
BD-B1
ext
• Internal Route int-S1 leaked into VRF-V2 VRF-V1 2
ext-S2 route not leaked into VRF-V1 yet… EPG-subnet C1 subnet
int-S1 ext-S2
• Shared-Service prefix list added to route-map
permitting advertisement of int-S1. External routers VRF-V2
can now learn int-S1 through OSPF, EIGRP, or BGP
on VRF-V2.
Assume: VRF-V2 has a route to ext-
No need to associate BD to shared L3Out, route
S2 through static or dynamic route
controlled by contract!
Forwarding Table
• Shared-Service contract programmed onto leaf to
allow traffic flow. VRF Route pcTag Flags
V1 int-S1 1 proxy
• Problems:
• VRF-V1 does not have return route to ext-S2 V2 ext-S2 ext2 L3Out
• Even though rule is programmed, return traffic V2
V2 int-S1
int-S1 E1 proxy, leak->V1
from VRF-V1 can’t derive destination pcTag so
no policy available to enforce

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 175
 Shared L3Out E1 L3Out-1
Completing the Configuration
BD-B1
ext
VRF-V1 2
Shared Route Control flag allows external route to be
leaked into EPG context. EPG-subnet C1 subnet
int-S1 ext-S2
• In this example, adding shared route control to the
external subnet allows ext-S2 to be leaked into VRF- Subnet: ext-S2/mask VRF-V2
V1, but pcTag set to reserved drop value. Scope:
 Shared Route Control
Shared Security Import is used to classify dataplane
packets into external EPG for policy enforcement for  Shared Security Import
shared prefixes Forwarding Table

• In this example, adding shared security import to the VRF Route pcTag Flags
external subnet created a prefix-based EPG in any- V1 int-S1 1 proxy
VRF* for the external subnet ext-S2 with pcTag of
EPG-ext2. V2 ext-S2 ext2 L3Out
V2 int-S1 E1 proxy, leak->V1
V1
V1 ext-S2
ext-S2 ext2
deny-tag L3Out, leak->V2

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 176
 Aggregate Shared Subnet: 8.8.0.0/16
Scope:
Supported for any prefix, not just 0.0.0.0!  Shared Route Control
 Shared Security Import
Aggregate Shared flag allows multiple  Aggregate Shared
prefixes from L3Out to be shared/leaked into
another VRF.
In this example, a /16 prefix is configured with
aggregate shared flag set. The external
router advertised multiple /24 subnets within
the range. Each are leaked into VRF-V1
E1 L3Out-1
Advertisement:
Restrictions - 8.8.8.0/24
Shared Route control subnets cannot be a BD-B1 - 8.8.9.0/24
subset of Shared Security import. For example: ext2 - 8.8.10.0/24
8.8.0.0/16 VRF-V1
C1
• shared security import + shared route VRF-V2
control + aggregate shared Forwarding Table
8.8.10.0/24
VRF Route pcTag Flags
• shared route control (only)
Traffic on VRF-V1 toward 8.8.10.0/24 dropped V1 8.8.8.0/24 ext2 L3Out, leak->V2
V1 8.8.9.0/24 ext2 L3Out, leak->V2
V1 8.8.10.0/24 ext2 L3Out, leak->V2
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 177
 Aggregate Shared 8.8.8.0/24
8.8.9.0/24
RD: L4:V2
8.8.10.0/24
RT:RD: L4:V2
How does this work? ASN:V2
RT:RD: L4:V2
ASN:V2
MP-BGP RT: ASN:V2
vpnv4 VRF-V1 overlay-1
• Leaf4 exports routes into MP-BGP with import
route-target for VRF V2 RT: ASN:V1
L1 vpnv4 VRF-V2
• Leaf1 imports routes with route-targets RT: ASN:V2 L4
export
from both VRF-V1 and VRF-V2 into RT: ASN:V2
V1 vrf. Routes are filtered with route-map
based on subnet control flags

E1 L3Out-1
Advertisement:
leaf101# show bgp process vrf V1
- 8.8.8.0/24
Import route-map V1-shared-svc-leak BD-B1 - 8.8.9.0/24
Import RT list:
ext2 - 8.8.10.0/24
ASN:V1 VRF-V1
ASN:V2 C1
... VRF-V2
route-map V1-shared-svc-leak, permit, sequence 1000*
Match clauses:
ip address prefix-lists: IPv4-V2-V1-shared-svc-leak

ip prefix-list IPv4-V2-V1-shared-svc-leak
seq 3 permit 8.8.0.0/16 le 32
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 178
 L3 External Subnet Review
• External Subnets for the External EPG (Security Import)
Used to classify dataplane packets into external EPG for policy enforcement

• Export Route Control

filter Transit Routes advertised out of the fabric.

• Import Route Control

filter External Routes received on an L3Out

• Shared Security Import

used to classify dataplane packets into external EPG for policy enforcement for shared/leaked prefixes

• Shared Route Control

Allows external route to be leaked into another VRF

• Aggregate Export - allows prefixes to be aggregated together in export direction (0/0 or ::/0 only)
• Aggregate Import - allows prefixes to be aggregated together in import direction (0/0 or ::/0 only)
• Aggregate Shared Route - allows prefixes to be aggregated together for shared route control

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 179
Agenda
• Introduction
• Building the Overlay
• Access Policies
• VRFs, Bridge Domains, and EPGs
• L2Outs and Loop Prevention
• Traversing the Overlay
• Learning, Forwarding, and Policy
Enforcement
• Shared Services and Route Leaking
• L3outs and Routing Protocols

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 197
 Cisco Webex Teams

Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session

How
1 Find this session in the Cisco Events Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space

cs.co/ciscolivebot#BRKACI-3101

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 198
Complete your online
session survey
• Please complete your Online Session
Survey after each session
• Complete 4 Session Surveys & the Overall
Conference Survey (available from
Thursday) to receive your Cisco Live T-
shirt
• All surveys can be completed via the Cisco
Events Mobile App or the Communication
Stations

Don’t forget: Cisco Live sessions will be available for viewing

on demand after the event at ciscolive.cisco.com

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 199
Continue Your Education

Demos in Meet the Related

Walk-in
the Cisco engineer sessions
self-paced
Showcase labs 1:1
meetings

BRKACI-3101 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 200
Thank you