VxLAN Routing and Control Plane on Nexus

9000 Series Switches

Lilian Quan Technical Marketing Engineering, INSBU
Chad Hintz TSA, US Commercial

Contributors and Acknowledgements

Lukas Krattiger

Leo Boulton

David Jansen

Victor Moreno

Vaughn Suazo

Kevin Corbin

Yves Louis

Dave Malik

Babi Seal

Lilian Quan

James Christopher

Mike Herbert

Jim Pisano

Juan Lage

Matt Smorto

Chad Hintz

Jason Pfiefer

Priyam Reddy

Errol Roberts

Lilian Quan

Brenden Buresh

Jason Gmitter

Cesar Obediente

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Agenda
VxLAN Overview
MP-BGP EVPN Basics
MP-BGP EVPN Control Plane
VXLAN Design Options
MP-BGP EVPN VXLAN Configuration

VxLAN Capability on Nexus 9000

Series Switches

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Agenda
VxLAN Overview
MP-BGP EVPN Basics
MP-BGP EVPN Control Plane
VXLAN Design Options
MP-BGP EVPN VXLAN Configuration

VxLAN Capability on Nexus 9000

Series Switches

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Data Center Fabric Journey

STP
VPC
FabricPath

Application Centric Infrastructure

VXLAN

APIC
MAN/WA
Application
Policy
N
Infrastructure
Controller

ACI Fabric
FabricPath
/BGP

VXLAN
/EVPN

MAN/WA
N
2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

MAN/WA
N

Trend: Flexible Data Center Fabrics

Create Virtual Networks on top
of an efficient IP network

Hosts

V
M

V
M

O
S

O
S

Workload Mobility
Workload Placement
Segmentation
Scale
Automation & Programmability
L2 + L3 Connectivity
Physical + Virtual
Open

Physical

Virtual

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Why Do We Need Overlays?

Location and Identity Separation
Traditional Behaviour
10.1.0.1

Loc/ID Overloaded Semantic

IP core

Device IPv4 or IPv6

Address Represents
Identity and Location

20.2.0.9

When the Device Moves, It Gets

a New IPv4 or IPv6 Address for
Its New Identity and Location

Overlay Behaviour
Loc/ID Split

IP core

10.1.0.1
Device IPv4 or IPv6 1.1.1.1
2.2.2.2
Address Represents
Identity Only.
Its Location Is Here!
Only the Location Changes
2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

When the Device Moves, Keeps

Its IPv4 or IPv6 Address.
It Has the Same Identity
10.1.0.1

Underlay Network:

IP Transport
Network

IP routing proven, stable, scalable

ECMP utilize all available network paths

Overlay Network:
VXLAN VNI
VTE
P

VTE
P

Local LAN

VTEP

LocalLAN
LANSegment Local LAN

2014 Cisco and/or its affiliates. All rights reserved.

VTEP

Local LAN

Cisco Public

Standards-based overlay

Layer-2 extensibility and mobility

Expanded Layer-2 name space

Scalable network domain

Multi-Tenancy

 No VXLAN control plane

Data driven flood-&-learn
Multicast transport for VXLAN BUM (Broadcast, Unknown Unicast and Multicast) traffic.
End System

End
System

End System
A
MAC-A
IP-A

VTEP 1
IP-1

2014 Cisco and/or its affiliates. All rights reserved.

VTEP
-3

VTEP
3
IP-3

VTEP-1

Multicast
Group
IP Network

Cisco Public

VTEP2
VTEP 2
IP-2

End System B
MAC-B
IP-B

Sound Familiar?

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

FabricPath
Shortest path any to any
MAC

IF

e1/1

s8, e1/2

FabricPath
e1/1

s3

s8

e1/2

Single address lookup at the ingress edge identifies the exit port
across the fabric
Traffic is then switched using the shortest path available
Reliable L2 and L3 connectivity any to any
(L2 as if it was within the same switch, no STP inside)
2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

The Secret Sauce is the Control

Plane, not the Encapsulation

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Agenda
VxLAN Overview
MP-BGP EVPN Basics
MP-BGP EVPN Control Plane
VxLAN Design Options
MP-BGP EVPN VXLAN Configuration

VxLAN Capability on Nexus 9000

Series Switches

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

MP-BGP with MPLS VPN Route Distribution

Exchange of VPN Policies Among PE Routers
Full mesh of BGP sessions
among all PE routers
BGP Route Reflector

Multi-Protocol BGP extensions

(MP-iBGP) to carry VPN policies

CE

PE-CE routing options

Blue VPN
Policy
Red VPN
CE Policy

BGP Route Reflector

PE-CE
Link

Static routes
eBGP
OSPF
IS-IS

PE

PE

Cisco Public

PE

CE
BlueVPN
Policy`
Red VPN
Policy CE

Label Switched Traffic

2014 Cisco and/or its affiliates. All rights reserved.

PE-CE
Link

PE

VPN Control Plane Processing

VRF Parameters
Make customer routes unique:
Route Distinguisher (RD):
8-byte field, VRF parameters; unique value to make VPN IP routes unique

VPNv4 address: RD + VPN IP prefix

Selective distribute VPN routes:
Route Target (RT): 8-byte field, VRF parameter, unique value to define the
import/export rules for VPNv4 routes
MP-iBGP: advertises VPNv4 prefixes + labels

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VPN Control Plane Processing

Interactions Between VRF and BGP VPN Signaling
1. CE1 redistribute IPv4 route to PE1
via eBGP
2. PE1 allocates VPN label for prefix
learnt from CE1 to create unique
VPNv4 route
3. PE1 redistributes VPNv4 route into
MP-iBGP, it sets itself as a next
hop and relays VPN site routes to
PE2
4. PE2 receives VPNv4 route and, via
processing in local VRF (green), it
redistributes original IPv4 route to
CE2
2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

BGP advertisement:
VPN-IPv4 Addr =
RD:16.1/16
BGP Next-Hop = PE1
Route Target = 100:1
Label=42

eBGP:
16.1/16
IP
Subnet
CE1

PE1

Blue VPN

eBGP:
16.1/16
IP Subnet
P

VRF
ip
vrfparameters:
blue-vpn
Name
= blue-vpn
RD 1:100
RD
= 1:100 export 1:100
route-target
Import
Route-Target
= 100:1
route-target
import 1:100
Export Route-Target = 100:1

PE2

CE2

EVPN Ethernet VPN

VXLAN Evolution
ControlPlane

DataPlane

EVPN MP-BGP
draft-ietf-l2vpn-evpn

Multi-Protocol Label Switching

(MPLS)

Provider Backbone Bridges

(PBB)

Network Virtualization Overlay

(NVO)

draft-ietf-l2vpn-evpn

draft-ietf-l2vpn-pbb-evpn

draft-sd-l2vpn-evpn-overlay

EVPN over NVO Tunnels (VXLAN, NVGRE, MPLSoE) for

Data Center Fabric encapsulations
Provides Layer-2 and Layer-3 Overlays over simple IP
Networks
2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

18

Ethernet VPN
Highlights
Next generation solution for Ethernet
multipoint connectivity services
Data-plane address
learning from Access

Leverage similarities with L3VPN

PEs run Multi-Protocol BGP to advertise &

learn MAC addresses over Core
Learning on PE Access Circuits via dataplane transparent learning

Control-plane address
advertisement / learning
over Core

PE1

PE3

VID 100
SMAC: M1
DMAC: F.F.F

CE1

CE3

MPLS

No pseudowire full-mesh required

Unicast: use MP2P tunnels
Multicast: use ingress replication over MP2P tunnels
or use LSM

Under standardization at IETF draft-ietfl2vpn-evpn

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

PE2

PE4
BGP MAC adv. Route
E-VPN NLRI
MAC M1 via PE1

EVPN
Multi-Protocol BGP (MP-BGP)
based Control-Plane using EVPN
NLRI (Network Layer Reachability
Information)
Make Forwarding decisions at
VTEPs for Layer-2 (MAC) and
Layer-3 (IP)
Discovery: BGP, using MPLS
VPN mechanisms (RT)

BGP advertisement:
L2VPN/EVPN Addr = CE1.MAC
BGP Next-Hop = PE1
Route Target = 100:1
Label=42

BGP RR
CE1

PE

PE

CE4

CE2

Signaling: BGP

PE

PE

Learning: Control plane (BGP)

Emulated Virtual Switch
2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

20

CE3

Host and Subnet Route Distribution

VXLAN/EVPN
Host Route Distribution decoupled
from the Underlay protocol
RR

Use MultiProtocol-BGP (MP-BGP)

on the Leaf nodes to distribute
internal Host/Subnet Routes and
external reachability information

RR

V2

V1

Route-Reflectors deployed for

scaling purposes
RR

V3

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

21

BGP Route-Reflector
iBGP Adjacency

Agenda
VxLAN Overview
MP-BGP EVPN Basics
MP-BGP EVPN Control Plane
VXLAN Design Options
MP-BGP EVPN VXLAN Configuration

VxLAN Capability on Nexus 9000

Series Switches

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Overlay services
Layer-2
Layer-3
Layer-2 + Layer-3

Tunnel
Encapsulation

Control Plane

Underlay transport
network

Data Plane

Peer discovery mechanism

Overlay L2/L3 Unicast traffic

Route learning and distribution

mechanism

Overlay Broadcast, Unknown

(Layer-2) traffic, Multicast traffic
(BUM traffic) forwarding

Local learning
Remote learning
2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Outer
IP Header

Outer
Mac Header

VXLAN
Header

UDP Header

Original L2 Frame
FCS

FCS

8 Bytes

For next-hop transport in

the underlay network

16

16

16

Source and Destination

VTEP addresses,
allowing transport across
the underlay IP network

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

16

Reserved

32

VNID

32

Reserved

UDP Length

16

Checksu
m
0x0000

UDP
Dst Port

UDP
Src. Port

72

Outer
Dst. IP

16

Outer
Src. IP

Ether Type
0x0800

16

Header
Checksum

VLAN ID
Tag

16

Protocol
0x11

VLAN Type
0x8100

48

IP Header
Misc Data

Src.
MAC Addr.

Dst.
MAC Addr.
48

20 Bytes

VXLAN
RRRR1RRR

8 Bytes
10 or 14 Bytes

24

24

Allows for possible

The well known VXLAN
16M segments
port 4789. Indicates a
VXLAN packet
Hash of the internal L2/L3/L4 header
of the original frame. Can be used as
entropy for better ECMP/LACP load
sharing

 IP routed Network
IP Transport Network

Flexible topologies

Recommend a network with redundant paths using ECMP for load sharing
Support any routing protocols --- OSFP, EIGRP, IS-IS, BGP, etc.

Multicast is needed if using multicast for overlay BUM replication and transport

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Fabric Design

3-Tier Design

DC Spine
DC Core

DC Aggregation

DC Access

DC Leaf
DC Interconnect

Collapsed Core/Aggregation
2-Tier Design

DC-2

DC-1
DC Core/ Aggregation

DC Access
WAN
2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VXLAN terminates its tunnels on VTEPs (Virtual Tunnel End Point).

Each VTEP has two interfaces, one is to provide bridging function for local hosts, the
other has an IP identification in the core network for VXLAN
encapsulation/decapsulation.

Transport IP Network

VTEP

VTEP

IP Interface

Local LAN Segment

End System

Local LAN Segment

End System

2014 Cisco and/or its affiliates. All rights reserved.

IP Interface

End System
Cisco Public

End System

Agenda
VxLAN Overview
MP-BGP EVPN Basics
MP-BGP EVPN Control Plane
VXLAN Design Options
MP-BGP EVPN VXLAN Configuration

VxLAN Capability on Nexus 9000

Series Switches

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Flood-&-Learn

EVPN Control Plane

Overlay Services

L2+L3

L2+L3

Underlay Network

IP network with ECMP

IP network with ECMP

Encapsulation

MAC in UDP

MAC in UDP

Peer Discovery

Data-driven flood-&-learn

MP-BGP

Peer Authentication

Not available

MP-BGP

Host Route Learning

Local hosts: Data-driven flood-&-learn

Remote hosts: Data-driven flood-&-learn

Local Host: Data-driven

Remote host: MP-BGP

Host Route Distribution

No route distribution.

MP-BGP

L2/L3 Unicast Forwarding

Unicast encap

Unicast encap

BUM Traffic forwarding

Multicast replication
Unicast/Ingress replication

Multicast replication
Unicast/Ingress replication

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

MP-BGP for EVPN

MP-BGP is the routing protocol for EVPN
Multi-tenancy construct using VRF (Rout
Distinguisher, Route Targets)

New address-family l2vpn evpn for distributing

EVPN routes

evpn
vni 20000 l2
rd auto
route-target import auto
route-target export auto
router bgp 100
router-id 10.1.1.11
log-neighbor-changes
address-family ipv4 unicast
address-family l2vpn evpn
neighbor 10.1.1.1 remote-as 100
update-source loopback0
address-family ipv4 unicast
address-family l2vpn evpn
send-community extended
vrf evpn-tenant-1
address-family ipv4 unicast
advertise l2vpn evpn

EVPN routes = [MAC] + [IP]

iBGP or eBGP support

2014 Cisco and/or its affiliates. All rights reserved.

vrf context evpn-tenant-1

vni 39000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn

Cisco Public

VXLAN EVPN Control Plane Functions in Bronte Release

Host MAC/IP advertisements through MP-BGP

VTEP Peer Auto-discovery and Authentication via MP-BGP

Anycast IP gateway
ARP Suppression
Ingress Replication with Head-end Auto-discovery (planned for
Bronte+)

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

MP-BGP
for VXLAN
EVPN
Control Plane
EVPN Control
Plane
Reachability
Distribution
EVPN Control Plane -- Host and Subnet Route Distribution
BGP Update

Spine

Host-MAC
Host-IP
Internal IP Subnet
External Prefixes
VTEP

VTEP

VTEP

VTEP

Leaf

Use MP-BGP with EVPN Address Family on VTEPs to distribute internal

host MAC/IP addresses, subnet routes and external reachability information

MP-BGP enhancements to carry up to 100s of thousands of routes with

reduced convergence time

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

C-1

Install host info to RIB/FIB:

Install host info to RIB/FIB:

H-MAC-1 MAC table

H-IP-1 VRF IP host table
Host
IP

VNI

H-IP-1

VNII-1

VTEP
VTEP-1

BGP Update:
H-MAC-1
H-IP-1
VTEP-1
VNI-1

Route
Reflector

BGP Update:
H-MAC-1
H-IP-1
VTEP-1
VNI-1

H-MAC-1 MAC table

4

2
BGP Update:
H-MAC-1
H-IP-1
VTEP-1
VNI-1

VTEP-3

MAC

Host
IP

VNI

H-MAC-1

H-IP-1
VNII-1
VTEP-1
Local learning of host info:

VTEP

H-MAC-1 (MAC table)

H-IP-1 (VRF IP host table )
2014 Cisco and/or its affiliates. All rights reserved.

VTEP-1

Cisco Public

H-MAC-1
H-IP-1
VLAN-1 /VNI-1

VTEP-2

H-IP-1 VRF IP host table

MAC

Host
IP

VNI

VTEP

H-MAC-1

H-IP-1

VNII-1

VTEP-1

VXLAN
BGP
Control
Plane
EVPN Control Plane --- Host Movement
NLRI:
Host MAC1, IP1
NVE IP 1
VNI 5000
Next-Hop: VTEP-3

NLRI:
Host MAC1, IP1
NVE IP 1
VNI 5000
Next-Hop: VTEP-1

Ext. Community:
Encapsulation: VXLAN
Cost/Sequence: 1

Ext. Community:
Encapsulation: VXLAN
Cost/Sequence: 0

VTEP-1

VTEP-2

VTEP-3

VTEP-4

Host 1
MAC1
IP 1
VNI 5000

1.
2.
3.
4.

MAC

IP

VNI

Next-Hop

Encap

Seq

MAC

IP

VNI

Next-Hop

Encap

Seq

MAC-1

IP-1

5000

VTEP-1

VXLAN

MAC-1

IP-1

5000

VTEP-3

VXLAN

VTEP-1 detects Host1 and advertise an EVPN route for Host1 with seq# 0
Host1 Moves behind VTEP-3
VTEP-3 detects Host1 and advertises an EVPN route for Host1 with seq #1
VTEP-1 sees more recent route and withdraws its advertisement
2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

# VLAN to VNI mapping

vlan 200
vn-segment 5200
# Anycast Gateway MAC, identically configured on all VTEPs
fabric forwarding anycast-gateway-mac 0002.0002.0002
# Distributed IP Anycast Gateway (SVI)
# Gateway IP address needs to be identically configured on all
VTEPs
interface vlan 200
no shutdown
vrf member Tenant-A
ip address 20.0.0.1/24
fabric forwarding mode anycast-gateway

The same anycast gateway virtual IP

address and MAC address need to
be configured on all VTEPs in the
VNI

SVI
GW IP
GW MAC

SVI
GW IP
GW MAC

VTEP

Host 1
MAC1
IP 1
VLAN A
VXLAN A

2014 Cisco and/or its affiliates. All rights reserved.

Host 2
MAC2
IP 2
VLAN A
VXLAN A

Cisco Public

SVI
GW IP
GW MAC

VTEP

Host 3
MAC3
IP 3
VLAN A
VXLAN A

SVI
GW IP
GW MAC

VTEP

Host 4
MAC4
IP 4
VLAN A
VXLAN A

VTEP

ARP Suppression in MP-BGP EVPN

ARP suppression reduces network flooding due to host learning
IP Address

MAC Address

VLAN

Physical Interface
Index (ifindex)

Flags

IP-1

MAC-1

10

E1/1

Local

IP-2

MAC-2

10

Null

Remote

IP-3

MAC-3

10

Null

Remote

VTEP-1 intercepts the ARP request and checks in its

ARP suppression cache. It finds a match for IP-2 in
VLAN 10 in its ARP suppression cache.*

VTEP-1 sends an ARP response back to Host-1 with

MAC-2.*

VTEP
1

VTEP
2

Host 1
MAC1
IP 1
VLAN 10
VXLAN 5000

VTEP
3

VTEP
4

Host 1
MAC1
IP 1
VLAN 10
VXLAN 5000

Host-1 in VLAN 10 sends an

ARP request for Host-2s IP-2
address.

Host-1 learns the IP-2 and MAC-2 mapping.

* If VTEP-1 doesnt have a match for IP-2 in its ARP suppression cache table, it will flood the ARP request to all other VTEPs in this VNI
2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

38

ARP Suppression in MP-BGP EVPN (Conted)

ARP Suppression can be enabled on

a per-VNI basis under the interface
nve1 configuration.

VTEP
1

VTEP
2

VTEP
3

VTEP
4

interface nve1
no shutdown
source-interface loopback0
host-reachability protocol bgp
member vni 20000
suppress-arp
mcast-group 239.1.1.1
member vni 21000
suppress-arp
mcast-group 239.1.1.2
member vni 39000 associate-vrf
member vni 39010 associate-vrf

n9396-vtep-1.sakommu-lab.com# sh ip arp suppression topo-info

ARP L2RIB Topology information
Topo-id ARP-suppression mode
100
L2 ARP Suppression
200
L2/L3 ARP Suppression
201
L2/L3 ARP Suppression

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Head-end Replication
Head-end Replication (aka. Ingress replication):
Eliminate the need for underlay multicast to transport overlay BUM
traffic
Spine

Multicast-Free
Underlay

VTEP-1 receives the overlay BUM traffic,

encapsulates the packets into unicast VXLAN
packets, sends one copy to each remote VTEP
peer in the same VXLAN VNI

VTEP
1

2014 Cisco and/or its affiliates. All rights reserved.

VTEP
2

Host-1 sends BUM

traffic into the
VXLAN VNI

Cisco Public

40

VTEP
3

VTEP
4

Leaf

Different integrated Route/Bridge (IRB) Modes

VXLAN Routing
Overlay Networks do follow two
slightly different integrated
Route/Bridge (IRB) semantics

Routing ?

Asymmetric
Uses different path from Source to
Destination and back

Symmetric

SVI
B

SVI
A
VTEP-1

Uses same path from Source to

Destination and back

Cisco follows Symmetric IRB

2014 Cisco and/or its affiliates. All rights reserved.

IP Transport Network

Cisco Public

VTEP-2

Host 1
H-MAC-1
H-IP-1
VNI-A

VTEP-3

VTEP-4

Host 2
H-MAC-2
H-IP-2
VNI-B

Asymmetric

Routing and Bridging on the ingress VTEP

Bridging on the egress VTEP

Both source and destination VNIs need to reside on the ingress VTEP

Ingress VTEP
routes packets
from source VNI to
destination VNI. DMAC in the inner
header is the
destination host
MAC

S-IP: VTEP-1
D-IP: VTEP-4
VNI: VNI-B

S-MAC: H-MAC-1
D-MAC: H-MAC-2
S-IP: H-IP-1
D-IP: H-IP-2
VNI
A

VNI
B
VTEP-1

S-MAC: H-MAC-1
D-MAC: H-MAC-2
S-IP: H-IP-1
D-IP: H-IP-2

VNI
A
VTEP-2

VTEP-3

Host 1
H-MAC-1
H-IP-1
VNI-A

2014 Cisco and/or its affiliates. All rights reserved.

VNI
B

VTEP-4

Host 2
H-MAC-2
H-IP-2
VNI-B
Cisco Public

S-MAC: H-MAC-1
D-MAC: H-MAC-2
S-IP: H-IP-1
D-IP: H-IP-2

Egress VTEP
bridges packets in
the destination
VNI

VXLAN
BGP
Control
Plane
VTEP VNI Membership Asymmetric IRB
Every VTEP needs to be in all
VNIs
Every VTEP needs to maintain
MAC tables for all VNIs, including
those they dont have local hosts
for.

SVI 100

SVI 200
VTEP

Host 1
MAC1
IP 1
VLAN 100
VXLAN 5100

1.
2.

SVI 100

SVI 200

SVI 100

SVI 100

SVI 200

VTEP

VTEP

VTEP

Host 2
MAC2
IP 2
VLAN 100
VXLAN 5100

SVI 200

Host 3
MAC3
IP 3
VLAN 200
VXLAN 5200

All VTEPs in a VNI can be the virtual IP gateway for the local hosts
Optimized south-north bound forwarding for routed traffic without hair-pinning
2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

 Routing on both ingress and egress VTEPs

Layer-3 VNI
Tenant VPN indicator
One per tenant VRF
VTEP Router MAC
Ingress VTEP routes packets onto the Layer-3 VNI
Egress VTEP routes packets to the destination Layer-2 VNI

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

vlan 200
vn-segment 20000
vlan 201
vn-segment 20100
vlan 3900
name l3-vni-vlan-for-tenant-1
vn-segment 39000

Layer-3 VNI
(VRF VNI)

interface Vlan3900
description l3-vni-for-tenant-1-routing
no shutdown
vrf member evpn-tenant-1
ip address 39.0.0.1/16
fabric forwarding mode anycast-gateway

Layer-2 VNI
(Network VNI)

VTEP

Layer-2 VNI
(Network VNI)

VTEP

VTEP

VTEP

vrf context evpn-tenant-1

vni 39000
rd auto
address-family ipv4 unicast
route-target import 39000:39000
route-target export 39000:39000
route-target both auto evpn
interface Vlan200
no shutdown
vrf member evpn-tenant-1
ip address 20.0.0.1/24
fabric forwarding mode anycast-gateway
interface Vlan201
no shutdown
vrf member evpn-tenant-1
ip address 20.1.0.1/24
fabric forwarding mode anycast-gateway

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Ingress VTEP
routes packets
from source VNI to
L3 VNI. D-MAC in
the inner header is
the egress VTEP
router MAC

S-IP: VTEP-1
D-IP: VTEP-4
VNI: L3 VNI

VNI
A

L3
VNI

VTEP-1
Router MAC-1

S-MAC: H-MAC-1
D-MAC: H-MAC-2
S-IP: H-IP-1
D-IP: H-IP-2

S-MAC: Router-MAC-1
D-MAC: Router-MAC-4
S-IP: H-IP-1
D-IP: H-IP-2
L3
VNI
VTEP-2

VTEP-3

Host 1
H-MAC-1
H-IP-1
VNI-A

2014 Cisco and/or its affiliates. All rights reserved.

VNI
B

VTEP-4
Router MAC-4

Host 2
H-MAC-2
H-IP-2
VNI-B

Cisco Public

Egress VTEP
routes packets
from L3 VNI to the
destination
VNI/VLAN

S-MAC: H-MAC-1
D-MAC: H-MAC-2
S-IP: H-IP-1
D-IP: H-IP-2

VXLAN
BGP
Control
Plane
VTEP VNI Membership Symmetric IRB
Every VTEP only needs to be in VNIs
that it has local hosts for.

VTEPs dont need to maintain MAC

tables for VNIs that they dont have local
hosts for.
SVI
100

Host 1
MAC1
IP 1
VLAN 100
VXLAN 5100

1.
2.

VTEP

SVI
100

VTEP

VTEP

Host 2
MAC2
IP 2
VLAN 100
VXLAN 5100

VTEP

SVI
200

Host 3
MAC3
IP 3
VLAN 200
VXLAN 5200

Optimal utilization of ARP and MAC tables

A VTEP only needs to be in the VNIs which it has local hosts for.
2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Use VTEP addresses in the outer

header to route encapsulated
packets to the egress VTEP

S-IP: VTEP-1
D-IP: VTEP-2
VNI: L3-VNI-A

IP Transport
Network

S-MAC: Router-MAC-1
D-MAC: Router-MAC-2
S-IP: H-IP-1
D-IP: H-IP-2

VTEP-1

S-MAC: HMAC-1
D-MAC: HMAC-2
S-IP: H-IP-1
D-IP: H-IP-2

VTEP

S-IP: VTEP-1
D-IP: VTEP-2
VNI: L3 VNI-A

2014 Cisco and/or its affiliates. All rights reserved.

VTEP-2

VTEP
S-MAC: HMAC-1
D-MAC: HMAC-2
S-IP: H-IP-1
D-IP: H-IP-2

Host 1
H-MAC-1
H-IP-1
VNI-A
L3-VNI-A
VRF-A

Cisco Public

S-MAC: Router-MAC-1
D-MAC: Router-MAC-4
S-IP: H-IP-1
D-IP: H-IP-2

Host 2
H-MAC-2
H-IP-2
VNI-B
L3-VNI-A
VRF-A

Tenant
Tenant AA
VRF-A
VRF-A
L3-VNI-A
L3-VNI-A
H-IP-2
H-IP-2

Tenant B
VRF-B
L3-VNI-B

Use L3-VNI to identify

the tenant VRF

Tenant C
VRF-C
L3-VNI-C

 Symmetric IRB has optimal utilization of ARP and MAC tables on a VTEP
Symmetric IRB scales better for end hosts
Symmetric IRB scales better in terms of the total number of VNIs a VXLAN
overlay network can support
Multi-vendor interoperability:
Some vendors implemented Asymmetric IRB
Its been agreed upon among multiple vendors that Symmetric IRB is the
ultimate solution
Cisco implemented Symmetric IRB
Cisco will introduce backward compatability with asymmetric IRB by adding the
support for it.
2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Optimal VXLAN Routing

with Symmetric IRB and Anycast Gateway
Host-based fabric routing and bridging
with optimal and flexible VXLAN VNI placement
Every VTEP is an anycast
gateway for its VXLAN subnets.
Anycast gateway VTEPs share:
The same virtual Gateway IP
The same virtual MAC
address

Spine

SVI
100
VTEP-1

SVI
100

SVI
300
VTEP-3

VTEP-2

Port

IP-A

Eth1/1

VLAN 100

IP-B

VTEP-4

SVI VLAN 100

VNI 5100
Host IP-A

2014 Cisco and/or its affiliates. All rights reserved.

Distributed inter-vxlan
host-based routing on
local VTEP
Cisco Public

Leaf

VTEP-5

VTEP-4

Remote host route learning through

MP-BGP
Host IP

SVI
200

SVI
200

Host IP

VTEP

IP-A

VTEP-2

IP-B

Eth1/1

VLAN 200
SVI VLAN 200
VNI 5200
Host IP-B

Local Scoping of VLANs ToR Local

16 million possible VNIs global scope

VNI 5000 maps to VLAN 10

VNI 5000 maps to VLAN 60

VLANS are Locally

Scoped at Top of
Rack/ Gateway

VLANS are Locally

Scoped at Top of Rack/
Gateway

Possible VLAN IDs 1-4K

2014 Cisco and/or its affiliates. All rights reserved.

Possible VLAN IDs 1-4K

Cisco Public

51

Local Scoping of VLANs Port Local*

* Available in
Q2CY2015

16 million possible VNIs global scope

(Eth1/1, Vlan10) => VNI 10000

(Eth1/2, Vlan10) => VNI 10001
(Eth1/2, Vlan11) => VNI 10000

VNI 5000 maps to (E1/1, VLAN 10)

VNI 5000 maps to (E1/2, VLAN 60

VLANS are Locally Scoped

VLAN to VNI mapping is per-port
significant
Possible VLAN IDs 1-4K
2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VLANS are Locally Scoped

VLAN to VNI mapping is per-port
significant

Possible VLAN IDs 1-4K

52

When vPC is enabled an anycast VTEP

address is programmed on both vPC
peers

Symmetrical forwarding behavior on both

peers provides

Multicast topology prevents BUM traffic

being sent to the same IP address across
the L3 network (prevents duplication of
flooded packets)

vPC peer-gateway feature must be

enabled on both peers

VXLAN header is not carried on the vPC

Peer link (MCT link)
2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VXLAN
vPC VTEP

vPC VTEP

vPC

VLAN

interface loopback0
ip address 10.1.1.13/32
ip address 10.1.1.134/32 secondary

Underlay IP Network

BGP Router ID 1

BGP Router ID 2
vPC VTEP-1

vPC VTEP-2

Virtual
PortChannel

vPC VTEP with

Anycast VTEP
Address

Layer 2 Link
Layer 3 Link

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

EVPN Control Plane Advantages

A multi-tenant fabric solution with host-based forwarding

Industry standard protocol for multi-vendor interoperability

Build-in multi-tenancy support

Truly scalable with protocol-driven learning

Leverage MP-BGP to deliver VXLAN with L3VPN characteristics

Host MAC/IP address advertisement through EVPN MP-BGP

Fast convergence upon host movements or network failures

MP-BGP protocol driven re-learning and convergence

Upon host movement, the new VTEP will send out a BGP update to advertise
the new location of the host
2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

EVPN Control Plane Advantages (Conted)

A multi-tenant fabric solution with host-based forwarding

Optimal traffic forwarding supporting host mobility

Anycast IP gateway for optimal forwarding for host generated traffic

No need for hair-pinning to to reach the IP gateway

ARP suppression

Minimize ARP flooding in overlay

Head-end Replication with dynamically learned remote-VTEP list

Head-end replication enables multicast-free underlay network

Dynamically learned remote-VTEP list minimizes the operational overhead of

head-end replication

VTEP peer authentication via MP-BGP authentication

Added security to prevent rogue VTEPs or VTEP spoofing

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Agenda
VxLAN Overview
MP-BGP EVPN Basics
MP-BGP EVPN Control Plane
VxLAN Design Options
MP-BGP EVPN VXLAN Configuration

VxLAN Capability on Nexus 9000

Series Switches

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

L2 Link
L3 Link

DC Core

DC Aggregation

VXLAN Overlay

DC Access
VTEP

VTEP

2014 Cisco and/or its affiliates. All rights reserved.

VTEP

Cisco Public

VTEP

RR

RR

Spine

MP-iBGP Sessions

VXLAN Overlay
MP-iBGP EVPN

VTEP

VTEP

VTEP

VTEP

VTEP Functions are on leaf layer

Spine nodes are iBGP route reflector
Spine nodes dont need to be VTEP

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VTEP

VTEP

Leaf

Spine switches are not capable of

running MP-BGP EVPN.

Spine

Leaf switches are chosen to provide

iBGP route-reflector functions to the
other iBGP VTEP leaf nodes. All other
leaf nodes peer with them through
iBGP.

VXLAN Overlay

Leaf

iBGP
VTEP

iBGP
VTEP

2014 Cisco and/or its affiliates. All rights reserved.

iBGP

iBGP RR

iBGP

VTEP

VTEP

VTEP

Cisco Public

RR

Spine switches dont need to be

able to run MP-BGP EVPN. They
are purely IP transport devices.

RR
RR

Spine

Leaf

iBGP
Cisco Nexus 9300
VTEP

iBGP
Cisco Nexus 9300

VTEP

iBGP
Cisco Nexus 9300
VTEP

iBGP

iBGP
Cisco Nexus 9300
VTEP

Dedicated MP-BGP EVPN route

reflectors provide better scalability
and control-plane performance.
They can be connected to the
fabric network in the same way as
a leaf node.

Cisco Nexus 9300

VTEP

All leaf VTEPs run iBGP sessions with the

dedicated route reflectors.

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

AS 65000

Spine

BGP on Spine needs to have the

following in address-family l2vpn
evpn:
BGP next-hop unchanged
retain route-target all

MP-eBGP Sessions

VTEP

AS 65001

VTEP

AS 65002

VTEP

VTEP

AS 65003

AS 65004

VTEP Functions are on leaf layer

Spine nodes are MP-eBGP Peers
Spine nodes dont need to be VTEP

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VTEP

AS 65005

VTEP

Leaf

AS 65006

Need to manually configure Routetargets on each VTEP

[BGP configuration on a spine switch as in Figure 16 design]

route-map permit-all permit 10
set ip next-hop unchanged
router bgp 65000
router-id 10.1.1.1
address-family ipv4 unicast
redistribute direct route-map permitall
address-family l2vpn evpn
nexthop route-map permit-all
retain route-target all
neighbor 192.167.11.2 remote-as 65001
address-family ipv4 unicast
address-family l2vpn evpn
send-community extended
route-map permit-all out
neighbor 192.168.12.2 remote-as 65002
address-family ipv4 unicast
address-family l2vpn evpn
send-community extended
route-map permit-all out
2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Set next-hop policy to not change

the next-hop attributes.
Retain routes with all route targets
when advertising the EVPN BGP
routes to eBGP peers.
Set outbound policy to advertise all
routes to this eBGP neighbor.

AS 65000

Spine

BGP on Spine needs to have the

following in address-family l2vpn
evpn:
BGP next-hop unchanged
retain route-target all
MP-eBGP Sessions

VTEP

AS 65100

VTEP

AS 65100

VTEP

VTEP

AS 65100

AS 65100

VTEP

AS 65100

VTEP

AS 65100

VTEP leafs are in the

same BGP AS

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Leaf

EVPN VXLAN Fabric External Routing

VXLAN Overlay
RR

RR

EVPN VRF/VRFs Space

Spine

VXLAN Overlay
EVPN MP-BGP
VTEP

VTEP

VTEP

VTEP

Border Leaf
VTEP

VTEP

Leaf

Routing
Protocol
of
Choice

Global Default VRF

IP Routing

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Or User Space VRFs

EVPN VXLAN Fabric External Routing (Conted)

Border Leaf

For Layer 3 interfaces, use

one per overlay VRF
instance. The routing
protocol neighbor is in the
EVPN VRF address family.
Layer 3 interfaces on the
external devices can be in
either tenant VRF
instances or the global
default VRF instance.

Overlay
EVPN
VRF A

Overlay
EVPN
VRF B

Overlay
EVPN
VRF C

VRF OSPF Process

VRFA

VRFB

VRFC

Tenant VRF or Default VRF

External Router

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Interface-Type Options:
Physical Routed Ports
Subinterfaces
VLAN SVIs over Trunk Ports

RR

RR

Spine

VXLAN Overlay
VTE
P

Border Leaf

EVPN
VRF VTE
Instance VTE
Space
VTE
P

Router bgp 100

vrf evpn-tenant-1
address-family ipv4 unicast
network 20.0.0.0/24
neighbor 30.10.1.2 remote-as 200
address-family ipv4 unicast
prefix-list outbound-no-hosts out

VTEP

interface Ethernet2/9.10
mtu 9216
encapsulation dot1q 10
vrf member evpn-tenant-1
ip address 30.10.1.1/30

interface Ethernet1/50.10
mtu 9216
encapsulation dot1q 10
ip address 30.10.1.2/30
IP Routing in the Default
VRF Instance

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

router bgp 200

address-family ipv4 unicast
network 100.0.0.0/24
network 100.0.1.0/24
neighbor 30.10.1.1 remote-as 100
address-family ipv4 unicast

On the VXLAN Border Leaf

router bgp 100
router-id 10.1.1.16
log-neighbor-changes
address-family ipv4 unicast
address-family l2vpn evpn
neighbor 10.1.1.1 remote-as 100
update-source loopback0
address-family ipv4 unicast
address-family l2vpn evpn
send-community extended
neighbor 10.1.1.2 remote-as 100
update-source loopback0
address-family ipv4 unicast
address-family l2vpn evpn
send-community extended
vrf evpn-tenant-1
address-family ipv4 unicast
network 20.0.0.0/24
neighbor 30.10.1.2 remote-as 200
address-family ipv4 unicast
prefix-list outbound-no-hosts out

The eBGP neighbor is on the outside.

Its in address-family ipv4 unicast of the tenant VRF routing
instance.

For better scalability, apply prefix-list to filter out /32 IP host routes.
Advertise prefix routes only to the external eBGP neighbor.

ip prefix-list outbound-no-hosts seq 5 deny 0.0.0.0/0 eq 32

ip prefix-list outbound-no-hosts seq 10 permit 0.0.0.0/0 le 32

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

This is the external route.

n9396-vtep-1# sh ip bgp vrf evpn-tenant-1 100.0.0.0

BGP routing table information for VRF evpn-tenant-1, address family IPv4 Unicast
BGP routing table entry for 100.0.0.0/24, version 70
Paths: (1 available, best #1)
The next hop
Flags: (0x08041a) on xmit-list, is in urib, is best urib route
vpn: version 75, (0x100002) on xmit-list

is the VTEP address of the border lea

Advertised path-id 1, VPN AF advertised path-id 1

Path type: internal, path is valid, is best path, no labeled nexthop
Imported from unknown dest
The tenant is VRF L3 VNI.
AS-Path: NONE, path sourced internal to AS
10.1.1.16 (metric 3) from 10.1.1.1 (10.1.1.1)
Origin IGP, MED not set, localpref 100, weight 0
10.1.1.16 is the BGP router ID of the border
Received label 39000
leaf. 10.1.1.1 is the spine route reflector.
Extcommunity: RT:100:39000 ENCAP:8 Router MAC:6412.2574.6ae7
Originator: 10.1.1.16 Cluster list: 10.1.1.1
VRF advertise information:
Path-id 1 not advertised to any peer
VPN AF advertise information:
Path-id 1 not advertised to any peer
n9396-vtep-1#
n9396-vtep-1# sh ip route vrf evpn-tenant-1 100.0.0.0/24
IP Route Table for VRF "evpn-tenant-1"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in
via
output
denotes
VRF
<string>
2014
Cisco
and/or its affiliates.
All rights
reserved.
Cisco Public

This is the iBGP route. The next hop is the VTEP

address of the border leaf.

RR

RR

Spine

VXLAN Overlay
VTE
P

EVPN VRF and VRF

VTE Instance
VTE Space VTE
P

VTEP

interface Ethernet1/50.10
mtu 9216
encapsulation dot1q 10
ip address 30.10.1.2/30
ip router ospf 1 area 0.0.0.0

route-map permit-bgp-ospf permit 10

route-map permit-ospf-bgp permit 10
router ospf 1
router-id 10.1.1.16
vrf evpn-tenant-1
redistribute bgp 100 route-map permit-bgp-ospf
router bgp 100
router-id 10.1.1.16
log-neighbor-changes
address-family ipv4 unicast
address-family l2vpn evpn
retain route-target all
vrf evpn-tenant-1
address-family ipv4 unicast
advertise l2vpn evpn
redistribute ospf 1 route-map permit-ospf-bgp
2014 Cisco and/or its affiliates. All rights reserved.

Border Leaf

interface Ethernet2/9.10
mtu 9216
encapsulation dot1q 10
vrf member evpn-tenant-1
ip address 30.10.1.1/30
ip router ospf 1 area 0.0.0.0

IP Routing in the Default

VRF Instance

Cisco Public

ip prefix-list bgp-ospf-no-hosts seq 5 permit 0.0.0.0/0 eq 32

route-map permit-bgp-ospf deny 5
match ip address prefix-list bgp-ospf-no-hosts
route-map permit-bgp-ospf permit 10
Redistribute BGP
route-map permit-ospf-bgp permit 10

routes to OSPF. Filter

out /32 IP host routes.

router ospf 1
router-id 10.1.1.16
vrf evpn-tenant-1
redistribute bgp 100 route-map permit-bgp-ospf
router bgp 100
router-id 10.1.1.16
log-neighbor-changes
address-family ipv4 unicast
address-family l2vpn evpn
retain route-target all
neighbor 10.1.1.1 remote-as 100
update-source loopback0
address-family ipv4 unicast
address-family l2vpn evpn
send-community extended
neighbor 10.1.1.2 remote-as 100
update-source loopback0
address-family ipv4 unicast
address-family
2014 Cisco and/or its l2vpn
affiliates. Allevpn
rights reserved.

A BGP router will modify route targets in l2vpn

evpn routes when it is an autonomous system
boundary router. The original route target must
be retained.

Redistribute OSPF to BGP. Advertise

the redistributed routes to L2VPN
EVPN.

Cisco Public

n9396-vtep-1# sh vrf evpn-tenant-1 detail

VRF-Name: evpn-tenant-1, VRF-ID: 3, State: Up
VPNID: unknown
RD: 10.1.1.11:3
VNI: 39000
Max Routes: 0 Mid-Threshold: 0
Table-ID: 0x80000003, AF: IPv6, Fwd-ID: 0x80000003, State: Up
Table-ID: 0x00000003, AF: IPv4, Fwd-ID: 0x00000003, State: Up

The external route learned through MPBGP EVPN is imported into the tenant
VRF.

n9396-vtep-1# sh bgp l2vpn evpn rd 10.1.1.11:3 100.0.0.0

BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 10.1.1.11:3
(L3VNI 39000)
BGP routing table entry for [5]:[0]:[0]:[24]:[100.0.0.0]:[0.0.0.0]/224, version 396
Paths: (1 available, best #1)
The next hop is the
Flags: (0x00001a) on xmit-list, is in l2rib/evpn

VTEP address of

the border leaf.

Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Imported from 10.1.1.16:3:[5]:[0]:[0]:[24]:[100.0.0.0]:[0.0.0.0]/120
AS-Path: NONE, path sourced internal to AS
This is the Layer 3 VNI
10.1.1.16 (metric 3) from 10.1.1.1 (10.1.1.1)
VRF routing instance.
Origin IGP, MED not set, localpref 100, weight 0
Received label 39000
Extcommunity: RT:100:39000 ENCAP:8 Router MAC:6412.2574.6ae7
Originator: 10.1.1.16 Cluster list: 10.1.1.1
2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

of the tenant

Alternative Design for EVPN VXLAN External Routing

EVPN MP-BGP

IP Routing

VXLAN Overlay

IP Routing MP-BGP

Routing Protocol of
Choice

EVPN VRF/VRFs
Space

Default VRF in
IP Routing

Spine/Aggr.

VXLAN Overlay
EVPN MP-BGP

Border Leaf
RR

VTEP

VTEP

VTEP

VTEP

VTEP

Overlay
EVPN
VRFs

VTEP

Default
VRF

Leaf
Global Default VRF
Or User Space VRFs

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Agenda
VxLAN Overview
MP-BGP EVPN Basics
MP-BGP EVPN Control Plane
VXLAN Design Options
MP-BGP EVPN VXLAN Configuration

VxLAN Capability on Nexus 9000

Series Switches

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Tenant A (VRF A)
SVI
A

SVI
B

SVI
N

VLAN A

VLAN B

VLAN N

Layer-2 VNI B

Layer-2 VNI N

Layer-2 VNI A

SVI
X

Layer-3 VNI X

VLAN X

One VLAN maps to one Layer-2 VNI Layer-2 VNI per

Layer-2 segment

1 Layer-3 VNI per

Tenant (VRF) for routing

A Tenant can have multiple VLANs, therefore multiple

Layer-2 VNIs

VNI X is used for routed

packets

Traffic within one Layer-2 VNI is bridged

Traffic between Layer-2 VINs is routed

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Initial configuration Per Switch

Enable VXLAN and MP-BGP EVPN Control Plane
feature nv overlay
feature vn-segment-vlan-based
feature bgp
nv overlay evpn

Enable VXLAN
Enable VLAN-based VXLAN (the currently only
mode)
Enable BGP

Enable EVPN control plane for VXLAN

Other features may need to be anabled

feature ospf
feature pim
feature interface-vlan

Enable OSPF if its chosen to be the underlay

IGP routing protocol

Enable IP PIM multicast routing in the

underlay network
Enable VLAN SVI interfaces if the VTEP
needs to be IP gateway and route for the
VXLAN VLAN IP subnet.

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

EVPN Tenant VRF

Create VXLAN tenant VRF

Create a VXLAN Tenant VRF

vrf context evpn-tenant-1

vni 39000
rd auto
address-family ipv4 unicast
route-target import 39000:39000
route-target export 39000:39000
route-target both auto evpn

vrf context evpn-tenant-2

vni 39010
rd auto
address-family ipv4 unicast
route-target import 39010:39010
route-target export 39010:39010
route-target both auto evpn
2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Specify the Layer-3 VNI for VXLAN routing

within the tenant VRF

Define VRF RD (route distinguisher)

Define VRF Route Target and import/export

policies in address-family ipv4 unicast

Example to create a 2nd tenant VRF following

the above steps

Layer-3 VNI Per Tenant for EVPN Routing

Configure Layer-3 VNI per EVPN Tenant VRF Routing Instant
vlan 3900
name l3-vni-vlan-for-tenant-1
vn-segment 39000
interface Vlan3900
description l3-vni-for-tenant-1-routing
no shutdown
vrf member evpn-tenant-1
vrf context evpn-tenant-1
vni 39000
rd auto
address-family ipv4 unicast
route-target import 39000:39000
route-target export 39000:39000
route-target both auto evpn

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Create the VLAN for the Layer-3 VNI.

One Layer-3 VNI per tenant VRF routing
instance

Create the SVI interface for the Layer-3 VNI

Put this SVI interface into the tenant VRF context

Associate the Layer-3 VNI with the tenant VRF

routing instance.

EVPN Layer-3 VNI Per Tenant for Routing Instance

Configure Layer-3 VNI per EVPN Tenant VRF Routing Instant
vlan 3901
name l3-vni-vlan-for-tenant-2
vn-segment 39010
interface Vlan3901
description l3-vni-for-tenant-2-routing
no shutdown
vrf member evpn-tenant-2
vrf context evpn-tenant-2
vni 39010
rd auto
address-family ipv4 unicast
route-target import 39010:39010
route-target export 39010:39010
route-target both auto evpn

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Define Layer-3 VNI for a 2nd tenant following

the same steps in the previous slide

EVPN Layer-2 Network VXLAN VNI

Map VLANs to VXLAN VNIs and Configure their MP-BGP EVPN Parameters
vlan 200
vn-segment 20000
vlan 210
vn-segment 21000
evpn
vni 20000 l2
rd auto
route-target
route-target
vni 21000 l2
rd auto
route-target
route-target

Map VLAN to VXLAN VNI

import auto
export auto

import auto
export auto

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Under EVPN configuration, define RD and

RT import/export policies for each Layer-2
VNIs

EVPN Layer-2 Network VXLAN VLAN SVI Interface

Create SVI interface for Layer-2 VNIs for VXLAN routing

interface Vlan200
no shutdown
vrf member evpn-tenant-1
ip address 20.1.1.1/8
fabric forwarding mode anycast-gateway

interface Vlan210
no shutdown
vrf member evpn-tenant-1
ip address 21.1.1.1/8
fabric forwarding mode anycast-gateway

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Create SVI interface for a Layer-2 VNI.

Associate it with the tenant VRF.
All VTEPs for this VLAN/VNI should have the
same SVI interface IP address as the
distributed IP gateway.
Enable distributed anycast gateway for this
VLAN/VNI

EVPN Distributed Gateway

Configure distributed gateway virtual MAC address
One virtual MAC per VTEP
All VTEPs should have the same virtual MAC
address

fabric forwarding anycast-gateway-mac 0002.0002.0002

interface Vlan210
no shutdown
vrf member evpn-tenant-2
ip address 21.1.1.1/8
fabric forwarding mode anycast-gateway

Configure virtual IP address

All VTEPs for this VLAN should have the same
virtual IP address

Enable distributed gateway for this VLAN

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VXLAN Tunnel Interface Configuration

Configure VXLAN tunnel interface nve1
interface nve1
no shutdown
source-interface loopback0
host-reachability protocol bgp
member vni 20000
suppress-arp
mcast-group 239.1.1.1
member vni 21000
suppress-arp
mcast-group 239.1.1.2
member vni 39000 associate-vrf
member vni 39010 associate-vrf
interface loopback 0
ip address
10.1.1.11/32
2014 Cisco and/or its
affiliates. All rights reserved.

Cisco Public

Specify loopback0 as the source interface

Define BGP as the mechanism for host
reachability advertisement
Associate tenant VNIs to the tunnel interface
nve1
Define the mcast group on a per-VNI basis
Enable arp suppression on a per-VNI basis

Add Layer-3 VNIs, one per tenant VRF

MP-BGP Configuration on VTEP

router bgp 100
router-id 10.1.1.11
log-neighbor-changes
address-family ipv4 unicast
address-family l2vpn evpn
neighbor 10.1.1.1 remote-as 100
update-source loopback0
address-family ipv4 unicast
address-family l2vpn evpn
send-community extended
neighbor 10.1.1.2 remote-as 100
update-source loopback0
address-family ipv4 unicast
address-family l2vpn evpn
send-community extended
vrf evpn-tenant-1
address-family ipv4 unicast
advertise l2vpn evpn
vrf evpn-tenant-2
address-family ipv4 unicast
advertise l2vpn evpn
2014 Cisco and/or its affiliates. All rights reserved.

Address-family ipv4 unicast for prefix-based

routing
Address-family l2vpn evpn for evpn host
routes
Define MP-BGP neighbors.
Under each neighbor define address-family
ipv4 unicast and l2vpn evpn

Send extended community in l2vpn evpn

address-family to distribute EVPN route
attributes
Under address-family ipv4 unicast of each
tenant VRF instance, enable advertising EVPN
routes

Cisco Public

MP-BGP Configuration on iBGP Route Reflectore

router bgp 100
router-id 10.1.1.1
log-neighbor-changes
address-family ipv4 unicast
address-family l2vpn evpn
retain route-target all
template peer vtep-peer
remote-as 100
update-source loopback0
address-family ipv4 unicast
send-community both
route-reflector-client
address-family l2vpn evpn
send-community both
route-reflector-client
neighbor 10.1.1.11
inherit peer vtep-peer
neighbor 10.1.1.12
inherit peer vtep-peer
neighbor 10.1.1.13
inherit peer vtep-peer
neighbor 10.1.1.14
inherit
vtep-peer
2014 Cisco peer
and/or its affiliates.
All rights reserved.
Cisco Public

Address-family ipv4 unicast for prefix-based

routing
Address-family l2vpn evpn for EVPN vxlan
host routes
Retain route-targets attributes

iBGP RR client peer template

Send both standard and extended community
in address-family ipv4 unicast
Send both standard and extended community
in address-family l2vpn evpn

Agenda
VxLAN Overview
MP-BGP EVPN Basics
MP-BGP EVPN Control Plane
VXLAN Design Options
MP-BGP EVPN VXLAN Configuration

VxLAN Capability on Nexus 9000

Series Switches

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VXLAN is supported across the Nexus 9000 series platforms. The VXLAN
Gateway functionality is supported across all form factors and line cards.
Integrated routing functionality is supported on Nexus 9300 switches and
ACI-enabled Modules for Nexus 9500 switches.

Nexus 9300 Series

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Nexus 9500 Series

VXLAN /VLAN

VXLAN Encapsulation and De-encapsulation

occur on T2

Bridging and Gateway are independent of the

port type (1/10/40G ports)

Encapsulation happens on the egress port

Decapsulation happens on the ingress port

ALE
(NorthStar)

Encap/Decap
(NFE)

VXLAN /VLAN

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VXLAN Routing is not supported currently on Broadcom

Additional recirculation required for VXLAN routing through NS

Insieme
Recirculate

Route
Packet

Insieme
Recirculate

Route
Packet

Encap/De
cap

VLAN VXLAN
Subnet 10.20.20.0/24 Subnet 10.10.10.0/24
2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VLAN
Subnet 10.20.20.0/24

Encap/De
cap

VXLAN
Subnet 10.10.10.0/24

VXLAN Scales on Nexus 9000 Series Switches

Scale Parameter

Bronte Target

VxLAN enabled VLANs

1000

VxLAN enabled VRFs

900

VxLAN SVIs

1000

Total VNIs (L2/L3)

1900 (1000/900)

ECMP paths

64

Local VTEPs

Remote VTEPs

255

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public