Building Data Center

Networks with VXLAN

EVPN Overlays

Lukas Krattiger, Principal Engineer

BRKDCT-3378
Cisco Spark
Questions?
Use Cisco Spark to chat with the
speaker after the session

How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space

Cisco Spark spaces will be cs.co/ciscolivebot#BRKDCN-3378

available until July 3, 2017.

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Session Reference
• BRKDCN-2304
• L4-L7 Service Integration in Multi-Tenant VXLAN EVPN Data Center Fabrics
• BRKDCN-2035
• VXLAN BGP EVPN based Multi-Pod, Multi-Fabric, Multi-Site
• BRKDCN-2125
• Overlay Management and Visibility with VXLAN
• Thursday, Jun 29, 10:30 am

• BRKDCN-2342
• Programmable Fabric Automation and Management with DCNM 10
• Thursday, Jun 29, 1:00 p.m.

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Agenda
• Introduction to Data Center Fabrics
• Leaf, Spine, Super-Spine (CLOS)
• Overlay
• Underlay

• VXLAN with BGP EVPN

• Control & Data Plane
• Multi-Tenancy
• Distributed Anycast Gateway
• VPC
• DCI

• A Deployment Story … for your reference

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Introduction to Data Center Fabrics
 Data Center “Fabric” Journey (Standalone)
Layer-3 HSRP HSRP

Layer-2

Spanning-Tree

Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Layer-2

Baremet al Hypervisor Hypervisor Hypervisor Baremet al Hypervisor Baremet al Baremet al Hypervisor Hypervisor

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
 VXLAN Evolves as the Control Plane Evolves!
Back Then
Yet Another Encapsulation
 Flood & Learn (Multicast-based)
 Data-Plane only Yesterday
VXLAN for the Data Center – Intra-DC
 Control-Plane
 Active VTEP Discovery
 Multicast and Unicast

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Agenda
• Introduction to Data Center Fabrics
• Leaf, Spine, Super-Spine (CLOS)
• Overlay
• Underlay

• VXLAN with BGP EVPN

• Control & Data Plane
• Multi-Tenancy
• Distributed Anycast Gateway
• VPC
• DCI

• A Deployment Story … for your reference

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
 The Leaf / Spine Topology (Clos* Network)

Spine Spine Spine Spine

• Wide ECMP: Unicast or Multicast

• Uniform Reachability
• Deterministic Latency Leaf Leaf Leaf Leaf Leaf Leaf Leaf

• High Redundancy
• On Node or Link Failure

*Clos, Charles (1953) "A study of non-blocking switching networks" BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
 A Scale Out Architecture

More Spine – More Bandwidth – More Resiliency

• Leaf Spine Spine Spine Spine

• Smallest Operational Entity

• Spines
• Wide vs. Big
Leaf Leaf Leaf Leaf Leaf Leaf Leaf

• Uplinks
• Symmetric to all Spines or Pods
More Leaf – More Ports – More Capacity
• SAYG: Scale as You Grow

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
The Super-Spine
SuperSpine

SuperSpine SuperSpine

Spine Spine Spine Spine Spine Spine Spine Spine

Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf

POD 1 POD 2

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
 The Super-Spine
SuperSpine

• Scale Out
SuperSpine SuperSpine

• Not Limited to Port Density

• Simpler Capacity Planning

• Beyond a single Server Room

• Allows Interconnecting Pods
Spine Spine Spine Spine Spine Spine Spine Spine

• Retains Intra-Pod Topology with

Flexible Inter-Pod Connectivity
Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf

POD 1 POD 2

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
 Data Center Fabric Properties

Spine Spine Spine Spine

• Any Subnet, Anywhere, Rapidly

• Any Network on Any Leaf

• Reduced Failure Domain

Leaf Leaf Leaf Leaf Leaf Leaf Leaf
• Any Default Gateway on Any Leaf
- Distributed

• Extensible Scale and Resiliency

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
 Overlay Based Data Center: Fabrics

• Mobility
• Segmentation
Spine Spine Spine Spine

Overlay • Scale

VTEP VTEP VTEP VTEP VTEP VTEP VTEP

• Automated and Programmable
• Abstracted Consumption Model
• Layer-2 and Layer-3 Service
• Physical and Virtual Workloads

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
 Overlay Based Data Center: Edge Devices
Network Overlays Host Overlays

Overlay Overlay
VTEP VTEP VTEP VTEP - - - -

Hybrid Overlays
VTEP VTEP VTEP VTEP
Baremetal Baremetal Baremetal Baremetal Hypervisor Hypervisor Hypervisor Hypervisor

• Router/Switch End-Points
• Virtual End-Points only
• Protocols for Resiliency/Loops Overlay • Single Admin Domain
• Traditional VPNs
• VXLAN, NVGRE, STT
• VXLAN, OTV, VPLS, LISP, FP - - VTEP VTEP

VTEP VTEP
Hypervisor Hypervisor Baremetal Baremetal

• Physical and Virtual

• Resiliency and Scale
• Cross-Organizations/Federation
• Open Standards
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Agenda
• Introduction to Data Center Fabrics
• Leaf, Spine, Super-Spine (CLOS)
• Overlay
• Underlay

• VXLAN with BGP EVPN

• Control & Data Plane
• Multi-Tenancy
• Distributed Anycast Gateway
• VPC
• DCI

• A Deployment Story … for your reference

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Overlay Taxonomy - Underlay

Layer-3
Interface Spine Spine Spine Spine

Peering

Underlay
Edge Device Leaf Leaf Leaf Leaf Leaf Leaf Leaf

LAN
Segment

Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal

Virtual
Server Physical
Server

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Overlay Taxonomy - Overlay

Tunnel Encapsulation
Spine
(VNI Namespace)
Spine Spine Spine

Overlay
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

LAN
Segment

Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal

Virtual
Server Physical VTEP: VXLAN Tunnel End-Point
Server VNI/VNID: VXLAN Network Identifier
BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Understanding Overlay Technologies
Overlay Services
• Layer-2 Underlay Transport
Tunnel Encapsulation
• Layer-3 Network
• Layer-2 and Layer-3

Data-Plane
Control-Plane • Overlay Layer-2/Layer-3 Unicast Traffic
• Peer-Discovery • Overlay Broadcast, Unknown Unicast,
• Route Learning and Distribution Multicast traffic (BUM traffic)
• Local Learning forwarding
• Remote Learning • Ingress Replication (Unicast)
• Multicast

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Agenda
• Introduction to Data Center Fabrics
• Leaf, Spine, Super-Spine (CLOS)
• Overlay
• Underlay

• VXLAN with BGP EVPN

• Control & Data Plane
• Multi-Tenancy
• Distributed Anycast Gateway
• VPC
• DCI

• A Deployment Story … for your reference

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
MTU and Overlays
• Data Center often require Jumbo
MTU
• Most Server NIC support up to
9000 Bytes

• Network Switches support MTU

up to 9216* Bytes
• Accommodates Jumbo MTU plus
Overlay overhead

• Avoid Fragmentation
• Adjust the Transport Network with
appropriate MTU

*Cisco Nexus 5600 only supports a MTU of 9192 Byte for Layer-3 Traffic BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
 Interface Principles

• Routed Ports and Interfaces

• Layer-3 Interfaces between Leaf Spine Spine Spine Spine

and Spine(no switchport)

• For each Point-2-Point (P2P) Underlay
connection, minimum /31
required Leaf Leaf Leaf Leaf Leaf Leaf Leaf

• Alternative, use IP Unnumbered

(/32)

• Use Loopback as Source-

Interface for VTEP (NVE*)

*NVE: Network Virtualization Edge BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
 IP Addressing Principles Routing
Identifier Rendezvous
• Prepare a IP Addressing Plan p2p* Links / IP Point
Unnumbered
• Separate Interface functions Spine Spine Spine Spine

through IP Addressing
(Aggregates) Underlay
• Unicast Routing – Routing
Protocol Peering (p2p*) Leaf Leaf Leaf Leaf Leaf Leaf Leaf

• Unicast Routing – Routing

Identifier (RID)
Routing
• VTEP and VPC VTEP
Identifier
• Multicast Rendezvous-Point (RP) Loopback

p2p Agg: 10.1.1.0/24

• IPv4 only (today) RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24
RP Agg: 10.254.254.0/24
*p2p: Point-to-Point BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
 IP Addressing Principles
interface loopback254
description RP
interface ethernet4/4 ip address 10.254.254.1/32
description p2p-to-Leaf interface loopback0
ip address 10.1.1.2/30 description RID
Spine Spine Spine Spine ip address 10.10.10.201/32

interface ethernet1/49
description p2p-to-Spine
ip address 10.1.1.1/30 Underlay
Leaf Leaf Leaf Leaf Leaf Leaf Leaf

interface loopback0
description RID
ip address 10.10.10.101/32 interface loopback1
Hypervisor Baremetal description VTEP
Hypervisor Hypervisor Baremetal Hypervisor Baremetal
p2p Agg: 10.1.1.0/24
Baremetal

ip address 10.200.200.101/32
RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24
RP Agg: 10.254.254.0/24
BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
 Some Math – IP Addressing Principles (P2P)

• Example from depicted Topology

• 4 Spine * 7 Leaf (28 Links) Spine Spine Spine Spine

• 11 Router ID (RID Loopback)

• 7 VTEP (Loopback) Underlay
• 28 Link * 2 (/31) = 56 IP Addresses
Leaf Leaf Leaf Leaf Leaf Leaf Leaf

• 11 Router ID (RID) = 11 IP Addresses

• 7 VTEP = 7 IP Addresses

• Total: 74 IP Addresses Required

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
 Simplifying the Math – IP Unnumbered

• Example from depicted Topology

• 4 Spine + 7 Leaf (11 Loopback) Spine Spine Spine Spine

• 11 Router ID (RID Loopback)

• 7 VTEP (Loopback) Underlay
• 11 Unnumbered IF = 11 IP Addresses
Leaf Leaf Leaf Leaf Leaf Leaf Leaf

• 11 Router ID (RID) = 11 IP Addresses

• 7 VTEP = 7 IP Addresses

• Total: 29 IP Addresses Required

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Unicast Routing – OSPF and IS-IS

• OSPF – watch your Network Type • IS-IS – what was this CLNS?
• Independent of IP (CLNS)
• Network Type Point-2-Point • Well suited for routed
• Preferred (only LSA type-1) interfaces/ports
• No DR/BDR election • No SPF calculation on Link change;
• Suits well for routed interfaces/ports only if Topology changes
(optimal from a LSA DB perspective) • Fast Re-convergence
• Full SPF calculation on Link Change • Not everyone is familiar with it

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
Unicast Routing – BGP

• eBGP Underlay Routing –

Service Provider style
• Two Different Models
• Two-AS
• Multi-AS

• BGP is a Distance Vector

Protocol
• actually Path Vector Protocol
• AS* are used to calculate the
Path (AS_Path)

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
 Unicast Routing – eBGP Two-AS Model

All-Spine AS#65500
• eBGP Two-AS, yes it works!
Spine Spine Spine Spine

• eBGP peering for Underlay

Underlay • Spine is not a Route-Reflector
(eBGP) – Retain Route-Targets
VTEP VTEP VTEP VTEP VTEP VTEP VTEP
• Disable BGP AS-Path check
• Next-Hop needs to be
Unchanged
All-Leaf AS#65501
• Underlay is Reachability!
• Advertise your Loopbacks

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
 Unicast Routing – eBGP Multi-AS Model
• eBGP Two-AS, yes it works!
All-Spine AS#65500
Spine Spine Spine Spine
• eBGP peering for Underlay
• Spine is not a Route-Reflector
(eBGP) – Retain Route-Targets
Underlay • Disable BGP AS-Path check
• Next-Hop needs to be
VTEP VTEP VTEP VTEP VTEP VTEP VTEP
Unchanged

• Underlay is Reachability!
• Advertise your Loopbacks

• Changes Overlay Routing Policy

• Manually define Route-Targets

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
 Unicast Routing – eBGP Model

Spine Spine Spine Spine • Two different BGP Peering

• eBGP peering for Underlay
Underlay • Global IPv4/v6 Address-Family
• Use Physical Interface IP
VTEP VTEP VTEP VTEP VTEP VTEP VTEP

• eBGP peering for Overlay

• Global EVPN Address-Family
• Use Loopback Interface IP
• BFD not so ok

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
 Unicast Routing – Why two different BGP Peering?
1) Interface Down – BGP Down
BGP Peering
Spine (IPv4/IPv6)
BGP Peering
(IPv4/IPv6)

Leaf Leaf
AS#65500

Spine

BGP Peering BGP Peering

(IPv4/IPv6) (IPv4/IPv6)

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
 Unicast Routing – Why two different BGP Peering?
1) Interface Down – BGP Down
BGP Peering
Spine • Point-2-Point
(IPv4/IPv6) Link Fails
BGP Peering
(IPv4/IPv6)
• BGP Peering is teared down
• Lights-Out Event or BFD
Leaf
• Fast reaction
Leaf
to Routing Table
AS#65500
• Underlay Network Converges
• ECMP kicks in if
available/configured
Spine

BGP Peering • IGPsBGP

doPeering
this Automatically
(IPv4/IPv6) (IPv4/IPv6)

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
 Unicast Routing – Why two different BGP Peering?
1) Interface Down – BGP Down

Spine
BGP Peering
2) Interface Down – BGP Not Down (EVPN)

Leaf Leaf
AS#65500

Spine

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
 Unicast Routing – Why two different BGP Peering?
1) Interface Down – BGP Down
• Point-2-Point
BGP Peering
Link Fails
Spine
2) Interface Down – BGP Not Down (EVPN)
• Loopback to Loopback Peering
remains Up
• If Alternate Path available
Leaf • Timers should
Leaf allow Time for
AS#65500 Network Re-Convergence
• No BFD

Spine
• Unchanged Overlay Reachability
• No Mass Delete/Re-Learn
• Underlay Path change only

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
 Unicast Routing – Why two different BGP Peering?
1) Interface Down – BGP Down

Spine
2) Interface Down – BGP Not Down

3) Leaf Down – Prefix are Withdrawn (RNH*)

Leaf Leaf
AS#65500

5 192.168.10.0/24
Spine

Next-Hop:
2 0000.3001.1101
10.200.200.102

2 0000.3001.1101, 192.168.10.101

*RNH: Recursive Next-Hop BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
 Unicast Routing – Why two different BGP Peering?
1) Interface Down – BGP Down

Spine
2) Interface Down – BGP Not Down

3) Leaf Down – Prefix are Withdrawn (RNH*)

Leaf Leaf
AS#65500

5 192.168.10.0/24
Spine

Next-Hop:
2 0000.3001.1101
10.200.200.102

2 0000.3001.1101, 192.168.10.101

*RNH: Recursive Next-Hop BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
 Unicast Routing – Why two different BGP Peering?
1) Interface Down – BGP Down
• Leaf and p2p Interfaces Fail
2) Interface Down – BGP Not Down
Spine
• Either IGP or BGP converges

• Loopback to Loopback Peering

3) Leaf Down – Prefix are Withdrawn (RNH*) remains Up
Leaf • BGP Dead-Timer
Leaf (180s)
AS#65500
• Recursive Next-Hop will trigger
Convergence Event
5 192.168.10.0/24
Spine
• Next-Hop (VTEP) disappeared in
Next-Hop:
Underlay
2 0000.3001.1101
10.200.200.102 • Overlay withdraws Prefixes
2 0000.3001.1101, 192.168.10.101

*RNH: Recursive Next-Hop BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
Underlay - Unicast Routing and Overlay
IGP + BGP for
• Generic Concept for Underlay / true Protocol
Overlay Separation Separation
• Use Different Routing Protocol BGP for single
• Use Same Routing Protocol Routing Protocol
Specific to BGP approach
as a Overlay • RNH* for Overlay works with ALL
Control-Protocol Underlay Routing Protocols
• Ensure /32-Reachability for
BGP Knobs can VTEPs
Help here • Other Routes can impact
(Aggregates, Default-Route)

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
 Underlay - Multicast Routing and Rendezvous-Point

• PIM Any-Source-Multicast (ASM) • Bidirectional PIM (Bidir)

• Platform Support • Platform Support
• Nexus 9000 / Nexus 7000 (F3/M3) • Nexus 5600 / Nexus 7000 (F3/M3)
• ASR 1000 / ASR 9000 • ASR 1000 / ASR 9000

• RP Redundancy • RP Redundancy
• PIM Anycast-RP or MSDP • Phantom-RP

• Source-Trees (Unidirectional) • Shared-Trees (Bidirectional)

• 1 Source Tree per VTEP per • 1 Shared Tree per Multicast Group
Multicast Group • Follows Unicast Routing Path

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Underlay – PIM ASM with PIM Anycast-RP

RP RP
Spine Spine Spine Spine

S,G S,G
S,G S,G
Underlay S,G

Leaf Leaf Leaf Leaf Leaf Leaf Leaf

Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
 PIM ASM – S,G for 5 VTEP
S,G

S=VTEP1 S,G

S=VTEP2 S=VTEP3
S,G

S,G S=VTEP4

S=VTEP7

S,G

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
 Underlay – PIM ASM with PIM Anycast-RP
ip pim anycast-rp 10.254.254.1 10.10.10.201
ip pim anycast-rp 10.254.254.1 10.10.10.202

ip pim rp-address 10.254.254.1

RP RP (Leaf&Spine)
interface loopback254 Spine Spine Spine Spine interface loopback254
description RP description RP
ip address 10.254.254.1/32 ip address 10.254.254.1/32
ip pim sparse-mode ip pim sparse-mode
Underlay
interface loopback0 interface loopback0
description RID description RID
ip address 10.10.10.201/32
Leaf Leaf Leaf Leaf Leaf Leaf ipLeaf address 10.10.10.202/32
ip pim sparse-mode ip pim sparse-mode

Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Underlay - Multicast Routing and Rendezvous-Point
• The Spine makes a good
Rendezvous-Point (RP)
• Use multiple RP for Redundancy
• Watch your Multicast-Group and
OIF* scale

• VXLAN uses Multicast for BUM

• Broadcast, Unknown Unicast,
Multicast
• 1:1 Multicast-to-VNI mapping
• 1:N Multicast-to-VNI mapping

• Ingress-Replication can be valid

as well

*OIF: Outgoing Interface BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
 Underlay – Ingress Replication
• A Packet Multiplication
• EVPN assists no Peer, VNI Topology
Spine Spine Spine Spine

• Various Platform Support

• Nexus 9000 Underlay
• Ingress Replication
•
Leaf Leaf Leaf Leaf Leaf Leaf Leaf
Host sends 1 Packet to Edge-Device
• Edge-Device Encapsulates 1 Packet
and multiplies it
• Ingress VTEP sends 1 Packet per
Neighbor

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
VXLAN with BGP EVPN
Agenda
• Introduction to Data Center Fabrics
• Leaf, Spine, Super-Spine (CLOS)
• Overlay
• Underlay

• VXLAN with BGP EVPN

• Control & Data Plane
• Multi-Tenancy
• Distributed Anycast Gateway
• VPC
• DCI

• A Deployment Story … for your reference

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
What is … ?
• VXLAN • EVPN
• Standards based Encapsulation • Standards based Control-Plane
• RFC 7348 • RFC 7432
• Uses UDP-Encapsulation • Uses Multiprotocol BGP

• Transport Independent • Uses Various Data-Planes

• Layer-3 Transport (Underlay) • VXLAN (EVPN-Overlay), MPLS,
Provider Backbone (PBB)
• Flexible Namespace
• 24-bit field (VNID) provides ~16M • Many Use-Cases Covered
unique identifier • Bridging, MAC Mobility, First-Hop &
• Allows Segmentations Prefix Routing, Multi-Tenancy (VPN)

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Introducing Ethernet VPN (EVPN)

EVPN MP-BGP – RFC 7432

MPLS Provider Backbone Bridges Overlay (NVO3)

(draft-ietf-l2vpn-evpn) (draft-ietf-l2vpn-pbb-evpn) (draft-ietf-bess-evpn-overlay)

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
VXLAN and EVPN related RFCs & Drafts (IETF)
ID Title Category

RFC 7348 Virtual Extensible Local Area Network Data Plane

RFC 7432 BGP MPLS based Ethernet VPNs Control Plane

draft-ietf-bess-evpn-overlay A Network Virtualization Overlay Solution using EVPN Control Plane

draft-ietf-bess-evpn-inter-subnet-forwarding Integrated Routing and Bridging in EVPN Control Plane

draft-ietf-bess-l2vpn-evpn-prefix-advertisement IP Prefix Advertisement in E-VPN Control Plane

draft-tissa-nvo3-oam-fm NVO3 Fault Management / OAM Management Plane

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Multiprotocol BGP (MP-BGP) Primer

Spine

Leaf Leaf
AS#65500

Spine

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
Multiprotocol BGP (MP-BGP) Primer

• Multiprotocol BGP (MP-BGP)

Spine

• Extension to Border Gateway

Protocol (BGP)
• RFC 4760
Leaf Leaf
AS#65500 • VPN Address-Family
• Allows different types of Address-
Families (i.e. VPNv4/v6, MVPN,
Spine
L2VPN, EVPN)
• Various Information transported
over single BGP Peering

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
 Multiprotocol BGP (MP-BGP) Primer
vrf context A vrf context A
rd 10.10.10.101:3 rd 10.10.10.102:8
address-family ipv4 unicast address-family ipv4 unicast
route-target import 65500:5000
route-target export 65500:5000
Spine
• VPNroute-target
Segmentation
route-target
import 65500:5000
for 65500:5000
export Tenant
Routing
• Route Distinguisher (RD)
Leaf • 8-byte field
Leaf
AS#65500
• A Value to make a VPN Prefix
unique
Spine
• RD + VPN Prefix
• [10.10.10.101:5000 + 192.168.10.0/24]

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
 Multiprotocol BGP (MP-BGP) Primer
vrf context A vrf context A
rd 10.10.10.101:3 rd 10.10.10.102:8
address-family ipv4 unicast address-family ipv4 unicast
route-target import 65500:5000 Spine route-target import 65500:5000
route-target export 65500:5000 route-target export 65500:5000

Leaf Leaf
AS#65500

Spine

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
 Multiprotocol BGP (MP-BGP) Primer
vrf context A vrf context A
rd auto rd auto
address-family ipv4 unicast address-family ipv4 unicast
route-target import 65500:5000 Spine route-target import 65500:5000
route-target export 65500:5000 route-target export 65500:5000
• Cisco provides automated Route
Distinguisher derivation
Leaf • Macros uses Type 1 format Leaf
AS#65500 • 4-byte Router ID (RID)
• 4-byte VRF ID (internal number)
• Example of auto derived RD:
Spine
• 10.10.10.101:3

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
 Multiprotocol BGP (MP-BGP) Primer
vrf context A vrf context A
rd auto rd auto
address-family ipv4 unicast address-family ipv4 unicast
route-target import 65500:5000 Spine route-target import 65500:5000
route-target export 65500:5000 route-target export 65500:5000

Leaf Leaf
AS#65500

Spine

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
 Multiprotocol BGP (MP-BGP) Primer
vrf context A vrf context A
rd auto rd auto
address-family ipv4 unicast • VPNaddress-family
Segmentation forunicast
ipv4 Tenant
route-target import 65500:5000 Spine route-target import 65500:5000
route-target export 65500:5000 Routing
route-target export 65500:5000

• Route Target (RT)

• 8-byte field
Leaf Leaf
AS#65500 • A Value to import/export a VPN
Prefix
• Each RD + VPN Prefix have an
Spine
RT
• [10.10.10.101:5000 + 192.168.10.0/24]
• [65500:5000, 65500:5000]

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
 Multiprotocol BGP (MP-BGP) Primer
vrf context A vrf context A
rd auto rd auto
address-family ipv4 unicast address-family ipv4 unicast
route-target import auto Spine route-target import auto
route-target export auto route-target export auto

Leaf Leaf
AS#65500

Spine

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
 Multiprotocol BGP (MP-BGP) Primer
vrf context A vrf context A
rd auto rd auto
address-family ipv4 unicast address-family ipv4 unicast
route-target import auto Spine route-target import auto
route-target export auto • Cisco provides automated
route-target export auto Route
Target derivation
• Macros uses following values
•
Leaf Leaf
AS#65500 4-byte Autonomous System
• 4-byte VNI
• Example of auto derived RD:
• Import, Export or Both
Spine
• 65500:5000

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
 Multiprotocol BGP (MP-BGP) Primer
vrf context A vrf context A
rd 10.10.10.101:3 rd 10.10.10.102:8
address-family ipv4 unicast address-family ipv4 unicast
route-target import 65500:5000 Spine route-target import 65500:5000
route-target export 65500:5000 route-target export 65500:5000

Leaf Leaf
101010110101
01010101010 AS#65500

Spine

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
 Multiprotocol BGP (MP-BGP) Primer
vrf context A vrf context A
rd 10.10.10.101:3 rd 10.10.10.102:8
address-family ipv4 unicast address-family ipv4 unicast
route-target import 65500:5000 Spine route-target import 65500:5000
route-target export 65500:5000 route-target export 65500:5000

RD Prefix Next-Hop Route Target

Leaf Leaf

10.10.10.101:3 192.168.10.0/24 10.200.200.101

101010110101
01010101010 AS#65500
65500:5000, 65500:5000

10.10.10.101:7 192.168.20.0/24 10.200.200.101 65500:5001, 65500:5001

Spine

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
 Multiprotocol BGP (MP-BGP) Primer
vrf context A vrf context A
rd 10.10.10.101:3 rd 10.10.10.102:8
address-family ipv4 unicast address-family ipv4 unicast
route-target import 65500:5000 Spine route-target import 65500:5000
route-target export 65500:5000 route-target export 65500:5000

RD Prefix Next-Hop Route Target

Leaf Leaf
101010110101
01010101010 AS#65500
10.10.10.101:3 192.168.10.0/24 10.200.200.101 65500:5000, 65500:5000

10.10.10.101:7 192.168.20.0/24 10.200.200.101 65500:5001, 65500:5001

Next-Hop:
Spine 5 192.168.10.0/24
10.200.200.101

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
 EVPN - Host and Subnet Route Distribution
• Host Route Distribution
RR RR decoupled from the Underlay
Spine Spine Spine Spine
protocol

Overlay • Use MultiProtocol-BGP (MP-

BGP) on the Leaf nodes to
Leaf Leaf Leaf Leaf Leaf Leaf Leaf distribute internal Host/Subnet
Routes and external reachability
information
• Route-Reflectors (RR) deployed
for scaling purposes

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
 EVPN Control Plane - Host and Subnet Routes
• BGP EVPN NLRI*

Spine Spine Spine Spine

• Host MAC (Route Type 2)
• MAC only, Single VNI, Single
Route Target
Overlay
• Host MAC+IP (Route Type 2)
Leaf Leaf Leaf Leaf Leaf Leaf Leaf • MAC and IP, Two VNI, Two Route
Target, Router MAC

• Internal and External Subnet

Prefixes (Route Type 5)
• IP Subnet Prefix, Single VNI,
Single Route Target

*NLRI: Network Layer Reachability Information (BGP Update Format) BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
 Host Advertisements
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.

2 0000.3001.1101 / 48 3001, 65500:3001 10.200.200.101

Spine Spine Spine Spine

2 0000.3001.1102 / 48 3001, 65500:3001 10.200.200.104

• Host MAC (Route Type 2)
2 0000.3002.2101 / 48
Overlay 3002, 65500:3002 • MAC
10.200.200.107

• MPLS Label1 (L2VNI*)

Leaf
101010110101
01010101010
Leaf Leaf Leaf
101010110101
01010101010
Leaf Leaf Leaf
101010110101
01010101010
• Route Target for MAC-VRF

• MAC attributes are Mandatory

Baremetal Baremetal Baremetal

Host A Host B Host C

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101

*L2VNI: VNI for all Bridging operation (”VLAN-VNI”) BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
 Ethernet Tag
Ethernet Identifier
V2# show bgp l2vpn evpn 0000.3001.1101 (Ethtag)
Segment
Identifier (ESI) MAC Address MAC
Route Type: Length Address
BGP routing table information
MAC/IP for VRF default, address family L2VPN EVPN
Route Distinguisher: 10.10.10.101:32777
BGP routing table entry for [2]:[0]:[0]:[48]:[0000.3001.1101]:[0]:[0.0.0.0]/216,
version 4
Paths: (1 available, best #1)
Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked

Advertised path-id 1
Next-Hop
IP Address
Path type: internal, path
L2VNIis valid, is best path, no labeled nexthop
AS-Path: NONE, path(MPLS
sourced internal to AS
Label1)
10.200.200.101 (metric 3) fromL2VNI
10.10.10.201 (10.10.10.201)
Encap:8
Route Target VXLAN
Origin IGP, MED not set, localpref 100, weight 0
Received label 3001
Extcommunity: RT:65500:3001 ENCAP:8
Originator: 10.10.10.101 Cluster list: 10.10.10.201

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
 Host Advertisements
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.

2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 • Host

5000, 65500:5000 MAC+IP (Route Type 2)
10.200.200.101
•
Spine Spine Spine Spine
MAC and IP
2 0000.3001.1102 / 48 3001, 65500:3001 192.168.10.102 /32 5000, 65500:5000
• 10.200.200.104
MPLS Label1 (L2VNI)
2 0000.3002.2101 / 48
Overlay 3002, 65500:3002 192.168.20.101 /32 5000, 65500:5000• Route Target for MAC-VRF
10.200.200.107
• MPLS Label2 (L3VNI*)
Leaf
101010110101
Leaf Leaf Leaf
101010110101
Leaf Leaf Leaf
101010110101
• Route Target for IP-VRF
•
01010101010 01010101010 01010101010

Router MAC

• IP Attributes are Optional

Baremetal Baremetal Baremetal • Populated through ARP/ND
Host A Host B Host C
MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101

*L3VNI: VNI for all Routing operation (”VRF-VNI”) BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
 Ethernet Tag
Ethernet Identifier
V2# show bgp l2vpn evpn 0000.3001.1101 (Ethtag)
Segment
Identifier (ESI) MAC Address MAC
Route Type: Length Address
BGP routing table information
MAC/IP for VRF default, address family L2VPN EVPN
Route Distinguisher: 10.10.10.101:32777
BGP routing table entry for [2]:[0]:[0]:[48]:[0000.3001.1101]:[32]:[192.168.10.101]/272,
version 4
Paths: (1 available, best #1) IP Address
Length
Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked IP Address

Advertised path-id 1
Next-Hop L3VNI
IP Address
Path type: internal,L2VNI
path is (MPLS
valid, is best path, no labeled nexthop
Label2)
AS-Path: NONE, path
(MPLSsourced
Label1) internal to AS
10.200.200.101 (metric 3) from 10.10.10.201 (10.10.10.201)
Encap:8
Origin IGP, MED not set, localpref 100, weight 0 VXLAN
Received label 3001 5000
Extcommunity: RT:65500:3001 RT:65500:5000 ENCAP:8 Router MAC:0200.0ade.de01
Originator: 10.10.10.101 Cluster list: 10.10.10.201
L2VNI L3VNI
Route Target Router MAC
Route Target

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
 Subnet Route Advertisements
Type IP / Length L3VNI / RT Next-Hop Seq.

5 192.168.10.0 /24 5000, 65500:5000 10.200.200.101

Spine Spine Spine Spine
• Internal and External Subnet
Prefixes (Route Type 5)
Overlay • IP Prefix
• MPLS Label (L3VNI)
Leaf Leaf Leaf Leaf Leaf Leaf Leaf
• Route Target for IP-VRF
•
101010110101
01010101010

Router MAC

• Populated through External

Routing Protocol
Subnet A
192.168.10.0/24

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
 Subnet Route Advertisements
Type IP / Length L3VNI / RT Next-Hop Seq.

5 192.168.10.0 /24
Spine
5000, 65500:5000
Spine Spine
10.200.200.101
Spine
• IP Prefix Learning
5 192.168.10.0 /24 5000, 65500:5000 10.200.200.104
• via BGP with VRF-Lite
• via LISP on Nexus 7000/7700
5 192.168.20.0 /24
Overlay
5000, 65500:5000 10.200.200.107 • via other routing protocol (static
or dynamic)
Leaf Leaf Leaf Leaf Leaf Leaf Leaf

• Default: Export of IP Host and IP

101010110101 101010110101
01010101010 01010101010

Prefix Routes advertisements

• Filter and Summarize where
appropriate
Subnet A Subnet A Subnet B
192.168.10.0/24 192.168.10.0/24 192.168.20.0/24

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
 Ethernet Tag
Ethernet Identifier
Segment
V2# show bgp l2vpn evpn 192.168.10.0 (Ethtag)
Identifier (ESI) IP Address
Route Type: IP Address
Length family
BGP routing table information
IP Prefix for VRF default, address L2VPN EVPN
Route Distinguisher: 10.10.10.101:3
BGP routing table entry for [5]:[0]:[0]:[24]:[192.168.10.101]/224,
version 4
Paths: (1 available, best #1)
Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked

Advertised path-id 1
Next-Hop
IP Address
Path type: internal, path
L3VNIis valid, is best path, no labeled nexthop
AS-Path: NONE, path(MPLS
sourced internal to AS
Label)
10.200.200.101 (metric 3) fromL3VNI
10.10.10.201 (10.10.10.201)
Encap:8
Origin IGP, MED not set, localpref 100, weight 0 Router MAC
Route Target VXLAN
Received label 5000
Extcommunity: RT:65500:5000 ENCAP:8 Router MAC:0200.0ade.de01
Originator: 10.10.10.101 Cluster list: 10.10.10.201

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
 Introducing VXLAN

MAC 802.1q IP Payload CRC

Src and Dst
Src, Dst VTEP VTEP IP
and Hop-by- UDP Dst VXLAN
Address Port 4789 VNI
Hop MAC Original Layer-2 Frame

Outer MAC Outer IP UDP VXLAN Inner MAC Inner IP Payload CRC

Data-Plane (VXLAN) 20-byte + 8-byte +8-byte + 14-byte* = 50 Bytes

of total overhead

UDP Src Port

Hash of L2/L3/L4
headers of original
Frame

*plus 4-byte if IEEE 802.1q exists as part of Inner MAC Header BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
VXLAN Frame Format – MAC in IP Encapsulation
Field Value Bites Total Field Value Bites Total

Dest. MAC Address Next-Hop MAC Address 48 Source Port L2/L3/L4 Hash 16

(4 Bytes Optional)

8 Bytes
Src. MAC Address Next-Hop MAC Address 48 Destination Port 4789 (UDP) 16

14 Bytes
VLAN Type 0x8100 16 UDP Length 16

VLAN ID Tag 16 Checksum 0x0000 16

Ether Type 0x0800 16

Outer MAC Outer IP UDP VXLAN Inner MAC Payload CRC

Field Value Bites Total

Field Value Bites Total
VXLAN Flags RRRRIRRR 8
IP Header Misc. Data 72

8 Bytes
Reserved 24
Protocol 0x11 (UDP) 8
20 Bytes

VNI 16M Possible Segments 24

Header Checksum Various 16
Reserved 8
Source IP Src, VTEP IP 32

Destination IP Dest. VTEP IP 32 BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
No Path Diversity

Spine

Leaf Leaf
101010110101010
10101010
Baremetal
AS#65500 Baremetal

Spine

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
No Path Diversity

Spine
• Equal Cost Multi-Pathing (ECMP)
uses Header information to form
Path Diversity

Leaf
• Some Tunnel Protocol provide no
Leaf
101010110101010
10101010
Baremetal
AS#65500 diversity in IP or Protocol Header Baremetal

• As a Result, all Packets travel

the same Path
Spine

• No Path Diversity or Entropy

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
Introducing VXLAN – Entropy

Spine

VTEP VTEP
101010110101010
10101010
Baremetal
AS#65500 Baremetal

Spine

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
Introducing VXLAN – Entropy

Spine
• VXLAN provides variable UDP
Source Port in Outer Header
• Hash of the inner Layer-2/Layer-
VTEP VTEP
101010110101010
10101010
Baremetal
AS#65500 3/Layer-4 Headers of the original Baremetal

Ethernet Frame.
• Enables entropy for ECMP Load
Spine
balancing in the Network

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
Introducing VXLAN – Entropy

Spine

VTEP VTEP
AS#65500

Entropy Spine

happens here

Outer MAC Outer IP UDP VXLAN Inner MAC Inner IP Payload CRC
Data-Plane (VXLAN)
BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
 VXLAN and BGP EVPN – Putting it Together
Control-Plane (BGP EVPN)
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.

3001 5000
2 0000.3001.1101/48 192.168.10.101/32 10.200.200.101
65500:3001 65500:5000

Dst VTEP IP L2VNI Dst MAC Dst IP

10.200.200.101 3001 0000.3001.1101 192.168.10.101

Outer MAC Outer IP UDP VXLAN Inner MAC Inner IP Payload CRC
Data-Plane (VXLAN)

Bridging

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
EVPN Layer-2 Services (1)

• Single Subnet per EVI

• VLAN-based

• Per EVI BGP Route Distinguisher / Router Target per EVI / VNI
• BGP Route-Target constrain mechanism to limit propagation (import/export)

• 1:1:1 mapping
• VNI to EVI to Single Broadcast
Domain (Bridge Domain)
• Ethernet Tag ID must be 0

VID = VLAN ID
VNI = VXLAN Network Identifier
EVI = EVPN Virtual Instance BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
EVPN Layer-2 Services (1)
(draft-ietf-bess-evpn-overlay – Section 5.1.2)

• VLAN-based

VID
VNI EVI
10

Route Target: 65000:30000

VID = VLAN ID
VNI = VXLAN Network Identifier
EVI = EVPN Virtual Instance BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
EVPN Layer-2 Services (1)
(draft-ietf-bess-evpn-overlay – Section 5.1.2)

• VLAN-based

VID
VNI EVI
10

Route Target: 65000:30000

VID = VLAN ID
VNI = VXLAN Network Identifier
EVI = EVPN Virtual Instance BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
EVPN Layer-2 Services (2)
(draft-ietf-bess-evpn-overlay – Section 6.1)

• VLAN-based
[2]:[0]:[0]:[48]:[0050.569f.d495]:[32]:[192.168.22.33]

VID
VNI EVI
10

Route Target: 65000:30000

VID = VLAN ID
VNI = VXLAN Network Identifier
EVI = EVPN Virtual Instance BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
EVPN Layer-2 Services (2)
(draft-ietf-bess-evpn-overlay – Section 6.1)

• VLAN-based
[2]:[0]:[0]:[48]:[0050.569f.d495]:[32]:[192.168.22.33]

VID
VNI EVI
10

Route Target: 65000:30000

VID = VLAN ID
VNI = VXLAN Network Identifier
EVI = EVPN Virtual Instance BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
 VXLAN and BGP EVPN – Putting it Together
Extended Community
Router MAC
Control-Plane (BGP EVPN) 0200.0ade.de01

Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.

3001 5000
2 0000.3001.1101/48 192.168.10.101/32 10.200.200.101
65500:3001 65500:5000

Dst VTEP IP L3VNI Router MAC Dst IP

10.200.200.101 5000 0200.0ade.de01 192.168.10.101

Outer MAC Outer IP UDP VXLAN Inner MAC Inner IP Payload CRC
Data-Plane (VXLAN)

Routing

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 91
 Routing and the Router MAC – Ethernet
Router MAC

SMAC DMAC SIP DIP

Payload
0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101

SMAC DMAC SIP DIP SMAC DMAC SIP DIP

Payload Payload
0000.3001.1101 2020:0000:AAAA 192.168.10.101 192.168.20.101 2020.0000AAAA 0000.3002.2101 192.168.10.101 192.168.20.101

Switch Switch
SVI10 SVI20
192.168.10.1 192.168.20.1

interface: Eth2/1 interface: Eth2/1

MAC: 0200.0ade.de01 MAC: 0200.0ade.de07
Baremetal IP: 10.200.200.1 IP: 10.200.200.7 Baremetal

Host A Host C
MAC: 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.20.101

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
 Routing and the Router MAC – VXLAN
Router MAC

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
10.200.200.101 10.200.200.107 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101

SMAC DMAC SIP DIP SMAC DMAC SIP DIP

Payload Payload
0000.3001.1101 2020:0000:AAAA 192.168.10.101 192.168.20.101 2020.0000AAAA 0000.3002.2101 192.168.10.101 192.168.20.101

SVI10 SVI20
192.168.10.1 VTEP
VXLAN VTEP 192.168.20.1

interface: NVE1 interface: NVE1

MAC: 0200.0ade.de01 MAC: 0200.0ade.de07
Baremetal IP: 10.200.200.1 IP: 10.200.200.7 Baremetal

Host A Host C
MAC: 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.20.101

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
EVPN IP-VRF Services (1)

• Interface-Less Model
• Route-Type 5 only
• Next-Hop is remote VTEP
• Two extended communities
• Encapsulation Extended
Community
• Router’s MAC Address (remote
VTEP)

Route Type 2 = MAC/IP Route

Route Type 5 = IP Prefix Route
BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 94
EVPN IP-VRF Services (2)
(draft-ietf-bess-evpn-prefix-advertisement – Section 5.4.1)

• Interface-Less

NVE IP: 10.22.22.34

EVPN Router MAC: 0200.0ADE.DE22

VTEP VTEP

[5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0]
BGP 10.22.22.34 (Next-Hop)
Update Encap:8 (VXLAN)
Router MAC:0200.0ade.de22

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 96
EVPN IP-VRF Services (2)
(draft-ietf-bess-evpn-prefix-advertisement – Section 5.4.1)

• Interface-Less

NVE IP: 10.22.22.34

EVPN Router MAC: 0200.0ADE.DE22

VTEP VTEP

[5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0]
BGP 10.22.22.34 (Next-Hop)
Update Encap:8 (VXLAN)
Router MAC:0200.0ade.de22

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 97
 Packet Walk – ARP Request
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.
2 0000.3001.1101 / 48 3001, 65500:3001 10.200.200.101

Spine Spine Spine Spine

SIP DIP VXLAN Overlay

SMAC DMAC
ARP Request for
192.168.10.102
10.200.200.101 239.0.0.1 3001 0000.3001.1101 FFFF.FFFF.FFFF

VTEP VTEP VTEP VTEP VTEP VTEP VTEP

ARP Request for 192.168.10.102

SMAC: DMAC:
ARP Request for 192.168.10.102 0000.3001.1101 FFFF.FFFF.FFFF

SMAC: DMAC:
0000.3001.1101 FFFF.FFFF.FFFF

Baremetal Baremetal Baremetal

Host A Host B Host C

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 102
Packet Walk – ARP Response
Type
Type MAC
MAC/ Length
/ Length L2VNI
L2VNI/ RT
/ RT IPIP/ Length
/ Length L3VNI
L3VNI/ RT
/ RT Next-Hop
Next-Hop Seq.
Seq.
22 0000.3001.1101
0000.3001.1101/ 48
/ 48 3001,
3001,65500:3001
65500:3001 192.168.10.101/32 5000, 65500:5000 10.200.200.101
10.200.200.101

Spine Spine Spine Spine

SIP DIP VXLAN Overlay

SMAC DMAC
ARP Response for
192.168.10.102
10.200.200.104 10.200.200.101 3001 0000.3001.1102 0000.3001.1101

VTEP VTEP VTEP VTEP VTEP VTEP VTEP

ARP Response for 192.168.10.102 ARP Response for 192.168.10.102

SMAC: DMAC: SMAC: DMAC:

0000.3001.1102 0000.3001.1101 0000.3001.1102 0000.3001.1101

Baremetal Baremetal Baremetal

Host A Host B Host C

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 103
 Packet Walk – Bridging
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101/32 5000, 65500:5000 10.200.200.101
2 0000.3001.1102 / 48 3001, 65500:3001 192.168.10.102/32 5000, 65500:5000 10.200.200.104
Spine Spine Spine Spine

SIP DIP VXLAN SMAC Overlay DMAC SIP DIP

Payload
10.200.200.101 10.200.200.104 3001 0000.3001.1101 0000.3001.1102 192.168.10.101 192.168.10.102

VTEP VTEP VTEP VTEP VTEP VTEP VTEP

SMAC DMAC SIP DIP

0000.3001.1101 0000.3001.1102 192.168.10.101 192.168.10.102

SMAC DMAC SIP DIP

0000.3001.1101 0000.3001.1102
Baremetal 192.168.10.101 192.168.10.102 Baremetal Baremetal

Host A Host B Host C

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 104
 Packet Walk – Routing
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101/32 5000, 65500:5000 10.200.200.101
2 0000.3002.2102 / 48 3002, 65500:3002 192.168.20.101/32 5000, 65500:5000 10.200.200.107
Spine Spine Spine Spine

SIP DIP VXLAN SMAC Overlay DMAC SIP DIP

Payload
10.200.200.101 10.200.200.107 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101

VTEP VTEP VTEP VTEP VTEP VTEP VTEP

SMAC DMAC SIP DIP

Router MAC
2020.0000.AAAA 0000.3002.2101 192.168.10.101 192.168.20.101

SMAC DMAC SIP DIP

0000.3001.1101 2020.0000.AAAA
Baremetal 192.168.10.101 192.168.20.101 Baremetal Baremetal

Host A Host B Host C

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 105
 Packet Walk – Routing (Silent Host)
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 5000, 65500:5000 10.200.200.101
5 192.168.20.0/24 5000, 65500:5000 10.200.200.107
Spine Spine Spine Spine

SIP DIP VXLAN SMAC Overlay DMAC SIP DIP

Payload
10.200.200.101 10.200.200.107 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101

VTEP VTEP VTEP VTEP VTEP VTEP VTEP

SMAC DMAC SIP DIP

0000.3001.1101 2020.0000.AAAA
Baremetal 192.168.10.101 192.168.20.101 Baremetal Baremetal

Host A Host B Host C

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 106
 Packet Walk – Routing (Silent Host)
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 5000, 65500:5000 10.200.200.101
5 192.168.20.0/24 5000, 65500:5000 10.200.200.107
Spine Spine Spine Spine
2 0000.3002.21o1 / 48 3002, 65500:3002 192.168.20.101 5000, 65500:5000 10.200.200.107

SIP DIP VXLAN SMAC Overlay DMAC SIP DIP

Payload
10.200.200.101 10.200.200.107 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101

VTEP VTEP VTEP VTEP VTEP VTEP VTEP

ARP Response for 192.168.20.101

SMAC: DMAC:
0000.3002.2101 2020.0000.AAAA

ARP Request for 192.168.20.101

SMAC: DMAC:
SMAC DMAC SIP DIP 2020.0000.AAAA FFFF.FFFF.FFFF
0000.3001.1101 2020.0000.AAAA
Baremetal 192.168.10.101 192.168.20.101 Baremetal Baremetal

Host A Host B Host C

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 107
Agenda
• Introduction to Data Center Fabrics
• Leaf, Spine, Super-Spine (CLOS)
• Overlay
• Underlay

• VXLAN with BGP EVPN

• Control & Data Plane
• Multi-Tenancy
• Distributed Anycast Gateway
• VPC
• DCI

• A Deployment Story … for your reference

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
What is Multi-Tenancy ?

• Segregation at Layer-2 • Segregation at Layer-3

• VLAN • VRF
• Layer-2 VNI (L2VNI) • Layer-3 VNI (L3VNI)

• VLAN Significance • VRF Significance

• Per-Fabric • Per-Fabric
• Per-Switch • Per-Switch
• Per-Port

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 109
 Layer-2 Multi-Tenancy – Bridge Domains
• Bridge Domain
• Layer-2 Segment from End-Point
Spine Spine Spine Spine
to End-Point

Overlay • Bridge Domains in VXLAN

VNI 3001 (L2VNI)
consists of
Leaf Leaf Leaf Leaf Leaf Leaf Leaf
• The Ethernet Segment (VLAN)
between Host and Edge Device
• The Hardware Resources within
VLAN 10 the Edge Device
VLAN 100
• The VXLAN Segment (VNI)
Baremetal Baremetal Baremetal
between Edge Device and Edge
Host A
VLAN 10
Host B
VLAN 100
Host C
VLAN 20
Device

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 110
 Layer-3 Multi-Tenancy – Routing Domains

• Routing Domain
Spine Spine Spine Spine
• Multiple Subnets sharing the
same Layer-3 forwarding policy
Overlay
VNI 5000 (L3VNI) • Routing Domain in VXLAN
consists of
Leaf Leaf Leaf Leaf Leaf Leaf Leaf • The Routing Domain local to the
Edge Device (VRF)
• The Routing Domain (VPN)
across the Edge Devices
Baremetal Baremetal Baremetal
• Multi-Protocol BGP with EVPN
Host A Host B Host C
Address-Family
192.168.10.101 192.168.10.102 192.168.20.101

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 111
Agenda
• Introduction to Data Center Fabrics
• Leaf, Spine, Super-Spine (CLOS)
• Overlay
• Underlay

• VXLAN with BGP EVPN

• Control & Data Plane
• Multi-Tenancy
• Distributed Anycast Gateway
• VPC
• DCI

• A Deployment Story … for your reference

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
 Distributed IP Anycast Gateway
• Distributed First-Hop Routing on
192.168.20.1
192.168.10.1 2020.0000.AAAA Edge Device
2020.0000.AAAA
•
Spine Spine Spine Spine
All Edge Device share same
Gateway IP and MAC address
Overlay • Pervasive Gateway approach

Leaf Leaf Leaf Leaf Leaf Leaf Leaf

• Gateway is always active
• No redundancy protocol for hello
or state exchange

• Distributed and smaller state

• Only local End-Points ARP
entries

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 113
 Distributed IP Anycast Gateway
• Distributed First-Hop Routing on
192.168.20.1
192.168.10.1 2020.0000.AAAA Edge Device
2020.0000.AAAA
•
Spine Spine Spine Spine
All Edge Device share same
Gateway IP and MAC address
Overlay • Pervasive Gateway approach

Leaf Leaf Leaf Leaf Leaf Leaf Leaf

• Gateway is always active
• No redundancy protocol for hello
or state exchange

• Distributed and smaller state

• Only local End-Points ARP
entries

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 114
 Anycast – One-to-Nearest Association

Spine Spine Spine Spine

• Network Addressing and Routing
Methodology
Overlay • Datagrams sent from a single
Sender to the Topologically
Leaf Leaf Leaf Leaf Leaf Leaf Leaf
Nearest Node
• Group of potential Receivers, all
identified by the same
Baremetal Baremetal Baremetal
Destination Address
Host A Host B Host C

*L3VNI: VNI for all Routing operation (”VRF-VNI”) BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 115
Symmetric or Asymmetric IRB

• Symmetric Inter-Subnet • Asymmetric Inter-Subnet

Forwarding Forwarding
• Bridge->Route/Route->Bridge • Bridge->Route->Bridge
• Symmetric VNI in both • Different (Asymmetric) VNI
directions depending on directions
• Adjacency contains Remote • Adjacency contains Remote
VTEP,VRF VTEP,VRF and End-Points
• Optimal for Scale • Potential Sub-Optimal for Scale
• Flexible Configuration • Consistent Configuration

VTEP = VXLAN Tunnel End-Point

VRF = Virtual Routing and Forwarding
VNI = VXLAN Network Identifier
BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 116
EVPN IRB Services (1)
(Traditional Bridging – Depending on EVPN Layer-2 Services)

• Symmetric IRB • Asymmetric IRB

VNI 30000 (L2VNI) VNI 30000 (L2VNI)

V1 V2 V1 V2

MA MA
MAC IP IP MAC MAC IP IP MAC
C C

192.168.22.33 192.168.22.44 192.168.22.33 192.168.33.44

192.168.22.44

Bridge Bridge

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 117
EVPN IRB Services (2)
(draft-ietf-bess-evpn-inter-subnet-forwarding – Section 4 and 5)

• Symmetric IRB • Asymmetric IRB

VNI 50000 (L3VNI) VNI 40000 (L2VNI)

VNI 30000 (L2VNI)

V1 V2 V1 V2

MA MA MA MA
IP IP IP IP
C C C C

192.168.22.33 192.168.33.44 192.168.22.33 192.168.33.44

Bridge -> Route -> Route -> Bridge Bridge -> Route -> Bridge

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 118
EVPN IRB Services (3)
(draft-ietf-bess-evpn-inter-subnet-forwarding – Section 5.1)

• Symmetric IRB
VNI 50000 (L3VNI)

V1 V2

MA MA
IP IP
C C

192.168.22.33 192.168.33.44
[2]:[0]:[0]:[48]:[0050.569f.d495]:[32]:[192.168.33.44]
BGP 10.22.22.34 (Next-Hop)
Update L2VNI: 30000
L3VNI: 50000

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 119
EVPN IRB Services (3)
(draft-ietf-bess-evpn-inter-subnet-forwarding – Section 5.1)

• Symmetric IRB
VNI 50000 (L3VNI)

V1 V2

MA MA
IP IP
C C

192.168.22.33 192.168.33.44
[2]:[0]:[0]:[48]:[0050.569f.d495]:[32]:[192.168.33.44]
BGP 10.22.22.34 (Next-Hop)
Update L2VNI: 30000
L3VNI: 50000

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 120
EVPN IRB Services (4)
(draft-ietf-bess-evpn-inter-subnet-forwarding – Section 4.1)

• Symmetric IRB • Asymmetric IRB

VNI 50000 (L3VNI) VNI 40000 (L2VNI)

VNI 30000 (L2VNI)

V1 V2 V1 V2

MA MA MA MA
IP IP IP IP
C C C C

192.168.22.33 192.168.33.44 192.168.22.33 192.168.33.44

[2]:[0]:[0]:[48]:[0050.569f.d495]:[32]:[192.168.33.44]
BGP 10.22.22.34 (Next-Hop)
Update L2VNI: 30000

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 121
EVPN IRB Services (4)
(draft-ietf-bess-evpn-inter-subnet-forwarding – Section 4.1)

• Symmetric IRB • Asymmetric IRB

VNI 50000 (L3VNI) VNI 40000 (L2VNI)

VNI 30000 (L2VNI)

V1 V2 V1 V2

MA MA MA MA
IP IP IP IP
C C C C

192.168.22.33 192.168.33.44 192.168.22.33 192.168.33.44

[2]:[0]:[0]:[48]:[0050.569f.d495]:[32]:[192.168.33.44]
BGP 10.22.22.34 (Next-Hop)
Update L2VNI: 30000

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 122
EVPN IRB Services (5)

• Symmetric IRB and Asymmetric IRB is NOT interoperable per-se

• Routing is implemented different
• Symmetric IRB: Bridge -> Route -> Route -> Bridge
• Asymmetric IRB: Bridge -> Route -> Bridge

• Symmetric IRB uses Route-Type 2 with two VNI

• L3VNI for routing and L2VNI for bridging
• Asymmetric IRB uses Route-Type2 and Route-Type 5
• Type 2 with L2VNI for bridging and inter-subnet forwarding (known VNI/VTEP)
• Type 5 with L3VNI for inter-subnet forwarding (see IP-VRF Services)
• If implemented

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 123
Agenda
• Introduction to Data Center Fabrics
• Leaf, Spine, Super-Spine (CLOS)
• Overlay
• Underlay

• VXLAN with BGP EVPN

• Control & Data Plane
• Multi-Tenancy
• Distributed Anycast Gateway
• VPC
• DCI

• A Deployment Story … for your reference

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
 VPC Gateway Redundancy
• VPC – Virtual Port-Channel
• Multi-Chassis Link Aggregation
• Layer-2 Multihoming Spine Spine Spine Spine

• Extended for VXLAN

• Host-side Overlay
• Dual-Connect Hosts VPC VPC

• Using Port-Channels Leaf VTEP VTEP Leaf Leaf VTEP VTEP

• Fabric-side
• Individual VTEPs
• Using a common Anycast VTEP
•
Baremetal Baremetal Baremetal

Seen as one VTEP from remote

Nodes

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 129
VPC Gateway Redundancy – A VXLAN perspective
Both sharing an
Overlay Anycast VTEP

VPC

VTEP VTEP

Individual Node Individual Node

with unique with unique
Identity Identity

Baremetal

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 130
 VPC Gateway Redundancy – A VXLAN perspective

Overlay
VPC

interface loopback0 VTEP VTEP interface loopback0

description RID description RID
ip address 10.10.10.102/32 ip address 10.10.10.103/32

interface loopback1 interface loopback1

description VTEP description VTEP
ip address 10.200.200.102/32 ip address 10.200.200.103/32
ip address 10.200.200.123/32 secondary ip address 10.200.200.123/32 secondary

Anycast VTEP Anycast VTEP

IP Address IP Address
Baremetal

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 131
Host Advertisements with VPC
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.

2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.123

Spine Spine Spine Spine

2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.123

Overlay
VPC VPC
Leaf VTEP VTEP Leaf Leaf VTEP VTEP
101010110101 101010110101
01010101010 01010101010

Baremetal Baremetal Baremetal

Host A Host B Host C

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 132
 Host Advertisements with VPC
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.

• Independent Devices
2
in0000.3001.1101
the EVPN / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.123
Control-Plane Spine Spine Spine Spine

• Individual Router and

2
Peering
0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.123

• Unique Route Distinguisher (RD) Overlay

• Independent Underlay Routing
VPC VPC
Devices Leaf VTEP VTEP Leaf Leaf VTEP VTEP
101010110101 101010110101
01010101010 01010101010

• Common VXLAN Device

• Next-Hop is Anycast VTEP
• Underlay ECMP Load Share to
Anycast VTEP Baremetal Baremetal Baremetal

Host A Host B Host C

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 133
ECMP to the Anycast VTEP – Underlay

Spine

VTEP

VTEP
VPC

Baremetal
AS#65500 101010110101010
10101010
Baremetal

VTEP
Host A Host B
MAC: 0000.3001.1101 MAC: 0000.3001.1102
IP: 192.168.10.101 IP: 192.168.10.102
Spine

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 134
Bridging to a VPC Domain – VXLAN
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
10.200.200.104 10.200.200.123 3001 0000.3001.1102 0000.3001.1101 192.168.10.102 192.168.10.101
Spine

VTEP

VTEP
VPC

Baremetal
AS#65500 Baremetal

VTEP
Host A Host B
MAC: 0000.3001.1101 MAC: 0000.3001.1102
IP: 192.168.10.101 IP: 192.168.10.102
Spine

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 135
Routing to a VPC Domain – VXLAN
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
10.200.200.107 10.200.200.123 5000 0200.0ade.de01 2020.2323.2323 192.168.20.101 192.168.10.101
Spine

Local Station
or
VTEP Virtual MAC

VTEP
VPC

Baremetal
AS#65500 Baremetal

VTEP
Host A Host C
MAC: 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.20.101
Spine

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 136
VPC Gateway Redundancy – A VXLAN perspective

• VPC provides Layer-2 Gateway

Redundancy
• From the VXLAN perspective,
the next-hop is always the
Anycast VTEP (VIP)
• Optimal for direct attached Hosts
• 1:1 Multicast-to-VNI mapping

• VPC operates at Layer-2

• Routing Tables are not Synchronized

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 137
 Subnet Route Advertisement with VPC
Type IP / Length L3VNI / RT Next-Hop Seq.

• Subnet Route Advertisement 5 192.168.11.0 /24

Spine
5000, 65500:5000
Spine Spine
10.200.200.123
Spine

• Route Type 5 5 192.168.22.0 /24 5000, 65500:5000 10.200.200.123

• Next-Hop is Anycast VTEP
Overlay
• Ensure Sync of Subnet VPC VPC
• Dual-Connect Networks (Point-2- Leaf VTEP VTEP Leaf Leaf VTEP VTEP

Point not Layer-3 over VPC)

• Synchronize Routing Table
• Advertise Route Type 5 with
individual VTEP IP (PIP) Baremetal

Host B
Subnet Y MAC: 0000.3001.1102
192.168.22.0/24
IP: 192.168.10.102
Subnet X
192.168.11.0/24
BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 138
Subnet Route Advertisement with VPC
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
10.200.200.107 10.200.200.123 5000 0200.0ade.de01 2020.2323.2323 192.168.20.101 192.168.11.101
Spine

VTEP

Subnet X VTEP
VPC

192.168.11.0/24 AS#65500 101010110101010

10101010
Baremetal

VTEP
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101
Spine

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 139
VPC – Dual-Attach Networks
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
10.200.200.107 10.200.200.123 5000 0200.0ade.de01 2020.2323.2323 192.168.20.101 192.168.11.101
Spine

VTEP

Subnet X VTEP
VPC

192.168.11.0/24 AS#65500 101010110101010

10101010
Baremetal

VTEP
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101
Layer-3 Point-2-Point Spine

(not Layer-3 over

VPC!)

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 140
VPC – Synchronizing the Routing
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
10.200.200.107 10.200.200.123 5000 0200.0ade.de01 2020.2323.2323 192.168.20.101 192.168.11.101
Spine

VTEP

Subnet X VTEP
VPC

192.168.11.0/24 AS#65500 101010110101010

10101010
Baremetal

VTEP
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101

Dedicated Routing Spine

Session (per-VRF)

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 141
 VPC – Advertise Subnet Individually (Advertise-PIP)
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
10.200.200.107 10.200.200.102 5000 0200.0ade.de07 0200.0ade.de02 192.168.20.101 192.168.11.101
Spine

VTEP

Subnet X VTEP
VPC

192.168.11.0/24 AS#65500 101010110101010

10101010
Baremetal

VTEP
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101

Type IP / Length L3VNI / RT Next-Hop Seq. Spine

5 192.168.11.0 /24 5000, 65500:5000 10.200.200.102

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 142
 VPC – Advertise Subnet Individually (Advertise-PIP)
SIP DIP VXLAN SMAC DMAC SIP DIP

10.200.200.107 10.200.200.102 5000 0200.0ade.de07 0200.0ade.de02 192.168.20.101 192.168.11.101 Payload

Spine
10.200.200.107 10.200.200.103 5000 0200.0ade.de07 0200.0ade.de03 192.168.20.101 192.168.11.101

VTEP

Subnet X VTEP
VPC

192.168.11.0/24 AS#65500 101010110101010

10101010
Baremetal

VTEP
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101

Type IP / Length L3VNI / RT Next-Hop Seq. Spine

5 192.168.11.0 /24 5000, 65500:5000 10.200.200.102

5 192.168.11.0/24 5000, 65500:5000 10.200.200.103

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 143
Agenda
• Introduction to Data Center Fabrics
• Leaf, Spine, Super-Spine (CLOS)
• Overlay
• Underlay

• VXLAN with BGP EVPN

• Control & Data Plane
• Multi-Tenancy
• Distributed Anycast Gateway
• VPC
• DCI

• A Deployment Story … for your reference

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
What is the Elephant in the Room?

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 145
Note sure if it is a Elephant
VXLAN for Interconnecting Networks

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 146
 VXLAN Evolves as the Control Plane Evolves!
Back Then
Yet Another Encapsulation
 Flood & Learn (Multicast-based)
 Data-Plane only Yesterday
VXLAN for the Data Center – Intra-DC
 Control-Plane
 Active VTEP Discovery Now!
 Multicast and Unicast
VXLAN for DCI – Inter-DC
 DCI Ready
 ARP/ND caching/suppress
 Multi-Homing
 Failure Domain Isolation
 Loop Protection
BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 147
 Single Fabric
Inter-X Connectivity with End-to-End
Encapsulation

• Multi-Pod Build Hierarchy in the

• aka Option 1 Underlay – Flatten it in
Multiple Fabrics • End-to-End Fabric (Stretched) the Overlay
Interconnect
using DCI
• Multi-Fabric
• aka Option 2
Multiple Fabrics – • Fabric Connected DCI (2-box) Multiple Fabrics
Normalized through with Integrated
Ethernet DCI
• Multi-Site
• aka Option 3 (Layer-2 Extension)
Integrated DCI –
• aka Option 4 (Layer-3 Extension)
Scaling within and
• Fabric Integrated DCI (1-box) between Fabrics

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 148
 Inter-X Connectivity
Multi-Pod Multi-Fabric Multi-Site

EVPN Control- BGP EVPN EVPN Control- EVPNFabric

Control-Plane EVPNFabric
Control-Plane EVPN Control-Plane BGP EVPN EVPN Control-Plane
Fabric #1 Fabric #2 #1 #2 Fabric #1 Fabric #2
Plane Domain 1 Plane Domain 2 Domain 1 Domain 2 Domain 1 Domain 2

Overlay Overlay Overlay Overlay Overlay Overlay

VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE
P P P P P P P P P P P P P P P P P P P P P P P P

Bar Bar Bar Bar

em em em em Bar Bar Bar Bar Bar Bar Bar Bar
eta eta eta eta em em em em em em em em
l l l l etal etal etal etal etal etal etal etal

DCI Data-Plane DCI

Single Data-Plane – End-to-End Data-Plane Domain 1 Data-Plane Domain 2 Data-Plane Domain 2
Data-Plane Domain 1 Data-Plane

• Multiple Fabrics – • Multiple Fabrics with

• Single Fabric with End-
Normalized through Integrated DCI
to-End Encapsulation
Ethernet • Integrated DCI –
• Build Hierarchy in the
• Multiple Fabrics Scaling within and
Underlay – Flatten it in
Interconnect using DCI between Fabrics
the Overlay
(Layer 2 and Layer 3) • The Happy Place 

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 149
Multi-Pod (using Border Leaf)

Underlay Extension
VTEP VTEP VTEP VTEP

Spine Spine Spine Spine Overlay Spine Spine Spine Spine

VTEP VTEP
10.1.1.1 VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
10.2.2.7
VTEP

Pod 1 Pod n
Baremetal Baremetal

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 150
Multi-Pod (using Super-Spine)
SuperSpine

SuperSpine SuperSpine

Underlay Extension

Spine Spine Spine Spine Overlay Spine Spine Spine Spine

VTEP VTEP
10.1.1.1 VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
10.2.2.7
VTEP

Pod 1 Pod n
Baremetal Baremetal

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 151
Multi-Pod Characteristics – ”The Single”

 Single Overlay Domain – End-to-End Encapsulation

 Single Overlay Control-Plane Domain – End-to-End EVPN Updates
 Single Underlay Domain End-to-End
 Single Replication Domain for BUM
 Single VNI Administrative Domain

Building Underlay Hierarchies – Non Hierarchical Overlay

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 152
 Multi-Fabric
L2 DCI

L2 DCI L2 DCI L2 DCI L2 DCI

L3 DCI No Underlay Extension L3 DCI
VTEP VTEP VTEP VTEP
VLAN Hand-Off VRF-Lite Hand-Off

Overlay Fabric 1
Spine Spine Spine Spine Overlay Fabric n
Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Fabric 1 Fabric n
Baremetal Baremetal

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 153
 Multi-Fabric Characteristics – ”The Separate”

• Separate Overlay Domains –Independent L2 and L3 DCI (complexity)

• Separate Overlay Control-Plane Domains – Manual Configuration
• Separate Underlay Domains - Isolated
• Separate Replication Domains for BUM – Independent BUM transport/DCI
• Dedicated Border Leaf – no local End-Point Attachment

Underlay Isolation – Separate DC Interconnection

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 154
Multi-Site
Overlay Multi-Site

No Underlay Extension
VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Overlay Site 1

Spine Spine Spine Spine Overlay Site n
Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Site 1 Site n
Baremetal Baremetal

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 155
Multi-Site Characteristics – ”The Multiple”

 Multiple Overlay Domains – Interconnected & Controlled

 Multiple Overlay Control-Plane Domains – Interconnected & Controlled
 Multiple Underlay Domains - Isolated
 Multiple Replication Domains for BUM – Interconnected & Controlled
 Multiple VNI Administrative Domains – Phase 2

Underlay Isolation – Overlay Hierarchies

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 156
Multi-Site Advantages – ”The Multiple”

 Multiple Overlay Domains – Interconnected & Controlled

• Scaling and Segregating VXLAN EVPN Networks
 Multiple Overlay Control-Plane Domains – Interconnected & Controlled
• Limited Overlay Control-Plane Update Propagation
 Multiple Underlay Domains - Isolated
• Isolated Underlay Domains – No need for Extension
 Multiple Replication Domains for BUM – Interconnected & Controlled
• Individual BUM flooding domain with Traffic control

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 157
Summary
Summary
• Multi-Tier Topologies based on Leaf and Spines (aka Clos)
• Overlays (VXLAN) for Network Virtualization
• Layer-3 in the Underlay – Defines the Topology
• Layer-2 and Layer-3 in the Overlay – Defines the Services
• End-Points State exists in the Overlay

• VXLAN evolved as the Control-Plane evolved

• Applicability changes over time – VXLAN EVPN Multi-Site for DCI
• BGP EVPN for integrated Layer-2 and Layer-3 Services
• Control-Plane driven
• Optimal Routing and Bridging
• Avoid hair pinging and reduced failure domains

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
 Complete Your Online
Session Evaluation
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 gift card.
• Complete your session surveys
through the Cisco Live mobile
app or on www.CiscoLive.com/us.

Don’t forget: Cisco Live sessions will be

available for viewing on demand after the
event at www.CiscoLive.com/Online.

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 161
Thank you
Agenda
• Introduction to Data Center Fabrics
• Leaf, Spine, Super-Spine (CLOS)
• Overlay
• Underlay

• VXLAN with BGP EVPN

• Control & Data Plane
• Multi-Tenancy
• Distributed Anycast Gateway
• VPC
• DCI

• A Deployment Story … for your reference

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
A Deployment Story
Story #1: Scalable Data Center Fabric

• VXLAN based Data Center Fabric

• BGP EVPN Control-Protocol (Overlay)
• OSPF for Underlay Routing (Unicast)
• PIM ASM with Anycast-RP for BUM Replication (Underlay)
• Distributed IP Anycast Gateway

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 166
 p2p Agg: 10.1.1.0/24
RID Agg: 10.10.10.0/24
A Deployment Story VTEP Agg: 10.200.200.0/24
RP Agg: 10.254.254.0/24

Spine Spine Spine Spine

Underlay
Leaf Leaf Leaf Leaf Leaf Leaf Leaf

Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 167
 p2p Agg: 10.1.1.0/24
RID Agg: 10.10.10.0/24
A Deployment Story VTEP Agg: 10.200.200.0/24
RP Agg: 10.254.254.0/24
interface loopback0 interface loopback0
ip address 10.10.10.101/32 ip address 10.10.10.201/32
ip router ospf UNDERLAY area 0.0.0.0 ip router ospf UNDERLAY area 0.0.0.0
interface loopback0
router ospf UNDERLAY
ip address 10.10.10.102/32 router ospf UNDERLAY
router-id
ip router 10.10.10.101
ospf UNDERLAY area 0.0.0.0 Spine Spine Spine Spine router-id 10.10.10.201
interface loopback0
interface
router Ethernet1/1
ospf
ip addressUNDERLAY
10.10.10.103/32 interface Ethernet1/1
mturouter-id
9192
ip router10.10.10.102
ospf UNDERLAY area 0.0.0.0 mtu 9192
ip address 10.1.1.1/30
ipinterface
ospf
router network
ospf point-to-point
Ethernet1/1
UNDERLAY
Underlay ip address 10.1.1.2/30
ip ospf network point-to-point
ip mtu
router
9192ospf 10.10.10.103
router-id UNDERLAY area 0.0.0.0 ip router ospf UNDERLAY area 0.0.0.0
ip ip
pimaddress
sparse-mode
10.1.1.5/30 ip pim sparse-mode
… ipinterface
ospf network point-to-pointLeaf
Ethernet1/1 Leaf Leaf Leaf Leaf Leaf Leaf
ip mtu
router
9192ospf UNDERLAY area 0.0.0.0 interface Ethernet1/2
ip ip
pimaddress
sparse-mode
10.1.1.9/30 mtu 9192
… ip ospf network point-to-point ip address 10.1.1.6/30
ip router ospf UNDERLAY area 0.0.0.0 ip ospf network point-to-point
ip pim sparse-mode ip router ospf UNDERLAY area 0.0.0.0
… ip pim sparse-mode

interface Ethernet1/3
Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal
mtu 9192
ip address 10.1.1.10/30
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
…
BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 168
 p2p Agg: 10.1.1.0/24
RID Agg: 10.10.10.0/24
A Deployment Story VTEP Agg: 10.200.200.0/24
RP Agg: 10.254.254.0/24
interface loopback0 interface loopback0
ip address 10.10.10.202/32 ip address 10.10.10.203/32
ip router ospf UNDERLAY area 0.0.0.0 ip router ospf UNDERLAY area 0.0.0.0

interface loopback254 interface loopback254

ip address 10.254.254.1/32 ip address 10.254.254.1/32
Spine Spine Spine Spine
ip router ospf UNDERLAY area 0.0.0.0 ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode ip pim sparse-mode

ip pim anycast-rp 10.254.254.1 10.254.254.202

ip pim anycast-rp 10.254.254.1 10.254.254.203 Underlay ip pim anycast-rp 10.254.254.1 10.254.254.202
ip pim anycast-rp 10.254.254.1 10.254.254.203

ip pim rp-address 10.254.254.1 ip pim rp-address 10.254.254.1

Leaf Leaf Leaf Leaf Leaf Leaf Leaf

interface loopback0
ip address 10.10.10.103/32
ip router ospf UNDERLAY area 0.0.0.0
interface loopback0
ip address 10.10.10.102/32
ip pim rp-address 10.254.254.1
ip router ospf Baremetal
UNDERLAY Hypervisor
area 0.0.0.0
Hypervisor Baremetal
interface
Hypervisor
loopback0
Hypervisor Baremetal Baremetal

ip address 10.10.10.101/32
ip pim rp-address 10.254.254.1
ip router ospf UNDERLAY area 0.0.0.0

ip pim rp-address 10.254.254.1

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 169
 p2p Agg: 10.1.1.0/24
RID Agg: 10.10.10.0/24
A Deployment Story VTEP Agg: 10.200.200.0/24
RP Agg: 10.254.254.0/24

Spine Spine Spine Spine

Underlay
Leaf Leaf Leaf Leaf Leaf Leaf Leaf

interface loopback1
ip address 10.200.200.103/32
interface loopback1
ip router ospf UNDERLAY area 0.0.0.0
ip address 10.200.200.102/32
interface loopback1
ip routerinterface
ospf UNDERLAY nve1 area 0.0.0.0
ip address 10.200.200.101/32
Hypervisor Baremetal Hypervisor Hypervisor
source-interface
Baremetal
loopback1
Hypervisor Baremetal Baremetal

ip routerinterface
ospf UNDERLAY nve1 area 0.0.0.0
host-reachability protocol bgp
source-interface loopback1
interface nve1 host-reachability protocol bgp
source-interface loopback1
host-reachability protocol bgp

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 170
 p2p Agg: 10.1.1.0/24
RID Agg: 10.10.10.0/24
A Deployment Story VTEP Agg: 10.200.200.0/24
RP Agg: 10.254.254.0/24
router bgp 65500 router bgp 65500
router-id 10.10.10.202 router-id 10.10.10.203
neighbor 10.10.10.0/24 remote-as 65500 neighbor 10.10.10.0/24 remote-as 65500
update-source loopback0 update-source loopback0
address-family l2vpn evpn address-family l2vpn evpn
send-community both send-community both
route-reflector-client Spine Spine Spine Spine
route-reflector-client

Underlay
Leaf Leaf Leaf Leaf Leaf Leaf Leaf
router bgp 65500
router bgp router-id
65500 10.10.10.103
router-id neighbor
10.10.10.102 10.10.10.202 remote-as 65500
router bgp 65500
update-source loopback0
router-id neighbor
10.10.10.101 10.10.10.202 remote-as 65500
update-source address-family
loopback0 l2vpn evpn
neighbor 10.10.10.202 remote-as 65500
address-family send-community both
update-source loopback0 l2vpn evpn
send-communityneighbor 10.10.10.203 remote-as 65500
address-family l2vpn evpn both
neighbor update-source
10.10.10.203 loopback0
remote-as 65500
Hypervisor Baremetal send-community
Hypervisor both
Hypervisor Baremetal Hypervisor Baremetal Baremetal

update-source address-family
loopback0 l2vpn evpn
neighbor 10.10.10.203 remote-as 65500
address-family send-community both
update-source loopback0 l2vpn evpn
address-family l2vpn evpn both
send-community
send-community both

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 171
 p2p Agg: 10.1.1.0/24
RID Agg: 10.10.10.0/24
A Deployment Story VTEP Agg: 10.200.200.0/24
RP Agg: 10.254.254.0/24
vlan 100
vn-segment 30001
name Blue
vlan 200
vn-segment 30002
name Green
Spine Spine Spine Spine

evpn
vni 30001
Overlay rd auto
route-target both auto
vni 30002
rd auto
route-target both auto
Leaf Leaf Leaf Leaf Leaf Leaf Leaf

interface nve1
source-interface loopback1
host-reachability protocol bgp
member vni 30001
mcast-group 239.239.239.1
member vni 30002
mcast-group 239.239.239.2
Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 172
 p2p Agg: 10.1.1.0/24
RID Agg: 10.10.10.0/24
A Deployment Story VTEP Agg: 10.200.200.0/24
RP Agg: 10.254.254.0/24
interface Vlan100 vrf context VRF-A
mtu 9192 vni 50001
vrf member VRF-A rd auto
ip address 192.168.1.1/24 tag 21921 address-family ipv4 unicast
fabric forwarding mode anycast-gateway route-target both auto
route-target both auto evpn
interface Vlan200 Spine Spine Spine Spine
address-family ipv6 unicast
mtu 9192 route-target both auto
vrf member VRF-A route-target both auto evpn
ip address 10.10.10.1/24 tag 21921
fabric forwarding mode anycast-gateway Overlay interface nve1
source-interface loopback1
host-reachability protocol bgp
member vni 50001 associate-vrf
Leaf Leaf Leaf Leaf Leaf Leaf Leaf

router bgp 65500

vrf VRF-A
address-family ipv4 unicast
advertise l2vpn evpn
redistribute direct route-map TAG

Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor route-map

Baremetal TAG Baremetal
permit 10
match tag 21921

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 173
 p2p Agg: 10.1.1.0/24
RID Agg: 10.10.10.0/24
A Deployment Story VTEP Agg: 10.200.200.0/24
RP Agg: 10.254.254.0/24
interface Ethernet 2/1.10
vrf member VRF-A interface Ethernet 1/15.21
ip address 172.16.0.1/30 vrf member VRF-A
encapsulation dot1q 5 ip address 172.16.0.2/30
encapsulation dot1q 5
interface Ethernet 2/1.20
vrf member VRF-B Spine Spine Spine Spine
interface Ethernet 1/15.22
ip address 172.16.0.1/30 vrf member VRF-B
encapsulation dot1q 6 ip address 172.16.0.2/30
router bgp 65500 Overlay encapsulation dot1q 6
vrf VRF-A router bgp 65599
address-family ipv4 unicast vrf VRF-A
advertise l2vpn evpn address-family ipv4 unicast
Leaf Leaf Leaf Leaf Leaf Leaf Leaf
aggregate-address 10.10.10.0/24 summary-only neighbor 172.16.0.1 remote-as 65500
aggregate-address 192.168.1.0/24 summary-only update-source Ethernet1/15.21
neighbor 172.16.0.1 remote-as 65599 address-family ipv4 unicast
update-source Ethernet2/1.10 …
address-family ipv4 unicast
…

Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal

Subnet B
192.168.20.0/24

BRKDCN-3378 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 174