BRKDCN 2933

#CiscoLive
Next Generation DCI

EVPN/VXLAN Multisite A Way To Retire Old
Technologies
Stephen McCabe, Technical Solutions Architect
BRKDCN-2933
#CiscoLive
Cisco Webex App
Questions?
Use Cisco Webex App to chat
with the speaker after the session
How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install the Webex App or go directly to the Webex space Enter your personal notes here
4 Enter messages/questions in the Webex space
Webex spaces will be moderated

by the speaker until June 9, 2023. https://ciscolive.ciscoevents.com/ciscolivebot/#BRKDCN-2933
#CiscoLive BRKDCN-2933 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
• Introduction
• What is EVPN Multi-Site?
• Use cases
• Multi-Site with DCI – A
Deeper Look
• Migration from Legacy to
new EVPN/VXLAN Fabric
Agenda • Failure Scenarios
• Automation and Observability
with Nexus Dashboard
• Conclusion
BRKDCN-2933 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Abstract
VXLAN is a widely adopted industry standard for encapsulation, and with MP-BGP,
EVPN provides extensive capabilities as a control-plane. With VXLAN and EVPN, we
have excellent capabilities for Data Center fabric deployments with an integrated
Layer-2 and Layer-3 approach. With the maturity of the control and data planes, new
capabilities for interconnecting multiple fabrics are experiencing growing interest with
VXLAN BGP EVPN. The goal of the session is to provide a better understanding of how
VXLAN EVPN Multi-Site architecture is a modern alternative to DCI technologies such
as OTV, VPLS, or EoMPLS, especially for interconnecting data center networks that
are solely built on legacy technologies (for example, STP, vPC, or Cisco FabricPath).
Important Note: The session is exclusively focused on NX-OS standalone VXLAN

EVPN and does not discuss the multi-pod and multi-site solutions offered with Cisco
ACI.
Introduction
Introduction
• A brief touchpoint of the work at the IETF (Internet Engineering Task
Force) and what RFC (Request for Comment) are Standard and
what Informational
• What is VXLAN EVPN Multisite?
• Use Cases – Focus on Enabling Migration Off Legacy Technologies
• Migration/Deployment Scenarios
• The Border Gateway (BGW)
• Automation and Observability
What is
Multisite?
RFC 9014
By the Standards Body
• Internet Engineering Task • RFC 9014

Force (IETF) Request for • https://datatracker.ietf.org/do
Comment (RFC) c/html/rfc9014
• Categorized for Standards

Track
• Internet Standard since
2021
• Existing Industry Adoption
• Interconnect Solution for
Ethernet VPN (EVPN)
Overlay Networks
• Co-Authored by Cisco
RFC 9014 - Overview
• DCI EVPN Overlay (aka RFC 9014)
• Interconnect Solution for Ethernet VPN (EVPN) Overlay Networks
• From the Abstract “extend the Layer 2 connectivity required for some tenants.”
BGP Autonomous System (AS) BGP Autonomous System (AS)

65001 65002
VXLAN BGP VXLAN BGP

EVPN EVPN
GW eBGP EVPN* GW
Address-Family
// Layer-2
//
// = tunnel stitching point at GW // = tunnel stitching point at GW
*RFC 9014 supports more than just EVPN for the Interconnect Network
RFC 9014 Gateway Model Side-by-Side
Decoupled and Integrated Gateway
Decoupled Gateway (Section 3) Integrated Gateway (Section 4)

BGP AS BGP AS
65001 65001
VXLAN BGP EVPN VXLAN BGP EVPN

Gateway Gateway
WAN Edge
VLAN Handoff
Layer-2 EVPN* Layer-2 EVPN*
BGP AS BGP AS
65002
WAN Edge
VLAN Handoff 65002
Gateway Gateway
RFC 9014 Gateway Model Side-by-Side
Decoupled and Integrated Gateway
Decoupled Gateway (Section 3) Integrated Gateway (Section 4)

BGP AS BGP AS
65001 65001

Gateway Gateway
WAN Edge
VLAN Handoff
Layer-2 EVPN* Layer-2 EVPN*
BGP AS BGP AS
65002
WAN Edge
VLAN Handoff 65002
Gateway Gateway
What about Layer-3?

Multi-Site Solution for
Ethernet VPN (EVPN)
Overlay
draft-sharma-bess-multi-site-evpn
What is Multi-Site?
• Internet Engineering Task • Multi-Site (BESS version)

Force (IETF) Request for • https://datatracker.ietf.org/doc/
Comment (RFC) html/draft-sharma-bess-multi-
site-evpn
• Categorized as Informational
• Pre-Cursor Draft (replaced
• Internet Draft since 2016 by BESS version)
• Currently in Version 3 • https://datatracker.ietf.org/doc/
• Overall, 7 versions html/draft-sharma-multi-site-
evpn
• Updated and Maintained by
BESS version of draft
• draft-sharma-bess-multi-site-
evpn
• Shipping since 2017
Multi-Site
• Multi-Site Solution for Ethernet VPN (EVPN) Overlay (draft-sharma-bess-multi-site-evpn)
• Interconnect Solution for Ethernet VPN (EVPN) Overlay Networks
• From the Abstract “support extension of Layer-2 and Layer-3, Unicast & Multicast, VPNs”

65001 65002
VXLAN BGP VXLAN BGP

EVPN eBGP EVPN EVPN
BGW Address-Family BGW
// Layer-2 & Layer-3

//
// = tunnel stitching point at BGW // = tunnel stitching point at BGW
RFC9014 and Multi-Site - Side by Side
DCI-EVPN-Overlay Multi-Site EVPN
(RFC 9014) (draft-sharma-bess-multi-site-evpn)
Interconnect Integrated (1-Box), Decoupled (2-Box) Integrated (1-Box)
DCI Encap VPLS, PBB-VPLS, EVPN-MPLS, PBB-EVPN, VXLAN VXLAN
Gateway Mode Multipath PIP Anycast VIP Multipath PIP
ECMP Underlay and Overlay Underlay Underlay and Overlay
EVPN RT-1 Consumed and Generated None Consumed and Generated
EVPN RT-2 Re-Originated with I-ESI Re-Originated with ESI 0 Re-Originated with I-ESI
EVPN RT-3 Consumed and Generated Consumed and Generated Consumed and Generated
EVPN RT-4 Consumed and Generated Consumed and Generated Consumed and Generated
EVPN RT-5 - (not part of RFC) Re-Originated Re-Originated
Route Distinguisher (RD) Separate RD for Intra and Inter DC Separate RD for VIP and PIP
Route-Target (RT) Separate RT for Intra and Inter DC Same RT for Intra and Inter DC
VNI Allocation Global and Downstream Global and Downstream
DF Election Based on EVPN RT-4 Based on EVPN RT-4
Identifier I-ESI I-ESI (= Site-ID)
Split Horizon Local Bias Local Bias
ESI-Type Type 0 (Operator Managed) Type 3 (MAC Based) or Type 5 (AS based)
BUM Tree # 2, GW stitched (Intra and Inter DC) 2, GW stitched (Intra and Inter DC)
RFC9014 and Multi-Site – Side by Side
In a Nutshell
BGP Autonomous System (AS) BGP Autonomous System (AS) BGP Autonomous System (AS) BGP Autonomous System (AS)
65001 65002 65001 65002
VXLAN BGP EVPN eBGP VXLAN BGP EVPN VXLAN BGP EVPN eBGP VXLAN BGP EVPN
GW EVPN GW GW EVPN GW
Address-Family Address-Family
// Layer-2
// // Layer-2 & Layer-3
//
RFC 9014 Multi-Site

Base Standard for Interconnecting EVPN Extends RFC 9014 for Interconnecting EVPN
Defines the Layer-2 Stitching Describes Layer-2 and Layer-3 Stitching
Two Gateway Model Single Gateway Model (Two BGW* Model)
Multiple Encapsulations Focus only on VXLAN Encapsulation
Leverages Overlay and Underlay ECMP Different ECMP model depending on BGW Model
*BGW – Border Gateway (BGW); Cisco’s name for the VXLAN EVPN to VXLAN EVPN Gateway
EVPN Multisite
Use Cases
Use Cases - Overview
VXLAN EVPN Multi-Site architecture is a design for VXLAN BGP
EVPN–based overlay networks. It allows interconnection of multiple
distinct VXLAN BGP EVPN fabrics or overlay domains, and it allows
new approaches to fabric scaling, compartmentalization, and DCI.
Use cases for EVPN Multisite:
• Compartmentalization
• Hierarchical scale-out approaches
• DCI
Areas of Focus
• Integration of legacy networks
Use Case #1: Compartmentalization
• Multiple Fabrics, single Data
DC Core / Super Spine
Center
• Single or Multiple Data Halls
S S S S
• Within a Geographic
Locations
• Control at BGW (Border
B B B B Gateway)
Fabric #1 Fabric #2
• Allows Extension of Layer-2
L L L L L L • Allows Extension of Layer-2
and Layer-3
Server Server Server Server • Allows Traffic Control
(BUM*)
• Defines VNI allocation and
stitching
• Optimizes BUM* Replication
BUM Optimization
Use Case #1 – Compartmentalization
Single Fabric BUM with Ingress Replication Multi-Site BUM with Ingress Replication
DC Core / Super Spine DC Core / Super Spine
S S S S S S S S
S S S S B B B B
Fabric #1 Fabric #2 Fabric #1 Fabric #2
L L L L L L L L L L L L
Server Server Server Server Server Server Server Server
Single BUM Packet, 5x Replicated Single BUM Packet, 3x Replicated

3 Replication over DC Core / Super Spine (Between) 1 Replication over DC Core / Super Spine (Between)
2 Replication for Fabric #1 (Local) 3 Replication for Fabric #1 (Local)
3 Replication for Fabric #2 (Local)
*BUM – Broadcast, Unknown Unicast, Multicast
Use Case #2 - Scale
• Multiple Fabrics , single or multiple Data
Center
• Single or Multiple Data Halls
S S S S
• Within or between Geographic Locations
• Control at BGW (Border Gateway)
• Reduces Remote VTEP Count
Up
B to 128
B Sites per Multi-SiteBDomain
B • Expands VTEP scale
Fabric #1 Fabric #128
• Scale through Hierarchy
• Multiply VTEP with Sites
L L L L L L
*TRM upto 15 sites
Up to 256 VTEP per Fabric Up to 256 VTEP per Fabric
Server Server Server Server *Number of BGWs per site 6 (Anycast), 2 (vPC)
32’768 VTEP to extend Layer-2 or/and Layer-3 segments to
VTEP Scale
Use Case #2 - Scale
Single Fabric or Multi-POD Multiple Fabric with Multi-Site

S S S S S S S S
S S S S B B B B
Fabric #1 Fabric #2 Fabric #1 Fabric #2
L L L L L L L L L L L L
Server Server Server Server Server Server Server Server
Leaf #1 sees every VTEP, 5 VTEP Peer Leaf #1 sees only local VTEP, 3 VTEP Peer
3 VTEP Peer for Fabric #2 (Between) 1 VTEP Peer for Exit, BGW (Between)
2 VTEP Peer for Fabric #1 (Local) 2 VTEP Peer for Fabric #1 (Local)
Use Case #3 – Data Center Interconnect (DCI)
• Multiple Fabrics, Geographically
Dispersed
• Classic DCI Use Case
B B
L L L
• Allows Extension of Layer-2 and Layer-3
B B
• Allows Traffic Control (BUM*)
L L L
• Defines VNI allocation and stitching
• Optimizes BUM* Replication
B B
B B
L L L
L L L
Works Within a Geographic Location – Works Between Geographic Locations

Use Case #4 – Integration with Legacy Networks
• Integrating Fabrics with Legacy
Networks
• BGW Frontends Legacy Network
S S S S
• BGW Frontends New Network
• Host Mobility and Migration
• Provides Distributed Default Gateway
B B B B • Allows Layer-2 Extension where needed
Fabric #1
• Benefits from all Multi-Site functions
Agg Agg
• Layer-2, Layer-3 Multicast and Unicast
L L L VPNs between different Networks for
Access Access Access Migration or Co-Existance
Server Server
Server Server
Much more on this shortly!
Multisite and the Role of
the Border Gateway
A Deeper Look
As we Talk about Scale
Hardware Support
Minimum Hardware and Software Requirements for BGW (Border Gateway)

Cisco Nexus 9300 EX platform
Cisco Nexus 9300 FX platform
Cisco Nexus 9300 FX2 platform
Cisco Nexus 9300 FX3 platform
Cisco Nexus 9300 GX platform
Cisco Nexus Hardware Cisco Nexus 9300 GX2 platform
Cisco Nexus 9364C platform
Cisco Nexus 9332C platform
Cisco Nexus 9500 platform with X9700-EX line card
Cisco Nexus 9500 platform with X9700-FX line card
Cisco Nexus 9500 platform with X9700-GX line card
Cisco Nexus Software (NX-OS) Cisco NX-OS Software Release 7.0(3)I7(1) or later*
*Check for Hardware Specific Support Releases
As we Talk about Scale
Scalability Values as of NX-OS 10.2(5)M
Multi-Site Scale
Number of Sites 128
Number of BGW per Site 6
Number of VTEP per Site (internal) 256
Border Gateway (BGW) Scale EX FX2 FX,FX3,GX,GX2 N9364C & N9332C

Number of Layer-2 VNI (VLAN) 3900
Number of Layer-3 VNI (VRF) 2000
MAC per BGW 92k
IPv4 Host Routes per BGW* 450k 450k 1.1M 96k
IPv4 Network Routes per BGW* 450k 450k 1.1M 8k
IPv6 Host Routes per BGW* 24k 260k 620k 48k
IPv6 Network Routes per BGW* 200k 290k 620k 2k
*The values provided in these tables focus on the scalability of one particular Route scale at a time
Some Notes on BGW and VXLAN Tunnels
Multi-Site
• Tunnels are Stitched at the BGW (Border Gateway)
• Intra Fabric Tunnel goes from Leaf to Leaf or Leaf to BGW
• Inter Fabric Tunnel goes from BGW to BGW

65001 65002
VXLAN BGP VXLAN BGP

EVPN eBGP EVPN EVPN
BGW Address-Family BGW
// Layer-2 & Layer-3

//
// = tunnel stitching point at BGW // = tunnel stitching point at BGW
Some Notes on the Interconnect and Underlay
Multi-Site
• Fabric #1 Underlay (VTEP, Point-2-Point, Loopback etc) is not aware of Fabric #2
• Each Fabric maintains their Unique Network Topology, Protocols and IP Addressing
• Only BGW IP Addressing must be Unique and Aligned between Sites

65001 65002
VXLAN BGP VXLAN BGP

EVPN EVPN Leaf
Leaf BGW BGW
Leaf Leaf
Leaf Leaf
Fabric #1 Underlay Fabric #2 Underlay

Leaf: Multi-Site Underlay Leaf:
10.1.1.1 10.2.2.1
BGW Fabric#1: BGW Fabric#2:
10.1.1.2 10.2.2.2
10.0.1.1 10.0.2.1
10.1.1.3 10.2.2.3
10.0.1.2 10.0.2.2
10.1.1.4 10.2.2.4
10.0.1.3 10.0.2.3
10.1.1.5 10.2.2.5
10.1.1.6 10.2.2.6
10.1.1.7 10.2.2.7
Border Gateway
Details
Border Gateways Deployment Considerations
Anycast Border
Leaf Gateway
Anycast Spine B B B vPC Spine
Border Gateway L L L Border Gateway
B B B B B B
S S S S
Fabric Fabric Fabric
L L L L L L L L L
• Border Gateways used for two main functions:

• Interconnecting each site to the Inter-Site network (for East-West traffic flows)
• Connecting each site to the external Layer 3 domain (for North-South traffic flows)
• NOTE: May also be used to connect endpoints and/or network service nodes (FWs, ADCs)
• Possible deployment models:
• Anycast Border Gateways
• vPC Border Gateways
• BGW function enablement in the VXLAN EVPN fabric:
• BGWs on Leaf node (Border Gateway Leaf)
• BGWs on Spine node (Border Gateway Spine)
Anycast Border Gateway
• Up to 6 Border Gateways
• Border Gateway
• Deploying as a Leaf node since release 7.0(3)I7(1)
• Deploying as a Spine node since release
7.0(3)I7(2)
B B B B • Two Modes of Operation:

• Can Operate as Multi-Site Anycast BGW with VIP
Fabric • Focuses on Scale and Convergence
L L L • Using Virtual IP (VIP) for Tunnel Stitching
• Uses Overlay ECMP
• Can Operate in RFC 9014 BGW Mode with PIP
• Focuses on 3rd Party Interop
• Using Primary IP (PIP) for Tunnel Stitching
• Uses Underlay and Overlay ECMP
vPC Border Gateway
vPC Border Gateway
• Up to 2 Border Gateways
• Border Gateway
• Deploying as a Leaf node since 9.2(1)
• Common Use Case

B B • Legacy Network Integration or Migration
Fabric
• Provides Multi-Chassis Link Aggregation
• Integrates with Ethernet and FabricPath
L L L
• Hosts the Distributed Anycast Gateway
• Attachment of Network Services
• Dual-Attachment of Firewalls and ADCs
• Acts like a vPC when it comes to Routing
When to use what BGW
B B B B B B
Fabric Fabric
L L L L L L
Anycast Border Gateway vPC Border Gateway

• Up to 6 BGW • 2 BGW with physical vPC Peer-Link
• Share Nothing • Small Deployments
• Simple Failure Scenarios • End-Point or Network Services
• Any Deployments Connectivity on BGW
• No End-Point or Network Services • Migration Use-Cases (Brownfield)
Connectivity on BGW • Classic Ethernet/FabricPath to VXLAN
• Greenfield Deployments EVPN
vPC Border Gateways
The Details
Details on the Different BGW
• Both Anycast and vPC Border Gateway needs to be configured with a common Multi-Site VIP
address and an individual Primary IP (PIP) address
• vPC Border Gateways share a secondary IP address to be used as vPC virtual IP (vPC VIP)
Multi-Site VIP
10.0.2.1
Multi-Site VIP vPC VIP
vPC Border Gateway

10.0.1.1 10.2.2.254
PIP PIP PIP PIP PIP PIP

10.1.1.1 10.1.1.2 10.1.1.3 10.1.1.4 10.2.2.1 10.2.2.2
B B B B B B
Fabric Fabric
L L L L L L
VXLAN EVPN Multi-Site with vPC BGW considerations
What’s What?
vPC BGWs’ logical interfaces:

• Unique logical interfaces (for example, vPC VIP1
loopback interfaces) must be defined on the B11.11.11.11
Multi-site VIP
vPC BGW devices to perform their duties
C100.100.100.1
00
B
interface loopback0 PIP PIP
description CP IP or RID A 10.1.10.1 10.1.20.1
ip address 10.1.1.1/32 tag 54321
! CP IP1 B B CP IP2
interface loopback1 A 10.1.1.1 10.1.2.1
description PIP1 VTEP
D
ip address 10.1.10.1/32 tag 54321 B
ip address 11.11.11.11/32 secondary tag 54321
!
interface loopback100
description Multi-Site VIP1 M-Site 1
ip address 100.100.100.100/32 tag 54321
C
! L
interface nve1 VTEP
host-reachability protocol bgp
source-interface loopback1 D
multisite border-gateway interface loopback100
What are the used for?
Control Plane IP address (CP IP):
• Used for control plane adjacencies for the
MP-BGP EVPN overlay with the remote BGW
devices.
SRC DST
Primary IP address (PIP): vPC VIP1 vPC VIP2 VXLAN Header Original Packet
• Unique IPs per BGW used to source traffic
originated from devices connected via Layer
3 and used to receive traffic from remote Inter-site
sites. North-South Traffic Network
vPC Virtual IP address (vPC VIP): vPC VIP1 vPC VIP2

11.11.11.11 22.22.22.22
• Secondary IP defined on both BGWs part of the
B B B B
same vPC Domain used for two purposes:
1. Sourcing BUM traffic for Layer 2
networks stretched to remote site(s)
2. Sourcing/receiving traffic for single- or M-Site 1 M-Site 2
dual-attached endpoints locally
connected at Layer 2 to the BGWs
Things to Think About
Multi-Site Virtual IP address (Multi-Site VIP):
• IP address on a dedicated loopback defined on

SRC DST
both BGW nodes that are part of the same vPC
Multi-site VIP1 Multi-site VIP2 VXLAN Header Original Packet
domain.
• IP address is used to source traffic destined to
remote sites and originated from endpoints
connected behind a leaf node in the local site. Inter-site
The same IP address is also used to receive Network
traffic originating from remote sites and
destined to endpoints connected behind a leaf vPC VIP1 vPC VIP2
11.11.11.11 22.22.22.22
node in the local site
B B B B
M-Site 1 M-Site 2
L L
VTEP VTEP
DCI & vPC Border
Gateways Connectivity
and Migration
A deeper look
1. Common Control plane & Data
plane
Architectural 2. Integrated Layer 2 and Layer 3

extension
Benefits of 3. Fault Containment
Introducing vPC 4. Transport Agnostic
Border Gateways 5. Multihoming
6. Multipath Load Sharing
7. Loop Prevention and STP
Isolation
8. Support for Multiple Sites
vPC Border Gateway Use
Cases
Integration with Legacy Networks
Distributed Anycast Gateway
Primary Use cases
• vPC BGW attached to the existing legacy network providing interconnect with a remote network
• Enabling migration of Legacy fabric workloads to a modern fabric built with VXLAN EVPN (DCI
Multisite)
• The vPC BGWs use a Distributed Anycast Gateway (DAG) to provide a consistent first-hop gateway.
This coupled with new EVPN/VXLAN fabric we can extended the anycast GWs to be available across
each fabric
S S S S
Distributed
B B B B
B B B B Anycast Gateway
Fabric #1
Agg Agg Agg Agg
L L L L
Access Access Access Access Access Access
Server Server
Server Server Server Server
EVPN/VXLAN Legacy Site Legacy Site

Fabric
Server
VXLAN EVPN Multi-Site with vPC BGWs
vPC BGW Use Case: #1 Legacy Site to VXLAN/EVPN Fabric
Capabilities/Benefits Achieved
• Integration/coexistence of a legacy site with a
VXLAN BGP EVPN site with EVPN Multi-Site VXLAN
• Provides ability to migration workloads to DC Core / Super Spine
EVPN/VXLAN Fabrics S S S S
• STP Configurations STP Root
• vPC BGW should be configured as STP
Root
• Best Practice is to configure STP Root-
Guard on VPC Connections between
BGWs and Legacy Network B B B B B B
Fabric #1 Enable
STP Root Agg Agg
L L L L Guard
Access Access Access Access
Server Server
Server Server
FW LB
EVPN/VXLAN
Fabric Legacy Site
Server
*Use case targets small-fabric deployments
vPC BGW Use Case: #1 Services Considerations
VIP Requested Traffic
Capabilities/Benefits Achieved
• Integration/coexistence of a legacy site
with a VXLAN BGP EVPN site with EVPN VXLAN
Multi-Site DC Core / Super Spine
• Provides ability to migration workloads to S S S S
EVPN/VXLAN Fabrics
• Considerations for Services
B B B B B B
Fabric #1
Agg Agg
L L L L
Server Server
Server Server
FW LB
EVPN/VXLAN
Fabric Legacy Site
Server
vPC BGW Use Case: #1 Service Migration
VIP Requested Traffic
• Load Balancer VIP/server
migration
• DNS
• Stateful firewalls DC Core / Super Spine
• PBR (Policy Based Routing) S S S S
• Elastic Service Redirection

• ePBR and ITD
• Is FHRP for hosts the Load
Balancer or FW? Options..
B B B B B B
• Stretch Cluster
• Migrate FHRP Fabric #1
Agg Agg
L L L L
Server Server
FW LB
Server Server
EVPN/VXLAN
Fabric Legacy Site
Server
Use Case #2 Small Site connectivity
Use Cases:
• Multisite connectivity for smaller VXLAN
EVPN/VXLAN sites DC Core / Super Spine
• Cost effective vs. deploying S S S S
dedicated Anycast BGWs
B B B B
FW LB FW LB
Spine Spine Spine Spine
Leaf Leaf Leaf Leaf Leaf Leaf
Small EVPN Site 1 Small EVPN Site 2
Migrating Away From Legacy
Using vPC Border Gateways
Migrating legacy to VXLAN EVPN fabrics using vPC BGWs
Steps involved
Step 1: Insert a pair of vPC BGWs in each legacy site, using Layer 2 double-sided vPC
Step 2: Configure vPC BGWs DCI underlay network
Step 3: Configure vPC BGWs DCI overlay network
Step 4: Configure vPC BGWs for DCI Layer 2 extension across sites
Step 5: Enable Anycast Gateway on vPC BGWs and keep it in shutdown state
Step 6: Migrate first-hop FHRP Gateway in the legacy site to the vPC BGW Anycast Gateway
Step 7: Transition legacy data centers to new Data Center Fabric
Step 1: Insert Pair of BGWs into Each Legacy Site
• If existing DC Aggregation devices support VPC/mLAG configure with double-sided VPC

• Double-sided VPC provides for active/active paths and removes need for STP to block paths
• *NOTE: When the aggregation switches do not support vPC or MLAG, local port-channels can be created from each
aggregation switch and the pair of vPC BGW nodes

STP Root
Double
R R R R
Sided VPC
B B B B Enable
B B B B STP Root
Enable Guard
Fabric #1 STP Root Agg Agg Agg Agg
Guard
L L L L
Server Server
EVPN/VXLAN Legacy Site Legacy Site

Fabric
Server
Step 1 – Cont’d: If Legacy Devices Don’t support VPC/mLAG
• If aggregation switches don’t support vPC/MLAG, local port-channels

can be created from each aggregation switch and the pair of vPC
BGW nodes
• STP block the Layer 2 loop created between the aggregation switches
and the BGWs – STP root will be on the vPC BGWs
R R R R STP Root R R R R STP Root
B B B B
Enable Enable
STP Root STP Root
Guard Agg Agg Guard Agg Agg
Layer 2 Legacy Site

Legacy Site Backdoor
Step 1: Configuration
• Define the vPC domain and properly tune the delay-restore and the reload-delay timers to optimize
convergence after a vPC peer reload event.
• Establish iBGP peering relationship along with associated IGP peering (OSPF, ISIS, etc.)
vlan 3600
interface Vlan3600
feature vpc
R R R R description VPC-Peer-Link SVI
no shutdown
vpc domain 1
mtu 9216
peer-switch
no ip redirects
peer-keepalive destination 172.19.217.122
ip address 10.1.10.49/30
source 172.19.217.123
no ipv6 redirects
delay-restore 150
B B ip ospf network point-to-point
peer-gateway
ip router ospf UNDERLAY area 0.0.0.0
auto-recovery reload-delay 360
ip pim sparse-mode
ipv6 nd synchronize Agg Agg
ip arp synchronize
system nve infra-vlans 3600
Access Access Access
interface port-channel10
router bgp 65501
vpc peer-link Server Server neighbor 10.1.10.50
remote-as 65501
Legacy Site address-family ipv4 unicast
• EVPN Multi-Site interface tracking is required on

the interface(s) connecting to the external Layer 3
AS 65099 core to detect the scenario where a given vPC
BGW node gets isolated from the external network
Site external
R R R R
10.55.41.1
interface Ethernet1/3
no switchport
Eth1/3 mtu 9216
AS 65520 10.55.41.2 ip address 10.55.41.2/30 tag 54321
B B evpn multisite dci-tracking
Lo0: 10.101.101.41
Agg Agg
Site internal
Server Server
Legacy Site
• Configure eBGP peering relationships between

each vPC BGW and external peers
• Activate the IPv4 unicast family (VRF default) to
AS 65099 redistribute required loopback prefixes and
Site external
directly connected interfaces

R R R R
router bgp 65520

router-id 10.101.101.41
AS 65520 log-neighbor-changes
B B address-family ipv4 unicast
Lo0: 10.101.101.41 redistribute direct route-map RMAP-REDIST-DIRECT
maximum-paths 4
Agg Agg
Site internal
neighbor 10.55.41.1
remote-as 65099
Access Access Access update-source Ethernet1/3
address-family ipv4 unicast
Server Server
route-map RMAP-REDIST-DIRECT permit 10

Legacy Site
match tag 54321
Step 3: Configure vPC BGWs DCI Overlay network
• Configure the remote BGW neighbor(s) with the EVPN address family type L2VPN EVPN enabled
• The IP address specified for the neighbor represents its loopback0 CP IP address
• ebgp-multihop command will likely be required to support remote BGW devices
• The peer-type fabric-external configuration is required for each remote Multi-Site BGW(s)
• The rewrite-evpn-rt-asn configuration is required to enable the rewriting of Route-Target values for prefixes
advertised to remote BGWs
router bgp 65520

MP-eBGP EVPN Peering router-id 10.101.101.41
log-neighbor-changes
neighbor 10.101.201.41
Core/WAN remote-as 65521
update-source loopback0
ebgp-multihop 5
peer-type fabric-external
B B B B B B address-family l2vpn evpn
send-community
Fabric #1
send-community extended
Agg Agg rewrite-evpn-rt-asn
L L L L
Server Server
Server Server
EVPN/VXLAN Fabric
AS 65521 Legacy Site
Server
AS 65520
Step 4: Configure vPC BGWs for DCI Layer 2 extension across sites
• Define the site-id on each vPC BGW - the pair of vPC BGWs at the same site must use the same site-id
value
• Define the loopback interface to be used as Multi-Site virtual IP address (Multi-Site VIP), and the loopback
interface to be used as Primary IP address (PIP) and vPC virtual IP address (vPC VIP)
• Map the VLANs to the corresponding Layer 2 VNIs.
evpn multisite border-gateway 2 VXLAN evpn multisite border-gateway 1
delay-restore time 300
R R R R interface loopback100
description Multi-Site VIP
ip address 10.10.12.1/32 tag 54321
ip pim sparse-mode
!
interface loopback1
B ip address 10.10.10.1/24 tag 54321 <-- The first
B B B B B IP is each BGW's PIP and is unique in the pair
Fabric #1 ip address 10.10.11.1/24 secondary tag 54321
Agg Agg
L L L L
vlan 5
Access Access Access vn-segment 30005
Server Server
vlan 6
Server Server
vn-segment 30006
EVPN/VXLAN
Fabric Legacy Site
Server
Step 4 – Con’t: Configure vPC BGWs for DCI Layer 2 extension across sites
• Associate the Layer 2 VNIs with the NVE interface (VTEP) for selective advertisement. Only the associated
Layer 2 VNIs are extended across the DCI.
• NOTE: If VLANs being extended in VXLAN are already extended via a traditional DCI solution (OTV, VPLS), it
is critical to avoid the creation of an end-to-end Layer 2 loop between data center sites. This can be
achieved in a couple of different ways (on a VLAN-by-VLAN basis):
• “Flip the switch” - Disable the VLAN extension in traditional DCI solution and start using VXLAN, or;
• Keep the VLAN extension function via the traditional DCI solution and avoid trunking the VLAN on
one of the two vPC connections between the legacy networks and the vPC BGW nodes.
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback1
multisite border-gateway interface loopback100
global ingress-replication protocol bgp
member vni 30005
multisite ingress-replication
ingress-replication protocol bgp
member vni 30006
multisite ingress-replication
mcast-group 239.1.1.1
Step 5: Enable Anycast Gateway on vPC BGWs and keep it in shutdown state
• Define the Anycast Gateway MAC address (2020.0000.00AA in this example) for all the
defined tenant SVIs
• Map one of the reserved VLANs to the L3 VNI to be used for a given VRF (tenant-1)
• Associate L3VNI to NVE interface (VTEP on BGW)
AS 65099
• Define the SVI to be used as Anycast Gateway and keep it in shutdown mode
Site external
R R R R
• Configure the VRF under the BGP process to be able to start exchanging L3 prefixes with
the remote BGW nodes:
Anycast GW • Associate route-map used to redistribute IP subnet information into the EVPN control
SVIs in down state plane – match on TAG
interface Vlan5
shutdown
B B vrf member tenant1
fabric forwarding anycast-gateway-mac
2020.0000.00AA ip address 10.1.5.1/24 tag 12345 NOTE: Tag to facilitate
Site internal
! redistribution
HSRP enabled SVIs in up state fabric forwarding mode anycast-gateway
vlan 2001
vn-segment 50001
vrf context tenant-1 router bgp 65520
Agg Agg
vni 50001 <-- Maps the tenant/VRF to L3VNI vrf tenant-1
! address-family ipv4 unicast
Access Access Access interface nve1 redistribute direct route-map FABRIC-RMAP-REDIST-SUBNET
member vni 50001 associate-vrf maximum-paths ibgp 2 only needed for local fabric
Server Server redistribute direct route-map FABRIC-RMAP-REDIST-SUBNET
maximum-paths ibgp 2  only needed for local fabric
!
Legacy Site route-map FABRIC-RMAP-REDIST-SUBNET permit 10
match tag 12345
Step 6: Migrate first-hop FHRP Gateway in the legacy site to the vPC BGW Anycast Gateway
• Align all FHRP Gateway MAC and IP addresses with the Multi-Site vPC BGW
distributed IP Anycast Gateway configuration. You must use the same virtual
MAC address for all of the different IP subnets, because the Anycast Gateway
AS 65099 virtual MAC address is a global configuration parameter on VXLAN EVPN VTEPs.
Site external
R R R R • Create a sub-interface per tenant and enable exchange of IPv4 routes with the
BGP neighbor.
interface vlan 20
vrf member Tenant-A
Eth1/1 B B ip address 192.168.20.201/24
192.168.20.2
Site internal
hsrp 10
ip 192.168.20.1
Eth1/1 mac-address 2020.0000.00aa interface Ethernet1/1.20
192.168.20.1 description L3 Link to vPC BGW1 (T1)
Agg Agg encapsulation dot1q 20
vrf member Tenant-A
ip address 192.168.20.4/31
Access Access Access router bgp 65520

router-id 100.100.100.1
Server Server vrf Tenant-A
neighbor 192.168.20.5
remote-as 65520
Legacy Site
Step 7: Transition legacy data centers to new Nexus 9000 EVPN/VXLAN fabric
Connect the new fabric spines to the pair of vPC BGWs with point-to-point Layer 3 links. Modify the
configuration on the vPC BGWs to integrate with the new VXLAN EVPN fabric. Those changes do not affect
the existing connectivity between the legacy networks.
VXLAN
B B B B
Agg Agg S S S S Agg Agg

L L L L L L
Server Server Server
Server
“Mixed Site “Mixed Site
Step 7: Continued
End state of the legacy data center migration to VXLAN EVPN fabrics with vPC BGW nodes
• Getting to this point - Migration of services (Firewall, Load Balancing, DNS, etc.), application workloads
and associated dependences have migrated to EVPN fabric
• Notice that the vPC BGW nodes perform the full BGW duties as they allow extending connectivity
between endpoints connected to local and remote VTEP devices. This is in contrast with original state in
the “legacy” zones, where there was no presence of VTEP nodes inside the local sites.
VXLA
N
B B B B
S S S S
L L L L L L L L L L

Server
EVPN VXLAN Fabric 1 EVPN VXLAN Fabric 2
Step 7: Continued
Converting vPC BGWs to Anycast BGWs (Optional, but recommended Last step)
• This is the recommended deployment model for interconnecting VXLAN EVPN fabrics, but it is only
possible if there are no endpoints connected to the original vPC BGWs that are using them as their
default gateway.
• Note: The conversion to Anycast mode can be performed one BGW at the time, in order not to
disrupt the Layer 2 and L3 connectivity between sites.
VXLA
N
Anycast BGWs Anycast BGWs
B B B B
S S S S
L L L L L L L L L L

Server
EVPN VXLAN Fabric 1 EVPN VXLAN Fabric 2
EVPN Multi-Site vPC
BGW failure scenarios
EVPN VXLAN Multi-site BGW Failure Scenarios
• EVPN Multi-Site dci-tracking: interface tracking is required
on the interface(s) connecting to the external Layer 3 core
to detect the scenario where a given vPC BGW node gets
isolated from the external network (Site External)
AS 65099 • EVPN multi-site fabric tracking: Interface tracking is the

mechanism implemented on each BGW node to detect a
Site external
S S S S potential loss of connectivity toward the site-internal or site-

external network, and be able to properly react to those
events
Eth1/1
B B
Eth1/2 description L3 Link to Site-External Network
ip address 10.111.111.1/30
Site internal
evpn multisite dci-tracking

Agg Agg
Access Access Access description L3 Link to Site-Internal Network
ip address 10.0.1.5/30
evpn multisite fabric-tracking
vPC BGW isolation from the site-external network
vPC BGW isolation from the site-external network
• Under these circumstances, the following sequence of

AS 65099 events will happen on the vPC BGW node isolated from the
site-external network:
Site external
R R R R
• The PIP1 and vPC VIP addresses continue to be advertised
toward the site-internal network and to the peer BGW via
vPC VIP1 the Layer 3 adjacency established on the vPC peer-link.
11.11.11.11 This is required to allow connectivity to the external network
B B and to local endpoints (only reachable via the isolated BGW
node) both from endpoints connected to the local site and
PIP1 PIP2
in remote sites.
Site internal
10.1.10.1 10.1.20.1
S S
L Multi-site VIP
VTEP 100.100.100.100
vPC BGW isolation from the site-internal network
vPC BGW isolation from the site-internal network
• Under these circumstances, all the logical interfaces on the

AS 65099 isolated BGW (PIP, vPC VIP, and Multi-Site VIP) remain
active and their addresses are still advertised toward the
Site external
R R R R site-external network (and to the peer BGW via the Layer 3

adjacency established on the vPC peer-link)
vPC VIP1 • This implies that 50 percent of the traffic flows incoming
11.11.11.11 from remote sites will need to be forwarded via the vPC
B B peer-link, together with the totality of flows originated from
endpoints or networks directly connected to the isolated
PIP1 PIP2
BGW node
Site internal
10.1.10.1 10.1.20.1
S S
L Multi-site VIP
VTEP 100.100.100.100
Automation and
Observability
Nexus Dashboard Fabric Controller
and Insights
Nexus Dashboard Fabric Controller (NDFC)
Need new Icons
Solution Benefits
Streamlined Automate and Maintain Extensive visibility, Expand your

lifecycle configure your compliance and monitoring and network with
management networks with ease detect errors modernized integrations with
topology views NDO and NDI
Fabric A Fabric B
Enhanced Classic LAN
Profile for Automating Migration of Legacy to EVPN/VXLAN
Fully automated fabric - Enhanced Classic

LAN
Support for greenfield and brownfield

deployments
Provisioning of 3tier architecture/

L2/L3 Networks and VRFs
Classic LAN Fabric VRF-Lite Between Agg and Core
Benefits
Best Practice Templates Simplified workflows Flexibility based on customer needs
Cisco NDFC & Nexus Insights
Seamless integration with Day 2 operations for in depth telemetry analytics
Network automation of your

Connectivity
data center environment
Single point of management

Operations
and control for daily
operations
End-to-end discovery,
Enhanced app experience
visibility and monitoring
Conclusion
Conclusion – Key Take-Aways
#1 #2
vPC Border Gateways VXLAN BGP EVPN Multi-Site
Provides an Industry Standard method to migrate off Legacy DC Tech A Simple add or drop-in
Flexible Integration model with older Network Gear First introduced in September 2017 – proven and deployed
Proven technology with documented Migration Plans A Solution beyond EVPN DCI Overlay (RFC9014)
Coordination with Application Teams once Migration Path is ready Provides Layer-2 and Layer-3 extension
Nexus Dashboard for Automation, Management and Visibility Wide Hardware Support
Flexible Deployment Option - Not just for VXLAN Fabrics
Nexus Dashboard for Automation, Management and Visibility
Fill out your session surveys!
Attendees who fill out a minimum of four session

surveys and the overall event survey will get
Cisco Live-branded socks (while supplies last)!
Attendees will also earn 100 points in the

Cisco Live Challenge for every survey completed.
These points help you get on the leaderboard and increase your chances of winning daily and grand prizes
• Visit the Cisco Showcase
for related demos
• Book your one-on-one

Meet the Engineer meeting
• Attend the interactive education

with DevNet, Capture the Flag,
Continue and Walk-in Labs
your education • Visit the On-Demand Library

for more sessions at
www.CiscoLive.com/on-demand
Thank you
#CiscoLive
Gamify your Cisco Live experience!
Get points for attending this session!
How:
1 Open the Cisco Events App.
2 Click on 'Cisco Live Challenge’ in the side menu.
3 Click on View Your Badges at the top.
4 Click the + at the bottom of the screen and scan the QR code:
#CiscoLive

BRKDCN 2933

Uploaded by

Copyright:

Available Formats

BRKDCN 2933

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKDCN 2933

Uploaded by

Copyright:

Available Formats

#CiscoLive

Next Generation DCI

2 Click “Join the Discussion”

4 Enter messages/questions in the Webex space

Webex spaces will be moderated

Important Note: The session is exclusively focused on NX-OS standalone VXLAN

• Internet Engineering Task • RFC 9014

• Categorized for Standards

BGP Autonomous System (AS) BGP Autonomous System (AS)

VXLAN BGP VXLAN BGP

Decoupled Gateway (Section 3) Integrated Gateway (Section 4)

VXLAN BGP EVPN VXLAN BGP EVPN

Layer-2 EVPN* Layer-2 EVPN*

VXLAN BGP EVPN VXLAN BGP EVPN

Decoupled Gateway (Section 3) Integrated Gateway (Section 4)

VXLAN BGP EVPN VXLAN BGP EVPN

Layer-2 EVPN* Layer-2 EVPN*

VXLAN BGP EVPN VXLAN BGP EVPN

What about Layer-3?

• Internet Engineering Task • Multi-Site (BESS version)

• Shipping since 2017

BGP Autonomous System (AS) BGP Autonomous System (AS)

VXLAN BGP VXLAN BGP

// Layer-2 & Layer-3

DCI Encap VPLS, PBB-VPLS, EVPN-MPLS, PBB-EVPN, VXLAN VXLAN

Gateway Mode Multipath PIP Anycast VIP Multipath PIP

ECMP Underlay and Overlay Underlay Underlay and Overlay

EVPN RT-1 Consumed and Generated None Consumed and Generated

EVPN RT-5 - (not part of RFC) Re-Originated Re-Originated

VNI Allocation Global and Downstream Global and Downstream

DF Election Based on EVPN RT-4 Based on EVPN RT-4

Identifier I-ESI I-ESI (= Site-ID)

Split Horizon Local Bias Local Bias

RFC 9014 Multi-Site

Server Server Server Server Server Server Server Server

Single BUM Packet, 5x Replicated Single BUM Packet, 3x Replicated

*BUM – Broadcast, Unknown Unicast, Multicast

32’768 VTEP to extend Layer-2 or/and Layer-3 segments to

Single Fabric or Multi-POD Multiple Fabric with Multi-Site

Server Server Server Server Server Server Server Server

*BUM – Broadcast, Unknown Unicast, Multicast

Works Within a Geographic Location – Works Between Geographic Locations

Much more on this shortly!

Minimum Hardware and Software Requirements for BGW (Border Gateway)

*Check for Hardware Specific Support Releases

Border Gateway (BGW) Scale EX FX2 FX,FX3,GX,GX2 N9364C & N9332C

BGP Autonomous System (AS) BGP Autonomous System (AS)

VXLAN BGP VXLAN BGP

// Layer-2 & Layer-3

BGP Autonomous System (AS) BGP Autonomous System (AS)

VXLAN BGP VXLAN BGP

Fabric #1 Underlay Fabric #2 Underlay

Fabric Fabric Fabric

• Border Gateways used for two main functions:

B B B B • Two Modes of Operation:

• Common Use Case

Anycast Border Gateway vPC Border Gateway

Multi-Site VIP vPC VIP

vPC Border Gateway