CUMULUS NETWORKS WHITEPAPER — VXLAN ROUTING WITH EVPN

VXLAN routing with EVPN

The VXLAN routing architectures,
operations, and standard configurations
for modern data center networks

Contents
Introduction 2

VXLAN routing overview 3

VXLAN routing architectures 4

Centralized architecture 4

Distributed architecture 5

VXLAN routing with EVPN overview 6

EVPN route types for VXLAN routing 6

EVPN integrated routing and bridging (IRB) models 9

Asymmetric IRB model 9

Symmetric IRB model 10

Cumulus EVPN VXLAN routing deployment scenarios and

configurations 12
Centralized VXLAN routing with bridging using EVPN 15

Distributed VXLAN routing with the asymmetric IRB model 23

Distributed VXLAN routing with the symmetric IRB model 27

NetQ for EVPN operations 32

Conclusion
 EVPN: Introduction

Introduction
Modern day data centers are moving from layer 2 to layer 3 architectures to take advantage
of a vast array of benefits. A layer 3 infrastructure provides smaller failure domains, less
spanning tree troubleshooting, fewer proprietary protocols such as MLAG, as well as
increased redundancy and resilience over a strict layer 2 data center.

However, some distributed applications, load balancers, and storage appliances still require
layer 2 connectivity to communicate. Therefore, VXLAN was born to run as an overlay on a
robust layer 3 data center to offer the best of both worlds: maintaining layer 2 connectivity
between racks while deploying a layer 3 infrastructure. VXLAN also offers additional
benefits over layer 2 VLANs such as scale and flexibility.

To provide the layer 2 connectivity, VLANs are bridged into VXLAN tunnels, which run over
the IP infrastructure. Since these packets are bridged, they cannot reach outside the same
VLAN/VXLAN without performing some type of routing. However, many server applications
require both layer 2 connectivity to the same VLAN on a different rack and layer 3
connectivity to other VXLANs/VLANs within in the data center in a different rack (or even
outside the data center altogether). For example, in Figure 1 below, the green application
only communicates via layer 2, while the orange application needs access to the Internet
or other VLANs. Without static configuration, the front-end server cannot reach the Internet
without some type of VXLAN routing.

Internet

Layer 3 IP Fabric

VLAN 10 VLAN 10

Front-end Server Back-end Database

Figure 1 - VXLAN routing example

In order to reach each other or the Internet, these individual tunnels must be integrated with
the larger routing fabric. Older ASICs in many data center switches, such as the Broadcom
Trident II, cannot support dual lookup routing — routing first on the outer header IP address
and then again on the decapsulated packet. Therefore, data centers with an external

2 CUMULUS NETWORKS — WHITE PAPER

EVPN: VXLAN routing overview

reachability requirement deployed external routing via a separate router/firewall or a

hyperloop cable to perform routing between the VXLAN tunnels and the Internet, usually
from a centralized location. The newer ASICs, such as the Broadcom Trident II+, Maverick,
Tomahawk (via internal loopback), and the Mellanox Spectrum can route directly on the
ASIC, which opens the door to more efficient VXLAN routing design options using BGP with
the Ethernet Virtual Private Network (EVPN) address family as the routing protocol.

This paper is meant as an addendum to the paper, BGP EVPN for VXLAN, which
describes using EVPN for VLAN/VXLAN bridging. Deploying BGP EVPN also as a layer
3 routing protocol for VXLAN dramatically simplifies the VXLAN routing deployment by
using the same protocol, BGP, for the underlay, for bridging between VLANs/VXLANs, and
for routing between VXLAN tunnels and to outside the data center. This paper provides a
VXLAN routing overview, describes some of the architectures and aspects of VXLAN routing
with EVPN, and provides some typical design examples along with configuration snippets.

VXLAN routing overview

VXLAN is an effective way to tunnel layer 2 frames over a layer 3 infrastructure, and EVPN can
be used to provide a resilient control plane for VXLAN tunnels. However, routing is also needed
for communication between VXLAN tunnels or between a VXLAN tunnel and the Internet.

In the past, customers with older generation silicon performed VXLAN routing using an
external router and/or a hyperloop. Newer generation ASICs such as the Broadcom Trident
II+, Trident III, Tomahawk1 , and Mellanox Spectrum support VXLAN routing directly on the
switch. For example, Figure 2 shows Host A on VLAN A communicating with a host on
VLAN B located on a different rack.

Routing

vni vni
Bridge
swp1 swp2

Send to host C
Host A VLAN A Host B VLAN B
on VLAN B on a
different rack

Figure 2 - Internal silicon-based VXLAN routing example

VXLAN routing internal to the switch allows for much more efficient architectures in a
network. You can efficiently deploy one of two different architectures to route across
VXLAN boundaries while still supporting VXLAN bridging when applicable. The two
architectures are discussed in the next section.
1 Tomahawk provides internal VXLAN routing via an internal loopback

CUMULUS NETWORKS — WHITE PAPER 3

 EVPN: VXLAN routing architectures

VXLAN routing architectures

VXLAN routing is primarily deployed using two architectures: either centralized or
distributed. A centralized architecture involves using a centralized router (or pair of routers)
to route between all the local VXLAN tunnels and between a VXLAN tunnel and the outside.
A centralized architecture does not use EVPN for layer 3 routing; all the routing happens on
a centralized switch but EVPN routes are leveraged for VXLAN bridging and features such
as ARP suppression and mobility.
A distributed architecture involves distributing EVPN routes that are used for both routing
and bridging VXLAN tunnels to each top of rack leaf switch. A distributed architecture
provides routing closest to the hosts.

CENTRALIZED ARCHITECTURE
In a centralized architecture, only one or a pair of routers provide access between VXLANs.
All inter-VLAN traffic or traffic intended for outside the local data center needs to be VXLAN
tunneled to the centralized router and back to the destination. This "tromboning" of traffic
means additional east-west traffic will occur in the data center to support VXLAN routing.
For each VLAN/VXLAN pair in the local network, the centralized router must have a VNI
and a switch virtual interface (SVI) that is used as the VLAN's default gateway. The default
gateway IP and MAC are advertised via BGP EVPN extended community to all the leaf
switches. See Figure 3.

Internet

Layer 3 Network
eBGP Peering
Spine01 Spine02 (ipv4 unicast)

Leaf01 VTEP Leaf02 VTEP VTEP

Bridging only Bridging only Centralized Router

SVI B (default gtwy)
SVI A (default gtwy)
VLAN A VLAN B VLAN A VLAN B

Server01 Server02 Server03 Server04

VXLAN Tunnel, VNI A

VXLAN Tunnel, VNI B

Figure 3 - Centralized routing architecture

For example, Server01 on VLAN A wants to communicate with Server04 on VLAN B. Traffic
exits Server01 and enters Leaf01 with a destination IP/MAC of the SVI on the centralized
router (the default gateway). Leaf01 bridges and encapsulates the traffic in a VXLAN tunnel

4 CUMULUS NETWORKS — WHITE PAPER

EVPN: VXLAN routing architectures

headed for the default gateway. The centralized router decapsulates the packet and routes
the packet to SVI B, which is seen as a connected route. The router then encapsulates the
packet in a new VXLAN header and sends it to Leaf02 on the destination VXLAN. Leaf02
decapsulates and bridges the packet to Server04.

DISTRIBUTED ARCHITECTURE
A distributed architecture involves configuring an SVI and thus enabling VXLAN routing
on each top of rack (ToR) leaf switch. Therefore, the VXLAN routing occurs closest to the
host, keeping traffic local which provides more efficient routing and lower latency than the
centralized architecture.

Typically, an anycast gateway is configured with a distributed architecture. Using an anycast

gateway, every leaf's SVI is configured with the same IP address per VLAN. For example,
looking at Figure 4 below, Leaf01 SVI A and Leaf02 SVI A would be configured with the
same IP address and subnet mask. Since all hosts within a VLAN are configured with the
same IP default gateway address, all hosts or VMs can be easily moved throughout the
data center without changing their configuration.

Internet

Layer 3 Network
eBGP Peering
Spine01 Spine02 (ipv4 unicast)

Leaf01 VTEP Leaf02 VTEP VTEP

Anycast gateway SVI A SVI B SVI A SVI B Exit01

VLAN A VLAN B VLAN A VLAN B

Server01 Server02 Server03 Server04

VXLAN Tunnel, VNI A

VXLAN Tunnel, VNI B

Figure 4 - Distributed architecture

Using Figure 4 as an example, Server01 on VLAN A wants to communicate with Server04

on VLAN B. Traffic would exit Server01 with a destination MAC address of Leaf01, its default
gateway. Leaf01 would recognize its own MAC address, perform a lookup, add the correct
VXLAN header and route the packet directly towards Leaf02. The Exit01 router is only used for
traffic to/from the local data center/POD to outside the data center/POD.

CUMULUS NETWORKS — WHITE PAPER 5

 EVPN: VXLAN routing with EVPN overview

VXLAN routing with EVPN overview

It is helpful to understand several aspects of EVPN that are specific to VXLAN routing.
We cover some of them in the next section to introduce the topics before moving into
deployment scenarios and configuration.

EVPN ROUTE TYPES FOR VXLAN ROUTING

BGP consists of multiple address families. Each address family exchanges a different type
of route. For example, the IPv4 address family exchanges IPv4 routes, and the IPv6 address
family exchanges IPv6 routes. EVPN also has its own BGP address family which exchanges
EVPN routes. Several different types of EVPN routes exist within the BGP EVPN address
family.

This paper covers the two route types directly used for VXLAN routing, type 2 and type 5.
Cumulus Linux also supports type 3, which is used for VTEP discovery.

Type 2 EVPN routes

A type 2 EVPN route is generally used for communication within a data center, although it could
used for "stretching" VXLAN tunnels between data centers. It consists of the following fields:

BGP EVPN Type 2 Route

Route Distinguisher

Ethernet Segment Identifier

Ethernet Tag ID

MAC Address Length

MAC Address

IP Address Length

IP Address

Label 1 (L2VNI)

Label 2 (L3VNI)

Figure 5 - BGP EVPN type 2 route

As seen in Figure 5, a type 2 route includes both an IP address and MAC address coupled
together. The MAC address in the advertisement enables all leafs to know the location of
every host and enables layer 2 reachability while eliminating data plane learning. Adding the IP
address onto the advertisement for each MAC address allows each leaf switch to route as well
as perform ARP suppression, which reduces broadcast traffic in the network.

6 CUMULUS NETWORKS — WHITE PAPER

EVPN: VXLAN routing with EVPN overview

A type 2 route is used for:

●● All VLAN/VXLAN bridging

●● Intra-data center or intra-POD VXLAN routing

●● Inter data center routing with stretched VXLAN tunnels (VXLAN tunnels between
data centers or PODs)

If external layer 3 connectivity is required, a separate route type, type 5, is used. Type 5 is
discussed below.

Type 5 EVPN routes

The type 5 EVPN route, IP prefix route, includes the following fields:

BGP EVPN Type 5 Route

Route Distinguisher

Ethernet Segment Identifier

Ethernet Tag ID

IP Prefix Length

IP Prefix

GW IP address

Label (L3VNI)

Figure 6 - Type 5 EVPN route

As is seen above in Figure 6, the route advertisement contains the IP address and IP prefix
length, but no MAC address. The type 5 route is used with a distributed architecture to
advertise external, inter-data center and/or inter-POD routes to all local leafs within a VRF
and can only be used with an L3VNI. More information about the L3VNI can be found in the
Symmetric IRB Model section of this paper.

Figure 7 depicts a setup where type 5 EVPN routes are used for external routing. In this
case, we are advertising the 172.16.0.0/16 route from the Internet router towards the border
leafs in the BGP IPv4 address family. It is inserted into vrf1 and sent to all the VTEPs in the
network via a type 5 EVPN route. A type 5 route may also be used for inter-POD or inter-
data center routing.

CUMULUS NETWORKS — WHITE PAPER 7

 EVPN: VXLAN routing with EVPN overview

Layer 3 Network

AS65020 AS65020
Internet

Advertise
172.16.0.0/16
eBGP Peering
(ipv4 unicast)
L2 VNI L2 VNI
VRF1 Leaf01 VRF1 Leaf02 VRF1 Border Leaf
L3 VNI L3 VNI
AS65011 AS65012 AS65041

Host A VLAN A Host A VLAN A

Rack 1 Rack 2

VXLAN Tunnel, L2 VNI (Type 2 routes)

VXLAN Tunnel, L3 VNI (Type 2/5 routes)

Figure 7 - Using Type 5 routes for external routing

cumulus@leaf01:mgmt-vrf:~$ net show bgp evpn route

BGP table version is 23, local router ID is 10.0.0.11
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-2 prefix: [2]:[ESI]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[ESI}[EthTag]:[Iplen]:[IP]:[GW}

   Network          Next Hop            Metric LocPrf Weight Path

Route Distinguisher: 10.0.0.11:2
*> [2]:[0]:[0]:[48]:[44:38:39:00:00:15]
                    10.0.0.112                         32768 i
*> [2]:[0]:[0]:[48]:[44:38:39:00:00:15]:[32]:[10.2.4.102]
                    10.0.0.112                         32768 I
<snip>
Route Distinguisher: 10.0.0.41:2
*  [5]:[0]:[0][16]:[172.16.0.0]
                    10.0.0.41                              0 65020 65041 i
*> [5]:[0]:[0][16]:[172.16.0.0]
                    10.0.0.41                              0 65020 65041 i

A type 5 EVPN route is used for:

●● All traffic destined to the Internet

●● Inter-data center or inter-POD VXLAN routing

8 CUMULUS NETWORKS — WHITE PAPER

EVPN: VXLAN routing with EVPN overview

EVPN INTEGRATED ROUTING AND BRIDGING (IRB) MODELS

The IETF integrated routing and bridging in EVPN draft identifies two possible
distributed IRB models for use with distributed EVPN: asymmetric and symmetric. Cumulus
Linux supports both models.

Each model is useful in certain situations and has unique characteristics. While some
vendors support either one model or the other, a Cumulus Linux EVPN with VXLAN routing
implementation can support either model. Supporting both provides interoperability with
various other vendors as well as a choice for which model is best for your data center. This
section discusses the benefits and operation of each model used for VXLAN integrated
routing and bridging.

ASYMMETRIC IRB MODEL

The asymmetric IRB model is considered asymmetric as routed traffic between two hosts
travels on different VNIs (always the destination VNI). Traffic entering the VNI is bridged
from the VLAN and may be routed to a different VNI, whereas traffic exiting the VNI is
always bridged.
Layer 3 Network

Orange VNI Orange VNI

Leaf01 Leaf02
Green VNI Green VNI
Routing on ingress Bridging on egress Routing on ingress Bridging on egress

Host A VLAN A Host B VLAN B

Host D VLAN B Host C VLAN A

VXLAN Tunnel, Orange VNI

VXLAN Tunnel, Green VNI

Figure 8 - Asymmetric IRB model

Consider the scenario in Figure 8. Host A on VLAN A wants to communicate with Host C
on VLAN A. Host A knows Host C is on its same subnet (using the destination address and
its own IP address and mask to determine) so it initiates the communication by ARPing for
Host C. Since Host C has already sent a frame in the past, Leaf02 learned its MAC address
and previously communicated it to Leaf01. Leaf01 responds to Host A's ARP request, telling
Host A Host C's MAC address. Host A then sends its packets directly to Host C, bridging
across the orange VXLAN tunnel.

CUMULUS NETWORKS — WHITE PAPER 9

 EVPN: VXLAN routing with EVPN overview

Host A wants to now communicate with Host B, which is located on a different VLAN and
thus reachable via a different VNI. Since the destination is a different subnet from Host A,
Host A sends the frame to its default gateway, which is Leaf01. The Leaf01 southbound
interface towards Host A is configured with an anycast IP address, which is discussed
more deeply in the architectures section. Leaf01 recognizes that the destination MAC
address is itself and uses the routing table to route the packet to the Green VNI. Leaf01
then tunnels the frame in the Green VNI to Leaf02. Leaf02 removes the VXLAN header from
the frame, and bridges the frame to Host B.

The return traffic behaves similarly. Host B sends a frame to Leaf02, which recognizes its
own destination MAC address and routes the packet to the Orange VNI. The packet is
tunneled within the Orange VNI to Leaf01. Leaf01 removes the VXLAN header from the
frame and bridges it to Host A.

With the asymmetric model, all the required source and destination VNIs (that is, Orange and
Green) must be present on each leaf, even if that leaf doesn't have a host in that associated
VLAN in its rack. In many instances, this is needed for VM mobility anyway. As a result, all
leafs would be required to hold all routes and all MAC addresses that communicate with each
other. Deploying an asymmetric model is a simple solution as no additional VNIs need to be
configured and fewer routing hops occur to communicate between VXLANs. However, it does
not scale as well as the symmetric model covered below.

In the case of multitenancy, each set of VLANs can be placed into separate VRFs and
routed between them.

Use the asymmetric model if:

●● You plan to deploy all VNIs on all leaf switches anyway

●● You have a small number of VNIs in your data center or POD
●● Your data center can be broken into PODs, with the VLANs/subnets contained in
each POD
SYMMETRIC IRB MODEL
Routed VXLAN traffic within the symmetric IRB model travels across the same "transit" VNI
in both directions. The "transit" VNI must also have an associated VLAN to accommodate
traffic to be routed between VXLAN tunnels. We call the new transit VNI "L3VNI". All traffic
that needs to be routed is bridged onto the associated VLAN and then routed onto the
L3VNI. The ingress traffic is routed on ingress to the L3VNI, tunneled across the layer 3
infrastructure, and then routed again on egress off the L3VNI to the associated VLAN and
ultimately bridged to the destination host.

10 CUMULUS NETWORKS — WHITE PAPER

EVPN: VXLAN routing with EVPN overview

Layer 3 Network

To Leaf03

Leaf01 Leaf02

VNI 24
Routing & bridging VNI 24 Routing & bridging
on ingress & egress on ingress & egress VNI 13
L3VNI
L3VNI
Host A VLAN 24 Host B VLAN 13

Host C VLAN 24

Rack 1 Rack 2

VXLAN Tunnel, Orange VNI 24 (Layer 2 communication only)

VXLAN Tunnel, Green VNI 13 (Layer 2 communication only (e.g. to Rack 3))

VXLAN Tunnel, L3VNI (Layer 3 communication only)

cumulus@leaf02:mgmt-vrf:~$ net show bgp evpn vni

Advertise Gateway Macip: Disabled
Advertise All VNI flag: Enabled
Number of L2 VNIs: 2
Number of L3 VNIs: 1
Flags: * - Kernel
VNI        Type RD              Import RT           Export RT         Tenant VRF                           
* 24         L2   10.0.0.12:2      65012:24             65012:13           vrf1                                 
* 13         L2   10.0.0.12:3      65012:13             5012:13            vrf1                                 
* 104001  L3  10.2.4.12:4      65012:104001        65012:104001       vrf1

Figure 9 - Symmetric IRB model

Using Figure 9 as an example, Host A on VLAN 24 needs to communicate with Host B on

VLAN 13. Since the destination is a different subnet from Host A, Host A sends the frame to
its default gateway, Leaf01.

Leaf01 recognizes that the destination MAC address is itself and uses the routing table to
route the packet to the egress leaf (Leaf02) over the L3VNI. The MAC address of Leaf02 is
communicated to Leaf01 via a BGP extended community. The VXLAN-encapsulated packet
has the egress leaf's MAC as the destination MAC address and the L3VNI as the VNI.
Leaf02 performs VXLAN decapsulation and recognizes that the destination MAC address
is itself and routes the packet to the destination VLAN to reach the destination host. The
return traffic is routed similarly over the same L3VNI. Routing and bridging happens on both
the ingress leaf and the egress leaf.

CUMULUS NETWORKS — WHITE PAPER 11

 EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations

With the symmetric model, the leaf switches only need to host the VNIs/VLANs that are
located on their rack, as well as the L3VNI and its associated VLAN, since the ingress leaf
switch doesn't route directly to the destination VNI. The ability to host only the local VNIs
(plus one extra) provides additional scale over the asymmetric model. However, the data
plane traffic is more complex as an extra routing hop occurs and an extra VXLAN tunnel
and VLAN in your network is required.

Multitenancy requires one L3VNI per VRF, and all switches participating in that VRF must be
configured with the same L3VNI.

Use the symmetric model if:

●● Your VLANs/subnets/VNIs are widely dispersed

●● Your VLANs/subnets/VNIs are provisioned on the fly
●● You need external routing with Cumulus Linux 3.5

Cumulus EVPN VXLAN routing deployment scenarios and

configurations
VXLAN routing can be deployed in either a centralized or distributed architecture.
Centralized architecture involves bridging on the leaf switches and routing between
VXLANs on a centralized switch, usually a border switch. A distributed architecture can
be deployed in either the asymmetric or the symmetric IRB model. This section discusses
deployment scenarios for each model and includes configuration snippets for each element
of the design. More information on configuring VXLAN routing with EVPN can be found in
the Cumulus Linux User Guide.

The three solutions of this section use the Cumulus Linux Reference Architecture, as
discussed in each section. Although the examples are shown with the default, or one data
plane VRF, the same procedure as shown here could be used with multiple VRFs as well.

Although not depicted in the diagrams, all switches and hosts are connected to an out-
of-band management network. Management VRF is configured on the switches and
eth0 (connected to the out-of-band management network) is located in the management
VRF. Although an out-of-band management network connected to the switch via the
management VRF is always recommended, it is not required to deploy these solutions.

We set up an eBGP unnumbered underlay between the leafs, the spine and the exit leafs
in all models outlined in this paper. Using eBGP as the underlay routing protocol provides
a robust scalable infrastructure for the layer 2 overlay and sets the stage to also use BGP
EVPN as the overlay routing protocol.

We configure VXLAN tunnels as the overlay — providing layer 2 connectivity over the layer
3 infrastructure. BGP EVPN is used as the VXLAN routing and bridging control plane and
provides routing between VXLAN tunnels on the local, directly-connected leaf. All hosts are

12 CUMULUS NETWORKS — WHITE PAPER

EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations

advertised as type 2 EVPN routes in all scenarios.

For each of these examples, we deploy four servers, two in VLAN13 and two in VLAN24. We
run MLAG on the leaf switches to provide redundancy to the hosts. We deploy a border leaf
connected to an Internet router. For purposes of these examples, the following addresses are
used. Note that not all the addresses depicted in Table 1 below apply to all scenarios.

Table 1 - Addresses

ADDRESSES

SWITCH LOOPBACK SVIs VNIs VLANs BGP AS

Internet 10.0.0.253/32 N/A N/A N/A 25253

104001
Exit01 10.0.0.41/32 N/A 4001 (mapped to L3VNI) 65041
(L3VNI)

104001
Exit02 10.0.0.42/32 N/A 4001 (mapped to L3VNI) 65042
(L3VNI)

Spine01 10.0.0.21/32 N/A N/A N/A 65020

Spine02 10.0.0.22/32 N/A N/A N/A 65020

10.1.3.11
10.1.3.1 (anycast gtwy)
13 (data) 13

10.2.4.11
Leaf01 10.0.0.11/32 24 (data) 24 65011
10.2.4.1 (anycast gtwy)

104001
N/A 4001
(L3VNI)

10.1.3.12/24
13 (data) 13
10.1.3.1/24 (anycast gtwy)

10.2.4.12/24
Leaf02 10.0.0.12/32 24(data) 24 65012
10.2.4.1/24 (anycast gtwy)

104001
N/A 4001
(L3VNI)

10.1.3.13/24
13 (data) 13
10.1.3.1/24

10.2.4.13/24
Leaf03 10.0.0.13/32 24(data) 24 65013
10.2.4.1/24 (anycast gtwy)

104001
N/A 4001
(L3VNI)

CUMULUS NETWORKS — WHITE PAPER 13

 EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations

10.1.3.14/24
13 (data) 13
10.1.3.1/24 (anycast gtwy)

10.2.4.14/24
Leaf04 10.0.0.14/32 24(data) 24 65014
10.1.3.1/24 (anycast gtwy)

104001
N/A 4001
(L3VNI)

In all scenarios the spine switches are configured the same. They are configured with eBGP
unnumbered with the EVPN address family, peering with each leaf and any border/exit
routers. A sample configuration is show below in Figure 10 where the loopback is 10.0.0.21
and the BGP AS is 65020. Switch ports 1-4 connect to each leaf switch, respectively, and
swp29 and 30 connect to the exit switches.

router bgp 65020 address-family ipv4 unicast

bgp router-id 10.0.0.21 network 10.0.0.21/32
coalesce-time 1300
bgp bestpath as-path multipath-relax address-family l2vpn evpn
neighbor swp1 interface remote-as external neighbor swp1 activate
neighbor swp2 interfac remote-as external neighbor swp2 activate
neighbor swp3 interface remote-as external neighbor swp3 activate
neighbor swp4 interface remote-as external neighbor swp4 activate
neighbor swp29 interface remote-as external neighbor swp29 activate
neighbor swp30 interface remote-as external neighbor swp30 activate

Figure 10 - Spine switch configuration snippet

CENTRALIZED VXLAN ROUTING WITH BRIDGING USING EVPN

An architecture describing a centralized routing scenario is depicted below in Figure 11.

Internet

Layer 3 Network
eBGP
Peering

Spines
Swp1-4, 29-30
Swp44
Swp51-52 Swp51-52

Leafs VTEP VTEP VTEP

Swp1-2 Swp1-2 Swp1-2 Swp1-2 Exit Routers

SVI 24 (default gtwy)
SVI 13 (default gtwy)
10.1.3.101 10.2.4.102 10.1.3.103 10.2.4.104

VLAN 13 VLAN 24 VLAN 13 VLAN 24

Server01 Server02 Server03 Server04

VXLAN Tunnel, VNI 13

VXLAN Tunnel, VNI 24

Figure 11 - Centralized architecture

14 CUMULUS NETWORKS — WHITE PAPER

EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations

Each of the two racks hosts two servers and two leaf switches. The servers on a rack are
on separate VLANs from each other. One server is in VLAN 13 and the other server is in
VLAN 24. The leaf switches are configured with MLAG and each server is configured with a
bond interface. However, each member of the bond is connected to a different leaf switch
in the same rack.

Two VNIs and an anycast VTEP exist on each leaf switch, VNI 13 and VNI 24. An anycast
address is used for the VTEP to allow for VXLAN redundancy with the MLAG. Only bridging
occurs on the leaf switches in this design.

Each leaf is connected to each spine. The spines are configured for eBGP unnumbered
as the underlay routing protocol, and with BGP EVPN for the overlay. The spines are
connected to the centralized routers.

The centralized (exit) routers also have the same VNIs configured as the leafs and an
anycast VTEP. Since it hosts the SVI and thus is the default gateway for all the VNIs, the
centralized routers also advertises themselves as the default gateway by using a BGP
extended community with a type 2 EVPN route. Since we are running MLAG with VRR
between the centralized routers, the virtual IP and MAC addresses are advertised as well as
the physical addresses. The centralized routers peer to the spines for the underlay with the
IPv4 address family, and peer to the spines with the L2VPN EVPN address family. They also
peer with the Internet router with the IPv4 unicast address family.

This design can be expanded to accommodate many more servers and racks.

A sample configuration snippet for a leaf switch is shown in Figure 12. The MLAG, peerlink
and bond interface towards the servers are left off for brevity.

CUMULUS NETWORKS — WHITE PAPER 15

 EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations

interface lo interface vlan13

address 10.0.0.11/32 vlan-id 13
clagd-vxlan-anycast-ip 10.0.0.112 vlan-raw-device bridge

interface bridge interface vlan24

bridge-ports bond01 bond02 peerlink vlan-id 24
vni13 vni24 vlan-raw-device bridge
bridge-pvid 1
bridge-vids 13 24 router bgp 65011
bridge-vlan-aware yes bgp router-id 10.0.0.11
bgp bestpath as-path multipath-relax
interface vni13 neighbor swp51 interface remote-as external
bridge-access 13 neighbor swp52 interface remote-as external
bridge-learning off
bridge-arp-nd-suppress on address-family ipv4 unicast
mstpctl-bpduguard yes network 10.0.0.11/32
mstpctl-portbpdufilter yes network 10.0.0.112/32
mtu 9000
vxlan-id 13 address-family l2vpn evpn
vxlan-local-tunnelip 10.0.0.11 neighbor swp51 activate # pointing to spine01
neighbor swp52 activate # pointing to spine02
interface vni24 advertise-all-vni
bridge-access 24
bridge-arp-nd-suppress on
bridge-learning off
mstpctl-bpduguard yes
mstpctl-portbpdufilter yes
mtu 9000
vxlan-id 24
vxlan-local-tunnelip 10.0.0.11

Figure 12 - Centralized architecture leaf switch configuration snippet

The centralized switches are configured with a VTEP and all the same VNIs. An SVI is also
configured on each VLAN on the centralized switches.

BGP EVPN is configured to send out the default gateway community to all the leaf switches,
as highlighted below. The tenant VLANS are advertised via the BGP network command,
and a route-map is used to send only the tenant VLAN subnets to the Internet router.
Alternatively, redistribute connected could also be used with a route map. A route-map is
also used to send only the loopback and VTEP anycast addresses of the exit routers to the
underlay. The MLAG configurations were left off for brevity. Figure 13 shows a configuration
snippet on a centralized router.

16 CUMULUS NETWORKS — WHITE PAPER

EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations

interface lo interface vlan24

address 10.0.0.41/32 address 10.2.4.13/24
clagd-vxlan-anycast-ip 10.0.0.142 address-virtual 44:39:39:ff:00:24 10.2.4.1/24
interface bridge vlan-id 24
bridge-ports peerlink vni13 vni24 vlan-raw-device bridge
bridge-pvid 1 router bgp 65041
bridge-vids 13 24 bgp router-id 10.0.0.41
bridge-vlan-aware yes coalesce-time 1100
interface vni13 bgp bestpath as-path multipath-relax
bridge-access 13 neighbor swp44 interface remote-as external
bridge-learning off neighbor swp51 interface remote-as external
mstpctl-bpduguard yes neighbor swp52 interface remote-as external
mstpctl-portbpdufilter yes address-family ipv4 unicast
mtu 9000 Network 10.0.0.142/32
vxlan-id 13 network 10.0.0.41/32
vxlan-local-tunnelip 10.0.0.41 network 10.1.3.0/24
interface vni24 network 10.2.4.0/24
bridge-access 24 neighbor swp44 route-map TENANT _ VLANS out
bridge-arp-nd-suppress on neighbor swp51 route-map NO _ TENANT _ VLANS _ UN-
bridge-learning off DERLAY out
mstpctl-bpduguard yes neighbor swp52 route-map NO _ TENANT _ VLANS _ UN-
mstpctl-portbpdufilter yes DERLAY out
mtu 9000 address-family l2vpn evpn
vxlan-id 24 neighbor swp51 activate
vxlan-local-tunnelip 10.0.0.41 neighbor swp52 activate
interface vlan13 advertise-all-vni
address 10.1.3.13/24 advertise-default-gw
address-virtual 44:39:39:ff:00:13 ip prefix-list TENANT _ VLANS seq 5 permit 10.2.4.0/24
10.1.3.1/24 ip prefix-list TENANT _ VLANS seq 10 permit 10.1.3.0/24
vlan-id 13 ip prefix-list NO _ TENANT _ VLANS seq 5 permit
vlan-raw-device bridge 10.0.0.41/32
ip prefix-list NO _ TENANT _ VLANS seq 10 permit
10.0.0.142/32
route-map TENANT _ VLANS permit 10
match ip address prefix-list TENANT _ VLANS
route-map NO _ TENANT _ VLANS _ UNDERLAY permit 10
match ip address prefix-list NO _ TENANT _ VLANS

Figure 13 - Centralized switch configuration snippet

The configuration from the Internet router is very basic: we configure BGP unnumbered,
and advertise the loopback (BGP router ID) and the default route. Figure 14 shows the
configuration snippet. The management VRF was left off for brevity.

CUMULUS NETWORKS — WHITE PAPER 17

 EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations

interface lo
address 10.0.0.253/32
interface eth0
address 192.168.0.253/24
interface swp1
ipv6 nd ra-interval 10
no ipv6 nd suppress-ra
router bgp 25253
bgp router-id 10.0.0.253
coalesce-time 1000
bgp bestpath as-path multipath-relax
neighbor swp1 interface remote-as external
neighbor swp2 interface remote-as external
address-family ipv4 unicast
network 10.0.0.253/32
neighbor swp1 default-originate
neighbor swp2 default-originate
line vty

Figure 14 - Internet router configuration

Moving back to the leaf, we can examine the EVPN route table. We see the addresses
to the virtual default gateways — 10.1.3.1 and 10.2.4.1— are installed as type 2 routes as
expected; each has two ECMP routes, one through each spine switch. The addresses to
the physical interfaces are displayed as well. Since the centralized architecture bridges at
the ToRs, EVPN sends only type 2 and 3 routes to the leafs. The EVPN routes in this case
are used for bridging only and are shown in Figure 15. Some routes to the other hosts were
left off for brevity.

18 CUMULUS NETWORKS — WHITE PAPER

EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations

cumulus@leaf01:mgmt-vrf:~$ net show bgp evpn route

BGP table version is 7, local router ID is 10.0.0.11
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-2 prefix: [2]:[ESI]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[ESI]:[EthTag]:[IPlen]:[IP]
<snip>
Route Distinguisher: 10.0.0.41:2
* [2]:[0]:[0]:[48]:[44:39:39:ff:00:13]:[32]:[10.1.3.1]
10.0.0.142 0 65020 65041 i
*> [2]:[0]:[0]:[48]:[44:39:39:ff:00:13]:[32]:[10.1.3.1]
10.0.0.142 0 65020 65041 i
* [2]:[0]:[0]:[48]:[44:39:39:ff:00:13]:[128]:[fe80::4639:39ff:feff:13]
10.0.0.142 0 65020 65041 i
*> [2]:[0]:[0]:[48]:[44:39:39:ff:00:13]:[128]:[fe80::4639:39ff:feff:13]
10.0.0.142 0 65020 65041 i
* [2]:[0]:[0]:[48]:[6a:eb:f2:b0:f3:f6]:[32]:[10.1.3.13]
10.0.0.142 0 65020 65041 i
*> [2]:[0]:[0]:[48]:[6a:eb:f2:b0:f3:f6]:[32]:[10.1.3.13]
10.0.0.142 0 65020 65041 i
* [2]:[0]:[0]:[48]:[6a:eb:f2:b0:f3:f6]:[128]:[fe80::68eb:f2ff:feb0:f3f6]
10.0.0.142 0 65020 65041 i
*> [2]:[0]:[0]:[48]:[6a:eb:f2:b0:f3:f6]:[128]:[fe80::68eb:f2ff:feb0:f3f6]
10.0.0.142 0 65020 65041 i
* [3]:[0]:[32]:[10.0.0.142]
10.0.0.142 0 65020 65041 i
*> [3]:[0]:[32]:[10.0.0.142]
10.0.0.142 0 65020 65041 i
Route Distinguisher: 10.0.0.41:3
* [2]:[0]:[0]:[48]:[44:39:39:ff:00:24]:[32]:[10.2.4.1]
10.0.0.142 0 65020 65041 i
*> [2]:[0]:[0]:[48]:[44:39:39:ff:00:24]:[32]:[10.2.4.1]
10.0.0.142 0 65020 65041 i
* [2]:[0]:[0]:[48]:[44:39:39:ff:00:24]:[128]:[fe80::4639:39ff:feff:24]
10.0.0.142 0 65020 65041 i
*> [2]:[0]:[0]:[48]:[44:39:39:ff:00:24]:[128]:[fe80::4639:39ff:feff:24]
10.0.0.142 0 65020 65041 i
* [2]:[0]:[0]:[48]:[6a:eb:f2:b0:f3:f6]:[32]:[10.2.4.13]
10.0.0.142 0 65020 65041 i
*> [2]:[0]:[0]:[48]:[6a:eb:f2:b0:f3:f6]:[32]:[10.2.4.13]
10.0.0.142 0 65020 65041 i
* [2]:[0]:[0]:[48]:[6a:eb:f2:b0:f3:f6]:[128]:[fe80::68eb:f2ff:feb0:f3f6]
10.0.0.142 0 65020 65041 i
*> [2]:[0]:[0]:[48]:[6a:eb:f2:b0:f3:f6]:[128]:[fe80::68eb:f2ff:feb0:f3f6]
10.0.0.142 0 65020 65041 i
* [3]:[0]:[32]:[10.0.0.142]
10.0.0.142 0 65020 65041 i
*> [3]:[0]:[32]:[10.0.0.142]
10.0.0.142 0 65020 65041 i
Route Distinguisher: 10.0.0.42:3
* [2]:[0]:[0]:[48]:[3a:ba:74:bb:6f:17]:[32]:[10.1.3.12]
10.0.0.142 0 65020 65042 i
*> [2]:[0]:[0]:[48]:[3a:ba:74:bb:6f:17]:[32]:[10.1.3.12]
10.0.0.142 0 65020 65042 i
* [2]:[0]:[0]:[48]:[3a:ba:74:bb:6f:17]:[128]:[fe80::38ba:74ff:febb:6f17]
10.0.0.142 0 65020 65042 i
*> [2]:[0]:[0]:[48]:[3a:ba:74:bb:6f:17]:[128]:[fe80::38ba:74ff:febb:6f17]
10.0.0.142 0 65020 65042 i
* [2]:[0]:[0]:[48]:[44:39:39:ff:00:13]:[32]:[10.1.3.1]
10.0.0.142 0 65020 65042 i
*> [2]:[0]:[0]:[48]:[44:39:39:ff:00:13]:[32]:[10.1.3.1]
10.0.0.142 0 65020 65042 i

Figure 15 - Leaf EVPN routing table with centralized architecture

CUMULUS NETWORKS — WHITE PAPER 19

 EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations

* [2]:[0]:[0]:[48]:[44:39:39:ff:00:13]:[128]:[fe80::4639:39ff:feff:13]
10.0.0.142 0 65020 65042 i
*> [2]:[0]:[0]:[48]:[44:39:39:ff:00:13]:[128]:[fe80::4639:39ff:feff:13]
10.0.0.142 0 65020 65042 i
* [3]:[0]:[32]:[10.0.0.142]
10.0.0.142 0 65020 65042 i
*> [3]:[0]:[32]:[10.0.0.142]
10.0.0.142 0 65020 65042 i
Route Distinguisher: 10.0.0.42:4
* [2]:[0]:[0]:[48]:[3a:ba:74:bb:6f:17]:[32]:[10.2.4.12]
10.0.0.142 0 65020 65042 i
*> [2]:[0]:[0]:[48]:[3a:ba:74:bb:6f:17]:[32]:[10.2.4.12]
10.0.0.142 0 65020 65042 i
* [2]:[0]:[0]:[48]:[3a:ba:74:bb:6f:17]:[128]:[fe80::38ba:74ff:febb:6f17]
10.0.0.142 0 65020 65042 i
*> [2]:[0]:[0]:[48]:[3a:ba:74:bb:6f:17]:[128]:[fe80::38ba:74ff:febb:6f17]
10.0.0.142 0 65020 65042 i
* [2]:[0]:[0]:[48]:[44:39:39:ff:00:24]:[32]:[10.2.4.1]
10.0.0.142 0 65020 65042 i
*> [2]:[0]:[0]:[48]:[44:39:39:ff:00:24]:[32]:[10.2.4.1]
10.0.0.142 0 65020 65042 i
* [2]:[0]:[0]:[48]:[44:39:39:ff:00:24]:[128]:[fe80::4639:39ff:feff:24]
10.0.0.142 0 65020 65042 i
*> [2]:[0]:[0]:[48]:[44:39:39:ff:00:24]:[128]:[fe80::4639:39ff:feff:24]
10.0.0.142 0 65020 65042 i
* [3]:[0]:[32]:[10.0.0.142]
10.0.0.142 0 65020 65042 i
*> [3]:[0]:[32]:[10.0.0.142]
10.0.0.142 0 65020 65042 i
Displayed 52 prefixes (94 paths)

Figure 15 - Leaf EVPN routing table with centralized architecture (continued)

We can see the default gateway community for the route distinguisher by using the net
show bgp l2vpn evpn route rd <rd> command, as shown below in Figure 16. Some of the
output has been snipped for brevity.

20 CUMULUS NETWORKS — WHITE PAPER

EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations

cumulus@leaf01:mgmt-vrf:~$ net show bgp l2vpn evpn route rd 10.0.0.41:2

EVPN type-2 prefix: [2]:[ESI]:[EthTag]:[MAClen]:[MAC]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[ESI]:[EthTag]:[IPlen]:[IP]
<snip>
BGP routing table entry for 10.0.0.41:2:[2]:[0]:[0]:[48]:[44:39:39:ff:00:13]:[32]:[10.1.3.1]
Paths: (1 available, best #1)
Advertised to non peer-group peers:
spine02(swp52)
Route [2]:[0]:[0]:[48]:[44:39:39:ff:00:13]:[32]:[10.1.3.1] VNI 13
65020 65041
10.0.0.142 from spine02(swp52) (10.0.0.22)
Origin IGP, localpref 100, valid, external, bestpath-from-AS 65020, best
Extended Community: RT:65041:13 ET:8 Default Gateway
AddPath ID: RX 0, TX 30
Last update: Thu Mar 8 19:58:56 2018
<snip>

Figure 16 - Default gateway community

Looking at the routing table on the leaf, we see only layer 3 routes to the loopbacks of the
switches; all the routing between VXLAN subnets is happening on the centralized switch.
Figure 17 depicts the output.

CUMULUS NETWORKS — WHITE PAPER 21

 EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations

cumulus@leaf01:mgmt-vrf:~$ net show route

show ip route
=============
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
> - selected route, * - FIB route
C>* 10.0.0.11/32 is directly connected, lo, 02:39:46
B>* 10.0.0.12/32 [20/0] via fe80::4638:39ff:fe00:54, swp51, 02:38:03
* via fe80::4638:39ff:fe00:25, swp52, 02:38:03
B>* 10.0.0.13/32 [20/0] via fe80::4638:39ff:fe00:54, swp51, 02:39:43
* via fe80::4638:39ff:fe00:25, swp52, 02:39:43
B>* 10.0.0.14/32 [20/0] via fe80::4638:39ff:fe00:54, swp51, 02:39:43
* via fe80::4638:39ff:fe00:25, swp52, 02:39:43
B>* 10.0.0.21/32 [20/0] via fe80::4638:39ff:fe00:54, swp51, 02:39:43
B>* 10.0.0.22/32 [20/0] via fe80::4638:39ff:fe00:25, swp52, 02:39:43
B>* 10.0.0.41/32 [20/0] via fe80::4638:39ff:fe00:54, swp51, 02:39:43
* via fe80::4638:39ff:fe00:25, swp52, 02:39:43
B>* 10.0.0.42/32 [20/0] via fe80::4638:39ff:fe00:54, swp51, 02:39:43
* via fe80::4638:39ff:fe00:25, swp52, 02:39:43
C>* 10.0.0.112/32 is directly connected, lo, 02:39:35
B>* 10.0.0.134/32 [20/0] via fe80::4638:39ff:fe00:54, swp51, 02:39:43
* via fe80::4638:39ff:fe00:25, swp52, 02:39:43
B>* 10.0.0.142/32 [20/0] via fe80::4638:39ff:fe00:54, swp51, 00:43:58
* via fe80::4638:39ff:fe00:25, swp52, 00:43:58
C>* 169.254.1.0/30 is directly connected, peerlink.4094, 02:38:04

Figure 17 - Leaf routing table

The routing table on the centralized switch is as expected, shown in Figure 18, where we
see the VLAN subnets as directly connected and the default pointing to the internet router:

22 CUMULUS NETWORKS — WHITE PAPER

EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations

cumulus@exit01:mgmt-vrf:~$ net show route

show ip route
=============
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
> - selected route, * - FIB route
B>* 0.0.0.0/0 [20/0] via fe80::4638:39ff:fe00:7, swp44, 18:06:20
B>* 10.0.0.11/32 [20/0] via fe80::4638:39ff:fe00:a, swp51, 02:40:33
* via fe80::4638:39ff:fe00:5b, swp52, 02:40:33
B>* 10.0.0.12/32 [20/0] via fe80::4638:39ff:fe00:a, swp51, 02:38:52
* via fe80::4638:39ff:fe00:5b, swp52, 02:38:52
B>* 10.0.0.13/32 [20/0] via fe80::4638:39ff:fe00:a, swp51, 19:15:53
* via fe80::4638:39ff:fe00:5b, swp52, 19:15:53
B>* 10.0.0.14/32 [20/0] via fe80::4638:39ff:fe00:a, swp51, 19:15:53
* via fe80::4638:39ff:fe00:5b, swp52, 19:15:53
B>* 10.0.0.21/32 [20/0] via fe80::4638:39ff:fe00:a, swp51, 19:15:53
B>* 10.0.0.22/32 [20/0] via fe80::4638:39ff:fe00:5b, swp52, 19:15:53
C>* 10.0.0.41/32 is directly connected, lo, 19:15:56
B>* 10.0.0.42/32 [20/0] via fe80::4638:39ff:fe00:a, swp51, 05:29:47
* via fe80::4638:39ff:fe00:5b, swp52, 05:29:47
B>* 10.0.0.112/32 [20/0] via fe80::4638:39ff:fe00:a, swp51, 02:39:11
* via fe80::4638:39ff:fe00:5b, swp52, 02:39:11
B>* 10.0.0.134/32 [20/0] via fe80::4638:39ff:fe00:a, swp51, 19:15:53
* via fe80::4638:39ff:fe00:5b, swp52, 19:15:53
C>* 10.0.0.142/32 is directly connected, lo, 18:17:39
B>* 10.0.0.253/32 [20/0] via fe80::4638:39ff:fe00:7, swp44, 18:06:20
C * 10.1.3.0/24 is directly connected, vlan13-v0, 18:17:43
C>* 10.1.3.0/24 is directly connected, vlan13, 18:17:43
C * 10.2.4.0/24 is directly connected, vlan24-v0, 05:27:17
C>* 10.2.4.0/24 is directly connected, vlan24, 18:17:43
C>* 169.254.1.0/30 is directly connected, peerlink.4094, 18:17:47
<snip>

Figure 18 - Centralized routing table

This setup is available virtually on Cumulus Networks GitHub site and all the full
configurations are depicted.

DISTRIBUTED VXLAN ROUTING WITH THE ASYMMETRIC IRB MODEL

An architecture outlining an asymmetric model is depicted below in Figure 19. This design
can be also expanded to accommodate many more racks.

CUMULUS NETWORKS — WHITE PAPER 23

 EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations

Layer 3 Network

Leaf01 Leaf02 Leaf03 Leaf04

VRF1 VTEP VRF1 VRF1 VTEP VRF1

Anycast gateway SVI A SVIs SVIs SVIs SVIs

10.1.3.101 10.2.4.102 10.1.3.101 10.2.4.102

VLAN 24 VLAN 13 VLAN 24 VLAN 13

Server01 Server02 Server03 Server04

VXLAN Tunnel, VNI 24

VXLAN Tunnel, VNI 13

Figure 19 - Architecture using asymmetric IRB model

Two servers are located in VLAN 24 and two servers are located in VLAN 13. The servers
are dual connected via MLAG to the leaf switches. Asymmetric VXLAN routing with EVPN is
deployed to provide inter-VLAN connectivity.

No special additional leaf (ToR) configuration is needed for VXLAN routing in the
asymmetric scenario. It is enabled by default when the associated VLAN is configured with
an SVI. However, to provide routing to a destination VLAN, the local switch must have a
local VNI and VLAN configured for that destination VLAN, even if no hosts on that same
destination VLAN exist on the rack.

A sample leaf switch configuration (leaf01) is shown in Figure 20. Each leaf is configured
similarly. The MLAG/bond configuration and the peerlink were left off for brevity. Note the
SVI is configured on each VLAN, along with Virtual Router Redundancy (VRR). For example,
a server on VLAN 13 would use 10.1.3.1 as its default gateway.

24 CUMULUS NETWORKS — WHITE PAPER

EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations

interface lo interface vlan13

address 10.0.0.11/32 address 10.1.3.11/24
alias loopback interface address-virtual 44:39:39:ff:00:13
clagd-vxlan-anycast-ip 10.0.0.112 10.1.3.1/24
interface bridge vlan-id 13
bridge-ports bond01 bond02 peerlink vni13 vlan-raw-device bridge
vni24 interface vlan24
bridge-pvid 1 address 10.2.4.11/24
bridge-vids 13 24 address-virtual 44:39:39:ff:00:24
bridge-vlan-aware yes 10.2.4.1/24
interface vni13 vlan-id 24
bridge-access 13 vlan-raw-device bridge
bridge-arp-nd-suppress on router bgp 65011
bridge-learning off bgp router-id 10.0.0.11
mstpctl-bpduguard yes bgp bestpath as-path multipath-relax
mstpctl-portbpdufilter yes neighbor swp51 interface remote-as ex-
mtu 9000 ternal
vxlan-id 13 neighbor swp52 interface remote-as ex-
vxlan-local-tunnelip 10.0.0.11 ternal
interface vni24 address-family ipv4 unicast
bridge-access 24 network 10.0.0.11/32
bridge-arp-nd-suppress on network 10.0.0.112/32
bridge-learning off address-family l2vpn evpn
mstpctl-bpduguard yes neighbor swp51 activate
mstpctl-portbpdufilter yes neighbor swp52 activate
mtu 9000 advertise-all-vni
vxlan-id 24
vxlan-local-tunnelip 10.0.0.11

Figure 20 - Asymmetrical model Leaf01 switch configuration snippet

VRFs may be added for multitenancy, and routing between VXLANs happens only within a VRF.

Looking at the EVPN routing table shown in Figure 21, the four server IP addresses along
with their MAC addresses appear in the table as type 2 routes: 10.1.3.101, 10.2.4.102,
10.1.3.103 and 10.2.4.104:

CUMULUS NETWORKS — WHITE PAPER 25

 EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations

cumulus@leaf01:mgmt-vrf:~$ net show bgp evpn route

BGP table version is 6, local router ID is 10.0.0.11
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-2 prefix: [2]:[ESI]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[ESI]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path

Route Distinguisher: 10.0.0.11:2
*> [2]:[0]:[0]:[48]:[44:38:39:00:08:01]
10.0.0.112 32768 i
*> [2]:[0]:[0]:[48]:[44:38:39:00:08:01]:[32]:[10.1.3.101]
10.0.0.112 32768 i
*> [2]:[0]:[0]:[48]:[44:38:39:00:08:01]:[128]:[fe80::4638:39ff:fe00:801]
10.0.0.112 32768 i
*> [2]:[0]:[0]:[48]:[46:38:39:00:08:01]
10.0.0.112 32768 i
*> [2]:[0]:[0]:[48]:[46:38:39:00:08:02]
10.0.0.112 32768 i
*> [3]:[0]:[32]:[10.0.0.112]
10.0.0.112 32768 i
Route Distinguisher: 10.0.0.11:3
*> [2]:[0]:[0]:[48]:[44:38:39:00:09:01]
10.0.0.112 32768 i
*> [2]:[0]:[0]:[48]:[44:38:39:00:09:01]:[32]:[10.2.4.102]
10.0.0.112 32768 i
*> [2]:[0]:[0]:[48]:[44:38:39:00:09:01]:[128]:[fe80::4638:39ff:fe00:901]
10.0.0.112 32768 i
*> [2]:[0]:[0]:[48]:[46:38:39:00:09:01]
10.0.0.112 32768 i
*> [2]:[0]:[0]:[48]:[46:38:39:00:09:02]
10.0.0.112 32768 i
*> [3]:[0]:[32]:[10.0.0.112]
10.0.0.112 32768 i
Route Distinguisher: 10.0.0.13:2
*> [2]:[0]:[0]:[48]:[44:38:39:00:0a:01]
10.0.0.134 0 65020 65013 i
*> [2]:[0]:[0]:[48]:[44:38:39:00:0a:01]:[32]:[10.1.3.103]
10.0.0.134 0 65020 65013 i
*> [2]:[0]:[0]:[48]:[44:38:39:00:0a:01]:[128]:[fe80::4638:39ff:fe00:a01]
10.0.0.134 0 65020 65013 i
*> [2]:[0]:[0]:[48]:[46:38:39:00:0a:01]
10.0.0.134 0 65020 65013 i
*> [2]:[0]:[0]:[48]:[46:38:39:00:0a:02]
10.0.0.134 0 65020 65013 i
*> [3]:[0]:[32]:[10.0.0.134]
10.0.0.134 0 65020 65013 i
Route Distinguisher: 10.0.0.13:3
*> [2]:[0]:[0]:[48]:[44:38:39:00:0b:01]
10.0.0.134 0 65020 65013 i
*> [2]:[0]:[0]:[48]:[44:38:39:00:0b:01]:[32]:[10.2.4.104]
10.0.0.134 0 65020 65013 i
*> [2]:[0]:[0]:[48]:[44:38:39:00:0b:01]:[128]:[fe80::4638:39ff:fe00:b01]
10.0.0.134 0 65020 65013 i
*> [2]:[0]:[0]:[48]:[46:38:39:00:0b:01]
10.0.0.134 0 65020 65013 i
*> [2]:[0]:[0]:[48]:[46:38:39:00:0b:02]
10.0.0.134 0 65020 65013 i
*> [3]:[0]:[32]:[10.0.0.134]
10.0.0.134 0 65020 65013 i

Figure 21 - EVPN routing table with asymmetric IRB model

26 CUMULUS NETWORKS — WHITE PAPER

EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations

Route Distinguisher: 10.0.0.14:2

*> [2]:[0]:[0]:[48]:[44:38:39:00:0a:01]
10.0.0.134 0 65020 65014 i
*> [2]:[0]:[0]:[48]:[44:38:39:00:0a:01]:[32]:[10.1.3.103]
10.0.0.134 0 65020 65014 i
*> [2]:[0]:[0]:[48]:[44:38:39:00:0a:01]:[128]:[fe80::4638:39ff:fe00:a01]
10.0.0.134 0 65020 65014 i
*> [2]:[0]:[0]:[48]:[46:38:39:00:0a:01]
10.0.0.134 0 65020 65014 i
*> [2]:[0]:[0]:[48]:[46:38:39:00:0a:02]
10.0.0.134 0 65020 65014 i
*> [3]:[0]:[32]:[10.0.0.134]
10.0.0.134 0 65020 65014 i
Route Distinguisher: 10.0.0.14:3
*> [2]:[0]:[0]:[48]:[44:38:39:00:0b:01]
10.0.0.134 0 65020 65014 i
*> [2]:[0]:[0]:[48]:[44:38:39:00:0b:01]:[32]:[10.2.4.104]
10.0.0.134 0 65020 65014 i
*> [2]:[0]:[0]:[48]:[44:38:39:00:0b:01]:[128]:[fe80::4638:39ff:fe00:b01]
10.0.0.134 0 65020 65014 i
*> [2]:[0]:[0]:[48]:[46:38:39:00:0b:01]
10.0.0.134 0 65020 65014 i
*> [2]:[0]:[0]:[48]:[46:38:39:00:0b:02]
10.0.0.134 0 65020 65014 i
*> [3]:[0]:[32]:[10.0.0.134]
10.0.0.134 0 65020 65014 i
Displayed 36 prefixes (36 paths)

Figure 21 - EVPN routing table with asymmetric IRB model (continued)

Looking at the kernel routing table, we can see that to get to either subnet, we access it
directly on the same leaf switch as shown in Figure 22.

cumulus@leaf01:mgmt-vrf:~$ net show route

show ip route
=============
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel,
> - selected route, * - FIB route
C>* 10.0.0.11/32 is directly connected, lo, 00:00:53
B>* 10.0.0.12/32 [20/0] via fe80::4638:39ff:fe00:701, swp52, 00:00:47
* via fe80::4638:39ff:fe00:601, swp51, 00:00:47
B>* 10.0.0.13/32 [20/0] via fe80::4638:39ff:fe00:701, swp52, 00:00:47
* via fe80::4638:39ff:fe00:601, swp51, 00:00:47
B>* 10.0.0.14/32 [20/0] via fe80::4638:39ff:fe00:701, swp52, 00:00:47
* via fe80::4638:39ff:fe00:601, swp51, 00:00:47
B>* 10.0.0.21/32 [20/0] via fe80::4638:39ff:fe00:601, swp51, 00:00:47
B>* 10.0.0.22/32 [20/0] via fe80::4638:39ff:fe00:701, swp52, 00:00:47
C>* 10.0.0.112/32 is directly connected, lo, 00:00:42
B>* 10.0.0.134/32 [20/0] via fe80::4638:39ff:fe00:601, swp51, 00:00:40
* via fe80::4638:39ff:fe00:701, swp52, 00:00:40
C * 10.1.3.0/24 is directly connected, vlan13-v0, 00:00:44
C>* 10.1.3.0/24 is directly connected, vlan13, 00:00:44
C * 10.2.4.0/24 is directly connected, vlan24-v0, 00:00:44
C>* 10.2.4.0/24 is directly connected, vlan24, 00:00:44
C>* 169.254.1.0/30 is directly connected, peerlink.4094, 00:00:49
<snip>

Figure 22 - Leaf routing table with asymmetric IRB model

CUMULUS NETWORKS — WHITE PAPER 27

 EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations

This setup is available on Cumulus Network GitHub site.

DISTRIBUTED VXLAN ROUTING WITH THE SYMMETRIC IRB MODEL

An architecture using the symmetric model is depicted below in Figure 23.

Internet

Lo=172.16.1.1
Layer 3 Network
(BGP unnumbered underlay) 0.0.0.0/0

Spine01 Spine02

Leaf01 Leaf02 Leaf03 Leaf04 VRF1 VRF1

VRF1 VTEP VRF1 VRF1 VTEP VRF1 VTEP VTEP

Anycast gateway SVIs SVIs SVIs SVIs Exit 01 Exit 02

10.1.3.101 10.2.4.102 10.1.3.103 10.2.4.104

VLAN 24 VLAN 13 VLAN 24 VLAN 13

Server01 Server02 Server03 Server04

VXLAN Tunnel, VNI 24

VXLAN T

VXLAN Tunnel, VNI 104001, L3VNI

Figure 23 - VXLAN routing with a symmetric IRB model architecture

For purposes of this example, we deploy four servers and two VLANs. The servers use
MLAG to connect to the leaf switches for redundancy and are configured with an anycast
default gateway. The same virtual anycast IP address is configured on each switch's
southbound SVI, using VRR, towards the servers.

Each of the four leafs host two SVIs, one terminating with VLAN 13 and one with VLAN
24. VXLAN distributed routing occurs on these leafs. Each leaf switch also has VNIs 13
and 24 configured for the VXLAN tunnels to layer 2 transport both VLAN 13 and VLAN 24
respectively. Specifically for the symmetric routing model, each leaf and exit switch also
hosts VLAN 4001 (the transport VLAN) and VNI 104001 (the L3VNI).

All leaf switches host vrf1, and all servers are located in vrf1. Since we are using EVPN,
there is no need to configure vrf1 on the spine switches.

This design has no hosts attached to border leafs, but attaching hosts to the border leafs
as well is fully supported.

For this design, we generate a default route (0.0.0.0/0) from the Internet router and advertise
that default route to the exit leafs via the BGP IPv4 address family. A sample snippet
configuration for peering and generating a default route via eBGP unnumbered is shown in
Figure 24:

28 CUMULUS NETWORKS — WHITE PAPER

EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations

interface swp1
ipv6 nd ra-interval 10
no ipv6 nd suppress-ra
interface swp2
ipv6 nd ra-interval 10
no ipv6 nd suppress-ra
router bgp 25253
bgp router-id 10.0.0.253
bgp bestpath as-path multipath-relax
neighbor swp1 interface remote-as external
neighbor swp2 interface remote-as external
!
address-family ipv4 unicast
network 10.0.0.253/32
neighbor swp1 default-originate
neighbor swp2 default-originate
exit-address-family

Figure 24 - Internet router configuration snippet

The exit leaf's interface that peers with the Internet router (swp44) is placed in vrf1, which
also places the default route in vrf1. The Exit01 router injects vrf1's default route into
vrf1's EVPN address family as a type 5 route with the command advertise ipv4 unicast, as
highlighted below. EVPN then advertises this route via the local VLAN4001, which is also in
vrf1, across the L3VNI (VNI 104001) to all other leafs in the network that participate in vrf1.
The local leafs inject the type 5 default route into its vrf1 routing table. No other VNIs are
configured on the exit leafs since no hosts are directly connected to the exit leafs.

A sample configuration snippet from Exit01 is shown in Figure 25. Note the advertise ipv4
unicast command is only needed on routers where you want to redistribute IPv4 address
family routes into L2VPN EVPN address family routes, which is generally on the border/
exit leafs. We advertise the VXLAN subnets to the Internet router. Also, in this case we
need only the L3VNI on the exit leaf as there are no hosts attached to that leaf. If hosts
were attached to those leafs, we would also need the VNIs/SVIs configured here as well, as
shown in the leaf snippet.

CUMULUS NETWORKS — WHITE PAPER 29

 EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations

interface bridge router bgp 65041

bridge-ports vxlan4001 bgp router-id 10.0.0.41
bridge-vlan-aware yes coalesce-time 1100
bgp bestpath as-path multipath-relax
interface vlan4001 neighbor swp51 interface remote-as external
vlan-id 4001 neighbor swp52 interface remote-as external
vlan-raw-device bridge
vrf vrf1 address-family ipv4 unicast
network 10.0.0.41/32
interface swp44
vrf vrf1 address-family l2vpn evpn
neighbor swp51 activate
interface vxlan4001 neighbor swp52 activate
bridge-access 4001 advertise-all-vni
bridge-learning off
vxlan-id 104001 router bgp 65041 vrf vrf1
vxlan-local-tunnelip 10.0.0.41 bgp router-id 10.0.0.41
network 10.1.3.0/24 # advertise VLAN to Internet
vrf vrf1 network 10.2.4.0/24 # advertise VLAN to Internet
vni 104001 coalesce-time 1050
neighbor swp44 interface remote-as external
interface swp44 vrf vrf1
ipv6 nd ra-interval 10 address-family l2vpn evpn
no ipv6 nd suppress-ra advertise ipv4 unicast # sends Type 5 routes

Figure 25 - Exit01 configuration snippet

The Exit02 switch is configured similarly to Exit01.

The spine switches are configured for both the IPv4 unicast address family as well as
the EVPN address family, which is fairly straightforward, as shown below. A snippet from
Spine01 is shown below. Interfaces swp1-4 connect to the leaf switches, and swp29-30
connect to the exit switches. Spine02 is configured similarly.

In a distributed architecture, VXLAN routing is done on the leaf switches. A snippet of a

sample configuration from a leaf switch is in Figure 26.

30 CUMULUS NETWORKS — WHITE PAPER

EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations

interface bridge interface vlan13

bridge-ports bond01 bond02 peerlink address 10.1.3.11/24
vni13 vni24 vxlan4001 address-virtual 44:39:39:ff:00:13 10.1.3.1/24
bridge-pvid 1 vlan-id 13
bridge-vids 13 24 vlan-raw-device bridge
bridge-vlan-aware yes vrf vrf1
interface vni13 interface vlan24
bridge-access 13 address 10.2.4.11/24
bridge-learning off address-virtual 44:39:39:ff:00:24 10.2.4.1/24
mstpctl-bpduguard yes vlan-id 24
mstpctl-portbpdufilter yes vlan-raw-device bridge
mtu 9000 vrf vrf1
vxlan-id 13 interface vlan4001
vxlan-local-tunnelip 10.0.0.11 hwaddress 44:39:39:FF:40:94
interface vni24 vlan-id 4001
bridge-access 24 vlan-raw-device bridge
bridge-arp-nd-suppress on vrf vrf1
bridge-learning off vrf vrf1
mstpctl-bpduguard yes vni 104001
mstpctl-portbpdufilter yes router bgp 65011
mtu 9000 bgp router-id 10.0.0.11
vxlan-id 24 bgp bestpath as-path multipath-relax
vxlan-local-tunnelip 10.0.0.11 neighbor swp51 interface remote-as external
interface vxlan4001 neighbor swp52 interface remote-as external
bridge-access 4001 address-family ipv4 unicast
bridge-learning off network 10.0.0.11/32
vxlan-id 104001 address-family l2vpn evpn
vxlan-local-tunnelip 10.0.0.11 neighbor swp51 activate
interface vrf1 neighbor swp52 activate
vrf-table auto advertise-all-vni
interface vlan13
address 10.1.3.11/24
address-virtual 44:39:39:ff:00:13
10.1.3.1/24
vlan-id 13
vlan-raw-device bridge
vrf vrf1

Figure 26 - Leaf01 switch configuration snippet for symmetric IRB model

Although not depicted above for brevity, the leaf's southbound ports are placed in vrf1
and a tenant VLAN. Each tenant VLAN is mapped to a VNI that provides the layer 2
VXLAN tunneling. In addition, VNI 104001 (associated with interface vxlan4001) and VLAN
4001 are configured to create the L3VNI. This provides the hop between the local VLANs
and the L3VNI.

Finally, let's look at the leaf's EVPN routing table in Figure 27, showing both type 2 and type
5 routes (the type 3 routes provide VTEP discovery). We can see the type 2 routes to the
servers in the data center, as well as the type 5 routes to reach outside the data center. Part
of the routing table has been left off for brevity.

CUMULUS NETWORKS — WHITE PAPER 31

 EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations

cumulus@leaf01:mgmt-vrf:~$ net show bgp evpn route

BGP table version is 10, local router ID is 10.0.0.11
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-2 prefix: [2]:[ESI]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[ESI]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path

Route Distinguisher: 10.0.0.11:2
*> [2]:[0]:[0]:[48]:[44:38:39:00:00:17]
10.0.0.112 32768 i
*> [2]:[0]:[0]:[48]:[44:38:39:00:00:17]:[32]:[10.1.3.101]
10.0.0.112 32768 i
* [3]:[0]:[32]:[10.0.0.134]
10.0.0.134 0 65020 65014 i
<snip>
Route Distinguisher: 10.0.0.14:3
* [2]:[0]:[0]:[48]:[44:38:39:00:00:32]
10.0.0.134 0 65020 65014 i
*> [2]:[0]:[0]:[48]:[44:38:39:00:00:32]
10.0.0.134 0 65020 65014 i
* [2]:[0]:[0]:[48]:[44:38:39:00:00:32]:[32]:[10.2.4.104]
10.0.0.134 0 65020 65014 i
*> [2]:[0]:[0]:[48]:[44:38:39:00:00:32]:[32]:[10.2.4.104]
10.0.0.134 0 65020 65014 i
* [2]:[0]:[0]:[48]:[44:38:39:00:00:32]:[128]:[fe80::4638:39ff:fe00:32]
10.0.0.134 0 65020 65014 i
*> [2]:[0]:[0]:[48]:[44:38:39:00:00:32]:[128]:[fe80::4638:39ff:fe00:32]
10.0.0.134 0 65020 65014 i
* [2]:[0]:[0]:[48]:[46:38:39:00:00:23]
10.0.0.134 0 65020 65014 i
*> [2]:[0]:[0]:[48]:[46:38:39:00:00:23]
10.0.0.134 0 65020 65014 i
*> [2]:[0]:[0]:[48]:[46:38:39:00:00:32]
10.0.0.134 0 65020 65014 i
* [2]:[0]:[0]:[48]:[46:38:39:00:00:32]
10.0.0.134 0 65020 65014 i
*> [3]:[0]:[32]:[10.0.0.134]
10.0.0.134 0 65020 65014 i
* [3]:[0]:[32]:[10.0.0.134]
10.0.0.134 0 65020 65014 i
Route Distinguisher: 10.0.0.41:2
* [5]:[0]:[0]:[0.0.0.0]
10.0.0.41 0 65020 65041 25253 i
*> [5]:[0]:[0]:[0.0.0.0]
10.0.0.41 0 65020 65041 25253 i
Route Distinguisher: 10.0.0.42:2
* [5]:[0]:[0]:[0]:[0.0.0.0]
10.0.0.42 0 65020 65042 25253 i
*> [5]:[0]:[0]:[0]:[0.0.0.0]
10.0.0.42 0 65020 65042 25253 i
Displayed 36 prefixes (61 paths)

Figure 27 - EVPN routing table with symmetric IRB model

Looking in the leaf's routing table in Figure 28, we can see all the routes are available in vrf1
as well as the default route that is reachable over vlan4001, which is attached to the L3VNI:

32 CUMULUS NETWORKS — WHITE PAPER

EVPN: NetQ for EVPN operations

cumulus@leaf01:mgmt-vrf:~$ net show route vrf vrf1

show ip route vrf vrf1

=======================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
> - selected route, * - FIB route

VRF vrf1:
B>* 0.0.0.0/0 [20/0] via 10.0.0.41, vlan4001 onlink, 00:05:53 #Type 5 with L3VNI
* via 10.0.0.42, vlan4001 onlink, 00:05:53
K * 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 00:10:03
B>* 10.0.0.253/32 [20/0] via 10.0.0.41, vlan4001 onlink, 00:05:53
* via 10.0.0.42, vlan4001 onlink, 00:05:53
C * 10.1.3.0/24 is directly connected, vlan13-v0, 00:10:03
C>* 10.1.3.0/24 is directly connected, vlan13, 00:10:03
B>* 10.1.3.103/32 [20/0] via 10.0.0.134, vlan4001 onlink, 00:04:56 #Type 2 with L3VNI
C * 10.2.4.0/24 is directly connected, vlan24-v0, 00:10:03
C>* 10.2.4.0/24 is directly connected, vlan24, 00:10:03
B>* 10.2.4.104/32 [20/0] via 10.0.0.134, vlan4001 onlink, 00:04:16 #Type 2 with L3VNI

Figure 28 - Symmetric routing table

A virtual setup with this topology is available at Cumulus Linux GitHub site.

NetQ for EVPN operations

Cumulus NetQ greatly simplifies EVPN deployment and operations. NetQ provides
configuration checks and simplifies troubleshooting EVPN in a network from one location
anywhere in the network.

Upon initial setup, it is wise to check the configurations to be sure nothing is missing
and all VNIs are configured — NetQ can accomplish this in simply one step, as shown
in Figure 29 below.

cumulus@oob-mgmt-server:~$ netq check evpn

Total Nodes: 10, Failed Nodes: 1, Total Sessions: 8 , Failed Sessions: 1, Total VNIs: 2
Hostname Peer Name Peer Hostname Error Last Changed
--------- ----------- ------------- ------------------------- ------------
leaf04 swp51 spine01 AFI/SAFI evpn not activated on peer 1m:35.655s

Figure 29 - NetQ `check evpn`

In this example, we are missing activating the L2VPN EVPN address family on spine01. We
can also check BGP and it will tell us the same information as shown in Figure 30.

CUMULUS NETWORKS — WHITE PAPER 33

 EVPN: NetQ for EVPN operations

cumulus@oob-mgmt-server:~$ netq check bgp

Total Nodes: 10, Failed Nodes: 2, Total Sessions: 16 , Failed Sessions: 2,
Hostname Peer Name Peer Hostname Reason Last Changed
---------- ---------- ------------- --------------------------------------------- ---------
leaf04 swp51 spine01 evpn address families not activated on peer 34.140580s
spine01 swp4 leaf04 evpn address families not activated on node 33.140629s

Figure 30 - NetQ `check bgp`

After this is fixed, all will look good, as shown in Figure 31 below.

cumulus@oob-mgmt-server:~$ netq check evpn

Total Nodes: 10, Failed Nodes: 0, Total Sessions: 8, Failed Sessions: 0, Total VNIs: 2
cumulus@oob-mgmt-server:~$ netq check bgp
Total Nodes: 10, Failed Nodes: 0, Total Sessions: 16, Failed Sessions: 0

Figure 31 - Good EVPN and BGP configurations

NetQ also displays all the VNIs in the entire network in one simple command, to check and
be sure the correct VNIs are where they belong, as shown in Figure 32:

cumulus@oob-mgmt-server:~$ netq show evpn vni 13

Matching evpn records:
Hostname VNI VTEP IP In Kernel Export RT Import RT Last Changed
------------ ---- ---------- --------- ---------- ----------- --------------
leaf01 13 10.0.0.112 yes 65011:13 65011:13 15m:57.782s
leaf02 13 10.0.0.112 yes 65012:13 65012:13 16m:0.939s
leaf03 13 10.0.0.134 yes 65013:13 65013:13 15m:56.334s
leaf04 13 10.0.0.134 yes 65014:13 65014:13 15m:57.467s

Figure 32 - Showing all VNIs on a node

We can also easily display all VNIs per node, as shown in Figure 33:

cumulus@oob-mgmt-server:~$ netq leaf01 show evpn

Matching evpn records:
Hostname VNI VTEP IP In Kernel Export RT Import RT Last Changed
---------- ------ --------- -------- ----------- ---------- ------------
leaf01 13 10.0.0.112 yes 65011:13 65011:13 17m:37.430s
leaf01 24 10.0.0.112 yes 65011:24 65011:24 17m:37.430s

Figure 33 - NetQ displays all VNIs on a node

This is just a few examples of how NetQ can help with operations in a EVPN environment.
There are many more features available. Visit our NetQ User's Guide for more information.

34 CUMULUS NETWORKS — WHITE PAPER

EVPN: Conclusion

Conclusion
Modern data centers deploy layer 3 with the BGP routing protocol in order to scale and
provide a robust, easy to troubleshoot infrastructure that also supports a multi-vendor
environment. However, since some applications still require layer 2 connectivity, VXLAN
tunnels, deployed over a layer 3 fabric, have become a popular way to achieve layer 2
connectivity between racks.

In order for the VXLAN tunnels to reach each other or the outside world, VXLAN routing
must be enabled. Newer merchant silicon supports this functionality directly in the ASIC,
thereby making a cost effective and simple deployment. EVPN, which is already popular
for VXLAN bridging, now integrates with VXLAN routing, unifying the control plane and
streamlining deployment.

Cumulus Linux EVPN is the ideal control plane solution for VXLAN routing. It uses the
same routing protocol preferred for data center infrastructures — BGP — for both VXLAN
bridging and routing. Additionally, it has inherent support for multitenancy.

Try it out for yourself on this ready-to-go demo using Cumulus VX with NetQ and Vagrant
or try it out in Cumulus in the Cloud.

ABOUT CUMULUS NETWORKS®

Cumulus Networks is leading the transformation of bringing web-scale networking to enterprise cloud. Its network
switch, Cumulus Linux, is the only solution that allows you to affordably build and efficiently operate your network like the
world’s largest data center operators, unlocking vertical network stacks. By allowing operators to use standard hardware
components, Cumulus Linux offers unprecedented operational speed and agility, at the industry’s most competitive cost.
Cumulus Networks has received venture funding from Andreessen Horowitz, Battery Ventures, Capital, Peter Wagner and
four of the original VMware founders.

For more information visit cumulusnetworks.com or follow @cumulusnetworks.

©2018 Cumulus Networks. All rights reserved. CUMULUS, the Cumulus Logo, CUMULUS NETWORKS, and the Rocket Turtle Logo (the “Marks”) are
trademarks and service marks of Cumulus Networks, Inc. in the U.S. and other countries. You are not permitted to use the Marks without the prior
written consent of Cumulus Networks. The registered trademark Linux® is used pursuant to a sublicense from LMI, the exclusive licensee of Linus
Torvalds, owner of the mark on a worldwide basis. All other marks are used under fair use or license from their respective owners.

04032018

CUMULUS NETWORKS — WHITE PAPER 35