06 - VXLAN Part VI VXLAN BGP EVPN - Basic Configurations
06 - VXLAN Part VI VXLAN BGP EVPN - Basic Configurations
06 - VXLAN Part VI VXLAN BGP EVPN - Basic Configurations
In my previous post “VXLAN Part V: Flood and Learn”, I have shown, how VXLAN works
without Control Plane protocol. In this post, I am going to show how to configure BGP
EVPN on VXLAN fabric.
In Figure 1, you can see the high-level overview of our example VXLAN fabric design. We
have one vrf context (=tenant) TENANT77 spread over the two VTEPs. We also have two
VLANs; VLAN 10 (attached to L2VNI 10000) and VLAN 20 (attached to L2VNI 20000). On
each VTEPs there are two connected hosts (Cafe and Abba on VTEP-101, Beef, and Babe
on VTEP-102). The cross VLAN flows between the hosts in different VTEPs is routed over
the L3VNI 10077. The reason why I start with the configurations is that I want to use
show commands as well as Wireshark captures while explaining the theory in my next
post.
Configuration
Enabling features
First, we need to enable vxlan and related features as well as routing protocols needed
for underlay and overlay:
nv overlay evpn
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
The rest of the configurations are divided into two main parts:
Control Plane and tenant configuration (BGP, VRF Context, and EVPN)
Adding a customer network to the tenant
Configuring BGP
In our example, all switches belong to AS65000. Spine-11 is BGP Route Reflector (RR)
and VTEPs are RR clients. I am going to use dedicated loopback IP addresses for the BGP
peering even though we could also use the same address used with OSPF RID. The
reason for dedicated IP address for BGP and OSPF is that I want to draw a clear line
between the protocols used in Underlay and Overlay networks. In this his way, we can
simplify the troubleshooting process.
We want to send and receive the BGP EVPN NLRIs (Network Layer Reachability
Information = routing updates), that is why the “address-family l2vpn evpn” is needed in
addition to ipv4 unicast afi. What address-family actually is? Well, it describes the type
of the information that is carried inside the NLRI (IPv4, IPv6, vpnv4, evpn…).The
Address-Family identifier (AFI) number for Layer2 NLRI information is 25 and the
Subsequent AFI (SAFI) for EVPN is 70. Under the l2vpn afi, we define the BGP
community types that we want to carry with BGP update messages. We are going to use
Route-Targets (RT) for importing/exporting routes to and from the BGP process. Since
RTs are extended communities and only standard BGP communities are added to NLRI
by default, we need to add them to the address-family l2vpn evpn configuration.
router bgp 65000
router-id 192.168.77.101
address-family ipv4 unicast
address-family l2vpn evpn
neighbor 192.168.77.11
remote-as 65000
description ** Spine-11 BGP-RR **
update-source loopback77
address-family l2vpn evpn
send-community extended
!
interface loopback77
description ** BGP peering **
ip address 192.168.77.101/32
ip router ospf UNDERLAY-NET area 0.0.0.0
A couple of words about the IP addressing and IP connectivity. In figure 2, we can see
that there are three logical Loopback interfaces in each VTEP switch.
Loopback 100: is used for VXLAN tunnel addressing. NVE 1 interface use Loopback 100
as a source interface.
Loopback 77: Is used for BGP peering. The “MP_REACH_NLRI” Path Attribute in BGP
Update message use the ip address of the NVE 1 interface in the “Next Hop Address”
field. The tunnel address has to be the next-hop-address of all NLRIs and if eBGP is used
Spine switches have to retain the original next-hop-address while forwarding the routing
update. Note that BGP RR does not change ANY of the Path Attributes of the reflected
route, so the source address in our case is retained automatically.
I have written the article “VXLAN Part X: Recovery issue when BGP EVPN peering uses
the same loopback interface as a source than VXLAN NVE1 interface” in which the
meaning of Loopback addresses is analyzed in more detail.
Figure 2: BGP and IP addressing
We can verify the BGP peering with show bgp l2vpn evpn summary.
VRF context in VXLAN fabric has a dedicated Virtual Network Id (VNI). When routing
traffic between two hosts behind the different VTEPs in different subnets, packets are
routed over the L3VNI (Figure 3). VXLAN headers for these routed packets uses L3VNI
instead of L2VNI. We are using symmetric Integrated Route and Bridge (IRB) model
where all routed traffic inside a tenant will use the same L3VNI.
Note! I am using term “vrf” for virtual routing inside a single box (local). I am using
term “tenant” while speaking about the virtual L2/l3 domain spread over the fabric
Figure 3: Routing over between different subnets.
We will set up the vrf context TENANT77 and attach L3VNI 10077 to it (Figure 4). Since
we use MP-BGP, we also need to define a Route Distinguisher (RD), as well as Route
Targets (RT) specified under the ipv4 unicast afi (routed traffic is Unicast).
Address-family IPv4 unicast in vrf context is used for exporting/importing routes with
BGP process. To be able to do that, we also need to attach RT values in each BGP NLRI
updates. Since RTs are used for import/export policy, RTs has to be consistent in each
VTEP switch. We will use RT auto format, which generates the RT values by combining
BGP AS number and L3VNI. Since we are using iBGP peering (all switches belongs to
same AS), we can use the auto-generation mode. If each VTEPs are in its own AS
(eBGP) then manual mode has to be used, otherwise we end up the situation where each
VTEP has a different value for RT and even though routes will successfully be exported to
BGP, no one will import those.
After creating the vrf context, we are going to attach it to BGP process.
As can be seen from the output below, the BGP RID for Leaf-101 is 192.168.77.101 and
the TENANT77 VRF_ID is 3. These together give us auto-generated RD value
192.168.77.101:3.
Leaf-101# sh vrf
VRF-Name VRF-ID State
Reason
TENANT77 3 Up --
For a routed packet, we need a layer 3 interface and layer 2 vlan. First, we create layer
2 vlan (in our case with id 77) and assign it to vn-segment 10077. Next, we create a
layer 3 interface for the vlan and attach it to the vrf context TENANT77. Layer 3 interface
does not have an ip address and we are going to use the command “ip forward”, which
allows ipv4 traffic on an interface that has no ip address.
As the last configuration step, I am going to add two customer subnets in our example
VXLAN fabric. We are going to create two VLANs 10 and 20. First, we create layer 2 vlan
and attach it to vn-segment (vlan 10 = VNI 10000 and vlan 20 = VNI 20000). We are
using anycast-gateway ip address (AGW IP), where the gateway ip for the specific
subnet is the same in all VTEPs (vlan 10 = 192.168.11.1 and vlan 20 = 192.168.12.1).
Anycast gateway in VXLAN fabric uses AGW MAC address, which is the same across all
VTEPs and all of the subnets. We are going to use AGW MAC 0001.0001.0001. Customer
layer 3 interfaces are attached to vrf context TENANT77.
The last thing to do is attach VNIs associated with vlan to NVE interface. Note that we
are using the same mcast group for bum traffic of both VLANs. We are also using ARP-
suppression to prevent unnecessary ARP flooding. Even though not shown in the
configuration we need to configure the host-facing interfaces to correct vlan.
Note! When a host joins to network, it might use some Address Conflict Detection
mechanism to prevent duplicate ip addresses. This can be done with Gratuitous ARP,
where a host sends an ARP request by using its own ip addresses in both Sender- and
Target IP address fields (see Figure 10 in Appendix 1.). Based on normal mac learning
process, VTEP switch learns the mac/ip addresses of connected host and then send a
BGP EVPN update to other VTEPs. Note also that the ARP suppression is L2VNI
specific.
We are going to test basic connectivity between the hosts with ping.
Cafe#ping 192.168.11.11
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.11.11, timeout is 2
seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max =
1/1/2 ms
Cafe#ping 192.168.12.11
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.12.11, timeout is 2
seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max =
2/8/13 ms
Cafe#ping 192.168.12.12
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.12.12, timeout is 2
seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max =
20/23/29 ms
That’s it. I will go through the operation and theory of the VXLAN BGP EVPN from both
Control and Data Plane in my next post.
References:
Building Data Center with VXLAN BGP EVPN – A Cisco NX-OS Perspective
ISBN-10: 1-58714-467-0 – Krattiger Lukas, Shyam Kapadia, and Jansen Davis
Gratuitous ARP
This Wireshark capture is taken during the time that host Cafe joins to the network for
the very first time.
Figure 10: Gratuitous ARP sends by host cafe when joining the network.
Leaf-101
Leaf-101# sh run
version 7.0(3)I7(1)
hostname Leaf-101
vdc Leaf-101 id 1
limit-resource vlan minimum 16 maximum 4094
limit-resource vrf minimum 2 maximum 4096
limit-resource port-channel minimum 0 maximum 511
limit-resource u4route-mem minimum 128 maximum 128
limit-resource u6route-mem minimum 96 maximum 96
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8
nv overlay evpn
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
no password strength-check
username admin password 5
$5$aV2kcO97$7ioNn2XTmsfuFj62MLL/wcMnEoJE9ifSY/AFfWPY2/
/ role network-admin
ip domain-lookup
ip host Spine-12 192.168.0.12
snmp-server user admin network-admin auth md5
0x223cfb63ca87c5b4856c960235329cff
priv 0x223cfb63ca87c5b4856c960235329cff localizedkey
rmon event 1 description FATAL(1) owner PMON@FATAL
rmon event 2 description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 description ERROR(3) owner PMON@ERROR
rmon event 4 description WARNING(4) owner PMON@WARNING
rmon event 5 description INFORMATION(5) owner PMON@INFO
interface Vlan1
no shutdown
interface Vlan10
no shutdown
vrf member TENANT77
ip address 192.168.11.1/24
fabric forwarding mode anycast-gateway
interface Vlan20
no shutdown
vrf member TENANT77
ip address 192.168.12.1/24
fabric forwarding mode anycast-gateway
interface Vlan77
no shutdown
vrf member TENANT77
ip forward
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback100
member vni 10000
suppress-arp
mcast-group 238.0.0.10
member vni 10077 associate-vrf
member vni 20000
suppress-arp
mcast-group 238.0.0.10
interface Ethernet1/1
no switchport
medium p2p
ip unnumbered loopback0
ip ospf network point-to-point
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
no switchport
medium p2p
ip unnumbered loopback0
ip ospf network point-to-point
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/3
switchport access vlan 10
interface Ethernet1/4
switchport access vlan 20
interface mgmt0
vrf member management
interface loopback0
description ** RID/Underlay **
ip address 192.168.0.101/32
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
interface loopback77
description ** BGP peering **
ip address 192.168.77.101/32
ip router ospf UNDERLAY-NET area 0.0.0.0
interface loopback100
description ** VTEP/Overlay **
ip address 192.168.100.101/32
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
line console
line vty
router ospf UNDERLAY-NET
router-id 192.168.0.101
name-lookup
router bgp 65000
router-id 192.168.77.101
address-family ipv4 unicast
address-family l2vpn evpn
neighbor 192.168.77.11
remote-as 65000
description ** Spine-11 BGP-RR **
update-source loopback77
address-family l2vpn evpn
send-community extended
vrf TENANT77
address-family ipv4 unicast
advertise l2vpn evpn
evpn
vni 10000 l2
rd auto
route-target import auto
route-target export auto
vni 20000 l2
rd auto
route-target import auto
route-target export auto
Leaf-101#
Leaf-102
Leaf-102# sh run
version 7.0(3)I7(1)
hostname Leaf-102
vdc Leaf-102 id 1
limit-resource vlan minimum 16 maximum 4094
limit-resource vrf minimum 2 maximum 4096
limit-resource port-channel minimum 0 maximum 511
limit-resource u4route-mem minimum 128 maximum 128
limit-resource u6route-mem minimum 96 maximum 96
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8
nv overlay evpn
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
interface Vlan1
no shutdown
interface Vlan10
no shutdown
vrf member TENANT77
ip address 192.168.11.1/24
fabric forwarding mode anycast-gateway
interface Vlan20
no shutdown
vrf member TENANT77
ip address 192.168.12.1/24
fabric forwarding mode anycast-gateway
interface Vlan77
no shutdown
vrf member TENANT77
ip forward
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback100
member vni 10000
suppress-arp
mcast-group 238.0.0.10
member vni 10077 associate-vrf
member vni 20000
suppress-arp
mcast-group 238.0.0.10
interface Ethernet1/1
no switchport
medium p2p
ip unnumbered loopback0
ip ospf network point-to-point
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
no switchport
medium p2p
ip unnumbered loopback0
ip ospf network point-to-point
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/3
switchport access vlan 10
interface Ethernet1/4
switchport access vlan 20
interface mgmt0
vrf member management
interface loopback0
description ** RID/Underlay **
ip address 192.168.0.102/32
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
interface loopback77
description ** BGP peering **
ip address 192.168.77.102/32
ip router ospf UNDERLAY-NET area 0.0.0.0
interface loopback100
description ** VTEP/Overlay **
ip address 192.168.100.102/32
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
line console
line vty
router ospf UNDERLAY-NET
router-id 192.168.0.102
name-lookup
router bgp 65000
router-id 192.168.77.102
address-family ipv4 unicast
address-family l2vpn evpn
neighbor 192.168.77.11
remote-as 65000
description ** Spine-11 BGP-RR **
update-source loopback77
address-family l2vpn evpn
send-community extended
vrf TENANT77
address-family ipv4 unicast
advertise l2vpn evpn
evpn
vni 10000 l2
rd auto
route-target import auto
route-target export auto
vni 20000 l2
rd auto
route-target import auto
route-target export auto
Leaf-102#
Spine-11
Spine-11# sh run
nv overlay evpn
feature ospf
feature bgp
feature pim
feature vn-segment-vlan-based
feature nv overlay
no password strength-check
username admin password 5
$5$60DVUPIV$uZWPu6ufHQOJSG18SK5b9/5kpZnV5E4/EFapzQP5CI
/ role network-admin
ip domain-lookup
ip host Spine-12 192.168.0.12
ip host Leaf-102 192.168.0.102
snmp-server user admin network-admin auth md5
0xd177fd3448eab21dd2feb16d54938469
priv 0xd177fd3448eab21dd2feb16d54938469 localizedkey
rmon event 1 description FATAL(1) owner PMON@FATAL
rmon event 2 description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 description ERROR(3) owner PMON@ERROR
rmon event 4 description WARNING(4) owner PMON@WARNING
rmon event 5 description INFORMATION(5) owner PMON@INFO
interface Ethernet1/1
no switchport
medium p2p
ip unnumbered loopback0
ip ospf network point-to-point
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
no switchport
medium p2p
ip unnumbered loopback0
ip ospf network point-to-point
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
no shutdown
interface mgmt0
vrf member management
interface loopback0
description ** RID/Underlay **
ip address 192.168.0.11/32
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
interface loopback77
description ** BGP peering **
ip address 192.168.77.11/32
ip router ospf UNDERLAY-NET area 0.0.0.0
interface loopback238
description ** Anycast-RP address **
ip address 192.168.238.6/29
ip ospf network point-to-point
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
line console
line vty
router ospf UNDERLAY-NET
router-id 192.168.0.11
name-lookup
router bgp 65000
router-id 192.168.77.111
address-family ipv4 unicast
address-family l2vpn evpn
neighbor 192.168.77.101
remote-as 65000
update-source loopback77
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 192.168.77.102
remote-as 65000
update-source loopback77
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
Spine-11#