EVPN IRB With Vxlan Underlay
EVPN IRB With Vxlan Underlay
eos.arista.com/eos-4-20-1f/evpn-irb-with-vxlan-underlay
Contents [hide]
To provide a more optimal forwarding model and avoid traffic tromboning, the EVPN inter-
subnet draft (draft-sajassi-l2vpn-evpn-inter-subnet-forwarding) proposes integrating the
routing and bridging (IRB) functionality directly onto the VTEP, thereby allowing the routing
operation to occur as close to the end host as possible. The draft proposes two forwarding
models for the IRB functionality, which are termed asymmetric IRB and symmetrical IRB.
Asymmetric IRB
In the asymmetric IRB model, the inter-subnet routing functionality is performed by the
ingress VTEP, with the packet after the routing action being VXLAN bridged to the
destination VTEP. The egress VTEP only then needs to remove the VXLAN header and
forward the packet onto the local layer 2 domain based on the VNI to VLAN mapping. In the
return path, the routing functionality is reversed with the destination VTEP now performing
the ingress routing and VXLAN bridging operation, hence the term asymmetric IRB.
1/19
For the asymmetric model to operate the sending VTEP needs the information for all the
tenant’s hosts (MAC and MAC to IP binding), to route and bridge the packet. This means
the VTEP needs to be member of all the tenant’s subnets/VNI and have an associated SVI
with an anycast IP for all the subnets, this will be required on all VTEPs participating in the
routing functionality for the tenant. As the data center grows, the amount of state (MAC and
ARP entries) required to be held on each VTEP in the asymmetric model does introduce
potential scaling issues.
Symmetric IRB
To address the scale issues of the asymmetric model, in the symmetrical model the VTEP
is only configured with the subnets that are present on the directly attached hosts,
connectivity to non-local subnets on a remote VTEP is achieved through an intermediate
IP-VRF. In this model, the ingress VTEP routes the traffic between the local subnet and the
IP-VRF, which both VTEPs are a member of, the egress VTEP then routes the frame from
the IP-VRF to the destination subnet. The forwarding model results in both VTEPs
performing a routing function, hence the term symmetrical IRB.
The EOS 4.20.1F release introduces support for both IRB models; symmetric and
asymmetric on the platform listed in the table below:
The introduction of the IRB functionality in the 4.20.1F release introduces new EVPN route
types and route formats:
Type-2 Route: The type-2 EVPN route is used to advertise host MAC and optionally
IP addresses. In the 4.20.1F release, to advertise MAC and IP bindings and provide
ARP suppression in the IRB models, Type-2 routes are now advertised with both
MAC and IP addresses. On receiving such a route, the neighbor will program
following two things:
A remote ARP entry; behind the corresponding VLAN interface.
A remote host route; in the corresponding IP-VRF.
2/19
Type-5 Route: The Type-5 EVPN route is used to advertise an IP prefix, rather than a
host MAC and IP binding. To advertise subnet connectivity in the symmetrical IRB
model, Type-5 routes are used, with the route learnt via the IP-VRF. The Type-5
route can also be used in both the symmetrical and asymmetric models to advertise
external IP prefixes into the EVPN domain. In the 4.20.1F release, Arista supports the
interface-less format of the type-5 route as defined in the internet draft
https://tools.ietf.org/html/draft-ietf-bess-evpn-prefix-advertisement-04,
Red lines represent L3 connections; blue lines represent L2 connections (trunk ports).
VTEP-1-1 and VTEP-1-2 are Mlag-Vtep.
Network configuration
To enable EVPN IRB with VXLAN, the following seven components need to be configured:
VTEP-1-1:
VTEP-1-2:
VTEP-2:
4/19
service routing protocols model multi-agent
!
interface Ethernet1
description core-facing interface connected to spine
no switchport
ip addr 50.0.20.0/31
!
ip routing
!
router bgp 65002
neighbor UNDERLAY peer-group
neighbor UNDERLAY remote-as 65999
neighbor 50.0.20.1 peer-group UNDERLAY
redistribute connected
!
Spine:
MLAG configuration
VTEP-1-1:
5/19
vlan 4094
trunk group mlag
!
no spanning-tree vlan 4094
!
interface Vlan4094
mtu 9214
no autostate
ip address 2.0.0.0/31
!
interface Ethernet10
description Connect to Vtep-1-2 et10
channel-group 2000 mode on
!
interface Port-Channel2000
Description Mlag peer link
switchport mode trunk
switchport trunk group mlag
!
mlag configuration
domain-id DC1-TOR1
local-interface Vlan4094
peer-address 2.0.0.1
peer-link Port-Channel2000
!
VTEP-1-2:
vlan 4094
trunk group mlag
!
no spanning-tree vlan 4094
!
interface Vlan4094
mtu 9214
no autostate
ip address 2.0.0.1/31
!
interface Ethernet10
description Connect to Vtep-1-1 et10
channel-group 2000 mode on
!
interface Port-Channel2000
Description Mlag peer link
switchport mode trunk
switchport trunk group mlag
!
mlag configuration
domain-id DC1-TOR1
local-interface Vlan4094
peer-address 2.0.0.0
peer-link Port-Channel2000
!
6/19
VTEP-1-1
vlan 4093
trunk group mlag
!
no spanning-tree vlan 4093
!
interface Vlan4093
mtu 9214
no autostate
ip address 4.0.0.0/31
!
router bgp 65001
neighbor MLAG-UNDERLAY peer-group
neighbor MLAG-UNDERLAY remote-as 65001
neighbor MLAG-UNDERLAY next-hop-self
neighbor 4.0.0.1 peer-group MLAG-UNDERLAY
!
VTEP-1-2
vlan 4093
trunk group mlag
!
no spanning-tree vlan 4093
!
interface Vlan4093
mtu 9214
no autostate
ip address 4.0.0.1/31
!
router bgp 65001
neighbor MLAG-UNDERLAY peer-group
neighbor MLAG-UNDERLAY remote-as 65001
neighbor MLAG-UNDERLAY next-hop-self
neighbor 4.0.0.0 peer-group MLAG-UNDERLAY
!
vlan 10, 20
!
vrf definition red
!
interface loopback0
description Vxlan tunnel source interface
ip address 1.1.1.1/32
!
interface vxlan 1
description Vxlan tunnel interface
vxlan source-interface loopback0
vxlan vlan 10 vni 1010
vxlan vlan 20 vni 1020
vxlan vrf red vni 8000
!
7/19
VTEP-2:
vlan 10, 30
!
vrf definition red
!
interface loopback0
description Vxlan tunnel source interface
ip address 1.1.1.2/32
!
interface vxlan 1
description Vxlan tunnel interface
vxlan source-interface loopback0
vxlan vlan 10 vni 1010
vxlan vlan 30 vni 1030
vxlan vrf red vni 8000
!
VTEP-1-1:
While using VXLAN EVPN IRB, SVI must configure “ip address virtual” — regular
IP/IPv6 are not supported.
To support ICMP sourced from “ip address virtual”, Linux kernel will automatically
create a src-NAT rule to replace the source IP address.
For each IP-VRF which has “ip address virtual” configured, users need to configure a
highest unique regular IP address for src-NAT.
In the example above, the highest unique IP address is 223.255.1.0/31.
Other VTEPs need to have reachability to this highest unique IP address:
Reachability from MLAG peer (VTEP-1-2):
VLAN 3000 is allowed across mlag-peer-link so VTEP-1-2 can reach
8/19
223.255.1.0 through Connected route.
Reachability from remote VTEPs (VTEP-2):
VTEP-1-1 will advertise 223.255.1.0/31 to its EVPN neighbors through an
EVPN Type 5 update since we enable “redistribute connected” under the
IP-VRF; note the configuration example entitled “BGP/EVPN MAC-VRF
and IP-VRF” below.
VTEP-1-2:
VTEP-2:
9/19
eBGP EVPN configuration for the network overlay
VTEP-1-1:
interface loopback1
Description BGP_EVPN router interface
ip address 3.3.3.1/32
!
router bgp 65001
neighbor EVPN peer-group
neighbor EVPN update-source Loopback1
neighbor EVPN fall-over bfd
neighbor EVPN send-community
neighbor EVPN ebgp-multihop
neighbor 3.3.3.99 peer-group EVPN
neighbor 3.3.3.99 remote-as 65999
!
address-family evpn
neighbor 3.3.3.99 activate
!
address-family ipv4
no neighbor 3.3.3.99 activate
!
VTEP-1-2:
interface loopback1
Description BGP_EVPN router interface
ip address 3.3.3.2/32
!
router bgp 65001
neighbor EVPN peer-group
neighbor EVPN update-source Loopback1
neighbor EVPN fall-over bfd
neighbor EVPN send-community
neighbor EVPN ebgp-multihop
neighbor 3.3.3.99 peer-group EVPN
neighbor 3.3.3.99 remote-as 65999
!
address-family evpn
neighbor 3.3.3.99 activate
!
address-family ipv4
no neighbor 3.3.3.99 activate
!
VTEP-2:
10/19
interface loopback1
Description BGP_EVPN router interface
ip address 3.3.3.3/32
!
router bgp 65002
neighbor EVPN peer-group
neighbor EVPN update-source Loopback1
neighbor EVPN fall-over bfd
neighbor EVPN send-community
neighbor EVPN ebgp-multihop
neighbor 3.3.3.99 peer-group EVPN
neighbor 3.3.3.99 remote-as 65999
!
address-family evpn
neighbor 3.3.3.99 activate
!
address-family ipv4
no neighbor 3.3.3.99 activate
!
interface loopback1
Description BGP_EVPN router interface
ip address 3.3.3.99/32
!
router bgp 65999
neighbor EVPN peer-group
neighbor EVPN update-source Loopback1
neighbor EVPN fall-over bfd
neighbor EVPN send-community
neighbor EVPN ebgp-multihop
neighbor 3.3.3.1 peer-group EVPN
neighbor 3.3.3.1 remote-as 65001
neighbor 3.3.3.2 peer-group EVPN
neighbor 3.3.3.2 remote-as 65001
neighbor 3.3.3.3 peer-group EVPN
neighbor 3.3.3.3 remote-as 65002
!
address-family evpn
bgp next-hop-unchanged
neighbor EVPN activate
!
address-family ipv4
no neighbor EVPN activate
!
VTEP-1-1:
11/19
router bgp 65001
vlan 10
rd 3.3.3.1:1010
route-target both 0:1010
redistribute learned
!
vlan 20
rd 3.3.3.1:1020
route-target both 0:1020
redistribute learned
!
vrf red
rd 3.3.3.1:8000
route-target both 0:8000
redistribute connected
!
VTEP-1-2:
VTEP-2:
Troubleshooting:
12/19
show interfaces vxlan 1:
VLAN 3001 is an internal VLAN (“show vlan internal allocation policy”), which is
automatically reserved for the IP-VRF to VNI mapping.
Both MLAG peers (VTEP-1-1 and VTEP-1-2) will choose the same internal VLAN.
VTEP-2:
13/19
VTEP-2:
Red lines represent L3 connections; blue lines represent L2 connections (trunk ports).
VTEP-1-1 and VTEP-1-2 are Mlag-Vtep.
Step 2: VTEP-1-1 advertises EVPN Type 2 MAC+IP update to its EVPN neighbors
14/19
Vtep-1-1(config)# show bgp evpn route-type mac-ip 0000.0010.0005 detail
BGP routing table entry for mac-ip 0000.0010.0005 100.0.10.5, Route Distinguisher:
3.3.3.1:1010
Paths: 1 available
Local
- from - (0.0.0.0)
Origin IGP, metric -, localpref -, weight 0, valid, local, best
Extended Community: Route-Target-AS:0:1010 Route-Target-AS:0:8000
TunnelEncap:tunnelTypeVxlan EvpnRouterMac:44:55:66:77:88:01
VNI: 1010 L3 VNI: 8000 ESI: 0000:0000:0000:0000:0000
BGP routing table entry for mac-ip 0000.0010.0005 100.0.10.5, Route Distinguisher:
3.3.3.1:1010
Paths: 1 available
65999 65001
1.1.1.1 from 3.3.3.99 (3.3.3.99)
Origin IGP, metric -, localpref 100, weight 0, valid, external, ECMP head,
best, ECMP contributor
Extended Community: Route-Target-AS:0:1010 Route-Target-AS:0:8000
TunnelEncap:tunnelTypeVxlan EvpnRouterMac:44:55:66:77:88:01
VNI: 1010 L3 VNI: 8000 ESI: 0000:0000:0000:0000:0000
Step 4: VTEP-2 imports this EVPN Type 2 MAC+IP update into the MAC-VRF, and
programs a remote ARP binding
15/19
Red lines represent L3 connections; blue lines represent L2 connections (trunk ports).
VTEP-1-1 and VTEP-1-2 are Mlag-Vtep.
Step 2: VTEP-1-1 advertises EVPN Type 2 MAC+IP update to its EVPN neighbors
BGP routing table entry for mac-ip 0000.0020.0005 100.0.20.5, Route Distinguisher:
3.3.3.1:1020
Paths: 1 available
Local
- from - (0.0.0.0)
Origin IGP, metric -, localpref -, weight 0, valid, local, best
Extended Community: Route-Target-AS:0:1020 Route-Target-AS:0:8000
TunnelEncap:tunnelTypeVxlan EvpnRouterMac:44:55:66:77:88:01
VNI: 1020 L3 VNI: 8000 ESI: 0000:0000:0000:0000:0000
16/19
This EVPN Type 2 MAC+IP route has two VNIs:
1020 is the VNI for VLAN 20
8000 is the VNI for IP-VRF red
44:55:66:77:88:01 is the System-MAC of VTEP-1-1.
BGP routing table entry for mac-ip 0000.0020.0005 100.0.20.5, Route Distinguisher:
3.3.3.1:1020
Paths: 1 available
65999 65001
1.1.1.1 from 3.3.3.99 (3.3.3.99)
Origin IGP, metric -, localpref 100, weight 0, valid, external, ECMP head,
best, ECMP contributor
Extended Community: Route-Target-AS:0:1020 Route-Target-AS:0:8000
TunnelEncap:tunnelTypeVxlan EvpnRouterMac:44:55:66:77:88:01
VNI: 1020 L3 VNI: 8000 ESI: 0000:0000:0000:0000:0000
Step 4: VTEP-2 imports this EVPN Type 2 MAC+IP update into its IP-VRF, and injects
a remote host route
Since Host-4 and Host-2 are in different IP subnets, Host-4 will send packets to its
default gateway, which is “interface vlan 30” on VTEP-2.
VTEP-2 will use this remote host route to route packets to Host-2:.
Packets will be encapsulated using VNI 8000,
The outer-dst-IP will be 1.1.1.1, which is VTEP-1-1’s VTI,
The inner-dst-MAC will be 44:55:66:77:88:01, which is VTEP-1-1’s System
MAC.
VTEP-1-1:
VTEP-1-2:
17/19
router bgp 65001
vlan 10
rd 3.3.3.2:1010
route-target both 0:1010
redistribute learned
no redistribute host-route
!
Step 3: VTEP-1-1 advertises EVPN Type 2 MAC+IP update to its EVPN neighbors —
without L3 VNI and IP-VRF route target
BGP routing table entry for mac-ip 0000.0010.0005 100.0.10.5, Route Distinguisher:
3.3.3.1:1010
Paths: 1 available
Local
- from - (0.0.0.0)
Origin IGP, metric -, localpref -, weight 0, valid, local, best
Extended Community: Route-Target-AS:0:1010 TunnelEncap:tunnelTypeVxlan
EvpnRouterMac:44:55:66:77:88:01
VNI: 1010 ESI: 0000:0000:0000:0000:0000
This EVPN Type 2 MAC+IP route doesn’t have route target for IP-VRF red (0:8000)
This EVPN Type 2 MAC+IP route doesn’t have VNI for IP-VRF red (8000)
BGP routing table entry for mac-ip 0000.0010.0005 100.0.10.5, Route Distinguisher:
3.3.3.1:1010
Paths: 1 available
65999 65001
1.1.1.1 from 3.3.3.99 (3.3.3.99)
Origin IGP, metric -, localpref 100, weight 0, valid, external, ECMP head,
best, ECMP contributor
Extended Community: Route-Target-AS:0:1010 TunnelEncap:tunnelTypeVxlan
EvpnRouterMac:44:55:66:77:88:01
VNI: 1010 ESI: 0000:0000:0000:0000:0000
Platform Compatibility
DCS-7050X2
DCS-7060X2
DCS-7050X
DCS-7060X
DCS-7250X
18/19
DCS-7300X
DCS-7320X
DCS-7280R
Vxlan EVPN IRB requires Vxlan Routing; some platforms need additional
configuration to enable Vxlan Routing.
Additional configuration for Vxlan Routing on 7050X, 7060X and 7260X
Additional configuration for Vxlan Routing on 7280R
Limitation
SVI with regular IP address is not supported.
SVI with IPv6 is not supported.
“Host Route Injection” for hosts in default IP-VRF is not supported.
L2 VTEP is not supported — if a host directly connects to a VTEP, then that VTEP
must be the gateway of the host.
While using the VLAN-aware Bundle service, all SVIs in the same bundle need to be
in the same IP-VRF.
Resources
[1] RFC7432 – BGP MPLS-Based Ethernet VPN
[2] draft-ietf-bess-evpn-overlay – A Network Virtualization Overlay Solution using
EVPN
[3] draft-ietf-bess-evpn-prefix-advertisement – IP Prefix Advertisement in EVPN
[4] draft-ietf-bess-evpn-prefix-advertisement – Integrated Routing and Bridging in
EVPN
19/19