BRKDCN-3040 Troubleshooting Vxlan Evpn 2019

BRKDCN-3040
Troubleshooting
VXLAN BGP EVPN
Shridhar V. Dhodapkar CCIE#6367
Technical Leader-Customer
Experience
Twitter - shridhardh
Agenda
• Introduction
• VXLAN Overview
• VXLAN Packet Flow
• Configuration
• Nexus 9000 Components
• Control Plane Troubleshooting
• Troubleshooting BUM Traffic
• Troubleshooting Tenant Routed Multicast
• Troubleshooting Tools
• Conclusion
BRKDCN-3040 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
VxLAN Overview
VXLAN Concepts
• VXLAN Overlay
- A VXLAN Overlay or VXLAN segment is a Layer-2 broadcast domain identified by
the VNID that extends or tunnels traffic from one VTEP to another.
• VXLAN Tunnel End Point (VTEP)
- A VTEP is a device that provides both encapsulation and de-capsulation of
classical Ethernet and VXLAN packets to and from a VXLAN segment
- Each VTEP may have the following types of interfaces:
 Switchport interfaces on the local LAN segment to support local endpoints
 Layer-3 interfaces to the transport IP network
 SVI interfaces
• VXLAN Gateway
- A VTEP that bridges traffic between VXLAN segments
VXLAN Encapsulation
Ethernet
Payload FCS
Header
Outer Outer Inner

Outer IP VXLAN Payload New FCS
Ethernet UDP Ethernet
Flags Reserved Instance ID Reserved

8 Bytes 24 Bytes 24 Bytes 8 Bytes
Rsvd Rsvd
Outer UDP Destination Port = VXLAN (originally 8472, recently updated to 4789)
1
Outer UDP Source Port = Hash of Inner Frame Headers (optional)
• The outer IP header has the source IP and destination IP of the VTEP
endpoints
• The outer Ethernet header has the source MAC of the source VTEP
and the destination MAC of the immediate Layer-3 next hop
VxLAN Packet Structure
F
Outer MAC Outer UDP VxLAN
Outer IP Header Original Layer 2 Frame C
Header Header Header
S
14 Bytes
(4 bytes optional) 8 Bytes Hash of inner L2/l3/L4 headers
Destination NH MAC Source Port
of original frame. Enables
16 entropy for ECMP Load
48 Address Addr
16 VxLAN Port UDP 4789 balancing
48 Source Address
16 UDP Length
16 VLAN Type
0x8100 16 Checksum
16 VLAN ID Tag
Src and Dst
16 Ether Type 20 Bytes Addresses of 8 Bytes
0x0800 16M possible
VTEP
72 IP Header Misc. VxLAN
Data 8 RRRR1RRR segments
8 Protocol 0x11 Reserved
24
Src VTEP MAC Addr
16 Header
24 VNI
32 Checksum
Source IP 8 Reserved
32
Destination IP
VxLAN Overview
VxLAN Gateway Types
• Layer 2 Gateway
− The layer 2 gateway is required when the layer 2 traffic (IEEE 802.1q
tagged traffic) comes from VLAN into VxLAN segment (encapsulation) or
− The Ingress VxLAN packet egresses out an 802.1q tagged interface (de-
encapsulation), where the packet is bridged to a new VLAN.
• Layer 3 Gateway
− A layer 3 gateway is used when there is a VxLAN to VxLAN routing
− The ingress packet is a VxLAN packet on a routed segment but the
packet egresses out on a tagged 802.1q interface and the packet is
routed to a new VLAN
End System End System
VxLAN Overview 3
ARP Request for IP B
VxLAN - Flood and Learn Src MAC: MAC-A
Dst MAC: FF:FF:FF:FF:FF:FF
MAC-3
IP- 192.168.3.3
VTEP 3
S-MAC: MAC-1
D-MAC: MAC VxLAN Remote
VTEP-3
00:01:5E:01:01:01 Address ID VTEP
Outer S-IP: 192.168.1.1 MAC-A 10000 192.168.1.1

4
Outer D-IP: 239.1.1.1
UDP ARP Response from IP B
Src MAC: MAC-B
VXLAN VNID: 10000
Dst MAC: MAC-A
7
Src MAC: MAC-A 2
Dst MAC: VTEP 2
ARP Response from IP B Mcast Group IP- 192.168.2.2
FF:FF:FF:FF:FF:FF End System B
Src MAC: MAC-B 239.1.1.1
MAC-2 MAC-B
Dst MAC: MAC-A IP - B
2
VTEP-1 VTEP-2 3
End System A S-MAC: MAC-2
MAC-A D-MAC: MAC-NH ARP Request for IP B
IP - A
VTEP 1 5 Src MAC: MAC-A
1 IP-192.168.1.1 Outer S-IP: 192.168.2.2 Dst MAC: FF:FF:FF:FF:FF:FF
6
MAC-1 Outer D-IP: 192.168.1.1
MAC VxLAN Remote UDP MAC VxLAN Remote
Address ID VTEP Address ID VTEP
Src MAC: MAC-A
Dst MAC: FF:FF:FF:FF:FF:FF VXLAN VNID: 10000
MAC-B 10000 192.168.2.2 MAC-A 10000 192.168.1.1
ARP Response
VxLAN Overview
VxLAN – Flood and Learn
• Data Plane learning technique for VxLAN
• VNI’s are mapped to a multicast group on a VTEP
• Local MACs are learnt over a VLAN (VNI) on a VTEP
• Broadcast, Unknown Unicast, Multicast (BUM Traffic) is flooded to
the delivery multicast group for that VNI
• Remote VTEPs part of same multicast group learn host MAC, VNI
and source VTEP as the next-hop for the host MAC from flooded
traffic
• Unicast packets to the host MAC are sent directly to source VTEP
as VxLAN encapsulated packet
VxLAN Overview
Ingress Replication
• Some customers not comfortable deploying multicast in their core
• With Ingress Replication (IR), BUM traffic ingress access side is
replicated to remote VTEP as unicast
• Static IR VETP tunnel is kept alive as long as the route to the VTEP
is available.
• Support multiple VTEPs per VNI and a VTEP in multiple VNIs
VxLAN Overview
Tenant
Tenant 1 (VRF 1)
SVI SVI SVI SVI

X A B N
Layer-3 VNI X’ Layer-2 VNI A’ Layer-2 VNI B’ Layer-2 VNI N’
VLAN X VLAN A VLAN B VLAN N
• 1 Layer-3 VNI per • 1 Layer-2 VNI per Layer-2 segment

Tenant (VRF) for
• Multiple Layer-2 VNIs per tenant
routing
• VNI A’ and B’ are used for bridged
• VNI X’ is used for
packets
routed packets
VxLAN Overview
VxLAN EVPN
BGP-EVPN BGP-EVPN
Mac_H1: VNI1 Mac_H3: VNI1

IP_H1: VNI3 IP Multicast Core IP_H3: VNI3
NHOP: VTEP-1 Or NHOP: VTEP-1
Mac_H2: VNI2 Unicast Core Mac_H4: VNI2
IP_H2: VNI3 VTEP-1 VTEP-2 IP_H4: VNI3
NHOP: VTEP-1 NHOP: VTEP-2
V V
VNI3 VRF VRF VNI3
Vlan-500 A A Vlan-500
L3 VNI L3 VNI
MAC_H1 MAC_H2 MAC_H3 MAC_H4

IP_H1 IP_H2 IP_H3 IP_H4
VNI1 VNI2 VNI1 VNI2

Vlan-100 Vlan-200 Vlan-100 Vlan-200
VxLAN Overview
Distributed Anycast Gateway
Spine1 Spine2
L3 Underlay
Unicast / Multicast Routing
VTEP-1 VTEP-2 VTEP-3 VTEP-3
SVI SVI SVI SVI

GW IP GW IP GW IP GW IP
GW MAC GW MAC GW MAC GW MAC
Host-A Host-B
Host-C
VxLAN Overview
Distributed Anycast Gateway - Configuration
• All VTEPs has same IP address for an L2 VNI
• Anycast Gateway MAC is global to each VTEP for all VNI’s for all Tenants
• One virtual MAC / VTEP
• All VTEPs should have same virtual MAC address
fabric forwarding anycast-gateway-mac 0001.0001.0001

!
interface Vlan100
no shutdown
vrf context test-evpn-tenant
ip address 172.16.1.254/24
fabric forwarding mode anycast-gateway
VxLAN Overview
ARP Suppression
• Hosts send out G-ARP when they
come online
IP Multicast Core
• Local leaf node receives G-ARP, 2 +
creates local ARP cache and VTEP-1 VxLAN EVPN VTEP-2
advertises to other leaf by BGP as 3 V V
route type 2 VRF VRF
A A
• Remote leaf node puts IP-MAC info
into remote ARP cache and supresses 4 MAC_H1
1 MAC_H2
incoming ARP request for this IP IP_H1 IP_H2
VNI1 VNI1
• If IP info not found in ARP suppression Vlan-100 Vlan-100
cache table, VTEP floods the ARP

request to other VTEPs
Configuration
L2 VNI – 10000, 20000
VxLAN Configuration L3 VNI - 50000
Topology Spine Spine
10.10.10.10 20.20.20.20
L3 Underlay
Unicast / Multicast Routing
VTEP-3
192.168.100.100
Leaf1 (VTEP-1) Leaf2 (VTEP-2) Leaf3 (VTEP-3) Leaf4 (VTEP-3)

192.168.1.1 192.168.2.2 192.168.3.3 192.168.4.4
VPC
Host-C
Host-A Host-B
Configuration
Feature Enablement
feature bgp For BGP Configuraiton

Feature pim
feature interface-vlan For PIM / Multicast Core
feature vn-segment-vlan-based
feature nv overlay SVI Configuration
!
For VxLAN Overlay
nv overlay evpn
! Map VLAN to VNI
Enables EVPN AFI

Not required to be
enabled on Spine
Underlay Configuration
Leaf Spine
router bgp 65000 router bgp 65001
router-id 192.168.1.1 router-id 192.168.10.10
address-family ipv4 unicast address-family ipv4 unicast
network 1.1.1.1/32 network 10.10.10.10/32
network 192.168.1.1/32 network 192.168.10.10/32
neighbor 10.1.101.10 neighbor 10.1.201.1
remote-as 65001 remote-as 65000
allowas-in 3 allowas-in 3
disable-peer-as-check disable-peer-as-check
neighbor 10.1.201.20 neighbor 10.1.202.2
remote-as 65001 remote-as 65000
allowas-in 3 allowas-in 3
disable-peer-as-check disable-peer-as-check
Spine Node Configuration
router bgp 65001
address-family l2vpn evpn
nexthop route-map permitall
interface loopback1 retain route-target all
ip address 192.168.10.10/32 neighbor 1.1.1.1
ip pim sparse-mode remote-as 65000
update-source loopback0
!
ebgp-multihop 3
interface loopback2 address-family l2vpn evpn
ip address 192.168.50.50/32 disable-peer-as-check
ip pim sparse-mode send-community extended
! route-map permitall out
ip pim rp-add 192.168.50.50 group-list 239.1.1.0/24 neighbor 2.2.2.2
remote-as 65000
ip pim anycast-rp 192.168.50.50 192.168.10.10
update-source loopback0
ip pim anycast-rp 192.168.50.50 192.168.20.20 ebgp-multihop 3
address-family l2vpn evpn
disable-peer-as-check
send-community extended
route-map permitall out
!
route-map permitall permit 10
set ip next-hop unchanged
Leaf Node Configuration – L2 VNI
vlan 100 fabric forwarding anycast-gateway-mac 0001.0001.0001
vn-segment 10000 interface Vlan100
no shutdown
! Create L2 VNI vrf member EVPN-TENANT
ip address 100.1.1.254/24
evpn fabric forwarding mode anycast-gateway
vni 10000 l2 !
rd 10000:1 router bgp 65000
route-target import 10000:1 neighbor 10.10.10.10
route-target export 10000:1 remote-as 65001
! update-source loopback0
interface nve1 ebgp-multihop 3
no shutdown address-family l2vpn evpn
source-interface loopback1 allowas-in 3
host-reachability protocol bgp disable-peer-as-check
member vni 10000 send-community extended
mcast-group 239.1.1.1 vrf EVPN-TENANT
suppress-arp address-family ipv4 unicast
advertise l2vpn evpn
Leaf Node Configuration – L3 VNI
vlan 500 interface nve1
vn-segment 50000 no shutdown
! source-interface loopback0
vrf context EVPN-TENANT host-reachability protocol bgp
vni 50000 member vni 50000 associate-vrf
rd 50000:1 !
address-family ipv4 unicast interface loopback200
route-target import 50000:1 vrf member EVPN-TENANT
route-target import 50000:1 evpn ip address 200.1.1.1/32
route-target export 50000:1 !
route-target export 50000:1 evpn router bgp 65000
! vrf EVPN-TENANT
interface Vlan500 address-family ipv4 unicast
no shutdown network 200.1.1.1/32
vrf member EVPN-TENANT advertise l2vpn evpn
ip forward
!
Leaf Node with VPC Configuration
vpc domain 10
peer-switch
peer-keepalive destination 10.1.34.4 source
10.1.34.3 VTEP IP. The secondary
delay restore 60 IP is same on both Leaf3
peer-gateway and Leaf 4 running VPC
ipv6 nd synchronize
ip arp synchronize
!
interface loopback0
ip address 192.168.3.3/32 Backup Routing SVI
ip address 192.168.100.100/32 secondary
ip router ospf 100 area 0.0.0.0 vlan 5
ip pim sparse-mode interface vlan 5
ip add 10.5.1.1/24
ip router ospf 100 area 0.0.0.0
Backup Routing SVI ip pim sparse-mode
Configured on both vPC
peers and part of global
routing table.
Leaf Node Configuration – L2 & L3 VNI for IPv6
vlan 100 fabric forwarding anycast-gateway-mac 0001.0001.0001
vn-segment 10000 interface Vlan100
vlan 500 no shutdown
vn-segment 50000 vrf member EVPN-TENANT
evpn ip address 100.1.1.254/24
vni 10000 l2 ipv6 address 2001::1/64
rd 10000:1 fabric forwarding mode anycast-gateway
route-target import 10000:1 !
route-target export 10000:1 interface nve1
! no shutdown
vrf context EVPN-TENANT source-interface loopback0
vni 20000 host-reachability protocol bgp
rd 20000:1 member vni 10000
address-family ipv4 unicast mcast-group 239.1.1.1
route-target both 50000:1 suppress-arp
route-target both 50000:1 evpn member vni 50000 associate-vrf
address-family ipv6 unicast !
route-target both 50000:1 router bgp 65000
route-target both 50000:1 evpn vrf EVPN-TENANT
! address-family ipv4 unicast
interface Vlan500 advertise l2vpn evpn
no shutdown address-family ipv6 unicast
vrf member EVPN-TENANT advertise l2vpn evpn
ip forward !
ipv6 address use-link-local-only vpc domain 10
ipv6 nd synchronize
VxLAN EVPN Configuration
Host Learning and Peer Discovery
Host Learning Data Plane Control Plane
CORE
Multicast Flood and Learn EVPN-Multicast
Peer Learning: Data Plane Peer Learning: BGP-RnH
Vlan 100 Vlan 100

vn-segment 10000 vn-segment 10000
Interface nve1 Interface nve1
Member vni 10000 host-reachability protocol bgp
Mcast-group 239.1.1.1 member vni 10000
Mcast-group 239.1.1.1
Unicast Static Ingress-Replication EVPN Ingress-Replication

Peer Learning - CLI Peer Learning – BGP-IMET
Vlan 150 Vlan 150
vn-segment 15000 vn-segment 15000
Interface nve1 Interface nve1
member vni 15000 host-reachability protocol bgp
Ingress-replication protocol member vni 15000
static peer x.x.x.x ingress-replication protocol bgp
VXLAN BGP EVPN
Control-Plane
Verification
Nexus 9000 VxLAN Architecture
BGP
URIB, VxLAN
U6RIB Manager L2RIB
(PI)
HMM
MRIB
VxLAN
Manager Adj. Mgr
(PD)
MFDM L2FM ARP IPv6 ND
Troubleshooting VxLAN EVPN
EVPN Prefix Types
 BGP EVPN uses 5 different route types for IP
prefixes and advertisement
 Type 1 - Ethernet Auto-Discovery (A-D) route
 Type 2 - MAC advertisement route  L2 VNI MAC/MAC-IP
 Type 3 - Inclusive Multicast Route  EVPN IR, Peer Discovery
 Type 4 - Ethernet Segment Route
 Type 5 - IP Prefix Route  L3 VNI Route
 Route type 2 or MAC Advertisement route is for

MAC and ARP resolution advertisement, MAC
or MAC-IP
 Route type 5 or IP Prefix route will be used for
the advertisement of prefixes, IP only
VxLAN EVPN Prefix Types
Leaf1# show bgp l2vpn evpn
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 10000:1 (L2VNI 10000)
*>l[2]:[0]:[0]:[48]:[523d.e706.ae1b]:[0]:[0.0.0.0]/216
192.168.1.1 100 32768 i
*>l[2]:[0]:[0]:[48]:[523d.e706.ae1b]:[32]:[100.1.1.1]/272
192.168.1.1 100 32768 i
*>l[3]:[0]:[32]:[192.168.1.1]/88
192.168.1.1 100 32768 i
*>e[3]:[0]:[32]:[192.168.2.2]/88
192.168.2.2 0 65001 65000 i
*>e[3]:[0]:[32]:[192.168.3.3]/88
192.168.3.3 0 65001 65000 I
Route Distinguisher: 192.168.2.2:32967
* e[3]:[0]:[32]:[192.168.2.2]/88
192.168.2.2 0 65001 65000 i
*>e 192.168.2.2 0 65001 65000 i
Route Distinguisher: 192.168.1.1:3 (L3VNI 50000)

*>e[5]:[0]:[0]:[32]:[100.100.100.2]:[0.0.0.0]/224
192.168.2.2 0 65001 65000 i
NVE Interface
Leaf1# show nve interface
Interface: nve1, State: Up, encapsulation: VXLAN
VPC Capability: VPC-VIP-Only [not-notified]
Local Router MAC: f40f.1b6f.926f
Host Learning Mode: Control-Plane
Source-Interface: loopback0 (primary: 192.168.1.1, secondary: 0.0.0.0)
Leaf1# show interface nve1

nve1 is up If NVE Interface status is down, ensure
admin state is up, Hardware: NVE that a no shut is performed on the
MTU 9216 bytes
Encapsulation VXLAN
interface.
Auto-mdix is turned off
RX
ucast: 40 pkts, 5400 bytes - mcast: 1 pkts, 118 bytes
TX
ucast: 54 pkts, 6256 bytes - mcast: 9 pkts, 1026 bytes
Local MAC Routes Learning
L2FM
Mac Learnt on Vlan 100
Leaf1#show mac address-table vlan 100

(T) - True, (F) - False
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+-------------
* 100 523d.e706.ae1b dynamic 0 F F Eth1/15
Leaf1# sh sys inter l2fm event-hist deb | in 523d.e706.ae1b

[104] l2fm_l2rib_add_delete_local_mac_routes(1095): To L2RIB: topo-id: 100,
macaddr: 523d.e706.ae1b, nhifindx: 0x1a001600 peer_addr 0x1a001600
[104] l2fm_macdb_insert(6360): slot 0 fe 0 mac 523d.e706.ae1b vlan 100 flags
0x400107 hints 0 E8 NL lc : if_index 0x1a001600 old_if_index 0
L2FM installs the MAC in the L2RIB
L2FM L2RIB Leaf1#show l2route evpn mac evi 100
Mac Address Prod Next Hop (s)
-------------- ------ ---------------
523d.e706.ae1b Local Eth1/15
Leaf1# show system internal l2rib event-history mac | in 523d.e706.ae1b

[06/01/16 22:31:55.201 UTC 5 9954] Received MAC ROUTE msg: addr: (100, 523d.e706.ae1b) vni: 0
admin_dist: 0 seq_num: 0 rt_flags: L soo: 0 dg_count: 0 res: 0 esi: (F) nh_count: 1
[06/01/16 22:31:55.202 UTC 7 9954] (100,8c60.4f93.5ffc):Mobility check for new rte from prod:
3
[06/01/16 22:31:55.202 UTC 8 9954] (100,523d.e706.ae1b):Current non-del-pending route
local:no, remote:no, linked mac-ip count:1
[06/01/16 22:31:55.202 UTC 9 9954] (523d.e706.ae1b,3):MAC route created with seq num:0,
flags:L (), soo:0, peerid:0
[06/01/16 22:31:55.205 UTC a 9954] (100,523d.e706.ae1b,3):Encoding MAC best route (ADD,
client id 5)
[06/01/16 22:31:55.207 UTC 3 9954] (100,523d.e706.ae1b):Bound MAC-IP(100.1.1.1) to MAC, Total
MAC-IP linked: 1
L2 VNI, MAC
L2FM L2RIB BGP L2VPN
Leaf1#show bgp l2vpn evpn vni-id 10000

Network Next Hop Metric LocPrf Weight Path
*>l[2]:[0]:[0]:[48]:[523d.e706.ae1b]:[0]:[0.0.0.0]/216
192.168.1.1 100 32768 i
Leaf1# show bgp internal event-history events | in 523d.e706.ae1b

2016 Jun 1 22:31:55.205989 bgp 100 [16855]: [16888]: (default) RIB: [L2VPN EVPN
] add prefix 10000:1:[2]:[0]:[0]:[48]:[523d.e706.ae1b]:[0]:[0.0.0.0] (flags 0x1)
: OK, total 2
2016 Jun 1 22:31:55.205655 bgp 100 [16855]: [16888]: EVT: Received from L2RIB MAC
route: Add topo 10000 mac 523d.e706.ae1b soo 0 seq 0
. . .
Local MAC Address in BGP L2VPN
L2FM L2RIB BGP L2VPN
Leaf1# show bgp l2vpn evpn 523d.e706.ae1b

BGP routing table information for VRF default, address family L2VPN EVPN
BGP routing table entry for
[2]:[0]:[0]:[48]:[523d.e706.ae1b]:[0]:[0.0.0.0]/216, version 318
Paths: (1 available, best #1)
Flags: (0x000102) (high32 00000000) on xmit-list, is not in l2rib/evpn
Advertised path-id 1
Path type: local, path is valid, is best path
AS-Path: NONE, path locally originated
192.168.1.1 (metric 0) from 0.0.0.0 (192.168.1.1)
Origin IGP, MED not set, localpref 100, weight 32768
Received label 10000
Extcommunity: RT:65000:10000 ENCAP:8
Path-id 1 advertised to peers:
10.10.10.10 20.20.20.20
Remote L2 MAC Route Installation via BGP EVPN
Leaf2# show bgp l2vpn evpn 523d.e706.ae1b
Route Distinguisher: 192.168.1.1:32867
BGP routing table entry for [2]:[0]:[0]:[48]:[523d.e706.ae1b]:[0]:[0.0.0.0]/216,
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Path type: external, path is valid, is best path
BGP L2VPN Imported to 1 destination(s)
AS-Path: 65001 65000 , path sourced external to AS
192.168.1.1 (metric 0) from 20.20.20.20 (192.168.20.20)
Path type: external, path is valid, not best reason: newer EBGP path
192.168.1.1 (metric 0) from 10.10.10.10 (192.168.10.10)
EVPN BGP Route Type 2 Fields
 Ethernet Tag ID, MAC Address Length, MAC Address, IP Address Length, and IP Address fields are
considered to be part of the prefix in the NLRI.
 Ethernet Segment Identifier, MPLS Label1, and MPLS Label2 are treated as route attributes, not part
of the "route". Both the IP and MAC address lengths are in bits.
Leaf1#show bgp l2vpn evpn 523d.e706.ae1b Route Distinguisher – 8 byte

BGP routing table information for VRF default, address family L2VPN
EVPN Ethernet Segment ID – 10 byte
BGP routing table entry for Ethernet Tag ID – 4 byte
[2]:[0]:[0]:[48]:[8c60.4f93.5ffc]:[0]:[0.0.0.0]/216, version 8
Paths: (1 available, best #1) MAC Address Length – 1 byte
Flags: (0x00010a) on xmit-list, is not in l2rib/evpn
MAC Address – 6 byte
Path type: local, path is valid, is best path, no labeled nexthop IP Address Length – 1 byte
192.168.1.1 (metric 0) from 0.0.0.0 (192.168.1.1) IP Address – 0, 4, 16 byte
Received label 10000 MPLS Label 1 – 3 byte, L2VNI
Extcommunity: RT:10000:1
MPLS Label 2 – 3 byte L3VNI
Remote L2 MAC Route Installation with BGP EPVN (Flow)
2a. Peer, VNI
Notification
BGP L2VPN
2b. Add (VNI, MAC-> Remote
VTEP IP)
VxLAN Mgr
VxLAN PD L2RIB
Lib.
5. Add (VNI,
3. Program data plane with
MAC -> Peer ID)
unicast encap/decap for VNI,
Allocate Peer ID
L2FM
UFDM
BGP to L2RIB
Leaf2# show bgp internal event-history events | in 523d.e706.ae1b

2016 Jun 2 02:53:14.844179 bgp 100 [9878]: [9890]: (default) IMP: bgp_tbl_ctx_import:
1812: [L2VPN EVPN] Importing
10000:1:[2]:[0]:[0]:[48]:[523d.e706.ae1b]:[0]:[0.0.0.0]/112 to RD 10000:1
2016 Jun 2 02:53:14.844167 bgp 100 [9878]: [9890]: (default) IMP: bgp_vrf_import:
2740: vrf default 10000:1:[2]:[0]:[0]:[48]:[523d.e706.ae1b]:[0]:[0.0.0.0]/112 result 1
2016 Jun 2 02:53:14.844130 bgp 100 [9878]: [9890]: (default) RIB: [L2VPN EVPN]: Send
to L2RIB 10000:1:[2]:[0]:[0]:[48]:[523d.e706.ae1b]:[0]:[0.0.0.0]/112 via 192.168.1.1
Add 1 EVPN MAC routes succeeded
Remote L2 MAC Route Installation via BGP EVPN
Leaf2# show nve internal bgp rnh database VxLAN Mgr
--------------------------------------------
Showing BGP RNH Database, size : 2 vni 0
VNI Peer-IP Peer-MAC Tunnel-ID Encap (A/S)

10000 192.168.1.1 0000.0000.0000 0x0 vxlan (1/0)
50000 192.168.1.1 f40f.1b6f.926f 0xc0a80101 vxlan (1/0)
BGP L2VPN
Leaf2# show l2route evpn mac evi 100

Mac Address Prod Next Hop (s) L2RIB
-------------- ------ ---------------
523d.e706.ae1b BGP 192.168.1.1
VxLAN Manager
Leaf2# show forwarding nve l3 peers
NVE cleanup transaction-id 0
tunnel_id Peer_id Peer_address Interface rmac origin state del count
--------------------------------------------------------------------------------------
0xc0a80101 1 192.168.1.1 nve1 f40f.1b6f.926f NVE merge-done no 1
Leaf2# show nve peers detail

Details of nve Peers:
----------------------------------------
Peer-Ip: 192.168.1.1
NVE Interface : nve1
VxLAN Mgr
Peer State : Up
Peer Uptime : 01:27:30 Hardware
Programs data plane with Router-Mac : f40f.1b6f.926f Programmed
unicast encap/decap for Peer First VNI : 20000
VNI, Allocate Peer ID
Time since Create : 01:27:30
Configured VNIs : 10000,20000
Provision State : add-complete
UFDM
Route-Update : Yes
Peer Flags : RmacL2Rib, TunnelPD, DisableLearn
Learnt CP VNIs : 10000,20000,50000
Peer-ifindex-resp : Yes
L2FM Verification
Leaf2# show system internal l2fm debugs | in 523d.e706.ae1b

[104] l2fm_macdb_insert(6327): slot 32 fe 0 mac 8c60.4f1b.e43c vlan 100 flags 0x7
hints 0 E8 NL lc : if_index 0x49080001 old_if_index 0
[104] l2fm_l2rib_mac_update(21832): Add L2RIB remote mac 523d.e706.ae1b
[104] l2fm_process_l2rib_remote_route_update(405): Type: 2 Len: 152 Seq: 0, del: 0
(Prod: 5) Flags: Ctrl=3 Rt=0, mac 8c60.4f1b.e43c topo_id 100
Leaf2# show mac address-table vlan 100

VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+-----------------
-
* 100 8c60.4f19.51fc dynamic 0 F F Eth1/13
* 100 523d.e706.ae1b dynamic 0 F F nve1(192.168.1.1)
Host IP and Host MAC local route
4. Route add: VRF test, Host-IP, L3VNI
HMM L2RIB
3. Adj. Notification: Host-IP, Host- 5. Route add: VRF test, Host-IP, L3 Table
MAC, SVI interface ID. Per prefix: L2 Table ID, Host-MAC
AM BGP
2. Create Adjacency: Host-IP, Host- L3 + L2 NLRI
MAC, Server facing SVI interface Prefix = Host-IP
MAC = Host-MAC
1b. Add an adjacency in AM, if the Label1: L3-VNI
ARP VLAN is enabled for L3 processing Label2: L2-VNI
VRF RT
1a. ARP Request/Reply BD RT
NH = VTEP IP
RMAC
ENCAP_VXLAN
Troubleshoot VxLAN EVPN
ARP from Host and ARP -> AM
Leaf1# show ip arp vrf EVPN-TENANT
IP ARP Table for context EVPN-TENANT
Total number of entries: 1
Address Age MAC Address Interface ARP
100.1.1.1 00:10:47 523d.e706.ae1b Vlan100
Leaf1# show forwarding vrf EVPN-TENANT adjacency

IPv4 adjacency information
next-hop rewrite info interface AM

-------------- --------------- -------------
100.1.1.1 523d.e706.ae1b Vlan100
Troubleshoot VxLAN EVPN AM
AM -> HMM -> L2RIB
Leaf1# show ip route vrf EVPN-TENANT
100.1.1.1/32, ubest/mbest: 1/0, attached
*via 100.1.1.1, Vlan100, [190/0], 02:41:57, hmm
100.1.1.254/32, ubest/mbest: 1/0, attached HMM
*via 100.1.1.254, Vlan100, [0/0], 02:59:46, local
Leaf1# show l2route evpn mac-ip evi 100

Mac Address Prod Host IP Next Hop (s)
-------------- ---- ------------------- --------------
-
L2RIB
523d.e706.ae1b HMM 100.1.1.1 N/A
L2RIB -> BGP
Leaf1# show bgp l2vpn evpn 100.1.1.1
version 6
Flags: (0x00010a) on xmit-list, is not in l2rib/evpn
Path type: local, path is valid, is best path, no labeled nexthop
192.168.1.1 (metric 0) from 0.0.0.0 (192.168.1.1)
Received label 10000 50000
Extcommunity: RT:10000:1 RT:50000:1

10.10.10.10
L2 + L3 Remote Route Installation
L3 + L2 NLRI BGP-EVPN
Prefix = Host-IP
MAC = Host-MAC EVPN Table
Label1: L3-VNI VTEP_IP,
Label2: L2-VNI RCAM, L3VNI,
VRF RT VRF Import Tunnel Info
L2VNI, VxLAN
BD RT BD Import
NH = VTEP IP RNH DB
RMAC VxLAN Manager
ENCAP_VXLAN Tunnel_ID
VRF BD
Peer-Id RMAC, VNI-
notification >Peer ID
VRF, Host-IP,
L3VNI, VTEP-IP
BD L2RIB
URIB
RMAC, VNI->Peer ID
VRF Program data plane
(L2-BD, MAC-
H1)-> Peer-ID
with unicast
VRF, Host-IP, encap/decap for
L3VNI, VTEP-IP L3VNI, RMAC,
Allocate Peer-ID
UFDM
FIB L2FM
L2RIB and URIB Information
Leaf2# show l2route evpn mac-ip evi 100

Mac Address Prod Host IP Next Hop (s)
-------------- ---- --------------------------- ---------------
523d.e706.ae1b BGP 100.1.1.1 192.168.1.1
Leaf2# show ip route vrf EVPN-TENANT 100.1.1.1

IP Route Table for VRF "EVPN-TENANT"
100.1.1.1/32, ubest/mbest: 1/0
*via 192.168.1.1%default, [200/0], 04:00:28, bgp-100, internal,
tag 100 (evpn) segid: 50000 tunnelid: 0xc0a80101 encap: VXLAN
Remote Host Prefix - EVPN
Leaf2# show bgp l2vpn evpn 100.1.1.1
version 5
Flags: (0x00021a) on xmit-list, is in l2rib/evpn, is not in HW, , is locked
Path type: internal, path is valid, imported same remote RD, is best path, no labeled
nexthop
AS-Path: NONE, path sourced internal to AS
192.168.1.1 (metric 5) from 10.10.10.10 (192.168.10.10)
Extcommunity: RT:10000:1 RT:50000:1 ENCAP:8 Router MAC:f40f.1b6f.926f
Originator: 192.168.1.1 Cluster list: 10.10.10.10
Path-id 1 not advertised to any peer

. . .
Remote Host Prefix – contd…
...
version 6
Flags: (0x00021a) on xmit-list, is in l2rib/evpn, is not in HW,
Path type: internal, path is valid, is best path, no labeled nexthop
Imported from 10000:1:[2]:[0]:[0]:[48]:[8c60.4f1b.e43c]:[32]:[100.1.1.1]/144
(VNI 10000)
AS-Path: NONE, path sourced internal to AS
192.168.1.1 (metric 5) from 192.168.10.10 (192.168.10.10)
Extcommunity: RT:10000:1 RT:50000:1 ENCAP:8 Router MAC:f40f.1b6f.926f
Originator: 192.168.1.1 Cluster list: 10.10.10.10
Path-id 1 not advertised to any peer

NVE Internal Platform Detail
Leaf1# show nve internal platform interface nve 1 detail
Printing Interface ifindex 0x49000001 detail
|======|=========================|===============|===============|=====|=====|
|Intf |State |PriIP |SecIP |Vnis |Peers|
|======|=========================|===============|===============|=====|=====|
|nve1 |UP |192.168.1.1 |0.0.0.0 |2 |1 |
|======|=========================|===============|===============|=====|=====|
SW_BD/VNIs of interface nve1:

================================================
|======|======|=========================|======|====|======|========
|Sw BD |Vni |State |Intf |Type|Vrf-ID|Notified
|======|======|=========================|======|====|======|========
|100 |10000 |UP |nve1 |CP |0 |Yes
|500 |50000 |UP |nve1 |CP |3 |Yes
|======|======|=========================|======|====|======|========
Peers of interface nve1:
============================================
Peer_ip: 192.168.2.2
Peer-ID : 1
State : UP
Learning : Disabled
TunnelID : 0xc0a80202
MAC : 88f0.312a.f2c1
Table-ID : 0x1
Encap : 0x1
NVE Internal Information for Leaf Nodes with VPC Peers
Leaf3# show nve internal platform interface nve 1 detail
Printing Interface ifindex 0x49000001 detail
|======|=========================|===============|===============|=====|=====|
|Intf |State |PriIP |SecIP |Vnis |Peers|
|======|=========================|===============|===============|=====|=====|
|nve1 |UP |192.168.3.3 |192.168.100.100|2 |2 |
|======|=========================|===============|===============|=====|=====|
SW_BD/VNIs of interface nve1:
================================================
|======|======|=========================|======|====|======|========
|Sw BD |Vni |State |Intf |Type|Vrf-ID|Notified
|======|======|=========================|======|====|======|========
|100 |10000 |UP |nve1 |CP |0 |Yes
|200 |20000 |UP |nve1 |CP |3 |Yes
|======|======|=========================|======|====|======|========
Peers of interface nve1:

============================================
Peer_ip: 192.168.1.1
Peer-ID : 2
State : UP
Learning : Disabled
TunnelID : 0xc0a80101
MAC : f40f.1b6f.926f
Table-ID : 0x1
Encap : 0x1
Leaf Node with VPC Consistency Check
sh vpc consistency-parameters vni
Legend:
Type 1 : vPC will be suspended in case of mismatch
Name Type Local Value Peer Value

------------- ---- ---------------------- ----------------------
-
Nve1 Adm St, Src Adm St, 1 Up, Up, Up, Up,
Sec IP, Host Reach, VMAC 192.168.100.100, CP, 192.168.100.100, CP,
Adv, SA,mcast l2, mcast FALSE, Disabled, FALSE, Disabled,
l3, IR BGP,MS Adm St, Reo 0.0.0.0, 0.0.0.0, 0.0.0.0, 0.0.0.0,
Disabled, Down, Disabled, Down,
0.0.0.0 0.0.0.0
Nve1 Vni, Mcast, Mode, 1 10011, 239.0.1.1, 10011, 239.0.1.1,
Type, Flags Mcast, L2, None Mcast, L2, None
Nve1 Vni, Mcast, Mode, 1 10010, 239.0.1.0, 10010, 239.0.1.0,
Type, Flags Mcast, L2, None Mcast, L2, None
Nve1 Vni, Mcast, Mode, 1 50000, 0.0.0.0, n/a, 50000, 0.0.0.0, n/a,
Type, Flags L3, L3VNI L3, L3VNI
Allowed VLANs - 1,5,9-12,99 ,100 1,5,10-12,99,100
Local suspended VLANs - 9 -
Leaf3#
ARP Suppression Cache
Leaf1# show ip arp suppression-cache local

Ip Address Age Mac Address Vlan Physical-ifindex Flags
100.1.1.1 00:15:48 8c60.4f1b.e43c 100 Ethernet1/12 L
Leaf1# show ip arp suppression-cache remote

Ip Address Age Mac Address Vlan Physical-ifindex Flags
100.1.1.2 00:05:19 8c60.4f19.51fc 100 (null) R
Leaf1(config)# hardware access-list tcam region arp-ether 256 double-wide
Troubleshooting VXLAN EVPN
Uplinks with SVI / Sub-Interfaces
• For SVI based uplinks, define the infra VLANs
• system nve infra-vlan <svi-vlan>
• Sub-Interfaces
• Not supported on ALE links (40G ports) (Documented)
• Check CCO documentation
• L3 Port-channel supported – Check CCO documentation
Troubleshooting BUM
Traffic
Troubleshooting BUM Traffic
BUM• Traffic over- Multicast
BUM Traffic Core (ARP and other broadcast packets), Multicast traffic from
Broadcast traffic
hosts, etc.
• Check the multicast group associated with the L2 VNI
• Get the Source VTEP IP and Router MAC
• Check if ARP Suppression is enabled or not
• Ask the right Questions:

• Check if the VTEP is a VPC VTEP or Standalone VTEP?
• Is the issue seen for IPv4 hosts or IPv6 hosts?
• Know the trigger and understand if the issue is continuously reproducible or not?
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Step 1 – Check if the BUM Traffic is making it to the source VTEP
LEAF45# ethanalyzer local interface inband capture-fil "arp" limit-captured-frames 0
Capturing on inband
2018-05-21 14:52:36.289960 20:20:00:00:00:aa -> 54:7f:ee:07:e1:41 ARP 10.150.1.254 is
at 20:20:00:00:00:aa
2018-05-21 14:52:36.295037 54:7f:ee:07:e1:41 -> ff:ff:ff:ff:ff:ff ARP Who has
10.150.1.35? Tell 10.150.1.36
2018-05-21 14:52:36.295425 2c:54:2d:f6:0f:bc -> 54:7f:ee:07:e1:41 ARP 10.150.1.35 is at
2c:54:2d:f6:0f:bc
2018-05-21 14:52:38.127594 54:7f:ee:07:e1:41 -> ff:ff:ff:ff:ff:ff ARP Who has
10.150.1.38? Tell 10.150.1.36
LEAF45# debug logfile arp

LEAF45# debug ip arp packet
14:52:38.127774 arp: (context 3) Receiving packet from Vlan1501, logical interface

Vlan1501 physical interface port-channel36, (prty 6) Hrd type 1 Prot type 800 Hrd len 6
Prot len 4 OP 1, Pkt size 46
14:52:38.127811 arp: Src 547f.ee07.e141/10.150.1.36 Dst ffff.ffff.ffff/10.150.1.38
Step 2 – Check the Mroute Entry – Src VTEP
LEAF45# show ip mroute 231.1.150.1 10.0.0.204 detail
IP Multicast Routing Table for VRF "default"
Total number of routes: 8
Total number of (*,G) routes: 2
Total number of (S,G) routes: 5
Total number of (*,G-prefix) routes: 1
(10.0.0.204/32, 231.1.150.1/32), uptime: 00:14:01, nve(0) mrib(0) ip(0) pim(1)

RPF-Source: 10.0.0.204 [0/0]
Data Created: No
Received Register stop
VXLAN Flags
VXLAN Encap
VPC Flags
RPF-Source Forwarder
Stats: 51/2601 [Packets/Bytes], 27.200 bps
Stats: Active Flow
Incoming interface: loopback1, RPF nbr: 10.0.0.204
Outgoing interface list: (count: 1) (bridge-only: 0)
Ethernet1/50, uptime: 00:09:52, pim
Step 3.1 – Check the Mroute Entry – Dest VTEP

(10.0.0.204/32, 231.1.150.1/32), uptime: 00:03:19, ip(0) mrib(1) pim(0)

RPF-Source: 10.0.0.204 [3/110]
Data Created: Yes
VXLAN Flags
VXLAN Decap
Stats: Inactive Flow
Incoming interface: Ethernet1/50, RPF nbr: 10.0.0.42
nve1, uptime: 00:03:19, mrib
Step 3.2 – Check the Mroute Entry – Dest VTEP
(10.0.0.204/32, 231.1.150.1/32), uptime: 00:03:57, ip(0) mrib(1) pim(0)

RPF-Source: 10.0.0.204 [3/110]
Data Created: Yes
VXLAN Flags
VXLAN Decap
VPC Flags
Incoming interface: Ethernet1/50, RPF nbr: 10.0.0.42
nve1, uptime: 00:03:57, mrib
Step 4 – Capturing BUM Traffic in Core
LEAF45(config)# monitor session 1
LEAF45(config-monitor)# source interface ethernet 1/50
LEAF45(config-monitor)# destination interface sup-eth 0
LEAF45(config-monitor)# no shut
LEAF45(config-monitor)# end
LEAF45# ethanalyzer local interface inband capture-filter "host 231.1.150.1" limit-

captured-frames 0
Capturing on inband
2018-05-21 16:21:01.985236 10.0.0.204 -> 231.1.150.1 UDP Source port: 41316
Destination port: 4789
Use the detail option with ethanalyzer to see the whole packet
Demo
Troubleshooting Tenant
Routed Multicast (TRM)
Tenant Routed Multicast (TRM)
Overview
• A BGP based solution for allowing multicast routing and snooping over
VXLAN EVPN fabric
• Sources and Receivers are connected to the VTEPs
• Multicast Source and Receiver info is propagated using BGP

• No PIM/IGMP packets sent to VXLAN fabric from any TRM VTEP
• Modes:
• L3 Mode
• L2/L3 Mixed Mode
• Both modes are supported only on N9k - EX or FX platforms

• Supported only with Multicast based core. IR not supported
Modes and RP
• L3 Mode
• RP configured on all VTEPs – (ip multicast overlay-spt-only command required)
• RP on selected VPC VTEP – not supported
• Internal RP – Supported from 7.0(3)I7(1)
• External RP – Supported from 7.0(3)I7(4) (upcoming release)
• L2/L3 Mode
• RP on all Distributed-DR – Supported on Tahoe
• RP on Core – not supported
• Internal RP - Supported from 7.0(3)I7(1)
• External RP – Not supported
EVPN – L3 Anycast
S S
V V V V
S R R R
Configuring Layer 3 Tenant Routed Multicast
feature ngmvpn interface loopback1000

Ip igmp snooping vxlan vrf member EVPN-TENANT
ip multicast overlay-spt-only ip address 209.165.200.1/32
advertise evpn multicast Ip pim sparse-mode
interface nve1 Vrf context EVPN-TENANT
no shutdown Ip pim rp-address 209.165.200.1 group-lost 224.0.0.0/4
source-interface loopback0 interface Vlan500
host-reachability protocol bgp no shutdown
member vni 50000 associate-vrf vrf member EVPN-TENANT
mcast-group 225.3.3.3 ip forward
router bgp 65000 ip pim sparse-mode
vrf EVPN-TENANT Interface Vlan100
address-family ipv4 unicast no shutdown
network 200.1.1.1/32 vrf member EVPN-TENANT
advertise l2vpn evpn ip address 100.1.1.254/24
address-family ipv4 mvpn fabric forwarding mode anycast-gateway
ip pim neighbor-policy NONE*
TRM Verification
Leaf1# show fabric multicast vrf all
Fabric Multicast Enabled VRFs
VRF Name VRF Vprime VN-Seg
ID If ID
default 1 Null 0
EVPN-TENANT 4 Vlan500 50000
Leaf1# show fabric multicast globals

Pruning: segment-based
Switch role: leaf
Fabric Control Seg: Null
Peer Fabric Control Address: 0.0.0.0
Advertising vPC RPF routes: Disabled
Created VNI List: -
Fwd Encap: Fabric
Overlay Distributed-DR: FALSE
Overlay spt-only: TRUE
Show bgp internal mvpn detail
Leaf1# show bgp internal mvpn detail
*************************************************
NGMVPN feature/server/role: VxLAN/1/VxLAN
NGMVPN registered: Yes (Jun 9 00:56:59.297696/never)
NGMVPN TRM mode: L3 (0x000002)
NGMVPN down: in-prg/up-defer: 0/0
NGMVPN register/failures: 1/0
NGMVPN deregister/failures: 0/0
NGMVPN Convergence sent: 0
NGMVPN local-req sent/skipped: 7/3
NGMVPN local-req sent: 4 (L2VNI)/ 2 (L3VNI)/ 1 (All VNIs)
NGMVPN remote-req rcvd: 0 (L2VNI)/ 0 (L3VNI)/ 1 (All VNIs)
NGMVPN del remote: 0 (L2VNI)/ 0 (L3VNI)/ 0 (All VNIs)
NGMVPN msgs sent/acks rcvd: 9/9
NGMVPN msgs rcvd/acks sent: 24/14
NGMVPN msg err/ack err/drops: 0/0/0
Last xid sent to NGMVPN: 9
Last xid ack by NGMVPN: 9
. . . .
Show bgp internal mvpn detail (contd…)
++++++++++++++++++++++++++++++++++++++++++
BGP MVPN RD Information for 192.168.1.1:4 (0xd625952c)
VNI ID : 50000
VRF : EVPN-TENANT
Global NGMVPN mode : L3 (1 L3 VRFs)
VRF L3 Mode : Yes Jun 9 00:57:00.203673
Enabled : Yes
Delete Pending : No
Cleanup Pending : No
Import Pending : No
Import In Progress : No
Created : Jan 9 00:56:55.570427
Enabled At : Jan 9 00:56:55.570471
. . .
Is Auto RT : No
Config VRF Import RT : 1
Import RT cfg list: 192.168.1.1:500
Active VRF Import RT : 1
Active VRF Import RT list : 192.168.1.1:500
VRF Import RT chg/chg-pending : 0/0
Join from Receiver
Leaf3# show ip mroute 239.0.0.1 detail vrf EVPN-TENANT
IP Multicast Routing Table for VRF ”EVPN-TEANT"

(*, 239.0.0.1/32), uptime: 00:00:55, igmp(1) ip(0) pim(0)

RPF-Source: 209.165.200.1 [0/0]
RD-RT ext comm Route-Import:
Data Created: No
VPC Flags
Incoming interface: loopback1000, RPF nbr: 209.165.200.1
Vlan100, uptime: 00:00:55, igmp (vpc-svi)
FHR VTEP sends SA-AD (Type-5 Route) using BGP to other VTEPs
Leaf1# show bgp ipv4 mvpn sa-ad detail vrf EVPN-TENANT
BGP routing table information for VRF default, address family IPv4 MVPN
BGP routing table entry for [5][10.0.0.1][239.0.0.1]/64, version 34
Flags: (0x000002) (high32 00000000) on xmit-list, is not in mvpn
0.0.0.0 (metric 0) from 0.0.0.0 (192.168.1.1)
Extcommunity: RT:65000:50000

10.10.10.10 20.20.20.20
LHR / Remote VTEPs build (S, G)
Leaf3# show ip mroute 239.0.0.1 10.0.0.1 detail vrf EVPN-TENANT
IP Multicast Routing Table for VRF ”EVPN-TEANT"

(10.0.0.1/32, 239.0.0.1/32), uptime: 00:01:55, ip(0) mrib(1) pim(0) ngmvpn(0)

RPF-Source: 10.0.0.1 [0/20]
RD-RT ext comm Route-Import: 0b c0 a8 01 01 05 dc 00 01 c0 a8 01 01 83 e7
Data Created: Yes
Fabric dont age route
VPC Flags
Incoming interface: Vlan500, RPF nbr: 192.168.1.1
Vlan100, uptime: 00:01:55, mrib (vpc-svi)
VTEP3/4 sends Type-7 back to FHR on L3VNI
Leaf3# show bgp ipv4 mvpn route-type 7 detail
Route Distinguisher: 192.168.1.1:33767 (Local VNI: 50000)
BGP routing table entry for [7][10.0.0.1][239.0.0.1][65000]/96, version 43
Flags: (0x000002) (high32 00000000) on xmit-list, is not in mvpn
0.0.0.0 (metric 0) from 0.0.0.0 (192.168.3.3)
Extcommunity: RT:192.168.1.1:500

10.10.10.10 20.20.20.20
VTEP1 Receives the Type-7 route
Leaf1# show bgp ipv4 mvpn route-type 7 detail
BGP routing table entry for [7][10.0.0.1][239.0.0.1][65000]/96, version 36
Flags: (0x00001a) (high32 00000000) on xmit-list, is in mvpn, is not in HW
Path type: external, path is valid, is best path, in rib
Imported from 192.168.1.1:33767:[7][10.0.0.1][239.0.0.1][65000]/96
192.168.100.100 (metric 0) from 10.10.10.10 (192.168.10.10)
Extcommunity: RT:192.168.1.1:500

20.20.20.20
VTEP1 adds L3VNI in the OIF List
IP Multicast Routing Table for VRF ”EVPN-TENANT"


RPF-Source: 10.0.0.1 [0/0]
RD-RT ext comm Route-Import:
Data Created: Yes
Received Register stop
Stats: Active Flow
Incoming interface: Vlan1000, RPF nbr: 10.0.0.1, internal
Outgoing interface list: (count: 2) (Fabric OIF) (bridge-only: 0)
Vlan500, uptime: 00:12:02, ngmvpn
Vlan1000, uptime: 00:12:03, mrib, (RPF)
VTEP3 adds Receiver Vlan in the OIF List
IP Multicast Routing Table for VRF "TRM"


RPF-Source: 10.0.0.1 [0/20]
RD-RT ext comm Route-Import: 0b c0 a8 01 01 05 dc 00 01 c0 a8 01 01 83 e7
Data Created: Yes
VPC Flags
Incoming interface: Vlan500, RPF nbr: 192.168.1.1
Vlan1000, uptime: 00:24:35, mrib (vpc-svi)
MFDM (Validate this info on both FHR and LHR)
Leaf1# show forwarding distribution multicast route vrf EVPN-TENANT group 239.0.0.1
source 10.0.0.1
(10.0.0.1/32, 239.0.0.1/32), RPF Interface: Vlan1000, flags:

Received Packets: 0 Bytes: 0
Number of Outgoing Interfaces: 2
Outgoing Interface List Index: 29
Vlan1000
( Mem L2 Ports: nve1 )
l2_oiflist_index: 1
Vlan500 (Vxlan Encap)
( Mem L2 Ports: nve1 )
l2_oiflist_index: 1
MFIB Database
Show forwarding multicast outgoing-interface-list l3 <index>
Troubleshooting Tools
VXLAN OAM
• VXLAN OAM feature introduced in 7.0(3)I5(2) – NGOAM
• Need a feature for Path verification and Path tracking with Telemetry
data
• Similar to Fabric Path OAM Enable NGOAM
Feature
• Provides 3 features
feature ngoam
• VxLAN Ping Create Profile
ngoam profile 1
• VxLAN Traceroute oam-channel 2
! Install ACL
• VxLAN PathTrace
ngoam install acl
VXLAN OAM
Usability
• Helps diagnose underlay / overlay reachability of VMs
• Covers exact path as Data Packet
• Path verification for all ECMP paths in Overlay
• Path tracking – Exact path host traffic takes in overlay and underlay
network
• Layer 2 – Traceroute / Ping to VM host from Leaf
• Layer 3 – Traceroute / Ping to Vm host from Leaf
• Flexible OAM channel supporting multiple drafts
• Tissa draft – nvo3
VXLAN OAM
VXLAN PING
• VxLAN Ping checks connectivity to the destination, where the destination
can be VM’s IP address or routed loopback addresses on the leaf switch
• Since there are multiple paths, only one path is followed based on the flow
parameters
• Ping for both VM / Host MAC and IP
• Default ping support – Ping based on just destination address and VNI
segment
• Allow users to specify flow parameters such as UDP port, destination and
source address
• This helps VxLAN ping follow the specific path the unicast ping will take to reach
the destination
VxLAN OAM
PING NVE MAC
Leaf1# ping nve mac 0050.56b3.bcef 200 port-channel 101 profile 1 vni 20000
verbose
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
'D' - Destination Unreachable, 'X' - unknown return code,
'm' - malformed request(parameter problem)
,'c' - Corrupted Data/Test, '#' - Duplicate response
Sender handle: 21
! sport 51932 size 39,Reply from 192.168.100.100,time = 5 ms
VxLAN OAM
VxLAN Traceroute
• VxLAN Traceroute – Used to trace the path a packet takes between source
and destination
• Only one path is traced based on the given flow parameters
• Trace will show uni-directional path the packet takes to the destination, but the
return path may be different
• Should be able to trace VTEPs, access switch and end-host. For the
default mode, the user should be able to trace the tunnel endpoint IP
address and the segment ID
• Actual path taken by a packet is dependent on all the L2/L3/L4 header
fields and network topology at the time the packet is sent
• Users can specify the flow parameters
VxLAN OAM
VxLAN PathTrace
• Similar to Traceroute, but uses Nvo3 channel
• Carries additional ingress / egress and load information of the path
• Provides additional information if the device supports nvo3 channel else its
same as traceroute
• Actual path taken by a packet is dependent on all the L2/L3/L4
header fields and network topology at the time the packet is sent
•
Leaf1# Users can specify the flow parameters
pathtrace nve mac 0050.56b3.bcef 200 port-channel 101 vni 20000
<snip>
Path trace Request to peer ip 192.168.100.100 source ip 192.168.99.99
Sender handle: 35Hop Code ReplyIP IngressI/f EgressI/f State
====================================================
1 !Reply from 10.101.1.10, Eth2/1 Eth1/17 UP / UP
2 !Reply from 192.168.100.100, Eth1/17 Unknown UP / DOWN
ELAM
• Embedded Logic Analyzer module (ELAM) – tool used to capture a packet
processed by a Cisco ASIC
• Depending on the N9k platform,
• ELAM on NS ASIC
• ELAM on TAHOE ASIC
• Useful in scenario’s where packet forwarding is impacted
• Can perform capture for raw packet from the host and even VxLAN
encapsulated packet towards the VxLAN Core
• Useful for Cisco TAC and Cisco Engineering for understanding the cause of
packet loss or impacted forwarding
ELAM on N9k Platform
TAHOE ELAM Steps
• Attach module X
• Debug platform internal tah elam asic [0 | 1]
• Trigger [init | reset] asic [num] slice [num] lu-a2d [0 | 1] in-
select [3-7] out-select [0-5] use-src-id [src-id]
• Lu-a2d 0 – used for reverse ELAM, where trigger is based on result
• Lu-a2d 1 – used for ELAM where trigger is based on packet attributes
• Set outer [ipv4 | l2 | l4] . . . .
• Start
• Status {Can be Armed / Triggered}
• Report [detail]
TAHOE ELAM
Spine
att mod 1
debug platform internal tah elam asic 0
trigger init asic 0 slice 0 lu-a2d 1 in-select 7 out-select 0 use-src-id 52
reset VxLAN Encapsulated Packet
set inner ipv4 src_ip 100.1.1.1 dst_ip 100.1.1.5
start
report
Leaf1
att mod 1
debug platform internal tah elam asic 0
trigger init asic 0 slice 0 lu 1 in-select 6 out-select 0
reset Host IP Packet
set outer ipv4 src_ip 100.1.1.1 dst_ip 100.1.1.5
start
report
Partial Output
• Dot1Q Header
module-1(TAH-elam-insel6)# report detail | grep pr_lu_vec_l2v.qtag0
GBL_C++: [MSG] pr_lu_vec_l2v.qtag0_vld: 0x1 << dot1q yes? 0x1
GBL_C++: [MSG] pr_lu_vec_l2v.qtag0_cos: 0x0
GBL_C++: [MSG] pr_lu_vec_l2v.qtag0_de: 0x0
GBL_C++: [MSG] pr_lu_vec_l2v.qtag0_vlan: 0x64 << VL 100
• VLAN
module-1(TAH-elam-insel6)# report detail | grep -1 fpx_lookup_vec.lkup.macsakey.key.fid
GBL_C++: [MSG] fpx_lookup_vec.lkup.macsakey.key.vld: 0x1
GBL_C++: [MSG] fpx_lookup_vec.lkup.macsakey.key.fid_type: 0x0
GBL_C++: [MSG] fpx_lookup_vec.lkup.macsakey.key.fid_vld: 0x0
GBL_C++: [MSG] fpx_lookup_vec.lkup.macsakey.key.fid: 0x64 << dec 0xa = VL 10
GBL_C++: [MSG] fpx_lookup_vec.lkup.macsakey.key.mac: 0xFEC80E2715
Partial Output
• Src & Dst IP
module-1(TAH-elam-insel6)# report detail | grep vec_l3v.ip.*a
GBL_C++: [MSG] pr_lu_vec_l3v.ip.da: 0x0000000000000000064010101 << 100.1.1.1
GBL_C++: [MSG] pr_lu_vec_l3v.ip.sa: 0x0000000000000000064010105 << 100.1.1.5
• Src MAC
module-1(TAH-elam-insel6)# report detail | grep -i fpx_lookup_vec.lkup.macsakey.key.mac
GBL_C++: [MSG] fpx_lookup_vec.lkup.macsakey.key.mac: 0xFEC80E2715 << 00fe.c80e.2715
Great works and great Folly
may be indistinguishable at
the outset
- Adam Steltzner
ELAM Wrapper - Demo
debug platform internal tah elam
trigger init
set outer ipv4 src-ip ip-address dst-ip
ip-address
start
report [detail]
Demo – Consistency
Checker (CC)
- test consistency-checker forwarding
[ipv4 unicast | vrf vrf-name] [module
slot] [stop]
- Show consistency-checker forwarding

Demo – L2 PI/PD
Troubleshooting
Show troubleshoot l2 mac mac-

address [vlan vlan-id]
Demo – L3 PI/PD
Troubleshooting
Show troubleshoot l3 [ipv4 | ipv6] v4/v6 address

[src-ip v4/v6 address] [vrf vrf-name]
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco Webex Teams
Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session
How
1 Find this session in the Cisco Events Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
cs.co/ciscolivebot#BRKDCN-3040
Complete your online
session survey
• Please complete your Online Session
Survey after each session
• Complete 4 Session Surveys & the Overall
Conference Survey (available from
Thursday) to receive your Cisco Live T-
shirt
• All surveys can be completed via the Cisco
Events Mobile App or the Communication
Stations
Don’t forget: Cisco Live sessions will be available for viewing

on demand after the event at ciscolive.cisco.com
Continue Your Education
Demos in Meet the Related

Walk-in
the Cisco engineer sessions
self-paced
Showcase labs 1:1
meetings
Thank you

BRKDCN-3040 Troubleshooting Vxlan Evpn 2019

Uploaded by

Copyright:

Available Formats

BRKDCN-3040 Troubleshooting Vxlan Evpn 2019

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKDCN-3040 Troubleshooting Vxlan Evpn 2019

Uploaded by

Copyright:

Available Formats

What is VXLAN?

What is VXLAN?

What are the components of VXLAN?

What are the components of VXLAN?

BRKDCN-3040

Outer Outer Inner

Flags Reserved Instance ID Reserved

Outer S-IP: 192.168.1.1 MAC-A 10000 192.168.1.1

SVI SVI SVI SVI

VLAN X VLAN A VLAN B VLAN N

• 1 Layer-3 VNI per • 1 Layer-2 VNI per Layer-2 segment

Mac_H1: VNI1 Mac_H3: VNI1

MAC_H1 MAC_H2 MAC_H3 MAC_H4

VNI1 VNI2 VNI1 VNI2

VTEP-1 VTEP-2 VTEP-3 VTEP-3

SVI SVI SVI SVI

fabric forwarding anycast-gateway-mac 0001.0001.0001

cache table, VTEP floods the ARP

Leaf1 (VTEP-1) Leaf2 (VTEP-2) Leaf3 (VTEP-3) Leaf4 (VTEP-3)

feature bgp For BGP Configuraiton

Enables EVPN AFI

Vlan 100 Vlan 100

Unicast Static Ingress-Replication EVPN Ingress-Replication

MFDM L2FM ARP IPv6 ND

 Route type 2 or MAC Advertisement route is for

Route Distinguisher: 192.168.1.1:3 (L3VNI 50000)

Leaf1# show interface nve1

Leaf1#show mac address-table vlan 100

Leaf1# sh sys inter l2fm event-hist deb | in 523d.e706.ae1b

Leaf1# show system internal l2rib event-history mac | in 523d.e706.ae1b

Leaf1#show bgp l2vpn evpn vni-id 10000

Leaf1# show bgp internal event-history events | in 523d.e706.ae1b

Leaf1# show bgp l2vpn evpn 523d.e706.ae1b

Leaf1#show bgp l2vpn evpn 523d.e706.ae1b Route Distinguisher – 8 byte

Leaf2# show bgp internal event-history events | in 523d.e706.ae1b

VNI Peer-IP Peer-MAC Tunnel-ID Encap (A/S)

Leaf2# show l2route evpn mac evi 100

Leaf2# show nve peers detail

Leaf2# show system internal l2fm debugs | in 523d.e706.ae1b

Leaf2# show mac address-table vlan 100

Leaf1# show forwarding vrf EVPN-TENANT adjacency

next-hop rewrite info interface AM

Leaf1# show l2route evpn mac-ip evi 100

Path-id 1 advertised to peers:

Leaf2# show l2route evpn mac-ip evi 100

Leaf2# show ip route vrf EVPN-TENANT 100.1.1.1

Path-id 1 not advertised to any peer

Path-id 1 not advertised to any peer

SW_BD/VNIs of interface nve1:

Peers of interface nve1:

Name Type Local Value Peer Value

Leaf1# show ip arp suppression-cache local

100.1.1.1 00:15:48 8c60.4f1b.e43c 100 Ethernet1/12 L

Leaf1# show ip arp suppression-cache remote

100.1.1.2 00:05:19 8c60.4f19.51fc 100 (null) R

Leaf1(config)# hardware access-list tcam region arp-ether 256 double-wide

• L3 Port-channel supported – Check CCO documentation

• Get the Source VTEP IP and Router MAC

• Check if ARP Suppression is enabled or not