#CLUS

Troubleshooting
VXLAN BGP EVPN
Vinit Jain
Technical Leader, Services
Twitter - @vinugenie
BRKDCN-3040

#CLUS
Agenda • Introduction
• VXLAN Overview
• VXLAN Packet Flow
• Configuration
• Nexus 9000 Components
• Control Plane Troubleshooting
• Troubleshooting BUM Traffic
• Troubleshooting Tenant Routed
Multicast
• Troubleshooting Tools
• Conclusion
#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
VxLAN Overview
VXLAN Concepts
• VXLAN Overlay
- A VXLAN Overlay or VXLAN segment is a Layer-2 broadcast domain identified by
the VNID that extends or tunnels traffic from one VTEP to another.
• VXLAN Tunnel End Point (VTEP)
- A VTEP is a device that provides both encapsulation and de-capsulation of
classical Ethernet and VXLAN packets to and from a VXLAN segment
- Each VTEP may have the following types of interfaces:
 Switchport interfaces on the local LAN segment to support local endpoints
 Layer-3 interfaces to the transport IP network
 SVI interfaces

• VXLAN Gateway
- A VTEP that bridges traffic between VXLAN segments
#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
 VXLAN Encapsulation
Ethernet
Payload FCS
Header

Outer Outer Inner

Outer IP VXLAN Payload New FCS
Ethernet UDP Ethernet

Flags Reserved Instance ID Reserved

8 Bytes 24 Bytes 24 Bytes 8 Bytes

Rsvd Rsvd
Outer UDP Destination Port = VXLAN (originally 8472, recently updated to 4789)
1
Outer UDP Source Port = Hash of Inner Frame Headers (optional)

• The outer IP header has the source IP and destination IP of the VTEP
endpoints
• The outer Ethernet header has the source MAC of the source VTEP
and the destination MAC of the immediate Layer-3 next hop
#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
VxLAN Packet Structure
F
Outer MAC Outer UDP VxLAN
Outer IP Header Original Layer 2 Frame C
Header Header Header
S

14 Bytes
(4 bytes optional) 8 Bytes Hash of inner L2/l3/L4
Destination NH MAC Source Port
headers of original frame.
16 Enables entropy for ECMP
48 Address Addr
16 VxLAN Port UDP 4789 Load balancing
48 Source Address
16 UDP Length
16 VLAN Type
0x8100 16 Checksum
16 VLAN ID Tag
Src and Dst
16 Ether Type 20 Bytes Addresses of 8 Bytes
0x0800
VTEP 16M possible
72 IP Header Misc. VxLAN
Data 8 RRRR1RRR segments
8 Protocol 0x11 Reserved
Src VTEP MAC Addr 24
16 Header
24 VNI
32 Checksum
Source IP 8 Reserved
32
Destination IP
#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
VxLAN Overview
VxLAN Gateway Types
• Layer 2 Gateway
− The layer 2 gateway is required when the layer 2 traffic (IEEE 802.1q
tagged traffic) comes from VLAN into VxLAN segment (encapsulation) or
− The Ingress VxLAN packet egresses out an 802.1q tagged interface (de-
encapsulation), where the packet is bridged to a new VLAN.
• Layer 3 Gateway
− A layer 3 gateway is used when there is a VxLAN to VxLAN routing
− The ingress packet is a VxLAN packet on a routed segment but the
packet egresses out on a tagged 802.1q interface and the packet is
routed to a new VLAN

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
 End System End System

VxLAN Overview 3
ARP Request for IP B
VxLAN - Flood and Learn Src MAC: MAC-A
Dst MAC: FF:FF:FF:FF:FF:FF

MAC-3
IP- 192.168.3.3
VTEP 3
S-MAC: MAC-1
D-MAC: MAC VxLAN Remote

VTEP-3
00:01:5E:01:01:01 Address ID VTEP

Outer S-IP: 192.168.1.1 MAC-A 10 192.168.1.1

4
Outer D-IP: 239.1.1.1
UDP ARP Response from IP B
Src MAC: MAC-B
VXLAN VNID: 10
Dst MAC: MAC-A
7
ARP Request for IP B
Src MAC: MAC-A 2
Dst MAC: VTEP 2
Mcast Group
ARP Response from IP B
FF:FF:FF:FF:FF:FF IP- 192.168.2.2 End System B
239.1.1.1
Src MAC: MAC-B MAC-2 MAC-B
Dst MAC: MAC-A IP - B
2
VTEP-1 VTEP-2 3
End System A S-MAC: MAC-2
MAC-A D-MAC: MAC-1 ARP Request for IP B
IP - A VTEP 1 5 Src MAC: MAC-A
1 IP-192.168.1.1 Outer S-IP: 192.168.2.2 Dst MAC: FF:FF:FF:FF:FF:FF
6
MAC-1 Outer D-IP: 192.168.1.1
MAC VxLAN Remote UDP MAC VxLAN Remote
ARP Request for IP B
Address ID VTEP Address ID VTEP
Src MAC: MAC-A
Dst MAC: FF:FF:FF:FF:FF:FF VXLAN VNID: 10
MAC-B 10 IP-2 MAC-A 10 192.168.1.1
ARP Response

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
VxLAN Overview
VxLAN – Flood and Learn
• Data Plane learning technique for VxLAN
• VNI’s are mapped to a multicast group on a VTEP
• Local MACs are learnt over a VLAN (VNI) on a VTEP
• Broadcast, Unknown Unicast, Multicast (BUM Traffic) is flooded to
the delivery multicast group for that VNI
• Remote VTEPs part of same multicast group learn host MAC, VNI
and source VTEP as the next-hop for the host MAC from flooded
traffic
• Unicast packets to the host MAC are sent directly to source VTEP
as VxLAN encapsulated packet
#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
VxLAN Overview
Ingress Replication
• Some customers not comfortable deploying multicast in their core
• With Ingress Replication (IR), BUM traffic ingress access side is
replicated to remote VTEP as unicast
• Static IR VETP tunnel is kept alive as long as the route to the VTEP
is available.
• Support multiple VTEPs per VNI and a VTEP in multiple VNIs

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
VxLAN Overview
Tenant

Tenant 1 (VRF 1)

SVI SVI SVI SVI

X A B N
Layer-3 VNI X’ Layer-2 VNI A’ Layer-2 VNI B’ Layer-2 VNI N’

VLAN X VLAN A VLAN B VLAN N

• 1 Layer-3 VNI per • 1 Layer-2 VNI per Layer-2 segment

Tenant (VRF) for
• Multiple Layer-2 VNIs per tenant
routing
• VNI A’ and B’ are used for bridged
• VNI X’ is used for
packets
routed packets

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
VxLAN Overview
VxLAN EVPN
BGP-EVPN BGP-EVPN

Mac_H1: VNI1 Mac_H3: VNI1

IP_H1: VNI3 IP Multicast Core IP_H3: VNI3
NHOP: VTEP-1 Or NHOP: VTEP-1
Mac_H2: VNI2 Unicast Core Mac_H4: VNI2
IP_H2: VNI3 VTEP-1 VTEP-2 IP_H4: VNI3
NHOP: VTEP-1 NHOP: VTEP-2
V V
VNI3 VNI3
Vlan- VRF VRF Vlan-
500 A A 500
L3 VNI L3 VNI

MAC_H1 MAC_H1 MAC_H3 MAC_H4

IP_H1 IP_H1 IP_H3 IP_H4

VNI1 VNI2 VNI1 VNI2

Vlan-100 Vlan-200 Vlan-100 Vlan-200

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
VxLAN Overview
Distributed Anycast Gateway
Spine1 Spine2

L3 Underlay
Unicast / Multicast Routing

VTEP-1 VTEP-2 VTEP-3 VTEP-3

SVI SVI SVI SVI

GW IP GW IP GW IP GW IP
GW MAC GW MAC GW MAC GW MAC

Host-A Host-B

Host-C
#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
 VxLAN Overview
Distributed Anycast Gateway - Configuration
• All VTEPs has same IP address for an L2 VNI
• Anycast Gateway MAC is global to each VTEP for all VNI’s for all Tenants
• One virtual MAC / VTEP
• All VTEPs should have same virtual MAC address

fabric forwarding anycast-gateway-mac 0001.0001.0001

!
interface Vlan100
no shutdown
vrf context test-evpn-tenant
ip address 172.16.1.254/24
fabric forwarding mode anycast-gateway

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
 VxLAN Overview
ARP Suppression
• Hosts send out G-ARP when they
come online
IP Multicast Core
• Local leaf node receives G-ARP, 2 +
creates local ARP cache and VTEP-1 VxLAN EVPN VTEP-2
advertises to other leaf by BGP as 3 V V
route type 2 VRF VRF
A A
• Remote leaf node puts IP-MAC info
into remote ARP cache and supresses 4 MAC_H1
1 MAC_H2
incoming ARP request for this IP IP_H1 IP_H2

VNI1 VNI1
• If IP info not found in ARP suppression Vlan-100 Vlan-100

cache table, VTEP floods the ARP

request to other VTEPs
#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Configuration
 L2 VNI – 10000, 20000
VxLAN Configuration L3 VNI - 50000
Topology Spine Spine
10.10.10.10 20.20.20.20

L3 Underlay
Unicast / Multicast Routing
VTEP-3
192.168.100.100

Leaf1 (VTEP-1) Leaf2 (VTEP-2) Leaf3 (VTEP-3) Leaf4 (VTEP-3)

192.168.1.1 192.168.2.2 192.168.3.3 192.168.4.4

VPC

Host-A

Host-B
Host-C
#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Configuration
Feature Enablement

feature bgp For BGP Configuration

Feature pim
feature interface-vlan For PIM / Multicast Core
feature vn-segment-vlan-based
feature nv overlay SVI Configuration
!
nv overlay evpn For VxLAN Overlay
!
Map VLAN to VNI

Enables EVPN AFI

Not required to be
enabled on Spine

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Underlay Configuration

Leaf Spine
router bgp 65000 router bgp 65001
router-id 192.168.1.1 router-id 192.168.1.1
address-family ipv4 unicast address-family ipv4 unicast
network 1.1.1.1/32 network 10.10.10.10/32
network 192.168.1.1/32 network 192.168.10.10/32
address-family l2vpn evpn address-family l2vpn evpn
neighbor 10.1.101.10 neighbor 10.1.201.1
remote-as 65001 remote-as 65000
address-family ipv4 unicast address-family ipv4 unicast
allowas-in 3 allowas-in 3
disable-peer-as-check disable-peer-as-check
neighbor 10.1.201.20 neighbor 10.1.202.2
remote-as 65001 remote-as 65000
address-family ipv4 unicast address-family ipv4 unicast
allowas-in 3 allowas-in 3
disable-peer-as-check disable-peer-as-check

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Leaf Node Configuration – L2 VNI
vlan 100 fabric forwarding anycast-gateway-mac
vn-segment 10000 0001.0001.0001
interface Vlan100
! Create L2 VNI no shutdown
vrf member EVPN-TENANT
evpn ip address 100.1.1.254/24
vni 10000 l2 fabric forwarding mode anycast-gateway
rd 10000:1 !
route-target import 10000:1 router bgp 65000
route-target export 10000:1 neighbor 10.10.10.10
! remote-as 65001
interface nve1 update-source loopback0
no shutdown ebgp-multihop 3
source-interface loopback0 address-family l2vpn evpn
host-reachability protocol bgp allowas-in 3
member vni 10000 disable-peer-as-check
mcast-group 239.1.1.1 send-community extended
suppress-arp vrf EVPN-TENANT
address-family ipv4 unicast
advertiseBRKDCN-3040
#CLUS
l2vpn evpn© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Leaf Node Configuration – L3 VNI
vlan 500 interface nve1
vn-segment 50000 no shutdown
! source-interface loopback0
vrf context EVPN-TENANT host-reachability protocol bgp
vni 50000 member vni 50000 associate-vrf
rd 50000:1 !
address-family ipv4 unicast interface loopback200
route-target import 20000:1 vrf member EVPN-TENANT
route-target import 20000:1 evpn ip address 200.1.1.1/32
route-target export 20000:1 !
route-target export 20000:1 evpn router bgp 65000
! vrf EVPN-TENANT
interface Vlan500 address-family ipv4 unicast
no shutdown network 200.1.1.1/32
vrf member EVPN-TENANT advertise l2vpn evpn
ip forward
!

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Leaf Node with VPC Configuration
vpc domain 10
peer-switch
peer-keepalive destination 10.1.34.4 source
10.1.34.3 VTEP IP. The secondary
delay restore 60 IP is same on both Leaf3
peer-gateway and Leaf 4 running VPC
ipv6 nd synchronize
ip arp synchronize
!
interface loopback0
ip address 192.168.3.3/32 Backup Routing SVI
ip address 192.168.100.100/32 secondary
ip router ospf 100 area 0.0.0.0 vlan 5
ip pim sparse-mode interface vlan 5
ip add 10.5.1.1/24
ip router ospf 100 area 0.0.0.0
Backup Routing SVI ip pim sparse-mode
Configured on both vPC
peers and part of global
routing table.

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
Spine Node Configuration
router bgp 65001
address-family l2vpn evpn
nexthop route-map permitall
interface loopback0 retain route-target all
ip address 192.168.10.10/32 neighbor 1.1.1.1
ip pim sparse-mode remote-as 65000
update-source loopback0
!
ebgp-multihop 3
interface loopback1 address-family l2vpn evpn
ip address 192.168.50.50/32 disable-peer-as-check
ip pim sparse-mode send-community extended
! route-map permitall out
ip pim rp-add 192.168.50.50 group-list neighbor 2.2.2.2
remote-as 65000
239.1.1.0/24
update-source loopback0
ip pim anycast-rp 192.168.50.50 192.168.10.10 ebgp-multihop 3
ip pim anycast-rp 192.168.50.50 192.168.20.20 address-family l2vpn evpn
disable-peer-as-check
send-community extended
route-map permitall out
!
route-map permitall permit 10
set ip next-hop unchanged
#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
Leaf Node Configuration – L2 & L3 VNI for IPv6
vlan 100 fabric forwarding anycast-gateway-mac 0001.0001.0001
vn-segment 10000 interface Vlan100
vlan 200 no shutdown
vn-segment 20000 vrf member EVPN-TENANT
evpn ip address 100.1.1.254/24
vni 10000 l2 ipv6 address 2001::1/64
rd 10000:1 fabric forwarding mode anycast-gateway
route-target import 10000:1 !
route-target export 10000:1 interface nve1
! no shutdown
vrf context EVPN-TENANT source-interface loopback0
vni 20000 host-reachability protocol bgp
rd 20000:1 member vni 10000
address-family ipv4 unicast mcast-group 239.1.1.1
route-target both 20000:1 suppress-arp
route-target both 20000:1 evpn member vni 20000 associate-vrf
address-family ipv6 unicast !
route-target both 20000:1 router bgp 100
route-target both 20000:1 evpn vrf EVPN-TENANT
! address-family ipv4 unicast
interface Vlan200 advertise l2vpn evpn
no shutdown address-family ipv6 unicast
vrf member EVPN-TENANT advertise l2vpn evpn
ip forward !
ipv6 address use-link-local-only vpc domain 10
ipv6 nd synchronize

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
VxLAN EVPN Configuration
Host Learning and Peer Discovery
Host Learning Data Plane Control Plane

CORE
Multicast Flood and Learn EVPN-Multicast
Peer Learning: Data Plane Peer Learning: BGP-RnH

Vlan 100 Vlan 100

vn-segment 10000 vn-segment 10000
Interface nve1 Interface nve1
Member vni 10000 host-reachability protocol bgp
Mcast-group 239.1.1.1 member vni 10000
Mcast-group 239.1.1.1

Unicast Static Ingress-Replication EVPN Ingress-Replication

Peer Learning - CLI Peer Learning – BGP-IMET
Vlan 150 Vlan 150
vn-segment 15000 vn-segment 15000
Interface nve1 Interface nve1
member vni 15000 host-reachability protocol bgp
Ingress-replication protocol member vni 15000
static peer x.x.x.x ingress-replication protocol
bgp

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
VXLAN BGP EVPN Control-Plane
Verification
Nexus 9000 VxLAN Architecture
BGP

URIB, VxLAN
U6RIB Manager L2RIB
(PI)
HMM
MRIB

VxLAN
Manager Adj. Mgr
(PD)

MFDM L2FM ARP IPv6 ND

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Troubleshooting VxLAN EVPN
VxLAN EVPN Prefix Types
Leaf1# show bgp l2vpn evpn
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 10000:1 (L2VNI 10000)
*>l[2]:[0]:[0]:[48]:[523d.e706.ae1b]:[0]:[0.0.0.0]/216
192.168.1.1 100 32768 i
*>l[2]:[0]:[0]:[48]:[523d.e706.ae1b]:[32]:[100.1.1.1]/272
192.168.1.1 100 32768 i
*>l[3]:[0]:[32]:[192.168.1.1]/88
192.168.1.1 100 32768 i
*>e[3]:[0]:[32]:[192.168.2.2]/88
192.168.2.2 0 65001 65000 i
*>e[3]:[0]:[32]:[192.168.3.3]/88
192.168.3.3 0 65001 65000 I
Route Distinguisher: 192.168.2.2:32967
* e[3]:[0]:[32]:[192.168.2.2]/88
192.168.2.2 0 65001 65000 i
*>e 192.168.2.2 0 65001 65000 i

Route Distinguisher: 192.168.1.1:3 (L3VNI 50000)

*>e[5]:[0]:[0]:[32]:[100.100.100.2]:[0.0.0.0]/224
192.168.2.2 0 65001 65000 i

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
 Troubleshooting VxLAN EVPN
EVPN Prefix Types
 BGP EVPN uses 5 different route types for IP
prefixes and advertisement Route TYPE - 8
 Type 1 - Ethernet Auto-Discovery (A-D) route
Length - 10
 Type 2 - MAC advertisement route  L2 VNI MAC/MAC-IP
 Type 3 - Inclusive Multicast Route  EVPN IR, Peer Discovery Route Type Specific
 Type 4 - Ethernet Segment Route
 Type 5 - IP Prefix Route  L3 VNI Route

 Route type 2 or MAC Advertisement route is for MAC

and ARP resolution advertisement, MAC or MAC-IP
 Route type 5 or IP Prefix route will be used for the
advertisement of prefixes, IP only

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Troubleshooting VxLAN EVPN
NVE Interface
Leaf1# show nve interface
Interface: nve1, State: Up, encapsulation: VXLAN
VPC Capability: VPC-VIP-Only [not-notified]
Local Router MAC: f40f.1b6f.926f
Host Learning Mode: Control-Plane
Source-Interface: loopback0 (primary: 192.168.1.1, secondary: 0.0.0.0)

Leaf1# show interface nve1

nve1 is up If NVE Interface status is down,
admin state is up, Hardware: NVE
ensure that a no shut is performed
MTU 9216 bytes
Encapsulation VXLAN on the interface.
Auto-mdix is turned off
RX
ucast: 40 pkts, 5400 bytes - mcast: 1 pkts, 118 bytes
TX
ucast: 54 pkts, 6256 bytes - mcast: 9 pkts, 1026 bytes

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Troubleshooting VxLAN EVPN
Local MAC Routes Learning
L2FM
Mac Learnt on Vlan 100

Leaf1#show mac address-table vlan 100

(T) - True, (F) - False
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+-------------
* 100 523d.e706.ae1b dynamic 0 F F Eth1/15

Leaf1# sh sys inter l2fm event-hist deb | in 523d.e706.ae1b

[104] l2fm_l2rib_add_delete_local_mac_routes(1095): To L2RIB: topo-id: 100,
macaddr: 523d.e706.ae1b, nhifindx: 0x1a001600 peer_addr 0x1a001600
[104] l2fm_macdb_insert(6360): slot 0 fe 0 mac 523d.e706.ae1b vlan 100 flags
0x400107 hints 0 E8 NL lc : if_index 0x1a001600 old_if_index 0

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Troubleshooting VxLAN EVPN
L2FM installs the MAC in the L2RIB Leaf1#show l2route evpn mac evi 100
Mac Address Prod Next Hop (s)
L2FM L2RIB -------------- ------ ---------------
523d.e706.ae1b Local Eth1/15

Leaf1# show system internal l2rib event-history mac | in 523d.e706.ae1b

[06/01/16 22:31:55.201 UTC 5 9954] Received MAC ROUTE msg: addr: (100, 523d.e706.ae1b) vni: 0
admin_dist: 0 seq_num: 0 rt_flags: L soo: 0 dg_count: 0 res: 0 esi: (F) nh_count: 1
[06/01/16 22:31:55.202 UTC 7 9954] (100,8c60.4f93.5ffc):Mobility check for new rte from prod:
3
[06/01/16 22:31:55.202 UTC 8 9954] (100,523d.e706.ae1b):Current non-del-pending route
local:no, remote:no, linked mac-ip count:1
[06/01/16 22:31:55.202 UTC 9 9954] (523d.e706.ae1b,3):MAC route created with seq num:0,
flags:L (), soo:0, peerid:0
[06/01/16 22:31:55.205 UTC a 9954] (100,523d.e706.ae1b,3):Encoding MAC best route (ADD,
client id 5)
[06/01/16 22:31:55.207 UTC 3 9954] (100,523d.e706.ae1b):Bound MAC-IP(100.1.1.1) to MAC, Total
MAC-IP linked: 1

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Troubleshooting VxLAN EVPN
L2 VNI, MAC
L2FM L2RIB BGP L2VPN

Leaf1#show bgp l2vpn evpn vni-id 10000

Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 10000:1 (L2VNI 10000)
*>l[2]:[0]:[0]:[48]:[523d.e706.ae1b]:[0]:[0.0.0.0]/216
192.168.1.1 100 32768 i

Leaf1# show bgp internal event-history events | in 523d.e706.ae1b

2016 Jun 1 22:31:55.205989 bgp 100 [16855]: [16888]: (default) RIB: [L2VPN EVPN
] add prefix 10000:1:[2]:[0]:[0]:[48]:[523d.e706.ae1b]:[0]:[0.0.0.0] (flags 0x1)
: OK, total 2
2016 Jun 1 22:31:55.205655 bgp 100 [16855]: [16888]: EVT: Received from L2RIB MAC
route: Add topo 10000 mac 523d.e706.ae1b soo 0 seq 0
. . .

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Troubleshooting VxLAN EVPN
Local MAC Address in BGP L2VPN
L2FM L2RIB BGP L2VPN

Leaf1# show bgp l2vpn evpn 523d.e706.ae1b

BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 192.168.1.1:32867 (L2VNI 10000)
BGP routing table entry for
[2]:[0]:[0]:[48]:[523d.e706.ae1b]:[0]:[0.0.0.0]/216, version 318
Paths: (1 available, best #1)
Flags: (0x000102) (high32 00000000) on xmit-list, is not in l2rib/evpn
Advertised path-id 1
Path type: local, path is valid, is best path
AS-Path: NONE, path locally originated
192.168.1.1 (metric 0) from 0.0.0.0 (192.168.1.1)
Origin IGP, MED not set, localpref 100, weight 32768
Received label 10000
Extcommunity: RT:65000:10000 ENCAP:8
Path-id 1 advertised to peers:
10.10.10.10 20.20.20.20

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
 Troubleshooting VxLAN EVPN
Remote L2 MAC Route Installation via BGP EVPN
Leaf2# show bgp l2vpn evpn 523d.e706.ae1b
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 192.168.1.1:32867
BGP routing table entry for [2]:[0]:[0]:[48]:[523d.e706.ae1b]:[0]:[0.0.0.0]/216,
Paths: (2 available, best #1)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW

Advertised path-id 1
Path type: external, path is valid, is best path
BGP L2VPN Imported to 1 destination(s)
AS-Path: 65001 65000 , path sourced external to AS
192.168.1.1 (metric 0) from 20.20.20.20 (192.168.20.20)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10000
Extcommunity: RT:65000:10000 ENCAP:8

Path type: external, path is valid, not best reason: newer EBGP path
AS-Path: 65001 65000 , path sourced external to AS
192.168.1.1 (metric 0) from 10.10.10.10 (192.168.10.10)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10000
Extcommunity: RT:65000:10000 ENCAP:8

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
 EVPN BGP Route Type 2 Fields
 Ethernet Tag ID, MAC Address Length, MAC Address, IP Address Length, and IP Address fields are
considered to be part of the prefix in the NLRI.
 Ethernet Segment Identifier, MPLS Label1, and MPLS Label2 are treated as route attributes, not part
of the "route". Both the IP and MAC address lengths are in bits.

Leaf1#show bgp l2vpn evpn 523d.e706.ae1b Route Distinguisher – 8 byte

BGP routing table information for VRF default, address family L2VPN
EVPN Ethernet Segment ID – 10 byte
Route Distinguisher: 10000:1 (L2VNI 10000)
BGP routing table entry for Ethernet Tag ID – 4 byte
[2]:[0]:[0]:[48]:[8c60.4f93.5ffc]:[0]:[0.0.0.0]/216, version 8
Paths: (1 available, best #1) MAC Address Length – 1 byte
Flags: (0x00010a) on xmit-list, is not in l2rib/evpn
MAC Address – 6 byte
Advertised path-id 1
Path type: local, path is valid, is best path, no labeled nexthop IP Address Length – 1 byte
AS-Path: NONE, path locally originated
192.168.1.1 (metric 0) from 0.0.0.0 (192.168.1.1) IP Address – 0, 4, 16 byte
Origin IGP, MED not set, localpref 100, weight 32768
Received label 10000 MPLS Label 1 – 3 byte, L2VNI
Extcommunity: RT:10000:1
MPLS Label 2 – 3 byte L3VNI

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Troubleshooting VxLAN EVPN
Remote L2 MAC Route Installation with BGP EPVN (Flow)
2a. Peer,
VNI
Notification
BGP L2VPN
2b. Add (VNI, MAC->
VxLAN Remote VTEP IP)
Mgr
VxLAN PD L2RIB
Lib.
5. Add (VNI,
3. Program data plane with
MAC -> Peer
unicast encap/decap for
ID)
VNI, Allocate Peer ID

L2FM
UFDM

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Troubleshooting VxLAN EVPN
BGP to L2RIB

Leaf2# show bgp internal event-history events | in 523d.e706.ae1b

2016 Jun 2 02:53:14.844179 bgp 100 [9878]: [9890]: (default) IMP: bgp_tbl_ctx_import:
1812: [L2VPN EVPN] Importing
10000:1:[2]:[0]:[0]:[48]:[523d.e706.ae1b]:[0]:[0.0.0.0]/112 to RD 10000:1
2016 Jun 2 02:53:14.844167 bgp 100 [9878]: [9890]: (default) IMP: bgp_vrf_import:
2740: vrf default 10000:1:[2]:[0]:[0]:[48]:[523d.e706.ae1b]:[0]:[0.0.0.0]/112 result 1
2016 Jun 2 02:53:14.844130 bgp 100 [9878]: [9890]: (default) RIB: [L2VPN EVPN]: Send
to L2RIB 10000:1:[2]:[0]:[0]:[48]:[523d.e706.ae1b]:[0]:[0.0.0.0]/112 via 192.168.1.1
Add 1 EVPN MAC routes succeeded

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
Troubleshooting VxLAN EVPN
Remote L2 MAC Route Installation via BGP EVPN
Leaf2# show nve internal bgp rnh database VxLAN Mgr
--------------------------------------------
Showing BGP RNH Database, size : 2 vni 0

VNI Peer-IP Peer-MAC Tunnel-ID Encap (A/S)

10000 192.168.1.1 0000.0000.0000 0x0 vxlan (1/0)
50000 192.168.1.1 f40f.1b6f.926f 0xc0a80101 vxlan (1/0)

BGP L2VPN

Leaf2# show l2route evpn mac evi 100

Mac Address Prod Next Hop (s) L2RIB
-------------- ------ ---------------
523d.e706.ae1b BGP 192.168.1.1

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
VxLAN Manager
Leaf2# show forwarding nve l3 peers
NVE cleanup transaction-id 0
tunnel_id Peer_id Peer_address Interface rmac origin state del count
--------------------------------------------------------------------------------------
0xc0a80101 1 192.168.1.1 nve1 f40f.1b6f.926f NVE merge-done no 1

Leaf2# show nve peers detail

Details of nve Peers:
----------------------------------------
Peer-Ip: 192.168.1.1
NVE Interface : nve1
VxLAN Mgr Peer State : Up
Peer Uptime : 01:27:30
Hardware
Programs data plane Router-Mac : f40f.1b6f.926f Programmed
with unicast Peer First VNI : 20000
encap/decap for VNI,
Allocate Peer ID Time since Create : 01:27:30
Configured VNIs : 10000,20000
Provision State : add-complete
UFDM
Route-Update : Yes
Peer Flags : RmacL2Rib, TunnelPD, DisableLearn
Learnt CP VNIs : 10000,20000,50000
Peer-ifindex-resp : Yes
#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Troubleshooting VxLAN EVPN
L2FM Verification

Leaf2# show system internal l2fm debugs | in 523d.e706.ae1b

[104] l2fm_macdb_insert(6327): slot 32 fe 0 mac 8c60.4f1b.e43c vlan 100 flags 0x7
hints 0 E8 NL lc : if_index 0x49080001 old_if_index 0
[104] l2fm_l2rib_mac_update(21832): Add L2RIB remote mac 523d.e706.ae1b
[104] l2fm_process_l2rib_remote_route_update(405): Type: 2 Len: 152 Seq: 0, del: 0
(Prod: 5) Flags: Ctrl=3 Rt=0, mac 8c60.4f1b.e43c topo_id 100

Leaf2# show mac address-table vlan 100

VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+-----------------
-
* 100 8c60.4f19.51fc dynamic 0 F F Eth1/13
* 100 523d.e706.ae1b dynamic 0 F F nve1(192.168.1.1)

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Troubleshooting VxLAN EVPN
Host IP and Host MAC local route
4. Route add: VRF test, Host-IP, L3VNI

HMM L2RIB
3. Adj. Notification: Host-IP, Host- 5. Route add: VRF test, Host-IP, L3 Table
MAC, SVI interface ID. Per prefix: L2 Table ID, Host-MAC

AM BGP
2. Create Adjacency: Host-IP, L3 + L2 NLRI
Host-MAC, Server facing SVI Prefix = Host-IP
interface MAC = Host-MAC
1b. Add an adjacency in AM, if Label1: L3-VNI
ARP the VLAN is enabled for L3 Label2: L2-VNI
processing VRF RT
1a. ARP Request/Reply BD RT
NH = VTEP IP
RMAC
ENCAP_VXLAN

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Troubleshoot VxLAN EVPN
ARP from Host and ARP -> AM
Leaf1# show ip arp vrf EVPN-TENANT
IP ARP Table for context EVPN-TENANT
Total number of entries: 1
Address Age MAC Address Interface ARP
100.1.1.1 00:10:47 523d.e706.ae1b Vlan100

Leaf1# show forwarding vrf EVPN-TENANT adjacency

IPv4 adjacency information

next-hop rewrite info interface AM

-------------- --------------- -------------
100.1.1.1 523d.e706.ae1b Vlan100

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
Troubleshoot VxLAN EVPN AM
AM -> HMM -> L2RIB
Leaf1# show ip route vrf EVPN-TENANT
100.1.1.1/32, ubest/mbest: 1/0, attached
*via 100.1.1.1, Vlan100, [190/0], 02:41:57, hmm
100.1.1.254/32, ubest/mbest: 1/0, attached HMM
*via 100.1.1.254, Vlan100, [0/0], 02:59:46, local

Leaf1# show l2route evpn mac-ip evi 100

Mac Address Prod Host IP Next Hop (s)
-------------- ---- ------------------- --------------
- L2RIB
523d.e706.ae1b HMM 100.1.1.1 N/A

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Troubleshooting VxLAN EVPN
L2RIB -> BGP
Leaf1# show bgp l2vpn evpn 100.1.1.1
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 10000:1 (L2VNI 10000)
BGP routing table entry for [2]:[0]:[0]:[48]:[523d.e706.ae1b]:[32]:[100.1.1.1]/272,
version 6
Paths: (1 available, best #1)
Flags: (0x00010a) on xmit-list, is not in l2rib/evpn

Advertised path-id 1
Path type: local, path is valid, is best path, no labeled nexthop
AS-Path: NONE, path locally originated
192.168.1.1 (metric 0) from 0.0.0.0 (192.168.1.1)
Origin IGP, MED not set, localpref 100, weight 32768
Received label 10000 50000
Extcommunity: RT:10000:1 RT:50000:1

Path-id 1 advertised to peers:

10.10.10.10

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
 L2 + L3 Remote Route Installation
L3 + L2 NLRI BGP-EVPN
Prefix = Host-IP
MAC = Host-MAC EVPN Table
Label1: L3-VNI VTEP_IP, RCAM,
Label2: L2-VNI L3VNI, L2VNI,
VRF RT VRF Import Tunnel Info
VxLAN
BD RT BD Import
NH = VTEP IP RNH DB VxLAN
RMAC
ENCAP_VXLAN Tunnel_ID Manager
VRF BD
Peer-Id RMAC,
notification VNI->Peer
VRF, Host-IP, ID
L3VNI, VTEP-IP
BD L2RIB
URIB

RMAC, VNI->Peer ID
VRF Program data

(L2-BD, MAC-
H1)-> Peer-ID
plane with unicast
VRF, Host-IP, encap/decap for
L3VNI, VTEP-IP L3VNI, RMAC,
Allocate Peer-ID
UFDM

FIB L2FM
#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Troubleshooting VxLAN EVPN
L2RIB and URIB Information

Leaf2# show l2route evpn mac-ip evi 100

Mac Address Prod Host IP Next Hop (s)
-------------- ---- --------------------------- ---------------
523d.e706.ae1b BGP 100.1.1.1 192.168.1.1

Leaf2# show ip route vrf EVPN-TENANT 100.1.1.1

IP Route Table for VRF "EVPN-TENANT"
100.1.1.1/32, ubest/mbest: 1/0
*via 192.168.1.1%default, [200/0], 04:00:28, bgp-100, internal,
tag 100 (evpn) segid: 50000 tunnelid: 0xc0a80101 encap: VXLAN

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
Troubleshooting VxLAN EVPN
Remote Host Prefix - EVPN
Leaf2# show bgp l2vpn evpn 100.1.1.1
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 10000:1 (L2VNI 10000)
BGP routing table entry for [2]:[0]:[0]:[48]:[523d.e706.ae1b]:[32]:[100.1.1.1]/272,
version 5
Paths: (1 available, best #1)
Flags: (0x00021a) on xmit-list, is in l2rib/evpn, is not in HW, , is locked
Advertised path-id 1
Path type: internal, path is valid, imported same remote RD, is best path, no labeled
nexthop
AS-Path: NONE, path sourced internal to AS
192.168.1.1 (metric 5) from 10.10.10.10 (192.168.10.10)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10000 50000
Extcommunity: RT:10000:1 RT:50000:1 ENCAP:8 Router MAC:f40f.1b6f.926f
Originator: 192.168.1.1 Cluster list: 10.10.10.10

Path-id 1 not advertised to any peer

. . .

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
 Troubleshooting VxLAN EVPN
Remote Host Prefix – contd…
...
Route Distinguisher: 50000:1 (L3VNI 20000)
BGP routing table entry for [2]:[0]:[0]:[48]:[523d.e706.ae1b]:[32]:[100.1.1.1]/272,
version 6
Paths: (1 available, best #1)
Flags: (0x00021a) on xmit-list, is in l2rib/evpn, is not in HW,

Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Imported from 10000:1:[2]:[0]:[0]:[48]:[8c60.4f1b.e43c]:[32]:[100.1.1.1]/144
(VNI 10000)
AS-Path: NONE, path sourced internal to AS
192.168.1.1 (metric 5) from 192.168.10.10 (192.168.10.10)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10000 50000
Extcommunity: RT:10000:1 RT:50000:1 ENCAP:8 Router MAC:f40f.1b6f.926f
Originator: 192.168.1.1 Cluster list: 10.10.10.10

Path-id 1 not advertised to any peer

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
 Troubleshooting VxLAN EVPN
NVE Internal Platform Detail
Leaf1# show nve internal platform interface nve 1 detail
Printing Interface ifindex 0x49000001 detail
|======|=========================|===============|===============|=====|=====|
|Intf |State |PriIP |SecIP |Vnis |Peers|
|======|=========================|===============|===============|=====|=====|
|nve1 |UP |192.168.1.1 |0.0.0.0 |2 |1 |
|======|=========================|===============|===============|=====|=====|

SW_BD/VNIs of interface nve1:

================================================
|======|======|=========================|======|====|======|========
|Sw BD |Vni |State |Intf |Type|Vrf-ID|Notified
|======|======|=========================|======|====|======|========
|100 |10000 |UP |nve1 |CP |0 |Yes
|500 |50000 |UP |nve1 |CP |3 |Yes
|======|======|=========================|======|====|======|========
Peers of interface nve1:
============================================
Peer_ip: 192.168.2.2
Peer-ID : 1
State : UP
Learning : Disabled
TunnelID : 0xc0a80202
MAC : 88f0.312a.f2c1
Table-ID : 0x1
Encap : 0x1

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
Troubleshooting VxLAN EVPN
ARP Suppression Cache

Leaf1# show ip arp suppression-cache local

Ip Address Age Mac Address Vlan Physical-ifindex Flags

100.1.1.1 00:15:48 8c60.4f1b.e43c 100 Ethernet1/12 L

Leaf1# show ip arp suppression-cache remote

Ip Address Age Mac Address Vlan Physical-ifindex Flags

100.1.1.2 00:05:19 8c60.4f19.51fc 100 (null) R

Leaf1(config)# hardware access-list tcam region arp-ether 256 double-wide

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
 Troubleshooting VxLAN EVPN
NVE Internal Information for Leaf Nodes with VPC Peers
Leaf3# show nve internal platform interface nve 1 detail
Printing Interface ifindex 0x49000001 detail
|======|=========================|===============|===============|=====|=====|
|Intf |State |PriIP |SecIP |Vnis |Peers|
|======|=========================|===============|===============|=====|=====|
|nve1 |UP |192.168.3.3 |192.168.100.100|2 |2 |
|======|=========================|===============|===============|=====|=====|
SW_BD/VNIs of interface nve1:
================================================
|======|======|=========================|======|====|======|========
|Sw BD |Vni |State |Intf |Type|Vrf-ID|Notified
|======|======|=========================|======|====|======|========
|100 |10000 |UP |nve1 |CP |0 |Yes
|200 |20000 |UP |nve1 |CP |3 |Yes
|======|======|=========================|======|====|======|========

Peers of interface nve1:

============================================
Peer_ip: 192.168.1.1
Peer-ID : 2
State : UP
Learning : Disabled
TunnelID : 0xc0a80101
MAC : f40f.1b6f.926f
Table-ID : 0x1
Encap : 0x1

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
 Troubleshooting VxLAN EVPN
NVE Internal Information for Leaf Nodes on Non-VPC Peers
Leaf1# show nve internal platform interface nve1 detail
Printing Interface ifindex 0x49000001 detail
|======|=========================|===============|===============|=====|=====|
|Intf |State |PriIP |SecIP |Vnis |Peers|
|======|=========================|===============|===============|=====|=====|
|nve1 |UP |192.168.1.1 |0.0.0.0 |2 |2 |
|======|=========================|===============|===============|=====|=====|
SW_BD/VNIs of interface nve1:
================================================
|======|======|=========================|======|====|======|========
|Sw BD |Vni |State |Intf |Type|Vrf-ID|Notified
|======|======|=========================|======|====|======|========
|100 |10000 |UP |nve1 |CP |0 |Yes
|200 |20000 |UP |nve1 |CP |3 |Yes
|======|======|=========================|======|====|======|========

Peers of interface nve1:

============================================
Peer_ip: 192.168.100.100
Peer-ID : 2
State : UP
Learning : Disabled
TunnelID : 0xc0a86464
MAC : 88f0.312b.9e4d
Table-ID : 0x1
Encap : 0x1

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
Troubleshooting VXLAN EVPN
Uplinks with SVI / Sub-Interfaces
• For SVI based uplinks, define the infra VLANs
• system nve infra-vlan <svi-vlan>
• Sub-Interfaces
• Not supported on ALE links (40G ports) (Documented)
• Check CCO documentation

• L3 Port-channel supported – Check CCO documentation

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Troubleshooting BUM
Traffic
Troubleshooting BUM Traffic
BUM Traffic over Multicast Core
• BUM Traffic - Broadcast traffic (ARP and other broadcast packets),
Multicast traffic from hosts, etc.
• Check the multicast group associated with the L2 VNI
• Get the Source VTEP IP and Router MAC
• Check if ARP Suppression is enabled or not
• Ask the right Questions:
• Check if the VTEP is a VPC VTEP or Standalone VTEP?
• Is the issue seen for IPv4 hosts or IPv6 hosts?
• Know the trigger and understand if the issue is continuously reproducible
or not?

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
Troubleshooting BUM Traffic
Step 1 – Check if the BUM Traffic is making it to the source VTEP
LEAF45# ethanalyzer local interface inband capture-fil "arp" limit-captured-frames 0

Capturing on inband
2018-05-21 14:52:36.289960 20:20:00:00:00:aa -> 54:7f:ee:07:e1:41 ARP 10.150.1.254 is
at 20:20:00:00:00:aa
2018-05-21 14:52:36.295037 54:7f:ee:07:e1:41 -> ff:ff:ff:ff:ff:ff ARP Who has
10.150.1.35? Tell 10.150.1.36
2018-05-21 14:52:36.295425 2c:54:2d:f6:0f:bc -> 54:7f:ee:07:e1:41 ARP 10.150.1.35 is at
2c:54:2d:f6:0f:bc
2018-05-21 14:52:38.127594 54:7f:ee:07:e1:41 -> ff:ff:ff:ff:ff:ff ARP Who has
10.150.1.38? Tell 10.150.1.36

LEAF45# debug logfile arp

LEAF45# debug ip arp packet

14:52:38.127774 arp: (context 3) Receiving packet from Vlan1501, logical interface

Vlan1501 physical interface port-channel36, (prty 6) Hrd type 1 Prot type 800 Hrd len 6
Prot len 4 OP 1, Pkt size 46
14:52:38.127811 arp: Src 547f.ee07.e141/10.150.1.36 Dst ffff.ffff.ffff/10.150.1.38

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Troubleshooting BUM Traffic
Step 2 – Check the Mroute Entry – Src VTEP
LEAF45# show ip mroute 231.1.150.1 10.0.0.204 detail
IP Multicast Routing Table for VRF "default"
Total number of routes: 8
Total number of (*,G) routes: 2
Total number of (S,G) routes: 5
Total number of (*,G-prefix) routes: 1

(10.0.0.204/32, 231.1.150.1/32), uptime: 00:14:01, nve(0) mrib(0) ip(0) pim(1)

RPF-Source: 10.0.0.204 [0/0]
Data Created: No
Received Register stop
VXLAN Flags
VXLAN Encap
VPC Flags
RPF-Source Forwarder
Stats: 51/2601 [Packets/Bytes], 27.200 bps
Stats: Active Flow
Incoming interface: loopback1, RPF nbr: 10.0.0.204
Outgoing interface list: (count: 1) (bridge-only: 0)
Ethernet1/50, uptime: 00:09:52, pim

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
Troubleshooting BUM Traffic
Step 3.1 – Check the Mroute Entry – Dest VTEP
LEAF43# show ip mroute 231.1.150.1 10.0.0.204 detail
IP Multicast Routing Table for VRF "default"

Total number of routes: 8

Total number of (*,G) routes: 2
Total number of (S,G) routes: 5
Total number of (*,G-prefix) routes: 1

(10.0.0.204/32, 231.1.150.1/32), uptime: 00:03:19, ip(0) mrib(1) pim(0)

RPF-Source: 10.0.0.204 [3/110]
Data Created: Yes
VXLAN Flags
VXLAN Decap
Stats: 1/51 [Packets/Bytes], 0.000 bps
Stats: Inactive Flow
Incoming interface: Ethernet1/50, RPF nbr: 10.0.0.42
Outgoing interface list: (count: 1) (bridge-only: 0)
nve1, uptime: 00:03:19, mrib

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
Troubleshooting BUM Traffic
Step 3.2 – Check the Mroute Entry – Dest VTEP
LEAF44# show ip mroute 231.1.150.1 10.0.0.204 detail
IP Multicast Routing Table for VRF "default"
Total number of routes: 8
Total number of (*,G) routes: 2
Total number of (S,G) routes: 5
Total number of (*,G-prefix) routes: 1

(10.0.0.204/32, 231.1.150.1/32), uptime: 00:03:57, ip(0) mrib(1) pim(0)

RPF-Source: 10.0.0.204 [3/110]
Data Created: Yes
VXLAN Flags
VXLAN Decap
VPC Flags
RPF-Source Forwarder
Stats: 1/51 [Packets/Bytes], 0.000 bps
Stats: Inactive Flow
Incoming interface: Ethernet1/50, RPF nbr: 10.0.0.42
Outgoing interface list: (count: 1) (bridge-only: 0)
nve1, uptime: 00:03:57, mrib

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Troubleshooting BUM Traffic
Step 4 – Capturing BUM Traffic in Core
LEAF45(config)# monitor session 1
LEAF45(config-monitor)# source interface ethernet 1/50
LEAF45(config-monitor)# destination interface sup-eth 0
LEAF45(config-monitor)# no shut
LEAF45(config-monitor)# end

LEAF45# ethanalyzer local interface inband capture-filter "host 231.1.150.1" limit-

captured-frames 0

Capturing on inband
2018-05-21 16:21:01.985236 10.0.0.204 -> 231.1.150.1 UDP Source port: 41316
Destination port: 4789

Use the detail option with ethanalyzer to see the whole packet

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
Demo
Troubleshooting Tenant
Routed Multicast (TRM)
Tenant Routed Multicast (TRM)
Overview
• A BGP based solution for allowing multicast routing and snooping over
VXLAN EVPN fabric
• Sources and Receivers are connected to the VTEPs

• Multicast Source and Receiver info is propagated using BGP

• No PIM/IGMP packets sent to VXLAN fabric from any TRM VTEP

• Modes:
• L3 Mode
• L2/L3 Mixed Mode

• Both modes are supported only on N9k - EX or FX platforms

• Supported only with Multicast based core. IR not supported
#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Tenant Routed Multicast (TRM)
Modes and RP
• L3 Mode
• RP configured on all VTEPs – (ip multicast overlay-spt-only command required)
• RP on selected VPC VTEP – not supported
• Internal RP – Supported from 7.0(3)I7(1)
• External RP – Supported from 7.0(3)I7(4) (upcoming release)
• L2/L3 Mode
• RP on all Distributed-DR – Supported on Tahoe
• RP on Core – not supported
• Internal RP - Supported from 7.0(3)I7(1)
• External RP – Not supported

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
Tenant Routed Multicast (TRM)
EVPN – L3 Anycast

S S

V V V V

S R R R

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
Tenant Routed Multicast (TRM)
TRM Verification
Leaf1# show fabric multicast vrf all
Fabric Multicast Enabled VRFs
VRF Name VRF Vprime VN-Seg
ID If ID
default 1 Null 0
TRM 4 Vlan1500 150000

Leaf1# show fabric multicast globals

Pruning: segment-based
Switch role: leaf
Fabric Control Seg: Null
Peer Fabric Control Address: 0.0.0.0
Advertising vPC RPF routes: Disabled
Created VNI List: -
Fwd Encap: Fabric
Overlay Distributed-DR: FALSE
Overlay spt-only: TRUE

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
Tenant Routed Multicast (TRM)
Show bgp internal mvpn detail
Leaf1# show bgp internal mvpn detail
*************************************************
NGMVPN feature/server/role: VxLAN/1/VxLAN
NGMVPN registered: Yes (Jun 9 00:56:59.297696/never)
NGMVPN TRM mode: L3 (0x000002)
NGMVPN down: in-prg/up-defer: 0/0
NGMVPN register/failures: 1/0
NGMVPN deregister/failures: 0/0
NGMVPN Convergence sent: 0
NGMVPN local-req sent/skipped: 7/3
NGMVPN local-req sent: 4 (L2VNI)/ 2 (L3VNI)/ 1 (All VNIs)
NGMVPN remote-req rcvd: 0 (L2VNI)/ 0 (L3VNI)/ 1 (All VNIs)
NGMVPN del remote: 0 (L2VNI)/ 0 (L3VNI)/ 0 (All VNIs)
NGMVPN msgs sent/acks rcvd: 9/9
NGMVPN msgs rcvd/acks sent: 24/14
NGMVPN msg err/ack err/drops: 0/0/0
Last xid sent to NGMVPN: 9
Last xid ack by NGMVPN: 9
. . . .

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Tenant Routed Multicast (TRM)
Show bgp internal mvpn detail (contd…)
++++++++++++++++++++++++++++++++++++++++++
BGP MVPN RD Information for 192.168.1.1:4 (0xd625952c)
VNI ID : 150000
VRF : TRM
Global NGMVPN mode : L3 (1 L3 VRFs)
VRF L3 Mode : Yes Jun 9 00:57:00.203673
Enabled : Yes
Delete Pending : No
Cleanup Pending : No
Import Pending : No
Import In Progress : No
Created : Jun 9 00:56:55.570427
Enabled At : Jun 9 00:56:55.570471
. . .
Is Auto RT : No
Config VRF Import RT : 1
Import RT cfg list: 192.168.1.1:1500
Active VRF Import RT : 1
Active VRF Import RT list : 192.168.1.1:1500
VRF Import RT chg/chg-pending : 0/0

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Tenant Routed Multicast (TRM)
Join from Receiver
Leaf3# show ip mroute 239.0.0.1 detail vrf TRM
IP Multicast Routing Table for VRF "TRM"

Total number of routes: 3

Total number of (*,G) routes: 1
Total number of (S,G) routes: 1
Total number of (*,G-prefix) routes: 1

(*, 239.0.0.1/32), uptime: 00:00:55, igmp(1) ip(0) pim(0)

RPF-Source: 99.99.99.99 [0/0]
RD-RT ext comm Route-Import:
Data Created: No
VPC Flags
RPF-Source Forwarder
Stats: 0/0 [Packets/Bytes], 0.000 bps
Stats: Inactive Flow
Incoming interface: loopback1000, RPF nbr: 99.99.99.99
Outgoing interface list: (count: 1) (bridge-only: 0)
Vlan1000, uptime: 00:00:55, igmp (vpc-svi)

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Tenant Routed Multicast (TRM)
FHR VTEP sends SA-AD (Type-5 Route) using BGP to other VTEPs
Leaf1# show bgp ipv4 mvpn sa-ad detail vrf TRM
BGP routing table information for VRF default, address family IPv4 MVPN
Route Distinguisher: 192.168.1.1:4 (L3VNI 150000)
BGP routing table entry for [5][10.0.0.1][239.0.0.1]/64, version 34
Paths: (1 available, best #1)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in mvpn

Advertised path-id 1
Path type: local, path is valid, is best path
AS-Path: NONE, path locally originated
0.0.0.0 (metric 0) from 0.0.0.0 (192.168.1.1)
Origin IGP, MED not set, localpref 100, weight 32768
Extcommunity: RT:65000:150000

Path-id 1 advertised to peers:

10.10.10.10 20.20.20.20

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Tenant Routed Multicast (TRM)
LHR / Remote VTEPs build (S, G)
Leaf3# show ip mroute 239.0.0.1 10.0.0.1 detail vrf TRM
IP Multicast Routing Table for VRF "TRM"

Total number of routes: 3

Total number of (*,G) routes: 1
Total number of (S,G) routes: 1
Total number of (*,G-prefix) routes: 1

(10.0.0.1/32, 239.0.0.1/32), uptime: 00:01:55, ip(0) mrib(1) pim(0) ngmvpn(0)

RPF-Source: 10.0.0.1 [0/20]
RD-RT ext comm Route-Import: 0b c0 a8 01 01 05 dc 00 01 c0 a8 01 01 83 e7
Data Created: Yes
Fabric dont age route
VPC Flags
RPF-Source Forwarder
Stats: 1/84 [Packets/Bytes], 0.000 bps
Stats: Inactive Flow
Incoming interface: Vlan1500, RPF nbr: 192.168.1.1
Outgoing interface list: (count: 1) (bridge-only: 0)
Vlan1000, uptime: 00:01:55, mrib (vpc-svi)

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
Tenant Routed Multicast (TRM)
VTEP3/4 sends Type-7 back to FHR on L3VNI
Leaf3# show bgp ipv4 mvpn route-type 7 detail
BGP routing table information for VRF default, address family IPv4 MVPN
Route Distinguisher: 192.168.1.1:33767 (Local VNI: 150000)
BGP routing table entry for [7][10.0.0.1][239.0.0.1][65000]/96, version 43
Paths: (1 available, best #1)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in mvpn

Advertised path-id 1
Path type: local, path is valid, is best path
AS-Path: NONE, path locally originated
0.0.0.0 (metric 0) from 0.0.0.0 (192.168.3.3)
Origin IGP, MED not set, localpref 100, weight 32768
Extcommunity: RT:192.168.1.1:1500

Path-id 1 advertised to peers:

10.10.10.10 20.20.20.20

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
Tenant Routed Multicast (TRM)
VTEP1 Receives the Type-7 route
Leaf1# show bgp ipv4 mvpn route-type 7 detail
BGP routing table information for VRF default, address family IPv4 MVPN
Route Distinguisher: 192.168.1.1:4 (L3VNI 150000)
BGP routing table entry for [7][10.0.0.1][239.0.0.1][65000]/96, version 36
Paths: (1 available, best #1)
Flags: (0x00001a) (high32 00000000) on xmit-list, is in mvpn, is not in HW

Advertised path-id 1
Path type: external, path is valid, is best path, in rib
Imported from 192.168.1.1:33767:[7][10.0.0.1][239.0.0.1][65000]/96
AS-Path: 65001 65000 , path sourced external to AS
192.168.100.100 (metric 0) from 10.10.10.10 (192.168.10.10)
Origin IGP, MED not set, localpref 100, weight 0
Extcommunity: RT:192.168.1.1:1500

Path-id 1 advertised to peers:

20.20.20.20

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
Tenant Routed Multicast (TRM)
VTEP1 adds L3VNI in the OIF List
Leaf1# show ip mroute 239.0.0.1 10.0.0.1 detail vrf TRM
IP Multicast Routing Table for VRF "TRM"

Total number of routes: 3

Total number of (*,G) routes: 1
Total number of (S,G) routes: 1
Total number of (*,G-prefix) routes: 1

(10.0.0.1/32, 239.0.0.1/32), uptime: 00:12:03, ip(0) mrib(1) pim(0) ngmvpn(1)

RPF-Source: 10.0.0.1 [0/0]
RD-RT ext comm Route-Import:
Data Created: Yes
Received Register stop
Fabric dont age route
Stats: 695/58380 [Packets/Bytes], 672.000 bps
Stats: Active Flow
Incoming interface: Vlan1000, RPF nbr: 10.0.0.1, internal
Outgoing interface list: (count: 2) (Fabric OIF) (bridge-only: 0)
Vlan1500, uptime: 00:12:02, ngmvpn
Vlan1000, uptime: 00:12:03, mrib, (RPF)

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
Tenant Routed Multicast (TRM)
VTEP1 adds L3VNI in the OIF List
Leaf3# show ip mroute 239.0.0.1 10.0.0.1 detail vrf TRM
IP Multicast Routing Table for VRF "TRM"

Total number of routes: 3

Total number of (*,G) routes: 1
Total number of (S,G) routes: 1
Total number of (*,G-prefix) routes: 1

(10.0.0.1/32, 239.0.0.1/32), uptime: 00:24:35, ip(0) mrib(1) pim(0) ngmvpn(0)

RPF-Source: 10.0.0.1 [0/20]
RD-RT ext comm Route-Import: 0b c0 a8 01 01 05 dc 00 01 c0 a8 01 01 83 e7
Data Created: Yes
Fabric dont age route
VPC Flags
RPF-Source Forwarder
Stats: 1/84 [Packets/Bytes], 0.000 bps
Stats: Inactive Flow
Incoming interface: Vlan1500, RPF nbr: 192.168.1.1
Outgoing interface list: (count: 1) (bridge-only: 0)
Vlan1000, uptime: 00:24:35, mrib (vpc-svi)

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
Tenant Routed Multicast (TRM)
MFDM (Validate this info on both FHR and LHR)
Leaf1# show forwarding distribution multicast route vrf TRM group 239.0.0.1 source
10.0.0.1

(10.0.0.1/32, 239.0.0.1/32), RPF Interface: Vlan1000, flags:

Received Packets: 0 Bytes: 0
Number of Outgoing Interfaces: 2
Outgoing Interface List Index: 29
Vlan1000
( Mem L2 Ports: nve1 )
l2_oiflist_index: 1
Vlan1500 (Vxlan Encap)
( Mem L2 Ports: nve1 )
l2_oiflist_index: 1

MFIB Database
Show forwarding multicast outgoing-interface-list l3 <index>

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
Troubleshooting Tools
 Troubleshooting Tools
VXLAN OAM
• VXLAN OAM feature introduced in 7.0(3)I5(2) – NGOAM
• Need a feature for Path verification and Path tracking with Telemetry
data
• Similar to Fabric Path OAM Enable NGOAM
Feature
• Provides 3 features
feature ngoam
• VxLAN Ping Create Profile
ngoam profile 1
• VxLAN Traceroute oam-channel 2
! Install ACL
• VxLAN PathTrace
ngoam install acl

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
VXLAN OAM
Usability
• Helps diagnose underlay / overlay reachability of VMs
• Covers exact path as Data Packet
• Path verification for all ECMP paths in Overlay
• Path tracking – Exact path host traffic takes in overlay and underlay
network
• Layer 2 – Traceroute / Ping to VM host from Leaf
• Layer 3 – Traceroute / Ping to Vm host from Leaf
• Flexible OAM channel supporting multiple drafts
• Tissa draft – nvo3
#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
VXLAN OAM
VXLAN OAM Message
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
. Circuit Header . (variable)
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ Transport Header + Technology dependent
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ VXLAN Header + 8 bytes
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
. OAM Message Channel . Variable
. .
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ Link Trailer + Variable
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
VXLAN OAM
VXLAN PING
• VxLAN Ping checks connectivity to the destination, where the destination
can be VM’s IP address or routed loopback addresses on the leaf switch
• Since there are multiple paths, only one path is followed based on the flow
parameters
• Ping for both VM / Host MAC and IP
• Default ping support – Ping based on just destination address and VNI
segment
• Allow users to specify flow parameters such as UDP port, destination and
source address
• This helps VxLAN ping follow the specific path the unicast ping will take to reach
the destination

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
VxLAN OAM
PING NVE MAC
Leaf1# ping nve mac 0050.56b3.bcef 200 port-channel 101 profile 1 vni 20000
verbose
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
'D' - Destination Unreachable, 'X' - unknown return code,
'm' - malformed request(parameter problem)
,'c' - Corrupted Data/Test, '#' - Duplicate response
Sender handle: 21
! sport 51932 size 39,Reply from 192.168.100.100,time = 5 ms
! sport 51932 size 39,Reply from 192.168.100.100,time = 4 ms
! sport 51932 size 39,Reply from 192.168.100.100,time = 4 ms
! sport 51932 size 39,Reply from 192.168.100.100,time = 4 ms
! sport 51932 size 39,Reply from 192.168.100.100,time = 4 ms

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
VxLAN OAM
VxLAN Traceroute
• VxLAN Traceroute – Used to trace the path a packet takes between source
and destination
• Only one path is traced based on the given flow parameters
• Trace will show uni-directional path the packet takes to the destination, but the
return path may be different
• Should be able to trace VTEPs, access switch and end-host. For the
default mode, the user should be able to trace the tunnel endpoint IP
address and the segment ID
• Actual path taken by a packet is dependent on all the L2/L3/L4 header
fields and network topology at the time the packet is sent
• Users can specify the flow parameters

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
 VxLAN OAM
VxLAN PathTrace
• Similar to Traceroute, but uses Nvo3 channel
• Carries additional ingress / egress and load information of the path
• Provides additional information if the device supports nvo3 channel else its
same as traceroute
• Actual path taken by a packet is dependent on all the L2/L3/L4
header fields and network topology at the time the packet is sent
•
Leaf1# Users can specify the flow parameters
pathtrace nve mac 0050.56b3.bcef 200 port-channel 101 vni 20000
<snip>
Path trace Request to peer ip 192.168.100.100 source ip 192.168.99.99
Sender handle: 35Hop Code ReplyIP IngressI/f EgressI/f State
====================================================
1 !Reply from 10.101.1.10, Eth2/1 Eth1/17 UP / UP
2 !Reply from 192.168.100.100, Eth1/17 Unknown UP / DOWN

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
Troubleshooting Tools
ELAM
• Embedded Logic Analyzer module (ELAM) – tool used to capture a
packet processed by a Cisco ASIC
• Depending on the N9k platform,
• ELAM on NS ASIC
• ELAM on TAHOE ASIC

• Useful in scenario’s where packet forwarding is impacted

• Can perform capture for raw packet from the host and even VxLAN
encapsulated packet towards the VxLAN Core
• Useful for Cisco TAC and Cisco Engineering for understanding the
cause of packet loss or impacted forwarding
#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
 ELAM on N9k Platform
Northstar (NS) ASIC
• NS datapath is divided into ingress and egress pipelines
• 2 ELAM’s present in each pipeline (Input & Output)
• Packets can be captured based on either input select lines or output select lines but not both

• Input Select Lines

• 3  Outerl2-outerl3-outerl4
• 4  Innerl2-innerl3-inner l4
• 5  Outerl2-innerl2
• 6  Outerl3-innerl3
• 7  Outerl4-innerl4

• Output Select Lines

• 0  Pktrw
• 5  Sideband
#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
ELAM on N9k Platform
NS ELAM Steps
• Attach module X

• Debug platform internal ns elam asic [0 | 1]

• Trigger [init | reset] [ingress | egress] in-select [3 –
7] out-select [0, 5]
• Ingress – For capturing packets from front panel to Fabric
• Egress – For capturing packets from Fabric to Front Panel
• Set outer [ipv4 | l2 | l4] . . . .
• Start
• Status {Can be Armed / Triggered}
• report
#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
ELAM on N9k Platform
Verify Port-Asic, Slice and Src-Id
N9K# show hardware internal tah interface e1/49
<snip>
IfIndex: 436232192
DstIndex: 5952
IfType: 26
Asic: 0
AsicPort: 56
SrcId: 48 <<
Slice: 1
PortOnSlice: 24

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
ELAM on N9k Platform
TAHOE ELAM Steps
• Attach module X

• Debug platform internal tah elam asic [0 | 1]

• Trigger [init | reset] asic [num] slice [num] lu-a2d [0 |
1] in-select [3-7] out-select [0-5] use-src-id [src-id]
• Lu-a2d 0 – used for reverse ELAM, where trigger is based on result
• Lu-a2d 1 – used for ELAM where trigger is based on packet attributes
• Set outer [ipv4 | l2 | l4] . . . .
• Start
• Status {Can be Armed / Triggered}
• Report [detail]
#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
 ELAM on N9k Platform
TAHOE ELAM
Spine
att mod 1
debug platform internal tah elam asic 0
trigger init asic 0 slice 0 lu-a2d 1 in-select 7 out-select 0 use-src-id 52
reset VxLAN Encapsulated Packet
set inner ipv4 src_ip 100.1.1.1 dst_ip 100.1.1.5
start
report

Leaf1
att mod 1
debug platform internal tah elam asic 0
trigger init asic 0 slice 0 lu 1 in-select 6 out-select 0
reset Host IP Packet
set outer ipv4 src_ip 100.1.1.1 dst_ip 100.1.1.5
start
report

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 91
 ELAM on N9k Platform
Partial Output
• Dot1Q Header
module-1(TAH-elam-insel6)# report detail | grep pr_lu_vec_l2v.qtag0
GBL_C++: [MSG] pr_lu_vec_l2v.qtag0_vld: 0x1 << dot1q yes? 0x1
GBL_C++: [MSG] pr_lu_vec_l2v.qtag0_cos: 0x0
GBL_C++: [MSG] pr_lu_vec_l2v.qtag0_de: 0x0
GBL_C++: [MSG] pr_lu_vec_l2v.qtag0_vlan: 0x64 << VL 100

• VLAN
module-1(TAH-elam-insel6)# report detail | grep -1 fpx_lookup_vec.lkup.macsakey.key.fid
GBL_C++: [MSG] fpx_lookup_vec.lkup.macsakey.key.vld: 0x1
GBL_C++: [MSG] fpx_lookup_vec.lkup.macsakey.key.fid_type: 0x0
GBL_C++: [MSG] fpx_lookup_vec.lkup.macsakey.key.fid_vld: 0x0
GBL_C++: [MSG] fpx_lookup_vec.lkup.macsakey.key.fid: 0x64 << dec 0xa = VL 10
GBL_C++: [MSG] fpx_lookup_vec.lkup.macsakey.key.mac: 0xFEC80E2715

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
 ELAM on N9k Platform
Partial Output
• Src & Dst IP
module-1(TAH-elam-insel6)# report detail | grep vec_l3v.ip.*a
GBL_C++: [MSG] pr_lu_vec_l3v.ip.da: 0x0000000000000000064010101 << 100.1.1.1
GBL_C++: [MSG] pr_lu_vec_l3v.ip.sa: 0x0000000000000000064010105 << 100.1.1.5

• Src MAC
module-1(TAH-elam-insel6)# report detail | grep -i fpx_lookup_vec.lkup.macsakey.key.mac
GBL_C++: [MSG] fpx_lookup_vec.lkup.macsakey.key.mac: 0xFEC80E2715 << 00fe.c80e.2715
GBL_C++: [MSG] fpx_lookup_vec.lkup.macsakey.key.mac: 0xFEC80E2715
GBL_C++: [MSG] fpx_lookup_vec.lkup.macsakey.key.mac: 0xFEC80E2715
GBL_C++: [MSG] fpx_lookup_vec.lkup.macsakey.key.mac: 0xFEC80E2715
GBL_C++: [MSG] fpx_lookup_vec.lkup.macsakey.key.mac: 0xFEC80E2715

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
ELAM Wrapper - Demo
debug platform internal tah elam
trigger init
set outer ipv4 src-ip ip-address dst-ip ip-address
start
report [detail]
Demo – Consistency Checker (CC)
- test consistency-checker forwarding [ipv4 unicast | vrf vrf-name]
[module slot] [stop]

- Show consistency-checker forwarding

Demo – L2 PI/PD Troubleshooting
Show troubleshoot l2 mac mac-address [vlan vlan-id]
Demo – L3 PI/PD Troubleshooting
Show troubleshoot l3 [ipv4 | ipv6] v4/v6 address [src-ip v4/v6 address] [vrf vrf-name]
#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 98
Cisco Webex Teams
Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session

How
1 Find this session in the Cisco Events App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space

Webex Teams will be moderated cs.co/ciscolivebot#BRKDCN-3040

by the speaker until June 18, 2018.

#CLUS © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 99
Complete your online session evaluation

Give us your feedback to be entered

into a Daily Survey Drawing.
Complete your session surveys through
the Cisco Live mobile app or on
www.CiscoLive.com/us.
Don’t forget: Cisco Live sessions will be available for viewing
on demand after the event at www.CiscoLive.com/Online.

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 100
Continue
your Demos in
the Cisco
Walk-in
self-paced
Meet the
engineer
Related
sessions
education campus labs 1:1
meetings

#CLUS BRKDCN-3040 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 101
Thank you

#CLUS
#CLUS