Brkrst-3320 BGP Cisco Live

Troubleshooting BGP
Vinit Jain
CCIE# 22854 (RS, SP, Sec, DC)
Mani Ganesan
CCIE# 27200 (RS, SP)
@vinugenie
[email protected]
@mani_cisco
[email protected]
BRKRST-3320
Introduction
Housekeeping
Cell Phones
Who we are?
Who are you?

Service Provider
Enterprise
Data Center
Studying for CCIE
Advanced Class
Assume BGP Operational Experience
Basic configuration
Show commands
Understand BGP attributes
Why should you care?
Most widely deployed protocol - literally makes the internet work
Impact of BGP going down is huge
Understanding the challenges for troubleshooting BGP
Break down and simplify the problem statement
Various tools and techniques
Best practices for troubleshooting BGP
Real life TAC examples
Introduction
Operating Systems
IOS vs. IOS-XR vs. NX-OS
Troubleshooting concepts / logic are the same
Some variation in show command syntax and output
Will use all three in this presentation
Agenda
BGP peering issues
session not coming up, dynamic peering, session Flapping
Platform issues due to BGP
High CPU, Memory utilization
BGP slow-peer
BGP for Enterprise

- Multi-homing, Load-sharing
BGP for Service Providers
MPLS VPN, BGP PIC
BGP for Datacenter

- VxLan BGP-EVPN
BRKRST-3320 - Troubleshooting BGP

Session Flow
Problem
Troubleshooting
Steps
TAC Example
Solution
Agenda
BGP peering issues
BGP Slow-peer
BGP for Enterprise


- MPLS VPN, BGP PIC
BGP for Datacenter

- VxLan BGP-EVPN
Scenario 1 - Failed BGP Peering

Problem Description
iBGP / eBGP is not establishing
Newly configured BGP session not coming up
Session was up before, but not coming up now

OSPF
R1
iBGP
OSPF
R2
AS65535
eBGP
Physical Topology
R3
AS65534
R1
iBG
P
R2
eBGP
AS65535
Logical Topology
R3
AS65534
Failed BGP peering

Configuration
OSPF
R1
iBGP
R2
eBGP
AS65535
router bgp 65535

bgp router-id 2.2.2.2
bgp log-neighbor-changes
neighbor 1.1.1.1 remote-as 65535
neighbor 1.1.1.1 update-source Loopback0
. . .
R3
AS65534
Check
AS Numbers
Peering IP
eBGP Multihop?
router bgp 65534

bgp router-id 3.3.3.3
bgp log-neighbor-changes
. . .
Failed BGP peering

Reachability
R1# ping 2.2.2.2
Sending 5, 100-byte ICMP Echos to 2.2.2.2,
timeout is 2 seconds:
Lo0 = 1.1.1.1/32
R1
AS65535
Rn
Packet sent with a source address of 10.12.12.1

!!!!!
Success rate is 0 percent (0/5)
R1# ping 2.2.2.2 source loopback0
Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
Lo0 = 2.2.2.2/32
R2
Failed BGP peering

Verify any Firewall / ACL in path for TCP port 179
ASA_FW# sh run access-list
access-list OUT extended permit icmp any any
access-list OUT extended permit ospf any any
AS65535
AS65534
R3
R2
access-list OUT extended permit tcp any any eq telnet

. . . .
Rn# sh ip access-list R1_R2
Lo0 = 1.1.1.1/32
AS65535
Lo0 = 2.2.2.2/32
permit icmp any any

permit ospf any any
R1
Rn
R2
permit tcp host 10.12.12.1 eq bgp 2.2.2.2

permit tcp host 10.12.12.1 2.2.2.2 eq bgp
. . . .
ACL
Failed BGP peering

Verify TCP session
R2#sh tcp brief
TCB
Local Address
65F19834 2.2.2.2.179
Foreign Address
1.1.1.1.46523
(state)
ESTAB
Quick test when BGP is down

R1#telnet 2.2.2.2 179 /source-interface loopback 0
Trying 2.2.2.2 ...
% Destination unreachable; gateway or host down
R1#
This means BGP Packets are being blocked between R1 and R2
Failed BGP peering

Blocked process in XR
Ensure
BGP process is in Run
state.
Check
for blocked BGP or

TCP process on the RP / LC
using show process blocked
command.
RP/0/RSP0/CPU0:ASR9010-B# show process bgp

Mon Jun 3 09:47:12.646 EST
Job Id: 1040
PID: 307494
Executable path: /disk0/iosxr-routing4.2.3/bin/bgp
Instance #: 1
Version ID: 00.00.0000
Respawn: ON
Respawn count: 1
Max. spawns per minute: 12
Last started: Tue May 28 14:35:50
2013
Process state: Run
Package state: Normal
Started on config: default
. . . .
Failed BGP peering

Show process bgp (contd. Output)
RP/0/RSP0/CPU0:ASR9010-B# show process bgp
<snip>
10
2 488K 10 Nanosleep
0:00:02:0004
0:00:00:0847 bgp
1049
13
3 488K 10 Receive
0:00:00:0811
6:36:52:0264 bgp
1049
14
3 488K 10 Condvar
14:56:55:0236
9:07:49:0890 bgp
1049
15
0 488K 10 Condvar
14:56:55:0240
25:09:49:0542 bgp
1049
16
3 488K 10 Running
0:00:00:0000
57:53:33:0110 bgp
1049
17
1 488K 10 Receive
0:00:28:0379
0:00:00:0066 bgp
1049
18
1 488K 10 Mutex
13:15:50:0870
3:31:49:0712 bgp
<snip>
You can also use show process blocked to check the blocked
processes
Failed BGP peering

Sniffer Capture
Use SPAN to get traffic to your sniffer

- monitor session 1 source interface Te2/4 rx
- monitor session 1 destination interface Te2/2
IOS-XR
- Only supported on ASR-9000
- Use ACLs to control what packets to SPAN
RSPAN
- RSPAN has all the features of SPAN, plus support for source ports and destination ports
that are distributed across multiple switches, allowing one to monitor any destination port
located on the RSPAN VLAN. Hence, one can monitor the traffic on one switch using a
device on another switch.
Failed BGP peering

Wireshark
Failed BGP peering

Platform Specific Packet Capture Tools
IOS
Embedded Packet Capture
6500 / 7600
ELAM
NETDR Capture
MPA (Mini Protocol Analyzer)
ASR9000
Network Processor Capture
Nexus (7k, 5k, 3k)
Ethanalyzer
Elam
For your
reference only
Dynamic BGP peering

BGP Dynamic Neighbors
Allows BGP peering to a group of remote neighbors that are defined by a range of
IP addresses
BGP passively listens to configured address range for incoming sessions
BGP neighbor dynamically created

- Remote address is source of TCP connection
- Config template associated with listen range is applied
Provisioning
- No manual config necessary on hub for new clients
- Significant reduction in config overhead
Dynamic BGP Peering

Configuration
router bgp 65535
Creating a global limit of BGP

dynamic subnet range
neighbors
neighbor Test peer-group

bgp listen limit 300
bgp listen range 192.168.0.0/16 peer-group Test
neighbor Test remote-as 300 alternate-as 200
Configuring subnet range

and associating with a peer
group
!
address-family ipv4 unicast
neighbor Test activate
Associating Autonomous
System numbers for listen range
peers
Max Listen Limit 5000
Alternate-as limit is 5 (Config only used with listen range peer-groups)
Dynamic BGP Peering

Verifying Dynamic BGP Peers
R2# show ip bgp summary
BGP router identifier 192.168.3.1, local AS number 65535
BGP table version is 1, main routing table version 1
Neighbor
V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
*192.168.3.2
4 200
2
2
1
0
0 00:00:37
0
* Dynamically created based on a listen range command
Dynamically created neighbors: 1/(300 max), Subnet ranges: 1
BGP peergroup groupTAC listen range group members:
192.168.0.0/16
Dynamic BGP Peering

Most common issues
MD5 related
- Make sure MD5 password is configured at both ends if see the error such as "MD5
received, but NOT expected from.." message
Resource issues in a scaled environment

Security issues if the range is not carefully defined
Lab TRY - Try removing 'bgp listen range' and add it back
- Only try it in lab for testing purposes, not in live production
TAC Case Example - 1

BGP Peering down
Customer
reported new iBGP peer not

coming up with a different vendor device
Configuration
verified
show
ip bgp summary shows BGP

state changes from Idle to Active and
then to Closing state
TCP
session goes to Established but

then immediately moves to CloseWait
AS100
R1
R2

show log | in BGP
R2#
*Jun 5 18:18:04.667: %BGP-3-NOTIFICATION: sent to neighbor
10.1.12.1 active 2/7 (unsupported/disjoint capability) 0 bytes
R2#
*Jun 5 18:18:04.671: %BGP-4-MSGDUMP: unsupported or malformatted message received from 10.1.12.1:
FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 002D 0104 0064 00B4
0101 0101 1002 0601 0400 0100 0102 0280 0002 0202 00

Resolution
Analysis
Capability 131 is set when BGP is trying to establish multisession
- The other side did not understand this capability i.e. Single-session
-
Resolution
-
Configure both sessions to use same capability i.e. Single-session / Multi-session

-
neighbor <ip-addr> transport single-session | multi-session
Disable capability negotiation during session establishment process

-
neighbor <ip-addr> dont-capability-negotiate
Scenario 2 - BGP Peer Flapping

Problem Description
Multiple BGP Sessions Flapping
Keeps oscillating between two states (Idle/Established)
Symptoms
AS 65534
- Hold time expired notifications

- High CPU
- Interface Input-Queue Drops
iBGP
eBGP
iBGP
AS 65535
BGP Peer Flapping

BGP States
Stuck in IDLE state:

- No connected route to the peer
Staying in ACTIVE state:
- No route to the peer address (IP connectivity is not there)
- Configuration error, update-source
Flapping IDLE/ACTIVE:
- TCP establishes but BGP negotiation fails Misconfigured AS
Flapping IDLE/Established:
- Bad update, TCP problem (MSS size in multi-hop)
AS65534
R2
AS65535
Failed BGP peering

Notifications
R2#
*Mar 24 20:25:47.262: %BGP-5-ADJCHANGE: neighbor 1.1.1.1 Down BGP
Notification sent
*Mar 24 20:25:47.262: %BGP-3-NOTIFICATION: sent to neighbor 1.1.1.1 4/0
(hold time expired) 0 bytes
BGP NOTIFICATIONs consist of an error code, sub-code and data

- All Error Codes and Sub-codes can be found here
- http://www.iana.org/assignments/bgp-parameters/bgp-parameters.xml
- http://tinyurl.com/bgp-notification-codes
- Data portion may contain what triggered the notification

-
Example: corrupt part of the UPDATE
Notifications Contd
%BGP-3-NOTIFICATION: sent to neighbor 2.2.2.2 2/2 (peer in wrong AS) 2 bytes 00C8
FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 002D 0104 00C8 00B4 0202 0202 1002 0601 0400
0100 0102 0280 0002 0202 00
Value
Name
Reference
Message Header Error
RFC 4271
OPEN Message Error
RFC 4271
UPDATE Message Error
RFC 4271
Hold Timer Expired
RFC 4271
Finite State Machine Error
RFC 4271
Cease
RFC 4271
The first 2 in 2/2 is the Error Code.so OPEN Message Error
Notifications Contd
Subcode #
Subcode Name
1
Unsupported BGP version
Bad Peer AS
Bad BGP Identifier
Unsupported Optional Parameter
Unacceptable Hold Time
Unsupported Capability
Subcode Description
The version of BGP the peer is running isnt compatible with
the local version of BGP
The AS this peer is locally configured for doesnt match the
AS the peer is advertising
The BGP router ID is the same as the local BGP router ID
There is an option in the packet which the local BGP
speaker doesnt recognize
The remote BGP peer has requested a BGP hold time
which is not allowed (too low)
The peer has asked for support for a feature which the local
router does not support
OPEN Message Subcodes shown above

The second 2 in 2/2 is the Error Subcode.so Bad Peer AS
Failed Peering
Notifications
10.1.2.2
x0064 = data of NOTIFICATION

x0064 = decimal 100
10.1.2.1
R2# show log | include NOTIFICATION

%BGP-3-NOTIFICATION: sent to neighbor 10.1.2.1 2/2 (peer in wrong AS)
R2
2 bytes 0064 FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 002D 0104
AS 100
0064 00B4 0101 0101 1002 0601 0400 0100 0102 0280 0002 0202 00
R3
AS 200
Sniff of BGP Notification Sent from R2 to R1
Failed BGP peering

MTU Mismatch
R2#show ip bgp neighbors 1.1.1.1 | in max data segment
Datagrams (max data segment is 9176 bytes):
R2# ping 1.1.1.1 source 2.2.2.2 df-bit size 1500
Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 1.1.1.1, timeout is
2 seconds:
Packet sent with the DF bit set
!!!!!
R2# ping 1.1.1.1 source 2.2.2.2 df-bit size 9216
Type escape sequence to abort.
Sending 5, 9216-byte ICMP Echos to 1.1.1.1, timeout is
2 seconds:
Packet sent with the DF bit set
.....
Lo0 = 1.1.1.1/32
AS65535
Lo0 = 2.2.2.2/32
R2
R1
1500
BGP OPENs and Keepalives

are small
UPDATEs can be much larger
Maybe small packets work but

larger packets do not?
MTU 20byte IP header 20 byte TCP header = MSS
Failed BGP peering

High CPU Victim BGP
High CPU can cause data and control

plane packet loss
High CPU can be caused due to

process or interrupt (traffic hitting CPU)
Example Packets with TTL set to 1

are punted to CPU
PE1_RTR# sh proc cpu sorted | ex 0.00
CPU utilization for five seconds: 92%/91%; one minute: 14%; five minutes: 8%
PID Runtime(ms)
Invoked
uSecs
5Sec
1Min
5Min TTY Process
82
7490
929
8062 0.29% 2.96% 1.52%
0 Exec
4
6308484
908039
6947 0.05% 0.08% 0.06%
0 Check heaps
Failed BGP peering

Interface Input-Queue Drops and CoPP Drops
PE_RTR# sh int gi 0/1 | in drop
Input queue: 5/75/4351/0 (size/max/drops/flushes); Total output drops: 0
PE_RTR# show policy-map control-plane
Control Plane
. . . .
Class-map: Routing (match-any)
1441 packets, 195499 bytes
5 minute offered rate 5000 bps, drop rate 1000 bps
Match: access-group name CPP-Routing
police:
cir 10000000 bps, bc 312500 bytes
conformed 271 packets, 40286 bytes; actions:
transmit
exceeded 57 packets, 61143 bytes; actions:
drop
conformed 5000 bps, exceed 1000 bps
Input-Queue drops can also

lead to multiple ControlPlane packet loss along
with data loss
Drops in CoPP policy can

cause BGP sessions to flap

BGP Session Flapping
AS100
PE
AS200
PRTR
Multiple
BGP sessions flapping noticed

on ATM circuit towards a partner router
following an IOS upgrade on the router
No
Changes in configuration
Previous
show ip bgp summary

shows all the other neighbors were
stable
Requested
show ip bgp neighbor

<nei-ip> command before and after
upgrade
router bgp 100
address-family vrf ABC

neighbor 12.12.12.2 password cisco
neighbor 12.12.12.2 activate

neighbor 12.12.12.2 send-community

MTU Analysis
Before Upgrade
Rcvd: 517011 (out of order: 1), with data: 232834, total data bytes: 4674384
Sent: 525432 (retransmit: 6796 fastretransmit: 5),with data: 295508, total data bytes: 6886010
After Upgrade
Rcvd: 166 (out of order: 0), with data: 76, total data bytes: 2203
Sent: 168 (retransmit: 2 fastretransmit: 0),with data: 92, total data bytes: 2555

MTU Calculation
Before Upgrade Wrong Calculation
MSS = MTU (IP Header + TCP Header) Optional Bytes

After Upgrade Correct Calculation
MSS = MTU (IP Header + TCP Header)

The old code had a different behavior of calculating the MSS value

Resolution
Globally change the TCP MSS negotiation value

- ip tcp mss <mss-value>
Remove password authentication for the affected neighbors
Troubleshooting Summary Peering Issues
BGP Peering Down
Verify BGP configuration (router-config, ACL, Firewall, etc.)

Verify reachability between peers sourcing the peering IP
Check the TCP table for the active sessions (show tcp brief)
Packet Captures
Peer Flapping
Analyze the Notifications generated in syslogs

Verify MTU in the path
Verify CPU utilization and history
Drops (interface, CoPP)
Agenda
BGP peering issues

- Session not coming up, dynamic peering, session Flapping

- High CPU, Memory utilization
BGP slow-peer
BGP for Enterprise


- MPLS VPN, BGP PIC
BGP for Datacenter

- VxLan BGP-EVPN
Scenario 3 High CPU due to BGP

Problem Description
High CPU noticed due to BGP

- BGP Scanner
- BGP Router
Symptoms
- Router unstable
- Traffic loss
- Loss of Manageability
High CPU due to BGP

BGP Scanner
Can be expected for short durations carrying large Internet routing table
High cpu condition varies with the number of neighbors and the number of
routes learned per neighbor
Verify from platform data-sheet on the scalability of the router

- E.g. Sup720-3BXL on 7600 series router has default IPv4 TCAM size of 512,000 routes
Make use of maximum-prefix knob where required

------------------ show process cpu -----------------CPU utilization for five seconds: 99%/0%; one minute: 28%; five minutes: 17%
143
. . . .
393745836
2319493
169755 71.49% 15.34% 12.38%
0 BGP Scanner
High CPU due to BGP

BGP Router
Router#show process cpu
CPU utilization for five seconds: 100%/0%; one minute: 99%; five minutes: 81%
....
139
6795740
1020252
6660 88.34% 91.63% 74.01%
0 BGP Router
Look at the scenario
Is BGP going through Initial Convergence?
Are there any route churns?
The high cpu on the device could also be due to the instability of the BGP table.
(Receiving two copies of routing table one from iBGP and one from eBGP)
Insufficient Memory
High CPU due to BGP

Route Churn (Flapping Routes)
How to identify route churn?

- Do sh ip bgp summary | in table, note the table version
- Wait 4-5 seconds
- Do sh ip bgp summary | in table, compare the table version from 4-5 seconds ago
You have 150k routes and see the table version increase by 300
- This is probably normal route churn
- Know how many bestpath changes you normally see per minute
You have 150k routes and see the table version increase by 150k
- This is bad and is the cause of your high CPU
High CPU due to BGP

Route Churn
Router#show ip route | in 00:00:0
B
10.1.0.0 [200/0] via 10.185.80.140,
B
10.2.0.0 [200/0] via 10.185.80.140,
B
10.3.0.0 [200/0] via 10.185.80.140,
B
10.4.0.0 [200/0] via 10.185.80.140,
B
10.5.0.0 [200/0] via 10.185.80.140,
. . . .
00:00:00
00:00:00
00:00:00
00:00:00
00:00:00
Router#Show ip bgp all sum | in tab

Router#
Over 1800 prefixes flapped

Router#Show ip bgp all sum | in tab
Router#
High CPU due to BGP

Embedded Event Manager (EEM)
Serves as a powerful tool for high CPU troubleshooting
Triggered based on event and thresholds
Multiple actions can be set based on events

event manager applet HIGHCPU
event snmp oid "1.3.6.1.4.1.9.9.109.1.1.1.1.3.1" get-type exact entry-op gt entry-val "90"
exit-op lt exit-val "70" poll-interval 5 maxrun 200
action 1.0 syslog msg "START of TAC-EEM: High CPU"
action 1.1 cli command "enable
action 1.4 cli command "debug netdr capture
action 2.0 cli command "sh clock | append disk0:proc_CPU"
action 2.1 cli command "show process cpu sorted | append disk0:proc_CPU
action 2.2 cli command "show proc cpu history | append disk0:proc_CPU"
action 2.3 cli command "show netdr | append disk0:proc_CPU"
action 3.1 cli command "show log | append disk0:proc_CPU"
action 4.0 syslog msg "END of TAC-EEM: High CPU"

High CPU due to BGP Router
RR1
PE1
Customer
reported one of the router

becomes inaccessible
They
had to reload the router to get the

access back
AS65535
RR2
Found CPU spiking to 100% due

to BGP Router process
PE1#show ip route | in 00:00:0

B
10.1.1.0 [200/0] via 10.1.12.2, 00:00:00
10.2.1.0 [200/0] via 10.1.12.2, 00:00:00
<snip>
B
PE2
10.100.10.0 [200/0] via 10.1.12.2, 00:00:00

High CPU due to BGP Router
RR1
To investigate further, we followed the path

through till router R2
On PE2 we noticed 11 flapping links
PE1
PE2
AS65535
RR2
Captured syslog for last 2 days

<SNIP>
1367861: Sep 11 2012 17:33:14: %LINK-SW1_SP-3-UPDOWN: Interface GigabitEthernet1/4/36, changed state
to up
1367862: Sep 11 2012 17:33:14: %LINEPROTO-SW1_SP-5-UPDOWN: Line protocol on Interface

GigabitEthernet1/4/36, changed state to up
1367919: Sep 11 2012 17:34:43: %LINK-3-UPDOWN: Interface GigabitEthernet1/4/36, changed state to down

Resolution
Configured IP event dampening and BGP Dampening
Alternatively we suggested to configure ACL to block the subnets learnt
Work on stabilizing the flapping links.
Scenario 4 High Memory / Memory Leak

Problem Description
High Memory consumption by BGP
Caused due to:

- Insufficient Memory
- Memory Leak
Symptoms
- Slow performance
- Malloc Failures
- Router reloads
High Memory / Memory Leak

Malloc Failures
Sep 20 22:43:01.831 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65556 bytes
failed from 0x400E04EC, alignment 16
Pool: Processor Free: 8952 Cause: Not enough free memory
Alternate Pool: Reclaimed Free: 25520 Cause: Not enough free memory
-Process= "BGP Router", ipl= 0, pid= 156
-Traceback= 40348B24 403FB928 403FDFA0 403F7238 40F5AC5C 4026B690 406B794C 40682DB0
406833FC 40884688 40884DF4 40885BB4 40F9B160 40885C68 40843E68
If Malloc error show the process as BGP doesnt mean BGP is the culprit
This error log can be the consequence of insufficient memory or a memory leak
condition
Get memory base line from your NMS tool
Run show memory debug leak [chunk] to identify a memory leak

Memory Leak in IOS XR
Use IOS XR memory comparator tool to track any incremental memory leak
Simple 3 step process

- Show memory compare start
- Show memory compare end
- Show memory compare report
RP/0/RP0/CPU0:XR_RTR# show memory compare report

Sun Apr 12 22:28:21.715 PDT
JID
name
mem before
mem after
restart/exit/new
----------------------1088 mibd_interface
232432236
232957764
1086 mibd_entity
1046476
1528332
1044 bgp
1037562644
1037827636
0
malloc_dump
0
22144
<snip>
4-5 min gap
difference mallocs
---------525528
481856
264992
22144
------33753
11
425
355

Memory Baseline
Memory usage baseline (Use of polling servers)
Understand since when the memory started increasing

What changes were made around the time the memory increased?
Understanding if the platform is having sufficient memory based on the services

its running
Also based on the amount of information learnt through BGP

Memory Leak
Customer reported show memory loses about 3-10 Mb of Free memory per
day
It was noticed that BGP Router process accumulated most of the holding
memory
Same behavior was noticed even after the reload.

Allocated
Freed
Holding
Process
458001556
61617656
240877764
BGP Router
459588300
61623540
241919840
BGP Router
463207104
61699296
244022952
BGP Router
467475524
61706168
246534736
BGP Router
The holding memory keeps on increasing

Memory Leak
Asked customer to run show memory debug leak chunk

-
Customer denied
Setup a lab to reproduce the problem

RR1
Pagent
PE1
AS65535
RR2
Still not able to reproduce the problem
Configured 400 BGP

neighbors and 100,000
routes

Memory Leak
Neighbor
State/PfxRcd
80.81.192.33
80.81.192.45
80.81.192.117
80.81.193.61
80.81.193.217
195.30.0.18
. . . . .
AS MsgRcvd MsgSent
4 1273
4 20646
4 3209
4 8220
4 3209
4 5539
0
0
0
0
0
0
0
0
0
0
0
0
TblVer
0
0
0
0
0
0
InQ OutQ Up/Down

0
0
0
0
0
0
0
0
0
0
0
0
never
never
never
never
never
never
Idle
Idle
Idle
Idle
Idle
Idle
(Admin)
(Admin)
(Admin)
(Admin)
(Admin)
(Admin)
Resolution
-
Removing the Idle / Admin down neighbors helped overcome the memory leak
Troubleshooting Summary Platform issues
High CPU
- Verify what is the cause of the high CPU
- Is it due interrupt or process?
- Verify if there are route churns happening in your network / partner network
- Verify if there was any recent changes made in your network (addition of new customer,
addition of routes) which triggered the impact in your network
- Configure EEM with appropriate outputs to gather relevant information in case the high
CPU condition is random
High Memory
- Verify Memory baseline, changes done
- How fast is the memory increasing?
- Trigger of memory increase
Agenda
BGP peering issues
BGP slow-peer
BGP for Enterprise


- MPLS VPN, BGP PIC
BGP for Datacenter

- VxLan BGP-EVPN
Scenario 5 BGP Slow Peer

Problem Description
Customer reports updates not getting across all PE routers
Symptoms
High CPU due to BGP

Updates not replicated to all peers
Router reloads
RR1
PE1
Im
Slow
PE3
PE2
BGP Slow-peer
BGP OutQ & Cache Size
OutQ column should show very high OutQ value
Should be reaching the maximum cache size for that update-group

RR# show ip bgp
..
Neighbor
12.123.67.97
12.122.78.19
vpnv4 all summary

V
4
4
AS MsgRcvd MsgSent
109
42
87065
109
42
87391
TblVer
0
0
InQ OutQ Up/Down State/PfxRcd

0 1000 00:10:00
0
0 674 00:10:00
0
RR# show ip bgp vpnv4 all replication

Index
1
2
3
4
Members
348
2
1
2
Leader
12.123.67.97
12.122.78.19
199.37.187.24
12.122.78.249
MsgFmt
MsgRepl
1726595727 1938155978
79434677
79398843
0
0
79219618
97412908
Current Next
Csize
Version Version
999/1000 1012333000/1012351142
0/200 1012351503/1012351503
0/100
0/0
0/200 1012351504/1012351504
BGP Slow-peer
TCP sndwnd
RR#show neighbor 10.1.0.1
..
iss: 3662804973 snduna: 3668039487
irs: 1935123434 rcvnxt: 1935222998
SRTT: 300 ms,
minRTT: 0 ms,
Status Flags:
Option Flags:
sndnxt: 3668039487
rcvwnd:
16003
sndwnd:
delrcvwnd:
0
381
RTTO: 303 ms, RTV: 3 ms, KRTT: 0 ms

maxRTT: 512 ms, ACK hold: 200 ms
passive open, gen tcbs
nagle, path mtu capable
Check for send window (sndwnd) and receive window (rcvwnd) using show ip
bgp neighbor <x.x.x.x>
For the TCP session for which outQ is high, we might notice that sndwd is very
low or zero.
On the remote end, we should see the rcvwnd value is very low or zero.
BGP Slow-peer
Static Slow peer
The manual knob to flag a peer as slow will create a separate update group for
the peer.
The advantage - there is a limit to the overhead that this feature will create.
The drawback - slow member update group will have to progress at the pace of
the slowest of the slow peers.
neighbor {<nbr-addr>/<peer-grp-name>} slow-peer split-update-group static
This command will manually mark a neighbor as slow peer.

The peer will be part of slow update group.
BGP Slow-peer
Dynamic Slow peer
IOS BGP will monitor the transmission speeds of the peers.
A peer will have to be exhibiting slowness for several minutes to be flagged.
Log message for when a slow peer is detected/recovered

bgp slow-peer detection [threshold <seconds>]
neighbor {<nbr-addr>/<peer-grp-name>} slow-peer detection [threshold < seconds >]
The threshold defines the threshold time in seconds to detect a peer as slow
peer.
The range is 120 seconds to 3600 seconds. Default is 300 seconds.
BGP Slow-peer
Slow peer protection
Depends on Dynamic Slow Peer feature
When a slow peer recovery is detected (the peer has converged), the peer will
be moved back to its original group
bgp slow-peer split-update-group dynamic [permanent]
neighbor {<nbr-addr>/<peer-grp-name>} slow-peer split-update-group dynamic [permanent]
When permanent is not configured, the slow peer will be moved to its regular
original update group, after it becomes regular peer (converges).
If permanent is configured, the peer will not be moved to its original update
group automatically
BGP Slow-peer
Syslog Messages
The below log message will be generated when a peer is detected as dynamic
slow peer.
"bgp neighbor %s in af %d is detected as slow-peer"
The below log message will be generated when a slow-peer recovers.

"slow bgp peer %s in af %d has recovered"
BGP Slow-peer
Show / Clear Commands
Show Commands
show ip bgp [AF/scope/topo] update-group summary slow
show ip bgp [AF/scope/topo] summary slow

show ip bgp [AF/scope/topo] neighbor slow
Clear Commands
Clear [ip] bgp <nbr-addr> slow
Clear [ip] bgp peer-group <group-name> slow
Clear [ip] bgp af * slow
Clear ip bgp * slow
TAC Case Example 5
RR1
BGP Slow Peer
Customer reported routes were stuck in BGP RR.
Their end-customer removed service from one of their locations but the routes
are still seen on their RR and other locations
Soft clearing the neighborship temporarily resolved the problem but reoccurred
again after sometime
PE1
PE3
RR1# sh ip bgp vpnv4 all replication

Index
1
2
5
Members
150
5
1
Leader
216.156.3.10
65.106.7.100
66.239.189.212
MsgFmt
274950548
41049479
16143960
MsgRepl
Csize
650809652 2000/2000
204232170
0/500
16143960
0/100
Current
Next
Version Version
421492656/421493582
421493582/0
421491282/421493582
PE2
TAC Case Example 5

Resolution
Two neighbors were identified to be showing slow peer symptoms
Customers RR router didnt had the slow peer capability in the IOS they were
running
Two workarounds / solutions:
Create a separate outbound policy for slow peers.
You can use the "neighbor <ip> advertise-interval <interval>".
Default for internal neighbors is 5 sec and for external is 30 seconds.
Agenda
BGP peering issues
BGP slow-peer
BGP for Enterprise


- MPLS VPN, BGP PIC
BGP for Datacenter

- VxLan BGP-EVPN
Scenario 5 Multi-Homing
Problem Description
In a multi-homed network, load sharing is not working as expected
Symptoms:
- Under or over utilized link / resource
B
A
90%
D
AS200
AS65000
Provider
5%
C
BGP Multi-homing
Overview
More than one link external to the local network

- Usually two external facing routers
Two design scenarios :

- Multi-homing to the same ISP
- Multi-homing to different ISPs
Each design categorized into two modes :

- Fail-over mode
- Load-sharing mode
Following examples assume that the multi-homed site has a /19 address block
For your
reference only
Best Path Algorithm

1
Weight
Highest wins
Scope is router only
LOCAL_PREFERENCE
Highest wins
Scope is AS only
Locally Originated
AS_PATH
Shortest wins
Skipped if bgp bestpath as-path ignore configured

AS_SET counts as 1
CONFED parts do not count
ORIGIN
Lowest wins
IGP < EGP < Incomplete
MED
Lowest wins
MEDs are compared only if the first AS in the AS_SEQUENCE

is the same
eBGP over iBGP
Metric to Next Hop
Lowest wins
IGP cost to the BGP NEXTHOP
Multiple Paths in RIB
Flag path as multipath is max-paths is configured
10
Oldest External Wins
Unless BGP best path compare router-id configured
11
BGP Router ID
Lowest
12
CLUSTER_LIST
Smallest
Shorter CLUSTER_LIST wins
13
Neighbor Address
Lowest
Lowest neighbor address
Redistribution or network statement favored over aggregateaddress
Multi-Homing with same ISP

Failover Mode
Applies when end-site has bought a large primary WAN link to their upstream a
small secondary WAN link as the backup
primary link:
- Outbound announce /19 unaltered
- Inbound receive default route
B
A
D
AS200
AS65000
backup
C
primary
backup link:
- Outbound announce /19 with increased metric
- Inbound received default, and reduce local preference
Multi-Homing with same ISP

Load-Balancing Mode
This example assumes equal capacity circuits - Unequal capacity circuits
requires more refinement
Announce /19 aggregate on each link

- Assumes multi-path feature is enabled on internal routers of receiving AS
Split /19 and announce as two /20s

- Advertise one with a lower MED and the other with a higher MED on each link
B
A
Link-1
D
AS200
AS65000
Link-2
C
Multi-Homing with two different ISPs

Failover Mode
One link primary, the other link backup only
Announce /19 aggregate on each link

- Primary link makes standard announcement
- Backup link lengthens the AS PATH by using AS PATH prepend
D
AS200
B
A
Internet
AS65000
C
E
AS300
Multi-Homing with two different ISPs

Load-Sharing Mode
Use both default from both ISPs for outbound
Two options for inbound :

1. Announce /19 aggregate on each link
- Assumes multipath enabled on upstream BGP speakers
2. Split /19 and announce as two /20s
- one is announced with lengthened AS by prepending and other is announced as is
on each link
D
AS200
B
A
Internet
AS65000
C
E
AS300
For your
reference only
Best Path
Algorithm
Quick bestpath review
Remember
BGP only advertises one path per prefixthe bestpath

Cannot advertise path from one iBGP peer to another
Bestpath selection process is a little lengthy
First eliminate paths that are ineligible for bestpath

1
Not synchronized
Only happens if sync is configured AND the route isnt in your IGP
Inaccessible NEXTHOP
IGP does not have a route to the BGP NEXTHOP
Received-only paths
Happens if soft-reconfig inbound is applied. A path will be received-only if it

was denied/modified by inbound policy.
BGP - Single-Path Protocol
Each router picks a single best route
From the routes learned from its neighbors
And announces that route to its neighbors

R2#sh ip bgp 7.4.4.0/24 bestpath
BGP routing table entry for 7.4.4.0/24, version 2
Paths: (20 available, best #13, table Default-IP-Routing-Table)
Flag: 0x820
Not advertised to any peer
100
192.150.6.11 from 192.150.6.11 (192.150.6.11)
Origin IGP, metric 0, localpref 100, valid, external, best
R2#
For your
reference only

BGP Multi-Homing
It was required to prefer router B as the primary router for all outgoing and
incoming traffic.
In case of any failure, the traffic must automatically be routed to C.
D
AS200
AS100
Internet
A
C
E
AS300

Resolution
Making router B the primary HSRP gateway for A will solve the outbound
requirement
Use of set as-path prepend will help solve this problem of inbound
requirement
Router C:
=========
access-list 100 permit <specific_traffic>
route-map foo permit 10
match ip address 1
set as-path prepend 100
!
router bgp 100
neighbor <AS_300_nei> route-map foo out
Agenda
BGP peering issues

- session not coming up, dynamic peering, session Flapping

BGP Slow-peer
BGP for Enterprise


- MPLS VPN, PIC
BGP for Datacenter

- VxLan BGP-EVPN
Scenario 7 MPLS VPN

Problem Description
Customer reports reachability issues for a Customer VRF Customer A

connected to PE1 is unable to reach its other site connected to PE2
Caused due to:
Wrong IGP/VPN Label propagation
Symptoms
PE1
RR1
PE2
No reachability
Packet loss
CE1
CE2
BGP in MPLS VPN

Troubleshooting steps
Verify VRF configuration on both PE routers

- IOS - show run vrf A
Verify local PE-CE Reachability (PE1 CE1) & (PE2 CE2)
Verify PE to PE loopback reachability

- PE1# ping <PE2_loopback> source lo0
PE1
Verify LSP path between PE routers

CE1
- ping mpls ipv4 <dst> <subnet>
- ping mpls traffic-eng tunnel <tunnel_num>
Verify PE to PE reachability for VRF

- PE1# ping vrf A <vrf_ip_on_PE2>
RR1
PE2
MPLS VPN
CE2
MPLS VPN
show mpls for 1.1.1.1
Labeling & Packet Flow
Verify VPN Label:

Show ip bgp vpnv4 vrf A 100.1.1.1
show mpls for vrf A
IGP: POP
IGP: 20
VPN:100
VPN:100
VPN:100
PE1
RR1
Lo0=1.1.1.1/32
PE2
Lo0=3.3.3.3/32
MPLS VPN
Show ip bgp vpnv4 vrf A 100.1.1.1

should have 100 as the Out label
advertised by PE1
Data Plane
Control Plane
CE1
CE2
Lo0=100.1.1.1/32
Lo0=200.1.1.1/32
PE1#sh ip bgp vpnv4 vrf A 100.1.1.1

BGP routing table entry for 1:1:100.1.1.1/32
192.168.10.2 from 0.0.0.0 (1.1.1.1)
<snip>
OSPF RT:0.0.0.0:2:0 OSPF ROUTER
ID:192.168.10.1:0
mpls labels in/out 100/nolabel
PE2#sh ip bgp vpnv4 vrf A 100.1.1.1

BGP routing table entry for 1:1:100.1.1.1/32
1.1.1.1 (metric 21) from 2.2.2.2 (2.2.2.2)
<snip>
OSPF RT:0.0.0.0:2:0 OSPF ROUTER
ID:192.168.10.1:0
Originator: 1.1.1.1, Cluster list: 2.2.2.2
mpls labels in/out nolabel/100
MPLS VPN
Checking All VPN Labels
PE1#sh ip bgp vpnv4 all labels
Network
Next Hop
In label/Out label
Route Distinguisher: 1:1 (A)
100.1.1.1/32
192.168.10.2
100/nolabel
192.168.10.0/30
0.0.0.0
19/nolabel(A)
192.168.20.0/30
3.3.3.3
nolabel/19
200.1.1.1/32
3.3.3.3
nolabel/20
PE1#
Troubleshooting Inter-AS MPLS VPN
For your
reference only
Understand the different types of Inter-AS MPLS VPN implementations
Back to Back VRF

Inter-AS Option B
Inter-AS Option C
Most important thing to verify is the LSP path
Use of ping mpls ipv4. Ensure MPLS OAM is enabled for this to work.
Understanding which node is supposed to allocate / modify VPN label
Understand which ASBR or RR will be exchanging customer routes across ASs.
Once the above understanding is gained, troubleshoot like regular MPLS VPN
or regular BGP troubleshooting

MPLS VPN
Customer reported that a newly configured MPLS Services for the customer are
not working. CE2 is unable to see the routes of CE2
Customer didnt have the access to AS1PE2 router initially as it was in a distant
location and managed by a different group.
AS1RR1
AS100
AS1PE1
AS1PE2
CE1
CE2
100.1.1.0/24
200.1.1.0/24

Troubleshooting Done
Debugging done to see if AS1RR1 was sending the update or not

debug ip bgp vpnv4 unicast update <neighbor> <acl> in
access-list 10 per 100.1.1.0 0.0.0.255
route-policy DEBUG_BGP
if destination in BGP_PREFIX then
pass
else
drop
endif
end-policy
prefix-set BGP_PREFIX
100.1.1.0/24
end-set
debug bgp update vpnv4 unicast [in | out] route-policy DEBUG_BGP
IOS
IOS XR

Other Imp Debugs / Traces / show techs
IOS XR
Show bgp trace [update] [error]

Show cef trace
Show tech routing bgp
IOS
Debug bgp <af> update

Debug ip bgp <af> trace
NXOS
Show tech bgp
For your
reference only

Resolution
Further access was requested to troubleshoot AS1PE2 router

ip vrf Cust_A
rd 15956:3772958
route-target import 15956:3772958
route-target export 15956:3772968
ip vrf Cust_A
rd 15832:3772958
route-target import
route-target export
route-target import
route-target import
route-target import
route-target import
route-target import
15832:3772958
15956:3772958
15956:3772958
12956:3772953
13956:3692952
14956:3832957
17956:3773959
It was noticed that there was a missing entry on the vrf import statement
Scenario 8 BGP PIC

Problem Description
Customer reported they have a multi-homed Customer. But when the primary
BGP session goes down, it takes time to converge and the customer
experiences a traffic loss
Symptoms
Traffic loss
PE2
CE1
PE1
AS100
CE2
PE3
What is PIC and BGP FRR?
Prefix Independent Convergence (PIC) in CEF and platform whereby cutover to any
backup path happens within sub-seconds and independent of the number of prefixes.
BGP Fast Re-Route (BGP FRR) enables BGP to use alternate paths within subseconds after a failure of the primary or active paths.
Without backup paths available to CEF, convergence is driven from the routing protocols
updating the RIB and CEF one prefix at a time, leading to convergence times directly
proportional to the number of affected prefixes.
When backup paths are available, CEF can use these to provide constant time and prefix
independent convergence, when a failure affecting a shared path-list occurs.
BGP PIC
PIC edge vs. PIC core
1
2
PE2
3
CE1
PE1
Customer
Site A
CE2
PE3
1. PIC
core when IGP path changes.
2. PIC
edge when remote PE node or its reach ability fails.
3. PIC
edge when PE-CE link fails.
Customer
Site B
BGP PIC
Configuration
BGP PIC Core is enabled by default. If disabled, re-enable by:

R1(config)cef table output-chain build favor convergence-speed
BGP PIC Edge

R1(config-bgp-vrf)bgp additional-path install
R1(config-bgp-vrf)bgp advertise-best-external
(on primary PE for PE-CE link protection)
BGP PIC
BGP Output
ASR-1K# sh ip bgp vpnv4 vrf site-111111 2.0.0.0
BGP routing table entry for 65300:111111:2.0.0.0/24, version 150035
Paths: (3 available, best #1, table sie-111111)
Additional-path-install
Advertised to update-groups:
105
Refresh Epoch 1
20570 20570
10.200.1.2 from 10.200.1.2 (10.200.1.2)
Origin incomplete, localpref 100, valid, external, best
Extended Community: RT:64300:111111 , recursive-via-connected
rx pathid: 0, tx pathid: 0x0
<snip>
Refresh Epoch 1
20570 20570
10.10.10.2 from 10.10.10.2 (5.5.5.5)
Origin incomplete, localpref 100, valid, internal, backup/repair
Extended Community: RT:64300:111111 , recursive-via-host
rx pathid: 0, tx pathid: 0
BGP PIC
CEF Output
ASR-1K# sh ip cef vrf site-111111 2.0.0.0 255.255.255.0 detail
2.0.0.0/24, epoch 0, flags rib only nolabel, rib defined all labels
recursive via 10.200.1.2
attached to GigabitEthernet0/1/1.200
recursive via 10.10.10.2, repair
attached to GigabitEthernet0/1/0.451
BGP PIC
Show Commands
BGP
RIB
sh ip bgp ipv4 unicast <x.x.x.x>

sh ip bgp vpnv4 unicast vrf FOO <x.x.x.x> (To check if backup/repair is set on a prefix)
sh ip bgp vpnv4 vrf FOO neighbor <neighbor_ip> (shows if PIC is enabled)
Show ip route vrf FOO <x.x.x.x>
CEF
show (ip | ipv6) cef [vrf XX] prefix/mask internal
show monitor event cef (ipv4 | ipv6 | bfd) all
show cef bfd
debug cef bfd
debug cef loadinfo map
debug cef path
debug cef filter fib (ipv4 | ipv6) prefix/mask

BGP PIC
Customer has implemented BGP PIC Edge feature and is also using BFD for
faster failover.
When the best path fails, the customer does not see the fast convergence
happening
P2
CE1
PE4
PE1
CE2
P3
PE5

CEF Output
ASR-1K#sh ip cef vrf site-111111 2.0.0.0 internal
2.0.0.0/30, epoch 0, flags rib only nolabel, rib defined all labels, RIB[B], refcount 6, perdestination sharing
sources: RIB
<snip>
GigabitEthernet0/1/1.350(21): 10.200.1.2
path 7FA7EB87AB00, path list 7FA7EB611F90, share 1/1, type recursive, for IPv4, flags recursivevia-connected
recursive via 10.200.1.2[IPv4:sie-111111], fib 7FA7ECBE2558, 1 terminal fib, v4:sie111111:10.200.1.2/32
path 7FA7EB87A630, path list 7FA7EB612C10, share 1/1, type adjacency prefix, for IPv4
attached to GigabitEthernet0/1/1.350, adjacency IP adj out of GigabitEthernet0/1/1.350, addr
10.200.1.2 7FA7EB643568
path 7FA7EB879DF0, path list 7FA7EB611F90, share 1/1, type recursive, for IPv4, flags repair,
recursive-via-host, unuseable
recursive via 10.10.10.2[IPv4:sie-111111], repair, fib 7FA7E2E01068, 1 terminal fib, v4:sie111111:10.10.10.2/32
<snip>

Resolution
router bgp 65300
!
address-family ipv4 vrf site-111111
bgp additional-paths install
bgp recursion host
network 3.3.3.0 mask 255.255.255.0
<snip>
neighbor 10.200.1.2 version 4
neighbor 10.200.1.2 fall-over bfd
neighbor 10.200.1.2 send-community
neighbor 10.200.1.2 next-hop-self
neighbor 10.200.1.2 soft inbound
exit-address-family
Removing this configuration, resolved the

issue. BGP Recursion host is useful when
performing node protection
Agenda
BGP peering issues

- session not coming up, dynamic peering, session Flapping

BGP Slow-peer
BGP for Enterprise


- MPLS VPN, PIC
BGP for Datacenter

- VxLan BGP-EVPN
BGP in VxLAN
Problem Description
Server virtualization has generated a large number of Virtual Machines (VMs) in

the Data Center
The growth of nodes in the Data Center requires that network administrators
segregate the VMs into isolated Layer 2 domains which has caused scalability
problems given the 4K VLAN limit
The demand of increasing number of isolated Layer 2 domains is push to higher

limits as Data Centers host multiple tenants and their applications.
Caused due to:
More host nodes (Physical and Virtual Machines) in Data Center

4K VLAN limit
VxLAN BGP EVPN

VXLAN Overview
VXLAN is a Layer 2 Overlay scheme over a Layer 3 network
VXLAN provides a way to extend Layer 2 networks across Layer 3 infrastructure

using MAC-in-UDP encapsulation and tunneling
VXLAN uses a 24-bit Virtual Network ID (VXLAN Network Identifier or VNID)

Allows for a maximum of 16 million VXLAN segments to coexist in the same
administrative domain
- Addresses the scalability limitations of traditional VLANs that use a 12-bit VLAN ID.
-
VXLAN packets are transferred through the underlying network based on its
Layer 3 header
Cisco Nexus switches are designed for hardware-based VXLAN
VxLAN BGP EVPN

VXLAN Overview
Edge Device
Edge Device
Local LAN
Segment
Physical
Host
Local LAN
Segment
IP Interface
Edge Device
Local LAN
Segment
Physical
Host
Virtual Switch
Virtual Hosts
104
VxLAN BGP EVPN

VXLAN Overview
VTEP
VTEP
V
Local LAN
Segment
Physical
Host
Local LAN
Segment
Encapsulation
VTEP
V
Local LAN
Segment
Physical
Host
Virtual Switch
VTEP VXLAN Tunnel End-Point

VNI/VNID VXLAN Network Identifier
Virtual Hosts
105
BGP for VxLAN

WHY?
Control plane learning for end host Layer-2 and Layer-3 reachability
information to build more robust and scalable VXLAN overlay networks.
Leverages the decade-long MP-BGP VPN technology to support scalable

multi-tenant VXLAN overlay networks.
EVPN address family carries both Layer 2 and Layer 3 reachability

information. This provides integrated bridging and routing in VXLAN overlay
networks.
BGP for VxLAN

Advantages
Minimizes network flooding through protocol-driven host MAC/IP route

distribution and ARP suppression on the local VTEPs.
Provides optimal forwarding for east-west and north-south bound traffic with the
distributed any-cast function
Provides VTEP peer discovery and authentication which mitigates the risk of
rouge VTEPs in the VXLAN overlay network.
VXLAN Concepts
VXLAN Overlay
-
For your
reference only
A VXLAN Overlay or VXLAN segment is a Layer-2 broadcast domain identified by the

VNID that extends or tunnels traffic from one VTEP to another.
VXLAN Tunnel End Point (VTEP)

A VTEP is a device that provides both encapsulation and de-capsulation of classical
Ethernet and VXLAN packets to and from a VXLAN segment
- Each VTEP may have the following types of interfaces:
-
Switchport interfaces on the local LAN segment to support local endpoints

Layer-3 interfaces to the transport IP network
SVI interfaces
VXLAN Gateway
-
A VTEP that bridges traffic between VXLAN segments
For your
reference only
VXLAN Encapsulation
Outer
Ethernet
Outer IP
Flags
VXLAN
Reserved
24 Bytes
8 Bytes
Rsvd
Outer
UDP
Rsvd
Ethernet
Header
Payload
FCS
Inner
Ethernet
Payload
New FCS
Instance ID
24 Bytes
Reserved
8 Bytes
Outer UDP Destination Port = VXLAN (originally 8472, recently updated to 4789)
Outer UDP Source Port = Hash of Inner Frame Headers (optional)
The outer IP header has the source IP and destination IP of the VTEP endpoints
The outer Ethernet header has the source MAC of the source VTEP and the
destination MAC of the immediate Layer-3 next hop
For your
reference only
VXLAN EVPN VNI Types

Tenant 1 (VRF 1)
SVI
X
SVI
A
SVI
B
Layer-3 VNI X
Layer-2 VNI A
Layer-2 VNI B
VLAN X
VLAN A
VLAN B
1 Layer-3 VNI per

Tenant (VRF) for
routing
VNI X is used for
routed packets
1 Layer-2 VNI per Layer-2 segment

Multiple Layer-2 VNIs per tenant
VNI A and B are used for bridged
packets
Initial Configuration Per Switch

Enable VXLAN and MP-BGP EVPN Control Plane:
feature nv overlay
feature vn-segment-vlan-based
feature bgp
nv overlay evpn
For your
reference only
Enable VXLAN
Enable VLAN-based VXLAN (the currently only

mode)
Enable BGP
Enable EVPN control plane for VXLAN
Other features that may need to be enabled:

feature ospf
feature pim
feature interface-vlan
Enable OSPF if its chosen to be the underlay IGP

routing protocol
Enable IP PIM multicast routing in the underlay network
Enable VLAN SVI interfaces if the VTEP needs to be IP

gateway and route for the VXLAN VLAN IP subnet.
For your
reference only
VXLAN EVPN Tenant VRF

Create VXLAN tenant VRF
vrf context evpn-tenant-1
vni 39000
rd auto
route-target both auto evpn
vni 39010
rd auto
Create a VXLAN Tenant VRF named evpn-tenant-1
Layer-3 VNI for VXLAN routing within the tenant
Define VRF RD (route distinguisher)
Define VRF Route Target and import/export policies in

Example to create a 2nd tenant VRF following the above

steps
Layer-3 VNI Per Tenant for EVPN Routing
For your
reference only
Configure Layer-3 VNI per EVPN Tenant VRF Routing Instant

vlan 3900
name l3-vni-vlan-for-tenant-1
vn-segment 39000
interface Vlan3900
description l3-vni-for-tenant-1-routing
no shutdown
vrf member evpn-tenant-1
ip address 39.0.0.1/16
fabric forwarding mode anycast-gateway
vni 39000
rd auto
Create VLAN 3900/VNI 39000 for Layer-3 VNI for

tenant evpn-tenant-1 vrf routing instance
Create the SVI interface for VLAN 3900/VNI 39000 for

VXLAN routing.
Put this SVI interface into the tenant VRF context
Associate VNI 39000 with the tenant VRF evpn-tenant-1

as its routing Layer-3 VNI
EVPN Layer-3 VNI Per Tenant for Routing Instance

Configure Layer-3 VNI per EVPN Tenant VRF Routing Instant
For your
reference only
vlan 3901
name l3-vni-vlan-for-tenant-2
vn-segment 39010
interface Vlan3901
description l3-vni-for-tenant-2-routing
no shutdown
vni 39010
rd auto
Define Layer-3 VNI for a 2nd tenant following the same

steps in the previous slide
VNI EVPN Layer-2 Network VXLAN VNI
For your
reference only
Map VLANs to VXLAN VNIs and Configure their MP-BGP EVPN Parameters
vlan 200
vn-segment 20000
vlan 210
vn-segment 21000
evpn
vni 20000 l2
rd auto
route-target
route-target
vni 21000 l2
rd auto
route-target
route-target
import auto
export auto
import auto
export auto
Map VLAN to VXLAN VNI
Under EVPN configuration, define RD and RT

import/export policies for each Layer-2 VNIs
EVPN Layer-2 Network VXLAN VLAN SVI Interface

Map VLANs to VXLAN VNIs and Configure their MP-BGP EVPN Parameters
interface Vlan200
no shutdown
interface Vlan210
no shutdown
For your
reference only
Create SVI interface vlan 200

Associate it with tenant VRF evpn-tenant-1
All VTEPs for this VLAN should have the same SVI
interface IP address as the distributed IP gateway
Enable distributed anycast gateway for vlan 200
subnet
Create SVI interface vlan 210
Associate it with tenant VRF evpn-tenant-2
All VTEPs for this VLAN should have the same SVI
interface IP address as the distributed IP gateway
Enable distributed anycast gateway for vlan 210
subnet
For your
reference only
EVPN Distributed Gateway
Configure distributed gateway virtual MAC address

One virtual MAC per VTEP
All VTEPs should have the same virtual MAC address
fabric forwarding anycast-gateway-mac 0002.0002.0002
interface Vlan210
no shutdown
Configure virtual IP address

All VTEPs for this VLAN should have the same virtual
IP address
Enable distributed gateway for this VLAN
VXLAN Tunnel Interface Configuration
For your
reference only
Configure VXLAN tunnel interface nve1

interface nve1
no shutdown
source-interface loopback0
host-reachability protocol bgp
member vni 20000
suppress-arp
mcast-group 239.1.1.1
member vni 21000
suppress-arp
mcast-group 239.1.1.2
member vni 39000 associate-vrf
member vni 39010 associate-vrf
Specify loopback0 as the source

interface
Define BGP as the mechanism for host reachability
advertisement
Associate tenant VNIs to the tunnel interface nve1
Define the mcast group on a per-VNI basis
Enable arp suppression on a per-VNI basis
Add Layer-3 VNIs, one per tenant VRF
MP-BGP Configuration on VTEP

router bgp 100
router-id 10.1.1.11
log-neighbor-changes
address-family l2vpn evpn
update-source loopback0
send-community extended
send-community extended
vrf evpn-tenant-1
advertise l2vpn evpn
<snip>
For your
reference only
address-family ipv4 evpn for vxlan host-based routing
Define MP-BGP neighbors. Under each neighbor

define address-family ipv4 unicast and l2vpn evpn
Send extended community for both MAC and Host

IP reachability information
Under address-family ipv4 unicast of each tenant

VRF instance, enable advertising EVPN routes
MP-BGP Configuration on iBGP Route Reflector

VTEP Configuration
router bgp 100
router-id 10.1.1.1
log-neighbor-changes
retain route-target all
template peer vtep-peer
remote-as 100
send-community both
route-reflector-client
send-community both
route-reflector-client
neighbor 10.1.1.11
inherit peer vtep-peer
<snip>
For your
reference only
address-family ipv4 unicast for prefix-based routing
address-family ipv4 evpn for vxlan host-based routing
iBGP RR client peer template
Send both standard and extended community

in address-family ipv4 unicast
Send both standard and extended community

in address-family l2vpn evpn
VxLAN BGP EVPN

Check Remote BGP neighbor
N9396-1# show bgp l2vpn evpn summary
BGP summary information for VRF default, address family L2VPN EVPN
BGP router identifier 10.40.1.1, local AS number 65040
BGP table version is 26, L2VPN EVPN config peers 1, capable peers 1
7 network entries and 7 paths using 696 bytes of memory
BGP attribute entries [4/576], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [1/4]
Neighbor
10.40.1.3
N9396-1#
V
AS MsgRcvd MsgSent
4 65040
50
50
Remote VTEP
TblVer
26
InQ OutQ Up/Down State/PfxRcd

0
0 00:34:16 2
VxLAN BGP EVPN

Verifying MAC and ARP learnt in BGP
N9396-1# show bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 26, local router ID is 10.40.1.1
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i
njected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup
Network
Next Hop
Metric
LocPrf
Route Distinguisher: 10.40.1.1:36569
(L2VNI 400502)
*>i[2]:[0]:[0]:[48]:[8cb6.4f2f.45c2]:[0]:[0.0.0.0]/216
10.40.1.2
100
*>i[2]:[0]:[0]:[48]:[8cb6.4f2f.45c2]:[32]:[8.40.2.100]/272
10.40.1.2
100
< snip >
MAC entry
Weight Path
0 i
0 I
This entry indicates that an ARP has been

seen by the VTEP
VxLAN BGP EVPN

Verifying VRF Context and Next-Hop
N9396-1# show bgp l2vpn evpn received-paths
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 26, local router ID is 10.40.1.1
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i
njected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup
Network
Next Hop
Metric
LocPrf
(L2VNI 400501)
(L2VNI 400502)
Weight Path
Paths learned for each L2VNI indicating the

next hop VTEP
<snip>
(L3VNI 38000)
L3VNI corresponding to
the VRF context
VxLAN BGP EVPN

Verifying paths received and sent
N9396-1# show bgp l2vpn evpn neighbors 10.40.1.3
BGP neighbor is 10.40.1.3, remote AS 65040, ibgp link,
index 1
BGP version 4, remote router ID 10.40.1.3
BGP state = Established, up for 00:36:20
Using loopback0 as update source for this peer
Enable logging neighbor events
<snip>
For address family: L2VPN EVPN
BGP table version 26, neighbor version 26
2 accepted paths consume 152 bytes of memory
First Update Rcvd
: 00:08:15
Last Update Rcvd
: 00:08:15
2 sent paths
First Update Sent
: 00:09:24
Last Update Sent
: 00:09:23
<SNIP>
Peer
VxLAN BGP EVPN

Checking BGP Event History
N9396-1# show bgp event-history events | grep "8cb6.4f2f.45c2\|L2VPN"
<snip>
2015 Apr 27 02:09:33.208286 bgp 65040 [3312]: [3323]: (default) RIB: [L2VPN EVPN] Add/delete
10.40.1.1:36569:[2]:[0]:[0]:[48]:[8cb6.4f2f.45c2]:[0]:[0.0.0.0]/112, flags=0x200, old: (nil) new:
0xe3f605ec(10.40.1.2) in_rib: no
2015 Apr 27 02:09:33.208274 bgp 65040 [3312]: [3323]: (default) IMP: [L2VPN EVPN]
(10.40.1.1:36569:[2]:[0]:[0]:[48]:[8cb6.4f2f.45c2]:[0]:[0.0.0.0]/112 (<default>)): imported/updated one path,
reeval=0
2015 Apr 27 02:09:33.208262 bgp 65040 [3312]: [3323]: (default) IMP: [L2VPN EVPN] Created import destination
entry for 10.40.1.1:36569:[2]:[0]:[0]:[48]:[8cb6.4f2f.45c2]:[0]:[0.0.0.0]/112
2015 Apr 27 02:09:33.208255 bgp 65040 [3312]: [3323]: (default) IMP: [L2VPN EVPN] Importing prefix
10.40.1.2:36569:[2]:[0]:[0]:[48]:[8cb6.4f2f.45c2]:[0]:[0.0.0.0]/112
2015 Apr 27 02:09:33.208249 bgp 65040 [3312]: [3323]: (default) IMP: bgp_tbl_ctx_import: 1849: [L2VPN EVPN]
OK IMPORTING 10.40.1.2:36569:[2]:[0]:[0]:[48]:[8cb6.4f2f.45c2]:[0]:[0.0.0.0]/112 from RD 10.40.1.2:36569 to
RD 10.40.1.1:36569
2015 Apr 27 02:09:33.208241 bgp 65040 [3312]: [3323]: (default) IMP: bgp_tbl_ctx_import: 1640: [L2VPN EVPN]
Importing 10.40.1.2:36569:[2]:[0]:[0]:[48]:[8cb6.4f2f.45c2]:[0]:[0.0.0.0]/112 to RD 10.40.1.1:36569
<snip>
2015 Apr 27 02:09:33.208122 bgp 65040 [3312]: [3322]: (default) UPD: [L2VPN EVPN] Received
10.40.1.2:36569:[2]:[0]:[0]:[48]:[8cb6.4f2f.45c2]:[0]:[0.0.0.0] route from peer 10.40.1.3
MAC address 8cb6.4f2f.45c2 received from
VTEP with RD 10.40.1.2:36569 is imported
into RIB
Key Takeaways What have we learned ?
Key challenges while troubleshooting BGP
Various tools and techniques
Real TAC examples
Best practices in BGP
Learnt what impact can a BGP slow peer have
Troubleshooting BGP in MPLS VPN deployment
Introduced VxLAN BGP-EVPN and things to keep in mind while troubleshooting
Call to Action
Explore most common BGP problems that you faced in your network
Baseline your network resources (CPU, Memory, BGP Prefixes, TCAM,)
What changed in BGP? (New prefixes, route-maps, filters, peers)
Try to narrow down the problem with techniques we discussed
Collect as much information with available show commands during problematic

condition Helps faster resolution
Leverage scripting tools for sporadic problems (EEM, TCL,)
Enable event tracing helps in forensic investigation for RCA
Debug is a last resort. Be cautious and be specific (with filters)
Complete Your Online Session Evaluation
Give us your feedback to be

entered into a Daily Survey
Drawing. A daily winner
will receive a $750 Amazon
gift card.
Complete your session surveys

though the Cisco Live mobile
app or your computer on
Cisco Live Connect.
Dont forget: Cisco Live sessions will be available
for viewing on-demand after the event at
CiscoLive.com/Online
Participate in the My Favorite Speaker Contest

Promote Your Favorite Speaker and You Could Be a Winner
Promote your favorite speaker through Twitter and you could win $200 of Cisco
Press products (@CiscoPress)
Send a tweet and include

- Your favorite speakers Twitter handle - @vinugenie, @mani_cisco
- Two hashtags: #CLUS #MyFavoriteSpeaker
You can submit an entry for more than one of your favorite speakers
Dont forget to follow @CiscoLive and @CiscoPress
View the official rules at http://bit.ly/CLUSwin
Continue Your Education
Demos in the Cisco campus
Walk-in Self-Paced Labs
Table Topics
Meet the Engineer 1:1 meetings
Related sessions
Q&A
Appendix
Scenario 9 BGP RT Constraint Filtering

Problem Description
Customer reported that their resource utilization on edge routers have increased
with the growth of their SP network where as no extra services has been
deployed on those edge devices
Symptoms
Unwanted resource utilization

Performance issues
PEn
PE(n-1)
PE2
CE1
PE1
CE2
RR
AS100
PE3
CE3
BGP RT Constraint Filtering

Overview
PE routers use Route Target (RT) extended communities to control the

distribution of routes into VRFs.
Ideally PE routers need to hold only routes marked with Route Targets
pertaining to VRFs that have local CE attachments.
When the distribution of VPNs is sparse, there is wastage of resources in

maintaining the unwanted routes at the PE routers and unnecessary distribution
of routes by the route-reflectors.
Route Target (RT)-constraint is a mechanism to prevent the propagation of VPN

NLRI to a PE that is not interested in the VPN.

Un-wanted Routes at RR & PE
VRF- Green
VRF- Purple
VRF- Blue
VRF- Red
Unwanted VRF routes for PE
RR-1
VRF- Red
VRF- Green
RR-2
Unwanted VRF routes for RR
VRF- Purple
VRF- Blue
PE-3 and PE-4 advertise VRF Blue, red and green VPN routes to RR-1
RR-1 send ALL its VPN routes to RR-2.
VRF-Red routes are really unwanted on RR-2 since PE-1 and PE-2 does not have VRF Purple.

Concept
By having BGP speakers exchanging the wanted Route Targets, this allows
BGP speaker to eliminate advertising unwanted VPN routes to its peer.
The wanted Route Targets are called RT membership.
MP-BGP UPDATE message to propagate RT membership information.

- Peers to advertise their RT membership.
- Restrict advertisement of VPN route based on received RT membership information.
Perform Constraint Route Distribution on VPN v4 and v6 Route advertisements

only

Enable RT Constraint on RR
RR-1#sh run | b router bgp
router bgp 65000
address-family vpnv4
neighbor 20.0.0.2 send-community both
neighbor 20.0.0.2 route-reflector-client
!
address-family rtfilter
neighbor 20.0.0.2 route-reflector-client
exit-address-family
20.0.0.2
20.0.0.1
PE-2
RR-1

Enable RT Constraint on PE
PE-2#sh run | b router bgp
router bgp 65000
address-family vpnv4
exit-address-family
!
address-family rtfilter
exit-address-family
20.0.0.2
20.0.0.1
PE-2
RR-1
RT Constraint Capability Exchange

20.0.0.2
PE-2
20.0.0.1
RR-1
*Apr 29 01:18:34.658: BGP: 20.0.0.1

active OPEN has CAPABILITY code: 1,
length 4
*Apr 29 01:16:24.896: BGP: 20.0.0.2

passive OPEN has CAPABILITY code: 1,
length 4
*Apr 29 01:18:34.658: BGP: 20.0.0.1

active OPEN has MP_EXT CAP for
afi/safi: 1/132
*Apr 29 01:16:24.896: BGP: 20.0.0.2

passive OPEN has MP_EXT CAP for
afi/safi: 1/132
*Apr 29 01:18:34.658: BGP: 20.0.0.1

accept RTC SAFI
*Apr 29 01:16:24.897: BGP: 20.0.0.2

accept RTC SAFI

Exchanging RTC between PE and RR
RT-Constraint:
RT-Constraint:
NLRI = {RT 1, RT 2}
NLRI = {RT 2, RT 3}
VRF- Red RT 2
VRF- Blue RT 1
VRF- Red RT 2
PE-3
PE-4
RR-1
Outbound Filter:
Outbound Filter:
Permit RT 1
Permit RT 3
Permit RT 2
Permit RT 2
Deny all
Deny all
1. PE-3 send RTC NLRI {RT 1, RT 2} to RR-1
2. PE-4 sends RTC NLRI {RT 2, RT 3} to RR-1

3. RR-1 install an outbound Filter (Permit RT 1, RT 2) for PE-3
4. RR-1 installs an outbound Filter (Permit RT 2, RT 3) for PE-4
VRF- Green RT 3
Thank you

Brkrst-3320 BGP Cisco Live

Uploaded by

Copyright:

Available Formats

Brkrst-3320 BGP Cisco Live

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Brkrst-3320 BGP Cisco Live

Uploaded by

Copyright:

Available Formats

Troubleshooting BGP

Who are you?

Why should you care?

Most widely deployed protocol - literally makes the internet work

Impact of BGP going down is huge

Understanding the challenges for troubleshooting BGP

Break down and simplify the problem statement

Various tools and techniques

Best practices for troubleshooting BGP

Real life TAC examples

IOS vs. IOS-XR vs. NX-OS

Troubleshooting concepts / logic are the same

Some variation in show command syntax and output

Will use all three in this presentation

BGP peering issues

session not coming up, dynamic peering, session Flapping

Platform issues due to BGP

High CPU, Memory utilization

BGP for Enterprise

BGP for Service Providers

MPLS VPN, BGP PIC

BGP for Datacenter

BRKRST-3320 - Troubleshooting BGP

BGP peering issues

session not coming up, dynamic peering, session Flapping

Platform issues due to BGP

High CPU, Memory utilization

BGP for Enterprise

BGP for Service Providers

BGP for Datacenter

Scenario 1 - Failed BGP Peering

iBGP / eBGP is not establishing

Newly configured BGP session not coming up

Session was up before, but not coming up now

Failed BGP peering

router bgp 65535

router bgp 65534

Failed BGP peering

Packet sent with a source address of 10.12.12.1

Failed BGP peering

access-list OUT extended permit tcp any any eq telnet

Rn# sh ip access-list R1_R2

permit icmp any any

permit tcp host 10.12.12.1 eq bgp 2.2.2.2

Failed BGP peering

Quick test when BGP is down

This means BGP Packets are being blocked between R1 and R2

Failed BGP peering

BGP process is in Run

for blocked BGP or

RP/0/RSP0/CPU0:ASR9010-B# show process bgp

Failed BGP peering

Failed BGP peering

Use SPAN to get traffic to your sniffer

Failed BGP peering

Failed BGP peering

Embedded Packet Capture

Network Processor Capture