Discovery 3: Troubleshoot the MPLS

VPN Backbone

Troubleshoot the MPLS VPN Backbone

In this lab, you will detect, analyze, and solve issues that affect the operation of a
service provider backbone and disrupt the connectivity between provider edge (PE in
topology) and provider backbone (P in topology) devices. Issues to solve are
categorized into three main areas:

 IGP operation (OSPF protocol)

 MPLS operation (LDP protocol)
 BGP operation (IBGP sessions inside the service provider backbone)

You will use commands and tools to first verify that the service provider network has
connectivity problems. Then, based on your networking and operational experience, and
by using commands and troubleshooting tools, you will find the root cause of every
detected issue and apply the corrective actions that are necessary to restore network
connectivity and return the service provider backbone to normal operation.

Detect, Analyze, and Solve OSPF Routing Issues in

PE1 and PE3 Routers
In this scenario, all end customers that attach to the PE1 and PE3 routers claim that
they lost connectivity to their VPN remote locations. Level 1 support has investigated
the situation and found that local connectivity to the end customers in PE1 and PE3 is
completely operational. However, the Level 1 support team has informed you that they
do not see IP routing information that relates to the internal service provider backbone
in PE1 and PE3. Your task to make sure that this report is accurate and, if it is, to
analyze and find the root causes for the issues and solve them to restore backbone
connectivity.

Step 1

Begin by troubleshooting the PE1 router. Click the PE1 router to open a serial console
session.

Step 2

Press Enter and then provide login information to enter privileged EXEC mode. Use
the access credentials that are provided in Job Aid.

Answer
User Access Verification

Username: cisco
Password: cisco

Step 3

Check if PE1 has an operational OSPF neighbor relationship with the P1 and P2
routers.

Note
Throughout the lab, you will see log messages from the operating system. Although
they may help you to know the reason for problems, ignore these messages and follow
the troubleshooting steps in this guide. They will allow you to analyze the problems in a
predefined order.

Answer

Use the show ospf neighbor command to verify the OSPF neighbor list and state.
Observe that PE1 has no neighbors.

RP/0/RP0/CPU0:PE1#show ospf neighbor

Mon Jul 22 20:21:53.486 UTC
RP/0/RP0/CPU0:PE1#

Step 4

Use a verification command on PE1 to check the OSPF status for the interface that
connects to P1. Default values for timers should be in use in the network. Is the
information that displays correct? Do you see any incorrect parameter values?

Note
Roles for the router in your pod (DR or BDR) may vary.

Answer

Use the show ospf interface Gi0/0/0/1 command to verify OSPF status and
operational parameters. Pay attention to reported parameters, such as area ID, process
ID, router ID, and timers. Do they match what you expect?

RP/0/RP0/CPU0:PE1#show ospf interface Gi0/0/0/1

Mon Jul 22 20:38:39.466 UTC

GigabitEthernet0/0/0/1 is up, line protocol is up

Internet Address 192.168.111.1/24, Area 0
Label stack Primary label 3 Backup label 3 SRTE label 6
Process ID 1, Router ID 10.1.1.1, Network Type BROADCAST, Cost: 1
Transmit Delay is 1 sec, State DR, Priority 1, MTU 1500, MaxPktSz
1500
Forward reference No, Unnumbered no, Bandwidth 10000000
 Designated Router (ID) 10.1.1.1, Interface address 192.168.111.1
No backup designated router on this network
Timer intervals configured, Hello 5, Dead 20, Wait 20, Retransmit 5
Hello due in 00:00:01:539
Index 2/2, flood queue length 0
Next 0(0)/0(0)
Last flood scan length is 0, maximum is 0
Last flood scan time is 0 msec, maximum is 0 msec
LS Ack List: current length 0, high water mark 0
Neighbor Count is 0, Adjacent neighbor count is 0
Suppress hello for 0 neighbor(s)
Multi-area interface Count is 0
RP/0/RP0/CPU0:Jul 25 13:39:03.568 UTC: ospf[1023]: %ROUTING-OSPF-4-
ERRRCV : Received invalid packet: mismatch area ID, from backbone area
must be virtual-link but not found from 192.168.121.12,
GigabitEthernet0/0/0/2

Step 5

On the PE1 router, use a verification command to check the OSPF status for the
interface that connects to P2. Default values for timers should be in use in the network.
Is the information that displays correct? Do you see any incorrect parameter values?

Note
Roles for the routers in your pod (DR or BDR) may vary.

Answer

Use the show ospf interface Gi0/0/0/2 command to verify OSPF status and
operational parameters. Pay attention to reported parameters, such as area ID, process
ID, router ID, and timers. Do they match what you expect?

RP/0/RP0/CPU0:PE1#show ospf interface Gi0/0/0/2

Tue Jul 23 12:38:39.323 UTC

GigabitEthernet0/0/0/2 is up, line protocol is up

Internet Address 192.168.121.1/24, Area 1
Label stack Primary label 3 Backup label 3 SRTE label 6
Process ID 1, Router ID 10.1.1.1, Network Type BROADCAST, Cost: 1
Transmit Delay is 1 sec, State DR, Priority 1, MTU 1500, MaxPktSz
1500
Forward reference No, Unnumbered no, Bandwidth 10000000
Designated Router (ID) 10.1.1.1, Interface address 192.168.121.1
No backup designated router on this
network
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
Hello due in 00:00:05:942
Index 1/3, flood queue length 0
Next 0(0)/0(0)
Last flood scan length is 0, maximum is 0
Last flood scan time is 0 msec, maximum is 0 msec
LS Ack List: current length 0, high water mark 0
Neighbor Count is 0, Adjacent neighbor count is 0
Suppress hello for 0 neighbor(s)
Multi-area interface Count is 0
RP/0/RP0/CPU0:Jul 25 13:39:03.568 UTC: ospf[1023]: %ROUTING-OSPF-4-
ERRRCV : Received invalid packet: mismatch area ID, from backbone area
must be virtual-link but not found from 192.168.121.12,
GigabitEthernet0/0/0/2

Step 6

Use a verification command on PE1 to display information about the OSPF processes
that run on the PE1 router. The backbone should be configured for single area OSPF. Is
the information that displays correct? Do you see any incorrect configurations?

Answer

Use the show protocols ospf command to display information about the OSPF
processes that run on the PE1 router. Pay attention to the areas and the interfaces that
are assigned to them. Do they match what you expect?

RP/0/RP0/CPU0:PE1#show protocols ospf

Tue Jul 23 12:51:12.156 UTC

Routing Protocol OSPF 1

Router Id: 10.1.1.1
Distance: 110
Non-Stop Forwarding: Disabled
Distribute list in: SUMFLT
Redistribution:
None
Area 0
Loopback0
GigabitEthernet0/0/0/1
Area 1
GigabitEthernet0/0/0/2

Step 7

Use a verification command to display information about the OSPF running

configuration on the PE1 router. Is the information that displays correct? Do you see
any incorrect configurations?

Answer

Use the show running-config router ospf command to display information about
the OSPF running configuration on the PE1 router. Pay attention to the areas and
interfaces that are assigned to them. Also look for any unexpected parameter values that
affect OSPF neighbor adjacency. Do they match what you expect?

RP/0/RP0/CPU0:PE1#show running-config router ospf

Tue Jul 23 12:53:10.234 UTC
router ospf 1
router-id 10.1.1.1
distribute-list SUMFLT in
address-family ipv4
area 0
interface Loopback0
!
 interface GigabitEthernet0/0/0/1
dead-interval 20
hello-interval 5
!
!
area 1
interface GigabitEthernet0/0/0/2
!
!
!

Step 8

Based on output from the verification commands, formulate your hypothesis about what
issues are preventing PE1 from establishing OSPF full adjacency with P1 and P2.

Note
You can compare your diagnostic with actual root causes by clicking the SHOW ME
button in this step.

Answer

From the process of using the verification commands, it should be evident that two
issues are preventing PE1 to form OSPF adjacency with P1 and P2:

1. Incorrect timers for the hello-interval and dead-interval were configured for the
interface that faces P1, as all devices in this design use default hello and dead
intervals.
2. An incorrect OSPF area (Area 1) addition to the PE1 OSPF configuration
conflicts with the single-area OSPF design. Furthermore, interface
GigabitEthernet0/0/0/2, which connects to P2, was incorrectly assigned to the
additional Area 1.

Step 9

On the PE1 router, correct the first issue, which affects OPSF adjacency with P1.
Remember to apply the changes. Return to OSPF configuration mode.

Answer

Use the configure terminal command to enter global configuration mode. Enter
OSPF configuration mode for process ID 1 by typing router ospf 1. Use the area 0
command followed by interface GigabitEthernet0/0/0/1 to enter configuration
mode for that interface in Area 0. Type the hello-interval 10 command followed by
the dead-interval 40 command to set both timers at their default values. Use the
commit command to apply the changes. Type exit twice to return to OSPF
configuration mode.

RP/0/RP0/CPU0:PE1#configure terminal
Tue Jul 23 13:03:37.104 UTC
RP/0/RP0/CPU0:PE1(config)#router ospf 1
RP/0/RP0/CPU0:PE1(config-ospf)#area 0
RP/0/RP0/CPU0:PE1(config-ospf-ar)#interface GigabitEthernet0/0/0/1
RP/0/RP0/CPU0:Jul 23 13:03:49.322 UTC: ospf[1023]: %ROUTING-OSPF-4-
ERRRCV : Received invalid packet: mismatch area ID, from backbone area
must be virtual-link but not found from 192.168.121.12,
GigabitEthernet0/0/0/2
RP/0/RP0/CPU0:PE1(config-ospf-ar-if)#hello-interval 10
RP/0/RP0/CPU0:PE1(config-ospf-ar-if)#dead-interval 40
RP/0/RP0/CPU0:PE1(config-ospf-ar-if)#commit
Tue Jul 23 13:04:16.091 UTC
RP/0/RP0/CPU0:Jul 23 13:04:17.048 UTC: config[68598]: %MGBL-CONFIG-6-
DB_COMMIT : Configuration committed by user 'cisco'. Use 'show
configuration commit changes 1000034192' to view the changes.
RP/0/RP0/CPU0:PE1(config-ospf-ar-if)#RP/0/RP0/CPU0:Jul 23 13:04:25.643
UTC: ospf[1023]: %ROUTING-OSPF-5-ADJCHG : Process 1, Nbr 10.11.11.11
on GigabitEthernet0/0/0/1 in area 0 from LOADING to FULL, Loading
Done, vrf default vrfid 0x60000000
RP/0/RP0/CPU0:Jul 23 13:04:26.426 UTC: ospf[1023]: %ROUTING-OSPF-4-
ERRRCV : Received invalid packet: mismatch area ID, from backbone area
must be virtual-link but not found from 192.168.121.12,
GigabitEthernet0/0/0/2
RP/0/RP0/CPU0:Jul 23 13:04:45.593 UTC: ospf[1023]: %ROUTING-OSPF-4-
ERRRCV : Received invalid packet: mismatch area ID, from backbone area
must be virtual-link but not found from 192.168.121.12,
GigabitEthernet0/0/0/2
RP/0/RP0/CPU0:Jul 23 13:05:04.954 UTC: ospf[1023]: %ROUTING-OSPF-4-
ERRRCV : Received invalid packet: mismatch area ID, from backbone area
must be virtual-link but not found from 192.168.121.12,
GigabitEthernet0/0/0/2
RP/0/RP0/CPU0:PE1(config-ospf-ar-if)#exit
RP/0/RP0/CPU0:PE1(config-ospf-ar)#exit

Step 10

Correct the second issue, which affects OPSF adjacency with the P2 router.

Answer

At this point, you should be in OSPF configuration mode. Type no area 1 to delete
Area 1 from the OSPF configuration. Use the area 0 command and then type the
interface GigabitEthernet0/0/0/2 command to associate the interface to OSPF
Area 0. Use the commit command to apply the changes. Type end to exit configuration
mode.

RP/0/RP0/CPU0:PE1(config-ospf)#no area 1
RP/0/RP0/CPU0:PE1(config-ospf)#area 0
RP/0/RP0/CPU0:PE1(config-ospf-ar)#RP/0/RP0/CPU0:Jul 23 13:10:26.642
UTC: ospf[1023]: %ROUTING-OSPF-4-ERRRCV : Received invalid packet:
mismatch area ID, from backbone area must be virtual-link but not
found from 192.168.121.12, GigabitEthernet0/0/0/2
RP/0/RP0/CPU0:PE1(config-ospf-ar)#interface GigabitEthernet0/0/0/2
RP/0/RP0/CPU0:PE1(config-ospf-ar-if)#RP/0/RP0/CPU0:Jul 23 13:11:04.636
UTC: ospf[1023]: %ROUTING-OSPF-4-ERRRCV : Received invalid packet:
mismatch area ID, from backbone area must be virtual-link but not
found from 192.168.121.12, GigabitEthernet0/0/0/2
RP/0/RP0/CPU0:PE1(config-ospf-ar-if)#commit
Tue Jul 23 13:11:15.889 UTC
RP/0/RP0/CPU0:Jul 23 13:11:16.786 UTC: config[68598]: %MGBL-CONFIG-6-
DB_COMMIT : Configuration committed by user 'cisco'. Use 'show
configuration commit changes 1000034193' to view the changes.
RP/0/RP0/CPU0:PE1(config-ospf-ar-if)#RP/0/RP0/CPU0:Jul 23 13:11:23.367
UTC: ospf[1023]: %ROUTING-OSPF-5-ADJCHG : Process 1, Nbr 10.12.12.12
on GigabitEthernet0/0/0/2 in area 0 from LOADING to FULL, Loading
Done, vrf default vrfid 0x60000000
RP/0/RP0/CPU0:PE1(config-ospf-ar-if)#end

Step 11

Verify that PE1 now has fully operational OSPF adjacencies with P1 and P2.

Note
Roles for routers in your pod (DR or BDR) may vary.

Answer

Use the show ospf neighbor command to verify the OSPF neighbor list and state.
Observe that two neighbors report with FULL as the adjacency status in the State
column.

RP/0/RP0/CPU0:PE1#show ospf neighbor

Tue Jul 23 13:29:52.235 UTC

* Indicates MADJ interface

# Indicates Neighbor awaiting BFD session up

Neighbors for OSPF 1

Neighbor ID Pri State Dead Time Address

Interface
10.11.11.11 1 FULL/DR 00:00:35 192.168.111.11
GigabitEthernet0/0/0/1
Neighbor is up for 00:01:26
10.12.12.12 1 FULL/DR 00:00:33 192.168.121.12
GigabitEthernet0/0/0/2
Neighbor is up for 00:00:39

Total neighbor count: 2

Step 12

Continue troubleshooting the PE3 router. Click PE3 to open a serial console session.

Step 13

Press Enter and then provide login information to enter privileged EXEC mode. Use
the access credentials that are provided in Job Aid.

Answer
User Access Verification
Username: cisco
Password: cisco

Step 14

Check if PE3 has an operational OSPF neighbor relationship with the P1 and P2
routers.

Note
The neighbor state and role on your pod may vary. The State column may vary, as
OSPF transitions through several phases while establishing adjacency.

Answer

Use the show ip ospf neighbor command to verify the OSPF neighbor list and state.
Observe that PE3 has only one neighbor, and it is not in the FULL state.

PE3#show ip ospf neighbor

Neighbor ID Pri State Dead Time Address

Interface
10.12.12.12 1 INIT/DROTHER 00:00:35 192.168.123.12
GigabitEthernet2

Step 15

On the PE3 router, use a verification command to check the OSPF status for the
interface that connects to P1. Is the information that displays correct? Do you see any
incorrect parameter values?

Note
Roles for routers in your pod (DR or BDR) may vary.

Answer

Use the show ip ospf interface Gi1 command to verify OSPF status and
operational parameters. Pay attention to reported parameters, such as area ID, process
ID, router ID, and timers. Do they match what you expect?

PE3#show ip ospf interface Gi1

GigabitEthernet1 is up, line protocol is up
Internet Address 192.168.113.3/24, Interface ID 19, Area 0
Attached via Network Statement
Process ID 1, Router ID 10.11.11.11, Network Type BROADCAST, Cost: 1
Topology-MTID Cost Disabled Shutdown Topology Name
0 1 no no Base
Transmit Delay is 1 sec, State DR, Priority 1
Designated Router (ID) 10.11.11.11, Interface address 192.168.113.3
No backup designated router on this network
 Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
oob-resync timeout 40
Hello due in 00:00:08
Supports Link-local Signaling (LLS)
Cisco NSF helper support enabled
IETF NSF helper support enabled
Can be protected by per-prefix Loop-Free FastReroute
Can be used for per-prefix Loop-Free FastReroute repair paths
Not Protected by per-prefix TI-LFA
Index 1/2/2, flood queue length 0
Next 0x0(0)/0x0(0)/0x0(0)
Last flood scan length is 0, maximum is 0
Last flood scan time is 0 msec, maximum is 0 msec
Neighbor Count is 0, Adjacent neighbor count is 0
Suppress hello for 0 neighbor(s)

Step 16

Use a verification command on PE3 to check the OSPF status for the interface that
connects to P2. Default timers should be in use in the network. Is the information that
displays correct? Do you see any incorrect parameter values?

Note
Roles for routers in your pod (DR or BDR) may vary.

Answer

Use the show ip ospf interface Gi2 command to verify OSPF status and
operational parameters. Pay attention to reported parameters, such as area ID, process
ID, router ID, and timers. Do they match what you expect?

PE3#show ip ospf interface Gi2

GigabitEthernet2 is up, line protocol is up
Internet Address 192.168.123.3/24, Interface ID 20, Area 0
Attached via Network Statement
Process ID 1, Router ID 10.11.11.11, Network Type BROADCAST, Cost:
10
Topology-MTID Cost Disabled Shutdown Topology Name
0 10 no no Base
Transmit Delay is 1 sec, State DR, Priority 1
Designated Router (ID) 10.11.11.11, Interface address 192.168.123.3
No backup designated router on this network
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
oob-resync timeout 40
Hello due in 00:00:08
Supports Link-local Signaling (LLS)
Cisco NSF helper support enabled
IETF NSF helper support enabled
Can be protected by per-prefix Loop-Free FastReroute
Can be used for per-prefix Loop-Free FastReroute repair paths
Not Protected by per-prefix TI-LFA
Index 1/3/3, flood queue length 0
Next 0x0(0)/0x0(0)/0x0(0)
Last flood scan length is 0, maximum is 0
Last flood scan time is 0 msec, maximum is 0 msec
Neighbor Count is 1, Adjacent neighbor count is 0
 Suppress hello for 0 neighbor(s)

Step 17

Use a verification command to display information about the OSPF processes that run
on the PE3 router. All routers should use the IP address on the Loopback 0 interfaces as
the OSPF router ID. Is the information that displays correct? Do you see any incorrect
configurations?

Answer

Use the show ip protocols | section ospf command to display information about
the OSPF processes that run on the PE3 router. Pay attention to the router ID, areas, and
the interfaces that are assigned to them. Do they match what you expect?

PE3#show ip protocols | section ospf

Routing Protocol is "ospf 1"
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
Router ID 10.11.11.11
Number of areas in this router is 1. 1 normal 0 stub 0 nssa
Maximum path: 4
Routing for Networks:
10.3.3.3 0.0.0.0 area 0
192.168.113.3 0.0.0.0 area 0
192.168.123.3 0.0.0.0 area 0
Routing Information Sources:
Gateway Distance Last Update
Distance: (default is 110)

Step 18

Use a verification command to display information about the OSPF running

configuration on the PE3 router. All routers should use the IP address on the Loopback
0 interface as the OSPF router ID. Is the information that displays correct? Do you see
any incorrect configurations?

Answer

Use the show running-config | section router ospf command to display

information about the OSPF running configuration on the PE3 router. Pay attention to
areas and the interfaces that are assigned to them. Also look for any unexpected
parameter values that may affect OSPF neighbor adjacency. Do they match what you
expect?

PE3#show running-config | section router ospf

router ospf 1
router-id 10.11.11.11
network 10.3.3.3 0.0.0.0 area 0
network 192.168.113.3 0.0.0.0 area 0
network 192.168.123.3 0.0.0.0 area 0

Step 19
On the PE3 router, debug OSPF events. Also debug OSPF adjacencies. Look for
messages that indicate the root causes of the issues that prevent adjacencies to establish.
After a few minutes, turn off debugging and analyze the output.

Answer

Use the debug ip ospf events and the debug ip ospf adj commands to turn on
debugging of OSPF events and adjacencies, respectively. Analyze the output. To turn
off all possible debugs, use the undebug all command.

PE3#debug ip ospf events

PE3#debug ip ospf adj
*Jul 29 11:16:19.799: OSPF-1 ADJ Gi2: Rcv DBD from 10.12.12.12 seq
0x1E3EDA11 opt 0x52 flag 0x7 len 32 mtu 1500 state INIT
*Jul 29 11:16:19.799: OSPF-1 ADJ Gi2: Nbr 10.12.12.12 has smaller
interface MTU
*Jul 29 11:16:19.799: OSPF-1 ADJ Gi2: 2 Way Communication to
10.12.12.12, state 2WAY
*Jul 29 11:16:19.799: OSPF-1 ADJ Gi2: Neighbor change event
*Jul 29 11:16:19.800: OSPF-1 ADJ Gi2: DR/BDR election
*Jul 29 11:16:19.800: OSPF-1 ADJ Gi2: Elect BDR 0.0.0.0
*Jul 29 11:16:19.800: OSPF-1 ADJ Gi2: Elect DR 10.12.12.12
*Jul 29 11:16:19.800: OSPF-1 ADJ Gi2: Elect BDR 10.11.11.11
*Jul 29 11:16:19.800: OSPF-1 ADJ Gi2: Elect DR 10.12.12.12
*Jul 29 11:16:19.800: OSPF-1 ADJ Gi2: DR: 10.12.12.12 (Id)
*Jul 29 11:16:19.800: OSPF-1 ADJ Gi2: BDR: 10.11.11.11 (Id)
*Jul 29 11:16:19.800: OSPF-1 ADJ Gi2: Nbr 10.12.12.12: Prepare dbase
exchange
*Jul 29 11:16:19.801: OSPF-1 ADJ Gi2: Send DBD to 10.12.12.12 seq
0x5AB8C83C opt 0x52 flag 0x7 len 32
*Jul 29 11:16:19.801: OSPF-1 ADJ Gi2: Set flush timer
*Jul 29 11:16:19.801: OSPF-1 ADJ Gi2: Remember old DR 10.11.11.11
(id)
*Jul 29 11:16:19.801: OSPF-1 ADJ Gi2: NBR Negotiation Done. We are
the SLAVE
PE3#
*Jul 29 11:16:19.801: OSPF-1 ADJ Gi2: Nbr 10.12.12.12: Summary list
built, size 1
*Jul 29 11:16:19.801: OSPF-1 ADJ Gi2: Send DBD to 10.12.12.12 seq
0x1E3EDA11 opt 0x52 flag 0x2 len 52
PE3#
*Jul 29 11:16:21.937: %OSPF-4-DUP_RTRID_NBR: OSPF detected duplicate
router-id 10.11.11.11 from 192.168.113.11 on interface
GigabitEthernet1
PE3#
*Jul 29 11:16:24.446: OSPF-1 ADJ Gi2: Rcv DBD from 10.12.12.12 seq
0x1E3EDA11 opt 0x52 flag 0x7 len 32 mtu 1500 state EXCHANGE
*Jul 29 11:16:24.446: OSPF-1 ADJ Gi2: Nbr 10.12.12.12 has smaller
interface MTU
*Jul 29 11:16:24.446: OSPF-1 ADJ Gi2: Send DBD to 10.12.12.12 seq
0x1E3EDA11 opt 0x52 flag 0x2 len 52
PE3#
*Jul 29 11:16:29.018: OSPF-1 ADJ Gi2: Rcv DBD from 10.12.12.12 seq
0x1E3EDA11 opt 0x52 flag 0x7 len 32 mtu 1500 state EXCHANGE
*Jul 29 11:16:29.018: OSPF-1 ADJ Gi2: Nbr 10.12.12.12 has smaller
interface MTU
*Jul 29 11:16:29.018: OSPF-1 ADJ Gi2: Send DBD to 10.12.12.12 seq
0x1E3EDA11 opt 0x52 flag 0x2 len 52
PE3#
*Jul 29 11:16:33.807: OSPF-1 ADJ Gi2: Rcv DBD from 10.12.12.12 seq
0x1E3EDA11 opt 0x52 flag 0x7 len 32 mtu 1500 state EXCHANGE
*Jul 29 11:16:33.807: OSPF-1 ADJ Gi2: Nbr 10.12.12.12 has smaller
interface MTU
*Jul 29 11:16:33.807: OSPF-1 ADJ Gi2: Send DBD to 10.12.12.12 seq
0x1E3EDA11 opt 0x52 flag 0x2 len 52
PE3#
*Jul 29 11:16:38.553: OSPF-1 ADJ Gi2: Rcv DBD from 10.12.12.12 seq
0x1E3EDA11 opt 0x52 flag 0x7 len 32 mtu 1500 state EXCHANGE
*Jul 29 11:16:38.553: OSPF-1 ADJ Gi2: Nbr 10.12.12.12 has smaller
interface MTU
*Jul 29 11:16:38.553: OSPF-1 ADJ Gi2: Send DBD to 10.12.12.12 seq
0x1E3EDA11 opt 0x52 flag 0x2 len 52
*Jul 29 11:16:38.896: OSPF-1 ADJ Gi2: Neighbor change event
*Jul 29 11:16:38.896: OSPF-1 ADJ Gi2: DR/BDR election
*Jul 29 11:16:38.896: OSPF-1 ADJ Gi2: Elect BDR 10.11.11.11
*Jul 29 11:16:38.896: OSPF-1 ADJ Gi2: Elect DR 10.12.12.12
*Jul 29 11:16:38.896: OSPF-1 ADJ Gi2: DR: 10.12.12.12 (Id)
PE3#
*Jul 29 11:16:38.897: OSPF-1 ADJ Gi2: BDR: 10.11.11.11 (Id)
PE3#un al
*Jul 29 11:16:43.352: OSPF-1 ADJ Gi2: Rcv DBD from 10.12.12.12 seq
0x1E3EDA11 opt 0x52 flag 0x7 len 32 mtu 1500 state EXCHANGE
*Jul 29 11:16:43.352: OSPF-1 ADJ Gi2: Nbr 10.12.12.12 has smaller
interface MTU
*Jul 29 11:16:43.352: OSPF-1 ADJ Gi2: Send DBD to 10.12.12.12 seq
0x1E3EDA11 opt 0x52 flag 0x2 len 52 PE3#undebug all
All possible debugging has been turned off

Step 20

Check the operational details on the PE3 router for interfaces that connect to the P1 and
P2 routers. MTU on all routers should be configured for 1500 bytes. Do you see any
unexpected values?

Answer

Use the show interfaces Gi1 and show interfaces Gi2 commands to verify details
in the interfaces that connect to neighboring routers, including IP addressing and MTU.

PE3#show interfaces Gi1

GigabitEthernet1 is up, line protocol is up
Hardware is 12xGE-2x10GE-FIXED, address is 7488.bba6.880c (bia
7488.bba6.880c)
Description: Link to P1 GigabitEthernet0/0/0/3
Internet address is 192.168.113.3/24
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 10000Mbps, link type is auto, media type is H10GB-CU1M
output flow-control is unsupported, input flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:02, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops:
0
Queueing strategy: fifo
 Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
2735 packets input, 233290 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
699 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 2735 multicast, 0 pause input
27417 packets output, 2926038 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

PE3#show interfaces Gi2

GigabitEthernet2 is up, line protocol is up
Hardware is 12xGE-2x10GE-FIXED, address is 7488.bba6.880d (bia
7488.bba6.880d)
Description: Link to P2 GigabitEthernet0/0/0/3
Internet address is 192.168.123.3/24
MTU 9216 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 10000Mbps, link type is auto, media type is H10GB-CU1M
output flow-control is unsupported, input flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:02, output 00:00:02, output hang never
Last clearing of "show interface" counters never
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops:
0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
4343 packets input, 369193 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
299 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 3929 multicast, 0 pause input
28518 packets output, 3032696 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

Step 21

Based on output from verification commands, formulate your hypothesis about what
issues are preventing PE3 to establish OSPF full adjacency with P1 and P2.

Note
You can compare your diagnostic with actual root causes by clicking the SHOW ME
button in this step.
Answer

By running these verification commands, it should be evident that two issues are
preventing PE3 to form OSPF adjacency with P1 and P2:

1. PE3 was configured with an incorrect and duplicated router ID for OSPF. It is
using the router ID for P1.
2. The interface that connects to P2 was configured with an MTU value that differs
from the one that P2 uses (1500 bytes).

Step 22

Correct the first issue, which affects OPSF adjacency with the P1 router. Restart the
OSPF process. Return to global configuration mode.

Answer

Use the configure terminal command to enter global configuration mode. Enter
OSPF configuration mode for process IP 1 by typing the router ospf 1 command.
Use the router-id 10.3.3.3 command to set the router ID. To force the OSPF
process to restart, use the do clear ip ospf process command. Answer yes to
reload the process. Type exit to return to global configuration mode.

PE3#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
PE3(config)#router ospf 1
PE3(config-router)#router-id 10.3.3.3
% OSPF: Reload or use "clear ip ospf process" command, for this to
take effect
PE3(config-router)#
*Jul 23 07:04:35.161: %OSPF-4-DUP_RTRID_NBR: OSPF detected duplicate
router-id 10.11.11.11 from 192.168.113.11 on interface
GigabitEthernet1
PE3(config-router)#do clear ip ospf process
Reset ALL OSPF processes? [no]: yes
PE3(config-router)#
*Jul 23 07:04:58.824: %OSPF-5-ADJCHG: Process 1, Nbr 10.12.12.12 on
GigabitEthernet2 from EXCHANGE to DOWN, Neighbor Down: Interface down
or detached
PE3(config-router)#
*Jul 23 07:05:08.191: %OSPF-5-ADJCHG: Process 1, Nbr 10.11.11.11 on
GigabitEthernet1 from LOADING to FULL, Loading Done
PE3(config-router)#
*Jul 23 07:05:12.339: %LDP-5-NBRCHG: LDP Neighbor 10.11.11.11:0 (1) is
UP
PE3(config-router)#
*Jul 23 07:05:14.412: %LDP-5-NBRCHG: LDP Neighbor 10.12.12.12:0 (2) is
UP
PE3(config-router)#exit

Step 23
Correct the second issue, which affects OPSF adjacency with the P2 router, and exit
global configuration mode. After some seconds, you should see a log message about
restoring the OSPF sessions.

Note
You may see a log message with the text: %TCP-6-BADAUTH. This alert relates to
the BGP process. Please ignore this warning. You will analyze it in later tasks.

Answer

Type the interface Gi2 command. Type the mtu 1500 command to restore the
default value for MTU in that interface. Type end to exit configuration mode.

PE3(config)#interface Gi2
PE3(config-if)#mtu 1500
PE3(config-if)#end
*Jul 23 07:14:02.935: %OSPF-5-ADJCHG: Process 1, Nbr 10.12.12.12 on
GigabitEthernet2 from LOADING to FULL, Loading Done
*Jul 23 07:14:59.532: %TCP-6-BADAUTH: No MD5 digest from
10.12.12.12(26127) to 10.3.3.3(179) tableid - 0

Step 24

Verify that PE3 now has fully operational OSPF adjacencies with P1 and P2.

Note
Roles for routers in your pod (DR or BDR) may vary.

Answer

Use the show ip ospf neighbor command to verify the OSPF neighbor list and state.
Observe that two neighbors report with FULL as the adjacency status in the State
column.

PE3#show ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface

10.12.12.12 1 FULL/DR 00:00:32 192.168.123.12
GigabitEthernet2
10.11.11.11 1 FULL/DR 00:00:38 192.168.113.11
GigabitEthernet1

Activity Verification
You have completed this task when you attain these results:

 On PE1, you detected issues and fixed incorrect OSPF timers to connect to P1.
 On PE1, you detected issues and fixed the incorrect OSPF area setup to P2.
 On PE1, you restored the OSPF neighbor relationship to P1 and P2.
  On PE3, you detected issues and fixed the duplicated OSPF router ID (with P1).
 On PE3, you detected issues and fixed the MTU value on the Gi2 interface to
P2.
 On PE3, you restored the OSPF neighbor relationship to P1 and P2.

Detect, Analyze, and Solve MPLS LDP Issues in PE2

and P Routers
The Level 1 support team has informed you that LDP sessions in PE2 to the P1 and P2
backbone routers seem to have failed. They also report another issue in the LDP
relationship between P1 and P2. You are asked to confirm if this report is correct and
solve any issues that affect the operation of the MPLS network.

Step 25

Begin by troubleshooting the issue between PE2 and the backbone routers. Click the
PE2 router to open a serial console session.

Step 26

Press Enter and then provide login information to enter privileged EXEC mode. Use
the access credentials that are provided in Job Aid.

Answer
User Access Verification

Username: cisco
Password: cisco

Step 27

Check if PE2 has active LDP sessions to P1 and P2. Observe that no LDP sessions are
reported.

Answer

Use the show mpls ldp neighbor command to verify the LDP neighbor list and state.
Observe that PE2 has no LDP sessions.

RP/0/RP0/CPU0:PE2#show mpls ldp neighbor

Tue Jul 23 16:07:10.840 UTC
RP/0/RP0/CPU0:PE2#

Step 28

Display the status of the LDP discovery process on PE2. Do you see normal status for
LDP sessions? Are the LDP router IDs the expected values?
Note
Network setup for this lab uses the IP address in the Loopback 0 interface as the LDP
router ID.

Answer

Use the show mpls ldp discovery command to verify any discovered LDP
neighbors. Observe that two neighbors are reported. However, the explicit state: oper
message does not display to indicate that a neighbor is operational. Verify that the two
neighbor routers are using the correct LDP router ID.

RP/0/RP0/CPU0:PE2#show mpls ldp discovery

Tue Jul 23 16:10:22.269 UTC

Local LDP Identifier: 10.2.2.2:0

Discovery Sources:
Interfaces:
GigabitEthernet0/0/0/1: xmit/recv
VRF: 'default' (0x60000000)
LDP Id: 10.11.11.11:0, Transport address: 10.11.11.11
Hold time: 10 sec (local:15 sec, peer:10 sec)
Established: Jul 22 17:15:18.353 (22:55:04 ago)

GigabitEthernet0/0/0/2: xmit/recv
VRF: 'default' (0x60000000)
LDP Id: 10.12.12.12:0, Transport address: 10.12.12.12
Hold time: 10 sec (local:15 sec, peer:10 sec)
Established: Jul 22 17:15:18.803 (22:55:03 ago)

Step 29

Make sure that the local PE2 router can reach the IP addresses that are used as LDP
router IDs in P1 and P2. Use the neighbor router IDs that displayed in previous
commands.

Answer

Use the ping command to check Layer 3 connectivity to the LDP router ID for P1 and
P2. Observe that these ping tests succeed.

RP/0/RP0/CPU0:PE2#ping 10.11.11.11
Tue Jul 23 16:19:49.441 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.11.11.11, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/9/27 ms

RP/0/RP0/CPU0:PE2#ping 10.12.12.12
Tue Jul 23 16:20:04.133 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.12.12.12, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/5/13 ms
Step 30

Click the P1 router to open a serial console session.

Step 31

Press Enter and then provide login information to enter privileged EXEC mode. Use
the access credentials that are provided in Job Aid

Answer
User Access Verification

Username: cisco
Password: cisco

Step 32

Verify that the LDP session to PE2 is nonoperational.

Answer

Use the show mpls ldp neighbor 10.2.2.2 command to verify the LDP neighbor
state. Observe that P1 has no LDP session with PE2.

RP/0/RP0/CPU0:P1#show mpls ldp neighbor 10.2.2.2

Tue Jul 23 09:18:02.718 UTC

RP/0/RP0/CPU0:P1#

Step 33

Check if P1 can reach the IP address that is used as the LDP router ID in PE2.

Answer

Use the ping 10.2.2.2 command to check Layer 3 connectivity to the LDP router ID
on PE2. Observe that this ping test fails.

RP/0/RP0/CPU0:P1#ping 10.2.2.2
Tue Jul 23 09:31:34.415 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.2.2.2, timeout is 2 seconds:
UUUUU
Success rate is 0 percent (0/5)

Step 34

Display the IP routing table in P1 and check if the network for the PE2 LDP router ID
is listed.
Answer

Use the show route ipv4 command to display the P1 routing table and check for
10.2.2.2/32. Observe that this network is not visible.

RP/0/RP0/CPU0:P1#show route ipv4

Tue Jul 23 09:35:23.265 UTC

Codes: C - connected, S - static, R - RIP, B - BGP, (>) - Diversion

path
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
i - ISIS, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, su - IS-IS summary null, * - candidate
default
U - per-user static route, o - ODR, L - local, G - DAGR, l -
LISP
A - access/subscriber, a - Application route
M - mobile route, r - RPL, t - Traffic Engineering, (!) - FRR
Backup path

Gateway of last resort is not set

S 10.0.0.0/8 is directly connected, 23:27:35, Null0

O 10.1.1.1/32 [110/2] via 192.168.111.1, 03:36:06,
GigabitEthernet0/0/0/1
O 10.3.3.3/32 [110/2] via 192.168.113.3, 00:33:50,
GigabitEthernet0/0/0/3
L 10.11.11.11/32 is directly connected, 23:27:27, Loopback0
O 10.12.12.12/32 [110/2] via 192.168.12.12, 23:25:27,
GigabitEthernet0/0/0/5
L 127.0.0.0/8 [0/0] via 0.0.0.0, 23:27:36
C 192.168.12.0/24 is directly connected, 23:26:17,
GigabitEthernet0/0/0/5
L 192.168.12.11/32 is directly connected, 23:26:17,
GigabitEthernet0/0/0/5
C 192.168.21.0/24 is directly connected, 23:26:17,
GigabitEthernet0/0/0/6
L 192.168.21.11/32 is directly connected, 23:26:17,
GigabitEthernet0/0/0/6
C 192.168.111.0/24 is directly connected, 23:26:17,
GigabitEthernet0/0/0/1
L 192.168.111.11/32 is directly connected, 23:26:17,
GigabitEthernet0/0/0/1
C 192.168.112.0/24 is directly connected, 23:26:17,
GigabitEthernet0/0/0/2
L 192.168.112.11/32 is directly connected, 23:26:17,
GigabitEthernet0/0/0/2
C 192.168.113.0/24 is directly connected, 23:26:17,
GigabitEthernet0/0/0/3
L 192.168.113.11/32 is directly connected, 23:26:17,
GigabitEthernet0/0/0/3
O 192.168.121.0/24 [110/2] via 192.168.12.12, 03:29:08,
GigabitEthernet0/0/0/5
[110/2] via 192.168.111.1, 03:29:08,
GigabitEthernet0/0/0/1
O 192.168.122.0/24 [110/2] via 192.168.12.12, 23:24:37,
GigabitEthernet0/0/0/5
 [110/2] via 192.168.112.2, 23:24:37,
GigabitEthernet0/0/0/2
O 192.168.123.0/24 [110/2] via 192.168.12.12, 23:25:27,
GigabitEthernet0/0/0/5

Step 35

Return to the PE2 console session. Verify if OSPF is active in all the necessary
interfaces.

Answer

Use the show ospf interface brief command to display interfaces for which the
OSPF protocol is enabled. Remember that OSPF will propagate IP networks on
interfaces for which the protocol is activated.

RP/0/RP0/CPU0:PE2#show ospf interface brief

Tue Jul 23 16:44:27.436 UTC

* Indicates MADJ interface, (P) Indicates fast detect hold down state

Interfaces for OSPF 1

Interface PID Area IP Address/Mask Cost

State Nbrs F/C
Gi0/0/0/1 1 0 192.168.112.2/24 10 BDR
1/1
Gi0/0/0/2 1 0 192.168.122.2/24 1 BDR
1/1

Step 36

Review the PE2 router OSPF configuration and pay special attention to interfaces that
are included in the OSPF process for Area 0. Look for any missing piece of
configuration.

Answer

Use the show running-config router ospf command to display interfaces for which
OSPF protocol is enabled. Remember that OSPF will propagate IP networks on
interfaces for which the protocol is activated.

RP/0/RP0/CPU0:PE2#show running-config router ospf

Tue Jul 23 16:45:59.122 UTC
router ospf 1
router-id 10.2.2.2
address-family ipv4
area 0
interface GigabitEthernet0/0/0/1
cost 10
!
interface GigabitEthernet0/0/0/2
!
!
!
Step 37

Based on output from the verification commands, formulate your hypothesis about what
issues are preventing PE2 from establishing LDP full adjacency with P1 and P2.

Note
You can compare your diagnostic with actual root causes by clicking the SHOW ME
button in this step.

Answer

By using the verification commands, the issue that prevents PE2 from forming LDP
adjacency with P1 and P2 should be evident. PE2 is not advertising its LDP router ID
in OSPF updates because the Loopback 0 interface was missing from the OPSF
configuration.

Step 38

On the PE2 router, correct the issue that affects LDP adjacency with P1 and P2. Apply
the changes and exit global configuration mode.

Answer

Use the configure terminal command to enter global configuration mode. Enter
OSPF configuration mode for process ID 1 by typing the router ospf 1 command.
Use the area 0 command and then type the interface loopback 0 command to
associate the interface to OSPF Area 0. Use the commit command to apply the changes.
Type end to exit configuration mode.

RP/0/RP0/CPU0:PE2#configure terminal
RP/0/RP0/CPU0:PE2(config)#router ospf 1
RP/0/RP0/CPU0:PE2(config-ospf)#area 0
RP/0/RP0/CPU0:PE2(config-ospf-ar)#interface loopback 0
RP/0/RP0/CPU0:PE2(config-ospf-ar-if)#commit
Tue Jul 23 16:50:11.744 UTC
RP/0/RP0/CPU0:Jul 23 16:50:12.663 UTC: config[66273]: %MGBL-CONFIG-6-
DB_COMMIT : Configuration committed by user 'cisco'. Use 'show
configuration commit changes 1000038989' to view the changes.
RP/0/RP0/CPU0:PE2(config-ospf-ar-if)#RP/0/RP0/CPU0:Jul 23 16:50:43.471
UTC: bgp[1060]: %ROUTING-BGP-5-ADJCHANGE : neighbor 10.12.12.12 Up
(VRF: default) (AS: 65000)
RP/0/RP0/CPU0:Jul 23 16:50:43.472 UTC: bgp[1060]: %ROUTING-BGP-5-
NSR_STATE_CHANGE : Changed state to Not NSR-Ready
RP/0/RP0/CPU0:PE2(config-ospf-ar-if)#end

Step 39

Check that PE2 now has active LDP sessions to the P1 and P2 routers.

Note
LDP neighbors may display in reverse order.

Answer

Use the show mpls ldp neighbor command to verify the LDP neighbor list and state.
Observe that two neighbors report adjacency in an operational state for each one.

RP/0/RP0/CPU0:PE2#show mpls ldp neighbor

Tue Jul 23 16:54:56.900 UTC

Peer LDP Identifier: 10.12.12.12:0

TCP connection: 10.12.12.12:21047 - 10.2.2.2:646
Graceful Restart: No
Session Holdtime: 180 sec
State: Oper; Msgs sent/rcvd: 22/28; Downstream-Unsolicited
Up time: 00:04:44
LDP Discovery Sources:
IPv4: (1)
GigabitEthernet0/0/0/2
IPv6: (0)
Addresses bound to this peer:
IPv4: (9)
10.12.12.12 192.168.12.12 192.168.21.12 192.168.121.12
192.168.122.12 192.168.123.12 209.165.200.225 209.165.201.1
209.165.202.129
IPv6: (0)

Peer LDP Identifier: 10.11.11.11:0

TCP connection: 10.11.11.11:15742 - 10.2.2.2:646
Graceful Restart: No
Session Holdtime: 180 sec
State: Oper; Msgs sent/rcvd: 22/22; Downstream-Unsolicited
Up time: 00:04:33
LDP Discovery Sources:
IPv4: (1)
GigabitEthernet0/0/0/1
IPv6: (0)
Addresses bound to this peer:
IPv4: (6)
10.11.11.11 192.168.12.11 192.168.21.11 192.168.111.11
192.168.112.11 192.168.113.11
IPv6: (0)

Step 40

Next you will troubleshoot the issue between P1 and P2. Resume your console
connection to P1. Check the status for the LDP session to P2.

Answer

Use the show mpls ldp neighbor 10.12.12.12 command to verify that 10.12.12.12
is not an LDP neighbor.

RP/0/RP0/CPU0:P1#show mpls ldp neighbor 10.12.12.12

Tue Jul 23 11:25:18.460 UTC
Step 41

Check interfaces where LDP is enabled on the P1 router. Is LDP enabled on links to
P2?

Answer

Use the show mpls ldp interface command to verify interfaces in which LDP is
enabled.

RP/0/RP0/CPU0:P1#show mpls ldp interface

Tue Jul 23 11:26:59.147 UTC
Interface GigabitEthernet0/0/0/1 (0x1000048)
VRF: 'default' (0x60000000)
Enabled via config: LDP interface
Interface GigabitEthernet0/0/0/2 (0x1000040)
VRF: 'default' (0x60000000)
Enabled via config: LDP interface
Interface GigabitEthernet0/0/0/3 (0x1000038)
VRF: 'default' (0x60000000)
Enabled via config: LDP interface
Interface GigabitEthernet0/0/0/5 (0x1000028)
VRF: 'default' (0x60000000)
Disabled:
Interface GigabitEthernet0/0/0/6 (0x1000020)
VRF: 'default' (0x60000000)
Disabled:

Step 42

Based on output from the verification commands, formulate your hypothesis about what
issues are preventing P1 from establishing LDP full adjacency with P2.

Note
You can compare your diagnostic with actual root causes by clicking the SHOW ME
button in this step.

Answer

By using the verification commands, the issue that prevents P1 from forming LDP
adjacency with P2 should be evident. LDP was accidentally disabled on the interfaces
that connect P1 to P2.

Step 43

Correct the issue that affects LDP adjacency between P1 and P2. Apply the changes and
exit global configuration mode.

Answer
Use the configure terminal command to enter global configuration mode. Enter LDP
configuration mode by typing mpls ldp. To activate LDP on the necessary interfaces,
use the interface GigabitEthernet0/0/0/5 and interface
GigabitEthernet0/0/0/6 commands. Type commit to apply the changes. Type end to
exit global configuration mode.

RP/0/RP0/CPU0:P1#configure terminal
Tue Jul 23 11:36:46.189 UTC
RP/0/RP0/CPU0:P1(config)#mpls ldp
RP/0/RP0/CPU0:P1(config-ldp)#interface GigabitEthernet0/0/0/5
RP/0/RP0/CPU0:P1(config-ldp-if)#interface GigabitEthernet0/0/0/6
RP/0/RP0/CPU0:P1(config-ldp-if)#commit
Tue Jul 23 11:42:41.926 UTC
RP/0/RP0/CPU0:Jul 23 11:42:42.066 UTC: mpls_ldp[1252]: %ROUTING-LDP-5-
NBR_CHANGE : VRF 'default' (0x60000000), Neighbor 10.12.12.12:0 is UP
(IPv4 connection)
RP/0/RP0/CPU0:Jul 23 11:42:42.365 UTC: config[67081]: %MGBL-CONFIG-6-
DB_COMMIT : Configuration committed by user 'cisco'. Use 'show
configuration commit changes 1000000026' to view the changes.
RP/0/RP0/CPU0:P1(config-ldp-if)#end
RP/0/RP0/CPU0:Jul 23 11:42:43.144 UTC: config[67081]: %MGBL-SYS-5-
CONFIG_I : Configured from console by cisco
RP/0/RP0/CPU0:P1#

Step 44

On the P1 router, check the status for the LDP session to P2. Is the session now
restored?

Answer

Use the show mpls ldp neighbor 10.12.12.12 command to verify the LDP
neighbor operational state. Observe that the neighbor relationship is established over the
two directly connected interfaces.

RP/0/RP0/CPU0:P1#show mpls ldp neighbor 10.12.12.12

Tue Jul 23 11:47:11.900 UTC

Peer LDP Identifier: 10.12.12.12:0

TCP connection: 10.12.12.12:39813 - 10.11.11.11:646
Graceful Restart: No
Session Holdtime: 180 sec
State: Oper; Msgs sent/rcvd: 21/28; Downstream-Unsolicited
Up time: 00:04:30
LDP Discovery Sources:
IPv4: (3)
GigabitEthernet0/0/0/6
GigabitEthernet0/0/0/5
Targeted Hello (10.11.11.11 -> 10.12.12.12, active)
IPv6: (0)
Addresses bound to this peer:
IPv4: (9)
10.12.12.12 192.168.12.12 192.168.21.12 192.168.121.12
192.168.122.12 192.168.123.12 209.165.200.225 209.165.201.1
209.165.202.129
IPv6: (0)
Activity Verification
You have completed this task when you attain these results:

 You verified that PE2 had no LDP neighbor relationship to P1 and P2.
 You analyzed the LDP issue and found that the LDP router ID for PE2 router
was not advertised by the OSPF protocol.
 On the PE2 router, you solved the issue that affected connectivity to the LDP
router ID.
 You confirmed the establishment of the LDP sessions in PE2 to P1 and P2.
 You verified the LDP session issues between P1 and P2.
 You analyzed the LDP issue on P1 and found that LDP was disabled on the
interfaces that connect to P2.
 On the P1 router, you enabled LDP on the GigabitEthernet0/0/0/5 and
GigabitEthernet0/0/0/6 interfaces.
 You confirmed the establishment of the LDP sessions between the P1 and P2
routers.

Detect, Analyze, and Solve MP-BGP Session Issues

The Level 1 support team has informed you that the BGP session for the VPNv4
address family between PE1 and P2 (the BGP route reflector) seems broken. You
receive the same report for the relationship between the PE3 and P2 routers. You are
asked to confirm if this report is correct and solve any issue that affects the
establishment of these MP-BGP sessions.

BGP Neighbor States

BGP starts a TCP session with a neighbor router (peer). The BGP session transitions
between the following states: Idle, Connect, Active, OpenSent, OpenReceive, and
Established. The meaning of each phase is as follows:

 Idle: The initial state that the BGP routing process enters when you enable the
routing process or reset the device. In this state, the device waits for a start
event, such as a peering configuration with a remote peer. After the device
receives a TCP connection request from a remote peer, the device initiates
another start event to wait for a timer before starting a TCP connection to a
remote peer. The peer resets if the device resets, and the BGP routing process
returns to the idle state.
 Connect: The BGP routing process detects that a peer is trying to establish a
TCP session with the local BGP speaker.
 Active: In this state, the BGP routing process tries to establish a TCP session
with a peer device by using the ConnectRetry timer. Start events are ignored
while the BGP routing process is in the active state. If you reconfigure the BGP
routing process or an error occurs, the BGP routing process will release system
resources and return to the idle state.
 OpenSent: The TCP connection is established, and the BGP routing process
sends an OPEN message to the remote peer and transitions to the OpenSent
 state. The BGP routing process can receive other OPEN messages in this state. If
the connection fails, the BGP routing process transitions to the active state.
 OpenReceive: The BGP routing process receives the OPEN message from the
remote peer and waits for an initial keepalive message from the remote peer.
When it receives a keepalive message, the BGP routing process transitions to the
established state. If it receives a notification message, the BGP routing process
transitions to the idle state. If an error or configuration change occurs that affects
the peering session, the BGP routing process sends a notification message with
the Finite State Machine (FSM) error code and then transitions to the idle state.
 Established: The initial keepalive is received from the remote peer. Peering is
now established with the remote neighbor and the BGP routing process starts
exchanging update message with the remote peer. The hold timer restarts when
an update or keepalive message is received. If the BGP process receives an error
notification, it will transition to the idle state.

Step 45

Begin by troubleshooting the PE1 router. Resume a serial console session to PE1.
Validate if a BGP session is established to P2 for the VPNv4 address family.

Note
You may see the state cycle between Idle and Active.

Answer

Use the show bgp vpnv4 unicast summary command to check for established MP-
BGP sessions.

RP/0/RP0/CPU0:PE1#show bgp vpnv4 unicast summary

Tue Jul 23 19:04:38.962 UTC
BGP router identifier 10.1.1.1, local AS number 65000
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0 RD version: 0
BGP main routing table version 11
BGP NSR Initial initsync version 9 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

BGP is operating in STANDALONE mode.

Process RcvTblVer bRIB/RIB LabelVer ImportVer SendTblVer

StandbyVer
Speaker 11 11 11 11 11
0

Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down

St/PfxRcd
10.12.12.12 0 65000 0 0 0 0 0 00:00:00
Active
Step 46

Check the status of BGP neighbor peering between PE1 and P2.

Answer

Use the show bgp neighbors command to check the status for established BGP
sessions.

RP/0/RP0/CPU0:PE1#show bgp neighbors

Tue Jul 23 19:06:44.357 UTC

BGP neighbor is 10.12.12.12

Remote AS 65000, local AS 65000, internal link
Remote router ID 0.0.0.0
BGP state = Active
NSR State: None
Last read 00:00:00, Last read before reset 00:00:00
Hold time is 180, keepalive interval is 60 seconds
Configured hold time: 180, keepalive: 60, min acceptable hold time:
3
<... output omitted ...>

Step 47

Debug TCP events in PE1 to analyze BGP TCP session issues. Wait for one minute,
turn off debugging, and interpret the results.

Answer

Use the debug tcp events command to look for issues that affect the MP-BGP
session. To turn off all possible debugs, use the undebug all command. Look for
messages that indicate that P2 is attempting a TCP session (BGP port 179) to the local
router and check the local destination IP address for that session.

RP/0/RP0/CPU0:PE1#debug tcp events

Tue Jul 23 19:10:52.701 UTC
RP/0/RP0/CPU0:PE1#RP/0/RP0/CPU0: Jul 23 19:10:54.716 UTC: tcp[264]:
[t5403] Tcp datagram get a packet from network layer: pak 0x6410b9f3
RP/0/RP0/CPU0: Jul 23 19:10:54.716 UTC: tcp[264]: [t5403] pak
0x0x6410b9f3, cache write into HSSD success, cache 0x55a734d6d5a8
RP/0/RP0/CPU0: Jul 23 19:10:54.716 UTC: tcp[264]: [t5403] Tcp datagram
pak:0x6410b9f3, seq=870702541, len=28 for input processing
RP/0/RP0/CPU0: Jul 23 19:10:54.716 UTC: tcp[264]: [t5403] Enqueuing
input pak 0x6410b9f3 queue index 0
RP/0/RP0/CPU0: Jul 23 19:10:54.717 UTC: tcp[264]: [t5664] Got pulsed
for internal queue 0
RP/0/RP0/CPU0: Jul 23 19:10:54.717 UTC: tcp[264]: [t5664] Dequeued
input pkt 0x6410b9f3, queue index 0
RP/0/RP0/CPU0: Jul 23 19:10:54.717 UTC: tcp[264]: [t5664] Queue pak
(0x6410b9f3): 10.12.12.12:47472 -> 10.1.1.1:179 len=0 PEER.ack =0
RP/0/RP0/CPU0: Jul 23 19:10:54.717 UTC: tcp[264]: [t5664] Attempt to
connect to 10.1.1.1:179 from 10.12.12.12:47472 (pak 0x6410b9f 3)
RP/0/RP0/CPU0: Jul 23 19:10:54.717 UTC: tcp[264]: [t5664]
tcp_dummy_tcpcb_send_segments:LINE=661 Tcp output: pak (0x6410b7bb)
10.
1.1.1:179 -> 10.12.12.12:47472, len=0
RP/0/RP0/CPU0: Jul 23 19:10:54.717 UTC: tcp[264]: [t5664] Tcp IQ read
no items, queue_index: 0

<... output omitted ...>

RP/0/RP0/CPU0:PE1#undebug all
All possible debugging has been turned off

Step 48

On the PE1 router, check the BGP configuration and pay special attention to the BGP
peering setup for P2. P2 is pointing to the IP address in the local Loopback 0 interface
to establish the BGP session. Do you see any misconfigurations?

Answer

To verify the BGP configuration, use the show running-config router bgp
command.

RP/0/RP0/CPU0:PE1#show running-config router bgp

Tue Jul 23 19:19:38.562 UTC
router bgp 65000
address-family vpnv4 unicast
!
neighbor 10.12.12.12
remote-as 65000
update-source GigabitEthernet0/0/0/1
address-family vpnv4 unicast
next-hop-self
!
<... output omitted ...>

Step 49

Based on output from the verification commands, formulate your hypothesis about the
issue that prevents PE1 from establishing BGP full adjacency with P2.

Note
You can compare your diagnostic with actual root causes by clicking the SHOW ME
button in this step.

Answer

By using the verification commands, the issue that prevents PE1 from forming BGP
adjacency with P2 should be evident. PE1 is sourcing the BGP packets from an IP
address that differs from the one which P2 is expecting it.

Step 50
Correct the issue that affects BGP adjacency between PE1 and P2. Apply the changes
and exit global configuration mode.

Answer

Use the configure terminal command to enter global configuration mode. Enter
BGP configuration mode by typing the router bgp 65000 command. Use the
neighbor 10.12.12.12 command to configure the BGP session to that router. Type
update-source loopback0 to force the local router to use the IP address in that
interface as source for BGP packets. Type commit to apply the changes. Type end to
exit global configuration mode.

RP/0/RP0/CPU0:PE1#configure terminal
Tue Jul 23 19:36:07.420 UTC
RP/0/RP0/CPU0:PE1(config)#router bgp 65000
RP/0/RP0/CPU0:PE1(config-bgp)#neighbor 10.12.12.12
RP/0/RP0/CPU0:PE1(config-bgp-nbr)#update-source Loopback0
RP/0/RP0/CPU0:PE1(config-bgp-nbr)#commit
Tue Jul 23 19:36:39.534 UTC
RP/0/RP0/CPU0:Jul 23 19:36:41.569 UTC: config[65923]: %MGBL-CONFIG-6-
DB_COMMIT : Configuration committed by user 'cisco'. Use 'show
configuration commit changes 1000034194' to view the changes.
RP/0/RP0/CPU0:PE1(config-bgp-nbr)#end
RP/0/RP0/CPU0:Jul 23 19:36:43.016 UTC: config[65923]: %MGBL-SYS-5-
CONFIG_I : Configured from console by cisco
RP/0/RP0/CPU0:PE1#RP/0/RP0/CPU0:Jul 23 19:36:57.303 UTC: bgp[1060]:
%ROUTING-BGP-5-ADJCHANGE : neighbor 10.12.12.12 Up (VRF: default) (AS:
65000)
RP/0/RP0/CPU0:Jul 23 19:36:57.304 UTC: bgp[1060]: %ROUTING-BGP-5-
NSR_STATE_CHANGE : Changed state to Not NSR-Ready

Step 51

On the PE1 router, verify that you restored the BGP neighbor relationship to P2.

Answer

To display information about BGP and TCP connections to neighbors, use the show
bgp neighbors command.

RP/0/RP0/CPU0:PE1#show bgp neighbors

Tue Jul 23 19:39:21.833 UTC

BGP neighbor is 10.12.12.12

Remote AS 65000, local AS 65000, internal link
Remote router ID 10.12.12.12
BGP state = Established, up for 00:02:24
NSR State: None
Last read 00:00:19, Last read before reset 00:00:00
Hold time is 180, keepalive interval is 60 seconds
Configured hold time: 180, keepalive: 60, min acceptable hold time:
3
Last write 00:00:19, attempted 19, written 19
Second last write 00:01:19, attempted 19, written 19
Last write before reset 00:00:00, attempted 0, written 0
Second last write before reset 00:00:00, attempted 0, written 0
 Last write pulse rcvd Jul 23 19:39:02.338 last full not set pulse
count 10
Last write pulse rcvd before reset 00:00:00
Socket not armed for io, armed for read, armed for write
Last write thread event before reset 00:00:00, second last 00:00:00
Last KA expiry before reset 00:00:00, second last 00:00:00
Last KA error before reset 00:00:00, KA not sent 00:00:00
Last KA start before reset 00:00:00, second last 00:00:00
Precedence: internet
Non-stop routing is enabled
Multi-protocol capability received
<... output omitted ...>

Step 52

Verify that PE1 is receiving two VPNv4 prefixes from P2 (route reflector).

Note
You should see two VPNv4 prefixes that PE2 advertises. At this moment, an issue
affects the BGP session between PE3 and P2. The PE2 router is correctly
preconfigured. PE2 is advertising one IP prefix for the CustomerA VRF and another
one for the CustomerB VRF.

Answer

The command show bgp vpnv4 unicast summary displays the VPNv4 session with
the route reflector and the number of received prefixes.

RP/0/RP0/CPU0:PE1#show bgp vpnv4 unicast summary

Tue Jul 23 19:54:42.877 UTC
BGP router identifier 10.1.1.1, local AS number 65000
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0 RD version: 0
BGP main routing table version 15
BGP NSR Initial initsync version 9 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

BGP is operating in STANDALONE mode.

Process RcvTblVer bRIB/RIB LabelVer ImportVer SendTblVer

StandbyVer
Speaker 15 15 15 15 15
0

Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down

St/PfxRcd
10.12.12.12 0 65000 27 22 15 0 0 00:17:45
2

Step 53
Next you will troubleshoot the PE3 router. Resume your serial console session to PE3.
Validate if a BGP session has established to P2 for the VPNv4 address family.

Note
The State column may show a different status as the BGP session transitions through
phases.

Answer

Use the show bgp vpnv4 unicast all summary command to verify that a neighbor
relationship has established to P2. Note that although a neighbor displays, no prefixes
are being received because the state is in a loop: Idle or Active.

PE3#show bgp vpnv4 unicast all summary

BGP router identifier 10.3.3.3, local AS number 65000
BGP table version is 3, main routing table version 3
2 network entries using 312 bytes of memory
2 path entries using 184 bytes of memory
2/2 BGP path/bestpath attribute entries using 352 bytes of memory
2 BGP extended community entries using 48 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 896 total bytes of memory
BGP activity 2/0 prefixes, 4/2 paths, scan interval 60 secs

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ

Up/Down State/PfxRcd
10.12.12.12 4 65000 0 0 1 0 0
never Idle

Step 54

Check the status of BGP neighbor peering between PE3 and P2.

Answer

Use the show bgp all neighbors command to verify that a neighbor relationship has
not established to the P2 router.

PE3#show bgp all neighbors

For address family: IPv4 Unicast
BGP neighbor is 10.12.12.12, remote AS 65000, internal link
BGP version 4, remote router ID 0.0.0.0
BGP state = Active, down for never
Neighbor sessions:
0 active, is not multisession capable (disabled)
Stateful switchover support enabled: NO for session 0
Message statistics:
InQ depth is 0
OutQ depth is 0

Sent Rcvd
Opens: 0 0
Notifications: 0 0
 Updates: 0 0
Keepalives: 0 0
Route Refresh: 0 0
Total: 0 0
Do log neighbor state changes (via global configuration)
Default minimum time between advertisement runs is 0 seconds

Address tracking is enabled, the RIB does have a route to

10.12.12.12
Route to peer address reachability Up: 1; Down: 0
Last notification 22:23:09
Connections established 0; dropped 0
Last reset never
Interface associated: (none) (peering address NOT in same link)
Transport(tcp) path-mtu-discovery is enabled
Graceful-Restart is disabled
SSO is disabled
No active TCP connection
<... output omitted ...>

Step 55

Observe and analyze the TCP-6-BADAUTH message that constantly appears in the
PE3 console.

Answer

This message repeatedly displays on the PE3 router console. MD5 is a hash function
that is used in cryptography to protect passwords.

PE3#
*Jul 24 05:46:21.385: %TCP-6-BADAUTH: No MD5 digest from
10.12.12.12(22359) to 10.3.3.3(179) tableid - 0
PE3#
*Jul 24 05:46:23.385: %TCP-6-BADAUTH: No MD5 digest from
10.12.12.12(22359) to 10.3.3.3(179) tableid - 0
PE3#
*Jul 24 05:46:27.397: %TCP-6-BADAUTH: No MD5 digest from
10.12.12.12(22359) to 10.3.3.3(179) tableid - 0
PE3#
*Jul 24 05:46:35.396: %TCP-6-BADAUTH: No MD5 digest from
10.12.12.12(22359) to 10.3.3.3(179) tableid - 0
PE3#

Step 56

Check the BGP configuration on the PE3 router and pay special attention to the BGP
peering setup. You have not used passwords for BGP neighbors in this lab setup. Do
you see any unexpected pieces of configuration?

Answer

Use the show running-config | section bgp command to review the BGP
configuration in this router.
PE3#show running-config | section router bgp
router bgp 65000
bgp log-neighbor-changes
neighbor 10.12.12.12 remote-as 65000
neighbor 10.12.12.12 password itiswrong
neighbor 10.12.12.12 update-source Loopback0
!
address-family ipv4
neighbor 10.12.12.12 activate
exit-address-family
!
address-family vpnv4
neighbor 10.12.12.12 activate
neighbor 10.12.12.12 send-community both
neighbor 10.12.12.12 next-hop-self
exit-address-family
!
address-family vpnv6
exit-address-family
!
address-family ipv4 vrf CustomerA
redistribute connected
exit-address-family
!
address-family ipv4 vrf CustomerB
redistribute connected
exit-address-family

Step 57

Based on output from the verification commands, formulate your hypothesis about what
issues prevent PE3 from establishing BGP full adjacency with P2.

Note
You can compare your diagnostic with actual root causes by clicking the SHOW ME
button in this step.

Answer

From the verification commands and alerts on the console, the issue that prevents PE3
from forming BGP adjacency with P2 should be evident. PE3 is configured for BGP
authentication, while P2 is not using it at all.

Step 58

To keep consistency with the lab setup, correct the issue that affects BGP adjacency
between PE3 and P2 by removing the authentication configuration on PE3. Exit global
configuration mode.

Note
In production environments, it is highly recommended that you protect all BGP sessions
with secure passwords.
Answer

Use the configure terminal command to enter global configuration mode. Enter
BGP configuration mode by typing the router bgp 65000 command. Use no
neighbor 10.12.12.12 password itiswrong to delete the authentication setup.
Type end to return to privileged EXEC mode.

PE3#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
PE3(config)#
*Jul 24 06:33:47.637: %TCP-6-BADAUTH: No MD5 digest from
10.12.12.12(42326) to 10.3.3.3(179) tableid - 0
PE3(config)#router bgp 65000
PE3(config-router)#no neighbor 10.12.12.12 password itiswrong
PE3(config-router)#end
PE3#
*Jul 24 06:34:37.615: %SYS-5-CONFIG_I: Configured from console by
cisco on console
PE3#
*Jul 24 06:34:45.651: %BGP-5-NBR_RESET: Neighbor 10.12.12.12 active
reset (BGP Notification sent)
*Jul 24 06:34:45.651: %BGP-5-ADJCHANGE: neighbor 10.12.12.12 Up

Step 59

On the PE3 router, verify that the BGP neighbor relationship to P2 router is restored.

Answer

Use the show bgp all neighbors command to verify that a neighbor relationship is
established to P2.

PE3#show bgp all neighbors

For address family: IPv4 Unicast
BGP neighbor is 10.12.12.12, remote AS 65000, internal link
BGP version 4, remote router ID 10.12.12.12
BGP state = Established, up for 00:58:05
Last read 00:00:59, last write 00:00:33, hold time is 180, keepalive
interval is 60 seconds
Neighbor sessions:
1 active, is not multisession capable (disabled)
Neighbor capabilities:
Route refresh: advertised and received(new)
Four-octets ASN Capability: advertised and received
Address family IPv4 Unicast: advertised and received
ipv4 MPLS Label capability: received
Address family IPv6 Unicast: received
Address family VPNv4 Unicast: advertised
Address family VPNv6 Unicast: received
Address family L2VPN Vpls: received
Enhanced Refresh Capability: advertised
Multisession Capability:
Stateful switchover support enabled: NO for session 1
Message statistics:
InQ depth is 0
OutQ depth is 0
 Sent Rcvd
Opens: 1 1
Notifications: 0 0
Updates: 4 10
Keepalives: 65 58
Route Refresh: 0 0
Total: 70 69
Do log neighbor state changes (via global configuration)
Default minimum time between advertisement runs is 0 seconds

Address tracking is enabled, the RIB does have a route to

10.12.12.12
Route to peer address reachability Up: 1; Down: 0
Last notification 1d00h
Connections established 1; dropped 0
Last reset never
Interface associated: (none) (peering address NOT in same link)
Transport(tcp) path-mtu-discovery is enabled
Graceful-Restart is disabled
SSO is disabled
Connection state is ESTAB, I/O status: 1, unread input bytes: 0
Connection is ECN Disabled, Mininum incoming TTL 0, Outgoing TTL 255
Local host: 10.3.3.3, Local port: 179
<... output omitted ...>

Step 60

Check if PE3 is receiving VPNv4 prefixes from the P2 router (route reflector). Observe
that no prefixes have been learned yet.

Answer

Use the show bgp vpnv4 unicast all summary command to verify that a neighbor
relationship has established to P2. Observe that prefixes have not been received yet
from P2.

PE3#show bgp vpnv4 unicast all summary

BGP router identifier 10.3.3.3, local AS number 65000
BGP table version is 3, main routing table version 3
2 network entries using 312 bytes of memory
2 path entries using 184 bytes of memory
2/2 BGP path/bestpath attribute entries using 352 bytes of memory
2 BGP AS-PATH entries using 48 bytes of memory
2 BGP extended community entries using 48 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 944 total bytes of memory
BGP activity 18/14 prefixes, 32/28 paths, scan interval 60 secs

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ

Up/Down State/PfxRcd
10.12.12.12 4 65000 0 0 1 0 0
00:03:15 (NoNeg)

Step 61
Next you will troubleshoot the P2 router. You may suspect that some configurations in
the route reflector are preventing PE3 router from learning routes. Click the P2 router to
open a serial console session.

Step 62

Press Enter and then provide login information to enter privileged EXEC mode. Use
the access credentials that are provided in Job Aid.

Answer
User Access Verification

Username: cisco
Password: cisco

Step 63

Explore and analyze the BGP configuration in P2. Focus on the configuration that
relates to PE3 and compares to PE1 and PE2.

Answer

Use the show running-config router bgp command to review the BGP
configuration in this router.

RP/0/RP0/CPU0:P2#show running-config router bgp

Wed Jul 24 08:51:11.813 UTC
router bgp 65000
<... output omitted ...>

neighbor 10.1.1.1
remote-as 65000
update-source Loopback0
address-family ipv4 unicast
route-reflector-client
!
address-family ipv4 labeled-unicast
route-reflector-client
!
address-family vpnv4 unicast
route-reflector-client
!
address-family ipv6 labeled-unicast
route-reflector-client
!
address-family vpnv6 unicast
route-reflector-client
!
address-family l2vpn vpls-vpws
route-reflector-client
!
!
neighbor 10.2.2.2
remote-as 65000
update-source Loopback0
 address-family ipv4 unicast
route-reflector-client
!
address-family ipv4 labeled-unicast
route-reflector-client
!
address-family vpnv4 unicast
route-reflector-client
!
address-family ipv6 labeled-unicast
route-reflector-client
!
address-family vpnv6 unicast
route-reflector-client
!
address-family l2vpn vpls-vpws
route-reflector-client
!
!
neighbor 10.3.3.3
remote-as 65000
update-source Loopback0
address-family ipv4 unicast
route-reflector-client
!
address-family ipv4 labeled-unicast
route-reflector-client
!
address-family ipv6 labeled-unicast
route-reflector-client
!
address-family vpnv6 unicast
route-reflector-client
!
address-family l2vpn vpls-vpws
route-reflector-client
!
<... output omitted ...>

Step 64

Based on output from the verification commands, formulate your hypothesis about what
issues are preventing PE3 from receiving VPNv4 prefixes.

Note
You can compare your diagnostic with actual root causes by clicking the SHOW ME
button in this step.

Answer

By using the verification commands, the issue that prevents PE3 from receiving VPNv4
prefixes from P2 should be evident. P2 is missing the address family VPNv4 and route
reflector client configuration in the PE3 BGP configuration.

Step 65
On the P2 router, correct the issue that affects BGP adjacency between PE3 and P2.
Apply the changes and exit global configuration mode.

Answer

Use the configure terminal command to enter global configuration mode. Enter
BGP configuration mode by typing the router bgp 65000 command. Use the
neighbor 10.3.3.3 command to enter the neighbor configuration mode. Use
address-family vpnv4 unicast to enable the VPNv4 address family. Use the
route-reflector-client command to configure the local router as the route reflector
for that neighbor. Type commit to apply the changes. Type end to exit global
configuration mode.

RP/0/RP0/CPU0:P2#configure terminal
Wed Jul 24 09:13:17.380 UTC
RP/0/RP0/CPU0:P2(config)#router bgp 65000
RP/0/RP0/CPU0:P2(config-bgp)#neighbor 10.3.3.3
RP/0/RP0/CPU0:P2(config-bgp-nbr)#address-family vpnv4 unicast
RP/0/RP0/CPU0:P2(config-bgp-nbr-af)#route-reflector-client
RP/0/RP0/CPU0:P2(config-bgp-nbr-af)#commit
Wed Jul 24 09:13:51.778 UTC
RP/0/RP0/CPU0:Jul 24 09:13:52.248 UTC: config[68631]: %MGBL-CONFIG-6-
DB_COMMIT : Configuration committed by user 'cisco'. Use 'show
configuration commit changes 1000000031' to view the changes.
RP/0/RP0/CPU0:P2(config-bgp-nbr-af)#RP/0/RP0/CPU0:Jul 24 09:13:53.829
UTC: bgp[1060]: %ROUTING-BGP-5-ADJCHANGE : neighbor 10.3.3.3 Down -
Address family activated (VRF: default) (AS: 65000)
RP/0/RP0/CPU0:Jul 24 09:14:07.932 UTC: bgp[1060]: %ROUTING-BGP-5-
ADJCHANGE : neighbor 10.3.3.3 Up (VRF: default) (AS: 65000)
RP/0/RP0/CPU0:P2(config-bgp-nbr-af)#end

Step 66

Resume your PE3 router console session and confirm that it now receives four VPNv4
prefixes from P2 (the route reflector).

Note
PE3 receives one IP prefix for the CustomerA VRF and another one for the
CustomerB VRF from PE1 and PE2.

Answer

Use the show bgp vpnv4 unicast all summary command to verify that a neighbor
relationship is established to P2. Observe that four prefixes were received from P2.

PE3#show bgp vpnv4 unicast all summary

BGP router identifier 10.3.3.3, local AS number 65000
BGP table version is 11, main routing table version 11
6 network entries using 936 bytes of memory
6 path entries using 552 bytes of memory
4/4 BGP path/bestpath attribute entries using 704 bytes of memory
2 BGP rrinfo entries using 48 bytes of memory
2 BGP AS-PATH entries using 48 bytes of memory
2 BGP extended community entries using 48 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 2336 total bytes of memory
BGP activity 22/14 prefixes, 38/30 paths, scan interval 60 secs

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ

Up/Down State/PfxRcd
10.12.12.12 4 65000 20 16 11 0 0
00:08:48 4

Activity Verification
You have completed this task when you attain these results:

 You found an issue and fixed the IP address that is used to source BGP packets
on PE1.
 You confirmed the establishment of the BGP session between PE1 and P2.
 You found an issue and removed the BGP session authentication on the PE3
router.
 You found an issue and added the address family VPNv4 and route reflector
client configuration under PE3 BGP configuration on the P2 router.
 You confirmed the establishment of the BGP session between the PE3 and P2
routers.
 You confirmed that PE3 is receiving four BGP prefixes from P2, which
correspond to the Customer A and Customer B VPNs.

Detect, Analyze, and Solve End-to-End LSPs Issues for

MPLS VPNs
A customer with three locations, represented in the topology by the CE1, CE2, and
CE3 devices, reports that connectivity between those sites is lost. Network Operations
Center personnel have reviewed the VRF and VPN configurations and find that all
related aspects are correct. Also, connectivity between the provider edge and customer
edge routers at each location were tested and verified. You suspect that something must
be wrong on the service provider network backbone. You plan to go deeper into this
troubleshooting project to find the root cause.

Step 67

Begin by confirming the lack of connectivity between sites in the same VPN. Click the
CE1 router to open a serial console session.

Note
Customer edge devices are preconfigured for two different VPNs with CustomerA and
CustomerB VRFs (Multi-VRF CE). They use static default routing that points to
provider edge routers to connect to remote locations.
Step 68

Press Enter and then provide login information to enter privileged EXEC mode. Use
the access credentials that are provided in Job Aid.

Answer
User Access Verification

Username: cisco
Password: cisco

Step 69

From the CustomerA VRF in CE1, test Layer 3 connectivity to the CustomerA CE2
and CustomerA CE3 IP addresses in the interface that connects those remote sites to
the service provider. You will find that these tests fail.

Answer

Use the ping vrf command to test Layer 3 connectivity to remote locations.

CE1#ping vrf CustomerA 192.168.102.12

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.102.12, timeout is 2
seconds:
.....
Success rate is 0 percent (0/5)

CE1#ping vrf CustomerA 192.168.103.13

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.103.13, timeout is 2
seconds:
.....
Success rate is 0 percent (0/5)
CE1#

Step 70

Validate that CE1 has the IP routing information that it needs to reach the remote
location for the CustomerA VRF.

Answer

Use the show ip route vrf command to verify that CE1 has a static default route to
reach any remote destination.

CE1#show ip route vrf CustomerA

Extended Host Mode is enabled

Routing Table: CustomerA

<... output omitted ...>

Gateway of last resort is 192.168.101.1 to network 0.0.0.0

S* 0.0.0.0/0 [1/0] via 192.168.101.1
10.0.0.0/32 is subnetted, 1 subnets
C 10.1.11.1 is directly connected, Loopback0
172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
C 172.16.11.0/24 is directly connected, Loopback2
L 172.16.11.1/32 is directly connected, Loopback2
192.168.101.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.168.101.0/24 is directly connected, GigabitEthernet1
L 192.168.101.11/32 is directly connected, GigabitEthernet1

Step 71

Now confirm that CE3 has no connectivity between sites in the same VPN. Click the
CE3 router to open a serial console session.

Step 72

Press Enter and then provide login information to enter privileged EXEC mode. Use
the access credentials that are provided in Job Aid.

Answer
User Access Verification

Username: cisco
Password: cisco

Step 73

From the CustomerA VRF in CE3, test Layer 3 connectivity to the CustomerA CE1
and CustomerA CE2 IP addresses in the interface that connects those remote sites to
the service provider. You will find that only one test fails.

Answer

Use the ping vrf command to test Layer 3 connectivity to remote locations.

CE3#ping vrf CustomerA 192.168.101.11

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.101.11, timeout is 2
seconds:
.....
Success rate is 0 percent (0/5)

CE3#ping vrf CustomerA 192.168.102.12

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.102.12, timeout is 2
seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/2 ms

Step 74
Validate that CE3 has the IP routing information that it needs to reach the remote
location on the CustomerA VRF.

Answer

Use the show ip route vrf command to verify that CE3 has a static default route to
reach any remote destination.

CE3#show ip route vrf CustomerA

Extended Host Mode is enabled

Routing Table: CustomerA

<... output omitted ...>

Gateway of last resort is 192.168.103.3 to network 0.0.0.0

S* 0.0.0.0/0 [1/0] via 192.168.103.3

10.0.0.0/32 is subnetted, 1 subnets
C 10.1.13.1 is directly connected, Loopback0
172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
C 172.16.13.0/24 is directly connected, Loopback2
L 172.16.13.1/32 is directly connected, Loopback2
192.168.103.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.168.103.0/24 is directly connected, GigabitEthernet1
L 192.168.103.13/32 is directly connected, GigabitEthernet1

Step 75

Resume your PE1 router console session and check if it has the IP routing information
that it needs to reach the provider edge–customer edge link at the two remote locations
(CE2 and CE3) for the associated VRF (CustomerA).

Answer

Use the show route vrf command to verify the information that PE1 uses to reach the
CE2 and CE3 remote destinations inside the VRFs.

RP/0/RP0/CPU0:PE1#show route vrf CustomerA

Wed Jul 24 21:35:33.983 UTC
<... output omitted ...>

Gateway of last resort is not set

C 192.168.101.0/24 is directly connected, 2d04h,

GigabitEthernet0/0/0/3
L 192.168.101.1/32 is directly connected, 2d04h,
GigabitEthernet0/0/0/3
B 192.168.102.0/24 [200/0] via 10.2.2.2 (nexthop in vrf default),
01:20:52
B 192.168.103.0/24 [200/0] via 10.3.3.3 (nexthop in vrf default),
01:20:52

Step 76
Check if PE1 has the IP routing information that it needs to reach the two remote
provider edge routers (PE2 and PE3). For this action, check the global routing table that
the OSPF protocol built. Observe no specific entries for 10.2.2.2/32 and 10.3.3.3/32,
because they are part of summary 10.0.0.0/8 network.

Answer

Use the show route ospf command to verify the information that PE1 uses to reach
PE2 and PE3. Observe no specific entries for 10.2.2.2/32 and 10.3.3.3/32, because they
are part of the summary 10.0.0.0/8 network.

RP/0/RP0/CPU0:PE1#show route ospf

Wed Jul 24 21:39:42.936 UTC

O E2 10.0.0.0/8 [110/1] via 192.168.121.12, 01:25:01,

GigabitEthernet0/0/0/2
[110/1] via 192.168.111.11, 01:25:01,
GigabitEthernet0/0/0/1
O 10.11.11.11/32 [110/2] via 192.168.111.11, 01:25:01,
GigabitEthernet0/0/0/1
O 10.12.12.12/32 [110/2] via 192.168.121.12, 01:25:01,
GigabitEthernet0/0/0/2
O 192.168.12.0/24 [110/2] via 192.168.121.12, 01:25:01,
GigabitEthernet0/0/0/2
[110/2] via 192.168.111.11, 01:25:01,
GigabitEthernet0/0/0/1
O 192.168.21.0/24 [110/3] via 192.168.121.12, 01:25:01,
GigabitEthernet0/0/0/2
[110/3] via 192.168.111.11, 01:25:01,
GigabitEthernet0/0/0/1
O 192.168.112.0/24 [110/2] via 192.168.111.11, 01:25:01,
GigabitEthernet0/0/0/1
O 192.168.113.0/24 [110/2] via 192.168.111.11, 01:25:01,
GigabitEthernet0/0/0/1
O 192.168.122.0/24 [110/2] via 192.168.121.12, 01:25:01,
GigabitEthernet0/0/0/2
O 192.168.123.0/24 [110/2] via 192.168.121.12, 01:25:01,
GigabitEthernet0/0/0/2

Step 77

Verify that, even when no specific routes to the PE2 and PE3 Loopback 0 interfaces
exist in the local routing table, PE1 is still able to reach the associated IP addresses in
PE2 and PE3. This verification confirms the Layer 3 connectivity inside the backbone.

Answer

Use the ping command to verify Layer 3 connectivity to the PE2 and PE3 Loopback 0
interfaces. This ping should succeed, and IP connectivity inside the backbone is
working as expected.

RP/0/RP0/CPU0:PE1#ping 10.2.2.2
Wed Jul 24 21:46:34.806 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.2.2.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms

RP/0/RP0/CPU0:PE1#ping 10.3.3.3
Wed Jul 24 21:46:39.579 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.3.3.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms

Verify End-to-End LSP Paths

You use the MPLS ping and MPLS traceroute commands for monitoring and
troubleshooting MPLS LSPs. These features help you check connectivity, isolate failure
points, and provide MPLS Operations, Administration, and Maintenance (OAM)
solutions. You use normal ICMP ping and traceroute commands to diagnose the root
cause when a forwarding failure occurs. However, they may not detect LSP failures,
because an ICMP packet can forward through IP to the destination when an LSP
breakage occurs. You can use MPLS LSP ping and traceroute to identify LSP
breakages. These commands use UDP packets with loopback destination addresses to
encode requests and label them with the selected FEC label. Enable MPLS OAM by
using the mpls oam command on all routers in the MPLS network.

The MPLS echo request uses the outgoing interface IP address as the source. It uses a
loopback IP address, which is configurable, as the destination (127.0.0.1). The TTL in
MPLS ping is set to 255. If the LSP breaks somewhere inside the MPLS domain, the
127/8 address in the IP header destination address field causes the packet to not forward
from any routers that use the IP header.

Step 78

On the PE1 router, use ping mpls with source on local Loopback0 to check if an end-
to-end LSP exists to the Loopback 0 interfaces on PE2 and PE3. Test failure implies
that the end-to-end LSP paths are broken. Broken LSPs disrupt MPLS VPN
connectivity.

Note
MPLS OAM, which is necessary to use ping mpls, is preconfigured on all routers.

Answer

You can use MPLS LSP echo request and reply packets to validate an LSP by using the
ping mpls ipv4 10.2.2.2/32 source 10.1.1.1 and ping mpls ipv4
10.3.3.3/32 source 10.1.1.1 commands. This test failure indicates that end-to-end
LSP paths are broken.

RP/0/RP0/CPU0:PE1#ping mpls ipv4 10.2.2.2/32 source 10.1.1.1

Wed Jul 24 21:54:37.476 UTC

Sending 5, 100-byte MPLS Echos to 10.2.2.2/32,

 timeout is 2 seconds, send interval is 0 msec:
<... output omitted ...>

Type escape sequence to abort.

QQQQQ
Success rate is 0 percent (0/5)

RP/0/RP0/CPU0:PE1#ping mpls ipv4 10.3.3.3/32 source 10.1.1.1

Wed Jul 24 21:55:04.313 UTC

Sending 5, 100-byte MPLS Echos to 10.3.3.3/32,

timeout is 2 seconds, send interval is 0 msec:
<... output omitted ...>

Type escape sequence to abort.

QQQQQ
Success rate is 0 percent (0/5)

Step 79

On the PE1 router, check the LDP label assignment and look for IP networks on
Loopback0 for PE2 and PE3.

Answer

Use the show mpls ldp binding command to explore the content of the LDP binding
table. Verify the labels for the loopback interfaces on the provider edge routers with /32
netmask and the label for /8 netmask that displays in the OSPF routing table.

RP/0/RP0/CPU0:PE1#show mpls ldp binding

Wed Jul 24 21:59:57.621 UTC

10.0.0.0/8, rev 24
Local binding: label: 64010
Remote bindings: (2 peers)
Peer Label
----------------- ---------
10.11.11.11:0 ImpNull
10.12.12.12:0 ImpNull
10.1.1.1/32, rev 2
Local binding: label: ImpNull
Remote bindings: (2 peers)
Peer Label
----------------- ---------
10.11.11.11:0 24004
10.12.12.12:0 24005
10.2.2.2/32, rev 0
No local binding
Remote bindings: (2 peers)
Peer Label
----------------- ---------
10.11.11.11:0 24006
10.12.12.12:0 24007
10.3.3.3/32, rev 0
No local binding
 Remote bindings: (2 peers)
Peer Label
----------------- ---------
10.11.11.11:0 24005
10.12.12.12:0 24006
<... output omitted ...>

Step 80

On the PE1 router, check the MPLS LDP swapping table (LFIB) and look for
information that relates to the loopback addresses on the provider edge routers. You will
not find label information for /32 Loopback0 networks on remote provider edge routers.

Answer

Use the show mpls forwarding command to explore the contents of the LFIB table.
Verify the labels for the loopback interfaces on the provider edge routers with /8
netmask. However, labels for specific loopback addresses with /32 netmask do not
display.

RP/0/RP0/CPU0:PE1#show mpls forwarding

Wed Jul 24 22:10:49.786 UTC
Local Outgoing Prefix Outgoing Next Hop
Bytes
Label Label or ID Interface
Switched
------ ----------- ------------------ ------------ --------------- ---
---------
64000 Aggregate CustomerA: Per-VRF Aggr[V] \
CustomerA 520
64001 Aggregate CustomerB: Per-VRF Aggr[V] \
CustomerB 0
64002 Pop 10.11.11.11/32 Gi0/0/0/1 192.168.111.11
165052
64003 Pop 10.12.12.12/32 Gi0/0/0/2 192.168.121.12
177203
64004 Pop 192.168.112.0/24 Gi0/0/0/1 192.168.111.11 0
64005 Pop 192.168.12.0/24 Gi0/0/0/1 192.168.111.11 0
Pop 192.168.12.0/24 Gi0/0/0/2 192.168.121.12 0
64006 Pop 192.168.21.0/24 Gi0/0/0/1 192.168.111.11 0
Pop 192.168.21.0/24 Gi0/0/0/2 192.168.121.12 0
64007 Pop 192.168.122.0/24 Gi0/0/0/2 192.168.121.12 0
64008 Pop 192.168.113.0/24 Gi0/0/0/1 192.168.111.11 0
64009 Pop 192.168.123.0/24 Gi0/0/0/2 192.168.121.12 0
64010 Pop 10.0.0.0/8 Gi0/0/0/1 192.168.111.11 520
Pop 10.0.0.0/8 Gi0/0/0/2 192.168.121.12 520

Step 81

On the PE1 router, use a verification command to display information about the OSPF
running configuration. Is the information that displays correct? Do you see any
unexpected configurations?

Answer
Use the show running-config router ospf command to display information about
the OSPF running configuration on the PE1 router. Pay attention to the Distribute List
feature. Does it match what you expect to find?

RP/0/RP0/CPU0:PE1#show running-config router ospf

Wed Jul 24 23:20:03.016 UTC
router ospf 1
router-id 10.1.1.1
distribute-list SUMFLT in
address-family ipv4
area 0
interface Loopback0
!
interface GigabitEthernet0/0/0/1
dead-interval 40
hello-interval 10
!
interface GigabitEthernet0/0/0/2
!
!
!

Step 82

On PE1, verify the content of the SUMFLT ACL. What is the effect of this access list?
Do you see any unexpected configurations in the access list?

Answer

Use the show access-lists SUMFLT command to display information for the ACL.
Pay attention to the ACL content. Does it match what you expect to find?

RP/0/RP0/CPU0:PE1#show access-lists SUMFLT

Mon Jul 29 21:00:00.996 UTC
ipv4 access-list SUMFLT
10 deny ipv4 host 10.1.1.1 any
20 deny ipv4 host 10.2.2.2 any (11 matches)
30 deny ipv4 host 10.3.3.3 any (25 matches)
40 permit ipv4 any any (70 matches)

Step 83

Resume your console session for the provider backbone routers. Check the MPLS LDP
swapping table (LFIB) on P1 and P2 and look for information that relates to the
loopback addresses on the provider edge routers. Observe that they have details for /32
networks on the provider edge routers.

Answer

Use the show mpls forwarding command to explore the content of the LFIB table.
Observe that the label exists for the loopback interface on the provider edge routers with
/32 netmask.

RP/0/RP0/CPU0:P1#show mpls forwarding

Wed Jul 24 15:08:12.064 UTC
Local Outgoing Prefix Outgoing Next Hop
Bytes
Label Label or ID Interface
Switched
------ ----------- ------------------ ------------ --------------- ---
---------
24000 Pop 10.12.12.12/32 Gi0/0/0/5 192.168.12.12
49215
24001 Pop 192.168.121.0/24 Gi0/0/0/5 192.168.12.12 0
Pop 192.168.121.0/24 Gi0/0/0/1 192.168.111.1 0
24002 Pop 192.168.122.0/24 Gi0/0/0/5 192.168.12.12 0
Pop 192.168.122.0/24 Gi0/0/0/2 192.168.112.2 84
24003 Pop 192.168.123.0/24 Gi0/0/0/5 192.168.12.12 0
24004 Pop 10.1.1.1/32 Gi0/0/0/1 192.168.111.1
17650
24005 Pop 10.3.3.3/32 Gi0/0/0/3 192.168.113.3
16478
24006 Pop 10.2.2.2/32 Gi0/0/0/2 192.168.112.2
18118

RP/0/RP0/CPU0:P2#show mpls forwarding

Wed Jul 24 15:08:19.789 UTC
Local Outgoing Prefix Outgoing Next Hop
Bytes
Label Label or ID Interface
Switched
------ ----------- ------------------ ------------ --------------- ---
---------
24000 Pop 10.11.11.11/32 Gi0/0/0/5 192.168.12.11
218261
24001 Pop 192.168.111.0/24 Gi0/0/0/5 192.168.12.11 0
Pop 192.168.111.0/24 Gi0/0/0/1 192.168.121.1 500
24002 Pop 192.168.112.0/24 Gi0/0/0/5 192.168.12.11 0
24003 Pop 192.168.113.0/24 Gi0/0/0/5 192.168.12.11 0
Pop 192.168.113.0/24 Gi0/0/0/3 192.168.123.3 0
24004 Aggregate Internet: Per-VRF Aggr[V] \
Internet 0
24005 Pop 10.1.1.1/32 Gi0/0/0/1 192.168.121.1
32663
24006 Pop 10.3.3.3/32 Gi0/0/0/3 192.168.123.3
30257
24007 Pop 10.2.2.2/32 Gi0/0/0/2 192.168.122.2
29095

Step 84

On the P1 and P2 routers, check the global routing table and look for information that
relates to the loopback addresses on the provider edge routers. Pay special attention to
the existence of a summarization entry in addition to specific /32 networks. Also,
observe that the summary route points to Null0.

Answer

Use the show route ipv4 command to display the content of the routing table.
Observe the entries for the loopback interface on the provider edge routers with /8
netmask. However, routes for specific loopback addresses with /32 netmask also
display. The /8 network is a summary. Also, observe that the summary route points to
Null0.

RP/0/RP0/CPU0:P1#show route ipv4

Wed Jul 24 15:13:44.647 UTC
<... output omitted ...>

Gateway of last resort is not set

S 10.0.0.0/8 is directly connected, 2d05h, Null0

O 10.1.1.1/32 [110/2] via 192.168.111.1, 02:33:09,
GigabitEthernet0/0/0/1
O 10.2.2.2/32 [110/2] via 192.168.112.2, 02:33:09,
GigabitEthernet0/0/0/2
O 10.3.3.3/32 [110/2] via 192.168.113.3, 02:33:09,
GigabitEthernet0/0/0/3
L 10.11.11.11/32 is directly connected, 2d05h, Loopback0
O 10.12.12.12/32 [110/2] via 192.168.12.12, 02:33:09,
GigabitEthernet0/0/0/5
L 127.0.0.0/8 [0/0] via 0.0.0.0, 2d05h
<... output omitted ...>

RP/0/RP0/CPU0:P2#show route ipv4

Wed Jul 24 15:13:55.476 UTC
<... output omitted ...>

Gateway of last resort is not set

S 10.0.0.0/8 is directly connected, 2d05h, Null0

O 10.1.1.1/32 [110/2] via 192.168.121.1, 02:33:30,
GigabitEthernet0/0/0/1
O 10.2.2.2/32 [110/2] via 192.168.122.2, 02:33:30,
GigabitEthernet0/0/0/2
O 10.3.3.3/32 [110/2] via 192.168.123.3, 02:33:30,
GigabitEthernet0/0/0/3
O 10.11.11.11/32 [110/2] via 192.168.12.11, 02:33:30,
GigabitEthernet0/0/0/5
L 10.12.12.12/32 is directly connected, 2d05h, Loopback0
L 127.0.0.0/8 [0/0] via 0.0.0.0, 2d05h
<... output omitted ...>

Step 85

On the P1 and P2 routers, use a verification command to display information about the
OSPF running configuration. Do you see any unexpected configurations? Pay attention
to redistribution commands.

Answer

Use the show running-config router ospf command to display information about
the OSPF running configuration on the P1 and P2 routers. Pay attention to the
redistribution feature. Does it match what you expect?

RP/0/RP0/CPU0:P1#show running-config router ospf

Wed Jul 24 15:35:29.483 UTC
router ospf 1
 router-id 10.11.11.11
redistribute static metric 1
area 0
mpls traffic-eng
interface Loopback0
!
interface GigabitEthernet0/0/0/1
!
interface GigabitEthernet0/0/0/2
!
interface GigabitEthernet0/0/0/3
!
interface GigabitEthernet0/0/0/5
!
interface GigabitEthernet0/0/0/6
cost 2
!
!
mpls traffic-eng router-id Loopback0
!

RP/0/RP0/CPU0:P2#show running-config router ospf

Wed Jul 24 15:35:30.167 UTC
router ospf 1
router-id 10.12.12.12
maximum paths 32
redistribute static metric 1
area 0
mpls traffic-eng
interface Loopback0
!
interface GigabitEthernet0/0/0/1
!
interface GigabitEthernet0/0/0/2
!
interface GigabitEthernet0/0/0/3
!
interface GigabitEthernet0/0/0/5
!
interface GigabitEthernet0/0/0/6
cost 2
!
!
mpls traffic-eng router-id Loopback0
!

Step 86

On the P1 and P2 routers, use a verification command to display information about the
static routing in the running configuration.

Answer

Use the show running-config router static command to display information

about the static routing in the running configuration on the P1 and P2 routers.

RP/0/RP0/CPU0:P1#show running-config router static

Wed Jul 24 15:54:32.349 UTC
router static
address-family ipv4 unicast
10.0.0.0/8 Null0
!
!

RP/0/RP0/CPU0:P2#show running-config router static

Wed Jul 24 15:54:26.629 UTC
router static
address-family ipv4 unicast
10.0.0.0/8 Null0
!
!

Step 87

Based on output from verification commands, formulate your hypothesis about the
issues that prevent VPN connectivity.

Note
You can compare your diagnostic with actual root causes by clicking the SHOW ME
button in this step.

Answer

By using the verification commands, it should be evident that two issues break the end-
to-end LSPs and prevent VPN connectivity:

1. An incorrect summarization for /32 IP addresses, which are the provider edge
LDP router IDs, was applied in the P1 and P2 backbone routers.
2. A filtering mechanism for inbound routes that applied on the PE1 router caused
only the summarized 10.0.0./8 network to present in the routing table. It
suppressed the more-specific /32 networks.

Step 88

On the P1 and P2 routers, remove the incorrect summarization. Apply the changes and
exit global configuration mode.

Answer

Use the configure terminal command to enter global configuration mode. Remove
the static route by typing the no router static address-family ipv4 unicast
10.0.0.0/8 command. Use the router ospf 1 command to enter OSPF configuration
mode. To remove the static redistribution, use the no redistribute static metric
1 command. Type commit to apply the changes. Type end to exit global configuration
mode.

RP/0/RP0/CPU0:P1#configure terminal
Wed Jul 24 15:55:55.970 UTC
RP/0/RP0/CPU0:P1(config)#no router static address-family ipv4 unicast
10.0.0.0/8
RP/0/RP0/CPU0:P1(config)#router ospf 1
RP/0/RP0/CPU0:P1(config-ospf)#no redistribute static metric 1
RP/0/RP0/CPU0:P1(config-ospf)#commit
Wed Jul 24 15:57:08.129 UTC
RP/0/RP0/CPU0:Jul 24 15:57:08.586 UTC: config[67954]: %MGBL-CONFIG-6-
DB_COMMIT : Configuration committed by user 'cisco'. Use 'show
configuration commit changes 1000000035' to view the changes.
RP/0/RP0/CPU0:P1(config-ospf)#end

RP/0/RP0/CPU0:P2#configure terminal
Wed Jul 24 15:58:09.861 UTC
RP/0/RP0/CPU0:P2(config)#no router static address-family ipv4 unicast
10.0.0.0/8
RP/0/RP0/CPU0:P2(config)#router ospf 1
RP/0/RP0/CPU0:P2(config-ospf)#no redistribute static metric 1
RP/0/RP0/CPU0:P2(config-ospf)#commit
Wed Jul 24 15:58:50.930 UTC
eRP/0/RP0/CPU0:Jul 24 15:58:51.457 UTC: config[67730]: %MGBL-CONFIG-6-
DB_COMMIT : Configuration committed by user 'cisco'. Use 'show
configuration commit changes 1000000038' to view the changes.
RP/0/RP0/CPU0:P2(config-ospf)#end

Step 89

Resume your PE1 router console session and remove the distribute list that is filtering
routes in OSPF.

Answer

Use the configure terminal command to enter global configuration mode. Use the
router ospf 1 command to enter OSPF configuration mode. To remove the distribute
list, use the no distribute-list SUMFLT in command. Type commit to apply the
changes. Type end to exit global configuration mode.

RP/0/RP0/CPU0:P1#configure terminal
Wed Jul 24 23:24:28.802 UTC
RP/0/RP0/CPU0:PE1(config)#router ospf 1
RP/0/RP0/CPU0:PE1(config-ospf)#no distribute-list SUMFLT in
RP/0/RP0/CPU0:PE1(config-ospf)#commit
Wed Jul 24 23:24:48.233 UTC
enRP/0/RP0/CPU0:Jul 24 23:24:49.159 UTC: config[68300]: %MGBL-CONFIG-
6-DB_COMMIT : Configuration committed by user 'cisco'. Use 'show
configuration commit changes 1000034198' to view the changes.
RP/0/RP0/CPU0:PE1(config-ospf)#end

Step 90

From the PE1 router, verify that end-to-end LSPs between provider edge routers are
restored.

Answer
Use the ping mpls command to verify end-to-end LSPs to the PE2 and PE3 Loopback
0 interfaces and source packets with the IP address in the Loopback 0 interface. You can
see that this ping mpls now succeeds.

RP/0/RP0/CPU0:PE1#ping mpls ipv4 10.2.2.2/32 source 10.1.1.1

Wed Jul 24 23:27:47.706 UTC

Sending 5, 100-byte MPLS Echos to 10.2.2.2/32,

timeout is 2 seconds, send interval is 0 msec:
<... output omitted ...>

Type escape sequence to abort.

!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 3/5/12 ms

RP/0/RP0/CPU0:PE1#ping mpls ipv4 10.3.3.3/32 source 10.1.1.1

Wed Jul 24 23:27:58.373 UTC

Sending 5, 100-byte MPLS Echos to 10.3.3.3/32,

timeout is 2 seconds, send interval is 0 msec:
<... output omitted ...>

Type escape sequence to abort.

!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 3/9/24 ms

Step 91

From CustomerA in the CE1 router, verify that connectivity for MPLS VPN is restored
to both remote sites.

Answer

Use the ping vrf command to test Layer 3 connectivity to remote locations.

RP/0/RP0/CPU0:PE1#ping vrf CustomerA 192.168.102.12

Wed Jul 24 23:33:48.536 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.102.12, timeout is 2
seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms

RP/0/RP0/CPU0:PE1#ping vrf CustomerA 192.168.103.13

Wed Jul 24 23:34:02.695 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.103.13, timeout is 2
seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms

Activity Verification
You have completed this task when you attain these results:

 You found and solved incorrect summarization issues for LDP router IDs that
applied on the P1 and P2 backbone routers.
 You found and removed incorrect filters for the OSPF routes (distribute list) that
applied to PE1.
 You confirmed that end-to-end LSPs are reestablished between PE1, PE2, and
PE3.
 You restored connectivity inside the VPN for end customers.