Layer 3 Troubleshooting

Objectives
• Network layer problems include any problem which
involves a Layer 3 protocol, both routed protocols
and routing protocols. This module will focus
primarily on IP routing protocols. Issues concerning
routed protocols, such as IP, and other Layer 3 IP
protocols, are discussed in the other modules.
• Network layer problems may also include issues at
the other layers. Configuring routing over a
nonbroadcast multi-access (NBMA) or dialup network
can involve specific configuration and
troubleshooting issues. Because these configurations
involve a specific Layer 2 technology such as Frame
Relay, ISDN, and so on, routing problems involving
these technologies will be discussed in the module
Troubleshooting Layer 2.
• This module will focus on the most common
troubleshooting issues regarding both static routing
and dynamic routing protocols.
Table of Content

1 Troubleshooting Network Layer Problems

2 Troubleshooting Static Routes
3 Common IGP Routing Protocol Issues,
Causes, and Solutions
4 Troubleshooting RIP
5 Troubleshooting IGRP
6 Troubleshooting EIGRP
7 Troubleshooting OSPF
8 Troubleshooting IS-IS
9 Troubleshooting BGP
10 Troubleshooting Redistribution
TROUBLESHOOTING NETWORK
LAYER PROBLEMS
What are network layer problems?
• Problems at the network layer can have the following
affects on a network:
• Network failure
– Network failure is when the network is nearly or completely
nonfunctional, affecting all users and applications using the
network. These failures are usually noticed quickly by users and
network administrators, and are obviously critical to the
productivity of a company.
• Failure to perform optimally
– Network optimization problems can be more difficult to discover
and sometimes harder to diagnose. These problems usually
involve a subset of users, applications, destinations, or a
particular type of traffic. Optimization issues in general can be
more difficult to detect and even harder to isolate and diagnose
as they usually involve multiple layers or even the host
computer itself. Determining that the problem is a network
layer problem can take time.
What are network layer problems?
Isolating the problem methodology
• It is important to note that there is no single
template for solving these Layer 3 problems.
• Routing problems are solved with a good methodical
process, using a series of commands to isolate and
diagnose the problem.
• General Network Issues:
Many times a change in the topology, such as a down
link, may have other affects on other areas of the
network which might not be obvious at the time. This
may include the installation of new routes, static or
dynamic, removal of other routes, and so on.
– Some of the things to consider include:
• Has anything in the network changed recently?
• Is there anyone currently working on the network
infrastructure?
Isolating the problem methodology
• Connectivity Issues:
Check for any equipment and connectivity problems including:
– Power problems: outages, intermittent problems, environmental
problems such as overheating, and so on.
– Layer 1 problems: cabling problems, bad ports, service provider
problems, and so on.
• Some commands, which can be used to check these and other
issues, include:
– ping
– traceroute
– show interface, show ip interface
• Neighbor issues – If the routing protocol establishes an
adjacency with a neighbor, check to see if there are any
problems with the routers forming neighbor relationships.
• Topology database – If the routing protocol uses a topology
table or database, check the table for anything unexpected,
such as missing entries or unexpected entries
• Routing table – Check the routing table for anything
unexpected, such as, missing routes or unexpected routes. Use
debug commands to view routing updates and routing table
Static routes, dynamic routing, summarization,
redistribution, and combinations
• Most networks are not configured with only static
routes or with only one type of dynamic routing
protocol.
• Most networks are a complex combination of static
routes, one or more dynamic routing protocols, manual
and automatic route aggregations, redistributions,
distribute-lists, access lists, and more.
• Understanding how these protocols and techniques
work together and what affect they can have on each
other is essential in understanding and troubleshooting
networks.
• What is important is to have a thorough understanding
of the routing protocols and configurations used in
many networks.
• The more time spent now understanding how the
routing protocols function and how they interoperate
with other routing protocols, static routes, and the
routing table, the less time will be spent later trying to
TROUBLESHOOTING STATIC
ROUTES
Static routes and classful lookups
• Routing table maintenance can be more complex for
static routes than dynamic routes.
• Static routes in the routing table can be invalid, in
other words, they can reference an inactive exit-
interface or an intermediate network address that
cannot be resolved.
– The static routing process within the IOS must be able to find
invalid static routes and remove them from the routing table,
as well as install new static routes that become valid or
available.
• When the routing table process checks for a
resolvable static route using an intermediate
address, this check is always done in classful
mode.
• This is regardless of whether or not the ip classless
command is used.
• If the intermediate address cannot be resolved
in the routing table in classful mode, the static
Static routes and classful lookups

The static route was removed because

the routing table uses the classful mode
for resolving intermediate (next hop)
addresses. There is a reason for using
classful mode for resolving intermediate
addresses of static routes. If classless
mode was used and a default route was
present, backup static routes with
higher administrative distances would
never be installed in the routing table if
Static routes and classful lookups
• It is important to remember that the Cisco IOS
Software stores all static routes, whether or not they
are installed in the routing table. The Cisco routing
table process invokes a static route function every 60
seconds which checks the routing table to install or
remove any static routes according to the dynamically
changing routing table.
• For example, if an interface has gone down, this
function will remove any static routes that were
resolved using this interface. This may be a static
route that includes this interface as its exit interface,
or a static route that has an intermediate address, but
eventually that intermediate address is ultimately
resolved via this downed interface.
• In addition, static routes may be installed in the
routing table when an interface becomes active or a
Static routes and intermediate addresses
• Static routes can be created
either by using an intermediate
network address or an exit
interface. In most cases, using an
exit interface will be more
efficient during the routing table
process for resolving the static
route.
• The static route that was created
uses an intermediate IP address
of 172.16.2.2.
• This is sometimes referred to as
the next-hop IP address, but the
IP address does not have to be
the physical next-hop.
• As long as the intermediate IP
address can be resolved in the
routing table, it does not have to
be the actual next-hop routers
interface.
• Ultimately, the static network
route, 172.16.3.0 in the example,
must finally be resolved to a route
in the routing table that has an
exit interface.
Static routes and intermediate addresses
• Notice that the static route that was
configured does not contain an exit
interface. Instead, the static routing
table entry contains the
intermediate address that was used
when configuring the static route.
• Whenever the routing table process
needs to use the static route entry
for the 172.16.3.0/24 network, it will
also need to resolve the
intermediate address, 172.16.2.2.
This is called a recursive lookup.
• Because this routing table entry for
the 172.16.3.0/24 route does not
contain an exit interface, but instead
the intermediate address
172.16.2.2, it cannot use this entry
alone to forward the packets.
• The routing table process uses the
intermediate address of 172.16.2.2
and does another lookup (recursive
lookup) in the routing table to find a
route for this 172.16.2.0 network.
• The routing table process finds the
directly connected network entry for
172.16.2.0.
Static routes optimization with serial networks
• There are ways to avoid recursive
table lookups. Although, there
may be times when recursive
table lookups are preferred and
configured by the network
administrator.
• Most of the time, static routes
over serial point-to-point
networks can easily avoid
recursive route lookups by using
an exit interface instead of the
intermediate or next-hop
address.
• Figure 1 shows an example of
creating a static route on RouterB
for RouterA LAN, using an exit
interface instead of an
intermediate address.
• Instead of using an intermediate
address, this route is resolved
with the exit interface, Serial0/0
that was configured with the
Static routes optimization with Ethernet networks
• Configuring static routes
without recursive lookups can
also be done over multi-access
networks like Ethernet. In this
instance, using only an exit-
interface instead of an
intermediate address causes a
problem. Because the network
is multi-access, there most
likely are multiple devices,
receivers, and so on, sharing
this network.
• Figure shows a network
between RouterA and RouterB
that is a multiaccess, Fast
Ethernet link.
• A static route using only an
intermediate address could be
configured. However, this will
cause a recursive routing table
lookup. Figure shows both the
static route command and its
installment in the routing
Static routes optimization with Ethernet networks
• In order to avoid the recursive route lookup, the solution is to use both an
intermediate address and an exit interface.
• Figure shows the recommended way to configure a static route in this case,
and the routing table entry. When using both the intermediate address and
the exit interface, only a single lookup is needed in the routing table lookup
process.
• The standard rule-of-thumb when configuring static routes is to use
an exit interface over point-to-point and exit interfaces with the
intermediate address over multiaccess networks. This will avoid
the recursive route lookups caused by static routing entries that
only contain an intermediate address.
Recurring static route installation and deletion
Recurring static route installation and deletion
• This process of adding and
deleting this static route is
repeated every 60 seconds.
Notice the times in the debug
output.
• This is also a demonstration of
how the static route process is
implemented every 60
seconds.
• This situation can create even
more instability if there are
other static routes resolved by
this route.
• One solution is to always
configure static routes to
use exit interfaces instead
of intermediate addresses
wherever possible.
Using discard routes
• An example of a typical customer
network is shown in Figure.
• The Customer Network is using a
dynamic routing protocol to route
the 172.16.0.0/24 and
192.168.1.0/24 traffic within its own
network.
• On RTA there is a 0.0.0.0/0 default
route configured, sending all default
traffic to ISP. RTA propagates this
default route to all other routers in
the Customer Network via a dynamic
routing protocol.
• There is no dynamic routing protocol
being used between RTA, the
Customer Network entrance router,
and the ISP router.
• The ISP router has two static routes
pointing to RTA for 172.16.0.0/16
and 192.168.1.0/24 networks.
• All of the networks are up, and there
is complete reachability throughout
the network and with the ISP.
However, there is the potential for a
routing loop problem.
Using discard routes
• If the routers are configured for
classless routing behavior, ip
classless, RTB will forward all
packets destined for
172.16.4.0/24 and
192.168.1.0/24 to RTA using
the default 0.0.0.0/0 route,
when the link between RTB &
RTC breaks down.
• RTA will also use the default
route to forward those packets
onto the ISP router.
• Figure shows an example of
pings being rerouted from RTA
to ISP, once RTA has removed
the 172.16.4.0/24 network
from its routing table, as the
link has broken down.
Using discard routes
• What does the ISP router do with
these packets for 172.16.4.1?
Since the ISP router has a static
route for the 172.16.0.0/16
network, forwarding traffic to
RTA, ISP will send those packets
destined for the 172.16.4.0/24
network back to RTA. RTA will
receive the packets from ISP and
forward them back to ISP, still
using its default route. Figure
shows the effect of this routing
loop on RTA serial interface as it
sends and receives these
packets (pings in this example)
on the link it shares with the ISP.
• This problem has now created a
blackhole in the network, with a
routing loop between RTA and
ISP.
• The packets will eventually
be dropped once the time-
to-live (TTL) field in the IP
headers gets decremented
Using discard routes
• Solution 1: Classful Mode Routing (no ip classless)
One solution is to change the routing behavior from classless to
classful using the command, no ip classless on all of the Customer
Network routers. Classful routing would cause a router searching its
routing table for a best match for 172.16.4.0 to drop the packets if
there are routes for other 172.16.0.0/24 subnets, but not for
172.16.4.0/24.
• With the no ip classless command, the router does not use any
supernet or default routes when the there is at least one known
subnet. The packets for 172.16.4.0 would be dropped by RTA and
RTB.
• This is usually not the preferred solution because it changes the
routing table lookup behavior for all packets. Packets destined for a
discontiguous subnet that rely on a supernet or default route to be
forwarded would also be dropped, unless that route is specifically in
the routing table.
• In addition, this does not solve the problem for packets destined for
the192.168.1.0/24 network. Even with using the no ip classless
command, all packets destined for 192.168.1.0/24 will still be
caught in a routing loop between RTA and ISP.
• In any case, modifying the route lookup process with no ip
classless is not always an ideal solution as it might have
unforeseen affects on the routing behavior in the network.
Using discard routes
• Solution 2: Using a Discard Route
A more elegant and scalable solution is
to use a discard route.
• A discard route is a route that sends
packets to null0, the bit-bucket, when
there is no specific match in the
routing table and it is undesirable to
have those packets forwarded using a
supernet or default route.
• Figure shows an example of a discard
route for RTA. This route will cause RTA
to drop all packets for subnets in the
172.16.0.0 network, that do not have a
specific route in the routing table.
• Using the failed route example and still
using classless routing (ip classless),
any 172.16.0.0 packets not matching
172.16.1.0/24 or 172.16.2.0/24 or
172.16.3.0/24 or 172.16.4.0/24, would
be routed to null0, using the discard
route. RTA would drop these packets
instead of forwarding them to ISP.
• The discard route will also keep
traffic with wrong IP addresses
from finding this blackhole in the
network.
Using discard routes
• Packets destined for 192.168.1.0/24
For any packets destined for the
192.168.1.0/24 network from RTA or RTB,
using classful mode routing, the no ip
classless command, would not help.
• Since this network is not a subnet of a
parent network in the routing tables of RTA
or RTB, the default route would be used
whether or not classful mode routing is
used.
• Once 192.168.1.0/24 is no longer
reachable, this route would be removed
from the routing tables of RTA and RTB. RTA
would eventually forward all packets to ISP.
Once again, the ISP would send these
packets back to RTA, causing another
blackhole.
• The solution is to configure another discard
route that will only be used if the primary
route fails. This can be done by modifying
the default administrative distance of the
static route to a value higher than the
administrative distance of the dynamic
routing protocol being used.
• RTA(config)#ip route 192.168.1.0
255.255.255.0 null0 200
• The discard route would only enter the
routing table for RTA, when the dynamic
route to 192.168.1.0/24 is removed.
• Figure shows how the discard route is
installed in the routing table, only after the
dynamic route is removed due to the link
 COMMON IGP ROUTING
PROTOCOL ISSUES, CAUSES, AND
SOLUTIONS
Introduction
• This section discusses several possible scenarios that can prevent routes
from being installed in the routing table. These are some of the problems
that are common to most routing protocols.
• If routes are not installed in the routing table, the router will not forward
the packets to the destinations that are missing. It is possible that the
packets could be incorrectly forwarded using a supernet or default route.
This scenario was previously discussed in the section, Using Discard
Routes.
• Three possibilities exist for routes not being installed in the routing table:
– Receiver problem – The router is receiving the updates, but it is not
installing the routes.
– Intermediate media problem, Layer 2 – The sender has sent the updates,
but they were lost along the way and the receiver did not receive them.
– Sender problem – The sender is not advertising the routes, so the receiving
side is not seeing the routes in the routing table.
• Some of the common causes for routes not being installed in the
routing table are:
– Missing or incorrect network or neighbor statement
– Layer 1/2 down
– Distribute-list in/out blocking (sender/receiver)
– Access list blocking
– Advertised Network Interface is down
– Passive interface
Missing or incorrect network or neighbor statement
• When configuring or modifying a network it
may become apparent that a route is missing
from the routing table. There can be many
reasons for this.
• One of the obvious things to check is to see
whether the network statement under router
configuration has been properly configured.
• For IGP routing protocols, the network
statement does two things:
– Enables the routing protocol on interfaces with IP
addresses that match the IP address in the
network statement, giving the interface the
capability to send and receive updates.
– Advertises that network in its own updates to other
routers.
Missing or incorrect network or neighbor statement

• This example shows two

routers running OSPF
between each other.
Debugs and Verification
• The output of the show ip
ospf neighbor command
shows an empty list. In a
normal working scenario, the
output would display the
OSPF adjacent neighbors.
• With some protocols like
OSPF and EIGRP, the routing
protocols can be enabled on
a per-interface basis using
wildcard masks.
• Careful configuration of the
wildcard mask is important
so interfaces are not
incorrectly included or
incorrectly left out.
Missing or incorrect network or neighbor statement
• An obvious place to begin is by
looking at the running-
configurations on both routers by
using the show running-config
command.
• Figure 1 shows the configuration of
Router R2. The configuration shows
that the network statement exists,
but a closer look reveals that the
wrong wildcard mask is used. The
network statement is determined in
OSPF in exactly the same way that
an access list would be defined.
The main idea here is to include the
range of addresses in the area. The
network statement of 131.108.0.0
with the wildcard mask of 0.0.0.255
will not cover 131.108.1.2. It covers
only the range from 131.108.0.0 to
131.108.0.255, as indicated by the
wildcard mask
• Figure 2 shows the output of the
show ip ospf interface command
and that OSPF is not enabled on the
Ethernet 0 interface of Router R2.
Missing or incorrect network or neighbor statement

• Depending upon the routing • show ip protocols

protocol, there are several • show ip interface
other commands that can
help troubleshoot this issue. • show ip interface brief
• The show ip protocols • show ip eigrp interfaces
command will display which • show ip eigrp neighbors
networks are originating • show ip ospf interface
from this router.
• debug ip routing
• The debug command can be
used to verify whether the • debug ip rip
routing update is being sent • debug ip igrp events
or received, or if there are • debug ip igrp
any mismatched timers, transactions
subnet masks, and so on.
• debug ip eigrp
• Commands used in this
example and some of the • debug ip ospf events
commands that can be • debug isis adj packets
used for other routing • debug isis update-
protocols include:
packets
Missing or incorrect network or neighbor statement
• Solution
The obvious solution is to correctly
configure the network statements to
enable the routing protocol on the
appropriate interfaces.
• During network configuration under
OSPF, a cut-and-paste of the OSPF
configuration might create this
problem. Therefore, it is always best
to look at the output of show ip ospf
interface for that specific interface
and confirm whether OSPF is enabled
on that interface. This specific
problem can be corrected by
reentering the network statement.
• Figure 1 shows the new configuration
that fixes the OSPF network problem.
In this example, the wildcard mask is
0.0.255.255, which means that it
covers the range from 131.108.0.0 to
131.108.255.255
• Figure 2 shows the output of show ip
ospf neighbor after applying the
correct network mask. Beginning with
Cisco IOS 12.0, the output of show ip
ospf interface does not display
anything if OSPF is not enabled on
the interface.
Layer 1 or 2 down
• One of the causes for routes
not being installed in the
routing table is that Layer 1 or
Layer 2 is down. If this is the
case, it is not a routing
protocol problem. Layer 1 or 2
could be down for several
reasons. The following is a list
of the most common things to
check if an interface or line
protocol is down:
– Unplugged cable
– Loose cable
– Bad cable
– Bad transceiver
– Bad port
– Bad interface card
– Layer 2 problem at the Telco
in the case of a WAN link
– Missing clockrate statement
in the case of back-to-back
serial connection
Layer 1 or 2 down
• Figure 1 shows the output of
debug ip igrp
transactions command.
• The output shows that the
router is not sending or
receiving any IGRP updates
because Layer 2 is down.
Solution
• To correct this problem, the
Layer 2 problem must be
fixed by checking previously
mentioned conditions. The
solution could be as simple
as plugging in a cable, or it
could be as complex as bad
hardware, in which case the
hardware must be replaced.
Distribute-list in/out blocking
• A distribute list is a filtering mechanism for routing
updates. Distribute lists are not supported for OSPF
or IS-IS. Routers running link state protocols determine
their routes based on information in their link state
database, rather than the advertised route entries
received from its neighbors.
• Route filters have no effect on link state
advertisements or the link state database. This is
because a basic requirement of link state routing
protocols is that routers in an area must have identical
link state databases.
• In addition, it is important to remember that distribute
lists do not permit or deny the actual packets from
entering the routers, only which routing updates a router
will send or receive. A distribute list calls on an access
list and checks which networks are supposed to be
permitted. If the access list does not contain the network,
it will automatically be denied. A distribute list can be
applied on incoming routing updates or outgoing
Distribute-list in/out blocking
Access list blocking
• Standard access lists are used to filter the traffic based on
source address. Extended access lists are used to filter the
traffic based on the source or destination address.
• These access lists can be applied on the interface with the
interface-level command ip access-group {access-list
number} {in|out} to filter the incoming or outgoing traffic.
• When the access list is applied, make sure that it does
not block the source address of the routing update, or
for routing protocols that form adjacencies that it is not
blocking the Hello packets from being sent or received.
• It is very common to implement an access list for security
measures at the interface level. In the case of routing protocols
such as OSPF that use multicasts to exchange Hellos, be sure to
permit the multicast Hello addresses in the access list.
Otherwise, the access list might block the OSPF multicast
address unknowingly and prevent OSPF from forming neighbors
on that interface.
• This situation happens only when the access list is blocking
Hellos on both routers. If only one side is blocking OSPF Hellos,
the output of show ip ospf neighbor will indicate that the
neighbor is stuck in INIT state.
Access list blocking
Debugs and Verification
• Figure 1 shows the OSPF
configuration of both routers R1
and R2, the access list is
permitting only incoming TCP
and UDP traffic. The inbound
access list checks only traffic
coming in on that interface.
Because there is an implicit
deny at the end of each access
list, this access list will block the
OSPF multicast address of
224.0.0.5. The access list 101
in command is defined for
debugging purposes only.
• Figure 2 shows the output of
debug ip packet 101 detail
command. This debug tracks
down the OSPF Hello packet only
on the Ethernet segment. The
debug shows that OSPF Hello
packet from router R1 is denied
on R2. This is because OSPF
packets are carried directly over
IP, and do not include a TCP or
Access list blocking
Solution
• To correct this problem, the
access list must be
reconfigured to permit OSPF
multicast Hellos.
• Figure 1 shows the configuration
that fixes this problem. In this
configuration, OSPF multicast
Hellos are permitted.
• Similarly, the access list on the
other side needs to be changed,
making sure that the OSPF
Hellos are permitted in the
access list.
• Figure 2 shows the OSPF
neighbor in FULL state after
fixing the configuration.
• Commands used in this example
and some of the commands that
can be used for other routing
protocols include:
– show ip eigrp neighbors
– show ip ospf neighbor
Advertised network interface is down
• IGP routing protocols will not
advertise any interface
network address that is
physically down.
• As soon as the advertised
network comes back up, the
routing protocol will start
advertising it again in its
updates.
• This example shows two routers
running RIP between each other.
Debugs and Verification
• Figure 2 shows that interface
Ethernet 0 is down.
• Figure 3 shows the output from
the debug ip rip command. In
this debug, router R1 is not
sending or receiving any RIP
updates because Layer 2 is
down. There is no output from
this debug command because of
Advertised network interface is down
• Solution
RIP, like other routing protocols, runs
above Layer 2. RIP cannot send or
receive any routes if Layer 2 is down.
To correct this problem, Layer 2 or
Layer 1 must be corrected.
• Figure 1 shows the interface Ethernet
0 after fixing the Layer 2 problem.
• Figure 2 shows the routing table of
router R2.
• Commands used in this example and
some of the commands that can be
used for other routing protocols
include:
• show ip interface
• show ip interface brief
• show ip eigrp interfaces
• show ip ospf interface
• debug ip routing
• debug ip rip
• debug ip igrp transactions
• debug ip eigrp
• debug isis adj packets
• debug isis update-packets
Passive interface
• The passive-interface command works differently with the
different IP routing protocols that support it.
• RIP/IGRP – With RIP and IGRP, routing updates are received, but
are not sent.
• EIGRP – With EIGRP, the router stops sending hello packets on
passive interfaces. When this happens, EIGRP cannot form
neighbor adjacencies on the interface, and routing updates can
neither be sent nor received.
• OSPF – With OSPF, routing information is neither sent nor
received on a passive interface. The network address of the
passive interface appears as a stub network in the OSPF domain.
• This example shows two routers running IGRP between each other.
Passive interface
Debugs and Verification
• Figure 1 shows the output of
the show ip protocols
command, which shows the
outgoing interface is defined
as passive.
• Figure 2 shows the
configuration of router R1,
which shows that the outgoing
interface is defined as passive.
Solution
• Figure and confirm that the
interface Ethernet 0 is defined
as passive, so router R1 is not
sending any updates on
Ethernet0
• Sometimes, it is desirable for
some networks to be
advertised and others to be
filtered. In this situation, a
distribute-list out would be a
Passive interface

• In this example, the assumption

is that the passive-interface
was configured by mistake, and
this command needs to be
removed to solve this problem.
• Figure 2 shows the new
configuration to solve this
problem.
• Figure 3 shows the routing table
entry on router R2 after fixing the
problem.
TROUBLESHOOTING RIP
Incompatible RIP version types
• When RIP is configured on a
router, by default, the
software receives RIP Version
1 (RIP v1) and RIP Version 2
(RIP v2) packets, but sends
only RIP v1 packets.
• To send and receive only RIP
v1 packets, the router must be
configured with the command
version 1 under router rip.
• To send and receive only RIP
v2 packets, the router must be
configured with the command
version 2 under router rip.
• When the version command
is used, by default, updates
from other routers sending
other than the specified
version are ignored.
• This example shows two
routers running RIP between
each other.
Incompatible RIP version types
• Figure 1 shows the output of the
show ip protocols command,
which indicates that the
Ethernet 0 interface is sending
and receiving RIP v1 packets.
This means that if a Version 2
packet is received on Ethernet 0
of R2, it will be ignored because
the interface can send and
receive only Version 1 packets.
• Figure 2 shows the configuration
of router R1. This shows that the
sender R1 is configured to only
send and receive Version 2
packets.
Incompatible RIP version types
Solution
• An obvious solution is to configure all
routers to run RIP v2. However, there
may be times when this is not possible,
and some routers can only run RIP v1.
Therefore, another solution is to configure
the appropriate interfaces to send and
receive the appropriate RIP v1 or RIP v2
packets.
• If the receiver, R2, is configured to receive
only RIP v1 packets, it will ignore the RIP
v2 updates. Router R1 must be configured
on the sender side so that it will send both
Version 1 and Version 2 packets. When R2
receives the Version 1 packet, it will install
the routes in the routing table. R2 will
ignore RIP v2 packets because it is
configured for RIP v1.
Mismatched authentication key
• One of the options in RIP v2 is
that RIP v2 updates can be
authenticated for increased
security. When authentication
is used, a password must be
configured on both sides. This
password is called the
authentication key. If this key
does not match the key on the
other side, the RIP v2 updates
will be ignored on both sides.
Debugs and Verification
• Figure 1 shows the
configurations of routers R1
and R2. In this configuration, a
different RIP authentication
key is configured on R1 and
R2.
Reaches RIP hop count limit
• The RIP metric maximum is 15
hops. If a network has more than
15 hops, RIP is not a suitable
protocol. Figure 3 shows a
network that produces a RIP hop-
count limit problem. Router R2 is
receiving an update for a RIP
route, which is more than 15
hops away. R2 does not install
that route in the routing table, as
demonstrated in the output.
Discontiguous networks
When a major network is separated by
another major network, this is called a
discontiguous network. Figure shows an
example of a discontiguous network.
Debugs and Verification
Figure 1 shows the configuration for
routers R1 and R2. RIP, with the default
of Version 1, is enabled on the Ethernet
interfaces of R1 and R2 with the correct
network statements.
Figure 2 shows the debug ip rip
command output for routers R1 and R2.
The debugs show that both routers are
sending to each other the summarized
major network address for
137.99.0.0, instead of their specific
network addresses. As a result, both
routers will ignore the less specific
137.99.0.0 update because they are
already connected to this major
network.
Discontiguous networks
• Solution
RIP is not installing the route
137.99.0.0 in the routing table because
RIP v1 does not support discontiguous
subnets. Several solutions to this
problem exist. Figure 1 shows that a
quick solution is to configure each
router with a static route to the
specific 137.99.0.0 subnet on the other
router. Since RIP v1 does not include
the subnet information in its routing
updates, configuring the static routes
is a “patch” to fix this problem. This
may be necessary if the routers in
question can only run RIP v1.
• Fig. 2 shows a better solution, to
enable the classless routing version of
RIP v2 with no auto-summary
configured on both routers. The no
auto-summary command will disable
auto-summarization of RIP v2 routes
when crossing a major network
boundary. It is important to disable the
auto-summarization or the same
unreachability problem will continue to
exist. With no auto-summary, the
specific subnet information is also
included in these updates. Figure 3
shows the routing table of R2 after
using this solution.
Invalid source address
• When RIP tells the routing table to
install the route, it performs a source-
validity check. If the source is not on
the same subnet as the local interface,
RIP ignores the update and does not
install the routes coming from this
source address in the routing table.
• Note: This same issue exists with
IGRP. In the case of EIGRP and OSPF,
routers will not be able to form
neighbor relationships if they are not
on the same subnet. OSPF performs
the subnet number and mask check on
all media except point-to-point and
virtual links.
• In Figure 1, the R1 Serial 0 interface is
unnumbered to Loopback 0. The
Router R2 serial interface is
numbered. When router R2 receives a
RIP update from R1, the source
address is considered invalid because
the source address is not on the same
subnet as the R2 serial interface.
• Debugs and Verification
Figure 2 shows the configuration of
both router R2 and R1. In this
configuration, the R1 Serial 0 interface
is unnumbered to Loopback 0. The R2
Serial interface is numbered.
Invalid source address
Solution
• When one side is numbered
and the other side is
unnumbered, this check must
be disabled. This is usually the
case in a dialup situation when
remotes are dialing into an access
router. The access routers dialup
interface is unnumbered, and all
remote routers get an IP address
assigned on their dialup
interfaces.
• This same solution can also
apply when both sides of the
link are unnumbered.
• Figure 1 shows the configuration
change on router R2 to fix this
problem.
• Figure 2 shows that after
changing the configuration of R2,
the route gets installed in the
routing table.
Flapping routes
• Running RIP in a complex
environment can sometimes cause
the flapping of routes.
• Route flapping refers to the
constant deletion and
reinsertion of a route within the
routing table.
• To check whether the routes are
indeed flapping, check the
routing table and look at the
age of the routes. If the ages
are constantly being reset to
00:00:00, this means that the
routes are flapping.
• One of the most common reasons is
packet loss, which occurs when the
packet is dropped on the sender's or
receiver's interface.
• The most common environment this
occurs in is Frame Relay, which is
used in this scenario.
• The existence of packet loss can be
verified by utilizing the show
interface command and examining
the interface statistics to determine
if the number of packet drops is
constantly incrementing.
Flapping routes
Solution
• The show interfaces serial 0
command further proves that there
is some problem at the interface
level. Too many drops are occurring
at the interface. This is the cause of
the route flapping.
• In the case of Frame Relay, the
Frame Relay broadcast queue
might have to be tuned. Several
white papers on Cisco’s web site
discuss how to tune the Frame Relay
broadcast queue.
• In a non-Frame Relay situation,
the input or output hold queue
might need to be increased.
• Figure 1 shows that after fixing the
interface drop problem, the route
flapping disappears.
• The output from the show ip route
rip command displays that the
routes are stable in the routing table
and that the timers are set at a
value lower than 30 seconds.
Large routing tables
• Large routing tables can be
problematic for a network by
increasing the amount of bandwidth
required for routing updates as well as
leading to increased memory
requirements.
• Figure 1 shows an example of a
network that could produce a large
routing table. In this example, there
are only three routes. In a real network,
however, the number of networks
could be much higher and the problem
of a large routing table, much worse.
• R2 is announcing several subnets of
131.108.0.0 network. Notice that the
link between R1 and R2 is also part of
the 131.108.0.0 network, so auto-
summarization cannot play any role to
solve the problem of receiving a subnet
route that could be summarized. The
auto-summarization feature can only
work if the link between R1 and R2 is
on a different major network.
• Debugs and Verification
Figure 2 shows that in the configuration
of R2, the ip summary-address
command is not used under the Serial
1 interface to summarize the routes.
Large routing tables
Solution
• In this situation, autosummary is on
but is not helpful because the whole
network in within one major network.
Imagine a network with Class B
address space with thousands of
subnets. Autosummary cannot play
any role here because no major
network boundary is crossed.
• A new feature of summarization
was introduced in RIP starting
with Cisco IOS Software Release
12.0.7T. This feature is similar to
EIGRP manual summarization.
• Figure 1 shows the new configuration
that solves this problem using the ip
summary-address rip command.
This configuration reduces the size
of the routing table.
• This command can be used with
different masks, so if a network
has contiguous blocks of a
subnet, the router could be
configured to summarize
subnets into smaller blocks. This
would reduce the routes
advertised to the RIP network.
TROUBLESHOOTING IGRP
Discontiguous networks
• When a major network is
separated by another major
network, this is called a
discontiguous network.
Figure shows an example of
a discontiguous network in
which 137.99.0.0 is a major
network. The subnets of this
major network are separated
by another major network,
131.108.0.0. This situation
produces a discontiguous
network problem.
• Debugs and Verification
Figure shows the
configuration of routers R1
and R2. IGRP is enabled on
the Ethernet interfaces of
routers R1 and R2 with the
correct network statements.
Discontiguous networks
• Figure 1 shows the debug ip igrp
transactions command output
for router R1 and R2. The debugs
show that both routers are
sending to each other the
summarized major network
address for 137.99.0.0, instead of
their specific network addresses.
As a result, both routers will
ignore the less specific 137.99.0.0
update because they are already
connected to this major network.
• Solution
IGRP does not support
discontiguous networks. Several
solutions to this problem exist.
One solution is to configure a
static route on each router to the
specific 137.99.0.0 subnet on the
other router as shown in Figure 2.
Since IGRP is a classful routing
protocol and does not include the
subnet information in its routing
updates, configuring the static
routes is a “patch” to fix this
problem.
Discontiguous networks
• Another solution is to replace the
classful IGRP routing protocol with a
classless routing protocol, such as
RIP v2, OSPF, EIGRP, or IS-IS. Unlike
RIP, there is not a version of IGRP
that is a classless routing protocol.
• If using IGRP is a must, there are two
solutions. The first solution is to
change the address on the link
between routers R1 and R2 to be
part of the 137.99.0.0 network. In
other words, assign another subnet
on this link, which is part of
137.99.0.0.
• If the addresses cannot be changed,
a Generic Routing Encapsulation
(GRE) tunnel can be configured
between routers R1 and R2. A
separate subnet address with the
same mask is configured on the
tunnel interface, as demonstrated.
• Discontiguous networks are
discouraged and should be avoided
even if a protocol supports it, as
they limit the summarization
capabilities. Figure shows the routing
table entry for R2 after fixing this
AS mismatch
• IGRP updates carry their autonomous
system (AS) number. When a receiver
receives an IGRP update and the AS
number of the sender does not match
with its own AS number, IGRP ignores
that update. As a result, no IGRP
routes are installed in the routing
table. Multiple IGRP processes can be
run under different AS numbers. These
processes are independent of each
other.
• Note: The same is true with EIGRP, as
the AS numbers must also match, or
EIGRP routers will not form a neighbor
relationship with its adjacent router.
• Debugs and Verification
Figure shows the configuration of both
routers R1 and R2. R1 is running IGRP
AS 1, and R2 is running IGRP AS 2.
• Figure shows the output of the debug
ip igrp transaction and debug ip
packet 100 detail commands on
routers R1 and R2. IGRP is ignoring
these updates because of the AS
number mismatch. Unfortunately, the
debug output does not show the
mismatch message.
AS mismatch
• Solution
In this example, the sender is
sending AS 1 in the update.
When R2 receives it, it
ignores this update because
R2 is running IGRP under AS
2. To fix this problem, change
the IGRP configurations so
that R1 and R2 both agree on
one AS number.
• Figure 1 shows the new
configuration on R2 that fixes
the problem.
• Figure 2 shows that after
fixing the AS mismatch
problem, IGRP routes get
installed in the routing table.
Misconfigured neighbor statement
• By default, IGRP sends out its
updates as a broadcast. IGRP
provides a unicast method of
sending IGRP updates using the
neighbor statement. IGRP will
not send the unicast update to
the wrong neighbor or
nonexistent neighbor. Figure
shows two routers running IGRP
between each other.
• Note: RIP v1 sends updates as
broadcasts, whereas RIP v2,
EIGRP, OSPF, and IS-IS use
multicasts. The same neighbor
statement can be used with RIP
v1.
• Debugs and Verification
Figure shows the IGRP
configuration for router R1. The
configuration shows that the
neighbor statement is
configured incorrectly. Instead of
131.108.1.2, as shown in Figure ,
the neighbor statement points
to 131.108.1.3, which does not
Misconfigured neighbor statement
• Solution
IGRP is sending a
unicast update to
131.108.1.3, a neighbor
that does not exist. To
resolve this problem,
the neighbor
statement must be
configured correctly.
• Figure 1 shows the
correct configuration on
router R1.
• Figure 2 shows the IGRP
routes installed in the
Maximum paths
• By default, for load balancing
purposes, Cisco routers
support only four equal
paths. The command
maximum-path can be
used for up to six equal-cost
paths. If the command is not
configured properly, it can
cause problems.
• Figure 1 shows a network
setup with multiple equal-
cost paths from router R1 to
the 131.108.2.0/24 network.
• Figure 2 shows the routing
table entry for router R1,
where only one route is
being installed in the routing
Maximum paths
• Debugs and Verification
Figure 1 shows the output of
debug ip igrp transactions
command on router R1, revealing
that R1 is receiving two equal-
cost path routes. However, the
routing table showed that only
one route is being installed.
• Figure 2 shows the current
configuration for router R1. The
maximum-path command has
been configured with the value of
1, which prevents IGRP from
installing more than one path in
the routing table.
• By default, the maximum-path
is set to four.
• If the command maximum-path
1 is used, it will only allow one
path to the destination even
though more than one path
exists. The maximum-path 1
command should only be used
when load balancing is not
desired.
Candidate default
• Except for IGRP, the way to set
the gateway of last resort for all
other routing protocols is to
define a static route to 0.0.0.0
with a mask of 0.0.0.0.
• That route can then be
propagated within RIP and OSPF
with the default-information
originate command, and within
EIGRP with the redistribute
static command.
• IGRP does not understand the
0.0.0.0 static route, so there is a
different way to set the gateway
of last resort in IGRP.
• Figure 2 shows a sample network
where router R1 will send default
traffic out the 155.155.155.0/24
network. Router R1 wants to
propagate this gateway of last
resort to router R2.
• Figure 3 shows the configuration
of router R1, without a gateway
of last resort configured.
Candidate default
• Debugs and Verification
Figure shows the routing table in R2,
which R2 is receiving
155.155.155.0/24, but it is not a
candidate for default.
• Solution
IGRP is incapable of carrying the
0.0.0.0/0 (also known as a default
route). Instead, IGRP uses the
default-network command to mark
a network as a candidate for default.
• In this example, R1 is sending
155.155.155.0/24 and it is desirable
to make R1 a candidate for default. To
do that, the configuration on R1 must
be changed to establish 155.155.0.0
as the default network. Upon doing
this, IGRP will automatically start
treating 155.155.155.0/24 as the
candidate for default and will set the
gateway of last resort on router R2.
• Figure 2 shows the configuration for
default-network on R1. This ip
default-network statement must
always point toward a major network,
not a subnet, otherwise, it will not set
the gateway of last resort.
TROUBLESHOOTING EIGRP
Mismatched K values
• For EIGRP to establish its neighbors, the K constant value to manipulate the EIGRP
metric must be the same. In the EIGRP metric calculation, the default for the K
value is set so that only the bandwidth and the delay of the interface are used to
calculate the EIGRP metric. At times, the network administrator might want other
interface values, such as load and reliability, to determine the EIGRP metric.
Therefore, the K values must be changed. Because only bandwidth and delay are
used in the calculations, the remaining K values are set to a value of 0 by default.
• Note: Cisco usually recommends that network administrators do not modify the
default metric of using bandwidth and delay.
• However, the K values must be the same for all the routers or EIGRP will not
establish a neighbor relationship. An example of two routers with mismatched K
values is illustrated.
• Modifying the K values will affect how much, if any, of bandwidth, delay, reliability,
and load will have on the metric calculation.
• Modifying these K values will have the following affect:
– K1 for bandwidth
– K2 for load
– K3 for delay
– K4 and K5 for reliability
Mismatched K values
• Debugs and Verification
Figure shows the K values on RTR B
have been changed to include load
and reliability, along with bandwidth
and delay. K1, K2, K3, and K4 have
all been set to 1.
• The metric weights command
shows that these values have been
modified. The first value is the type
of service (TOS) which is not
supported, but must be included.
• The command show ip eigrp
neighbors on both RTR A and RTB B
would show an empty list.
Troubleshooting this problem
requires careful scrutiny of the
routers configuration.
• Solution
The solution is to configure the same
K values on all routers in the EIGRP
domain. Figure shows the
modification made to RTR A to
match the K values on RTR B. At this
point both routers will be able to
establish a neighbor relationship and
exchange routing information.
Mismatched AS number
• EIGRP will not form any neighbor
relationships with routers that have
different autonomous system
numbers.
• If the AS numbers are mismatched,
no adjacency is formed. This problem
is usually caused by misconfiguration
on the routers.
• Figure 1 shows two routers running
EIGRP.
• Debugs and Verification
Figure 2 shows the configurations of
both routers.
• Because the two routers have
different AS numbers, using the
command show ip eigrp neighbors
on both RTR A and RTB B would show
an empty list.
• Solution
The solution is to modify one of the
two routers so the AS numbers are
the same.
• Figure 3 shows both routers
configured with the same AS number,
EIGRP AS 1. At this point, both routers
will be able to establish a neighbor
relationship and exchange routing
Stuck in active – determining the problem
• When troubleshooting an EIGRP stuck in active problem, two questions
need to be answered:
– Why is the route active?
– Why is the route stuck?
• Determining why the route is active is not necessarily a difficult task.
Sometimes, the route that constantly is going active could be due to a
flapping link. Or, if the route is a host route (/32 route), it is possible
that it is from a dial-in connection that gets disconnected. A more
important and difficult task is determining why an active route
becomes stuck. Usually, an active route gets stuck for one of the
following reasons:
– Bad or congested link
– Low router resources, such as low memory or high CPU processing
on the router
– Long query range
– Excessive redundancy
• By default, the stuck in active timer is only 180 seconds. If the EIGRP
neighbor does not hear a reply for the query in 180 seconds, neighbors
are reset. This adds difficulty in troubleshooting EIGRP routes stuck in
active because when an active route is stuck, there is only 180 seconds
to track down the active route query path and find the cause.
Stuck in active – determining the problem
• Debugs and Verification
The tool that helps troubleshoot the
EIGRP stuck in active error is the
show ip eigrp topology active
command.
• Figure 1 shows sample output from
this command. This command shows
what routes are currently active, how
long the routes have been active, and
which neighbors have and have not
replied to the query. From the output,
it can be determined which neighbors
have not replied to the query. Track
the query path and find the status of
the query by hopping to the
neighbors that have not replied.
• Figure 2 shows another example of
the show ip eigrp topology active
command output. The only difference
between the two outputs is the list of
neighbors that have not replied to the
router.
• However, this does not mean that all
of the neighbors have replied to the
queries. The neighbor 1.1.1.2 has a
lowercase r next to the address of
1.1.1.2. This also means that the
neighbor has not replied to the
Stuck in active – determining the problem
• In other words, the router has two ways of representing neighbors that
have not replied to the queries. One way, is to have them listed under the
Remaining replies: section. The other is to have an r next to the neighbor
interface IP address. When using the show ip eigrp topology active
command, the router can use any combination of these methods to
represent neighbors that have not yet replied to the queries.
• The neighbors that have not replied to the queries are 1.1.1.2 and
10.1.5.2. Only one of the non-replying neighbors 10.1.5.2 is listed under
the Remaining replies: section. The other neighbor, 1.1.1.2, that has not
replied is listed with the other replying neighbor. To summarize, when
issuing the show ip eigrp topology active command, the most
important part to look for is the neighbors that have not replied to the
query. To look for such a neighbor, look for neighbors that have
the r next to their interface IP address.
Stuck in active – methodology for troubleshooting
Stuck in active – methodology for troubleshooting
• Consider the network shown for
an example of troubleshooting the
EIGRP “stuck in active” problem.
Router A has a FastEthernet
interface with network
20.2.1.0/24 that just went away.
Router A does not have a feasible
successor to go to as a backup
route. Router A has no choice but
to put the 20.2.1.0/24 route into
active state and query its
neighbor, Router B.
• Notice the output of show ip
eigrp topology active in Router
A. The 20.2.1.0/24 route has gone
away for 1 minute and 12
seconds, and the neighbor that
has not responded is listed as
10.1.1.2 from Serial 0/0, which is
Router B. The next step is to
Telnet to Router B to see the
active route status in Router B.
Figure 3 shows the active route
status in Router B by performing
the command show ip eigrp
topology active.
Stuck in active – methodology for troubleshooting
• The command show ip eigrp topology active on Router B
shows that the route 20.2.1.0/24 is also in active status in
Router B and that it has gone active for 1 minute and 23
seconds. Most importantly, Router B cannot reply to Router A
about 20.2.1.0/24 because Router B is still waiting for the
neighbor with IP address of 10.1.3.2 (Router D) from Serial 1/2
to reply to the query. The next step is to go to Router D to see
the status of the active route 20.2.1.0/24 to see why Router D
has not replied to the query. Figure 1 shows the output of show
ip eigrp topology active on Router D.
Stuck in active – methodology for troubleshooting
• Router D also put the router 20.2.1.0/24 in active state, and it
has been in active state for 1 minute and 43 seconds. Router D
can not answer the query from Router B because Router D is
waiting for the router with the IP address of 10.1.5.2 from Serial
1/2 (Router E) to respond to the query. The next step is to go to
Router E to see the status of the active route 20.2.1.0/24 and to
find out why Router E is not replying to the query. Figure shows
the status of the active route on Router E.
Stuck in active – methodology for troubleshooting
• The output for show ip eigrp topology active did not show
anything for Router E. This indicates that, as far as Router E is
concerned, there are no routes in active state. The next step is
to Telnet back to Router D to double-check whether the router is
still in the active state for route 20.2.1.0/24. Telnetting back to
Router D shows that Router D is still in active state for route
20.2.1.0/24, but Router E does not have any routes in active
state.
• To summarize the event so far:
– Router A went active for route 20.2.1.0/.24 and is waiting for
Router B to reply to its query.
– Router B cannot reply because it is waiting for the query response
from Router D.
– Router D cannot reply because it is waiting for Router E to reply to
the query.
– Finally, the show ip eigrp topology active command in Router E
shows that Router E does not think that any routes are active,
while going back to Router D shows that the route 20.2.1.0/24 is
still in active state.
• From this sequence of events, you can see that there is clearly
a discrepancy between Router D and Router E. More
• A look at Router D and Router E CPU
utilization and memory usage does
not show a problem. The CPU
utilization and available memory are
normal for both routers. Look at the
Router D neighbor list to see if there
is a problem with the neighbors.
Figure 1 shows the Router D EIGRP
neighbor list.
• Notice that there is a problem in
Router D with EIGRP sending a
reliable packet to the neighbor with IP
address of 10.1.5.2 (Router E). The Q
count is 1, and performing the show
ip eigrp neighbors command a few
times in succession shows that the Q
count is not decrementing.
• The RTO counter is at its maximum
value of 5000ns. This indicates that
Router D is trying to send a reliable
packet to Router E, but Router E
never acknowledges the reliable
packet back to Router D. Because
Router E does not appear to have a
high CPU or memory problem, the
link should be tested for reliability
between Router D and Router E.
Figure shows the output of a ping
Stuck in active – the ultimate solution
• The ultimate solution for preventing
the EIGRP stuck in active problem is
to manually summarize the routes
whenever possible and to have a
hierarchical network design.
• The more networks EIGRP
summarizes, the less work EIGRP
has to do when a major convergence
takes place. Therefore, this reduces
the number of queries being sent
out and ultimately reduces the
occurrence of EIGRP stuck in active
errors.
• An example of a poor network
design that will not scale in a large
EIGRP network is shown in Figure 1.
• The best practice is to readdress the
IP addressing scheme. One region
should take only a block of IP
addresses, this way, the core routers
would be capable of summarizing
the routes into the core, resulting in
a reduced routing table in the core.
The routers and the query would be
contained only in one region. Figure
2 shows an improved and more
scalable EIGRP network design
Duplicate router IDs
• There are times that EIGRP will
not install routes because of a
duplicate router ID problem.
EIGRP does not use router ID
as extensively as OSPF. EIGRP
uses the notion of router ID
only for external routes to
prevent loops. EIGRP chooses
the router ID based on the
highest IP address of the
loopback interfaces on the
router. If the router does not
have any loopback interface,
the highest active IP address
of all the interfaces is chosen
as the router ID for EIGRP.
Figure 1 shows the network
setup for this example on
EIGRP router IDs. Figure 2
shows the pertinent
configurations for the cause of
Duplicate router IDs
• Debugs and Verification
The problem is that Router A is
not installing the 150.150.0.0/16
route in the routing table. As a
matter of fact, Router A is not
showing the 150.150.0.0/16 route
in its topology table. Going back
to Router B, the route is in the
routing table, and the topology
table appears.
• Solution
The solution to the duplicate
router ID problem is to change the
IP address of the loopback
interface of Router X or to change
the IP address of Ethernet 0 on
Router A. The rule of thumb is to
never configure the same IP
address on two places in the
network. Figure shows the change
the loopback IP address of Router
X to 192.168.9.1/32 to fix this
problem. The result of the IP
address change in Router X is the
installment of the 150.150.0.0/16
route in Router A.
EIGRP error messages
• DUAL-3-SIA
This message means that the primary route is gone and no
feasible successor is available. The router has sent out the
queries to its neighbor and has not heard the reply from a
particular neighbor for more than three minutes. The route
state is now stuck in active state.
• Neighbor is not on common subnet
This message means that the router has heard a hello packet
from a neighbor that is not on the same subnet as the router.
• DUAL-3-BADCOUNT
Badcount means that EIGRP believes that it knows of more
routes for a given network than actually exists. It is typically,
but not always, seen in conjunction with DUAL-3-SIAs, but it is
not believed to cause any problems itself.
• Unequal, <route>, dndb=<metric>, query=<metric>
This message indicates that there is an EIGRP internal error.
However, the router is coded to fully recover from this internal
error. The EIGRP internal error is caused by a software problem
and should not affect the operation of the router. The plan of
action is to report this error to the Cisco TAC and have the
experts decode the traceback message. At some point the Cisco
EIGRP error messages
• IP-EIGRP: Callback: callback_routes
At some point, EIGRP attempted to install routes to the destinations
and failed, most commonly because of the existence of a route with
a better administrative distance. When this occurs, EIGRP registers
its route as a backup route. When the better route disappears from
the routing table, EIGRP is called back through callback_routes so
that it can attempt to reinstall the routes that it is holding in the
topology table.
• Error: EIGRP: DDB not configured on interface
This means that when the routers interface receives an EIGRP hello
packet and the router attempts to associate the packet with a DUAL
descriptor block (DDB) for that interface, it does not find one that
matches. This means that the router is receiving a hello packet on
the interface that does not have EIGRP configured.
• Poison squashed
The router threads a topology table entry as poison in reply to an
update (the router set up for poison reverse). While the router is
building the packet that contains the poison reverse, the router
realizes that it does not need to send it. For example, if the router
receives a query for that route from the neighbor, it is currently
threaded to poison.
EIGRP error messages
TROUBLESHOOTING OSPF
Mismatched parameters
• For OSPF to form and maintain
neighbor adjacencies, several
parameters must match.
• OSPF neighbors exchange Hello
packets periodically to form neighbor
relationships. If specific parameters
do not match, the neighbor adjacency
will not be formed and the routers will
not exchange OSPF updates.
• Most mismatch issues can be seen
using the debug ip ospf adj
command.
Hello/Dead Interval Mismatch
• The output of R2 debug ip ospf adj,
when the neighbor Hello interval does
not match with Router R2, is shown in
Fig. 2. R refers to “Received” and C
refers to “Configured”.
• Figure 3 shows the configuration of
routers R1 and R2. R2 has the default
Hello interval of 10 seconds, whereas
R1 has been configured with the Hello
interval of 15 seconds. Both the Hello
and the Dead intervals must match
for routers to form an adjacency.
• Figure 4 shows the corrected
configuration of R1.
Mismatched parameters
• Figure 1 displays the output of the
show ip ospf neighbor
command. This shows that after
configuring the same Hello
interval, OSPF forms an
adjacency.
Mismatched Authentication Type
• The output of R2 debug ip ospf
adj indicates that the R2 neighbor
is configured for MD5
authentication and that R2 is
configured for plain-text
authentication.
• In Figure 3, the configurations of
routers R1 and R2 show that both
are using different authentication
methods.
• To fix this problem, both routers
must be configured to use the
same type of authentication,
which can be verified using the
show ip ospf neighbor
Mismatched parameters
Mismatch Area ID
• OSPF sends area information in
Hello packets. If both sides do not
agree that both routers are
members of a common area, no
OSPF adjacency will be formed.
The area information is part of the
OSPF protocol header.
• The output of debug ip ospf adj
on R1, as shown in Figure 1,
indicates that R1 is receiving an
OSPF packet from R2 and that the
OSPF header has area 0.0.0.1.
Looking at the configurations on
both routers, shown in Figure 2
proves that the two routers are
configured for different areas.
• The solution to this problem is to
configure the same area across
this common link. To solve this
problem, change the area of
router R1 to area 1. Using the
show ip ospf neighbor
command will verify that the
routers have formed a neighbor
Mismatched parameters
Mismatched Stub/Transit/NSSA
Area Options
• When OSPF exchanges Hello packets
with a neighbor, one of the things that
it exchanges is an optional capability
represented by 8 bits. One of the
option fields is for the E bit, which is
the OSPF stub flag. When the E bit is
set to 0, the area with which the route
is associated is a stub area, and no
external LSAs are allowed in this area.
• If the E bit does not match on both
sides, an adjacency will not be
formed. This is called an optional
capabilities mismatch.
• The output of debug ip ospf adj on
R1 indicates a stub/transit bit
mismatch. Figure 2 verifies the
configuration mismatch.
• The solution would be to configure the
routers with the appropriate stub
commands. A configuration change on
R1 that fixes the problem is shown in
Fig. 3. Router R1 is now also a part of
the stub area. Using the show ip
ospf neighbor command will verify
that the routers have formed a
neighbor relationship
OSPF state issues
• OSPF Stuck in ATTEMPT
This problem is only valid for NMBA networks in which neighbor
statements have been defined.
• Stuck in ATTEMPT means that a router is trying to contact a
neighbor by sending its Hello, but it has not received a response.
The output of show ip ospf neighbor indicates this problem
exists.
• The most common possible causes of the problem are:
– Misconfigured neighbor statement
– Unicast connectivity is broken on NBMA, which could be due
to a wrong DLCI, an access list, or NAT translating the unicast.
OSPF state issues
• OSPF Stuck in INIT
The INIT state indicates that a router sees hello packets from the neighbor, but
two-way communication has not been established. A Cisco router includes the
router IDs of all neighbors in the INIT (or higher) state in the neighbor field of its
hello packets. For two-way communication to be established with a neighbor, a
router also must see its own router ID in the neighbor field of the hello packets
coming from the neighbor router. The output of the show ip ospf neighbor
command indicates that the router is stuck in INIT.
• The most common possible causes of this problem are:
– An access list on one side is blocking OSPF Hellos.
– Multicast capabilities are broken on one side (switch problem).
– Authentication is enabled on only one side.
– The frame-relay map/dialer map statement on one side is missing
the broadcast keyword.
– Hellos are getting lost on one side at Layer 2.
OSPF state issues
• OSPF Stuck in 2-WAY
The 2-WAY state is an indication that the router has seen its own
Router ID in the neighbor field of the neighbor hello packet.
• The reception of a Database Descriptor (DBD) packet from a
neighbor in the INIT state will also cause a transition to 2-way
state. The OSPF neighbor 2-WAY state is not a cause for concern.
• On networks that require a DR/BDR election to take place, if all
routers are configured with priority 0, there will be no election
process.
• All routers will remain in 2-WAY state, as the show ip ospf
neighbor command will indicate. The solution is to make sure
at least one router has an ip ospf priority of at least 1.
OSPF state issues
• OSPF Stuck in EXSTART/EXCHANGE
OSPF neighbors that are in EXSTART or EXCHANGE state are in the process of
trying to exchange DBD packets. The router and its neighbor form a
master/slave relationship. The adjacency should continue past this state. If it
does not, there is a problem with the DBD exchange, such as MTU mismatch, or
the receipt of an unexpected DBD sequence number.
• The most common possible causes of this problem are:
– Mismatched interface MTU.
– Duplicate Router IDs on neighbors.
– Inability to ping across with more than certain MTU size.
– Broken unicast connectivity, which could be due to a wrong DLCI,
an access list, or NAT translating the unicast.
• The debug ip ospf adj command can help diagnose these problems, as shown
in Figure , which indicates an MTU interface mismatch.
OSPF state issues
• OSPF Stuck in LOADING
In the LOADING state, routers send link-state
request packets. During the adjacency, if a
router receives an outdated or missing link-
state advertisement (LSA), it requests that
LSA by sending a link-state request packet.
Neighbors that do not transition beyond this
state are most likely exchanging corrupted
LSAs. This problem is usually accompanied
by a "%OSPF-4-BADLSA" console message.
– Since this is not a common
problem, it is recommend that the
network administrator contacts the
Cisco TAC.
• If a neighbor does not reply or a neighbor
reply never reaches the local router, the
router will also be stuck in the LOADING
state.
• The most common possible causes of the
problem are:
– Mismatched MTU
– Corrupted link-state request
packet
• The output of show ip ospf neighbor
indicates that the R2 neighbor is stuck in
LOADING.
• The debug ip ospf adj command can help
diagnose these problems.
– Command output can indicate an MTU
interface mismatch.
– Command output can also indicate a
possible problem due to packet
One side of point-to-point link is unnumbered
• Figure shows a network setup in which the R1 serial link is
unnumbered to loopback interface, while the R2 serial link is
numbered.
• When OSPF creates a router LSA for point-to-point links, it
adheres to the following rule to fill the Link ID and Link Data
fields within the router LSA.
• The Link Data field for unnumbered point-to-point links should
have a MIB-II Index value. Because the link data value of the
numbered interface does not match that of the unnumbered
interface, this creates a discrepancy in the OSPF database.
One side of point-to-point link is unnumbered
• Debugs and
Verification
Figure 1 shows the
discrepancy in the OSPF
database.
• The R1 router LSA
shows the MIB-II IfIndex
value in the Link Data
field for Serial 0, while
the R2 router LSA shows
the router serial
interface in the Link
Data Field.
One side of point-to-point link is unnumbered
• Figure shows the
configuration on both R1
and R2, with the R1
serial interface
unnumbered to a
loopback address, and
the R2 serial interface
numbered.
• Solution
To fix this problem, both
sides need to be either a
numbered point-to-point
link or an unnumbered
point-to-point link.
Figure shows the new
configuration that solves
ABR not generating Type 4 summary LSA
• One of the functions of a Type
4 summary LSA is to
announce the reachability of
an ASBR to other areas. The
Type 4 LSA is not required if
the ASBR exists in the same
area.
• The ASBR does not generate
the Type 4 summary LSA if it
is not connected to area 0. To
generate a summary LSA of
Type 3 or Type 4, a router
must have a connection to
area 0. As a result, the
external routes will not be
installed in the network.
• Figure 1 shows a network in
which R3 redistributes RIP
routes into OSPF.
• Figure 2 shows that R1 is not
installing the external route
200.200.200.0/24 into the
ABR not generating Type 4 summary LSA
• Debugs and Verification
The output of show ip ospf
database external command in
Figure 1 shows that the route exists in
the external OSPF database of router
R1.
• However, the output of show ip ospf
database asbr-summary command
in Figure 2 shows that there is no Type
4 LSA for this route.
• The next logical step is to go on the
ABR and see if it is indeed an ABR. If
it is an ABR it will generate a
summary LSA of Type 3 or Type 4.
• If it is not an ABR, it will not generate
that summary. The output of the
show ip ospf command in Figure 3
shows that router R2, which is
between two areas and may look like
an ABR, is not identifying itself as an
ABR. If R2 were an ABR, the output
would display that is an “Area Border
Router.”
• Note: All areas in an OSPF
autonomous system must be
physically or virtually connected to
the backbone area (area 0). For a
router to be an ABR, one of its
ABR not generating Type 4 summary LSA
• Solution
In this example, router R2 does
not generate the Type 4 summary
LSA because it is not connected to
area 0. To generate a summary
LSA of Type 3 or Type 4, a router
must have a connection into area
0.
• To solve this problem, router R2
must be connected to area 0,
either physically or virtually by
creating a virtual link, as shown in
Figure 1. By configuring a virtual
link on R2, the router is now
virtually connected to area 0;
therefore, it now considers itself
an ABR. After connecting R2 to
area 0, the output of show ip
ospf shows that it is now an ABR.
• After the configuration change on
R2, as shown in Figure 3, R1 is
now generating a Type 4 summary
LSA into area 3. Because the Type
4 LSA is now being received, R1
installs the external route
200.200.200.0/24 into its routing
Forwarding address is not known through the intra-area
or interarea route
• When OSPF learns an external LSA, it
makes sure that the forwarding address is
known through an OSPF intra-area or
interarea route before it installs the route
into the routing table. In accordance with
the RFC 2328 standard, if the forwarding
address is not known through an intra-
area or interarea route, OSPF will not
install the route in the routing table.
• Figure 1 shows a network with the
following specifications:
– R3 is an ASBR and is redistributing
RIP routes into OSPF.
– R4 is running RIP with R3.
– R4 is learning 200.200.200.0/24
through RIP.
– R2 is running OSPF with R3.
– R2 is the ABR.
• The network is experiencing a problem of
external routes not getting installed in
the routing table.
• Figure 2 shows the output of show ip
route for 200.200.200.0/24 on router R1.
This network resides in a RIP domain.
Because RIP is being redistributed into
OSPF on router R3, all OSPF routers
should see this external OSPF route.
However, router R1 is not seeing it in its
routing table.
Forwarding address is not known through the intra-area
or interarea route
Debugs and Verification
• Figure 1 shows the external LSA for
200.200.200.0/24 on R1. The output of
the show ip ospf database external
command shows that the external LSA
exists in the OSPF database, but the route
is still not being installed in the routing
table. Note the forwarding address
involved in this external LSA.
• Figure 2 shows that the route to the
forwarding address of 131.108.0.4, is
known through an OSPF external route.
The ABR is summarizing 131.108.0.0/24
with the area range command, so the
more specific intra-area routes are
summarized into one route.
• This range summarizes all routes under
the 131.108.0.0/26 range. (Fig. 3)
• Figure 4 shows that the ASBR is doing the
redistribution from RIP into OSPF.
• It also shows that the connected
networks in the range of 131.108.0.0/26
are being redistributed into OSPF because
RIP owns those connected routes.
• This configuration redistributes
131.108.0.4/26, which is a connected
interface on router R3.
• This subnet covers the forwarding
address that appeared in Figure 2.
Forwarding address is not known through the intra-area
or interarea route
Solution
• The problem can be solved in one of
two ways:
• Do not summarize at the ABR.
– To implement the first solution,
remove the area range
command on the ABR.
• Filter the connected subnet from
being redistributed into OSPF at
the ASBR.
– To implement the second
solution, add a filter to control
the redistributed routes on the
ASBR.
– A route-map is added to the
redistribute rip command, as
shown in Figure 2 to prevent the
route 131.108.0.0/26 from
getting redistributed into OSPF,
permitting only the
200.200.200.0/24 route.
• Figures 3 and 4 show the changes in
the R1 routing table for the external
200.200.200.0/24 route and that the
forwarding address is now known
through OSPF interarea instead of
Route summarization problems
• Route summarization allows a group of contiguous networks to be
summarized as fewer addresses in order to help reduce the size of the
routing table, thus increasing the performance of OSPF and other routing
protocols.
• OSPF can use two types of summarization:
– Interarea route summarization that can be done on the ABR.
– External route summarization that can be done on the ASBR.
• Figure 1 shows a network in which a router is not summarizing interarea
routes.
• Debugs and Verification
Router R1 has the area range command for summarization of area 3
routes. The syntax of the command is correct, but the problem is that R1
is not an ABR. Router R1 is included in area 0, but it is not connected to
area 3, and thus cannot summarize area 3 routes.
Route summarization problems
• A way to check if a router is
summarizing the routes properly is
to use the show ip ospf command.
The output of show ip ospf shows
that the area 3 range is passive.
• Passive means that no addresses
within this area fall within this range.
In fact, R1 does have the routes in
this range; however, because R1 is
not connected to area 3, the range
appears as passive.
• Because summarization is not
happening, R1 is receiving four
routes in the routing table instead of
one summarized route.
Solution
• The solution to this problem is to
configure the area range command
on the ABR, which is R2.
• After changing the configuration and
removing the area range command
on R1, verify that the summarization
is indeed active by using show ip
ospf on R2.
• The summarized route is now being
installed in the routing table of R1.
Route summarization problems
• Note: There are similar issues
with external summarization,
which is done on the ASBR.
• External summarization uses the
summary-address command
only on the ASBR. The show ip
ospf summary-address
command can be used to help
troubleshoot external
summarization issues.
• A metric of infinity, shown in
Figure 1, means that no valid
addresses belong to this range
of summarization and that there
is a problem (16,777,215
equates to infinity because the
external LSA metric is 24 bits
long and 224 is equal to
16,777,215).
• Figure 2 shows the same show
ip ospf summary-address
command, when summarization
is configured correctly on the
ASBR, with a metric less than
CPUHOG problems
• When OSPF forms an adjacency, it floods all the link-state update packets to its
neighbors. Sometimes, the flooding process takes a lot of time, depending upon
the router resources. CPUHOG messages appear in the log if the resources of the
router become exhausted due to flooding.
• CPUHOG messages usually appear in two significant stages:
– Neighbor formation process
– LSA refresh process
• Neighbor formation process
When OSPF forms an adjacency, it floods all of its link-state packets to its
neighbor. This flooding sometimes takes a lot of CPU processing. Starting with
Cisco IOS Software 12.0T, packet pacing was included to help prevent CPU
processing problems from occurring. Prior to 12.0T, the IOS did not support
packet pacing.
• Figure shows the log messages on a router showing CPUHOG during adjacency
formation. Packet pacing introduces a delay of 33 ms between packets and 66
ms between retransmissions. This pacing interval reduces the CPUHOG
messages, and the adjacency is formed more quickly.
• If a router is experiencing this problem with an IOS prior to 12.0T, the
solution is to upgrade to 12.0T or higher.
CPUHOG problems
• LSA refresh process
Starting with Cisco IOS 12.0, the LSA group pacing feature was
introduced to eliminate a CPU processing problem that can
occur every 30 minutes.
• Prior to IOS 12.0, all LSAs are flooded every 30 minutes to
refresh the MAXAGE times in the link state database. Without
this flooding, the LSAs would expire after 60 minutes. This
flooding is also known as a paranoid update.
• This flooding causes CPUHOG messages every 30 minutes,
especially in cases where a couple of thousand LSAs are all
being flooded at the same time. The CPUHOG messages appear
in the router log every 30 minutes.
• LSA group packing looks at the LSA periodic interval (every 4
minutes, by default) and refreshes only those LSAs that are past
their refresh time. This is an efficient way of reducing a large
flood by chopping it down to smaller LSA floods.
• No extra configuration is required for this feature, but for large
numbers of LSAs (generally 10,000 or more), it is recommended
to use small intervals (for example, every 2 minutes); for a few
hundred LSAs, use a large interval, such as 20 minutes.
CPUHOG problems
• If 10,000 LSAs need to be refreshed, keeping the refresh
interval smaller will check the LSA every 2 or 4 minutes to see
how many LSAs have reached the refresh interval, which is 30
minutes.
• The advantage of checking this frequently is that few LSAs
would need to be refreshed every 2 or 4 minutes, and this will
not cause a huge storm of LSA updates.
• If the number of LSAs is small, it really does not matter whether
the refresh occurs at 2 minutes or 20 minutes. This is why it is
better to increase the timer so that all the LSAs, which are few
in number, can be refreshed at once.
• Figure shows how to configure the LSA refresh interval
SPF calculation and route flapping
• Whenever there is a change in • The following is an example of SPF
topology, OSPF runs the SPF running constantly due to an interface
algorithm to compute the Shortest flap within the network. This is a common
Path First tree again. problem in OSPF. Whenever there is a link
flap in an area, OSPF runs SPF within that
• Unstable links existing within the area.
OSPF network could cause constant • So, if a network has unstable links, it can
SPF calculations. cause constant SPF calculations. SPF
• itself is not a problem because OSPF is
Some of the most common just adjusting to the change in the
reasons for SPF running database through calculating SPF.
constantly are: • The real problem occurs if there are small
– Interface flaps within the routers in the area and a constant SPF
area run might cause a CPU spike in a router.
– Neighbor interface flaps • Because R1 is also included in area 0, any
within the area link flap in area 0 causes all routers in
area 0 to run SPF.
– Duplicate router ID
• Topology changes in an OSPF
network cause SPF calculations
within that area. The SPF is not
recalculated if the topology change
is in another area.
• Actually, OSPF distributes interarea
(between areas) topology
information using a distance-vector
method. The ABRs forward routing
information between areas using a
distance vector technique similar to
SPF calculation and route flapping
Debugs and Verification
• A link flap in an area causes SPF to
run. If a link is flapping constantly, this
can increase the number of SPF
calculations in an area. A constant
number of SPF calculations in not a
problem, but if the number is
incrementing constantly, it is an
indication of a problem.
• The output of show ip ospf, shows
that there is a huge counter for SPF in
area 0.
• The easiest way to find out which
particular LSA is flapping is to turn on
debug ip ospf monitor. This debug
command in Figure 2 shows that a
router LSA is flapping in area 0.
• The next step is to determine which
router LSA is flapping and check the
log for any interface flap.
• The show log command in Figure 3
shows the log of the router with router
ID 192.168.1.129, the ID displayed in
the output of the debug ip ospf
monitor command. The log shows
that a serial link keeps going up and
down. Whenever there is an interface
flap, it causes SPF to run.
SPF calculation and route flapping
Solution
• There are two solution to this problem:
– Fix the link that is flapping.
– Redefine the area boundaries.
• Sometimes, the first solution might not be manageable
because the link is flapping as the result of a telco
outage that is outside the network boundary. One way to
fix this temporarily is to manually shut down the
interface.
• The second solution requires some redesigning. If the
link flap is happening too often, it might be possible to
redefine the area, exclude this router from the area, and
make it a member of a totally stubby area. Sometimes,
this is also difficult to implement, depending upon the
physical location of this link.
• In short, link flaps are realities; if there are too
many link flaps, the number of routers in an area
should be decreased so that fewer routers are
TROUBLESHOOTING IS-IS
IS-IS adjacency problems
• IS-IS adjacency-related problems normally
are caused by link failures and configuration
errors. On Cisco routers, inspecting the
output of the show interface command can
easily identify link failures
• Also, because IS-IS routing is not required to
establish IP connectivity to directly attached
routers, it is easy to discern whether the
problem is media-related or specific to the
IS-IS configuration.
• The show clns neighbors command is
usually the starting point for troubleshooting
IS-IS adjacency problems. The output of this
command should list all neighbors expected
to be adjacent to the router being
investigated
• The command show clns is-neighbors
provides similar output, but it is intended to
list only neighbor routers or IS-IS
adjacencies; whereas show clns neighbors
lists all types of adjacencies, both for IS-IS
and ES-IS.
• Using the output from the show clns
neighbor and show clns neighbor detail
commands, IS-IS adjacencies can be
examined.
• Problems with IS-IS adjacency formation can
be registered by the presence of fewer
neighbors than expected or a situation in
which the status of an expected adjacency is
not up. Another symptom could be that the
neighbor is known through ES-IS protocol
instead of IS-IS.
IS-IS adjacency problems
• Figure also shows output from show clns neighbors, but shows
problems with the adjacencies formed with RT2 and RT5. In this
example, the IS-IS adjacency with RT2 is in INIT state instead of UP. The
protocol is correctly shown as IS-IS. The adjacency with RT5 shows UP,
however the protocol is ES-IS instead of IS-IS. As explained previously,
the ES-IS protocol runs independently of IS-IS; therefore, the ES-IS
adjacency formed between RT1 and RT5 has nothing to do with IS-IS.
• These routers cannot form an IS-IS adjacency with each other,
apparently because of a problem in the configuration or the IS-IS
environment. Most adjacency problems related to the IS-IS
environment can be debugged with the debug isis adj-packets
command. The output of this command can be daunting if the router
under inspection has a lot of neighbors because the display shows all
the hellos transmitted and received by the local routers.
Some or all of the adjacencies are not coming up
• The absence of some expected
IS-IS adjacencies means that
the affected routers will not be
capable of exchanging routing
information, effectively creating
reachability problems to certain
destinations in the network.
• Figure 1 shows a simple
network in which four daisy-
chained routers are grouped in
twos and placed in separate
areas.
• Figure 2 and 3 display different
outputs of the show clns
neighbors command captured
on RT1. Figure displays two
neighbors, while Figure displays
only one neighbor instead of
the expected two. RT2 is listed,
but RT5 is missing.
Some or all of the adjacencies are not coming up
Steps and Possible Causes
• Step 1: Checking for link failures
The first step is to check for link failures. On Cisco routers this can be
done using the show ip interface brief command. If there is a
problem, the appropriate interfaces will display something other than
the up/up state. Shown is an example if there is a link failure problem.
The solution here would be to correct the Layer 1/2 problem.
• Step 2: Checking for configuration errors
If the link is fine, the next step would be to verify the IS-IS
configuration. Make sure the IS-IS process is defined and that an NSAP
and NET is configured. Unlike other IP routing protocols, the IS-IS
configuration does not use network statements to enable IS-IS routing
on the router interfaces.
– To enable IS-IS routing for IP on a Cisco router, the ip router isis command
must be configured on the appropriate interfaces. Make sure the router-level
passive-interface command is not being used to disable IS-IS routing on
these interfaces. When an interface is made passive, the ip router isis
command is automatically removed from the interface.
Some or all of the adjacencies are not coming up
• Step 3: Checking for mismatched Level 1 and Level 2 interfaces
If the configuration looks correct, the next step is to check for mismatched Level 1
and Level 2 interfaces. IS-IS supports a two-level routing hierarchy in which
routing within an area is designated as Level 1 and routing between areas in
designated as Level 2. An IS-IS router can be configured to participate in Level 1
routing only (Level 1 router), Level 2 routing only (Level 2 router) or both (Level 1-
2 router). Level 1-2 routers act as border routers between IS-IS areas and facilitate
interarea communication.
• In the default mode of operation, Cisco routers have Level 1-2 capability. Two
directly connected routers with a common area ID will form a Level 1-2 adjacency
by default, even though only a Level 1 adjacency is necessary for them to
communicate. Use the router-level is-type command to change this behavior.
• In Figure 1, it is desirable to make RT5 a Level 1-only router while RT1 remains
capable of Level 1-2. This requires RT5 to be configured with the is-type level-1
command, but nothing needs to be done on RT1. If RT1 is made a Level 2-only
router with is-type level-2-only command, it will not be capable of forming a
Level 1 adjacency with RT5. The proper setup is to make RT5 a Level 1 router only
if it has limited resources (memory and CPU); RT1 should be a Level 1-2 router for
it to communicate with RT5 at Level 1 and with RT2 at Level 2 because RT2 is in
another area. Just as with RT5, RT6 can be a Level 1-only router, if necessary.
Some or all of the adjacencies are not coming up
• Step 4: Checking for area
misconfiguration
Two routers in different areas with
different area IDs consequently
are assigned to different areas,
therefore, form only a Level 2
adjacency. RT5 is configured as
Level 1 only but its area ID is
misconfigured to be different from
the area ID of RT1. These two
routers will not form any
adjacency. The configuration in
Figure 1, even though RT1 is
expected to be in area 49.001,
has been configured with an area
ID of 49.005 placing it in a
different area from RT5.
Therefore, RT5 must be Level 2-
capable to form adjacencies with
RT1. However, it has been made a
Level-1 router with the command
is-type level-1. Therefore, no IS-
IS adjacency will be formed
between RT1 and RT5. The
solution here would be to change
the area on RT5 to the proper
Some or all of the adjacencies are not coming up
• Step 5: Checking for misconfigured
subnets
In recent releases of Cisco IOS Software,
particularly in 12.0S, 12.0T, and 12.0T
release trains, adjacencies will not be
formed between two neighbors if the
directly connected interfaces are not in
the same IP subnet. In Figure 1, the IP
address of RT1 is changed to that of
another subnet. In Figure 2, the output of
debug isis adj-packet shows RT1 is
rejecting the hello received from RT5
because the interface address 10.1.1.5
advertised by the latter is not on subnet
10.1.8.0/24.
• In earlier Cisco IOS Software releases, it
did not matter whether the routers
belonged to the different IP subnets
because IS-IS adjacency formation occurs
primarily within the CLNP framework,
where IP addresses are irrelevant.
However, in IP applications, directly
connected routers must be on the same
subnet, except when IP unnumbered is
used. Therefore, the recent behavior
provides an extra check for the IP
configuration while introducing sanity into
IS-IS data structures for tracking IP
information.
• In summary, it is important to make sure
that directly connected routers that need
to form IS-IS adjacencies for IP routing are
Some or all of the adjacencies are not coming up
• Step 6: Check for duplicate
systems IDs
If the previous steps checked out
okay, but a specific neighbor is
not in the show clns neighbor
output, it is possible that
adjacency is not being formed
because that neighbor has a
duplicate system ID with the local
router. An IS-IS router will not
form an adjacency with a router
in the same area that has a
duplicate system ID. It also logs
duplicate system ID errors.
• Use the show logging command
to display entries in the log. If
duplicate system ID errors are
found, the source of the conflict
can be determined from the
output of debug adj-packets.
This points to the interface where
the hellos with the duplicate
system ID are coming from.
Adjacency is stuck in INIT state
• The most common causes for an
adjacency getting stuck in INIT state
are mismatched interface MTU and
authentication parameters.
• Figure 1 shows two routers running
IS-IS.
• The output from show clns
neighbors shows what an
adjacency would look like when
stuck in INIT.
Steps and Possible Causes
• Step 1: Checking authentication
If IS-IS authentication is configured,
the first step in tackling this problem
is to address potential issues in this
area.
• The Cisco implementation allows
authentication to be configured in
three ways: at the domain, area, or
interface levels.
• It is important to be sure that the
appropriate method is properly
configured and that the passwords
used are consistent. The output of
the debug isis adj-packets
command indicates authentication
Adjacency is stuck in INIT state
• Step 2: Checking for
mismatched MTUs
If there are no authentication
issues, the next possibility is
mismatched MTUs. The show
clns interface command can
quickly verify the MTUs on the
other side of the link. The
output of debug isis adj-
packets shows when the MTU
is changed to produce a
mismatch.
Adjacency is stuck in INIT state
• Step 3: Checking for Disabling of
IS-IS Hello Padding
Cisco IOS Software releases starting
with 12.0S and 12.0ST, allow hello
padding to be disabled to reduce
significant and unnecessary
bandwidth consumption in some
network environments.
• Hello padding is disabled with the
assumption that the MTUs match.
• This step suggest making sure that
hello padding is configured
consistently on either side. In general,
only suppressing hello padding should
not affect the adjacency, as long as
the hellos sent out on the transmitting
side are smaller than the MTU on the
receiving side.
• Also, disabling hello padding does not
disable verification of the maximum
acceptable size of received hello
packets.
• The debug isis adj-packets
command can be used to troubleshoot
these issues.
• The show clns interface command
in Figure 2 shows the status of hello
padding on an interface.
ES-IS adjacency instead of IS-IS adjacency formed

• Cisco routers running IS-IS in IP environments still listen to ISHs,

intermediate-system hellos, generated by ES-IS protocol, in
conformance with ISO 10589 requirements. When the physical
and data-link layers are operational, an ES-IS adjacency can be
formed even if appropriate conditions do not exist for establishing
an IS-IS adjacency.
• Figure 1 shows what the output for the show clns neighbors
command looks like when an ES-IS adjacency is formed instead of
an IS-IS adjacency. This is usually because IS-IS hellos are not
being processed, as a result of interface MTU mismatch or
misconfigured authentication.
Route advertisement problems
• Most route advertisement problems
can be narrowed down to
configuration problems at the source
or LSP propagation issues.
• Because IS-IS is a link-state protocol,
IS-IS routers depend on LSP flooding
to obtain topology and routing
information. During stable conditions,
each IS-IS router in an area will have
the same Level 1 link-state database,
which contains the LSPs generated by
every router in the area.
• Dijkstra’s algorithm is run over the LS
database to obtain the best path to
every advertised route. If a route is
missing in a section of the area, it is
because the routers in that section
did not receive the original LSP, or
the LSP was received corrupted,
therefore, was purged.
• An even simpler reason could be that
the route was not even put into the
LSP at the source. The outputs of
debug isis update-packets and
debug isis snp-packets could help
decipher this sort of problem as it is
related to LSP flooding or issues with
link-state database synchronization.
Route flapping problem
• Route flaps normally are caused by unstable links between the
source of the route and the location where the flap is being
observed. Typically, multiple routes are impacted at the same
time because of an adjacency change affecting many LSPs, all of
which might carry numerous IP prefixes. Also, route flaps could
induce consistent running of SPF process, causing dangerously
high CPU utilization on route processors that might crash
affected routers. If the advertised LSP changes affect only IP
prefixes, only partial route calculation (PRC) is run instead of full
SPF calculations. PRC is less costly in terms of CPU cycles than
full SPF runs.
• Apart from certain destinations not being reachable, obviously
implying routing problems, high CPU utilization by the SPF
process (show process cpu command) certainly should flag
instabilities in the network. High CPU utilization can be observed
through the IOS command line interface, network management
applications, or special network-performance analysis tools, such
as CiscoWorks.
• The show process cpu command can obtain CPU utilization
information. If any indication of high CPU utilization caused by
the SPF process is determined, the show isis spf-log command
can determine SPF-related events that might cause the high CPU
Route flapping problem
• The output of the show isis
spf-log command lists SPF
process runs by time, trigger,
and duration.
• LSPs are refreshed every 15
minutes, triggering periodic
SPF calculations.
• Such events are labeled
periodic in the show isis
spf-log output.
• It also shows that at
03:25:25, the process was
run over an insignificant
period of time because of
receipt of a new LSP from
RT5.
• Figure 2 lists and describes
Route flapping problem
• The following is a list of IS-IS
debugging commands that can
also be used to narrow SPF-
related problems:
– debug isis spf-events
– debug isis spf-triggers
– debug isis spf-statistics
– debug isis update-packets
– debug isis adj-packets
• Use caution when enabling
debugging in a situation in
which the route processor CPU
already is overtaxed. • The show isis spf-log command can
provide an indication of which LSPs
• Output of the debug isis spf-
are changing the most frequently and
events command displays the
events following an interface are triggering the SPF calculations.
shut down to simulate a link • Similar clues can be gleaned by
flap. Lines indicating LSPs enabling debugging of the update
flagged for recalculation as a process with debug isis update-
result of this event are packets.
highlighted. Also, events,
flagging computation of the • This should be done with care so
Level 1 and Level 2 SPF trees that the CPU is not overloaded.
TROUBLESHOOTING BGP
Troubleshooting BGP neighbor relationships
• Troubleshooting neighbor relationship issues should follow the OSI
reference model. First, Layer 1/2 should be checked, followed by
IP connectivity (Layer 3), TCP connections (Layer 4), and finally
the BGP configuration. The following is a list of problems most
commonly seen when forming BGP neighbor relationships:
– Directly connected external BGP neighbors not initializing
– Non-directly connected external BGP neighbors not
initializing
– Internal BGP neighbors not initializing
– BGP neighbors (external and internal) not initializing
• Directly connected external BGP neighbors not initializing
The autonomous system (AS) will not send or receive any IP
prefix updates to or from a neighboring AS unless the neighbor
relationship reaches the Established state, which is the final stage
of BGP neighbor establishment.
– When an AS has a single EBGP connection, no IP connectivity
can occur until BGP finalizes its operation of sending and
receiving IP prefixes.
Troubleshooting BGP neighbor relationships
• Figure shows a network in which
an external BGP neighbor
relationship is configured
between AS 109 and AS 110. The
most common possible causes of
directly connected external BGP
neighbors not initializing are:
– Layer 2 is down, preventing
communication with directly
connected EBGP neighbor.
– An incorrect neighbor IP
address is in the BGP
configuration.
• The BGP neighbor relationship
can be verified by using the
show ip bpg summary and
show ip bgp neighbors
commands. This command
shows that the BGP neighbor is
in Active state.
• This state indicates that no
successful communication
between the neighbors has taken
place and the BGP has failed to
form a neighbor relationship.
Troubleshooting BGP neighbor relationships
• Misconfiguration of the neighbor
IP address is a fairly common
mistake, and it can be caught
with a visual inspection of the
configuration. However, in a
large IP network, this might not
be a trivial task.
• The debug ip bgp command,
as demonstrated in Figure 1 can
help diagnose this problem.
• Observe that router R2 is
having difficulty communicating
with host 131.108.1.11, which
has a misconfigured neighbor
address on R2, as shown in
Figure 2.
• The solution here would be to
correct the neighbor address in
the configuration of router R2.
Troubleshooting BGP neighbor relationships
• Nondirectly connected external
BGP neighbors not initializing
In some cases, EBGP neighbors are
not directly connected. BGP
neighbor relationships can be
established between routers trying
to make an EBGP neighbor
relationship that are separated by
one or more routers. Such a
neighbor relationship is termed
EBGP multihop in Cisco IOS
Software.
• Peering between loopbacks • The most common possible causes
between EBGP typically is done for nondirectly connected external
when multiple interfaces exist BGP neighbors not initializing are:
between the routers, and IP traffic
– The route to the nondirectly
needs to be load-shared among
connected peer address is
those interfaces. Such a
missing from the routing table.
connection is considered
nondirectly connected. – The ebgp-multihop command is
missing in BGP configuration.
• Figure 1 shows an example of a
nondirectly connected EBGP – The update-source interface
session between two loopback command is missing.
Troubleshooting BGP neighbor relationships
• The show ip bpg summary and show ip
bgp neighbors commands can be used to
show that the neighbor relationship is in
Active state.
• In the case of ebgp-multihop command
missing in the BGP configuration, these
commands will show that the BGP neighbor
is in Idle state, because no resources are
allocated to the BGP neighbor. This might
be because the other side has not received
any BGP negotiation
• The solutions to these possible causes vary
and depend upon the exact situation.
• Using the scenario in Figure 2, the
following are some possible solutions.
• The solutions for the route to the
nondirectly connected peer address is
missing from the routing table is to either
use a static route (common practice) to
the connected peer address or a use an
IGP dynamic routing protocol such as OSPF.
This is usually an issue when peering is
done between peers using loopback
addresses.
• A possible solution to the ebgp-multihop
command missing in the BGP configuration
is to properly configure this command.
• Figure 3 shows an example of configuring
this command on router R1.
Troubleshooting BGP neighbor relationships
• In the case of the update-source interface command missing,
using the example in Figure 1, R1 and R2 should be configured with
the update-source command. The update-source command
ensures that the source address is that of Loobback 0.
Internal BGP neighbors not initializing
• IBGP can experience similar issues to EBGP in neighbor relationship.
IBGP is an important piece of BGP-running networks. The causes of
IBGP neighbor relationship issues are identical to those of EBGP:
– The route to the nondirectly connected IBGP neighbor is missing.
– The update-source interface command is missing in BGP configuration.
• The same troubleshooting and configuration techniques can be
used for IBGP neighbor issues that are used for EBGP neighbor
issues.
Troubleshooting BGP neighbor relationships
• BGP neighbors (external and
internal) not initializing
• Interface access list/filters are
another common cause of BGP
neighbor activation problems. If
an interface access list
unintentionally blocks TCP
packets that carry BGP protocol
packets, the BGP neighbor will
not come up.
• Figure 1 shows a sample access
list 101 that explicitly blocks TCP.
Access list 102 has an implicit
deny of BGP because of the
implicit deny at the end of each
access list.
• Figure 2 shows the revised
access configuration that permits
the BGP port (TCP port 179). All
BGP packets will be permitted
because of the second line in
Troubleshooting BGP route advertisements
• Another common problem that BGP operators face occurs in
BGP route advertisement/origination and receiving. BGP
originates routes only by configuration. However, it needs no
configuration in receiving routes.
• Larger ISPs originate new BGP routes for their customers on a
daily basis, whereas small-enterprise BGP networks mostly
configure BGP route origination upon first setup. Problems in
route originating can occur because of either configuration
mistakes or a lack of BGP protocol understanding.
• It is beyond the scope of this section to discuss all of the
possible problems that can occur, but a list of possible
problems that will be briefly discussed are:
– A BGP route not getting originated
– Problem in propagating/originating a BGP route to
IBGP/EBGP neighbors
– Problem in propagating a BGP route to an IBGP neighbor
but not to an EBGP neighbor
– Problem in propagating an IBGP route to an IBGP/EBGP
neighbor
Troubleshooting BGP route advertisements
• A BGP route not getting originated
• BGP originates IP prefixes and announces them to neighboring BGP speakers (IBGP and EBGP) so
that the Internet can reach those prefixes.
• For example, if an IP address associated with www.cisco.com fails to originate because of a BGP
configuration mistake or lack of protocol requirements, the Internet will never know about the IP
address of www.cisco.com, resulting in no connectivity to this web site. Therefore, it is essential
to look at BGP route-origination issues in detail.
• Several causes in failure of BGP route origination exists, the most common of which
are:
– The IP routing table does not have a matching route.
– A configuration error has occurred.
– BGP is autosummarizing to a classful/network boundary.
• BGP requires the IP routing table to have an exact matching entry for the prefix that BGP is trying
to advertise using the network and redistribute commands. The prefix and mask of the
network that BGP is trying to advertise must be identical in the IP routing table and in the BGP
configurations.
• Figure shows a misconfiguration in BGP to adverstise 100.100.100.0/24 using the network
statement. The static route for 100.100.100.0 has a mask of /23, whereas BGP is configured to
advertise /24. Therefore, BGP will not consider /24 as a valid advertisement because an exact
match does not exist in the routing table
Troubleshooting BGP route advertisements
• Problem in propagating/originating a
BGP route to IBGP/EBGP neighbors
• A scenario might arise in which the BGP
configuration to originate and propagate
routes looks good, but BGP neighbors are
not receiving the routes.
• The originator’s BGP table shows all the
routes.
• There is a possibility that configured
distribute-list filters are the cause of the
problem or a problem in the policy routing.
• Problem in propagating a BGP route
to an IBGP neighbor but not to an
EBGP neighbor
• In some cases, certain routes are not
propagated to IBGP neighbors, but are
propagated only to EBGP neighbors.
• When IBGP speakers in an AS are not fully
meshed and have no route reflector or
confederation configuration, any route
that is learned from an IBGP neighbor will
not be given to any other IBGP neighbor.
Such routes are advertised only to EBGP
neighbors.
• Figure 2 shows the configuration of the
routers R8, R1, and R2.
• This example shows that the IBGP network
is not fully meshed and that the IBGP-
learned route will not be given to any
other IBGP neighbor.
Troubleshooting BGP route advertisements
• Verify this with the show ip
bgp command. “Not
advertised to any peer” in
R1 means that even though
R2 is the neighbor, it is an
IBGP neighbor, therefore,
100.100.100.0/24 will not be
advertised.
• It is essential that IBGP-
learned routes are
propagated to other BGP
speakers.
• BGP operators can use
three methods to address
this problem:
– Use IBGP full mesh.
– Design a route-reflector
model.
– Design a confederation
Troubleshooting BGP route advertisements
Troubleshooting BGP route advertisements
• Problem in propagating an IBGP route to an IBGP/EBGP neighbor
A problem might arise in which an IBGP learned route is not propagated to
any BGP neighbor, whether it is an IBGP or EBGP neighbor. One case could be
that when an IBGP-learned route is not synchronized, that route is not
considered as a candidate to advertise to other BGP neighbors. A BGP route is
synchronized only if it has been first learned through an IGP or static route.
• In Cisco IOS Software, BGP advertises only what it considers the best path to
its neighbors. If an IBGP path is not synchronized, it is not included in the best
path calculation. The output of the show ip bgp command will show
unsynchronized routes in the BGP table. The solution would be to either turn
off synchronization or make the routes synchronized by redistributing them in
the IGP at the router that first introduced this route into the IBGP domain.
Troubleshooting routes not being installed in the IP
routing table
• A common problem is with BGP routes not
getting installed into the IP routing table. If
a router must forward an IP packet by
looking at the IP destination address of the
IP packet, the router must have an IP
routing table entry for the subnet of the IP
destination address.
• If the BGP process fails to create an IP
routing table entry, all traffic destined for
missing IP subnets in the routing table will
be dropped. This is generic behavior of
hop-by-hop IP packet forwarding done by
routers.

• The most common causes of IBGP-learned routes not

getting installed in the IP routing table are:
– IBGP routes are not synchronized.
– The BGP next hop is not reachable.
• The most common causes of EBGP-learned routes not
getting installed in the IP routing table are:
– BGP routes are dampened.
– The BGP next hop is not reachable in the case of multihop
EBGP.
– The multiexit discriminator (MED) value is infinite.
The BGP next hop is not reachable
• A common problem in BGP occurs
when the next hop is not reachable.
The cause of this problem, IBGP-
learned route is not getting installed
into the IP routing table, is most
common in IBGP-learned routes
where BGP next-hop address should
have been learned through an IGP.
Failure to reach the next hop is an
IGP problem, and BGP is merely a
victim. With BGP, when IP prefixes
are advertised to an IBGP neighbor,
the NEXT-HOP attribute of the prefix
does not change. The IBGP receiver
must have an IP route to reach this
next hop.
• Figure 1 shows the NEXT-HOP of BGP
routes advertised to IBGP neighbors
are not changed and might result in
route installation failure.
• Debugs and Verification
R8 is advertising the
100.100.100.0/24 route to its EBGP
peer R1, which will advertise this
route to R2. However, on R2, the
problem of the next hop appears.
Figure 2 shows the configuration of
BGP on RT8, RT1, and RT2.
The BGP next hop is not reachable
• In Figure 1, R2 is an IBGP peer of
R1. R2 receives this BGP route,
100.100.100.0/24 with a NEXT-
HOP of 206.56.89.1, but fails to
install it in its IP routing table
(Fig. 2).
Solution
• Remember that for EBGP
sessions, the next hop is the IP
address of the neighbor that
announced the route. For IBGP
sessions, routes that originated
inside the AS, the next-hop is
the IP address of the neighbor
that announced the route.
• For routes injected into the AS
via EBGP, the next hop learned
from EBGP is carried unaltered
into IBGP. The next hop is the IP
address of the EBGP neighbor
from which the route was
The BGP next hop is not reachable
• Two common solutions exist for addressing this
problem:
– Announce the EBGP next hop through an IGP
using a static route or redistribution.
– Change the next hop to an internal peering
address, using the next-hop-self command.
• Change the configuration of R1 so that the
subnet 206.56.89.0/30 is included in its OSPF
advertisements of R1 to R2, which would include
the address of the next-hop IP address of R8.
Figure 2 shows that R2 receives this route
through OSPF.
• Another solution is to change the BGP next hop
address on R1 to its loopback address when
advertising IBGP routes to R2.
• Figure 3 shows the configuration on R1 that
modifies the BGP next hop to be changed to its
own loopback address.
• The command neighbor 131.108.10.2 next-
hop-self changes the next-hop to its own
loopback 0 (131.108.10.1).
• The neighbor 131.108.10.2 update-source
loopback 0 command makes R1 loopback 0 the
source of all BGP packets sent to R2.
• The command show ip bgp in Figure 4 shows
this change reflected in R2. The exterior next-
hop changed to the loopback of R1,
BGP routes getting dampened
• Dampening is the way to minimize instability in a local BGP network caused by
unstable BGP routes from EBGP neighbors. RFC 2439, “BGP Route Flap
Dampening,” describes in detail how dampening works. In short, dampening is the
way to assign a penalty for a flapping BGP route.
• A withdrawal of a prefix is considered a flap. A penalty of 1000 is assigned for
each flap; if the flap penalty reaches the suppress limit because of continued flaps
(default 2000), the BGP path is suppressed and is taken out of the routing table.
This penalty is decayed exponentially based on the half-life time (default 15
minutes). When the penalty reaches the reuse value (default 750), the path is
unsuppressed and is installed in the routing table and advertised to other BGP
neighbors. Any dampened path can be suppressed only until the maximum
suppress time is reached (default 60 minutes). Dampening is applied only to EBGP
neighbors, not to IBGP neighbors.
• BGP dampening is off by default. The following BGP command activates
dampening:
– router bgp 109
bgp dampening
• Cisco IOS Software allows dampening parameters to be changed and are defined
as follows:
– bgp dampening half-life-time reuse suppress maximum-suppress-time
• The range of values for these options are:
• half-life-time – 1 to 45 minutes, Default is 15 minutes.
• reuse – 1 to 20,000, Default is 750
• suppress – 1 to 20,000, Default is 2,000.
• maximum-suppress-time – Maximum duration that a route can be
BGP routes getting dampened
• Figure 1 shows a simple EBGP
network between R1 and R2 in AS
109 and AS 110, respectively. R2 has
advertised 100.100.100.0/24 to R1.
To show how dampening works, R2 is
made to flap 100.100.100.0/24
multiple times. Removing the route
in R2 routing table and putting it
back again can simulate flapping. R1
receives these flaps and, if
configured with dampening, assigns
penalties per flap.
• Debugs and Verification
Figure 2 shows the necessary debug
commands, debug ip bgp
dampening and debug ip bpg
updates, to observe the dampening
feature in R1. Most debugs can be
run along with an access list to limit
the output created by these debugs.
Access list 1 is permitting only the
100.100.100.0 network.
• Figure 3 shows the debug output and
flap statistics in BGP output.
Highlighted debug and show
command output shows that
100.100.100.0/24 has flapped four
times in 3 minutes and 13 seconds.
BGP routes getting dampened
Solution
• Note: Actually, route dampening
can be considered the solution
regarding ill-behaved Internet
routes and keeping the Internet
routers in a stable state.
• If R1 wants to reinstall
100.100.100.0/24, it can do the
following:
– Wait for the penalty to go below
the reuse limit (750).
– Remove dampening altogether
from the BGP configuration.
– Clear the flap statistics.
• Figure 1 shows how the dampened
path can be cleared and
immediately get installed in the
routing table. The output in Figure
1 is from the debug ip bgp
update 1 command which is on
to display the activity in the BGP
process.
• This output shows
100.100.100.0/24 going into the IP
routing table.
Troubleshooting outbound and inbound BGP policy
issues
• The real power of BGP is in managing IP traffic flows coming in
and going out of the AS. BGP in general and Cisco IOS Software
in particular offer a great deal of flexibility in manipulating BGP
attributes LOCAL_PREFERENCE, MED, and so forth to control BP
best-path calculation.
• This best-path decision determines how traffic exits the AS.
With the large size of BGP networks today, it is crucial that BGP
operators understand how BGP attributes should be managed.
• Some of the most common problems encountered in
managing outbound traffic flow:
– Multiple exit points exist, but traffic goes out through one
or a few exit routers.
– Traffic takes a different interface from what is shown in the
routing table.
– A multiple BGP connection exists to the same BGP neighbor,
but traffic goes out through only one connection.
– Asymmetrical routing occurs and it causes a problem
especially when NAT and time-sensitive applications are
used.
Troubleshooting outbound and inbound BGP policy
issues
• Some of the most common problems in
managing inbound IP traffic in an AS using BGP
are:
– Multiple connections exist to an AS, but all the traffic
comes in through one BGP neighbor, in the same AS.
– A BGP neighbor in an AS should just be a backup
provider, but some traffic from the Internet still comes
through that AS.
– Asymmetrical routing occurs.
– Traffic to a certain subnet should come through a
particular connection, but it is coming from
somewhere else.
• Each of these problems, both with outbound and
inbound policy issues, could require extensive
troubleshooting. In the next section, the problem of
traffic taking a different interface from what is shown
in the routing table will be examined more thoroughly
Traffic takes a different interface than what is shown in
routing table
• In some scenarios, BGP and the
routing table path to a certain
destination prefix show Exit A, but
actual traffic leaves through Exit B.
Packet forwarding to a destination
takes place from the routing table,
and network operators do expect to
see this behavior.
• However, in most cases, the next
hops of prefixes in the routing table
are not directly connected and
packet forwarding eventually takes
place based on the next-hop path.
• Figure shows that R1 and R2 are two
route reflectors, with R6 and R8 as
their clients. R6 is advertising
100.100.100.0/24 to R2 and R1, and
both reflect this advertisement to R8
with a next hop of 172.16.126.6.
• Now assume that R8 has a BGP
policy that chooses the path for
100.100.100.0/24 from R2 (the
upper path) as the best path that it
will install in its routing table.
However, in the same router, R8, the
best IGP path to reach 172.16.126.6
(next hop of 100.100.100.0/24) is
Traffic takes a different interface than what is shown in
routing table
Debugs and Verification
• The show ip bgp command output
in Figure 1 shows the output of
100.100.100.0/24 in R8. The next
hop is 172.16.126.6. When traffic is
sent to 100.100.100.0/24, it actually
is sent to the interface that provides
a better route for 172.16.126.6.
• In R8, 172.16.126.6 is the next hop
for 100.100.100.0/24 advertised by
R2 and is considered the best route;
therefore, it will be installed in the IP
routing table.
• Figure 2 shows that the best way to
reach 172.16.126.6, the next hop of
100.100.100.0/24, is through R1, not
through R2.
• The highlighted 172.16.18.1 is the
next hop for 172.16.126.6 (next hop
of 100.100.100.0/24).
• Therefore, all traffic from or through
R8 destined for 100.100.100.0/24
will go though 172.16.18.1 (R1).
• The output of a traceroute done
from R8 to 100.100.100.1. shows
that traffic flows through
TROUBLESHOOTING
REDISTRIBUTION
Redistribution problems with RIP
• The most prevalent problem encountered
with redistribution is that redistributed
routes are not being installed in the routing
table of the routers receiving these routes.
The most common cause of this is a metric
that is not defined during redistribution into
RIP.
• In RIP, the metric for a route is treated as a
hop count that shows the number of routers
that exist along this route. The maximum
hop count that RIP supports is 15; anything
greater than 15 is treated as the infinite
metric and, upon receipt, is dropped.
• Figure 1 shows a network that could
produce the problem in which redistributed
routes do not get installed in the routing
table of the receiver. R1 and R3 are running
OSPF in area 0, whereas R1 and R2 are
running RIP. R3 is announcing
131.108.6.0/24 through OSPF to R1. In R1,
OSPF routes are being redistributed into
RIP, but R2 is not receiving 131.108.6.0/24
through RIP.
• Debugs and Verification
The first step is to investigate whether R1 is
receiving 131.108.6.0/24. Figure 2 shows
that R3 is advertising 131.108.6.0/24
through OSPF to R1. Figure 3 shows that R1
is redistributing OSPF into RIP on R1
• The next step is to check whether or not R2
is receiving the 131.108.6.0/24 route.
Figure 4 shows that in R2, 131.108.6.0/24 is
Redistribution problems with RIP
• The output of debug ip rip on R2 which
shows that R2 is receiving the
131.108.6.0/24 route, but with a metric of
16 hops (infinity).
Solution
• In RIP, 16 is considered to be an infinite
metric. Any update with a metric greater
than 15 will not be considered for entry
into the routing table.
• In this example, an OSPF route in R1 for
131.108.6.0/24 has a metric of 20. When
OSPF is redistributed into RIP in R1, OSPF
advertised 131.108.6.0/24 with the default
metric of 20, which exceeds the maximum
metric allowed in RIP.
• OSPF knows only cost as a metric, whereas
RIP utilizes hop count. No metric translation
facility exists, so the administrator must
configure a metric to be assigned to
redistributed routes.
• To correct this problem, R1 needs to assign
a valid metric when configuring the
redistribution. This can be done with either
the metric option in the redistribute
command or with default-metric
command.
• In this configuration all redistributed routes
from OSPF into RIP get a metric of 1. This
metric is treated as hop count by R2.
• Figure 3 shows that R2 is now receiving the
correct route with a metric of 1.
Redistribution problems with IGRP/EIGRP
• IGRP has a composite metric made up of
bandwidth, delay, reliability, load, and
MTU. By default, it utilizes only bandwidth
and delay. Other routing protocols use
different metrics. For example, OSPF uses a
metric based on interface cost. OSPF cost
is derived from the bandwidth of the link.
Cisco uses 100,100,000/bandwidth to get
that cost.
• IGRP does not understand the metrics of
other protocols (except EIGRP) so it is
necessary to input a default metric when
doing redistribution.
• Figure 1 shows a network that could
produce the problem in which redistributed
routes do not get installed in the routing
table of the receiver. OSPF is redistributed
into IGRP at R1, but R2 is not receiving the
IGRP routes.
• R1 and R3 are running OSPF in area 0,
whereas R1 and R2 are running IGRP. R3 is
announcing 131.108.6.0/24 through OSPF
to R1. In R1, OSPF routes are being
redistributed into IGRP, but R2 is not
receiving 131.108.6.0/24 through IGRP.
• Debugs and Verification
Figure 2 shows that R3 is advertising
131.108.6.0/24 through OSPF to R1.
• Figure 3 shows that R1 is redistributing
OSPF into IGRP. However, examining the
routing table in R2, Figure 4 shows that
131.108.6.0/24 is not being installed in the
Redistribution problems with IGRP/EIGRP
• Solution
To solve this problem, R1 needs
to include the metric option
with the redistribute
statement, so that it can convert
the OSPF metric properly.
• Figure 1 shows the new
configuration for R1. OSPF is
redistributed into IGRP with
metric values of bandwidth,
delay, reliability, load, and MTU.
Even if reliability and load are
not being used in the IGRP
metric, these values must be
included in the redistribute
command.
• Another solution is to use the
default-metric command
under router igrp. Figure 2
shows the syntax for using the
default-metric command, with
the same values of bandwidth,
delay, reliability, load and MTU.
• The route is now installed in the
Redistribution problems with OSPF
• When a router in OSPF does redistribution, it becomes an ASBR.
The routes that are redistributed into OSPF could be directly
connected routes, static routes, or routes from another routing
protocol or another OSPF process.
• The two most common problems associated with OSPF
and redistribution are:
– OSPF is not installing external routes in the routing table.
– ASBR is not advertising redistributed routes.
OSPF is not installing external routes in the routing table
• When OSPF redistributes any routes whether connected, static,
or from a different routing protocol, it generates a Type 5 LSA
for those external routes. These Type 5 routes are flooded into
every OSPF router, with the exception of those in stub and NSSA
areas. Sometimes, the problem is that the external routes are in
the OSPF database but are not being installed in the routing
table.
• The most common causes of this problem are:
– The forwarding address is not known through the intra-area
or interarea route.
– The ABR is not generating Type 4 Summary LSAs.
Redistribution problems with OSPF
ASBR is not advertising redistributed routes
• Whenever a route is known to be connected or static, or when
any other routing protocol is redistributed into OSPF, an
external LSA is generated for that route. If an OSPF router is not
advertising the external route even after the redistribution, this
indicates a problem on a router that is doing the redistribution.
Mostly, the problem stems from configuration mistakes.
• The most common causes of this problem are:
– The subnets keyword is missing from the ASBR configuration.
– distribute-list out command is blocking the routes.
• Distribute list issues are discussed in the section on Common
IGP Routing Protocol Issues, Causes and Solutions.
• The next slide is an example of a problem caused by the
missing subnets keyword on the ASBR configuration.
• When any protocol is redistributed into OSPF, if the networks
that are being redistributed are subnets, the subnets keyword
must be used under the OSPF configuration.
• If the subnets keyword is not added, OSPF will ignore all
the subnetted routes when generating the external LSA.
Redistribution problems with OSPF
• Figure 1 shows a network
setup that is redistributing
into OSPF.
Debugs and Verification
• Figure 2 shows the output of
show ip ospf database
external for 132.108.3.0.
• The output shows no LSA
information, which means that
R1 is not even originating the
external LSA for 132.108.3.0.
• Figure 3 shows the OSPF
configuration of R1, and that
the redistribute rip
command under OSPF is
missing the subnets
keyword.
Redistribution problems with OSPF
• Solution
The solution to this problem is to add
the subnets keyword to the subnets
command under OSPF. After this option
has been added, OSPF will redistribute
all of the routes that are subnetted; for
example 131.108.3.0 with the /24
mask.
• Figure 2 shows that R1 is now
generating the external LSA for
132.108.3.0/24 and 132.108.4.0/24.
Redistribution problems with IS-IS
• Cisco IOS Software allows IP routes to be imported into IS-IS. Examples of the
external sources are static routes and other dynamic routing protocols such as RIP
and OSPF.
• The IP external reachability TLV is used for adding external routes into the IS-IS
domain. Even though RFC 1195 specifies the IP external reachability for only Level 2
LSPs, Cisco IOS Software provides a special capability for using them in Level 1 LSPs,
which allows external routes into a Level 1 area.
• Most service provider networks use IS-IS as the IGP in large single-area Level 1-only
or Level 2-only domains. For those with Level 1-only backbones, the capability to
redistribute into Level 1 provides flexibility to import external routes into the IS-IS
domain.
• Even though this behavior is not standardized, it should not pose interoperability
issues with other vendors’ routers because both existing IS-IS standards, ISO 10589
and RFC 1195, require IS-IS implementations to ignore unsupported or unknown
optional TLVs encountered while parsing IS-IS-packets.
• The IOS router-level command redistribute enables redistribution. This command
takes on other options, such as metric value, metric type, route map, and so on.
• In the Cisco implementation of IS-IS, CLNS static routes are automatically distributed
into IS-IS. However, IP static routes are redistributed only by manual configuration.
• When static IP routes need to be redistributed, the redistribute command requires
the keyword ip to go with it, in addition to the other arguments previously
mentioned. The metric type for external routes can be either internal or external.
• Internal metrics are comparable to metrics used for internal routes. External metrics
require the I/E bit (bit 7) of the metric field to be set in addition to the actual metric,
resulting in higher metric values.
• In current Cisco IOS Software releases, when using narrow metrics, bit 8 of the
default metric field is set for external metrics, resulting in an increase of the metric
Redistribution problems with IS-IS
• By default, the internal metric
type is assigned if nothing is
specified in the configuration.
Also, the external routes are
added into Level 2 unless
Level 1 is explicitly stated in
the configuration.
• Figure 1 illustrates basic
examples of redistribution in
IS-IS.
• In Figure 2, only the ip
keyword is used with the
redistribute command.
• Note that below show
running-config, the internal
metric type has been
assigned by default and the
metric applied is 0. The
output of show isis
database on RT1 shows that
the external static route has
been added to only the Level
Redistribution problems with IS-IS
• The metric type is explicitly set to
external in this configuration, but no
metric value is applied. As explained
previously, the I/E bit needs to then be
set for the external metric type,
effectively increasing the metric value
by 64.
• However, Cisco IOS Software set bit 8 of
the narrow metric instead of bit 7,
consequently adding 128 instead to the
original value of 0. The Level 2 LSP
displayed shows 128 as the metric value
for the external route, 172.16.1.0/24.
• The IP routing table output from RT2
shows the external route, 172.16.1.0/24,
which was redistributed from a static
source into IS-IS on router RT1.
• The metric entered for this route, 138, is
the total of the metric on the outgoing
interface from RT2 to RT1 (10) plus the
metric of 128 advertised by RT1. Other
routes received from RT1 (10.0.0.1/32
and 10.1.1.0/24) are registered with a
metric 20 (10 advertised by RT1 and
additional 10 for the metric from RT2 to
RT1).
Redistribution problems with IS-IS
• The route-map option of the
redistribute command provides
more flexibility for configuring
redistribution, such as selective
importation of external routes
into the IS-IS environment,
applying special tags, and even
setting the metric of redistributed
routes.
• When used for selective
importation of routes into IS-IS,
route maps provide a filtering
effect by controlling which
elements from an external source
are allowed or denied into IS-IS.
• In Figure, static routes are
redistributed into IS-IS while
filtering through the route map
TEST.
• Route map TEST matches the
static routes against access list 1,
which permits only 172.16.2.0/24
into the IS-IS environment. RT1
LSP is shown from RT2. Also
shown is the routing table of RT2.
Redistribution problems with IS-IS

In Figure, the route map

approach is used to set
the metric for routes
imported into IS-IS.
Redistribution problems with BGP
• In this example of • In Figure, for instance, AS
redistribution with BGP, the
discussion will focus more on 300 is advertising two
design and configuration routes: 192.168.250.0/24
issues, than with actual and 192.168.1.212/30. The
troubleshooting problems.
IGP for AS 300 and the
• On an AS border router,
outgoing route advertisements configuration or router Tahoe
affect incoming traffic, and are unimportant to this
incoming route advertisements example.
affect outgoing traffic. As a
result, outgoing and incoming
advertisements should be
considered separately. The
example in this section will
discuss the example of
injecting BGP routes into an
IGP, as this would be the
situation for most enterprise
networks.
• Prefixes that are learned from
an EBGP neighbor are
automatically added to the
Redistribution problems with BGP
• The important observations
are that the prefixes
advertised by Tahoe to its
external BGP peer are
displayed in the Taos routing
table as reachable and that
pings to a destination in AS
300 are successful.
• An extended ping is used
because the subnet of the
Taos serial interface,
192.168.1.224/30, is not
advertised.
• The BGP-learned routes are
tagged in the routing table
with a B.
Redistribution problems with BGP
• Although the networks of AS
300 are reachable from Taos,
the BGP routes must be
advertised into EIGRP before
the networks are reachable
from AS 200 interior routers.
One way to accomplish this
is with the redistribution at
Taos, as demonstrated by the
configuration in Figure 1.
• Figure 2 shows that AS 300
prefixes are advertised to
AngelFire and that the
destinations are reachable.
However, redistribution picks
up every BGP route, but the
administrator might want
only a subset of the BGP
routes to be redistributed. In
such a case, route filters are
required to suppress the
unwanted routes.
Redistribution problems with BGP
• Another vitally important reason
exists for not redistributing BGP
routes into an IGP.
• A full Internet routing table consists
of more than 100,000 prefixes, and
an IGP process will “choke” trying to
process so many routes.
• Redistribution of a full Internet table,
or even a large partial table, will
inevitably cause a major network
crash. Never use a BGP-to-IGP
redistribution on an Internet facing
router.
• For more control over which routes
are advertised into AS 200, static
routes can be used. In this
configuration only 192.168.250.0/24
is advertised into the AS.
• As Figure 2 shows, AngelFire has no
knowledge of subnet
192.168.1.212/30.
• Using static routes in configuration
has the added benefit of protecting
AS 200 from instabilities.
• If network 192.168.250.0 flaps in AS
300, the changes are not advertised
Redistribution problems with BGP
• Of course, in a single-homed AS,
such as AS 200 in Figure 1 little
reason exists to advertise any
external routes into the AS.
Unless there is a need to
advertise specific routes into the
AS, a default route suffices. In this
configuration, Taos generates a
default route and advertises it to
all EIGRP speakers; however BGP
can also be configured to
generate a default route. To
advertise a default from Vail to its
BGP neighbors, the configuration
in Figure 3 could be used.
• A default route to the Null0
interface is created statically, and
the route is advertised with the
network command. The
assumption with the configuration
is that Vail has full routing
information. All packets are
forward to Vail; any destination
address that cannot be matched
to a more-specific route matches
the static route and is dropped.
Redistribution problems with BGP
• In some design cases, a default should be sent to some neighbors,
but not to others. To send a default from Vail to Taos, but not to
any of Vail’s other neighbors, use the configuration in Figure.
• The BGP neighbor default-originate command is similar to the
OSPF default information-originate always command in that a
default is advertised whether the router actually has a default
route or not.
• Notice in the configuration that the static route from the preceding
configuration is no longer present; however, a route to 0.0.0.0/0 is
still advertised to Taos.
Redistribution problems with BGP
• Figure 1 shows the routing table of
Tahoe, and unlike Taos, Tahoe does
not have an entry for 0.0.0.0/0.
• The advertisement of a default route
to a BGP neighbor does not suppress
the more-specific routes.
• The routes from AS 300 are still
present in the Taos routing table. In
some cases, this can be desirable.
• For example, an ISP might send to a
customer the routes to all of its
other customers (a partial Internet
table), as well as a default to the
rest of the Internet.
• Such a case is useful when
multihomed to the same ISP. The
customer network can then make
best-path choices to the ISP’s
customers and use the default route
for all other external destinations.
• If only the default is to be sent, a
router must use a filter to suppress
all more-specific routers.
• The configuration in Figure 2, using
the neighbor distribute-list
command, is just one way to filter
Summary
• Using a methodology for troubleshooting Layer 3
problems is important to isolate and identify the
problem as quickly as possible.
• In most networks static routes are used in
combination with dynamic routing protocols.
Improper configuration of static routes can lead to
less than optimal routing and in some cases create
routing loops or parts of the network to become
unreachable.
• Troubleshooting dynamic routing protocols require a
thorough understanding of how the specific routing
protocol functions. Some problems are common to all
routing protocols while other problems are particular
to the individual routing protocol.
• Troubleshooting redistribution between routing
protocols requires an understanding of how the
particular routing protocols function and how to
redistribute their different metrics.
Q&A