Layer 4 Troubleshooting

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1-1
}Objectives

• This module covers the operation of various

transport-layer networking technologies used on
routers and hosts, including:
– Transport Control Protocol
– User Datagram Protocol
– NetBIOS
– Network Address Translation
– Extended Access Lists
• This module also discusses tools and methodologies
that can be used to aid in troubleshooting
transport-layer networking issues.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1-2
}Table of Content

1 Characteristics of Transport Layer Technologies

2 Troubleshooting Transport Layer Issues on the
Router
3 Troubleshooting Transport Layer Issues on
Network Hosts
4 Troubleshooting Complex Network Systems

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1-3
CHARACTERISTICS OF
TRANSPORT LAYER
TECHNOLOGIES

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1-4
}Common Transport Layer Technologies
• The transport layer provides end-to-end traffic accountability. Layer 4 technologies ensure
reliable data delivery using acknowledgments, sequence numbers, and flow control
mechanisms.
• The transport layer is the first layer that provides end-user functions.
• Problems at the transport layer can present symptoms ranging from sub-optimal network
operation to complete network communications failure.
• There are at least 35 recognized transport layer protocols. Some of the more common of
these are:
– User Datagram Protocol (UDP)
– Transport Control Protocol (TCP)
– Sequenced Packet Exchange (SPX)
– AppleTalk Transaction Protocol (ATP)
– NetBIOS

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1-5
 }Common Transport Layer Technologies

– Within the TCP/IP protocol suite, Layer 4 operations are primarily handled by UDP and TCP.
UDP and TCP rely on IP at the network layer and use port numbers to identify what higher layer
application traffic is contained in the packet. ICMP is a protocol from the TCP/IP suite that
operates at the network layer and it too relies on IP. Unlike UDP and TCP, ICMP does not carry
user data. ICMP is primarily used by network devices for self-management and self-tuning
functions and by network engineers for troubleshooting network problems.
– The Network Basic Input/Output System (NetBIOS) was developed for IBM in 1983 by Sytek
Corporation and officially defines a session level interface and a data transport protocol. NetBIOS
was extended by IBM in 1985 to create the NetBIOS Extended User Interface (NetBEUI) protocol.
NetBEUI supports NetBIOS operations at the network layer. NetBEUI and NetBIOS are
commonly used in Microsoft and IBM LANs.
– NetBEUI operates at the network layer and interfaces directly with ISO’s Logical Link Control 2
(LLC2) at the data-link layer. NetBIOS interfaces with NetBEUI and with IBM’s Server Message
Block (SMB) protocol at the application layer. Together, NetBIOS and NetBEUI can be
considered to be operating from the network layer through to the presentation layer. NetBIOS and
NetBEUI depend on other hierarchical protocols, such as IP or IPX, to operate in a routed network.
– Novell’s proprietary protocol suite uses Sequenced Packet Exchange (SPX) at the transport layer
to implement reliable data delivery. In the early days of local area networking, Novell’s suite of
protocols was commonly implemented.
– AppleTalk Transaction Protocol (ATP) is used at the transport layer of legacy AppleTalk
networks and relies on AppleTalk’s Datagram Delivery Protocol (DDP) at the network layer.
Because ATP is incompatible with IP, new Mac networks usually use the TCP/IP protocol suite.
– Although legacy networks using IPX/SPX and AppleTalk still exist, troubleshooting these protocol

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1-6
suites is not in the scope of this course and will not be discussed further in this curriculum.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1-6
}User Datagram Protocol
• UDP is connectionless and is considered
unreliable because it does not guarantee
packet delivery. UDP operates on ‘best
effort delivery’ basis and does not use
packet sequencing, acknowledgment or
retransmission mechanisms for flow
control and error detection/correction. If
flow control and error
detection/correction features are required
for a UDP-based data flow, these features
must be implemented in higher layer
protocols or applications.
• Because UDP does not retransmit lost
packets and does not consume bandwidth
with acknowledgments, it is relatively
light-weight, fast, and is suitable for both
one-to-one and one-to-many
communications. Over congestion-free
and error-free networks, UDP is ideal for
transferring small amounts of data, and
for supporting streaming applications
such as voice communications and video
multicasts. However, using UDP over
congested or error-prone networks often
results in high degrees of data loss with
higher-layer protocols, applications, or
even users having to initiate data
retransmission.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1-7
}User Datagram Protocol

• Many higher-layer protocols and applications make

use of UDP, including:
– Trivial File Transfer Protocol (TFTP)
– Domain Name Service (DNS)
– NetBIOS Name Resolution (NetBIOS-NS)
– Windows Internet Name Service (WINS)
– Bootstrap Protocol (BootP)
– Dynamic Host Configuration Protocol (DHCP)
– Network Time Protocol (NTP)
– Remote Authentication Dial-In User Service (RADIUS)
– Terminal Access Control Access Control Server (TACACS)

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1-8
}Transport Control Protocol
• Unlike UDP, TCP is connection-
oriented. Because TCP implements
packet sequencing,
acknowledgment, and retransmission
mechanisms at the transport layer, it
is considered to be an inherently
reliable protocol. These additional
features at Layer 4 give TCP a larger
operational overhead, which do not
carry a data payload and consume
bandwidth. Because of these
reliability features, TCP is better
suited to one-to-one communications
and is rarely used for streaming and
one-to-many communications.
• TCP implements two main
mechanisms for maximizing reliability
and efficiency. They are the three-
way handshake and windowing.
Understanding these technologies is
important to troubleshooting network
performance issues and failures.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1-9
 }Transport Control Protocol
• TCP three-way
handshake
The TCP three-way
handshake occurs during
TCP connection
establishment and
consists of three stages:
• A similar process is
followed at TCP
connection termination,
ensuring full and complete
transfer of data and the
orderly termination of
processes.

1. Session request (TCP SYN) – the initiating host sends a TCP synchronization packet, which
has the SYN bit set on, and contains the initiating host’s own sequence number (seq = x) for
the connection. This packet also contains information about the initiating hosts TCP receive-
window size.
2. Session request acknowledgment (TCP SYN-ACK) – the target host responds to the packet
from the initiating host by sending its own ‘synchronization acknowledgment’ packet. This
packet has both the ACK and SYN bits set on, and contains the acknowledgment number
generated by incrementing the sequence number from the initiating host by one (ack = x+1)
plus the new sequence number from the target host (seq = y). The purpose of this packet is to
inform the initiating host that the target host has received and understood the information
from the initiating host and inform the initiating host of the TCP receive-window size of the
target.
3. Session acknowledgment (TCP ACK) – the initiating hosts received the TCP SYN/ACK
packet from the target host and responds with a session acknowledgment packet. This packet
has the SYN bit set off and the ACK bit set on. The acknowledgment number in this packet
is generated by incrementing the sequence number from the target host by one (ack = y + 1).
The purpose of this packet is to inform the target host that the initiating host has received and
understood the information sent by the target host. Once the target host has received this
packet, the TCP session has been established, and reliable data exchange can begin.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 10
 }Transport Control Protocol
• TCP sliding windows
TCP sliding window is the mechanism used to implement flow control and packet
retransmission. Before discussing the operation of TCP windows, several terms
should be explained:
• Transmit window – where segmented data is sequenced and buffered for
transmission on a sending host.
• Receive window – where segments are buffered for re-sequencing on the
receiving host.
• Segment size – The maximum size TCP will make each segment (before
transmission) when processing data from higher layers.
• Acknowledgment – A packet sent from the receiving host to the sending host,
acknowledging the receipt of particular packets.
• Acknowledgment trigger – An event that triggers the transmission of an
acknowledgment packet on the receiving host, such as when two sequential
segments are received or when a predefined window threshold is reached.
• Delayed acknowledgment timer – the amount of time a receiving host will
wait for an acknowledgment trigger before sending ‘delayed acknowledgment’.
• Delayed acknowledgment – an acknowledgment packet triggered by the
‘delayed acknowledgment timer’.
• Retransmission timer – the amount of time the sending host will wait for an
acknowledgment before retransmitting a packet.

– Recall that TCP receive-window sizes are exchanged between hosts as part of the TCP connection
establishment.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 11
}Transport Control Protocol
1. The sending host initiates the TCP connection establishment process, during which, each host
transmits its own receive-window size to the other, and sets its own transmit window to the same size
as the receive window of the other host.
2. TCP on the sending host processes data from higher layer protocols into segments (according to the
configured segment size) and buffers them for transmission in the transmit window. Segments waiting
to be put into the transmit window are buffered in the transmit buffer.
3. Segments in the transmit window are copied, and the copies are passed to the IP process for
transmission. At this point, the retransmit timer for each segment is started. When passing the
segments to IP, TCP tags each of them with a TCP header containing a sequence number. A copy of
each packet will remain in the transmit window until an acknowledgment is received accounting for
each segment.
4. The receiving host places the received segments in its receive window in sequence according to the
sequence number of each segment and starts a delayed acknowledgment timer for each. When two
sequential segments are received or a pre-defined window threshold is reached, the receiving host
sends an acknowledgment packet for the received segments to the sending host. This
acknowledgment also contains the current size of the receive window of the receiving host.
5. If an acknowledgment is not triggered on the receiving host before the delayed acknowledgment
timer for a segment expires, the receiving host sends a delayed acknowledgment packet for that
segment.
6. The sending host receives the acknowledgment (or delayed acknowledgment) from the receiving host
and discards the acknowledged segments from its own transmit window. The transmit window now
‘slides’ past the acknowledged segments and accepts new segments waiting for transmission. These
new segments are passed to the IP process for transmission (as described in Step 3).
7. If the sending host does not receive the acknowledgment (or delayed acknowledgment) before the
retransmit timer expires on a segment (because either the data segment or the acknowledgment was
lost in transit), the sending host will resend that data segment and reset the appropriate retransmit
timer to double its original time.
8. Once all segments have been transmitted and acknowledged, the sending host initiates the TCP
connection termination process to properly terminate the connection and associated processes on
each host.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 12
 }Transport Control Protocol
• Because TCP window size is important to the efficient operation of the
network, TCP window size is able to dynamically change during the
course of the transmission. In other words, TCP is effectively self-tuning
its efficiency according to the conditions on the network. This self-tuning
mechanism depends on the number of TCP packets successfully received
by the receiving host over a percentage of the receive-window size.
• If the receiving host is receiving a large number of sequential data within
a short period of time, it will exceed a threshold defined as a percentage
of the receive-window size. This causes the receiving host to increase
the size of its receive window and informs the sending host of the
change by putting this information into one of the outbound
acknowledgement packets.
• Once the sending host receives the acknowledgement packet with the
new receive-window size for the receiving host, the sending host
increases the size of the transmit window. The receive-window size of
the receiving host will continue to get bigger until the growth threshold
of the window is the same as the rate at which the network can support
data transfer.

– This process will continue until the capacity of the slowest network link is saturated and network
contention is starting to occur. As the gap between the data transfer rate and threshold get smaller,
the rate at which the window grows gets smaller. This explains why, when downloading large files
from the Internet or across a wide area network, the transfer rate initially increases rapidly, but
plateaus to a fairly constant transfer rate for the remainder of the download.
– If the network becomes congested from another source, packets and acknowledgements in existing
traffic flows will be lost. This will cause the same process to operate in reverse, reducing the
receive-window size of the receiving host until there is little or no gap between the data transfer
rate and the threshold of the window.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 13
}Standard access control lists
• Recall that access control lists (ACLs) can be implemented on routers to permit
and deny traffic that matches predefined profiles. Traffic profiles can be
configured to match individual hosts, parts of networks, or entire ranges of
networks, and can apply to all IP traffic, or traffic using a specific source or
destination port. Access lists can also be used to filter traffic for other operations
on the network equipment, particularly network management traffic destined to
a network switch or router.
• Standard access lists examine only the source address of packets. This means
they must be implemented at each ingress point for a protected destination.
Implementing the standard access list as close to the protected destination as
possible reduces the total number of access lists required.
– Numbered standard access lists can be used for filtering traffic being forwarded or
received by a router or switch. Numbered standard access lists use the number ranges
1-99 and 1300-1999.
– Named standard ACLs have the same capabilities and limitations as numbered
standard ACLs. The main advantage of a named ACL is better documentation, as the
ACL can be named according to its purpose. One significant disadvantage is that other
processes, such as SNMP security, cannot reference named standard ACLs.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 14
}Standard access control lists

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 15
}Extended access control lists
• Standard IP access lists can permit or deny packets based only on the source IP
address of the packet. Extended access lists have more features and can be used
to target very specific traffic based on any combination of the following:
– Source address
– Destination address
– Source port
– Destination port
• Extended access lists can also be used to filter ICMP traffic based on ICMP type
and code.
• Because extended access lists can filter on destination address, they should be
implemented as close as possible to the source of the traffic being filtered,
typically at the edge of an organization's network. This enables traffic to be
examined and filtered before crossing expensive and congested WAN links within
the organization.
• Numbered extended access lists are primarily used to filter traffic being
forwarded through a router and can be configured to use the number ranges
100-199 and 2000-2699.
• Named extended access lists are also primarily used to filter traffic being
forwarded through a router and are often used to provide better documentation
of the configuration on the router.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 16
}Extended access control lists

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 17
 }Static IP Network Address Translation
• ‘The IP Network Address Translator’ is a technology defined in RFC 1631 that
allows one group of addresses to be represented by another group of addresses.
NAT is technically a network layer technology, but has some features that extend
into the transport layer.
• As IPv4 started to run out of addresses, several measures were developed to
alleviate the pressure. One of these measures was the introduction of reserved
private network address spaces in RFC 1918. This RFC reserves three address
spaces which are not recognized or routed by the Internet and can be re-used by
networks which do not need to connect to other networks or to the Internet.
Internet core routers are configured to drop any packets either sourced from or
destined to an address in any of these reserved address spaces. The three
address spaces reserved by RFC 1918 are shown in Figure 1.
• NAT is a process which allows networks originally configured with one of these
reserved address spaces to connect to other privately addressed networks and to
the Internet without having to re-address the internal network. NAT is normally
an additional process that is run on routers operating on the boundary between
two discrete networks.
• NAT works, either statically or dynamically, by using a table of IP addresses to
re-write the addressing information in an IP packet header.

– In its simplest form, the network engineer manually builds the NAT table. The table has addresses
used inside the network mapped to addresses used outside the network. Each table entry represents
a single host inside the network.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 18
}Static IP Network Address Translation
• The following process shows the operation of NAT:
– User at host 10.4.1.1 opens a connection to host B.
– The first packet that the router receives from 10.4.1.1 causes the router to check its
NAT table. A translation is found because it has been statically configured.
– The router replaces the inside local IP address 10.4.1.1 with the selected inside global
address (2.2.2.2) and forwards the packet.
– Host B receives the packet and responds to 10.4.1.1 using the inside global IP address
2.2.2.2.
– When the router receives the packet with the inside global IP address of 2.2.2.2, the
router performs a NAT table lookup using the inside global address as the reference.
The router then translates the address back to 10.4.1.1 and forwards the packet to
the host.
– The host with the 10.4.1.1 address receives the packet and continues the
conversation.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 19
 }Dynamic IP Network Address Translation
• NAT can be configured in a number of ways:
– Static table entries
• Static tables are manually defined by a network engineer and traditionally define
a one-to-one mapping between inside and outside IP addresses so that only the
IP address portion of the packet header is altered.
• Static NAT has the benefit of offering both connectivity and security because
hosts on either side of the NAT router cannot communicate if the administrator
has not defined an appropriate NAT table entry.
– Dynamic table entries
• Dynamic NAT has neither of the problems associated with static NAT. Dynamic
NAT creates entries in the NAT table as required and removes them after they
have remained idle for a predefined period. Dynamic NAT allows a large number
of inside hosts to share a small number of outside addresses. Dynamic NAT also
offers greater security than static NAT, because unlike static NAT, entries in a
dynamic NAT table are deleted after they have been idle for a short time.

– The main disadvantage of static NAT is that it requires one outside address for each inside address
that needs to be translated. Because it is unlikely that all of the inside hosts will need to
communicate through the NAT router at the same time, static NAT wastes precious outside
addresses. Another significant disadvantage of static NAT is that it is must be manually
administered. Manual administration of a NAT table can be difficult for a small network and is
likely to be impossible for a large network.
– The main problem with dynamic NAT is that it can be more complex to troubleshoot as the
mapping entries in the table change. Sometimes this results in intermittent symptoms and
problems. Entries that fail to automatically expire from the NAT table also cause issues with
network operation as they do not release outside addresses that would have otherwise become
available.
– Dynamic NAT operates by examining the packets as they are processed for forwarding. If a
suitable entry already exists in the NAT table, the timer for that entry is reset, the address of the
packet is updated as appropriate, and the packet is forwarded. If the NAT table does not already
have a suitable entry, the NAT process uses an address from the pool of available outside addresses
to create a new entry before setting the expiration timer for the entry. It will then update the header
of the packet with the new addressing information, and forward the packet.
– If all of the outside addresses are currently in use, the NAT process drops the packet and returns an
error to the inside host.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 20
 }Dynamic IP Network Address Translation
• NAT with overload and Port Address Translation
Dynamic NAT also has a feature called overloading, or Port Address Translation.
NAT without overloading operates at the network layer only, and only IP address
information is substituted in packet headers. NAT with overloading extends the
operation of NAT into the transport layer and UDP and TCP port numbers are
included with entries in the NAT table.
• Recall from previous curriculum that UDP and TCP have roughly 65,535
associated ports each. It is highly unlikely that a single client host accessing
network resources will legitimately need to use all of these ports at the same
time.
• Because a single inside host does not require all of the ports available on an
outside address, NAT overload allows multiple inside hosts to make use of the
unused ports on a common outside address. It does this by including the port
numbers for a given session in the translation table.
• Recall that a single entry in a normal static or dynamic NAT table represents a
single host inside the network. In a table for NAT with overload, a single entry
now represents a single transport-layer session.

– A significant consideration for implementing NAT with overload is that it can only be used for
sessions from network clients. Servers hosting specific applications must listen for session request
traffic on specific ports and therefore cannot make use of port address translation technology.
– Typically, a single host being used for normal Internet activities will maintain around a dozen
sessions through a NAT router at any given time. Consider these typical Internet browsing
activities:
o How many web pages are open at the same time?
o Is email or newsgroup software updating messages in the background?
o How many file download sessions are active?
o Are any IM applications running in the background?
o What background processes, like DNS, are supporting other applications?

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 21
}NetBIOS and NetBEUI
• NetBIOS and NetBEUI are a pair of protocols that work together to
provide easily configured, broadcast-based networking service for
computers running the Microsoft Windows family of operating systems.
NetBIOS supports the following network services:
– Network name registration and verification
– Session establishment, termination, and management
– Reliable session data transfer (connection-oriented)
– Unreliable datagram data transfer (connectionless)
– Monitoring and management of network interfaces and lower-layer
protocols
• One of the main advantages of using NetBIOS and NetBEUI is that they
are simple to configure. The main failing of NetBIOS is that it uses a
broadcast-based non-hierarchical namespace, forcing it to rely on other
network layer protocols if used over a routed network.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 22
}NetBIOS and NetBEUI
• NetBIOS Names
The NetBIOS name space is 16 bytes
long. The first 15 characters are the
computer name defined by the user
during configuration.
• The last character is a hexadecimal
digit representing the type of data in
the NetBIOS name and network
service using that data on a given
machine. NetBIOS computers
advertise this information to show
what services they can offer to the
remainder of the network. NetBIOS
names can also be used to identify a
machine as being part of a
workgroup or domain.
• The 16th character is also used to
identify a NetBIOS name as being
either unique as in a computer or
user name, or group such as a
Microsoft Windows Workgroup or
Domain name.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 23
 }NetBIOS and NetBEUI
• NetBIOS in Operation
There are three main stages in the operation of NetBIOS:
– Name registration
– Name discovery
– Name release
• When a Windows NT computer starts up, the various services running on
the machine register themselves using the unique NetBIOS name of the
machine and the appropriate hexadecimal character at the 16th byte
position. This process is called NetBIOS name registration and uses
either a broadcast to the local network or a unicast to a local NetBIOS
name server if one has been configured for the computer.
• When one NetBIOS host needs to communicate with another, it needs
the NetBIOS name of the remote machine. This name can either be input
by the user or can be obtained from a network browse list. The browse
list is a list of network names and services available on the network that
is maintained by a master browser machine. The browse list is built from
the NetBIOS information broadcast during the NetBIOS name
registration process during the start up of a machine. This list is
maintained by the master browser and is sent to one or more backup
browsers in the network.

– Opening Network Neighborhood in Windows displays the browse list for the network.
Periodically, the NetBIOS host computer contacts the master browser for a list of backup browsers
from which one is selected. The host computer then contacts the backup browser, retrieves the
browse list, and displays it to the user.
– If the client cannot locate a master browser on the network, it initiates a browser election to ensure
a master browser and at least one backup browser is elected in the network.
– The computer browser service only supplies NetBIOS name information. The NetBIOS name
discovery process is what occurs when a NetBIOS name needs to be resolved to a lower-level
network address. On a local network, this process is completed using either broadcasts to the local
network or using a unicast message to a NetBIOS name server if present. The NetBIOS name
resolution process is discussed later in this section.
– When an application or networking service on a host is stopped, the NetBIOS name for that service
on that host is available for use by another host. This process is called “NetBIOS Name Release”
and is the process that is used to remove services from the network when a network host is shut
down.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 24
 }NetBIOS and NetBEUI
• NetBIOS Name Discovery and 1. Windows computers maintain a local cache of
recently resolved NetBIOS names. This speeds
Resolution up resolution of names for computers that are
The operation of NetBIOS-based connected to regularly.
networks is dependent on resolving 2. A NetBIOS Name Server (NBNS), normally a
NetBIOS names to lower-layer Microsoft Windows NT server running the
addresses. Because resolving names Windows Internet Name Service (WINS)
service, maintains name and address
to addresses is so important, information for all computers on the network. If
Windows computers use the the NBNS does not contain name resolution
following process to efficiently check information for the requested machine name,
up to six sources of information the Windows client sends a local broadcast in
an attempt to contact the remote host directly.
when attempting to connect to If there is no response, the Windows computer
another computer: checks locally stored files.
1. Check the local NetBIOS name 3. Administrators of Windows computers can
configure them with files containing hard-coded
cache name-to-address resolution information. The
2. Contact a NetBIOS name server first of these files is called LMHOSTS (LAN
manager hosts) and is a remnant of older
3. Broadcast the name resolution implementations of Windows networking.
request locally 4. If a suitable entry is not found in the LMHOSTS
file, another locally stored file called HOSTS is
4. Check the locally stored checked. The HOSTS file is similar to the
LMHOSTS file (Windows only) LMHOSTS file and is a remnant of older
implementations of TCP/IP networking.
5. Check the locally stored HOSTS 5. If both of these local files have not been
file (Windows only) configured with a hard-coded resolution of the
desired NetBIOS name and address, and the
6. Contact a Domain Name System Windows client is configured to use a DNS
server (Windows only) server, the DNS server is queried.

– During startup, the client computer requests its own name from the name server to check that no
other computer on the network has the same computer name. Recall that NetBIOS names must be
unique on the network. If the client computer is already registered in the network, the name
address currently being used by the client computer is returned. If the client computer is new to
the network, the server returns a message indicating that the requested information is not found,
and the client computer registers its own name, address, and services provided in the WINS
database. This information remains in the database until the machine is shut down when the client
computer requests its registration be released from the database.
• Windows computers configured to use a NBNS server do a number of things
during startup and shutdown.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 25
}NetBIOS and NetBEUI
• NetBIOS and Other Network Layer Protocols
By default, NetBIOS uses the NetBEUI Frame (NBF) protocol at the lower
layers of the OSI model for network communications. NetBEUI has the
advantage of being very simple to configure, making it highly suitable to
small networks unlikely to have a full-time administrator. The main
limitation of NetBEUI is that it is broadcast-based and cannot operate
over routed networks. To work on a larger scale, NetBIOS must replace
NetBEUI with either the IPX/SPX or TCP/IP protocol suites at the
network layer of the OSI model. The TCP/IP protocol suite is most
commonly used.
• When NetBIOS interfaces with the TCP/IP protocol stack, it creates the
NetBIOS over TCP/IP (NetBT) protocol. This is slightly misleading,
because NetBIOS actually interfaces with UDP, not TCP, at the transport
layer of the TCP/IP stack, using the NetBIOS Data Protocol to form
NetBT. Recall that NetBIOS provides connection and session
management at higher layers in the OSI model. This means NetBIOS
does not need to rely on TCP for these functions and is able to use the
more bandwidth-efficient UDP at transport layer.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 26
 }NetBIOS and NetBEUI
• NetBIOS Node Type
Recall that NetBIOS is broadcast-based, but TCP/IP is primarily unicast-based. This means
that the behavior of the NetBIOS name resolution process over TCP/IP can be configured
depending on the size and nature of the network. Misconfiguration of the NetBIOS node
type is a common cause of network problems at the transport layer. The behavior of
NetBIOS nodes over TCP can be configured as one of four options:
– B-node (Broadcast node)
– P-node (Peer-to-peer node)
– M-node (Mixed node)
– H-node (Hybrid node)
• NetBIOS hosts configured as NetBIOS broadcast nodes use only UDP datagram broadcasts
for NetBIOS name registration and resolution. In large networks, this has the negative
impact of increasing the load on the network. By default, routers contain packets generated
by NetBIOS name registration and resolution from broadcast nodes. This means that
NetBIOS B-node operation is not suitable where resources need to be accessed across a
router.
• Peer-to-peer NetBIOS hosts do not use broadcasts. These hosts rely on having a NetBIOS
name server configured to support the operation of NetBIOS name registration and
resolution activities. Although this enabled NetBIOS computers to communicate across
routers, it makes the network completely reliant on the operation of the NetBIOS name
server. If the NetBIOS name server were to fail, the NetBIOS clients would not be able to
communicate with each other because they could not broadcast to locate each other.
• Computers with mixed node configuration use both B-node and P-node operation for
NetBIOS name registration and resolution. The default mode of operation for a mixed node
host is broadcast.

– Therefore, if a broadcast fails to return a positive response, the host reverts to peer-to-peer
operation. This enables a computer to locate resources on a local network easily and to use a
NetBIOS name server if a required resource is not located on the local network segment.
– NetBIOS hosts configured as hybrid node clients also use both peer-to-peer and broadcast
operation for NetBIOS name registration and resolution. Unlike Mixed nodes, Hybrid nodes
default to using peer-to-peer and revert to using broadcast when peer-to-peer fails to return a
positive response.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 27
TROUBLESHOOTING TRANSPORT
LAYER ISSUES ON THE ROUTER

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 28
}Common issues with extended ACL's
• Recall that access lists are used to filter all traffic entering
and leaving the router. Obviously, the most common issues
with extended access lists will be the result of
misconfiguration by the network engineer. There are eight
areas where misconfigurations commonly occur:
– Selection of traffic flow
– Order of access control elements
– Implicit “deny any any”
– Addresses and wildcard masks
– Selection of transport layer protocol
– Source and destination port(s)
– Use of the ‘established’ keyword
– Uncommon protocols

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 29
}Common issues with extended ACL's
• Selection of traffic flow
Although these points are not listed in any particular order, the most common
router misconfiguration of extended access lists is applying the access list to the
incorrect traffic. Traffic is defined by both the router interface through which the
traffic is traveling, as well as the direction in which this traffic is traveling. Once
defined, an access list must be applied to the correct interface and the correct
traffic direction must be selected in order to function properly.
• Order of Access Control Elements
Another common mistake made when configuring ACLs is the order in which
access control elements (ACEs) are configured. Although an access list may have
an element to specifically permit a particular traffic flow, packets will never
match that element if they are being denied by another element earlier in the
list.
• Recall that the guideline for configuring an access control list is specific to
general. This means that the most specific elements are configured at the top of
the list and the less specific elements are configured at the end. The more
information defined in an element, the more specific that element is. For
example, the element permit tcp 10.0.0.0 0.255.255.255 any eq 110
established is more specific than the element permit tcp 10.0.0.0
0.255.255.255 any eq 110 because the first element used the additional
keyword established. The element permit udp host 10.32.96.7 eq 53 any
is more specific than both of these because it matches a more specific (smaller)
range of source addresses.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 30
}Common issues with extended ACL's
• Implicit “deny any any”
Every extended access list has a deny any any element implied as the
final entry in the list. This serves as a security catch all, ensuring traffic
that does not match any of the administratively configured access
control elements is not allowed through. This does not normally pose an
issue when configuring firewall routers, as the guideline for configuring
highly secure access lists is to deny everything and specifically permit
particular traffic flows. In a situation where high security is not required
on the access list, forgetting about this implicit access control element
may be the cause of an access list misconfiguration.
• Addresses and wildcard masks
Although setting a source or destination address may seem like
something difficult to get wrong, it still happens quite often. There are,
however, a number of things that make correctly selecting source and
destination addresses more complex:
– Running NAT on the router
– Using complex wildcard masks to select patterns of addresses

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 31
 }Common issues with extended ACL's
• If the router is running both access
lists and NAT, the order in which
each of these technologies is applied
to a traffic flow is important. The
order of operations in the switching
path is quite complex, as shown in
Figure. The important points to
remember here are:
– Inbound traffic is processed by the
inbound access list before being
processed by outside-to-inside NAT.
– Outbound traffic is processed by the
outbound access list after being
processed by inside-to-outside NAT.
• Wildcard masks are typically used to
select ranges of addresses. For
example, the address 198.162.10.0
and wildcard mask 0.0.0.255 could
be used to select all hosts in the
Class C network address-space
198.162.10.0. Generally, these sorts
of address and wildcard mask
combinations are difficult to get
wrong.

– More complex wildcard masks can also be used to select patterns of addresses. For example, the
address 10.0.32.0 and wildcard mask 0.0.32.15 would select the first 15 host addresses in either the
10.0.0.0 network or the 10.0.32.0 network. Complex wildcard masks like this can provide
significant improvements in efficiency, especially in large networks with structured and controlled
IP addressing schemes. They also require that the network engineer have detailed and thorough
knowledge of the network address when designing these complex access list elements.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 32
}Common issues with extended ACL's
• Selection of transport layer protocol
When configuring ACLs, it is important that only the correct transport layer
protocols be specified in the element. Many network engineers, when unsure if a
particular traffic flow uses a TCP port or a UDP port, will configure both. The first
problem with doing this is that it opens a hole through the firewall, possibly
giving intruders an avenue into the network. The other problem is that it
introduces an extra element into the ACL. This means the ACL takes longer to
process, introducing more latency into network communications.
• Network engineers might also make a mistake and unintentionally configure an
ACL to use the incorrect transport layer protocol. For example, an ACL may be
intended to permit HTTP traffic, but be configured with UDP port 80 (instead of
TCP port 80).
• Source and destination port(s)
Correctly specifying source and destination ports is usually fairly simple, but can
be quite complex. In one example of a simple traffic flow, the client end of the
connection uses a random high-numbered port to initiate a connection to a
specific port at the server-end. Defining the correct source and destination ports
is not overly complex in this situation.
• A more difficult concept is understanding the flow of traffic between two hosts
and building the ACLs to properly control the traffic. Simple traffic flows require
symmetric access control elements for inbound and outbound access lists. In
other words, address and port information for traffic generated by a replying
host is the mirror image of address and port information for traffic generated by
the initiating host.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 33
 }Common issues with extended ACL's
1. A user wants to check his e-mail on a remote 5. Having satisfied an element of the outbound
ISP mail server. access list, the router forwards the traffic to the
2. The client PC opens the mail client and initiates next hop on the way to the ISP mail server.
a connection to a POP3 mail server. 6. The ISP mail server has mail waiting for the
3. The request for new mail is generated and sent user. When the ISP mail server receives the
by the client PC. request, the mail server responds with the mail
items it has for the user. Because the traffic is
– The source address of the request packets is the
IP address of the client PC. now returning to the client PC, the address and
port information from the request packets have
– The source port of the request packets will be the source details swapped with the destination
randomly chosen from above 1024 (for example,
TCP port 2113). details.
– The source address of the reply packets is the IP
– The destination address of the request packets is
the IP address of the mail server. address of the ISP mail server.
– The destination port of the request packets is – Because the reply is coming from the POP3 mail
TCP port 110 (the POP3 mail server). server, the source port of the reply packets is
TCP port 110.
– The packets traverse the LAN, get to the firewall – The destination address of the reply packets is
router, and are processed by the access list the IP address of the client PC.
controlling outbound traffic on the external
router port. – The client PC needs to know which process to
– A control element permitting traffic from any pass the reply packets to when it receives them.
port with an internal address to TCP port 110 on This means the destination port of the reply
the ISP mail server address is configured near to packets is set as TCP port 2113.
the top of the access list. 7. The mail server sends the reply packets onto
– The ‘request for mail’ traffic matches this the Internet and they eventually get back to the
element and is allowed through the firewall firewall router. The traffic must satisfy an
router. element of the access list controlling inbound
4. The packets traverse the LAN, get to the traffic on the external router port before being
firewall router, and are processed by the access forwarded to the LAN.
list controlling outbound traffic on the external 8. The ‘reply’ traffic reaches the client, where the
router port. mail items are displayed by the e-mail client
software.

– Examine these descriptions of a simple e-mail traffic flow traversing a firewall router to
understand the relationship for addresses and ports between initiating and responding traffic:

4. Step 4.
• A control element permitting traffic from any port with an internal address to TCP port
110 on the ISP mail server address is configured near to the top of the access list.
• The ‘request for mail’ traffic matches this element and is allowed through the firewall
router.
7. Step 7
• A control element permitting traffic from TCP port 110 from the IP address of the ISP
mail server to any port on any internal IP address is configured near to the top of the
access list.
• The ‘reply with mail’ traffic matches this element and is allowed through the firewall
router.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 34
 }Common issues with extended ACL's
• Other traffic flows are quite involved and network engineers must
understand these flows before attempting to control them using access
lists. One such complex traffic flow is the File Transfer Protocol, better
known as FTP. FTP uses TCP at the transport layer and is associated
with two port numbers. They are ports 20 and 21. TCP port 21 is used
for FTP control messages, while TCP port 20 is used for FTP data
messages.
• When an FTP client connects to an FTP server, a control session is
established. The port used at the client end will be randomly chosen
from port numbers above 1023, and the port used to reference the
server end will be TCP port 21. When the user-based application issues a
‘get’ command to the FTP client, it is requesting a file from the FTP
server. This file must be transferred using an FTP data connection so
that the user-based application can still control the FTP server and
interrupt the file transfer if necessary. The FTP client binds to an
additional local port (above 1024), and sends this information to the FTP
server through the FTP control connection. The server responds by
attempting to open a connection with a source port of TCP port 20 and
the destination port supplied by the FTP client.

– This process requires that the firewall router controlling this traffic would need to have a single
element permitting outbound traffic to TCP port 21 for FTP control data, but would require two
elements to allow inbound traffic. The first element would need to allow reply FTP control traffic
from TCP port 21 on the FTP server. The other element would need to allow new FTP data traffic
from TCP port 20 on the FTP server.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 35
 }Common issues with extended ACL's
• Use of the ‘established’ keyword
In a TCP session, all packets after the first have the ACK bit
set. The initiating host sets the SYN bit of the first packet to
on but does not set the ACK bit. Subsequent packets have
both bits set to on. Using the "established" keyword means a
packet must match the specified source and destination IP
addresses and ports and must also have the ACK bit set
before a complete match is possible.
• Obviously, the established keyword can be used to
increase the security provided by an access list. If this
established keyword is applied to an outbound access list,
unexpected results may occur. Again, network engineers
need to have a thorough understanding of network traffic
flows before implementing the established keyword in
access lists.

– Use of the ‘established’ keyword

o This characteristic of TCP packets in a connection can be used by an access list to control the
allowed source of sessions. For example, an access list watching inbound traffic may have an
element to permit traffic from a remote Telnet server (source port TCP port 23). By adding the
‘established’ keyword to this element, the access list can be configured to permit traffic from a
remote Telnet server, but only if the Telnet connection was initiated from inside the firewall
router.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 36
}Common issues with extended ACL's
• Uncommon protocols
Extended access lists can be used to control traffic for IP-encapsulated
protocols other than TCP, UDP, and ICMP. Misconfigured access lists
often cause problems for less common protocols.
• One group of uncommon protocols that is gaining popularity are Virtual
Private Networking and encryption protocols, including Layer 2 Tunneling
Protocol (L2TP), Generic Routing Encapsulation (GRE), Internet Key
Exchange (IKE), Internet Security Association and Key Management
Protocol (ISAKMP), and Encapsulating Security Payload (ESP).
• Because Virtual Private Networks (VPNs) may need to run through
firewall routers, network engineers need to understand the specifics of
the traffic flows required by the VPN.
• IPSec, for example, uses the ISAKMP protocol for connection setup.
ISAKMP requires communications where the source and destination of a
packet are both UDP port 500. The network engineer must take this and
other unusual requirements of network traffic for uncommon protocols
into account when configuring the firewall router access lists.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 37
}Gathering information on ACL operation
• One of the most useful commands for viewing access list operation is the
log keyword on access list entries. This keyword instructs the router to
place an entry in the system log whenever that entry condition is
matched. The logged event includes details of the packet that matched
the access list element.
• The log keyword can be especially useful for troubleshooting access list
operation. It can also provide information on intrusion attempts being
blocked by the access list. For example, if the last element in an
extended ACL is configured as deny ip any any log, the details of any
packet not matching a condition higher in the ACL is recorded in the
system log. Because this element shows all packets not being matched
by a statement earlier in the ACL, it can be useful both for
troubleshooting when a certain traffic flow cannot communicate through
the firewall router, as well as for showing when an intruder is attempting
to access the network.
• This log output can either be buffered and viewed on the local system or
forwarded to an external syslog server where it can become part of a
larger network management system. Use the show logging command
to view the locally buffered copy of the system log.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 38
}Gathering information on ACL operation
• The command show ip access-list [number | name] is
particularly useful for troubleshooting IP access lists. This
command displays the detailed elements of a specific
access-list in the correct order and the number of packets
that have been matched against each element. Alternatively,
if no access list number is specified, details of all access lists
are shown.
• Figure shows the typical output from the show ip access-
list command.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 39
}Gathering information on ACL operation
• When viewing the number of access list matches, the hit
counters should sometimes be reset using the clear ip
access-list counter [number | name] command. This
command resets access list counters to zero, making it
easier to spot changes in the counters and heavily-matched
access list elements. Like the show ip access-list
command, this command can be used to clear the counters
for only a specific access list by specifying its name or
number. It will clear the counters for all IP access lists if no
access list name or number is specified. An alternative
command clear access-list counters [number | name]
can also be used to clear IP access list statistics.
• Figure shows an example of using the clear command.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 40
}Gathering information on ACL operation
• The command show ip
interface shows
information about the
configuration of interfaces
running the IP protocol,
including information on
any access lists configured
for inbound and outbound
traffic on the interface.
• Figure shows an example of
the output from this
command.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 41
 }Optimizing access list operation
• Routers handling a lot of traffic and
multiple access lists can introduce
significant latency into network
communications. In order to
minimize the impact that access lists
have on network latency, network
engineers should optimize existing
access lists. The concept is quite
simple, although the actual
implementation can be a little more
complex.
• Using the show ip access-list
command, network administrators
can gather information on which
access list elements are being
heavily used and which ones are not.
Using this information, they can then
re-write the access list such that the
most heavily used elements are
nearest the top. Figures shows this
process for a simple access list.

– While this can be achieved fairly easily in simple access lists, using the approach on more complex
access lists can be a bit more difficult. Consider an access list which must permit hosts 1-5 and 7-
20 access to remote web servers, but block access to that remote service for hosts 6 and 21-31.
Obviously, changing the order of the access list elements in this access list without regard to the
overall reason of the order would ‘break’ the access list.
– In order to optimize complex access lists, the network administrator must identify groups of access
list elements by purpose or intention. These groups can then be ordered such that the most heavily-
used group of elements is nearest the top.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 42
}Common issues with IP NAT
• The biggest problem with all NAT technologies is
interoperability with other network technologies, especially
those that contain or derive information from host network
addressing in the packet. Some of these technologies
include:
– BootP and DHCP
– DNS and WINS
– SNMP
– Tunneling and encryption protocols
• BootP and DHCP
DHCP was developed from BootP. Both protocols are used to
manage the automatic assignment of IP addresses to clients.
Recall that the first packet that a new client sends is a
DHCP-Request broadcast IP packet. The DHCP-Request
packet has a source IP address of 0.0.0.0. Because NAT
requires both a valid destination and source IP address,
BootP and DHCP can have difficulty operating over a router
running either static or dynamic NAT.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 43
}Common issues with IP NAT
• DNS and WINS
DNS and WINS are both name resolution services for networks. It is
essential to the correct operation of a network that the information they
contain be an accurate representation of the network. Because a router
running dynamic NAT will be changing the relationship between inside
and outside addresses regularly (as table entries expire and are re-
created as required), a DNS or WINS server outside the NAT router does
not have an accurate representation of the network inside the router.
• Additionally, DNS and WINS reply packets contain IP address information
in the data payload of the packet. A DNS or WINS server on the
network inside a NAT router resolves the hostname to a network address
on the inside network. When the NAT router processes the reply packet,
the NAT process translates the address in the packet header
appropriately, but is not able to alter the contents of the data payload.
The outside host is given the inside address of the inside host. Because
the inside network is hidden by the router, the outside network cannot
route packets directly to or from it.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 44
}Common issues with IP NAT
• SNMP
SNMP traffic often contains the IP address of managed network
equipment in the data payload of the packet. Similar to DNS packets,
NAT is not able to alter the addressing information stored in the data
payload of the packet. Because of this, an SNMP management station on
one side of a NAT router may not be able to contact SNMP agents on the
other side of the NAT router.
• Tunneling and Encryption Protocols
Tunneling and encryption protocols are concerned with both encrypting
data to protect packet contents from being intercepted, as well as
ensuring that the packet has not been interfered with during transit. Not
all encryption protocols have problems with NAT. Generally speaking,
encryption services operating at the application layer such as Pretty
Good Privacy (PGP) and Gnu Privacy Guard (GPG) are not impacted by
NAT routers. Encryption and tunneling protocols at OSI model layers 2 to
4 however, will not usually operate through a NAT router.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 45
 }Common issues with IP NAT
• Tunneling and Encryption Protocols
Tunneling and encryption protocols are concerned with both encrypting
data to protect packet contents from being intercepted, as well as
ensuring that the packet has not been interfered with during transit. Not
all encryption protocols have problems with NAT. Generally speaking,
encryption services operating at the application layer such as Pretty
Good Privacy (PGP) and Gnu Privacy Guard (GPG) are not impacted by
NAT routers. Encryption and tunneling protocols at OSI model layers 2 to
4 however, will not usually operate through a NAT router.
• Encryption and tunneling protocols often require that traffic be sourced
from a specific UDP or TCP port, or use a protocol at the transport layer
that cannot be processed by NAT. Some examples of this are:
– Internet Key Exchange (IKE) requires that UDP packets be sent to and
received from UDP port 500.
– IPSec tunnels use Encapsulating Security Payload (ESP) at the transport
layer, and Generic Routing Encapsulation (GRE) tunnels use GRE at the
transport layer. Neither ESP nor GRE protocols can be processed by NAT.

– If encryption or tunneling protocols must be run through a NAT router, the network administrator
can create a static NAT entry for the required port for a single IP address on the inside of the NAT
router.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 46
 }Common issues with IP NAT
• Common NAT misconfigurations
One of the more common misconfigurations of NAT is forgetting that it
affects both inbound and outbound traffic. An inexperienced network
administrator might pre-configure a static NAT entry to redirect inbound
traffic to a specific inside ‘backup’ host. In the event of a failure on the
primary system, traffic could be automatically re-directed to the backup
system without the administrator having to do anything. This static NAT
statement will also change the source address of traffic from that host,
possibly resulting in an undesirable (and unexpected) set of behaviors.
At best, this is likely to result in sub-optimal operation.
• Misconfigured timers can also result in unexpected network behavior and
suboptimal operation of dynamic NAT. If NAT timers are too short,
entries in the NAT table may expire before replies are received and
packets will be discarded. This means the intended traffic did not get
through and the loss of the packets generates retransmissions,
consuming more bandwidth. The NAT router log will also be filled with
errors about closed ports.

– If timers are too long, entries may stay in the NAT table longer than necessary, consuming the
available connection pool. In particularly busy networks, this may lead to memory problems on the
router and hosts may be unable to establish connections if the dynamic NAT table is full.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 47
 }Gathering information on NAT configuration and
operation
• show commands
There are two commands in the
show ip nat group of commands.
The show ip nat statistics
command is used to display statistics
on static and dynamic translations on
the router, as shown in Figure 1.
• The show ip nat translations
command displays the NAT table
currently in operation on the router,
listing both static and dynamic NAT
entries. (Fig. 2)
• debug commands
There is a range of debug
commands available for reporting on
NAT traffic. A commonly used
command is debug ip nat. This
command can also have the
information displayed limited to
events for specific protocols and
processes using the keywords h323,
port, pptp, route, skinny, and
detailed. (Fig. 3)

– The show ip nat translations command also has the optional keywords icmp, pptp, tcp and udp,
which allow the network engineer to limit the type of entries displayed. The network administrator
can also use the verbose keyword to display additional information about the entries in the table.
– Note that the debug command can also use a standard access list to limit the information being
displayed by the debug process to traffic matching the permit statements in the ACL.
– clear commands
When debugging NAT problems, it can be useful to reset NAT statistics or to clear the NAT table
of any dynamic entries. Use the command clear ip nat statistics to reset the NAT traffic statistics
counters. Use the command clear ip nat translations * to clear dynamic entries from the NAT
table.
– Other keywords can be used when clearing the NAT table. The forced keyword clears all IP NAT
translations even if they are currently in use. The inside keyword removes all inside addresses and
ports from the table, while the outside keyword removes all outside addresses and ports. Using the
tcp keyword only removes TCP-related entries. Using the udp keyword only removes UDP-
related NAT entries.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 48
}Other useful information
• There are a range of other tools that
can be used to help troubleshoot
transport layer problems on network
devices. These include:
– Protocol analyzers
– Network device system logs
– Centralized logging system (using
Syslog)
– Network Management systems
• Protocol analyzers
Protocol analyzers can be used to
collect information on network
operations from the data-link layer to
the application layer. A good protocol
analyzer is able to provide a network
engineer with a source of
information on network transactions
at the transport layer.
• Protocol analyzers have been
discussed in Module 2.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 49
 }Other useful information
• Local system logging
Configuring buffered local system logging
can also provide a rich source of
information when troubleshooting
network problems.
• A local system log can also provide
historical information on past events.
Logging on local systems is highly
configurable and can be used to capture
general router events as well as other
information of interest, such as debug
messages.
• To configure a router to keep its local log,
use the following commands from global
configuration mode:
– Router(config)#logging on
Router(config)#logging buffered
[buffer size] [logging level]
• There are seven levels of logging, from 0
for emergency messages (indicating that
the router is unusable), to 7 for
debugging messages generated by
engineer-configured debug commands.
These levels are summarized in Figure.

– The system log buffer uses volatile memory and is cleared by rebooting the router. Because of this,
it is recommended that system log events be redirected to an external system (discussed later in this
section).
– The first step is to ensure system logging is enabled (note that local system logging is on by
default). When configuring the logging buffer, set the size of the log buffer and the level of
message to log.
– The show logging command can be used to display the state of Syslog error and event logging,
including host addresses, and whether console logging is enabled. This command also displays
SNMP configuration parameters and protocol activity.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 50
}Other useful information
• Cisco routers support the Syslog protocol for delivering
system log messages to a centralized system. The Syslog
protocol uses UDP port 514 making it a lightweight, fast, but
unreliable delivery mechanism.
• Syslog servers are machines that can listen on UDP port 514,
and collate information from a number of sources (network
devices) simultaneously. This information is stored in a
central location, such as a database, from where it can be
used to build a report. Such reports can then be used to spot
patterns and trends in network traffic, including current and
potential problems.
• Configuration of Syslog for centralized logging is discussed in
detail in Module 7.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 51
}Other useful information
• Setting the real time clock
To make the logged information more useful, network engineers should
set the real-time clock of the router and enable date and time stamps for
log messages. If messages are logged without the real time clock of the
device being set, the date and time stamp uses the uptime of the router.
• The router real time clock can be manually set or can use a Network
Time Protocol source for its information. Using an NTP time source is
recommended for a number of reasons:
– The real time clock of the router is reset when the router is rebooted.
– NTP time sources are much more accurate.
– Having the real-time clock of all devices synchronized can help trace traffic
patterns and trends through the network.
• Unlike much of the router configuration, setting the real time clock is not
done from global configuration mode. To manually set the router real
time clock from privileged mode, use the clock set [hh:mm:ss] [Day
of the month] [Month] [Year] command. Note that the order of Day
of the month and Month does not matter. For example, the following
two commands have the same effect:
– Router#clock set 15:42:00 12 March 2005
Router#clock set 15:42:00 March 12 2005

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 52
 }Other useful information
• Using Network Time Protocol
Network Time Protocol servers are available for time queries
on the Internet and are arranged in a hierarchy of
importance. A list of NTP servers publicly available from the
Internet is available from
http://www.eecis.udel.edu/~mills/ntp/servers.html.
Alternatively, Network Engineers can build their own internal
NTP servers or can purchase and use NT network appliances
(often synchronized to geo-synchronous satellites) to provide
a reliable and accurate time source for network host devices.
• To configure the router to query an NTP time source, use
these commands from global configuration mode:
– Router(config)#ntp peer [NTP server IP address]
Router(config)#ntp peer authenticate

– Note that NTP sends traffic to and from UDP port 123. This needs to be allowed when configuring
the firewall router access list.
– The network administrator should also set the time zone local to the router so that the router knows
how far to adjust the UTC time signal received from the NTP time source. Use these commands to
configure the local time zone of the router:
– Router(config)#clock timezone [timezone-name] [hours-offset] [minutes-offset]
– This command can also be used to uniquely identify log messages from the router by specifying a
unique time zone name. Note that the timezone-name parameter is limited to eight characters.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 53
}Other useful information
• Figure 1 shows how to configure a
router with NTP and a local time
zone.
• Enabling date and time stamps
on logged messages
To enable date and time stamps for
logged messages, use the following
command from global configuration
mode:
• Router(config)#service
timestamps debug datetime
[localtime] [msec] [show-
timezone]
• The keywords localtime, msec, and
show-timezone can all be used to
add extra information to the logged
messages. It is recommended that
the msec keyword is included,
especially on busy routers.
• Figure 2 shows messages with the
date and time stamp information.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 54
TROUBLESHOOTING TRANSPORT
LAYER ISSUES ON NETWORK
HOSTS

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 55
 }Common transport layer issues with IP networks
• TCP window size can dramatically impact the performance of the
network.
• If the TCP window size is too small, the result will be additional network
overhead from segment acknowledgments. A lot of time is also wasted
when the sending host waits for acknowledgments. Increasing the
window size allows the sending host to transmit more than one packet at
a time, increase network bandwidth utilization and decrease the time
spent waiting for acknowledgements.
• Having a TCP window too large can also cause problems, especially on a
congested network. If the network is dropping a lot of TCP packets and
acknowledgements, the sending host wastes a lot of time waiting for
retransmit timers to expire. The sending host also adds to the congestion
problem by sending large amounts of retransmission traffic. Making the
window smaller forces the sending host to send less data at once,
reduces its impact on the congestion, and improves the chances of TCP
packets and acknowledgements getting through the network
successfully.
• When operating correctly, TCP self-tunes its behavior according to
network conditions.

– Recall from previous content that TCP window sizes are self-tuning. This self-tuning mechanism
can be seen in action when downloading a large file from the Internet. When the download starts,
the rate of transfer is quite slow, as the initial window size is relatively small. As the download
proceeds, the rate of download increases rapidly as the window size expands. As the TCP window
approaches its optimal size, the increase in the download rate slows until the rate is almost
constant. For the remainder of the download, the window size self-tunes to suit prevailing network
conditions.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 56
 }Gathering transport layer information on Windows
machines
• Telnet
Although Telnet normally refers to an
application layer protocol and program, it
can be a useful tool for troubleshooting
problems at other layers, particularly the
transport layer. The usefulness of Telnet
in troubleshooting comes from its ability
to connect to a specific destination port
on a remote server, allowing it to be used
to confirm correct traffic flow from a
client to remote server. Telnet can be run
as either a command line utility or with a
graphical user interface (GUI). Figure
shows the command line Telnet
application being used to confirm
connectivity to a POP3 server.
• ipconfig command
The ipconfig /all command can provide
the network engineer with a lot of
information about the configuration of a
Windows-based machine. This
information covers details from the MAC
address at Layer 2, the IP address at
Layer 3, NetBIOS information at Layers 4
and 5, to DNS information at Layers 6
and 7

– Although never intended as a troubleshooting tool, Telnet is particularly useful for transport layer
diagnostics.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 57
 }Gathering transport layer information on UNIX
machines
• There are many variants of the UNIX operating system. The most
common are SunOS, FreeBSD, SCO, and Linux. All of these operating
systems have tools that can be used to troubleshoot network issues.
Note that while these tools often have the same name and purpose, the
command line options required to use them vary greatly from platform to
platform. Consult the online manual of the local operating software,
using the command man [tool_name], to get more information about
the command line options of the tool to use.
• It is important to note that almost all newer versions of UNIX variants
support native packet-filtering firewall features. When troubleshooting
transport layer problems with UNIX hosts, network engineers should be
aware of the possibility that the problem could be caused by the local
machine firewall configuration.
• There are three common commands used to configure UNIX firewall
features:
– On older UNIX variants using kernel version 2.0, use the ipfwadm
command
– On UNIX variants using kernel version 2.2, use the ipchains command
– On newer UNIX variants using kernel version 2.4 or later, use the iptables
command

– These tools have been covered in greater detail elsewhere in this curriculum and are mentioned
here in the context of troubleshooting options for transport layer issues on UNIX hosts.
– Each new version of the firewall capability has produced a tool with significantly greater features
than its predecessor.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 58
}Gathering transport layer information on UNIX
machines
• ifconfig command
The ifconfig command is a low-level command used to administer network
connections on a UNIX host. This command is used to configure network layer
address and DHCP operation interface design. ifconfig can also be used to
configure data-link layer addressing if the network card has a configurable MAC
address, and is used to enable and disable specific interfaces.
• netstat command
netstat on UNIX hosts operates in much the same way as it does on Windows
hosts. On UNIX hosts, netstat also serves to provide the network engineer with
information on current network connections and sockets and can also filter
information displayed.
• route command
The route command, not surprisingly, is used to add, delete, and manage IP
routing information on the UNIX host.
• ip command
ip is a command available in some newer UNIX variants. This command is a
powerful unified network configuration tool, and supports some unique
configuration capabilities not available in other tools. One important capability of
the ip tool is its support for multiple protocols, including IPv4, IPv6, and IPX.
• Telnet
This client can be used in the same way for UNIX as for Windows hosts to help
troubleshoot transport layer connectivity issues. Another Telnet-like
troubleshooting tool is netcat.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 59
}Common issues with NetBIOS networks
• There are four common NetBIOS-related • Obviously, configuring the NetBIOS node
issues: type as peer-to-peer in a network that
– NetBIOS node type misconfiguration does not have a NetBIOS name server
– NetBIOS workgroup mismatch stops the network from functioning.
Conversely, using broadcast-mode in a
– NetBIOS scope mismatch routed network stops the hosts from
– Duplicate NetBIOS names accessing resources across the router.
• These issues are usually a result of • The NetBIOS node type is usually set
misconfiguration by a user or network using a DHCP server. Recall that DHCP
engineer. Note that several of these can be used to configure more than just a
problems can be caused by DHCP server client IP address. DHCP can be used to
misconfiguration. set a number of optional settings on the
• One of the most common client. The settings are referred to as
misconfigurations on a small NetBIOS DHCP options.
network is the NetBIOS node type. Recall • Because Cisco routers can be configured
from previous content that there are four as DHCP servers, NetBIOS DHCP options
possible settings for the NetBIOS node need to be taken into account. Use the
type: commands netbios-name-server
– B-node (Broadcast node) name-server-address and netbios-
– P-node (Peer-to-peer node) node-type type from DHCP-
– M-node (Mixed node)
configuration mode to implement
NetBIOS DHCP options from a Cisco
– H-node (Hybrid node) router-based DHCP server.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 60
 }Common issues with NetBIOS networks
• DHCP options 44 and 46 are commonly used in NetBIOS networks with
NetBIOS name servers.
• The NetBIOS node type can also be set manually on Windows hosts. The
registry of the Windows machine must be edited in order to do this.
Further information on this can be found in knowledge-base article
160177 at http://support.microsoft.com/.
• When two NetBIOS computers have different values defined for the
workgroup name, these computers are unable to communicate. The
workgroup name is usually configured during the initial installation of the
computer operating system or when networking is being added to the
computer. The workgroup name is not case-sensitive.
• The NetBIOS Scope ID is a left-over piece of NetBIOS technology rarely
used in modern networking. A NetBIOS Scope is a group of computers
that can only communicate with each other. Because the NetBIOS Scope
ID is blank by default, a single computer misconfigured with a NetBIOS
Scope ID is not able to communicate with other hosts on the network.

– Because this is a setting that is not often used, it can be quite difficult to troubleshoot. Note that
like the NetBIOS node type, the NetBIOS Scope ID can also be controlled centrally from the
DHCP server, using DHCP option 47. Unlike the node type however, the Scope ID can also be
easily altered on the local machine through the network interface GUI.
– When duplicate NetBIOS names exist in a network, the duplicate hosts are unable to connect to the
network until they are configured with a unique NetBIOS name. Duplicate NetBIOS names occur
most often when there is no structure to the naming scheme in a NetBIOS network.
– In small networks, the lack of a structure poses no real issue. In larger networks however, a system
for ensuring unique NetBIOS naming is strongly recommended.
– Note that the host name can be set using DHCP option 12. This option is intended for use when
using DHCP reservations (where an IP address is specifically reserved for a given MAC address),
as it allows for full host-specific network information to be controlled from a single administrative
interface. Obviously, using DHCP option 12 with dynamic DHCP clients would cause significant
problems on the network.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 61
 }Gathering NetBIOS information
• netstat is a command line utility that
displays information on protocol statistics
and current TCP/IP network connections.
netstat supports IP, IPv6, ICMP,
ICMPv6, TCP, TCPv6, UDP, and UDPv6,
and can be used to display information
for:
– All connections and listening ports
running on the host
– Ethernet (Layer 2) statistics for the
host, such as numbers of bytes,
packets, and errors received and
sent for the host.
– Port and address information for
connections to and from the host
– Which connections the host initiated
– Per-protocol connection information
– Per-protocol statistics
– Routing table entries
• nbtstat is a command line utility that
displays information on protocol statistics
and current NetBIOS connections running
over TCP/IP. Unlike netstat, nbtstat is
also able to interrogate remote machines
for connection information.

– Several tools are discussed in other modules for troubleshooting network problems, such as ping,
ipconfig, and winipcfg. Some additional tools useful for gathering TCP and NetBIOS information
are:
o netstat
o Nbtstat
– netstat can also be configured to re-query and re-display information at a configured interval. By
configuring netstat to re-query at intervals and directing this output to a text file, netstat can be
used to build a profile of the behavior of the host TCP/IP transport layer operations over time.
– Recall that NetBIOS uses a flat namespace, and that accurate NetBIOS name-to-IP address
mapping is important for correct operation of a NetBIOS-based network. The functions of nbtstat
are centered around reporting on and resetting the information in name tables on local machines,
remote machines, and central NetBIOS name servers.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 62
TROUBLESHOOTING COMPLEX
NETWORK SYSTEMS

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 63
 }Identifying complex transport layer problems
• As a network increases in complexity, the chances of network failure and
sub optimal performance increase. Problems that might have remained
undetected, in simpler networks, may combine to cause problems in a
more complex one.
• Complex network problems are usually caused by the combination of
smaller problems, which by themselves appear to have no immediate
impact on the network. To resolve these problems, develop a profile of
the symptoms and use this as a starting point for troubleshooting
activities.
• In each of these situations, the network engineer would need to be able
to recognize the misconfiguration as such, repair the configuration, and
even though the problem is not immediately resolved, be confident that
it was a contributing factor

– Also keep in mind that some network problems can be triggered or made more obvious as a result
of user behavior. For example, staff at a remote site might need to backup data to a central site
every afternoon before going home causing the WAN link to become congested. This activity is
likely to impact other users who are still working over the link. This particular problem may be
solved by changing the work process (stop backing up across the WAN link), by provisioning
more bandwidth, or possibly by implementing superior routing technologies such as Link
Fragmentation and Interleaving to better utilize the link bandwidth. When the symptom is regular
and predictable, it makes it easier to find the cause and solve the problem.
– Some network problems are intermittent. These sorts of network problems occur with no obvious
pattern and often just go away and reappear at will. Intermittent network problems are significantly
more complicated to troubleshoot because the ability to collect solid information becomes
increasingly difficult.
– Again, intermittent problems may be caused by user behavior. For example, assume a user is
loading a large file across the WAN. This obviously impacts the performance of the WAN link,
generating a support call to the IT help desk. By the time the IT help desk has been made aware of
the problem and investigates the issue, the file transfer has finished and the WAN link performance
has returned to normal.
– Intermittent problems can also be caused by the interaction of various technologies. Firewalls
running dynamic NAT, configuring NAT for load balancing, and running parallel links between
systems can all present intermittent problems in network communications. The key to solving these
sorts of issues is in understanding the technologies involved.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 64
 }Disassembling the problem
• A good way to solve a complex problem is one piece at a time. If
transport layer connectivity is failing, it may be because of a
misconfiguration on more than one device or technology. Following the
traffic flow and correcting each problem as it is encountered can be a
valid troubleshooting approach. Examining the symptoms and generating
a list of possible causes is less likely to be a successful troubleshooting
methodology when dealing with more complex issues.
• Even though it is a good practice to reverse changes that have no effect
when troubleshooting simple problems, this is unlikely to help solve
complex problems. With complex problems, the network engineer has to
start relying on their own experience and judgment as to what is a
probable cause of a problem and what is not.

– Mechanisms for disassembling complex problems include gathering information from midpoints in
the network communications chain and disabling parts of the system to exclude them as being the
cause. This can involve recabling components of the network in order to insert monitoring hosts or
other troubleshooting tools, or to bypass suspected equipment. Gathering detailed log information
from key points in the communications chain can also help pinpoint specific problem areas.
– When resolving complex network problems, the network engineer should always keep a record of
changes being made. Keeping a log has the advantage of providing a record in case the
configuration changes need to be reversed. A log of activities performed also removes any doubt as
to whether a certain activity has been performed, helping the network engineer avoid repeating
troubleshooting activities.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 65
 }Solving the component problems
• Using the example shown in Figure , assume
that Host A is a Telnet client attempting to
access the Telnet server on Host E.
• The network engineer can use a protocol
analyzer on the Host A network to confirm that
the packets are being generated and sent to
the router. At this point, the network engineer
should notice that the configuration and
operation of Host A appears to be correct and
that no reply packets are being received. A
protocol analyzer running on the remote
network is reporting that no Telnet packets are
being received. Based on this information, the
network engineer can assume there is a • Because the problem has still not been solved,
problem with at least one of the routers. the network engineer now moves to the
• Because the access list on Router C is quite configuration of Router D. The ACL filtering
complex, there does not appear to be any
problem when the network engineer gives the inbound traffic on the serial interface is
configuration a visual check using show ip permitting the Telnet traffic and the packet
access-list. To be sure however, the deny ip counter against the appropriate ACE is
any any log statement is configured to incrementing with traffic.
highlight any packets not being permitted
through the ACL. The messages generated by • Using a protocol analyzer, the network engineer
the ACL logging highlight a misconfiguration confirms that the Telnet packets are now
that would have otherwise gone unnoticed, reaching the network of Host E and that replies
which is fixed by the network engineer. The
ACL is updated and the show ip access-list are being sent back to Host A. The protocol
command is used again to confirm that packets analyzer on the network of Host A, however, is
are being matched by the new access list not able to see any of these Telnet packets. It
element entered for the Telnet traffic. appears as though there is another problem on
one of the routers.

– The access lists on Router D are not as complex as those on Router C and the network engineer
immediately spots and corrects a configuration error.
– The next test works and the problem is considered resolved. The final activity the network engineer
should perform is to remove any unnecessary configuration changes to the network. Using the log
of activities generated during troubleshooting, the network administrator identifies that the use of
the deny ip any any log command only provided diagnostic information and can be removed from
the configuration.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 66
 }Dynamic NAT and extended ACLs
• The interaction of Dynamic NAT and extended access lists can generate complex
network problems, particularly regarding the use of addressing and ports.
• Addressing considerations
Recall that the order of processing inbound traffic on a router is that the inbound
traffic is processed by the inbound access list before being processed by outside-
to-inside NAT. When designing access lists for implementation on NAT routers,
remember that the destination address of inbound traffic will be the IP address
used by the outbound NAT translation.
• Dynamic NAT timeouts
When configuring dynamic NAT, different timeout values can be configured for
different types of traffic. Figure shows the commands used to change these
values for translations built with and without overloading.

– As discussed in previous content, highly tuned translation timeouts combined with network
congestion can be the cause of intermittent problems in network communications. Different
transport layer protocols also have different timeout values by default and can be configured
individually. This can mislead network engineers when troubleshooting, as discussed below.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 67
 }Dynamic NAT and extended ACLs
• Misconfiguration example
Figure shows a router with multiple
configuration errors. The symptom presented is
that neither Host A, nor Host B can establish a
reliable connection to download files from
Server D using TFTP.
• Following the traffic flow, the network engineer
checks the local network segment and finds
nothing wrong. The access lists filtering
outbound traffic are examined and no faults are
found. The NAT process on the router is also
building the appropriate translation for the
outbound traffic and the access list filtering
inbound traffic has an entry to permit traffic
from UDP port 69 on the TFTP server to any
UDP port on the private IP addresses allocated
to Hosts A and B. The engineer can ping Server • The network engineer examines the
D from both Hosts A and B and from Router C. configuration of the router and notices the
• At this stage, the network engineer configures following block in the configuration script:
the deny ip any any log command to find out
if the traffic is returning from the TFTP server, – ip nat translation udp-timeout 18
but is being blocked by the access list. The ip nat translation dns-timeout 120
logged messages indicate that TFTP traffic from ip nat translation tcp-timeout 3600
Server D is getting back to the router, but is
addressed to the IP address of the router serial • Assuming the first statement is an error, the
interface being used as the NAT overload network engineer replaces it with ip nat
outside address. The network engineer corrects translation udp-timeout 180 and tests the
the problems with the ACL and tries another file transfer again.
TFTP download.

– Host A can now establish a connection with Server D, but loses connection when doing a large file
transfer. There is not any pattern as to the point during the transfer at which the connection is lost.
The network engineer suspects that this may be a problem with UDP packets being lost in transit,
but uses a protocol analyzer on the WAN link to make sure (installing the WAN protocol analyzer
interrupts network communications, so the engineer waits until everyone has gone out to lunch).
– Results from the protocol analyzer show that the router is receiving more TFTP packets than it
should. This means that there is a problem with the router. Examining the router system log in
greater detail, the network engineer finds some error messages stating that router has ‘…received
packets for which no translation exists…’.
– The large file is transferred successfully and the problem is considered resolved. After updating the
appropriate documentation, the network engineer removes the unnecessary additions to the
network configuration, such as the WAN protocol analyzer and the deny ip any any log statement
in the inbound ACL.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 68
}TCP load distribution with NAT
• NAT on Cisco routers allows
network engineers to provide
TCP Load Distribution among
hosts. A graphical representation
of TCP Load Distribution is given
in Figure 1. Steps to configure
TCP load distribution with NAT
are in Figure 2.
• Although TCP load distribution
can improve the performance of
some types of network
transactions (such as accessing a
corporate intranet web service),
it can also be a source of
network complexity, resulting in
intermittent fault behaviors when
something goes wrong.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 69
}TCP load distribution with NAT
• Example 1 – host fault • Example 2 – host misconfiguration
Assume that the network is configured as Using the same network configuration
depicted in Figure 1. The router is from Figure 1, assume a new network
distributing TCP connections evenly engineer is instructed to build a new
among the hosts, as it should. Overnight, server. Because the new network
power to Host 1.1.1.2 fails and the device engineer does not understand how the
is no longer on the network. Because the TCP load distribution system works, the
router does not know this, it continues to new server is configured with the IP
forward TCP connection requests to the address assigned to the virtual host on
failed host. The result of this is that one the NAT router. When the new server is
in three connection attempts fail, powered up, it detects an IP address
presenting a seemingly intermittent fault. conflict and cannot establish a connection
• After a little investigation, the network to the network. The new engineer
engineer realizes that every third examines the interface configuration on
connection attempt is failing and the router to locate and confirm the
immediately suspects that one of the duplicate IP address, but can not find it in
hosts may have failed. Power is restored the interface configuration.
to Host 1.1.1.2 and network performance • After discussing the problem with a more
returns to normal. experienced engineer, the new engineer
reconfigures the new server with correct
IP settings, reboots, and connects to the
network.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 70
}Summary

• By completing this module, students should have

gained an understanding of the operation of various
transport layer networking technologies on routers
and hosts. These technologies include:
– Transport Control Protocol
– User Datagram Protocol
– NetBIOS
– Network Address Translation
– Extended access lists
• Students should also have gained an appreciation of
the various tools and methodologies that can assist
with troubleshooting transport layer issues.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 71
 }Q&A

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 72
Application Layer Troubleshooting

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1-1
 }Objectives
• The application layer is the top layer in the TCP/IP reference
model. When the ISO developed the OSI Reference Model,
the application layer functions were divided into three
separate, more detailed layers. Although the OSI version is
more detailed, it is more common to refer to the application
layer of TCP/IP since it is more encompassing.
• The application layer is the interface that separates
application software from the transport layer, and deals with
high-level protocols rather than segments, bytes, packets, or
bits.
• It provides network services for users and their programs
and is the layer in which user-access network processes
reside. These processes include all of those that users
interact with directly, as well as other processes of which the
users are not aware.

– This layer includes all application layer protocols that use the host-to-host transport protocols to
deliver data. Other functions that process user data such as data encryption, decryption,
compression and decompression, can also reside at the application layer.
– Most of the application layer protocols provide user services. Application layer protocols are
typically used for network management, file transfer, distributed file services, terminal emulation,
and electronic mail. However, new user services are often added (for example, VPNs, VoIP, and
so on).

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1-2
 }Objectives
• The most widely known and implemented • Other application layer protocols are listed
TCP/IP application layer protocols are listed below.
below: – Finger — User Information Protocol
– Telnet enables users to establish terminal – IMAP4 — Internet Message Access Protocol
session connections with remote hosts. – IPDC — IP Device Control
– HyperText Transfer Protocol (HTTP)
supports the exchanging of text, graphic images, – ISAKMP — Internet Message Access Protocol
sound, video, and other multimedia files on the – LDAP — Lightweight Directory Access Protocol
World Wide Web. – NTP — Network Time Protocol
– File Transfer Protocol (FTP) performs – POP3 — Post Office Protocol version 3
interactive file transfers between hosts.
– RLOGIN — Remote Login
– Trivial File Transfer Protocol (TFTP)
performs basic interactive file transfers typically – RTSP — Real-time Streaming Protocol
between hosts and networking devices (for – SCTP — Stream Control Transmission Protocol
example, routers, switches, and so on). – S-HTTP — Secure Hypertext Transfer Protocol
– Simple Mail Transfer Protocol (SMTP) – SLP — Service Location Protocol
supports basic message delivery services.
– TFTP — Trivial File Transfer Protocol
– Post Office Protocol (POP) is used to connect
to mail servers and download e-mail. – WCCP — Web Cache Coordination Protocol
– Simple Network Management Protocol – X-Window
(SNMP) is used to collect management
information from network devices.
– Domain Name Service (DNS) maps IP
addresses to the names assigned to network
devices. Commonly called name service.
– Network File System (NFS) enables
computers to mount drives on remote hosts and
operate them as if they were local drives.
Originally developed by Sun Microsystems, it
combines with two other application layer
protocols, external data representation (XDR),
and remote-procedure call (RPC), to allow
transparent access to remote network resources.

– These and other network applications use the services of TCP/IP and other lower-layer Internet
protocols to provide users with basic network services.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1-3
}Table of Content

1 Troubleshooting the Application Layer

2 Gathering Information on Application Layer
Problems
3 Troubleshooting TCP/IP Application Layer
Protocols
4 Troubleshooting TCP/IP Application Layer
Problems

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1-4
TROUBLESHOOTING THE
APPLICATION LAYER

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1-5
 }Overview
• The primary responsibility of the upper layers of the OSI model is to
provide services such as e-mail, file transfer, and data transport.
Application layer problems result when data is not delivered to the
destination or network performance degrades to a level where
productivity is affected.
• The same general troubleshooting process used to isolate problems at
the lower layers can be used to isolate problems at the application layer.
The ideas stay the same, but the technological focus has shifted to
involve things such as refused or timed out connections, access lists, and
DNS issues.
• Problem isolation is vital to successfully troubleshoot any problem.
Merely isolating the problem will not bring the types of changes
necessary to return network functions to the documented baseline. To
meet the troubleshooting objective of resolving the problem, use the
tools and resources that are provided to correctly configure the
properties of a properly functioning network.

– Application layer problems prevent services from being provided to application programs. A
problem at the application layer can result in unreachable or unusable resources when the physical,
data link, network, and transport layers are functional. It is possible to have full network
connectivity but the application simply cannot provide data.
– Another type of problem at the application layer occurs when the physical, data link, network, and
transport layers are functional, but the data transfer and requests for network services from a single
network service or application do not meet the normal expectations of a user.
– A problem at the application layer may cause users to complain that the network or the particular
application that they are working with is sluggish or slower than usual when transferring data or
requesting network services.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1-6
}Eliminating Layers 1-3
• When an application program cannot successfully connect to
the destination host, establish at which layer the problem
resides. Is it a lower layer problem or a higher layer
problem?
• For example, assume the problem is the inability to connect
to a remote FTP server. To determine whether this is an
application layer problem and not a lower layer problem, the
first step is to verify Layer 3 connectivity. If successful, Layer
3 and lower can be eliminated as the source of the problem.
• To troubleshoot use the following steps:
– Ping the default gateway. If successful, Layer 1 and Layer 2
services are functioning properly.
– Verify end-to-end (host-to-host) connectivity. Use an
extended ping if attempting the ping from a Cisco router.
• If these pings are successful, then Layer 1 through Layer 3
can be eliminated. Since they are functioning properly, the
issue must exist at a higher layer.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1-7
}Eliminating Layer 4
• Layer 4 is the home of UDP and TCP protocols and is not as easy to
eliminate.
• For example, assume there are FTP connection problems. To
troubleshoot Layer 4, use the following steps:
1. Use the show access-list command. Are there any access-lists that could
be stopping traffic? Notice which access lists have matches.
2. Clear the access-list counters with the clear access-list counters
command and try to establish an FTP connection again.
3. Verify the access-list counters. Have any increased? Should they increase?
• Improperly configured access lists are common problem areas. Be sure
the implications of each access list statement are understood. This may
sound strange but it sometimes helps to think like the packet.
• However, if the access lists are functioning as expected, then the
problem must lie in a higher layer.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1-8
}Isolating application layer problems
• Even though there may be IP connectivity between a source and a
destination, problems may still exist for a specific upper-layer protocol
such as FTP, HTTP, or Telnet. These protocols ride on top of the basic IP
transport but are subject to protocol specific problems relating to packet
filters and firewalls. It is possible that everything except mail will work
between a given source and destination.
• Before troubleshooting at this level, it is important to establish whether
IP connectivity exists between the source and the destination. If IP
connectivity exists, then the issue must be at the application layer.
• The following list outlines possible issues:
– A packet filter/firewall issue might have arisen for the specific protocol, data
connection, or return traffic.
– The specific service could be down on the server.
– An authentication problem might have occurred on the server for the source
or source network.
– There could be a version mismatch or incompatibility with the client and
server software.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1-9
 }Isolating application layer problems
• Troubleshooting an upper-layer
protocol connectivity problem
requires understanding the process
of the protocol. This information is
usually found in the latest RFC for
the protocol or on the developer web
page.
• Questions that should be answered
to make certain the functions of the
protocol are understood include the
following:
– What IP protocols does the protocol
use (TCP, UDP, ICMP, IGMP)?
• Move the client outside the firewall or address
– What TCP or UDP port numbers are translation device.
used by the protocol?
• Verify whether the client can connect to a
– Does the protocol require any
server on the same subnet as the client.
inbound TCP connections or inbound
UDP packets? • Capture a network trace at the client LAN and
– Does the protocol embed IP on the LAN closest to the server or preferably,
addresses in the data portion of the on the server LAN.
packet? • If the service is ASCII based, telnet to the port
– Are the protocols being used on a of the service from the router closest to the
client or a server? server, then work backward into the network
toward the client.

– If the protocol embeds IP addresses in the data portion of the packet and NAT has been configured
anywhere along the path of the packet, the NAT gateway will need to know how to deal with that
particular protocol or the connection will fail. NAT gateways typically change information in the
data portion of a packet only when they have been specifically coded to do so. Some examples of
protocols that embed IP addresses in the data portion of the packet are FTP, SQLNet, and
Microsoft WINS.
– If there is a question regarding whether a firewall or router is interfering with the flow of data for a
particular application or protocol, several steps can be taken to see what exactly is happening.
These steps may not all be possible in every situation.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 10
}Identifying support resources
• Some application problems can
be resolved by reading technical
documentation at the software
vendor or developer’s website.
These sites also have patches
and version updates that a
troubleshooter can download to
repair bugs or incompatibilities.
• When troubleshooting network
problems, network administrators
must know where to find
information.
• Good sources of information
include:
– Standard organizations
– Technical forums
– Cisco Technical Assistance
Center
– Discussion groups

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 11
 }Accessing support resources
• In most cases, network problems can be resolved without
assistance from any outside technical support. However,
some problems may seem to be too elusive and professional
help is required. This is when Cisco Systems Technical
Assistance Center (TAC) should be utilized.
• It is suggested the following be completed before
calling Cisco (TAC):
– Have the service contract number ready. TAC will ask for
it.
– Have a diagram of the network, or the affected portion of
the network. Make sure all IP addresses and associated
network masks or prefix lengths are listed.
– List the steps already taken and their results compiled for
the TAC engineer.
– If the problem appears to be with only a few routers
(fewer than four), capture the output from show tech
command from these routers.

– Dial-in or Telnet access also helps considerably in effective problem resolution.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 12
}Correcting application layer problems
• Use the following steps:
1. Make a backup. Before proceeding, ensure that a valid configuration has
been saved for any device on which the configuration may be modified. This
provides for recovery to a known initial state.
2. Make initial hardware and software configuration changes. If the correction
requires more than one change, make only one change at a time.
3. Evaluate and document the change and the results of each change. If the
results of any problem-solving steps are unsuccessful, immediately undo the
changes. If the problem is intermittent, wait to see if the problem occurs
again before evaluating the effect of any change.
4. Verify that the change actually fixed the problem without introducing any
new problems. The network should be returned to the baseline operation
and no new or old symptoms should be present. If the problem is not
solved, undo all the changes. If new or additional problems are discovered,
modify the correction plan.
5. Stop making changes when the original problem appears to be solved.
6. If necessary, get input from outside resources. This may be a coworker,
consultant, or Cisco Technical Assistance Center (TAC). On rare occasions a
core dump may be necessary, which creates output that a specialist at Cisco
Systems can analyze.
7. Once the problem is resolved, document the solution.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 13
GATHERING INFORMATION ON
APPLICATION LAYER PROBLEMS

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 14
}Overview
• To make quick and accurate troubleshooting decisions, a network
administrator must be able to get the right information at the right time.
• There are several tools available to help in this troubleshooting process.
However, the best time to learn about these tools is not when a problem
is encountered. The best time to explore and learn these tools is when
the network is functioning correctly. This way network baselines can be
established and recorded. When problems occur, administrators should
refer to the normal baseline to identify inconsistencies more quickly.
• In short, an administrator must not only know about the tools, but they
must also be able to recognize and decipher the pertinent information
provided by the various tools.
• An administrator should be fluent with all the following tools:
– Command line (UNIX, DOS, Cisco IOS)
– Windows, UNIX, IOS utilities
– Protocol Analyzers
– Network Management Systems
– System logs

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 15
}Common TCP/IP commands
• The TCP/IP protocol suite offers several commands to help
troubleshoot Application Layer problems. Most of these
commands should be very familiar while others may be new.
Take time to fully understand and appreciate the value of
these commands.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 16
}Common TCP/IP commands
• Ping
Ping is the most frequently used network monitoring and troubleshooting
tool. Although it basically tests Layer 3 connectivity, it can be used to
help solve application layer problems.
• For example, a troubleshooting strategy using ping can be used to
identify a DNS application layer problem.
• If there is high latency due to congestion, it may cause application layer
problems because of timeout issues. In a WAN setting, latency between
packets should be expected. However, in a LAN setting, excessive
latency between packets could be an indication of network problems.
Ping is an excellent tool for identifying latency issues.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 17
 }Common TCP/IP commands
• Traceroute
Traceroute can be used to pinpoint a network problem. It identifies each
intermediate router on the way from host A to host B.
• As shown in Figure , traceroute sends the first packet with a TTL value of 1.
The first router decrements this and since the value drops to zero, the router
discards the packet and sends an ICMP Time-to-live Exceeded message back to
the sender. Traceroute then sends a packet with a TTL value of 2, which the first
router decrements and routes. But the second router decrements it to zero, and
sends an ICMP error message back. Ultimately, the TTL gets high enough for the
packet to reach the destination host, and traceroute is done, or some
maximum value (usually 30) is reached and traceroute ends the trace.

– Please note that most traceroute programs send a UDP datagram to a randomly selected high UDP
port. Microsoft’s tracert uses an ICMP echo request message (a ping packet) instead, which may
explain why some trace results do not match those of other users.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 18
}Common TCP/IP commands
• Pathping
Pathping is a Windows NT/2000/XP feature that combines the features of
the ping and tracert commands with additional information-gathering
features. The pathping command sends packets to each router on the
way to a final destination over a period of time and then computes
results based on the packets returned from each hop. Pathping displays
the degree of packet loss at any given router or link. This makes it easier
to determine which routers or links might be causing network problems.
• Nslookup
The most useful tool for troubleshooting DNS problems is nslookup. It
lets a user enter a host name (for example, cisco.com) and find out the
corresponding IP address. It will also do reverse name lookup and find
the host name for a specified IP address.
• Nslookup sends a domain name query packet to a designated (or
defaulted) domain name system (DNS) server. Depending on the system
being used, the default may be the local DNS name server at the service
provider, some intermediate name server, or the root server system for
the entire domain name system hierarchy.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 19
}Common TCP/IP commands
• Netstat
Netstat is used to report on the
routing table of the system, TCP and
UDP protocols, open connections
(ports), and the remote systems
ports. It gets this networking
information by reading the routing
tables in the memory, and then
provides an ASCII format at the
terminal.
• Every machine connected to an IP
network has an IP routing table.
How this information is displayed is
platform dependent. The output of
netstat – n and netstat – r on a
Windows platform (netstat –r
produces the same output as route
print) is shown in Figure .
• Other useful netstat commands
include netstat -a, which displays
all connections, and netstat -e,
which displays Ethernet statistics.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 20
}Platform specific TCP/IP utilities
• The traffic requirements of various
platforms influences how network
devices are configured. Five
situations where traffic requirements
would affect router setup are shown
in Figure.
• TCP/IP troubleshooting combines
facts gathered from network devices
such as routers and switches, and
facts gathered from a client or
server.
• To check the local host configuration
on a Windows NT/2000/XP system,
open a DOS command window on
the host and enter the ipconfig /all
command. The resulting output
displays the TCP/IP address
configuration, default gateway,
DHCP server, and Domain Name
System (DNS) server addresses. If
any IP addresses are incorrect or if
no IP address is displayed,
determine the correct IP address and
edit it or enter it for the local host.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 21
}Platform specific TCP/IP utilities
• The Windows NT/2000/XP
platform will log most
incorrect IP address or
subnet mask errors in the
Event Viewer. Examine the
Event Viewer system log
and look for any entry with
TCP/IP or DHCP as the
source.
• Read the appropriate
entries by double-clicking
them.
• Because DHCP configures
TCP/IP remotely, DHCP
errors cannot be corrected
from the local computer.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 22
}Platform specific TCP/IP utilities
• Also, check the configurations on the
NT/2000/XP server. If a connection using
an IP address is possible but the
connection cannot be made using
Microsoft networking (for example,
Network Neighborhood), try to isolate a
problem with the Windows NT/2000/XP
server configuration. Problem areas with
Microsoft networking relate to NetBIOS
support and associated mechanisms used
to resolve non-IP entities with IP
addresses. Non-IP problems can be
checked using the nbtstat command.
• As a last resort, try rebooting the
Windows system. Although this practice is
not encouraged, it frequently repairs the
problem.
• Figure 2 shows some general commands
used for isolating application layer
problems. While many of these
commands display lower layer
information, the commands are still useful
because they highlight problems in the
application layer.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 23
}Platform specific TCP/IP utilities

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 24
 }Cisco IOS commands

– The Cisco IOS software offers powerful commands to help in monitoring and troubleshooting
network problems. The following highlights some of the most common and useful commands.
– The show commands help monitor installation behavior and normal network behavior, as well as
isolate problem areas.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 25
 }Cisco IOS commands

– The debug commands assist in the isolation of protocol and configuration problems.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 26
}Cisco IOS commands
• The router show commands are among the most important tools for understanding the
status of a router, detecting neighboring routers, monitoring the network in general, and
isolating problems in the network.
• These commands are essential in almost any troubleshooting and monitoring situation. Use
show commands for the following activities:
– Monitoring router behavior during initial installation
– Monitoring normal network operation
– Isolating problem interfaces, nodes, media, or applications
– Determining when a network is congested
– Determining the status of servers, clients, or other neighbors
• The debug EXEC commands can provide a wealth of information about the traffic being
seen (or not seen) on an interface, error messages generated by nodes on the network,
protocol-specific diagnostic packets, and other useful troubleshooting data. Be conservative
with debug commands as these commands often generate quite a bit of extraneous data.
• Use debug commands to isolate problems, not to monitor normal network operation. Use
debug commands to look for specific types of traffic or problems after narrowing the
problems to a likely subset of causes.
• Figure shows examples of IOS troubleshooting commands.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 27
 }System logs
• Logging enables the router or switch to keep track of events that occur.
Logging can help find trends, system error messages, outages, and a
variety of other network events.
• The logging facility:
– Provides logging information for monitoring and troubleshooting
– Allows selection of the types of logging information captured
– Allows selection of the destination of captured logging information
• There are several types of events that can be monitored. Messages are
classified in terms of levels of severity. Level 0 is the highest level (most
severe) and level 7 is the lowest level (least severe). System messages
can be saved based on the type of facility and the severity level.

– How an administrator chooses to implement system logging and manage logging data may affect
their ability to manage their networks and effectively troubleshoot problems. Time should be taken
to develop a logging strategy that will provide reliable data when required.
– Monitoring activity in the log files is an important aspect of network management and should be
conducted regularly. Monitoring the log files allows the execution of appropriate and timely action
when problems are detected, such as breaches of security or events that are likely to lead to a
potential security breach.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 28
 }System logs
• Syslog messages can be categorized as follows:
– Warning, Errors, Critical, Alerts, and Emergencies are Error
level messages generated by software or hardware malfunctions.
– Notification level messages generated by interface up/down
transitions and system restart messages.
– Informational level messages generated by reload requests and
low-process stack messages.
– Debugging level messages generated by output from the debug
commands.
• Which event an administrator decides to capture depends
largely on the information they are seeking.
• The logging facility can also be configured to send captured
logging information to select destinations.
• By default, switches and routers normally log significant system
messages to their internal buffer and the system console.

– For example, logs can be invaluable in characterizing and responding to security incidents. To do
so, the most important events to log include change of interface status, changes to the system
configuration, access list matches, and events detected by the optional firewall and intrusion
detection features.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 29
}System logs
• The four destinations that syslog messages can be forwarded to are listed below:
– Console terminal
– Virtual terminals
– Internal buffer
– Syslog server
• Be aware that the debugging destination that is used, affects system overhead.
Logging to the console produces very high overhead, whereas logging to a
virtual terminal produces less overhead. Logging to a syslog server produces
even less, and logging to an internal buffer produces the least overhead of any
method.
• Time, specifically timestamp, is a valuable piece of information used to
determine when a problem arose. The idea behind this is that many network
problems can often be correlated to system configuration changes, modifications
to the network topology (both intentional and unintentional), and so on. For this
reason, syslog messages should be time-stamped to enhance real-time
debugging and management.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 30
}Syslog destinations
• Message logging is enabled by default.
However, the default could have been
disabled with the no logging on
command.
• To enable message logging to all
supported destinations other than the
console (the default), enter the following:
• Router(config)#logging on
• The logging process controls the
distribution of logging messages to the
various destinations, such as the logging
buffer, terminal lines, or syslog server. To
turn logging on and off for these
destinations individually use the logging
buffered, logging monitor, and • Additionally, the logging process logs messages
logging global configuration commands. to the console and the various destinations
after the processes that generated them have
• If the no logging on command has been
configured, no messages will be sent to completed. When the logging process is
these destinations. Only the console will disabled, messages are displayed on the
receive messages. console as soon as they are produced, often
appearing in the middle of command output.
• However, disabling the logging on
command will substantially slow down the • The logging synchronous line configuration
router. Any process that is generating command also affects the displaying of
debug or error messages will wait until messages to the console. When configured,
the messages have been displayed on the messages will appear only after the user types
console before continuing. a carriage return.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 31
}Syslog destinations
• Selecting Console Logging Levels
Different logging levels and corresponding keywords can be used when setting
logging levels. The highest level message is Level 0, emergencies. The lowest
level is Level 7, debugging, which also displays the largest number of messages.
• To limit the types of messages that are logged to the console, use the logging
console command. The full syntax of this command follows:
– Router(config)#logging console level
• The logging console command limits the logging of messages displayed on the
console terminal to the specified level and (numerically) lower levels. The level
number or level name can be entered.
• For example, the following sets the console logging to the warnings level. This
will display all warnings (4), as well as errors (3), critical (2), alerts (1), and
emergencies (0) messages.
– Router(config)#logging console warnings or logging console 4
• The no logging console command disables logging to the console terminal.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 32
}Syslog destinations
• Logging to the Internal Buffer
To log messages to an internal buffer, use the logging buffered router configuration
command. The full syntax of this command follows:
– Router(config)#logging buffered
• The logging buffered command copies logging messages to an internal buffer instead of
writing them to the console terminal. The buffer is circular in nature. Therefore, newer
messages overwrite older messages.
• To limit the types of messages that are logged to the buffer, use the logging buffered
level command. The level argument is one of the keywords listed in Figure . The full
syntax of this command follows:
– Router(config)#logging buffered level
• To display the messages that are logged in the buffer, use the privileged EXEC command
show logging. Use the clear logging command to reset the logging buffer. The no
logging buffered command cancels the use of the buffer and writes messages to the
console terminal (the default).

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 33
}Syslog destinations
• Logging to the Terminal Lines
To log messages logged to the terminal lines (VTY), use the logging monitor router configuration
command. The full syntax of this command follows:
• Router(config)#logging monitor level
• The logging monitor command limits the logging messages displayed on terminal lines other than the
console line to messages with a level up to and including the specified level argument.
• To display logging messages on a terminal (virtual console), use the privileged EXEC command
terminal monitor.
• Logging to a Syslog Server
Messages can also be logged to a syslog server. The host is required to be running a Syslog Server
application such as Unix Syslog server (native in most Unix implementation) or Kiwi Syslog Daemon
(Win9x, ME, XP, NT4, and 2000). Commands to set up a Unix Syslog server are covered later in this
module.
• To log messages to the syslog server host, use the logging ip-address configuration command. The
full syntax of this command follows:
• Router(config)#logging ip-address
• The logging command identifies a syslog server host to receive logging messages. The ip-address
argument is the IP address of the host. By issuing this command more than once, a list of syslog servers
to receive logging messages is created.
• The no logging command deletes the syslog server with the specified address from the list of syslogs.
• To limit the number of messages sent to the syslog servers, use the logging trap router configuration
command. The full syntax of this command follows:
• Router(config)#logging trap level
• The logging trap command limits the logging messages sent to syslog servers to messages with a
level up to and including the specified level argument. The default trap level is informational. The no
logging trap command disables logging to syslog servers.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 34
 }Deciphering syslog messages
• All messages begin with a percent sign, and are
displayed in the following format:
• %FACILITY-SEVERITY-MNEMONIC:
Message-text
• FACILITY is a code, consisting of two to five
uppercase letters, indicating the facility to
which the message refers. A facility may be a
hardware device, a protocol, or a module of the
system software. The IOS has over 500 service
identifiers.
• SEVERITY is a single-digit code from 0 to 7
that reflects the severity of the condition. The
lower the number, the more serious the
situation. MNEMONIC is a code, consisting of
uppercase letters that uniquely identify the
message.
• Message-text is a text string describing the
condition. This portion of the message
sometimes contains detailed information about
the event being reported, including terminal
port numbers, network addresses, or addresses
that correspond to locations in the system
memory address space. Because the
information in these variable fields changes
from message to message (see below), it is
represented here by short strings enclosed in
square brackets ([ ]). For example, a decimal
number is represented as [dec].

– Some example error messages could be as follows:

– Error message: %HELLO-2-NORDB: Redistributed IGRP without rdb
– In this message, HELLO is the facility, 2 is the severity, and NORDB is the MNEMONIC. This
message indicates that an internal software error has occurred and technical support should be
contacted for assistance.
– Error message: %IP-4-DUPADDR Duplicate address [inet] on [chars], sourced by [enet]
– This error message indicates that another system on the network segment is using this IP address
and that the IP address on one of the two systems should be changed.
– If one or more error messages reoccur after the recommended action has been taken, contact Cisco
or a local field service organization.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 35
}Protocol analyzers
• Network management involves using
network and protocol analysis tools to
establish a network system baseline and
to monitor and optimize performance.
• Protocol analyzers are almost always
software-based. They are used to gather
information about traffic flows and are
very useful for establishing a network
baseline. Although they do not decode
the contents of frames, protocol
analyzers are often used for solving Layer
2 and higher problems.
• They can be used to assist in locating
traffic overloads, planning for network
expansion, detecting intruders, • Note: Some devices may come equipped
establishing baseline performance, and with traffic monitoring capabilities. For
distributing traffic more efficiently. example, the Cisco Catalyst® 6500 Series
• Using these tools effectively is not easy. switch can be equipped with a Network
Administrators must be able to decipher Analysis Module (NAM). The NAM is an
and interpret the information generated. integrated and powerful traffic monitoring
system. It comes with an embedded web-
• Examples of protocol analyzers include based Traffic Analyzer, which provides full
Fluke’s Protocol Inspector and Sniffer Pro scale remote monitoring and
Protocol Analyzer. troubleshooting capabilities accessible
through a web browser.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 36
}Network management systems
• Network management systems are always software-
based tools. They continually monitor the network.
There are various types of network management
systems and not all are equal. Some are better at
status monitoring and fault management tracking
while others are better at service-level reporting.
The choice is sometimes confusing since features
overlap.
• Network Management System (NMS) functions can
be categorized into three main categories:
– Operations management
– Device management
– Service management

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 37
}Network management systems
• Operations management tools are used for active monitoring of day-
to-day network administration. The software provides features such as
network topology discovery, status monitoring, fault management, and
basic real-time performance data. Major vendors include HP OpenView
(current market leader), Computer Associates, and IBM Tivoli.
• Device management tools are typically vendor specific. They are used
to manage a vendor's network components to make configuration
changes to network devices and to apply rules and policies. Most provide
graphical tools to interact with actual devices. Examples of device
management tools include Cisco Systems’ CiscoWorks (Cisco), Navis
iEngineer (Lucent), and Optivity (Nortel).
• Service management tools focus on QoS and service-level guarantee
issues. They collect performance data over time that is then used for
establishing a baseline, trend analysis, historical usage analysis, and
service-level reporting. The tools focus on comparing the expected
quality of network resources with actual results. Major vendors include
HP, Lucent, and NetScout Systems.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 38
}Network management systems
• SNMP
Network management tools use the Simple Network Management
Protocol (SNMP) to capture and communicate device data. NMS
periodically polls the devices it manages, sending queries for their
current status. The monitored devices respond by transmitting the
requested data and by sending traps (called notifications in SNMPv2).
• A trap is an unsolicited message to the NMS, generated when a
monitored parameter reaches unacceptable levels. For example, an
environmental monitoring device may send a trap when the temperature
level is too low or too high. Traps are useful because they provide a
method for a device to signal that something unexpected has occurred.
• In SNMP, the term manager refers both to the monitoring software
running on the NMS and the actual device running the software.
Similarly, the term agent refers to the device being monitored and to the
software used by the monitored devices to generate and transmit their
status data.
• SNMP is a client-server protocol that normally communicates on TCP and
UDP ports 161. SNMP traps use TCP and UDP ports 162. Some vendors
use nonstandard ports for traps (for example, Cisco uses TCP and UDP
ports 1993).

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 39
TROUBLESHOOTING TCP/IP
APPLICATION LAYER
PROTOCOLS

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 40
}Overview
• Application layer protocols can be very difficult to isolate.
Test and eliminate any problems in the lower layers before
attempting to isolate upper layer problems.
• This section focuses on how to isolate problems with various
application layer protocols such as:
– Telnet
– HTTP
– SMTP, POP, and IMAP
– FTP, TFTP
– DNS
– SNMP
– NTP
– DHCP

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 41
}Client-server systems
• A client-server model is a network architecture in which a computer
(client) requests access to services offered on another remote host
(server). The model provides a convenient way to remotely interconnect
programs located in different locations. Computer transactions using the
client-server model are very common.
• Clients are PCs or workstations on which users run applications. Clients
rely on servers for resources, such as files, devices, and even processing
power.
• A client is defined as a requester of services and a server is defined as
the provider of services. A single machine can be both a client and a
server depending on the software configuration.
• Servers are powerful computers dedicated to managing disk drives (file
servers), printers (print servers), or network traffic (network servers). A
server receives a request and, after any necessary processing, the
requested file is returned to the client. Typically, multiple client programs
share the services of a common server.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 42
 }Terminals and consoles
• Telnet is the standard terminal emulation protocol in the TCP/IP
protocol stack. Telnet is defined in RFC 854 and operates over the TCP
port 23. Telnet and FTP were the first two services available on
ARPANET.
• To understand how Telnet works, it is necessary to first make a
distinction between console and terminal. A console refers to a keyboard
and monitor that are directly connected to the computer system. In
mainframe computing, a console was also referred to as a dumb terminal
since it only operated by using the resources of the remote server. A
microcomputer is now more commonly used as a console.
• All consoles require a terminal connection to enable users to log in to
remote systems and use resources (for example, CPU, applications, and
storage) as if they were connected to a local system. A terminal is a
console that artificially emulates the physical hookup of a console. The
destination host assumes it has a direct connection to the client since the
terminal just provides a communication channel for the user's input and
output. A terminal program is commonly used to connect to a central
server over the network.
• Use the terminal monitor IOS command to redirect the output to any
of the VTY ports. Keep the amount of debugging that is enabled to a
minimum.

– The term "terminal emulator" refers to a terminal application that is implemented in software.
Clients can use the Telnet program to establish a terminal connection. Other software such as
HyperTerminal or TeraTerm can also be used and they typically offer more advanced features.
– Note: Debug output is normally sent to the console port. This means that if the connection is
established through the Telnet port, the debug output will not be seen. Use the terminal monitor
IOS command to redirect the output to any of the VTY ports. Keep the amount of debugging that is
enabled to a minimum.
– Note: RDP Protocol (Terminal Server) is a little known Telnet port (port 3389) used by Remote
Desktop Protocol (RDP). RDP is the remote Windows terminal protocol used by Microsoft
Windows NT 4.0, Terminal Server Edition operating system and Windows 2000 Terminal
Services. Keep this in mind when troubleshooting.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 43
 }Terminals and consoles
• Telnet for Troubleshooting • Source Telnet Interface
Network administrators often overlook Finally, a useful IOS command to use
Telnet as a troubleshooting tool. when testing an access list is the ip
However, Telnetting to a host allows telnet source-interface command. This
better verification of network status than specifies the IP address of an interface as
just using ping. Telnet runs on top of the the source address for Telnet
TCP protocol, so it establishes a more connections. To reset the source address
reliable indication of accessibility than to the default for each connection, use
ICMP echo requests can. It also tests the no form of this command.
higher-level functions of the destination • By default, Telnet will use the IP address
host system. A server may be inaccessible of the closest interface to the destination
for application layer functions, but still as the source address. However,
answers pings since those are handled by sometimes another interface may be
the lower layer protocols. preferred as the source. Conceptually,
• Telnet also has an additional feature that this is similar to specifying another source
makes it valuable for troubleshooting IP address when using an extended ping
application layer protocols. Telnet client command.
applications allow the user to select the • The following example forces the IP
destination port number to be used. It address for FastEthernet interface 0/1 as
can be used to connect to other TCP the source address for Telnet
ports on destination hosts to test out connections:
other functions. That means that Telnet
can contact network application programs • Router(config)#ip telnet source-
other than a Telnet server. This can be interface FastEthernet 0/1
useful as a substitute for a client
application program.

– For example, Telnetting to port 25 (SMTP) will verify that the e-mail server is answering. Telnet to
port 80 (HTTP) to verify the Web server is answering.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 44
 }Web traffic
• Hypertext Transfer Protocol (HTTP) is the protocol used to transfer the
files that make up web pages. Although the HTTP specification allows for
data to be transferred on port 80 using either TCP or UDP, most
implementations use TCP.
• HTTPS is a secure version of the HTTP protocol. Aside from the initial
connection and setup, HTTPS and HTTP are basically the same. The
difference lies in the initial setup between client and server. HTTPS uses
the Secure Socket Layer (SSL) protocol. SSL was created in order to
secure credit card purchases over the Internet. It requires that both
sides of a connection be authenticated and that data be encrypted and
decrypted. It uses port 443 to initiate a secure connection.
• HTTP connectivity can be tested using any Telnet application that allows
a port number to be specified by Telnetting to the IP address of the
destination server using port 80.
– If the connection failed, a message will display stating that the Telnet
application could not open a connection to the host on port 80.
– If the connection was successful, a hello message may be displayed or a
Telnet window will open, but there will be no response. This indicates HTTP
connectivity to the server.
– To have the Web server respond, type GET / HTTP/1.0, then press
the enter key twice.

– Note: By default characters are not echoed in MS Telnet upon successful connection to a Web
Server unless 'local echo' is enabled in the preferences.
– The following table presents commands that can be used to troubleshoot a World Wide Web
network application. A troubleshooter uses the information from these commands to isolate
problems at the application layer that are related to the Web and the HTTP protocol.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 45
 }Electronic mail
• Simple Mail Transport Protocol (SMTP) is used to transport e-mail messages in ASCII
format using TCP between clients and servers. Other protocols such as Post Office Protocol
(POP) or Internet Access Message Protocol (IMAP) are used to retrieve e-mails from mail
servers.
• POP v3 is the current version of the protocol and it is incompatible with earlier versions.
POP3 downloads user e-mails to the local computer. For this reason, POP3 is best suited in
situations where users retrieve their e-mail from the same computer. If users use different
computers, their e-mails will likely be spread around several computers.
• IMAP v4 is another alternative that lets users download their e-mail at any time to any
computer.
• Because these different protocols are used to send and receive mail, it is possible that mail
clients can perform one task and not the other. Therefore, when verifying the configuration
of a mail client, both the mail relay (SMTP) server and mail (POP or IMAP) servers should
be verified.
• SMTP, IMAP, and POP connectivity can be tested using any Telnet application that allows a
port number to be specified. Telnet to the IP address of the destination server using ports
25, 143, and 110 respectively.
• The following commands can be used to isolate application layer problems related to email
and the POP3, SMTP, and IMAP protocols.

– For example, assume User-A wants to send e-mail to User-B. When User-A clicks on the Send
button, e-mail is sent to the local e-mail server using the SMTP protocol. The e-mail server will
then send the e-mail using SMTP to User-B’s e-mail server. It remains stored there until User-B
collects it. Later, User-B connects to the local e-mail server and downloads the e-mails using either
POP3 or IMAP 4.
– If there is a problem with the receiving system, the user should see a text error message. If the
connection was successful, a hello message will be displayed or an unresponsive Telnet window
will open. This indicates connectivity to the server. At this point, the user could use POP3 or
SMTP text-based commands to perform basic e-mail procedures such as authenticate, read, delete,
or send messages.
– For example, basic POP commands include user, pass, stat, list, top, uidl, retr, dele, noop,
rset, and quit.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 46
 }File transfer
• File Transfer Protocol (FTP) is used for uploading and downloading files between remote
computer systems on a network. Servers run FTP services or FTP daemons, and clients
connect by way of the TCP/IP FTP client command line interface or with a third party
commercial program that offers a graphical user interface (for example, WS_FTP Pro, UNIX
NcFTP Client, and Linux IglooFTP PRO). A Web browser can also make FTP requests to
download programs selected from a Web page.
• This data connection can be opened in several different ways:
– Traditional (or active)—The FTP server opens a TCP connection back to the client's port 20.
This method will not work on a multi-user system because many users may make simultaneous
FTP requests, and the system will not be capable of matching incoming FTP data connections to
the appropriate user.
– Multi-user traditional (or active)—The FTP client instructs the FTP server to open a
connection on some random port in the range 1024 through 65,535. This method creates a rather
large security hole because it requires system administrators to permit inbound TCP connections to
all ports greater than 1023. Although firewalls that monitor FTP traffic and dynamically allow
inbound connections help close this security hole, many corporate networks do not permit this
type of traffic. Most command-line FTP clients default to this method of transfer and offer a
passive command (or something similar) to switch to passive mode.
– Passive mode—The FTP client instructs the FTP server that it wants a passive connection, and
the server replies with an IP address and port number to which the FTP client can open a TCP data
connection. This method is by far the most secure because it requires no inbound TCP connections
to the FTP client. Many corporate networks permit only this type of FTP transfer. Most web
browsers default to this method of FTP transfer.
• As an example, assume a typical FTP connection process to connect to an FTP server and
download a file called README. Once logged in to an FTP server, the user could type help
to get a listing of acceptable commands.
• Some of the more popular FTP commands include ascii, binary, cd, dir, get, help,
Is, mkdir, put, pwd, and quit.

– FTP uses two or more TCP connections to accomplish data transfers. To start a session, the FTP
client opens a TCP connection to port 21 on the FTP server. This connection is called the control
connection and is used to pass commands and results between the client and the server. No data,
such as file transfers or directory listings, is passed over the control connection. Instead, data is
transferred over a separate TCP connection called the data connection.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 47
}File transfer
• An FTP connection can be tested using • TFTP
any Telnet application that allows a port Trivial File Transfer Protocol (TFTP) is a
number to be specified. Telnet to the IP simplified version of FTP. Unlike FTP that
address of the destination server using uses the TCP transport protocol, TFTP
port 21. If the connection is successful, a operates over port 69 and makes use of
hello message will be displayed or an the UDP protocol. UDP makes TFTP faster
unresponsive Telnet window will open. at uploading and downloading files.
This indicates connectivity to the server. • A client can only read or write a file to a
At this point the user may want to type in TFTP server. Unlike FTP, TFTP does not
help to see which commands are support directory-browsing, file renaming,
available. Since the connection to the FTP logging in, or statistics. For this reason, a
server is by way of Telnet, the choice of user must know the filename of the file
commands will vary. they wish to download.
• In some instances, a router can be • A common TFTP application is to back up
configured to act as an FTP server. FTP and restore router configuration files and
clients can copy files to and from certain IOS images.
directories on the router. For example,
the FTP Server allows retrieval of files, • The following commands display
such as syslog files, from the disk file information about file management
system on the router. applications. A troubleshooter uses the
information from these commands to
• When the router receives a request for an isolate problems at the application layer
FTP connection, the FTP Server process is that are related to the FTP and TFTP
started. At this point, the user is typically protocols.
prompted for a username and password.
After supplying a valid username and
password, various commands can be
entered.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 48
}Network management and time protocols
• NTP
Logging time is very important in determining when a problem started. Most network
problems can be narrowed down to a configuration change or modifications to the network
topology. A synchronized time enables correlation of syslog and Cisco IOS debug output to
specific events. While the primary goal of problem resolution is to fix the problem, it is also
quite helpful to know when the problem originated so that the problem can be resolved
and avoided in the future.
• The Network Time Protocol (NTP) synchronizes timekeeping among a set of distributed
time servers and clients. This synchronization allows events to be correlated when system
logs are created and other time-specific events occur. For timestamps to be of use, it is a
good idea for all the routers and switches in the network to derive time from a common
network time source.
• Configuring time services on routers requires exec and configuration commands. To
configure the time zone properties on the router, the configuration commands clock
timezone and clock summer-time are used. The commands ntp server ip-addr and
ntp source interface define the NTP server(s) and the source IP address of the NTP
requests.
• The internal clock of the router is set using the EXEC command clock set. To view NTP
peer status information, use the show ntp associations and show ntp status
commands.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 49
 }Network management and time protocols
• SNMP
Simple Network Management Protocol (SNMP) is an application-layer protocol
that facilitates the exchange of management information between network
devices. It is part of the Transmission Control Protocol/Internet Protocol (TCP/IP)
protocol suite.
• Although troubleshooting is necessary to recover from problems, the ultimate
goal of the network administrator is to avoid problems. That is also the goal of
network management software. The network management software used on
TCP/IP networks is based on the Simple Network Management Protocol (SNMP).
• SNMP is a client/server protocol. In SNMP terminology, it is described as a
manager/agent protocol. The agent (the server) runs on the device being
managed, which is called the Managed Network Entity. The agent monitors the
status of the device and reports that status to the manager.
• The manager (the client) runs on the Network Management Station (NMS). The
NMS collects information from all of the different devices that are being
managed, consolidates it, and presents it to the network administrator. This
design places all of the data manipulation tools and most of the human
interaction on the NMS. Concentrating the bulk of the work on the manager
means that the agent software is small and easy to implement. This is why most
TCP/IP network equipment comes with an SNMP management agent.
• SNMP is a request/response protocol. UDP port 161 is its well-known port. SNMP
uses UDP as its transport protocol because it has no need for the overhead of
TCP.

– Reliability is not required because each request generates a response. If the SNMP application does
not receive a response, it simply re-issues the request. Sequencing is not needed because each
request and each response travels as a single datagram.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 50
 }Network management and time protocols
• SNMP
Polling also reduces the burden on the network
because the polls originate from a single
system at a predictable rate. The shortcoming
of polling is that it does not allow for real-time
updates. If a problem occurs on a managed
device, the manager does not find out until the
agent is polled. To handle this, SNMP uses a
modified polling system called trap-directed
polling.
• A trap is an interrupt signaled by a predefined
event. When a trap event occurs, the SNMP
agent does not wait for the manager to poll.
Instead it immediately sends information to the
manager. Traps allow the agent to inform the
manager of unusual events while allowing the
manager to maintain control of polling. SNMP
traps are sent on UDP port 162. The manager
sends polls on port 161 and listens for traps on
port 162.
• The commands in Figure 1 display information
about network management applications. A
troubleshooter uses the information from these
commands to isolate problems at the
application layer that are related to the SNMP
and NTP protocols.
• Figure 2 lists commands which make
configuration changes that troubleshooters can
use to correct problems with network
management protocols at the application layer.

– The NMS periodically requests the status of each managed device (GetRequest) and each agent
responds with the status of its device (GetResponse).
– Making periodic requests is called polling. Polling reduces the burden on the agent because the
NMS decides when polls are needed, and the agent simply responds.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 51
 }Name resolution
• Domain Name Service (DNS) is the service that translates computer and server
names to IP addresses. These names are referred to as Fully-Qualified Domain
Names (FQDN).
• DNS Hierarchy
There are many DNS servers throughout the Internet. However, each DNS server
stores only a portion of the entire Internet namespace. A DNS hierarchy enables
DNS servers to find their neighbors and ask each other for information about a
specific host.
• A domain is a label in the DNS hierarchy. Each node in the DNS hierarchy
represents a domain. Domains under the top-level domains represent individual
organizations or entities. These domains can be further divided into subdomains
to ease administration of an organization's host computers. Domains, starting
with the top-level domains and branching out below, divide the total DNS name
space. The top-level domain names are closely controlled by the InterNIC, a
division of the Internet Assigned Numbers Authority (IANA) responsible for
assigning these names.

– Before DNS, network servers were identified using the IP addresses. However, this became very
cumbersome. Eventually, individuals started writing HOSTS files, which contained names of servers and IP
addresses assigned to them. This way, users would FTP or Telnet to a system by using their names instead
of the IP addresses. This worked well, and so the HOSTS file was placed on every system on the Internet.
– Because administrators of each system maintained the files independently, this created new problems. First,
if the text database changed, there was no way to update it automatically on every system. Essentially, the
response was to create a centralized HOSTS file, which would be the definitive HOSTS file on the Internet.
Routinely, administrators checked this central file for any changes and would update the HOSTS files on
their local systems when there were changes.
– This system had many problems. For example, with only one HOSTS file on the whole Internet, if that site
went down, nobody else knew what any of the DNS names were. Secondly, as more and more systems were
added, the HOSTS file started to get very big. Finally, the HOSTS names did not provide for any kind of
hierarchy. Therefore if somebody at one site wanted to have a computer named Admin, nobody else in the
whole world could have a computer named Admin.
– The answer to these problems was Domain Name Service (DNS). DNS allows computer systems to resolve
FQDN to IP addresses.
– Top-level domain names are part of most URLs. For example, “.com,” “.edu,” “.net,” “.gov,” and “.org” are
top-level domain names. These top-level domains contain the basis for the rest of the domain naming
structure. Individual organizations are granted second-level domain names within one or more of these top-
level domains. Because names have to be unique in a domain, they must be registered.
– When an organization wishes to acquire a second-level domain name, it must submit a request to the
Internet Network Information Center (InterNIC). If the domain name is available and the InterNIC does not
have a problem with the name, it is assigned to the organization in exchange for a small biannual fee. The
organization itself is responsible for assigning third-level and lower domains.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 52
}Name resolution
• How DNS is resolved
In Figure 1, the client makes a request to the
corporate DNS server. The DNS server checks
its cache to see if the query has already been
resolved. In this situation, the corporate DNS
server has no record of this query. Therefore,
the corporate DNS switches roles and now acts
as a client and issues an iterative query to the
local ISP.
• The ISP name server has no record of this
resolved request. The ISP server replies back
with a hint to query the root domain server.
• The DNS server issues an iterative query at the
top of the DNS hierarchy to the root level
server. After each query and response the
server goes down the DNS tree until it finally
finds the correct resolved name.
• Nslookup
The most effective command for testing and
resolving DNS issues is the nslookup
command.
• If the lookup request fails, nslookup prints an
error message. Figure 3 lists possible error
messages.
• DNS and Routers
A router can be configured to use DNS lookups
so that ping or traceroute commands can be
used with a hostname rather than an IP
address.
• Use the commands in Figure 4 to do so.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 53
}Dynamic Host Configuration Protocols (DHCP)
• Dynamic Host Configuration Protocol
(DHCP) is used to dynamically assign
IP addresses to hosts. Although it is
not a true TCP/IP application
program, it is important to cover it to
some detail.
• DHCP uses a client-server structure
to provide configuration parameters
to hosts. It consists of a protocol
that provides host-specific
configuration parameters from a
DHCP server (or collection of DHCP
servers) to a host and a mechanism
to allocate network addresses to a
host.
• The commands in Figure 1 display
information about the Dynamic Host
Configuration Protocol (DHCP)
application.
• A troubleshooter uses the
information from these commands to
isolate problems with DHCP.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 54
 TROUBLESHOOTING TCP/IP
APPLICATION LAYER PROBLEMS

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 55
 }Troubleshooting Telnet problems
• Troubleshooting Telnet Example
The second-level network engineer for a
company in Toronto would like to remotely
manage a router in Calgary. However, the
engineer is unable to establish a Telnet
connection to it from her office computer. This
is odd because Telnet to the router was
possible the day before.
• The computer has IP connectivity to a switch
named Toronto_SW and the switch is
connected to a router named Toronto. The
engineer also has console access to both
devices. Her division supports the
172.22.0.0/16 subnet.
• Therefore, the engineer consoles into
Toronto_SW to see if she can ping the Calgary
router.
• Toronto_SW can ping Calgary. Therefore, it
appears that the lower OSI layers between
these devices are working.
• Next, the engineer tries to Telnet from the
Toronto switch to the Calgary router, but this
attempt is unsuccessful. It is possible that
Telnet has been disabled, moved to a port
other than 23 on the Calgary router, or is being
blocked by an inbound access-list.

– The following section will provide common application layer problems and the suggested steps
required to solve these problems. The focus of this section is to develop an awareness of steps
required to logically solve problems.
– Many problems can stop a Telnet session from being established. The steps to troubleshoot
particular problems will change depending on the specific problem. However, a good
troubleshooter will be able to solve these problems by methodically eliminating potential issues.
– By consoling into the Toronto router and opening a Telnet session to Calgary, the possibility that
Telnet has been disabled, moved to another port, or is blocked by an inbound access-list has been
eliminated.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 56
 }Troubleshooting Telnet problems

– While telnetted into the Calgary router, signs of recent configuration changes are checked by using
the show logging and show clock commands. No configuration changes have been made on
Calgary for several days, so the engineer returns to the console session on Toronto.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 57
 }Troubleshooting Telnet problems

– The engineer looks for signs of recent configuration changes on Toronto with the show logging
and show clock commands. Although changes made to the running configuration cannot be
confirmed, the fact that someone else was in configuration mode on Toronto in the last few hours
can be confirmed.
– So far it is known that pings to Calgary from Toronto_SW are successful, but Telnet sessions are
not. Telnet sessions are possible from the Toronto router and possibly another administrator could
have made configuration changes on the Toronto router.
– Because of the facts, it is suspected that the problem is probably with an extended access list
filtering too much traffic. To confirm this suspicion, use the show access-lists command on the
Toronto router to review the current access lists configured.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 58
 }Troubleshooting Telnet problems

– The only extended access list configured is called Traffic. Notice that it explicitly permits ICMP,
FTP, WWW, and TFTP traffic. However, the implicit deny at the end of the list would block
Telnet traffic that came from Toronto_SW.
– To determine which interface on Toronto is being used to forward traffic to Calgary, use the show
ip route command. This reveals that traffic for Calgary is sent across the interface named
Serial0/0:0.
– Finally, verify that the access list named Traffic is applied to Serial0/0:0 of Toronto with the show
ip interface serial 0/0:0 command. To see how Traffic is configured, review the access list in the
running configuration.
– The issue is now isolated. The outbound access list named Traffic does not include a permit
statement for Telnet. All Telnet traffic from the LAN traffic connected to the Toronto switch is
being filtered. The remark statement for the access list Traffic states that it should support
outbound TCP Telnet connections.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 59
 }Troubleshooting Telnet problems

– The engineer corrects the extended access list named Traffic and adds a line to support Telnet
traffic from Toronto.
– Finally, verify the configuration change by consoling into Toronto_SW and Telnet to Calgary.
– The incomplete extended access list has been updated to support Telnet. The Application problem
has been resolved by correcting the transport layer problem and the baseline configuration has been
restored.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 60
}Troubleshooting HTTP problems
• Problems with HTTP connectivity can be hard to narrow down. Although Web
browsers are not the greatest utilities for detailed troubleshooting of the HTTP
protocol, they are nonetheless useful for determining whether clients on an
Internet may connect to a specific Web server. Even if a Web server responds
correctly to HTTP commands using the Telnet utility, this fact does not guarantee
that it will accomplish its goal of serving Web pages to the Internet public. For
this, the only choice is to connect to the Web server by using a popular Web
browser.
• When managing Web servers, it is a good idea to keep a variety of different Web
browsers on hand. All Web servers and Web pages should be tested with both
Netscape Navigator and Microsoft Internet Explorer.
• Be sure to try accessing the Web server from various hosts to eliminate
individual computer browser problems.
• The following commands make router configuration changes that troubleshooters
can use to correct problems with Web protocols at the application layer.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 61
}Troubleshooting e-mail problems
• Troubleshooting e-mail problems can be easy. However, sometimes there are
other factors that can affect users from properly retrieving or sending e-mail. A
mistyped setting can cause a lot of problems. Careful configuration is key to the
success of using an e-mail server.
• E-Mail Troubleshooting Example
In a fairly short period of time, a large number of network users call to report
that they cannot send email, but they can receive it. Remember, that the
network has separate servers for sending and receiving email. There is an SMTP
server that is used to send e-mail and a POP3 server is used to receive and save
e-mail.
• Since the users are receiving email, it is doubtful that the POP3 server is
malfunctioning. The problem of sending email could be isolated to the server
running the SMTP protocol.
• Testing the physical, data link, and network layers reveals no problems.
• To test the Transport layer, attempt to Telnet into the SMTP server through the
port number for the SMTP protocol (25). A hello message is not received from
the server. This indicates problems at either the transport or application layer.
• Verify the following:
– Is the router denying access to port 25?
– Is the e-mail client properly configured ?
– Is the address being used to Telnet the SMTP server?

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 62
}Troubleshooting FTP problems
• Generally, if a client has connectivity by way of the control connection
but cannot retrieve directory listings or transfer files, there is an issue
with opening the data connection. Try specifying passive mode because
this is permitted by most firewalls.
• Another common problem with FTP is being able to transfer small files
but not large files, with the transfer generally failing at the same place or
time in every file. Remember that the data connection (and the transfer)
will be closed if the control connection closes. This is because the control
connection is typically dormant during large file transfers. It is possible
for the connection to close in NAT/PAT environments in which there is a
timeout on TCP connections. Increasing the timeout on dormant TCP
connections may resolve this problem. If an FTP client is not properly
coded, this problem may occur.
• Because FTP file transfers generally create packets of maximum size, an
MTU mismatch problem will almost always cause file transfers to fail in a
single direction (gets may fail, but puts may work). A server located on a
LAN media that supports larger MTUs (such as Token Ring, which can
have an MTU of 4096 or larger) can be the cause of this problem.
Normally this problem is resolved automatically by fragmentation, but
misconfigurations or having the IP Don't Fragment option set in the IP
datagrams can prevent automatic resolution of these types of problems.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 63
 }Troubleshooting FTP problems

– Troubleshooting TFTP Example

The second-level network engineer for Orlando has console access to the distribution router named
Orlando and IP connectivity to all other devices in his division. The division supports the
172.21.0.0/16 subnet.
– The engineer knows from the base configuration information that there is at least a 100 MB
FastEthernet link between Orlando and Baltimore, so this would be a good source to use for
downloading the IOS image. He connects to the console port on Orlando to assess the situation.
– First, the available commands from within ROMMON mode are reviewed. Note that the
ROMMON prompt is on 14 and 15, which indicates that several commands have already been
issued.
– Next, he enters the ROMMON boot command to try to boot the router in case the Cisco IOS image
is not really missing. Entering the boot command did not work.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 64
 }Troubleshooting FTP problems

– The engineer attempts to reset the router to see if that will restore the IOS image. Entering the
reset command did not restore the image.
– He decides to look for the IOS image in the file system.
– Biff is correct. There is no IOS image in flash memory. At 5858 bytes, base.cfg is not big enough
to be an IOS image. The engineer learns from Biff that the image was erased while someone was
saving a backup configuration file.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 65
 }Troubleshooting FTP problems

– It is decided that TFTP should be used to recover the image. The engineer reviews the commands
available from ROMMON level.
– The engineer decides that tftpdnld is the command necessary to download the image from
Baltimore. Baltimore is running a TFTP server that is offering an image with the file name
flash:c1700-sv8y-mz.122-8.YL.bin.
– He enters the tftpdnld command. The attempt fails and the command output displays messages
that are symptoms of an issue with the TFTP application layer protocol.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 66
 }Troubleshooting FTP problems

– The engineer has isolated the TFTP issues to the tftpdnld ROMMON command needing the IP
address and mask for the local router, the default gateway for the local router, the IP address of the
TFTP server, and the name of the file to be transferred.
– A list of these values is shown below:
– Local IP address = 172.21.128.129
Local Mask = 255.255.255.128
Local Default Gateway = 172.21.128.130
TFTP Server = 172.22.128.129
File = c1700-sv8y-mz.122-8.YL.bin
– By reviewing the relevant information on http://www.cisco.com, the engineer realizes he needs to
enter variable_name=variable to set these variables.
– The engineer configures the TFTP variables on the Orlando router. This step is extremely case
sensitive. The variables have been configured, after he removes an extra space.
– The engineer now invokes the TFTP program. After configuring the parameters to support TFTP,
the TFTP download process seems to work. It appears that the application layer issue of missing
TFTP parameters has been resolved.
– The engineer now needs to boot the router using the new image.
– The IOS image has been restored. He finishes the task by restoring the baseline configuration files.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 67
 }Troubleshooting DNS problems
DNS name resolution can fail even when IP 3. Verify the name of the DNS server that should
connectivity works properly. To troubleshoot be used to help resolve the name. This can be
this problem, use one of the following methods found in different places on each operating
to determine if DNS is resolving the name of system. If unsure of how to find it, consult the
the destination: device manual. The following describes the
1. Ping the destination by name and look for an instructions for several common platforms:
error message indicating the name could not be – On a Cisco router, type show run and look for
resolved. the name-server.
– On Windows 9x and Windows Me, use
2. If working on a UNIX machine, use nslookup winipcfg.exe.
<fully-qualified domain name> to perform – On Windows XP, 2000, or NT, use ipconfig.exe.
a DNS lookup on the destination. If it is
successful, the address of the host should be – On a UNIX platform, type cat /etc/resolv.conf
at a command prompt.
displayed:
• unix% nslookup www.somedomain.com 4. Verify that the name server can be pinged
Server: localhost using its IP address. If the ping fails, then the
Address: 127.0.0.1 problem is at a lower layer.
• Non-authoritative answer: 5. Verify that names can be resolved within the
Name: www.somedomain.com local domain. For example, if a host is
Address: 10.1.1.1 host1.test.com, the names of other hosts, such
as host2.test.com, in the test.com domain
• If nslookup fails, the output should be similar to should resolve to an IP address.
the following: 6. Verify that one or more domain names outside
• unix% nslookup www.somedomain.com the local domain can be resolved. If names
Server: localhost from all domains except that of the destination
Address: 127.0.0.1 can be resolved, it is possible there is a
• *** localhost cannot find problem with the DNS for the destination host.
www.notvalid.com: Non-existent Contact the administrator of the destination
host/domain device.

– Cisco IOS: Use show run and look for the name-server
– Windows 9x,ME: Use winipcfg.exe
– Windows XP, 2000, NT: Use ipconfig.exe
– UNIX Plattaform: Use cat /etc/resolv.conf
– If names within the local domain or a large number of external domains cannot be resolved, contact
the DNS administrator, since there may be a problem with the local DNS (or the local host could
be using the wrong domain server).

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 68
}Summary

• By completing this module, students should have

gained an understanding of the operation of various
transport layer networking technologies on routers
and hosts. These technologies include:
– Transport Control Protocol
– User Datagram Protocol
– NetBIOS
– Network Address Translation
– Extended access lists
• Students should also have gained an appreciation of
the various tools and methodologies that can assist
with troubleshooting transport layer issues.

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 69
 }Q&A

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

© 2005, The USFQ – A Regional Cisco Networking Academy

CCNP4 v3 – Layer 3 Troubleshooting
1 - 70