CIS 188 CCNP TSHOOT

(Troubleshooting)
Ch. 8 Troubleshooting Converged
Networks Part 2
Rick Graziani
Cabrillo College
[email protected]
Fall 2014

Materials
Book:
Troubleshooting and Maintaining
Cisco IP Networks (TSHOOT)
Foundation Learning Guide:
Foundation learning for the CCNP
TSHOOT 642-832
By Amir Ranjbar
Book
ISBN-10: 1-58705-876-6
ISBN-13: 978-1-58705-876-9
eBook
ISBN-10: 1-58714-170-1
ISBN-13: 978-1-58714-170-6

Topics
Part 1
Troubleshooting Wireless Issues in a Converged Network
Part 2
Troubleshooting Unified Communications Issues in a Converged
Network

Common Unified Communications

Integration Issues

To have data and voice application traffic coexist in harmony, mechanisms

to differentiate types of traffic and to offer priority processing to voice traffic,
which is sensitive to delay, are needed to be deployed.

 These items result in challenging troubleshooting scenarios that increasingly

involve multiple components of the network:
Quality of Service:
Bandwidth, delay, jitter, packet loss, Network QoS Readiness, Trust
Boundaries, Switch QoS
High Availability:
STP/RSTP, HSRP/GLBP/VRRP
Security:
Traffic Segregation (Voice versus Data VLANs), Firewalling/Filtering
Provisioning and Management:
PoE, DHCP, TFTP, NTP, CDP, Trunking, VLANs
5

 An IP Phone is like other IP devices and has similar processes:

The following the is the list of the IP Phone boot process steps:
1. The IP phone powers on.
2. The phone performs a power-on self-test, or POST.
3. The phone boots.
4. The phone uses CDP to learn the voice VLAN.
5. The phone initializes the IP stack.
6. The IP phone sends DHCP broadcasts
7. The DHCP server returns information, including option 150 (TFTP
address).
8. The IP phone initializes, applying the IP configuration to the IP stack.
9. The IP phone requests a configuration file from the TFTP server defined
in option 150.

Voice VLANs
The VLAN architecture is very important and
knowing the voice and data VLANs is
crucial.
Knowing how voice and data traffic are
carried across switch ports help our
troubleshooting efforts.
Most Cisco IP Phones contain a three port
switch:
Connecting to the upstream port
Connection to PC (usually)
Internal VoIP data stream
The internal VoIP and external PC ports:
Access ports
Upstream data port:
Access port (single VLAN) or
802.1Q trunk (well, kind of)
10

Voice VLAN Configuration

switch

switch

Switch(config)# interface type mod/num

Switch(config-if)# switchport voice vlan {vlan-id | dot1p |
untagged | none}

The uplink port can be:

Access link
Trunk link
The IP Phone is considered a switch
so this port is considered an uplink
port.
Trunks to IP phones are automatically
negotiated by Dynamic Trunking
Protocol (DTP) and CDP.
Several option exist
11

switchport voice vlan none

Voice:
Untagged: Access VLAN
Data:
Untagged: Access VLAN

switchport voice vlan dot1p

Voice:
Tagged as VLAN 0

802.1Q trunk
CoS in 802.1p bits

Data:
Untagged: Native VLAN

switchport voice vlan untagged

Voice:
Untagged: Native VLAN
Data:
Untagged: Native VLAN

switchport voice vlan vvid

Voice:
Tagged as vvid

Recommended Option

802.1Q trunk
CoS in 802.1p bits

Data:
Untagged: Native VLAN
12

switchport voice vlan vvid

Voice:
Tagged as vvid

Recommended Option

802.1Q trunk
CoS in 802.1p bits

Data:
Untagged: Native VLAN

Switch(config)# interface type mod/num

Switch(config-if)# switchport voice vlan vlan-id
Instructs the Cisco IP phone to forward all voice traffic through the specified VLAN.
By default, the Cisco IP phone forwards the voice traffic with an 802.1Q priority of 5.
Creates a special 802.1Q trunk (more about this so called trunk later)
Negotiated by DTP and CDP (provisioning of the vvid)
CoS (Class of Service) in 802.1p bits (later)
vvid puts:
Voice packets on voice VLAN
Voice VLAN is configured.
Data packets in Native VLAN
VLAN 1 by default unless modified on the switch
Can configure the data VLAN to be a a VLAN other than Native or Voice
16

Configuring Voice VLAN Operation

Voice:
Tagged as voice VLAN 200

Recommended Option

802.1Q trunk
CoS in 802.1p bits

Data:
Untagged: Native VLAN
Tagged as VLAN 100

Switch(config)# interface FastEthernet0/24

Switch(config-if)# switchport voice vlan 200
Switch(config-if)# switchport access vlan 100

Portfast is automatically enabled with voice VLAN.

Switch# show run
interface FastEthernet0/24
switchport voice vlan 200
switchport access vlan 100
spanning-tree portfast
18

The Voice VLAN: Trunk or No Trunk?

http://cciepursuit.wordpress.com/2009/01/01/group-study-good-expla
nation-of-the-voice-vlan/

19

Example 1: Port Security and Voice VLAN Issues

The problem is that the IP phones will not boot and initialize.
They have no access to the IP network.
The problem occurs in multiple areas of the network, but not all.
The issue seems to be permanent, and not intermittent.
In those switches where the problem IP phones are connected,
it is not clear whether all IP phones have the same problem.

 The change logs show a recent change on VLAN Trunking Protocol (VTP)
domains and configuration.
We will start at the switch, with the show interface status command.
The errdisable state can have multiple causes:
Duplex mismatches
Late collisions
Spanning tree issues

SW1# show interfaces g0/21 status

Port
Name
Status
Vlan Duplex Speed
Gi0/21 Phone number 1
err-disabled 20
auto
auto
10/100/1000BaseTX

Type

 The output of the show port-security interface command shows

that the maximum allowed MAC addresses on the port is set to 1.
That setting is probably why the problem has occurred.
Some phones have PCs connected to them, and both the phone and the PC
send packets.
This means that two MAC addresses will be reported on the port, which is
beyond the maximum allowed.

SW1# show port-security interface g0/21

Port Security
: Enabled
Port Status
: Secure-shutdown
Violation Mode
: Shutdown
Aging Time
: 0 mins
Aging Type
: Absolute
SecureStatic Address Aging
: Disabled
Maximum MAC Addresses
: 1
Total MAC Addresses
: 1
Configured MAC Addresses
: 1
Sticky MAC Addresses
: 0
Last Source Address:vlan
: 0021.7098.30ab:20
Security Violation Count
: 1

 You are informed that this setting is not needed on IP phone switch ports.
Use the show running interface command to display the
configuration for the interfaces.
Port security allows a single static MAC address.
Notice there is only a data VLAN configured (hm?)

SW1# sh run int g0/21

Building configuration...
Current configuration : 200 bytes
!
Interface GigabitEthernet0/21
description Phone number 1
switchport access vlan 20
switchport mode access
switchport port-security
switchport port-security mac-address 000b.8572.1810
end

 To remove the port security configuration, use the no version of

all commands related to port security.
Before removing the erroneous commands, reset the interface
using the shutdown command.
Enter the no shutdown command afterwards.
Check the status of the interface and the status shows as
connected.
However, there is still a problem and the IP phones are down.
SW1# conf t
Enter configuration commands, one per line. End with CNTL/Z.
SW1(config)# int g0/21
SW1(config-if)# shutdown
SW1(config-if)# no switchport port-security
SW1(config-if)# no switchport port-security mac-address 000b.8572.1810
SW1(config-if)# no shutdown
SW1(config-if)# end
SW1# show interface g0/21 status
Port
Name
Status
Gi0/21 Phone number 1 connected
10/100/1000BaseTX

Vlan
20

Duplex
a-full

Speed
a-1000

Type

 A review of the running configuration shows that voice VLAN is not configured for the port.
A review of the configuration template for IP phone switch ports reveals that the interfaces are
missing the trust boundary settings and have no voice VLAN configuration
Configure one interface according to the configuration template for testing.
Set the voice VLAN using the switchport voice vlan 10 command
Set trust IP phone markings using the mls qos trust cos and mls qos trust
device ip-phone commands.

SW1(config)# int g0/21

SW1(config-if)# switchport voice vlan 10
SW1(config-if)# mls qos trust cos
SW1(config-if)# mls qos trust device cisco-phone
SW1(config-if)#

 Trust IP phone markings using:

mlsqostrustcos
mlsqostrustdeviceipphone
Uses CDP to detect whether or not a Cisco IP phone is attached to the port.
If CDP detects a Cisco IP phone, the interface applies the configured mls qos trust cos
command.
If CDP does not detect a Cisco IP phone, QoS ignores any configured non-default trust
state.

SW1(config)# int g0/21

SW1(config-if)# switchport voice vlan 10
SW1(config-if)# mls qos trust cos
SW1(config-if)# mls qos trust device cisco-phone
SW1(config-if)#

 show interface switchport shows the administrative status and the operational
status of the interface
We hear from the support team that the phone is now initializing and operational, so
our job here has been completed.
We now proceed with replicating the change to other affected interfaces, and we do
similar verifications for those ports.

SW1(config)# int g0/21

SW1(config-if)# switchport voice vlan 10
SW1(config-if)# mls qos trust cos
SW1(config-if)# mls qos trust device cisco-phone
SW1(config-if)#
SW1# show interfaces switchport g0/21
Name: Gi0/21
Switchport: Enabled
Administrative Mode: static access
Operational Mode: static access
Administrative Trunking Encapsulation: negotiate
Operational Trunking Encapsulation: native
Negotiation of Trunking: Off
Access Mode VLAN: 20 (VLAN0020)
Trunking Native Mode VLAN: 1 (default)
Administrative Native VLAN tagging: enabled
Voice VLAN: 10 (VLAN0010)
<output omitted>

Quality of Service

36

Overview

Previously an organization would use separate networks for:

Voice
Video
Data traffic
Now common practice to combine these into a single multi-service
network in which the varied traffic types coexist.
37

Overview

QoS Issues over non-QoS networks:

Stop-start and choppy Internet streaming video performance
Harsh audio when using Internet based IP phone
38

Quality of Service
defined

QoS refers to the ability of a network to provide improved service to

selected network traffic over various underlying technologies
including Frame Relay, ATM, Ethernet and IP-routed networks.
QoS features provide improved and more predictable network service by
offering the following services:
Dedicated bandwidth
Improved loss characteristics
Congestion management and avoidance
Traffic Shaping
Prioritization of traffic
39

QoS

QoS is only activated during times of congestion.

No need to go to the head of the line if you are already at the head of the
line.
40

Quality of Service defined

The goal is to move information from one point to another and the characteristics
that define the quality of this movement are:
Delay
Delay Variation (also known as Jitter)
Loss
41

Loss

Loss refers to the percentage of packets that fail to reach their

destination.
Loss can result from:
Errors in the network
Corrupted frames
Congested networks
42

Loss
TCP Header
UDP Header

Packet loss in a healthy network are actually deliberately dropped by

networking devices to avoid congestion.
TCP:
TCPs retransmission mechanism
UDP:
Some loss may be acceptable
As a guide, a highly available network should suffer less than 1% loss and for
voice traffic the loss should approach 0%.

43

Delay or latency

Delay or latency refers to the:

Time it takes for the packet to be sent across the wire (delay)
Time it takes for a packet to travel from the source to the destination
(latency)
Fixed delays
Serialization and encoding/decoding.
For example, a bit takes a fixed 100ns to exit a 10Mb Ethernet interface.
Variable delays
Congestion and time packets spend in network buffers waiting for access
to the media.
Router/Switch table lookups and decision making
As a design rule the total time it takes a voice packet to cross the network should
44
be less than 150ms (ms, millisecond = 1,000th of a second).

Delay variation or jitter

Delay variation or jitter is the difference in the delay times of

consecutive packets.

Jitter is the uneven arrival of packets.

A jitter buffer is used to smooth out arrival times.
Increases total network delay.
Audio streams are particularly susceptible to jitter.
In general, traffic requiring low latency also requires a minimum variation in
latency.
45

Delay variation or jitter

For example, consider that in a Voice over IP (VoIP) conversation:

Packet 1arrives.
Then, 20 ms later, packet 2 arrives.
After another 70 ms, packet 3 arrives.
And then 20 ms later packet 4 arrives.
This variation in arrival times (that is, variable delay) is not dropping packets, but
this jitter can be interpreted by the listener as dropped packets.
Jitter in excess of 30ms will result in degraded audio performance.
Excessive jitter in a streaming video environment will result in:
Jerky motion
Loss of video quality
Loss of video
46
As a design rule, voice networks cannot cope with more than 30ms of jitter.

Quality of Service requirements for video

Streaming video applications have more lenient QoS requirements

due to application buffering.

47

 Design rules of thumb for voice, video, and data traffic:

Voice:
No more than 150 ms of one-way delay
No more than 30 ms of jitter
No more than 1 percent packet loss
Video:
No more than 150 ms of one-way delay for
interactive voice applications (for example,
videoconferencing)
No more than 30 ms of jitter
No more than 1 percent packet loss
Data:
Applications have varying delay and loss
characteristics.
Therefore, data applications should be categorized
into predefined classes of traffic, where each
class is configured with specific delay and loss
characteristics.

48

Network availability

Highly available network uses:

Redundancy
Dynamic routing protocols
Hot Standby Routing Protocol (HSRP)
Spanning Tree Protocol (STP)
49

Provisioning

Bandwidth is not listed as an element of QoS.

Inadequate bandwidth inflates latency
It is not possible to meet QoS requirements if network LAN and WAN links
have insufficient bandwidth simply adding bandwidth, (also known as overprovisioning) will not solve the problem.
Over-provisioned network:
Good News: Less likely to be congested
Bad News: If it does become congested, the network may not perform
as well as a lower bandwidth network that makes use of QoS features.

50

Quality of Service mechanisms

Once the QoS requirements of the network have been defined, an appropriate service model must be
selected.
A service model is a general approach or a design philosophy for handling the competing streams
of traffic within a network.
There are three service models from which to chose;
Best-effort
Integrated Service Model
Differentiated Service Model

51

Best-Effort service
(single interface outbound queue)

(one packet at a time)

(relative time of arrival)

Best effort is a single service model in which an application sends data:

Whenever it must
In any quantity
Without requesting permission or first informing the network
No real QoS
For best-effort service, the network delivers data if it can, without any assurance of:
Reliability
Delay
Throughput

52

Best-Effort service
(single interface outbound queue)

(one packet at a time)

(relative time of arrival)

Cisco IOS QoS implements best-effort service is FIFO queuing.

FIFO is the default method of queuing for LAN and high speed WAN
interfaces on switches and routers.
Best-effort service is suitable:
General file transfers
E-mail
Web browsing

53

Best-Effort service

This would be like a fire truck having to wait in normal traffic lanes
with everyone else.
No priority.

54

Integrated services model (FYI)

Integrated service or IntServ
The application requests a
specific kind of service from
the network before it sends
data.
The Cisco IOS IntServ model makes
use of the IETF Resource
Reservation Protocol (RSVP)
Used by applications to signal
their QoS requirements to the
router.
Drawbacks
Not scalable
Require continuous signalling
from network devices
55

Integrated services model

This would be like a fire truck having to radio ahead to each

intersection before it left the firehouse.
Police at each intersection would contact each other to announce
that the fire truck was coming and give it priority.

56

Differentiated services

Differentiated services or DiffServ

Each network device handles packets on an individual basis.
Each device is configured with QoS policies to follow.
No advance reservations
QoS information is contained in each packet header.
Fire Truck analogy: Police at each intersection but dont know fire
truck is coming until they see the lights or hear the siren (packet
header QoS), then they give priority to the fire truck.

57

Differentiated services model

Differentiated Service or DiffServ architecture

Each packet is classified upon entry into the network.
These are represented using the Type of Service (ToS) field.
IP packet header:
IP precedence or
Differential Services Code Point (DSCP)
Either of these can be used to signify the QoS requirements of an IP
packet.

58

Differentiated services model

Once packets are classified at the edge by

Access layer switches
Border routers
Hosts/Applications
Unlike the IntServ model, DiffServ does not require network
applications be QoS aware.

59

DiffServ QoS at Layer 3: ToS and DSCP

ToS byte can be used to mark packets.

IP Precedence: 3 bits
DSCP: 6 bits
DiffServ keeps the existing IP ToS byte but uses it in a more scalable fashion.
Known as the DSCP (DiffServ Code Point)
Also known as the DS Byte
Terminology: ToS byte = DS byte
Same location but different interpretation
DSCP has been arranges so it is backward compatible with the IP
precedence bits.

60

ToS

Class
Selector

Drop
Precendence

ToS
IP DSCP value is the first 6 bits
IP Precedence value is the first 3 bits
The IP Precedence value is actually part of the IP DSCP value.
Therefore, both values cannot be set simultaneously.
DSCP supersedes IP Precedence.
A maximum of:
8 different IP precedence markings
64 different IP DSCP markings

61

DiffServ QoS at Layer 2: CoS

Data Link Layer:

Ethernet frames have no fields to signify its QoS requirements.
ISL or 802.1Q provides a 3 bit Class of Service (CoS) field.
Gives Layer 2 switches the ability to prioritize traffic.
802.1Q
3 bit Priority field indicates the frame CoS.
Untagged frames will receive a default CoS configured on the receiving
switch.
ISL
62
Similar but slightly different format.

CoS

The 3 bit CoS field present allows eight levels of priority.

0 lowest priority to 7 highest priority
Switches set a layer 2 CoS value for traffic based on their
ingress port
Hosts/applications can also set the CoS
Routers can translate the CoS value into an equivalent IP
Precedence or DSCP value

63

Trusting the CoS

If Edge device (IP phone or application) is capable of setting the CoS

bits then other devices must decide whether to trust the device or not.
The default action of switches:
Not to trust edge devices
Any frames that enter the switch have their CoS re-written to the lowest
priority of 0.
If the edge device can be trusted:
Default behaviour must be overridden
Access switch must be configured to simply switch the frame leaving
the CoS bits untouched.
64

Configuring CoS trust using the IOS

switch(config)# mls qos

Required on both the Catalyst 3550 and 6500.
The Catalyst 2950 has QoS enabled by default.

Depending on the switch model it may be necessary to first activate QoS using the command:

65

Configuring CoS trust using the IOS

The trust is configured on the switch port using the command:
switch(config-if)# mls qos trust cos
Any ISL or 802.1Q/P frames that enter the switch port will
now have its CoS passed, untouched, through the switch.
If an untagged frame arrives at the switch port, the
switch will assign a default CoS to the frame before
forwarding it. (How it will be treated within the switch.)
Default CoS = 0
Can be changed using the interface configuration
command:

switch(config-if)# mls qos cos default-cos

default-cos is a number between 0 and 7

66

Assigning CoS on
a per-port basis

switch(config-if)# mls qos trust cos

switch(config-if)# mls qos cos default-cos
If the incoming frame has a CoS, maintain the same CoS.
If the incoming frame has no CoS (0), apply the default CoS.

67

Re-writing the CoS

Switch(config-if)# mls qos cos override

switch(config-if)# mls qos cos default-cos

May be desirable not to trust any CoS value that may be present
in frames sourced from an edge device.
Override parameter - ignores any existing CoS value
Apply the default value. (Default = 0)

68

Traffic marking
Layer 2

Layer 3
The decision of whether to mark traffic at layers 2 or 3 or both is not trivial
and should be made after consideration of the following points:
Layer 2 marking of frames can be performed for non IP traffic.
Layer 2 marking of frames is the only QoS option available for
switches that are not IP aware
Layer 3 marking will carry the QoS information end-to-end
Older IP equipment may not understand DSCP
69

Mapping Layer 2 and Layer 3

When an IP packet is marked with DSCP, for example, and it needs to

traverse a series of Layer 2 switches or 802.1Q Trunks.
How will it be queued in these Layer 2 devices?
To accomplish this, there is a mapping that takes place between the Layer 3
mapping field (TOS) and the Layer 2 CoS fields.
I will show how this works soon.
Mapping is vendor specific.
On Cisco devices, this is taken care of for you through a mapping
process.

70

Putting it all together

71

 Routine
Default class, Class 0
Offers only best-effort delivery

72

 Classes 1 through 4 are called Assured Forwarding (AF) levels.

Packets with lower class numbers are more likely to be dropped.
Packet with AF Class 4 is more likely to be delivered than a packet with
an AF Class of 3
73

 Class 5 is known as expedited forwarding (EF).

Packets with this class number are given premium service and are least
likely to be dropped.
Used for time-critical traffic such as voice.
74

 Class 6 and 7 are used for operations necessary to keep the network functioning properly.
Used by routers and switches for packets containing STP, routing protocols, etc.

75

 Each DSCP class has three levels of drop

precedence
1: Low (least likely to be dropped)
2: Medium
3: High (most likely to be dropped)
NOTE: Packets with a higher drop precedence have
the potential for being dropped before those with a
lower value.
This gives DSCP a finer granularity to the decision of
what packet to drop when necessary.
76

 DSCP value can be given a codepoint name with

the class selector providing:
two letters (AF or EF)
followed by the Class Selector number
followed by the Drop Precedence number
Class AF Level 2 with a drop precedence of 1 (Low)
is written and referred to as:
AF21
The DSCP is commonly give a decimal value.
For AF21 the decimal value is:
18 (the decimal equivalent of the six binary bits)

77

 These values are not as random as they may seem.

Class selector: First three bits
Drop Precedence: Next three bits, 3rd bit is always 0
1: 010 (1, if you remove the last bit)
2: 100 (2 , if you remove the last bit)
3: 110 (3 , if you remove the last bit)
And remember, the decimal value is just the bits in
decimal.
78

 When a frame is marked with DSCP, and it needs to traverse a series of Layer 2
switches or 802.1Q Trunks how will it be queued in these Layer 2 devices?
Mapping that takes place between the Layer 3 DSCP (or ToS) to the Layer 2 CoS
fields.
The CoS value is the value of the 3 ToS bits or the first 3 bits of the DSCP (same
values) with the last three bits of 000.

79

Troubleshooting Example: Invalid Marking of

VoIP Packets

Users from one building complain about their experience with voice calls and
claim that it is choppy, they lose connections frequently, and at some point
voice conversations are intermittent.
The problem is worse for branch-to-branch calls.
Our task is to determine whether the network is to blame, and if it is, locate
where the problem is occurring.

80

 While gathering information, we need to ask the following questions:

How often do you observe the reported symptoms?
Is there a particular time of the day in which they commonly occur?
Is the perceived quality the same when calling internal extension numbers and as
it is when calling outside numbers?
How often are you unable to obtain a dial tone? For how long does this condition
remain?
Which locations of the network are experiencing the problem (building/branch)?
Are the problematic devices connected to the same wiring closet?
Now we can reduce the scope of our search.
We suspect a certain wiring closet where the devices in our diagram are located.
All symptoms (intermittent connections, choppy voice, disconnections) seem to be
related to QoS.
The latency numbers are definitely telling us that a QoS issue exists.

81

One of the possible issues is high CPU utilization at the switch level.
show processes CPU shows that the 5-minute averages go to around
25 percent utilization.
Our baseline at peak hours is 34% so this is okay.

SW1# show processes cpu

CPU utilization for five seconds: 99%/22%; one minute: 58%, five minutes: 25%
PID Runtime(ms) Invoked uSecs
5Sec 1Min 5Min TTY
Process
1
0
15
0 0.00% 0.00% 0.00%
0
Chunk Manager
2
9
1131
7 0.00% 0.00% 0.00%
0
Load Meter
3
0
1
0 0.00% 0.00% 0.00%
0
CEF RP IPC <output
omitted>

 The next step is a port-by-port analysis.

The interface Gi0/11 has a phone attached to it.
show interface is used inspect its bandwidth utilization averages
Around 15 percent of the total interface bandwidth.
The other reported numbers on this output dont look bad either

SW1# show interfaces gi0/11

5 minute input rate 729000 bits/sec, 847 packets/sec
5 minute output rate 14150000 bits/sec, 1129 packets/sec
104911 packets input, 13035040 bytes, 0 no buffer
Received 22020 broadcasts (110 multicasts)
<output omitted>

 Next things we need to investigate are the trunks, which aggregate traffic uplink to
the distribution layer.
show interface2 gig 0/13
The utilization is naturally higher because it is a trunk link
Consistent to the numbers recorded in the baseline

SW1# show interfaces gi0/13

5 minute input rate 2478000 bits/sec, 1642packets/sec
5 minute output rate 2194000 bits/sec, 690packets/sec
917323 packets input, 172833916 bytes, 0 no buffer
Received 913155 broadcasts (26001 multicasts)
<output omitted>

 Now shift our focus to QoS.

We should check and see if the QoS classes, and their corresponding markings, are
being enforced in the network.
From the documentation we learn that IP phones represent the trust boundary, and that
the DSCP markings are being used throughout the network.
Phones are allowed to tag their own packets with high priorities, in this instance DSCP
value EF (Expedited Forwarding).
We should check and see if our switch ports are maintaining those tags, and not resetting
them.
show mls qos interface on gig 0/11 reveals that the port:
Is trusted
DSCP values are being maintained and not reset.
Can conclude that the access switch is configured properly.
SW1# show mls qos int g0/11
GigabitEthernet0/11
trust state: trust dscp
trust mode: trust dscp
trust enabled flag: ena
COS override: dis
Default COS: 0
DSCP Mutation Map: Default DSCP Mutation Map
Trust device: none
qos mode: port-based

The distribution layer in this network is collapsed at the branch router level. Verify
QoS settings on R1.
The show policy-map interface command reveals that policy Reclassify
Is Applied to Fa0/0 inbound.

R1# show policy-map interface

FastEthernet0/0
Service-policy input: reclassify
Class-map: signaling (match-any)
0 packets, 0 bytes
5 minute offered rate 0 bps, drop rate 0 bps
Match: protocol h323
0 packets, 0 bytes
5 minute rate 0 bps
Match: protocol sip
0 packets, 0 bytes
5 minute rate 0 bps
Match: protocol mgcp
0 packets, 0 bytes
5 minute rate 0 bps
Class-map: voice (match-all)
0 packets, 0 bytes
5 minute offered rate 0 bps, drop rate 0 bps
Match: protocol rtp audio
QoS Set
dscp af31
Packets marked 0
<output omitted>

Policy called reclassify is attached to the router fa0/0 interface.

In case we want to reclassify and re-mark packets coming into this interface.
This is done because this device is the WAN edge device, and the service provider may
require a different marking to maintain QoS policies in their network.
VOICE is being reclassified and tagged with the DSCP value AF31.
Voice traffic is typically classified with DSCP value EF, the highest priority.
Looks like the voice traffic class is being reclassified into a lower priority.
R1# show policy-map interface
FastEthernet0/0
Service-policy input: reclassify
Class-map: signaling (match-any)
0 packets, 0 bytes
5 minute offered rate 0 bps, drop rate 0 bps
Match: protocol h323
0 packets, 0 bytes
5 minute rate 0 bps
Match: protocol sip
0 packets, 0 bytes
5 minute rate 0 bps
Match: protocol mgcp
0 packets, 0 bytes
5 minute rate 0 bps
Class-map: voice (match-all)
0 packets, 0 bytes
5 minute offered rate 0 bps, drop rate 0 bps
Match: protocol rtp audio
QoS Set
dscp af31
Packets marked 0
<output omitted>

QoS policies such as bandwidth reservation, priority queuing, and preferred path
selection, are not being applied.
Voice traffic is suffering because of a voice reclassification mistake.
Once this error is fixed, the VOICE problems are solved. (Beyond CCNP)
Router(config)#dialpeervoice1voip
Router(configdialpeer)#ipqosdscpef
R1# show policy-map interface
FastEthernet0/0
Service-policy input: reclassify
Class-map: signaling (match-any)
0 packets, 0 bytes
5 minute offered rate 0 bps, drop rate 0 bps
Match: protocol h323
0 packets, 0 bytes
5 minute rate 0 bps
Match: protocol sip
0 packets, 0 bytes
5 minute rate 0 bps
Match: protocol mgcp
0 packets, 0 bytes
5 minute rate 0 bps
Class-map: voice (match-all)
0 packets, 0 bytes
5 minute offered rate 0 bps, drop rate 0 bps
Match: protocol rtp audio
QoS Set
dscp ef
Packets marked 0
<output omitted>

Example: IP Multicast Configuration Error

This network simulates an IGMP network, with R1 acting as an IGMP client,
similar to a PC running a video application and joining multicast groups.
R2 acts as the first-hop router, listening to IGMP join and leave transactions.
R3 acts as the video server, pushing multicast traffic downstream. The video
server is simulated by the loopback interface on R3.
R2 and R3 are preconfigured to communicate multicast group information
through Protocol Independent Multicast (PIM).
R1 and R2 are preconfigured to use IGMP to allow R1 to join multicast
groups.

Video Troubleshooting Example 2 Cont.

The problem is that users in the R1 LAN are not able to watch
the video stream.
They are able to connect to the server and request the video,
but the video stream is not reaching them after that.
The application team has verified that the software is installed
correctly and the server is configured properly, and they suspect
the network is to blame.
The video application is the only one that is not working so IP
reachability and routing issues are not likely the problem.

Video Troubleshooting Example 2 Cont.

This is a multicast issue and end devices must join a multicast
group before they can receive traffic directed to that group.
On R2, use the show ip igmp groups command to see the
multicast groups the hosts in this LAN have joined.
R1 is not joining any group. The group in the example output is
on the S0/0/0 interface, while R1 is on the LAN interface Fa0/0.
R2# show ip igmp group
IGMP Connected Group Membership
Group Address Interface
Uptime
224.0.1.40
Serial0/0/0 00:08:48

Expires
Stopped

Last Reporter Group Accounted

10.23.23.2

Video Troubleshooting Example 2 Cont.

The show ip igmp membership command, which shows all members
of all groups, does not list the IP address of R1 (10.12.12.1) anywhere.
R2# show ip igmp membership
Flags: A - aggregate, T - tracked
L - Local, S - static, V - virtual, R - Reported through v3
I - v3lite, U - Urd, M - SSM (S,G) channel
1,2,3 - The version of IGMP the group is in
Channel/Group-Flags:
/ - Filtering entry (Exclude mode (S,G), Include mode (*,G)
Reporter:
<mac-or-ip-address> - last reporter if group is not explicitly tracked
<n>/<m> - <n> reporter in include mode, <m> reporter in exclude
Channel/Group
*.224.0.1.40

Reporter
10.23.23.2

Uptime
00:09:24

Exp
stop

Flags
2LA

Interface
Se0/0/0

Video Troubleshooting Example 2 Cont.

Activate debug ip igmp on R2.
From R1 to simulate joining a group by entering the
command ip igmp joingroup.
The debug output on R2 shows no activity.

R2# debug ip igmp

IGMP debugging is on
R2#

R1# config t
Enter configuration commands, one per line. End with CNTL/Z.
R1(config)# interface fa0/0
R1(config-if)# ip igmp join-group 224.8.8.8
R1(config-if)#

Video Troubleshooting Example 2 Cont.

On R2, the only interface where IGMP Is enabled is S0/0/0. IGMP is not
enabled on R2s Fa0/0 interface so R1 could not join the multicast group.
R2# show ip igmp interface
Serial0/0/0 is up, line protocol is up
Internet address is 10.23.23.2/24
IGMP is enabled on interface
Current IGMP host version is 2
Current IGMP router version is 2
IGMP query interval is 60 seconds
IGMP querier timeout is 120 seconds
IGMP max query response time is 10 seconds
Last member query count is 2
Last member query response interval is 1000 ms
Inbound IGMP access group is not set
IGMP activity: 1 joins, 0 leaves
Multicast routing is enabled on interface
Multicast TTL threshold is 0
IGMP querying router is 0.0.0.0 (this system)
Multicast groups joined by this system (number of users):
224.0.1.40(1)

Video Troubleshooting Example 2 Cont.

Configure IGMP on router R2s Fa0/0 interface by enabling PIM on this interface. The
debug output shows R2 sending IGMP Version 2 query and receiving a report from R1
(10.12.12.1 joining the multicast group 224.8.8.8).
R1# config t
Enter configuration commands, one per line. End with CNTL/Z.
R1(config)# interface fa0/0
R1(config-if)# ip pim sparse-dense-mode
R1(config-if)#
R2#
IGMP(0): Send v2 init Query on FastEthernet0/0
%PIM-5-DRCHG: Dr change from neighbor 0.0.0.0 to 10.12.12.2 on interface
FastEthernet0/0
IGMP(0): Received v2 Report on FastEthernet0/0 from 10.12.12.1 for 224.8.8.8
IGMP(0): Received Group record for group 224.8.8.8, mode 2 from 10.12.12.1 for
0
sources
IGMP(0): WAVL Insert group: 224.8.8.8 interface: FastEthernet0/0Successful
IGMP(0): Switching to EXCLUDE mode for 224.8.8.8 on FastEthernet0/0
IGMP(0): Updating EXLUDE group timer for 224.8.8.8
IGMP(0): MRT Add/Update FastEthernet0/0 for (*,224.8.8.8) by 0

Video Troubleshooting Example 2 Cont.

IGMP Is now enabled on both the R2 S0/0/0 and Fa0/0 Interfaces.

R2# show ip igmp interface

Serial0/0/0 is up, line protocol is up
Internet address is 10.23.23.2/24
IGMP is enabled on interface
Current IGMP host version is 2
Current IGMP router version is 2
IGMP query interval is 60 seconds
<output omitted>
FastEthernet0/0 is up
Internet address is 10.12.12.2/24
IGMP is enabled on interface
Current IGMP host version is 2
Current IGMP router version is 2
IGMP query interval is 60 seconds
<output omitted>

Video Troubleshooting Example 2 Cont.

Multicast group 224.8.8.8 is now known on Fa0/0 with last reporter as R1 (10.12.12.1).
A Ping to the multicast address 224.8.8.8 from R3 receives a reply from R1.
R2# show ip igmp group
IGMP Connected Group Membership
Group
Interface
Uptime Expires
Last Reporter
Address
224.8.8.8 FastEthernet0/0 00:08:48 00:02:51 10.12.12.1
224.0.1.40 Serial0/0/0
00:19:43 stopped
10.23.23.2

Group Accounted

R3# ping 224.8.8.8

Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 224.8.8.8, timeout is 2 seconds:
Reply to request 0 from 10.12.12.1, 1 mss

Troubleshooting
Video Issues in a
Converged
Network

Section Overview
This section addresses the challenge of troubleshooting the network
infrastructure supporting video and rich media traffic.
Streaming and broadcast types of video applications include:
Digital signage
Video on demand (VoD)
Video surveillance.
Video applications have different characteristics in terms of:
Interactivity
Network traffic volume
Audience
Requirements for underlying network infrastructure and services.

Common Video-Integration Issues

Several components and infrastructure services are shared between video
and voice applications.
Sometimes the endpoints are the same, or at least integrated and some of
the critical protocols, such as SIP, are also the same.
SIP is a signaling protocol that is used to initiate, manage, and terminate
voice calls but also video sessions.
The end user experiences an integrated service.
Devices supporting video services in a campus are shown here.

Common Video-Integration Issues

Both Video and voice applications need end-to-end QoS.
Video is much more bandwidth intensive and very bursty.
A high-definition stream can require more than 20-Mbps
bandwidth for delivery over the network.
Video packet sizes are much larger.
Each type of video application has unique requirements and
characteristics.
The table shows the QoS requirements for some of the main
video applications.
Metric

Video
Collaboration

Cisco
TelePresence

Video
Surveillance

Latency

200 ms

150 ms

500 ms

Jitter

10 ms

10 ms

10 ms

Loss

0.05%

0.05%

0.5%

Common Video-Integration Issues Cont.

Video applications require high availability and millisecond-level network
service recovery.
Video traffic cannot accept unpredictable or large network recovery
timeouts.
Convergence targets will be higher, and packet loss due to network outage
must be minimal.
Redundancy design, convergence of routing protocols and spanning tree
are extremely critical.
Building a multicast-aware network is another important consideration.
Security in a video-enabled network, similar to voice deployments, might
need to permit protocols such as:

SIP
H.323
SCCP (Skinny)
RTP
RTCP
Possibly others

Multicast Operation
Multicast traffic is used to send the same data packets to multiple receivers
efficiently. If unicast were used, the transmitter would send one copy for each
receiver.

Multicast Operation Cont.

The sender sends only one copy of a single data packet addressed to a
group of receivers.
Multicast groups IP addresses that use the Class D address space.
Class D addresses are denoted by the high-order 4 bits of the address set
to 1110. This results in the range of addresses 224.0.0.0 through
239.255.255.255.
Downstream multicast routers replicate and forward the data packet to all
those branches where subscribers exist.
Receivers express their interest in multicast traffic by registering at their
first-hop router.
This model and resulting protocols saves reduces resource utilization on
routers and switches and improves QoS and the user experience.
There are two main protocols involved:

Protocol Independent Multicast (PIM) routers advertise multicast receivers

Internet Group Management Protocol (IGMP) receivers subscribe to and
leave groups

Multicast Operation Cont.

The figure illustrates a multicast client joining a multicast group using IGMP.
Members joining a multicast group send an unsolicited report indicating their
interest.
This action reduces join latency for the end system joining if no other members
are present.
Once the Membership Report is received by the router, it advertises to the rest
of the network.
Multicast sources will forward traffic directed to the group to this router.

Multicast Operation Cont.

The multicast group remains active and is advertised by
the router as long as there are members in the group
within that network segment.
As long as there is at least one member, the group will
remain active.

Multicast Operation Cont.

When a user terminates a multicast-based application the application
sends a leave message to the router.
The router then sends a query, just to verify whether there are still
members of the group in the segment.
If a device replies, the group remains active and the router advertises it.
If no reports are received, the router stops advertising the group to the
rest of the network.

Troubleshooting Video Integration Issues

Common video-integration issues include the following:

Excessive bandwidth utilization

Poor quality (lack of QoS)
Security issues (filtering of key protocols, and stateful requirements)
Multicast issues

QoS is a common problem due to the bursty nature of video traffic

Video traffic tends to monopolize the available bandwidth but is also delaysensitive.
Network security can interfere with video traffic. Firewalls, ACLs, and other
security controls can get in the way of protocols such as RTP, RTCP, SIP,
H.323, and others.
Multicast configuration, if enabled in the network, is always a source of
potential issues.

Common IGMP problems are related to group filtering, where routers might not
accept join request from certain multicast group addresses.
Another potential multicast issue is related to differences in IGMP versions
between the router and the hosts sending multicast traffic.

Example 2: IP Multicast Configuration Error

This network simulates an IGMP network, with R1 acting as an IGMP client,
similar to a PC running a video application and joining multicast groups.
R2 acts as the first-hop router, listening to IGMP join and leave transactions.
R3 acts as the video server, pushing multicast traffic downstream. The video
server is simulated by the loopback interface on R3.
R2 and R3 are preconfigured to communicate multicast group information
through Protocol Independent Multicast (PIM).
R1 and R2 are preconfigured to use IGMP to allow R1 to join multicast
groups.

Video Troubleshooting Example 2 Cont.

The problem is that users in the R1 LAN are not able to watch
the video stream.
They are able to connect to the server and request the video,
but the video stream is not reaching them after that.
The application team has verified that the software is installed
correctly and the server is configured properly, and they suspect
the network is to blame.
The video application is the only one that is not working so IP
reachability and routing issues are not likely the problem.

Video Troubleshooting Example 2 Cont.

This is a multicast issue and end devices must join a multicast
group before they can receive traffic directed to that group.
On R2, use the show ip igmp groups command to see the
multicast groups the hosts in this LAN have joined.
R1 is not joining any group. The group in the example output is
on the S0/0/0 interface, while R1 is on the LAN interface Fa0/0.
R2# show ip igmp group
IGMP Connected Group Membership
Group Address Interface
Uptime
224.0.1.40
Serial0/0/0 00:08:48

Expires
Stopped

Last Reporter Group Accounted

10.23.23.2

Video Troubleshooting Example 2 Cont.

The show ip igmp membership command, which shows all members
of all groups, does not list the IP address of R1 (10.12.12.1) anywhere.
R2# show ip igmp membership
Flags: A - aggregate, T - tracked
L - Local, S - static, V - virtual, R - Reported through v3
I - v3lite, U - Urd, M - SSM (S,G) channel
1,2,3 - The version of IGMP the group is in
Channel/Group-Flags:
/ - Filtering entry (Exclude mode (S,G), Include mode (*,G)
Reporter:
<mac-or-ip-address> - last reporter if group is not explicitly tracked
<n>/<m> - <n> reporter in include mode, <m> reporter in exclude
Channel/Group
*.224.0.1.40

Reporter
10.23.23.2

Uptime
00:09:24

Exp
stop

Flags
2LA

Interface
Se0/0/0

Video Troubleshooting Example 2 Cont.

Activate debug ip igmp on R2.
From R1 to simulate joining a group by entering the
command ip igmp joingroup.
The debug output on R2 shows no activity.

R2# debug ip igmp

IGMP debugging is on
R2#

R1# config t
Enter configuration commands, one per line. End with CNTL/Z.
R1(config)# interface fa0/0
R1(config-if)# ip igmp join-group 224.8.8.8
R1(config-if)#

Video Troubleshooting Example 2 Cont.

On R2, the only interface where IGMP Is enabled is S0/0/0. IGMP is not
enabled on R2s Fa0/0 interface so R1 could not join the multicast group.
R2# show ip igmp interface
Serial0/0/0 is up, line protocol is up
Internet address is 10.23.23.2/24
IGMP is enabled on interface
Current IGMP host version is 2
Current IGMP router version is 2
IGMP query interval is 60 seconds
IGMP querier timeout is 120 seconds
IGMP max query response time is 10 seconds
Last member query count is 2
Last member query response interval is 1000 ms
Inbound IGMP access group is not set
IGMP activity: 1 joins, 0 leaves
Multicast routing is enabled on interface
Multicast TTL threshold is 0
IGMP querying router is 0.0.0.0 (this system)
Multicast groups joined by this system (number of users):
224.0.1.40(1)

Video Troubleshooting Example 2 Cont.

Configure IGMP on router R2s Fa0/0 interface by enabling PIM on this interface. The
debug output shows R2 sending IGMP Version 2 query and receiving a report from R1
(10.12.12.1 joining the multicast group 224.8.8.8).
R1# config t
Enter configuration commands, one per line. End with CNTL/Z.
R1(config)# interface fa0/0
R1(config-if)# ip pim sparse-dense-mode
R1(config-if)#
R2#
IGMP(0): Send v2 init Query on FastEthernet0/0
%PIM-5-DRCHG: Dr change from neighbor 0.0.0.0 to 10.12.12.2 on interface
FastEthernet0/0
IGMP(0): Received v2 Report on FastEthernet0/0 from 10.12.12.1 for 224.8.8.8
IGMP(0): Received Group record for group 224.8.8.8, mode 2 from 10.12.12.1 for
0
sources
IGMP(0): WAVL Insert group: 224.8.8.8 interface: FastEthernet0/0Successful
IGMP(0): Switching to EXCLUDE mode for 224.8.8.8 on FastEthernet0/0
IGMP(0): Updating EXLUDE group timer for 224.8.8.8
IGMP(0): MRT Add/Update FastEthernet0/0 for (*,224.8.8.8) by 0

Video Troubleshooting Example 2 Cont.

IGMP Is now enabled on both the R2 S0/0/0 and Fa0/0 Interfaces.

R2# show ip igmp interface

Serial0/0/0 is up, line protocol is up
Internet address is 10.23.23.2/24
IGMP is enabled on interface
Current IGMP host version is 2
Current IGMP router version is 2
IGMP query interval is 60 seconds
<output omitted>
FastEthernet0/0 is up
Internet address is 10.12.12.2/24
IGMP is enabled on interface
Current IGMP host version is 2
Current IGMP router version is 2
IGMP query interval is 60 seconds
<output omitted>

Video Troubleshooting Example 2 Cont.

Multicast group 224.8.8.8 is now known on Fa0/0 with last reporter as R1 (10.12.12.1).
A Ping to the multicast address 224.8.8.8 from R3 receives a reply from R1.
R2# show ip igmp group
IGMP Connected Group Membership
Group
Interface
Uptime Expires
Last Reporter
Address
224.8.8.8 FastEthernet0/0 00:08:48 00:02:51 10.12.12.1
224.0.1.40 Serial0/0/0
00:19:43 stopped
10.23.23.2

Group Accounted

R3# ping 224.8.8.8

Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 224.8.8.8, timeout is 2 seconds:
Reply to request 0 from 10.12.12.1, 1 mss

CIS 188 CCNP TSHOOT

(Troubleshooting)
Ch. 8 Troubleshooting Converged
Networks Part 2
Rick Graziani
Cabrillo College
[email protected]