Cisco uBR10012 Universal Broadband Router Troubleshooting Guide
Cisco uBR10012 Universal Broadband Router Troubleshooting Guide
Cisco uBR10012 Universal Broadband Router Troubleshooting Guide
Corporate Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: 408 526-4000
800 553-NETS (6387)
Fax: 408 526-4100
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT
SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE
OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.
The following information is for FCC compliance of Class A devices: This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant
to part 15 of the FCC rules. These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial
environment. This equipment generates, uses, and can radiate radio-frequency energy and, if not installed and used in accordance with the instruction manual, may cause
harmful interference to radio communications. Operation of this equipment in a residential area is likely to cause harmful interference, in which case users will be required
to correct the interference at their own expense.
The following information is for FCC compliance of Class B devices: The equipment described in this manual generates and may radiate radio-frequency energy. If it is not
installed in accordance with Cisco’s installation instructions, it may cause interference with radio and television reception. This equipment has been tested and found to
comply with the limits for a Class B digital device in accordance with the specifications in part 15 of the FCC rules. These specifications are designed to provide reasonable
protection against such interference in a residential installation. However, there is no guarantee that interference will not occur in a particular installation.
Modifying the equipment without Cisco’s written authorization may result in the equipment no longer complying with FCC requirements for Class A or Class B digital
devices. In that event, your right to use the equipment may be limited by FCC regulations, and you may be required to correct any interference to radio or television
communications at your own expense.
You can determine whether your equipment is causing interference by turning it off. If the interference stops, it was probably caused by the Cisco equipment or one of its
peripheral devices. If the equipment causes interference to radio or television reception, try to correct the interference by using one or more of the following measures:
• Move the equipment to one side or the other of the television or radio.
• Plug the equipment into an outlet that is on a different circuit from the television or radio. (That is, make certain the equipment and the television or radio are on circuits
controlled by different circuit breakers or fuses.)
Modifications to this product not authorized by Cisco Systems, Inc. could void the FCC approval and negate your authority to operate the product.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public
domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH
ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT
LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF
DEALING, USAGE, OR TRADE PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING,
WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO
OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
CCVP, the Cisco Logo, and the Cisco Square Bridge logo are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn is a service mark of Cisco Systems,
Inc.; and Access Registrar, Aironet, BPX, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press,
Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Enterprise/Solver, EtherChannel, EtherFast, EtherSwitch, Fast Step, Follow Me Browsing,
FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, IP/TV, iQ Expertise, the iQ logo, iQ Net Readiness Scorecard, iQuick Study, LightStream, Linksys,
MeetingPlace, MGX, Networking Academy, Network Registrar, Packet, PIX, ProConnect, RateMUX, ScriptShare, SlideCast, SMARTnet, StackWise, The Fastest Way to Increase
Your Internet Quotient, and TransPath are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.
All other trademarks mentioned in this document or Website are the property of their respective owners. The use of the word partner does not imply a partnership relationship
between Cisco and any other company. (0704R)
Preface vii
Purpose vii
Audience vii
Obtaining Documentation ix
Cisco.com ix
Ordering Documentation ix
Documentation Feedback ix
Troubleshooting the OC-12 Dynamic Packet Transport Spatial Reuse Protocol WAN Card 4-14
INDEX
This guide documents processes and procedures for user level hardware troubleshooting on the
Cisco uBR10012 universal broadband router. For complete configuration instructions, please refer to the
Cisco uBR10012 Universal Broadband Router Software Configuration Guide and the documents listed
in the “Related Documentation” section on page viii.
• Purpose, page vii
• Audience, page vii
• Document Organization, page viii
• Related Documentation, page viii
• Obtaining Documentation, page ix
• Documentation Feedback, page ix
• Obtaining Technical Assistance, page x
• Obtaining Additional Publications and Information, page xi
Purpose
The Cisco uBR10012 router provides data and Voice over IP (VoIP) services to cable modems (CMs)
and customer premises equipment (CPE) devices over a cable TV (CATV) network, supplying
high-speed Internet and voice connectivity over the coaxial cable that provides TV and other signals.
Many of the Cisco uBR10012 modules are available in redundant configurations, so that the failure of
one module does not affect systems operations. This guide provides troubleshooting steps for a failed
component that you can take before system failure occurs and before intervention from higher level
support agencies becomes necessary.
Audience
To benefit from this guide, you must be experienced using Cisco IOS and have some responsibility for
installing, configuring, or operating the Cisco uBR10012 router. Knowledge of basic cable data network
operations and of the Data-Over-Cable Service Interface Specifications (DOCSIS), which define the
transmission of data and other services over a coaxial cable TV network.
Document Organization
The sections of this guide are as follows:
Chapter Description
Chapter 1, “Basic Troubleshooting Tasks and Basic procedures that users should perform before undertaking a detailed
Startup Issues” troubleshooting analysis of the Cisco uBR10012 router or logging a case
with the Cisco Technical Assistance Center (TAC).
Chapter 2, “PEM Faults and Fan Assembly Methods for troubleshooting faults involving the Cisco uBR10012 Power
Failures” Entry Modules (PEMs) and blower modules.
Chapter 3, “Troubleshooting PRE-1 Modules” How to troubleshoot Performance Routing Engine (PRE-1) modules. It
provides information on troubleshooting PRE-1 fault states, the
management Ethernet port, and the serial port.
Chapter 4, “Troubleshooting Line Cards” Troubleshooting faults for all following Cisco uBR10012 line cards.
Chapter 5, “Replacing or Recovering How to recover a lost enable or console login password, and how to replace
Passwords” a lost enable secret password on the Cisco uBR10012 router.
Appendix A, “Unsupported Commands” A list of the commands that are not supported in Cisco IOS Release
12.2(15)BC1 for the Cisco uBR10012 router.
Appendix B, “Recommended Tools and Test A list of basic tools and test equipment necessary to perform maintenance
Equipment” and troubleshooting tasks on the Cisco uBR10012 router.
Related Documentation
When troubleshooting the Cisco uBR10012 router, you should use the Cisco uBR10012 Universal
Broadband Router Troubleshooting Guide with the following documents:
• Cisco uBR10012 Universal Broadband Router Release Notes—Provides the most up-to-date
information about software version requirements for using the router. It also provides information
about bugs and workarounds. See the following URL:
http://www.cisco.com/univercd/cc/td/doc/product/cable/ubr10k/ub10krns/index.htm
• Cisco uBR10012 Universal Broadband Router Software Configuration Guide—Contains detailed
information on the configuration and administration of the Cisco uBR10012 router. See the
following URL:
http://www.cisco.com/univercd/cc/td/doc/product/cable/ubr10k/ubr10012/scg/index.htm
• Cisco uBR10012 Universal Broadband Router Hardware Installation Guide—Contains information
about the hardware of the Cisco uBR10012 router, how to install the router, connect its cables, and
start the system up for the first time. See the following URL:
http://www.cisco.com/univercd/cc/td/doc/product/cable/ubr10k/ubr10012/hig/index.htm
For more information about the IOS software that runs on the Cisco uBR10012 router, see the Cisco IOS
command reference books and configuration guides:
• Cisco Broadband Cable Command Reference Guide—Describes the cable specific commands used
on the Cisco uBR10012 router. See the following URL:
http://www.cisco.com/univercd/cc/td/doc/product/cable/bbccmref/i ndex.htm
• Cisco IOS Release 12.2 Configuration Guides and Command References—Describes the commands
and configuration used in Cisco IOS Release 12.2. See the following URL:
http://www.cisco.com/univercd/cc/td/doc/product/software/ios122/ 122cgcr/index.htm
Obtaining Documentation
Cisco documention and additional literature are available on Cisco.com. Cisco also provides several
ways to obtain technical assistance and other technical resources. These sections explain how to obtain
technical information from Cisco Systems.
Cisco.com
You can access the most current Cisco documentation on the World Wide Web at this URL:
http://www.cisco.com/univercd/home/home.htm
You can access the Cisco website at this URL:
http://www.cisco.com
International Cisco websites can be accessed from this URL:
http://www.cisco.com/public/countries_languages.shtml
Ordering Documentation
You can find instructions for ordering documentation at this URL:
http://www.cisco.com/univercd/cc/td/doc/es_inpck/pdi.htm
You can order Cisco documentation in these ways:
• Registered Cisco.com users (Cisco direct customers) can order Cisco product documentation from
the Ordering tool:
http://www.cisco.com/en/US/partner/ordering/index.shtml
• Nonregistered Cisco.com users can order documentation through a local account representative by
calling Cisco Systems Corporate Headquarters (California, USA) at 408 526-7208 or, elsewhere in
North America, by calling 800 553-NETS (6387).
Documentation Feedback
You can submit e-mail comments about technical documentation to bug-doc@cisco.com.
You can submit comments by using the response card (if present) behind the front cover of your
document or by writing to the following address:
Cisco Systems
Attn: Customer Document Ordering
170 West Tasman Drive
San Jose, CA 95134-9883
We appreciate your comments.
• Training—Cisco offers world-class networking training. Current offerings in network training are
listed at this URL:
http://www.cisco.com/en/US/learning/index.html
This section describes the basic procedures that users should perform before undertaking a detailed
troubleshooting analysis of the Cisco uBR10012 router or logging a case with the Cisco Technical
Assistance Center (TAC).
These basic troubleshooting checks are organized as follows:
• Basic Troubleshooting Checklist, page 1-1
• Confirming the Hardware Installation, page 1-2
• Displaying the Cisco IOS Software Version, page 1-3
• Displaying System Environment Information, page 1-4
• Hardware Troubleshooting Flowchart, page 1-4
• Cisco uBR10012 System Startup Sequence, page 1-5
125440K bytes of ATA PCMCIA card at slot 1 (Sector size 512 bytes).
32768K bytes of Flash internal SIMM (Sector size 256KB).
Configuration register is 0x2102
UBR10K-ROUTER1#
125440K bytes of ATA PCMCIA card at slot 1 (Sector size 512 bytes).
32768K bytes of Flash internal SIMM (Sector size 256KB).
Configuration register is 0x2102
UBR10K-ROUTER1#
UBR10K-ROUTER1#
Turn
on
PEM
*Miswire No Correct
LED input
off wiring
?
Yes
See section
PEM No PEM No "PEM Faults"
Power OK Troubleshoot Power OK
LED on PEM LED on
? ?
Yes Yes
See section
"PRE Module
PRE Reseat
Status No PRE No Faults"
PRE status LEDs
LEDs OK and restart
? OK?
Yes Yes
See section
ICC+ "Troubleshooting
Status No Reseat ICC+ No TCC+"
LEDs OK TCC+ status LEDs
? OK?
Yes Yes
See appropriate
line card fault
No Reseat Line No section
Line card line card card LEDs
LEDs on and restart OK
? ?
Yes Yes
Obtain
103381
technical
System assistance
startup
successful
6. The Power LED on each TCC+ card turns green. The Status LED on each TCC+ lights
yellow. After a few seconds, the Status LED on the primary TCC+ card lights green, and the
Status LED on the backup TCC+ card begins blinking green.
Cisco IOS software loads 1. If the system is set to boot from the slot0: file system, the green slot LED lights.
2. The message BOOT IMGE appears on the alphanumeric display on the active PRE-1
module.
3. The console displays a series of pound signs (#) as the IOS software image is decompressed.
4. The following messages appear on the alphanumeric display on the active PRE-1 module.
• IOS STRT
• IOS EXC
• IOS FPGA
• IOS FPOK
• IOS FILE
• IOS STBY
• IOS DRVR
• IOS LIB
• IOS MGMT
• IOS CONF
5. The console displays the bootup screen, followed by the prompt:
Press RETURN to get started!
6. The message IOS RUN appears in the alphanumeric display on the active PRE-1 module. In
a redundant configuration, the message IOS STBY appears on the alphanumeric display of
the standby PRE-1 module.
If the boot process fails, no console access is available. If you cannot boot the
Cisco uBR10012 router, call Cisco TAC.
The following sections provide methods for troubleshooting faults involving the Cisco uBR10012 DC
Power Entry Modules (PEMs), the optional 2400W AC-input power shelf, and fan assembly module.
This chapter contains the following major sections:
• AC PEM Faults, page 2-1
• DC PEM Faults, page 2-3
• 2400W AC-Input Power Shelf, page 2-5
• Other Electrical Problems, page 2-6
• Fan Assembly Module Faults, page 2-7
AC PEM Faults
On the Cisco uBR10012 router, two AC PEMs are installed in a redundant configuration, which allows
one AC PEM to fail without affecting system operations. A single PEM can power the router for
sufficient time to request and install a new PEM to replace the one that failed.
Tip To quickly check the functional status of your PEMs, use the show environment command.
POWER
FAULT
AC power
cord clips
AC power plug
AC power switch
62520
Table 2-2 lists the AC PEM fault symptoms and corrective actions.
PEM experiences 1. Ensure that the input power to both slots is correct.
problems in one
2. Verify that no connections have been made to the DC-power connectors
slot but operates
underneath each PEM.
normally in a
different slot 3. If the problem persists, contact Cisco TAC.
Fault LED is lit 1. Verify that no connections have been made to the DC-power connectors
yellow underneath each PEM.
2. Verify that the PEM is fully inserted into the power bay and that its captive
screws have been tightened.
3. Check to see if the power switch is set to the standby position. If so, set the
switch to the ON position.
4. If the problem persists, flip the power switch on the PEM to the standby
position, wait several seconds, and then back to the ON position.
5. Replace PEM with a known good replacement.
6. Contact Cisco TAC.
Tip Securely tighten the captive screws on your PEMs to prevent heightened levels of electromagnetic
interference.
DC PEM Faults
On the Cisco uBR10012 router, two DC PEMs are in a redundant configuration, which allows one DC
PEM to fail without affecting system operations. A single PEM can usually power the router for
sufficient time to request and install a new PEM to replace the one that failed.
Tip To quickly check the functional status of your PEMs, use the show environment command.
POWER
MISWIRE
FAULT
56480
Figure 2-3 DC PEM Front Panel (current model with alarm connector, UBR10-PWR-DC-M)
POWER
MISWIRE
FAULT
62518
LED Description
Power (green) PEM is powered on and is operational.
Fault (yellow) PEM is not operating correctly or the circuit breaker is in the OFF position.
Miswire (yellow) Input DC power cables are wired incorrectly and should be reversed.
Table 2-4 lists the DC PEM fault symptoms and corrective actions.
Tip Securely tighten the captive screws on your PEMs to prevent heightened levels of electromagnetic
interference.
The power shelf includes three 1200-watt (W) AC-input power modules that plug into a common power
backplane in the 2400W AC-input power shelf. Two 1200W AC-input power modules are capable of
powering a fully configured Cisco uBR10012 router. The third power module provides full redundancy.
During normal operation, the three AC-input power modules provide automatic load-sharing with each
power module supporting 33 percent of the power load. When you remove one of the AC-input power
modules, the remaining power modules immediately ramp up to full power and maintain uninterrupted
system power for a limited time. This allows you to replace the affected module without impacting
system operations.
Faults on the 2400W AC-input power shelf can occur for the following reasons:
• The AC-input power to one or more power modules has failed.
• The AC power plug to one or more power modules has been removed or unplugged.
• One or more power modules has failed and must be replaced.
Figure 2-4 illustrates the AC PEM and its indicators. Table 2-5 describes the indicators.
DC OK AC OK FAULT
DC OK AC OK FAULT
DC OK AC OK FAULT
36137
DC OK AC OK FAULT
FANS OK
SINGLE FAN FAILURE
MULTIPLE FAN FAILURE
Fan
assembly
CISCO
10000
CISCO
10000
56479
AU
X
AU
X
A
C
TI
V
IT
E
TH Y A
C
E TI
V
LI RN IT
N E E Y
K T TH
E
LI RN
N
K ET
The Cisco uBR10012 fan assembly module contains four fans in a redundant configuration. One fan can
fail without affecting system operations. If more than one fan fails, however, the fan assembly module
must be replaced immediately to avoid overheating the system.
The fan assembly module draws air in from the bottom front of the Cisco uBR10012 router, through the
air filter at the bottom of the front bezel. The air is drawn up through the line cards, and then exits
through the vents at the top rear of the router.
Figure 2-6 shows the air circulation pattern of the Cisco uBR10012 router when two DC PEMs are
installed. The air flow when two AC PEMs are installed is similar. The front bezel is not shown for
clarity.
CISCO POWER
10000 CISCO
10000
MISWIR POWER
E
C
O
N
FAULT MISWIR
S
O
LE C
O
N
S
O
E
FAULT
LE
AU
X
AU
X
A
C
TIV
IT A
E C
TH Y TIV
E IT
LIN RN E Y
SL
SL
E TH
K T
OT
E
OT
LIN RN
SL
SL
0
K ET
1
OT
OT
0
ALARMS
A ALARMS
C
O
A
C
O
C
R
IT
IC C
A R
M L IT
A IC
JO
R A
M L
M A
IN JO
PERFORMANCE ROUTING ENGINE
O R
R M
IN
PERFORMANCE ROUTING ENGINE
O
R
S
TA
TU
S S
FA TA
IL TU
FA S
IL
56430
The LEDs on the front panel indicate the current status of the fans. Table 2-6 lists the fan assembly
module fault indications and recommended actions.
Table 2-6 Fan Assembly Module Fault Indications and Recommended Action
This chapter describes how to troubleshoot Performance Routing Engine (PRE-1) modules. It provides
information on troubleshooting PRE-1 fault states, the management Ethernet port, and the serial port.
• Information Required for Troubleshooting PRE-1 Modules, page 3-1
• PRE Module Not Supported, page 3-2
• PRE-1 Module Status Screen, page 3-2
• Booting Up with Redundant PRE-1 Modules, page 3-3
• PRE-1 Module Faults, page 3-4
• Ethernet Connection Problems, page 3-6
• Console Port Serial Connection Problems, page 3-7
• Troubleshooting Common System Problems, page 3-8
https://www.cisco.com/cgi-bin/Support/OutputInterpreter/home.pl
Step 3 Capture the complete bootup sequence, especially if the router is reporting errors at bootup.
Step 4 If the router is unresponsive, or if it refuses to boot to the Cisco IOS prompt, reboot the router to the
ROMMON prompt and capture a stack trace, using the stack ROMMON command. For more
information on this procedure, see the Obtaining a Stack Trace from ROM Monitor section in the
Troubleshooting Router Hangs document, at the following URL:
http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a0080106fd7.shtml
To correct this error, replace the PRE modules in the router with PRE-1 modules. To continue using the
original PRE modules, you must be reload the router with Cisco IOS Release 12.2(4)BC1 or an earlier
12.2 BC release.
Note For information on the replacement of PRE modules with PRE-1 modules, see the Field Notice,
Cisco uBR10000 Proactive Upgrade of PRE to PRE1, at the following URL:
http://www.cisco.com/en/US/products/hw/cable/ps2209/products_field_notice09186a00800946c5.sht
ml
Message Description
BLDRSTRT The PRE-1 module is starting the boot loader software.
BLDREXC The boot loader software has begun to execute.
BLDRMEM The boot loader software is initializing the memory on the PRE-1 module.
BLDRFILE The boot loader software is initializing the router’s file systems.
BLDRDRVR The boot loader software is initializing the driver subsystems.
BLDRLIB The boot loader software is initializing the subsystem libraries.
BLDRPROT The boot loader software is initializing the protocol subsystems.
BLDRMGMT The boot loader software is initializing the management subsystems.
BLDRINTF The boot loader software is initializing the router’s interfaces.
BLDRSTBY The boot loader software is running and the PRE-1 module is running as the
standby PRE-1 module.
LOADIOS The boot loader software has finished initializing and has begun to load the
Cisco IOS software.
IOS STRT The PRE-1 module is starting the Cisco IOS software.
IOS EXC The Cisco IOS software has begun to execute.
IOS MEM The Cisco IOS software is initializing the memory on the PRE-1 module.
IOS FILE The Cisco IOS software is initializing the router’s file systems.
Message Description
IOS DRVR The Cisco IOS software is initializing the driver subsystems.
IOS LIB The Cisco IOS software is initializing the subsystem libraries.
IOS PROT The Cisco IOS software is initializing the protocol subsystems.
IOS MGMT The Cisco IOS software is initializing the management subsystems.
IOS INTF The Cisco IOS software is initializing the router’s interfaces.
IOS CONF The Cisco IOS software has begun to load the startup configuration file.
IOS RUN The Cisco IOS software is running and the PRE-1 module is running as the
active PRE-1 module. This could indicate that the PRE-1 module originally
booted up as the active module, or that a switchover put this module into the
active state.
Note This message indicates that the Cisco IOS router is running a Cisco IOS
software image. This is typically the full Cisco IOS image that was
found on a Flash disk or TFTP server. However, if an error occurs during
bootup, this could be the boot Cisco IOS image that is permanently
written in the router’s bootflash and is used when the router cannot boot
the full Cisco IOS image.
IOS STBY The Cisco IOS software is running and the PRE-1 module is running as the
standby PRE-1 module. This could indicate that the PRE-1 module originally
booted up as the standby module, or that the PRE-1 module was originally the
active PRE-1module, but that a switchover put it into the standby state.
• Verify that both PRE-1 modules are booting the same version of Cisco IOS software. Slight
variations in the loading of different images could allow the slot B PRE-1 module to boot first.
CISCO
10000
LED Status Description
Ethernet port LEDs
CO
NS
OL
Activity Green Packets are being transmitted
E
AU
X
and received.
AC
T IV
IT
Off No activity.
ET Y
HE
LIN
K
RN
ET
Link Green Carrier detected; the port is
SL
SL
OT
OT
CR
IT
IC
Green PRE is active (primary).
AL
MA
MIN
JO
R
Flashing PRE is standby (secondary).
PERFORMANCE ROUTING ENGINE
OR
green
Off No power to PRE.
ST
AT
US
FA
IL
Fail Yellow A major failure has disabled
the PRE.
29995
Off The PRE is operating properly.
Table 3-2 lists the PRE-1 fault indications and recommended actions. The information contained in the
table is based on the assumption that you have a nonredundant configuration.
The PRE-1 initializes, but you cannot 1. Ensure that the terminal settings are properly set.
establish a console connection
2. If you still cannot connect, check the console cable. Is it firmly connected? Is it
the correct type of cable with proper connectors?
3. If the cable checks out and you cannot establish a console or Telnet session,
reinsert the PRE-1 module. If the problem persists, replace the PRE-1 module.
4. Enter show log to review console messages recorded in the system log.
Card cannot be fully inserted into its Make sure that you are using the correct slot (A or B) for the PRE-1 module.
slot
An alarm LED is lit 1. Enter the show facility-alarm status command and examine the output to
determine which system component raised the alarm.
2. Troubleshoot using a procedure appropriate to the module or FRU responsible for
the alarm.
Note The show interface command also shows that there is an Ethernet interface (E0/0/0) on the PRE-1
module, but this is an internal interface that the router uses to communicate between PRE-1 modules and
line cards. This Ethernet interface is not configurable and can be used only by the router’s internal
subsystems.
If the Link LED is lit (steady green), but the Ethernet port is not working properly, make sure that the
port in question is configured properly and is not administratively shut down. If you have a working
console connection, perform the following steps:
Step 1 At the switch prompt, enter show interface fastethernet0/0/0. If the port is administratively down, enter
these commands to enable it:
c10000# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
c10000(config)#interface fastethernet0/0/0
c10000(config-if)# no shut
c10000(config-if)# exit
c10000(config)# exit
c10000#
Step 2 Check that the Ethernet port in question is assigned a valid IP address.
For more information about configuring Ethernet ports, refer to the Cisco uBR10012 Universal
Broadband Router Software Configuration Guide.
If the cable, connections, power, and configuration all check out, and you still cannot connect to the
Ethernet port on the module, replace the module in question. If the problem persists, contact the Cisco
TAC for further assistance. Refer to the “Obtaining Technical Assistance” section on page x for
instructions on contacting the Cisco TAC.
Step 1 Refer to the “Cisco uBR10012 System Startup Sequence” section on page 1-5. If the display stops
responding during this process, there is no console output.
Step 2 Check the console cable and make sure it is properly connected to the console port on the active PRE-1
module at one end and to your terminal equipment or terminal server at the other end.
Note You cannot connect to the console port on the standby PRE-1 module. You must connect to
the console port on the currently active PRE-1 module. If a switchover occurs, you must
switch the serial cable to the new active PRE-1 module to maintain the console connection.
Step 3 Verify that you are using the right type of cable and adapter. For information about pin-out connections
and installation instructions, refer to the Cisco uBR10012 Universal Broadband Router Hardware
Installation Guide.
Step 4 Make sure the cable is not defective or broken. Replace the cable with another high quality cable if
possible, and check to see if the console port starts working.
Step 5 Check that the terminal equipment is configured with the correct settings for the console port. The
default console port settings are:
• 9600 baud
• 8 data bits
• 1 stop bit
• No parity
• No flow control
Step 6 Check the LEDs on the PRE-1 faceplate to make sure it has powered up properly. If necessary, remove
and reinsert both PRE-1 modules to power them up again. Also, make sure the terminal equipment is
working properly.
Step 7 The console can appear frozen if the PRE-1 processor is busy performing other tasks, such as parsing a
large configuration file or passing a large burst of traffic. These periods are usually only temporary, and
normal reaction resumes after a few moments.
Step 8 The console can be frozen if the PRE-1 process is generating a large volume of debug messages. If this
is the case, hit the return key a couple of times and type no debug all to attempt to turn off the debug
messages. This will not work if the router is in global configuration mode, but try typing do no debug
all to execute this EXEC mode command in global configuration mode.
If the cable, connections, power, and terminal settings all check out and you still cannot connect to the
console port on the module, replace the module in question. If the problem persists, contact the Cisco
TAC for further assistance.
Note Registered Cisco.com users can decode the output of these show commands by using the Output
Interpreter tool, which is at the following URL:
https://www.cisco.com/cgi-bin/Support/OutputInterpreter/home.pl
For additional information on troubleshooting system crashes, see the following URLs:
• Troubleshooting Router Crashes, at the following URL:
http://www.cisco.com/en/US/products/sw/iosswrel/ps1835/products_tech_note09186a00800b4447
.shtml
• Less Common Types of System Crashes, at the following URL:
http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_note09186a008010876d
.shtml
http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a00800a70f2.shtml
ARP Traffic
High volumes of Address Resolution Protocol (ARP) requests and responses can occupy a significant
portion of the CPU time, because the router cannot use fast-switching to process ARP packets, but must
instead forward them to the route processor (RP). Because of this, processing a large volume of ARP
traffic can also prevent the router from handling normal traffic.
Theft-of-service and denial-of-service (DNS) attacks also often generate a large number of ARP packets
on the network. Many viruses also use ARP requests to discover computers that might be vulnerable to
attack, and if these computers become infected, they are used to propagate the virus, generating even
more ARP traffic on the network.
ARP requests are broadcast packets, so they are broadcast to all devices on that particular network
segment. In some cases, a router can also forward ARP broadcasts to an ARP proxy for further
processing. Some low-end routers commonly used by subscribers for home networks can also incorrectly
respond to all ARP requests, which generates even more traffic.
In addition, the Cisco CMTS router automatically monitors ARP traffic and enters the IP addresses
found in ARP requests into its own ARP table, in the expectation that a device will eventually be found
with that IP address. Unacknowledged IP addresses remain in the router’s ARP table for 60 seconds,
which means that a large volume of ARP traffic can fill the router’s ARP table.
If ARP traffic is excessive, you can try the following ways to limit this traffic:
Step 1 Disable the forwarding of ARP requests on a cable interface by using the no cable arp command in
interface configuration mode.
Step 2 Disable the use of proxy-ARP on a cable interface by using the no cable proxy-arp command in
interface configuration mode.
Note Using the no cable arp and no cable proxy-arp commands shifts all responsibility for the
management of the IP addresses used by CMs and CPE devices to the DHCP server and
provisioning system.
Another approach would be to identify the cable modems and customer premises equipment (CPE) that
are generating the ARP traffic. A simple way of doing this is by using an access list to log requests for
an unassigned IP address in the subnet being used on a cable interface.
Step 1 Reserve at least one IP address on each cable interface’s subnet and ensure that it is not being assigned
to any cable modems or CPE devices. For example, if a cable interface is using the subnet
192.168.100.0/24, you could choose to reserve IP address 192.168.100.253 for this purpose. Ensure that
the IP addresses you have chosen are not assigned to devices by your provisioning system.
Step 2 If you currently have an access list applied to the cable interface, add a line that logs requests for this
particular IP address. If you are not currently using an access list on the cable interface, create one for
this purpose. In both cases, the relevant line would be:
Router(config)# access-list number permit ip any host 192.168.100.253 log
where number is the number for the access-list. Change the IP address to whatever address you have
selected to be reserved for this cable interface.
Note If you are creating a new access list, ensure that the last line of the list is access-list number
permit ip any any. Otherwise, all other traffic will be blocked on the interface.
Step 3 Apply the access list to the cable interface using the ip access-group command:
Router(config-if)# ip access-group number in
Step 4 After applying the access list, regularly examine the message log to find the devices that are attempting
to access the reserved IP address. If a cable modem or CPE device is repeatedly sending ARP requests
or replies for this IP address, it could be part of a virus or theft-of-service attack, or it could indicate a
cable modem with defective software.
Step 5 After identifying these devices, you can further investigate the matter, and if necessary, block these
devices from further network access.
CPUHOG Errors
The router displays a %SYS-3-CPUHOG error message when a process is using an excessive amount of
processor cycles. For example, using the logging buffered command to allocate a significant amount of
memory (for example, 200 MB) for log buffers could generate a %SYS-3-CPUHOG message, because
allocating such an amount of memory requires a large amount of processor time.
For more information on what could cause this problem and how to resolve it, see the document What
Causes %SYS-3-CPUHOG Messages, at the following URL:
http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_note09186a00800a6ac4.sht
ml
Router#
2. Disable console messages by using the no logging console command in global configuration mode:
Router# configure terminal
Router(config)# no logging console
Router(config)#
To keep the logging of console messages, but to limit the number of messages that can be displayed,
use the logging rate-limit command. You can rate-limit all messages (including debug messages),
or just the console messages, using one of the following commands:
Router(config)# logging rate-limit console number-of-messages-per-second
3. If you have logged into the router using a Telnet connection, you can disable debug messages using
the terminal default monitor command in privileged EXEC mode:
Router# terminal default monitor
Router#
For information on resolving problems with high CPU usage caused by the Exec and Virtual EXEC
processes, see the document High CPU Utilization in Exec and Virtual Exec Processes, at the following
URL:
http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2ae4.shtml
http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml
We recommend using the default configuration, which can be restored by giving the default scheduler
allocate command in global configuration mode:
Router(config)# default scheduler allocate
Router(config)#
IP Input Processing
The Cisco IOS software uses a process named IP input to process IP packets that cannot be processed
using the fast-switching process. If the router is process-switching a lot of IP traffic, it could result in
excessively high CPU usage.
The most common reasons for excessive IP Input processing are that fast-switching has been disabled
on one or more interfaces, and that the router is receiving a large volume of traffic that must be
process-switched. For more information on resolving problems with the IP Input process, see the
Troubleshooting High CPU Utilization in IP Input Process document at the following URL:
http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af3.shtml
For more information, see the document Troubleshooting High CPU Utilization Due to Processes, at the
following URL:
http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af6.shtml
http://www.cisco.com/univercd/cc/td/doc/product/software/ios122/122cgcr/fipr_c/ipcprt1/1cfip.htm
Tip If you are using Ciscoworks to manage your network, consider using the Ciscoworks Access
Control List Manager to manage access lists.
SNMP Traffic
High volumes of Simple Network Management Protocol (SNMP) traffic can occupy a significant portion
of the CPU time, as the processor receives SNMP requests and sets the appropriate attributes on the
router, or sends the appropriate information back to the SNMP manager. For information on controlling
SNMP traffic, see the Application Note, IP Simple Network Management Protocol (SNMP) Causes High
CPU Utilization, at the following URL:
http://www.cisco.com/en/US/tech/tk648/tk362/technologies_tech_note09186a00800948e6.shtml
Bus Errors
Bus errors occur when the router tries to access a memory location that either does not exist (which
indicates a software error) or that does not respond (which indicates a hardware error). Bus errors
generated by the PRE-1 module typically cause a crash and force the router to reload.
Use the following procedure to determine the cause of a bus error and to resolve the problem. Perform
these steps as soon as possible after the bus error, before manually reloading or power cycling the router.
Step 1 Use the show version command to display the reason for the last system reload:
Router# show version
...
Router#
Step 2 Determine whether the memory address for the bus error is a valid address. If the address is valid, the
problem is most likely a hardware problem. If the address is an invalid address (such as the above
example of 0x0), the problem is software-related.
Step 3 If the problem is hardware-related, you can map the memory address to a particular hardware component
by using the show region command.
Router# show region
Region Manager:
Router#
Step 4 When you have identified the hardware that is generating the bus error, try removing and reinserting the
hardware into the chassis. If this does not correct the problem, replace the DRAM chips on the hardware.
If the problem persists, replace the hardware.
Step 5 If the problem is software-related, verify that you are running a released version of software, and that
this release of software supports all of the hardware that is installed in the router. If necessary, upgrade
the router to the latest version of software.
Step 6 To further troubleshoot the problem, registered users on Cisco.com can also decode the output of
multiple show commands by using the Output Interpreter tool, which is at the following URL:
https://www.cisco.com/cgi-bin/Support/OutputInterpreter/home.pl
Tip The most effective way of using the Output Interpreter tool is to capture the output of the
show stacks and show tech-support commands and upload the output into the tool. If the
problem appears related to a line card, you can also try decoding the show context command.
For more information on troubleshooting bus errors, see the Troubleshooting Bus Error Crashes
document, at the following URL:
http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_note09186a00800cdd51.sht
ml
Memory Problems
This section describes the following types of memory problems:
• Alignment Errors, page 3-15
• Low Memory Errors, page 3-16
• Memory Parity Errors, page 3-16
• Particle Pool Fallbacks, page 3-17
• Spurious Interrupts, page 3-18
• Spurious Memory Accesses, page 3-19
Alignment Errors
Alignment errors occur when the software attempts to read or write data using a data size that is not
aligned with the memory address being used. For example, an alignment error occurs when attempting
to read two bytes from a memory address that is not an even multiple of two bytes.
Alignment errors are always caused by a software bug, and can be either correctable or fatal. See the
following sections for more information. Also see the document Troubleshooting Spurious Accesses,
Alignment Errors, and Spurious Interrupts, at the following URL:
http://www.cisco.com/en/US/products/sw/iosswrel/ps1828/products_tech_note09186a00800a65d1.sht
ml
The Cisco IOS software can automatically correct most alignment errors. When it does so, the router
generates a system error message similar to the following:
%ALIGN-3-CORRECT: Alignment correction made at 0x60262478 reading/writing 0x60A9FF5C
Occasional alignment errors do not necessarily require operator intervention, because the Cisco IOS
software can correct these errors and continue with normal operations. However, correcting alignment
errors consumes processor resources and could impact performance if the errors continuously repeat.
If the alignment error was a fatal error, it displays a message similar to the following:
%ALIGN-1-FATAL: Corrupted program counter error.
ERROR: Slot 0, NPE300/IOFE2/VXR, CACHE, External Data Cache Memory Test:
*** Data Expected= 0x99999999 ***
Fatal alignment errors are most likely a hardware fault on the processor card. The card itself could be
faulty, or the memory on the card could be faulty. Try replacing the processor card and rebooting the
router. If a replacement card is not available, try replacing the memory on the processor card, making
sure that the new memory meets the specifications that are required by the card.
http://www.cisco.com/warp/customer/63/mallocfail.shtml
System returned to ROM by processor memory parity error at PC 0x60301298, address 0x0 at
17:19:47 PDT Mon Dec 15 2003
System restarted at 17:19:47 PDT Mon Dec 15 2003
...
Router#
Step 1 Determine whether this is a soft parity error or a hard parity error. Soft parity errors are 10 to 100 times
more frequent than hard parity errors. Therefore, wait for a second parity error before taking any action.
Monitor the router for several weeks after the first incident, and if the problem reoccurs, assume that the
problem is a hard parity error and proceed to the next step.
Step 2 When a hard parity error occurs (two or more parity errors at the same memory location), try removing
and reinserting the PRE-1 module, making sure to fully insert the card and to securely tighten the
restraining screws on the front panel.
Step 3 If this does not resolve the problem, remove and reseat the DRAM chips. If the problem continues,
replace the DRAM chips.
Step 4 If parity errors occur, the problem is either with the PRE-1 module or the router chassis. Replace the
PRE-1 module.
Step 5 If the problems continue, contact Cisco TAC for further instructions.
For more information about parity errors, see the Processor Memory Parity Errors document, at the
following URL:
http://www.cisco.com/en/US/products/hw/routers/ps341/products_tech_note09186a0080094793.shtml
The system also allocates public pools in a number of different sizes for more general use. If a packet
requires special handling, or if a packet cannot be completely processed at interrupt time, the system
copies the packet into a portion of contiguous memory in the public pool, so it can be processed
switched.
Tip Use the show buffers command to display the current status of the router’s particle pools.
Fallbacks with particle pools occur when bursts of traffic produce more packets than would fit in the
available buffer space. When an interface runs out of space in the private particle pools, it falls back to
using the normal public memory. Fallbacks are expected during periods of bursty traffic, and the router
should be considered to be operating normally in these situations.
If fallbacks occur more frequently, however, it could indicate a problem. In particular, if the private
particle pools are consistently producing fallbacks, it could result in the router using excessive amounts
of public memory for packet processing, reducing the resources that are available to the other router
processes. If this is the case, look for the following possible causes.
• Extremely fast interfaces are handling large volumes of traffic with a high rate of throughput that is
approaching the maximum rate on the interface.
• The Fast Ethernet interfaces on the processor card could be heavily loaded.
• The Cisco IOS software has a memory leak that is not releasing the memory in the private particle
pool after the interface has finished processing a packet.
For more information on resolving problems with particle pool buffers, see the document Buffer Tuning,
at the following URL:
http://www.cisco.com/warp/public/63/buffertuning.html
Also see the document Troubleshooting Buffer Leaks, at the following URL:
http://www.cisco.com/warp/public/63/bufferleak_troubleshooting.html
Spurious Interrupts
A spurious interrupt occurs when the Cisco IOS software generates an unnecessary interrupt for packet
that has been processed already. This is a software error that is usually caused by an improper
initialization of interrupt handling routines, or by a race condition where two processes compete to
handle the same process.
Spurious interrupts can occasionally be expected during normal operations, and the occasional spurious
interrupt has no discernible impact on the router’s performance. However, action might be needed if the
number of spurious interrupts is high or increasing, and performance is being degraded, with packets
being dropped.
For information on resolving the problem with spurious interrupts, see the document Troubleshooting
Spurious Accesses, Alignment Errors, and Spurious Interrupts, at the following URL:
http://www.cisco.com/en/US/products/sw/iosswrel/ps1828/products_tech_note09186a00800a65d1.sht
ml
Where possible, the Cisco IOS software handles spurious memory accesses by returning a value of zero
to the calling routine, and then displaying the above error message. If this is not possible, the router
crashes with a Segment Violation (SegV) error. In either case, the cause of the error is almost always a
bug in the Cisco IOS software.
If possible, upgrade to the latest release of the Cisco IOS software. If the bug still exists on the router,
see the section Spurious Accesses in the document Troubleshooting Spurious Accesses, Alignment
Errors, and Spurious Interrupts, at the following URL:
http://www.cisco.com/en/US/products/sw/iosswrel/ps1828/products_tech_note09186a00800a65d1.sht
ml
This chapter discusses troubleshooting faults on the following Cisco uBR10012 line cards:
• General Information for Troubleshooting Line Card Crashes, page 4-2
• Troubleshooting the Timing, Communication, and Control Plus Card, page 4-8
• Troubleshooting the OC-12 Packet-Over-SONET Line Card, page 4-12
• Troubleshooting the OC-12 Dynamic Packet Transport Spatial Reuse Protocol WAN Card, page
4-14
• Troubleshooting the Cisco uBR10012 OC-48 DPT/POS Line Card, page 4-16
• Troubleshooting the Gigabit Ethernet Line Card, page 4-18
Command Description
show version Provides general information about the system's hardware and software
configurations
show logging Displays the general logs of the router
show diag [slot/subslot] Provides specific information about a particular slot: type of engine,
hardware revision, firmware revision, memory configuration, and so on.
show context [summary | Provides context information about the most recent crashes. This is
slot [slot/subslot] ] often the most useful command for troubleshooting line card crashes.
Use the following procedure if you suspect that a line card has crashed.
Step 1 If you can identify the particular card that has crashed or is experiencing problems, first use the other
sections in this chapter to perform basic troubleshooting. In particular, ensure that the line card is fully
inserted into the proper slot, and that all cables are properly connected.
Step 2 If any system messages were displayed on the console or in the SYSLOG logs at the time of the crash,
consult the Cisco CMTS System Messages guide and the Cisco IOS System Messages Guide for possible
suggestions on the source of the problem.
Step 3 Line cards can crash or appear to crash when an excessive number of debug messages are being
generated. In particular, this can happen when using the verbose or detail mode of a debug command,
or if the debug command is dumping the contents of packets or packet buffers. If the console contains a
large volume of debug output, turn off all debugging with the no debug all command.
Step 4 If the system message log contains messages that indicate the line card is not responding (for example,
%IPCOIR-3-TIMEOUT), and the card’s LEDs are not lit, the line card might have shut down because of
overheating. Ensure that all chassis slots either have the proper card or module installed in them. If a slot
is blank, ensure that the slot has a blank front panel installed, so that proper airflow and cooling can be
maintained in the chassis.
Step 5 Use the show context summary command to identify all of the line cards that have experienced a crash:
Router# show context summary
Router#
Step 6 After identifying the particular card that crashed, use the show context command again to display more
information about the most recent crash. For example:
Router# show context slot 2/0
CRASH INFO: Slot 2/0, Index 1, Crash at 19:57:56 PDT Wed Nov 27 2002
VERSION:
7200 Software (UBR10KCLC-LCK8-M), Version 12.2(122BC.021127.), CISCO DEVELOPMENN
Compiled Wed 27-Nov-02 12:57 by
Card Type: UNKNOWN, S/N CAB0544L6F5
System exception: sig=10, code=0x8000000C, context=0x60A1BDE4
STACK TRACE:
traceback 601C28FC 601C29B4 601B9E8C 600F99B0 600F999C
CONTEXT:
$0 : 00000000, AT : 60930000, v0 : FFFFFFFF, v1 : 60940000
a0 : 00000000, a1 : 00000000, a2 : 00000001, a3 : 0000EA60
t0 : FFFFFFFF, t1 : FFFFA91C, t2 : 601284E0, t3 : FFFF00FF
t4 : 601284D8, t5 : 00000062, t6 : 00000000, t7 : D1B71759
s0 : 00000000, s1 : 00000008, s2 : 00000000, s3 : 60CD0998
s4 : 60CD0990, s5 : 00000000, s6 : 00000002, s7 : 60940000
t8 : 60D98C2C, t9 : 0000001B, k0 : 3040D001, k1 : BE840244
gp : 6093BD60, sp : 60CD0968, s8 : 60A70000, ra : 601C2900
EPC : 0x601C28F8, SREG : 0x3400F903, Cause : 0x8000000C
ErrorEPC : 0xCF1998F2
Router#
Step 7 Look for the SIG Type in the line that starts with “System exception” to identify the reason for the crash.
Table 4-2 lists the most common SIG error types and their causes.
Step 8 The vast majority of line card crashes are either Cache Parity Exception (SIG type=20), Bus Error
Exception (SIG type=10), and Software-forced Crashes (SIG type=23). Use the following sections to
further troubleshoot these problems:
• Cache Parity Errors, page 4-4
• Bus Errors, page 4-5
• Software-Forced Crashes, page 4-6
If the line card crashed for some other reason, capture the output of the show tech-support command.
Registered Cisco.com users can decode the output of this command by using the Output Interpreter tool,
which is at the following URL:
https://www.cisco.com/cgi-bin/Support/OutputInterpreter/home.pl
Step 9 If you cannot resolve the problem using the information from the Output Interpreter, collect the
following information and contact Cisco TAC:
• All relevant information about the problem that you have available, including any troubleshooting
you have performed.
• Any console output that was generated at the time of the problem.
• Output of the show tech-support command.
• Output of the show log command (or the log that was captured by your SYSLOG server, if
available).
For information on contacting TAC and opening a case, see the “Obtaining Technical Assistance” section
on page x.
• Hard parity errors occur when a hardware defect in the DRAM or processor board causes data to be
repeatedly corrupted at the same address. In general, a hard parity error occurs when more than one
parity error in a particular memory region occurs in a relatively short period of time (several weeks
to months).
When parity occurs, take the following steps to resolve the problem:
Step 1 Determine whether this is a soft parity error or a hard parity error. Soft parity errors are 10 to 100 times
more frequent than hard parity errors. Therefore, wait for a second parity error before taking any action.
Monitor the router for several weeks after the first incident, and if the problem reoccurs, assume that the
problem is a hard parity error and proceed to the next step.
Step 2 When a hard parity error occurs (two or more parity errors at the same memory location), try removing
and reinserting the line card, making sure to fully insert the card and to securely tighten the restraining
screws on the front panel.
Step 3 If this does not resolve the problem, remove and reseat the DRAM chips. If the problem continues,
replace the DRAM chips.
Step 4 If parity errors occur, the problem is either with the line card or the router chassis. Try removing the line
card and reinserting it. If the problem persists, try removing the line card from its current slot and
reinserting it in another slot, if one is available. If that does not fix the problem, replace the line card.
Step 5 If the problems continue, collect the following information and contact Cisco TAC:
• All relevant information about the problem that you have available, including any troubleshooting
you have performed.
• Any console output that was generated at the time of the problem.
• Output of the show tech-support command.
• Output of the show log command (or the log that was captured by your SYSLOG server, if
available).
For information on contacting TAC and opening a case, see the “Obtaining Technical Assistance” section
on page x.
Bus Errors
Bus errors (SIG type is 10) occur when the line card tries to access a memory location that either does
not exist (which indicates a software error) or that does not respond (which indicates a hardware error).
Use the following procedure to determine the cause of a bus error and to resolve the problem.
Perform these steps as soon as possible after the bus error. In particular, perform these steps before
manually reloading or power cycling the router, or before performing an Online Insertion/Removal
(OIR) of the line card, because doing so eliminates much of the information that is useful in debugging
line card crashes.
Step 1 Capture the output of the show stacks, show context, and show tech-support commands. Registered
Cisco.com users can decode the output of this command by using the Output Interpreter tool, which is
at the following URL:
https://www.cisco.com/cgi-bin/Support/OutputInterpreter/home.pl
Step 2 If the results from the Output Interpreter indicate a hardware-related problem, try removing and
reinserting the hardware into the chassis. If this does not correct the problem, replace the DRAM chips
on the hardware. If the problem persists, replace the hardware.
Step 3 If the problem appears software-related, verify that you are running a released version of software, and
that this release of software supports all of the hardware that is installed in the router. If necessary,
upgrade the router to the latest version of software.
Tip The most effective way of using the Output Interpreter tool is to capture the output of the
show stacks and show tech-support commands and upload the output into the tool. If the
problem appears related to a line card, you can also try decoding the show context command.
Upgrading to the latest version of the Cisco IOS software eliminates all fixed bugs that can cause line
card bus errors. If the crash is still present after the upgrade, collect the relevant information from the
above troubleshooting, as well as any information about recent network changes, and contact Cisco TAC.
Software-Forced Crashes
Software-forced crashes (SIG type is 23) occur when the Cisco IOS software encounters a problem with
the line card and determines that it can no longer continue, so it forces the line card to crash. The original
problem could be either hardware-based or software-based.
The most common reason for a software-forced crash on a line card is a “Fabric Ping Timeout,” which
occurs when the PRE-1 module sends five keepalive messages (fabric pings) to the line card and does
not receive a reply. If this occurs, you should see error messages similar to the following in the router’s
console log:
%GRP-3-FABRIC_UNI: Unicast send timed out (4)
%GRP-3-COREDUMP: Core dump incident on slot 4, error: Fabric ping failure
Fabric ping timeouts are usually caused by one of the following problems:
• High CPU Utilization—Either the PRE-1 module or line card is experiencing high CPU utilization.
The PRE-1 module or line card could be so busy that either the ping request or ping reply message
was dropped. Use the show processes cpu command to determine whether CPU usage is
exceptionally high (at 95 percent or more). If so, see the “High CPU Utilization Problems” section
on page 3-9 for information on troubleshooting the problem.
• CEF-Related Problems—If the crash is accompanied by system messages that begin with “%FIB,”
it could indicate a problem with Cisco-Express Forwarding (CEF) on one of the line card’s
interfaces. For more information, see Troubleshooting CEF-Related Error Messages, at the
following URL:
http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a0080110d68.s
html
• IPC Timeout—The InterProcess Communication (IPC) message that carried the original ping
request or the ping reply was lost. This could be caused by a software bug that is disabling interrupts
for an excessive period of time, high CPU usage on the PRE-1 module, or by excessive traffic on the
line card that is filling up all available IPC buffers.
If the router is not running the most current Cisco IOS software, upgrade the router to the latest
software release, so that any known IPC bugs are fixed. If the show processes cpu shows that CPU
usage is exceptionally high (at 95 percent or more), or if traffic on the line card is excessive, see the
“High CPU Utilization Problems” section on page 3-9.
http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a00800945a1.s
html
• Hardware Problem—The card might not be fully inserted into its slot, or the card hardware itself
could have failed. In particular, if the problem began occurring after the card was moved or after a
power outage, the card could have been damaged by static electricity or a power surge. Only a small
number of fabric ping timeouts are caused by hardware failures, so check for the following before
replacing the card:
– Reload the software on the line card, using the hw-module slot reset command.
– Remove and reinsert the line card in its slot.
– Try moving the card to another slot, if one is available.
If software-forced crashes continue, collect the following information and contact Cisco TAC:
• All relevant information about the problem that you have available, including any troubleshooting
you have performed.
• Any console output that was generated at the time of the problem.
• Output of the show tech-support command.
• Output of the show log command (or the log that was captured by your SYSLOG server, if
available).
For information on contacting TAC and opening a case, see the “Obtaining Technical Assistance” section
on page x.
Note Because the TCC+ card is considered a half-height card, use slot numbers 1/1 or 2/1 to display
information for the TCC+ card using the show diag command. The show cable clock and show
controllers clock-reference commands also use these slot numbers when displaying clock-related
information.
The front panel on the TCC+ card has seven LEDs. Table 4-3 describes each LED on the TCC+ card.
When performing any troubleshooting on the TCC+ cards, first check the LEDs as follows:
1. Check the POWER LEDs on each TCC+ card. Are the POWER LEDs on each TCC+ card on
(green)?
– If no, remove the TCC+ card and reinsert it, making sure that it firmly connects to the backplane
and that both captive screws are tightly connected.
– If yes, proceed to the next step.
2. Is the STATUS LED on the primary TCC+ card on (green) to indicate that it is the primary card? Is
the STATUS LED on the secondary TCC+ card flashing (green) to indicate that it is the redundant
card?
Use Table 4-4 to continue troubleshooting the TCC+ cards.
CISCO
10000
LED Status Description
Fail Yellow A major failure has disabled the
FA
IL
line card.
Off The line card is operating
properly.
Rx (receive) Green Receiving traffic.
Off Not receiving traffic.
Tx (transmit) Green Transmitting traffic.
Off Not transmitting traffic.
Carrier Green Carrier detected.
CA
RR
29993
IE
RX
TX
0
OC–12/STM–4 POS SM–IR
Table 4-5 describes fault conditions on the OC-12 POS line card and recommended corrective actions.
Table 4-5 OC-12 POS Line Card Fault Indications and Recommended Action
68499
Table 4-6 Cisco uBR10-SRP-OC12SML DPT WAN Line Card LEDs
Table 4-6 Cisco uBR10-SRP-OC12SML DPT WAN Line Card LEDs (continued)
FA
CISCO
LI
10000
FA
IL
TX
TX
Warning
Invisible laser radiation may
RX
be emitted from
RX
disconnected fibers or
connectors. Do not stare
into beams or view directly
EN
AB
SR
PO
EN
LE
P
S
SR
PO
LE
P
S
RX
CD
TX
W C
PA RU
SY
THAP
R
SS
N
RX
CD
TX
PA RU
W C
TH P
SY
SS
RA
OC–48/STM–16 POS/SRP SM–LR
N
P/N ESR-10C48/P/SRPSMS
Ejector lever
82923
Captive screw
Tip Make sure that the gigabit Ethernet Interface Converter (GBIC) type on the Cisco uBR10012 router
matches the GBIC type at the other end of the fiber optic cable.
Figure 4-5 Gigabit Ethernet Line Card Faceplate and LED Descriptions
CISCO
10000
LED Status Description
Fail Yellow A major failure has disabled the
FA
IL
line card.
Off The line card is operating
properly.
Rx (receive) Green Receiving traffic.
Off Not receiving traffic.
Tx (transmit) Green Transmitting traffic.
Off Not transmitting traffic.
Link Green Carrier detected; the port is
able to pass traffic.
LIN
RX
TX
K
29992
not able to pass traffic.
GIGABIT ETHERNET
Table 4-8 describes the gigabit Ethernet line card fault indications and suggests responses to each.
Table 4-8 Gigabit Ethernet Line Card Faults and Recommended Responses
This section describes how to recover a lost enable or console login password, and how to replace a lost
enable secret password on the Cisco uBR10012 router.
Note It is possible to recover the enable or console login password. The enable secret password is encrypted,
however, and must be replaced with a new enable secret password.
Note To recover a lost password if the break function is disabled on the router, you must have physical access
to the router.
Step 2 Configure the terminal to operate at 9600 baud, 8 data bits, no parity, and 1 stop bit.
Step 3 If you can log in to the router as a nonprivileged user, enter the show version command to display the
existing configuration register value, then go to Step 6. If you cannot log in to the router at all, go to the
next step.
Step 4 Press the Break key or send a break signal from the console terminal.
• If break is enabled, the router enters the ROM monitor, indicated by the ROM monitor prompt
(rommon>). Go to Step 6.
• If break is disabled, power cycle the router (turn off the router or unplug the power cord, and then
restore power). Then go to Step 5.
Step 5 Within 60 seconds of restoring the power to the router, press the break key or send a break signal. This
action causes the router to enter the ROM monitor and display the ROM monitor prompt ( rommon>).
Step 6 Set the configuration register using the configuration register utility. Enter the confreg command at the
ROM monitor prompt as follows:
rommon> confreg
Answer yes to the enable “ignore system config info?” Press the return key at all other prompts to accept
the existing value.
Step 7 Reboot the router by entering the reset command:
rommon> reset
The router initializes, the configuration register is set to 0x142, and the router boots the system image
from Flash memory and enters the system configuration dialog (setup):
--- System Configuration Dialog --
Step 8 Enter no in response to the system configuration dialog prompts until the following message appears:
Press RETURN to get started!
Step 10 Enter the enable command to enter privileged EXEC mode. Then enter the show startup-config
command to display the passwords in the configuration file as follows:
Router# show startup-config
Step 11 Scan the configuration file display, looking for the passwords (the enable passwords are usually located
near the beginning of the file, and the console login or user EXEC password is near the end). The
passwords displayed appear similar to the following:
enable secret 5 $1$ORPP$s9syZt4uKn3SnpuLDrhuei
enable password 23skiddoo
.
.
line con 0
password onramp
The enable secret password is encrypted and cannot be recovered; it must be replaced. Go to the next
step to replace an enable secret, console login, or enable password. If there is no enable secret password,
note the enable and console login passwords. If the enable and console login passwords are not
encrypted, go to Step 16.
Caution Do not execute the next step unless you have determined you must change or replace the enable, enable
secret, or console login passwords. Failure to follow the steps as shown might cause you to erase the
router configuration.
Step 12 Enter the copy startup-config running-config command to load the startup configuration file into
running memory. This action allows you to modify or replace passwords in the configuration.
Router# copy startup-config running-config
Step 13 Enter the privileged EXEC command configure terminal to enter configuration mode:
Router# configure terminal
Change only the passwords necessary for your configuration. You can remove individual passwords by
using the no form of the above commands. For example, entering the no enable secret command
removes the enable secret password.
Step 15 You must configure all interfaces to avoid having the system be administratively shut down:
Router(config)# interface fastethernet 0/0
Router(config-int)# no shutdown
Enter the equivalent commands for all interfaces that were originally configured. If you omit this step,
all interfaces are administratively shut down and unavailable when the router is restarted.
Step 16 Use the config-register command to set the configuration register to the original value noted in Step 3
or Step 7, or to the factory default value 0x2102.
Router(config)# config-register 0x2102
Step 17 Press Ctrl-Z (hold down the Control key while you press Z) or enter end to exit configuration mode
and return to the EXEC command interpreter.
Caution Do not execute the next step unless you have changed or replaced a password. If you skipped Step 12
through Step 15, go to Step 19. Failure to observe this caution causes you to erase the router
configuration file.
Step 18 Enter the copy running-config startup-config command to save the new configuration to NVRAM.
Step 19 Enter the reload command to reboot the router.
Step 20 Log in to the router using the new or recovered passwords.
The following are lists of the commands that are present but not supported for the
Cisco uBR10012 router in various releases of the Cisco IOS software.
• Unsupported Frame Relay Commands, page A-1
• HCCP Commands, page A-2
• MLPPP Commands, page A-2
• Unsupported MPLS VPN Commands, page A-3
• Unsupported PPP Commands, page A-3
• Spectrum Management Commands, page A-3
• Unsupported Telco-Return Commands, page A-3
Warning Cisco strongly advises against using unsupported Cisco IOS commands, even if described, because such
commands can have undesirable effects upon the performance of the Cisco CMTS. In particular, Cisco advises
against using any unsupported commands that pertain to service-policy or to Modular Quality of Service
command-line interface (MQC) while Parallel Express Forwarding (PXF) is running on the Cisco CMTS.
Such commands may cause the Cisco CMTS to hang with unpredictable recovery times.
HCCP Commands
The following commands are supported in Cisco IOS Release 12.2(8)BC2 and later 12.2 BC releases.
These commands are not supported in previous releases.
• hccp authenticate
• hccp authenticate key-chain
• hccp ds-switch
• hccp lockout
• hccp protect
• hccp revert
• hccp reverttime
• hccp switch
• hccp timers
• hccp track
• hccp unlockout
• hccp working
• show hccp
• show hccp interface
• debug hccp authentication
• debug hccp events
• debug hccp sync
MLPPP Commands
The following commands are supported in Cisco IOS Release 12.2(4)BC1 and later 12.2 BC releases.
These commands are not supported in previous releases.
• multilink load-threshold
• multilink max-fragments
• ppp multilink fragment-delay
• ppp multilink fragmentation
• ppp multilink idle-link
• ppp multilink interleave
Table B-1 lists the basic tools and test equipment necessary to perform general maintenance and
troubleshooting tasks on the Cisco uBR10012 router.
Equipment Description
Number 2 Phillips and flat-head Small and medium-sized.
screwdrivers
Voltage tester Refer to the “Testing with Digital Multimeters and Cable
Testers” section on page B-1.
Optical fiber test equipment Refer to the “Testing with Digital Multimeters and Cable
Testers” section on page B-1.
Cable testing equipment Refer to the “Testing with Digital Multimeters and Cable
Testers” section on page B-1.
ESD-preventive wrist or ankle —
strap with connection cord
The following sections describe advanced testing equipment to aid in complex problem isolation.
Test fiber-optic cable both before installation (on-the-reel testing) and after installation. Continuity
testing of the fiber requires either a visible light source or a reflectometer. Light sources capable of
providing light at the three predominant wavelengths, 850 nanometers (nm), 1300 nm, and 1550 nm, are
used with power meters that can measure the same wavelengths and test attenuation and return loss in
the fiber.
BERT B-3
D
bit error rate testers B-3
BLERT B-3 DC PEM faults 2-3 to 2-5
block error rate testers B-3 debug messages 3-11
blower failures 2-7 digital multimeters B-1
see also show commands displaying environmental information 1-4
blower module faults 2-7 displaying software version 1-3
breakout boxes B-3 documentation viii
bus errors 3-13
line cards 4-5
E