System Health Monitoring

Monitoring critical system resources is very important to maintain stability of the network. We
recommend that you monitor the switch CPU, memory, file systems, and environmental resources on a
regular basis.
This workflow discusses the commonly used commands and procedures to monitor and maintain system
health.

Prerequisites for System Health Monitoring

Obtain information about your switch such as the running software release, duration of switch run time,
and the reason for the most recent reload. To obtain this information, use the show version command.
The command with the pipe feature gives the duration of uptime and any reload information.

show version|inc software|uptime|Last

Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software
(CAT3K_CAA-UNIVERSALK9-M), Version 03.03.02.SE RELEASE SOFTWARE (fc2)
3850-access-Bld1Flr1 uptime is 5 weeks, 3 days, 2 hours, 59 minutes
Last reload reason: reload

Show Running Status

Identify the reasons for uptime and reload. Over time, switches can crash and reload without your
knowledge.

Step 1 Use the show version command to retrieve the overall switch status.
If you are only interested in the switch uptime and last reload, you can run a more direct command using
the pipe “|” feature built into Cisco IOS XE (and Cisco IOS) software.
This example shows that Cisco IOS XE release 3.3.2 SE was running for five weeks before a privileged
user initiated a switch reload.

Cisco Systems, Inc.

www.cisco.com
 System Health Monitoring
Run a System Baseline for Core Resources

show version|inc software|uptime|Last

Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software
(CAT3K_CAA-UNIVERSALK9-M), Version 03.03.02.SE RELEASE SOFTWARE (fc2)

3850-access-Bld1Flr1 uptime is 5 weeks, 3 days, 2 hours, 59 minutes

Last reload reason: reload

Run a System Baseline for Core Resources

Set your system baseline usage during normal production time and determine if there is a change from
your expected resource values. If the increase in usage is not justified, investigate to find the cause.
Ideally, it is best to setup some form of Network Monitoring System (NMS) to automatically monitor
these values, however it is also important to learn how to manually poll these values.
After you have identified the switch running status, examine core resources to ensure that they are all at
optimal values.

Obtain CPU and Core Processor Usage

Step 2 Use the show process cpu command to display CPU and core processor usage.
To find CPU usage due to the subprocesses and tasks operating under a specific process, use the show
process cpu detailed command. To sort for high activity usage, use show process cpu sorted command.
CPU usage can be monitored on a per-switch basis in a stacked environment.
At periodic intervals, we recommend that you run the following variations of the show process cpu
command.

Note The switch is a multicore platform that is different from its predecessors. A single core can experience
high CPU, so it is important to monitor each core when running these commands.

This output shows the five-second, one-minute, and five-minute periods on each CPU core. It also
shows the Forwarding Engine Driver (FED), IOS daemon IOSd, and Wireless Controller Module
(WCM) processes have the highest CPU utilization.

Best Practice User Guide for the Catalyst 3850 and Catalyst 3650 Switch Series
104
System Health Monitoring
Run a System Baseline for Core Resources

show process cpu sorted | ex 0.00

Core 0: CPU utilization for five seconds: 4%; one minute: 5%; five minutes:
5%
Core 1: CPU utilization for five seconds: 2%; one minute: 1%; five minutes:
1%
Core 2: CPU utilization for five seconds: 0%; one minute: 0%; five minutes:
0%
Core 3: CPU utilization for five seconds: 1%; one minute: 2%; five minutes:
1%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
5639 1598657 15898882 68 0.98 1.06 1.08 1088 fed
8503 1554112 10180648 52 0.54 0.50 0.44 0 iosd
8499 982266 14501353 18 0.20 0.15 0.15 0 wcm
5640 427135 54197163 16 0.05 0.10 0.11 0
platform_mgr
6170 502150 9040937 55 0.05 0.01 0.01 0 obfld
6177 2057130 87345912 23 0.05 0.01 0.03 0 pdsd

Step 3 Use the history command to display a graph of sustained CPU utilization.
This graph helps to formulate patterns. For example, if you observe a spike to 100 percent every 30
minutes, you can conclude that something might be polling the switch on a regular schedule. Examine
your SNMP configuration to help determine the cause.

show process cpu history

History information for system:

1111122222222222222222222
111111111111111111111111111111222225555588888888886666666666
100
90
80
70
60
50
40
30 ********************
20 *************************
10 *************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)

Reference:
For detailed information to help troubleshoot your high CPU usage concerns, see the Catalyst 3850
Series Switch High CPU Usage Troubleshooting document.

Best Practice User Guide for the Catalyst 3850 and Catalyst 3650 Switch Series
105
 System Health Monitoring
Run a System Baseline for Core Resources

Obtain Switch Memory Usage

Step 4 Use the show process memory command to display the state of memory usage on your switch.
To find memory usage due to the subprocesses and tasks operating under a specific process, use the show
process memory detailed command. To sort for high activity usage, use the show process memory
detailed sorted command.
Memory usage can be monitored on a per-switch basis in a stacked environment.

show process memory sorted

System memory : 3930840K total, 1487028K used, 2443812K free, 222004K kernel
reserved
Lowest(b) : 1915568076
PID Text Data Stack Heap RSS Total Process
5681 9988 269088 92 476 233060 584844 fed
10162 72268 34364 104 288 206548 343980 iosd
10158 24260 519732 88 10628 108612 662328 wcm

Monitor File Systems Usage

Step 5 At regular intervals, use the show file systems command to monitor the file systems within the switch
to ensure that there is always sufficient space available.
Unlike previous platforms, the switch writes crash files to a separate directory. For example, the show
file systems command output shows that the crashifo folder is populated. Compare the size of the folder
against the free space available.
The switch has different file systems that can be listed by using the show file systems command.

show file systems

File Systems:

Size(b) Free(b) Type Flags Prefixes

248354816 148799488 disk rw crashinfo: crashinfo-1:
248512512 178782208 disk rw crashinfo-2: stby-crashinfo:
* 1621966848 346673152 disk rw flash: flash-1:
1622147072 350224384 disk rw flash-2: stby-flash:

Note An (*) asterisk indicates the default file system. If the file system has a dash (-) or a zero (0) for the
Size(b) field, that indicates that the file system is not present or not recognized.

Step 6 Use the dir filesystem or the show filesystem command to list the files under a specific files system.
When you find crash files, it is important to immediately retrieve them to diagnose a system failure or
unexpected crash.

Best Practice User Guide for the Catalyst 3850 and Catalyst 3650 Switch Series
106
System Health Monitoring
Run a System Baseline for Environmental Resources

This example shows that crash files were created in the directory.
dir crashinfo
Directory of crashinfo:/

6073 drwx 1024 Jul 17 2013 17:53:48 +00:00 ap_crash

12 -rwx 0 Jan 1 1970 00:00:06 +00:00 koops.dat
11 -rwx 357 Jun 1 2014 13:05:15 +00:00 last_systemreport_log
13 -rwx 1128623 Nov 22 2013 12:33:27 +00:00
system-report_2_20131122-123229-UTC.gz
14 -rwx 39 Jun 1 2014 13:05:15 +00:00 last_systemreport
15 -rwx 657766 Jun 5 2013 09:17:03 +00:00
system-report_1_20130605-091616-UTC.gz
16 -rwx 737390 Jun 26 2013 22:48:22 +00:00
system-report_1_20130626-224726-UTC.gz

Run a System Baseline for Environmental Resources

Step 7 Use the show environment command to display an overview of switch health.
It is important to monitor environmental resource values because something as small as a fan failure can
lead to a serious hardware problem. If your switches provide Power Over Ethernet (POE), then the show
environment command will also provide a view into the power supplies and if they are performing as
expected.

show environment all

Switch 1 FAN 1 is OK
Switch 1 FAN 2 is OK
Switch 1 FAN 3 is OK
FAN PS-1 is OK
FAN PS-2 is OK
Switch 1: SYSTEM TEMPERATURE is OK
SW PID Serial# Status Sys Pwr PoE Pwr Watts
-- ------------------ ---------- --------------- ------- ------- -----
1A PWR-C1-715WAC LIT171310MT OK Good Good 715
1B PWR-C1-715WAC LIT171310PS OK Good Good 715

Step 8 If your switches are in a stack, run the show environment stack command to view all of the
environmental outputs stack wide.
Although some of settings are adjustable, we recommend leaving the settings with their default values.

Best Practice User Guide for the Catalyst 3850 and Catalyst 3650 Switch Series
107
 System Health Monitoring
Other System Monitoring Considerations

show environment stack

SWITCH: 1
Switch 1 FAN 1 is OK
Switch 1 FAN 2 is OK
Switch 1 FAN 3 is OK
FAN PS-1 is OK
FAN PS-2 is OK
Switch 1: SYSTEM TEMPERATURE is OK
Inlet Temperature Value: 34 Degree Celsius
Temperature State: GREEN
Yellow Threshold : 41 Degree Celsius
Red Threshold : 56 Degree Celsius
Hotspot Temperature Value: 45 Degree Celsius
Temperature State: GREEN
Yellow Threshold : 105 Degree Celsius
Red Threshold : 125 Degree Celsius
SWITCH: 2
Switch 2 FAN 1 is OK
Switch 2 FAN 2 is OK
.
.
.

Other System Monitoring Considerations

Spanning Tree Monitoring

Spanning tree design is beyond the scope of this document, however, the goal of this procedure is to
provide simple spanning tree monitoring commands. It is important to always understand your spanning
tree topology within your network. There are a number of simple commands that you can run to verify
that your switch is performing the expected spanning tree role.
Step 9 Use the show spanning-tree summary command to periodically monitor the stability of your spanning
tree environment and ensure a loop-free environment.
This example output shows that the switch is actually operating as the root bridge for all of the VLANs
which can cause extreme network degradation if incorrectly configured.

Best Practice User Guide for the Catalyst 3850 and Catalyst 3650 Switch Series
108
System Health Monitoring
Other System Monitoring Considerations

show spanning-tree summary

Switch is in pvst mode
Root bridge for: VLAN0001, VLAN0011, VLAN0015, VLAN0100-VLAN0101
VLAN0881-VLAN0883
Extended system ID is enabled
Portfast Default is disabled
PortFast BPDU Guard Default is disabled
Portfast BPDU Filter Default is disabled
Loopguard Default is disabled
EtherChannel misconfig guard is enabled
UplinkFast is disabled
BackboneFast is disabled
Configured Pathcost method used is short

Name Blocking Listening Learning Forwarding STP Active

---------------------- -------- --------- -------- ---------- ----------
VLAN0001 0 0 0 2 2
VLAN0011 0 0 0 1 1
VLAN0015 0 0 0 1 1
VLAN0100 0 0 0 1 1
VLAN0101 0 0 0 1 1
VLAN0777 0 0 0 2 2
VLAN0881 0 0 0 1 1

Name Blocking Listening Learning Forwarding STP Active

---------------------- -------- --------- -------- ---------- ----------
VLAN0882 0 0 0 1 1
VLAN0883 0 0 0 1 1
---------------------- -------- --------- -------- ---------- ----------
9 vlans 0 0 0 11 11

Step 10 Use the show spanning-tree detail command to frequently check STP stability.
This command displays network stability information about the number of topology changes within each
VLAN, the last time a TCN was received, and so forth. Frequently monitoring this information is critical
to maintaining overall health of the switch and network.

Best Practice User Guide for the Catalyst 3850 and Catalyst 3650 Switch Series
109
 System Health Monitoring
Other System Monitoring Considerations

show spanning-tree detail |inc ieee|occur|from|is|exec

VLAN0001 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 55 last change ed 4d07h ago
from GigabitEthernet1/0/1
VLAN0011 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 7 last change ed 4d07h ago
from GigabitEthernet1/0/1
VLAN0015 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 7 last change ed 4d07h ago
from GigabitEthernet1/0/1
VLAN0100 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 7 last change ed 4d07h ago
from GigabitEthernet1/0/1
VLAN0101 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 7 last change ed 4d07h ago
from GigabitEthernet1/0/1
VLAN0777 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 12 last change ed 4d07h ago
from GigabitEthernet1/0/1
VLAN0881 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 7 last change ed 4d07h ago
from GigabitEthernet1/0/1

Best Practice User Guide for the Catalyst 3850 and Catalyst 3650 Switch Series
110