Troubleshooting High CPU and memory related

issues on Cisco ASA platforms

Puneesh Chhabra & Aditya Ganjoo (HTTS-SECURITY)

HIGH CPU CAUSES

• Most cases of high CPU utilization occur because the Dispatch Unit
process is high. Common causes of high utilization include:
• Oversubscription
• Routing loops
• Host with a high number of connections
• Excessive system logs
• Unequal traffic distribution
Oversubscription
• In order to determine if the ASA is reaching capacity, run the show traffic command, the
clear traffic command, then the show traffic command again over a period of 1-3
minutes. Since the output from the show traffic command is an average rate, use the
clear traffic command between the show traffic commands in order to have cleared,
accurate statistics.

• Calculate the statistics from the Aggregated Traffic on Physical Interface section for
received and transmitted bytes per second and for packets per second. Compare those
statistics with the ASA model specifications in order to see if the ASA is approaching its
limits.
Routing Loops

• Routing loops are a very common cause of high CPU utilization issues,
especially with ASA 5505s, since those are devices with a very low
maxiumum throughput. Routing loops can make the ASA send a packet
where it came from, based on the routing table, and forward it again and
again endlessly until the time to live (TTL) is expired. Since the ASA does
not decrement the TTL by default, the other device that is sending the
packets directly to the ASA is the only device doing this.
Host with High Number of
Connections
If you suspect the high CPU utilization is related to the traffic through the ASA, double-check the amount of
connections/xlates with these commands:

•show conn count

•show xlate count
•show perfmon
You can sort the number of connections for each host with the show local-host | in host|count/limit
command. Output from this command displays, for each host, the number of TCP, UDP, and embryonic
connections.

As you troubleshoot an ASA on version 8.x, you can also see specific kinds of traffic. You can choose
between TCP, UDP, and embryonic connections and specify the number of connections you want to see. The
command format is:
show local-host connection { tcp | udp | embryonic } start-number | in host|count/limit

For example, in order to see the host with more than 500 embryonic connections, the command is:

show local-host connection embryonic 500 | in host|count/limit

Excessive System Logs

The ASA may generate a lot of traffic and increase CPU utilization if:

• The ASA is generating a large number of syslogs.

• There are multiple syslog servers configured at debugging level.
• There are multiple syslog servers sending logs via Simple Network Management
Protocol (SNMP) traps.
• In such cases, the SNMP Notify Thread and Logger take some of the CPU
utilization.
• Remove some of the syslog/SNMP server logs or change the severity of the logging
to a lower value in order to try to reduce utilization.
High CPU caused by Crypto Keys

High CPU caused by 2048 bit keys

Percentage of hits per function

@@@@@@@@@@@@@@@@.................................. 33.58% : _bn_mul_add_words
This is caused by the crypto/math operations since those keys are processed in software and
not hardware.

In order to make the ASA process those operations in hardware, use the crypto engine large-
mod-accel command.
CPU Hog Explanation
Show proc cpu-hog' output explanation
For each recorded CPU hog, the output includes
• The process name: Note that a given process may appear several times since CPU hogs at different locations in
the process code are counted separately. See also the PC field.
• The PC (program counter) field: The PC field contains either
- the address of the code instruction where the process suspended, in the case where the hog was recorded by the
scheduler. In this case, you will see "(suspend)" after the PC address.
OR
- the address of the code instruction when the timer interrupt fired, in the case where the hog was recorded by the
timer interrupt handler. In this case you will see "(interrupt)" after the PC address. This value can be decoded
using the ASA traceback decoder.
• The call stack field: Not always present. The sequence of function calls on the process stack that led to the
current execution point (PC). This value can be decoded just like a crash traceback using the ASA traceback
decoder.
• Either a PROC_PC_TOTAL value or a NUMHOG value: A PROC_PC_TOTAL value is printed if no
traceback has been recorded for this hog. A NUMHOG value is printed if we do have a traceback for the hog. In
both cases, the value indicates the number of times this hog was recorded
CPU Hog Explanation

We note that often we will see hogs recorded for the same PC value, both with and without a stack
trace.This is because the ASA rate-limits traceback recording, since this is, in itself, an expensive
operation in terms of CPU cycles.
• The MAXHOG field gives the duration of the longest recorded hog with this PC value, in
milliseconds.
• The LASTHOG field gives the duration of the last recorded hog with this PC value, in milliseconds.
• The LASTHOG At field supplies the timestamp of the last recorded instance of this hog

The output also includes -

• The CPU hog threshold : This is a platform specific value that defines the length of time a
process must have been running for a hog to be recorded
• Last cleared : A timestamp for when the user last entered the exec command clear process cpu-
hog
Sample Output
ciscoasa# show process cpu-hog
• Process: Dynamic Filter updater, PROC_PC_TOTAL: 144, MAXHOG: 12, LASTHOG: 11
• LASTHOG At: 06:30:16 UTC Nov 18 2012
• PC: 8ceb7bc (suspend)

• Process: Dynamic Filter updater, NUMHOG: 144, MAXHOG: 12, LASTHOG: 11

• LASTHOG At: 06:30:16 UTC Nov 18 2012
• PC: 8ceb7bc (suspend)
• Call stack: 8ceb7bc 8d00bc8 8d01d01 8e14f13 8e16139 8e16e02 8062873

• Process: Dynamic Filter updater, PROC_PC_TOTAL: 144, MAXHOG: 1789, LASTHOG: 1783
• LASTHOG At: 06:30:16 UTC Nov 18 2012
• PC: 8e16d2a (suspend)
Maximum ACL Limits

ACL table size is only bound by available memory

• Compiled into binary structure, no performance advantage from order
• Each ACE uses a minimum of 212 bytes of RAM
• Connection rate is impacted beyond maximum recommended values

Issue show access-list | include elements to see how many ACEs you have
ACE Optimisation

• ACE Explosion with Object Groups

• All configured ACLs are expanded before programming
• Nested Object Groups magnify the impact – Add a new source Object
Group with 25 additional objects – Result: (10+25) x 21 x 33 = 24,255
rules (ACEs)  ACL Optimisation prevents the Object Group expansion –
Significant reduction in memory utilisation, not so much on CPU
 CPU monitoring
On ASA we have several tools to monitor CPU utilization, but none of them is perfect.

We have a syslog message which is produced in admin context when CPU utilization reaches 95%
or more and stays there for a period of 5 minutes:

%ASA-2-321005: System CPU utilization reached 95%

Since 8.4.1 we have SNMP traps with configurable "rising" threshold value and
configurable measurement interval:

snmp cpu trap threshold rising <10-94%> <minutes>

snmp-server enable traps cpu threshold rising
The default threshold is 70% and the interval is 1 minute. Also, there is another hardcoded 95%
threshold which cannot be disabled. The system sends SNMP trap when any of the above
thresholds is crossed, if "snmp-server enable traps cpu threshold rising" and "snmp-server”
are configured.
Use the CPU Profiler
• The CPU Profiler is a valuable tool that can help you determine which process
consumes more CPU.
• A CPU profile captures the address of the process that was running on the CPU
when the timer interrupt fires. This occurs every 10 ms, regardless of CPU
load. For example, 5,000 samples take exactly 50 seconds. The samples are
stored in an array of size n-samples.
• Use the cpu profile activate n-samples command in order to enable the
profiler; n-samples is 1,000 by default.
• If you enter the cpu profile activate command while the CPU Profiler is
collecting data, the profiler notifies you that the collection is in progress.Once
data collection is complete, the profiler reports that it is done and notes the
number of samples. You can check this with the show cpu profile command
 High memory Causes

• Event logging: Event logging can consume large amounts of memory. In order to resolve this issue,
install and log all events to an external server, such as a syslog server.
• Memory Leakage: A known issue in the security appliance software can lead to high memory
consumption. In order to resolve this issue, upgrade the security appliance software.
• Debugging Enabled: Debugging can consume large amounts of memory. In order to resolve this
issue, disable debugging with the undebug all command.
• Blocking Ports: Blocking ports on the outside interface of a security appliance cause the security
appliance to consume high amounts of memory to block the packets through the specified ports.In
order to resolve this issue, block the offending traffic at the ISP end.
• Threat-Detection: The threat detection feature consists of different levels of statistics gathering for
various threats, as well as scanning threat detection, which determines when a host is performing a
scan. Turn off this feature to consume less memory.
 Troubleshooting high Memory

Show memory can be utilized to view the current utilization of an ASA:

hostname# show memory Free memory: 87764304 bytes (33%) Used memory: 180671152
bytes (67%) ------------- ---------------- Total memory: 268435456 bytes (100%)

The above indicates that 33% of memory (87 MB) is free. The memory above is measured in
bytes.

The total memory of the box is 256MB, or 256 x 1024 x 1024 = 268435456 bytes.
 Helpful Commands

 Show memory detail

 Show memory top-usage

 show memory binsize x

From the outputs of "show memory binsize x", generate a list of PCs that make a lot of calls for
memory in your "suspect binsizes". These are your "suspect PCs".
 Types of High Memory on ASA

General consumption: In these situations the memory is not being leaked but the memory
consumption is expected as a result of certain features (ex: Threat Detection, VPN, WebVPN)
or capacity (high amount of/conncurrent connections/xlates).

Memory leak: A bug in the code causes memory to be allocated by a feature or process, but
not freed (or returned to the pool) when the feature no longer needs it. Over time, this cycle
repeats and the memory on the box is depleted and problems arise, such as packet forwarding
failures.
Enabling memory Tracking

Memory Tracking
• Issue the command "show memory tracking" at regular intervals to see the change in
memory allocation
• Issue the command "show memory tracking address | i <PC-Counter>" where <PC-
Counter> is the pc counter (in hex) of the largest growing process from the previous step
• Gather the output "show memory tracking dump <address>" for any of the memory
address locations picked at random from the output of the previous step
Block Exhaustion
• show blocks
• show blocks old (several pages of this)
• show blocks pool <size> dump (several pages of this) ---> be careful when you run this as this can
generate lot of output especially for block size 1550 - not recommended to run on console unless you
have to
• show blocks old dump | b size <size> (several pages of this) --->be careful when you run this as this
can generate lot of output especially for block size 1550 - not recommended to run on console unless
you have to
• show block address <address_of_block> dump (if you identify a block to investigate)
• show blocks queue history detail
• show blocks queue history core-local
• show blocks old core-local
• show blocks exhaustion snapshot
• show blocks assigned
Block Exhaustion
Memory Related Symptoms
• Unable to send packets (SNMP, Syslogs, etc)
• Unable to processing traffic and forward packets through the ASA
• Unable to save the configuration
• Unable to synchronize failover
• Unable to perform URL filtering to a websense server
• Console messages printed to the screen stating "process_create: out of stack
memory"
• Unable to create new connections through the ASA
• The ASA will reload by itself and possibly write a crashinfo file
• Unable to telnet, SSH or ASDM to the ASA
• Unable to add new ACL elements
Questions ?????
Thanks