SAP Knowledge Base Article

2100040 - FAQ: SAP HANA CPU

Component: HAN-DB-PERF (SAP HANA > SAP HANA Database > SAP HANA Database Performance), Version: 272, Released On: 23.03.2025

Symptom
You want to understand the CPU consumption of SAP HANA.

Environment
SAP HANA

Cause
1. Which indications exist for SAP HANA CPU problems?
2. Where do I find more information related to SAP HANA CPU analysis?
3. Is a high CPU consumption generally an issue?
4. How can I collect information about the CPU consumption?
5. How can additional CPU resource tracking be activated?
6. Is it possible to capture SQL statements with a particularly high CPU consumption?
7. What is the difference between system, user and I/O wait CPU consumption?
8. How can the CPU consumption of SAP HANA hosts be optimized?
9. How can CPU intensive operations in SAP HANA be identified and optimized?
10. Is it possible to limit the amount of consumed CPU?
11. How can I determine physical CPU details on SAP HANA side?
12. How can I trigger automatic follow-up actions in case of a high CPU consumption?
13. Is the CPU consumption shown by SAP HANA always correct?
14. Can the available CPU resources be limited on OS level?
15. Is it recommended to use hyperthreading in SAP HANA environments?
16. Which thread states and call stacks indicate CPU consumption?
17. How can the CPU performance of the SAP HANA database server be tested and compared?
18. What are reasons for slowness in combination with a limited CPU consumption?
19. What can be done in case of high CPU consumption in SAP HANA Cloud environments?
20. How can CPU spikes be analyzed?

Resolution

1. Which indications exist for SAP HANA CPU problems?

The administration overview screen or the load graph of SAP HANA Studio shows a high current or historic CPU consumption.
The following SAP HANA alerts indicate problems in the CPU area:

Alert Name SAP Note Description

5 Host CPU Usage 1909670 Determines the percentage CPU idle time on the host and therefore whether or not CPU resources are running low.

SQL: "HANA_Configuration_MiniChecks" (SAP Notes 1969700, 1999993) returns a potentially critical issue (C = 'X') for one of the following individual checks:

Check ID Details

M0210 Minimum CPU rate (MHz)

M0211 Hosts with varying CPU rates

M0212 CPU discrepancy host info vs. host agent

M0213 Missing host agent metrics

M0215 Hosts with outdated CPU type

M0220 Current CPU utilization (%)

M0221 Peak CPU utilization (%, last day)

M0222 Time since CPU utilization > 95 % (h)

 M0226 Peak system CPU utilization (%, last day)

M0227 External CPU utilization (%, last hour)

M0228 Erroneous system CPU calculation

M0229 Discrepancy between NUMA nodes and sockets

M0232 Hyperthreading active in critical context

M0233 NUMA nodes with high CPU utilization

M0234 CPU steal share (%)

M0893 Users with high thread limit

SQL: "HANA_Threads_Callstacks_MiniChecks" (SAP Notes 1969700, 2313619) reports critical check IDs.
The HDBCPUUsage alert is raised in SAP HANA Cloud environments.
A high amount of database threads are active with status "Running" (see SAP Note 2000000).

2. Where do I find more information related to SAP HANA CPU analysis?

The SAP HANA Troubleshooting and Performance Analysis Guide contains - among others - information about analyzing and optimizing the SAP HANA CPU consumption.

3. Is a high CPU consumption generally an issue?

A high CPU consumption can be normal and acceptable, because SAP HANA uses available CPUs efficiently by parallelizing complex operations. Therefore, a high CPU consumption can be intended and a consequence of a massively parallelized processing of complex operations. If on the other
hand simple database requests or other processes consume a high amount of CPU, this can result in a critical CPU bottleneck situation which can be avoided. This SAP Note provides information how to analyze and optimize unnecessary CPU consumption.

4. How can I collect information about the CPU consumption?

The following options exist to check the current CPU utilization of the SAP HANA database server:
SAP HANA Studio -> Administration -> Overview -> CPU Usage
SAP HANA Studio -> Administration -> Performance -> Load -> [System] CPU
The following SQL statements are available via SAP Note 1969700 in order to understand the CPU consumption of the SAP HANA hosts:

SQL statement name Description

SQL: "HANA_Resources_CPU_UtilizationPerCore" Current CPU utilization per core, breakdown to NUMA node possible

SQL: "HANA_Resources_CPUAndMemory_CurrentDet Current CPU information based on Host Agent information

ails"

SQL: "HANA_Resources_CPUAndMemory_History" Among others this SQL statement is able to check for current and historic CPU consumption on a granular basis. It displays the CPU utilization (in %), the number of CPUs busy with user activities, the number of CPUs busy with system activities and the n
umber of idle CPUs.

SQL: "HANA_Configuration_InfrastructureReport" This command provides information about the used CPUs including frequency and number.

SQL: "HANA_LoadHistory_Hosts" Host wide CPU consumption (SAP Note 2222110)

SQL: "HANA_LoadHistory_Services" Indexserver CPU information (total, system) (SAP Note 2222110)

Additionally you should use operating system tools (e.g. top) in order to understand which processes are responsible for the highest CPU consumption.

5. How can additional CPU resource tracking be activated?

An exact measurement of CPU consumption of a SAP HANA request isn't feasible because a lot of often parallel and short-running threads can contribute to the execution and activating a precise measurement would be a significant overhead.
As of SAP HANA 1.0 SPS 12 a CPU approximation based on the states of active threads is used. When an active thread doesn't have to wait (e.g. for locks, I/O or network), it is supposed to consume CPU. See SAP Note 2114710 for more information related to SAP HANA threads. As a
consequence, you always have CPU figures available without having to activate specific measurements. Indeed, there is no other way to activate CPU time measurement mode with SAP HANA 1.0 SPS 12 and higher.
The following monitor view columns contain CPU information:
M_SERVICE_THREADS.CPU_TIME_SELF
M_SERVICE_THREADS.CPU_TIME_CUMULATIVE
M_EXPENSIVE_STATEMENTS.CPU_TIME

6. Is it possible to capture SQL statements with a particularly high CPU consumption?

You can capture SQL statements exceeding a defined CPU consumption via expensive statements trace (see SAP Note 2119087) using the following parameters:

Parameter Default Unit Details

false Main switch, 'true' activates the expensive statement trace

global.ini -> [expensive_statement] -> enable
 us Threshold for minimum CPU utilization in micro seconds (Rev. >= 94)
global.ini -> [expensive_statement] -> threshold_cpu_time

7. What is the difference between system, user and I/O wait CPU consumption?
CPU utilization is typically separated in system CPU, user CPU and I/O wait CPU consumption:

CPU utilization type Description

User CPU consumption originating from coding outside of the operating system kernel, e.g.:

Expensive operation in SAP HANA like scan of large table or counting a high amount of distinct values
Expensive operation of non-SAP HANA product running on same host

System CPU consumption originating from the operating system kernel, e.g.:

Operating system background activities like memory defragmentation

Implicit OS calls of SAP HANA like active waits for locks or I/O requests

I/O Wait CPUs waiting for I/O requests to complete, can usually be considered as idle time

8. How can the CPU consumption of SAP HANA hosts be optimized?

The following approaches exist to optimize CPU consumption and reduce the risk of running into CPU bottlenecks:

Possible symptoms Optimization Optimization Details

High system CPU consumption ca Optimize OS configurat Make sure that the OS recommendations of SAP Note 2000003 ("How can the configuration and performance of the SAP HANA hardware, firmware and operating system be checked?") are implemented.
used by OS processes ion
A more detailed analysis of activities involving system CPU consumption can be performed via OS kernel stack backtrace as described in SAP Note 2166414. Additionally the Linux perf package can be used to trace operating system activities, e.g. "perf t
op ..." for generating and displaying a performance counter profile in real time.
High system CPU consumption ca Optimize OS calls trigg System CPU consumption related to SAP HANA traces is typically caused by the following scenarios:
used by SAP HANA processes ered by SAP HANA
Active (yield) waits
Frequent executions of expensive operating system calls
Implicit triggering of undesired low-level activities like TLB shootdowns
External software using Linux kernel taints, e.g. Cortex XDR, Crowdstrike Falcon, Imperva, K-TAP (e.g. as part of Guardium), Microsoft Dependency Agent or Trend Micro (tmhook)
External software using Linux kprobe interface (see kernel probes for details), e.g. the SentinelOne agent or the system-probe component of Datadog Agent
External software triggering network packet capturing, e.g. Dynatrace OneAgent (oneagentnetwork)
Configured Linux syscall auditing
Syscall interpositioning, e.g. Dynatrace OneAgent

See "How can CPU intensive operations in SAP HANA be identified and optimized?" for more details.

For kernel taints and kprobe instrumentation the call stacks are often network related (methods like __libc_recv, __libc_send, __writev, __writev_nocancel) and the threads frequently show "no SQL (NoAction)" (SAP Note 2114710).

In case the SAP HANA call stacks don't contain relevant information, a more detailed analysis of activities involving system CPU consumption can be performed via OS kernel stack backtrace as described in SAP Note 2166414. Additionally the Linux pe
rf package can be used to trace operating system activities, e.g. "perf top ..." for generating and displaying a performance counter profile in real time or "sar -P all 1".
For supportability considerations when running 3rd party solutions see SAP Note 3359552.

Below you can find typical scenarios of system CPU consumption of SAP HANA introduced by activities and configurations on lower layers.

Linux kernel taints:

Linux kernel taints can be identified by sapsysinfo.sh available via SAP Note 618104. Kernel taints or often no problem, but taints flagged with N (unsupported), E (unsigned) or P (proprietary) should be checked more in detail. A closer look at the OS k
ernel stack backtrace is useful to identify suspicious taints. For example, the following call stack indicates a spin lock (_raw_spin_lock) being called by a module called krg_622832_imSLE15sp3_KS_SMP64:

_raw_spin_lock+0x1b/0x20
pseudolock_trylock_read+0x32/0x1a0 [krg_622832_imSLE15sp3_KS_SMP64]
? memory_manager_alloc+0x70/0x4f0 [krg_622832_imSLE15sp3_KS_SMP64]
? readv_pre_code+0xec0/0xec0 [krg_622832_imSLE15sp3_KS_SMP64]
? writev_pre_code+0xd1/0xeb0 [krg_622832_imSLE15sp3_KS_SMP64]
? cshook_network_ops_inet6_sockraw_recvmsg+0x185a9/0x21b10 [falcon_lsm_serviceable]
? cshook_systemcalltable_pre_ia32_pread64+0x23eb7/0x2cf40 [falcon_lsm_serviceable]
? indexes_bit_mask_is_index_set+0xb8/0x280 [krg_622832_imSLE15sp3_KS_SMP64]
? _generic_fd_write_postcode+0x2e0/0x2e0 [krg_622832_imSLE15sp3_KS_SMP64]
? _orig_write+0x210/0x210 [krg_622832_imSLE15sp3_KS_SMP64]
? readv_pre_code+0xec0/0xec0 [krg_622832_imSLE15sp3_KS_SMP64]
? syscall_wrappers_generic_flow_with_param+0x6ee/0x830 [krg_622832_imSLE15sp3_KS_SMP64]
? syscall_wrappers_generic_flow_with_param+0x6ee/0x830 [krg_622832_imSLE15sp3_KS_SMP64]
? SYS_writev_common_wrap+0x86/0x100 [krg_622832_imSLE15sp3_KS_SMP64]
? SYS_writev_common_wrap+0x86/0x100 [krg_622832_imSLE15sp3_KS_SMP64]
? unload_network_ops_symbols+0x52f3/0x70f0 [falcon_lsm_pinned_13707]
? do_syscall_64+0x5b/0x1e0
? entry_SYSCALL_64_after_hwframe+0x44/0xa9

Spin locks in the Linux kernel consume system CPU. The sapsysinfo.sh output helps to map the module to an external software of company Imperva. After having deactivated the related tool, the system CPU consumption went down to reasonable level
s. With this result the external vendor can be approached, asking for an optimization.

As another example, the following call stack indicates spin locking in context of the Microsoft Dependency Agent:

queued_spin_lock_slowpath+0x7/0xe
_raw_spin_lock_irqsave+0x36/0x40
bs_acquire_spinlock+0xe/0x20 [microsoft_dependency_agent]
hash_table_get+0x26/0x90 [microsoft_dependency_agent]
bs_is_flow_tracing_disabled+0xa/0x30 [microsoft_dependency_agent]
bs_handle_tcp_xfer+0xb5/0x190 [microsoft_dependency_agent]
? tcp_sendmsg_locked+0x314/0xb50
inet_sendmsg_rethook+0xa0/0xc0 [microsoft_dependency_agent]

This specific issue was resolved after an upgrade of the Microsoft Dependency Agent from version 9.3 to version 9.10.

Yet two other example call stacks, related to K-TAP (e.g. used in context of Guardium) and Trend Micro (tmhook):

_raw_spin_lock_irqsave+0x3b/0x50
__wake_up_common_lock+0x61/0xc0
gb_copy_done+0x1c/0x60 [ktap_115135 34a4b0de797b7391d7858aff0ee9d90d54d29e40]
do_gb_put_PDU+0x12e0/0x1d30 [ktap_115135 34a4b0de797b7391d7858aff0ee9d90d54d29e40]
? sk_reset_timer+0x14/0x60
? tcp_schedule_loss_probe+0xf5/0x180
gb_put_PDU+0x343/0x3e0 [ktap_115135 34a4b0de797b7391d7858aff0ee9d90d54d29e40]
ktap_post_process+0x2f9/0x4b0 [ktap_115135 34a4b0de797b7391d7858aff0ee9d90d54d29e40]
? release_sock+0x43/0x90
ktap_tcpv4_sendmsg+0x541/0x840 [ktap_115135 34a4b0de797b7391d7858aff0ee9d90d54d29e40]
ktap_tcpv4_sendmsg_4_1+0x21/0x30 [ktap_115135 34a4b0de797b7391d7858aff0ee9d90d54d29e40]
ktap_tcpv4_sendmsg_4_1_wrapper+0x3e/0x50 [ktap_115135 34a4b0de797b7391d7858aff0ee9d90d54d29e40]
? sock_sendmsg+0x5f/0x70
 _raw_spin_lock+0x1e/0x30
tmhook_invoke_posthooks+0x5a/0xb0 [tmhook]
tmhook_handler+0x1be/0x2a0 [tmhook]
? do_syscall_64+0x5b/0x1e0
? tmhook_invoke+0xa0/0xa0 [tmhook]
do_syscall_64+0x5b/0x1e0
entry_SYSCALL_64_after_hwframe+0x61/0xc6

Linux kprobe interface

Tools using the Linux kprobe interface don't show up in the OS kernel stack backtrace. Instead "kprobe" or "kretprobe" is visible. For example, the following call stacks indicate spin locks related to kprobe:

_raw_spin_lock_irqsave+0x36/0x50
pre_handler_kretprobe+0xfc/0x1c0
? tcp_sendmsg+0x5/0x40
aggr_pre_handler+0x40/0x80
? tcp_sendmsg+0x1/0x40
kprobe_ftrace_handler+0xab/0x100
ftrace_ops_assist_func+0xb1/0x150
? tcp_sendmsg_locked+0xbd0/0xbd0
? tcp_sendmsg+0x1/0x40
tcp_sendmsg+0x5/0x40

_raw_spin_lock+0x20/0x30
__pcpu_freelist_pop+0x82/0xb0
alloc_htab_elem+0x1f9/0x250
htab_map_update_elem+0x2a2/0x350
? tcp_sendmsg+0x5/0x40
? tcp_sendmsg+0x5/0x40
? trace_call_bpf+0x6d/0x90
? kprobe_perf_func+0x17b/0x230
? tcp_sendmsg+0x5/0x40
? kprobe_dispatcher+0x2f/0x50
? aggr_pre_handler+0x40/0x80
? tcp_sendmsg+0x1/0x40
? kprobe_ftrace_handler+0xab/0x100
? ftrace_ops_assist_func+0xb1/0x150
? tcp_sendmsg_locked+0xbd0/0xbd0
? tcp_sendmsg+0x1/0x40
? tcp_sendmsg+0x5/0x40

_raw_spin_lock+0x20/0x30
recycle_rp_inst+0x46/0xa0
? copy_oldmem_page_encrypted+0x20/0x20
? sock_sendmsg+0x3a/0x50
trampoline_handler+0x1f9/0x230
kretprobe_trampoline+0x25/0x50
? tcp_sendmsg+0x31/0x40

Apart from kprobe and the spin lock these call stacks of this example also have in common that they are related to method tcp_sendmsg.

Active kernel probes can be identified via:

cat /sys/kernel/debug/kprobes/list

Now you can check for tracing instrumentation in directory /sys/kernel/debug/tracing/events/kprobes. Among others, the following files related to tcp_sendmsg were found:

/sys/kernel/debug/tracing/events/kprobes/r_tcp_sendmsg_net_14442/format
/sys/kernel/debug/tracing/events/kprobes/r_tcp_sendmsg_net_14442/hist
/sys/kernel/debug/tracing/events/kprobes/r_tcp_sendmsg_net_14442/trigger
/sys/kernel/debug/tracing/events/kprobes/r_tcp_sendmsg_net_14442/filter
/sys/kernel/debug/tracing/events/kprobes/r_tcp_sendmsg_net_14442/id
/sys/kernel/debug/tracing/events/kprobes/r_tcp_sendmsg_net_14442/enable

The trailing number 14442 is the Linux process ID of the tool that uses this tracing. In this specific case this process ID pointed to /opt/datadog-agent/... (identified via "ps -ef | grep 14442"). After having disabled the Datadog Agent, the system CPU co
nsumption reduced significantly. As a next step, the tool configuration may be adjusted to work in a less intrusive manner and the provider of this tool can be involved in order to reduce the spin lock contention.

Network packet capturing

External software capturing network packages can lead to spin locks in context of af_packet, e.g.:

<IRQ>
_raw_spin_lock+0x25/0x30
tpacket_rcv+0xa30/0xc70 [af_packet aae77b834dcc1a04b129ed1b495e8972008ea008]

Another symptom is high CPU consumption of Linux processes of type ksoftirqd. This scenario has e.g. been observed in context of Dynatrace Oneagent (oneagentnetwork) after an upgrade to SLES 15.5. Root cause is the Linux fix for net/packet: tpack
et_rcv: avoid a producer race condition. Only TPACKET_V2 is impacted, so redesigning the monitoring tool for TPACKET_V3 or higher will resolve the issue.
 Linux syscall auditing

Call stacks like the following indicate overhead in Linux syscall auditing:

<TASK>
__audit_syscall_exit+0x1e3/0x2a0
syscall_exit_to_user_mode_prepare+0x16e/0x1b0
syscall_exit_to_user_mode+0x28/0x40
do_syscall_64+0x67/0x80

<TASK>
__audit_syscall_entry+0x103/0x130
schedule+0x5c/0xc0
__do_sys_sched_yield+0xa/0x20
do_syscall_64+0x5b/0x80

Configured auditing can be determined via:

sudo auditctl -l

Check if you can set up more lightweight Linux syscall auditing or even switch it off in order to minimize system CPU overhead.

Linux syscall interpositioning:

External software may interposition operating system calls (syscall). The related call stack on SAP HANA shows two syscall methods, one related to the usual libc library and one to the additional external library. For example, the following call stack in
dicates syscall interpositioning by Dynatrace OneAgent:

syscall+0x19 (libc.so.6)
syscall+0x172 (liboneagentproc.so)
Synchronization::Mutex::lockInternal

This interpositioning is not supported in context of SAP HANA and external tools need to be reconfigured to no longer use this feature. Check ID C0440 ("Syscall interpositioning by Dynatrace OneAgent") of SQL: "HANA_Threads_Callstacks_MiniCh
ecks" (SAP Note 1969700) reports syscall interpositioning by the Dynatrace OneAgent.

High system CPU consumption ca Optimize OS calls of ot Identify why application is responsible for the processes with the high system CPU utilization and optimize or reduce these calls.
used by other processes her processes
A typical reason for high system CPU consumption in this context is the configuration of Transparent Huge Pages (see SAP Note 2131662). Make sure that you disable this feature based on the provided recommendations.

A more detailed analysis of activities involving system CPU consumption can be performed via OS kernel stack backtrace as described in SAP Note 2166414.

High CPU consumption of non SA Optimize non SAP HA Reduce unnecessary high CPU requirements of non SAP HANA software running on SAP HANA hosts. If required, get in touch with the vendor of the software in order to ask for assistence.
P HANA processes NA software

High SAP HANA user CPU consum Analyze expensive SAP See "How can CPU intensive operations in SAP HANA be identified and optimized?" for more details.
ption HANA operations

9. How can CPU intensive operations in SAP HANA be identified and optimized?
In order to identify CPU intensive operations within SAP HANA you can use SQL: "HANA_Threads_ThreadSamples_FilterAndAggregation" (SAP Note 1969700) in the following way:
If there is a specific time of high resource consumption, restrict BEGIN_TIME and END_TIME accordingly.
Set the filter THREAD_STATE = 'Running', because typically the threads in state 'Running' are responsible for CPU consumption.
Set AGGREGATE_BY to 'HASH', so that the statement hashs of the top SQL statements are displayed.
If no statement has exists for most samples, you can set AGGREGATE_BY to 'THREAD_TYPE, THREAD_METHOD, THREAD_DETAIL' (or a subset) to understand which types of threads are responsible.
If you have identified top statement hashs, you can check if these SQL statements can be tuned. See SAP Note 2000002 for more information related to SQL optimization.
For more detailed information like the call stacks you can collect additional runtime information (SAP Note 2313619):
SQL: "HANA_Threads_Callstacks" (SAP Note 1969700) can be used to retrieve the call stacks of the currently active threads.
A runtime dump with call stacks and further runtime information can be created (SAP Note 1813020).
Even more details can be retrieved with a kernel profiler trace (SAP Note 2800030).
Additionally make sure that the SAP HANA parameters are set based on the standard recommendations (see SAP Note 2186744). For example, SAP Note 2207501 describes CPU issues if a specific undocumented parameter is activated.
Typical scenarios of high SAP HANA CPU consumption are:

Symptom / Call stack CPU Details

Type

ptime::DeletedPageList::recycle Syste Due to a bug with SAP HANA Rev. 102.01 to 102.02 a blocked garbage collection can result after some time in a high CPU consumption during recycling of the deleted page list. Therefore you have to put a particular focus on garbage collection issues (SAP Note
m 2169283) with these Revisions in order to avoid the escalation. See SAP Note 2253017 for more information.

__GI___sched_yield Syste Up to SAP HANA 1.0 SPS 10 the row store uses active lock waits at different locations. Several of these active waits were removed as of SAP HANA 1.0 SPS 11.
m
With SAP HANA 1.0 <= SPS 11 it can happen that ptime::PageHeader::latch is responsible for __GI___sched_yield waits, This problem is fixed starting with SAP HANA 1.0 SPS 12.
__GI___sched_yield Syste This specific yield wait is linked to the function cache (SAP Note 2502256). The lock holder is typically active in LinkHash::find. Entries in the function cache are removed when the related SQL cache entries are evicted. In most cases contention is a consequenc
ptime::Futex::lock m e of too many SQL cache evictions, so you need to make sure that evictions remain on a reasonable level (SAP Note 2124112).
expr::FunctionMap::_matchID
You can use SQL: "HANA_SQL_SQLCache_Overview" (SAP Note 1969700) to check for evictions. Check IDs M1129 ("SQL cache evictions / h (short-term)") and M1130 ("SQL cache evictions / h") of the SAP HANA Mini Checks (SAP Note 1999993) return a p
otentially critical rating in case the evictions are higher than expected.

Starting with SAP HANA 2.00.048.00 you can disable the function cache using the following SAP HANA parameter:

indexserver.ini -> [sql] -> enable_function_cache = false

Starting with SAP HANA 2.0 SPS 06 the function cache implementation no longer exists (issue number 236643) and the parameter has no effect and can be removed.

Network related operating system calls like: Syste If you observe increased system CPU calls in context of network related call stack methods and increased statement load, Linux kernel taints or Linux kprobe instrumentation by external tools can be responsible. See Linux kernel taints and kprobe instrumentat
m ion for more information.
__libc_recv
__libc_send
__writev
__writev_nocancel

Execution::Context::getExecutionInstrume Syste This scenario can happen if CPU tracking is activated on SAP HANA <= 1.0 SPS 11:
ntationSnapshot m
Execution::ContextStatistics::getCurrentEx global.ini -> [resource_tracking] -> cpu_time_measurement_mode
ecutionSnapshot
Execution::ExecutionInstrumentationSnap Deactivate CPU tracking if many threads are working in the mentioned call stacks.
shot::takeMeasurement
Execution::Thread::getSysExecutionTime
Execution::TraceContextAggregatable::cont
inueKPI
TRexUtils::CPUTimeMeasurement::calcula
teCurrentTimes
TRexUtils::CPUTimeMeasurement::getAgg
regatedTimes
TRexUtils::CPUTimeMeasurement::pause
Measurement
TRexUtils::CPUTimeMeasurement::resume
Measurement
TRexUtils::CPUTimeMeasurement::retriev
eTimes
TRexUtils::CPUTimeMeasurement::startM
easurement
TRexUtils::KPIManager::startMeasurement
TRexUtils::KPIManager::stopMeasurement

__lll_lock_wait_private Syste These call stacks are typically linked to timezone conversions. You can consider the following aspects to reduce the CPU consumption:
__lll_unlock_wake_private m
Make sure that the TZ environment variable (if defined at all) of the user starting SAP HANA is configured with a proper POSIX value (SAP Note 2360909). Be aware that 'UTC' isn’t a proper value, instead you have to use e.g. 'UTC0'.
__tzfile_compute
__tzset_parse_tz For ABAP systems make sure that the following SAP HANA parameters point to the proper timezone tables and client 000 (SAP Notes 1791342):
__tz_convert
_L_lock indexserver.ini -> [global] -> timezone_default_data_client_name = 000
indexserver.ini -> [global] -> timezone_default_data_schema_name = <abap_schema_with_TZ_tables>
_L_unlock
abapSysTimezone
Afterwards execute the relevant “After updating the tables” steps of SAP Note 1791342.
abapSysTimezone_null
Evaluator::ThreeCodeProcessor::now Make sure that the configured timezone (M_HOST_INFORMATION -> timezone_name) is configured in table TTZZ. Switch to a different timezone (e.g. INDIA instead of IST) if possible. Be aware that M_HOST_INFORMATION may display a summer time
while in TTZZ only the basis timezone is contained, this is okay (e.g. CEST vs. CET, PDT vs. PST).

Adjust SQL statements responsible for a high amount of timezone conversion requests (e.g. by replacing calls like now() with fix time values determined beforehand).
__cxa_throw Syste A high system CPU consumption in these call stacks is typically related to costly exception handling. See SAP Note 2313619 -> C0400 ("Exception unwinding") for more information and analysis steps.
__cxa_throw.cold m
__cxxabiv1::__cxa_throw
__dl_iterate_phdr
__GI___pthread_mutex_lock
__lll_lock_wait
__lll_unlock_wake
_L_lock_<id>
ltt::exception::do_throw
ltt::impl::throw_check
pthread_mutex_lock
__pthread_mutex_unlock_usercnt
_Unwind_Find_FDE
_Unwind_GetTextRelBase
_Unwind_RaiseException
_Unwind_Resume

Persistence garbage collectors (GCJob*) wai Syste If persistence garbage collectors (GCJob*) repeatedly wait for the FileIDMapping lock and consume significant amount of system CPU, open a SAP case for further analysis.
ting for FileIDMapping locks m
In order to reduce the CPU consumption caused by parallel garbage collection you can set the following parameter to a lower value (details see further below):

global.ini -> [persistence] -> max_gc_parallelity

Translation lookaside buffer shootdowns Syste If you observer a high system CPU consumption related to translation lookaside buffer (TLB) shootdowns on operating system level, the following scenarios may appy:
m
SAP Note 2040144: Collecting Diagnosis Information from the SAP HANA Studio can cause TLB shootdowns (Rev. <= 1.00.74.04)
SAP Note 2206354: High System CPU Consumption Caused by Plan Trace
Issue number 294080: TLB shootdowns when hitting limit of persistent memory (SAP Note 2700084), e.g. in context of method MemoryManager::NVMProviderSPI::createBlock

DataContainer::VirtualFileStatsProxy::getO Syste A high system CPU consumption on secondary sites of system replication environments (SAP Note 1999880) with replication mode logreplay can be caused by a high number of LogRecoveryQueue<id> threads. See SAP Note 1999998 ("vfLOBStatsMapLock") f
rCreateVirtualFileLOBStatistics m or more details and consider reducing the number of log recovery threads with the following parameter (SAP Note 2222250):
DataContainer::VirtualFileStatsProxy::rem
oveVirtualFileLOBStatistics global.ini -> [persistence] -> recovery_queue_count

ptime::SparsePagePool::Container::Partitio Syste As a consequence of other problems (e.g. MemoryReclaim waits, SAP Note 1999998) a high system CPU consumption is possible in this module because of a spin lock implementation. This problem is fixed with Rev. 1.00.122.07 and 2.00.002.
n::Entry::exePopForAlloc m

__GI_kill Syste In context of allocation limit checks (SAP Note 1999997) a "kill -0" is sent to different SAP HANA services (module __GI_kill) in order to perform a process existence check. In specific scenarios this operation can be quite slow, consuming significant amounts
System::ProcessInformation::isProcessAliv m of system CPU. This can slow down different operations querying the effective allocation limit, e.g. the nameserver ping (SAP Note 2222110). If you face this scenario you should check on operating system side why the system CPU consumption is so high, and e
e liminate the root cause. With SAP HANA >= 1.00.122.15, >= 2.00.012.04 and >= 2.00.24 the __GI_kill based existence check will be disabled in context of memory management.
MemoryManager::GlobalMemoryHandler::
processIsAlive
MemoryManager::GlobalMemoryInfos::get
CurrentEffectiveAllocationLimitRespecting
OtherProcesses
syscall Syste When memory defragmentation happens, contention and spin locks are possible during munmap calls, resulting in increased system CPU consumption. See SAP Note 1999997 ("What is memory garbage collection?") for more details.
MemoryManager::munmapOverride m
System::UX::munmap
System::memFreeSystemPages
MemoryManager::SystemPageMemoryRele
aser::freeSystemPagesInternal
MemoryManager::SystemPageMemoryRele
aser::freeSystemPages
MemoryManager::freeSystemPagesInternal
MemoryManager::MemorySource::dealloca
teSystemMemory
MemoryManager::BigBlockAllocator::mun
mapGarbage
MemoryManager::BigBlockAllocator::clean
upGarbage
MemoryManager::BigBlockAllocator::collec
tGarbageFull
MemoryManager::MemoryPool::collectGar
bageBBA
MemoryManager::MemoryPool::collectGar
bage
MemoryManager::PoolAllocator::collectGar
bage
MemoryManager::ReclaimMemoryTaskHa
ndler::TaskBase::executeTask
MemoryManager::ReclaimMemoryTaskHa
ndler::executeOneTasks
MemoryManager::GlobalMemoryHandler::
pmParallelSupportProvideMemory
MemoryManager::GlobalMemoryHandler::
pmAcquireIPMMLockOrSupportProvideMe
moryHlp
MemoryManager::GlobalMemoryHandler::
pmAcquireIPMMLockOrSupportProvideMe
mory

Linux call stack: Syste A Linux soft-lockup due to a high number of directory entries can result in high system CPU consumption and a hanging database (SAP Note 2753342).
m
evict
__dentry_kill
shrink_dentry_list
shrink_dcache_parent
proc_flush_task
release_task
do_exit
do_group_exit
get_signal_to_deliver
do_signal
do_notify_resume
int_signal

Increased system and user CPU consumptio Syste The problem may be caused by a high number of changes to a specific table record in row store, which results in version consolidation overhead.
n m, Us
Check from application side if a particularly high amount of changes is executed against single table records and try to reduce the change frequency. If the problem is caused by UPDATE operations on table QIWKTAB you should check for a proper RFC queue c
SQLExecutor threads in status "Running" w er
onfiguration. For example, SMQ2 entries in "STOP" state can result in a permanent re-execution of the QIWKTAB UPDATE operations and should be avoided.
ith slow access to specific table
MVCCGarbageCollector thread active or thr From a SAP HANA perspective the row store version consolidation of single records is further improved with SPS 09 and will also reduce the probability of performance escalations.
eads in CollectVersion method
JoinEvaluator::TableAccess::acquireSearch Syste In some cases there is an active wait for index handles (SAP Note 1999998) that can result in a massive increase of system and user CPU consumption:
LockAndReplay m, Us
When a delta merge has to wait for the index handle (TRexConfig_IndexMgrIndex_FastLock), it blocks the index handle for all other requests to the table and other sessions may spin on CPU actively.
TRexAPI::Search::TableHandleProxy::initH er
When an index handle request is started without a dedicated timeout, it checks once per microsecond for a cancellation signal.
andle
TRexConfig::IndexHandle::acquire These scenarios are fixed starting with SAP HANA 1.00.122.08.
TRexConfig::IndexHandle::fastSharedAcqu
ire
TRexConfig::IndexMgr::getBucket
Synchronization::ReadWriteLock::timedWa
itLockExclusive

Index handle related lock waits like:

TRexConfig_IndexMgrIndex_Lock
TRexConfig_IndexMgrIndex_FastLock

Execution::JobOpenQueue::cleanupAfterG Syste These methods (often related to jx-pq* mutex waits, SAP Note 1999998) are prominent in case a high number of concurrent, small jobs is executed. Analyze and optimize expensive database requests with a high JobWorker demand (SAP Note 2000002).
et m, Us
With SAP HANA >= 2.00.066 the internal SAP HANA job handling is optimized (issue number 296880) which reduces the user and system CPU overhead.
Execution::JobOpenQueue::getNextJobFro er
m
Execution::JobExecutorImpl::getNextSpeci
ficFromNeighborNumaNodes

syscall Syste These syscalls are used to determine the current thread ID (getCurrentThreadID). In particular when the number of threads is significant (e.g. due to lock contention or overload scenarios) the performance of these calls can deteriorate and result in an increase
ptime::TcpReceiver::doWork m, Us d CPU and system CPU consumption. Proceed as follows to optimize this scenario:
ptime::TcpReceiver::runInner er
Eliminate the main issue like lock contention or overload. Then also the risk of running into this follow-up issue is reduced.
ptime::TcpReceiver::run
Upgrade to SAP HANA >= 2.00.059.07 or >= 2.00.067 where the syscalls have been replaced with a more light-weight context ID retrieval (issue number 299078).
syscall
This scenario typically results in several threads reported with method NoAction for a longer time (SAP Note 2114710).
SessionLayer::CommEventHolder::se
tPendingEvent At the time of the issue call stack generation with runtime dumps or for the statistics server call stack history is impacted.
SessionLayer::EventIterator::reset
SessionLayer::TcpCommMgr::waitEvents
ptime::TcpReceiver::doWork
ptime::TcpReceiver::runInner
ptime::TcpReceiver::run

High user CPU consumption caused by spec User See SAP Note 2000002 and optimize the SQL statement.
ific SQL statement

ptime::TransTokenMgr::checkTIDBufferFl User If SAP HANA SPS <= 1.0 SPS 09 is used and threads with ptime::TransTokenMgr::checkTIDBufferFlush in their call stacks are responsible for a high CPU consumption, the issue is linked to a transaction ID buffer flush check. You can set the following paramet
ush er as a workaround:

indexserver.ini -> [transaction] -> maxtidvectorsizefortidbuffer = '0'

Starting with SAP HANA SPS 10 this problem no longer exists.

High CPU consumption with Lumira User If you experience high CPU consumption in the context of Lumira, you can check if disabling the jobs as described in SAP Note 2106466 reduces the CPU utilization, and make sure that you only use supported Lumira functions (SAP Note 2195103).

__GI___sched_yield User This call stack is linked to parsing activities (SAP Note 2332551). Check according to SAP Note 2124112 if the SQL cache is configured and used optimally in order to minimize parsing activities.
Synchronization::impl::SpinLock::lock
ptime::Query::PlanHandle::validate
ptime::PreparedStatement::validate_
ptime::Statement::execute_

Synchronization modules like Syste The Synchronization::Mutex modules administer mutexes from a SAP HANA perspective, e.g. they record details that are accumulated and displayed in monitoring view M_MUTEXES. See SAP Note 1999998 for more information related to locks and mutexes i
m, Us n SAP HANA environments.
Synchronization::Mutex::lock
er
Synchronization::Mutex::setOwner A particularly high amount of lock / unlock activities can result in increased user and system CPU consumption. Known scenarios are described in SAP Note 2313619:
Synchronization::Mutex::unlock
Check ID C1020 ("Unlocking of mutex")
Synchronization::NonrecursiveMutex::unlo
Check ID C1021 ("Unlocking of non-recursive mutex")
ck
INSERT / UPDATE / DELETE threads with User A high CPU consumption in these modules can be a consequence of a high amount of accesses to the SAP HANA resource container. In this context you can often see threads of type "Resource Load Wait" or waiting for UndoAnchor.
modules like:
Most commonly this issue happens in context of undo file processing (module DataAccess::UndoFileAnchor::allocateFile). In this case the following optimizations exist:
PageAccess::LogicalPageAccessImpl::loadP
Starting with SAP HANA 1.00.122.06 and SAP HANA 2 you can increase the size of the undo file cache (default: 64 entries) by increasing the parameter indexserver.ini -> [persistence] -> undo_file_cache_size. This will eliminate the repeated creation an
ageInternal
d deletion of new undo files.
PageAccess::LogicalPageControlBlock::isId
Starting with SAP HANA 1.00.122.12 the handling is partially improved, so that an upgrade to that Revision level or higher can improve the situation (SAP Note 2547543).
entifiedBy
Upgrade to SAP HANA 2.00.001 or higher where undo file creation is optimized.
ResourceManager::ResourceContainerImpl
::lookupResourceInList
ResourceManager::ResourceContainerImpl
::putResourceHeaderIntoHashBucket
ResourceManager::ResourceContainerImpl
::readResource
ResourceManager::ResourceContainerImpl
::testEquivTryAndIncreaseRefWithDisp
ResourceManager::ResourceHeader::addRe
ferenceIfManaged
ResourceManager::ResourceHeader::releas
eAndUpdateTime

IndexingQueue / Request threads with thre User This can be a problem of the French lingware. See SAP Note 2428875 and implement SAP HANA Rev. 1.00.122.07 or 2.00.001.
ad method indexing / prep/indexing are act
ive in:

inxight::fsmindex::fst_lookup_forward
inxight::fsm_lexicon::lookup
inxight::simple_fsm_spec_lookup_for_co
mpound::lookup
inxight::priority_union_spec_lookup::look
up
inxight::tagged_stemming_implementatio
n::word_tag_lookup
inxight::tagged_stemming_implementatio
n::stem_tagged_word
inxight::tagged_stemming_implementatio
n::stem_tagged_sentence
inxight::tagged_stemming_implementatio
n::tagged_stemming_implementation

TRexUtils::BitVector::iterator::skip_ User A scan of an indexed column with SPARSE compression can consume significant CPU resources. This problem is fixed with SAP HANA >= 1.00.122.10, >= 2.00.002.01 and >= 2.00.012. See SAP Note 2112604 and consider an uncompression of the related colu
unchecked mns.
AttributeEngine::SparseIndexScanVecOut
Helper
AttributeEngine::SpDocuments::scanWithI
ndexInternal
AttributeEngine::SpDocuments::scan<Attri
buteEngine::ScanVectorBinSearchPredicate
AttributeEngine::SpAttribute<TrexTypes::S
tringAttributeValue, AttributeEngine::Valu
eDict
AttributeEngine::AttributeApi::jeReadInde
x
JoinEvaluator::TableAccess::jeReadIndex

MemoryManager::PoolAllocator::add User Due to a bug there can be an overhead when updating reference counts of memory objects that can result in high CPU consumption. The behavior is optimized with SAP HANA >= 1.00.122.11, >= 2.00.012.01 and >= 2.00.020.
Reference
As a workaround memory_tracking can be disabled (SAP Note 1999997):
TRexUtils::TRexKPIMgrCB::createMemory
StatisticsCollector global.ini -> [resource_tracking] -> memory_tracking = 'off'
Execution::TraceContextAggregatable::ena
bleResourceTracking Be aware that disabling this feature also deactivates the statement memory limit and so you are at risk that a single query consumes a very high amount of memory. Therefore you should additionally set the following parameter to a reasonable lower size of the r
Execution::TraceContextAggregatable::rese esource container (SAP Note 1993128):
tAggregations
indexserver.ini -> [memoryobjects] -> unload_lower_bound = '<min_size_of_resource_container>'
MemoryManager::PoolAllocator::rel
ease
TRexUtils::MemoryStatistics::~MemorySta
tistics>
TRexUtils::MemoryStatisticsCollector::~Me
moryStatisticsCollector
ltt::allocated_refcounted::destroyImp
JoinEvaluator::LoopJob::findJoinPairsTL_ User This module indicates expensive join engine operations. It often correlates to a large size of heap allocator Pool/JoinEvaluator/JECreateNTuple (SAP Note 1999997). A related SAP HANA bug is fixed with Rev. 2.00.002 and higher. As a workaround the NO_G
native ROUPING_SIMPLIFICATION hint (SAP Note 2142945) can be used. If triggered by BW / MDX, you can also disable the RSADMIN parameter MDX_F4_USE_SQL (SAP Note 1865554).

Execution::JobFinalObjectImpl::initi User This call stack indicates contention related to resource limitation settings via parameter default_statement_concurrency_limit (SAP Note 2222250). As a workaround you can disable the default_statement_concurrency_limit setting. This issue is fixed with SA
alizeStatementConcurrencyLimit P HANA >= 1.00.122.11, 2.00.002.02 and 2.00.010.
Execution::JobFinalObjectImpl::JobFinalO
bjectImpl
Execution::JobExecutorImpl::execute
Execution::JobContextImpl::startExecution

Stream::NetworkChannelSSLFilter::receive User A significant CPU consumption in SSL network modules can be caused by the SAP HANA bug described in SAP Note 2385992.
Raw

QueryEntryPruningEvaluator::doesPartpro User This module indicates BW multiprovider pruning activities. As a workaround you can disable multiprovider pruning by setting the following parameter:
viderMatchInternal
indexserver.ini -> [calcengine] -> disabled_patterns = 38

A SAP HANA optimization is available with >= 1.00.122.13, >= 2.00.012.02 and >= 2.00.021.

TRexCommonObjects::ItabLeakTraceWrap User These call stack methods are linked to the itab leak trace that can be used to understand memory consumption in heap allocator Pool/itab (SAP Note 1999997). Make sure that the itab leak trace is only activated as rarely and shortly as possible in order to avoid
per::addCol locking and performance overhead. The following related SAP HANA parameters must not be set, otherwise the trace may be active:
TRexCommonObjects::ItabLeakTraceWrap
per::removeCol indexserver.ini -> [itab] -> trace_leaks
indexserver.ini -> [itab] -> trace_leakcallstack

AttributeEngine::ClusterPredScanBvOutHe User In general these modules indicate column scans, e.g. caused by an inappropriate application queries or insufficient index design.
lper
A unusual high thread activity in AttributeEngine::*Scan*OutHelper modules with SAP HANA 1.00.122.10 - 1.00.122.11, 2.00.000 - 2.00.012.01 and 2.00.020 can be caused by the bug described in SAP Note 2516807. As a workaround you can set the SAP HAN
AttributeEngine::ClusterPredScanVecOutH
A parameter
elper
AttributeEngine::ClusterScanBvOutHelper indexserver.ini -> [optimize_compression] -> singleindex_consider_for_compressed_columns = false
AttributeEngine::ClusterScanVecOutHelper
AttributeEngine::IndirectPredScanBvOutH and recompress the involved tables.
elper
AttributeEngine::IndirectPredScanVecOut
Helper
AttributeEngine::IndirectScanBvOutHelper
AttributeEngine::IndirectScanVecOutHelpe
r
AttributeEngine::SparseBvScanBvOutHelpe
r
AttributeEngine::SparseBvScanVecOutHelp
er
AttributeEngine::SparsePredScanBvOutHel
per
AttributeEngine::SparsePredScanVecOutH
elper
AttributeEngine::SparseRangeScanBvOutH
elper
AttributeEngine::SparseRangeScanVecOut
Helper

ltt::impl::casInt8 User Active spinning in context of Execution::JobOpenFastUnsorted::getNextJobFrom can happen on systems with more than 8 CPU sockets in unspecific situations, particularly in cases of already increased CPU consumption. This problem is fixed starting with SA
ltt::impl::AtomicOperation<8ul>::cas P HANA 1.00.122.15.
ltt::casPtr<Execution::Context>
Execution::JobOpenFastUnsorted::ge
tNextJobFrom
Execution::JobExecutorImpl::getNextJobF
romNumaNodes
Metadata::GrantedPrivInfo::getPrivId User These call stack methods are linked to catalog authorization checks that are typically linked to SAP HANA monitoring and dictionary views that are defined with a HASANYPRIVILEGES privileges check. You can avoid these expensive checks by assigning the C
Metadata::GrantedPrivInfo::getPrivInfosOf ATALOG READ role to the executing user.
Obj
HASANYPRIVILEGES is removed from the definition of certain views with the following SAP HANA Revision levels:
Metadata::GrantedPrivInfo::getPrivInfos
Metadata::AssignedRoleInfo::searchTransit Views Optimized with
iveClosureByGranteeId
FUNCTIONS >= 1.00.122.10
Authorization::CatalogPrincipalStora
M_CS_ALL_COLUMNS >= 2.00.002.02
geManager::getTransitiveClosureOfA M_CS_COLUMNS >= 2.00.012
ssignedRolesForGrantee M_CS_TABLES
Authorization::CatalogAuthorization M_RS_TABLES
M_TABLES
StorageManager::isAuthorized
OWNERSHIP
Authorization::Principal::readTransitiveRol PROCEDURES
eClosureFromStorage SYNONYMS
Authorization::Principal::getTransitiveRole TABLE_COLUMNS
TABLES
Closure
TRIGGERS
Authorization::AuthorizationManager::has VIEW_COLUMNS
AnyPrivilege VIEWS
Authorization::AuthorizationManager::isAu
M_TEMPORARY_TABLES >= 1.00.122.14
thorizedRegardlessOfOwnership
>= 2.00.012.04
Authorization::AuthorizationManager::isAu >= 2.00.023
thorized
Authorization::AuthorizationManager::isAu
thorizedSystemPrivilege
__hasanyprivileges__String_BigInt_String
_String_String
__hasanyprivileges__String_BigInt_
String_String

ltt::allocator_statistics::getCountInUse User Long runtimes in these call stacks on SAP HANA 1.00.122.14 and 2.00.012.03 can be caused by the SAP HANA bug described in SAP Note 2580435.
MemoryManager::PoolAllocator::checkDela
yedCleanup
MemoryManager::PoolAllocator::release

sigprocmask
sigjmp_save
MemoryManager::PoolBlockEnumerator::
moveToNextBlock
MemoryManager::Allocator::dumpInUseBl
ocks
MemoryManager::Allocator::dumpInUseBl
ocks
MemoryManager::PoolAllocator::handleMe
moryLeaks

Synchronization::Mutex::lock User When late materialization is used (SAP Note 1975448) it can happen in rare scenarios that a high amount of requests for mutex "LateMatColumnSource pageLock" are executed that negatively impact performance. In this case you see many threads in module S
TRexAPI::LateMatColumnSource::lockLoa ynchronization::Mutex::lock, but not in a waiting operating system call (syscall). As a temporary workaround you can consider increasing the late materialization threshold so that less queries use late materialization (default: 20,000), e.g. to 1 million:
dAndActivatePage
TRexAPI::LateMatColumnSource::getNativ indexserver.ini -> [search] -> late_materialization_threshold = 1000000
eTEMP
Be aware that this increase can result in higher memory requirements.

Diagnose::impl::TraceBuffer::sync User A high CPU consumption in these modules indicates massive generation of database trace information (SAP Note 2380176). Check for what reason so many trace entries are written (activated trace, problem scenario, ...) and take appropriate actions to reduce it
Diagnose::TraceOutputFileHandlerImpl Syste .
Diagnose::TraceSegment::reserveEntry m
Diagnose::TraceSegment::waitReady
Diagnose::TraceStream::flushTraceBuffer
ptime::qo_Case::is_same_exp User These modules are traversed when the query optimizer is used to optimize the plan in calculation engine environments. Long runtimes are typically caused by complex models. For testing purposes or as temporary workarounds you can set the following calcula
ptime::qo_Comp::is_same_pred tion view execution hint:
ptime::qo_Comp::less_than
ptime::qo_Conj::is_same_pred ce2qo_for_nested_views_with_sql_execution = 0
ptime::qo_Func::is_same_exp
ptime::qo_Func::isSharable In order to disable it on global level, the following SAP HANA parameter can be set:
ptime::qo_Normalizer::applyRule
indexserver.ini -> [calcengine] -> ce2qo_for_nested_views_with_sql_execution = 0
ptime::qo_Normalizer::case_normalize
ptime::qo_Normalizer::eliminate_duplicat
Be aware that this setting can result in performance degradations, so it should only be implemented with care.
e_terms
ptime::qo_Normalizer::exp_pred_normaliz Even with this setting the call stack can still show up in certain scenarios. In this case further simplifications like the hint OPTIMIZATION_LEVEL(MINIMAL) may be helpful (SAP Note 2142945).
e
ptime::qo_Normalizer::is_subtree_groupin
g_exp
ptime::qo_Normalizer::normalize
ptime::qo_Normalizer::normalize_disj
ptime::qo_Normalizer::pred_normalize
ptime::qo_Normalizer::push_down_select
ptime::qo_Normalizer::push_down_select
_one_rel
ptime::qo_Normalizer::rel_pred_normaliz
e
ptime::qo_Normalizer::rewrite
ptime::qo_Normalizer::select_thru_join
ptime::qo_Normalizer::simplify_compariso
n
ptime::qo_Normalizer::simplify_pred
ptime::qo_PredicateDerivation::apply
ptime::qo_PredicateNormalization::apply
ptime::qo_Rule::checkAndApply
ptime::qo_SelectPushDown::apply
ptime::qo_VisitorUtil::qo_check_relations
ptime::qo_VisitorUtil::qo_visit_preds
ptime::qo_VisitorUtilImpl::check_all_inter
nal
ptime::qo_VisitorUtilImpl::visit_all_intern
al
ptime::QueryOptimizer::optimize_plan
ptime::CommonApiImpl::optimizeQoPlan
TrexCalculationEngine::ceQOExecuto
r::execute

Auditing::ComplexObjectCallback::getAllU User These call stacks are used in context of SAP HANA auditing (SAP Note 2159014) when auditing is globally activated with the following SAP HANA parameter:
nderlyingObjectsWithOwner
Auditing::ViewObjectHierarchyCallback::ge global.ini -> [auditing configuration] -> global_auditing_state = true
tBaseObjects
Auditing::AuditEventFactory::getRelatedEv Activation of this parameter is useful and required when auditing policies are defined, in other cases it should not be set to true in order to avoid unnecessary overhead in terms of CPU and memory (Pool/Auditing).
entList
Auditing::AuditEventFactory::generateEve
ntsFromActionList
Auditing::AuditEventFactory::collectDMLA
uditRelevantCheckEntries

Diagnose::ModuleInfo::getModuleForAddr User This call stack indicates the generation of call stacks (SAP Note 2313619) in context of configured traces like allocation stack trace (SAP Note 2222218) or memory fencing (SAP Note 2222276).
ess
Check IDs M0661 ("Heap allocators with allocation stack trace"), M0662 ("Heap allocators with other memory traces") and M0663 ("Time since activated memory trace (days)") of the SAP HANA Mini Checks (SAP Note 1999993) can be used to monitor for act
Diagnose::UnwindContext::decodeUnwindI
ivated memory traces.
nfo
Diagnose::UnwindContext::step SQL: "HANA_Memory_Heap_Flags" (ONLY_TRACE_FLAGS = 'X') available via SAP Note 1969700 can be used to check for heap allocators with activated trace flags.
Diagnose::CallStack::fastStackUnwind
Diagnose::CallStack::fillTrace
Diagnose::CallStackManager::saveStackLi
mited
MemoryManager::CheckedBlockHan
dler::createStacktrace
__cxa_free_exception User This call stack is used in context of modification commands that change columns that are used for partitioning (SAP Note 2044468) - this imposes the possibility that the records need to be moved from one partition to the other and for that reason a significant
TRexUtils::BitVector::resize overhead is possible.
TrexStore::UdivListContainerMVCC::reCre
Among others this scenario can happen when you have partitioned a table by the complete primary key and any of the primary key columns is adjusted. As suggested in SAP Note 2044468 it is usually helpful to partition tables with HASH on the most selective
ateUdivListEqualSSN
primary key column. This reduces the risk that an updated column is part of the partition columns and so partition moves are less likely.
TrexStore::UdivListManager::getEqu
alSSNUdivs The following optimizations exist in this scenario:
qo3::MethodInfo::validUdivs Use SAP HANA >= 2.0 SPS 01 where this check is implemented more efficiently.
qo3::MethodInfo::allDocs Partition the table based on a column that isn't modified.
Reduce the modification rate of columns being part of the partitioning criteria by changes on application side.
Perform more frequent delta merges (SAP Note 2057046) on the underlying table to keep the critical delta storage at a rather small size (trade-off between performance gain and delta merge overhead).

[vgettimeofday] User The operating system call __vdso_gettimeofday is a potentially slower fallback mechanism to retrieve time information when RDTSC / RDTSCP (Read Time Stamp Counter) isn't properly available for SAP HANA. This call happens in user space, so you see use
__vdso_gettimeofday r CPU, not system CPU. The trace file contains a warning like:
TimerImpl::gettimeofday_MicroTimer
Fallback to system call for HR timer
Basis::Timer::getMicroTimer
Usage of system timer is forced because the clocksource file '/sys/devices/system/clocksource/clocksource0/current_clocksource' has no 'tsc' clocksource
Basis::Timer::getHiPerfTimer
ptime::scoped_timer_for_result_t_open:: In this case check ID T0868 ("Fallback to system call for high resolution timer") may be reported by SQL: "HANA_TraceFiles_MiniChecks" (SAP Note 2380176).
_end In order to avoid overhead caused by a slow timer you need to make sure that RDTSC / RDTSCP is properly configured on Linux level or that an efficient fallback mechanism is in place. See SAP Note 2753418 for more information.
ptime::scoped_timer_for_result_t_open::
~scoped_timer_for_result_t_open For different environments problems are already known:

AWS (SAP Note 2915232)

Azure (SAP Note 2791572)
SLES 12.3 (SAP Note 2859234)
VMWare (SAP Note 2823760)

clock_gettime User This is another call stack indicating overhead in context of retrieving time information. See SAP Note 2753418 and make sure that all timer recommendations like a proper clocksource configuration are in place.
__clock_gettime_2
__GI___clock_gettime
fallback_ticks
ltt::timer::us
ltt::timer::ns_monotonic
rte::timer:ms
rte::timer:us

__wcscpy_chk User This call stack can be a consequence of significant FDA READ activities in context of fast data access (FDA, SAP Note 2399993) that is required to create and populate the internal table (itab). This bottleneck can be a reason of fluctuating response times of quic
ptime::SessionCodecNewdb::eResultSetIta Syste k database requests. In the worst case, contention in OS call __wcscpy_chk can also impact other database requests that don't use FDA READ, but use memory operations like __wcscpy_chk in different contexts.
b m
As a temporary workaround you can consider to disable FDA READ by setting the following SAP ABAP profile parameters (SAP Note 2399993):
Thread method: ExecQidItab
rsdb/supports_fda_prot = 0
dbs/hdb/supports_fda_prot = 0

Starting with SAP HANA 2.00.024.04 and 2.00.32 the overhead of FDA READ is significantly reduced.

Catalyst::TransactionBrokerClient::acquire User This contention can be caused by a SAP HANA bug with Revisions 2.00.024.07 - 2.00.024.08 and 2.00.034 - 2.00.035 (SAP Note 2748836).
SnapshotTimestamp

businessDB::QueryTools::linearizeDictiona User These call stack methods indicate processing in context of currency conversion. You can reduce the overhead by activating the currency conversion cache (SAP Note 2502256).
ryOf
When businessDB::CacheInstanceImplementation appears in the call stack, the currency conversion cache is already activated and used.
businessDB::QueryTools::linearizeDictiona
riesOf
businessDB::SimpleQuery::run
businessDB::SimpleQuery::operator
businessDB::ERPCurrencyConversion::fillC
onversionRatesTable
businessDB::ConversionMethodsRegistry::
createRatesTable
businessDB::Conversion::computeConversi
onRatesTable
businessDB::Conversion::convert
businessDB::Conversion::operator

QueryMediator::FilterTransportOptimizati User SAP Note 2785840 describes a problem with SAP HANA <= 1.00.122.16, <= 2.00.024.02 and 2.00.030 where filter transport estimations in the OLAP engine can be quite expensive in context of concurrent DML operations. As a workaround it may help to disa
on::apply ble precise filter transport estimations:
OlapEngine::OlapEngine::executeOLAPQuery
indexserver.ini -> [query_mediator] -> use_precise_filtertransport_estimations = false
TRexUtils::OutputVec::fill User This call stack indicates a full scan with HEX (SAP Note 2570371). On SAP HANA 2.00.040 it can be caused by the bug described in SAP Note 2793751. In general it can also be caused by an expensive database request that can be analyzed and optimized based
TRexUtils::IndexVectorGenericImpl::mgetSearch on SAP Note 2000002.
Bv
TRexUtils::IndexVectorRef::mgetSearch
AttributeEngine::MainDataVectorScanReader
hex::cs::DataVectorScanOrLookupBaseOp
hex::LeafOperator::xf_run2
hex::PullOperator::pull
hex::cs::ConjunctionOp::run
hex::PullOperator::pull
hex::cs::PostFilterConjOp::run
hex::cs::FragmentScanOpWithRowPos::run
hex::cs::TableScanControlOp::run
hex::impl::runNextImpl
hex::WsTaskJob::run

TransactionManager::MinReadTSTracker: User A high amount of metadata requests in scale-out systems isn't processed efficiently on the master node with SAP HANA 1.0 and a high CPU utilization can be observed in module TransactionManager::MinReadTSTracker::registerMinReadTS. Consider the follo
:registerMinReadTS wing steps:
Newdb::DistMDCacheServer::handle
Upgrade to SAP HANA 2.0
Optimization of metadata requests (SAP Note 2222200 -> "What can be reasons for threads in network related states?" -> "Metadata access").

JoinEvaluator::TupleList::partitionFind User This call stack is linked to cyclic join processing, the related thread method is ParallelLoopWithEqJob (SAP Note 2114710). If the high CPU consumption can't be resolved differently (e.g. by adjusting application or SAP HANA configuration), you can disable cy
JoinEvaluator::TupleList::partition_iterator::oper clic joins via SAP HANA hint NO_CYCLIC_JOIN (SAP Note 2142945).
ator++
Issue number 299491 is used to check if offering a global parameter for controlling cyclic joins is possible.
JoinEvaluator::LoopWithEqJob::recurse
WithEq
AttributeEngine::fnJeReadIndexIndividual User These call stack methods can become more dominant in case memory management is a bottleneck because they either take care for the memory management itself or they typically perform an increased number of memory allocations / deallocations. If you sus
AttributeEngine::ValueArray::init Syste pect that memory management is a bottleneck, you can check it by executing SQL: "HANA_Tests_MemoryOperations" (and optimally also SQL: "HANA_Tests_ArithmeticOperations" and SQL: "HANA_Tests_StringOperations") a few times and evaluate the
__cxa_free_exception m result using SQL: "HANA_Tests_Results" (SAP Note 1969700).
Diagnose::StatisticsData::addChild
Example: (increased memory operations runtime)
__GI___mremap
IndexVectorAligned::get ------------------------------------------------------------------------------------
introsortLoopAux |TEST_STATEMENT |EXPECTATION |RESULT |
JoinEvaluator::TableAccess::jeReadIndex ------------------------------------------------------------------------------------
JoinEvaluator::TranslationTableImplVector::resiz |SQL: "HANA_Tests_ArithmeticOperations"|<= 500 ms / execution|307 ms / execution |
eVector |SQL: "HANA_Tests_MemoryOperations" |<= 1000 ms / execution|41454 ms / execution|
ltt_adp::vector::push_back |SQL: "HANA_Tests_StringOperations" |<= 1000 ms / execution|710 ms / execution |
ltt::allocator::allocate ------------------------------------------------------------------------------------
ltt::allocate_raw_chunk
ltt::impl::vectorFill You can use SQL: "HANA_Threads_Callstacks_History" (ONLY_MEMORY_MANAGEMENT_CALLSTACKS = 'X') available via SAP Note 1969700 in order to filter (and aggregate) historic call stacks linked to memory management. This can allow you to ide
ltt::impl::vectorResize
ntify trends like an increase of reported samples for specific days:
ltt::string_base::string_base Example: (AGGREGATE = 'TIME', TIME_AGGREGATE_BY = 'DAY')
ltt::uninitialized_fill
memcpy_impl ------------------------
MemoryManager::PrechargeBase::remapMemory |SAMPLE_TIME |COUNT|
MemoryManager::PrechargeBase::remapBlocksFr ------------------------
|2021/11/19 (FRI)| 3274|
omLocalFreeListWithoutUpdateValues
|2021/11/18 (THU)| 9315|
MemoryManager::PrechargeBase::getMemory
|2021/11/17 (WED)| 9886|
MemoryManager::Precharge::prepare
|2021/11/16 (TUE)|11087|
MemoryManager::MemoryPool::reserveMemoryA |2021/11/15 (MON)|17608|
ndAllocateBigOrHugeBlock |2021/11/14 (SUN)| 262|
MemoryManager::MemoryPool::reserveMemoryA |2021/11/13 (SAT)| 722|
ndAllocateHugeBlock |2021/11/12 (FRI)| 6412|
MemoryManager::MemoryPool::allocateBigOrHu |2021/11/11 (THU)| 7372|
geBlock ------------------------
MemoryManager::MemoryPool::allocate
MemoryManager::PoolAllocator::allocateNoThro In this case some misconfigurations slowed down the database after a restart on 15th of November and a fix was provided on 19th of November. During the time of the issue the increased amount of memory related operations is visible.
wImpl If you suspect a bottleneck in the memory management area, you can check SAP Note 1999997 and optimize the memory utilization and configuration, e.g.:
MemoryManager::TempAllocator::allocateNoThr
owImpl Make sure that the SAP HANA free memory cache doesn't have and adverse effect on performance and consider disabling it for SAP HANA 2.0 SPS 03 and SPS 04 (SAP Notes 2857553, 2600030). Also disabling it on SAP HANA 2.0 SPS 05 can sometimes
memset_impl be useful. Starting with SAP HANA 2.0 SPS 06 it is disabled per default.
mremap Optimize database request being responsible for a particularly high memory allocation rate.
__open_nocancel Take care for proper memory sizing and configuration to make sure that memory reclaims, compactions and shrinks remain on a reasonable level.
partitionFind Avoid frequent and aggressive manual memory compactions (e.g. "mm gc -f" with hdbcons) and configuration of parameters async_free_target or async_free_threshold.
TRexCommonObjects::VariableMem::sort
Make sure that proactive reclaims (parameters gc_unused_memory_threshold_rel, gc_unused_memory_threshold_abs) are configured reasonably and in line with SAP Note 2600030.
TRexUtils::BitVector::highestBitSet SQL: "HANA_Threads_Callstacks_MiniChecks" (SAP Note 2313619) must not report check ID C0100 ("Proactive memory compaction") as potentially critical.
TRexUtils::convertToHigher
TRexUtils::IndexVectorRef::mset
TRexUtils::Mset::fill
TRexUtils::MsetLoop::run
TRexUtils::MsetWord::run
TRexUtils::MultiColumnHashMap::init
UnifiedTable::NBitDataAccessor64::msetValueAr
ray::init

Page faults

TRexAPI::UdivLookupWithoutConcat::look User This call stack method indicates an access via inverted individual index (SAP Note 2600076). In addition to a normal expensive database request (SAP Note 2000002) this index type can be particularly expensive if the index column with the highest number of
upWithoutConcat distinct values has some quite frequent values that are also used when accessing the table. In this case you can consider to use an inverted value or inverted hash index instead.

DataStatistics::DataStatisticsAdviserCache: User High CPU consumption in these modules indicates that the Data Statistics Adviser Cache is activated (SAP Note 2502256). This cache can be responsible for significant CPU overhead, so it should be only activated with care. Make sure that the following SAP H
:createNewEntries ANA parameters aren't set to true to disable this adviser and its overhead:
DataStatistics::DataStatisticsAdviserCache:
:createNewEntry indexserver.ini -> [cache] -> datastatisticsadvisercache_enabled
DataStatistics::DataStatisticsAdviserCache: indexserver.ini -> [cache] -> datastatistics_adviser
:insert
Alternatively it can be configured in SAP HANA Cockpit (SAP Note 2800006) -> "Recommendations" -> "Configure" -> "Data Statistics Advisor".
DataStatistics::DataStatisticsAdviserCache:
:lookup
DataStatistics::CSAccessStatisticsCache::in User These call stacks are related to the maintenance of the column store access statistics cache (SAP Note 2502256).
sert
See SAP Notes 2785533 and 2970271 and make sure that the column store access statistics cache is only activated as short and targeted as possible. The following parameter controls the behavior (true -> activated):
DataStatistics::CSAccessStatisticsCache::lo
okup indexserver.ini -> [cs_access_statistics] -> collection_enabled
AttributeEngine::AccessCounterHelpers::in
crementCSAccessStatisticsCacheValue
AttributeEngine::AccessCounterHelpers::in
crementCSAccessStatisticsCacheValueForK
eyInternal
AttributeEngine::AccessCounterHelpers::in
sertIntoCSAccessStatisticsCache

AttributeEngine::RowOrderOptimizer::dete User This call stack is related to a specific optimize compression phase (SAP Note 2112604) that can be quite expensive, using a high amount of concurrent JobWorkers. The parallelism can be controlled by the following SAP HANA parameter:
rmineValueIdCounts
AttributeEngine::SortRestRangeJob:: indexserver.ini -> [optimize_compression] -> row_order_optimizer_threads
run
See SAP Notes 2222250 for workload settings in general and SAP Note 2600030 for workload parameter recommendations in particular to make sure that the concurrency settings (in particular row_order_optimizer_threads, max_concurrency_hint) are in pl
ace based on the SAP best practices.

On top of the parallelism itself also NUMA effects can be triggered that can result in an overall increased user CPU consumption, looking similar to a slow-down of the CPU speed. In this case you need to make sure that NUMA and also infrastructure details lik
e firmware, BIOS and Intel specification updates are configured optimally (SAP Notes 2470289, 2632231). As a temporary workaround you can set the following parameter:

indexserver.ini -> [optimize_compression] -> numa_awareness_no_steal_jobs = true

Be aware that this parameter must only be set if there are severe NUMA effects beyond the actual SortRestRangeJob activities.

Catalyst::HashMap::Stripe:reserveEl User A high CPU consumption in Catalyst::HashMap coding can be caused by other threads getting stuck in global deadlock detection (module TransactionManager::WaitGraph::checkGlobalDeadlock) for a long time, e.g. due to waits for the "Transaction object latc
ement h" mutex (SAP Note 1999998). In the worst case (i.e. in case of an internal deadlock with no progress) it is required to restart SAP HANA.
Catalyst::HashMap::Stripe::resize
Catalyst::HashMap::reserve
ptime::Transaction::start_logging
ptime::Transaction::set_write_trans
ptime::Query::lock_table_globally

MemoryManager::StripedAllocator:: User A high CPU consumption in MemoryManager::StripedAllocator::allocateStriped is typically caused by factors like:
allocateStriped
Many client connections
ltt::allocator_statistics::allocator_statistics
Small ABAP statement cache size
Execution::ContextAllocator::initImplicitSt
global.ini -> [memorymanager] -> composite_statistics_striping set to NUMA
atementMemoryBooking
Execution::StatementContext::initImplicitS In this case you the following parameter in order to deactivate internal striping:
tatementMemoryBooking
global.ini -> [memorymanager] -> statistics_type = 7
MemoryManager::PoolAllocator::PoolAlloc
ator
See SAP Note 1999997 -> "How can allocations in Pool/Statistics be analyzed and optimized?" for further details about these aspects.
MemoryManager::PoolAllocator::createCo
mpositeAllocator
MemoryManager::PoolAllocator::createSub
AllocatorInternal
ltt::allocator_statistics::getCountAlloc User Heuristic reordering in context of query rewrites can introduce significant overhead if on lower layers expensive operations happen repeatedly, e.g. counting M_CONTEXT_MEMORY entries repeatedly like in the left example call stack. You can check if disabli
ltt::allocator_statistics::isActive ng heuristic reordering with hint OPTIMIZATION_LEVEL(RULE_BASED) (SAP Note 2142945) can help to reduce the overhead. The SAP HANA Mini Checks (SAP Notes 1969700, 1999993) were extended by this hint for that purpose.
ltt::allocator_statistics::isVisible
Diagnose::RowCountFunctor::execute
Diagnose::StatisticsWrapper::traverseNode
sRecursiveImpl
Diagnose::StatisticsWrapper::traverseNode
sRecursiveImpl
Diagnose::StatisticsWrapper::traverseNode
sRecursiveImpl
Diagnose::StatisticsWrapper::traverseNode
sRecursiveImpl
Diagnose::StatisticsWrapper::traverseNode
sRecursive
TypedStatisticsWrapper__M_CONTEXT_
MEMORY::traverseNodesImpl
Diagnose::StatisticsWrapper::traverseNode
sBase
Diagnose::StatisticsRegistry::traverseNodes
ptime::StatisticsMonitorHandle::getRowCo
untEstimation
ptime::StatisticsAPIWrapper::getMonitorVi
ewRowCount
ptime::TableSizeEstimator::fetchHistogram
OfMonitorView
ptime::TableSizeEstimator::Estimate
ptime::qo_size_estimation::estimate_one_
rel
ptime::qo_size_estimation::head_estimati
on
ptime::qo_size_estimation::rel_estimation
ptime::qo::heuristic::heuristic_reorder_joi
n
ptime::qo::heuristic::heuristic_reord
er
ptime::qo_HeuristicJoinReorder::apply
ptime::qo_Rule::checkAndApply
ptime::qo_Normalizer::applyRule
ptime::qo_Normalizer::rewrite

ptime::Monitor::Monitor User This call stack is linked to queries on monitoring view M_MERGED_TRACES. SAP Note 2088971 -> "M_MERGED_TRACES" describes possibilities to reduce the data available in the underlying trace files.
ptime::MergedTracesMonitor::Iterat
or::fetch
ptime::Monitor_scan::fetch_with_itr
ptime::Monitor_scan::do_fetch
ptime::Table::fetch

Configuration::ParameterBase::id User Large call stacks with repeated occurrences of these modules are linked to an inefficient evaluation of nested expressions in context of HEX (SAP Note 2570371). The following options exist to improve the situation:
Configuration::Parameter::value
Upgrade to SAP HANA 2.0 >= SPS 05 where the problem is fixed.
ptime::qo::hex::Enumerator::isHexCompati
Disable HEX via hint or globally (SAP Note 2570371 -> "How can the usage of HEX be controlled"?).
bleStorageType
Reduce the amount of used expressions, e.g. simplify complex / nested CASE conditions
ptime::qo::hex::Enumerator::hasUnsupport
edColumnInUse
ptime::qo::hex::CheckExpHexSupportVisit
or::operator
ptime::qo_VisitorUtilImpl::check_all_inter
nal
ptime::qo_VisitorUtil::qo_check_all
_exprs_pre_order
ptime::qo::hex::CheckExpHexSuppor
tVisitor::checkExpression
Diagnose::StatisticsData::addChild User Expensive read write lock maintenance in context of AbapSysTimeZoneSingleCache is related to lock ptime::AbapTimeZone (SAP Note 1999998). It can e.g. happen in context of huge aggregations via STRING_AGG. A fix is available with SAP HANA >= 2.00.0
Diagnose::StatisticsData::StatisticsData 48.00.
Synchronization::ReadWriteLockStatistics::
ReadWriteLockStatistics
Synchronization::ReadWriteLock::ReadWri
teLock
ptime::Env::AbapSysTimeZoneSingle
Cache::AbapSysTimeZoneSingleCach
e

AttributeEngine::MainInvertedIndexLooku User This call stack indicates index lookups in context of HEX (SAP Note 2570371). The following options may reduce the overhead:
pReader::lookup
SQL statement tuning (SAP Note 2000002)
hex::cs::IndexLookupOp::run
SAP HANA >= 2.00.046: Adjustment of predicate term sampling (SAP Note 2124112 -> "What kind of advanced parsing features exist?" -> "Predicate term sampling")
As a temporary workaround: Deactivation of HEX (SAP Note 2570371), optimally only on statement level via hint NO_USE_HEX_PLAN (SAP Note 2142945)

rte::diag_topic::change_level_user User This call stack is linked to a trace activated on user level, e.g. a user-specific trace or an application_user setting in contexts like the expensive statements trace. Make sure that you activate user related traces only as focused and as short as possible. See SAP Not
rte::diag_topic::context_levels::set_level e 2119087 for more details related to SAP HANA traces.
rte::diag_topic::context_levels::reset
Diagnose::CountedUserTraceContext
::operator=
Diagnose::CountedUserTraceContext
::removeUserTrace
Execution::StatementContext::setRo
otStatementHash
Execution::StatementContext::setStat
ementHash
Execution::TraceContext::~TraceCon
text
rte::diag_topic::change_level_user
rte::diag_topic::context_levels::set_level
rte::diag_topic::context_levels::context_lev
els
Diagnose::CountedUserTraceContext
::addUserTrace

TRexUtils::BitVector::BitVector User These call stacks indicate BitVector processing. A BitVector is a low level memory structure that is used in different contexts. The related heap allocator is Pool/BitVector. See SAP Note 1999997 -> "Pool/BitVector" for typical reasons of heavy BitVector processi
TRexUtils::BitVector::numSet_AVX512 ng.
TRexUtils::BitVector::resize

Configuration::ParameterBase::id User When a thread waits for a packet to arrive on a network socket, it uses the epoll call with a timeout defined with the following parameter (default: 100, unit: ms):
ptime::TcpReceiver::doWork
ptime::TcpReceiver::run indexserver.ini -> [session] -> event_wait_timeout
TrexThreads::PoolThread::run
TrexThreads::PoolThread::run When no packet arrived within the timeout period, some checks are done and with SAP HANA >= 2.00.059.07 and >= 2.00.067 among others the current value for the following parameter is determined (default: 500, unit: ms):
Execution::Thread::staticMainImp
indexserver.ini -> [session] -> tcp_receiver_idle_check_threshold
Execution::pthreadFunctionWrapper
start_thread
With SAP HANA 2.0 SPS 05 the parameter check is expensive to a certain extent and in cases where many threads are active in network receive, the regular 100 ms check of this parameter can become dominant in terms of CPU consumption (issue number 311
__GI___clone
538). Thus, general contention scenarios blocking many threads (e.g. ConsistentChangeLock, slow commits) can over time result in a CPU bottleneck when more and more threads get stuck in network receive.
gstack (normal call stacks may not work):
As a workaround the event_wait_timeout parameter can be increased so that the tcp_receiver_idle_check_threshold parameter access happens less frequently, e.g. to 1000 ms:
set<Configuration::ParameterID> () a
t smart_ptr.hpp:319 indexserver.ini -> [session] -> event_wait_timeout = 1000
smart_ptr () at smart_ptr.hpp:527
id () at Basis/Configuration/impl/Paramete An optimization is available with SAP HANA >= 2.00.059.10 but the Configuration::ParameterBase::delegate call stack can still be responsible for CPU overhead. In this case eliminate the root cause for the high number of threads piling up (e.g. avoid starting
r.cpp:138 ABAP application servers while SAP HANA is still loading the column store).
value () at Basis/Configuration/Parameter.
hpp:241
doWork () at ptime/session/tcp_receiver.
cc:272
run () at ptime/session/tcp_receiver.cc:492

Configuration::ParameterBase::deleg
ate
Configuration::DelegatingParameterValue
Manager::getValue
ptime::TcpReceiver::doWork
10. Is it possible to limit the amount of consumed CPU?
See SAP Note 2222250 ("How can workload management be configured for CPU and threads?") for an overview of parameters and settings that can be used to influence parallelism and CPU consumption.

11. How can I determine physical CPU details on SAP HANA side?
General CPU information can be retrieved via monitoring view M_HOST_INFORMATION or with the related command SQL: "HANA_Hosts_Overview" (SAP Note 1969700).
Example:

---------------------------------------------------------
|CPU_DETAILS|CPU_MHZ|CPU_MODEL |
---------------------------------------------------------
|60*2294MHz | 2294|Intel Xeon CPU E7-8880 v3 @ 2.30GHz|
---------------------------------------------------------

If hyperthreading is used, the number of logical CPU cores is included in brackets after the CPU number (e.g. "60(120)*2294MHz").
Further details like the number of CPUs per socket can't be determined via SQL. On SAP HANA side you can alternatively use the following hdbcons command (SAP Note 2222218):

hdbcons 'jexec info'

Example:

Using 2 numa nodes

numa_features: 1
max_concurrency: 20 (cfg=0, dyn=20)
max_concurrency_hint: 20 (cfg=0)
min_concurrency_hint: 10 (cfg=0)
0 statement limiters
System info:
2 CPU sockets, 20 physical cores, 20 logical cores, 20 total possibles cores
Using global restriction to a subset of cores: [11111111 11111111 1111]
Numa node 0: usable cores=10
workers: [11111111 11000000 0000]
others: [00000000 00000000 0000]
full: [11111111 11000000 0000]
max_concurrency: 10, dyn=10
Numa node 1: usable cores=10
workers: [00000000 00111111 1111]
others: [00000000 00000000 0000]
full: [00000000 00111111 1111]
max_concurrency: 10, dyn=10
[OK]

In this example you can see that 20 physical cores are available in two sockets, so each socket contains 10 cores.
For more information related to NUMA in SAP HANA environments see SAP Note 2470289.

12. How can I trigger automatic follow-up actions in case of a high CPU consumption?
For an optimal root cause analysis it is important to have as much analysis data available for times of high CPU consumption as possible. In order to support the collection of analysis data (e.g. via runtime dump, SAP Note 2400007) you can schedule SAP HANASitter (SAP Note 2399979).
Example:

Command Details

python hanasitter.py -cpu 0,5,2,95 Standard behavior and additional trigger of actions when the CPU consumption is at least 95 % during 5 checks with an interval of 2 seconds

13. Is the CPU consumption shown by SAP HANA always correct?

In most cases you can rely on the displayed CPU figures, but for the following reasons it can be wrong:

Scenario SAP No Details
te

Wrong system CPU numbers in load history 2222110 If the system CPU values are identical to the user CPU values, you most see wrong values. In this case you can check the real system CPU consumption in a different way, e.g. via SQL: "HANA_Resources_CPUAndMemory_History" (
SAP Note 1969700).

NULL values for system CPU in HOST_RESOURCE_UTILIZATI 2527122 Due to a bug in the view definition of HOST_RESOURCE_UTILIZATION_STATISTICS NULL values can be reported for system CPU consumption. This problem is fixed with SAP HANA >= 2.00.020. Alternatively you can adjust th
ON_STATISTICS e view definition based on the instructions in the SAP Note.

Estimated CPU time measurement Starting with SAP HANA 1.0 SPS 12 CPU times are estimated rather than computed, so there can be some deviations from the actual CPU consumption.

14. Can the available CPU resources be limited on OS level?
In SAP HANA environments you should generally avoid to limit CPU resources on OS level, because this limitation is often intransparent and certain SAP HANA tasks can significantly suffer in case of insufficient CPU resources.
Examples for undesired CPU limitations introduced on OS level:

Feature SAP Note Details

CPU power-save mode 2519630 If a CPU power-save mode is active, the frequency of CPUs is reduced dynamically in case of reduced workload.

CPU folding 2205917 CPU folding dynamically deactivates CPU in case of reduced workload. The related cpufoldd is part of the pseries-energy package available on IBM on Power systems. You can use the following OS command to make sure that it is not installed:
2292690
rpm -q pseries-energy

15. Is it recommended to use hyperthreading in SAP HANA environments?

In general hyperthreading can be used in SAP HANA environments, but on Intel based systems with more than 8 CPUs it should usually be deactivated. See SAP Note 2711650 for more information.

16. Which thread states and call stacks indicate CPU consumption?
Only thread state "Running" represents CPU consumption. All other thread states indicate waiting (e.g. for locks or network communications). You can use SQL: "HANA_Threads_ThreadSamples_FilterAndAggregation" (THREAD_STATE = 'Running') available via SAP Note 1969700 to
display thread activities consuming CPU.
SAP Note 2929927 describes an exception where a network activity with the following call stack is accounted as running instead of "Network Poll", thus confusing the JobWorker concurrency calculations:

__poll_nocancel
ldap_int_poll
ldap_int_tls_start
ldap_start_tls_s
Authentication::LDAP::Session::bind
Authentication::LDAP::SessionFactory::createNewSession

Call stacks working in __libc_recv are also mapped to thread state "Running" although the CPU consumption is usually minor (issue number 290193).
As described in SAP Note 1999998, there can be some "hidden" lock waits of threads in status "Running". Related call stacks (SAP Note 2313619) contain:
__GI___pthread_mutex_lock
__GI___sched_yield
__lll_lock_wait
_L_lock_1008
pthread_mutex_lock
pthread_mutex_lock_internal
ptime::Futex::lock
Synchronization::impl::SpinLock::lock
Synchronization::impl::SpinLock::_lock
Synchronization::SystemMutex::lock
For threads created outside of the normal SAP HANA context the thread state "Unknown" is used that can mean both CPU consumption or wait states. See SAP Note 2114710 for more details.
In call stacks (SAP Note 2313619) the first module is of interest. It is indicating CPU consumption as long as it doesn't indicate an idle system call (e.g. syscall, do_futex_wait, io_queue_run, __nanosleep_nocancel, pthread_cond_wait) or network operations (e.g. __GI_epoll*, __GI___poll*,
__epoll*, __poll*, __read*, __write*). You can use SQL: "HANA_Threads_Callstacks_Current" or SQL: "HANA_Threads_Callstacks_History" (ACTIVITY_TYPE = 'CPU') available via SAP Note 1969700 in order to filter for call stacks indicating CPU consumption.

17. How can the CPU performance of the SAP HANA database server be tested and compared?
The performance of a CPU depends on various factors like the CPU type, the system load, the NUMA configuration, the power-saving scenario or the activation of special features like mitigations for Spectre / Meltdown vulnerabilities. Thus, it can sometimes be useful to measure certain basic
CPU operations on SAP HANA side and compare it with usual expectations and between systems. For that purpose test commands are available via SAP Note 1969700:
SQL: "HANA_Tests_ArithmeticOperations": Executes arithmetic operations
SQL: "HANA_Tests_StringOperations": Executes string operations
The runtimes of these commands can subsequently be evaluated with SQL: "HANA_Tests_Results" (SAP Note 1969700).
Example:

---------------------------------------------------------------------------------------------
|TEST_STATEMENT |EXECUTIONS|EXPECTATION |RESULT |
---------------------------------------------------------------------------------------------
|SQL: "HANA_Tests_ArithmeticOperations"| 5|<= 500 ms / execution|226 ms / execution|
|SQL: "HANA_Tests_StringOperations" | 5|<= 1000 ms / execution|506 ms / execution|
---------------------------------------------------------------------------------------------

18. What are reasons for slowness in combination with a limited CPU consumption?
The following scenarios are most typical explanation why SAP HANA can be slow while at the same time the available CPU resources are utilized only to a minor extent:
Transactional or internal lock contention (SAP Note 1999998)
Restrictive CPU workload settings like affinity, default_statement_concurrency_limit, max_concurrency or max_concurrency_hint (SAP Note 2222250)
 NUMA issues (SAP Note 2470289)
Memory management bottlenecks

19. What can be done in case of high CPU consumption in SAP HANA Cloud environments?
In case of high CPU consumption in SAP HANA Cloud (SHC) environments, e.g. reported via HDBCPUUsage alert, you can proceed according to SAP Note 3123732 in order to analyze and optimize the CPU utilization or adjust the sizing.

20. How can CPU spikes be analyzed?

In most cases, temporary CPU spikes are caused by expensive database requests. SAP Note 3365645 describes how to identify the responsible database requests. SAP Note 2999990 provides a general approach how to analyze episodes of reduced performance, including high CPU scenarios.

Keywords
Performance
Alert
Host CPU Usage
threshold_cpu_time
M_SERVICE_THREADS.CPU_TIME_SELF
M_SERVICE_THREADS.CPU_TIME_CUMULATIVE
M_EXPENSIVE_STATEMENTS.CPU_TIME
CPUTimeMeasurement
liboneagentproc.so%

Attributes
Key Value

Other Components SAP HANA > SAP HANA Database (HAN-DB)

Products
Products

SAP BusinessObjects BI 4.1, add-on for SAP Lumira

SAP HANA, platform edition all versions

SAP Lumira 1.0, server version for the SAP BusinessObjects BI platform

SAP Lumira, desktop edition 1.0

SAP Lumira, server edition 2.0

This document refers to

SAP Note/KBA Component Title

3365645 HAN-DB-PERF

2999990 HAN-DB-PERF How-To: SAP HANA Performance Analysis

2915232 HAN-DB-MON HANA timer fallback alert on AWS due to incorrect configuration

2823760 BC-OP-LNX-ESX HANA Timer Fallback on VMware based servers

2800030 HAN-DB-PERF FAQ: SAP HANA Kernel Profiler

2753342 BC-OP-LNX-SUSE BUG: soft lockup - CPU#x stuck for xxs!

2700084 HAN-DB FAQ: SAP HANA Persistent Memory

2600076 HAN-DB FAQ: SAP HANA Inverted Individual Indexes

2600030 HAN-DB Parameter Recommendations in SAP HANA Environments

2570371 HAN-DB-ENG FAQ: SAP HANA Execution Engine (HEX)

2519630 HAN-DB Recommendation: Check whether power save mode is active

2502256 HAN-DB-ENG FAQ: SAP HANA Caches

2470289 HAN-DB FAQ: SAP HANA Non-Uniform Memory Access (NUMA)

2400007 HAN-DB-PERF FAQ: SAP HANA Runtime Dumps

2399993 HAN-DB FAQ: SAP HANA Fast Data Access (FDA)

2399979 HAN-DB-PERF How-To: Configuring automatic SAP HANA Data Collection with SAP HANASitter
2380176 HAN-DB FAQ: SAP HANA Database Trace

2313619 HAN-DB-MON How-To: Generating and Evaluating SAP HANA Call Stacks

2222276 HAN-DB How-To: Activating SAP HANA Memory Fencing

2222250 HAN-DB-PERF FAQ: SAP HANA Workload Management

2222218 HAN-DB FAQ: SAP HANA Database Server Management Console (hdbcons)

2222200 HAN-DB FAQ: SAP HANA Network

2222110 HAN-DB-PERF FAQ: SAP HANA Load History

2195103 BI-LUM-SRV SAP Lumira server version for SAP HANA and SAP Lumira Cloud are retiring

2186744 HAN-DB FAQ: SAP HANA Parameters

2169283 HAN-DB-ENG FAQ: SAP HANA Garbage Collection

2159014 HAN-DB-SEC FAQ: SAP HANA Security

2142945 HAN-DB-PERF FAQ: SAP HANA Hints

2135965 BI-LUM-SRV How to turn off the background processing jobs for Lumira Server

2124112 HAN-DB-PERF FAQ: SAP HANA Parsing

2119087 HAN-DB How-To: Configuring SAP HANA Traces

2116157 HAN-DB FAQ: SAP HANA Consistency Checks and Corruptions

2114710 HAN-DB-MON FAQ: SAP HANA Threads and Thread Samples

2112604 HAN-DB-ENG FAQ: SAP HANA Compression

2106466 BI-LUM-SRV Lumira Server: is it possible to disable / configure the VDSchedulerJob_* that are enabled by Lumira Server installation?

2088971 HAN-DB-MON How-To: Controlling the Amount of Records in SAP HANA Monitoring Views

2081591 HAN-DB-PER FAQ: SAP HANA Table Distribution

2057046 HAN-DB-ENG FAQ: SAP HANA Delta Merges

2044468 HAN-DB-ENG FAQ: SAP HANA Partitioning

2000002 HAN-DB-PERF FAQ: SAP HANA SQL Optimization

1999998 HAN-DB-PERF FAQ: SAP HANA Lock Analysis

1999997 HAN-DB FAQ: SAP HANA Memory

1999993 HAN-DB-MON How-To: Interpreting SAP HANA Mini Check Results

1999880 HAN-DB-HA FAQ: SAP HANA System Replication

1909670 HAN-DB-MON How to handle HANA Alert 5: ‘Host CPU Usage'

1813020 HAN-DB How to generate a runtime dump on SAP HANA

3359552 BC-OP-LNX Linux: Supportability considerations when running 3rd-party software

3123732 HAN-CLS-HC Analyze Reason for Alert HDB CPU Usage

2929927 HAN-DB-ENG When Jobworkers Are Occupied to Make LDAP Connection But Fail,Then Other Operations May Not Function Properly

2859234 BC-OP-LNX-SUSE SAP HANA Timer Fallback alert on SLES 12 SP3

2857553 HAN-DB Overall System Performance Degrading After HANA or OS Kernel Upgrade

2793751 HAN-DB High CPU Consumption With Many Threads Waiting in hex::impl::WsAllTasksRefState::waitForAllTasks

2791572 HAN-DB Performance Degradation Because of Missing VDSO Support in Microsoft Azure VM

2785840 HAN-DB Degraded Performance of the Select Statement Running on the Table Heavily Affected by DML Queries

2753418 HAN-DB Potential Performance Degradation Due to Timer Fallback

2748836 HAN-DB Indexserver Busy With High CPU on Catalyst::TransactionBrokerClient::acquireSnapshotTimestamp

2711650 HAN-DB SAP HANA on Intel server systems with more than 8 CPU´s / sockets

2632231 HAN-DB SAP HANA and Non-Uniform Memory Access (NUMA)

2580435 HAN-DB Memory Leak in Pool/RowEngine/Session

2547543 HAN-DB High CPU Used by DataContainer::VarSizeEntryUserDataHandler::getUnusedEntryFromFreeList

2527122 HAN-DB-MON Values in _SYS_STATISTICS.HOST_RESOURCE_UTILIZATION_STATISTICS.TOTAL_CPU_SYSTEM_TIME_DELTA are NULL

2516807 HAN-DB jeReadIndex on Compressed Column of Index Type FULL is Slow

2385992 HAN-DB SAP HANA DB: Hanging Connection Requests When Trying to Open Secure Connections to HANA in Parallel
2292690 HAN-DB SAP HANA DB: Recommended OS settings for RHEL 7

2205917 HAN-DB SAP HANA DB: Recommended OS settings for SLES 12 / SLES for SAP Applications 12

1993128 HAN-DB SAP HANA: column store table unloads and unloading behavior of Memory Objects Container

1975448 HAN-DB Problems due to Late Materialization and chunk-wise Data Processing

1969700 HAN-DB SQL Statement Collection for SAP HANA

1865554 BW-BEX-OT-MDX MDX: Access type F4 help / improved error update

1791342 HAN-DB Time Zone Support in SAP HANA

784391 BC-OP-LNX SAP support terms and 3rd-party Linux kernel drivers

618104 BC-OP-LNX "sapsysinfo" - Collecting system information on Linux

SAP HANA Troubleshooting and Performance Analysis Guide

Kernel Probes

net/packet: tpacket_rcv: avoid a producer race condition

This document is referenced by

SAP Note/KBA Component Title

3549830 SAP HANA Minicheck findings M0551 and M0552 in regards to timezone

3110663 HAN-DB How-To: Check total number of existing CPU thread for HANA DB

2222200 HAN-DB FAQ: SAP HANA Network

2999990 HAN-DB-PERF How-To: SAP HANA Performance Analysis

3273961 HAN-DB-PERF The Orphan JobWorker threads utilize 99% CPU.

3257190 HAN-DB-PERF High CPU usage caused by column store access statistics cache

3065607 CA-LT-MC Performance tips & tricks for SAP S/4HANA Migration Cockpit: Migrate Data Using Staging Tables

3011192 HAN-DB-PERF Analysing High User CPU in SAP HANA DB

3024031 HAN-DB-PERF CPU spikes every hour.

2915232 HAN-DB-MON HANA timer fallback alert on AWS due to incorrect configuration

2800008 HAN-DB-ENG-TXT FAQ: SAP HANA Fulltext Indexes

2975817 HAN-DB-PERF Locks caused by enabled Data Statistics Adviser Cache

2970921 HAN-DB-MON HANA CPU spikes to 80-100%

2952074 HAN-DB High CPU Caused by Optimize Compression

2313619 HAN-DB-MON How-To: Generating and Evaluating SAP HANA Call Stacks

2092196 HAN-DB-ENG How-To: Terminating Sessions in SAP HANA

2918507 HAN-DB-MON HANA CPU usage is higher than expected at intermittent intervals based on configured CPU Workload Parameters

2399990 HAN-DB How-To: Analyzing ABAP Short Dumps in SAP HANA Environments

2908925 HAN-DB-HA Takeover slow during optimized compression phase

1999997 HAN-DB FAQ: SAP HANA Memory

2502256 HAN-DB-ENG FAQ: SAP HANA Caches

2843506 HAN-DB-PERF 100% CPU usage on Time Stamp Counter (TSC)

1616401 BC-UPG-TLS-TLA Parallelism in the Upgrades, EhPs and Support Packages implementations

2785814 HAN-DB-BAC HANA high CPU utilization caused by large backup catalog

2699939 HAN-DB SAP HANA Emergency Suitcase

2399993 HAN-DB FAQ: SAP HANA Fast Data Access (FDA)

2380176 HAN-DB FAQ: SAP HANA Database Trace

2498739 HAN-DB-PERF How-To: Analyzing Runtime Dumps with SAP HANA Dump Analyzer

2000000 HAN-DB-PERF FAQ: SAP HANA Performance Optimization

2600030 HAN-DB Parameter Recommendations in SAP HANA Environments

2494336 HAN-DB Cannot turn off CPU Measurement in HANA 1.0 SPS12
2142945 HAN-DB-PERF FAQ: SAP HANA Hints

2112604 HAN-DB-ENG FAQ: SAP HANA Compression

2470289 HAN-DB FAQ: SAP HANA Non-Uniform Memory Access (NUMA)

2400007 HAN-DB-PERF FAQ: SAP HANA Runtime Dumps

2044468 HAN-DB-ENG FAQ: SAP HANA Partitioning

2400024 HAN-DB-MON How-To: SAP HANA Administration and Monitoring

2399979 HAN-DB-PERF How-To: Configuring automatic SAP HANA Data Collection with SAP HANASitter

2222218 HAN-DB FAQ: SAP HANA Database Server Management Console (hdbcons)

2000002 HAN-DB-PERF FAQ: SAP HANA SQL Optimization

2222250 HAN-DB-PERF FAQ: SAP HANA Workload Management

1999998 HAN-DB-PERF FAQ: SAP HANA Lock Analysis

2222110 HAN-DB-PERF FAQ: SAP HANA Load History

2116157 HAN-DB FAQ: SAP HANA Consistency Checks and Corruptions

2186744 HAN-DB FAQ: SAP HANA Parameters

2180165 HAN-DB FAQ: SAP HANA Expensive Statements Trace

2119087 HAN-DB How-To: Configuring SAP HANA Traces

2169283 HAN-DB-ENG FAQ: SAP HANA Garbage Collection

2154870 HAN-DB How-To: Understanding and defining SAP HANA Limitations

2114710 HAN-DB-MON FAQ: SAP HANA Threads and Thread Samples

1999993 HAN-DB-MON How-To: Interpreting SAP HANA Mini Check Results

2000003 HAN-DB-ENG FAQ: SAP HANA