SAP Knowledge Base Article

2000000 - FAQ: SAP HANA Performance Optimization

Component: HAN-DB-PERF (SAP HANA > SAP HANA Database > SAP HANA Database Performance), Version:
117, Released On: 26.08.2024

Symptom
The SAP HANA performance is not optimal.

Environment
SAP HANA

Cause
1. Where do I find information about SAP HANA performance optimization provided by SAP?
2. Which prerequisites should be fulfilled in order to be able to perform a detailed performance analysis?
3. How can I check if a performance issue is related to SAP HANA?
4. How can the main contributors to the SAP HANA response time be identified?
5. Which types of SAP HANA performance views exist?
6. Which general checks can be performed in order to optimize the performance of the SAP HANA database?
7. How can the performance of backup, restore and recovery be analyzed?
8. How can long SAP HANA startup times be analyzed?
9. Which SAP services are available for analyzing and optimizing the performance of SAP HANA?
10. What should be considered if SAP HANA hangs completely?
11. How can the performance of commit operations be optimized?
12. How can the performance of rollbacks be optimized?
13. How can the performance of DDL operations be optimized?
14. How can a representative workload be replayed in a different environment?
15. How can automatic actions be triggered in case of bad performance?
16. What are reasons for a non-responsive SAP HANA database without obvious resource or locking issue?
17. Which factors have an impact on the performance of shutdowns?
18. What do I have to consider in terms of database trigger performance?
19. How can I optimize the performance of migrations based on SUM / SWPM / R3load?
20. How can performance issues establishing connections be monitored?

Resolution

1. Where do I find information about SAP HANA performance optimization provided by SAP?
The SAP HANA Troubleshooting and Performance Analysis Guide contains comprehensive information how to optimize
performance of SAP HANA databases.
The Supportability Tools for SAP HANA provide tool-based support for analyzing SAP HANA issues with a focus on
performance.
The SAP HANA Performance Guide for Developers contains performance related information that is particularly useful for
developers.

2. Which prerequisites should be fulfilled in order to be able to perform a detailed performance analysis?
The performance analysis capabilities significantly improve with newer SAP HANA patch levels (e.g. more comprehensive
history information, utilization of STATEMENT_HASH for SQL statements), so it is of advantage to have a recent SAP
HANA patch level in place. See SAP Note 2115815 for more information related to SAP HANA patches and upgrades.
Make sure that the SAP HANA parameters are set based on the SAP recommendations (SAP Note 2186744)
because in this way you implicitly make sure that important monitoring features are enabled, e.g. the expensive statement
trace (SAP Note 2180165) and the thread sample details (SAP Note 2114710).
As of SAP HANA Revision 74 it is recommended to switch to the embedded statistics server (SAP Note 2147247). Among
others this change allows to store important thread sample information for longer times in
HOST_SERVICE_THREAD_SAMPLES.
In order to make sure that sufficient historic information is kept and also comparisons with activities of the previous month
are possible (e.g. comparison of current month's end closing activities with previous month's end closing activities) the
statistics server history retention should be set to 42 days as recommended in SAP Note 2147247. Be aware that this
change increases the memory requirements of the statistics server.

3. How can I check if a performance issue is related to SAP HANA?

In case of performance problems it is essential that you understand at first in which area the regression happens. If you e.g.
perform a migration to SAP HANA and the resulting performance is not as good as expected, you tend to suspect that SAP
HANA is responsible. In a lot of real-life cases this assumption turned out to be wrong and the regression was introduced for
different reasons, e.g. because of the SAP kernel or because of an increased network latency.
In order to avoid unnecessary analysis on SAP HANA side you should at first identify the portion of time spent on SAP HANA
server side, on application side (e.g. within ABAP) and in the network between SAP HANA and the application. Only if the
SAP HANA server side is responsible for a significant fraction of the overall response time, a more detailed investigation on
SAP HANA side is useful.
For more information see SAP Notes 2222200 and 805934. Although the second SAP Note explains details based on the
Oracle database, the principles are also valid for SAP HANA.
If you have upgraded to SAP kernel 7.4x, critical performance regressions can be caused by this change rather than by SAP
HANA. SAP Note 2031037 describes mandatory patch level requirements and actions to make sure that SAP kernel 7.4x is
running optimally. SAP Note 2013043 discusses performance issues linked to the 7.4x enqueue work process.

4. How can the main contributors to the SAP HANA response time be identified?
In order to focus on the most important performance factors it is required to understand where most of the time is lost. On
SAP HANA server side this time distribution can be determined based on the THREAD_STATE (and in case of locks
LOCK_WAIT_NAME) information in views M_SERVICE_THREADS and M_SERVICE_THREAD_SAMPLES. See SAP Note
2114710 for more information.

5. Which types of SAP HANA performance views exist?

The following sources with performance information exist:

Source Location Details

M_* Views Memory Monitoring views containing information since last startup

M_*_RESET Views Memory Resettable counterparts of monitoring views containing information since last res
et

A manual reset can be performed with the following command:

ALTER SYSTEM RESET MONITORING VIEW <reset_view>

Executing this command requires the RESOURCE ADMIN privilege.

HOST_* and GLOBAL_* View Memory + Dis History views populated by statistics server (see SAP Note 2147247)
s k

For more details see the SAP HANA SQL and System Views Reference.

6. Which general checks can be performed in order to optimize the performance of the SAP HANA database?
The following areas should be checked in order to make sure that the SAP HANA database runs as efficiently as possible:

Area Details
SAP HANA pa Current SAP HANA patch levels usually include additional performance optimizations and resolve performance bugs
tch level . Therefore you should make sure that you are on a reasonably new revision level. See SAP Note 2115815 for more det
ails.

SAP HANA an See SAP Notes 2186744 and 2600030 and make sure that the SAP HANA and SAP ABAP parameters are configured
d SAP ABAP p optimally.
arameter setti
ngs

Statement hin Make sure that the statement hints delivered via SAP Note 2700051 are properly implemented so that known scenari
ts os of wrong optimizer decisions are eliminated.

Configuration See SAP Note 2000003 ("How can the configuration and performance of the SAP HANA hardware, firmware and op
of operating s erating system be checked?") and make sure that all recommendations related to operating system and hardware are
ystem and har implemented. Pay particular attention for SAP Note 2131662 and make sure that transparent hugepages are disabled
dware on Linux level.

With SLES >= 12.2 you also have to make sure that the entry [Login] -> UserTasksMax=infinity is defined in /etc/sy
stemd/logind.conf.d/sap.conf as suggested in SAP Note 2205917. Otherwise the operating system may restrict the n
umber of threads that can be opened and this can result in various issues:

Terminations of thread starts with "Thread not started: <thread_type> not started" or "<thread_type> thread
creation error: 11 (Resource temporarily unavailable)" errors
Threads can be stuck in a long time waiting for JobWorker threads (e.g. "JobBarrier Wait for Jobs").

Network Make sure that the network bandwidth and latency fulfills your expectations, both between SAP HANA scale-out nod
es and between SAP HANA and the client application servers. See SAP Note 1100926 for more information.

Make sure that SAP HANA statement routing is configured optimally, so that the amount of network communication
s is minimized (see SAP Note 2200772).

I/O analysis Slow I/O can impact startup times, savepoints, commits or reloads. See SAP Note 1999930 for more information.

SQL statemen Check for expensive SQL statements and optimize them from SAP HANA or application perspective. See SAP Note 2
t optimization 000002 ("FAQ: SAP HANA SQL Optimization") for more information.

Lock analysis Check for lock waits and optimize them from SAP HANA or application perspective. See SAP Note 1999998 ("FAQ: S
AP HANA Lock Analysis") for more information.

Activated trac Activated traces can have an impact on performance, so you should only activate as few traces as possible. The activa
es tion and modification state of some (but not of all) traces can be checked using SQL: "HANA_Traces_ActivatedAnd
ModifiedTraces" (SAP Note 1969700).

Writing database or user trace information (SAP Note 2403162) can result in many threads being blocked by trace fl
ushes. In this case the call stack contains modules like:

TrexTrace::endl
TrexTrace::TraceStream::flush
stream& ltt::impl::ostreamFlush
Diagnose::impl::TraceBuffer::sync
Diagnose::TraceStream::flushTraceBuffer
Diagnose::TraceOutputFileHandlerImpl::doOutput
Diagnose::TraceSegment::reserveEntry

See SAP Note 2380176 and make sure that the database trace (and related traces like the user-specific trace) are only
activated temporarily and as focused as possible.

Connection ha SAP Note 2385992 describes situations and optimizations when initial connections to the database require significa
ndling nt time.

Garbage collec See SAP Note 2169283 and make sure that garbage collection works correctly and isn't permanently blocked.
tion

Application o Make sure that the application efficiently utilizes the SAP HANA database. For S/4HANA environments you can che
ptimization ck SAP Note 2689405 for a collection of performance best practices.
 NUMA See SAP Note 2470289 in order to make sure that NUMA is properly configured and doesn't cause issues.

External tools Make sure that external tools running on the same machine don't have a negative impact on the SAP HANA database
. Possible problem scenarios are:

Unexpectedly high CPU consumption of external tools

Unexpectedly high memory consumption of external tools
Significant use of Linux page cache by external tools: If a particularly high amount of Linux page cache is used
and released by external tools, this can severely impact the SAP HANA database and even temporary standstill
s are possible. It has for example been observed that an unlimited page cache usage of the backup tool azcopy r
esulted in standstills and logshipping timeouts of the SAP HANA database.

Several of these checks can be executed by running the SAP HANA Mini Checks (SQL: "HANA_Configuration_MiniChecks",
SAP Note 1969700). See SAP Note 1999993 for more information.

7. How can the performance of backup, restore and recovery be analyzed?
SAP Note 1835075 describes information that needs to be collected in case of performance issues in the area of backup,
restore and recovery. In many cases the performance is disk I/O bound (SAP Note 1999930).
The following specific backup performance issues are known:
2.00.040 - 2.00.048.00: Increased backup runtimes due to small I/O requests (SAP Note 2930882)
Starting with SAP HANA 2.0 SPS 01 the recovery progress can be monitored in the system DB using monitoring view
SYS_DATABASES.M_RECOVERY_PROGRESS respectively SQL: "HANA_Recovery_Progress_MDC" (SAP Note 1969700).
See SAP Note 3436585 for more information.
If recovery takes long, you can also check for progress information in the database trace files (SAP Note 2380176). Typically
you can find out what kind of operation currently takes place (e.g. recovery, rollback or garbage collection).
Additionally you can generate context lists with hdbcons (SAP Note 2222218) using 'context l -s' in order to understand what
the threads are actually doing. Depending on the call stacks (typically of the LogRecoveryQueue<id> threads) you can draw
the following conclusions:

Call stack Details

 This call stack indicates that a delta storage container is loaded. This can be a consequence of frequent delta
AttributeEngine:
:BTreeNodeGroupN merges during recovery with SAP HANA 1.0 because the delta storage of concat attributes / multi column ind
ode::getEntryInd exes isn't persisted (SAP Note 2160391 -> "Are indexes persisted to disk?") while at the same time the log repl
ex ay maintains persistence pages. Therefore the delta storage of the concat attribute has to be recreated repeate
AttributeEngine: dly.
:BtreeAttribute:
If you repeatedly suffer from this scenario for some tables, you can consider to adjust the auto merge decision
:Delta::DeltaBTr
ee::findLeafNode
function or the smart merge logic in order to minimize delta merges during normal production operation (SA
AttributeEngine: P Note 2057046).
:BtreeAttribute: Starting with SAP HANA 2.0 SPS 01 the log replay updates the in-memory representation and so the overhea
:Delta::DeltaBTr d no longer exists.
ee::insert_load
AttributeEngine:
:BtreeAttribute:
:Delta::DeltaBTr
ee::load
AttributeEngine:
:BtreeAttribute:
:DeltaContainer:
:load
AttributeEngine:
:BTreeAttribute:
:load
AttributeEngine:
:BTreeAttribute:
:open
AttributeEngine:
:openAvc
AttributeEngine:
:AttributeStore:
:loadAttribute
AttributeEngine:
:AttributeApi::l
oadAttribute
AttributeEngine:
:AttributeApi::l
oadAttributes
__pread_nocancel
This call stack indicates that a rollback takes place. See "How can the performance of rollbacks be optimized?"
System::UX::prea for details.
d
FileAccess::file
Read
FileAccess::Loca
lFile::read
PageAccess::Data
VolumeImpl::gene
ricPageIO
PageAccess::Data
VolumeImpl::read
Page
PageAccess::Page
IOImpl::readPage
PageAccess::Logi
calPageAccessImp
l::loadPageInter
nal
PageAccess::Logi
calPageAccess::l
oadPage
DataContainer::P
ageChainContaine
rImpl::deletePag
eImpl
DataContainer::V
irtualFileImpl::
deletePageIntern
alDoWork
DataContainer::V
irtualFileImpl::
deletePageIntern
al
DataContainer::V
irtualFileImpl::
deletePageInUndo
DataContainer::V
irtualFileUndo::
undoDoWork
DataContainer::V
irtualFileUndo::
undo
DataAccess::Undo
FileAnchor::proc
ess
DataAccess::Pers
istenceSession::
txRollbackCallba
ck
TransactionManag
er::RollbackLogR
ecord::redo
 syscall
This step at the end of a restore makes sure that all restored pages are flushed to disk. A SubmitThread-DATA
Synchronization: -0 is triggered that performs the actual I/O writes in FileAccess::ioSubmit.
:Semaphore::time From the trace files it can appear like there is no real progress for some time. The last lines of the backup log i
dWait ndicate that the recovery progress is already 100 %:
ResourceManager:
:ResourceFlushCo 2019-10-06T17:05:29+05:30 INFO RECOVERY progress of service: indexserver, hanahost:32103,
unterAndState::w volume: 3, 95% 448824082432/468017217536
aitFor 2019-10-06T17:07:04+05:30 INFO RECOVERY progress of service: indexserver, hanahost:32103,
ResourceManager: volume: 3, 100% 468017217536/468017217536
:WaitAndSwitchCo
unterResource::w
In the indexserver trace the following messages are printed on a regular basis:
aitForFlush
DataAccess::Pers
i Service_Startup GlobalTransCommImpl.cpp(06358) : Processing Slave_GetMasterLogPos for volu
istenceManagerIm
me 2
pl::RecoveryHand
i Service_Startup GlobalTransCommImpl.cpp(06366) : Finished processing Slave_GetMasterLogPos
lerImpl::finish
, lsn=0xffffffffffffffff
DataAccess::Data
RecoveryHandler:
This scenario can remain for a longer time in case of bottlenecks in the I/O area. You can check the SAP HAN
:finish
A I/O behavior based on SAP Note 1999930. In context of the restore you can use the following hdbcons com
Backup::BackupSr
c_CommonSnapshot
mand to extract I/O figures from the database:
::executeRestore
hdbcons 'statreg print -h -n M_VOLUME_IO_TOTAL_STATISTICS'
Backup::BackupEx
e_JobRestore::ru
nBackupJob Before being written down to disk, restored pages are stored in the following heap allocator (SAP Note 19999
97, name is version dependent):

Pool/PersistenceManager/PersistentSpace/DefaultLPA/Page
Pool/PersistenceManager/PersistentSpace/DefaultLPA/DataPage

8. How can long SAP HANA startup times be analyzed?

See SAP Note 2222217 for analyzing and optimizing SAP HANA startup times.

9. Which SAP services are available for analyzing and optimizing the performance of SAP HANA?
The most important service for analyzing and optimizing the performance of SAP HANA is:
SAP Technical Performance Optimization - HANA (TPO)
As part of this service SAP collects current architecture, runtime and performance data, identifies issues and risks related to
SAP HANA and analyzes the longest running SQL statements. It can also include an empowering component, i.e. a workshop
with a joint SAP HANA analysis and detailed technical discussions.
Several other SAP services can also include SAP HANA performance analysis:
Business Process Performance Optimization Service (BPPO)
Volume Test Optimization Service (VTO)
Customer Program Optimization Service (CPO)
SAP GoingLive Support Service (GLS)
SAP EarlyWatch Check
SAP HANA System Administration Service
Solution Manager Self Service "SQL Statement Tuning" (SAP Note 1601951)
For more information see the SAP Support Programs and Services Overview.

10. What should be considered if SAP HANA hangs completely?

While a database performance issue is usually local and still allows to perform analysis, a hang situation prevents every access
to SAP HANA. Both local performance issues and hang situations can be linked to the same root causes, so the checks
described in "Which general checks can be performed in order to optimize the performance of the SAP HANA database?"
should also be considered in case of a hang situation. In addition SAP Note 1999020 describes typical reasons for hang
situations and possible checks.
11. How can the performance of commit operations be optimized?
During commits the changes of the transaction are persisted to disk and made visible for other transactions. The following
factors can influence the time of commit operations:

Reaso Details
n

Disk I/ During commits the change information is written to the log files. If writing down the data is slow, the commit times can inc
O write rease. See SAP Note 1999930 and make sure that writing to the logs is performed with an acceptable performance.
perfor
mance
to logs

Synchr If a synchronous system replication mode is activated, changes are propagated to the replication side during a commit, and
onous the local session has to wait for an acknowledgement. Among others, semaphore waits for synchronousReplicationSemapho
system re may be observed (SAP Note 1999998). If you suffer from long commit times in a system with activated synchronous syste
replica m replication, you should at first check if a switch to a less critical system replication mode (e.g. SYNC -> SYNCMEM or SY
tion m NCMEM -> ASYNC) improve the performance. If yes, you can analyze if there are unnecessary replication delays, e.g. due t
ode o limited network bandwidth or high latency times. See SAP Note 1999880 for more information related to SAP HANA syst
em replication.

Synchr If a transaction performs changes on a table with activated optimistic synchronous table replication (OSTR), the replicas ne
onous t ed to be refreshed during commit. This can have an adverse impact on the commit performance.
able re
Delta storage contention on replicas can significantly impact the runtime of refreshes and AsyncReplicationDMLLogReplay
plicatio
er threads have call stacks similar to:
n
syscall
Synchronization::BinarySemaphore::timedWait
AttributeEngine::Delta::_acquireRetry
AttributeEngine::BTreeAttribute::commitOptimize
CSWriter::AttributeIndexJob::run
AttributeEngine::AttributeApi::commitOptimizeAttributes
CSAccessor::Dml::DmlExecutor::writeData
DmlExecutorFactory.cpp.
CSDmlAccessorApi::processInsertContext
TRexAPI::TableUpdateWriter::writeIntoTableContainer
TRexAPI::TableUpdateWriter::execute
TRexAPI::Dml::ExecuteWriter::execute
TRexAPI::TableUpdate::writeDataToReplicate
TRexAPI::TableUpdateRequestHandler::handleAsyncReplication
Distribution::ReplayerInterface::TUHandleAsyncReplication
DMLLogReplayer.cpp
Distribution::DMLLogQueueProcessor::processDMLLog
Distribution::DMLLogQueueProcessorThread::run

In this case you can proceed as follows:

Reduce delta storage contention (SAP Note 1999998 -> "BTree GuardContainer")
Upgrade to SAP HANA >= 2.00.070 where the impact of table replication refresh delays on commits is reduced (issue
number 282475).

See SAP Note 2340450 for more information related to SAP HANA table replication.

Wait st You can check the commit related thread states (e.g. using SQL: "HANA_Threads_ThreadSamples_FilterAndAggregation
ates ", THREAD_METHOD = 'CommitTrans', available via SAP Note 1969700) and analyze states different from "Running" wit
h a significant contribution of thread samples more in detail. See SAP Note 2114710 for more details. For example, significa
nt "Network Poll" waits can indicate weaknesses in the infrastructure or configuration area (SAP Note 2222200).

High a An unnecessary high amount of commit requests can result in overhead and performance regressions. You can use the follo
mount wing analysis commands available via SAP Note 1969700 to do a drill-down:
of com
SQL: "HANA_Workload" (column COMMIT_PER_S): Commit frequency over time
mit req
SQL: "HANA_Connections_Statistics" (columns COMMIT, COM_PER_S): Commits per connection
uests
 High a The commit performance can be significantly impacted by a high amount of active versions. See SAP Note 1694864 for mor
mount e details.
of activ
e versi
ons

Integra If a SAP HANA integrated liveCache is used, the SAP HANA commit performance can be impacted by related commits on li
ted live veCache side. You can use SQL: "HANA_liveCache_Procedures" (OBJECT_NAME = 'TRANS-END') available via SAP Note
Cache 1969700 to check for a commit time breakdown on liveCache side.

The time for the 'Kernel-Commit' method is part of the SAP HANA commit time. Other related times like 'Flush-Cache', 'Co
mmit-Invalidate-Callback' and 'Validate-Callback' have to be considered on top of the SAP HANA commit time.

During a liveCache commit a potentially large amount of data has to be flushed, so in case of high commit times you should
always consider the amount of flushed data to judge if it is more an I/O issue or a data volume issue.

See SAP Note 2593571 for more details related to the integrated liveCache.

Log vol Encryption of log volumes imposes a certain overhead when writing down redo log data to disk and so commit times can inc
ume en rease. Usually the overhead is limited and acceptable. See SAP Note 2159014 for more information.
cryptio
Encryption of log volumes is available starting with SAP HANA 2.0. You can use SQL: "HANA_Encryption_Overview" (SA
n
P Note 1969700) in order to check the encryption configuration.

Log ba When log backups are executed, I/O read requests are executed against the log file system that can content with the I/O wri
ckups te requests of commits. This contention can be reduced by lowering the log backup buffer size (default: 128 MB) e.g. to 1 MB
(SAP Note 2618564):

global.ini -> [backup] -> log_backup_buffer_size = 1

Be aware that this reduction is only useful when commit time peaks correspond to times of log backups.

Issue i Committing is a global operation in a multi-tenant system, so a problem in any of the involved database (system DB, tenant
n other s) can be responsible for longer commit times. If you can't identify a root cause in the local database, you need to perform th
MDC d e same checks also in the system DB and other tenants.
atabas
e

SAP H The following SAP HANA bugs can be responsible for increased commit times or hanging commits:
ANA b
SAP Note / issue num Impacted Revisi Details
ugs
ber on

2439289 122.06 Overhead in internal commit processing design

2.00.000

317865 PostCommitExecutor deadlock related to liveCache with lock "OMS RWLock: AREAID: 1000013, L
OCKID: 1"

The related call stacks are:

syscall
Synchronization::ReadWriteLock::timedWaitLockSharedLL
Synchronization::ReadWriteLock::lockShared
OMS_RWLock::enter
OmsReleasableScope::OmsReleasableScope
...
ptime::PostCommitExecutor::run

This scenario can only be resolved with a SAP HANA restart.

If commits take very long, a termination with error "snapshot timestamp synchronization failed" is possible (SAP Note
2399990).
SQL: "HANA_Transactions_History" (SAP Note 1969700) can be used to display commit related times of recent transaction
executions. Important columns are:
COMMIT_MS: Last commit duration (ms)
 PREP_COMM_MS: Prepare commit duration (ms), can be increased in scale-out when two-phase commit is delayed,
not related to actual log I/O
PREP_LOG_MS: Prepare log duration (ms)

12. How can the performance of rollbacks be optimized?

Rollbacks are usually not critical, because they don't happen too frequently and they don't have a global impact, but there are
certain exceptions. For example, a rollback can delay the SAP HANA startup significantly. See SAP Note
2222217 ("Termination of indoubt transactions") for more information. If you face a long running rollback during startup,
you can proceed in the following way:

Action Details

Generat See SAP Note 2313619 and generate call stacks of the SAP HANA threads in order to identify the activities of threads res
e call sta ponsible for the rollback.
cks
The threads are called LogRecoveryQueue<id> and a typical call stack would be:

54220[thr=120221]: LogRecoveryQueue53 at
1: 0x00007fef1f223113 in __pread_nocancel+0x27 (libpthread.so.0)
2: 0x00007fef228504e5 in System::UX::pread (libhdbbasis.so)
3: 0x00007fef22665bb5 in FileAccess::fileRead (libhdbbasis.so)
4: 0x00007fef22643a03 in FileAccess::LocalFile::read (libhdbbasis.so)
5: 0x00007fef2d25837e in PageAccess::DataVolumeImpl::genericPageIO (libhdbdataaccess.so)
6: 0x00007fef2d258fcd in PageAccess::DataVolumeImpl::readPage (libhdbdataaccess.so)
7: 0x00007fef2d29e0a9 in PageAccess::PageIOImpl::readPage (libhdbdataaccess.so)
8: 0x00007fef2d27dbab in PageAccess::LogicalPageAccessImpl::loadPageInternal (libhdbdataaccess.so)
9: 0x00007fef2d27ea01 in PageAccess::LogicalPageAccess::loadPage (libhdbdataaccess.so)
10: 0x00007fef2d0d1c85 in DataContainer::PageChainContainerImpl::deletePageImpl (libhdbdataaccess.so)
11: 0x00007fef2d0f1597 in DataContainer::VirtualFileImpl::deletePageInternalDoWork (libhdbdataaccess.s
o)
12: 0x00007fef2d0f1a47 in DataContainer::VirtualFileImpl::deletePageInternal (libhdbdataaccess.so)
13: 0x00007fef2d10507b in DataContainer::VirtualFileImpl::deletePageInUndo (libhdbdataaccess.so)
14: 0x00007fef2d156c46 in DataContainer::VirtualFileUndo::undoDoWork (libhdbdataaccess.so)
15: 0x00007fef2d15b897 in DataContainer::VirtualFileUndo::undo (libhdbdataaccess.so)
16: 0x00007fef2cfe5bc8 in DataAccess::UndoFileAnchor::process (libhdbdataaccess.so)
17: 0x00007fef2cf6c521 in DataAccess::PersistenceSession::txRollbackCallback (libhdbdataaccess.so)
18: 0x00007fef2d2bb0da in TransactionManager::RollbackLogRecord::redo (libhdbdataaccess.so)
19: 0x00007fef2d1e2685 in DataRecovery::ProxyPseudoLogRecord::redo (libhdbdataaccess.so)
20: 0x00007fef2d205c33 in DataRecovery::RecoveryQueue::run (libhdbdataaccess.so)
21: 0x00007fef22601f29 in Execution::Thread::staticMainImp (libhdbbasis.so)
22: 0x00007fef22602dfd in Execution::Thread::staticMain (libhdbbasis.so)

Modules like __pread_nocancel indicate read I/O activities (SAP Note 1999930).

Check m Once you have identified the main activities during rollback based on the call stacks, you can check SAP HANA monitori
onitorin ng views for details. During startup no SQL can be executed against the database, but instead you can use hdbcons for t
g views hat purpose. For example, you can evaluate M_VOLUME_IO_DETAILED_STATISTICS via hdbcons in the following w
ay in case the call stacks indicate I/O activities (SAP Note 2222218):

Action Command

Reset detailed I/O statistics

hdbcons 'statreg reset -n M_VOLUME_IO_DETAILED_STATISTICS_RESET'

Wait some time (>= 1 minute) to record current I/O behavior

Display I/O activities since reset

hdbcons 'statreg print -h -n M_VOLUME_IO_DETAILED_STATISTICS_RESET'

You can then load the resulting output into Excel and determine the times spent for different I/O types and page sizes. I
n this way you can find out how many pages of which size have been read, how long the average read time was and how l
ong reading took in total.
 Optimiz If you conclude that I/O is the main contributor to the rollback runtime, the following options exist to improve it:
e I/O
If the average I/O times determined in the last step are higher than expected: Check in collaboration with your OS
and hardware partners if there is a bottleneck or misconfiguration in the I/O stack that has a negative impact on t
he average I/O times (SAP Note 1999930).
If you face a high number of reasonably fast I/O requests, you can only consider a - temporary - move of the under
lying data file to a really fast storage (e.g. SSD) or a RAM disk. Prefetching and page cache on Linux level doesn't h
elp, because SAP HANA always performs direct I/O bypassing the operating system page cache.

Implem Starting with SAP HANA 1.00.122.11 and 2.00.020 the performance of rollback is optimized, so an upgrade to these min
ent SAP imum Revision levels may help to speed up rollbacks.
HANA fi
xes

Proactiv In order to avoid critical rollback times in the future, you should consider the following:
e steps
Avoid long running update transactions with a very high number of modifications from a business perspective, bec
ause a rollback in case of a termination or crash can take significant times.
Monitor the system - in particular before performing a planned SAP HANA shutdown - in order to detect a signific
ant rollback / undo generation, e.g. via check ID M0856 ("Max. undo size of current transaction (MB)" of the SAP
HANA Mini Checks (SAP Note 1999993) or via SQL: "HANA_Transactions_UndoAndRedoLog" (SAP Note 19697
00).
A rollback of row store table information typically requires more I/O activities than a rollback of column store tabl
es. Therefore you should check if you can move critical tables to the column store. See SAP Note 2222277 for more
information.
A high amount of individual 4 KB page reads can be linked to row store tables with hybrid LOBs (SAP Note 22206
27), because every LOB container is read from disk individually. Check if you can move tables with a high amount
of disk LOB activity from row store to column store (SAP Note 2222277). An optimization is available with SAP H
ANA 1.00.121.

13. How can the performance of DDL operations be optimized?

For the performance of DDL operations similar rules apply like for normal SQL statements. The table below contains specific
details that are particularly helpful for certain DDL operations:

DD SA Optimizations
Lt P
ype N
ot
e
 Ind 21 Consider the following optimizations:
ex c 60
If possible, create the index before the underlying table is populated with data.
reat 39
If the underlying uniqueness check executed by the join engine takes quite long, multi-threaded execution may imp
ion 1
rove the performance. This can be enforced by setting the parameter indexserver.ini -> [joins] -> single_thread_exe
cution_for_partitioned_tables to 'false'.
In case of a column store table and a unique / primary index it is helpful to perform a delta merge (SAP Note 20570
46) and make sure that the delta storage remains empty while the index is created. This eliminates the need to com
bine main and delta dictionaries required for uniqueness check purposes.
Starting with SAP HANA 2.00.045 inverted value indexes are created bulk wise to reduce memory consumption. Th
is can negatively impact the performance. You can set the parameter indexserver.ini -> [global] -> create_index_op
timize_bulksize (default: 40000000 for SAP HANA <= 2.0 SPS 06, 0 respectively individual calculation based on
memory limits for SAP HANA >= 2.0 SPS 07) to a higher value in order to shift resource consumption from CPU to
memory. A value of 0 or higher than the table rows deactivates bulk processing completely and provides optimal per
formance at the cost of high memory consumption. Be aware that this setting only applies to column loadable index
es, so if page loadable / NSE (SAP Note 2799997) is active for the index, it can't be used.
For indexes on partitioned tables the parameter indexserver.ini -> [ddl] -> max_number_of_data_jobs can be used
to balance between performance, CPU and memory. Per default, this parameter is only limited by the general concu
rrency settings (SAP Note 2222250), resulting in potentially high CPU and memory consumption and good perform
ance. By reducing it you can reduce the number of concurrently processed partitions and as a consequence also the
peak memory utilization. For example, max_number_of_data_jobs = 2 makes sure that only two partitions are pro
cessed at the same time and so also the CREATE INDEX memory overhead is lower compared to a parallel processi
ng of more than two partitions. At the same time the overall CREATE INDEX runtime can increase.

Tab 20 See SAP Note 2044468 ("How can the performance of repartitioning activities be influenced?") for details.
le p 44
arti 46
tion 8
ing

14. How can a representative workload be replayed in a different environment?

Optimal testing of a system change like a SAP HANA upgrade or a modification of the hardware layout requires the possibility
to replay a representative workload in the new environment. Starting with SAP HANA SPS 12 the built-in tool Capture and
Replay is available for that purpose. See SAP Note 2669621 for more information.

15. How can automatic actions be triggered in case of bad performance?

It is quite important to collect additional information like runtime dumps (SAP Note 2400007) at times of an unresponsive or
slow system in order to make sure that the later root cause analysis can be performed successfully. You can schedule SAP
HANASitter (SAP Note 2399979) in order to make sure that analysis data is collected during suspicious time frames.

16. What are reasons for a non-responsive SAP HANA database without obvious resource or locking issue?
In most cases a slow or hanging SAP HANA database is caused by lock scenarios (SAP Note 1999998) or high resource
consumption (e.g. CPU, SAP Note 2100040). Nevertheless there can also be scenarios where the database is slow or stuck and
where you can find neither obvious locks nor high resource consumption:

Scenario SAP Details

Note

Wrong parame 2600 Wrong parameters (e.g. a very small max_concurrency value or a restricted affinity setting) can significantly
ter settings 030 restrict the resources that can be used by SAP HANA so you have to make sure that all parameters are config
ured based on SAP best practices.

CPU restrictio In some scenarios it is possible that on layers below SAP HANA the amount of assigned CPUs or the CPU aff
ns on lower lay inity is limited for SAP HANA. Make sure that no unintended limitations exist.
ers

Call stack gene 2313 If you use gstack to generate SAP HANA call stacks, it can block the whole SAP HANA instance while it is ex
ration via gsta 619 ecuted. Neither CPU consumption is increased nor there are any dedicated locks visible. All activities are sus
ck pended until gstack is finished.
17. Which factors have an impact on the performance of shutdowns?
Starting with SAP HANA 2.0 SPS 06 you can use SQL: "HANA_Startup_StepDuration" (DATA_SOURCE = 'HISTORY',
PHASE = 'STOP%') to list steps that were done during historic service shutdowns including runtimes. This can sometimes
help to identify reasons for increased shutdown durations.
The following overview lists typical scenarios that can result in long SAP HANA shutdown times. When a shutdown takes too
long, it may be terminated with a timeout. See SAP Note 2177064 ("What happens when a shutdown takes a long time?") for
more details.

Sce Details
nari
o

Roll In case transactions with many changes were still running at the time of the shutdown, they have to be rolled back. You can us
back e SQL: "HANA_Transactions_UndoAndRedoLog" (SAP Note 1969700) to check how much undo was already created by runn
ing transactions.

Me Releasing allocated memory back to the operating system requires some time. Typically it takes around 1 to 2 minutes to relea
mor se 1 TB of memory. So on systems with a particularly high amount of allocated memory a successful shutdown can be delayed
y rel significantly.
ease
You can use SQL: "HANA_Memory_Overview" (SAP Note 1969700) to check for the memory that is currently allocated and
used by SAP HANA. If the allocated memory is much higher than the used memory, you can consider a manual memory defra
gmentation using the "mm gc -f" option of hdbcons (SAP Note 2222218) before doing the actual shutdown. With this comman
d the allocated memory will already be defragmented and released while the system is still up and running and so the subsequ
ent memory release time during shutdown will be shorter.

See SAP Note 1999997 for more information related to SAP HANA memory.

If you try to start the same service again while the previous one is still releasing memory (being in "defunct" state), the startup
can fail with the following errors (SAP Note 2222200):

can't listen on port <host>:<port>: (host unknown)

rc=98: Address already in use

18. What do I have to consider in terms of database trigger performance?

See SAP Note 2800020 -> "What do I have to consider in terms of database trigger performance?" for more about
performance aspects of SAP HANA triggers.

19. How can I optimize the performance of migrations based on SUM / SWPM / R3load?
The document Best Practice Guide - Classical Migration of SAP NetWeaver AS ABAP to SAP HANA provides a lot of useful
details including aspects relevant for performance. SAP Note 2000002 -> "How can the performance of INSERTs and data
loads be tuned?" can be helpful to optimize the import performance.

20. How can performance issues establishing connections be monitored?

With SAP HANA >= 2.00.059.04 and >= 2.00.063 long runtimes or timeouts when establishing connections to the SAP
HANA database are reported.
The following parameters control this features:

Parameter Defa Uni Details

ult t

2000 ms Minimum connection establish time to be reported i

indexserver.ini -> [session] -> connection_establish
_statistics_threshold n the database trace

true Activation / deactivation of the feature

indexserver.ini -> [session] -> enable_connection_es
tablish_statistics
When an connection establishment exceeds the configured limit, details about the different runtime contributors are written
to the database trace (SAP Note 2380176).
Attention: This feature can result in crashes due to race conditions on the underlying connStatMutex lock (issue number
303538). As a workaround it can be disabled via:

indexserver.ini -> [session] -> enable_connection_establish_statistics = false

Example: (time spent within SAP HANA server)

CONNECTION ESTABLISHMENT:
ESTABLISHED OPEN_TS=1667287667374733 (2022-11-01 08:27:47.374733)
ESTABLISHED_TS=1667287671737762 (2022-11-01 08:27:51.737762)
TOTAL_ELAPSED_TIME=4363029
- TOTAL_CLIENT_ROUND_TIME=1958
- TOTAL_SERVER_TIME=4361071
[0] INITIAL REQUEST TIME=4342083
- RECEIVE_TS=1667287667375027 (2022-11-01 08:27:47.375027)
- RECEIVE_TIME=4341754
- RECEIVE_CALL_TIME=4322898
- RECEIVE_CALL_COUNT=2
- SERVER_PROCESSING_TIME=306
- JOB_DISPATCH_WAIT_TIME=86
- SSL_ACTIVATE_TIME=18853
- SEND_TIME=23
- SENT_TS=1667287671716816 (2022-11-01 08:27:51.716816)
[1] AUTHENTICATE REQUEST TIME=737
- CLIENT_ROUND_TIME=1704
- RECEIVE_TS=1667287671718520 (2022-11-01 08:27:51.718520)
- RECEIVE_TIME=49
- RECEIVE_CALL_TIME=42
- RECEIVE_CALL_COUNT=1
- SERVER_PROCESSING_TIME=644
- JOB_DISPATCH_WAIT_TIME=74
- AUTHENTICATE_EVALUATE_TIME=413
- AUTHENTICATE_METHOD=SCRAMSHA256=0
- SEND_TIME=44
- SENT_TS=1667287671719257 (2022-11-01 08:27:51.719257)
[2] CONNECT REQUEST TIME=18251
- CLIENT_ROUND_TIME=254
- RECEIVE_TS=1667287671719511 (2022-11-01 08:27:51.719511)
- RECEIVE_TIME=15
- RECEIVE_CALL_TIME=13
- RECEIVE_CALL_COUNT=1
- SERVER_PROCESSING_TIME=18206
- JOB_DISPATCH_WAIT_TIME=50
- AUTHENTICATE_EVALUATE_TIME=49
- AUTHENTICATE_TIME/PRE=134
- AUTHENTICATE_TIME/POST/LDAP_AUTHORIZATION=68
- AUTHENTICATE_TIME/POST=12202
- AUTHENTICATE_TIME=12414
- SEND_TIME=30
- SENT_TS=1667287671737762 (2022-11-01 08:27:51.737762)

In this example the overall connection establishment time is around 4.36 seconds and most of the time is spent with the SAP
HANA database server (+ network transfer time). Thus, you can start to analyze the SAP HANA database for (temporary)
performance issues and bottlenecks that can contribute to increased server times.
Example: (time spent on client side)
 CONNECTION ESTABLISHMENT:
ESTABLISHED OPEN_TS=1667295051307880 (2022-11-01 10:30:51.307880)
ESTABLISHED_TS=1667295053945292 (2022-11-01 10:30:53.945292)
TOTAL_ELAPSED_TIME=2637412
- TOTAL_CLIENT_ROUND_TIME=2574608
- TOTAL_SERVER_TIME=62804
[0] INITIAL REQUEST TIME=42182
- RECEIVE_TS=1667295051308140 (2022-11-01 10:30:51.308140)
...

In this case the client (SAP Note 2393013) and network connectivity (SAP Note 2222200) need to be inspected.
Example: (JobWorker bottleneck)

CONNECTION ESTABLISHMENT:
ABORTED OPEN_TS=1680790399893438 (2023-04-06 16:13:19.893438)
TOTAL_ELAPSED_TIME=88343191
- TOTAL_CLIENT_ROUND_TIME=78068
...
- SENT_TS=1680790400121801 (2023-04-06 16:13:20.121801)
[2] CONNECT REQUEST TIME=88059526 - ABORTED
- CLIENT_ROUND_TIME=55302
- RECEIVE_TS=1680790400177103 (2023-04-06 16:13:20.177103)
- RECEIVE_TIME=53
- RECEIVE_CALL_TIME=21
- RECEIVE_CALL_COUNT=1
- SERVER_PROCESSING_TIME=88059473
- JOB_DISPATCH_WAIT_TIME=87831032
- AUTHENTICATE_EVALUATE_TIME=62
- AUTHENTICATE_TIME/PRE=230
- AUTHENTICATE_TIME/POST/LDAP_AUTHORIZATION=92
- AUTHENTICATE_TIME/POST=181988
- AUTHENTICATE_TIME=182338
- SEND_TIME=0
- SENT_TS=0 (<none>)

High JOB_DISPATCH_WAIT_TIME values typically indicate JobWorker bottlenecks. Make sure that workload settings are
properly in place (SAP Note 2222250), the system isn't overloaded by database requests (SAP Note 2000002) and NUMA is
properly configured (SAP Note 2470289).

Keywords
SAP HANA performance stuck hang troubleshooting slow

Attributes
Key Value

Other Components SAP HANA > SAP HANA Database (HAN-DB)

Products
Products

SAP HANA, platform edition all versions

 This document refers to
SAP Note/KBA Component Title

3436585 HAN-DB-BAC How to check recovery status of HANA system via SQL

2800020 HAN-DB FAQ: SAP HANA Triggers

2799997 HAN-DB-NSE FAQ: SAP HANA Native Storage Extension (NSE)

2700051 HAN-DB Delivery of Statement Hints (SAP HANA >= 1.00.122.03)

2689405 XX-SER-MCC FAQ: SAP S/4HANA Performance Best Practices - Collective Note

2600030 HAN-DB Parameter Recommendations in SAP HANA Environments

2593571 BC-DB-LCA FAQ: SAP HANA Integrated liveCache

2470289 HAN-DB FAQ: SAP HANA Non-Uniform Memory Access (NUMA)

2400007 HAN-DB-PERF FAQ: SAP HANA Runtime Dumps

2399990 HAN-DB How-To: Analyzing ABAP Short Dumps in SAP HANA Environments

2399979 HAN-DB-PERF How-To: Configuring automatic SAP HANA Data Collection with SAP HANASitter

2393013 HAN-DB FAQ: SAP HANA Clients

2380176 HAN-DB FAQ: SAP HANA Database Trace

2340450 HAN-DB FAQ: SAP HANA Table Replication

2313619 HAN-DB-MON How-To: Generating and Evaluating SAP HANA Call Stacks

2222277 HAN-DB-PER FAQ: SAP HANA Column Store and Row Store

2222250 HAN-DB-PERF FAQ: SAP HANA Workload Management

2222218 HAN-DB FAQ: SAP HANA Database Server Management Console (hdbcons)

2222217 HAN-DB How-To: Troubleshooting SAP HANA Startup Times

2222200 HAN-DB FAQ: SAP HANA Network

2220627 HAN-DB FAQ: SAP HANA LOBs

2200772 HAN-DB FAQ: SAP HANA Statement Routing and Client Distribution Mode

2186744 HAN-DB FAQ: SAP HANA Parameters

2185955 HAN-DB SAP HANA startup hangs due to orphan nameserver process

2180165 HAN-DB FAQ: SAP HANA Expensive Statements Trace

2177604

2177064 HAN-DB-ENG FAQ: SAP HANA Service Restarts and Crashes

2169283 HAN-DB-ENG FAQ: SAP HANA Garbage Collection

2160391 HAN-DB-ENG FAQ: SAP HANA Indexes

2159435 HAN-DB How-To: Keeping SAP HANA Row Store in Memory when restarting

2159014 HAN-DB-SEC FAQ: SAP HANA Security

2147247 HAN-DB-MON FAQ: SAP HANA Statistics Server

2119087 HAN-DB How-To: Configuring SAP HANA Traces

2115815 HAN-DB FAQ: SAP HANA Database Patches and Upgrades

2114710 HAN-DB-MON FAQ: SAP HANA Threads and Thread Samples

2100040 HAN-DB-PERF FAQ: SAP HANA CPU

2095794 HAN-DB-HA Nameserver startup hangs due to socket wait

2057046 HAN-DB-ENG FAQ: SAP HANA Delta Merges

2044468 HAN-DB-ENG FAQ: SAP HANA Partitioning

2000002 HAN-DB-PERF FAQ: SAP HANA SQL Optimization

1999998 HAN-DB-PERF FAQ: SAP HANA Lock Analysis

1999997 HAN-DB FAQ: SAP HANA Memory

1999993 HAN-DB-MON How-To: Interpreting SAP HANA Mini Check Results

1999930 HAN-DB FAQ: SAP HANA I/O Analysis

1999880

1813020 HAN-DB How to generate a runtime dump on SAP HANA

2930882 HAN-DB-BAC Potentially Increased Data Backup Or Full Data Shipment Runtime in SAP HANA 2 SPS04

2669621 HAN-DB FAQ: SAP HANA Capture and Replay

2618564 HAN-DB Optimizing High DML Commit Time During Log Backups

2439289 HAN-DB Multiple PostCommitExecutor Threads in Status ConditionalVariable Wait

2205917 HAN-DB SAP HANA DB: Recommended OS settings for SLES 12 / SLES for SAP Applications 12

1969700 HAN-DB SQL Statement Collection for SAP HANA

Supportability Tools for SAP HANA

SAP HANA SQL and System Views Reference

SAP Support Programs and Services

SAP HANA Performance Guide for Developers

Best Practice Guide - Classical Migration of SAP NetWeaver AS ABAP to SAP HANA

This document is referenced by

SAP Component Title
Note/KBA

2177064 FAQ: SAP HANA Service Restarts and Crashes

2100009 HAN-DB-ENG FAQ: SAP HANA Savepoints

2222250 HAN-DB-PERF FAQ: SAP HANA Workload Management

2999990 HAN-DB-PERF How-To: SAP HANA Performance Analysis

3133914 CA-LT-SLT TIME_OUT dump when replicating tables in SLT. - SAP SLT

3051992 HAN-DB-ENG SAP HANA takes longer to start after a crash or ungraceful shutdown

3065607 CA-LT-MC Performance tips & tricks for SAP S/4HANA Migration Cockpit: Migrate Data Using Staging
Tables

2800020 HAN-DB FAQ: SAP HANA Triggers

3007114 HAN-DB-BAC Memory size requirement for HANA recovery

2346988 BC-CTS Long Runtime in transport step "Generation of Programs and Screens"
<SourceSID>G9<number>.<targetSID>

2666775 HAN-DB-ENG DML Operations through SAP Replication Server (SRS) to HANA are sub-optimal

2908549 HAN-DB-ENG SAP HANA Database requests fail with error code 1038

2313619 HAN-DB-MON How-To: Generating and Evaluating SAP HANA Call Stacks

2800008 HAN-DB-ENG- FAQ: SAP HANA Fulltext Indexes

TXT

2160391 HAN-DB-ENG FAQ: SAP HANA Indexes

1999997 HAN-DB FAQ: SAP HANA Memory

2699939 HAN-DB SAP HANA Emergency Suitcase

2159014 HAN-DB-SEC FAQ: SAP HANA Security

2600030 HAN-DB Parameter Recommendations in SAP HANA Environments

2380176 HAN-DB FAQ: SAP HANA Database Trace

2393013 HAN-DB FAQ: SAP HANA Clients

1999930 HAN-DB FAQ: SAP HANA I/O Analysis

2399990 HAN-DB How-To: Analyzing ABAP Short Dumps in SAP HANA Environments

2400024 HAN-DB-MON How-To: SAP HANA Administration and Monitoring

2399979 HAN-DB-PERF How-To: Configuring automatic SAP HANA Data Collection with SAP HANASitter

1999998 HAN-DB-PERF FAQ: SAP HANA Lock Analysis

2222218 HAN-DB FAQ: SAP HANA Database Server Management Console (hdbcons)

2222217 HAN-DB How-To: Troubleshooting SAP HANA Startup Times

2222277 HAN-DB-PER FAQ: SAP HANA Column Store and Row Store

2147247 HAN-DB-MON FAQ: SAP HANA Statistics Server

2222200 HAN-DB FAQ: SAP HANA Network

2000002 HAN-DB-PERF FAQ: SAP HANA SQL Optimization

2222110 HAN-DB-PERF FAQ: SAP HANA Load History

2180165 HAN-DB FAQ: SAP HANA Expensive Statements Trace

2114710 HAN-DB-MON FAQ: SAP HANA Threads and Thread Samples

948066 HAN-DB-MON Performance Analysis: Transactions to use

2105467 CA-MDG MDG Performance