SAP R/3 Performance

Troubleshooting
(a semi-systematic approach)

Volker Rieckert
HP Outsourcing Services Europe
Enterprise Application Operations

April 23, 2024

 Before You Start

Don’t logon to the system before you’ve collected

information about

• when the problem started and whether it still exists?

(If exists for > 1 week  EarlyWatch and/or resizing)

• what has been changed just before problem started (e.g. imports,
new users, parameter changes, new release, patches)

• which application server(s) exhibit bad performance (single server

vs. whole system)

• which user(s) experience bad performance (single user vs. all users)

• which transactions perform badly (single transaction or batch job

vs. general performance problem)

04/23/24 Volker Rieckert / HP MS EMEA – SAP Operations / Consulting & Architecture Design Europe page 2
 Unsystematic Part (“Quick Guesses”):
Check CPU and Paging on DB server

On the DB server (OS-level):

• Run “top” to see if there are non-SAP/non-DB processes among the top 10
CPU consumers

• Start “glance” and leave it running during the whole time of your analysis (to
get accumulated statistics):

100% Disk Utilization does not necessarily indicate a problem.

Neither does 100% Memory Utilization

• Glance  Memory Report  KB Paged Out:

Cumulative Rate should be < 20 KB/sec (for DB server only)
(excessive paging  more RAM)

04/23/24 Volker Rieckert / HP MS EMEA – SAP Operations / Consulting & Architecture Design Europe page 3
 Unsystematic Part Cont.:
Reset statistics

Logon to the SAP application server with the

performance problem (SAPGUI):

• First reset some Oracle statistics (ST04) in order to get

values relevant for the current situation:
- Reset ST04-Overview
- Reset filesystem statistics (ST04  Detail analysis menu
 File system requests  I/O per path)
- Reset wait events (ST04  Detail analysis menu 
Display V$ values  v$system_event)

• What is your impression from the last few dialog steps:

Is there a general performance problem or did you observe
rather satisfactory response time for your own transactions?

04/23/24 Volker Rieckert / HP MS EMEA – SAP Operations / Consulting & Architecture Design Europe page 4
 Unsystematic Part Cont.:
Check Disk Performance

ST04  Detail analysis menu  File system requests  I/O per path:

• Compare “Since DB start” to “Since reset” values

(note: Oracle timed statistics is buggy!! Especially “Since DB start”
values may be nonsense.):

- Avg. Blk Reads:< 7 ms (very fast: < 3ms)

- Avg. Blk Write: < 2 ms

• If file system statistics is not available, then try v$system_event:

“db file sequential read” < 7 ms Avg. Wait

(very fast: < 3 ms)

04/23/24 Volker Rieckert / HP MS EMEA – SAP Operations / Consulting & Architecture Design Europe page 5
 Unsystematic Part Cont.:
Check some more DB statistics

• Check “Wait events” (or v$system_event):

- ignore idle events (SQL*Net message from client; rdbms ipc
message; pmon timer; smon timer)
- remaining wait time should be completely dominated (>90%) by
event “db file sequential read”. Anything else indicates a bottleneck in
the resource associated with the particular event.
Report “/SDF/RSORAVSE” or “RSORAVSE” (start via SE38)
displays a history of Oracle Wait events (extremely useful!!!)
• Desirable values for ST04-Overview Page:
- User/Recursive calls: >7 (Shared Pool Size)
- Reads/User call < 20 (bad SQL/missing index)
- Redo Logging: > 1 MB/sec (JBOD)
> 2 MB/sec (Disk Arrays)
- Cache Hit Ratio: > 92% (4.x with CBO)
> 98% (3.x with RBO)

04/23/24 Volker Rieckert / HP MS EMEA – SAP Operations / Consulting & Architecture Design Europe page 6
 Unsystematic Part Cont.:
What’s currently going on?

SM66: Systemwide Work Process Overview:

• Are there many Dialog WPs in PRIV mode? ( extended

memory shortage)
• Are there many long running dialog tasks?
• Put down users, reports and wait reasons of long running
dialog tasks
• Are they all accessing the same table?

Use the DB Alert Monitor (AL02) to find out which task is

currently consuming most DB resources. (Concentrate on
“Physical reads” and possibly “DB block changes”.

04/23/24 Volker Rieckert / HP MS EMEA – SAP Operations / Consulting & Architecture Design Europe page 7
 Systematic Part:
How does the system spend its time?

ST03/ST03N: Compare current response time values with history data (

Expert Mode  Load History  Total  By Month) in order to identify
the response time component mainly responsible for the increase)

• Response time: Starts when user request enters the dispatcher queue. Ends
when the following screen is sent back to the user. Does not include network
time between application server and front end.

• Wait time: Starts when user request enters the dispatcher queue. Ends when
request starts to be processed by work process.

• Load time: Time needed to load and generate objects like ABAP source
code, CUA and dynpro information from database

04/23/24 Volker Rieckert / HP MS EMEA – SAP Operations / Consulting & Architecture Design Europe page 8
 Systematic Part:
Components of Average Response Time

• Roll-in/out time: Time needed to roll in/out user context

information from/to roll area (shared memory or roll file) to/from
the work process.

• Roll wait time (see also speaker’s notes): Time within a dialog
step during which the user context is rolled out, waiting for
response (from some RFC-client or from the front end). During
this “waiting time in the roll area”, the work process can handle
other requests. As of 4.6 the major part of communiction between
application server and front end (typ. 100 – 200 ms per dialog
step) is thus reflected in the roll wait time.

04/23/24 Volker Rieckert / HP MS EMEA – SAP Operations / Consulting & Architecture Design Europe page 9
 Systematic Part:
Components of Average Response Time

• DB request time: Starts when a DB requests is put through to the

DB interface (part of disp+work!). Ends when DB interface
delivers the result. If the requested data cannot be found in the
R/3 buffers (“Generic key”, “Single record”) on the application
server, the DB request results in a DB call to the database. In this
latter case, the DB request time contains some network time as
well as the response time of the underlying database system.

• Enqueue time: Time needed to request enqueues

04/23/24 Volker Rieckert / HP MS EMEA – SAP Operations / Consulting & Architecture Design Europe page 10
 Av. response time = Av. wait time
+ Av. processing time (not listed)
+ Av. load time
+ Av. Roll in+wait time
+ Av. DB request time
+ Av. enqueue time
(but not Av. CPU time!!!)

Av. CPU time: av. CPU time used by SAP work process, i.e. some
fraction of “av. response time – av. wait time”

04/23/24 Volker Rieckert / HP MS EMEA – SAP Operations / Consulting & Architecture Design Europe page 11
 Systematic Part:
“Normal Dialog Response Times”

Av. response time < 1 sec

Av. wait time < 1% Av. response time
Av. load time < 10% Av. response time
Av. roll in < 3 ms
Av. roll wait time should be subtracted from av. dialog
response time to obtain values that
characterize SAP server performance
only (as up to 4.5);
values >200 ms might be an indica-
tion for slow User-LAN or slow clients
Av. DB req. time < 70% Av. response time
Av. CPU time < 40% Av. response time, but
> 40% of “Av. Response time
minus DB request time”

04/23/24 Volker Rieckert / HP MS EMEA – SAP Operations / Consulting & Architecture Design Europe page 12
 Systematic Part:
“Normal Time per DB request”

Time per DB request < 4 ms (very fast: 2 ms)

Direct reads < 1 ms (very fast: 0.3 ms)
Sequential reads < 12 ms (very fast: 3 ms)
Changes + commits large variation

Av. Bytes requested 30 – 100 KB

If “Time per DB request” is fast, but “Av. DB request time” is

high, then the application is simply issueing many DB requests
per dialog step ( application problem).

04/23/24 Volker Rieckert / HP MS EMEA – SAP Operations / Consulting & Architecture Design Europe page 13
 Systematic Part:
“How to proceed”

• Performance problem is confined to specific transactions

 Done (application problem)

• If general performance problem and “Av. DB request time” is too

high:

If “Time per DB request” is o.k.  application problem, if not:

- make sure DB performance is only limited by disk performance (

Wait events) and that disk performance is fast!!
- check for exclusiv lock waits (DB01) and discuss with customer
whether lock holding task may be aborted
- check R/3 buffers and DB Cache (Hit ratios)
- check missing indices (DB02)
- check if valid optimizer statistics exists (e.g. DB14)

04/23/24 Volker Rieckert / HP MS EMEA – SAP Operations / Consulting & Architecture Design Europe page 14
 Systematic Part:
“How to proceed”

• If Av. Wait time is high:

- only increase # of work processes, if all other bottlenecks have

been removed. High wait time is often only a secondary
symptom of a primary bottleneck that leads to long running
transactions

• If Av. CPU time > 50% Av. Response time:

 application problem (inefficient ABAP)

• If Av. Load time is high: check ABAP program buffer

04/23/24 Volker Rieckert / HP MS EMEA – SAP Operations / Consulting & Architecture Design Europe page 15
 Systematic Part:
“How to proceed”

• If Av. CPU time << “Av. Resp. time – Av. DB

req. time”:

- check for CPU or memory bottleneck

- for batch tasks this may not be a real problem,
since batch WPs may be idle while doing RFC to
other application servers (e.g. for distributed
processing of MRP run).

04/23/24 Volker Rieckert / HP MS EMEA – SAP Operations / Consulting & Architecture Design Europe page 16
 Additional Information:
Understanding R/3 Memory Mgmt.

A dialog task allocates memory in the following order:

• Roll Memory for User Context (ztta/roll_first)

• Extended Memory (ztta/roll_extension), if extended memory is
not already exhausted (em/initial_size_MB)
• Roll Memory (ztta/roll_area – already allocated)
• Heap (abap/heap_area_dia): this is process local memory (i.e. not
shared memory)  the WP starts to grow in size and goes into
PRIV mode. If the size of the WP exceeds “abap/heaplimit”, it
will be restarted at the end of the dialog transaction in order to
free the heap memory.

On Windows and LINUX, the same memory allocation order is also

used for all other tasks (batch, update, spool, ….)
04/23/24 Volker Rieckert / HP MS EMEA – SAP Operations / Consulting & Architecture Design Europe page 17
 Additional Information:
Understanding R/3 Memory Mgmt.

A batch task on HP-UX allocates memory in the following order:

• Roll Memory (ztta/roll_area):

• Heap (abap/heap_area_nondia): this is process local memory (i.e.
not shared memory)  the WP starts to grow in size (batch
processes are always “PRIV”). If the size of the WP exceeds
“abap/heaplimit”, it will be restarted at the end of the transaction
(generally consisting of several dialog steps) in order to free the
heap memory.
• Extended Memory (ztta/roll_extension), if extended memory is
not already exhausted (em/initial_size_MB)

04/23/24 Volker Rieckert / HP MS EMEA – SAP Operations / Consulting & Architecture Design Europe page 18
 Additional Information:
Understanding R/3 Memory Mgmt.

• Abap/heap_area_total limits the amount of heap memory that all

WPs together can allocate. This is to prevent them from
exhausting swap space and/or consuming too much memory
resources.

• Dialog or batch tasks will abort, if they cannot allocate any more
memory of any kind.

• For 32-bit R/3 kernels, it may make sense to decrease

ztta/roll_extension to as little as 20MB, in order to force memory
hungry dialog programs into PRIV mode before they use up all
available extended memory.

04/23/24 Volker Rieckert / HP MS EMEA – SAP Operations / Consulting & Architecture Design Europe page 19
 Additional Information:
Understanding R/3 Memory Mgmt.

• On HP-UX, if enough swap space is available,

em/initial_size_MB should be set to rather large values even if
much less RAM is available. This prevents dialog WPs from
going into PRIV mode, while OS-paging remains the same,
independent of whether large amounts of heap (process local) or
extended (shared) memory are being allocated.

This is especially true if the “disclaiming” features of the SAP

extended memory management are being used (see OSS-notes
724140 and 735566), which should be the case for all SAP
systems running on HP-UX 11.23 or higher with SAP kernels
4.6D or higher.

04/23/24 Volker Rieckert / HP MS EMEA – SAP Operations / Consulting & Architecture Design Europe page 20
 Additional Information:
First steps towards Perf. Analysis

• ST04  Detail analysis menu  SQL requests

- sort by “disk reads”

- look for “Bufgets/record” >> 1 (missing or bad index)
- look for “Records/execution” >> 1 (poor selectivity of SQL-
request  tune ABAP)

Starting with 4.6B, you can directly jump to the location in the
ABAP program from where the SQL-statement was called for the
first time (since DB start or reset)!

• ST05: SQL-trace for tracing specific user on specific application

server (find expensive SQL in transactions with high DB request
time)

04/23/24 Volker Rieckert / HP MS EMEA – SAP Operations / Consulting & Architecture Design Europe page 21