Analysis on Linux and Solaris - Application Server Infrastructure - SCN... https://wiki.scn.sap.

com/wiki/display/SI/Analysis+on+Linux+and+Solaris

Getting Started Store

Community WIKI SAP Community Welcome, Guest Login Register Search the Community

Application Server Infrastructure / … / Analyzing the heap memory usage of an ABAP work process

Analysis on Linux and Solaris

Created by Isaias Freitas, last modified on Apr 03, 2018

As mentioned in the main page of this WIKI, using the "ps" command to analyze heap memory usage is not helpful because "ps -elf" can (and most
likely will) include the shared memory areas used by the work process. This can happen in both Linux and Solaris.
The appropriate tool for heap memory analysis would be "pmap".
You can execute "pmap <PID of SAP work process>" and look for "heap" and "anon" blocks. Depending on the OS, pmap will also provide a
summary at the end of its output.
Using a test system (running on Linux) to simulate a heap memory issue, the following outputs were collected before the issue was triggered:

servername:sidadm 60> ps –elf

F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
(...)
1 S sidadm 16912 16853 0 80 0 - 1748628 - 19:00 ? 00:00:42 dw.sapSID_DVEBMGS20 pf=/usr
/sap/SID/SYS/profile/SID_DVEBMGS20_servername
(...)
servername:sidadm 61> pmap 16912 | egrep '(START|writable|anon|heap)'
START SIZE RSS PSS DIRTY SWAP PERM MAPPING
0000000004109000 72892K 7012K 4798K 7012K 0K rw-p [heap]
0000000008838000 57636K 25028K 25028K 25028K 0K rw-p [heap]
00007f1912623000 4K 0K 0K 0K 0K ---p [anon]
00007f1912624000 8192K 8K 8K 8K 0K rw-p [anon]
00007f194e791000 1204K 108K 108K 108K 0K rw-p [anon]
00007f195527e000 3412K 792K 792K 792K 0K rw-p [anon]
00007f195b03b000 1960K 1828K 1828K 1828K 0K rw-p [anon]
00007f196b184000 184K 0K 0K 0K 0K rw-p [anon]
00007f196bb7b000 652K 48K 48K 48K 0K rw-p [anon]
00007f196bffb000 316K 48K 48K 48K 0K rw-p [anon]
00007f196c6bd000 108K 28K 28K 28K 0K rw-p [anon]
00007f197735e000 380K 256K 15K 256K 0K rw-p [anon]
00007f1a90450000 1232K 952K 944K 952K 0K rw-p [anon]
00007f1aa1593000 8K 4K 0K 4K 0K rw-p [anon]
00007f1aa9c08000 4K 0K 0K 0K 0K rw-p [anon]
00007f1aab596000 12K 8K 8K 8K 0K rw-p [anon]
00007f1aab910000 20K 20K 20K 20K 0K rw-p [anon]
00007f1aabb2e000 16K 4K 4K 4K 0K rw-p [anon]
00007f1aac2b2000 84K 16K 0K 16K 0K rw-p [anon]
00007f1aac79f000 12K 12K 0K 12K 0K rw-p [anon]
00007f1aadb23000 40K 40K 2K 40K 0K rw-p [anon]
00007f1aaf631000 12K 0K 0K 0K 0K rw-p [anon]
00007f1aafd83000 260K 4K 4K 4K 0K rw-p [anon]
00007f1ab099d000 8K 8K 0K 8K 0K rw-p [anon]
00007f1ab1911000 28K 28K 16K 28K 0K rw-p [anon]
00007f1ab1d74000 176K 4K 4K 4K 0K rw-p [anon]
00007f1ab1dcc000 772K 768K 48K 768K 0K rw-p [anon]
00007f1ab1e8d000 256K 256K 15K 256K 0K rw-p [anon]
00007f1ab1f0c000 48K 48K 29K 48K 0K rw-p [anon]
00007f1ab1f24000 4K 0K 0K 0K 0K rw-p [anon]
00007f1ab1f40000 4K 4K 4K 4K 0K rw-p [anon]
00007f1ab1f42000 8K 8K 4K 8K 0K rw-p [anon]
00007f1ab1f46000 4K 4K 4K 4K 0K rw-p [anon]
155272K writable-private, 225480K readonly-private, 6613760K shared, and 428468K referenced
servername:sidadm 62>

If "egrep" is available at the OS in use, you can use it (as it is shown above) in order to have a filtered output from pmap, showing only the relevant
lines (header line, all "heap" and "anon" blocks, and the summary line from the end).
The summary line (highlighted in bold) shows that this work process (the work process itself) is using around 380MB of memory: around 155MB of
heap (private) writable memory plus around 225MB of heap readonly memory. The latter would be the executable itself (the "disp+work" binary file)

1 of 4 01/03/2019, 20:53
Analysis on Linux and Solaris - Application Server Infrastructure - SCN... https://wiki.scn.sap.com/wiki/display/SI/Analysis+on+Linux+and+Solaris

that is loaded into the memory so it can be executed.

But the "ps" output showed that this same process is using 1748628 blocks of memory. In the OS used to collect these outputs, the size of each
block is 4KB. Therefore, "ps" would be showing that this process would be using around 6.9GB of memory.
If we go back to the "pmap" summary line, we will notice that this process is "using" around 6.6GB of shared memory. The "catch" is that each work
process is using the same 6.6GB shared memory area (thus the name "shared memory" :-)).
These are the outputs from the same commands above, but executed after the heap memory issue was triggered.

servername:sidadm 70> ps –elf

F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
(...)
1 S sidadm 16912 16853 0 80 0 - 1787029 semtim 19:00 ? 00:00:42 dw.sapSID_DVEBMGS20 pf=/usr
/sap/SID/SYS/profile/SID_DVEBMGS20_servername
(...)
servername:sidadm 71> pmap 16912 | egrep '(START|writable|anon|heap)'
START SIZE RSS PSS DIRTY SWAP PERM MAPPING
0000000004109000 72892K 7012K 4798K 7012K 0K rw-p [heap]
0000000008838000 57636K 25028K 25028K 25028K 0K rw-p [heap]
00007f1909022000 153604K 8K 8K 8K 0K rw-p [anon]
00007f1912623000 4K 0K 0K 0K 0K ---p [anon]
00007f1912624000 8192K 8K 8K 8K 0K rw-p [anon]
00007f194e791000 1204K 108K 108K 108K 0K rw-p [anon]
00007f195527e000 3412K 792K 792K 792K 0K rw-p [anon]
00007f195b03b000 1960K 1828K 1828K 1828K 0K rw-p [anon]
00007f196b184000 184K 0K 0K 0K 0K rw-p [anon]
00007f196bb7b000 652K 48K 48K 48K 0K rw-p [anon]
00007f196bffb000 316K 48K 48K 48K 0K rw-p [anon]
00007f196c6bd000 108K 28K 28K 28K 0K rw-p [anon]
00007f197735e000 380K 256K 15K 256K 0K rw-p [anon]
00007f1a90450000 1232K 952K 944K 952K 0K rw-p [anon]
00007f1aa1593000 8K 4K 0K 4K 0K rw-p [anon]
00007f1aa9c08000 4K 0K 0K 0K 0K rw-p [anon]
00007f1aab596000 12K 8K 8K 8K 0K rw-p [anon]
00007f1aab910000 20K 20K 20K 20K 0K rw-p [anon]
00007f1aabb2e000 16K 4K 4K 4K 0K rw-p [anon]
00007f1aac2b2000 84K 16K 0K 16K 0K rw-p [anon]
00007f1aac79f000 12K 12K 0K 12K 0K rw-p [anon]
00007f1aadb23000 40K 40K 2K 40K 0K rw-p [anon]
00007f1aaf631000 12K 0K 0K 0K 0K rw-p [anon]
00007f1aafd83000 260K 4K 4K 4K 0K rw-p [anon]
00007f1ab099d000 8K 8K 0K 8K 0K rw-p [anon]
00007f1ab1911000 28K 28K 16K 28K 0K rw-p [anon]
00007f1ab1d74000 176K 4K 4K 4K 0K rw-p [anon]
00007f1ab1dcc000 772K 768K 48K 768K 0K rw-p [anon]
00007f1ab1e8d000 256K 256K 15K 256K 0K rw-p [anon]
00007f1ab1f0c000 48K 48K 29K 48K 0K rw-p [anon]
00007f1ab1f24000 4K 0K 0K 0K 0K rw-p [anon]
00007f1ab1f40000 4K 4K 4K 4K 0K rw-p [anon]
00007f1ab1f42000 8K 8K 4K 8K 0K rw-p [anon]
00007f1ab1f46000 4K 4K 4K 4K 0K rw-p [anon]
308876K writable-private, 225480K readonly-private, 6613760K shared, and 428600K referenced
servername:sidadm 72>

Notice that the "ps" output does not change significantly (because it shows usage of 4KB blocks), but it still shows much more memory than the work
process itself is using (it is still including the shared memory areas).
At the "pmap" output, we can see a new "anon" segment of about 150MB. We also see that the writable heap memory area has increased (at the
summary line).
The corresponding work process was idle during the time all the commands above were collected.

In order to determine whether the incorrect memory usage was triggered within the SAP kernel, we can use the transaction SM50, menu path
Administration -> Work process -> Write stack (prerequisite: the SAP kernel in use must be 742 patch level 315 / 745 patch level 35, or higher).
Then, the corresponding trace file (dev_w8, in this case) will have the following information:

2 of 4 01/03/2019, 20:53
Analysis on Linux and Solaris - Application Server Infrastructure - SCN... https://wiki.scn.sap.com/wiki/display/SI/Analysis+on+Linux+and+Solaris

M Mon Jun 27 21:33:04 2016

M
M ********** STACKDUMP 1 - begin **********
M
M C-STACK
dw.sapSID_DVEBMGS20[S](LinStackBacktrace+0x8c)[0x65656d]
dw.sapSID_DVEBMGS20[S](LinStack+0x35)[0x1aafd95]
dw.sapSID_DVEBMGS20[S](CTrcStack2+0x4e)[0x656401]
dw.sapSID_DVEBMGS20[S](ThStackHandler()+0x102)[0x4f5923]
dw.sapSID_DVEBMGS20[S](DpTrcOnOff(int)+0x212)[0x45ac89]
<signal handler called> [0x7f1aab924850]
libc.so.6[S](__GI_semtimedop+0xa)[0x7f1aab678faa]
dw.sapSID_DVEBMGS20[S](SemTimedOp+0x47)[0x1780bd7]
dw.sapSID_DVEBMGS20[S](EvtWtRst+0x72)[0x1a2ab82]
dw.sapSID_DVEBMGS20[S](RqQWorkerWaitForRequests(unsigned int, unsigned char,
Q_Q_PRIO, unsigned int*, int)+0x1fa)[0x19e7c1a]
dw.sapSID_DVEBMGS20[S](ThRqCheckQueues(int, REQUEST_BUF**)+0x37c)[0x1a39d5c]
dw.sapSID_DVEBMGS20[S](ThRqGetNextRequest(int, REQUEST_BUF**)+0x2a)[0x1a3d5fa]
dw.sapSID_DVEBMGS20[S](ThRqWaitFor(int, REQUEST_BUF**)+0x37)[0x1a3d787]
dw.sapSID_DVEBMGS20[S](ThRqAcceptImpl(unsigned char, int,
EQUEST_BUF**)+0xf2)[0x1a3ad32]
dw.sapSID_DVEBMGS20[S](ThRqAcceptNextRequest(int, REQUEST_BUF**)+0x16)[0x1a3c1f6]
dw.sapSID_DVEBMGS20[S](ThReceive()+0x28)[0x1a3c238]
dw.sapSID_DVEBMGS20[S](TskhLoop()+0x5e)[0x1a5114e]
dw.sapSID_DVEBMGS20[S](ThStart()+0x26c)[0x4f5471]
dw.sapSID_DVEBMGS20[S](DpMain+0x389)[0x19e09e9]
libc.so.6[S](__libc_start_main+0xe6)[0x7f1aab5b7c36]
M
M Top 50 PROC memory consumers by user tag ID (in bytes)
M
M Tag Allocations Smallest Biggest Sum
M 'FCLASS' 1 157.286.424 157.286.424 157.286.424
M 'DBI_MM' 12 102.424 4.194.568 17.948.112
M '<free>' 36 56 2.096.928 9.966.120
M 'DBSL ' 1464 144 3.200.656 9.882.184
M 'DYCBOX' 1 5.540.128 5.540.128 5.540.128
M 'CUA___' 3 500.128 500.128 1.500.384
M 'PCACHE' 1 1.376.384 1.376.384 1.376.384
M 'AbFdaM' 1 1.048.704 1.048.704 1.048.704
M 'DYTEMP' 1 1.000.128 1.000.128 1.000.128
M 'DIAGAR' 2 250.128 250.128 500.256
M 'D010TI' 7 1.344 165.168 389.984
M '*RTM_M' 3 16.128 160.128 216.384
M 'DBSLDA' 38 1.920 16.256 167.936
M 'NTALXX' 4 20.128 60.128 160.512
M 'AUTHSO' 1 148.128 148.128 148.128
M 'SHORT_' 1 102.424 102.424 102.424
M 'NTPGTF' 1 77.072 77.072 77.072
M 'NTAB ' 1 65.664 65.664 65.664
M 'BLIN ' 1 65.648 65.648 65.648
M 'BLOUT ' 1 65.648 65.648 65.648
M 'DBDYNP' 2 32.128 32.896 65.024
M 'AbShCo' 1 42.528 42.528 42.528
M 'RI LBL' 1 32.896 32.896 32.896
M 'BUFFER' 1 32.896 32.896 32.896
M 'RELOBB' 1 32.128 32.128 32.128
M 'RTFDSC' 2 512 28.800 29.312
M '*PXABM' 1 28.800 28.800 28.800
M 'DBSLTL' 2 12.128 12.128 24.256
M 'STAAPC' 4 2.096 6.848 21.968
M 'RSPOD1' 1 20.608 20.608 20.608
M 'STATSM' 4 3.392 6.896 20.384
M 'DIAGUP' 1 20.128 20.128 20.128
M 'STATWE' 2 352 18.480 18.832
M 'STAAMC' 4 3.680 5.120 16.928
M 'APQILS' 2 816 15.984 16.800
M 'PCHASH' 1 16.512 16.512 16.512
M
M Mon Jun 27 21:33:04 2016
M 'RSCRSR' 13 256 5.504 14.784
M 'AbRegx' 3 1.152 8.320 11.648
M '*PXAML' 1 10.528 10.528 10.528
M '*PXATL' 1 10.528 10.528 10.528
M 'STATTR' 2 512 9.872 10.384
M 'TRANFL' 1 10.128 10.128 10.128
M 'STAHC2' 1 9.632 9.632 9.632

3 of 4 01/03/2019, 20:53
Analysis on Linux and Solaris - Application Server Infrastructure - SCN... https://wiki.scn.sap.com/wiki/display/SI/Analysis+on+Linux+and+Solaris

M 'STATES' 3 704 6.176 9.264

M 'STAHS2' 1 9.104 9.104 9.104
M 'STAHC1' 1 9.056 9.056 9.056
M 'abtsor' 2 824 8.024 8.848
M 'DIAGOK' 1 8.320 8.320 8.320
M 'STAHS1' 1 8.096 8.096 8.096
M 'STARS2' 1 7.952 7.952 7.952
M
M ********** STACKDUMP 1 - end **********
M

We can see that the "FCLASS" tag is allocating exactly the 150MB of memory that should not be there.
Thus, we should search for SAP notes that would address a memory issue ("memory leak") related to the "FCLASS" tag.
Notice that:
"FCLASS" was a tag name used for the example above. It may not exist;
This was a straightforward example, when one tag made only one allocation that matched the issue exactly. It is possible that the memory leak
would occur over several small allocations instead of a single large allocation.
If the information at the trace file does not indicate that any of the tags is causing the issue, then you should continue with the analysis from different
areas, like the OS libraries and the database client / libraries.
<back to main page>
No labels

Contact Us SAP Help Portal

Privacy Terms of Use Legal Disclosure Copyright Cookie Preferences Follow SCN

4 of 4 01/03/2019, 20:53