®

IBM Software Group

How to Analyze verbosegc trace with

IBM Pattern Modeling and Analysis Tool
for IBM Java Garbage Collector
IBM Pattern Modeling and Analysis Tool Architect / Developer
Jinwoo Hwang ([email protected]) IBM WebSphere Application Server Support

WebSphere® Support Technical Exchange

 IBM Software Group

Agenda
ƒ What is PMAT?
ƒ Features
ƒ How does it work?
ƒ Verbose garbage collection trace
ƒ Prerequisite
ƒ Definitions of headers in tables/charts
ƒ Views
ƒ How to run PMAT and analyze data
ƒ Case Studies ( I, II and III )
ƒQ&A

WebSphere® Support Technical Exchange 2

 IBM Software Group

What is PMAT?
ƒ Pattern Modeling and Analysis Tool for IBM Java Garbage
Collector (PMAT) parses IBM SDK verbose Garbage Collect
(GC) logs and provides statistics, charts, analysis, and
recommendations for key Java™ heap configurations.
ƒ PMAT parses the IBM verbose GC trace and provides a
comprehensive analysis of the Java heap usage. It then
recommends key configurations by first executing a diagnosis
engine and then employing a pattern modeling algorithm in
order to make recommendations to optimize the Java heap
usage for a given JVM cycle.
ƒ If there are any errors related with Java heap exhaustion or
fragmentation in the verbose GC trace, PMAT can diagnose
the root cause of failures. PMAT provides rich chart features
that graphically display Java heap usage.

WebSphere® Support Technical Exchange 3

 IBM Software Group

Features
ƒ Diagnoses Java heap potential problems by
analyzing verbose GC traces.

ƒ Recommends optimal heap configurations by

employing a pattern modeling engine to track and
evaluate historical trends.

ƒ Includes a user-friendly graphical representation of

Java heap usage.

WebSphere® Support Technical Exchange 4

 IBM Software Group

How does it work? (1/2)

ƒ PMAT parses IBM SDK verbose garbage collector trace and
retrieves Java heap usage, garbage collector time, mark
phase time, sweep phase time, compact phase time, available
Java heap size, freed Java heap size, and total Java heap size.

ƒ It then calculates overhead ratio of each garbage collector

cycle, runs a garbage collector diagnostics engine to report any
problems recorded in the trace, and provides a root cause
analysis of the problems (Java heap fragmentation, Java heap
exhaustion, or too large a Java heap request).

ƒ The garbage collector diagnostics engine analyzes patterns of

historic data and provides diagnostic information based on
patterns stored in a knowledge database.

WebSphere® Support Technical Exchange 5

 IBM Software Group

How does it work?(2/2)

ƒ It also runs a pattern modeling engine based on historic Java
heap usage and provides optimal Java heap configuration
recommendations. For example, it can recommend optimal
Java maximum heap size and allocation size for class blocks in
kCluster.

ƒ The pattern modeling engine uses historical activity data of the

Java heap usage to predict future performance trends, thus
avoiding potential memory leaks or memory consumption
instability.

ƒ For advanced users, graphical representations of the Java

heap usage history and detailed elapsed time for each phase
of the garbage collector cycles is generated.

WebSphere® Support Technical Exchange 6

 IBM Software Group

What is verbosegc?
ƒ Verbose GC is a command-line option that you supply to the
JVM at startup time. The format is: -verbose:gc or –
verbosegc. This option switches on a substantial trace of
every garbage collection cycle. The format for the generated
information is not architected and therefore varies from
platform to platform and release to release.
ƒ This trace should allow you to see the gross heap usage in
every garbage collection cycle. For example, you could
monitor the output to see the changes in the free heap space
and the total heap space. You can use this information to
determine:
Whether garbage collections are taking too long to run
Whether too many garbage collections are occurring
Whether the JVM crashed during garbage collection

WebSphere® Support Technical Exchange 7

 IBM Software Group

Sample verbosegc trace (Sov. JVM )

<AF[1]: Allocation Failure. need 1544 bytes, 0 ms since last AF>
<AF[1]: managing allocation failure, action=1 (0/3983128)
(209640/209640)>
<GC(1): GC cycle started Tue Apr 24 10:49:58 2001>
<GC(1): freed 1421672 bytes, 38% free (1631312/4192768), in 9 ms>
<GC(1): mark: 8 ms, sweep: 1 ms, compact: 0 ms>
<GC(1): refs: soft 0 (age >= 32), weak 17, final 16, phantom 0>
<AF[1]: completed in 10 ms>
ƒ Notes:
1. GC(1): The 1st garbage collection cycle in this JVM.
2. freed 1,421,672 bytes: An indication of the amount of activity since the
last garbage collection cycle.

WebSphere® Support Technical Exchange 8

 IBM Software Group

Sample verbosegc trace ( J9 JVM )

<gc type="global" id="5" totalid="5" intervalms="18.880">
<compaction movecount="9282" movebytes="508064"
reason="forced compaction" />
<expansion type="tenured" amount="1048576" newsize="3145728"
timetaken="0.011" reason="insufficient free space following gc" />
<refs_cleared soft="0" weak="0" phantom="0" /> <finalization
objectsqueued="0" />
<timesms mark="7.544" sweep="0.088" compact="9.992"
total="17.737" />
<tenured freebytes="1567256" totalbytes="3145728" percent="49" >
<soa freebytes="1441816" totalbytes="3020288" percent="47" />
<loa freebytes="125440" totalbytes="125440" percent="100" />
</tenured>
</gc>
ƒ Notes:
1. Type=″global″ indicates that this was a global collection (mark, sweep, possibly compact).

WebSphere® Support Technical Exchange 9

 IBM Software Group

Prerequisite
ƒ Java 2 Runtime Environment 1.4.1 or higher
ƒ Log files with verbose:gc enabled on IBM SDK
1.3.x and 1.4.x. (Log files taken from Non-IBM
SDKs are not supported yet)

WebSphere® Support Technical Exchange 10

 IBM Software Group

Definitions in tables/charts (1/2)

ƒ Since Time(millisecond) elapsed since last allocation failure.
ƒ Freed Size(byte) of space that was freed during garbage
collection.
ƒ Needed/Requested Size(byte) of space that was requested
during allocation failure.
ƒ Free Size(byte) of space that was free after garbage
collection.
ƒ Total Size(byte) of Java heap after garbage collection.
ƒ Completed Time(millisecond) spent during allocation failure.

WebSphere® Support Technical Exchange 11

 IBM Software Group

Definitions in tables/charts (2/2)

ƒ GC Completed or GC Time(millisecond) spent during
garbage collection.
ƒ Overhead Time(%) spent in allocation failure
ƒ Mark Time(millisecond) spent in mark phase
ƒ Sweep Time(millisecond) spent in sweep phase.
ƒ Compact Time(millisecond) spent in compact phase.
ƒ Exhausted Whether there was insufficient space to satisfy
allocation failure.

WebSphere® Support Technical Exchange 12

 IBM Software Group

Views
ƒ GC analysis
ƒ GC Table View
ƒ AF summary
ƒ GC usage summary
ƒ GC duration summary
ƒ GC Graph View
ƒ GC trend analysis
ƒ Zoom in/out/selection/center of graph view

WebSphere® Support Technical Exchange 13

 IBM Software Group

How to run PMAT

ƒ You need to use the Java 2 Platform, Standard
Edition version 1.4.1 or higher Java Runtime
Environment (JRE) to run IBM PMAT.
ƒ Usage <Java path>java –Xmx[heap size] –jar
ga<PMAT version>.jar
ƒ For example, java –Xmx1000m –jar ga10.jar
ƒ If there’s java.lang.OutOfMemoryError while you
are processing verbosegc log, please try
increasing the maximum heap size (-Xmx) value to
give the JVM more memory.
ƒ Maximum heap size should not be larger than the
size of available physical memory size for this tool
due to performance issue.

WebSphere® Support Technical Exchange 14

 IBM Software Group

Initial screen
ƒ IBM PMAT is
displayed with
console
window.

WebSphere® Support Technical Exchange 15

 IBM Software Group

Open verbosegc log

ƒ Open
verbosegc log
by clicking on
File->Open

WebSphere® Support Technical Exchange 16

 IBM Software Group

Select verbosegc log

WebSphere® Support Technical Exchange 17

 IBM Software Group

Processing verbosegc log

ƒ Progress bars are displayed while processing verbosegc log

WebSphere® Support Technical Exchange 18

 IBM Software Group

Analysis view (1/6)

ƒ Statistics of
verbosegc data
is displayed as
well as
analysis of
each errors.

WebSphere® Support Technical Exchange 19

 IBM Software Group

Analysis view (2/6)

ƒ File name : Location and file name of verbosegc trace
ƒ Number of verboseGC cycles : Number of JVM restart
ƒ Number of Garbage Collections : GC frequency
ƒ Number of Allocation failures : AF frequency
ƒ First Garbage Collection : Timestamp of the first GC
ƒ Last Garbage Collection : Timestamp of the last GC
ƒ Number of Java heap exhaustion : Number of OutOfMemoryError
ƒ Maximum AF overhead : Ratio of time spent in AF and time
between AFs
ƒ Number of 100% overhead : Number of AF overhead 100%
ƒ Maximum size of Large Object Request : The largest object
request and timestamp
ƒ Number of Large Object Requests : Number of object request
(>900KB)
ƒ List of Java heap failure : Timestamp, Requested Java heap size,
Type of failure and available Java heap size.

WebSphere® Support Technical Exchange 20

 IBM Software Group

Analysis view (3/6)

ƒ Analysis and
Recommendat
ions report
provides
analysis and
recommendatio
ns for each
JVM runtime

WebSphere® Support Technical Exchange 21

 IBM Software Group

Analysis view (4/6)

Analysis and Recommendations report

ƒ Garbage collection start / finish : Timestamps of
JVM start and stop
ƒ Analysis :
Type of Java heap issue is displayed if there’s any
problem with Java heap.
No Java heap exhaustion found is displayed if there’s
no Java heap exhaustion.

WebSphere® Support Technical Exchange 22

 IBM Software Group

Analysis view (5/6)

Analysis and Recommendations report
ƒ Recommendations :
If there’s no Java heap issue, optimal maximum heap size is provided by
usage analysis.

If there’s any possibility that Java heap usage might increase, IBM PMAT
provides a ratio(%) based on Java Heap Occupancy Model.

Ratio of 100% means Java heap occupancy could go up to L x 2 bytes at

the time of T0 + (T1 – T0) x 2 where L (bytes) is Java heap occupancy at
the time of T1 (T0:JVM start time T1:JVM stop time)

Ratio of 0% means Java heap occupancy could stay L bytes at the time
of T0 + (T1 – T0) x 2 where L (bytes) is Java heap occupancy at the time
of T1 (T0:JVM start time T1:JVM stop time)

WebSphere® Support Technical Exchange 23

 IBM Software Group

Analysis view (6/6)

Analysis and Recommendations report
ƒ Recommendations :
If the ratio is close to 0, Java heap usage is stable based on model in the
specific JVM runtime.
If the ratio is non-zero, Java heap usage could increase over time.

The percentage error is calculated by comparing Java Heap Occupancy

Model with current data.
ƒ Note
If there’s not enough activities in a JVM runtime cycle, the model might
not fully represent actual JVM.

The percentage error does not represent accuracy of the ratio.

If percentage error is large(>50%), the model might not be reliable.

WebSphere® Support Technical Exchange 24

 IBM Software Group

GC View
ƒ Display GC
table view by
clicking on
Analysis -> GC
View

WebSphere® Support Technical Exchange 25

 IBM Software Group

GC View
ƒ You can sort
each column
by clicking on
column header.

WebSphere® Support Technical Exchange 26

 IBM Software Group

GC View sorted by Free

WebSphere® Support Technical Exchange 27

 IBM Software Group

Graph View

WebSphere® Support Technical Exchange 28

 IBM Software Group

Chart View
Red vertical dotted
line represents
“Exhausted”,
where there was
insufficient space
to satisfy
allocation failure.
You can click on
buttons to display
other data (Used,
Free and so on)

WebSphere® Support Technical Exchange 29

 IBM Software Group

Chart buttons (1-2)

ƒ Since Time(millisecond) elapsed since last allocation failure.
ƒ Freed Size(byte) of space that was freed during garbage
collection.
ƒ Requested Size(byte) of space that was requested during
allocation failure.
ƒ Free Size(byte) of space that was free after garbage
collection.
ƒ Total Size(byte) of Java heap after garbage collection.
ƒ Completed Time(millisecond) spent during allocation failure.

WebSphere® Support Technical Exchange 30

 IBM Software Group

Chart buttons (2-2)

ƒ GC Completed Time(millisecond) spent during garbage
collection.
ƒ Overhead Ratio(%) time spent in allocation failure vs. time
between AF
ƒ Zoom In Zoom in X axis
ƒ Zoom Out Zoom out X axis
ƒ Center Moves a point to center
ƒ Select Brings up GC View of a point

WebSphere® Support Technical Exchange 31

 IBM Software Group

Graph View
ƒ Free,
used, total
size are
displayed

WebSphere® Support Technical Exchange 32

 IBM Software Group

Graph View
ƒ Compact
and Sweep
buttons are
enabled.
ƒ You can
click on
Zoom In/Out
and Center
to navigate
chart in
various
levels.

WebSphere® Support Technical Exchange 33

 IBM Software Group

Graph View
ƒ You can
select a
point to
display
its table
view

WebSphere® Support Technical Exchange 34

 IBM Software Group

Usage Summary
ƒ Display
usage
summary
by clicking
on
Analysis-
>Usage
Summary

WebSphere® Support Technical Exchange 35

 IBM Software Group

Usage Summary

WebSphere® Support Technical Exchange 36

 IBM Software Group

Duration Summary
ƒ Duration
Summary
by clicking
on
Analysis-
.Duration
Summary

WebSphere® Support Technical Exchange 37

 IBM Software Group

Duration Summary

WebSphere® Support Technical Exchange 38

 IBM Software Group

AF Summary
ƒ AF Summary
by clicking
on Analysis-
>AF
Summary

WebSphere® Support Technical Exchange 39

 IBM Software Group

AF Summary
ƒ Display
usage
summary
by clicking
on
Analysis-
>Usage
Summary

WebSphere® Support Technical Exchange 40

 IBM Software Group

Option
ƒ Display
option by
clicking on
View-
>Option

WebSphere® Support Technical Exchange 41

 IBM Software Group

Option

ƒ You can change default directory

as well as the following settings:
ƒ Verbose Mode: display messages
in console
ƒ Save option: Saves options to
configuration file during exit
ƒ Mark Terminals: Mark points of
start/end
ƒ Mark non-terminals: Mark points
except for start and end

WebSphere® Support Technical Exchange 42

 IBM Software Group

Option
ƒ You can also change
color of chart

WebSphere® Support Technical Exchange 43

 IBM Software Group

Option
ƒ You can also
change color of
chart

WebSphere® Support Technical Exchange 44

 IBM Software Group

Clear Console
ƒ You can
clear logs
in console
by clicking
on View-
>Clear
Console

WebSphere® Support Technical Exchange 45

 IBM Software Group

Case Study I (1/8)

WebSphere® Support Technical Exchange 46

 IBM Software Group

Case Study I (2/8)

WebSphere® Support Technical Exchange 47

 IBM Software Group

Case Study I (3/8)

WebSphere® Support Technical Exchange 48

 IBM Software Group

Case Study I (4/8)

WebSphere® Support Technical Exchange 49

 IBM Software Group

Case Study I (5/8)

WebSphere® Support Technical Exchange 50

 IBM Software Group

Case Study I (6/8)

WebSphere® Support Technical Exchange 51

 IBM Software Group

Case Study I (7/8)

WebSphere® Support Technical Exchange 52

 IBM Software Group

Case Study I (8/8)

WebSphere® Support Technical Exchange 53

 IBM Software Group

Case Study II (1/5)

WebSphere® Support Technical Exchange 54

 IBM Software Group

Case Study II (2/5)

WebSphere® Support Technical Exchange 55

 IBM Software Group

Case Study II (3/5)

WebSphere® Support Technical Exchange 56

 IBM Software Group

Case Study II (4/5)

WebSphere® Support Technical Exchange 57

 IBM Software Group

Case Study II (5/5)

WebSphere® Support Technical Exchange 58

 IBM Software Group

Case Study III (1/5)

WebSphere® Support Technical Exchange 59

 IBM Software Group

Case Study III (2/5)

WebSphere® Support Technical Exchange 60

 IBM Software Group

Case Study III (3/5)

WebSphere® Support Technical Exchange 61

 IBM Software Group

Case Study III (4/5)

ƒ What is kCluster?

Objects that are on the Java heap are usually mobile; that is, the
garbage collector (GC) can move them around if it decides to
resequence the heap. Some objects, however, cannot be moved
either permanently or temporarily and might cause problems. One
of them is a class block.

The GC allocates a kCluster as the first object at the bottom of the

heap.
A kCluster is an area of storage that is used exclusively for class
blocks.
It is large enough to hold 1280 entries.
Each class block is 256 bytes long.

WebSphere® Support Technical Exchange 62

 IBM Software Group

Case Study III (5/5)

ƒ How can I find out the number of classes?

GC trace data obtained by setting

-Xtgc2 ( SDK 1.3.1 SR7 or higher)
or
-Dibm.dg.trc.print=st_verify ( SDK 1.4.2 )
provides a guide for the optimum size of the kCluster.

ƒ How can I set kCluster ?

-Xknnnn

where nnnn specifies the maximum number of classes the kCluster

contains. -Xk instructs the JVM to allocate space for nnnn class
blocks in kCluster.

WebSphere® Support Technical Exchange 63

 IBM Software Group

Generic JVM settings on admin. console

ƒ On WebSphere Application Server V5
Servers > Application Servers > server_name > Process Definition >
Java Virtual Machine> Generic JVM Arguments

ƒ On WebSphere Application Server V6

Servers > Application Servers > server_name > Java and Process
Management > Process definition > Java Virtual Machine > Generic
JVM Arguments

ƒ On WebSphere Application Server V4

1.Select the Application Server and go to the JVM Settings tab.
2.Click on the Advanced JVM Settings button.
3.Enter the values in the Generic JVM Arguments section.
4.Click OK.
5.Click Apply.
6.Stop and re-start the Application Server.

WebSphere® Support Technical Exchange 64

 IBM Software Group

Useful links
ƒ IBM SDK Diagnostics Guides
http://www-106.ibm.com/developerworks/java/jdk/diagnosis/
ƒ IBM Pattern Modeling and Analysis Tool for IBM Java
Garbage Collector http://www.alphaworks.ibm.com/tech/pmat
ƒ IBM HeapAnalyzer
http://www.alphaworks.ibm.com/tech/heapanalyzer
ƒ Webcast replay: Using IBM HeapAnalyzer to diagnose Java
heap issues
http://www.ibm.com/support/docview.wss?uid=swg27006624

WebSphere® Support Technical Exchange 65

 IBM Software Group

Additional WebSphere Product Resources

ƒ Discover the latest trends in WebSphere Technology and
implementation, participate in technically-focused briefings,
webcasts and podcasts at:
www.ibm.com/developerworks/websphere/community/
ƒ Learn about other upcoming webcasts, conferences and events:
www.ibm.com/software/websphere/events_1.html
ƒ Join the Global WebSphere User Group Community:
www.websphere.org
ƒ Access key product show-me demos and tutorials by visiting IBM
Education Assistant: ibm.com/software/info/education/assistant
ƒ Learn about the Electronic Service Request (ESR) tool for submitting
problems electronically:
www.ibm.com/software/support/viewlet/probsub/ESR_Overview_vie
wlet_swf.html
ƒ Sign up to receive weekly technical support emails:
www.ibm.com/software/support/einfo.html

WebSphere® Support Technical Exchange 66

 IBM Software Group

Questions?

WebSphere® Support Technical Exchange 67