IBM® Developer Kit and Runtime Environment, Java™ 2

Technology Edition, Version 1.4.2 򔻐򗗠򙳰

Diagnostics Guide for AIX 64-bit, z/OS64

and AMD64 platforms (for SAP)
IBM® Developer Kit and Runtime Environment, Java™ 2
Technology Edition, Version 1.4.2 򔻐򗗠򙳰

Diagnostics Guide for AIX 64-bit, z/OS64

and AMD64 platforms (for SAP)
 Note
Before using this information and the product it supports, read the information in Appendix H, “Notices,” on page 341.

Eighth Edition, limited availability (November 2006)

This limited availability edition applies to all the platforms that are included in the IBM Developer Kit and Runtime
Environment, Java 2 Technology Edition, Version 1.4.2 for AIX 64-bit, z/OS64, and AMD64 platforms and to all
subsequent releases and modifications until otherwise indicated in new editions.
© Copyright International Business Machines Corporation 2003, 2006. All rights reserved.
US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract
with IBM Corp.
Contents
Figures . . . . . . . . . . . . . . . ix Interaction of the Garbage Collector with
applications . . . . . . . . . . . . . . 18
Tables . . . . . . . . . . . . . . . xi How to coexist with the Garbage Collector . . . . 18
Root set . . . . . . . . . . . . . . 18
Thread local heap . . . . . . . . . . . 18
About this book . . . . . . . . . . xiii Bug reports . . . . . . . . . . . . . 19
What does the ″Java Virtual Machine (JVM)″ mean? xiii Finalizers . . . . . . . . . . . . . . 19
Who should read this book . . . . . . . . . xiii Manual invocation . . . . . . . . . . . 20
Before you read this book . . . . . . . . . xiii Summary . . . . . . . . . . . . . . 21
How to read this book . . . . . . . . . . xiii Frequently asked questions about the Garbage
Other sources of information . . . . . . . . xiv Collector . . . . . . . . . . . . . . . 21
Reporting problems in the JVM . . . . . . . xiv
Conventions and terminology used in this book . . xiv
Chapter 3. Understanding the class
How to send your comments . . . . . . . . xv
Contributors to this book . . . . . . . . . . xv loader . . . . . . . . . . . . . . . 25
Summary of changes . . . . . . . . . . . xv The parent-delegation model . . . . . . . . 25
Name spaces and the runtime package . . . . . 26
Why write your own class loader? . . . . . . . 26
Part 1. Understanding the IBM Virtual How to write your own class loader . . . . . . 27
Machine for Java . . . . . . . . . . 1
Chapter 4. Understanding the JIT . . . 29
Chapter 1. The building blocks of the JIT overview . . . . . . . . . . . . . . 29
IBM Virtual Machine for Java . . . . . . 3 How the JIT optimizes code . . . . . . . . . 30
Java application stack . . . . . . . . . . . 4 1) Inlining . . . . . . . . . . . . . . 30
IBM Virtual Machine for Java subcomponents . . . 4 2) Local optimizations . . . . . . . . . . 30
JVM API . . . . . . . . . . . . . . . 5 3) Control flow optimizations . . . . . . . 30
Diagnostics component . . . . . . . . . . 5 4) Global optimizations . . . . . . . . . 31
Memory management . . . . . . . . . . 5 5) Native code generation . . . . . . . . 31
Class loader . . . . . . . . . . . . . 6 Frequently asked questions about the JIT . . . . 31
Interpreter . . . . . . . . . . . . . . 6
Platform port layer . . . . . . . . . . . 6 Chapter 5. Understanding the ORB . . 33
CORBA . . . . . . . . . . . . . . . . 33
Chapter 2. Understanding the Garbage RMI and RMI-IIOP . . . . . . . . . . . . 33
Collector . . . . . . . . . . . . . . 7 Java IDL or RMI-IIOP? . . . . . . . . . . 34
RMI-IIOP limitations . . . . . . . . . . . 34
Overview of garbage collection . . . . . . . . 7
Further reading . . . . . . . . . . . . . 34
Object allocation . . . . . . . . . . . . 7
Examples of client–server applications . . . . . 34
Reachable objects . . . . . . . . . . . . 8
Interfaces . . . . . . . . . . . . . . 34
Garbage collection . . . . . . . . . . . 8
Remote object implementation (or servant) . . . 35
Heap size . . . . . . . . . . . . . . 8
Stub and ties generation . . . . . . . . . 35
Allocation . . . . . . . . . . . . . . . 9
Server code . . . . . . . . . . . . . 36
Heap lock allocation . . . . . . . . . . 10
Summary of major differences between RMI
Cache allocation . . . . . . . . . . . . 10
(JRMP) and RMI-IIOP . . . . . . . . . . 39
Detailed description of garbage collection . . . . 10
Using the ORB . . . . . . . . . . . . . 40
Mark phase . . . . . . . . . . . . . 10
How the ORB works . . . . . . . . . . . 43
Sweep phase . . . . . . . . . . . . . 12
The client side . . . . . . . . . . . . 43
Compaction phase . . . . . . . . . . . 13
The server side . . . . . . . . . . . . 47
Reference objects . . . . . . . . . . . 14
Features of the ORB . . . . . . . . . . . 49
Final reference processing . . . . . . . . 14
Portable object adapter . . . . . . . . . 49
JNI weak reference . . . . . . . . . . . 15
Fragmentation . . . . . . . . . . . . 51
Heap expansion . . . . . . . . . . . . 15
Portable interceptors . . . . . . . . . . 51
Heap shrinkage . . . . . . . . . . . . 15
Interoperable naming service (INS) . . . . . 54
How to do heap sizing . . . . . . . . . . 16
Other features . . . . . . . . . . . . 55
Initial and maximum heap sizes . . . . . . 16
Using verbosegc . . . . . . . . . . . . 17
Using fine tuning options. . . . . . . . . 17

© Copyright IBM Corp. 2003, 2006 iii

contents

Chapter 6. Understanding the Java Sending files to IBM support . . . . . . . . 86

Native Interface (JNI). . . . . . . . . 57 Getting files from IBM support . . . . . . . . 86
Overview of JNI . . . . . . . . . . . . . 57 Using your own ftp server . . . . . . . . . 87
The JNI and the Garbage Collector . . . . . . 58 Compressing core files . . . . . . . . . . . 87
Garbage Collector and object references . . . . 58 When you will receive your fix . . . . . . . . 87
Garbage Collector and global references . . . . 59
Garbage Collector and retained garbage . . . . 59 Part 3. Problem determination . . . 89
Copying and pinning . . . . . . . . . . . 60
Handling local references . . . . . . . . . . 60 Chapter 12. First steps in problem
Local reference scope . . . . . . . . . . 60
Summary of local references . . . . . . . . 63
determination . . . . . . . . . . . . 91
Local reference capacity . . . . . . . . . 63
Manually handling local references . . . . . 63 Chapter 13. Working in a WebSphere
Handling global references . . . . . . . . . 64 Application Server environment . . . . 93
Global reference capacity . . . . . . . . . 64
Handling exceptions . . . . . . . . . . . 64 | Chapter 14. AIX problem determination 95
Using the isCopy flag . . . . . . . . . . . 65 Setting up and checking your AIX environment . . 95
Using the mode flag . . . . . . . . . . . 65 | Enabling full AIX core files . . . . . . . . 96
A generic way to use the isCopy and mode flags . . 66 General debugging techniques . . . . . . . . 97
Synchronization . . . . . . . . . . . . . 66 Starting Javadumps in AIX . . . . . . . . 97
Debugging the JNI . . . . . . . . . . . . 67 Starting Heapdumps in AIX . . . . . . . . 97
JNI checklist . . . . . . . . . . . . . . 68 AIX debugging commands . . . . . . . . 97
| DBX Plug-in . . . . . . . . . . . . . 106
Chapter 7. Understanding Java Remote Diagnosing crashes . . . . . . . . . . . 107
Method Invocation . . . . . . . . . . 71 Documents to gather . . . . . . . . . . 107
The RMI implementation . . . . . . . . . . 71 Locating the point of failure . . . . . . . 108
Thread pooling for RMI connection handlers . . . 72 Debugging hangs . . . . . . . . . . . . 109
Understanding Distributed Garbage Collection AIX deadlocks . . . . . . . . . . . . 109
(DGC) . . . . . . . . . . . . . . . . 72 AIX busy hangs . . . . . . . . . . . 109
Debugging applications involving RMI . . . . . 73 Poor performance on AIX . . . . . . . . 111
Understanding memory usage . . . . . . . . 111
32- and 64-bit JVMs . . . . . . . . . . 112
Part 2. Submitting problem reports 75 The 32-bit AIX Virtual Memory Model . . . . 112
The 64-bit AIX Virtual Memory Model . . . . 113
Chapter 8. Overview of problem Changing the Memory Model (32-bit JVM) . . 113
submission . . . . . . . . . . . . . 77 The native and Java heaps . . . . . . . . 114
How does IBM service Java?. . . . . . . . . 77 The AIX 32-bit JVM default memory models . . 114
Submitting Java problem reports to IBM . . . . . 77 Monitoring the native heap . . . . . . . . 114
Java duty manager . . . . . . . . . . . . 77 Native heap usage . . . . . . . . . . . 115
Specifying MALLOCTYPE . . . . . . . . 116
Chapter 9. MustGather: Collecting the Monitoring the Java heap . . . . . . . . 116
Receiving OutOfMemory errors . . . . . . 116
correct data to solve problems . . . . 79
Is the Java or native heap exhausted? . . . . 117
Before you submit a problem report . . . . . . 79
Java heap exhaustion . . . . . . . . . . 117
Data to include . . . . . . . . . . . . . 79
Native heap exhaustion . . . . . . . . . 117
Things to try . . . . . . . . . . . . . . 80
AIX fragmentation problems . . . . . . . 117
Factors that affect JVM performance . . . . . . 80
Submitting a bug report . . . . . . . . . 118
Test cases . . . . . . . . . . . . . . . 80
Debugging performance problems . . . . . . 119
Performance problems – questions to ask . . . . 81
Finding the bottleneck . . . . . . . . . 119
CPU bottlenecks . . . . . . . . . . . 120
Chapter 10. Advice about problem Memory bottlenecks . . . . . . . . . . 123
submission . . . . . . . . . . . . . 83 I/O bottlenecks. . . . . . . . . . . . 124
Raising a problem report . . . . . . . . . . 83 JVM heap sizing . . . . . . . . . . . 124
What goes into a problem report? . . . . . . . 83 JIT compilation and performance . . . . . . 124
Problem severity ratings . . . . . . . . . . 83 Application profiling . . . . . . . . . . 124
Escalating problem severity . . . . . . . . . 84 MustGather information for AIX . . . . . . . 125
Getting AIX technical support . . . . . . . . 125
Chapter 11. Submitting data with a
problem report . . . . . . . . . . . 85 Chapter 15. Linux problem
IBM internal only (javaserv) . . . . . . . . . 85 determination . . . . . . . . . . . 127

iv Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 contents

Setting up and checking your Linux environment 127 Chapter 17. z/OS problem
Working directory . . . . . . . . . . . 127 determination . . . . . . . . . . . 151
Linux core files . . . . . . . . . . . . 127 Setting up and checking your z/OS environment 151
Threading libraries . . . . . . . . . . 128 Maintenance . . . . . . . . . . . . . 151
General debugging techniques . . . . . . . . 128 LE settings . . . . . . . . . . . . . 151
Starting Javadumps in Linux . . . . . . . 128 Environment variables . . . . . . . . . 151
Starting heapdumps in Linux . . . . . . . 128 Private storage usage . . . . . . . . . . 151
Using the dump extractor on Linux . . . . . 128 Setting up dumps . . . . . . . . . . . 152
Using core dumps . . . . . . . . . . . 129 General debugging techniques . . . . . . . . 152
Using system logs . . . . . . . . . . . 129 Starting Javadumps in z/OS . . . . . . . 152
Linux debugging commands . . . . . . . 130 Starting Heapdumps in z/OS . . . . . . . 152
Diagnosing crashes . . . . . . . . . . . 134 Using IPCS commands . . . . . . . . . 152
Checking the system environment . . . . . 134 Using dbx . . . . . . . . . . . . . 153
Gathering process information . . . . . . . 134 Interpreting error message IDs . . . . . . 153
Finding out about the Java environment . . . 134 Diagnosing crashes . . . . . . . . . . . 153
Debugging hangs . . . . . . . . . . . . 135 Documents to gather . . . . . . . . . . 153
Debugging memory leaks . . . . . . . . . 135 Determining the failing function . . . . . . 154
Debugging performance problems . . . . . . 135 Working with TDUMPs using IPCS . . . . . 155
System performance . . . . . . . . . . 135 Debugging hangs . . . . . . . . . . . . 160
JVM performance . . . . . . . . . . . 137 The process is deadlocked . . . . . . . . 160
JIT . . . . . . . . . . . . . . . . 138 The process is looping . . . . . . . . . 160
Collecting data from a fault condition in Linux . . 138 The process is performing badly . . . . . . 161
Collecting core files . . . . . . . . . . 138 Debugging memory leaks . . . . . . . . . 161
Producing Javadumps . . . . . . . . . 138 Allocations to LE HEAP . . . . . . . . . 161
Using system logs . . . . . . . . . . . 139 z/OS virtual storage . . . . . . . . . . 161
Determining the operating environment . . . 139 OutOfMemoryErrors . . . . . . . . . . 162
Sending information to Java Support . . . . 139 Debugging performance problems . . . . . . 163
Collecting additional diagnostic data . . . . 139 Collecting data from a fault condition in z/OS . . 164
Known limitations on Linux . . . . . . . . 140
Threads as processes . . . . . . . . . . 140
Floating stacks limitations . . . . . . . . 140
Chapter 18. Debugging the ORB . . . 165
glibc limitations . . . . . . . . . . . 140 Identifying an ORB problem . . . . . . . . 165
Font limitations . . . . . . . . . . . 141 What the ORB component contains . . . . . 165
CORBA limitations . . . . . . . . . . 141 What the ORB component does not contain . . 166
Platform-dependent problem . . . . . . . 166
JIT problem . . . . . . . . . . . . . 166
Chapter 16. Windows problem Fragmentation . . . . . . . . . . . . 166
determination . . . . . . . . . . . 143 Packaging . . . . . . . . . . . . . 166
Setting up and checking your Windows ORB versions . . . . . . . . . . . . 166
environment . . . . . . . . . . . . . . 143 Debug properties . . . . . . . . . . . . 167
Setting up your Windows environment for data ORB exceptions . . . . . . . . . . . . 168
collection . . . . . . . . . . . . . . 144 User exceptions . . . . . . . . . . . 168
General debugging techniques . . . . . . . . 145 System exceptions . . . . . . . . . . . 168
Starting Javadumps in Windows . . . . . . 145 Completion status and minor codes . . . . . 169
Starting Heapdumps in Windows . . . . . 145 Java2 security permissions for the ORB . . . . 169
Using the Cross-Platform Dump Formatter . . 145 Interpreting the stack trace . . . . . . . . . 170
System dump . . . . . . . . . . . . 145 Description string . . . . . . . . . . . 170
Diagnosing crashes in Windows . . . . . . . 146 Nested exceptions . . . . . . . . . . . 171
Data to send to IBM . . . . . . . . . . 147 Interpreting ORB traces . . . . . . . . . . 171
Debugging hangs . . . . . . . . . . . . 147 Message trace . . . . . . . . . . . . 171
Analyzing deadlocks . . . . . . . . . . 147 Comm traces . . . . . . . . . . . . 172
Getting a dump from a hung JVM . . . . . 147 Client or server . . . . . . . . . . . . 173
Debugging memory leaks . . . . . . . . . 147 Service contexts . . . . . . . . . . . 173
The Windows memory model . . . . . . . 148 Common problems . . . . . . . . . . . 174
Classifying leaks . . . . . . . . . . . 148 ORB application hangs . . . . . . . . . 174
Tracing leaks . . . . . . . . . . . . 148 Running the client without the server running
Using HeapDump to debug memory leaks . . 149 before the client is invoked . . . . . . . . 175
Debugging performance problems . . . . . . 149 Client and server are running, but not naming
Data required for submitting a bug report . . . 149 service. . . . . . . . . . . . . . . 175
Frequently reported problems . . . . . . . 150 Running the client with MACHINE2 (client)
Collecting data from a fault condition in Windows 150 unplugged from the network . . . . . . . 176

Contents v
contents

IBM ORB service: collecting data . . . . . . . 176 Using VerboseGC to obtain heap information . . . 208
Preliminary tests . . . . . . . . . . . 176
Data to be collected . . . . . . . . . . 177 Chapter 23. JVM dump initiation . . . 209
Overview. . . . . . . . . . . . . . . 209
Chapter 19. NLS problem Settings . . . . . . . . . . . . . . . 210
determination . . . . . . . . . . . 179 Platform-specific variations . . . . . . . . . 211
Overview of fonts . . . . . . . . . . . . 179 z/OS . . . . . . . . . . . . . . . 211
Font specification properties . . . . . . . 179 Windows . . . . . . . . . . . . . . 212
Fonts installed in the system . . . . . . . 179 Linux . . . . . . . . . . . . . . . 212
The font.properties file . . . . . . . . . . 180
The Linux font.properties file . . . . . . . 180 Chapter 24. Using dump agents . . . 213
The Windows font.properties file . . . . . . 181 Help options . . . . . . . . . . . . . 213
Font utilities . . . . . . . . . . . . . . 181 Dump types and triggering . . . . . . . . . 214
Font utilities on Linux, AIX, and z/OS . . . . 181 Types of dump agents - examples . . . . . . 215
Font utilities on Windows systems . . . . . 181 Console dumps . . . . . . . . . . . . 215
Common problems and possible causes . . . . 182 System dumps . . . . . . . . . . . . 215
Tool option . . . . . . . . . . . . . 216
Part 4. Using diagnostic tools . . . 185 Javadumps . . . . . . . . . . . . . 216
Heapdumps . . . . . . . . . . . . . 216
Default dump agents . . . . . . . . . . . 217
Chapter 20. Overview of the available Default settings for dumps . . . . . . . . . 218
diagnostics . . . . . . . . . . . . 187 Limiting dumps using filters . . . . . . . . 218
Categorizing the problem . . . . . . . . . 187 Removing dump agents . . . . . . . . . . 218
Platforms . . . . . . . . . . . . . . . 187
Summary of cross-platform tools . . . . . . . 187 Chapter 25. Using method trace . . . 219
Javadump (or Javacore) . . . . . . . . . 188 Running with method trace . . . . . . . . 219
Heapdump . . . . . . . . . . . . . 188 Examples of use . . . . . . . . . . . . 220
Cross-platform dump formatter . . . . . . 188 Where does the output appear? . . . . . . . 220
JVMPI tools . . . . . . . . . . . . . 188 Advanced options . . . . . . . . . . . . 220
JPDA Tools . . . . . . . . . . . . . 189 Real example . . . . . . . . . . . . . 220
JVM trace . . . . . . . . . . . . . 189
JVMRI . . . . . . . . . . . . . . . 189
Application trace . . . . . . . . . . . 190
Chapter 26. Using the dump formatter 223
Method trace . . . . . . . . . . . . 190 What the dump formatter is . . . . . . . . 223
JVM command-line parameters . . . . . . 190 Problems to tackle with the dump formatter . . . 224
JVM environment variables . . . . . . . . 190 Supported commands . . . . . . . . . . 224
Platform tools . . . . . . . . . . . . . 190 General commands . . . . . . . . . . 224
Commands for analysing the memory . . . . 226
Commands for working with classes . . . . 227
Chapter 21. Using Javadump. . . . . 191 Commands for working with objects . . . . 227
Enabling a Javadump. . . . . . . . . . . 191 Commands for working with Heapdumps. . . 228
The location of the generated Javadump . . . . 191 Commands for working with trace . . . . . 229
Triggering a Javadump . . . . . . . . . . 192 Example session . . . . . . . . . . . . 229
Interpreting a Javadump . . . . . . . . . 192
Javadump tags . . . . . . . . . . . . 193
Locks, monitors, and deadlocks (LOCKS) . . . 193
Chapter 27. JIT problem determination 237
Javadump sample output 1 (z/OS) . . . . . 196 Disabling the JIT . . . . . . . . . . . . 237
Javadump sample output 2 (Linux) . . . . . 202 Selectively disabling the JIT . . . . . . . . 237
Javadump sample output 3 (Windows) . . . . 202 Locating the failing method . . . . . . . . 238
Identifying JIT compilation failures . . . . . . 239
Performance of short-running applications . . . 240
Chapter 22. Using Heapdump . . . . 205
Summary of Heapdump . . . . . . . . . . 205
Information for users of previous releases of
Chapter 28. Garbage Collector
Heapdump . . . . . . . . . . . . . . 205 diagnostics . . . . . . . . . . . . 241
Enabling a Heapdump . . . . . . . . . . 205 How does the Garbage Collector work? . . . . 241
Explicit generation of a Heapdump . . . . . 206 Common causes of perceived leaks . . . . . . 241
Triggered generation of a Heapdump . . . . 206 Listeners . . . . . . . . . . . . . . 241
Enabling text formatted (″classic″) Heapdumps 207 Hash tables . . . . . . . . . . . . . 242
Location of the generated Heapdump . . . . . 207 Static data . . . . . . . . . . . . . 242
Producing a Heapdump using jdmpview . . . . 207 JNI references . . . . . . . . . . . . 242
Available tools for processing Heapdumps . . . 208 Premature expectation . . . . . . . . . 242

vi Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 contents

Objects with finalizers . . . . . . . . . 242 Chapter 31. Using the Reliability,

Basic diagnostics (-verbosegc) . . . . . . . . 242 Availability, and Serviceability
Garbage collection triggered by System.gc() . . 242
Interface . . . . . . . . . . . . . 283
Allocation failures . . . . . . . . . . . 243
Preparing to use JVMRI . . . . . . . . . . 283
Global collections . . . . . . . . . . . 245
Writing an agent . . . . . . . . . . . 283
Scavenger collections . . . . . . . . . . 245
Registering a trace listener . . . . . . . . 284
Concurrent mark . . . . . . . . . . . 246
Changing trace options . . . . . . . . . 285
Advanced diagnostics . . . . . . . . . . 249
Launching the agent . . . . . . . . . . 285
-Xdisableexplicitgc . . . . . . . . . . . 249
Building the agent . . . . . . . . . . . 285
-Xgcthreads . . . . . . . . . . . . . 250
Agent design . . . . . . . . . . . . 286
-Xclassgc . . . . . . . . . . . . . . 250
JVMRI functions . . . . . . . . . . . . 286
-Xnoclassgc . . . . . . . . . . . . . 250
API calls provided by JVMRI . . . . . . . . 286
-Xcompactgc. . . . . . . . . . . . . 250
CreateThread . . . . . . . . . . . . 286
-Xnocompactgc . . . . . . . . . . . . 250
DumpDeregister . . . . . . . . . . . 287
-Xcompactexplicitgc . . . . . . . . . . 250
DumpRegister . . . . . . . . . . . . 287
-Xnocompactexplicitgc . . . . . . . . . 250
DynamicVerbosegc . . . . . . . . . . 287
TGC tracing . . . . . . . . . . . . . . 250
GenerateHeapdump . . . . . . . . . . 288
-Xtgc:backtrace . . . . . . . . . . . . 250
GenerateJavacore . . . . . . . . . . . 288
-Xtgc:compaction . . . . . . . . . . . 251
GetComponentDataArea . . . . . . . . . 288
-Xtgc:concurrent . . . . . . . . . . . 251
GetRasInfo . . . . . . . . . . . . . 288
-Xtgc:dump . . . . . . . . . . . . . 251
InitiateSystemDump . . . . . . . . . . 289
| -Xtgc:excessiveGC . . . . . . . . . . . 252
InjectOutOfMemory . . . . . . . . . . 289
-Xtgc:freelist . . . . . . . . . . . . . 252
InjectSigSegv . . . . . . . . . . . . 289
-Xtgc:parallel . . . . . . . . . . . . 253
NotifySignal . . . . . . . . . . . . . 289
-Xtgc:references . . . . . . . . . . . . 253
ReleaseRasInfo . . . . . . . . . . . . 290
-Xtgc:scavenger . . . . . . . . . . . . 253
RunDumpRoutine . . . . . . . . . . . 290
-Xtgc:terse . . . . . . . . . . . . . 254
SetOutOfMemoryHook . . . . . . . . . 290
Heap and native memory use by the JVM . . . . 254
TraceDeregister . . . . . . . . . . . . 290
Native code . . . . . . . . . . . . . 255
TraceRegister . . . . . . . . . . . . 291
Large native objects . . . . . . . . . . 255
TraceResume . . . . . . . . . . . . 291
TraceResumeThis . . . . . . . . . . . 291
Chapter 29. Class-loader diagnostics 257 TraceSet . . . . . . . . . . . . . . 291
Class-loader command-line options . . . . . . 257 TraceSnap . . . . . . . . . . . . . 292
Class-loader runtime diagnostics . . . . . . . 257 TraceSuspend . . . . . . . . . . . . 292
Loading from native code . . . . . . . . . 258 TraceSuspendThis . . . . . . . . . . . 292
RasInfo structure . . . . . . . . . . . . 292
Chapter 30. Tracing Java applications RasInfo request types . . . . . . . . . . . 293
and the JVM . . . . . . . . . . . . 259 Intercepting trace data . . . . . . . . . . 293
What can be traced? . . . . . . . . . . . 259 The -Xtrace:external=<option> . . . . . . . 293
Tracing methods . . . . . . . . . . . 259 Calling external trace . . . . . . . . . . . 294
Tracing applications . . . . . . . . . . 259 Formatting . . . . . . . . . . . . . . 294
Internal trace . . . . . . . . . . . . 260
Where does the data go? . . . . . . . . . 260 Chapter 32. Using the JVMPI . . . . . 297
Placing trace data into in-storage buffers . . . 260 The HPROF profiler . . . . . . . . . . . 297
Placing trace data into a file . . . . . . . 261 Explanation of the HPROF output file . . . . . 298
External tracing . . . . . . . . . . . 261
Tracing to stderr . . . . . . . . . . . 261 Chapter 33. Using DTFJ . . . . . . . 303
Trace combinations . . . . . . . . . . 261 Overview of the DTFJ interface . . . . . . . 303
Controlling the trace . . . . . . . . . . . 261 DTFJ example application . . . . . . . . . 307
Specifying trace options . . . . . . . . . 262
Trace options summary . . . . . . . . . 262
Detailed descriptions of trace options . . . . 264 Part 5. Appendixes . . . . . . . . 311
Using the trace formatter . . . . . . . . 275
Trace properties file . . . . . . . . . . 276 Appendix A. Compatibility tables . . . 313
What to trace . . . . . . . . . . . . 276
Determining the tracepoint ID of a tracepoint . . 276 Appendix B. ORB tracing for
Application trace . . . . . . . . . . . . 277
Implementing application trace . . . . . . 277
WebSphere Application Server
version 5 . . . . . . . . . . . . . 315
Enabling trace at server startup . . . . . . . 315

Contents vii
contents

Changing the trace on a running server . . . . 316 z/OS environment variables . . . . . . . . 326
Selecting ORB traces . . . . . . . . . . . 316
Appendix F. Command-line options 329
Appendix C. CORBA GIOP message Specifying command-line options. . . . . . . 329
format . . . . . . . . . . . . . . 317 General command-line options . . . . . . . 329
GIOP header . . . . . . . . . . . . . 317 System property command-line options . . . . 330
Request header . . . . . . . . . . . . . 318 Nonstandard command-line options . . . . . . 331
Request body . . . . . . . . . . . . . 318 JIT command-line options . . . . . . . . . 333
Reply header . . . . . . . . . . . . . 318 Garbage Collector command-line options . . . . 334
Reply body (based on reply status) . . . . . . 319 General Garbage Collection options . . . . . 334
Cancel request header . . . . . . . . . . 319 Scavenger options . . . . . . . . . . . 336
Locate request header . . . . . . . . . . 319 Compact options . . . . . . . . . . . 336
Locate reply header . . . . . . . . . . . 320 Concurrent options . . . . . . . . . . 336
Locate reply body . . . . . . . . . . . . 320 Trace GC options . . . . . . . . . . . 336
Fragment message . . . . . . . . . . . . 320
Fragment header (GIOP 1.2 only) . . . . . . . 320 Appendix G. Default settings for the
JVM . . . . . . . . . . . . . . . 339
Appendix D. CORBA minor codes . . 321
Appendix H. Notices . . . . . . . . 341
Appendix E. Environment variables 323 Trademarks . . . . . . . . . . . . . . 342
Displaying the current environment . . . . . . 323
Setting an environment variable . . . . . . . 323 Index . . . . . . . . . . . . . . . 345
Separating values in a list . . . . . . . . . 323
JVM environment settings . . . . . . . . . 323

viii Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Figures
1. The components of a typical Java Application 6. Thread stack pointing to an object so that the
Stack and the IBM JRE . . . . . . . . . 4 Garbage Collector can see the object . . . . 62
2. Subcomponent structure of the IBM Virtual 7. Thread stack not pointing to an object so that
Machine for Java . . . . . . . . . . . 5 the Garbage Collector cannot see the object . . 62
3. The ORB client side . . . . . . . . . . 43 8. The AIX 32–Bit Memory Model with
4. Relationship between the ORB, the object MAXDATA=0 (default) . . . . . . . . 112
adapter, the skeleton, and the object 9. Screenshot of the ReportEnv tool . . . . . 144
implementation . . . . . . . . . . . 49 10. Diagram of the DTFJ interface . . . . . . 306
5. Simple portable object adapter architecture 51

© Copyright IBM Corp. 2003, 2006 ix

x Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Tables
1. Commands for stubs and ties (skeletons) 35 13. Options that control tracepoint selection 263
2. Stub and tie files . . . . . . . . . . . 36 14. Options that indirectly affect tracepoint
3. Deprecated Sun properties . . . . . . . 42 selection . . . . . . . . . . . . . 263
4. JNI checklist . . . . . . . . . . . . 68 15. Triggering and suspend or resume . . . . 263
5. Usage of ulimit . . . . . . . . . . . 127 16. Options that specify output files . . . . . 264
6. Methods affected when running with Java 2 17. MiscellaneousTrace control options . . . . 264
SecurityManager . . . . . . . . . . 141 18. CORBA GIOP messages . . . . . . . . 317
7. Packaging . . . . . . . . . . . . . 166 19. JVM environment settings — general options 324
8. Methods affected when running with Java 2 20. Basic JIT options . . . . . . . . . . 325
SecurityManager . . . . . . . . . . 169 21. Javadump and Heapdump options . . . . 325
9. Signal mappings on different platforms 211 22. Diagnostics options . . . . . . . . . 326
10. Usage from java -Xdump:help . . . . . . 213 23. Cross platform defaults . . . . . . . . 339
11. Types of dump . . . . . . . . . . . 213 24. Platform specific defaults . . . . . . . 340
12. Keywords . . . . . . . . . . . . . 214

© Copyright IBM Corp. 2003, 2006 xi

xii Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
About this book
This book describes debugging techniques and the diagnostic tools that are
available to help you solve problems with Java™ JVMs. It also gives guidance on
how to submit problems to IBM®.

Note that although this book covers Windows AMD64. this platform is available
only internally to IBM for testing purposes.

What does the ″Java Virtual Machine (JVM)″ mean?

The installable Java package supplied by IBM comes in two versions:
v The Java Runtime Environment (JRE)
v The Java Software Development Kit (SDK)

The JRE provides runtime support for Java applications. The SDK provides the
Java compiler and other development tools. The SDK includes the JRE.

The JRE (and, therefore, the SDK) includes a Java Virtual Machine (JVM). This is
the application that executes a Java program. A Java program requires a JVM to
run on a particular platform, such as Linux™, z/OS®, or Windows®.

The IBM SDK, Version 1.4.2 on z/OS 64 and AMD64 platforms, contains a different
implementation of the JVM and the Just-In-Time compiler (JIT) from releases of the
IBM SDK on other platforms (such as 31-bit z/OS, or 32-bit Intel). You can identify
this implementation from the string “IBM J9SE VM” in the output from the java
-version command. For diagnostics information on other IBM SDKs, see
http://www-106.ibm.com/developerworks/java/jdk/diagnosis/.

This book describes problem determination and diagnostics for the JVM.

Who should read this book

This book is for anyone who is responsible for solving problems with Java.

Before you read this book

Before you can use this book, you must have a good understanding of Java
Developer Kits and the Runtime Environment.

How to read this book

This book is to be used with the IBM SDK 1.4.2, for Linux and Windows on
AMD64 platforms and z/OS 64-bit.

Check the full version of your installed JVM. If you do not know how to do this,
see Chapter 12, “First steps in problem determination,” on page 91. Some of the
diagnostic tools described in this book apply only to this version or later.

You can use this book in three ways:

© Copyright IBM Corp. 2003, 2006 xiii

How to read this book

v As an overview of how the IBM Virtual Machine for Java operates, with
emphasis on the interaction with Java. Part 1 of the book provides this
information. You might find this information helpful when you are designing
your application.
v As straightforward guide to determining a problem type, collecting the
necessary diagnostic data, and sending it to IBM. Part 2 and Part 3 of the book
provide this information.
v As the reference guide to all the diagnostic tools that are available in the IBM
Virtual Machine for Java. This information is given in Part 4 of the book.

The parts overlap in some ways. For example, Part 3 refers to chapters that are in
Part 4 when those chapters describe the diagnostics data that is required. You will
be able to more easily understand some of the diagnostics that are in Part 4 if you
read the appropriate chapter in Part 1.

The appendixes provide supporting reference information that is gathered into

convenient tables and lists.

Other sources of information

v For the latest tools and documentation, see IBM developerWorks at:

http://www.ibm.com/developerworks/java/
v For Java documentation, see:

http://java.sun.com/products/jdk/1.4/docs/index.html
v For the IBM Java SDKs, see IBM Java downloads at:

http://www.ibm.com/developerworks/java/jdk/index.html

Reporting problems in the JVM

If you want to use this book only to determine your problem and to send a
problem report to IBM, go to Part 3, “Problem determination,” on page 89 of the
book, and to the chapter that relates to your platform. Go to the section that
describes the type of problem that you are having. This section might offer advice
about how to correct the problem, and might also offer workarounds. The section
will also tell you what data IBM service needs you to collect to diagnose the
problem. Collect the data and send a problem report and associated data to IBM
service, as described in Part 2, “Submitting problem reports,” on page 75 of the
book.

Conventions and terminology used in this book

Command-line options, system parameters, and class names are shown in bold.
For example:
v -Xresettable
v -Xinitsh
v -Dibm.jvm.trusted.middleware.class.path
v java.security.SecureClassLoader

Functions and methods are shown in a monospaced font. For example:

v ResetJavaVM()

xiv Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 conventions and terminology

v QueryJavaVM()

Options shown with values in braces signify that one of the values must be
chosen. For example:

-Xverify:{remote | all | none}

with the default underscored.

Options shown with values in brackets signify that the values are optional. For
example:

-Xrunhprof[:help][:<suboption>=<value>,...]

In this book, any reference to Sun is intended as a reference to Sun Microsystems,

Inc.

How to send your comments

Your feedback is important in helping to provide accurate and useful information.
If you have any comments about this book, you can send them by e-mail to
[email protected]. Include the name of the book, the part number of the
book, the platform you are using, the version of your JVM, and, if applicable, the
specific location of the text you are commenting on (for example, a page number
or table number).

Do not use this method for sending in bug reports on the JVM. For these, use the
usual methods, as described in Part 2, “Submitting problem reports,” on page 75.

Contributors to this book

This new edition of the Diagnostics Guide has been put together by members of the
IBM Java Technology Center development and service departments in Hursley,
Bangalore, Austin, Toronto, and Ottawa.

Summary of changes
For the August 2006 edition (SC34-6359-06) of this book, minor changes have been
made.

For the April 2006 edition (SC34-6359-05) of this book, minor changes have been
made.

For the January 2006 edition (SC34-6359-04) of this book, the following changes
have been made:
v A new chapter, Chapter 33, “Using DTFJ,” on page 303

For the September 2005 edition (SC34-6359-03) of this book, the following changes
have been made:
v Further changes to Chapter 22, “Using Heapdump,” on page 205.

For the June 2005 edition (SC34-6359-02) of this book the following changes were
made:
v Inclusion of the items in the addenda file.
v Major changes to Chapter 22, “Using Heapdump,” on page 205.

About this book xv

contributors

v Small updates to Chapter 23, “JVM dump initiation,” on page 209.

v Almost a complete rewrite of Chapter 24, “Using dump agents,” on page 213
v Additions to the CORBA properties and minor codes.

For the January 2005 update to this book, the information about jdkiv has been
removed from Chapter 17, “z/OS problem determination,” on page 151.

xvi Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Part 1. Understanding the IBM Virtual Machine for Java
The information in this part of the book will give you a basic understanding of the
JVM. It provides:
v Background information to explain why some diagnostics work the way they do
v Useful information for application designers
v An explanation of some parts of the JVM
A fairly large amount of information about the garbage collector is provided,
because the garbage collector often seems to be the most difficult part of the JVM
to understand.

Other sections provide a summary, especially where guidelines about the use of
the JVM are appropriate. This part is not intended as a description of the design of
the JVM, except that it might influence application design or promote an
understanding of why things are done the way that they are.

This part also provides a chapter that describes the IBM Object Request Broker
(ORB) component. The IBM ORB ships with the JVM and is used by the IBM
WebSphere® Application Server. It is one of the enterprise features of the Java 2
Standard Edition. The ORB is a tool and runtime component that provides
distributed computing through the OMG-defined CORBA IIOP communication
protocol. The ORB runtime consists of a Java implementation of a CORBA ORB.
The ORB toolkit provides APIs and tools for both the RMI programming model
and the IDL programming model.

The chapters in this part are:

v Chapter 1, “The building blocks of the IBM Virtual Machine for Java,” on page 3
v Chapter 2, “Understanding the Garbage Collector,” on page 7
v Chapter 3, “Understanding the class loader,” on page 25
v Chapter 4, “Understanding the JIT,” on page 29
v Chapter 5, “Understanding the ORB,” on page 33
v Chapter 6, “Understanding the Java Native Interface (JNI),” on page 57
v Chapter 7, “Understanding Java Remote Method Invocation,” on page 71

© Copyright IBM Corp. 2003, 2006 1

2 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 1. The building blocks of the IBM Virtual Machine for
Java
The IBM Java Virtual Machine (JVM) is a core component of the IBM Java Runtime
Environment (JRE). The JVM is a virtualized computing machine that follows a
well-defined specification for the runtime requirements of the Java programming
language. It is called ″virtual″ because it provides a machine interface that is
independent of the underlying operating system and machine hardware
architecture. This independence from hardware and operating system is a
cornerstone of the write-once run-anywhere value of Java programs. Java programs
are compiled into ″bytecodes″ that target the abstract virtual machine; the JVM is
responsible for implementing concretely the bytecodes on the specific operating
system and hardware combinations.

The JVM specification also defines several other runtime characteristics. All JVMs :
v Must conform to the same format of runtime files
v Provide fundamental runtime security such as bytecode verification
v Provide intrinsic operations such as performing arithmetic and allocating new
objects
JVMs that implement the specification completely and correctly are called
″compliant″. The IBM Virtual Machine for Java is certified as compliant. However,
not all compliant JVMs are identical. JVM implementers have a wide degree of
freedom to define characteristics of the JVM that are beyond the scope of the
specification. For example, implementers might choose to favour performance or
memory footprint; they might design the JVM for rapid deployment on new
platforms or for various degrees of serviceability. All the JVMs that are currently
used commercially come with a supplementary compiler that takes bytecodes and
outputs platform-dependent machine-code. This compilers works in conjunction
with the JVM to select parts of the Java program that would benefit from the
compilation of bytecode, and replaces the JVM’s virtualized interpretation of these
areas of bytecode with concrete code. This is called ″just-in-time compilation″ (JIT).
IBM’s JIT compiler is described in Chapter 4, “Understanding the JIT,” on page 29.

The IBM Virtual Machine for Java contains a number of private and proprietary
technologies that distinguish it from other implementations of the JVM. In this
release, IBM has made a significant change to the JVM and JIT compiler that were
provided in earlier releases, while retaining full Java compliance. When you read
this Diagnostics Guide, bear in mind that the particular unspecified behavior of
this release of the JVM might be different to the behavior that you experienced in
previous releases. Java programmers should not rely on the unspecified behavior
of a particular JRE for this reason.

This Diagnostics Guide is not a JVM specification; it discusses the characteristics of

the IBM JRE that might affect the non-functional behavior of your Java program.
This guide also provides information to assist you with tracking down problems
and offers advice, from the point of view of the JVM implementer, on how you can
tune your applications. There are many other sources for good advice about Java
performance, descriptions of the semantics of the Java runtime libraries, and tools
to profile and analyze in detail the execution of applications.

© Copyright IBM Corp. 2003, 2006 3

Java application stack

Java application stack

Figure 1 shows the components of a typical Java Application Stack and the IBM
JRE.

Java Application Stack

Java Application
Java Code

Java Class
Extensions
Class Libraries ORB

Native Libraries
Native Code

NativeOpt. User
IBM JVM Packages Native
Exts.

Others

Platform

Figure 1. The components of a typical Java Application Stack and the IBM JRE

A Java application uses the Java class libraries that are provided by the JRE to
implement the application-specific logic. The class libraries, in turn, are
implemented in terms of other class libraries and, eventually, in terms of primitive
native operations that are provided directly by the JVM. In addition, some
applications must access native code directly.

The JVM facilitates the invocation of native functions by Java applications and a
number of well-defined Java Native Interface functions for manipulating Java from
native code (for more information, see Chapter 6, “Understanding the Java Native
Interface (JNI),” on page 57).

IBM Virtual Machine for Java subcomponents

The IBM Virtual Machine for Java technology comprises a set of subcomponents
that implement a logical grouping of function. Figure 2 shows the subcomponent
structure of the IBM Virtual Machine for Java.

Figure 2 on page 5 shows subcomponent structure of the IBM Virtual Machine for
Java.

4 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 JVM API

JVM API

Diagnostics Memory Class loader Interpreter

management

Platform interface

Figure 2. Subcomponent structure of the IBM Virtual Machine for Java

JVM API
The JVM API encapsulates all the interaction between external programs and the
JVM. Examples include:
v Creation and initialization of the JVM through the invocation APIs.
v Interaction with the standard Java launchers, including handling command-line
directives.
v Presentation of public JVM APIs such as JNI, JVMDI, JVMPI, and so on.
v Presentation and implementation of private JVM APIs used by core Java classes.

Diagnostics component
The diagnostics component provides Reliability, Availability, and Serviceability
(RAS) facilities to the JVM.

The IBM Virtual Machine for Java is distinguished by its extensive RAS
capabilities. The IBM Virtual Machine for Java is designed to be deployed in
business-critical operations and includes several trace and debug utilities to assist
with problem determination.

If a problem occurs in the field, IBM’s service engineers can use the capabilities of
the diagnostics component to trace the runtime function of the JVM and identify
the cause of the problem. The diagnostics component can produce output
selectively from various parts of the JVM and the JIT. Part 4, “Using diagnostic
tools,” on page 185 describes various uses of the diagnostics component.

Memory management
The memory management subcomponent is responsible for the efficient use of
system memory by a Java application.

Java programs run in a managed execution environment. When a Java program

requires storage, the memory management subcomponent allocates the application
a discrete region of unused memory. After the application no longer refers to the

Chapter 1. The building blocks of the IBM Virtual Machine for Java 5
Memory management

storage, the memory management subcomponent must recognize that the storage
is unused and reclaim the memory for subsequent reuse by the application or
return it to the operating system.

The memory management subcomponent has several policy options that you can
specify when you deploy the application. Chapter 2, “Understanding the Garbage
Collector,” on page 7 discusses memory management in the IBM Virtual Machine
for Java.

Class loader
The class loader subcomponent is responsible for supporting Java’s dynamic code
loading facilities, including:
v Reading standard Java .class files.
v Resolving class definitions in the context of the current runtime environment.
v Verifying the bytecodes defined by the class file to determine whether the
bytecodes are language-legal.
v Initializing the class definition after it is accepted into the managed runtime
environment.
v Various reflection APIs for introspection on the class and its defined members.

Interpreter
The interpreter is the implementation of the stack-based bytecode machine that is
defined in the JVM specification. Each bytecode affects the state of the machine
and, as a whole, the bytecodes define the logic of the application.

The interpreter executes bytecodes on the operand stack, calls native functions,
contains and defines the interface to the JIT compiler, and provides support for
intrinsic operations such as arithmetic and the creation of new instances of Java
classes.

The interpreter is designed to execute bytecodes very efficiently. It can switch

between running bytecodes and handing control to the platform-specific
machine-code produced by the JIT compiler. The JIT compiler is described in
Chapter 4, “Understanding the JIT,” on page 29.

Platform port layer

The ability to reuse the code for the JVM across numerous operating systems and
processor architectures is made possible by the platform port layer.

The platform port layer is an abstraction of the native platform functions that are
required by the JVM. Other subcomponents of the JVM are written in terms of the
platform-neutral platform port layer functions. Further porting of the JVM requires
the provision of concrete implementations of the platform port layer facilities
rather than wholesale changes to the JVM code.

6 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 2. Understanding the Garbage Collector
This chapter describes the Garbage Collector under these headings:
v “Overview of garbage collection”
v “Allocation” on page 9
v “Detailed description of garbage collection” on page 10
v “How to do heap sizing” on page 16
v “Interaction of the Garbage Collector with applications” on page 18
v “How to coexist with the Garbage Collector” on page 18
v “Frequently asked questions about the Garbage Collector” on page 21

For detailed information about diagnosing Garbage Collector problems, see

Chapter 28, “Garbage Collector diagnostics,” on page 241.

For reference information about the Garbage Collector command-line parameters,

see “Garbage Collector command-line options” on page 334.

Overview of garbage collection

This chapter provides:
v A summary of some of the diagnostic techniques that are described elsewhere in
this book
v Knowledge of how the Garbage Collector works so that you can design
applications accordingly

The Garbage Collector allocates areas of storage in the heap. These areas of storage
define objects and arrays. When allocated, an object continues to be live while a
reference (pointer) to it exists somewhere in the active state of the JVM; therefore
the object is reachable. When an object ceases to be referenced from the active state,
it becomes garbage and can be reclaimed for reuse. When this reclamation occurs,
the Garbage Collector must process a possible finalizer and also ensure that any
internal JVM resources that are associated with the object are returned to the pool
of such resources.

Object allocation
Object allocation is driven by requests from inside the JVM for storage for Java
objects, arrays, or classes. Every allocation nominally requires a heap lock to be
acquired to prevent concurrent thread access. To optimize this allocation, particular
areas of the heap are dedicated to a thread, and that thread can allocate from its
local heap area without the need to lock out other threads. This technique delivers
the best possible allocation performance for small objects. Objects are allocated
directly from a thread local allocation buffer, which the thread has previously
allocated from the heap. A new object is allocated from this cache without the need
to grab the heap lock. All objects less than 512 bytes (768 bytes on 64-bit platforms)
are allocated from the cache. Larger objects are allocated from the cache if they can
be contained in the existing cache. This cache is often referred to as the thread local
heap or TLH.

© Copyright IBM Corp. 2003, 2006 7

Reachable objects

Reachable objects
The active state of the JVM is made up of the set of stacks that represents the
threads, the statics that are inside Java classes, and the set of local and global JNI
references. All functions that are invoked inside the JVM itself cause a frame on the
thread stack. This information is used to find the roots. These roots are then used to
find references to other objects. This process is repeated until all reachable objects
are found.

Garbage collection
When the JVM cannot allocate an object from the current heap because of lack of
space, a memory allocation fault occurs, and the Garbage Collector is invoked. The
first task of the Garbage Collector is to collect all the garbage that is in the heap.
This process starts when any thread calls the Garbage Collector either indirectly as
a result of allocation failure, or directly by a specific call to System.gc(). The first
step is to acquire exclusive control on the virtual machine to prevent any further
Java operations. Garbage collection can then begin. It occurs in three phases:
v Mark
v Sweep
v Compaction (optional)

Mark phase
In the mark phase, all the objects that are referenced from the thread stacks, statics,
interned strings, and JNI references are identified. This action creates the root set of
objects that the JVM references. Each of those objects might, in turn, reference
others. Therefore, the second part of the process is to scan each object for other
references that it makes. These two processes together generate a bit vector that
defines the beginning of all the live objects.

Sweep phase
The sweep phase uses the mark bit vector generated by the mark phase to identify
the chunks of heap storage that can be reclaimed for future allocations; these
chunks are added to the pool of free space.

Compaction phase
When the garbage has been reclaimed from the heap, the Garbage Collector can
consider compacting the resulting set of objects to remove the spaces that are
between them. Because compaction can take a long time, the Garbage Collector
only compacts when it is absolutely necessary. Compaction is, therefore, a rare
event.

Heap size
The maximum heap size is controlled by the -Xmx option. If this option is not
specified, the default applies as follows:
Windows
Half the real storage with a minimum of 16 MB and a maximum of
2 GB -1.
z/OS 64 MB.
Linux Half the real storage with a minimum of 16 MB and a maximum of
512 MB -1.

The initial size of the heap is controlled by the -Xms option. If this option is not
specified, the default applies as follows:

8 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Heap size

Windows and Linux

4 MB
z/OS 1 MB

Some basic heap sizing problems

For the majority of applications, the default settings work well. The heap expands
until it reaches a steady state, then remains in that state, which should give a heap
occupancy (the amount of live data on the heap at any given time) of 70%. At this
level, the frequency and pause time of garbage collection should be acceptable.

For some applications, the default settings might not give the best results. Listed
here are some problems that might occur, and some suggested actions that you can
take. Use verbosegc to help you monitor the heap.
The frequency of garbage collections is too high until the heap reaches a steady
state.
Use verbosegc to determine the size of the heap at a steady state and set -Xms
to this value.
The heap is fully expanded and the occupancy level is greater than 70%.
Increase the -Xmx value so that the heap is not more than 70% occupied, but
for best performance try to ensure that the heap never pages. The maximum
heap size should, if possible, be able to be contained in physical memory to
avoid paging.
At 70% occupancy the frequency of garbage collections is too great.
Change the setting of -Xminf. The default is 0.3, which tries to maintain 30%
free space by expanding the heap. A setting of 0.4, for example, increases this
free space target to 40%, and reduces the frequency of garbage collections.
Pause times are too long.
Try using -Xgcpolicy:optavgpause. This reduces the pause times and makes
them more consistent when the heap occupancy rises. It does, however, reduce
throughput by approximately 5%, although this value varies with different
applications.

Here are some useful tips:

v Ensure that the heap never pages; that is, the maximum heap size must be able
to be contained in physical memory.
v Avoid finalizers. You cannot guarantee when a finalizer will run, and often they
cause problems. If you do use finalizers, try to avoid allocating objects in the
finalizer method. A verbosegc trace shows whether finalizers are being called.
v Avoid compaction. A verbosegc trace shows whether compaction is occurring.
Compaction is usually caused by requests for large memory allocations. Analyze
requests for large memory allocations and avoid them if possible. If they are
large arrays, for example, try to split them into smaller arrays.

Allocation
The Garbage Collector is the JVM memory manager and is therefore responsible
for allocating memory in addition to collecting garbage. Because the task of
memory allocation is small, compared to that of garbage collection, the term
“garbage collection” usually also means “memory management”.

Chapter 2. Understanding the Garbage Collector 9

Heap lock allocation

Heap lock allocation

Heap lock allocation occurs when the allocation request cannot be satisfied in the
existing cache; see “Cache allocation.” As its name implies, heap lock allocation
requires a lock and is therefore avoided, if possible, by using the cache.

If the Garbage Collector cannot find a big enough chunk of free storage, allocation
fails and the Garbage Collector must perform a garbage collection. After a garbage
collection cycle, if the Garbage Collector created enough free storage, it searches
the freelist again and picks up a free chunk. If the Garbage Collector does not find
enough free storage, it returns out of memory. The heap lock is released either after
the object has been allocated, or if not enough free space is found.

Cache allocation
Cache allocation is specifically designed to deliver the best possible allocation
performance for small objects. Objects are allocated directly from a thread local
allocation buffer that the thread has previously allocated from the heap. A new
object is allocated from this cache without the need to grab the heap lock;
therefore, cache allocation is very efficient.

All objects less than 512 bytes (768 bytes on 64-bit platforms) are allocated from the
cache. Larger objects are allocated from the cache if they can be contained in the
existing cache; if not a locked heap allocation is performed.

The cache block is sometimes called a thread local heap (TLH). The size of the
TLH varies from 2 KB to 128 KB, depending on the use of the TLH.

Detailed description of garbage collection

Garbage collection is performed when an allocation failure occurs in heap lock
allocation, or if a specific call to System.gc() occurs. The thread that has the
allocation failure or the System.gc() call takes control and performs the garbage
collection. The first step is to acquire exclusive control on the Virtual machine to
prevent any further Java operations. Garbage collection then goes through the
three phases: mark, sweep, and, if required, compaction. The IBM Garbage
Collector is a stop-the-world (STW) operation, because all application threads are
stopped while the garbage is collected.

Mark phase
In this phase, all the live objects are marked. Because unreachable objects cannot be
identified singly, all the reachable objects must be identified. Therefore, everything
else must be garbage. The process of marking all reachable objects is also known as
tracing.

The mark phase uses:

v A pool of structures called work packets. Each work packet contains a mark stack.
A mark stack contains references to live objects that have not yet been traced.
Each marking thread refers to two work packets; an input packet from which
references are popped and an output packet to which unmarked objects that
have just been discovered are pushed. When the input packet becomes empty, it
is added to a list of empty packets and replaced by a non-empty packet. When
the output packet becomes full it is added to a list of non-empty packets and
replaced by a packet from the empty list.
v A bit vector called the mark bit array whose bits identify the objects that are
reachable and have been visited. The mark bit array contains one bit for each 4

10 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Mark phase

bytes of heap space on 31- and 32-bit architectures and one bit for each 8 bytes
of heap space on 64-bit architectures. The bit that corresponds to the start
address for each reachable object is set when it is first visited.

The first stage of tracing is the identification of root objects. The active state of the
JVM is made up of the saved registers for each thread, the set of stacks that
represent the threads, the statics that are in Java classes, and the set of local and
global JNI references. All functions that are invoked in the JVM itself cause a frame
on the C stack. This frame might contain references to objects as a result of either
an assignment to a local variable, or a parameter that is sent from the caller. All
these references are treated equally by the tracing routines.

All the mark bits for all root objects are set and references to the roots pushed to
the mark stack. After this is done, tracing can proceed by iteratively popping a
reference off the mark stack and then scanning the referenced object for references
to other objects. If there are any references to unmarked objects, that is, mark bit
off, the object is marked by setting the appropriate bit in the mark bit array and
the reference is pushed to the mark stack. This process continues until all the work
packets are on the empty list, at which point all the reachable (live) objects, have
been identified.

Mark stack overflow

Because the set of work packets has a finite size, it can overflow. If an overflow
occurs, the Garbage Collector empties one of the work packets by popping its
references one at a time, and chaining the referenced objects off their owning class
by using the class pointer slot in the object header. All classes with overflow
objects are also chained together. Tracing can then continue as before. If a further
mark stack overflow occurs, more packets are emptied in the same way.

When a marking thread asks for a new non-empty packet and all work packets are
empty, the GC checks the list of overflow classes. If the list is not empty, the GC
traverses this list and repopulates a work packet with the references to the objects
on the overflow lists. These packets are then processed as described above. Tracing
is complete when all the work packets are empty and the overflow list is empty.

Parallel mark
The goal of Parallel Mark is to not degrade Mark performance on a uniprocessor,
and to increase typical Mark performance on a multiprocessor system.

Object marking is increased through the addition of helper threads that share the
use of the pool of work packets; for example, full output packets that are returned
to the pool by one thread can be picked up as new input packets by another
thread. Parallel Mark still requires the participation of one application thread that
is used as the master coordinating agent. This thread performs very much as it
always did, but the helper threads assist both in the identification of the root
pointers for the collection and in the tracing of these roots. Mark bits are updated
by using atomic primitives that require no additional lock.

A platform with N processors also has N-1 new helper threads, that work with the
master thread to complete the marking phase of garbage collection. You can
override the default number of threads by using the -Xgcthreads option. If you
specify a value of 1, there will be no helper threads. The -Xgcthreads option
accepts any value greater than 0, but you gain little by setting it to more than N-1.

Chapter 2. Understanding the Garbage Collector 11

Concurrent mark

Concurrent mark
Concurrent mark gives reduced and consistent garbage collection pause times
when heap sizes increase. It starts a concurrent marking phase before the heap is
full. In the concurrent phase, the Garbage Collector scans the roots by asking each
thread to scan its own stack. These roots are then used to trace live objects
concurrently. Tracing is done by a low-priority background thread and by each
application thread when it does a heap lock allocation.

While the Garbage Collector is marking live objects concurrently with application
threads running, it has to record any changes to objects that are already traced. It
uses a write barrier that is activated every time a reference in an object is updated.
The write barrier flags when an object reference update has occurred, to force a
rescan of part of the heap. The heap is divided into 512-byte sections and each
section is allocated a byte in the card table. Whenever a reference to an object is
updated, the card that corresponds to the start address of the object that has been
updated with the new object reference is marked with 0x01. A byte is used instead
of a bit for two reasons: a write to a byte is quicker than a bit change, and the
other bits are reserved for future use. An STW collection is started when one of the
following occurs:
v An allocation failure
v A System.gc
v Concurrent mark completes all the marking that it can do

The Garbage Collector tries to start the concurrent mark phase so that it completes
at the same time as the heap is exhausted. The Garbage Collector does this by
constant tuning of the parameters that govern the concurrent mark time. In the
STW phase, the Garbage Collector scans all roots, uses the marked cards to see
what must be retraced, then sweeps as normal. It is guaranteed that all objects that
were unreachable at the start of the concurrent phase are collected. It is not
guaranteed that objects that become unreachable during the concurrent phase are
collected.

Reduced and consistent pause times are the benefits of concurrent mark, but they
come at a cost. Application threads must do some tracing when they are requesting
a heap lock allocation. The overhead varies depending on how much idle CPU
time is available for the background thread. Also, the write barrier has an
overhead.

This parameter enables concurrent mark:

-Xgcpolicy:<optthruput | optavgpause | gencon>

Setting -Xgcpolicy to optthruput disables concurrent mark. If you do not have

pause time problems (as seen by erratic application response times), you get the
best throughput with this option. Optthruput is the default setting. Setting
-Xgcpolicy to optavgpause enables concurrent mark with its default values. If you
are having problems with erratic application response times that are caused by
normal garbage collections, you can reduce those problems at the cost of some
throughput, by using the optavgpause option. The gencon option requests the
combined use of concurrent and generational GC to help minimize the time that is
spent in any garbage collection pause.

Sweep phase
On completion of the mark phase the mark bit vector identifies the location of all
the live objects in the heap. The sweep phase uses this to identify those chunks of
12 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Sweep phase

heap storage that can be reclaimed for future allocations; these chunks are added
to the pool of free space. To avoid filling this free space pool with lots of small
chunks of storage, only chunks of at least a certain size are reclaimed and added to
the free pool. The minimum size for a free chunk is currently defined as 512 bytes
(768 bytes on 64-bit platforms).

A free chunk is identified by examining the mark bit vector looking for sequences
of zeros, which identify possible free space. GC ignores any sequences of zeroes
that correspond to a length less than the minimum free size. When a sequence of
sufficient length is found, the Garbage Collector checks the length of the object at
the start of the sequence to determine the actual amount of free space that can be
reclaimed. If this amount is greater than or equal to the minimum size for a free
chunk, it is reclaimed and added to the free space pool.

The small areas of storage that are not on the freelist are known as ″dark matter″,
and they are recovered when the objects that are next to them become free, or
when the heap is compacted. It is not necessary to free the individual objects in the
free chunk, because it is known that the whole chunk is free storage. When a
chunk is freed, the Garbage Collector has no knowledge of the objects that were in
it.

Parallel bitwise sweep

Parallel Bitwise Sweep improves the sweep time by using available processors. In
Parallel Bitwise Sweep, the Garbage Collector uses the same helper threads that are
used in Parallel Mark, so the default number of helper threads is also the same
and can be changed with the -Xgcthreads<n> option. The heap is divided into
sections of 256KB and each thread (helper or master) takes a section at a time and
scans it, performing a modified bit-wise sweep. The results of this scan are stored
for each section. When all sections have been scanned, the freelist is built.

Compaction phase
When the garbage has been removed from the heap, the Garbage Collector can
consider compacting the resulting set of objects, to remove the spaces that are
between them. The process of compaction is complicated because if any object is
moved, the Garbage Collector must change all the references that exist to it.

The following analogy might help you understand the compaction process. Think
of the heap as a warehouse that is partly full of pieces of furniture of different
sizes. The free space is the gaps between the furniture. The free list contains only
gaps that are above a particular size. Compaction pushes everything in one
direction and closes all the gaps. It starts with the object that is closest to the wall,
and puts that object against the wall. Then it takes the second object in line and
puts that against the first. Then it takes the third and puts it against the second,
and so on. At the end, all the furniture is at one end of the warehouse and all the
free space is at the other.

To keep compaction times to a minimum, the helper threads are used again.

Compaction occurs if any of the following are true and -Xnocompactgc has not
been specified:
v -Xcompactgc has been specified.
v Following the sweep phase, not enough free space is available to satisfy the
allocation request.
v A System.gc() has been requested and the last allocation failure garbage
collection did not compact.

Chapter 2. Understanding the Garbage Collector 13

Compaction phase

v At least half the previously available memory has been consumed by TLH
allocations (ensuring an accurate sample) and the average TLH size falls below
1000 bytes.
v Less than 5% of the active heap is free.
v Less than 128 KB of the active heap is free.

Reference objects
When a reference object is created, it is added to a list of reference objects of the
same type. Instances of SoftReference, WeakReference, and PhantomReference are
created by the user and cannot be changed; they cannot be made to refer to objects
other than the object that they referenced on creation. Each reference results in the
creation of a reference object. Objects whose classes define a finalize method result
in a pointer to that object being placed on a list of objects that require finalization.

During garbage collection, immediately following the mark phase, these lists are
processed in a specific order:
1. Soft
2. Weak
3. Final
4. Phantom

Soft, weak, and phantom reference processing

For each element on a list GC determines if the reference object is eligible for
processing and then if it is eligible for collection.

An element is eligible for processing if it is marked and has a non-null referent

field. The referent is the object the reference object points to. If this is not the case,
the reference object is removed from the reference list, resulting in it being freed
during the sweep phase.

If an element is determined to be eligible for processing, GC must determine if it is

eligible for collection. The first criterion here is simple. Is the referent marked? If it
is marked, the reference object is not eligible for collection and GC moves onto the
next element of the list.

If the referent is not marked, GC has a candidate for collection. At this point the
process differs for each reference type. Soft references are collected if their referent
has not been marked for the previous two garbage collection cycles. If there is a
shortage of available storage, all soft references are cleared. Currently this happens
only on System.gc() calls and if an allocation failure could not be satisfied
following a garbage collection.

Weak references are always collected if their referent is not marked.

Phantom references are collected if their referent is not marked or they are no
longer reachable. When a phantom reference is processed, its referent is marked so
it will persist until the following garbage collection cycle or until the phantom
reference is processed if it is associated with a reference queue. When it is
determined that a reference is eligible for collection, it is either queued to its
associated reference queue or simply removed from the reference list.

Final reference processing

The processing of objects that require finalization is more straightforward. The list
of objects is processed and any element that is not marked is processed by marking
and tracing the object and then creating an entry on the finalizable object list for
14 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Final reference processing

the object. Then GC removes the element on the unfinalized object list. The final
method for the object is then run at an undetermined point in the future by the
reference handler thread.

JNI weak reference

JNI weak references provide the same capability as WeakReference objects do, but
the processing is very different. A JNI routine can create a JNI Weak reference to
an object and later delete that reference. The Garbage Collector clears any weak
reference where the referent is unmarked, but no equivalent of the queuing
mechanism exists. Note that failure to delete a JNI Weak reference causes a
memory leak in the table and performance problems. This is also true for JNI
global references. The processing of JNI weak references is handled last in the
reference handling process. The result is that a JNI weak reference can exist for an
object that has already been finalized and had a phantom reference queued and
processed.

Heap expansion
Heap expansion occurs after garbage collection while exclusive access of the virtual
machine is still held. The active part of the heap is expanded up to the maximum
if one of the following is true:
v The Garbage Collector did not free enough storage to satisfy the allocation
request.
v Free space is less than the minimum free space, which you can set by using the
-Xminf parameter. The default is 30%.
v More than 13% of the time is being spent in garbage collection.

The amount to expand the heap is calculated as follows:

v If the heap is being expanded because less than -Xminf (default 30%) free space
is available, the Garbage Collector calculates how much the heap needs to
expand to get -Xminf free space.
If this is greater than the maximum expansion amount, which you can set with
the -Xmaxe parameter (default of 0, which means no maximum expansion), the
calculation is reduced to -Xmaxe.
If this is less than the minimum expansion amount, which you can set with the
-Xmine parameter (default of 1 MB), it is increased to -Xmine.
v If the heap is expanding and the JVM is spending more than 13% for any other
reason, the Garbage Collector calculates how much expansion is needed to
expand the heap by 17% free space. This is adjusted as above, depending on
-Xmaxe and -Xmine.
v Finally, the Garbage Collector ensures that the heap is expanded by at least the
allocation request if garbage collection did not free enough storage.

All calculated expansion amounts are rounded up to a 256-byte boundary (512

bytes if Concurrent mark is used) on 32-bit architecture, or a 1024 bytes boundary
on 64-bit architecture.

Heap shrinkage
Heap shrinkage occurs after garbage collection while exclusive access of the virtual
machine is still held. Shrinkage does not occur if any of the following are true:
v The Garbage Collector did not free enough space to satisfy the allocation
request.

Chapter 2. Understanding the Garbage Collector 15

Heap expansion

v The maximum free space, which can be set by the -Xmaxf parameter (default is
60%), is set to 100%.
v The heap has been expanded in the last three garbage collections.
v This is a System.gc() and the amount of free space at the beginning of the
garbage collection was less than -Xminf (default is 30%) of the live part of the
heap.
v If none of the above is true and more than -Xmaxf free space exists, the Garbage
Collector must calculate how much to shrink the heap to get it to -Xmaxf free
space, without going below the initial (-Xms) value. This figure is rounded
down to a 256-byte boundary (512 bytes if Concurrent mark is used) on 32-bit
architecture, or a 1024 bytes boundary on 64-bit architecture.

A compaction occurs before the shrink if all the following are true:
v A compaction was not done on this garbage collection cycle.
v No free chunk is at the end of the heap, or the size of the free chunk that is at
the end of the heap is less than 10% of the required shrinkage amount.
v The Garbage Collector did not shrink and compact on the last garbage collection
cycle.

Note that, on initialization, the JVM allocates the whole heap in a single
contiguous area of virtual storage. The amount that is allocated is determined by
the setting of the -Xmx parameter. No virtual space from the heap is ever freed
back to the native operating system. When the heap shrinks, it shrinks inside the
original virtual space.

Whether any physical memory is released depends on the ability of the native
operating system. If it supports paging; that is, the ability of the native operating
system to commit and decommit physical storage to the virtual storage, the
Garbage Collector uses this function. In this case, physical memory can be
decommitted on a heap shrinkage.

To summarize. You never see the amount of virtual memory that is used by the
JVM decrease. You might see physical memory free size increase after a heap
shrinkage. The native operating system determines what it does with decommitted
pages.

Also note that, where paging is supported, the Garbage Collector allocates physical
memory to the initial heap to the amount that is specified by the -Xms parameter.
Additional memory is committed as the heap grows.

How to do heap sizing

This section describes how to do heap sizing to suit your requirements. Generally:
v Do not start with a minimum heap size that is the same as the maximum heap
size.
v Use verbosegc to tailor the minimum and maximum settings.
v Investigate the use of fine-tuning options.

Initial and maximum heap sizes

When you have established the maximum heap size that you need, you might
want to set the minimum heap size to the same value; for example, -Xms 512M
-Xmx 512M. Using the same values is not usually a good idea, because it delays the
start of garbage collection until the heap is full. The first time that the Garbage

16 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 How to do heap sizing

Collector runs, therefore, becomes a very expensive operation. Also, the heap is
most likely to be very fragmented when a need to do a heap compaction occurs.
Again, this is a very expensive operation. The recommendation is to start your
application with the minimum heap size that it needs. When it starts up, the
Garbage Collector will run often and, because the heap is small, efficiently.

The Garbage Collector takes these steps:

1. If the Garbage Collector finds enough garbage, it exits.
If it cannot find enough garbage, it goes to the next step.
2. The Garbage Collector runs compaction.
If it cannot find enough garbage, it goes to the next step.
3. The Garbage Collector expands the heap.

Therefore, an application normally runs until the heap is full. Then, successive
garbage collection cycles recover garbage. When the heap is full of live objects, the
Garbage Collector compacts the heap. If and when the heap is full of live objects
and cannot be compacted, the Garbage Collector expands the heap size.

From the above description, you can see that the Garbage Collector compacts the
heap as the needs of the application rise, so that as the heap expands, it expands
with a set of compacted objects in the bottom of the original heap. This is an
efficient way to manage the heap, because compaction runs on the
smallest-possible heap size at the time that compaction is found to be necessary.
Compaction is performed with the minimum heap sizes as the heap grows. Some
evidence exists that an application’s initial set of objects tends to be the key or root
set, so that compacting them early frees the remainder of the heap for more
short-lived objects.

Eventually, the JVM has the heap at maximum size with all long-lived objects
compacted at the bottom of the heap. The compaction occurred when compaction
was in its least expensive phase. The overheads of expanding the heap are almost
trivial compared to the cost of collecting and compacting a very large fragmented
heap.

Using verbosegc
The verbosegc output is fully described in Chapter 28, “Garbage Collector
diagnostics,” on page 241. Switch on verbosegc and run up the application with no
load. Check the heap size at this stage. This provides a rough guide to the start
size of the heap (-Xms option) that is needed. If this value is much larger than the
defaults (see Appendix G, “Default settings for the JVM,” on page 339), think about
reducing this value a little to get efficient and rapid compaction up to this value,
as described in “Initial and maximum heap sizes” on page 16.

By running an application under stress, you can determine a maximum heap size.
Use this to set your max heap (-Xmx) value.

Using fine tuning options

Refer to the description of the following command-line parameters and consider
applying to fine tune the way the heap is managed:

-Xmaxe
-Xmine
-Xmaxf
-Xminf

Chapter 2. Understanding the Garbage Collector 17

How to do heap sizing

These are described in “Heap expansion” on page 15 and “Heap shrinkage” on

page 15.

Interaction of the Garbage Collector with applications

This interaction can be expressed as a contract between the Garbage Collector and
an application. The Garbage Collector honors this contract:
1. The Garbage Collector will collect unused objects.
2. The Garbage Collector will not collect live objects.
3. The Garbage Collector will stop all threads when it is running.
4. Garbage Collector invocation:
a. The Garbage Collector will not run itself except when a memory fault
occurs.
b. The Garbage Collector will honor manual invocations.
5. The Garbage Collector will collect garbage at its own convenience, sequence,
and timing, subject to clause 4b.
6. The Garbage Collector will honor all command-line variables, environment
variables, or both.
7. Finalizers:
a. Are not run in any particular sequence
b. Are not run at any particular time
c. Are not guaranteed to run at all
d. Will run asynchronously to the Garbage Collector

This contract is used in the following section for some advice.

Note clause 4b. The specification says that a manual invocation of the Garbage
Collector (for example, through the System.gc() call) suggests that a garbage
collection cycle might be run. In fact, the call is interpreted as “Do a full garbage
collection scan unless a garbage collection cycle is already executing”.

How to coexist with the Garbage Collector

Root set
Consider the root set. It is mainly a pseudo-random set of references from what
happened to be in the stacks and registers of the JVM threads at the time that the
Garbage Collector was invoked. This means that the graph of reachable objects that
the Garbage Collector constructs in any given cycle is nearly always different from
that traced in another cycle. (See clause 5). This has significant consequences for
finalizers (clause 7), which are described more fully in “Finalizers” on page 19.

Thread local heap

The heap is subject to concurrent access by all the threads that are running in the
JVM. Therefore, it must be protected by a resource lock so that one thread can
complete updates to the heap before another thread is allowed in. Access to the
heap is therefore single-threaded. However, the Garbage Collector also maintains
areas of the heap as thread caches or thread local heap (TLH). These TLHs are
areas of the heap that are allocated as a single large object, marked noncollectable,
and allocated to a thread. The thread can now suballocate from the TLH, objects
that are below a defined size. No heap lock is needed, so allocation is very fast and

18 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 How to coexist with the Garbage Collector

efficient. When a cache becomes full, a thread returns the TLH to the main heap
and grabs another chunk for a new cache.

A TLH is not subject to a garbage collection cycle; it is a reference that is dedicated

to a thread.

Bug reports
Attempts to predict the behavior of the Garbage Collector are frequent underlying
causes of bug reports. An example of a regular bug report to Java service of the
hello-world variety is one in which a simple programme allocates some object or
objects, clears references to these objects, then initiates a garbage collection cycle.
The objects are not seen as collected, usually because the application has attached a
finalizer that reports when it is run.

It should be clear from the contract and the unpredictable nature of the Garbage
Collector that more than one valid reason exists for this:
v An object reference exists in the thread stack or registers, and the objects are
retained garbage.
v The Garbage Collector has not chosen to run a finalizer cycle at this time.

See clause 1 on page 18. True garbage is always found eventually, but it is not
possible to predict when (clause 5 on page 18).

Finalizers
The Java service team strongly recommends that applications avoid the use of
finalizers as far as possible. The JVM specification states that finalizers should be
used as an emergency clear-up of, for example, hardware resources. The service
team recommends that this should be the only use of finalizers. They should not
be used to clean up Java software resources or for closedown processing of
transactions.

The reasons for this recommendation are partly in the nature of finalizers and how
they are permanently linked to garbage collection, and partly in the contract that is
described in “Interaction of the Garbage Collector with applications” on page 18.
These topics are examine more closely in the following sections.

Nature of finalizers
The JVM specification says nothing about finalizers, except that they are final in
nature. Nothing states when, how, or even whether a finalizer is run. The only rule
is that if and when it is run, it is final.

Final, in terms of a finalizer, means that the class object is known not to be in use
any more. Clearly, this can happen only when the object is not reachable. Only the
Garbage Collector can determine this. Therefore, when the Garbage Collector runs,
it determines which are the unreachable objects that have a finalizer method.
Normally, such objects would be collected, and the Garbage Collector would be
able to satisfy the memory allocation fault. Finalized garbage, however, must have
its finalizer run before it can be collected. Therefore, no finalized garbage can be
collected in the cycle that actually finds it. Finalizers therefore make a garbage
collection cycle longer (the cycle has to detect and process the objects) and less
productive. Finalizers are an overhead on garbage collection. Because garbage
collection is a stop-the-world operation, it makes sense to reduce this overhead as
far as possible.

Chapter 2. Understanding the Garbage Collector 19

How to coexist with the Garbage Collector

Note that the Garbage Collector cannot run finalizers itself when it finds them.
This is because a finalizer might run an operation that takes a long time, and the
Garbage Collector cannot risk locking out the application while this operation is
running. So finalizers must be collected into a separate thread for processing. This
task adds more overhead into the garbage collection cycle.

Finalizers and the garbage collection contract

Garbage Collector contract clause 7 on page 18, which shows the nonpredictable
behavior of the Garbage Collector, has particular significant results:
v Because the graph of objects that the Garbage Collector finds is basically
random, the sequence in which finalized objects are located has no relationship
to the sequence in which they were created nor to the sequence in which their
objects became garbage (contract subclause 7a on page 18). Similarly, the
sequence in which finalizers are run is also random.
v Because the Garbage Collector has no knowledge of what is in a finalizer, or
how many finalizers exist, it tries to satisfy an allocation without needing to
process finalizers. If a garbage collection cycle cannot produce enough normal
garbage, it might decide to process finalized objects. So it is not possible to
predict when a finalizer is run (contract subclause 7b on page 18).
v Because a finalized object might be retained garbage, it is possible that a
finalizer might not run at all (contract subclause 7c on page 18).

How finalizers are run

If and when the Garbage Collector decides to process unreachable finalized objects,
those objects are placed onto a queue that is input to a separate finalizer thread.
When the Garbage Collector has ended and the threads are unblocked, this thread
starts to perform its function. It runs as a high-priority thread and runs down the
queue, running the finalizer of each object in turn. When the finalizer has run, the
finalizer thread marks the object as collectable and the object is (probably) collected
in the next garbage collection cycle. See contract subclause 7d on page 18. Of
course, if running with a large heap, the next garbage collection cycle might not
happen for quite a long time.

Summary
v Finalizers are an expensive overhead.
v Finalizers are not dependable.

The Java service team would recommend that :

v Finalizers are not used for process control
v Finalizers are not used for tidying Java resources
v Finalizers are not used at all as far as possible

For tidying Java resources, think about the use of a clean up routine. When you
have finished with an object, call the routine to null out all references, deregister
listeners, clear out hash tables, and so on. This is far more efficient than using a
finalizer and has the useful side-benefit of speeding up garbage collection. The
Garbage Collector does not have so many object references to chase in the next
garbage collection cycle.

Manual invocation
The Garbage Collector contract subclause 4b on page 18 notes that the Garbage
Collector always honors a manual invocation; for example, through the System.gc
() call. This call nearly always invokes a garbage collection cycle, which is
expensive.

20 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 How to coexist with the Garbage Collector

The Java service team recommend that this call is not used, or if it is, it is
enveloped in conditional statements that block its use in an application runtime
environment. The Garbage Collector is carefully adjusted to deliver maximum
performance to the JVM. Forcing it to run severely degrades JVM performance

From the previous sections, you can see that it is pointless trying to force the
Garbage Collector to do something predictable, such as collecting your new
garbage or running a finalizer. It might happen; it might not. Let the Garbage
Collector run in the parameters that an application selects at start-up time. This
method nearly always produces best performance.

Several actual customer applications have been turned from unacceptable to

acceptable performance simply by blocking out manual invocations of the Garbage
Collector. One actual enterprise application was found to have more than four
hundred System.gc() calls.

Summary
Do not try to control the Garbage Collector or to predict what will happen in a
given garbage collection cycle. You cannot do it. This unpredictability is handled,
and the Garbage Collector is designed to run well and efficiently inside these
conditions. Set up the initial conditions that you want and let the Garbage
Collector run. It will honor the contract (described in “Interaction of the Garbage
Collector with applications” on page 18), which is within the JVM specification.

Frequently asked questions about the Garbage Collector

What are the default heap sizes?
See “Heap size” on page 8.
If I don’t specify -Xmx and -Xms, what values will Java use?
See Appendix G, “Default settings for the JVM,” on page 339.
What are default values for the native stack (-Xss) and Java stack (-Xoss)?
The Native stack size is machine-dependent, because it is based on the
platform’s C stack usage. The Java stack size is 400*1024
What is the difference between the GC policies optavgpause, optthruput and
gencon?
optthruput disables concurrent mark. If you do not have pause time problems
(indicated by erratic application response times), you should get the best
throughput with this option.
optavgpause enables concurrent mark. If you have problems with erratic
application response times in garbage collection, you can alleviate them at the
cost of some throughput when running with this option.
gencon requests the combined use of concurrent and generational GC to help
minimize the time that is spent in any garbage collection pause.
What is the default GC mode (optavgpause, optthruput or gencon)?
optthruput - that is, generational collector and concurrent marking are off.
How many GC helper threads are spawned? What is their work?
A platform with n processors will have n-1 helper threads. These threads work
along with the main GC thread during:
v Parallel mark phase
v Parallel bitwise sweep phase
v Parallel compaction phase

Chapter 2. Understanding the Garbage Collector 21

Frequently asked questions about the Garbage Collector

You can control the number of GC helper threads with the -Xgcthreads option.
Passing the -Xgcthreads1 option to Java results in no helper threads at all.

You gain little by setting -Xgcthreads to more than n-1 other than possibly
alleviating mark-stack overflows, if you suffer from them.
How can I prevent Java heap fragmentation?
Note that the following suggestions might not help avoid fragmentation in all
cases.
v Start with a small heap. Set -Xms far lower than -Xmx. It might be
appropriate to allow -Xms to default, because the default is a low value.
v Increase the maximum heap size, -Xmx.
v If the application uses JNI, make sure JNI references are properly cleared. All
objects being referenced by JNI are pinned and not moved during
compaction, contributing significantly to heap fragmentation.
What is Mark Stack Overflow? Why is MSO bad for performance?
Mark stacks are used for tracing all object reference chains from the roots. Each
such reference that is found is pushed onto the mark stack so that it can be
traced later. Mark stacks are of fixed size, so they can overflow. This situation
is called Mark Stack Overflow (MSO). The algorithms to handle this situation
are very expensive in processing terms, and so MSO is a big hit on GC
performance.
How can I prevent Mark Stack Overflow?
There is nothing an application can do to avoid MSO, except to reduce the
number of objects it allocates. The following suggestions are not guaranteed to
avoid MSO:
v Increase the number of GC helper threads using -Xgcthreads command-line
option
v Decrease the size of the Java heap using the -Xmx setting.
v Use a small initial value for the heap or use the default.
When and why does the Java heap expand?
The JVM starts with a small default Java heap, and it expands the heap based
on an application’s allocation requests until it reaches the value specified by
-Xmx. Expansion occurs after GC if GC is unable to free enough heap storage
for an allocation request, or if the JVM determines that expanding the heap is
required for better performance.
When does the Java heap shrink?
Heap shrinkage occurs when GC determines that there is a lot of free heap
storage, and releasing some heap memory is beneficial for system performance.
Heap shrinkage occurs after GC, but when all the threads are still suspended.
Does the IBM GC guarantee that it will clear all the unreachable objects?
The IBM GC guarantees only that all the objects that were not reachable at the
beginning of the mark phase will be collected. While running concurrently, our
GC guarantees only that all the objects that were unreachable when concurrent
mark began will be collected. Some objects might become unreachable during
concurrent mark, but they are not guaranteed to be collected.
I am getting an OutOfMemoryError. Does this mean that the Java heap is
exhausted?
Not necessarily. Sometimes the Java heap has free space but an
OutOfMemoryError can occur. The error could occur because of
v Shortage of memory for other operations of the JVM.

22 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Frequently asked questions about the Garbage Collector

v Some other memory allocation failing. The JVM throws an OutOfMemoryError

in such situations.
v Excessive memory allocation in other parts of the application, unrelated to
the JVM, if the JVM is just a part of the process, rather than the entire
process (JVM through JNI, for instance).
How can I confirm if the OutOfMemoryError was caused by the Java heap
becoming exhausted?
Run with the -verbosegc option. VerboseGC will show messages such as
Insufficient heap space to satisfy allocation request when the Java heap
is exhausted
When I see an OutOfMemoryError, does that mean that the Java program will
exit?
Not always. Java programs can catch the exception thrown when OutOfMemory
occurs, and (possibly after freeing up some of the allocated objects) continue to
run.
How do I figure out if the Java heap is fragmented?
When you see (from verboseGC) that the Java heap has a lot of free space, but
the allocation request still fails, it usually points to a fragmented heap.
In verboseGC output, sometimes I see more than one GC for one allocation
failure. Why?
You see this when GC decides to clear all soft references. The GC is called once
to do the regular garbage collection, and might run again to clear soft
references. So you might see more than one GC cycle for one allocation failure.

Chapter 2. Understanding the Garbage Collector 23

Frequently asked questions about the Garbage Collector

24 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 3. Understanding the class loader
The Java 2 JVM introduced a new class loading mechanism with a
parent-delegation model. The parent-delegation architecture to class loading was
implemented to aid security and to help programmers to write custom class
loaders.

The class loader loads, verifies, prepares and resolves, and initializes a class from a
JVM class file.
v Loading involves obtaining the byte array representing the Java class file.
v Verification of a JVM class file is the process of checking that the class file is
structurally well-formed and then inspecting the class file contents to ensure that
the code does not attempt to perform operations that are not permitted.
v Preparation involves the allocation and default initialization of storage space for
static class fields. Preparation also creates method tables, which speed up virtual
method calls, and object templates, which speed up object creation.
v Initialization involves the execution of the class’s class initialization method, if
defined, at which time static class fields are initialized to their user-defined
initial values (if specified).

Symbolic references within a JVM class file, such as to classes or object fields that
reference a field’s value, are resolved at runtime to direct references only. This
resolution might occur either:
v After preparation but before initialization
v Or, more typically, at some point following initialization, but before the first
reference to that symbol.
The delay is generally to increase execution speed. Not all symbols in a class file
are referenced during execution. So, by delaying resolution, fewer symbols might
have to be resolved, giving you less runtime overhead. Additionally, the cost of
resolution is gradually reduced over the total execution time.

The parent-delegation model

The delegation model requires that any request for a class loader to load a given
class is first delegated to its parent class loader before the requested class loader
tries to load the class itself. The parent class loader, in turn, goes through the same
process of asking its parent. This chain of delegation continues through to the
bootstrap class loader (also known as the primordial or system class loader). If a
class loader’s parent can load a given class, it returns that class. Otherwise, the
class loader attempts to load the class itself.

The JVM has three class loaders, each possessing a different scope from which it
can load classes. As you descend the hierarchy, the scope of available class
repositories widens, and normally the repositories are less trusted:
Bootstrap
|
Extensions
|
Application

© Copyright IBM Corp. 2003, 2006 25

Parent-delegation model

At the top of the hierarchy is the bootstrap class loader. This class loader is
responsible for loading only the classes that are from the core Java API. These are
the most trusted classes and are used to bootstrap the JVM.

The extensions class loader can load classes that are standard extensions packages
in the extensions directory.

The application class loader can load classes from the local file system, and will
load files from the CLASSPATH. The application class loader is the parent of any
custom class loader or hierarchy of custom class loaders.

Because class loading is always delegated first to the parent of the class loading
hierarchy, the most trusted repository (the core API) is checked first, followed by
the standard extensions, then the local files that are on the class path. Finally,
classes that are located in any repository that your own class loader can access, are
accessible. This system prevents code from less-trusted sources from replacing
trusted core API classes by assuming the same name as part of the core API.

Name spaces and the runtime package

Loaded classes are identified by both the class name and the class loader that
loaded it. This separates loaded classes into name spaces that the class loader
identifies.

A name space is a set of class names that are loaded by a specific class loader.
When an entry for a class has been added into a name space, it is impossible to
load another class of the same name into that name space. Multiple copies of any
given class can be loaded because a name space is created for each class loader.

Name spaces cause classes to be segregated by class loader, thereby preventing

less-trusted code loaded from the application or custom class loaders from
interacting directly with more trusted classes. For example, the core API is loaded
by the bootstrap class loader, unless a mechanism is specifically provided to allow
them to interact. This prevents possibly malicious code from having guaranteed
access to all the other classes.

You can grant special access privileges between classes that are in the same
package by the use of package or protected access. This gives access rights
between classes of the same package, but only if they were loaded by the same
class loader. This stops code from an untrusted source trying to insert a class into a
trusted package. As discussed above, the delegation model prevents the possibility
of replacing a trusted class with a class of the same name from an untrusted
source. The use of name spaces prevents the possibility of using the special access
privileges that are given to classes of the same package to insert code into a
trusted package.

Why write your own class loader?

The three main reasons for wanting to create your own class loader are:
v To allow class loading from alternative repositories.
This is the most common case, in which an application developer might want to
load classes from other locations, for example, over a network connection.
v To partition user code.
This case is less frequently used by application developers, but widely used in
servlet engines.

26 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Why write your own class loader?

v To allow the unloading of classes.

This case is useful if the application creates large numbers of classes that are
used for only a finite period. Because a class loader maintains a cache of the
classes that it has loaded, these classes cannot be unloaded until the class loader
itself has been dereferenced. For this reason, system and extension classes are
never unloaded, but application classes can be unloaded when their classloader
is.

How to write your own class loader

Under the Java 1 class loading system, it was a requirement that any custom class
loader must subclass java.lang.ClassLoader and override the abstract loadClass()
method that was in the ClassLoader. The loadClass() method had to meet several
requirements so that it could work effectively with the JVM’s class loading
mechanism, such as:
v Checking whether the class has previously been loaded
v Checking whether the class had been loaded by the system class loader
v Loading the class
v Defining the class
v Resolving the class
v Returning the class to the caller

The Java 2 class loading system has simplified the process for creating custom class
loaders. The ClassLoader class was given a new constructor that takes the parent
class loader as a parameter. This parent class loader can be either the application
class loader, or another user-defined class loader. This allows any user-defined
class loader to be contained easily into the delegation model.

Under the delegation model, the loadClass() method is no longer abstract, and as
such does not need to be overridden. The loadClass() method handles the
delegation class loader mechanism and should not be overridden, although it is
possible to do so, so that Java 1 style ClassLoaders can run on a Java 2 JVM.

Because the delegation code is handled in loadClass(), in addition to the other

requirements that were made of Java 1 custom class loaders, custom class loaders
should override only the new findClass() method, in which the code to access the
new class repository should be placed. The findClass() method is responsible only
for loading the class bytes and returning a defined class. The method defineClass()
can be used to convert class bytes into a Java class:
class NetworkClassLoader extends ClassLoader {
String host;
int port;

public Class findClass(String name) {

byte[] b = loadClassData(name);
return defineClass(name, b, 0, b.length);
}
private byte[] loadClassData(String name) {
// load the class data from the connection
}
}

Chapter 3. Understanding the class loader 27

How to write your own class loader

28 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 4. Understanding the JIT
The Just-In-Time compiler (JIT) is not part of the JVM, but is a basic component of
the SDK. This chapter summarizes the relationship between the JVM and the JIT,
and gives a short description of how the JIT works.

JIT overview
Java is an interpreted language, so it has a Write Once Run Anywhere (WORA)
capability. The Java compiler reads Java source files and outputs strings of
bytecodes, which are platform-neutral pseudo-machine code. At runtime, the JVM
reads these bytecodes, interprets the semantics of each individual bytecode, and
performs the appropriate computation. This means that a JVM that is interpreting
bytecodes has a slower performance than a native application consisting of
machine code generated by a native compiler.

The JIT is therefore important because it helps improve the performance of Java
applications. The JIT comes into use whenever a Java method is called; it compiles
the bytecodes of that method into native machine code, compiling it ″just in time″
to execute. After a method is compiled, the JVM calls that the compiled code of
that method directly instead of interpreting it. However, when the JVM starts,
thousands of methods are executed. There is a significant overhead on all the
methods because of the time it takes the JIT to run and compile them. This means
that if you run without the JIT, the JVM starts up quickly but runs slowly.
Conversely, if you run with the JIT, the JVM starts up slowly, then runs quickly. In
some applications, you might find that it takes longer to start the JVM than to run
the application itself.

In practice, not all methods are compiled the first time they are called. For each
method, the JVM maintains a call count, which is incremented every time the
method is invoked. The JVM interprets a method until its call count exceeds a JIT
threshold. Therefore, often-used methods are compiled soon after the JVM has
started; conversely, the methods that are used less often are compiled much later
or perhaps not at all. In fact, the compilation of methods is spread out over the life
of the JVM. This way, the JVM starts up quickly, but the program does not suffer
performance loss because methods are compiled to native code when their call
counts reach the JIT threshold. The threshold is carefully selected to obtain the
maximum balance between startup times and run-time performance.

After a method is compiled, its call count is reset to zero; subsequent calls to the
method continue to increment its count. When the call count of a method reaches a
JIT recompilation threshold, it is compiled a second time, this time applying a larger
selection of optimizations than on the previous compilation (because the method
has proven to be a significant part of the whole program). The recompilation
process is iterative; the call count of a recompiled method is reset again and, as it
reaches succeeding thresholds, triggers recompilations at increasing optimization
levels. Thus, the busiest methods of a Java program are always optimized most
aggressively, maximizing the performance benefits of using the JIT. The JIT can
also measure operational data at run time, and use that data to improve the quality
of further recompilations.

© Copyright IBM Corp. 2003, 2006 29

How the JIT optimizes code

How the JIT optimizes code

When a method is chosen for compilation, the JVM feeds its bytecodes to the JIT.
The JIT needs to understand the semantics and syntax of the bytecodes before it
can compile the method correctly. To help the JIT analyze the method, its
bytecodes are first reformulated in an internal representation called trees, which
resembles machine code more closely than bytecodes. Analysis and optimizations
are then performed on the trees of the method. At the end, the trees are translated
into native code. This chapter does not contain much detail, but provides a
summary of the phases of JIT compilation. For more information, see Chapter 27,
“JIT problem determination,” on page 237.

The compilation consists of the following phases:

1. Inlining
2. Local optimizations
3. Control flow optimizations
4. Global optimizations
5. Native code generation
All phases except native code generation are cross-platform code.

1) Inlining
Inlining is the process by which the trees of smaller methods are merged, or
″inlined″, into the trees of their callers. This speeds up frequently executed method
calls. Two inlining algorithms with different levels of aggressiveness are used,
depending on the current optimization level. Optimizations performed in this
phase include:
v Trivial inlining
v Call graph inlining
v Tail recursion elimination
v Virtual call guard optimizations

2) Local optimizations
Local optimizations analyze and improve a small section of the code at a time.
Many local optimizations implement tried and tested techniques used in classic
static compilers. The optimizations include:
v Local data flow analysis and optimizations
v Register usage optimization
v Simplifications of Java idioms
These techniques are applied repeatedly, especially after global optimizations,
which might have pointed out more opportunities for improvement.

3) Control flow optimizations

The following optimizations analyze the flow of control within a method (or
specific sections of it) and rearrange code paths to improve their efficiency:
v Code reordering, splitting and removal
v Loop reduction and inversion
v Loop striding and loop-invariant code motion
v Loop unrolling and peeling
v Loop versioning and specialization

30 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 How the JIT optimizes code

v Exception-directed optimization
v Switch analysis

4) Global optimizations
Global optimizations work on the entire method at once. They are more
″expensive″, requiring larger amounts of compilation time, but can provide a great
increase in performance:
v Global data flow analyses and optimizations
v Partial redundancy elimination
v Escape analysis
v GC and memory allocation optimizations
v Synchronization optimizations

5) Native code generation

The process of native code generation varies, depending on the platform
architecture. Native code generation is generally divided between x86 architectures
and RISC-type architectures (PowerPC is an example of the latter). During this
phase of the compilation, the trees of a method are translated into machine code
instructions; some small optimizations are performed according to architecture
characteristics. The compiled code is placed into a part of the JVM process space
called the code cache; the location of the method within the code cache is recorded,
so that future calls to it will invoke the compiled code. At any given time, the JVM
process consists of the JVM executables and a set of JIT-compiled code that is
linked dynamically to the bytecode interpreter in the JVM.

Therefore, if a Java program crashes or hangs in the JVM process space, but
outside the range of the JVM executable code in that process, the problem is likely
to be within the code cache.

Frequently asked questions about the JIT

Can I disable the JIT?
Yes. Set the appropriate command-line parameter (see Appendix F,
“Command-line options,” on page 329). Alternatively, delete or rename the JIT
library, which is located with the JVM executables and called j9jit22.dll (on
Windows) or libj9jit22.so (on other platforms).
Can I use another vendor’s JIT?
No.
Can I use any version of the JIT with the JVM?
No. The two are tightly coupled. You must use the version of the JIT that
comes with the JVM package that you use.
Can the JIT ″decompile″ methods?
That is, can compiled code be canceled? No.
Can I control the JIT compilation?
Yes. See Chapter 27, “JIT problem determination,” on page 237. Advanced
diagnostics are available to IBM engineers.
Can I dynamically control the JIT?
No. You can set JIT parameters only at JVM startup time. The JIT can be
started up only at the same time as the JVM.

Chapter 4. Understanding the JIT 31

Frequently asked questions about the JIT

32 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 5. Understanding the ORB
This chapter describes the Object Request Broker (ORB). The topics are:
v “CORBA”
v “RMI and RMI-IIOP”
v “Java IDL or RMI-IIOP?” on page 34
v “RMI-IIOP limitations” on page 34
v “Further reading” on page 34
v “Examples of client–server applications” on page 34
v “Using the ORB” on page 40
v “How the ORB works” on page 43
v “Features of the ORB” on page 49

CORBA
Common Object Request Broker Architecture (CORBA) is an open,
vendor-independent specification for distributed computing. It is published by the
Object Management Group (OMG). Using the Internet Inter-ORB Protocol (IIOP), it
allows objects on different architectures, operating systems, and networks to
interoperate. This interoperability is obtained by the use of the Interface Definition
Language (IDL), which specifies the syntax that is used to invoke operations on
objects. IDL is programming-language independent.

Developers define the hierarchy, attributes, and operations of objects in IDL, then
use an IDL compiler (such as IDLJ for Java) to map the definition onto an
implementation in a programming language. The implementation of an object is
encapsulated. Clients of the object can see only its external IDL interface.

OMG have produced specifications for mappings from IDL to many common
programming languages, including C, C++, and Java. Central to the CORBA
specification is the Object Request Broker (ORB). The ORB routes requests from
client to remote object, and responses to their destinations. Java contains an
implementation of the ORB that communicates by using IIOP.

RMI and RMI-IIOP

RMI is Java’s traditional form of remote communication. Basically, it is an
object-oriented version of Remote Procedure Call (RPC). It uses the
nonstandardized Java Remote Method Protocol (JRMP) to communicate between
Java objects. This provides an easy way to distribute objects, but does not allow for
interoperability between programming languages.

RMI-IIOP is an extension of traditional Java RMI that uses the IIOP protocol. This
protocol allows RMI objects to communicate with CORBA objects. Java programs
can therefore interoperate transparently with objects that are written in other
programming languages, provided that those objects are CORBA-compliant.
Objects can still be exported to traditional RMI (JRMP) however, and the two
protocols can communicate.

© Copyright IBM Corp. 2003, 2006 33

RMI and RMI-IIOP

A terminology difference exists between the two protocols. In RMI (JRMP), the
server objects are called skeletons; in RMI-IIOP, they are called ties. Client objects
are called stubs in both protocols.

Java IDL or RMI-IIOP?

RMI-IIOP is the method that is chosen by Java programmers who want to use the
RMI interfaces, but use IIOP as the transport. RMI-IIOP requires that all remote
interfaces are defined as Java RMI interfaces. Java IDL is an alternative solution,
intended for CORBA programmers who want to program in Java to implement
objects that are defined in IDL. The general rule that is suggested by Sun is to use
Java IDL when you are using Java to access existing CORBA resources, and
RMI-IIOP to export RMI resources to CORBA.

RMI-IIOP limitations
In a Java-only application, RMI (JRMP) is more lightweight and efficient than
RMI-IIOP is, but less scalable. Because it has to conform to the CORBA
specification for interoperability, RMI-IIOP is a more complex protocol. The
developing of an RMI-IIOP application is much more similar to CORBA than it is
to RMI (JRMP).

You must take care if you try to deploy an existing CORBA application in a Java
RMI-IIOP environment. An RMI-IIOP client cannot necessarily access every
existing CORBA object. The semantics of CORBA objects that are defined in IDL
are a superset of those of RMI-IIOP objects. That is why the IDL of an existing
CORBA object cannot always be mapped into an RMI-IIOP Java interface. It is only
when the semantics of a specific CORBA object are designed to relate to those of
RMI-IIOP that an RMI-IIOP client can call a CORBA object.

Further reading
Object Management Group website: http://www.omg.org contains CORBA
specifications that are available to download.

OMG - CORBA Basics: http://www.omg.org/gettingstarted/corbafaq.htm.

Remember that some features discussed here are not implemented by all ORBs.

You can find the RMI-IIOP Programmer’s Guide in your SDK installation directory
under docs/rmi-iiop/rmi_iiop_pg.html. Example programs are provided in
demo/rmi-iiop.

Examples of client–server applications

Here, CORBA, RMI (JRMP), and RMI-IIOP approaches are going to be used to
present three client-server hello-world applications. All the applications exploit the
RMI-IIOP IBM ORB.

Interfaces
These are the interfaces that are to be implemented:
v CORBA IDL Interface (Foo.idl):
interface Foo { string message(); };
v JAVA RMI Interface (Foo.java):

34 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Examples of client–server applications

public interface Foo extends java.rmi.Remote

{ public String message() throws java.rmi.RemoteException; }

These two interfaces define the characteristics of the remote object. The remote
object implements a method, named message, that does not need any parameter,
and it returns a string. For further information about IDL and its mapping to Java
see, the OMG specifications (www.omg.org).

Remote object implementation (or servant)

The possible RMI(JRMP) and RMI-IIOP implementations (FooImpl.java) of this
object could be:
public class FooImpl extends javax.rmi.PortableRemoteObject implements Foo {
public FooImpl() throws java.rmi.RemoteException { super(); }
public String message() { return "Hello World!"; }
}

In the early versions of Java RMI (JRMP), the servant class had to extend the
java.rmi.server.UnicastRemoteObject class. Now, you can use the class
PortableRemoteObject for both RMI over JRMP and IIOP, thereby making the
development of the remote object virtually independent of the protocol that is
used. Also, the object implementation does not need to extend
PortableRemoteObject, especially if it already extends another class (single-class
inheritance). However, in this case, the remote object instance must be exported in
the server implementation (see below). By exporting a remote object, you make
that object available to accept incoming remote method requests. When you extend
javax.rmi.PortableRemoteObject, your class is exported automatically on creation.

The CORBA or Java IDL implementation of the remote object (servant) is:
public class FooImpl extends _FooPOA {
public String message() { return "Hello World"; }
}

This implementation conforms to the Inheritance model in which the servant

extends directly the IDL-generated skeleton FooPOA. You might want to use the
Tie or Delegate model instead of the typical Inheritance model if your
implementation must inherit from some other implementation. In the Tie model,
the servant implements the IDL-generated operations interface (such as
FooOperations). However, the Tie model introduces a level of indirection; one extra
method call occurs when you invoke a method. The server code describes the extra
work that is required in the Tie model, so you can decide whether to use the Tie or
the Delegate model. In RMI-IIOP however, you can use only the Tie or Delegate
model.

Stub and ties generation

The RMI-IIOP code provides the tools to generate stubs and ties for whatever
implementation exists of the client and server. Table 1 shows what command
should be run to get stubs and ties (or skeletons) for the three techniques.
Table 1. Commands for stubs and ties (skeletons)
CORBA RMI(JRMP) RMI-IIOP
idlj Foo.idl rmic FooImpl rmic -iiop Foo

The compilation generates the files that are shown in Table 2 on page 36. (Use the
-keep option with rmic if you want to keep the intermediate .java files).

Chapter 5. Understanding the ORB 35

Examples of client–server applications

Table 2. Stub and tie files

CORBA RMI(JRMP) RMI-IIOP
Foo.java FooImpl_Skel.class _FooImpl_Tie.class
FooHolder.java FooImpl_Stub.class _Foo_Stub.class
FooHelper.java Foo.class (Foo.java present) Foo.class (Foo.java present)
FooOperations.java FooImpl.class (only FooImpl.class (only
compiled) compiled)
_FooStub.java
FooPOA.java (-fserver, -fall,
-fserverTie, -fallTie)
FooPOATie.java (-fserverTie,
-fallTie)
_FooImplBase.java
(-oldImplBase)

In the J2SE v.1.4 ORB, the default object adapter (see the OMG CORBA
specification v.2.3) is the portable object adapter (POA). Therefore, the default
skeletons and ties that the IDL compiler generates can be used by a server that is
using the POA model and interfaces. By using the idlj -oldImplBase option, you
can still generate older versions of the server-side skeletons that are compatible
with servers that are written in J2SE 1.3 and earlier.

Server code
The server application has to create an instance of the remote object and publish it
in a naming service. The Java Naming and Directory Interface (JNDI) defines a set
of standard interfaces that are used to query a naming service or to bind an object
to that service.

The implementation of the naming service can be a CosNaming Service in the

CORBA environment or the RMI registry for a RMI (JRMP) application. Therefore,
you can use JNDI in CORBA and in RMI cases, thereby making the server
implementation independent of the naming service that is used. For example, you
could use the following code to obtain a naming service and bind an object
reference in it:
Context ctx = new InitialContext(...); // get hold of the initial context
ctx.bind("foo", fooReference); // bind the reference to the name "foo"
Object obj = ctx.lookup("foo"); // obtain the reference

However, to tell the application which naming implementation is in use, you must
set one of the following Java properties:
v java.naming.factory.initial: Defined also as
javax.naming.Context.INITIAL_CONTEXT_FACTORY, this property specifies the
class name of the initial context factory for the naming service provider. For RMI
registry, the class name is com.sun.jndi.rmi.registry.RegistryContextFactory. For
the CosNaming Service, the class name is com.sun.jndi.cosnaming.CNCtxFactory.
v java.naming.provider.url: This property configures the root naming context, the
ORB, or both. It is used when the naming service is stored in a different host,
and it can take several URI schemes:
– rmi
– corbaname
– corbaloc

36 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Examples of client–server applications

– IOR
– iiop
– iiopname
For example:
rmi://[<host>[:<port>]][/<initial_context>] for RMI registry
iiop://[<host>[:<port>]][/<cosnaming_name>] for COSNaming

To get the previous properties in the environment, you could code:

Hashtable env = new Hashtable();
Env.put(Context.INITIAL_CONTEXT_FACTORY,
"com.sun.jndi.cosnaming.CNCtxFactory");

and pass the hashtable as an argument to the constructor of InitialContext.

For example, with RMI(JRMP), you do not need to do much other than create an
instance of the servant and follow the previous steps to bind this reference in the
naming service.

With CORBA (Java IDL), however, you must do some extra work because you
have to create an ORB. The ORB has to make the servant reference available for
remote calls. This mechanism is usually controlled by the object adapter of the
ORB.
public class Server {
public static void main (String args []) {
try {
ORB orb = ORB.init(args, null);

// Get reference to the root poa & activate the POAManager

POA poa = (POA)orb.resolve_initial_references("RootPOA");
poa.the_POAManager().activate();

// Create a servant and register with the ORB

FooImpl foo = new FooImpl();
foo.setORB(orb);

// TIE model ONLY

// create a tie, with servant being the delegate and
// obtain the reference ref for the tie
FooPOATie tie = new FooPOATie(foo, poa);
Foo ref = tie._this(orb);

// Inheritance model ONLY

// get object reference from the servant
org.omg.CORBA.Object ref = poa.servant_to_reference(foo);
Foo ref = FooHelper.narrow(ref);

// bind the object reference ref to the naming service using JNDI
..........(see previous code) .....
orb.run();
}
catch(Exception e) {}
}
}

For RMI-IIOP:
public class Server {
public static void main (String args []) {
try {
ORB orb = ORB.init(args, null);

// Get reference to the root poa & activate the POAManager

Chapter 5. Understanding the ORB 37

Examples of client–server applications

POA poa = (POA)orb.resolve_initial_references("RootPOA");

poa.the_POAManager().activate();

// Create servant and its tie

FooImpl foo = new FooImpl();
_FooImpl_Tie tie = (_FooImpl_Tie)Util.getTie(foo);

// get an usable object reference

org.omg.CORBA.Object ref = poa.servant_to_reference((Servant)tie);

// bind the object reference ref to the naming service using JNDI
..........(see previous code) .....
}
catch(Exception e) {}
}
}

To use the previous POA server code, you must use the -iiop -poa options together
to enable rmic to generate the tie. If you do not use the POA, the RMI(IIOP) server
code can be reduced to instantiating the servant (FooImpl foo = new FooImpl())
and binding it to a naming service as is usually done in the RMI(JRMP)
environment. In this case, you need use only the -iiop option to enable rmic to
generate the RMI-IIOP tie. If you omit -iiop, the RMI(JRMP) skeleton is generated.

You must remember also one more important fact when you decide between the
JRMP and IIOP protocols. When you export an RMI-IIOP object on your server,
you do not necessarily have to choose between JRMP and IIOP. If you need a
single server object to support JRMP and IIOP clients, you can export your
RMI-IIOP object to JRMP and to IIOP simultaneously. In RMI-IIOP terminology,
this action is called dual export.

RMI Client example:

public class FooClient {
public static void main(String [] args) {
try{
Foo fooref
//Look-up the naming service using JNDI and get the reference
.........
// Invoke method
System.out.println(fooRef.message());
}
catch(Exception e) {}
}
}

CORBA Client example:

public class FooClient {
public static void main (String [] args) {
try {
ORB orb = ORB.init(args, null);
// Look-up the naming service using JNDI
......
// Narrowing the reference to the right class
Foo fooRef = FooHelper.narrow(o);
// Method Invocation
System.out.println(fooRef.message());
}
catch(Exception e) {}
}
}

RMI-IIOP Client example:

38 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Examples of client–server applications

public class FooClient {

public static void main (String [] args) {
try{
ORB orb = ORB.init(args, null);
// Retrieving reference from naming service
........
// Narrowing the reference to the correct class
Foo fooRef = (Foo)PortableRemoteObject.narrow(o, Foo.class);
// Method Invocation
System.out.println(fooRef.message());
}
catch(Exception e) {}
}
}

Summary of major differences between RMI (JRMP) and

RMI-IIOP
This section examines the major differences in development procedures between
RMI (JRMP) and RMI-IIOP. The points discussed here also represent work items
that are necessary when you convert RMI (JRMP) code to RMI-IIOP code.

Because the usual base class of RMI-IIOP servers is PortableRemoteObject, you

must change this import statement accordingly, in addition to the derivation of the
implementation class of the remote object. After completing the Java coding, you
must generate a tie for IIOP by using the rmic compiler with the -iiop option.
Next, run the CORBA CosNaming tnameserv as a name server instead of
rmiregistry.

For CORBA clients, you must also generate IDL from the RMI Java interface by
using the rmic compiler with the -idl option.

All the changes in the import statements for server development apply to client
development. In addition, you must also create a local object reference from the
registered object name. The lookup() method returns a java.lang.Object, and you
must then use the narrow() method of PortableRemoteObject to cast its type. You
generate stubs for IIOP using the rmic compiler with the -iiop option.

Summary of differences in server development

v Import statement:
import javax.rmi.PortableRemoteObject;
v Implementation class of a remote object:
public class FooImpl extends PortableRemoteObject implements Foo
v Name registration of a remote object:
NamingContext.rebind("Foo",ObjRef);
v Generate a tie for IIOP with rmic -iiop
v Run tnameserv as a name server
v Generate IDL with rmic -idl for CORBA clients

Summary of differences in client development

v Import statement:
import javax.rmi.PortableRemoteObject;
v Identify a remote object by name:
Object obj = ctx.lookup("Foo")

MyObject myobj = (MyObject)PortableRemoteObject.narrow(obj,MyObject.class);

Chapter 5. Understanding the ORB 39

Examples of client–server applications

v Generate a stub for IIOP with rmic -iiop

Using the ORB

To use the ORB, you need to understand the properties that the ORB contains.
These properties change the behavior of the ORB as described in this section. All
property values are specified as strings.
v com.ibm.CORBA.AcceptTimeout: (range: 0 through 5000) (default: 0=infinite
timeout)
The maximum number of milliseconds for which the ServerSocket waits in a call
to accept(). If this property is not set, the default 0 is used. If it is not valid, 5000
is used.
v com.ibm.CORBA.AllowUserInterrupt:
Set this property to true so that you can call Thread.Interrupt() on a thread that
is currently involved in a remote method call and thereby interrupt that thread’s
wait for the call to return. Interrupting a call in this way causes a
RemoteException to be thrown, containing a CORBA.NO_RESPONSE runtime
exception with the RESPONSE_INTERRUPTED minor code.
If this property is not set, the default behavior is to ignore any
Thread.Interrupt() received while waiting for a call to complete.
v com.ibm.CORBA.ConnectTimeout: (range: 0 through 300) (default: 0=infinite
timeout)
The maximum number of seconds that the ORB waits when opening a
connection to another ORB. By default, no timeout is specified.
v com.ibm.CORBA.BootstrapHost:
The value of this property is a string. This string can be a host name or the IP
address (for example, 9.5.88.112). If this property is not set, the local host is
retrieved by calling one of the following methods:
– For applications: InetAddress.getLocalHost().getHostAddress()
– For applets: <applet>.getCodeBase().getHost(
The hostname is the name of the machine on which the initial server contact for
this client resides.

Note: This property is deprecated. It is replaced by -ORBInitRef and

-ORBDefaultInitRef.
v com.ibm.CORBA.BootstrapPort: (range: 0 through 2147483647=Java max int)
(default: 2809)
The port of the machine on which the initial server contact for this client is
listening.

Note: This property is deprecated. It is replaced by -ORBInitRef and

-ORBDefaultInitRef.
v com.ibm.CORBA.BufferSize: (range: 0 through 2147483647=Java max int)
(default: 2048)
The number of bytes of a GIOP message that is read from a socket on the first
attempt. A larger buffer size increases the probability of reading the whole
message in one attempt. Such an action might improve performance. The
minimum size used is 24 bytes.
v com.ibm.CORBA.SendingContextRunTimeSupported: (default: true)

40 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Using the ORB

Set this property to false to disable the CodeBase SendingContext RunTime

service. This means that the ORB will not attach a SendingContextRunTime
service context to outgoing messages.
v com.ibm.CORBA.enableLocateRequest: (default: false)
If this property is set, the ORB sends a LocateRequest before the actual Request.
v com.ibm.CORBA.FragmentSize: (range: 0 through 2147483647=Java max int)
(default:1024)
Controls GIOP 1.2 fragmentation. The size specified is rounded down to the
nearest multiple of 8, with a minimum size of 64 bytes. You can disable message
fragmentation by setting the value to 0.
v com.ibm.CORBA.FragmentTimeout: (range: 0 through 600000 ms) (default:
300000)
The maximum length of time for which the ORB waits for second and
subsequent message fragments before timing out. Set this property to 0 if
timeout is not required.
v com.ibm.CORBA.GIOPAddressingDisposition: (range: 0 through 2) (default: 0)
When a GIOP 1.2 Request/LocateRequest/Reply/LocateReply is created, the
addressing disposition is set depending on the value of this property:
– 0 = Object Key
– 1 = GIOP Profile
– 2 = full IOR
If this property is not set or is passed an invalid value, the default 0 is used.
v com.ibm.CORBA.InitialReferencesURL:
The format of the value of this property is a correctly-formed URL; for example,
″http://w3.mycorp.com/InitRefs.file. The actual file contains a name/value pair
like: NameService=<stringified_IOR>. If you specify this property, the ORB does
not attempt the bootstrap approach. Use this property if you do not have a
bootstrap server and want to have a file on the webserver that serves the
purpose.

Note: This property is deprecated.

v com.ibm.CORBA.ListenerPort: (range: 0 through 2147483647=Java max int)
(default: next available system assigned port number)
The port on which this server listens for incoming requests. If this property is
specified, the ORB starts to listen during ORB.init().
v com.ibm.CORBA.LocalHost:
The value of this property is a string. This string can be a host name or the IP
address (ex. 9.5.88.112). If this property is not set, retrieve the local host by
calling: InetAddress.getLocalHost().getHostAddress(). This property represents
the host name (or IP address) of the machine on which the ORB is running. The
local host name is used by the server-side ORB to place the host name of the
server into the IOR of a remote-able object.
v com.ibm.CORBA.LocateRequestTimeout: (range: 0 through 2147483647)
(default: 0=infinity)
Defines the number of seconds to wait before timing out on a LocateRequest
message.
v com.ibm.CORBA.MaxOpenConnections: (range: 0 through 255) (default: 240)
Determines the maximum number of in-use connections that are to be kept in
the connection cache table at any one time.
v com.ibm.CORBA.MinOpenConnections: (range: 0 through 255) (default: 100)

Chapter 5. Understanding the ORB 41

Using the ORB

The ORB cleans up only connections that are not busy from the connection cache
table, if the size is of the table is higher than the MinOpenConnections.
v com.ibm.CORBA.NoLocalInterceptors: (default: false)
If this property is set to true, no local PortableInterceptors are driven. This
should improve performance if interceptors are not required when invoking a
co-located object.
v com.ibm.CORBA.ORBCharEncoding: (default: ISO8859_1)
Specifies the ORB’s native encoding set for character data.
v com.ibm.CORBA.ORBWCharDefault: (default: UCS2 )
Indicates that wchar codeset UCS2 is to be used with other ORBs that do not
publish a wchar codeset.
v com.ibm.CORBA.RequestTimeout: (range: 0 through 2147483647) (default:
0=infinity)
Defines the number of seconds to wait before timing out on a Request message.
v com.ibm.CORBA.SendVersionIdentifier: (default: false)
Tells the ORB to send an initial dummy request before it starts to send any real
requests to a remote server. This action determines the partner version of the
remote server ORB from that ORB’s response.
v com.ibm.CORBA.ServerSocketQueueDepth: (range: 50 through 2147483647 )
(default: 0)
The maximum queue length for incoming connection indications (a request to
connect). If a connection indication arrives when the queue is full, the
connection is refused. If the property is not set, the default 0 is used. If the
property is not valid, 50 is used.
v com.ibm.CORBA.ShortExceptionDetails: (default: false)
When a CORBA SystemException reply is created, the ORB, by default, includes
the Java stack trace of the exception in an associated ExceptionDetailMessage
service context. If you set this property to any value, the ORB includes a
toString of the Exception instead.
v com.ibm.tools.rmic.iiop.Debug: (default: false)
The rmic tool automatically creates import statements in the classes that it
generates. If set to true, this property causes rmic to output the mappings of
fully qualified class names to short names.
v com.ibm.tools.rmic.iiop.SkipImports: (default: false)
If this property is set to true, classes are generated with rmic using fully
qualified names only.

Table 3 shows the Sun properties that are now deprecated and the IBM properties
that have replaced them .
Table 3. Deprecated Sun properties
Sun property IBM property
com.sun.CORBA.ORBServerHost com.ibm.CORBA.LocalHost
com.sun.CORBA.ORBServerPort com.ibm.CORBA.ListenerPort
org.omg.CORBA.ORBInitialHost com.ibm.CORBA.BootstrapHost
org.omg.CORBA.ORBInitialPort com.ibm.CORBA.BootstrapPort
org.omg.CORBA.ORBInitialServices com.ibm.CORBA.InitialReferencesURL

42 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Using the ORB

Note that none of these properties are OMG standard properties, despite their
names.

How the ORB works

This section describes a simple, typical RMI-IIOP session in which a client accesses
a remote object on a server by implementing an interface named Foo, and invokes
a simple method called message(). This method returns a Hello World string. (See
the examples that are given earlier in this chapter.)

Firstly, this section explains the client side, and describes what the ORB does under
the cover and transparently to the client. Then, the important role of the ORB in
the server-side is explained

The client side

The subjects discussed here are:
v “Stub creation”
v “ORB initialization” on page 44
v “Getting hold of the remote object” on page 44
v “Remote method invocation” on page 46

Stub creation
In a simple distributed application, the client needs to know (in almost all the
cases) what kind of object it is going to contact and which method of this object it
needs to invoke. Because the ORB is a general framework you must give it general
information about the method that you want to invoke.

For this reason, you implement a Java interface, Foo, which contains the signatures
of the methods that can be invoked in the remote object (see Figure 3).

RMI Java interface

RMI (Foo.java) RMI
Java Java
client server

Stub TIE
rmic-iiop
_Foo_Stub.java _Foo_Tie.java

IIOP
ORB ORB

Figure 3. The ORB client side

The client relies on the existence of a server that contains an object that is that Foo
interface. You must, therefore, create a proxy. This proxy is an object, called stub
that acts as an interface between client application and ORB.

To create the stub, run the RMIC compiler on the Java interface: rmic -iiop Foo.
This action generates a file/object that is named _Foo_Stub.

Chapter 5. Understanding the ORB 43

How the ORB works

The presence of a stub is not always mandatory for a client application to operate.
When you use particular CORBA features such as the DII (Dynamic Invocation
Interface), you do not require a stub because the proxy code is implemented
directly by the client application. You can also upload a stub from the server to
which you are trying to connect. See the CORBA specification for further details

ORB initialization
In a standalone Java application, the client has to create an instance of the ORB by
calling the static method init(...); for example:
ORB orb = ORB.init(args,props);

The parameters that are passed to the method are:

v A string array that contains pairs property-value
v A Java Properties object

For an applet, a similar method is used in which a Java Applet is passed instead of
the string array.

The first step of the ORB initialization is the processing of the ORB properties. The
properties are processed in the following sequence:
1. Check in the applet parameter or application string array
2. Check in the properties parameter (if the parameter exists)
3. Check in the system properties
4. Check in the orb.properties file that is in the <user-home> directory (if the file
exists)
5. Check in the orb.properties file that is in the <java-home>/lib directory (if the
file exists)
6. Fall back on a hardcoded default behavior

The two properties ORBClass and ORBSingletonClass determine which ORB class
has to be instantiated.

The ORB loads its native libraries. Libraries are not mandatory, but they improve
performance.

After this, the ORB starts and initializes the TCP transport layer. If the ListenerPort
property was set, the ORB also opens a ServerSocket that is listening for incoming
requests, as a server-side ORB usually does. At the end of the init() method, the
ORB is fully functional and ready to support the client application.

Getting hold of the remote object

Several methods exist by which the client can get a reference for the remote object.
Usually, this reference is in a stringified form, called an IOR (Interoperable Object
Reference). For example:
IOR:000000000000001d524d493a5......

This reference contains all the information that is necessary to find the remote
object. It also contains some details of the settings of the server to which the object
belongs.

Generally, the client ORB is not supposed to understand the details of the IOR, but
use it as a sort of a key; that is, a reference to the remote object. However, when
client and server are both using an IBM ORB, extra features are coded in the IOR.

44 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 How the ORB works

For example, the IBM ORB adds into the IOR a proprietary field that is called
IBM_PARTNER_VERSION. This field looks like:
49424d0a 00000008 00000000 1400 0005

where:
v The three initial bytes (from left to right) are the ASCII code for IBM, followed
by 0x0A, which specifies that the following bytes handle the partner version.
v The next four bytes encode the length of the remaining data (in this case 8 bytes)
v The next four null bytes are for future use.
v The two bytes for the Partner Version Major field (0x1400) define the release of
the ORB that is being used (1.4.0 in this case).
v The Minor field (0x0005) distinguishes in the same release, service refreshes that
contain changes that have affected the backward compatibility.

Because the IOR is not visible to application-level ORB programmers and the client
ORB does not know where to look for it, another step has to be made. This step is
called the bootstrap process. Basically, the client application needs to tell the ORB
where the remote object reference is located.

A typical example of bootstrapping is if you use a naming service: the client

invokes the ORB method resolve_initial_references(″NameService″) that returns
(after narrowing) a reference to the name server in the form of a NamingContext
object. The ORB looks for a name server in the local machine at the port 2809 (as
default). If no name server exists , or the name server is listening in another port,
the ORB returns an exception. The client application can specify a different host,
port, or both by using the -ORBInitRef and -ORBInitPort options.

Using the NamingContext and the name with which the Remote Object has been
bound in the name service, the client can retrieve a reference to the remote object.
The reference to the remote object that the client holds is always an instance of a
Stub object; that is, your _Foo_Stub.

ORB.resolve_initial_references() causes a lot of activity under the covers. Mainly,

the ORB starts a remote communication with the name server. This communication
might include several requests and replies. Usually the client ORB first checks
whether a name server is listening, then asks for the specified remote reference. In
an application where performance is considered important, caching the remote
reference is a better alternative to repetitive use of the naming service. However,
because the naming service implementation is a transient type, the validity of the
cached reference is tied to the time in which the naming service is running.

The IBM ORB implements an Interoperable Naming Service as described in the

CORBA 2.3 specification. This service includes a new string format that can be
passed as a parameter to the ORB methods string_to_object() and
resolve_initial_references(). By invoking the previous two methods where the string
parameter has a corbaloc (or corbaname) format as, for example:
corbaloc:iiop:[email protected]:1050/AService

the client ORB uses GIOP 1.0 to send a request with a simple object key of
AService to port 1050 at host aserver.aworld.aorg. There, the client ORB expects to
find a server for the Aservice that is requested, and returns a reference to itself.
You can then use this reference to look for the remote object.

Chapter 5. Understanding the ORB 45

How the ORB works

This naming service is transient. It means that the validity of the contained
references expires when the name service or the server for the remote object is
stopped.

Remote method invocation

At this point, the client should hold a reference to the remote object that is an
instance of the stub class. The next step is to invoke the method on that reference.
The stub implements the Foo interface and therefore contains the message()
method that the client has invoked. It is that method that is executed.

First, the stub code determines whether the implementation of the remote object is
located on the same ORB instance and can be accessed without using the internet.

Note: In this discussion, the remote object will be called FooImpl, which in CORBA
language is referred to as a servant.

If the implementation of the remote object is located on the same ORB instance, the
performance improvement can be significant because a direct call to the object
implementation is done. If no local servant can be found, the stub first asks the
ORB to create a request by invoking its _request() method, specifying the name of
the method to invoke and whether a reply is expected or not.

Note that the CORBA specification imposes an extra indirection layer between the
ORB code and the stub. This layer is commonly known as delegation. CORBA
imposes the layer by using an interface named Delegate. This interface specifies a
portable API for ORB-vendor-specific implementation of the
org.omg.CORBA.Object methods. Each stub contains a delegate object, to which all
org.omg.CORBA.Object method invocations are forwarded. This allows a stub that
is generated by one vendor’s ORB to work with the delegate from another
vendor’s ORB.

When creating a request, the ORB first checks whether the enableLocateRequest
property is set to true. If it is, a LocateRequest is created. The steps of creating this
request are similar to the full Request case.

The ORB gets hold of the IOR of the remote object (the one that was retrieved by a
naming service, for example) and passes the information that is contained in the
IOR (Profile object) to the transport layer.

The transport layer uses the information that is in the IOR (IP address, port
number, object key) to create a connection if it does not already exist. The ORB
TCP/IP transport has an implementation of a table of cached connections for
improving performances, because the creation of a new connection is a
time-consuming process. The connection at this point is not an open
communication channel to the server host. It is only an object that has the potential
to create and deliver a TCP/IP message to a location on the internet. Usually that
involves the creation of a Java socket and a reader thread that is ready to intercept
the server reply. The ORB.connect() is invoked as part of this process.

When the ORB has the connection, it proceeds to create the Request message. In
the message are the header and the body of the request. The CORBA 2.3
specification specifies the exact format. The header contains, for example, local and
remote IP addresses and ports, message size, version of the CORBA stream format
(GIOP 1.x with x=0,1,2), byte sequence convention, request types, and Ids. (See
Chapter 18, “Debugging the ORB,” on page 165 for a detailed description and
example).

46 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 How the ORB works

The body of the request contains several service contexts and the name and
parameters of the method invocation. Parameters are typically serialized.

A service context is some extra information that the ORB includes in the request or
reply, to add several other functions. CORBA defines a few service contexts, such
as the codebase and the codeset service contexts. The first is used for the call-back
feature (see the CORBA specification), the second to specify the encoding of
strings.

In the next step, the stub calls _invoke(). Again it is the delegate invoke() method
that is executed. The ORB in this chain of events calls the send() method on the
connection that will write the request to the socket buffer and flush it away. The
delegate invoke() method waits for a reply to arrive. The reader thread that was
spun during the connection creation gets the reply message, demarshals it, and
returns the correct object.

The server side

Typically, a server is an application that makes available one of its implemented
objects through an ORB instance. The subjects discussed here are:
v “Servant implementation”
v “Tie generation”
v “Servant binding”
v “Processing a request” on page 48

Servant implementation
The implementations of the remote object can either inherit from
javax.rmi.PortableRemoteObject, or implement a remote interface and use the
exportObject() method to register themselves as a servant object. In both cases, the
servant has to implement the Foo interface. Here, the first case is described. From
now, the servant is called FooImpl.

Tie generation
Again, you must put an interfacing layer between the servant and the ORB code.
In the old RMI(JRMP) naming convention “skeleton” was the name given to the
proxy that was used on the server side between ORB and the object
implementation. In the RMI-IIOP convention, the proxy is called a Tie.

You generate the RMI-IIOP tie class at the same time as the stub, by invoking the
rmic compiler. These classes are generated from the compiled Java programming
language classes that contain remote object implementations; for example, rmic
-iiop FooImpl generates the stub _Foo_Stub.class and the tie _Foo_Tie.class.

Servant binding
The server implementation is required to do the following tasks:
1. Create an ORB instance; that is, ORB.init(...)
2. Create a servant instance; that is, new FooImpl(...)
3. Create a Tie instance from the servant instance; that is, Util.getTie(...)
4. Export the servant by binding it to a naming service

As described for the client side, you must create the ORB instance by invoking the
ORB static method init(...). The usual steps for that method are:
1. Retrieve properties
2. Get the system class loader

Chapter 5. Understanding the ORB 47

How the ORB works

3. Load and instantiate the ORB class as specified in the ORBClass property
4. Initialize the ORB as determined by the properties

Then, the server needs to create an instance of the servant class FooImpl.class.
Something more than the creation of an instance of a class happens under the
cover. Remember that the servant FooImpl extends the PortableRemoteObject class,
so the constructor of PortableRemoteObject is executed. This constructor calls the
static method exportObject(...) whose parameter is the same servant instance that
you try to instantiate. The programmer must directly call exportObject() if it is
decided that the servant will not inherit from PortableRemoteObject.

The exportObject() method first tries to load a rmi-iiop tie. The ORB implements a
cache of classes of ties for improving performances. If a tie class is not already
cached, the ORB loads a tie class for the servant. If it cannot find one, it goes up
the inheritance tree, trying to load the parent class ties. It stops if it finds a
PortableRemoteObject class or a java.lang.Object, and returns null. Otherwise, it
returns an instance of that tie that is kept in a hashtable that is paired with the
instance of the tie’s servant. If the ORB cannot get hold of the tie, it guesses that an
RMI (JRMP) skeleton might be present and calls the exportObject method of the
UnicastRemoteObject class. Finally, if all fails, a null tie and exception is thrown.
At this point, the servant is ready to receive remote methods invocations. However,
it is not yet reachable.

In the next step, the server code has to get hold of the tie itself (assuming the ORB
has already done this successfully) to be able to export it to a naming service. To
do that, the server passes the newly-created instance of the servant into the static
method javax.rmi.CORBA.Util.getTie(). This, in turn, fetches the tie that is in the
hashtable that the ORB created. The tie contains the pair of tie-servant classes.

When in possession of the tie, the server must get hold of a reference for the
naming service and bind the tie to it. As in the client side, the server invokes the
ORB method resolve_initial_references(“NameService”). It then creates a
NameComponent, a sort of directory tree object that identifies in the naming
service the path and the name of the remote object reference, and binds together
this NameComponent with the tie. The naming service then makes the IOR for the
servant available to anyone requesting. During this process, the server code sends
a LocateRequest to get hold of the naming server address. It also sends a Request
that requires a rebind operation to the naming server.

Processing a request
During the ORB initialization, a listener thread was created. The listener thread is
listening on a default port (the next available port at the time the thread was
created). You can specify the listener port by using the
com.ibm.CORBA.ListenerPort property. When a request comes in through that
port, the listener thread first creates a connection with the client side. In this case,
it is the TCP transport layer that takes care of the details of the connection. As seen
for the client side, the ORB caches all the connections that it creates.

By using the connection, the listener thread spawns a reader thread to process the
incoming message. When dealing with multiple clients, the server ORB has a
single listener thread and one reader thread for each connection or client.

The reader thread does not fully read the request message, but instead creates an
input stream for the message to be piped into. Then, the reader thread picks up
one of the worker threads in the implemented pool (or creates one if none is
present), and delegates the reading of the message. The worker threads read all the

48 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 How the ORB works

fields in the message and dispatch them to the tie, which unmarshals any
parameters and invokes the remote method.

The service contexts are then created and written to the response output stream
with the return value. The reply is sent back with a similar mechanism, as
described in the client side. After that, the connection is removed from the reader
thread which eventually stops.

Features of the ORB

This section describes:
v “Portable object adapter”
v “Fragmentation” on page 51
v “Portable interceptors” on page 51
v “Interoperable naming service (INS)” on page 54
v “Other features” on page 55

Portable object adapter

An object adapter is the primary way for an object to access ORB services such as
object reference generation. An object adapter exports a public interface to the
object implementation, and a private interface to the skeleton. The main
responsibilities of an object adapter are:
v Generation and interpretation of object references
v Method invocation
v Object and implementation activation and deactivation
v Mapping object references to the corresponding object implementations

Figure 4 shows how the object adapter relates to the ORB, the skeleton, and the
object implementation.

Figure 4. Relationship between the ORB, the object adapter, the skeleton, and the object
implementation

In CORBA 2.1 and below, all ORB vendors had to implement an object adapter,
which was known as the basic object adapter. Because the basic object adapter was
never completely specified with a standard CORBA IDL, vendors implemented it
in many different ways. Therefore, for example, programmers could not write
server implementations that could be truly portable between different ORB
products. A first attempt to define a standard object adapter interface was done in

Chapter 5. Understanding the ORB 49

ORB - features

CORBA 2.1. With CORBA v.2.3, the OMG group released the final corrected
version for a standard interface for the object adapter. This adapter is known as the
portable object adapter (POA).

Some of the main features of the POA specification are:

v Allow programmers to construct object and server implementations that are
portable between different ORB products.
v Provide support for persistent objects; that is, objects whose lifetimes span
multiple server lifetimes.
v Support transparent activation of objects and the ability to associate policy
information to objects.
v Allow multiple distinct instances of the POA to exist in one ORB.

For more details of the POA, see the CORBA v.2.3 (formal/99-10-07) specification.

The IBM J2SE v.1.4 ORB supports both the POA specification and the proprietary
basic object adapter that is already present in previous IBM ORB versions. As
default, the rmic compiler, when used with the -iiop option, generates RMI-IIOP
ties for servers. These ties are based on the basic object adapter. When a server
implementation uses the POA interface, you must add the -poa option to the rmic
compiler to generate the relevant ties.

If you want to implement an object that is using the POA, the server application
must obtain a POA object. When the server application invokes the ORB method
resolve_initial_reference(RootPOA), the ORB returns the reference to the main POA
object that contains default policies (see the CORBA specification for a complete
list of all the POA policies). You can create new POAs as children of the RootPOA,
and these children can contain different policies. This in turn allows you to
manage different sets of objects separately, and to partition the name space of
objects IDs.

Ultimately, a POA handles Object IDs and active servants. An active servant is a
programming object that exists in memory and has been registered with the POA
by use of one or more associated object identities. The ORB and POA cooperate to
determine on which servant the client-requested operation should be invoked. By
using the POA APIs, you can create a reference for the object, associate an object
ID, and activate the servant for that object. A map of object IDs and active servants
is stored inside the POA. A POA provides also a default servant that is used when
no active servant has been registered. You can register a particular implementation
of this default servant and also of a servant manager, which is an object for
managing the association of an object ID with a particular servant. A simple POA
architecture is represented in Figure 5 on page 51.

50 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 ORB - features

RootPOA POA Child1

Object ID Default servant

Object ID User-supplied servant

POA
User-supplied Object ID User-supplied servant
servant manager
Object ID User-supplied servant

Adapter activator

Figure 5. Simple portable object adapter architecture

The POA Manager is an object that encapsulates the processing state of one or
more POAs. You can control and change the state of all POAs by using operations
on the POA manager.

The adapter activator is an object that an application developer uses to activate

child POAs.

Fragmentation
CORBA specification introduced the concept of fragmentation to handle the
growing complexity and size of marshaled objects in GIOP messages. Graphs of
objects are linearized and serialized inside a GIOP message under the IDL
specification of valuetypes. Fragmentation specifies the way a message can be split
into several smaller messages (fragments ) and sent over the net.

The system administrator can set the properties FragmentSize and

FragmentTimeout to obtain best performance in the existing net traffic. As a
general rule, the default value of 1024 bytes for the fragment size is a good
trade-off in almost all conditions. The fragment time-out should not be set to too
low a value, or time-outs might occur unnecessarily.

Portable interceptors
CORBA implementations have long had proprietary mechanisms that allow users
to insert their own code into the ORB’s flow of execution. This code, known as
interceptors, is called at particular stages during the processing of requests. It can
directly inspect and even manipulate requests.

Because this message filtering mechanism is extremely flexible and powerful, the
OMG standardized interceptors in the CORBA 2.4.2 specification under the name
“portable interceptors”. The idea is to define a standard interface to register and
execute application-independent code that, among other things, takes care of
passing service contexts. These interfaces are stored in the package
org.omg.PortableInterceptor.* . The implementation classes are in the
com.ibm.rmi.pi.* package of the IBM ORB. All the interceptors implement the
Interceptor interface.

Two classes of interceptors are defined: request interceptors and IOR interceptors.
Request interceptors are called during request mediation. IOR interceptors are
Chapter 5. Understanding the ORB 51
ORB - features

called when new object references are created so that service-specific data can be
added to the newly-created IOR in the form of tagged components.

The ORB calls request interceptors on the client and the server side to manipulate
service context information. Interceptors must register with the ORB for those
interceptor points that are to be executed.

Five interception points are on the client side:

v send_request (sending request)
v send_poll (sending request)
v receive_reply (receiving reply)
v receive_exception (receiving reply)
v receive_other (receiving reply)

Five interception points are on the server side:

v receive_request_service_contexts (receiving request)
v receive_request (receiving request)
v send_reply (sending reply)
v send_exception (sending reply)
v send_other (sending reply)

The only interceptor point for IOR interceptors is establish_component. The ORB
calls this interceptor point on all its registered IOR interceptors when it is
assembling the set of components that is to be included in the IOP profiles for a
new object reference. Registration of interceptors is done using the interface
ORBInitializer.

Example:
package pi;

public class MyInterceptor extends org.omg.CORBA.LocalObject

implements ClientRequestInterceptor, ServerRequestInterceptor
{

public String name() { return "MyInterceptor"; }

public void destroy() {}

// ClientRequestInterceptor operations
public void send_request(ClientRequestInfo ri)
{ logger(ri, "send_request"); }

public void send_poll(ClientRequestInfo ri)

{ logger(ri, "send_poll"); }

public void receive_reply(ClientRequestInfo ri)

{ logger(ri, "receive_reply"); }

public void receive_exception(ClientRequestInfo ri)

{ logger(ri, "receive_exception"); }
public void receive_other(ClientRequestInfo ri)
{ logger(ri, "receive_other"); }

// Server interceptor methods

public void receive_request_service_contexts(ServerRequestInfo ri)
{ logger(ri, "receive_request_service_contexts"); }

public void receive_request(ServerRequestInfo ri)

52 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 ORB - features

{ logger(ri, "receive_request"); }

public void send_reply(ServerRequestInfo ri)

{ logger(ri, "send_reply"); }

public void send_exception(ServerRequestInfo ri)

{ logger(ri, "send_exception"); }

public void send_other(ServerRequestInfo ri)

{ logger(ri, "send_other"); }

// Trivial Logger
public void logger(RequestInfo ri, String point)
{
System.out.println("Request ID:" + ri.request_id() +
" at " name() + "." + point);
}
}
}

The interceptor class extends org.omg.CORBA.LocalObject to ensure that an

instance of this class does not get marshaled, because an interceptor instance is
strongly tied to the ORB with which it is registered. This trivial implementation
prints out a message at every interception point.

You can do a simple registration of the interceptor by using the ORBInitializer

class. Because interceptors are intended to be a means by which ORB services
access ORB processing, by the time the init() method call on the ORB class returns
an ORB instance, the interceptors have already been registered. It follows that
interceptors cannot be registered with an ORB instance that is returned from the
init() method call.

First, you must create a class that implements the ORBInitializer class. This class
will be called by the ORB during its initialization:
public class MyInterceptorORBInitializer extends LocalObject implements ORBInitializer {

public static Interceptor interceptor;

public String name() { return ""; }

public void pre_init(ORBInitInfo info) {

try {
interceptor = new MyInterceptor();
} catch (Exception ex) {}
}

public void post_init(ORBInitInfo info) {}

}

Then, in the server implementation, add the following code:

Properties p = new Properties();
p.put("org.omg.PortableInterceptor.ORBInitializerClass.pi.MyInterceptorORBInitializer", "");

orb = ORB.init((String[])null, p);

During the ORB initialization, the ORB runtime gets hold of the ORB properties
that begin with org.omg.PortableInterceptor.ORBInitializerClass;. The remaining
portion is extracted and the corresponding class is instantiated. Then, the pre_init()
and post_init() methods are called on the initializer object.

Chapter 5. Understanding the ORB 53

ORB - features

Interoperable naming service (INS)

CosNaming that is implemented in the IBM ORB is another name for the CORBA
Naming Service that observes the OMG Interoperable Naming Service specification
(INS, CORBA 2.3 specification). It stands for Common Object Services Naming. The
name service maps names to CORBA object references. Object references are stored
in the namespace by name and each object reference-name pair is called a name
binding. Name bindings can be organized under naming contexts. Naming contexts
are themselves name bindings, and serve the same organizational function as a file
system subdirectory does. All bindings are stored under the initial naming context.
The initial naming context is the only persistent binding in the namespace.

This implementation includes a new string format that can be passed as a

parameter to the ORB methods string_to_object() and resolve_initial_references()
such as the corbaname and corbaloc formats.

Corbaloc URIs allow you to specify object references that can be contacted by IIOP,
or found through ORB::resolve_initial_references(). This new format is easier than
IOR is to manipulate. To specify an IIOP object reference, use a URI of the form
(see the CORBA 2.4.2 specification for full syntax):
corbaloc:iiop:<host>:<port>/<object key>

For example, the following corbaloc URI specifies an object with key MyObjectKey
that is in a process that is running on myHost.myOrg.com listening on port 2809.
corbaloc:iiop:myHost.myOrg.com:2809/MyObjectKey

Corbaname URIs (see the CORBA 2.4.2 specification) cause string_to_object() to

look up a name in a CORBA naming service. They are an extension of the corbaloc
syntax:
corbaname:<corbaloc location>/<object key>#<stringified name>

For example:
corbaname::myOrg.com:2050#Personal/schedule

where the portion of the reference up to the hash mark (#) is the URL that returns
the root naming context. The second part is the argument that is used to resolve
the object on the NamingContext.

The INS specified two standard command-line arguments that provide a portable
way of configuring ORB::resolve_initial_references():
v -ORBInitRef takes an argument of the form <ObjectId>=<ObjectURI>. So, for
example, with command-line arguments of:
-ORBInitRef NameService=corbaname::myhost.example.com

resolve_initial_references(″NameService″) returns a reference to the object with

key NameService available on myhost.example.com, port 2809.
v -ORBDefaultInitRef provides a prefix string that is used to resolve otherwise
unknown names. When resolve_initial_references() cannot resolve a name that
has been specifically configured (with -ORBInitRef), it constructs a string that
consists of the default prefix, a `/’ character, and the name requested. The string
is then fed to string_to_object(). So, for example, with a command-line of:
-ORBDefaultInitRef corbaloc::myhost.example.com

a call to resolve_initial_references(″MyService″) returns the object reference that

is denoted by corbaloc::myhost.example.com/MyService.

54 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 ORB - features

You can specify -ORBInitRef and -ORBDefaultInitRef also as system properties;

for example:
-Dcom.ibm.CORBA.ORBInitRef.NameService="corbaloc:..."
-Dcom.ibm.CORBA.ORBDefaultInitRef="corbaloc:..."

Other features
Among all the other differences with previous versions of IBM ORBs, it is
important to outline the support for GIOP 1.2, an extended and improved RAS
facility.

Chapter 5. Understanding the ORB 55

ORB - features

56 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 6. Understanding the Java Native Interface (JNI)
The specification for the Java Native Interface (JNI) is maintained by Sun
Microsystems Inc. IBM recommends that you read the JNI specification. Go to
http://java.sun.com/ and search the site for JNI. Sun Microsystems Inc maintains
a combined programming guide and specification at http://java.sun.com/docs/
books/jni/.

This chapter gives additional information to help you with JNI operation and
design.

The topics that are discussed in this chapter are:

v “Overview of JNI”
v “The JNI and the Garbage Collector” on page 58
v “Copying and pinning” on page 60
v “Handling local references” on page 60
v “Handling global references” on page 64
v “Handling exceptions” on page 64
v “Using the isCopy flag” on page 65
v “Using the mode flag” on page 65
v “A generic way to use the isCopy and mode flags” on page 66
v “Synchronization” on page 66
v “Debugging the JNI” on page 67
v “JNI checklist” on page 68

Overview of JNI
The JNI is a set of wrapper functions that enables C or C++ code to access Java
code, and Java code to access C or C++ code. The JNI does very little management;
it mostly provides a vehicle for the code.

Note: In this chapter, C/C++ code is always called native code because it runs
directly on the target platform, unlike Java code, which requires a JVM.

You can use the JNI in two ways:

v You can write some C or C++ code in a library, and call it from your Java
application.
v You can embed a JVM in your native application so that you can write some
parts of that application in Java. This way is the normal runtime mode of Java;
that is, you start a native Java executable, which then embeds a JVM to execute
the Java code that you specify to that executable.

The JNI specification does not have a complete set of rules about how the JNI is to
be implemented. Therefore, different vendors implement JNI in different ways. The
Sun trademark specification and the Java Compatibility Kit (JCK) ensure
compliance to the specification, but not to the implementation. It is a common
mistake to write native JNI code that assumes implementation methods instead of
conforming strictly to the specification. Although this code might not cause any

© Copyright IBM Corp. 2003, 2006 57

Overview of JNI

problems at first, it could cause many problems if it is moved from one vendor’s
JVM to another, or if a vendor changes an implementation strategy.

The JNI and the Garbage Collector

Before you read about the two main JNI topics (“Handling local references” on
page 60 and “Handling global references” on page 64), you need to understand
why and how references are maintained, and how the Garbage Collector is
involved.

Three main interactions occur between the Garbage Collector and the JNI. Those
interactions are:
1. Garbage Collector and object references
2. Garbage Collector and global references
3. Garbage Collector and retained garbage

The first two interactions manage Java objects in native code. The third is a result
of the design of the IBM Garbage Collector.

Garbage Collector and object references

The Garbage Collector reclaims garbage, which is defined as anything on the Java
heap that is not reachable. However, if you access a Java object from your native
code, the reference for that access might not exist in a form that the Garbage
Collector can trace. The Garbage Collector, therefore, is likely to deduce that
objects that you have referenced or created are garbage. The Garbage Collector can,
from its root set of object pointers, trace only references to objects that are in the
Java heap (see Chapter 2, “Understanding the Garbage Collector,” on page 7).

To avoid this problem, the JNI automatically creates a local reference to any object
that is referenced across it. The local reference that it creates for your object is a
pointer to your object. It is created in the stack of the thread that is running your
code. When the Garbage Collector runs, it finds that local reference as part of its
root set of object pointers (see Chapter 2, “Understanding the Garbage Collector,”
on page 7) and therefore does not collect your object.

You can think of local references as invisible automatic variables that are in the
function or method that you use to access a Java object. The invisible variable is
passed on (invisibly) to all the functions that are called within the function that
declares the local reference, and to all the functions that are called by them, and so
on. As with all automatic variables, the local reference goes out of scope when you
exit the function in which it was declared.

Therefore, you have two elements of data for objects to which you refer across the
JNI. You have a real object that exists on the Java heap, and you have a reference to
that object. This reference exists on the stack of your native thread. When the
reference disappears, it does not directly affect the object to which you referred,
but the object might become unreachable and therefore able to be collected by a
future garbage collection cycle. An object can have more than one native reference
to it, and remains uncollectable as long as one or more references exist.

Here is some JNI code:

58 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Garbage Collector and object references

static void JNIcode (...)

{
jobject myObject = env->NewObject ()

env->GetObjectClass (myObject)
}

Here is how the same code would look if you used a local variable to create an
object reference (invisible code is in italics):
static void JNIcode (...)
{
void * myObjectlocalRef;

jobject myObject = env->NewObject ()

myObjectLocalRef = *myObject

env->GetObjectClass (myObject, myObjectLocalRef)

}// myObjectLocalRef goes out of scope here

The myObjectLocalRef is created in the scope of the function or method that creates
the object for which the local reference exists. This imaginary automatic variable
refers to myObject so that it cannot be garbage collected in the scope of the local
reference. The analogy has been expanded a little by the passing of the automatic
variable into all the functions that are called inside the scope. The idea is that the
local reference in JNIcode remains active in the GetObjectClass function, and in
any other functions that it calls. Only when you exit the function (or method) in
which a local reference is created does it become invalid (or out of scope). How
this affects your application is discussed in more detail in “Handling local
references” on page 60.

Garbage Collector and global references

“Garbage Collector and object references” on page 58 showed how local references
are automatically created and deleted. The scope of local references, however, is
limited. If you want to use an object outside the scope of a local reference, you
must manually create a reference to it. Obviously, you are also responsible for
deleting such a reference. These references are known as global references. Global
references are stored in a space that is reserved by the JVM. This space is in the
native heap space for the Java process. The Garbage Collector always checks in this
special space to determine whether a reference exists to an otherwise unreachable
object.

Another class of references is available. These references are known as weak global
references whose typical function is to cache objects. For more information about
weak global references, see your JNI documentation.

Garbage Collector and retained garbage

Retained garbage is space that is unused in the heap, but not recognized as unused
by the Garbage Collector. Therefore, the space is not reclaimed, it is retained.
Retained garbage is garbage that might not be collected when you think it should
be. For example, you know that a particular object is garbage but find that, after a
garbage collection cycle, it has not been collected.

You cannot directly solve this problem; it usually solves itself. Eventually, the
Garbage Collector finds the garbage. Do not assume that you can determine when
garbage should be collected. If this simple answer is enough for you, go to
“Handling local references” on page 60. Otherwise, continue here.

Chapter 6. Understanding the Java Native Interface (JNI) 59

Garbage Collector and retained garbage

The retained garbage is a result of the conservative nature of the Garbage Collector
reclamation and the use of JNI. You cannot always determine whether a value in
the stack frame is a reference to a Java object, or whether it is a native parameter
value that has been pushed onto the stack.

The Java threads execute as native threads on the native platform. The thread of
execution is defined by the set of frames that is on the native stack. The Garbage
Collector finds part of its set of root objects by scanning the native stack. When a
mixture of native and Java frames exists on the stack, the Garbage Collector might
scan native stack frames and create false root objects. These actions lead to retained
garbage. The JVM attempts to store the limit of the heap when it changes from
Java code to C/C++ code, so that it can control a garbage collection scan.
However, nested or recursive JNI calls (for example, from native code -> Java ->
native code -> Java) cause Java and native frames to become interleaved on the
stack, and the Garbage Collector is forced to scan an area that does not contain
valid heap references. As a result, false root objects are found, and the garbage of
any object graph to which such a root object refers might be kept.

Copying and pinning

Objects that are on the Java heap are usually mobile; that is, the Garbage Collector
can move them around if it decides to resequence the heap. Some objects, however,
cannot be moved either permanently, or temporarily. Such immovable objects are
known as pinned objects.

When native code, by way of the JNI, creates or refers to an object that is on the
heap, the JVM can do either of these actions:
v Make a copy of the object in local storage, and return this copy to the caller
v Pin the actual object on the heap, and return a pointer to the caller

The caller is told whether the object is a copy or is pinned, by way of a flag in the
appropriate API call.

The IBM Virtual Machine for Java usually uses a pinning implementation instead
of a copy implementation.

Handling local references

Local reference scope

You must understand the scoping rules of local references before you can
understand the problems that this section discusses. Ensure that you have read
“The JNI and the Garbage Collector” on page 58 or have visited the Sun website at
http://www.sun.com and read the documentation or specification that is given
there.

It is very easy to lose a local reference accidentally. That is, the local reference goes
out of scope, but you continue to use the objects to which it used to refer. When
you lose a local reference in this way, the object is not pinned down, and problems
will occur later. The loss of a local reference does not invalidate the object to which
it refers. Your application continues to work normally and to use the object, until a
garbage collection cycle occurs. However, until the space on the heap is moved or
reused, you can continue to use the object. Your code is pointing to invalid space,
but that space continues to hold the valid data that you put into it.

60 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Local reference scope

So your application might seem to work well, but at random intervals, it fails
when an object that you think is valid suddenly disappears. This is the type of
problem that usually occurs late in a product cycle. It can be quite difficult to
isolate. If you always have this type of problem shortly after a garbage collection
cycle with compaction, when objects are moved, it is a good hint that local
references are being misused.

Consider this example code:

jobject myJNIfunc1 ()
{
return env->NewObject ()
}

void myJNIfunc2 ()
{
jobject obj;

obj = myJNIfunc1 ()

..
..
}

When an object is created in myJNIfunc1, a local reference is created. This reference

immediately goes out of scope when JNIfunc1 returns. When obj is set in
myJNIfunc2, no local reference to obj exists, and the Garbage Collector can collect
it.

The reason for this is that the references to objects in the C code are not references
that the Garbage Collector normally follows. They are C or C++ type references
that are generated by the native compiler and are, therefore, not truly pointers into
the Java heap, which are what the Garbage Collector uses to find its root set of
objects. The native reference to a Java object is translated appropriately in a JNI
function. So, to make it work, the JVM creates a special stack frame when a JNI
function is entered, and reserves a set number of locations in the frame for any
pointers (local references) that the function might need. When the Garbage
Collector looks for root objects, it always looks at this area of the stack frame.

This is how it works:

1. When the call to NewObject returns, the JNI function NewObject has created a
local reference for you and stored it in the stack frame. Now, the stack has a
pointer to the object that the Garbage Collector can see (see Figure 6 on page
62).

Chapter 6. Understanding the Java Native Interface (JNI) 61

Local reference scope

Thread stack Java heap

Local reference area Object

myJNIFunc1
stack frame
Native data

Local reference area

myJNIFunc2 obj
stack frame
Other native data

Figure 6. Thread stack pointing to an object so that the Garbage Collector can see the object

2. In Figure 7, function myJNIFunc1 has returned, so the myJNIFunc1 stack frame

has been popped, and the local reference is therefore lost. A reference to Object,
to which obj can refer, exists on the stack (), and JNI functions can use this
reference to reach object on the Java heap. However, the stack contains no
reference that the Garbage Collector can see, that points to the Java heap object.
Therefore, the Java heap object is unreachable and eligible for garbage
collection.

Thread stack Java heap

myJNIFunc1
stack frame
popped Object

Local reference area

obj
myJNIFunc2
stack frame
Object

Other native data

Figure 7. Thread stack not pointing to an object so that the Garbage Collector cannot see the object

The use of obj now can be fatal because a garbage collection cycle can overwrite
the data to which obj refers. Clearly, a local reference operates like an automatic
reference. You cannot rely on it outside the function in which the local reference
was generated.

Another incorrect way to refer to an object outside the scope of its local reference
is:
static jclass cls = 0;

void myJNIfunc
{

62 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Local reference scope

...
if (cls == 0)
{
cls = (*env)->GetObjectClass(env, obj);
if (cls == 0)
{
... /* error */
}
}
// there’s no local ref to cls at this point
...
}

The error occurs because any local reference to cls goes out of scope on the first
exit from myJNIfunc. Therefore, the object could be garbage collected although the
static variable, which the Garbage Collector cannot examine, still contains the
reference. Again, no area in the stack directly refers to cls.

Summary of local references

Local references cannot be shared between separate functions or methods. Because
local references are like automatic variables, you cannot share them between
threads.

Local reference capacity

Occasionally, you might see a message such as:
"***ALERT: JNI local ref creation exceeded capacity

This message does not indicate an error. It is warning from the JVM that your
application has more local references than can be contained in the storage that you
first allocated for them. The local reference storage was described in the previous
section (see Figure 6 on page 62). The message suggests that you might want to
check your JNI code to see why you have many outstanding local references, and
decide whether it would be better if you managed them yourself (see “Manually
handling local references”). Normally, it is assumed that a function or method will
not hold many references at the same time. If, however, you have designed you
code to hold many references, you can ignore the message.

The JVM does not stop storing local references when this message appears; it
extends the storage capacity, as necessary. The execution of your application is not
affected in any way by this message, except for a small processing overhead. If
your application is designed this way and the message becomes annoying, or if
you are not willing to accept the overhead of recreating stack frames, JNI calls are
available that enable you to increase the capacity of the local reference storage.

The JNI specification does not set the local reference capacity of a JVM, nor does it,
require (or deny) use of this message. Therefore, this message might or might
appear. If it does, it might appear at different times for different JVMs.

Manually handling local references

You can control the storage capacity and freeing of local references, but you cannot
control whether they are created or not. You can create extra local references if you
want to. IBM strongly recommends that you do not create new local references in
an attempt to keep an object alive outside its automatic local reference scope. If
you do, it is almost certain that a window will remain through which data is lost
in a garbage collection cycle. Use global references instead.

Chapter 6. Understanding the Java Native Interface (JNI) 63

Local reference scope

Ensure that you do not refer to an object after you delete its local reference unless
you have a global reference to it. It might be good housekeeping to throw away a
local reference to an object when you have attached a global reference to it.

Handling global references

Use a global reference to refer to a JNI object where the scope of the local reference
is too restricted. You can use global references across threads and between
functions and methods. The Garbage Collector always finds objects that are
accessed through global references. Every “create global reference” call must have
a corresponding “free global reference” call. Otherwise, the global references
accumulate and cause a memory leak, because the objects that they reference are
never collected. The JVM does not (cannot) police or check global references.
Global references are completely under the JNI programmer’s control.

Leaks in global references eventually lead to an out-of-memory exception. They

can be quite difficult to solve, especially if you do not manage JNI exception
handling (see “Handling exceptions”).

Global reference capacity

The JNI specification does not define what the capacity of the JVM to hold global
references should be. The IBM Virtual Machine for Java has a fairly small limit, on
the order of 10³. Other JVMs have a much larger capacity or perhaps an unlimited
capacity (subject only to overriding process or platform sizes). This implementation
detail can cause problems. If you have a reference leak, it might not show up for a
very long time on some JVMs, although it will eventually. That same leak would
show up much more quickly on the IBM Virtual Machine for Java. This difference
can lead you to think mistakenly that your application works on the vendor’s JVM,
but not on the IBM Virtual Machine for Java.

Handling exceptions
Exceptions give you a way to handle errors in your application. Java has a clear
and consistent strategy for the handling of exceptions, but C/C++ code does not.
Therefore, the Java JNI does not throw an exception when it detects a fault because
it does not know how, or even if, the native code of an application can handle it.

The JNI specification requires exceptions to be deferred; it is the responsibility of

the native code to check whether an exception has occurred. A set of JNI APIs are
provided for this purpose. Note that a JNI function with a return code always sets
an error if an exception is pending. That is, you do not need to check for
exceptions if a JNI function returns “success”, but you do need to check for an
exception in an error case. If you do not check, the next time you go through the
JNI, the JNI code will detect a pending exception and throw it. Clearly, an
exception can be difficult to debug if it is thrown later and, possibly, at a different
point in the code from the point at which it was actually created.

Note: The JNI ExceptionCheck function might be a cheaper way of doing

exception checks than the ExceptionOccurred call, because the
ExceptionOccurred call has to create both an object to which you can refer,
and a local reference.

64 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Using the isCopy flag

Using the isCopy flag

Many of the JNI functions have a copy flag as a parameter (jboolean *isCopy). On
return, the flag is set to state TRUE if the data that is returned is a copy, or to
FALSE if that data is pinned. Whether to copy or pin data is an implementation
detail (see “Copying and pinning” on page 60).

The isCopy flag is an output parameter. You cannot set it, on entry to a JNI
function, to specify whether you want copy or pin. You do not have to use this
flag at all. You can pass NULL into the JNI function to indicate that you do not
care what the result is.

If the flag indicates a copy, a copy of the data has been taken. If the flag indicates
pinning, the data that is on the heap has been marked as referenced and pinned.
Pinned data cannot be moved in a compaction cycle, nor collected. If the data is
pinned, you effectively have a direct pointer to the data that is on the Java heap.

Clearly, you must free the space that is used for a copy of the data. Also, you must
free the data when it is pinned. By doing this, you tell the JVM that it can unpin
the data again. For example, the GetBooleanArrayElements call must always be
followed by a ReleaseBooleanArrayElements call, whatever the setting of the
isCopy flag.

The IBM Virtual Machine for Java generally uses the pin implementation. A
common mistake is to think that only copied data needs to be freed. If you assume
that you need free only data that is copied, the heap gradually becomes more and
more fragmented with bits of uncollectable, pinned data. Eventually, a failure
occurs.

Use of the isCopy flag is one of the JNI specification details in which you might
accidentally code to a JVM that prefers the copy method. Everything works
correctly if you accidentally free only copied data. If you swap to a pinning JVM
(or the JVM that you use changes its algorithm), code that was working fails if it is
not written to specification.

The JNI specification also states: “It is not possible to predict whether any given
JVM will copy or pin data on any particular JNI call”. If the flag indicates that a
copy has been used, another trap opens in which you must be sensitive to the
mode flag in the corresponding release call (see “Using the mode flag”).

Always call the Release<something> function after a function that is using the
isCopy flag.

Using the mode flag

This flag is used in Release<something>Array calls. For example:
ReleaseBooleanArrayElements
(JNIEnv *env, jbooleanArray array, jboolean *elems, jint mode);

You must use this flag correctly with respect to the setting of the corresponding
isCopy flag. You need to know what the isCopy flag is telling you (see “Using the
isCopy flag”). If the isCopy flag indicates that the returned data is pinned, any
preceding changes that you made to the data have been copied directly into the
Java heap, and the mode parameter is ignored.

Chapter 6. Understanding the Java Native Interface (JNI) 65

Using the mode flag

If, however, the isCopy flag indicates that the returned data is a copy, you must
use the mode flag to ensure that all changes that you made are actually actioned.

The possible settings of the mode flag are:

0 Update the data on the Java heap and free the space used by the copy.
JNI_COMMIT
Update the data on the Java heap and do not free the space used by the
copy.
JNI_ABORT
Do not update the data on the Java heap and free the space used by the
copy.

If you do not change the array data that you got as a copy, use JNI_ABORT
because it prevents unnecessary copying. If you do change the data, use 0 or
JNI_COMMIT to ensure that your changes actually happen, or use JNI_ABORT if
appropriate.
v If the isCopy flag indicates that the data is pinned, use the JNI_ABORT setting.
v If the isCopy flag indicates that the data is a copy, use the appropriate setting.

A generic way to use the isCopy and mode flags

Here is a generic way to use the isCopy and mode flags that works with all JVMs,
and ensures that changes are committed and leaks do not occur:
v Do not use the isCopy flag. Pass in null or 0.
v Always set the mode flag to zero.

A complicated use of these flags is necessary only if you want to do some special
optimization. This generic way does not release you from the need to think about
synchronization (see “Synchronization”).

Synchronization
When you get array elements through a Get<something>ArrayElements call, you
must think about synchronization. Whether or not the data is pinned, two entities
are involved in accessing the data:
v The Java code in which the data entity is declared and used
v The native code that accesses the data through the JNI

It is likely that these two entities are separate threads, in which case contention
occurs.

Consider the following scenario in a copying JNI implementation:

1. A Java program creates a large array and partially fills it with data.
2. The Java program calls native write function to write the data to a socket.
3. The JNI native that implements write() calls GetByteArrayElements.
4. GetByteArrayElements copies the contents of the array into a buffer, and
returns it to the native.
5. The JNI native starts writing a region from the buffer to the socket.
6. While the thread is busy writing, another thread (Java or native) runs and
copies more data into the array (outside the region that is being written).
7. The JNI native completes writing the region to the socket.

66 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Synchronization

8. The JNI native calls ReleaseByteArrayElements with mode 0, to indicate that it

has completed its operation with the array.
9. The VM, seeing mode 0, copies back the whole contents of the buffer to the
array, and overwrites the data that was written by the second thread.

In this particular scenario, note that the code would work with a pinning JVM.
Because each thread writes only its own bit of the data and the mode flag is
ignored, no contention occurs. This is another example of how code that is not
strictly to specification would work with one JVM implementation and not with
another. Although this scenario involves an array elements copy, you can see that
pinned data can also be corrupted when two threads access it at the same time.
Take care if the getter method says the data is pinned.

Be very careful about how you synchronize access to array elements. The JNI
interfaces allow you to access regions of Java entities to reduce problems in this
sort of interaction. In the above scenario, the thread that is writing the data should
write into its own region, and the thread that is reading the data should read only
its own region. This works whatever the JNI implementation is.

Debugging the JNI

Errors in JNI code can occur in several ways:
v The program crashes while it is executing a native method (most common).
v The program crashes some time after returning from the native method, often
during GC (less common).
v Bad JNI code causes deadlocks shortly after returning from a native method
(occasional).

If you think that you have a problem with the interaction between user-written
native code and the JVM (that is, a JNI problem), you can run diagnostics that help
you check the JNI transitions. to invoke these diagnostics, specify the -Xcheck:jni
option when you start up the JVM.

The -Xcheck:jni option activates a set of wrapper functions around the JNI
functions. The wrapper functions perform checks on the incoming parameters such
as:
v Whether the call and the call that initialized JNI are on the same thread.
v Whether the object parameters are valid objects.
v Whether local or global references refer to valid objects.
v The type matching, in get or set field operations.
v The validity of static and nonstatic field IDs.
v Whether strings are valid and non-null.
v Whether array elements are non-null.
v The types on array elements.

Output from jnichk appears on the standard output stream, and looks like:
JNI warning in FindClass: argument #2 is a malformed identifier ("invalid.name")
Warning occurred in com/ibm/examples/JNIExample.nativeMethod() [Ljava/lang/String];

The first line indicates:

v The error level (error, warning or advice).
v The JNI API in which the error was detected.

Chapter 6. Understanding the Java Native Interface (JNI) 67

Debugging the JNI

v An explanation of the problem.

The last line indicates what native method was being executed when the error was
detected.

You can specify additional sub-options by using -Xcheck:jni:<sub-option>[,<...>].

Useful suboptions are:
all check application and system classes
verbose
trace certain JNI functions and activities
trace
trace all JNI functions
nobounds
do not perform bounds checking on strings and arrays
nonfatal
do not exit when errors are detected
nowarn
do not display warnings
noadvice
do not display advice
novalist
do not check for va_list reuse (see note below)
pedantic
perform more thorough, but slower checks
help
print help information

The -Xcheck:jni option introduces some overhead because it is very thorough

when it validates input parameters.

Note: On some platforms, reusing a va_list in a second JNI call (for example,
when calling CallStaticVoidMethodV() twice with the same arguments)
causes the va_list to be corrupted and the second call to fail. To ensure that
the va_list is not corrupted, use the standard C macro va_copy() in the first
call. By default, -Xcheck:jni ensures that va_lists are not being reused. Use
the novalist suboption to disable this check only if your platform allows
reusing va_list without va_copy.

JNI checklist
Table 4. JNI checklist
Remember Outcome of nonadherence
Check your code to ensure that you do not Random crashes (depending on what you
accidentally lose local references. If in doubt, pick up in the overwritten object space)
create a global reference and ensure that you happen at random intervals.
delete that global reference when
appropriate.
Local references cannot be saved in global As above.
variables.

68 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 JNI checklist

Table 4. JNI checklist (continued)

Remember Outcome of nonadherence
Do not attempt to manipulate local As above. This problem might occur only in
references. small windows, very infrequently.
Ensure that every global reference created Memory leak. It might throw a native
has a path that deletes that global reference. exception if the global reference storage
overflows. It can be difficult to isolate.
Always check for exceptions (or return Unexplained exception in apparently perfect
codes) on return from a JNI function. code
Always handle a deferred exception
immediately you detect it.
Ensure that array and char elements are A small memory leak. It might fragment the
always freed. heap and cause other problems to occur
first.
Ensure that you use the isCopy and mode Memory leaks, heap fragmentation, or both.
flags correctly (see “A generic way to use
the isCopy and mode flags” on page 66).
When you update a Java object in native Memory corruption.
code, ensure synchronization of access.

Chapter 6. Understanding the Java Native Interface (JNI) 69

JNI checklist

70 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 7. Understanding Java Remote Method Invocation
Java Remote Method Invocation (Java RMI) enables you to create distributed Java
technology-based applications that can communicate with other such applications,
in which the methods of remote Java objects can be invoked from other Java
virtual machines, possibly on different hosts. RMI uses object serialization to
marshal and unmarshal parameters and does not truncate types, supporting true
object-oriented polymorphism.

The RMI implementation

The RMI implementation consists of three abstraction layers:
1. The Stub and Skeleton layer, which intercepts method calls made by the client
to the interface reference variable and redirects these calls to a remote RMI
service.
2. The Remote Reference layer below understands how to interpret and manage
references made from clients to the remote service objects.
3. The bottom layer is the Transport layer, which is based on TCP/IP connections
between machines in a network. It provides basic connectivity, as well as some
firewall penetration strategies.
On top of the TCP/IP layer, RMI uses a wire-level protocol called Java Remote
Method Protocol (JRMP), which works like this:
1. Objects that require remote behavior should extend the RemoteObject class,
typically through the UnicastRemoteObject subclass.
a. The UnicastRemoteObject subclass exports the remote object to make it
available for servicing incoming RMI calls.
b. Exporting the remote object creates a new server socket, which is bound to
a port number.
c. A thread is also created that listens for connections on that socket. The
Server is registered with a registry.
d. A client obtains details of connecting to the server from the registry.
e. Using the information from the registry, which includes the hostname and
the port details of the server’s listening socket, the client connects to the
server.
2. When the client issues a remote method invocation to the server, it creates a
TCPConnection object, which opens a socket to the server on the port specified
and sends the RMI header information and the marshalled arguments through
this connection using the StreamRemoteCall class.
3. On the server side:
a. When a client connects to the server socket, a new thread is assigned to
deal with the incoming call. The original thread can continue listening to
the original socket so that additional calls from other clients can be made.
b. The server reads the header information and creates a RemoteCall object of
its own to deal with unmarshalling the RMI arguments from the socket.
c. The serviceCall() method of the Transport class services the incoming call by
dispatching it
d. The dispatch() method calls the appropriate method on the object and
pushes the result back down the wire.

© Copyright IBM Corp. 2003, 2006 71

The RMI implementation

e. If the server object throws an exception, the server catches it and marshals it
down the wire instead of the return value.
4. Back on the client side:
a. The return value of the RMI is unmarshalled and returned from the stub
back to the client code itself.
b. If an exception is thrown from the server, that is unmarshalled and thrown
from the stub.

Thread pooling for RMI connection handlers

As explained in the previous section, on the server side, when a client connects to
the server socket, a new thread is forked to deal with the incoming call. The IBM
SDK implements thread pooling in the sun.rmi.transport.tcp.TCPTransport class.
Thread pooling is not enabled by default. Enable it with this command-line setting:
-Dsun.rmi.transport.tcp.connectionPool=true

(or use a non-null value instead of true).

With the connectionPool enabled, threads are created only if there is no thread in
the pool that can be reused. In the current implementation of the connection Pool,
the RMI connectionHandler threads are added to a pool and are never removed.
Because you cannot currently fine tune the number of threads in the pool, enabling
thread pooling is not recommended for applications that have only limited RMI
usage. Such applications have to live with these threads during the RMI off-peak
times as well. Applications that are mostly RMI intensive can benefit by enabling
the thread pooling because the connection handlers will be reused and there is no
overhead if these threads are created for every RMI call.

Understanding Distributed Garbage Collection (DGC)

The RMI subsystem implements reference counting-based Distributed Garbage
Collection (DGC) to provide automatic memory management facilities for remote
server objects.

The DGC abstraction is used for the server side of Distributed Garbage Collection.
This interface contains two methods: dirty() and clean(). A dirty() call is made
when a remote reference is unmarshalled in a client (the client is indicated by its
VMID). A corresponding clean() call is made when no more references to the
remote reference exist in the client. A failed dirty() call must schedule a strong
clean() call so that the call’s sequence number can be retained in order to detect
future calls received out of order by the distributed garbage collector.

A reference to a remote object is leased for a period of time by the client holding
the reference. The lease period starts when the dirty call is received. The client has
to renew the leases, by making additional dirty calls, on the remote references it
holds before such leases expire. If the client does not renew the lease before it
expires, the distributed garbage collector assumes that the remote object is no
longer referenced by that client.

DGCClient implements the client side of the RMI Distributed Garbage Collection
system. The external interface to DGCClient is the registerRefs() method. When a
LiveRef to a remote object enters the JVM, it must be registered with the
DGCClient to participate in distributed garbage collection. When the first LiveRef
to a particular remote object is registered, a dirty call is made to the server-side
distributed garbage collector for the remote object, which returns a lease

72 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Understanding Distributed Garbage Collection (DGC)

guaranteeing that the server-side DGC will not collect the remote object for a
certain period of time. While LiveRef instances to remote objects on a particular
server exist, the DGCClient periodically sends more dirty calls to renew its lease.
The DGCClient tracks the local availability of registered LiveRef instances using
phantom references. When the LiveRef instance for a particular remote object is
garbage collected locally, a clean() call is made to the server-side distributed
garbage collector, indicating that the server no longer needs to keep the remote
object alive for this client. The RenewCleanThread handles the asynchronous
client-side DGC activity by renewing the leases and making clean calls. So this
thread would wait until the next lease renewal or until any phantom reference is
queued for generating clean requests as necessary.

Debugging applications involving RMI

The list of exceptions that can occur when using RMI and their context is included
in the RMI Specification document at:

http://java.sun.com/j2se/1.4.2/docs/guide/rmi/spec/rmi-exceptions.html#3601

Properties settings that are useful for tuning, logging, or tracing RMI servers and
clients can be found at:

http://java.sun.com/j2se/1.4.2/docs/guide/rmi/javarmiproperties.html

Solutions to some common problems and answers to frequently asked questions

related to RMI and object serialization can be found at:

http://java.sun.com/j2se/1.4.2/docs/guide/rmi/faq.html

Network monitoring tools like netstat and tcpdump are useful for debugging RMI
problems at the network level.

Chapter 7. Understanding Java Remote Method Invocation 73

74 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Part 2. Submitting problem reports
This part describes how to gather data about a problem and how to send that data
to IBM service.

The chapters are:

v Chapter 8, “Overview of problem submission,” on page 77
v Chapter 9, “MustGather: Collecting the correct data to solve problems,” on page
79
v Chapter 10, “Advice about problem submission,” on page 83
v Chapter 11, “Submitting data with a problem report,” on page 85

© Copyright IBM Corp. 2003, 2006 75

76 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 8. Overview of problem submission
This chapter gives an overview of Java service and how you can send problem
reports.

How does IBM service Java?

Java is not a product that IBM sells; it is a supporting technology.

No traditional level 1, level 2, and level 3 service exists for Java. However, the Java
Technology Centre (JTC) maintains a Java L3 service team. Initially, your problem
report will probably go to the L2 service team for the product that you are using.
They will forward to the JTC if necessary. You can also send problem reports direct
to the JTC, as described in this part of the book.

Java L3 service is in Hursley (England), Bangalore (India), and Ottawa (Canada).

This geographical split is transparent to you for the purpose of submitting problem
reports. However, if you have to communicate directly with a service engineer, be
aware that:
v Hursley operates on GMT and uses Daylight Savings Time (DST).
v Bangalore operates on Indian Standard Time (IST), which is GMT + 4.5 and does
not use DST.
v Ottawa operates on Eastern Standard Time (EST), which is GMT –5 and uses
DST.

Submitting Java problem reports to IBM

Three methods are available:
v Create a Problem Management Report (PMR): If you are inside IBM, you can
do this directly. Your PMR will arrive on the Java PMR queue. If you are outside
IBM, your IBM representative will do this for you. As noted above, a PMR
might be created against the product that you are using. The product service
team will forward that PMR to the JTC if L3 Java analysis is required. If you are
outside IBM and would like access to the PMR system, ask your IBM
representative for details.
v By the web: This route is available only if you have access to the IBM intranet.
Go to http://eureka.hursley.ibm.com. This is a front end to the PMR system. Fill
in the form, and the server will create a PMR for you and queue it directly to
the Java queue.
v Direct contact: If you have direct contacts in the JTC, you can use them.
However, this is not the most desirable route because you are dependent on one
engineer, and that engineer might be absent for various reasons.

Java duty manager

A Java duty manager is available 24 hours per day, seven days per week. The duty
manager will call out staff if necessary. To call out the duty manager, you must
have a PMR number. Ask your IBM representative for the telephone number of the
Java duty manager.

© Copyright IBM Corp. 2003, 2006 77

78 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 9. MustGather: Collecting the correct data to solve
problems
This chapter gives general guidance about how to generate a problem report and
which data to include in it:
v “Before you submit a problem report”
v “Data to include”
v “Things to try” on page 80
v “Factors that affect JVM performance” on page 80
v “Test cases” on page 80
v “Performance problems – questions to ask” on page 81

See Part 3, “Problem determination,” on page 89 for specific information for your
platform.

Before you submit a problem report

To obtain a quicker response to your problems, you must try all the suitable
diagnostics and provide as much information as possible. By doing this, you
ensure that your initial submission contains the maximum information for IBM
support to track down your problem. If all the data is not there, you will get a
request for more information from IBM support and, therefore, increase the
turnaround time.

Data to include
The following checklist describes the information that you could include in your
problem report:
v Version information (-version from the command-line).
v Command-line options.
v Environment, non-default settings.
v OS and OS version.
v OS distribution (if applicable).
v Javadump.
v For crashes or hangs you should send a full system dump. These have the form
core.<date>.<time>.<pid>.dmp. For other situations, send a raw heap dump –
this is usually named heapdump.<date>.<time>.<pid>.dmp. (See Chapter 12,
“First steps in problem determination,” on page 91 for instructions on how to
enable these dumps.)

Note: For diagnosing suspected Java heap problems or Java memory leaks, the
raw heap dump files are more useful to IBM service than the system
dump files.
v XML file generated from the raw heap or a system dump, which is usually
named <original dump name>.xml. This file is used by the jdmpview tool for
dump analysis (see Chapter 15, “Linux problem determination,” or Chapter 16,
“Windows problem determination,” as appropriate, for instructions about how
to use the jextract command to create the xml file from the core dump).

© Copyright IBM Corp. 2003, 2006 79

Data to include

v CEEDUMP and transactional dumps for z/OS; see Chapter 17, “z/OS problem
determination.”
v Verbose output, where required.
v Data from any diagnostics that you run.
v Data from JIT diagnostics.
v Platform-specific data.
For information on how to gather this data, see Part 3, “Problem determination,”
on page 89.

Things to try
Refer to Chapter 12, “First steps in problem determination,” on page 91.

Factors that affect JVM performance

v Runtime flags.
v Environment variables.
v Set stack and heap size, Memory size (MAXDATA setting and -Xms, -Xmx ,
-Xss, and -Xoss settings). The values that are being used can be obtained by the
-verbose:sizes option.
v The search path to the class libraries (class path, mostly used classpath should
come first).
v Garbage collection .
v System limits.
v The quality of the code.
v System thread parameters.
v The machine configuration.
v I/O disk size and speed.
v Number and speed of CPUs.
v Network and network adapters number and speed.

Test cases
It is easier for IBM Service to solve a problem when a test case is available. Include
a test case with your problem report wherever possible.

If your application is too large or too complex to reduce into a test case, provide, if
possible, some sort of remote login so that IBM can see the problem in your
environment. (For example, install a VNC/Remote Desktop server and provide
logon details in the problem report.) This option is not very effective because IBM
has no control over the target JVM.

If no test case is available, analysis takes longer. IBM might send you
specially-instrumented JVMs that require the collection of the diagnostics data
while you are using them. This method often results in a series of interim fixes,
each providing progressively more instrumentation in the fault area. This operation
obviously increases the turnaround time of the problem. It might be quicker for
you to invest time and effort into a test case instead of having a costly cycle of
installing repeated JVM instrumentation onto your application.

80 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Performance problems – questions

Performance problems – questions to ask

When someone reports a performance problem, it is not enough only to gather
data and analyze it. Without knowing the characteristics of the performance
problem, you might waste time analyzing data that might not be related to the
problem that is being reported.

Always obtain and give as much detail as possible before you attempt to collect or
analyze data. Ask the following questions about the performance problem:
v Can the problem be demonstrated by running a specific test case or a sequence
of events?
v Is the slow performance intermittent?
v Does it become slow, then disappear for a while?
v Does it occur at particular times of the day or in relation to some specific
activity?
v Are all, or only some, operations slow?
v Which operation is slow? For example, elapsed time to complete a transaction,
or time to paint the screen?
v When did the problem start occurring?
v Has the condition existed from the time the system was first installed or went
into production?
v Did anything change on the system before the problem occurred (such as adding
more users or upgrading the software installed on the system)?
v If you have a client and server operation, can the problem be demonstrated
when run only locally on the server (network versus server problem)?
v Which vendor applications are running on the system, and are those
applications included in the performance problem? For example, the IBM
WebSphere Application Server?
v What effect does the performance problem have on the users?
v Which part of your analysis made you decide that the problem is caused by a
defect in the SDK?
v What hardware are you using? Which models; how many CPUs; what are the
memory sizes on the affected systems; what is the software configuration in
which the problem is occurring?
v Does the problem affect only a single system, or does it affect multiple systems?
v What are the characteristics of the Java application that has the problem?
v Which performance objectives are not being met?
v Did the objectives come from measurements on another system? If so, what was
the configuration of that system?

Two more ways in which you can help to get the problem solved more quickly are:
v Provide a clear written statement of a simple specific example of the problem,
but be sure to separate the symptoms and facts from the theories, ideas, and
your own conclusions. PMRs that report “the system is slow” require extensive
investigation to determine what you mean by slow, how it is measured, and
what is acceptable performance.
v Provide information about everything that has changed on the system in the
weeks before the problem first occurred. By missing something that changed,
you can block a possible investigation path and delay the solution of the
problem. If all the facts are available, the team can quickly reject those that are
not related.

Chapter 9. MustGather: Collecting the correct data to solve problems 81

Performance problems – questions

82 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 10. Advice about problem submission
This chapter describes how to submit a problem report, and explains the
information that you should include in that report:
v “Raising a problem report”
v “What goes into a problem report?”
v “Problem severity ratings”
v “Escalating problem severity” on page 84

Raising a problem report

See “Submitting Java problem reports to IBM” on page 77.

What goes into a problem report?

v All the data that you can collect; see below
v Contact numbers
v A brief description of your application and how Java is part of it
v An assessment of the severity of the problem

Problem severity ratings

Here is a guide to how to assess the severity of your problem. You can attach a
severity of 1, 2, 3, or 4 to your problem, where:
Sev 1
v In development: You cannot continue development.
v In service: Customers cannot use your product.
Sev 2
v In development: Major delays exist in your development.
v In service: Users cannot access a major function of your product.
Sev 3
v In development: Major delays exist in your development, but you have
temporary workarounds, or can continue to work on other parts of your
project.
v In service: Users cannot access minor functions of your product.
Sev 4
v In development: Minor delays and irritations exist, but good
workarounds are available.
v In service: Minor functions are affected or unavailable, but good
workarounds are available.

An artificial increase of the severity of your problem does not result in quicker
fixes. IBM queries your assessed severity if it seems too high. Problems that are
assessed at Sev 1 require maximum effort from the IBM Service team and,
therefore, 24-hour customer contact to enable Service Engineers to get more
information.

© Copyright IBM Corp. 2003, 2006 83

Escalating problem severity

Escalating problem severity

For problems below Sev 1, ask IBM Service to raise the severity if conditions
change. Do this, for example, when you discover that the problem is more
wide-ranging than you first thought, or if you are approaching a deadline and no
fix is forthcoming, or if you have waited too long for a fix.

For problems at Sev 1, you can escalate the severity higher into a ’critsit’. This
route is available only to customers who have service contracts and to internal
customers.

84 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 11. Submitting data with a problem report
Having followed the advice that is given in the previous two chapters, you
probably have a large amount of data to send to IBM in one or more files. This
chapter describes how to transmit data to IBM Java service. Data can be sent to
IBM in three ways:
v Java service maintain an anonymous ftp server, named ’javaserv’, for sending or
receiving data, This server is behind the IBM firewall and is therefore accessible
only inside IBM. Ask your SE to transmit the data.
v IBM also maintains an anonymous ftp public server. Java service prefer the use
of the javaserv ftp because the IBM server is not under the control of the IBM
Java Technology Center.
v You can also use an ftp server of your own if you want to. In your PMR, include
details of how to log on, and where the data is. Java service might need to send
data to you; for example an interim fix (see “When you will receive your fix” on
page 87). IBM uses the same server to send (PUT) data as Java service did to
receive (GET) it. If you use your own server, provide an address that Java
service can use to write to your server.

This chapter includes:

v “IBM internal only (javaserv)”
v “Sending files to IBM support” on page 86
v “Getting files from IBM support” on page 86
v “Using your own ftp server” on page 87
v “Compressing core files” on page 87
v “When you will receive your fix” on page 87

IBM internal only (javaserv)

ftp to javaserv like this:
ftp javaserv.hursley.ibm.com
1. Log in anonymously.
2. Change to directory pmrs and create a directory called 12345 (assuming your
PMR is 12345.xxx.xxx).
3. Change into 12345.
4. Set bin mode.
5. PUT your files.
Your output should look like this:
H:\crashes > ftp javaserv.hursley.ibm.com
Connected to fat.hursley.ibm.com.
220 fat.hursley.ibm.com FTP server (Version 4.1 Tue Sep 8 17:35:59 CDT 1998) ready.
User (fat.hursley.ibm.com:(none)): anonymous
331 Guest login ok, send ident as password.
Password:
230 Guest login ok, access restrictions apply.
ftp> cd pmrs
250 CWD command successful.
ftp> mkdir 12345
257 MKD command successful.
ftp> cd 12345

© Copyright IBM Corp. 2003, 2006 85

Submitting data with a problem report

250 CWD command successful.

ftp> bin
200 Type set to I.
ftp> put mytestcase

Sending files to IBM support

1. ftp to testcase.boulder.ibm.com
2. Change to <platform>/toibm. For example:
v For Windows, change to aix/toibm
v For Linux, change to linux/toibm
v For z/OS, change to zOS/toibm
3. Set binary mode.
4. PUT your file

Your output should look like this:

H:\website\AMD64 > ftp testcase.boulder.ibm.com
Connected to testcase.boulder.ibm.com.
220 testcase.boulder.ibm.com FTP server (Version wu-2.6.1(1) Thu Aug 16 13:39:44
MDT 2001) ready.
User (testcase.boulder.ibm.com:(none)): anonymous
331 Guest login ok, send your complete e-mail address as password.
Password: [email protected]
230-Please read the file README
230- it was last modified on Wed Oct 31 08:42:25 2001 - 29 days ago
230-Please read the file README_PS.TXT
230- it was last modified on Wed Oct 31 08:42:11 2001 - 29 days ago
230 Guest login ok, access restrictions apply.
ftp> cd windows
250 CWD command successful.
ftp> cd toibm
250 CWD command successful.
ftp> bin
ftp> put myfile

Files are kept on the server for only a short time, so notify IBM support
immediately after you have sent the files.

Getting files from IBM support

You can get files from IBM support in two ways:
1.
a. Point your browser to http://testcase.software.ibm.com
b. Click the TESTCASE SERVER.
c. Click the <platform>/fromibm icons. For example:
v For Windows, change to aix/fromibm
v For Linux, change to linux/fromibm
v For s/390, change to s390/fromibm
d. Click on the file that you want.
2. ftp to the server as above, and GET the data.
Remember that the files are on the server for only a short time.

86 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 submitting data with a problem report

Using your own ftp server

1. Dump the files and include the server address and log-in data in your problem
report.
2. Give read and write access to IBM service for this area of your server.

Compressing core files

Compress core files before sending them to IBM service. Use winzip or an
equivalent on Windows. and tar and gzip for Unix platforms. Use the testcase
server for sending the core files to IBM and do not send the zip file by e-mail.

When you will receive your fix

Java builds are performed daily at IBM. When an engineer has identified your
problem and produced a fix, that fix goes into the overnight build.

IBM periodically produces service refreshes of Java. After you have been notified
that your problem has been solved, you must obtain the next service refresh.

Service refreshes are fully supported by IBM. The version number in your JVM
(see Part 3, “Problem determination,” on page 89) identifies the service refresh level
that you are using. In some cases (for example when you urgently need a fix for a
Sev 1 problem), IBM service provides you with an overnight build as an electronic
fix (interim fix). An interim fix is a set of the Java binaries that contains a fix for
your problem. IBM support sends you this set of binaries to replace your original
binaries. Interim fixes are ftp’d to you through the same server that you used to
send in your problem data. Interim fixes are used to validate that a fix is good in
your environment, or to allow you to continue work on your project while waiting
for the next service refresh. Interim fixes are not supported by Java service,
because they have not been officially certified as Java-compatible. If you receive an
interim fix, you must get the next service refresh immediately it becomes available.

Chapter 11. Submitting data with a problem report 87

88 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Part 3. Problem determination
This part of the book is the problem determination guide. It is intended to help
you find the kind of fault you have and from there to do one or more of the
following tasks:
v Fix the problem
v Find a good workaround
v Collect the necessary data with which to generate a bug report to IBM

To use this part, go to the chapter that relates to your platform. If your application
runs on more than one platform and is exhibiting the same problem on them all,
go to the chapter about the platform to which you have the easiest access.

If you use the IBM WebSphere Application Server, the above guidance applies to
you, but read Chapter 13, “Working in a WebSphere Application Server
environment,” on page 93 first, because the platform-specific chapters discuss
subjects such as environment variables, and you will need the additional
information that is given in the chapter for the WebSphere Application Server.

A couple of JVM issues do not fit neatly into the platform model, and these have
their own chapters:
v Chapter 18, “Debugging the ORB,” on page 165
v Chapter 19, “NLS problem determination,” on page 179

If you have problems in these areas, check out the appropriate chapter in addition
to general diagnostics about your platform.

The chapters in this part are:

v Chapter 12, “First steps in problem determination,” on page 91
v Chapter 13, “Working in a WebSphere Application Server environment,” on page
93
v Chapter 14, “AIX problem determination,” on page 95
v Chapter 15, “Linux problem determination,” on page 127
v Chapter 16, “Windows problem determination,” on page 143
v Chapter 17, “z/OS problem determination,” on page 151
v Chapter 18, “Debugging the ORB,” on page 165
v Chapter 19, “NLS problem determination,” on page 179

© Copyright IBM Corp. 2003, 2006 89

90 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 12. First steps in problem determination
Ask these questions before going any further:
Have you enabled core dumps?
Core dumps are essential to enable IBM Service to debug a problem.
Depending on the platform, core dumps might not be enabled by default (see
Chapter 23, “JVM dump initiation,” on page 209 for details). To enable core
dumps, set the environment variable JAVA_DUMP_OPTS to:
JAVA_DUMP_OPTS="ONERROR(JAVADUMP,SYSDUMP) ONEXCEPTION(JAVADUMP,SYSDUMP),
ONDUMP(JAVADUMP)"

See Appendix E, “Environment variables,” on page 323 for details on setting

environment variables.
Can you reproduce the problem with the latest Service Refresh?
The problem might also have been fixed in a recent service refresh. Make sure
you are using the latest service refresh.
Are you using a supported Operating System (OS) with the latest patches
installed?
It is important to use an OS or distribution that supports the JVM and to have
the latest patches for operating system components. For example, upgrading
system libraries can solve problems. Moreover, later versions of system
software can provide a richer set of diagnostic information. (See platform
specific, ″Setting up and checking environment″ sections in chapters Chapter 13
through Chapter 17).
Have you installed the latest patches for other software that interacts with the
JVM? For example, the IBM WebSphere Application Server and DB2®.
The problem could be related to configuration of the JVM in a larger
environment and might have been solved already in a Fix Pack. The problem
could be related to native code executed by the JVM on behalf of other
software. If this is so, the issue might have been resolved in a later version of
any relevant software, for example DB2 or the WebSphere Application Server.
(See Chapter 13, “Working in a WebSphere Application Server environment,”
on page 93.)
Is the problem reproducible on the same machine?
Knowing that this defect occurs every time the described steps are taken, is
one of the most helpful things you can know about it and tends to indicate a
straightforward programming error. If, however, it occurs at alternate times, or
at one time in ten or a hundred, thread interaction and timing problems in
general would be much more likely.
Is the problem reproducible on another machine?
A problem that is not evident on another machine could help you find the
cause. A difference in hardware could make the problem disappear; for
example, the number of processors. Also, differences in the operating system
and application software installed might make a difference to the JVM. For
example, the visibility of a race condition in the JVM or a user Java application
might be influenced by the speed at which certain operations are performed by
the system.
Does the problem appear on multiple platforms?
If the problem appears only on one platform, it could be related to a

© Copyright IBM Corp. 2003, 2006 91

first steps in problem determination

platform-specific part of the JVM or native code used within a user

application. If the problem occurs on multiple platforms, the problem could be
related to the user Java application or a cross-platform part of the JVM; for
example, Java Swing API. Some problems might be evident only on particular
hardware; for example, Intel32. A problem on particular hardware could
possibly indicate a JIT problem.
Does turning off the JIT help?
If turning off the JIT prevents the problem, there might be a problem with the
JIT. This can also indicate a race condition within the user Java application
which surfaces only in certain conditions. If the problem is intermittent,
reducing the JIT compilation threshold to 0 might help reproduce the problem
more consistently. (See Chapter 27, “JIT problem determination,” on page 237.)
Have you tried reinstalling the JVM or other software and rebuilding relevant
application files?
Some problems occur from a damaged or invalid installation of the JVM or
other software. It is also possible that an application could have inconsistent
versions of binary files or packages. Inconsistency is particularly likely in a
development or testing environment and could potentially be solved by getting
a completely fresh build or installation.
Is the problem particular to a multiprocessor (or SMP) platform? If you are
working on a multiprocessor platform, does the problem still exist on a
uniprocessor platform?
This information is valuable to IBM Service.

92 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 13. Working in a WebSphere Application Server
environment
The WebSphere Application Server depends on the JVM and ORB technology.
Refer to Appendix A, “Compatibility tables,” on page 313 for WebSphere
Application Server/JVM/ORB compatibility tables.

For aspects of WebSphere Application Server JVM support (for example,

information on how to set JVM runtime parameters or how to get heapdumps
from the WebSphere environment) visit the WebSphere Application Server support
and service site at http://www.ibm.com/software/webservers/appserv/was/
support/. Click on the Technotes link and search for the topic that interests you to
find relevant documents.

© Copyright IBM Corp. 2003, 2006 93

94 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
|

| Chapter 14. AIX problem determination

This chapter describes problem determination on AIX.
v “Setting up and checking your AIX environment”
v “General debugging techniques” on page 97
v “Diagnosing crashes” on page 107
v “Debugging hangs” on page 109
v “Understanding memory usage” on page 111
v “Debugging performance problems” on page 119
v “MustGather information for AIX” on page 125
v “Getting AIX technical support” on page 125

Setting up and checking your AIX environment

Set up the right environment for the AIX JVM to run correctly during AIX
installation from either the installp image or the product with which it is packaged.

Note that the 64-bit JVM can work on a 32-bit kernel if the hardware is 64-bit. In
that case, you have to enable a 64-bit application environment using smitty:System
Environments -> Enable 64-bit Application Environment.

Occasionally the configuration process does not work correctly, or the environment
might be altered, affecting the operation of the JVM. In these conditions, you can
make checks to ensure that the JVM’s required settings are in place:
1. Check that the SDK and JRE files have been installed in the correct location and
that the correct permissions are set. See the User Guide for more information on
expected files and their location. Test the java and javac commands to ensure
they are executable.
The default installation directory is in /usr/java142_64. For developer kits
packaged with other products, the installation directory might be different;
consult your product documentation.
2. Ensure that the PATH environment variable points to the correct Java
executable (using which java), or that the application you are using is pointing
to the correct Java directory. You must include /usr/java142_64/jre/bin:/usr/
java142_64/bin in your PATH environment variable . If it is not present, add it
by using export PATH=/usr/java142_64/jre/bin:/usr/java142_64/bin:$PATH.
3. Ensure that the LANG environment variable is set to a supported locale. You
can find the language environment in use using echo $LANG, which should
report one of the supported locales as documented in the User Guide shipped
with the SDK.
4. Ensure that all the prerequisite AIX maintenance and APARs have been
installed. The prerequisite APARs and filesets will have been checked during an
install using smitty or installp. You can find the list of prerequisites in the
User Guide that is shipped with the SDK. Use lslpp -l to find the list of
current filesets. Use instfix -i -k <apar number> to test for the presence of an
APAR and instfix -i | grep _ML to find the installed maintenance level.

The ReportEnv tool, available from the Java service team, plugs into your JVM and
reports on the JVM environment in real time. Your JVM environment affects the

© Copyright IBM Corp. 2003, 2006 95

 AIX - setting up and checking your environment

operation of the JVM. ReportEnv reports on environment variables and

command-line parameters. It is a GUI tool, although it can be run without a GUI.
The GUI allows you to browse your environment and, to some extent, dynamically
change it. The tool also has a mechanism to generate reports to tell you the exact
state of your JVM environment. A screenshot of the tool is shown in “Setting up
and checking your Windows environment” on page 143. The ReportEnv tool is
available on request from [email protected].

| Enabling full AIX core files

| When a failure occurs, the most important diagnostic data to obtain is the process
| core file. The majority of the JVM settings are suitable by default, but, to ensure
| that this file is generated for the JVM on AIX, you must make a number of
| operating system settings.
| Operating system settings
| 1. To obtain full core files, set the following ulimit options:
| ulimit -c unlimited turn on corefiles with unlimited size
| ulimit -n unlimited allows an unlimited number of open file
| descriptors
| ulimit -d unlimited sets the user data limit to unlimited
| ulimit -f unlimited sets the file limit to unlimited

| You can display the current ulimit settings with:

| ulimit -a

| These values are the ″soft″ limit, and are applied for each user. These
| values cannot exceed the ″hard″ limit value. To display and change the
| ″hard″ limits, you can run the same ulimit commands using the
| additional -H flag.
| The ulimit -c value for the soft limit is ignored and the hard limit
| value is used so that the core file is generated. You can disable the
| generation of core files by using this Java command-line option:
| -Xdump:system:none
| 2. Set the following in smitty:
| a. Start smitty as root
| b. Go to System Environments -> Change/Show Characteristics of
| Operating System
| c. Set the Enable full CORE dump option to TRUE
| Alternatively, you can run chdev -l sys0 -a fullcore=’true’ as root.
| You can check the setting with lsattr -D -c sys -a fullcore -H,
| which should produce output similar to this:
| attribute deflt description user_settable
| fullcore false Enable full CORE dump True
| Java Virtual Machine settings
| The JVM settings should be in place by default, but you can check these
| settings using the following instructions.
| To check that the JVM is set to produce a core file when a failure occurs,
| run the following:
| java -Xdump:what

| which should include something like the following:

96 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 AIX - setting up and checking your environment

| -Xdump:system:
| events=gpf+abort,
| label=/u/cbailey/core.%Y%m%d.%H%M%S.%pid.dmp,
| range=1..0,
| priority=999,
| request=serial

| At least events=gpf must be set to generate a core file when a failure

| occurs.

| You can change and set options using the command-line option -Xdump,
| which is described in Chapter 24, “Using dump agents,” on page 213.
| Available disk space
| You must ensure that the disk space available is sufficient for the core file
| to be written to it. The core file is written to the directory specified in the
| label option. Up to 2 GB of free space might be required for 32-bit and
| over 6 GB for 64-bit system dumps. The Java process must have the correct
| permissions to write to that location.

General debugging techniques

Below is a short guide to the JVM provided diagnostic tools and AIX commands
that can be useful when diagnosing problems with the AIX JVM. In addition to the
information given below, you can obtain AIX publications from the IBM Web site.
Go to http://www.ibm.com/servers/aix/, click Library, and then choose the
documentation link for your platform. Of particular interest are:
v Performance management and tuning
v Programming for AIX
You might also find this Redbook helpful: ″C and C++ Application Development on
AIX″ (SG24-5674) , available from: http://www.redbooks.ibm.com.

Starting Javadumps in AIX

See Chapter 21, “Using Javadump,” on page 191.

Starting Heapdumps in AIX

See Chapter 22, “Using Heapdump,” on page 205.

AIX debugging commands

bindprocessor –q
This command lists the available processors.

bootinfo –K
This command shows if the 64–bit kernel is active.

bootinfo –y
This command shows whether the hardware in use is 32-bit or 64-bit.

dbx
The AIX debugger. Examples of use can be found throughout this chapter.

The Java 142 SDK also includes a dbx Plug-in for additional help debugging Java
applications. See “DBX Plug-in” on page 106 for more information.

Chapter 14. AIX problem determination 97

AIX - general debugging techniques

iostat
Use this command to determine if a system has an I/O bottleneck. The read and
write rate to all disks is reported. This tool is useful in determining if you need to
’spread out’ the disk workload across multiple disks. iostat also reports the same
CPU activity that vmstat does.

lsattr
This command details characteristics and values for devices in the system. To
obtain the type and speed of processor 0, use:
# lsattr -El proc0
state enable Processor state False
type PowerPC_POWER3 Processor type False
frequency 200000000 Processor Speed False

Processor 0 may not be available to you if you are using an LPAR. Use
bindprocessor -q to list the available processors.

lsconf
This command shows basic hardware and configuration details. For example:
System Model: IBM,7040-681
Machine Serial Number: 835A7AA
Processor Type: PowerPC_POWER4
Number Of Processors: 8
Processor Clock Speed: 1100 MHz
CPU Type: 64-bit
Kernel Type: 64-bit
LPAR Info: 5 JAVADEV1 - kukicha
Memory Size: 10240 MB
Good Memory Size: 10240 MB
Platform Firmware level: 3H041021
Firmware Version: IBM,RG041021_d78e05_s
Console Login: enable
Auto Restart: true
Full Core: true

Network Information
Host Name: bb1p5-1.hursley.ibm.com
IP Address: 9.20.136.92
Sub Netmask: 255.255.255.128
Gateway: 9.20.136.1
Name Server: 9.20.136.11
Domain Name: hursley.ibm.com

Paging Space Information

Total Paging Space: 512MB
Percent Used: 21%

Volume Groups Information

==============================================================================
rootvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk0 active 546 290 109..06..04..65..106
==============================================================================

INSTALLED RESOURCE LIST

The following resources are installed on the machine.

+/- = Added or deleted from Resource List.
* = Diagnostic support not available.

Model Architecture: chrp

Model Implementation: Multiple Processor, PCI bus

98 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 AIX - general debugging techniques

+ sys0 System Object

+ sysplanar0 System Planar
* vio0 Virtual I/O Bus
* vsa0 LPAR Virtual Serial Adapter
* vty0 Asynchronous Terminal
* pci12 U1.5-P2 PCI Bus
* pci11 U1.5-P2 PCI Bus
* pci10 U1.5-P2 PCI Bus
* pci9 U1.5-P1 PCI Bus
* pci14 U1.5-P1 PCI Bus
+ scsi0 U1.5-P1/Z2 Wide/Ultra-3 SCSI I/O Controller
+ hdisk0 U1.5-P1/Z2-A8 16 Bit LVD SCSI Disk Drive (73400 MB)
+ ses0 U1.5-P1/Z2-Af SCSI Enclosure Services Device
* pci8 U1.5-P1 PCI Bus
* pci7 U1.5-P1 PCI Bus
* pci6 U1.9-P2 PCI Bus
* pci5 U1.9-P2 PCI Bus
* pci4 U1.9-P2 PCI Bus
* pci13 U1.9-P2 PCI Bus
+ ent0 U1.9-P2-I3/E1 Gigabit Ethernet-SX PCI Adapter (14100401)
* pci3 U1.9-P1 PCI Bus
* pci2 U1.9-P1 PCI Bus
* pci1 U1.9-P1 PCI Bus
* pci0 U1.18-P1-H2 PCI Bus
+ L2cache0 L2 Cache
+ mem0 Memory
+ proc11 U1.18-P1-C3 Processor
+ proc12 U1.18-P1-C3 Processor
+ proc13 U1.18-P1-C3 Processor
+ proc16 U1.18-P1-C4 Processor
+ proc17 U1.18-P1-C4 Processor
+ proc18 U1.18-P1-C4 Processor
+ proc22 U1.18-P1-C4 Processor
+ proc23 U1.18-P1-C4 Processor

netpmon
This command uses the trace facility to obtain a detailed picture of network
activity during a time interval. It also displays process CPU statistics that show:
v The total amount of CPU time used by this process,
v The CPU usage for the process as a percentage of total time
v The total time that this process spent executing network-related code.
For example,
netpmon -o /tmp/netpmon.log; sleep 20; trcstop

is used to look for a number of things such as CPU usage by program, first level
interrupt handler, network device driver statistics, and network statistics by
program. Add the -t flag to produce thread level reports. The following output
shows the processor view from netpmon.
Process CPU Usage Statistics:
-----------------------------
Network
Process (top 20) PID CPU Time CPU % CPU %
----------------------------------------------------------
java 12192 2.0277 5.061 1.370
UNKNOWN 13758 0.8588 2.144 0.000
gil 1806 0.0699 0.174 0.174
UNKNOWN 18136 0.0635 0.159 0.000
dtgreet 3678 0.0376 0.094 0.000
swapper 0 0.0138 0.034 0.000
trcstop 18460 0.0121 0.030 0.000
sleep 18458 0.0061 0.015 0.000

Chapter 14. AIX problem determination 99

AIX - general debugging techniques

The adapter usage is shown here:

----------- Xmit ----------- -------- Recv ---------
Device Pkts/s Bytes/s Util QLen Pkts/s Bytes/s Demux
------------------------------------------------------------------------------
token ring 0 288.95 22678 0.0%518.498 552.84 36761 0.0222
...
DEVICE: token ring 0
recv packets: 11074
recv sizes (bytes): avg 66.5 min 52 max 1514 sdev 15.1
recv times (msec): avg 0.008 min 0.005 max 0.029 sdev 0.001
demux times (msec): avg 0.040 min 0.009 max 0.650 sdev 0.028
xmit packets: 5788
xmit sizes (bytes): avg 78.5 min 62 max 1514 sdev 32.0
xmit times (msec): avg 1794.434 min 0.083 max 6443.266 sdev 2013.966

The following shows the java extract:

PROCESS: java PID: 12192
reads: 2700
read sizes (bytes): avg 8192.0 min 8192 max 8192 sdev 0.0
read times (msec): avg 184.061 min 12.430 max 2137.371 sdev 259.156
writes: 3000
write sizes (bytes): avg 21.3 min 5 max 56 sdev 17.6
write times (msec): avg 0.081 min 0.054 max 11.426 sdev 0.211

To see a thread level report, add the -t as shown here.

netpmon -O so -t -o /tmp/netpmon_so_thread.txt; sleep 20; trcstop

The extract below shows the thread output:

THREAD TID: 114559
reads: 9
read sizes (bytes): avg 8192.0 min 8192 max 8192 sdev 0.0
read times (msec): avg 988.850 min 19.082 max 2106.933 sdev 810.518
writes: 10
write sizes (bytes): avg 21.3 min 5 max 56 sdev 17.6
write times (msec): avg 0.389 min 0.059 max 3.321 sdev 0.977

You can also request that less information is gathered. For example to look at
socket level traffic use the ″-O so″ option:
netpmon -O so -o /tmp/netpmon_so.txt; sleep 20; trcstop

netstat
Use this command with the –m option to look at mbuf memory usage, which will
tell you something about socket and network memory usage. By default, the
extended netstat statistics are turned off in /etc/tc.net with the line:
/usr/sbin/no -o extendednetstats=0 >>/dev/null 2>&1

To turn on these statistics, change to extendednetstats=1 and reboot. You can also
try to set this directly with no. When using netstat -m, pipe to page because the
first information is some of the most important:
67 mbufs in use:
64 mbuf cluster pages in use
272 Kbytes allocated to mbufs
0 requests for mbufs denied
0 calls to protocol drain routines
0 sockets not created because sockthresh was reached

100 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 AIX - general debugging techniques

-- At the end of the file:

Streams mblk statistic failures:
0 high priority mblk failures
0 medium priority mblk failures
0 low priority mblk failures

Use netstat -i <interval to collect data> to look at network usage and

possible dropped packets.

nmon
Nmon is a free interactive software tool that gives much of the same information
as topas, but saves the information to a file in Lotus 123 and Excel formats. The
download site is www.ibm.com/servers/esdd/articles/analyze_aix/. The
information that is collected includes CPU, disk, network, adapter statistics, kernel
counters, memory, and the ’top’ process information.

no
Use the no command to configure network attributes. The no command sets or
displays current network attributes in the kernel.

For example, to see the size of the wall use:

# no -a | grep wall
thewall = 524288
# no -o thewall =
1000000

The wall is the maximum amount of memory assigned to the network memory
buffer.

ps
The Process Status (ps) is used to monitor:
v A process.
v Whether the process is still consuming CPU cycles.
v Which threads of a process are still running.

To invoke ps to monitor a process, type:

ps -fp <PID>

Your output should be:

UID PID PPID C STIME TTY TIME CMD
user12 29730 27936 0 21 Jun - 12:26 java StartCruise

Where
UID
The user ID of the process owner. The login name is printed under the -f flag.
PPID
The Parent Process ID.
PID
The Process ID.
C CPU utilization, incremented each time the system clock ticks and the process
is found to be running. The value is decayed by the scheduler by dividing it
by 2 every second. For the sched_other policy, CPU utilization is used in
determining process scheduling priority. Large values indicate a CPU intensive

Chapter 14. AIX problem determination 101

AIX - general debugging techniques

process and result in lower process priority whereas small values indicate an
I/O intensive process and result in a more favorable priority.
STIME
The start time of the process, given in hours, minutes, and seconds. The start
time of a process begun more than twenty-four hours before the ps inquiry is
executed is given in months and days.
TTY
The controlling workstation for the process.
TIME
The total execution time for the process.
CMD
The full command name and its parameters.

To see which threads are still running, type:

ps -mp <PID> -o THREAD

Your output should be:

USER PID PPID TID ST CP PRI SC WCHAN F TT BND COMMAND
user12 29730 27936 - A 4 60 8 * 200001 pts/10 0 java StartCruise
- - - 31823 S 0 60 1 e6007cbc 8400400 - 0 -
- - - 44183 S 0 60 1 e600acbc 8400400 - 0 -
- - - 83405 S 2 60 1 50c72558 400400 - 0 -
- - - 114071 S 0 60 1 e601bdbc 8400400 - 0 -
- - - 116243 S 2 61 1 e601c6bc 8400400 - 0 -
- - - 133137 S 0 60 1 e60208bc 8400400 - 0 -
- - - 138275 S 0 60 1 e6021cbc 8400400 - 0 -
- - - 140587 S 0 60 1 e60225bc 8400400 - 0 -

Where
USER
The user name of the person running the process.
TID
The Kernel Thread ID of each thread.
ST
The state of the thread:
O Nonexistent.
R Running.
S Sleeping.
W Swapped.
Z Canceled.
T Stopped.
CP
CPU utilization of the thread.
PRI
Priority of the thread.
SC
Suspend count.
ARCHON
Wait channel.

102 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 AIX - general debugging techniques

F Flags.
TAT
Controlling terminal.
BAND
CPU to which thread is bound.

For more details, see the manual page for ps.

sar
Use the sar command to check the balance of CPU usage across multiple CPU’s. In
this example below, two samples are taken every five seconds on a 2-processor
system that is 80% utilized.
# sar -u -P ALL 5 2

AIX aix4prt 0 5 000544144C00 02/09/01

15:29:32 cpu %usr %sys %wio %idle

15:29:37 0 34 46 0 20
1 32 47 0 21
- 33 47 0 20
15:29:42 0 31 48 0 21
1 35 42 0 22
- 33 45 0 22

Average 0 32 47 0 20
1 34 45 0 22
- 33 46 0 21

svmon
This command captures snapshots of virtual memory. Using svmon to take
snapshots of the memory usage of a process over regular intervals allows you to
monitor memory usage. The following usage of svmon generates regular snapshots
of a process memory usage and writes the output to a file:
svmon -P [process id] -m -r -i [interval] > output.file

Gives output like:

Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd
25084 AppS 78907 1570 182 67840 N Y
Vsid Esid Type Description Inuse Pin Pgsp Virtual Addr Range
2c7ea 3 work shmat/mmap 36678 0 0 36656 0..65513
3c80e 4 work shmat/mmap 7956 0 0 7956 0..65515
5cd36 5 work shmat/mmap 7946 0 0 7946 0..65517
14e04 6 work shmat/mmap 7151 0 0 7151 0..65519
7001c d work shared library text 6781 0 0 736 0..65535
0 0 work kernel seg 4218 1552 182 3602 0..22017 :
65474..65535
6cb5a 7 work shmat/mmap 2157 0 0 2157 0..65461
48733 c work shmat/mmap 1244 0 0 1244 0..1243
cac3 - pers /dev/hd2:176297 1159 0 - - 0..1158
54bb5 - pers /dev/hd2:176307 473 0 - - 0..472
78b9e - pers /dev/hd2:176301 454 0 - - 0..453
58bb6 - pers /dev/hd2:176308 254 0 - - 0..253
cee2 - work 246 17 0 246 0..49746
4cbb3 - pers /dev/hd2:176305 226 0 - - 0..225
7881e - pers /dev/e2axa702-1:2048 186 0 - - 0..1856
68f5b - pers /dev/e2axa702-1:2048 185 0 - - 0..1847
28b8a - pers /dev/hd2:176299 119 0 - - 0..118
108c4 - pers /dev/e2axa702-1:1843 109 0 - - 0..1087
24b68 f work shared library data 97 0 0 78 0..1470
64bb9 - pers /dev/hd2:176311 93 0 - - 0..92
74bbd - pers /dev/hd2:176315 68 0 - - 0..67

Chapter 14. AIX problem determination 103

AIX - general debugging techniques

3082d 2 work process private 68 1 0 68 65287..65535

10bc4 - pers /dev/hd2:176322 63 0 - - 0..62
50815 1 pers code,/dev/hd2:210969 9 0 - - 0..8
44bb1 - pers /dev/hd2:176303 7 0 - - 0..6
7c83e - pers /dev/e2axa702-1:2048 4 0 - - 0..300
34a6c a mmap mapped to sid 44ab0 0 0 - -
70b3d 8 mmap mapped to sid 1c866 0 0 - -
5cb36 b mmap mapped to sid 7cb5e 0 0 - -
58b37 9 mmap mapped to sid 1cb66 0 0 - -
1c7c7 - pers /dev/hd2:243801 0 0 - -

in which:
Vsid
Segment ID
Esid
Segment ID: corresponds to virtual memory segment. The Esid maps to the
Virtual Memory Manager segments. By understanding the memory model that
is being used by the JVM, you can use these values to determine whether you
are allocating or committing memory on the native or Java heap.
Type
Identifies the type of the segment:
pers Indicates a persistent segment.
work Indicates a working segment.
clnt Indicates a client segment.
mmap Indicates a mapped segment. This is memory allocated using mmap in
a large memory model program.
Description
If the segment is a persistent segment, the device name and i-node number of
the associated file are displayed.
If the segment is a persistent segment and is associated with a log, the string
log is displayed.
If the segment is a working segment, the svmon command attempts to
determine the role of the segment:
kernel
The segment is used by the kernel.
shared library
The segment is used for shared library text or data.
process private
Private data for the process.
shmat/mmap
Shared memory segments that are being used for process private data,
because you are using a large memory model program.
Inuse
The number of pages in real memory from this segment.
Pin
The number of pages pinned from this segment.
Pgsp
The number of pages used on paging space by this segment. This value is
relevant only for working segments.
104 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 AIX - general debugging techniques

Addr Range
The range of pages that have been allocated in this segment. Addr Range
displays the range of pages that have been allocated in each segment, whereas
Inuse displays the number of pages that have been committed. For instance,
Addr Range might detail more pages than Inuse because pages have been
allocated that are not yet in use.

tprof
The tprof command reports CPU usage for individual programs and the system as
a whole. This command is a useful tool for anyone with a Java program that might
be CPU-bound and who wants to know which sections of the program are most
heavily using the CPU.

The tprof command can charge CPU time to object files, processes, threads,
subroutines (user mode, kernel mode and shared library) and even to source lines
of programs or individual instructions. Charging CPU time to subroutines is called
profiling and charging CPU time to source program lines is called micro-profiling.

topas
Topas is a useful graphical interface that will give you immediate information
about system activity. The screen looks like this:
Topas Monitor for host: aix4prt EVENTS/QUEUES FILE/TTY
Mon Apr 16 16:16:50 2001 Interval: 2 Cswitch 5984 Readch 4864
Syscall 15776 Writech 34280
Kernel 63.1 |################## | Reads 8 Rawin 0
User 36.8 |########## | Writes 2469 Ttyout 0
Wait 0.0 | | Forks 0 Igets 0
Idle 0.0 | | Execs 0 Namei 4
Runqueue 11.5 Dirblk 0
Network KBPS I-Pack O-Pack KB-In KB-Out Waitqueue 0.0
lo0 213.9 2154.2 2153.7 107.0 106.9
tr0 34.7 16.9 34.4 0.9 33.8 PAGING MEMORY
Faults 3862 Real,MB 1023
Disk Busy% KBPS TPS KB-Read KB-Writ Steals 1580 % Comp 27.0
hdisk0 0.0 0.0 0.0 0.0 0.0 PgspIn 0 % Noncomp 73.9
PgspOut 0 % Client 0.5
Name PID CPU% PgSp Owner PageIn 0
java 16684 83.6 35.1 root PageOut 0 PAGING SPACE
java 12192 12.7 86.2 root Sios 0 Size,MB 512
lrud 1032 2.7 0.0 root % Used 1.2
aixterm 19502 0.5 0.7 root NFS (calls/sec) % Free 98.7
topas 6908 0.5 0.8 root ServerV2 0
ksh 18148 0.0 0.7 root ClientV2 0 Press:
gil 1806 0.0 0.0 root ServerV3 0 "h" for help

trace
This command captures a sequential flow of time-stamped system events. The
trace is a valuable tool for observing system and application execution. While
many of the other tools provide high level statistics such as CPU and I/O
utilization, the trace facility helps expand the information about where the events
happened, which process is responsible, when the events took place, and how they
are affecting the system. The curt postprocessing tool can extract information from
the trace. It provides statistics on CPU utilization and process and thread activity.
Another postprocessing tool is splat, the Simple Performance Lock Analysis Tool.
This tool is used to analyze lock activity in the AIX kernel and kernel extensions
for simple locks.

truss
This command traces a process’s system calls, dynamically loaded user-level
function calls, received signals, and incurred machine faults.

Chapter 14. AIX problem determination 105

 AIX - general debugging techniques

vmstat
Use this command to give multiple statistics on the system. The vmstat command
reports statistics about kernel threads in the run and wait queue, memory paging,
interrupts, system calls, context switches, and CPU activity. The CPU activity is
percentage breakdown of user mode, system mode, idle time, and waits for disk
I/O.

The general syntax of this command is:

vmstat <time_between_samples_in_seconds> <number_of_samples> -t

A typical output looks like this:

kthr memory page faults cpu time
----- ----------- ------------------------ ------------ ----------- --------
r b avm fre re pi po fr sr cy in sy cs us sy id wa hr mi se
0 0 45483 221 0 0 0 0 1 0 224 326 362 24 7 69 0 15:10:22
0 0 45483 220 0 0 0 0 0 0 159 83 53 1 1 98 0 15:10:23
2 0 45483 220 0 0 0 0 0 0 145 115 46 0 9 90 1 15:10:24

In this output, look for:

v Columns r (run queue) and b (blocked) starting to go up, especially above 10.
This rise usually indicates that you have too many processes competing for
CPU.
v Values in the pi, po (page in/out) columns at non-zero, possibly indicating that
you are paging and need more memory. It might be possible that you have the
stack size set too high for some of your JVM instances.
v cs (contact switches) going very high compared to the number of processes. You
might need to tune the system with vmtune.
v In the cpu section, us (user time) indicating the time being spent in programs.
Assuming Java is at the top of the list in tprof, you need to tune the Java
application. In the cpu section, if sys (system time) is higher than expected, and
you still have id (idle) time left, you might have lock contention. Check the tprof
for lock–related calls in the kernel time. You might want to try multiple
instances of the JVM.
v The -t flag, which adds the time for each sample at the end of the line.

| DBX Plug-in
| The Plug-in for the AIX DBX debugger gives DBX users enhanced features when
| working on Java processes or core files generated by Java processes.

| The Plug-in requires a version of DBX that supports the Plug-in interface. Use the
| DBX command pluginload to find out whether your version of DBX has this
| support. All supported AIX versions include this support.

| To enable the Plug-in, use the DBX command pluginload:

| pluginload /usr/java142_64/jre/bin/libdbx_j9.so

| You can also set the DBX_PLUGIN_PATH environment variable to

| /usr/java142_64/jre/bin. DBX automatically loads any Plug-ins found in the path
| given.

| The commands available after loading the Plug-in can be listed by running:
| plugin java help

| from the DBX prompt.

106 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 AIX - general debugging techniques

| You can also use DBX to debug your native JNI code by specifying the full path to
| the java program as follows:
| dbx /usr/java142_64/jre/bin/java

| Under DBX, issue the command:

| (dbx) run <MyAppClass>

| Before you start working with DBX, you must set the $java variable. Start DBX and
| use the dbx set subcommand. Setting this variable causes DBX to ignore the
| non-breakpoint traps generated by the JIT. You can also use a pre-edited command
| file by launching DBX with the -c option to specify the command file:
| dbx -c .dbxinit

| where .dbxinit is the default command file.

| Although the DBX Plug-in is supplied as part of the SDK, it is not supported.
| However, IBM will accept bug reports.

Diagnosing crashes
A crash can occur because of a fault in the JVM or because of a fault in native
(JNI) code being run in the Java process. Therefore, if the application does not
include any JNI code and does not use any third-party packages that have JNI
code (for example, JDBC application drivers), the fault must be in the JVM, and
should be reported to IBM Support through the normal process.

If a crash occurs, you should gather some basic documents. These documents
either point to the problem that is in the application or third party package JNI
code, or help the IBM JVM Support team to diagnose the fault.

Documents to gather
When a crash takes place, the following diagnostic data is required to help
diagnose the problem:
| v The output of stackit.sh run against the core file located as specified by the
| label field of -Xdump:what. stackit.sh is a script that runs a dbx session and is
| available from your Support Representative or from [email protected].
| v The output of Jjextract run against the core file:
| jextract [core file]

| and collect the core.{date}.{time}.{pid}.dmp.zip output. See “jextract” on page 223

| for details about jextract.
| v Collect the javadump file. This file should be in the location detailed against the
| label field in -Xdump:what for javadumps
| v Collect any stdout and stderr output generated by the Java process
| v Collect the system error report:
| errpt -a > errpt.out
| The above steps should leave you with the following files:
v stackit.out
v core.{date}.{time}.{pid}.dmp.zip
v javacore.{date}.{time}.{pid}.txt
v Snap<seq>.<date>.<time>.<pid>.trc
v errpt.out
Chapter 14. AIX problem determination 107
 AIX - diagnosing crashes

v stderr/stdout files

Locating the point of failure

If a stack trace is present, examining the function running at the point of failure
should give you a good indication of the code that caused the failure, and whether
the failure is in IBM’s JVM code, or is caused by application or third party JNI
code. If dbx or stackit.sh produce no stack trace, the crash usually has two possible
causes:
v A stack overflow of the native AIX stack.
| v Java code is running (either JIT compiled or interpreted)

A failing instruction reported by dbx or stackit.sh as ″stwu″ indicates that there

might have been a stack overflow. For example:
Segmentation fault in strlen at 0xd01733a0 ($t1)
0xd01733a0 (strlen+0x08) 88ac0000 stwu r1,-80(r1)

You can check for the first cause by using the dbx command thread info and
looking at the stack pointer, stack limit, and stack base values for the current
thread. If the value of the stack pointer is close to that of the stack base, you might
have had a stack overflow. A stack overflow occurs because the stack on AIX
grows from the stack limit downwards towards the stack base. If the problem is a
native stack overflow, you can solve the overflow by increasing the size of the
native stack from the default size of 400K using the command-line option
-Xss<size>. You are recommended always to check for a stack overflow, regardless
of the failing instruction. To reduce the possibility of a JVM crash, you must set an
appropriate native stack size when you run a Java program using a lot of native
stack.
(dbx) thread info 1
thread state-k wchan state-u k-tid mode held scope function
>$t1 run running 85965 k no sys oflow

general:
pthread addr = 0x302027e8 size = 0x22c
vp addr = 0x302057e4 size = 0x294
thread errno = 0
start pc = 0x10001120
joinable = yes
pthread_t = 1
scheduler:
kernel =
user = 1 (other)
event :
event = 0x0
cancel = enabled, deferred, not pending
stack storage:
base = 0x2df23000
size = 0x1fff7b0
limit = 0x2ff227b0
sp = 0x2df2cc70

For the second cause, currently dbx (and therefore stackit.sh) does not understand
the structure of the JIT and Interpreter stack frames, and is not capable of
generating a stack trace from them. The javadump, however, does not suffer from
this limitation and can be used to examine the stack trace.

108 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 AIX - debugging hangs

Debugging hangs
The JVM is hanging if the process is still present but is not responding in some
sense. This lack of response can be caused because:
v The process has come to a complete halt because of a deadlock condition
v The process has become caught in an infinite loop
v The process is running very slowly

AIX deadlocks
For an explanation of deadlocks and how the Javadump tool is used to diagnose
them, see “Locks, monitors, and deadlocks (LOCKS)” on page 193.

If the process is not taking up any CPU time, it is deadlocked. Use the ps -fp
[process id] command to investigate whether the process is still using CPU time.
The ps command is described in “AIX debugging commands” on page 97. For
example:
$ ps -fp 30450
UID PID PPID C STIME TTY TIME CMD
root 30450 32332 2 15 May pts/17 12:51 java ...

If the value of ’TIME’ increases over the course of a few minutes, the process is
still using the CPU and is not deadlocked.

AIX busy hangs

If there is no deadlock between threads, consider other reasons why threads are
not carrying out useful work. Usually, this state occurs for one of the following
reasons:
1. Threads are in a ’wait’ state waiting to be ’notified’ of work to be done.
2. Threads are in explicit sleep cycles.
3. Threads are in I/O calls waiting to do work.
The first two reasons imply a fault in the Java code, either that of the application,
or that of the standard class files included in the SDK.

The third reason, where threads are waiting (for instance, on sockets) for I/O,
requires further investigation. Has the process at the other end of the I/O failed?
Do any network problems exist?

To see how the javadump tool is used to diagnose loops, see “Locks, monitors, and
deadlocks (LOCKS)” on page 193. If you cannot diagnose the problem from the
javadump and if the process still seems to be using processor cycles, either it has
entered an infinite loop or it is suffering from very bad performance. Using ps -mp
[process id] -o THREAD allows individual threads in a particular process to be
monitored to determine which threads are using the CPU time. If the process has
entered an infinite loop, it is likely that a small number of threads will be using
the time. For example:
$ ps -mp 43824 -o THREAD
USER PID PPID TID ST CP PRI SC WCHAN F TT BND COMMAND
wsuser 43824 51762 - A 66 60 77 * 200001 pts/4 - java ...
- - - 4021 S 0 60 1 22c4d670 c00400 - - -
- - - 11343 S 0 60 1 e6002cbc 8400400 - - -
- - - 14289 S 0 60 1 22c4d670 c00400 - - -
- - - 14379 S 0 60 1 22c4d670 c00400 - - -
...
- - - 43187 S 0 60 1 701e6114 400400 - - -

Chapter 14. AIX problem determination 109

AIX - debugging hangs

- - - 43939 R 33 76 1 20039c88 c00000 - - -

- - - 50275 S 0 60 1 22c4d670 c00400 - - -
- - - 52477 S 0 60 1 e600ccbc 8400400 - - -
...
- - - 98911 S 0 60 1 7023d46c 400400 - - -
- - - 99345 R 33 76 0 - 400000 - - -
- - - 99877 S 0 60 1 22c4d670 c00400 - - -
- - - 100661 S 0 60 1 22c4d670 c00400 - - -
- - - 102599 S 0 60 1 22c4d670 c00400 - - -
...

Those threads with the value ’R’ under ’ST’ are in the ’runnable’ state, and
therefore are able to accumulate processor time. What are these threads doing? The
output from ps shows the TID (Kernel Thread ID) for each thread. This can be
mapped to the Java thread ID using dbx. The output of the dbx thread command
gives an output of the form of:
thread state-k wchan state-u k-tid mode held scope function
$t1 wait 0xe60196bc blocked 104099 k no sys _pthread_ksleep
>$t2 run blocked 68851 k no sys _pthread_ksleep
$t3 wait 0x2015a458 running 29871 k no sys pthread_mutex_lock
...
$t50 wait running 86077 k no sys getLinkRegister
$t51 run running 43939 u no sys reverseHandle
$t52 wait running 56273 k no sys getLinkRegister
$t53 wait running 37797 k no sys getLinkRegister
$t60 wait running 4021 k no sys getLinkRegister
$t61 wait running 18791 k no sys getLinkRegister
$t62 wait running 99345 k no sys getLinkRegister
$t63 wait running 20995 k no sys getLinkRegister

By matching the TID value from ps to the k-tid value from the dbx thread
command, you can see that the currently running methods in this case are
reverseHandle and getLinkRegister.

Now you can use dbx to generate the C thread stack for these two threads using
the dbx thread command for the corresponding dbx thread numbers ($tx). To
obtain the full stack trace including Java frames, map the dbx thread number to
the threads pthread_t value, which is listed by the Javadump file, and can be
obtained from the ExecEnv structure for each thread using the Dump Viewer. Do
this with the dbx command thread info [dbx thread number], which produces an
output of the form:
thread state-k wchan state-u k-tid mode held scope function
$t51 run running 43939 u no sys reverseHandle
general:
pthread addr = 0x220c2dc0 size = 0x18c
vp addr = 0x22109f94 size = 0x284
thread errno = 61
start pc = 0xf04b4e64
joinable = yes
pthread_t = 3233
scheduler:
kernel =
user = 1 (other)
event :
event = 0x0
cancel = enabled, deferred, not pending
stack storage:
base = 0x220c8018 size = 0x40000
limit = 0x22108018
sp = 0x22106930

110 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 AIX - debugging hangs

Showing that the TID value from ps (k-tid in dbx) corresponds to dbx thread
number 51, which has a pthread_t of 3233. Looking for the pthread_t in the
Javadump file, you now have a full stack trace:
"Worker#31" (TID:0x36288b10, sys_thread_t:0x220c2db8) Native Thread State:
ThreadID: 00003233 Reuse: 1 USER SUSPENDED Native Stack Data : base: 22107f80
pointer 22106390 used(7152) free(250896)
----- Monitors held -----
java.io.OutputStreamWriter@3636a930
com.ibm.servlet.engine.webapp.BufferedWriter@3636be78
com.ibm.servlet.engine.webapp.WebAppRequestDispatcher@3636c270
com.ibm.servlet.engine.srt.SRTOutputStream@36941820
com.ibm.servlet.engine.oselistener.nativeEntry.NativeServerConnection@36d84490 JNI pinning lock

----- Native stack -----

_spin_lock_global_common pthread_mutex_lock - blocked on Heap Lock

sysMonitorEnterQuicker sysMonitorEnter unpin_object unpinObj
jni_ReleaseScalarArrayElements jni_ReleaseByteArrayElements
Java_com_ibm_servlet_engine_oselistener_nativeEntry_NativeServerConnection_nativeWrite

------ Java stack ------ () prio=5

com.ibm.servlet.engine.oselistener.nativeEntry.NativeServerConnection.write(Compiled Code)
com.ibm.servlet.engine.srp.SRPConnection.write(Compiled Code)
com.ibm.servlet.engine.srt.SRTOutputStream.write(Compiled Code)
java.io.OutputStreamWriter.flushBuffer(Compiled Code)
java.io.OutputStreamWriter.flush(Compiled Code)
java.io.PrintWriter.flush(Compiled Code)
com.ibm.servlet.engine.webapp.BufferedWriter.flushChars(Compiled Code)
com.ibm.servlet.engine.webapp.BufferedWriter.write(Compiled Code)
java.io.Writer.write(Compiled Code)
java.io.PrintWriter.write(Compiled Code)
java.io.PrintWriter.write(Compiled Code)
java.io.PrintWriter.print(Compiled Code)
java.io.PrintWriter.println(Compiled Code)
pagecompile._identifycustomer_xjsp.service(Compiled Code)
javax.servlet.http.HttpServlet.service(Compiled Code)
com.ibm.servlet.jsp.http.pagecompile.JSPState.service(Compiled Code)
com.ibm.servlet.jsp.http.pagecompile.PageCompileServlet.doService(Compiled Code)
com.ibm.servlet.jsp.http.pagecompile.PageCompileServlet.doGet(Compiled Code)
javax.servlet.http.HttpServlet.service(Compiled Code)
javax.servlet.http.HttpServlet.service(Compiled Code)

And, using the full stack trace, it should be possible to identify any infinite loop
that might be occurring. The above example shows the use of
spin_lock_global_common, which is a busy wait on a lock, hence the use of CPU
time.

Poor performance on AIX

If no infinite loop is occurring, look at the process that is working, but having bad
performance. In this case, change your focus from what individual threads are
doing to what the process as a whole is doing. This is described in the AIX
documentation.

Understanding memory usage

Before you can properly diagnose memory problems on AIX, first you must have
an understanding of the AIX virtual memory model and how the JVM interacts
with it.

Chapter 14. AIX problem determination 111

AIX - understanding memory usage

32- and 64-bit JVMs

Most of the information in this section about altering the memory model and
running out of native heap is relevant only to the 32-bit model, because the 64-bit
model does not suffer from the same kind of memory constraints. The 64-bit JVM
can suffer from memory leaks in the native heap, and the same methods can be
used to identify and pinpoint those leaks. The information regarding the Java heap
relates to both 32- and 64-bit JVMs.

The 32-bit AIX Virtual Memory Model

AIX assigns a virtual address space partitioned into 16 segments of 256 MB.
Processing address space to data is managed at the segment level, so a data
segment can either be shared (between processes), or private.

Kernel 0x0

Application program text 0x1

Application program data and application stack 0x2

0x3

0x4

0x5

0x6

0x7
Shared memory and mmap services
0x8

0x9

0xA

0xB

0xC

Shared library text 0xD

Miscellaneous kernel data 0xE

Application shared library data 0xF

Figure 8. The AIX 32–Bit Memory Model with MAXDATA=0 (default)

v Segment 0 is assigned to the kernel.

v Segment 1 is application program text (static native code).
v Segment 2 is the application program data and application stack (primordial
thread stack and private data).
v Segments 3 to C are shared memory available to all processes.
v Segments D and F are shared library text and data areas respectively.
v Segment E is also shared memory and miscellaneous kernel usage.

112 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 AIX - understanding memory usage

The 64-bit AIX Virtual Memory Model

The 64-bit model allows many more segments, although each segment is still 256
MB. Again, the address space is managed at segment level, but the granularity of
function for each segment is much finer.

With the large address space available to the 64-bit process, you are unlikely to
encounter the same kind of problems with relation to native heap usage as
described later in this chapter, although you might still suffer from a leak in the
native heap.

Changing the Memory Model (32-bit JVM)

Three memory models are available on the 32-bit JVM.

The small memory model

With the default small memory model for an application (as shown above), the
application has only one segment, segment 2, in which it can malloc() data and
allocate additional thread stacks. It does, however, have 11 segments of shared
memory into which it can mmap() or shmat() data.

The large memory model

This single segment for data that is allocated by using malloc() might not be
enough, so it is possible to move the boundary between Private and Shared
memory, providing more Private memory to the application, but reducing the
amount of Shared memory. You move the boundary by altering the o_maxdata
setting in the Executable Common Object File Format (XCOFF) header for an
application.

You can alter the o_maxdata setting by:

v Setting the value of o_maxdata at compile time by using the -bmaxdata flag with
the ld command.
v Setting the o_maxdata value by using the LDR_CNTRL=MAXDATA=0xn0000000
(n segments) environment variable.

| The very large memory model

| Activate the very large memory model by adding ″@DSA″ onto the end of the
| MAXDATA setting. It provides two additional capabilities:
| v The dynamic movement of the private and shared memory boundary between a
| single segment and the segment specified by the MAXDATA setting. This
| dynamic movement is achieved by allocating private memory upwards from
| segment 3 and shared memory downwards from segment C. The private
| memory area can expand upwards into a new segment if the segment is not
| being used by the shmat or mmap routines.
| v The ability to load shared libraries into the process private area. If you specify a
| MAXDATA value of 0 or greater than 0xAFFFFFFF, the process will not use
| global shared libraries, but load them privately. So the shmat and mmap
| procedures begin allocating at higher segments because they are no longer
| reserved for shared libraries. In this way, the process has more contiguous
| memory.
| Altering the MAXDATA setting applies only to a 32-bit process and not the 64-bit
| JVM.

| Further details of the AIX Memory Models can be found at: http://
| publib.boulder.ibm.com/infocenter/pseries/....

Chapter 14. AIX problem determination 113

 AIX - understanding memory usage

The native and Java heaps

The JVM maintains two memory areas, the Java heap, and the native (or system)
heap. These two heaps have different purposes and are maintained by different
mechanisms.

The Java heap contains the instances of Java objects and is often referred to simply
as ’the heap’. It is the Java heap that is maintained by Garbage Collection, and it is
the Java heap that is changed by the command-line heap settings. The Java heap is
allocated using mmap, or shmat if large page support is requested. The maximum
size of the Java heap is preallocated during JVM startup as one contiguous area,
even if the minimum heap size setting is lower. This allocation allows the artificial
heap size limit imposed by the minimum heap size setting to move toward the
actual heap size limit with heap expansion. See Chapter 2, “Understanding the
Garbage Collector,” on page 7 for more information.

The native, or system heap, is allocated by using the underlying malloc and free
mechanisms of the operating system, and is used for the underlying
implementation of particular Java objects; for example:
v Motif objects required by AWT and Swing
v Buffers for Inflaters and Deflators
v Malloc allocations by application JNI code
v Compiled code generated by the Just In Time (JIT) Compiler
v Threads to map to Java threads

The AIX 32-bit JVM default memory models

| The AIX 142 Java launcher alters its MAXDATA setting in response to the
| command-line options to optimize the amount of memory available to the process.
| The default are as follows:
| -Xmx <= 2304M 0xA0000000@DSA
| 2304M < -Xmx <= 3072M 0xB0000000@DSA
| 3072M < -Xmx 0x0@DSA

| Monitoring the native heap

You can monitor the memory usage of a process by taking a series of snapshots
over regular time intervals of the memory currently allocated and committed. Use
svmon like this:
svmon -P [pid] -m -r -i [interval] > output.filename

Use the -r flag to print the address range.

| Because the Java heap is allocated using mmap() or shmat(), it is clear whether
| memory allocated to a specific segment of memory (under ’Esid’) is allocated to
| the Java or the native heap. The type and description fields for each of the
| segments allows the determination of which sections are native or Java heap.
| Segments allocated using mmap or shmat are listed as ″mmap mapped to″ or
| ″extended shm segments″ and are the Java heap. Segments allocated using malloc
| will be marked as ″working storage″ and are in the native heap. This demarcation
| makes it possible to monitor the growth of the native heap separately from the
| Java heap (which should be monitored using verbose GC).

Here is the svmon output from the command that is shown above:

114 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 AIX - understanding memory usage

| -------------------------------------------------------------------------------
| Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd LPage
| 29670 java 87347 4782 5181 95830 N Y N
|
| Vsid Esid Type Description LPage Inuse Pin Pgsp Virtual
| 50e9 - work - 41382 0 0 41382
| Addr Range: 0..41381
| 9dfb - work - 28170 0 2550 30720
| Addr Range: 0..30719
| ddf3 3 work working storage - 9165 0 979 10140
| Addr Range: 0..16944
| 0 0 work kernel seg - 5118 4766 1322 6420
| Addr Range: 0..11167
| c819 d work text or shared-lib code seg - 2038 0 283 6813
| Addr Range: 0..10219
| 2ded f work working storage - 191 0 20 224
| Addr Range: 0..4150
| f5f6 - work - 41 14 4 45
| Addr Range: 0..49377
| 6e05 2 work process private - 35 2 23 58
| Addr Range: 65296..65535
| 1140 6 work other segments - 26 0 0 26
| Addr Range: 0..32780
| cdf1 - work - 2 0 0 2
| Addr Range: 0..5277
| e93f - work - 0 0 0 0
| 3164 c mmap mapped to sid 1941 - 0 0 - -
| 2166 - work - 0 0 0 0
| 496b b mmap mapped to sid 2166 - 0 0 - -
| b51e - clnt /dev/fslv00:44722 - 0 0 - -
| Addr Range: 0..207
| ee1c a mmap mapped to sid e93f - 0 0 - -
| 1941 - work - 0 0 0 0
| 1081 7 mmap mapped to sid 9dfb - 0 0 - -
| edf5 8 mmap mapped to sid 50e9 - 0 0 - -
| c01b 9 mmap mapped to sid cdf1 - 0 0 - -

| The actual memory values for the mmap allocated segments are stored against a
| Vsid of type ″work″. For example, the memory usage in segment 7 (Java heap):
| 1081 7 mmap mapped to sid 9dfb - 0 0 - -

| is described against Vsid 9dfb, which reads as follows:

| 9dfb - work - 28170 0 2550 30720 Addr Range: 0..30719

| Native heap usage

The native heap usage will normally grow to a stable level, and then stay at
around that level. You can monitor the amount of memory committed to the native
heap by observing the number of ’Inuse’ pages in the svmon output. However,
note that as JIT compiled code is allocated to the native heap with malloc(), there
might be a steady slow increase in native heap usage as little used methods reach
the threshold to undergo JIT compilation.

| You can monitor the JIT compiling of code to avoid confusing this behavior with a
| memory leak. To do this, run with the command-line option
| -Xjit:verbose={compileStart|compileEnd}. This command causes each method
| name to print to stderr as it is being compiled and, as it finishes compiling, the
| location in memory where the compiled code is stored.
| (warm) Compiling java/lang/System.getEncoding(I)Ljava/lang/String;
| + (warm) java/lang/System.getEncoding(I)Ljava/lang/String; @ 0x02BA0028-0x02BA0113
| (2) Compiling java/lang/String.hashCode()I
| + (warm) java/lang/String.hashCode()I @ 0x02BA0150-0x02BA0229
| (2) Compiling java/util/HashMap.put(Ljava/lang/Object;Ljava/lang/Object;)

Chapter 14. AIX problem determination 115

 AIX - understanding memory usage

| Ljava/lang/Object;
| + (warm) java/util/HashMap.put(Ljava/lang/Object;Ljava/lang/Object;)
| Ljava/lang/Object; @ 0x02BA0270-0x02BA03F7
| (2) Compiling java/lang/String.charAt(I)C
| + (warm) java/lang/String.charAt(I)C @ 0x02BA0430-0x02BA04AC
| (2) Compiling java/util/Locale.toLowerCase(Ljava/lang/String;)
| Ljava/lang/String;
| + (warm) java/util/Locale.toLowerCase(Ljava/lang/String;)Ljava/lang/String;
| @ 0x02BA04D0-0x02BA064C

| When you have monitored how much native heap you are using, you can increase
| or decrease the maximum native heap available by altering the size of the Java
| heap. This relationship between the heaps occurs because the process address
| space not used by the Java heap is available for the native heap usage.

| You must increase the native heap if the process is generating errors relating to a
| failure to allocate native resources or exhaustion of process address space. These
| errors can take the form of a JVM internal error message or a detail message
| associated with an OutOfMemoryError. The message associated with the relevant
| errors will make it clear that the problem is native heap exhaustion.

Specifying MALLOCTYPE
You can set the MALLOCTYPE=watson environment variable, available in AIX 5.3,
for use with the IBM 142 JVM, but for most applications the performance gains are
likely to be small. It particularly benefits any application that makes heavy use of
malloc calls in native code. For more information, see this article:
http://www-128.ibm.com/developerworks/eserver/library/es-appdev-aix5l.html.

Monitoring the Java heap

| The most straightforward, and often most useful, way of monitoring the Java heap
| is by seeing what garbage collection is doing. Turn on garbage collection’s verbose
| tracing using the command-line option -verbose:gc to cause a report to be written
| to stderr each time garbage collection occurs. You can also direct this output to a
| log file using:
| -Xverbosegclog:[DIR_PATH][FILE_NAME]

| where:
| [DIR_PATH] is the directory where the file should be written
| [FILE_NAME] is the name of the file to write the logging to

| See “Using VerboseGC to obtain heap information” on page 208 for more
information on verbose GC output and monitoring.

Receiving OutOfMemory errors

Any OutOfMemory condition that occurs could be the result of either running out
| of Java heap or native heap. If the process address space (that is, the native heap)
| is exhausted, an error message is received that explains that a native allocation has
| failed. In either case, it is entirely possible that there is not a memory leak as such,
just that the steady state of memory usage that is required is higher than that
available. Therefore the first step is to determine which heap is being exhausted,
and increase the size of that heap.

If the problem is occurring because of a real memory leak, increasing the heap size
will not solve the problem, but will delay the onset of the OutOfMemory or error
conditions, which can be of help on any production system.

116 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 AIX - understanding memory usage

The 32-bit JVM has these limits:

v The maximum size of an object that can be created is 1 GB.
v For an array object, the maximum number of array elements supported is (228
-1). So, for a byte array, the maximum size of an array object is 256 MB.

Is the Java or native heap exhausted?

Some OutOfMemory conditions also carry an explanatory message, including an
error code, which might point to the origin of the error, either native or Java heap.

If no error message is present, the first stage is to monitor the Java and native heap
usages. The Java heap usage can be monitored by using -verbose:gc as detailed
above, and the native heap using svmon.

Java heap exhaustion

The Java heap becomes exhausted when garbage collection cannot free enough
objects to make a new object allocation. Garbage collection can free only objects
that are no longer referenced by other objects, or are referenced from the thread
stacks (see Chapter 2, “Understanding the Garbage Collector,” on page 7 for more
details).

Java heap exhaustion can be identified from the -verbose:gc output by garbage
collection occurring more and more frequently, with less memory being freed.
Eventually the JVM will fail, and the heap occupancy will be at, or almost at, 100%
(See Chapter 2, “Understanding the Garbage Collector,” on page 7 for more details
on -verbose:gc output).

If the Java heap is being exhausted, and increasing the Java heap size does not
solve the problem, the next stage is to examine the objects that are on the heap,
and look for suspect data structures that are referencing large numbers of Java
objects that should have been released. Use Heapdump Analysis, as detailed in
“Available tools for processing Heapdumps” on page 208. Similar information can
be gained by using other tools, such as JProbe and OptimizeIt.

Native heap exhaustion

You can identify native heap exhaustion by monitoring the svmon snapshot output
as discussed above. Each segment is 256 MB of space, which corresponds to 65535
pages. (Inuse is measured in 4 KB pages.)

If each of the segments has approximately 65535 Inuse pages, the process is
suffering from native heap exhaustion. At this point, extending the native heap
size might solve the problem, but you should investigate the memory usage profile
to ensure that you do not have a leak.

f DB2 is running on your AIX system, you can change the application code to use
the ″net″ (thin client) drivers and, in the case of WebSphere MQ you can use the
″client″ (out of process) drivers.

AIX fragmentation problems

Native heap exhaustion can also occur without the Inuse pages approaching 65535
Inuse pages. It can be caused by fragmentation of the AIX malloc heaps, which is
how AIX handles the native heap of the JVM.

Chapter 14. AIX problem determination 117

AIX - understanding memory usage

This kind of OutOfMemory condition can again be identified from the svmon
snapshots. Whereas previously the important column to look at for a memory leak
is the Inuse values, for problems in the AIX malloc heaps it is important to look at
the ’Addr Range’ column. The ’Addr Range’ column details the pages that have
been allocated, whereas the Inuse column details the number of pages that are
being used (committed).

It is possible that pages that have been allocated have not been released back to
the process when they have been freed. This leads to the discrepancy between the
number of allocated and committed pages.

You have a range of environment variables to change the behavior of the malloc
algorithm itself and thereby solve problems of this type:
MALLOCTYPE=3.1
This option allows the system to move back to an older version of memory
allocation scheme in which memory allocation is done in powers of 2. The 3.1
Malloc allocator, as opposed to the default algorithm, frees pages of memory
back to the system for reuse. The 3.1 allocation policy is available for use only
with 32-bit applications.
MALLOCMULTIHEAP=heaps:n,considersize
By default, the malloc subsystem uses a single heap. MALLOCMULTIHEAP
allows users to enable the use of multiple heaps of memory. Multiple heaps of
memory can lead to memory fragmentation, and so the use of this
environment variable is not recommended
MALLOCTYPE=buckets
Malloc buckets provides an optional buckets-based extension of the default
allocator. It is intended to improve malloc performance for applications that
issue large numbers of small allocation requests. When malloc buckets is
enabled, allocation requests that fall within a predefined range of block sizes
are processed by malloc buckets. Because of variations in memory
requirements and usage, some applications might not benefit from the memory
allocation scheme used by malloc buckets. Therefore, it is not advisable to
enable malloc buckets system-wide. For optimal performance, enable and
configure malloc buckets on a per-application basis.

Note: The above options might cause a percentage of performance hit. Also
the 3.1 malloc allocator does not support the Malloc Multiheap and
Malloc Buckets options.
MALLOCBUCKETS=number_of_buckets:128,bucket_sizing_factor:64,blocks_per_
bucket:1024: bucket_statistics: pathname of file for malloc statistics>
See above.

Submitting a bug report

If the data is indicating a memory leak in native JVM code, contact the IBM service
team. If the problem is Java heap exhaustion, it is much less likely to be an SDK
issue, although it is still possible. The process for raising a bug is detailed in
Chapter 8, “Overview of problem submission,” on page 77, and the data that
should be included in the bug report is listed below:
v Required:
1. The OutOfMemoryCondition. The error itself with any message or stack trace
that accompanied it.
2. -verbose:gc output. (Even if the problem is determined to be native heap
exhaustion, it can be useful to see the verbose gc output.)

118 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 AIX - understanding memory usage

v As appropriate:
1. The svmon snapshot output
2. The Heapdump output
| 3. The javacore.txt file

Debugging performance problems

Locating the causes of poor performance is often difficult. Although many factors
can affect performance, the overall effect is generally perceived as poor response or
slow execution of your program.

Correcting one performance problem might cause more problems in another area.
By finding and correcting a bottleneck in one place you might only shift the cause
of poor performance to other areas. To improve performance, experiment with
tuning different parameters, monitoring the effect, and retuning until you are
satisfied that your system is performing acceptably

Finding the bottleneck

The aspects of the system that you are most interested in measuring are CPU
usage and memory usage. It is possible that even after extensive tuning efforts the
CPU is not powerful enough to handle the workload, in which case a CPU
upgrade is required. Similarly, if the program is running in an environment in
which it does not have enough memory after tuning, you must increase memory
size.

Given that any performance problem could be caused by any one of several
factors, you must look at several areas to eliminate each one. First, determine
which resource is constraining the system:
v CPU
v Memory
v Input/Output (I/O)
To do this, use the vmstat command. The vmstat command produces a compact
report that details the activity of these three areas:
> vmstat 1 10

outputs:
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
0 0 189898 612 0 0 0 3 11 0 178 606 424 6 1 92 1
1 0 189898 611 0 1 0 0 0 0 114 4573 122 96 4 0 0
1 0 189898 611 0 0 0 0 0 0 115 420 102 99 0 0 0
1 0 189898 611 0 0 0 0 0 0 115 425 91 99 0 0 0
1 0 189898 611 0 0 0 0 0 0 114 428 90 99 0 0 0
1 0 189898 610 0 1 0 0 0 0 117 333 102 97 3 0 0
1 0 189898 610 0 0 0 0 0 0 114 433 91 99 1 0 0
1 0 189898 610 0 0 0 0 0 0 114 429 94 99 1 0 0
1 0 189898 610 0 0 0 0 0 0 115 437 94 99 0 0 0
1 0 189898 609 0 1 0 0 0 0 116 340 99 98 2 0 0

The example above shows a system that is CPU bound. This can be seen as the
user (us) plus system (sy) CPU values either equal or are approaching 100. A
system that is memory bound shows significant values of page in (pi) and page

Chapter 14. AIX problem determination 119

 AIX - debugging performance problems

out (po). A system that is disk I/O bound will show an I/O wait percentage (wa)
exceeding 10%. More details of vmstat can be found in “AIX debugging
commands” on page 97.

CPU bottlenecks
| If vmstat has shown that the system is CPU-bound, the next stage is to determine
| which process is using the most CPU time. The recommended tool is tprof:
| > tprof -s -k -x sleep 60

| outputs:
| Mon Nov 28 12:40:11 2005
| System: AIX 5.2 Node: voodoo Machine: 00455F1B4C00
|
| Starting Command sleep 60
| stopping trace collection
| Generating sleep.prof
| > cat sleep.prof
| Process Freq Total Kernel User Shared Other
| ======= ==== ===== ====== ==== ====== =====
| ./java 5 59.39 24.28 0.00 35.11 0.00
| wait 4 40.33 40.33 0.00 0.00 0.00
| /usr/bin/tprof 1 0.20 0.02 0.00 0.18 0.00
| /etc/syncd 3 0.05 0.05 0.00 0.00 0.00
| /usr/bin/sh 2 0.01 0.00 0.00 0.00 0.00
| gil 2 0.01 0.01 0.00 0.00 0.00
| afsd 1 0.00 0.00 0.00 0.00 0.00
| rpc.lockd 1 0.00 0.00 0.00 0.00 0.00
| swapper 1 0.00 0.00 0.00 0.00 0.00
| ======= ==== ===== ====== ==== ====== =====
| Total 20 100.00 64.70 0.00 35.29 0.00
|
| Process PID TID Total Kernel User Shared Other
| ======= === === ===== ====== ==== ====== =====
| ./java 467018 819317 16.68 5.55 0.00 11.13 0.00
| ./java 467018 766019 14.30 6.30 0.00 8.00 0.00
| ./java 467018 725211 14.28 6.24 0.00 8.04 0.00
| ./java 467018 712827 14.11 6.16 0.00 7.94 0.00
| wait 20490 20491 10.24 10.24 0.00 0.00 0.00
| wait 8196 8197 10.19 10.19 0.00 0.00 0.00
| wait 12294 12295 9.98 9.98 0.00 0.00 0.00
| wait 16392 16393 9.92 9.92 0.00 0.00 0.00
| /usr/bin/tprof 421984 917717 0.20 0.02 0.00 0.18 0.00
| /etc/syncd 118882 204949 0.04 0.04 0.00 0.00 0.00
| ./java 467018 843785 0.03 0.02 0.00 0.00 0.00
| gil 53274 73765 0.00 0.00 0.00 0.00 0.00
| gil 53274 61471 0.00 0.00 0.00 0.00 0.00
| /usr/bin/sh 397320 839883 0.00 0.00 0.00 0.00 0.00
| rpc.lockd 249982 434389 0.00 0.00 0.00 0.00 0.00
| /usr/bin/sh 397318 839881 0.00 0.00 0.00 0.00 0.00
| swapper 0 3 0.00 0.00 0.00 0.00 0.00
| afsd 65776 274495 0.00 0.00 0.00 0.00 0.00
| /etc/syncd 118882 258175 0.00 0.00 0.00 0.00 0.00
| /etc/syncd 118882 196839 0.00 0.00 0.00 0.00 0.00
| ======= === === ===== ====== ==== ====== =====
| Total 100.00 64.70 0.00 35.29 0.00
|
| Total Samples = 24749 Total Elapsed Time = 61.88s

| This output shows that the Java process with Process ID (PID) 467018 is using the
| majority of the CPU time. You can also see that the CPU time is being shared
| among four threads inside that process (Thread IDs 819317, 766019, 725211, and
| 712827).

120 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 AIX - debugging performance problems

| By understanding what the columns represent, you can gather an understanding of

| what these threads are doing:
| Total
| The total percentage of CPU time used by this thread or process.
| Kernel
| The total percentage of CPU time spent by this thread or process inside Kernel
| routines (on behalf of a request by the JVM or other native code).
| User
| The total percentage of CPU time spent executing routines inside the
| executable. Because the Java executable is a thin wrapper that loads the JVM
| from shared libraries, this CPU time is expected to be very small or zero.
| Shared
| The total percentage of CPU time spent executing routines inside shared
| libraries. Time shown under this category covers work done by the JVM itself,
| the act of JIT compiling (but not the running of the subsequent code), and any
| other native JNI code.
| Other
| The total percentage of CPU time not covered by Kernel, User, and Shared. In
| the case of a Java process, this CPU time covers the execution of Java
| bytecodes and JIT-compiled methods themselves.

| From the above example, notice the Kernel and Shared values: these account for all
| of the CPU time used by this process, indicating that the Java process is spending
| its time doing work inside the JVM (or some other native code).

| To understand what is being done during the Kernel and Shared times, the
| relevant sections of the tprof output can be analyzed.

| The shared library section shows which shared libraries are being invoked:
| Shared Object %
| ============= ======
| /j9vmap3223-20051123/inst.images/rios_aix32_5/sdk/jre/bin/libj9gc23.so 17.42
| /usr/lib/libc.a[shr.o] 9.38
| /usr/lib/libpthreads.a[shr_xpg5.o] 6.94
| j9vmap3223-20051123/inst.images/rios_aix32_5/sdk/jre/bin/libj9thr23.so 1.03
| j9vmap3223-20051123/inst.images/rios_aix32_5/sdk/jre/bin/libj9prt23.so 0.24
| /j9vmap3223-20051123/inst.images/rios_aix32_5/sdk/jre/bin/libj9vm23.so 0.10
| j9vmap3223-20051123/inst.images/rios_aix32_5/sdk/jre/bin/libj9ute23.so 0.06
| j9vmap3223-20051123/inst.images/rios_aix32_5/sdk/jre/bin/libj9jit23.so 0.05
| /usr/lib/libtrace.a[shr.o] 0.04
| j9vmap3223-20051123/inst.images/rios_aix32_5/sdk/jre/bin/libj9trc23.so 0.02
| p3223-20051123/inst.images/rios_aix32_5/sdk/jre/bin/libj9hookable23.so 0.01
|

| This section shows that almost all of the time is being spent in one particular
| shared library, which is part of the JVM installation: libj9gc23.so. By understanding
| the functions that the more commonly used JVM libraries carry out, it becomes
| possible to build a more accurate picture of what the threads are doing:
| libbcv23.so
| Bytecode Verifier
| libdbg23.so
| Debug Server (used by the Java Debug Interface)
| libj9gc23.so
| Garbage Collection

Chapter 14. AIX problem determination 121

 AIX - debugging performance problems

| libj9jextract.so
| The dump extractor, used by the jextract command
| libj9jit23.so
| The Just In Time (JIT) Compiler
| libj9jpi23.so
| The JVMPI interface
| libj9jvmti23.so
| The JVMTI interface
| libj9prt23.so
| The ″port layer″ between the JVM and the Operating System
| libj9shr23.so
| The shared classes library
| libj9thr23.so
| The threading library
| libj9ute23.so
| The trace engine
| libj9vm23.so
| The core Virtual Machine
| libj9zlib23.so
| The compressed file utility library
| libjclscae_23.so
| The Java Class Library (JCL) support routines

| In the example above, the CPU time is being spent inside the garbage collection
| (GC) implementation, implying either that there is a problem in GC or that GC is
| running almost continuously.

| Again, you can obtain a more accurate understanding of what is occurring inside
| the libj9gc23.so library during the CPU time by analyzing the relevant section of
| the tprof output:
| Profile: /work/j9vmap3223-20051123/inst.images/rios_aix32_5/sdk/jre/bin/
| libj9gc23.so
|
| Total % For All Processes (/work/j9vmap3223-20051123/inst.images/rios_aix32_5/
| sdk/jre/bin/libj9gc23.so) = 17.42
|
| Subroutine % Source
| ========== ====== ======
| Scheme::scanMixedObject(MM_Environment*,J9Object*) 2.67 MarkingScheme.cpp
| MarkingScheme::scanClass(MM_Environment*,J9Class*) 2.54 MarkingScheme.cpp
| .GC_ConstantPoolObjectSlotIterator::nextSlot() 1.96 jectSlotIterator.cpp
| lelTask::handleNextWorkUnit(MM_EnvironmentModron*) 1.05 ParallelTask.cpp
| orkPackets::getPacket(MM_Environment*,MM_Packet**) 0.70 WorkPackets.cpp
| cheme::fixupRegion(J9Object*,J9Object*,bool,long&) 0.67 CompactScheme.cpp
| WorkPackets::putPacket(MM_Environment*,MM_Packet*) 0.47 WorkPackets.cpp
| rkingScheme::scanObject(MM_Environment*,J9Object*) 0.43 MarkingScheme.cpp
| sweepChunk(MM_Environment*,MM_ParallelSweepChunk*) 0.42 allelSweepScheme.cpp
| ment*,J9IndexableObject*,J9Object**,unsigned long) 0.38 MarkingScheme.cpp
| M_CompactScheme::getForwardingPtr(J9Object*) const 0.36 CompactScheme.cpp
| ObjectHeapIteratorAddressOrderedList::nextObject() 0.33 dressOrderedList.cpp
| ckets::getInputPacketFromOverflow(MM_Environment*) 0.32 WorkPackets.cpp
| .MM_WorkStack::popNoWait(MM_Environment*) 0.31 WorkStack.cpp
| WorkPackets::getInputPacketNoWait(MM_Environment*) 0.29 WorkPackets.cpp
| canReferenceMixedObject(MM_Environment*,J9Object*) 0.29 MarkingScheme.cpp
| MarkingScheme::markClass(MM_Environment*,J9Class*) 0.27 MarkingScheme.cpp

122 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 AIX - debugging performance problems

| ._ptrgl 0.26 ptrgl.s

| _MarkingScheme::initializeMarkMap(MM_Environment*) 0.25 MarkingScheme.cpp
| .MM_HeapVirtualMemory::getHeapBase() 0.23 eapVirtualMemory.cpp
|

| This output shows that the most-used functions are:

| MarkingScheme::scanMixedObject(MM_Environment*,J9Object*)
| 2.67 MarkingScheme.cpp
| MarkingScheme::scanClass(MM_Environment*,J9Class*)
| 2.54 MarkingScheme.cpp
| ObjectSlotIterator.GC_ConstantPoolObjectSlotIterator::nextSlot()
| 1.96 ObjectSlotIterator.cpp
| ParallelTask::handleNextWorkUnit(MM_EnvironmentModron*)
| 1.05 ParallelTask.cpp

| The values show that the time is being spent during the Mark phase of GC.
| Because the output also contains references to the Compact and Sweep phases, it is
| likely that GC is completing but that it is occurring continuously. You could
| confirm that likelihood by running with -verbosegc enabled.

| The same methodology shown above can be used for any case where the majority
| of the CPU time is shown to be in the Kernel and Shared columns. If, however, the
| CPU time is classed as being ″Other″, a different methodology is required because
| tprof does not contain a section that correctly details which Java methods are being
| run.

| In the case of CPU time being attributed to ″Other″, you can use a javacore file (or
| a series of javacore files) to determine the stack trace for the TIDs shown to be
| taking the CPU time, and therefore provide an idea of the work that it is doing.
| Map the value of TID shown in the tprof output to the correct thread in the
| javacore file by taking the tprof TID, which is stored in decimal, and convert it to
| hexadecimal. The hexadecimal value is shown as the ″native ID″ in the javacore
| file.

| For the example above:

| Process PID TID Total Kernel User Shared Other
| ======= === === ===== ====== ==== ====== =====
| ./java 7018 819317 16.68 5.55 0.00 11.13 0.00

| This thread is the one using the most CPU; the TID in decimal is 819317. This
| value is C8075 in hexadecimal, which can be seen in the javacore file:
| 3XMTHREADINFO "main" (TID:0x300E3500, sys_thread_t:0x30010734,
| state:R, native ID:0x000C8075) prio=5
| 4XESTACKTRACE at java/lang/Runtime.gc(Native Method)
| 4XESTACKTRACE at java/lang/System.gc(System.java:274)
| 4XESTACKTRACE at GCTest.main(GCTest.java:5)

| These entries show that, in this case, the thread is calling GC, and explains the
| time spent in the libj9gc23.so shared library.

Memory bottlenecks
If the results of vmstat point to a memory bottleneck, you must find out which
processes are using large amounts of memory, and which, if any, of these are
growing. Use the svmon tool:
> svmon -P -t 5

This command outputs:

Chapter 14. AIX problem determination 123

AIX - debugging performance problems

-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd
38454 java 76454 1404 100413 144805 N Y
-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd
15552 X 14282 1407 17266 19810 N N
-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd
14762 dtwm 3991 1403 5054 7628 N N
-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd
15274 dtsessi 3956 1403 5056 7613 N N
-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd
21166 dtpad 3822 1403 4717 7460 N N

This output shows that the highest memory user is Java, and that it is using 144805
pages of virtual memory (144805 * 4 KB = 565.64 MB). This is not an unreasonable
amount of memory for a JVM with a large Java heap - in this case 512 MB.

If the system is memory-constrained with this level of load, the only remedies
available are either to obtain more physical memory or to attempt to tune the
amount of paging space that is available by using the vmtune command to alter
the maxperm and minperm values.

If the Java process continues to increase its memory usage, an eventual memory
constraint will be caused by a memory leak.

I/O bottlenecks
This book does not discuss conditions in which the system is disk- or
network-bound. For disk-bound conditions, use filemon to generate more details of
which files and disks are in greatest use. For network conditions, use netstat to
determine network traffic. A good resource for these kinds of problems is
Accelerating AIX by Rudy Chukran (Addison Wesley, 1998).

JVM heap sizing

The Java heap size is one of the most important tuning parameters of your JVM.
See “Heap size” on page 8.

JIT compilation and performance

The JIT is another area that can affect the performance of your program. When
deciding whether or not to use JIT compilation, you must make a balance between
faster execution and increased compilation overhead. The performance of
short-running applications can be improved by using the -Xquickstart
command-line parameter. The JIT is switched on by default, but you can use -Xint
to turn it off. You also have considerable flexibility in controlling JIT processing.
For more details about the JIT, see Chapter 4, “Understanding the JIT,” on page 29
and Chapter 27, “JIT problem determination,” on page 237.

Application profiling
You can learn a lot about your Java application by using the hprof profiling agent.
Statistics about CPU and memory usage are presented along with many other
options. The hprof tool is discussed in detail in Chapter 32, “Using the JVMPI,” on
page 297. -Xrunhprof:help gives you a list of suboptions that you can use with
hprof.

124 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 AIX - collecting data from a fault condition

MustGather information for AIX

The information that is most useful at a point of failure depends, in general, on the
type of failure that is experienced. These normally have to be actively generated
and as such is covered in each of the sections on the relevant failures. However,
some data can be obtained passively:
The AIX core file
If the environment is correctly set up to produce full AIX Core files (as
detailed in “Setting up and checking your AIX environment” on page 95),
a core file is generated when the process receives a terminal signal (that is,
SIGSEGV, SIGILL, or SIGABORT). The core file is generated into the
current working directory of the process, or at the location pointed to by
| the label field specified using -Xdump.
For complete analysis of the core file, the IBM support team needs:
v The core file
v A copy of the Java executable that was running the process
v Copies of all the libraries that were in use when the process core
dumped
When a core file is generated:
| 1. Run the jextract utility against the core file like this
| jextract <core file name>

| to generate a file called dumpfilename.zip in the current directory. This

| file is compressed and contains the required files. Running jextract
| against the core file also allows the subsequent use of the Dump
| Viewer
| 2. If the jextract processing fails, use the snapcore utility to collect the
| same information. For example, snapcore -d /tmp/savedir core.001
| /usr/java142/jre/bin/java creates an archive (snapcore_pid.pax.Z) in
| the file /tmp/savedir.
You also have the option of looking directly at the core file by using dbx or
a canned dbx session. However, dbx does not have the advantage of
understanding Java frames and the JVM control blocks that the Dump
Viewer does. Therefore, you are recommended to use the Dump Viewer in
preference to dbx.
The javacore file:
| When a javacore file is written, a message (JVMDUMP010I) is written to
stderr telling you the name and full path of the javacore file. In addition, a
javacore file can be actively generated from a running Java process by
sending it a SIGQUIT (kill -3 or Ctrl-\) command.
The Error Report
The use of errpt -a generates a complete detailed report from the system
error log. This report can provide a stack trace, which might not have been
generated elsewhere. It might also point to the source of the problem
where it would otherwise be ambiguous.

Getting AIX technical support

See these web pages:

Chapter 14. AIX problem determination 125

AIX - getting technical support

http://techsupport.services.ibm.com/server/nav?fetch=a4ojc
http://techsupport.services.ibm.com/server/nav?fetch=a5oj

126 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 15. Linux problem determination
This chapter describes problem determination on Linux in:
v “Setting up and checking your Linux environment”
v “General debugging techniques” on page 128
v “Diagnosing crashes” on page 134
v “Debugging hangs” on page 135
v “Debugging memory leaks” on page 135
v “Debugging performance problems” on page 135
v “Collecting data from a fault condition in Linux” on page 138
v “Known limitations on Linux” on page 140

Setting up and checking your Linux environment

Note: Linux operating systems undergo a large number of patches and updates. It
is impossible for IBM personnel to test the JVM against every patch. The intention
is to test against the most recent releases of a few distributions. In general, you
should keep systems up-to-date with the latest patches. See http://www.ibm.com/
developerworks/java/jdk/linux/tested.html for an up-to-date list of releases and
distributions that have been successfully tested against.

The Java service team has a tool named ReportEnv that plugs into your JVM and
reports on the JVM environment in real time. Your JVM environment affects the
operation of the JVM. ReportEnv reports on environment variables and
command-line parameters. It is a GUI tool, although it can be run without a GUI.
The GUI allows you to browse your environment and, to some extent, dynamically
change it. The tool also has a mechanism to generate reports to tell you the exact
state of your JVM environment. A screenshot of the tool is shown in “Setting up
and checking your Windows environment” on page 143. The ReportEnv tool is
available on request from [email protected].

Working directory
The current working directory of the JVM process is where core files, Java dumps,
heap dumps, and the JVM trace outputs, including Application Trace and Method
trace, are outputted. Enough free disk space must be available for this directory.
Also, the JVM must have write permission.

Linux core files

A core file is an image of a process that is created by the operating system when
the process terminates unexpectedly. This file can be very useful in determining
what went wrong with a process. The production of core files can be enabled by
default, depending on the distribution and version of Linux that you have.

Because truncated files are of no practical use, set the size of the Linux core file to
″unlimited″.
Table 5. Usage of ulimit
Usage Action
ulimit -c # check the current corefile limit

© Copyright IBM Corp. 2003, 2006 127

setting up and checking your Linux environment

Table 5. Usage of ulimit (continued)

Usage Action
ulimit -c 0 # turn off corefiles
ulimit -c x # set the maximum corefile size to x number
of 1024-bytes
ulimit -c unlimited # turn on corefiles with unlimited size
ulimit -n unlimited # allows an unlimited number of open file
descriptors
ulimit -p # size of pipes
ulimit -s # maximum native stack size for a process
ulimit -u # number of user processes
help ulimit #list of other options

The core file is placed into the current working directory of the process, subject to
write permissions for the JVM process and free disk space.

Depending on the kernel level, a useful kernel option is available that gives
corefiles more meaningful names. As root user, the option sysctl -w
kernel.core_users_pid=1 ensures that core files have a name of the form
″Core.PID″.

Threading libraries
Most Linux distributions provide an implementation of the POSIX threads
standard known as the LinuxThreads library. Some newer distributions (for
example RHEL3 and SLES9) now provide the enhanced Native POSIX Threads
Library for Linux (NPTL). For information on the threading libraries which are
supported by the IBM Virtual Machine for Java on specific Linux platforms, see
http://www.ibm.com/developerworks/java/jdk/linux/tested.html.

You can discover your glibc version by changing to the /lib directory and running
the file libc.so.6. The Linux command ldd prints information that should help you
to work out the shared library dependency of your application.

General debugging techniques

This section provides a guide to the JVM-provided diagnostic tools and Linux
commands that can be useful when you are diagnosing problems that occur with
the Linux JVM.

Starting Javadumps in Linux

See Chapter 21, “Using Javadump,” on page 191.

Starting heapdumps in Linux

See Chapter 22, “Using Heapdump,” on page 205.

Using the dump extractor on Linux

When a dump occurs, the structure and contents of the core file produced differ
depending on platform. A cross-platform dump formatter can automate some of
the tasks that are involved with studying a corefile. For the dump formatter to
function, all corefiles must be converted to a common format. The Linux Dump

128 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Linux - general debugging techniques

Extractor converts a corefile obtained on a Linux machine to a corefile suitable for

use by the dump formatter. To use the Linux Dump extractor, run the command:
jextract <corefile>

This command produces a modified core file with a .sdff file extension, which you
might be asked to send to IBM service. See Chapter 26, “Using the dump
formatter,” on page 223 for details of the Cross Platform Dump Formatter.

Using core dumps

The commands objdump and nm both display information about object files. If a
crash occurs and a corefile is produced, these commands help you analyze the file.
objdump
Use this command to disassemble shared objects and libraries. After you have
discovered which library or object has caused the problem, use objdump to
locate the method in which the problem originates. To invoke objdump, type:
objdump <option> <filename>
nm
This command lists symbol names from object files. These symbol names can
be either functions, global variables, or static variables. For each symbol, the
value, symbol type, and symbol name are displayed. Lower case symbol types
mean the symbol is local, while upper case means the symbol is global or
external. To use this tool, type: nm <option> <filename>

You can see a complete list of options by typing objdump -H. The -d option
disassembles contents of executable sections

Run these commands on the same machine as the one that produced the core files
to get the most accurate symbolic information available. This output (together with
the core file, if small enough) is used by IBM Java Support to diagnose a problem.

Using system logs

The kernel provides useful environment information. Use the following commands
to view this information:
v ps -elf
v top
v vmstat

The ps command displays process status. Use it to gather information about native
threads. Some useful options are:
v -e: Select all processes
v -l: Displays in long format
v -f: Displays a full listing

The top command displays the most CPU- or memory-intensive processes in real
time. It provides an interactive interface for manipulation of processes and allows
sorting by different criteria, such as CPU usage or memory usage. The display is
updated every five seconds by default, although this can be changed by using the
s (interactive) command. The top command displays several fields of information
for each process. The process field shows the total number of processes that are
running, but breaks this down into tasks that are running, sleeping, stopped, or
undead. In addition to displaying PID, PPID, and UID, the top command displays
information on memory usage and swap space. The mem field shows statistics on

Chapter 15. Linux problem determination 129

Linux - general debugging techniques

memory usage, including available memory, free memory, used memory, shared
memory, and memory used for buffers. The Swap field shows total swap space,
available swap space, and used swap space.

The vmstat command reports virtual memory statistics. It is useful to perform a

general health check on your system, although, because it reports on the system as
a whole, commands such as ps and top can be used afterwards to gain more
specific information about your programs operation. When you use it for the first
time during a session, the information is reported as averages since the last reboot.
However, further usage will display reports that are based on a sampling period
that you can specify as an option. Vmstat 3 4 will display values every 3 seconds
for a count of 4 times. It might be useful to start vmstat before the application,
have it direct its output to a file and later study the statistics as the application
started and ran. The basic output from this command appears in five sections;
processes, memory, swap, io, system, and cpu.

The processes section shows how many processes are awaiting run time, blocked,
or swapped out.

The memory section shows the amount of memory (in kilobytes) swapped, free,
buffered, and cached. If the free memory is going down during certain stages of
your applications execution, there might be a memory leak.

The swap section shows the kilobytes per second of memory swapped in from and
swapped out to disk. Memory is swapped out to disk if RAM is not big enough to
store it all. Large values here can be a hint that not enough RAM is available
(although it is normal to get swapping when the application first starts).

The io section shows the number of blocks per second of memory sent to and
received from block devices.

The system section displays the interrupts and the context switches per second.
There is overhead associated with each context switch so a high value for this may
mean that the program does not scale well.

The cpu section shows a break down of processor time between user time, system
time, and idle time. The idle time figure shows how busy a processor is, with a
low value indicating that the processor is very busy. You can use this knowledge to
help you understand which areas of your program are using the CPU the most.

In Linux, each native thread is a distinct process with a unique process ID (PID).
The kernel can therefore provide very useful information about your threads
through commands such as ps and top.

Linux debugging commands

ps
On Linux, Java threads are implemented as system threads and might be visible in
the process table, depending on the Linux distribution. Running the ps command
gives you a snapshot of the current processes. The ps command gets its
information from the /proc filesystem. Here is an example of using ps.
ps -efwH

UID PID PPID C STIME TTY TIME CMD

cass 1234 1231 0 Aug07 ? 00:00:00 /bin/bash
cass 1555 1234 0 Aug07 ? 00:00:02 java app

130 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Linux - general debugging techniques

cass 1556 1555 0 Aug07 ? 00:00:00 java app

cass 1557 1556 0 Aug07 ? 00:00:00 java app
cass 1558 1556 0 Aug07 ? 00:00:00 java app
cass 1559 1556 0 Aug07 ? 00:00:00 java app
cass 1560 1556 0 Aug07 ? 00:00:00 java app
e Specifies to select all processes.
f Ensures that a full listing is provided.
m Shows threads if they are not shown by default.
w An output modifier that ensures a wide output.
H Useful when you are interested in Java threads because it displays a
hierarchical listing. With a hierarchical display, you can determine which
process is the primordial thread, which is the thread manager, and which are
child threads. In the example above, process 1555 is the primordial thread,
while process 1556 is the thread manager. All the child processes have a
parent process id pointing to the thread manager.

Tracing
Tracing is a technique that presents details of the execution of your program. If
you are able to follow the path of execution, you will gain a better insight into
how your program runs and interacts with its environment. Also, you will be able
to pinpoint locations where your program starts to deviate from its expected
behavior.

Three tracing tools on Linux are strace, ltrace and mtrace. The command man
<strace> will show a full set of available options.
strace
The strace tool traces system calls. You can either use it on a process that is
already active, or start it with a new process. strace records the system calls
made by a program and the signals received by a process. For each system call,
the name, arguments, and return value are used. strace allows you to trace a
program without requiring the source (no recompilation is required). If you use
it with the -f option, it will trace child processes that have been created as a
result of a forked system call. strace is often used to investigate plug-in
problems or to try to understand why programs do not start properly.
ltrace
The ltrace tool is distribution-dependent. It is very similar to strace. This tool
intercepts and records the dynamic library calls as called by the executing
process. strace does the same for the signals received by the executing process.
mtrace
mtrace is included in the GNU toolset. It installs special handlers for malloc,
realloc, and free, and enables all uses of these functions to be traced and
recorded to a file. This tracing decreases program efficiency and should not be
enabled during normal use. To use mtrace, set IBM_MALLOCTRACE to 1,
and set MALLOC_TRACE to point to a valid file where the tracing
information will be stored. You must have write access to this file.

gdb
The GNU debugger (gdb) allows you to examine the internals of another program
while the program executes or retrospectively to see what a program was doing at
the moment that it crashed. The gdb allows you to examine and control the
execution of code and is very useful for evaluating the causes of crashes or general
incorrect behavior. gdb does not handle Java processes, so it is of limited use on a
pure Java program. It is useful for debugging native libraries and the JVM itself.

Chapter 15. Linux problem determination 131

Linux - general debugging techniques

You can run gdb in three ways:

Starting a program
Normally the command: gdb <application> is used to start a program under
the control of gdb. However, because of the way that Java is launched, you
must invoke gdb by setting an environment variable and then calling Java:
export IBM_JVM_DEBUG_PROG=gdb
java

Then you receive a gdb prompt, and you supply the run command and the
Java arguments:
r<java_arguments>
Attaching to a running program
If a Java program is already running, you can control it under gdb. The
process id of the running program is required, and then gdb is started with the
Java executable as the first argument and the pid as the second argument:
gdb <Java Executable> <PID>

When gdb is attached to a running program, this program is halted and its
position within the code is displayed for the viewer. The program is then
under the control of gdb and you can start to issue commands to set and view
the variables and generally control the execution of the code.
Running on a corefile
A corefile is normally produced when a program crashes. gdb can be run on
this corefile. The corefile contains the state of the program when the crash
occurred. Use gdb to examine the values of all the variables and registers
leading up to a crash. With this information, you should be able to discover
what caused the crash. To debug a corefile, invoke gdb with the Java
executable as the first argument and the corefile name as the second argument:
gdb <Java Executable> <corefile>

When you run gdb against a corefile, it will initially show information such as
the termination signal the program received, the function that was executing at
the time, and even the line of code that generated the fault.

When a program comes under the control of gdb, a welcome message is displayed
followed by a prompt (gdb). The program is now waiting for your input and will
continue in whichever way you choose.

There are a number of ways of controlling execution and examination of the code.
Breakpoints can be set for a particular line or function using the command:

breakpoint linenumber
or
breakpoint functionName

After you have set a breakpoint, use the continue command to allow the program
to execute until it hits a breakpoint.

Set breakpoints using conditionals so that the program will halt only when the
specified condition is reached. For example, using breakpoint 39 if var = = value
causes the program to halt on line 39 only if the variable is equal to the specified
value.

132 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Linux - general debugging techniques

If you want to know where as well as when a variable became a certain value you
can use a watchpoint. Set the watchpoint when the variable in question is in scope.
After doing so, you will be alerted whenever this variable attains the specified
value. The syntax of the command is: watch var = = value.

To see which breakpoints and watchpoints are set, use the info command:

info break
info watch

When gdb reaches a breakpoint or watchpoint, it prints out the line of code it is
next set to execute. Note that setting a breakpoint on line 8 will cause the program
to halt after completing execution of line 7 but before execution of line 8. As well
as breakpoints and watchpoints, the program also halts when it receives certain
system signals. By using the following commands, you can stop the debugging
tool halting every time it receives these system signals:

handle sig32 pass nostop noprint

handle sigusr2 pass nostop noprint

When the correct position of the code has been reached, there are a number of
ways to examine the code. The most useful is backtrace (abbreviated to bt), which
shows the call stack. The call stack is the collection of function frames, where each
function frame contains information such as function parameters and local
variables. These function frames are placed on the call stack in the order that they
are executed (the most recently called function appears at the top of the call stack),
so you can follow the trail of execution of a program by examining the call stack.
When the call stack is displayed, it shows a frame number to the very left,
followed by the address of the calling function, followed by the function name and
the source file for the function. For example:

#6 0x804c4d8 in myFunction () at myApplication.c

To view more in-depth information about a function frame, use the frame
command along with a parameter specifying the frame number. After you have
selected a frame, you can display its variables using the command print var.

Use the print command to change the value of a variable; for example, print var
= newValue.

The info locals command displays the values of all local variables in the selected
function.

To follow the exact sequence of execution of your program, use the step and next
commands. Both commands take an optional parameter specifying the number of
lines to execute, but while next treats function calls as a single line of execution,
step will step through each line of the called function.

When you have finished debugging your code, the run command causes the
program to run through to its end or its crash point. The quit command is used to
exit gdb.

Other useful commands are:

ptype
Prints datatype of variable.

Chapter 15. Linux problem determination 133

Linux - general debugging techniques

info share
Prints the names of the shared libraries that are currently loaded.
info functions
Prints all the function prototypes.
list
Shows the 10 lines of source code around the current line.
help
The help command displays a list of subjects, each of which can have the help
command invoked on it, to display detailed help on that topic.

Diagnosing crashes
Many approaches are possible when you are trying to determine the cause of a
crash. The process normally involves isolating the problem by checking the system
setup and trying various diagnostic options.

Checking the system environment

The system might have been in a state that has caused the JVM to crash. For
example, this could be a resource shortage (such as memory or disk) or a stability
problem. Check the Javadump file, which contains various system information (as
described in Chapter 21, “Using Javadump,” on page 191). The Javadump file tells
you how to find disk and memory resource information. The system logs can give
indications of system problems.

Gathering process information

It is useful to find out what exactly was happening leading up to the crash.

Analyze the core file (as described in Chapter 26, “Using the dump formatter,” on
page 223) to produce a stack trace, which will show what was running up to the
point of the crash. This could be:
v JNI native code.
v JIT compiled code. If you have a problem with the JIT, try running with JIT off
by setting the -Xint option.
v JVM code.
Other tracing methods:
v ltrace
v strace
v mtrace - can be used to track memory calls and determine possible corruption
v RAS trace, described in Chapter 31, “Using the Reliability, Availability, and
Serviceability Interface,” on page 283.

Finding out about the Java environment

Use the Javadump to determine what each thread was doing and which Java
methods were being executed. Match function addresses against library addresses
to determine the source of code executing at various points.

Use the verbosegc option to look at the state of the Java heap and determine if:
v There was a shortage of Java heap space and if this could have caused the crash.

134 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Linux - diagnosing crashes

v The crash occurred during garbage collection, indicating a possible garbage

collection fault. See Chapter 2, “Understanding the Garbage Collector,” on page
7.
v The crash occurred after garbage collection, indicating a possible memory
corruption.

For more information about the Garbage Collector, see Chapter 2, “Understanding
the Garbage Collector,” on page 7.

Debugging hangs
For an explanation of deadlocks and diagnosing them using the Javadump tool,
see “Locks, monitors, and deadlocks (LOCKS)” on page 193.

A hang is caused by a wait or a loop. A wait or deadlock sometimes occurs

because of a wait on a lock or monitor. A loop or livelock can occur similarly or
sometimes because of an algorithm making little or no progress towards
completion. The following approaches are most useful in this situation:
v Monitoring process and system state (as described in “Collecting data from a
fault condition in Linux” on page 138).
v Java Dumps give monitor and lock information.
v verbosegc information is useful. It indicates:
– Excessive garbage collection because of lack of Java heap space causing the
system to appear to be in livelock
– Garbage collection causing of hang or memory corruption which later causes
hangs

Debugging memory leaks

The mtrace tool from GNU is available for tracking memory calls. This tool enables
you to trace memory calls such as malloc and realloc so that you can detect and
locate memory leaks.

For more details about analyzing the Java Heap, see Chapter 22, “Using
Heapdump,” on page 205.

Debugging performance problems

Locating the causes of poor performance is often difficult, because, although many
factors can affect performance, the overall effect is often the same; that is, poor
response or slow execution of your program.

Whether you want to find obvious performance bottlenecks, or tune general

performance, find out as much as possible about your system and how it performs.
Also, remember that when you correct one set of problems, you might cause more
problems in another area. By finding and correcting a bottleneck in one place, you
might only shift the cause of poor performance to other areas. So, to really
improve performance, you must experiment by tuning different parameters,
monitoring their effect, and retuning until you are satisfied that your system is
performing acceptably.

System performance
Several tools are available that enable you to measure system components and
establish how they are performing and under what kind of workload. Although

Chapter 15. Linux problem determination 135

Linux - debugging performance problems

most of these tools have been introduced earlier in this chapter, it is still worth
mentioning them here, and discussing how you can use them to specifically debug
performance issues.

The aspects of the system that you are most interested in measuring are CPU
usage and memory usage. If you can prove that the CPU is not powerful enough
to handle the workload, any amount of tuning makes not much difference to
overall performance. Nothing less than a CPU upgrade might be required.
Similarly, if a program is running in an environment in which it does not have
enough memory, an increase in the memory is going to make a much bigger
change to performance than any amount of tuning does.

CPU usage
You might typically experience Java processes consuming 100% of processor time
when a process reaches its resource limits. Ensure that ulimit settings are
appropriate to the application requirement. Some of the most-used ulimit
parameters are discussed in Table 5 on page 127.

The /proc file system provides information about all the processes that are running
on your system, including the Linux kernel. Because Java threads are run as
system processes, you can learn valuable information about the performance of
your application. See /proc man for more information about viewing /proc
information. /proc/version gives you information about the Linux kernel that is
on your system.

The top command provides real-time information about your system processes.
The top command is useful for getting an overview of the system load. It quite
clearly displays which processes are using the most resources. Having identified
the processes that are probably causing a degraded performance, you can take
further steps to improve the overall efficiency of your program. More information
is provided about the top command in “Using system logs” on page 129.

Memory usage
If a system is performing poorly because of lack of memory resources, it is
memory bound. By viewing the contents of /proc/meminfo, you can view your
memory resources and see how they are being used. /proc/swap gives
information on your swap file.

Swap space is used as an extension of the systems virtual memory. Therefore, not
having enough memory or swap space causes performance problems. A general
guideline is that swap space should be at least twice as large as the physical
memory.

A swap space can be either a file or disk partition. A disk partition offers better
performance than a file does. fdisk and cfdisk are the commands that you use to
create another swap partition. It is a good idea to create swap partitions on
different disk drives because this distributes the I/O activities and so reduces the
chance of further bottlenecks.

VMstat is a tool that enables you to discover where performance problems might
be caused. For example, if you see that high swap rates are occurring, it is likely
that you do not have enough physical or swap space. The free command displays
your memory configuration, while swapon -s displays your swap device
configuration. A high swap rate (for example, many page faults) means that it is
quite likely that you need to increase your physical memory. More details on how
to use VMstat are provided in “Using system logs” on page 129.

136 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Linux - debugging performance problems

Network problems
Another area that often affects performance is the network. Obviously, the more
you know about the behavior of your program, the easier it is for you to decide
whether this is a likely source of performance bottleneck. If you think that your
program is likely to be I/O bound, netstat is a useful tool. In addition to providing
information about network routes, netstat gives a list of active sockets for each
network protocol and can give overall statistics, such as the number of packets that
are received and sent. Using netstat, you can see how many sockets are in a
CLOSE_WAIT or ESTABLISHED state and you can tune the respective TCP/IP
parameters accordingly for better performance of the system. For example, tuning
/proc/sys/net/ipv4/tcp_keepalive_time will reduce the time for socket waits in
TIMED_WAIT state before closing a socket. If you are tuning /proc/sys/net file
system, the effect will be on all the applications running on the system. However,
to make a change to an individual socket or connection, you have to use Java
Socket API calls (on the respective socket object). Use netstat -p (or the lsof
command) to find the right PID of a particular socket connection and its stack
trace from a javacore file taken with the kill -3 <pid> command.

You can also use IBM’s RAS trace, -Xtrace:print=net, to trace out network-related
activity within the JVM. This technique is helpful when socket-related Java thread
hangs are seen. Correlating output from netstat -p, lsof, JVM net trace, and ps
-efH can help you to diagnose the network-related problems.

Providing summary statistics that are related to your network is useful for
investigating programs that might be underperforming because of TCP/IP
problems. The more you understand your hardware capacity, the easier it is for
you to tune with confidence the parameters of particular system components that
will improve the overall performance of your application. You can also determine
whether only system tuning and tweaking will noticeably improve performance, or
whether actual upgrades are required.

JVM performance
In addition to looking at your overall hardware and system performance, you can
tune several JVM parameters to further increase performance of your Java
application. These parameters are normally set as Java command-line options.
java [-options] class [args...]

OR
java -jar [-options] jarfile [args...]

where options include:

v -Xgcpolicy:optavgpause
v -Xmx
v -Xms
v -Xgcpolicy:optthruput

The Java heap size is one of the most important tunable parameters of your JVM.
It is especially important if you are running several processes and JVMs on your
system. The heap contains all Java objects (live and dead) and free memory.

Garbage collection is based on how full your heap is. Therefore, a large heap size
delays the frequency of garbage collection, but when garbage collection does occur,
it takes longer to complete.

Chapter 15. Linux problem determination 137

Linux - debugging performance problems

What you consider to be an acceptable heap size depends on your application; you
will certainly need to experiment. In addition to balancing the frequency and
length of garbage collections, you must also remember that memory that is
allocated to one applications heap is not available to other applications. This is an
example of fixing a bottleneck in one area, by increasing heap size to decrease
frequency of garbage collection, and causing problems somewhere else. For
example, other processes might have to use paging to supplement their diminished
memory. Under no circumstances should heap size be larger than physical
memory.

-Xms sets the initial heap size and -Xmx sets the maximum heap size.

After you have set the heap size, the verbosegc command shows you information
about garbage collection. The default garbage collection policy is optthruput,
which generally gives the fastest throughput. However, by specifying optavgpause,
you can help programs that are displaying erratic response times, although
throughput will be slower. See Chapter 28, “Garbage Collector diagnostics,” on
page 241 for more information.

JIT
The JIT is another area that can affect the performance of your program. When
deciding whether to use JIT compilation, you must make a balance between faster
execution and increased compilation overhead. The JIT is on by default; you can
turn it off by using -Xint.

It is useful to investigate the JIT when you are debugging performance problems.
For more details about the JIT, see Chapter 4, “Understanding the JIT,” on page 29
and Chapter 27, “JIT problem determination,” on page 237.

You can learn much about your Java application by using hprof, the nonstandard
profiling agent. Statistics about CPU and memory usage are presented along with
many other options. The hprof tool is discussed in detail in Chapter 32, “Using the
JVMPI,” on page 297. -Xrunhprof:help gives you a list of suboptions that you can
use with hprof.

Collecting data from a fault condition in Linux

When a problem occurs, the more information known about the state of the system
environment, the easier it is to reach a diagnosis of the problem. A large set of
information can be collected, although only some of it will be relevant for
particular problems. The following sections tell you the data to collect to help IBM
Java Service solve the problem.

Collecting core files

Collect corefiles to help diagnose many types of problem. Process the corefile with
jextract. The resultant sdff file is useful for service (see “jextract” on page 223).

Producing Javadumps
In some conditions (a crash, for example), a Javadump is produced, usually in the
current directory. In others (for example, a hang) you might have to prompt the
JVM for this by sending the JVM a SIGQUIT (kill -3 <PID>) signal. This is
discussed in more detail in Chapter 21, “Using Javadump,” on page 191.

138 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 collecting data from a fault condition in Linux

Using system logs

The kernel logs system messages and warnings. The system log is located in the
/var/log/messages file. Use it to observe the actions that led to a particular
problem or event. The system log can also help you determine the state of a
system. Other system logs are in the /var/log directory.

Determining the operating environment

The following commands can be useful to determine the operating environment of
a process at various stages of its lifecycle:
uname -a
Provides operating system and hardware information.
df Displays free disk space on a system.
free
Displays memory use information.
ps -ef
Gives a full process list.
lsof
Lists open file handles.
top
Displays process information (such as processor, memory, states) sorted by
default by processor usage.
vmstat
Provides general memory and paging information.

In general, the uname, df, and free output is useful. The other commands may be
run before and after a crash or during a hang to determine the state of a process
and to provide useful diagnostic information.

Sending information to Java Support

When you have collected the output of the commands listed in the previous
section, put that output into files. Compress the files (which could be very large)
before sending them to Java Support. You should compress the files at a very high
ratio.

The following command builds an archive from files {file1,..,fileN} and compresses
them to a file whose name has the format filename.tar.gz:
tar czf filename.tgz file1 file2...filen

Collecting additional diagnostic data

Depending on the type of problem, the following data can also help you diagnose
problems. The information available depends on the way in which Java is invoked
and also the system environment. You will probably have to change the setup and
then restart Java to reproduce the problem with these debugging aids switched on.

proc file system

The /proc file system gives direct access to kernel level information. The /proc/N
directory contains detailed diagnostic information about the process with PID
(process id) N, where N is the id of the process.

Chapter 15. Linux problem determination 139

collecting data from a fault condition in Linux

The command cat /proc/N/maps lists memory segments (including native heap)
for a given process.

strace, ltrace, and mtrace

Use the commands strace, ltrace, and mtrace to collect further diagnostic data. See
“Tracing” on page 131.

Known limitations on Linux

Threads as processes
The JVM for Linux implements Java threads as native threads. On NPTL-enabled
systems such as RHEL3 and SLES9, these are implemented as threads. However
using the LinuxThreads library results in each thread being a separate Linux
process. If the number of Java threads exceeds the maximum number of processes
allowed, your program might:
v Get an error message
v Get a SIGSEGV error
v Hang

Before kernel 2.4, the maximum number of threads available is determined by the
minimum of:
v The user processes setting (ulimit -u) in /etc/security/limits.conf.
v The limit MAX_TASKS_PER_USER defined in /usr/include/linux/tasks.h.
(This change requires the Linux kernel to be recompiled.)
v The limit PTHREAD_THREADS_MAX defined in libpthreads.so. (This change
requires the Linux kernel to be recompiled.)
However, you might run out of virtual storage before reaching the maximum
number of threads.

In kernel 2.4, the native stack size is the main limitation when running a large
number of threads. Use the -Xss option to reduce the size of the thread stack so
that the JVM can handle the required number of threads. For example, set the stack
size to 32 KB on startup.

For more information, see The Volano Report at http://www.volano.com/report/

index.html.

Floating stacks limitations

If you are running without floating stacks, regardless of what is set for -Xss, a
minimum native stack size of 256 KB for each thread is provided. On a floating
stack Linux system, the -Xss values are used. Thus, if you are migrating from a
non-floating stack Linux system, ensure that any -Xss values are large enough and
are not relying on a minimum of 256 KB. (See also “Threading libraries” on page
128.)

glibc limitations
If you receive a message indicating that the libjava.so library could not be loaded
because of a symbol not found (such as __bzero), you might have a down-level
version of the GNU C Runtime Library, glibc, installed. The SDK for Linux thread
implementation requires glibc version 2.1 or greater.

140 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Known limitations on Linux

Font limitations
When you are installing on a Red Hat system, to allow the font server to find the
Java TrueType fonts, run:
/usr/sbin/chkfontpath --add /opt/IBMJava2-131/jre/lib/fonts

You must do this at install time and you must be logged on as ″root″ to run the
command. For more detailed font issues, particularly with regard to Japanese fonts,
see the User Guide for your SDK.

CORBA limitations
Bidirectional GIOP is not supported.

When running with a Java 2 SecurityManager, invocation of some methods in the

CORBA API classes might cause permission checks to be made that could result in
a SecurityExecption. Here is a selection of affected methods:
Table 6. Methods affected when running with Java 2 SecurityManager
Class/Interface Method Required permission
org.omg.CORBA.ORB init java.net.SocketPermission resolve
org.omg.CORBA.ORB connect java.net.SocketPermission listen
org.omg.CORBA.ORB resolve_initial_references java.net.SocketPermission connect
org.omg.CORBA. _is_a java.net.SocketPermission connect
portable.ObjectImpl
org.omg.CORBA. _non_existent java.net.SocketPermission connect
portable.ObjectImpl
org.omg.CORBA. OutputStream _request (String, java.net.SocketPermission connect
portable.ObjectImpl boolean)
org.omg.CORBA. _get_interface_def java.net.SocketPermission connect
portable.ObjectImpl
org.omg.CORBA. invoke java.net.SocketPermission connect
Request
org.omg.CORBA. send_deferred java.net.SocketPermission connect
Request
org.omg.CORBA. send_oneway java.net.SocketPermission connect
Request
javax.rmi. narrow java.net.SocketPermission connect
PortableRemoteObject

Chapter 15. Linux problem determination 141

Known limitations on Linux

142 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 16. Windows problem determination
The Windows SDK is available only internally to IBM for testing purposes. This
chapter describes problem determination on Windows in:
v “Setting up and checking your Windows environment”
v “General debugging techniques” on page 145
v “Diagnosing crashes in Windows” on page 146
v “Debugging hangs” on page 147
v “Debugging memory leaks” on page 147
v “Debugging performance problems” on page 149
v “Collecting data from a fault condition in Windows” on page 150

Setting up and checking your Windows environment

The installation process of the SDK or JRE sets up everything for you. The installer
uses the Windows InstallShield software. If you are using an IBM product with
embedded Java (for example, WebSphere Application Server or WebSphere MQSI),
the product installation process installs Java for you.

The install process is the same on all versions of Windows.

If you experience any difficulty after the installation:

v If you installed Java as part of an IBM product, refer to the manuals for that
product.
v If you installed Java as a standalone product or if you installed Java manually,
check the following environment variables.
PATH
The PATH variable must point to the directory of your Java installation that
contains the file java.exe. Ensure that PATH includes the \bin directory of
your Java installation.
CLASSPATH
The JRE uses this environment variable to find the classes it needs when it
runs. This is useful when the class you want to run uses classes that are
located in other directories. By default, this is blank. If you install a product
that uses the JRE, CLASSPATH is automatically set to point to the JAR files
that the product needs.

A known problem for first-time users is to install Java and then set up a work
directory and compile a ’Hello World’ program. If you cannot run HelloWorld,
possibly the CLASSPATH variable is not pointing to your .CLASS file. A solution
is to type set CLASSPATH=., which always allows you to find classes in your
current directory.

The Java service team has a tool named ReportEnv that plugs into your JVM and
reports on the JVM environment in real time. Your JVM environment affects the
operation of the JVM. ReportEnv reports on environment variables and
command-line parameters. It is a GUI tool, although it can be run without a GUI.
The GUI allows you to browse your environment and, to some extent, dynamically
change it. The tool also has a mechanism to generate reports to tell you the exact

© Copyright IBM Corp. 2003, 2006 143

setting up and checking your Windows environment

state of your JVM environment. A screenshot of the tool is shown in “Setting up

and checking your Windows environment” on page 143. The ReportEnv tool is
available on request from [email protected].

Figure 9 shows the ReportEnv tool.

Figure 9. Screenshot of the ReportEnv tool

Setting up your Windows environment for data collection

Setting up for dump extraction
To enable the JVM to generate a dump for use by the cross platform debugger, see
Chapter 26, “Using the dump formatter,” on page 223.

Setting up for Javadump and Heapdump

Refer to Chapter 21, “Using Javadump,” on page 191 and Chapter 22, “Using
Heapdump,” on page 205.

Native Windows tools

Generating a user dump file in a hang condition: Windows provides a facility
that generates a user dump file for any process (even if it is hung) through a utility
called userdump.exe. This utility is provided by Microsoft and you can download
it from their Web site: www.microsoft.com.

Usage:
userdump -p
Lists all the processes and their pids.
userdump xxx
Creates a dump file of a process that has a pid of xxx (processname.dmp file is
created in the current directory from where userdump.exe is run).

For more information about generating a user dump file in a hang condition, see
“Debugging hangs” on page 147.

144 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Windows - general debugging techniques

General debugging techniques

This section provides a guide to the JVM-provided diagnostic tools and Windows
commands that can be useful when you are diagnosing problems that occur with
the Windows JVM.

Starting Javadumps in Windows

See Chapter 21, “Using Javadump,” on page 191.

Starting Heapdumps in Windows

See Chapter 22, “Using Heapdump,” on page 205.

Using the Cross-Platform Dump Formatter

The IBM Java Cross-Platform Dump Formatter is a powerful tool for debugging
many fault scenarios. As the name implies, it is a cross-platform tool and takes its
input from a predefined data source or code plug-in. The data source must be
generated by platform code because crash dumps vary according to the
architecture. See Chapter 26, “Using the dump formatter,” on page 223 for details.

System dump
When a JVM crash occurs, the JVM requests the operating system to generate a
system dump.

A system dump consists of all the memory that is being used by the JVM; this
includes the application heap, along with all JVM and user libraries. System
dumps allow the IBM service personnel to look at the state of the JVM at the time
of crash, and help them with the problem determination process. Because a system
dump contains all of the memory allocated by the JVM process, system dump files
can be very large.

You can find the location of the generated system dump in the output that is
displayed in the console after the crash. Here is an example of the output:
Unhandled exception
Type=GPF vmState=0x00000003
Target=2_20_20040813_1848_lHdSMR (Windows 2000 5.0 build 2195 Service Pack 4)
CPU=x86 (1 logical CPUs) (0x1ff7c000 RAM)
ExceptionCode=c0000005 ExceptionAddress=1130B074 ContextFlags=0001003f Handler1=1130B07C
Handler2=1130B080
EDI=00074af0 ESI=0000001e EAX=0006f978 EBX=00000000
ECX=00000000 EDX=00230608 EBP=0006f924
EIP=7800f4a2 ESP=0006f6cc
Module=C:\WINNT\system32\MSVCRT.dll
Module_base_address=78000000 Offset_in_DLL=0000f4a2
(I)DUMP0006 Processing Dump Event "gpf", detail "" - Please Wait.
(I)DUMP0007 JVM Requesting System Dump using ’D:\core.20040817.131302.2168.dmp’
(I)DUMP0010 System Dump written to D:\core.20040817.131302.2168.dmp
(I)DUMP0007 JVM Requesting Java Dump using ’D:\javacore.20040817.131319.2168.txt’
(I)DUMP0010 Java Dump written to D:\javacore.20040817.131319.2168.txt
(I)DUMP0013 Processed Dump Event "gpf", detail "".

In this example, the JVM has generated the dump in the file
D:\core.20040817.131302.2168.dmp.

The JVM attempts to generate the system dump file in one of the following
directories (listed in descending order):
1. The directory pointed to by environment variable IBM_COREDIR.

Chapter 16. Windows problem determination 145

Windows - general debugging techniques

2. The current directory.

3. The directory pointed to by the environment variable TMPDIR.
4. The C:\Temp directory

You might want to keep system dumps more private by setting the environment
variable IBM_COREDIR, if you are concerned about passwords and other security
details that are contained in a system dump.

Diagnosing crashes in Windows

You generally see a crash either as an unrecoverable exception thrown by Java or
as a pop-up window notifying you of a General Protection Fault (GPF). The
pop-up usually refers to java.exe as the application that caused the crash. Crashes
can occur because of a fault in the JVM, or because of a fault in native (JNI) code
being run in the Java process.

Try to determine whether the application has any JNI code or uses any third-party
packages that use JNI code (for example, JDBC application drivers, and JVMPI
profiling plug-ins). If this is not the case, the fault must be in the JVM. Otherwise,
the fault must be in other code. Try and find out which is the case so that you can
pinpoint a problem.

As a general rule, try to recreate the crash with minimal dependencies (in terms of
JVM options, JNI applications, or profiling tools).

In a crash condition, gather as much data as possible for the IBM Java service
team. You should:
v Collect the Javadump. See Chapter 21, “Using Javadump,” on page 191 for more
details on using Javadump.
v Collect the crash dump. See “Setting up and checking your Windows
environment” on page 143 for details.
v Run with the JIT turned off. See Chapter 27, “JIT problem determination,” on
page 237 for details.
v If the problem occurs with or without the JIT, specify the JVM option
-Xjit:count=n. -Xjit:count=n specifies the number of times a method is invoked
before it is compiled. This way, the JVM starts up reasonably quickly (there is no
overhead of ″JITting″ all the basic methods) and keeps the advantages of having
a JIT. During the default JIT operation, some methods in your code are
interpreted and some are executed as native code, depending on whether they
have hit the threshold. If you pass the option -Xjit:count=0 to the JVM, the JIT
starts ″JITting″ all methods (that is, no code is interpreted). Run your application
with -Xjit:count=0 and collect the Javadump and the log. This is the opposite of
the previous scenario where no code was ″JIT’d″.
v Collect the Javadump log if the problem still occurs.
v Try some JIT compile options. If the problem disappears with the JIT turned off,
try some JIT compile options to see if the problem can be narrowed down
further. You could find that you can continue using the JVM, albeit with reduced
JIT performance, while giving the service team a running start with your bug
report. For information on using the basic JIT compile options, see Chapter 27,
“JIT problem determination,” on page 237.
v Try adjusting the garbage collection parameters. See Chapter 2, “Understanding
the Garbage Collector,” on page 7 for details. Make a note of any changes in
behavior.

146 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Diagnosing crashes in Windows

v Try running on a uniprocessor box. If your problem is occurring on a

multiprocessor system, test your application on a uniprocessor box. You can use
the BIOS options on your SMP box to reset the processor affinity to 1 to make it
behave like a uniprocessor. If the problem disappears, make a note in your bug
report. Otherwise, collect the crash dump.

Data to send to IBM

At this point you potentially have several sets of either logs or dumps, or both (for
example one set for normal running, one set with JIT off, and so on). Label them
appropriately and make them available to IBM. (See Part 2, “Submitting problem
reports,” on page 75 for details.) The required files are:
v JVM-produced Javadump file (Javacore)
v XML file generated by jextract

Debugging hangs
Hangs refer to the JVM locking-up or refusing to respond. A hang can occur when:
v Your application entered an infinite loop.
v A deadlock has occurred
To determine which of these situations applies, open the Windows Task Manager
and select the Performance tab. If the CPU time is 100% and your system is
running very slowly, the JVM is very likely to have entered an infinite loop.
Otherwise, if CPU usage is normal, you are more likely to have a deadlock
situation.

Analyzing deadlocks
For an explanation of deadlocks and diagnosing them using the Javadump tool,
see “Locks, monitors, and deadlocks (LOCKS)” on page 193.

Getting a dump from a hung JVM

The Windows JVM is configured to do a dump extraction if it terminates
abnormally. Also, you can cause a dump by configuring the JVM to respond
appropriately to a SIGBREAK signal. This signal is tied, by default, to the Ctrl +
Break key combination. However, neither of these methods is particularly useful if
the JVM is hung up somehow.

For these conditions, the IBM Java service team can supply a small stand-alone
utility program that is called jvmdump.exe. This program takes a single parameter
that is the PID of a process. When run, the programme generates a minidump that
you can analyze through WinDbg, or translate into a dump-formatter dump in the
usual way. (See Chapter 26, “Using the dump formatter,” on page 223 for details.)
The jvmdump application is provided as-is. If you would like a copy, e-mail
[email protected].

Alternatively, if you have the Microsoft debugging tools installed, you can use
Windbg to generate a minidump. See “Generating a user dump file in a hang
condition” on page 144 for more information.

Debugging memory leaks

This section begins with a discussion of the Windows memory model and the Java
heap to provide background understanding before going into the details of
memory leaks.

Chapter 16. Windows problem determination 147

Windows - debugging memory leaks

The Windows memory model

Native memory leaks are not usually relevant to Java so these are discussed very
briefly.

Windows memory is virtualized. Applications do not have direct access to memory

addresses, so allowing Windows to move physical memory and to swap memory
in and out of a swapper file (called pagefile.sys).

Allocating memory is usually a two-stage process. Simply allocating memory

results in an application getting a handle. No physical memory is reserved. There
are more handles than physical memory. To use memory, it must be ’committed’.
At this stage, a handle references physical memory. This might not be all the
memory you requested.

For example, the stack allocated to a thread is normally given a small amount of
actual memory. If the stack overflows, an exception is thrown and the operating
system allocates more physical memory so that the stack can grow.

Memory manipulation by Windows programmers is hidden inside libraries

provided for the chosen programming environment. In the C environment, the
basic memory manipulation routines are the familiar malloc and free functions.
Windows APIs sit on top of these libraries and generally provide a further level of
abstraction.

From the point of view of a programmer, Windows provides a flat memory model,
in which addresses run from 0 up to the limit allowed for an application.
Applications can choose to segment their memory. On a dump, the programmer
sees sets of discrete memory addresses.

Classifying leaks
The following scenarios are possible :
v Windows memory usage is increasing, Java heap is static:
– Memory leak in application.
– Memory leak in JNI.
– Leak with hybrid Java and native objects (very rare).
v Windows memory usage increases because the heap keeps increasing:
– Memory leak in application Java code. (See “Common causes of perceived
leaks” on page 241 below.)
– Memory leak internal to JVM.

Tracing leaks
–Xrunjnichk option
You can use the -Xrunjnichk option to trace JNI calls that are made by your JNI
code or by any JVM components that use JNI. This helps you to identify incorrect
uses of JNI libraries from native code, and can help you to diagnose JNI memory
leaks. Note that -Xrunjnichk is equivalent to -Xcheck:jni. See “Debugging the JNI”
on page 67 for information on the -Xrunjnichk suboptions.

–memorycheck option
The -memorycheck option can help you identify memory leaks inside the JVM.
The -memorycheck option traces the JVM calls to the operating system’s malloc()

148 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Windows - debugging memory leaks

and free() functions, and identifies any JVM mistakes in memory allocation. See
Appendix F, “Command-line options,” on page 329 for more information.

Some useful techniques are built into the JVM:

v The-verbosegc option
v HeapDump: See Chapter 22, “Using Heapdump,” on page 205
v HPROF tools

Using HeapDump to debug memory leaks

For details about analyzing the Java Heap, see Chapter 22, “Using Heapdump,” on
page 205.

Debugging performance problems

Performance-related problems occur when:
v Applications consume 100% CPU when not required.
v Unnecessary events that can hinder performance are generated from the virtual
machine.
v Memory consumption with JVM is abnormal, but the program seems to be
running normally.
v Your application is very slow.

When a Java application seems to be running slowly, you should check the various
JIT options and ensure that a suitable JIT compiler exists for the virtual machine
before you try anything else. Refer to Chapter 27, “JIT problem determination,” on
page 237.

Use the hprof tool, which can help find the CPU usage problems with
applications. Different CPU options can be used to identify the method or thread
that consumes more CPU time. Hprof does not calculate the count of CPU
utilization by internal methods, but flattens the hierarchy of the methods and adds
the counts to the method that is at a lower level in the stack trace. Refer to java
-Xrunhprof:help (in Chapter 32, “Using the JVMPI,” on page 297) for further
options.

The memory consumption performance issues can be addressed by various

garbage collection options. Refer to Chapter 28, “Garbage Collector diagnostics,” on
page 241. Verify that the OS is tuned with sufficient paging memory for Java heap
management. The application heap tuning also plays a vital role. Using System.gc()
is not a good option because it is totally virtual machine dependent and cannot be
used to optimize the memory usage. Instead, your applications should take proper
care in managing the memory allocated to different objects. If you do use
System.gc(), try making it optionally compilable and switch it off to check if this is
impacting your performance. You can find general guidance on good garbage
collection practice in Chapter 2, “Understanding the Garbage Collector,” on page 7.

Other tools, such as JProf, ProGuard, and JinSight, can give further inputs on
various parameters of a program running in Java.

Data required for submitting a bug report

IBM service requires:
v Description of performance issue.

Chapter 16. Windows problem determination 149

Windows - debugging performance problems

v A heapdump (see Chapter 22, “Using Heapdump,” on page 205) if you think
that you have a memory consumption problem.
v Javadump snapshots (see Chapter 21, “Using Javadump,” on page 191) of the
JVM before performance became a problem and after.
v If performance is a permanent problem, send a couple of snapshots that are
separated by approximately 10 minutes, by using the dump extractor (see
“General debugging techniques” on page 145) after the point at which
performance became a problem.

Frequently reported problems

IBM service often receives problems that are caused by:
v Garbage collection cycles consuming too much processor time:
1. System.gc() check. Check for and remove any unwanted System.gc() calls in
your code. If you want to use this call, make it conditionally compilable and
check whether switching it off addresses performance issues.
2. Heap management check. If your heap is too small, for example, the Garbage
Collector will continually run into allocation faults. Refer to Chapter 28,
“Garbage Collector diagnostics,” on page 241 and Chapter 2, “Understanding
the Garbage Collector,” on page 7 for data to help you to set the correct heap
size and tune the way garbage collection runs.
v Unused objects are not being collected.
See “Common causes of perceived leaks” on page 241.
v Heap never shrinks.
Refer to Chapter 2, “Understanding the Garbage Collector,” on page 7 for
conditions under which this can occur.

Collecting data from a fault condition in Windows

The more information that you can collect about a problem, the easier it is to
diagnose that problem. A large set of data can be collected, although some is
relevant to particular problems. The following list describes a typical data-set that
you can collect to assist IBM service to fix your problem.
v Javadumps. These can be generated automatically or manually. Automatic
dumps are essential for IBM service.
v Heapdumps. If generated automatically, they are essential. They are also
essential if you have a memory or performance problem.
v System dump generated by the JVM. See “System dump” on page 145. This is
the key to most problems.
v WebSphere Application Server logs (see Chapter 13, “Working in a WebSphere
Application Server environment,” on page 93), if you are working in a
WebSphere Application Server environment.
v Other data, as determined by your particular problem.

150 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 17. z/OS problem determination
This chapter describes problem determination on z/OS in:
v “Setting up and checking your z/OS environment”
v “General debugging techniques” on page 152
v “Diagnosing crashes” on page 153
v “Debugging hangs” on page 160
v “Debugging memory leaks” on page 161
v “Debugging performance problems” on page 163
v “Collecting data from a fault condition in z/OS” on page 164

Setting up and checking your z/OS environment

Maintenance
The Java for z/OS website at:

http://www-1.ibm.com/servers/eserver/zseries/software/java/

has up-to-date information about any changing operating system prerequisites for
correct JVM operation. In addition, any new prerequisites are described in PTF
HOLDDATA.

LE settings
Language Environment (LE) Runtime Options (RTOs) affect operation of C and
C++ programs such as the JVM. In general, the options that developers set by
using C #pragma statements in the code should not be overridden because they
are generated as a result of testing to provide the best operation of the JVM.

Environment variables
Environment variables that change the operation of the JVM in one release can be
deprecated or change meaning in a following release. Therefore, you should review
environment variables that are set for one release, to ensure that they still apply
after any upgrade.

Private storage usage

The single most common class of failures after a successful install of the SDK are
those related to insufficient private storage. As discussed in detail in “Debugging
memory leaks” on page 161, LE provides storage from Subpool 2, key 8 for C/C++
programs like the JVM that use C RTL calls like malloc() to obtain memory. The LE
HEAP refers to the areas obtained for all C/C++ programs that run in a process
address space and request storage.

This area is used for the allocation of the Java heap where instances of Java objects
are allocated and managed by Garbage Collection. The area is used also for any
underlying allocations that the JVM makes during operations. For example, the JIT
compiler obtains work areas for compilation of methods and to store compiled
code.

© Copyright IBM Corp. 2003, 2006 151

z/OS - setting up and checking the environment

Because the JVM must preallocate the maximum Java heap size so that it is
contiguous, the total private area requirement is that of the maximum Java heap
size that is set by the -Xmx option (or the 64 MB default if this is not set), plus an
allowance for underlying allocations. A total private area of 140 MB is therefore a
reasonable requirement for an instance of a JVM that has the default maximum
heap size.

If the private area is restricted by either a system parameter or user exit, failures to
obtain private storage occur. These failures show as OutOfMemoryErrors or
Exceptions, failures to load dlls, or failures to complete subcomponent initialization
during startup.

Setting up dumps
The JVM, by default, generates a Javadump and System Transaction Dump
(SYSTDUMP) when any of the following occurs:
v A SIGQUIT signal is received
v The JVM aborts because of a fatal error
v An unexpected native exception occurs (for example, a SIGSEGV, SIGILL, or
SIGFPE signal is received)

You can use JAVA_DUMP_OPTS to change the dumps that are produced on the
various types of signal. You can use JAVA_DUMP_TDUMP_PATTERN to change
the naming convention to which the SYSTDUMP is written as an MVS dataset.
Both of these variables are described in Chapter 23, “JVM dump initiation,” on
page 209.

General debugging techniques

Starting Javadumps in z/OS

See Chapter 21, “Using Javadump,” on page 191.

Starting Heapdumps in z/OS

See Chapter 22, “Using Heapdump,” on page 205.

Using IPCS commands

Here are some sample IPCS commands that you might find useful during your
debugging sessions. In this case, the address space of interest is ASID(x’7D’).
ip verbx ledata ’nthreads(*)’
This command formats out all the C-stacks (DSAs) for threads in the
process that is the default ASID for the dump.
ip setd asid(x’007d’)
This command is to set the default ASID use command setdef; for
example, to set the default asid to x’007d’.
ip verbx ledata ’all,asid(007d),tcb(tttttt)’
In this command, the all report formats out key LE control blocks such as
CAA, PCB,
ZMCH, CIB. In particular, the CIB/ZMCH captures the PSW and GPRs at
the time the program check occurred.
ip verbx ledata ’cee,asid(007d),tcb(tttttt)’
This command formats out the traceback for one specific thread.

152 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 z/OS - general debugging techniques

ip summ regs asid(x’007d’)

This command formats out the TCB/RB structure for the address space. It
is rarely useful for JVM debugging.
ip verbx sumdump
Then issue find ’slip regs sa’ to locate the GPRs and PSW at the time a
SLIP TRAP s matched. This command is useful for the case where you set
a SA (Storage Alter) trap to catch an overlay of storage.
ip omvsdata process detail asid(x’007d’)
This command generates a report for the process showing the thread status
from a USS kernel perspective.
ip select all
This command generates a list of the address spaces in the system at the
time of the dump, so you can tie up the ASID with the JOBNAME.
ip systrace asid(x’007d’) time(gmt)
This command formats out the system trace entries for all threads in this
address space. It is useful for diagnosing loops. time(gmt) converts the
TOD Clock entries in the system trace to a human readable form.

Using dbx
The dbx utility has been improved for z/OS V1R6. You can use dbx to analyze
transaction dumps and to debug a running application. For information about dbx,
see the z/OS documentation (z/OS V1R6.0 Unix System Services Programming Tools).

Interpreting error message IDs

While working in the OMVS, if you get an error message and if you want to
understand exactly what the error message means, go to: http://www-1.ibm.com/
servers/s390/os390/bkserv/lookat/lookat.html and enter the message ID. Then
select your OS level and then press enter. The output will give a better
understanding of the error message. To decode the errno2 values, use the following
command:
bpxmtext <reason_code>

Reason_code is specified as 8 hexadecimal characters. Leading zeroes may be

omitted.

Diagnosing crashes
A crash should occur only because of a fault in the JVM, or because of a fault in
native (JNI) code that is being run inside the Java process. A crash is more strictly
defined on z/OS as a program check that is handled by z/OS UNIX as a fatal
signal (for example, SIGSEGV for PIC4, 10 or 11 or SIGILL for PIC1).

Documents to gather
When one of these fatal signals occurs, the JVM Signal Handler takes control. The
default action of the signal handler is to request an unformatted transaction dump
through the BCP IEATDUMP service and to produce a formatted dump of internal
JVM state, which is known as the Javadump. Output should be written to the
message stream that is written to stderr in the form of:

Chapter 17. z/OS problem determination 153

z/OS - diagnosing crashes

Unhandled exception
Type=GPF vmState=0x00000000
Target=2_20_20040813_1849_BHdSMr (z/OS 06.00)
CPU=s390x (2 logical CPUs) (0x0 RAM)
signal=0000000b
gpr0=00000000000003e7 gpr1=0000000000000000 gpr2=0000000100006160 gpr3=0000000000000010
gpr4=00000001082fe780 gpr5=00000000000000c0 gpr6=0000000000000000 gpr7=00000000122c66e8
gpr8=0000000000000007 gpr9=00000000122c6708 gpr10=0000000108377e70 gpr11=000000010c83fb78
gpr12=0000000108300c60 gpr13=0000000108377e00 gpr14=000000007cd18938 gpr15=0000000000000000
fpr0=4841230c72000000 fpr1=4580000000000000 fpr2=4e80000117ddc374 fpr3=3ff0000000000000
fpr4=406f000000000000 fpr5=0000000000000000 fpr6=0000000000000000 fpr7=0000000000000000
fpr8=0000000000000000 fpr9=0000000000000000 fpr10=0000000000000000 fpr11=0000000000000000
fpr12=0000000000000000 fpr13=0000000000000000 fpr14=0000000000000000 fpr15=0000000000000000
psw0=0785240180000000 psw1=00000000122c66f8 fpc=0000000000000000
JVMDUMP0006I Processing Dump Event "gpf", detail "" - Please Wait.
JVMDUMP0007I JVM Requesting System Dump using ’CWHITE.JVM.TDUMP.CWHITE9.D040818.T085946’
IEATDUMP success for DSN=’CWHITE.JVM.TDUMP.CWHITE9.D040818.T085946’
CEEDUMP success for FILE=’/u/cwhite/work/Test/dumpTest/CEEDUMP.20040818.085947.67240533’
JVMDUMP0007I JVM Requesting CEE Dump using ’/u/cwhite/work/Test/dumpTest/CEEDUMP.20040818.090014.67240533’
CEEDUMP success for FILE=’/u/cwhite/work/Test/dumpTest/CEEDUMP.20040818.090014.67240533’
JVMDUMP0010I CEE Dump written to /u/cwhite/work/Test/dumpTest/CEEDUMP.20040818.090014.67240533
JVMDUMP0007I JVM Requesting Java Dump using ’/u/cwhite/work/Test/dumpTest/javacore.20040818.090040.67240533.txt’
JVMDUMP0010I Java Dump written to /u/cwhite/work/Test/dumpTest/javacore.20040818.090040.67240533.txt
JVMDUMP0013I Processed Dump Event "gpf", detail "".

The output shows the location in HFS into which the Javadump file was written
and the name of the MVS dataset to which the transaction dump is written. These
locations are configurable and are described in Chapter 20, “Overview of the
available diagnostics,” on page 187 and Chapter 23, “JVM dump initiation,” on
page 209.

These two documents (that is, the Javadump file and the transaction dump)
provide the ability to determine the failing function, and therefore decide which
product owns the failing code, be it the JVM, application JNI code, or third part
native libraries (for example native JDBC drivers).

Determining the failing function

Any one of the three documents that you gathered, (see “Documents to gather” on
page 153) should be enough to determine the failing function, and therefore
determine to which IBM support group the problem should be routed, or whether
application native C code is at fault.

The most practical way to find where the exception occurred is to review either the
CEEDUMP or the Javadump. Both of these show where the exception occurred and
the native stack trace for the failing thread. The same information can be obtained
from the transaction dump by using either IPCS LEDATA VERB Exit, or the
svcdump.jar toolset. These generate a report that is similar to the CEEDUMP.

In each case, the report shows the C-Stack (or native stack, which is separate from
the Java stack that is built by the JVM because one method gives control to
another). The C-stack frames are also known on z/OS as DSAs, because this is the
name of the control block that LE provides as a native stack frame for a C/C++
program. The following traceback from a CEEDUMP shows where a failure
occurred:

154 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 z/OS - diagnosing crashes

Traceback:
DSA Entry E Offset Load Mod Program Unit Service Status
00000001 __cdump +00000000 CELQLIB HLE7709 Call
00000002 @@WRAP@MULTHD
+00000266 CELQLIB Call
00000003 j9dump_create
+0000035C *PATHNAM j040813 Call
00000004 doSystemDump+0000008C *PATHNAM j040813 Call
00000005 triggerDumpAgents
+00000270 *PATHNAM j040813 Call
00000006 vmGPHandler +00000C4C *PATHNAM j040813 Call
00000007 gpHandler +000000D4 *PATHNAM j040813 Call
00000008 __zerro +00000BC4 CELQLIB HLE7709 Call
00000009 __zerros +0000016E CELQLIB HLE7709 Call
0000000A CEEHDSP +00003A2C CELQLIB CEEHDSP HLE7709 Call
0000000B CEEOSIGJ +00000956 CELQLIB CEEOSIGJ HLE7709 Call
0000000C CELQHROD +00000256 CELQLIB CELQHROD HLE7709 Call
0000000D CEEOSIGG -08B3FBBC CELQLIB CEEOSIGG HLE7709 Call
0000000E CELQHROD +00000256 CELQLIB CELQHROD HLE7709 Call
0000000F Java_dumpTest_runTest
+00000044 *PATHNAM Exception
00000010 RUNCALLINMETHOD
-0000F004 *PATHNAM Call
00000011 gpProtectedRunCallInMethod
+00000044 *PATHNAM j040813 Call
00000012 j9gp_protect+00000028 *PATHNAM j040813 Call
00000013 gpCheckCallin
+00000076 *PATHNAM j040813 Call
00000014 callStaticVoidMethod
+00000098 *PATHNAM j040813 Call
00000015 main +000029B2 *PATHNAM j904081 Call
00000016 CELQINIT +00001146 CELQLIB CELQINIT HLE7709 Call

DSA DSA Addr E Addr PU Addr PU Offset Comp Date Attributes

00000001 00000001082F78E0 000000001110EB38 0000000000000000 ******** 20040312 XPLINK EBCDIC POSIX IEEE
00000002 00000001082F7A20 00000000110AF458 0000000000000000 ******** 20040312 XPLINK EBCDIC POSIX Floating Point
00000003 00000001082F7C00 0000000011202988 0000000000000000 ******** 20040817 XPLINK EBCDIC POSIX IEEE
00000004 00000001082F8100 0000000011213770 0000000000000000 ******** 20040817 XPLINK EBCDIC POSIX IEEE
00000005 00000001082F8200 0000000011219760 0000000000000000 ******** 20040817 XPLINK EBCDIC POSIX IEEE
00000006 00000001082F8540 000000007CD4BDA8 0000000000000000 ******** 20040817 XPLINK EBCDIC POSIX IEEE
00000007 00000001082F9380 00000000111FF190 0000000000000000 ******** 20040817 XPLINK EBCDIC POSIX IEEE
00000008 00000001082F9480 00000000111121E0 0000000000000000 ******** 20040312 XPLINK EBCDIC POSIX IEEE
00000009 00000001082FA0C0 0000000011112048 0000000000000000 ******** 20040312 XPLINK EBCDIC POSIX IEEE
0000000A 00000001082FA1C0 0000000010DB8EA0 0000000010DB8EA0 00003A2C 20040312 XPLINK EBCDIC POSIX Floating Point
0000000B 00000001082FCAE0 0000000010E3D530 0000000010E3D530 00000956 20040312 XPLINK EBCDIC POSIX Floating Point
0000000C 00000001082FD4E0 0000000010D76778 0000000010D76778 00000256 20040312 XPLINK EBCDIC POSIX Floating Point
0000000D 00000001082FD720 0000000010E36C08 0000000010E36C08 08B3FBB0 20040312 XPLINK EBCDIC POSIX Floating Point
0000000E 00000001082FE540 0000000010D76778 0000000010D76778 00000256 20040312 XPLINK EBCDIC POSIX Floating Point
0000000F 00000001082FE780 00000000122C66B0 0000000000000000 ******** 20040802 XPLINK EBCDIC POSIX IEEE
00000010 00000001082FE880 000000007CD28030 0000000000000000 ******** ^C"^22^04^FF^FDu^58 XPLINK EBCDIC POSIX IEEE
00000011 00000001082FEC80 000000007CD515B8 0000000000000000 ******** 20040817 XPLINK EBCDIC POSIX IEEE
00000012 00000001082FED80 00000000111FF948 0000000000000000 ******** 20040817 XPLINK EBCDIC POSIX IEEE
00000013 00000001082FEE80 000000007CD531A8 0000000000000000 ******** 20040817 XPLINK EBCDIC POSIX IEEE
00000014 00000001082FEF80 000000007CD4F148 0000000000000000 ******** 20040817 XPLINK EBCDIC POSIX IEEE

Notes:
1. The stack frame that has a status value of Exception indicates where the crash
occurred. In this example, the crash occurs in the function
Java_dumpTest_runTest.
2. The value under Service for each DSA is the service string. The string is built in
the format of jyymmdd, where j is the identifier for the code owner and yymmdd
is the build date. A service string like this indicates that the function is part of
the JVM. Similarly, any program unit whose z/OS UNIX filename begins with
/u/sovbld is part of the JVM. All functions should have the same build date,
unless you have been supplied with a dll by IBM Service for diagnostic or
temporary fix purposes.

Working with TDUMPs using IPCS

A TDUMP or Transaction Dump is generated from the MVS service IEATDUMP by
default in the event of a program check or exception in the JVM. You can disable
the generation of a TDUMP, but IBM Service does not recommended you to do
that.

Chapter 17. z/OS problem determination 155

z/OS - diagnosing crashes

The normal way to inspect a TDUMP is by using IPCS (see “Using IPCS
commands” on page 152). You can also inspect a TDUMP using a Java application
such as svcdump, or jdmpview, if the dump data set has been transferred in binary
mode to the inspecting system.

A TDUMP can contain multiple Address Spaces. It is important to work with the
correct address space associated with the failing java process.

Adding the dump file to the IPCS inventory

To work with a TDUMP in IPCS, here is a sample set of steps to add the dump file
to the IPCS inventory:
1. Browse the dump data set to check the format and to ensure that the dump is
correct.
2. In IPCS option 3 (Utility Menu), sub option 4 (Process list of data set names)
type in the TSO HLQ (for example, DUMPHLQ) and press Enter to list data sets.
You must ADD (A in the command-line alongside the relevant data set) the
uncompressed (untersed) data set to the IPCS inventory.
3. You may select this dump as the default one to analyze in two ways:
v In IPCS option 4 (Inventory Menu) type SD to add the selected data set name
to the default globals.
v In IPCS option 0 (DEFAULTS Menu), change Scope and Source
Scope ==> BOTH (LOCAL, GLOBAL, or BOTH)

Source ==> DSNAME(’DUMPHLQ.UNTERSED.SIGSEGV.DUMP’)

Address Space ==>
Message Routing ==> NOPRINT TERMINAL
Message Control ==> CONFIRM VERIFY FLAG(WARNING)
Display Content ==> NOMACHINE REMARK REQUEST NOSTORAGE SYMBOL

If you change the Source default, IPCS displays the current default address
space for the new source and ignores any data entered in the address space
field.
4. To initialize the dump, select one of the analysis functions, such as IPCS option
2.4 SUMMARY - Address spaces and tasks, which will display something like
the following and give the TCB address. (Note that non-zero CMP entries
reflect the termination code.)
TCB: 009EC1B0
CMP...... 940C4000 PKF...... 80 LMP...... FF DSP...... 8C
TSFLG.... 20 STAB..... 009FD420 NDSP..... 00002000
JSCB..... 009ECCB4 BITS..... 00000000 DAR...... 00
RTWA..... 7F8BEDF0 FBYT1.... 08
Task non-dispatchability flags from TCBFLGS5:
Secondary non-dispatchability indicator
Task non-dispatchability flags from TCBNDSP2:
SVC Dump is executing for another task

SVRB: 009FD9A8
WLIC..... 00000000 OPSW..... 070C0000 81035E40
LINK..... 009D1138

PRB: 009D1138
WLIC..... 00040011 OPSW..... 078D1400 B258B108
LINK..... 009ECBF8
EP....... DFSPCJB0 ENTPT.... 80008EF0

PRB: 009ECBF8
WLIC..... 00020006 OPSW..... 078D1000 800091D6
LINK..... 009ECC80

156 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 z/OS - diagnosing crashes

Useful IPCS commands and some sample output

In IPCS option 6 (COMMAND Menu) type in a command and press the Enter
key:
ip st
Provides a status report.
ip select all
Shows the Jobname to ASID mapping:
ASID JOBNAME ASCBADDR SELECTION CRITERIA
---- -------- -------- ------------------
0090 H121790 00EFAB80 ALL
0092 BPXAS 00F2E280 ALL
0093 BWASP01 00F2E400 ALL
0094 BWASP03 00F00900 ALL
0095 BWEBP18 00F2EB80 ALL
0096 BPXAS 00F8A880 ALL
ip systrace all time(local)
Shows the system trace:
PR ASID,WU-ADDR- IDENT CD/D PSW----- ADDRESS- UNIQUE-1 UNIQUE-2 UNIQUE-3
UNIQUE-4 UNIQUE-5 UNIQUE-6

09-0094 009DFE88 SVCR 6 078D3400 8DBF7A4E8AA6C648 0000007A 24AC2408

09-0094 05C04E50 SRB 070C0000 8AA709B800000094 02CC90C0 02CC90EC
009DFE88 A0
09-0094 05C04E50 PC ... 0 0AA70A06 0030B
09-0094 00000000 SSRV 132 00000000 0000E602 00002000 7EF16000
00940000

For suspected loops you might need to concentrate on ASID and exclude any
branch tracing:
ip systrace asid(x’3c’) exclude(br)
ip summ format asid(x’94’)
To find the list of TCBs, issue a find command for ″T C B″.
ip verbx ledata ’ceedump asid(94) tcb(009DFE88)
Obtains a traceback for the specified TCB.
ip omvsdata process detail asid(x’94’)
Shows a USS perspective for each thread.
ip verbx vsmdata ’summary noglobal’
Provides a summary of the local data area:
LOCAL STORAGE MAP
___________________________
| |80000000 <- Top of Ext. Private
| Extended |
| LSQA/SWA/229/230 |80000000 <- Max Ext. User Region Address
|___________________________|7F4AE000 <- ELSQA Bottom
| |
| (Free Extended Storage) |
|___________________________|127FE000 <- Ext. User Region Top
| |
| Extended User Region |
|___________________________|10D00000 <- Ext. User Region Start
: :
: Extended Global Storage :
=======================================<- 16M Line
: Global Storage :
:___________________________: A00000 <- Top of Private
| |
| LSQA/SWA/229/230 | A00000 <- Max User Region Address

Chapter 17. z/OS problem determination 157

z/OS - diagnosing crashes

|___________________________| 9B8000 <- LSQA Bottom

| |
| (Free Storage) |
|___________________________| 7000 <- User Region Top
| |
| User Region |
|___________________________| 6000 <- User Region Start
: System Storage :
:___________________________: 0

Input Specifications:

Region Requested => 3600000

IEFUSI/SMF Specification => SMFL : FFFFFFFF SMFEL: FFFFFFFF
SMFR : FFFFFFFF SMFER: FFFFFFFF
Actual Limit => LIMIT: 9FA000 ELIM : 7F606000

Summary of Key Information from LDA (Local Data Area) :

STRTA = 6000 (ADDRESS of start of private storage area)

SIZA = 9FA000 (SIZE of private storage area)
CRGTP = 7000 (ADDRESS of current top of user region)
LIMIT = 9FA000 (Maximum SIZE of user region)
LOAL = 1000 (TOTAL bytes allocated to user region)
HIAL = 43000 (TOTAL bytes allocated to LSQA/SWA/229/230 region)
SMFL = FFFFFFFF (IEFUSI specification of LIMIT)
SMFR = FFFFFFFF (IEFUSI specification of VVRG)

ESTRA = 10D00000 (ADDRESS of start of extended private storage area)

ESIZA = 6F300000 (SIZE of extended private storage area)
ERGTP = 127FE000 (ADDRESS of current top of extended user region)
ELIM = 7F606000 (Maximum SIZE of extended user region)
ELOAL = 1AFD000 (TOTAL bytes allocated to extended user region)
EHIAL = B36000 (TOTAL bytes allocated to extended LSQA/SWA/229/230)
SMFEL = FFFFFFFF (IEFUSI specification of ELIM)
SMFER = FFFFFFFF (IEFUSI specification of EVVRG)
ip verbx ledata ’nthreads(*)’
Obtains the tracebacks for all threads.
ip status regs
Shows the PSW and registers:
CPU STATUS:
BLS18058I Warnings regarding STRUCTURE(Psa) at ASID(X’0001’) 00:
BLS18300I Storage not in dump
PSW=00000000 00000000
(Running in PRIMARY key 0 AMODE 24 DAT OFF)
DISABLED FOR PER I/O EXT MCH
ASCB99 at FA3200 JOB(JAVADV1) for the home ASID
ASXB99 at 8FDD00 and TCB99G at 8C90F8 for the home ASID
HOME ASID: 0063 PRIMARY ASID: 0063 SECONDARY ASID: 0063

General purpose register values

Left halves of all registers contain zeros
0-3 00000000 00000000 00000000 00000000
4-7 00000000 00000000 00000000 00000000
8-11 00000000 00000000 00000000 00000000
12-15 00000000 00000000 00000000 00000000

Access register values

0-3 00000000 00000000 00000000 00000000
4-7 00000000 00000000 00000000 00000000
8-11 00000000 00000000 00000000 00000000
12-15 00000000 00000000 00000000 00000000

158 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 z/OS - diagnosing crashes

Control register values

0-1 00000000_5F04EE50 00000001_FFC3C007
2-3 00000000_5A057800 00000001_00C00063
4-5 00000000_00000063 00000000_048158C0
6-7 00000000_00000000 00000001_FFC3C007
8-9 00000000_00000000 00000000_00000000
10-11 00000000_00000000 00000000_00000000
12-13 00000000_0381829F 00000001_FFC3C007
14-15 00000000_DF884811 00000000_7F5DC138
ip cbf rtct
Helps you to find the ASID by looking at the ASTB mapping to see which
ASIDs are captured in the dump.
ip verbx vsmdata ’nog summ’
Provides a summary of the virtual storage management data areas:
DATA FOR SUBPOOL 2 KEY 8 FOLLOWS:

-- DQE LISTING (VIRTUAL BELOW, REAL ANY64)

DQE: ADDR 12C1D000 SIZE 32000

DQE: ADDR 1305D000 SIZE 800000
DQE: ADDR 14270000 SIZE 200000
DQE: ADDR 14470000 SIZE 10002000
DQE: ADDR 24472000 SIZE 403000
DQE: ADDR 24875000 SIZE 403000
DQE: ADDR 24C78000 SIZE 83000
DQE: ADDR 24CFB000 SIZE 200000
DQE: ADDR 250FD000 SIZE 39B000
FQE: ADDR 25497028 SIZE FD8
DQE: ADDR 25498000 SIZE 735000
FQE: ADDR 25BCC028 SIZE FD8
DQE: ADDR 25D36000 SIZE 200000
DQE: ADDR 29897000 SIZE 200000
DQE: ADDR 2A7F4000 SIZE 200000
DQE: ADDR 2A9F4000 SIZE 200000
DQE: ADDR 2AC2F000 SIZE 735000
FQE: ADDR 2B363028 SIZE FD8
DQE: ADDR 2B383000 SIZE 200000
DQE: ADDR 2B5C7000 SIZE 200000
DQE: ADDR 2B857000 SIZE 1000

***** SUBPOOL 2 KEY 8 TOTAL ALLOC: 132C3000 ( 00000000 BELOW, 132C3000

ip verbx ledata ’all asid(54) tcb(009FD098)’

Finds the PSW and registers at time of the exception:
+000348 MCH_EYE:ZMCH
+000350 MCH_GPR00:00000000 000003E7 MCH_GPR01:00000000 00000000
+000360 MCH_GPR02:00000001 00006160 MCH_GPR03:00000000 00000010
+000370 MCH_GPR04:00000001 082FE780 MCH_GPR05:00000000 000000C0
+000380 MCH_GPR06:00000000 00000000 MCH_GPR07:00000000 127FC6E8
+000390 MCH_GPR08:00000000 00000007 MCH_GPR09:00000000 127FC708
+0003A0 MCH_GPR10:00000001 08377D70 MCH_GPR11:00000001 0C83FB78
+0003B0 MCH_GPR12:00000001 08300C60 MCH_GPR13:00000001 08377D00
+0003C0 MCH_GPR14:00000000 112100D0 MCH_GPR15:00000000 00000000
+0003D0 MCH_PSW:07852401 80000000 00000000 127FC6F8 MCH_ILC:0004
+0003E2 MCH_IC1:00 MCH_IC2:04 MCH_PFT:00000000 00000000
+0003F0 MCH_FLT_0:48410E4F 6C000000 4E800001 31F20A8D
+000400 MCH_FLT_2:406F0000 00000000 00000000 00000000
+000410 MCH_FLT_4:45800000 00000000 3FF00000 00000000
+000420 MCH_FLT_6:00000000 00000000 00000000 00000000
+0004B8 MCH_EXT:00000000 00000000

Chapter 17. z/OS problem determination 159

z/OS - diagnosing crashes

blscddir dsname(’DUMPHLQ.ddir’)
Creates an IPCS DDIR.
runc addr(2657c9b8) link(20:23) chain(9999) le(x’1c’) or runc addr(25429108)
structure(tcb)
Runs a chain of control blocks using the RUNCHAIN command.
addr: the start address of the first block
link: the link pointer start and end bytes within the block (decimal)
chain: the maximum number of blocks to be searched (default=999)
le: the length of data from the start of each block to be displayed (hex)
structure: control block type

Debugging hangs
A hang refers to a process that is still present, but has become unresponsive. This
lack of response can be caused by any one of these reasons:
v The process has become deadlocked, so no work is being done. Usually, the
process is taking up no CPU time.
v The process has become caught in an infinite loop. Usually, the process is taking
up high CPU time.
v The process is running, but is suffering from very bad performance. This is not
an actual hang, but is normally initially thought to be one.

The process is deadlocked

A deadlocked process does not use any CPU time. You can monitor this condition
by using the USS ps command against the Java process:
UID PID PPID C STIME TTY TIME CMD
CBAILEY 253 743 - 10:24:19 ttyp0003 2:34 java -classpath .Test2Frame

If the value of TIME increases in a few minutes, the process is still using CPU, and
is not deadlocked.

For an explanation of deadlocks and how the Javadump tool is used to diagnose
them, see “Locks, monitors, and deadlocks (LOCKS)” on page 193.

The process is looping

If no deadlock exists between threads and the process appears to be hanging but is
consuming CPU time, look at what work the threads are doing. To do this, take a
console- initiated dump as follows:
1. Use the operating system commands (D OMVS,A=ALL) or SDSF (DA =
Display Active) to locate the ASID of interest.
2. Use the DUMP command to take a console-initiated dump both for hangs and
for loops:
DUMP COMM=(Dump for problem 12345)
r xx,asid=(53,d),DSPNAME=(’OMVS ’.*),CONT
R yy,SDATA=(GRSQ,LSQA,RGN,SUM,SWA,TRT,LPA,NUC,SQA)

When the console dump has been generated, you can view the Systrace in IPCS to
identify threads that are looping. You can do this in IPCS as follows:
ip systrace asid(x’007d’) time(gmt)

160 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 z/OS - debugging hangs

This command formats out the system trace entries for all threads that are in
address space 0x7d. The time(gmt) option converts the TOD clock entries, which
are in the system trace, to a human readable form.

From the output produced, you can determine which are the looping threads by
identifying patterns of repeated CLCK and EXT1005 interrupt trace entries, and
subsequent redispatch DSP entries. You can identify the instruction address range
of the loop from the PSWs (Program Status Words) that are traced in these entries.

The process is performing badly

If you have no evidence of a deadlock or an infinite loop, it is likely that the
process is suffering from very bad performance. This can be caused because
threads have been placed into explicit sleep calls, or by excessive lock contention,
long garbage collection cycles, or for several other reasons. This condition is not
actually a hang and should be handled as a performance problem. See “Debugging
performance problems” on page 163 for more information.

Debugging memory leaks

Memory problems can occur in the Java process through two mechanisms:
v A native (C/C++) memory leak that causes increased usage of the LE HEAP,
which can be seen as excessive usage of Subpool2, Key 8, or storage, and an
excessive Working Set Size of the process address space
v A Java object leak in the Java-managed heap. The leak is caused by
programming errors in the application or the middleware. These object leaks
cause an increase in the amount of live data that remains after a garbage
collection cycle has been completed.

Allocations to LE HEAP
The Java process makes two distinct allocation types to the LE HEAP.

The first type is the allocation of the Java heap that garbage collection manages.
The Java heap is allocated during JVM startup as a contiguous area of memory. Its
size is that of the maximum Java heap size parameter. Even if the minimum,
initial, heap size is much smaller, you must allocate for the maximum heap size to
ensure that one contiguous area will be available should heap expansion occur.

The second type of allocation to the LE HEAP is that of calls to malloc() by the
JVM, or by any native JNI code that is running under that Java process. This
includes application JNI code, and third party native libraries; for example, JDBC
drivers.

z/OS virtual storage

To debug these problems, you must understand how C/C++ programs, such as the
JVM, use virtual storage on z/OS. To do this, you need some background
understanding of the z/OS Virtual Storage Management component and LE.

The process address space on 31-bit z/OS has 31-bit addressing that allows the
addressing of 2 GB of virtual storage. The process address space on 64-bit z/OS
has 64-bit addressing that allows the addressing of over 2 GB of virtual storage.
This storage includes areas that are defined as common (addressable by code
running in all address spaces) and other areas that are private (addressable by
code running in that address space only).

Chapter 17. z/OS problem determination 161

z/OS - debugging memory leaks

The size of common areas is defined by several system parameters and the number
of load modules that are loaded into these common areas. On many typical
systems, the total private area available is about 1.4 GB. From this area, the Java
heap is allocated at startup, along with any subsequent calls to malloc(). A leak of
Java objects, therefore, does not cause VSM to issue an abend878 rc10 because of
lack of private storage. This abend can be caused only by unbounded growth of
storage that is allocated through malloc() for underlying JVM resources requested
by JVM components such as AWT or the JIT, or by calls to malloc() from
application JNI code and third party native libraries.

If you change the LE HEAP setting, you are asking LE to GETMAIN different
amounts of initial or incremental storage for use by all C applications. This has no
effect on a Java application throwing an OutOfMemoryError. If errors are received
because of lack of private storage, you must ensure that the region size is big
enough to allocate for the Java heap and for the underlying JVM resources. Note
that for TSO/E address spaces, the REGION size for USS processes that are like
the JVM inherit from the TSO/E address space, whereas in the case of rlogin or
telnet sessions, the region size is determined by the BPXPRMxx parameter
MAXASSIZE.

OutOfMemoryErrors
The JVM throws a java.lang.OutOfMemoryError (OOM) when the heap is full, and
it cannot find space for object creation. Heap usage is a result of the application
design, its use and creation of object populations, and the interaction between the
heap and the garbage collector.

The operation of the JVM’s Garbage Collector is such that objects are continuously
allocated on the heap by mutator (application) threads until an object allocation
fails. At this point, a garbage collection cycle begins. At the end of the cycle, the
allocation is retried. If successful, the mutator threads resume where they stopped.
If the allocation request cannot be fulfilled, an OutOfMemory exception is thrown.

The Garbage Collector uses a mark and sweep algorithm. That is, the Garbage
Collector marks every object that is referenced from the stack of a thread, and
every object that is referenced from a marked object. Any object on the heap that
remains unmarked is cleared up during the sweep phase because it is no longer
live.

An OutOfMemory exception occurs when the live object population requires more
space than is available in the Java managed heap. It is possible that this can occur
not because of an object leak, but because the Java heap is not large enough for the
application that is being run. In this case, you can use the -Xmx option on the JVM
invocation to increase the heap size and remove the problem, as follows:
java -Xmx320m MyApplication

If the failure is occurring under javac, remember that the compiler is a Java
program itself. To pass parameters to the JVM that is created for the compile, use
the -J option to pass the parameters that you would normally pass directly. For
example, the following passes a 128 MB maximum heap to javac:
javac -J-Xmx128m MyApplication.java

In the case of a genuine object leak, the increased heap size does not solve the
problem, but increases the time for a failure to occur.

162 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 z/OS - debugging memory leaks

OutOfMemory errors are also generated when a JVM call to malloc() fails. This
should normally have an associated error code.

Should an OutOfMemoryError be generated, and no error message is produced, it

is assumed that this is because of Java heap exhaustion. At this point, increase the
maximum Java heap size to allow for the possibility that the heap is not big
enough for the application that is running. Also enable the z/OS heapdump, and
switch on verbosegc output.

The -verbosegc (-verbose:gc) switch causes the JVM to print out messages when a
garbage collection cycle begins and ends. These messages indicate how much live
data remains on the heap at the end of a collection cycle. In the case of a Java
object leak, the amount of free space on the heap after a garbage collection cycle
will be seen to decrease over time. See “Basic diagnostics (-verbosegc)” on page
242.

These actions are listed in order of severity. As the number increases, the Garbage
Collector is becoming more desperate for memory. A high action number is a good
indication of a significant shortage of Java heap space.

A Java object leak is caused when an application retains references to objects that
are no longer in use. In a C application, a developer in required to free memory
when it is no longer required. A Java developer is required to removed references
to objects that are no longer required. The developer normally does this by setting
references to null. When this does not happen, the object, and anything that that
object references in turn, continues to reside on the Java heap and cannot be
removed. This typically occurs when data collections are not managed correctly;
that is, the mechanism to remove objects from the collection is either not used, or
used incorrectly.

Debugging performance problems

Check whether the JIT compiler is activated. To do this, ensure that:
v You have not unset the environment variable JAVA_COMPILER
v You have set the environment variable JAVA_COMPILER to something other
than j9jit22 or jitc.
v You have set the system property -Djava.compiler to null.
The JIT compiler makes a significant difference to performance. Do not disable it
unless under the direction of IBM Service. All areas of JIT optimization are
individually switchable, and the JIT allows for selective disablement of compilation
for identified methods, so you should always be able to bypass a problem without
disabling the JIT compiler completely.

Check whether the system is tuned to cope with the Java managed heap size that
you have specified. If the Java managed heap size is large, on a system without
large amounts of real storage you might see a performance degradation caused by
excessive paging.

If the system intermittently sees high CPU usage for the process in which Java is
running, this might be a symptom of excessive garbage collection pauses. The
garbage collector is a ″Stop The World″ type, and collection cycles are normally so
short (from 5-500 milliseconds, for example) that they are not observed externally.
If the collection cycle takes longer for some reason, or occurs more frequently than
expected, this will be observed as high CPU. This is because the garbage collection
code is CPU-intensive, and the collector uses helper threads for marking objects.

Chapter 17. z/OS problem determination 163

z/OS - debugging performance problems

These helper threads could possibly be running on all available CPUs. In addition,
some mutator threads might be in short ″busy waits″ for the cycle to end. In this
case, turn on switch -verbose:gc to see how often the cycles are occurring and
what their duration is. Pause times over several seconds are worth further
investigation. You should also use switch -Xgcpolicy:optavgpause to activate JVM
use of concurrent marking, to reduce and smooth out pause times, at some small
reduction in overall throughput. If this does not help resolve the problem, contact
IBM Service before gathering more information.

Collecting data from a fault condition in z/OS

The data collected from a fault situation in z/OS depends on the problem
symptoms, but could include some or all of the following:
v Transaction dump - an unformatted dump requested by the MVS BCP
IEATDUMP service. This dump can be post-processed with IPCS (Interactive
Problem Control System).
v CEEDUMP - formatted application level dump, requested by the cdump system
call.
v JAVADUMP - formatted internal state data produced by the IBM Virtual
Machine for Java.
v Binary or formatted trace data from the JVM internal high performance trace.
v Debugging messages written to stderr (for example, the output from the JVM
when switches like -verbose:gc, -verbose, or -Xtgc are used).
v Java stack traces when exceptions are thrown.
v Other unformatted system dumps obtained from middleware products or
components (for example, SVC dumps requested by WebSphere for z/OS).
v SVC dumps obtained by the MVS Console DUMP command (typically for loops
or hangs).
v Trace data from other products or components (for example LE traces or the
Component trace for z/OS UNIX).
v Heapdump - this is the same as a Transaction dump and it is also taken by the
MVS BCP IEATDUMP service. The difference between a Heapdump and a
Transaction dump is that the Heapdump is taken when the Java heap is
guaranteed to be stable and can be post-processed using either jextract or
jdmpview tools.

The JVM on z/OS makes use of the IEATDUMP service to capture unformatted
dumps. These dumps can then be processed with IPCS on z/OS. The internal high
performance trace allows for the creation of binary trace files, which can be
post-processed on any platform that supports Java.

164 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 18. Debugging the ORB
One of the first tasks that you must do when debugging an ORB problem is to
determine whether the problem is in the client-side or in the server-side of the
distributed application. Think of a typical RMI-IIOP session as a simple,
synchronous communication between a client that is requesting access to an object,
and a server that is providing it. During this communication, a problem might
occur in the execution of one of the following steps:
1. The client writes and sends a request to the server.
2. The server receives and reads the request.
3. The server execute the task in the request.
4. The server writes and sends a reply back.
5. The client receives and reads the reply.

It is not always easy to identify where the problem occurred. Often, the
information that the application returns, in the form of stack traces or error
messages, is not enough for you to make a decision. Also, because the client and
server communicate through their ORBs, it is likely that if a problem occurs, both
sides will record an exception or unusual behavior.

This chapter describes all the clues that you can use to find the source of the ORB
problem. It also describes a few common problems that occur more frequently. The
topics are:
v “Identifying an ORB problem”
v “Debug properties” on page 167
v “ORB exceptions” on page 168
v “Interpreting the stack trace” on page 170
v “Interpreting ORB traces” on page 171
v “Common problems” on page 174
v “IBM ORB service: collecting data” on page 176

Identifying an ORB problem

When you find a problem that you think is related to CORBA or RMI, a
knowledge of the constituents of the IBM ORB component can be very helpful.

What the ORB component contains

The ORB component contains the following:
v IBM Java ORB and rmi-iiop runtime (com.ibm.rmi.*, com.ibm.CORBA.*)
v rmi-iiop API (javax.rmi.CORBA.*,org.omg.CORBA.*)
v IDL to Java implementation (org.omg.* and IBM versions com.ibm.org.omg.*)
v Transient name server (com.ibm.CosNaming.*, org.omg.CosNaming.*) -
tnameserv
v -iiop and -idl generators (com.ibm.tools.rmi.rmic.*) for the rmic compiler - rmic
v idlj compiler (com.ibm.idl.*)

© Copyright IBM Corp. 2003, 2006 165

identifying an ORB problem

What the ORB component does not contain

The ORB component does not contain:
v RMI-JRMP (also known as Standard RMI)
v JNDI and its plug-ins

Therefore, if the problem is in java.rmi.* or sun.rmi.* , it is not an ORB problem.

Similarly, if the problem is in com.sun.jndi.*, it is not an ORB problem.

Platform-dependent problem
If possible, run the test case on more than one platform. All the ORB code is
shared. You can nearly always reproduce genuine ORB problems on any platform.
If you have a platform-specific problem, it is likely to be in some other component.

JIT problem
JIT bugs are very difficult to find. They might show themselves as ORB problems.
When you are debugging or testing an ORB application, it is always safer to switch
off the JIT by setting the option -Xint.

Fragmentation
Disable fragmentation when you are debugging the ORB. Although fragmentation
does not add complications to the ORB’s functioning, a fragmentation bug can be
difficult to detect because it will most likely show as a general marshalling
problem. The way to disable fragmentation is to set the ORB property
com.ibm.CORBA.FragmentSize=0. You must do this on the client side and on the
server side.

Packaging
Table 7. Packaging
IBM Platforms Non-IBM Platform
Runtime classes jre/lib/ibmorb.jar jre/lib/endorsed/ibmorb.jar
Tools classes lib/tools.jar lib/ibmtools.jar
CORBA API classes jre/lib/ibmorbapijar jre/lib/endorsed/ibmorbapijar
Runtime support None jre/lib/endorsed/ibmext.jar
rmic wrapper None ibm_bin/rmic
ibm_bin/rmic.bat
idlj wrapper None ibm_bin/idlj
ibm_bin/idlj.bat

ORB versions
The ORB component carries a few version properties that you can display by
invoking the main method of the following classes:
1. com.ibm.CORBA.iiop.Version (ORB runtime version)
2. com.ibm.tools.rmic.iiop.Version (for tools; for example, idlj and rmic)
3. rmic -iiop -version (run the command-line for rmic)

Note: Items 2 and 3 are alternative methods for reaching the same class.

166 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 ORB - debug properties

Debug properties
Attention: Do not turn on tracing for normal operation, because it might cause
performance degradation. Even if you have switched off tracing, FFDC (First
Failure Data Capture) is still working, so that only serious errors are reported. If a
debug output file is generated, examine it to check on the problem. For example,
the server might have stopped without performing an ORB.shutdown().

You can use the following properties to enable the ORB traces:
v com.ibm.CORBA.Debug: This property turns on trace, message, or both. If you
set this property to trace only traces are turned on; if you set it to message, only
messages are turned on. Any other value, or no value, turns on traces and
messages. The only way not to set this property is not to specify it. A value of
false enables it anyway. When enabling any kind of tracing, it is safe to turn this
property on.
v com.ibm.CORBA.Debug.Output: This property redirects traces to a file, which
is known as a trace log. When this property is not specified, or it is set to an
empty field, the file name defaults to the format
orbtrc.DDMMYYYY.HHmm.SS.txt, where D=Day; M=Month; Y=Year; H=Hour
(24 hour format); m=Minutes; S=Seconds. Note that if the application (or Applet)
does not have the privilege that it requires to write to a file, the trace entries go
to stderr.
v com.ibm.CORBA.CommTrace: This property turns on wire tracing. Every
incoming and outgoing GIOP message will be output to the trace log. You can
set this property independently from Debug; this is useful if you want to look
only at the flow of information, and you are not too worried about debugging
the internals. The only two values that this property can have are true and false.
The default is false.

Here is an example of common usage example:

java -Dcom.ibm.CORBA.Debug=true -Dcom.ibm.CORBA.Debug.Output=trace.log -Dcom.ibm.CORBA.CommTrace=true <classname>

For rmic -iiop or rmic -idl, the following diagnostic tools are available:
v -J-Djavac.dump.stack=1: This tool ensures that all exceptions are caught.
v -Xtrace: This tool traces the progress of the parse step.

If you are working with an IBM SDK, you can obtain CommTrace for the transient
name server (tnameserv) by using the standard environment variable
IBM_JAVA_OPTIONS. In a separate command session to the server or client
SDKs, you can use:
set IBM_JAVA_OPTIONS=-Dcom.ibm.CORBA.CommTrace=true -Dcom.ibm.CORBA.Debug=true

or the equivalent platform-specific command.

The setting of this environment variable affects each Java process that is started, so
use this variable carefully. Alternatively, you can use the -J option to pass the
properties through the tnameserv wrapper, as follows:
tnameserv -J-Dcom.ibm.CORBA.Debug=true

Chapter 18. Debugging the ORB 167

ORB exceptions

ORB exceptions
You are using this chapter because you think that your problem is related to the
ORB. Unless your application is doing nothing or giving you the wrong result, it is
likely that your log file or terminal is full of exceptions that include the words
“CORBA” and “rmi” many times. All unusual behavior that occurs in a good
application is highlighted by an exception. This principle is also true for the ORB
with its CORBA exceptions. Similarly to Java, CORBA divides its exceptions into
user exceptions and system exceptions.

User exceptions
User exceptions are IDL defined and inherit from org.omg.CORBA.UserException.
These exceptions are mapped to checked exceptions in Java; that is, if a remote
method raises one of them, the application that invoked that method must catch
the exception. User exceptions are usually not fatal exceptions and should always
be handled by the application. Therefore, if you get one of these user exceptions,
you know where the problem is, because the application developer had to make
allowance for such an exception to occur. In most of these cases, the ORB is not the
source of the problem.

System exceptions
System exceptions are thrown transparently to the application and represent an
unusual condition in which the ORB cannot recover gracefully, such as when a
connection is dropped. The CORBA 2.6 specification defines 31 system exceptions
and their mapping to Java. They all belong to the org.omg.CORBA package. The
CORBA specification defines the meaning of these exceptions and describes the
conditions in which they are thrown.

The most common system exceptions are:

v BAD_OPERATION: This exception is thrown when an object reference denotes
an existing object, but the object does not support the operation that was
invoked.
v BAD_PARAM: This exception is thrown when a parameter that is passed to a
call is out of range or otherwise considered illegal. An ORB might raise this
exception if null values or null pointers are passed to an operation.
v COMM_FAILURE: This exception is raised if communication is lost while an
operation is in progress, after the request was sent by the client, but before the
reply from the server has been returned to the client.
v DATA_CONVERSION: This exception is raised if an ORB cannot convert the
marshaled representation of data into its native representation, or cannot convert
the native representation of data into its marshaled representation. For example,
this exception can be raised if wide character codeset conversion fails, or if an
ORB cannot convert floating point values between different representations.
v MARSHAL: This exception indicates that the request or reply from the network
is structurally not valid. This error typically indicates a bug in either the
client-side or server-side runtime. For example, if a reply from the server
indicates that the message contains 1000 bytes, but the actual message is shorter
or longer than 1000 bytes, the ORB raises this exception.
v NO_IMPLEMENT: This exception indicates that although the operation that
was invoked exists (it has an IDL definition), no implementation exists for that
operation.
v UNKNOWN: This exception is raised if an implementation throws a
non-CORBA exception, such as an exception that is specific to the

168 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 ORB exceptions

implementation’s programming language. It is also raised if the server returns a

system exception that is unknown to the client. (This can happen if the server
uses a later version of CORBA than the version that the client is using, and new
system exceptions have been added to the later version.)

Completion status and minor codes

Each system exception has two pieces of data that are associated with it:
v A completion status, which is an enumerated type that has three values:
COMPLETED_YES, COMPLETED_NO and COMPLETED_MAYBE. These values
indicate either that the operation was executed in full, that the operation was
not executed, or that this cannot be determined.
v A long integer, called minor code, that can be set to some ORB vendor specific
value. CORBA also specifies the value of many minor codes.

Usually the completion status is not very useful. However, the minor code can be
essential when the stack trace is missing. In many cases, the minor code identifies
the exact location of the ORB code where the exception is thrown (see the section
below) and can be used by the vendor’s service team to localize the problem
quickly. However, for standard CORBA minor codes, this is not always possible.
For example:
org.omg.CORBA.OBJECT_NOT_EXIST: SERVANT_NOT_FOUND minor code: 4942FC11 completed: No

Minor codes are usually expressed in hexadecimal notation (except for SUN’s
minor codes, which are in decimal notation) that represents four bytes. The OMG
organization has assigned to each vendor a range of 4096 minor codes. The IBM
vendor-specific minor code range is 0x4942F000 through 0x4942FFFF. Appendix D,
“CORBA minor codes,” on page 321 gives diagnostic information for the
most-common minor codes.

System exceptions might also contain a string that describes the exception and
other useful information. You will see this string when you interpret the stack
trace.

The ORB tends to map all Java exceptions to CORBA exceptions. A runtime
exception is mapped to a CORBA system exception, while a checked exception is
mapped to a CORBA user exception.

More exceptions other than the CORBA exceptions could be generated by the ORB
component in a code bug. All the Java unchecked exceptions and errors and others
that are related to the ORB tools rmic and idlj must be considered. In this case, the
only way to determine whether the problem is in the ORB, is to look at the
generated stack trace and see whether the objects involved belong to ORB
packages.

Java2 security permissions for the ORB

When running with a Java 2 SecurityManager, invocation of some methods in the
CORBA API classes might cause permission checks to be made that could result in
a SecurityException. Here is a selection of affected methods:
Table 8. Methods affected when running with Java 2 SecurityManager
Class/Interface Method Required permission
org.omg.CORBA.ORB init java.net.SocketPermission resolve
org.omg.CORBA.ORB connect java.net.SocketPermission listen
org.omg.CORBA.ORB resolve_initial_references java.net.SocketPermission connect

Chapter 18. Debugging the ORB 169

ORB exceptions

Table 8. Methods affected when running with Java 2 SecurityManager (continued)

Class/Interface Method Required permission
org.omg.CORBA. _is_a java.net.SocketPermission connect
portable.ObjectImpl
org.omg.CORBA. _non_existent java.net.SocketPermission connect
portable.ObjectImpl
org.omg.CORBA. OutputStream _request (String, java.net.SocketPermission connect
portable.ObjectImpl boolean)
org.omg.CORBA. _get_interface_def java.net.SocketPermission connect
portable.ObjectImpl
org.omg.CORBA. invoke java.net.SocketPermission connect
Request
org.omg.CORBA. send_deferred java.net.SocketPermission connect
Request
org.omg.CORBA. send_oneway java.net.SocketPermission connect
Request
javax.rmi. narrow java.net.SocketPermission connect
PortableRemoteObject

If your program uses any of these methods, ensure that it is granted the necessary
permissions.

Interpreting the stack trace

Whether the ORB is part of a middleware application or you are using a Java
standalone application (or even an applet), you must retrieve the stack trace that is
generated at the moment of failure. It could be in a log file, or in your terminal or
browser window, and it could consist of several chunks of stack traces.

The following example describes a stack trace that was generated by a server ORB
running in the WebSphere Application Server:

org.omg.CORBA.MARSHAL: com.ibm.ws.pmi.server.DataDescriptor; IllegalAccessException minor code: 4942F23E completed: No

at com.ibm.rmi.io.ValueHandlerImpl.readValue(ValueHandlerImpl.java:199)
at com.ibm.rmi.iiop.CDRInputStream.read_value(CDRInputStream.java:1429)
at com.ibm.rmi.io.ValueHandlerImpl.read_Array(ValueHandlerImpl.java:625)
at com.ibm.rmi.io.ValueHandlerImpl.readValueInternal(ValueHandlerImpl.java:273)
at com.ibm.rmi.io.ValueHandlerImpl.readValue(ValueHandlerImpl.java:189)
at com.ibm.rmi.iiop.CDRInputStream.read_value(CDRInputStream.java:1429)
at com.ibm.ejs.sm.beans._EJSRemoteStatelessPmiService_Tie._invoke(_EJSRemoteStatelessPmiService_Tie.java:613)
at com.ibm.CORBA.iiop.ExtendedServerDelegate.dispatch(ExtendedServerDelegate.java:515)
at com.ibm.CORBA.iiop.ORB.process(ORB.java:2377)
at com.ibm.CORBA.iiop.OrbWorker.run(OrbWorker.java:186)
at com.ibm.ejs.oa.pool.ThreadPool$PooledWorker.run(ThreadPool.java:104)
at com.ibm.ws.util.CachedThread.run(ThreadPool.java:137)

Description string
The example stack trace shows that the application has caught a CORBA
org.omg.CORBA.MARSHAL system exception. After the MARSHAL exception,
some extra information is provided in the form of a string. This string should
specify minor code, completion status, and other information that is related to the
problem. Because CORBA system exceptions are alarm bells for an unusual
condition, they also hide inside what the real exception was.

Usually, the type of the exception is written in the message string of the CORBA
exception. The trace shows that the application was reading a value (read_value())
when an IllegalAccessException occurred that was associated to class
com.ibm.ws.pmi.server.DataDescriptor. This is a hint of the real problem and
should be investigated first.

170 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 ORB - interpreting the stack trace

Nested exceptions
In the example, the ORB mapped a Java exception to a CORBA exception. This
exception is sent back to the client later as part of a reply message. The client ORB
reads this exception from the reply. It maps it to a Java exception
(java.rmi.RemoteException according to the CORBA specification) and throws this
new exception back to the client application.

Along this chain of events, often the original exception becomes hidden or lost, as
does its stack trace. On early versions of the ORB (for example, 1.2.x, 1.3.0) the
only way to get the original exception stack trace was to set some ORB debugging
properties. Newer versions have built-in mechanisms by which all the nested stack
traces are either recorded or copied around in a message string. When dealing with
an old ORB release (1.3.0 and earlier), it is a good idea to test the problem on
newer versions. Either the problem is not reproducible (known bug already solved)
or the debugging information that you obtain is much more useful.

Interpreting ORB traces

The ORB trace file contains messages, trace points, and wire tracing. This section
describes the various types of trace.

Message trace
Here is a simple example of a message:

16:02:33.978 com.ibm.rmi.util.Version logVersions:88 P=953197:O=0:CT ORBRas[default] IBM Java ORB build cndev-20030114

This message records the time, the package, and the method name that was
invoked. In this case, logVersions() prints out to the log file, the version of the
running ORB.

After the first colon in the example message, the line number in the source code
where that method invocation is done is written (88 in this case). Next follows the
letter P that is associated with the process number that was running at that
moment. This number is related (by a hash) to the time at which the ORB class
was loaded in that process. It unlikely that two different processes load their ORBs
at the same time.

The following O=0 (alphabetic O = numeric 0) indicates that the current instance of
the ORB is the first one (number 0). CT specifies that this is the main (control)
thread. Other values are: LT for listener thread, RT for reader thread, and WT for
worker thread.

The ORBRas field shows which RAS implementation the ORB is running. It is
possible that when the ORB runs inside another application (such as a WebSphere
application), the ORB RAS default code is replaced by an external implementation.

The remaining information is specific to the method that has been logged while
executing. In this case, the method is a utility method that logs the version of the
ORB.

This example of a possible message shows the logging of entry or exit point of
methods, such as:

Chapter 18. Debugging the ORB 171

interpreting ORB traces

14:54:14.848 com.ibm.rmi.iiop.Connection <init>:504 LT=0:P=650241:O=0:port=1360 ORBRas[default] Entry

.....
14:54:14.857 com.ibm.rmi.iiop.Connection <init>:539 LT=0:P=650241:O=0:port=1360 ORBRas[default] Exit

In this case, the constructor (that is, <init>) of the class Connection is invoked. The
tracing records when it started and when it finished. For operations that include
the java.net package, the ORBRas logger prints also the number of the local port
that was involved.

Comm traces
Here is an example of comm (wire) tracing:
// Summary of the message containing name-value pairs for the principal fields
OUT GOING:
Request Message // It is an out going request, therefore we are dealing with a client
Date: 31 January 2003 16:17:34 GMT
Thread Info: P=852270:O=0:CT
Local Port: 4899 (0x1323)
Local IP: 9.20.178.136
Remote Port: 4893 (0x131D)
Remote IP: 9.20.178.136
GIOP Version: 1.2
Byte order: big endian

Fragment to follow: No // This is the last fragment of the request

Message size: 276 (0x114)
--

Request ID: 5 // Request Ids are in ascending sequence

Response Flag: WITH_TARGET // it means we are expecting a reply to this request
Target Address: 0
Object Key: length = 26 (0x1A) // the object key is created by the server when exporting
// the servant and retrieved in the IOR using a naming service
4C4D4249 00000010 14F94CA4 00100000
00080000 00000000 0000
Operation: message // That is the name of the method that the client invokes on the servant
Service Context: length = 3 (0x3) // There are three service contexts

Context ID: 1229081874 (0x49424D12) // Partner version service context. IBM only
Context data: length = 8 (0x8)
00000000 14000005

Context ID: 1 (0x1) // Codeset CORBA service context

Context data: length = 12 (0xC)
00000000 00010001 00010100

Context ID: 6 (0x6) // Codebase CORBA service context

Context data: length = 168 (0xA8)
00000000 00000028 49444C3A 6F6D672E
6F72672F 53656E64 696E6743 6F6E7465
78742F43 6F646542 6173653A 312E3000
00000001 00000000 0000006C 00010200
0000000D 392E3230 2E313738 2E313336
00001324 0000001A 4C4D4249 00000010
15074A96 00100000 00080000 00000000
00000000 00000002 00000001 00000018
00000000 00010001 00000001 00010020
00010100 00000000 49424D0A 00000008
00000000 14000005
Data Offset: 11c
// raw data that goes in the wire in numbered rows of 16 bytes and the corresponding ASCII
decoding
0000: 47494F50 01020000 00000114 00000005 GIOP............
0010: 03000000 00000000 0000001A 4C4D4249 ............LMBI
0020: 00000010 14F94CA4 00100000 00080000 ......L.........

172 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 interpreting ORB traces

0030: 00000000 00000000 00000008 6D657373 ............mess

0040: 61676500 00000003 49424D12 00000008 age.....IBM.....
0050: 00000000 14000005 00000001 0000000C ................
0060: 00000000 00010001 00010100 00000006 ................
0070: 000000A8 00000000 00000028 49444C3A ...........(IDL:
0080: 6F6D672E 6F72672F 53656E64 696E6743 omg.org/SendingC
0090: 6F6E7465 78742F43 6F646542 6173653A ontext/CodeBase:
00A0: 312E3000 00000001 00000000 0000006C 1.0............l
00B0: 00010200 0000000D 392E3230 2E313738 ........9.20.178
00C0: 2E313336 00001324 0000001A 4C4D4249 .136...$....LMBI
00D0: 00000010 15074A96 00100000 00080000 ......J.........
00E0: 00000000 00000000 00000002 00000001 ................
00F0: 00000018 00000000 00010001 00000001 ................
0100: 00010020 00010100 00000000 49424D0A ... ........IBM.
0110: 00000008 00000000 14000005 00000000 ................

Note: The italic comments that start with a double slash have been added for
clarity; they are not part of the traces.

In this example trace, you can see a summary of the principal fields that are
contained in the message, followed by the message itself as it goes in the wire. In
the summary are several field name-value pairs. Each number is in hexadecimal
notation.

Appendix C, “CORBA GIOP message format,” on page 317 gives details of the
structure of a GIOP message. See also CORBA specification chapters 13 and 15.)

Client or server
From the first line of the summary of the message, you can identify whether the
host to which this trace belongs is acting as a server or as a client. OUT GOING
means that the message has been generated in the machine where the trace was
taken and is sent to the wire.

In a distributed-object application, a server is defined as the provider of the

implementation of the remote object to which the client connects. In this work,
however, the convention is that a client sends a request while the server sends
back a reply. In this way, the same ORB can be client and server in different
moments of the rmi-iiop session.

The trace shows that the message is an outgoing request. Therefore, this trace is a
client trace, or at least part of the trace where the application acts as a client.

Time information and host names are reported in the header of the message.

The Request ID and the Operation (“message” in this case) fields can be very
helpful when multiple threads and clients destroy the logical sequence of the
traces.

The GIOP version field can be checked if different ORBs are deployed. If two
different ORBs support different versions of GIOP, the ORB that is using the more
recent version of GIOP should fall back to a common level. By checking that field,
however, you can easily check whether the two ORBs speak the same language.

Service contexts
The header also records three service contexts, each consisting of a context ID and
context data. A service context is extra information that is attached to the message

Chapter 18. Debugging the ORB 173

interpreting ORB traces

for purposes that can be vendor-specific (such as the IBM Partner version that is
described in the IOR in Chapter 5, “Understanding the ORB,” on page 33).

Usually, a security implementation makes extensive use of these service contexts.

Information about an access list, an authorization, encrypted IDs, and passwords
could travel with the request inside a service context.

Some CORBA-defined service contexts are available. One of these is the Codeset.

In the example, the codeset context has ID 1 and data 00000000 00010001
00010100. Bytes 5 through 8 specify that characters that are used in the message are
encoded in ASCII (00010001 is the code for ASCII). Bytes 9 through 12 instead are
related to wide characters.

The default codeset is UTF8 as defined in the CORBA specification, although

almost all Windows and UNIX platforms communicate normally through ASCII.
Mainframes such as zSeries systems are based on the IBM EBCDIC encoding.

The other CORBA service context, which is present in the example, is the Codebase
service context. It stores information about how to call back to the client to access
resources in the client such as stubs, and class implementations of parameter
objects that are serialized with the request.

Common problems
This section describes some of the problems that you might find.

ORB application hangs

One of the worst conditions is when the client, or server, or both, hang. If this
happens, the most likely condition (and most difficult to solve) is a deadlock of
threads. In this condition, it is important to know whether the machine that on
which you are running has more than one CPU.

A simple test that you can do is to keep only one CPU running and see whether
the problem disappears. If it does, you know that you must have a synchronization
problem in the application.

Also, you must understand what the application is doing while it hangs. Is it
waiting (low CPU usage), or it is looping forever (almost 100% CPU usage)? Most
of the cases are a waiting problem.

You can, however, still identify two cases:

v Typical deadlock
v Standby condition while the application waits for a resource to arrive

An example of a standby condition is where the client sends a request to the server
and stops while waiting for the reply. The default behavior of the ORB is to wait
indefinitely.

You can set a couple of properties to avoid this condition:

v com.ibm.CORBA.LocateRequestTimeout
v com.ibm.CORBA.RequestTimeout

174 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 ORB application hangs

When the property com.ibm.CORBA.enableLocateRequest is set to true (the default

is false), the ORB first sends a short message to the server to find the object that it
needs to access. This first contact is the Locate Request. You must now set the
LocateRequestTimeout to a value other than 0 (which is equivalent to infinity). A
good value could be something around 5000 milliseconds.

Also, set the RequestTimeout to a value other than 0. Because a reply to a request
is often large, allow more time; for example, 10000 milliseconds. These values are
suggestions and might be too low for slow connections. When a request times out,
the client receives an explanatory CORBA exception.

When an application hangs, consider also another property that is called

com.ibm.CORBA.FragmentTimeout. This property was introduced in IBM ORB
1.3.1, when the concept of fragmentation was implemented to increase
performance. You can now split long messages into small chunks or fragments and
send one after the other across the net. The ORB waits for 30 seconds (default
value) for the next fragment before it throws an exception. If you set this property,
you disable this time-out, and problems of waiting threads might occur.

If the problem appears to be a deadlock or hang, capture the Javadump

information. Do this once, then wait for a minute or so, and do it again. A
comparison of the two snapshots shows whether any threads have changed state.
For information about how to do this operation, see “Triggering a Javadump” on
page 192.

In general, stop the application, enable the orb traces (see previous section) and
restart the application. When the hang is reproduced, the partial traces that can be
retrieved can be used by the IBM ORB service team to help understand where the
problem is.

Running the client without the server running before the client
is invoked
This operation outputs:
(org.omg.CORBA.COMM_FAILURE)
Hello Client exception:
org.omg.CORBA.COMM_FAILURE:minor code:1 completed:No
at com.ibm.rmi.iiop.ConnectionTable.get(ConnectionTable.java:145)
at com.ibm.rmi.iiop.ConnectionTable.get(ConnectionTable.java:77)
at com.ibm.rmi.iiop.GIOPImpl.createRequest(GIOPImpl.java:98)
at com.ibm.rmi.iiop.GIOPImpl.createRequest(GIOPImpl.java:75)
at com.ibm.rmi.corba.ClientDelegate.createRequest(ClientDelegate.java:440)
at com.ibm.rmi.corba.ClientDelegate.is_a(ClientDelegate.java:571)
at org.omg.CORBA.portable.ObjectImpl._is_a(ObjectImpl.java:74)
at org.omg.CosNaming.NamingContextHelper.narrow(NamingContextHelper.java:58)
com.sun.jndi.cosnaming.CNCtx.callResolve(CNCtx.java:327)

Client and server are running, but not naming service

The output is:
Hello Client exception:Cannot connect to ORB
Javax.naming.CommunicationException:
Cannot connect to ORB.Root exception is org.omg.CORBA.COMM_FAILURE minor code:1 completed:No
at com.ibm.rmi.iiop.ConnectionTable.get(ConnectionTable.java:145)
at com.ibm.rmi.iiop.ConnectionTable.get(ConnectionTable.java:77)
at com.ibm.rmi.iiop.GIOPImpl.createRequest(GIOPImpl.java:98)
at com.ibm.rmi.iiop.GIOPImpl.createRequest(GIOPImpl.java:75)
at com.ibm.rmi.corba.ClientDelegate.createRequest(ClientDelegate.java:440)
at com.ibm.rmi.corba.InitialNamingClient.resolve(InitialNamingClient.java:197)

Chapter 18. Debugging the ORB 175

ORB application hangs

at com.ibm.rmi.corba.InitialNamingClient.cachedInitialReferences(InitialNamingClient.j
at com.ibm.rmi.corba.InitialNamingClient.resolve_initial_references(InitialNamingClien
at com.ibm.rmi.corba.ORB.resolve_initial_references(ORB.java:1269)
.........

You must start the Java IDL name server before an application or applet starts that
uses its naming service. Installation of the Java IDL product creates a script
(Solaris: tnameserv) or executable file (Windows NT: tnameserv.exe) that starts the
Java IDL name server.

Start the name server so that it runs in the background. If you do not specify
otherwise, the name server listens on port 2809 for the bootstrap protocol that is
used to implement the ORB resolve_initial_references() and list_initial_references()
methods.

Specify a different port, for example, 1050, as follows:

tnameserv -ORBInitialPort 1050

Clients of the name server must be made aware of the new port number. Do this
by setting the org.omg.CORBA.ORBInitialPort property to the new port number
when you create the ORB object.

Running the client with MACHINE2 (client) unplugged from the

network
Your output is:
(org.omg.CORBA.TRANSIENT CONNECT_FAILURE)

Hello Client exception:Problem contacting address:corbaloc:iiop:machine2:2809/NameService

javax.naming.CommunicationException:Problem contacting address:corbaloc:iiop:machine2:2809/N
is org.omg.CORBA.TRANSIENT:CONNECT_FAILURE (1)minor code:4942F301 completed:No
at com.ibm.CORBA.transport.TransportConnectionBase.connect(TransportConnectionBase.jav
at com.ibm.rmi.transport.TCPTransport.getConnection(TCPTransport.java:178)
at com.ibm.rmi.iiop.TransportManager.get(TransportManager.java:79)
at com.ibm.rmi.iiop.GIOPImpl.createRequest(GIOPImpl.java:131)
at com.ibm.rmi.iiop.GIOPImpl.createRequest(GIOPImpl.java:98)
at com.ibm.CORBA.iiop.ClientDelegate._createRequest(ClientDelegate.java:2096)
at com.ibm.CORBA.iiop.ClientDelegate.createRequest(ClientDelegate.java:1264)
at com.ibm.CORBA.iiop.ClientDelegate.createRequest(ClientDelegate.java:1177)
at com.ibm.rmi.corba.InitialNamingClient.resolve(InitialNamingClient.java:252)
at com.ibm.rmi.corba.InitialNamingClient.cachedInitialReferences(InitialNamingClient.j
at com.ibm.rmi.corba.InitialNamingClient.resolve_initial_references(InitialNamingClien
at com.ibm.rmi.corba.InitialReferenceClient.resolve_initial_references(InitialReferenc
at com.ibm.rmi.corba.ORB.resolve_initial_references(ORB.java:3211)
at com.ibm.rmi.iiop.ORB.resolve_initial_references(ORB.java:523)
at com.ibm.CORBA.iiop.ORB.resolve_initial_references(ORB.java:2898)
..........

IBM ORB service: collecting data

This section describes how to collect data about ORB problems.

Preliminary tests
The ORB is affected by problems with the underlying network, hardware, and
JVM. When a problem occurs, the ORB can throw an org.omg.CORBA.* exception,
some text that describes the reason, a minor code, and a completion status. Before
you assume that the ORB is the cause of problem, ensure the following:
v The scenario can be reproduced (not only on customers’ machines, but on a
similar setup configuration).

176 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 IBM ORB service: collecting data

v The JIT is disabled (see Chapter 27, “JIT problem determination,” on page 237).

Also:
1. Disable additional CPUs.
2. Eliminate memory dependencies with the client or server. The lack of physical
memory can be the cause of slow performance, apparent hangs, or crashes. To
remove these problems, ensure that you have a reasonable headroom of
memory. Remember that even with 1 GB of physical RAM, Java can use only
512 MB independently of what -Xmx is set to.
3. Check physical network problems (firewalls, com links, routers, DNS name
servers, and so on). These are the major causes of CORBA COMM_FAILURE
exceptions. As a test, ping your own machine name.
4. If the application is using a database such as DB2, switch to the most reliable
driver. For example, to isolate DB2 AppDriver, switch to Net Driver, which is
slower and uses sockets, but is more reliable.

Data to be collected
If after all these verifications, the problem is still present, collect at all nodes of the
problem the following:
v Operating system name and version.
v Output of java -version.
v Output of java com.ibm.CORBA.iiop.Version.
v Output of rmic -iiop -version, if rmic is involved.
v ASV build number (WebSphere Application Server only).
v If you think that the problem is a regression, include the version information for
the most recent known working build and for the failing build.
v If this is a runtime problem, collect debug and communication traces of the
failure from each node in the system (as explained earlier in this chapter).
v If the problem is in rmic -iiop or rmic -idl, set the options:
-J-Djavac.dump.stack=1 -Xtrace, and capture the output.
v Normally this step is not necessary. If it looks like the problem is in the buffer
fragmentation code, IBM service will return the defect asking for an additional
set of traces, which you can produce by executing with
-Dcom.ibm.CORBA.FragmentSize=0.

A testcase is not essential, initially. However, a working testcase that demonstrates

the problem by using only the Java SDK classes will speed up the resolution time
for the problem.

Chapter 18. Debugging the ORB 177

IBM ORB service: collecting data

178 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 19. NLS problem determination
The JVM contains built-in support for different locales. This chapter provides an
overview of locales, with the main focus on fonts and font management.
v “Overview of fonts”
v “The font.properties file” on page 180
v “Font utilities” on page 181
v “Common problems and possible causes” on page 182

Overview of fonts
When you want to display text, either in SDK components (AWT or Swing), on the
console or in any application, characters have to be mapped to glyphs. A glyph is
an artistic representation of the character, in some typographical style, and is
stored in the form of outlines or bitmaps. Glyphs might not correspond
one-for-one with characters. For instance, an entire character sequence can be
represented as a single glyph. Also a single character may be represented by more
than one glyph (for example, in Indic scripts).

A font is a set of glyphs, where each glyph is encoded in a particular encoding

format, so that the character to glyph mapping can be done using the encoded
value. Almost all of the available Java fonts are encoded in Unicode and provide
universal mappings for all applications.

The most commonly available font types are TrueType and OpenType fonts.

Font specification properties

Specify fonts according to the following characteristics:
Font family
A font family is a group of several individual fonts that are related in
appearance. For example: Times, Arial, and Helvetica.
Font style
Font style specifies that the font be displayed in various faces. For example:
Normal, Italic, and Oblique
Font variant
This property determines whether the font should be displayed in normal caps
or in small caps. A particular font might contain only normal caps, only small
caps, or both types of glyph.
Font weight
This refers to the boldness or the lightness of the glyph to be used.
Font size
This property is used to modify the size of the displayed text.

Fonts installed in the system

On Linux or Unix platforms
To see the fonts that are either installed in the system or available for an

© Copyright IBM Corp. 2003, 2006 179

NLS - overview of fonts

application to use, type the command: xset -q "". If your PATH also points
to the SDK (as it should be), xset -q output also shows the fonts that are
bundled with the Developer Kit.
Use xset +fp and xset -fp to add and remove the font path respectively.
On Windows platforms
Most text processing applications have a drop-down list of the available system
fonts, or you can use the Settings->Control Panel->Fonts application.

The font.properties file

The JVM has a font.properties file that controls how Java adds fonts to its runtime.
This is platform specific.

The Linux font.properties file

The font.properties file consists of several sections. The first section associates java
font names to platform fonts.

On Linux and USS platforms, a typical font.properties entry looks like this:
serif.3=-monotype-timesnewromanwt-medium-r-normal--*-%d-75-75-p-*-ibm-udcjp

This can be interpreted as follows:

<General font name>.[<Style>.]<index>=<Platform font name>

where:
General font name is the font name that Java understands.
Style can be normal, italic, bold, bolditalic, and so on. The default is ″normal″.
Index specifies the sequence of searching for matching font glyphs, with zero
the highest priority.
Platform font name is the name of the font in the system.

Other entries can be:

Font name /font file mapping
Entries in the font.properties help Java to map the font name with the font file
filename.timesnewromanwt_medium_r=tnrwt_j.ttf
This shows that the system font timesnewromanwt is defined in the font file
tnrwt_j.ttf.
Font substitution
When the font is missing, try to map the missing font with another:
substitute.0=-timesnewromanwt=timesnewromanwt30

Here, if timesnewromanwt font is not found in the system, it is substituted

with timesnewromanwt30.

If the JVM cannot load any fonts from the system, the characters are displayed
as small squares.
Font CharSet
These entries control the converter to be used to convert unicode strings.
fontcharset.serif.0=sun.iof.CharToByteISO8859_1
This indicates that to draw the font that is specified by serif.0, the
sun.iof.CharToByteISO8859_1 converter is used.

180 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 NLS - font.properties file

Alias
This is used to map one Java font to another.
alias.timesnewroman=serif
The font definitions for serif are used for the font timesnewroman.
Fontset
The fontset entry is used to match fonts specifically for TextArea and TextField
objects.
fontset.serif.plain=\
-jdk-lucidabright-medium-r-normal--*-%d-75-75-p-*-iso8859-1,\
-monotype-timesnewromanwt-medium-r-normal--*-%d-75-75-*-*-jisx0208.1983-0,\
-monotype-timesnewromanwt-medium-r-normal--*-%d-75-75-*-*-jisx0201.1976-0,\
-monotype-timesnewromanwt-medium-r-normal--*-%d-75-75-p-*-ibm-udcjp

The Windows font.properties file

Modification of this file is risky and is not supported. See http://java.sun.com/
products/jdk/1.2/docs/guide/internat/fontprop.html for more information.

Note: The Windows font.properties file refers to Arial Unicode MS. The Arial
Unicode MS font is part of Office 2000 and above. You can download it from
Microsoft if you have a license for Microsoft Office or related products.

Font utilities

Font utilities on Linux, AIX, and z/OS

xfd
Use the command xfd -fn <physical font name> in AIX to find out about the
glyphs and their rendering capacity. For example: Xfd -fn
monotype-sansmonowt-medium-r-normal--*-%d-75-75-m-*-ibm-udcjp brings up
a window with all the glyphs that are in that font.
xlsfonts
Use xlsfonts to check whether a particular font is installed on the system. For
example: xlsfonts | grep ksc will list all the Korean fonts in the system.
iconv
Use to convert the character encoding from one encoding to other. Converted
text is written to standard output. For example: iconv -f oldset -t newset
[file ...]
Options are:
-f oldset
Specifies the source codeset (encoding).
-t newset
Specifies the destination codeset (encoding).
file
The file that contain the characters to be converted; if no file is specified,
standard input is used.

Font utilities on Windows systems

There are no built-in utilities similar to those offered by Linux and z/OS.

Chapter 19. NLS problem determination 181

NLS - common problems and possible causes

Common problems and possible causes

Why do I see a square box or ??? (question marks) in the SDK components?
This effect is caused mainly because Java is not able to find the correct font file
to display the character. If a Korean character should be displayed, the system
should be using the Korean locale, so that Java can take the correct font file. If
you are seeing boxes or queries, check the following:
For AWT components:
1. Check your locale with locale.
2. To change the locale, export LANG=zh_TW (for example)
3. If this still does not work, try to log in with the required language.

For Swing components:

1. Check your locale with locale
2. To change the locale, export LANG=zh_TW (for example)
3. If you know which font you have used in your application, such as serif,
try to get the corresponding physical font from font.properties; then look
into the fontpath to check for the existence of the font. If the font file is
missing, try adding it there.
Character not displayed in TextArea or TextField
These components are Motif components (Linux and USS). Java gives a set of
fonts to Motif to render the character. If the characters are not displayed
properly, use the following Motif application to check whether the character is
displayable by your Motif.
#include <stdio.h>
#include <locale.h>
#include <Xm/Xm.h>
#include <Xm/PushB.h>
main(int argc, char **argv)
{
XtAppContext context;
Widget toplevel, pushb;
Arg args[8];
Cardinal i, n;
XmString xmstr;
char ptr[9];
/* ptr contains the hex. Equivalent of unicode value */
ptr[0] = 0xc4; /*4E00*/
ptr[1] = 0xa1;
ptr[2] = 0xa4; /*4E59*/
ptr[3] = 0x41;
ptr[4] = 0xa4; /*4EBA*/
ptr[5] = 0x48;
ptr[6] = 0xa4; /* 4E09 */
ptr[7] = 0x54;
ptr[8] = 0x00;

setlocale(LC_ALL, "");
toplevel = XtAppInitialize(&context, "", NULL, 0, &argc, argv,
NULL, NULL, 0);
n=0;
XtSetArg(args[n], XmNgeometry, "=225x225+50+50"); n++;
XtSetArg(args[n], XmNallowShellResize, True); n++;
XtSetValues(toplevel, args, n);
xmstr =XmStringCreateLocalized(ptr);
n=0;
XtSetArg(args[n], XmNlabelString, xmstr); n++;
pushb = XmCreatePushButton(toplevel, "PushB", args, n);
XtManageChild(pushb);

182 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 NLS - common problems and possible causes

XtRealizeWidget(toplevel);
XtAppMainLoop(context);
}
Compilation: cc -lXm -lXt -o motif motif.c

Note that the Motif library is statically linked into the Linux JVMs, so it is not
possible to use this technique there.

Chapter 19. NLS problem determination 183

NLS - common problems and possible causes

184 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Part 4. Using diagnostic tools
This part of the book describes how to use the diagnostic tools that are available.
The chapters are:
v Chapter 20, “Overview of the available diagnostics,” on page 187
v Chapter 21, “Using Javadump,” on page 191
v Chapter 22, “Using Heapdump,” on page 205
v Chapter 23, “JVM dump initiation,” on page 209
v Chapter 24, “Using dump agents,” on page 213
v Chapter 25, “Using method trace,” on page 219
v Chapter 26, “Using the dump formatter,” on page 223
v Chapter 27, “JIT problem determination,” on page 237
v Chapter 28, “Garbage Collector diagnostics,” on page 241
v Chapter 29, “Class-loader diagnostics,” on page 257
v Chapter 30, “Tracing Java applications and the JVM,” on page 259
v Chapter 31, “Using the Reliability, Availability, and Serviceability Interface,” on
page 283
v Chapter 32, “Using the JVMPI,” on page 297

Note: JVMMI is not supported on the v1.4.2 platforms described in this book.

© Copyright IBM Corp. 2003, 2006 185

186 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 20. Overview of the available diagnostics
This chapter describes the diagnostic tools used during problem determination.
The purpose of this chapter is to describe what is available, with a broad look at
how and when you might use a particular tool.

Note that Java on any given platform comprises two parts:

v The Java Virtual Machine (JVM) that interfaces Java to the native operating
system and
v The Java classes that provide the infrastructure.
There are no tools that can ″cross the barrier″ between these two parts. In other
words, if you have a Java problem you need a Java diagnostic tool and if you have
a problem in the JVM you need a JVM diagnostic tool.

This book addresses the IBM Virtual Machine for Java. The diagnostics in this book
are all JVM diagnostics.

Categorizing the problem

Problems are considered to fall into four categories:
1. Crashes
2. Hangs
3. Memory leaks
4. Poor performance

You need different tools to solve problems in each category. Most of the tools
described in this book are from IBM, either built into the JVM or as external
monitoring tools.

Platforms
IBM provides and supports Java on a number of platforms. These platforms can be
divided into the groups:
1. Linux
2. Windows
3. z/OS (previously called S/390)

The platform architectures are very different. You will find that:
v Some tools exist only for a given platform.
v Some tools have different versions for different platforms.
v Some tools are cross-platform.

Summary of cross-platform tools

IBM has several cross-platform diagnostic tools. They apply to the different types
of problem described above. The following sections provide a brief description of
the tools and indicate the sort of problem determination to which they are suited.

© Copyright IBM Corp. 2003, 2006 187

diagnostics - cross-platform tools

Javadump (or Javacore)

Javadump is also known as ″Javacore″.

The code that creates Javadumps is part of the JVM. You can control it by using
environment variables and runtime switches. By default, a Javadump is produced
when the JVM terminates unexpectedly (crashes) because of an operating system
signal or when the user enters a reserved key-combination (for example, Ctrl-Break
on Windows). A Javadump is a text file that summarizes the state of the JVM at
the instant the signal occurred.

Much of the content of the Javadump is IBM-specific; that is, it is present only in
the IBM Virtual Machine for Java. See Chapter 21, “Using Javadump,” on page 191
for details. Javadump is an automatic tool.

Heapdump
The IBM Virtual Machine for Java can generate a Heapdump, which is a record of
all the Java objects in the Java heap. Heapdump can generate Heapdump files at
the request of the user, in an out-of-memory condition, or when the JVM
terminates unexpectedly (a crash). Each Heapdump file contains details of every
object in the heap at the time it was generated. This is useful for diagnosing
several kinds of problems, in particular, memory-related problems.

Heapdump is IBM-specific; that is, it is present only in the IBM Virtual Machine
for Java. See Chapter 22, “Using Heapdump,” on page 205 for details.

Cross-platform dump formatter

The cross-platform dump formatter is a more advanced tool than Javadump. It
uses the dump files that the operating system generates to resolve data relevant to
the JVM. This tool is provided in two parts:
1. Platform code to extract data from the dump generated by the native operating
system (jextract).
2. A Java tool to analyze that data (jdmpview).

The formatter understands the JVM and can be used to analyze its internals. Thus,
it is a useful tool to debug JVM crashes. You must have a basic knowledge of the
JVM internals to use this tool. The formatter is really for use on postmortem
dumps. However, it is also useful for checking if leak problems occur in JVM
resources.

The cross-platform dump formatter is an IBM-specific tool; that is, it is present

only in the IBM Virtual Machine for Java. You need a long time to master the
dump formatter; it is not a simple tool to use. However, it is the deepest and most
complete post-mortem analysis tool that is available.

For more information, see Chapter 26, “Using the dump formatter,” on page 223.

JVMPI tools
JVMPI is officially described by Sun as “an experimental interface for profiling”. It
is not yet a standard profiling interface. It is provided for the benefit of tools
vendors who have an immediate need for profiling hooks in the Java virtual
machine. Sun states that the JVMPI will continue to evolve, based on feedback
from customers and tools vendors. IBM fully supports the current JVMPI

188 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 diagnostics - cross-platform tools

specification and is fully compatible with the current Sun release of the technology.
Visit Sun’s website (java.sun.com/j2se/1.3/docs/guide/jvmpi) for more
information.

JVMPI tools conform to the JVM Profiling Interface that is common across all
JVMs. The IBM Virtual Machine for Java is fully JVMPI compatible. Any tool
conforming to JVMPI can be used to profile the IBM Virtual Machine for Java.

JVMPI tools help with problems involving leaks and performance, although profile
logs might give useful hints to the state of the JVM just before a crash or hang
problem.

The JVMPI is intended for interested parties to write profilers, but IBM provides a
useful agent with the IBM SDK.

For more information, see Chapter 32, “Using the JVMPI,” on page 297.

JPDA Tools
Java Platform Debugging Architecture (JPDA) is a common standard for debugging
JVMs. The IBM Virtual Machine for Java is fully JPDA compatible.

Any JPDA debugger can be attached to the IBM Virtual Machine for Java. Being
debuggers, these tools are best suited to tracing leaks or the conditions prior to a
crash or hang, if these are repeatable.

An example of such a tool is the debugger that is bundled with Eclipse for Java.

JVM trace
JVM trace is a key diagnostic tool for the JVM.

The IBM Virtual Machine for Java contains a large amount of embedded trace.
Naturally, this tracing is switched off by default. Command-line options allow you
to turn trace on, set exactly what is to be traced, and specify where the trace
output is to go.

Trace applies to performance and leak problem determination, although the trace
file might provide clues to the state of a JVM before a crash or hang.

Trace is an IBM-specific tool; that is, it is present only in the IBM Virtual Machine
for Java. See Chapter 30, “Tracing Java applications and the JVM,” on page 259 for
details. You need some considerable effort to master trace. However, it is an
extremely effective tool.

JVMRI
The JVMRI (JVM RAS Interface, where RAS stands for Reliability, Availability,
Serviceability). JVMRI allows you to control several JVM operations
programmatically.

For example, the IBM Virtual Machine for Java contains a large amount of
embedded trace. Tracing is switched off by default. A JVMRI agent acts as a
plug-in to allow real-time control of trace information. You use the -Xrun
command-line option so that the agent is loaded by the JVM itself at startup time.
When loaded, a JVMRI agent can dynamically switch JVM trace on and off, control

Chapter 20. Overview of the available diagnostics 189

diagnostics - cross-platform tools

the trace level, and capture the trace output. The JVMRI applies to performance
and leak problem determination, although the trace file might provide clues to the
state of a JVM before a crash or hang.

The RAS plug-in interface is an IBM-specific interface; that is, it is present only in
the IBM Virtual Machine for Java. See Chapter 31, “Using the Reliability,
Availability, and Serviceability Interface,” on page 283 for details. You need some
programming skills and tools to be able to use this interface.

Application trace
Application trace allows you to place tracepoints in Java code to provide trace data
that is combined with other forms of trace. You can control the tracepoints at
start-up or enable them dynamically. For more information, see Chapter 30,
“Tracing Java applications and the JVM,” on page 259.

Application trace is an IBM-specific tool; that is, it is present only in the IBM
Virtual Machine for Java. See Chapter 30, “Tracing Java applications and the JVM,”
on page 259 for details. You need some considerable effort to master trace.
However, it is an extremely effective tool.

Method trace
Method trace permits the tracing of Java methods using the existing JVM trace
facility. The trace has entry, exit, and, optionally, input parameters. You can select
classes and methods for trace using wildcards. You start method trace by
command-line options at JVM startup time, or by using a JVMRI agent.

Method trace is an IBM-specific tool; that is, it is present only in the IBM Virtual
Machine for Java. See Chapter 25, “Using method trace,” on page 219 for details.
Basic method trace is simple to use, and very effective.

JVM command-line parameters

The IBM Virtual Machine for Java has a rich set of command-line parameters that
allow you to control various functions. See Appendix F, “Command-line options,”
on page 329.

JVM environment variables

The IBM Virtual Machine for Java has a rich set of environment variables that you
can use to affect its running; for example, controlling the JIT.

The variables are separately described for the tools and diagnostics to which they
apply, and are also all gathered together for reference in Appendix E,
“Environment variables,” on page 323

Platform tools
Platform-specific tools are documented in the appropriate sections that follow. All
platforms (except z/OS) have a dump extractor tool that feeds the cross-platform
dump formatter. For the other tools, each platform has a different toolset. Some
tools have versions for two or more platforms.

The Java service team has a prototype Java application that displays and analyses
the Java environment variables. If you want more details about this prototype,
send an e-mail to [email protected].

190 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 21. Using Javadump
Javadump produces files that contain diagnostic information related to the JVM
and a Java application captured at a point during execution. For example, the
information can be about the operating system, the application environment,
threads, stacks, locks, and memory. The exact contents depend on the platform on
which you are running. The files produced by Javadump are called ″Javadump
files″. By default, a Javadump occurs when the JVM terminates unexpectedly. A
Javadump can also be triggered by sending specific signals to the JVM.

Note: Javadump is also known as Javacore. This is NOT the same as a core file
(that is an operating system feature that can be produced by any program,
not just the JVM).

This chapter describes:

v “Enabling a Javadump”
v “The location of the generated Javadump”
v “Triggering a Javadump” on page 192
v “Interpreting a Javadump” on page 192

Note: “Interpreting a Javadump” on page 192 is the main part of this chapter.

Enabling a Javadump
Javadumps are enabled by default. To turn them off, set the environment variable
DISABLE_JAVADUMP to TRUE.

You can use the JAVA_DUMP_OPTS environment variable to control exactly when
a Javadump is produced; see Chapter 23, “JVM dump initiation,” on page 209 for
more information. You can also use the -Xdump agent option to get more
fine-grained control over Javadumps; see Chapter 24, “Using dump agents,” on
page 213 for more information.

The location of the generated Javadump

The JVM checks each of the following locations for existence and write-permission,
and stores the Javadump in the first one available. Note that you must have
enough free disk space (possibly up to 2.5 MB) for the Javadump file to be written
correctly.
1. The location specified by the IBM_JAVACOREDIR environment variable if set
(_CEE_DMPTARG on z/OS).
2. The current working directory of the JVM processes.
3. The location specified by the TMPDIR environment variable, if set.
4. The /tmp directory or, on Windows, C:\Temp.
5. If the Javadump cannot be stored in any of the above, it is put to STDERR.

The file name is of the following form: javacore.%Y%m%d.%H%M%S.%pid.txt (where

%pid is the process ID).

© Copyright IBM Corp. 2003, 2006 191

Triggering a Javadump

Triggering a Javadump
The Javadump is generated when one of the following occurs:
v A fatal native exception occurs in the JVM (not a Java Exception).
v The JVM has completely run out of heapspace.

Note: You can disable this option by setting the

IBM_JAVADUMP_OUTOFMEMORY=FALSE environment variable.
v You send a signal to the JVM from the operating system.
v You use the JavaDump() method within Java code that is being executed.

The exact conditions in which you get a Javadump vary depending on the
JAVA_DUMP_OPTS environment variable. For example, you can optionally get a
Javadump when the JVM terminates normally (on an interrupt). See Chapter 23,
“JVM dump initiation,” on page 209 for more information.

A ″fatal″ exception is one that causes the JVM to terminate. The JVM handles this
by producing a System dump followed by a Javadump and then terminates the
process. The behavior of the JVM in a failure is not affected by the Javadump and
should not affect the production of core files. However, it is possible that the
processing that is done to generate a Javadump might itself find a problem. In this
unlikely event, switch off Javadumps with the DISABLE_JAVADUMP=TRUE
environment variable.

Note: The exact format and content might be different from what is documented
at this stage.

In the user-controlled cases (the latter two), the JVM stops execution, performs the
dump, and then continues execution.

The signal for Linux is SIGQUIT. Use the command kill -3 n to send the signal to
a process with process id (PID) n. Alternatively, press Ctrl+\ in the shell window
that started Java.

The signal for z/OS is Ctrl+V.

In Windows, the dump is initiated by using Ctrl+Break in the command window

that started Java.

The class com.ibm.jvm.Dump contains a static JavaDump() method that causes

Java code to initiate a Javadump. In your application code, add a call to
com.ibm.jvm.Dump.JavaDump(). This call is subject to the same Javadump
environment variables as are described in “Enabling a Javadump” on page 191.

You can get a Javadump in a ″totally out of heapspace″ condition; that is, at the
same time as the Java application receives an OutOfMemory error. This feature is
enabled by default. You can disable it by using the
IBM_JAVADUMP_OUTOFMEMORY=FALSE environment variable.

Interpreting a Javadump
The information that is provided in a Javadump file is essentially the same,
regardless of the platform on which you are running the JVM. However, certain
platforms might provide more information about fatal exceptions.

192 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 interpreting a Javadump

Notes:
1. In some conditions, information might be missing because of the nature of a
crash.
2. The cross-platform sections of the dump are fully documented in the z/OS
example. The Linux and Windows examples build on top of this to describe
platform specifics.

Javadump tags
The Javadump files contain tags. This metadata makes it easier to parse and
perform simple analysis on the contents of Javadump files. An example tag is:
1CIJAVAVERSION J2RE 1.4.2 IBM J9 2.2 z/OS s390x-64 build 20040819_1437_BHdSMr

Normal tags have these characteristics:

v Tags are up to 15 characters long (padded with spaces).
v The first digit is a nesting level.
v The second and third characters identify the component that wrote the message
(for example, CI, LK, XE).
v The remainder is a unique string.
Special tags have these characteristics:
v A tag of ″NULL″ means the line is just to aid readability.
v Every section is headed by a tag of ″0SECTION″ with the section title.

Here is an example of some tags:

NULL ------------------------------------------------------------------------
0SECTION TITLE subcomponent dump routine
NULL ===============================
1TISIGINFO Dump Event "user" (00004000) received
1TIDATETIME Date: 2004/08/23 at 10:20:50
1TIFILENAME Javacore filename: /u/mcculls/test/javacore.20040823.102050.16908401.txt
NULL ------------------------------------------------------------------------
0SECTION GPINFO subcomponent dump routine
NULL ================================
2XHOSLEVEL OS Level : z/OS 06.00
2XHCPUS Processors -
3XHCPUARCH Architecture : s390x
3XHNUMCPUS How Many : 2
NULL

Note: For the rest of the chapter, the tags are removed to aid readability.

Locks, monitors, and deadlocks (LOCKS)

Here is an example of the LOCKS component part of the dump (this is practically
the same on all platforms). The LOCKS component handles locking in the JVM.

A lock prevents more than one entity from accessing a shared resource. Each object
in Java has an associated lock (gained by using a synchronized block or method).
In the case of the JVM, threads compete for various resources in the JVM and locks
on Java objects.

A monitor is a special kind of locking mechanism that is used in the JVM to allow
flexible synchronization between threads. For the purpose of this section, read the
terms monitor and lock interchangeably.

Chapter 21. Using Javadump 193

interpreting a Javadump

To avoid having a monitor on every object, the JVM usually uses a flag in a class
or method block to indicate that the item is locked. Most of the time, a piece of
code will transit some locked section without contention. Therefore, the guardian
flag is enough to protect this piece of code. This is called a flat monitor. However,
if another thread wants to access some code that is locked, a genuine contention
has occurred. The JVM must now create (or inflate) the monitor object to hold the
second thread and arrange for a signaling mechanism to coordinate access to the
code section. This monitor is now called an inflated monitor.
------------------------------------------------------------------------
LOCKS subcomponent dump routine
===============================

Monitor pool info:

Current total number of monitors: 2

Monitor Pool Dump (flat & inflated object-monitors):

sys_mon_t:0x000000013AD760E0 infl_mon_t: 0x000000013AD76120:
java/lang/Object@00000001088991A0/00000001088991B8: owner "Thread-3" (0x000000013AD78C00),
entry count 1
Waiting to enter:
"Thread-5" (0x000000013AD7EF00)
"Thread-6" (0x000000013AD7F600)
Waiting to be notified:
"Thread-1" (0x000000013AD73900)
"Thread-2" (0x000000013AD74000)
"Thread-4" (0x000000013AD79300)
sys_mon_t:0x000000013AD761B8 infl_mon_t: 0x000000013AD761F8:
java/lang/Object@000000010889A370/000000010889A388: owner "Thread-5" (0x000000013AD7EF00),
entry count 1
Waiting to be notified:
"Thread-3" (0x000000013AD78C00)

JVM System Monitor Dump (registered monitors):

Thread global lock (0x00000001083149C0): <unowned>
NLS hash table lock (0x0000000108314A08): <unowned>
VM thread list lock (0x0000000108314BB8): <unowned>
VM exclusive access lock (0x0000000108314C00): <unowned>
VM bind native lock (0x0000000108314C48): <unowned>
VM class loader blocks lock (0x0000000108314C90): <unowned>
VM class table lock (0x0000000108314CD8): <unowned>
VM string table lock (0x0000000108314D20): <unowned>
VM segment lock (0x0000000108314D68): <unowned>
VM JNI frame lock (0x0000000108314DB0): <unowned>
VM GC finalize master lock (0x0000000108314DF8): <unowned>
VM GC finalize run finalization lock (0x0000000108314E40): <unowned>
VM memory space list lock (0x0000000108314E88): <unowned>
VM sig quit lock (0x0000000108314F18): <unowned>
VM monitor table lock (0x0000000108314F60): <unowned>
VM volatile long lock (0x0000000108314FA8): <unowned>
VM mem segment list lock (0x0000000108314FF0): <unowned>
VM mem segment list lock (0x0000000108315038): <unowned>
VM mem segment list lock (0x0000000108315080): <unowned>
MM_ParallelDispatcher::slaveThread lock (0x00000001083150C8): <unowned>
Waiting to be notified:
"Gc Slave Thread" (0x0000000119815400)
MM_ParallelDispatcher::synchronize lock (0x0000000108315110): <unowned>
MM_WorkPackets::inputList lock (0x0000000108315158): <unowned>
MM_GCExtensions::gcStats lock (0x00000001083151A0): <unowned>
FinalizeListManager lock (0x0000000108315230): <unowned>
JIT sampling thread lock (0x0000000108315590): <unowned>
CompilationQueueMonitor lock (0x00000001083155D8): <unowned>
Waiting to be notified:
"[system]" (0x0000000108378300)
QueueSlotMonitor-0 lock (0x0000000108315620): <unowned>

194 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 interpreting a Javadump

The component dump is split into the following sections:

v Monitor pool info: Keeps track of the state of the LOCKS component.
v Monitor Pool Dump (flat & inflated object-monitors): The objects that threads
are waiting to lock.

Consider the monitor that is described by the part:

sys_mon_t:0x000000013AD760E0 infl_mon_t: 0x000000013AD76120:
java/lang/Object@00000001088991A0/00000001088991B8: owner "Thread-3" (0x000000013AD78C00),
entry count 1
Waiting to enter:
"Thread-5" (0x000000013AD7EF00)
"Thread-6" (0x000000013AD7F600)
Waiting to be notified:
"Thread-1" (0x000000013AD73900)
"Thread-2" (0x000000013AD74000)
"Thread-4" (0x000000013AD79300)
v The first line gives the address of some JVM monitor structures.
v The second line shows that a lock is on the java/lang/
Object@00000001088991A0/00000001088991B8 object and Thread-3 has this lock.
The entry count 1 shows that Thread-3 has entered this lock only once.
v The fourth line shows that a thread called Thread-5 with its JVM sys_thread_t
structure at 0x000000013AD7EF00 is waiting to get the lock.
v The seventh line shows that a thread called Thread-1 with its JVM sys_thread_t
structure at 0x000000013AD73900 has released the lock previously and is waiting
to be notified.

The entry count can be higher than 1 because a thread could enter a synchronized
method more than once (for example when an application uses recursion).

JVM system monitor dump (registered monitors)

This is a list of monitors that are maintained for use by the JVM. Each lock
contains details of which thread (including their respective JVM sys_thread_t data
structure addresses) holds the lock, if applicable.

Using the LOCKS component dump to diagnose a deadlock

Deadlocks are usually caused by an inconsistency in the locking semantics of the
application, or possibly some aspect of the Runtime Environment. This leads to
one of the following conditions:
v Thread 1 has lock A and wants lock B
v Thread 2 has lock B and wants lock A

That is: Thread 1 waits for B is locked by Thread 2 waits for A is locked by Thread
1..... - a cycle in the “waits for/locked by” graph.

Neither thread can proceed until the other releases the relevant lock; this cannot
happen. This situation could be more complex, involving three or more threads
with interdependent locks, but the principle remains the same. Other threads
usually end up blocked on one or other of the locks involved, thereby causing a
totally deadlocked Java application.
sys_mon_t:0x000000013AE8B410 infl_mon_t: 0x000000013AE8B450:
java/lang/Object@0000000108899168/0000000108899180: Flat locked by "Thread-2" (0x000000013AE04000),
entry count 1
Waiting to enter:
"Thread-3" (0x000000013AE8C600)
sys_mon_t:0x000000013AE8B458 infl_mon_t: 0x000000013AE8B498:

Chapter 21. Using Javadump 195

interpreting a Javadump

java/lang/Object@00000001088A1288/00000001088A12A0: Flat locked by "Thread-3" (0x000000013AE8C600),

entry count 1
Waiting to enter:
"Thread-2" (0x000000013AE04000)

The above LOCKS component dump is an example of a deadlock. First notice that
Thread-3 is waiting to lock the java/lang/Object@0000000108899168/
0000000108899180 object. Now Thread-2 has the lock of this object. Conversely,
Thread-2 is waiting to lock java/lang/Object@00000001088A1288/
00000001088A12A0, which is held by Thread-3. This is a clear (and in this case
simple) deadlock.
sys_mon_t:0x000000013AD760E0 infl_mon_t: 0x000000013AD76120:
java/lang/Object@00000001088991A0/00000001088991B8: owner "Thread-3" (0x000000013AD78C00), entry count 1
Waiting to enter:
"Thread-5" (0x000000013AD7EF00)
"Thread-6" (0x000000013AD7F600)
Waiting to be notified:
"Thread-1" (0x000000013AD73900)
"Thread-2" (0x000000013AD74000)
"Thread-4" (0x000000013AD79300)
sys_mon_t:0x000000013AD761B8 infl_mon_t: 0x000000013AD761F8:
java/lang/Object@000000010889A370/000000010889A388: owner "Thread-5" (0x000000013AD7EF00), entry count 1
Waiting to be notified:
"Thread-3" (0x000000013AD78C00)

The above dump is a more complex example of a deadlock. Thread-5 is waiting to

lock java/lang/Object@00000001088991A0/00000001088991B8, but Thread-3 has the
lock. However, Thread-3 is waiting to be notified on java/lang/
Object@000000010889A370/000000010889A388 - which is held by Thread-5. No
thread can get the lock to notify Thread-3 (as it is owned by Thread-5) and
Thread-5 cannot notify Thread-3, because it is blocked on another lock.

Note: Some categories of deadlock cannot be diagnosed automatically; they require

understanding of the synchronization in the application. For example, if
threads have interdependencies on wait()/notify() operations, you cannot be
aware, from the diagnostic information, of which thread would be expected
to notify some thread that is waiting.

The dump formatter (see Chapter 26, “Using the dump formatter,” on page 223)
can also diagnose deadlocks.

Javadump sample output 1 (z/OS)

This is a sample of the top section of a Javadump file that was produced on z/OS.

The following Javadump sample output 1 (z/OS) section applies to Linux and
Windows also.
------------------------------------------------------------------------
TITLE subcomponent dump routine
===============================
Dump Event "gpf" (00002000) received
Date: 2004/08/23 at 11:21:17
Javacore filename: /u/mcculls/javacore.20040823.112117.16908826.txt
------------------------------------------------------------------------
GPINFO subcomponent dump routine
================================
OS Level : z/OS 06.00
Processors -
Architecture : s390x
How Many : 2

196 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 interpreting a Javadump (z/OS)

signal: 000000000000000B

Registers:
gpr0:0000000000000000
gpr1:0000000000000000
gpr2:000000013AD6D168
gpr3:00000000112D5A68
gpr4:00000001082FE780
gpr5:000000013AD33E00
gpr6:000000001278F658
gpr7:000000001120901E
gpr8:0000000000000007
gpr9:0000000000000000
gpr10:0000000108377D70
gpr11:000000013AD6D168
gpr12:0000000108300C60
gpr13:0000000108377D00
gpr14:0000000011208918
gpr15:0000000000000000
fpr0:41400000C8A848BF
fpr1:4580000000000000
fpr2:0000000000000000
fpr3:3FF0000000000000
fpr4:406F000000000000
fpr5:0000000000000000
fpr6:0000000000000000
fpr7:0000000000000000
fpr8:0000000000000000
fpr9:0000000000000000
fpr10:0000000000000000
fpr11:0000000000000000
fpr12:0000000000000000
fpr13:0000000000000000
fpr14:0000000000000000
fpr15:0000000000000000
psw0:0785040180000000
psw1:000000001278F67C
fpc:0000000000000000

VM flags:00000000

------------------------------------------------------------------------
ENVINFO subcomponent dump routine
=================================
J2RE 1.4.2 IBM J9 2.2 z/OS s390x-64 build 20040819_1437_BHdSMr (JIT enabled - r7_level20040818_1801)
Running as a standalone JVM
java -Xmx4m -classpath test -Xbootclasspath/p:fix.jar gpf
Java Home Dir: /u/mcculls/sdk/J1.4_64
Java DLL Dir: /u/mcculls/sdk/J1.4_64/bin
Sys Classpath: fix.jar;/u/mcculls/sdk/J1.4_64/lib/jclSC14/classes.zip;/u/mcculls/sdk/J1.4_64/
lib/core.jar;
/u/mcculls/sdk/J1.4_64/lib/charsets.jar;/u/mcculls/sdk/J1.4_64/lib/graphics.jar;
/u/mcculls/sdk/J1.4_64/lib/security.jar;/u/mcculls/sdk/J1.4_64/lib/ibmpkcs.jar;/u/mcculls/sdk/
J1.4_64/lib/ibmorb.jar;
/u/mcculls/sdk/J1.4_64/lib/ibmorbapi.jar;/u/mcculls/sdk/J1.4_64/lib/ibmjcefw.jar;
/u/mcculls/sdk/J1.4_64/lib/ibmjssefips.jar;
/u/mcculls/sdk/J1.4_64/lib/ibmjgssprovider.jar;/u/mcculls/sdk/J1.4_64/lib/ibmjsseprovider.jar;
/u/mcculls/sdk/J1.4_64/lib/ibmjaaslm.jar;
/u/mcculls/sdk/J1.4_64/lib/ibmjaasactivelm.jar;/u/mcculls/sdk/J1.4_64/lib/ibmcertpathprovider.jar;
/u/mcculls/sdk/J1.4_64/lib/server.jar;
/u/mcculls/sdk/J1.4_64/lib/xml.jar;
UserArgs:
-Xjcl:jclscar_22
-Dcom.ibm.oti.vm.bootstrap.library.path=/u/mcculls/sdk/J1.4_64/bin
-Dsun.boot.library.path=/u/mcculls/sdk/J1.4_64/bin
-Djava.library.path=/u/mcculls/sdk/J1.4_64/bin/j9vm/libjvm.so:/u/mcculls/sdk/J1.4_64/bin/j9vm:
/u/mcculls/sdk/J1.4_64/bin/:/u/mcculls/javasrc/bin:/u/mcculls/javasrc/bin/j9vm:

Chapter 21. Using Javadump 197

interpreting a Javadump (z/OS)

/lib:/usr/lib:.
-Djava.home=/u/mcculls/sdk/J1.4_64
-Djava.ext.dirs=/u/mcculls/sdk/J1.4_64/lib/ext
-Duser.dir=/u/mcculls
_j2se_j9
vfprintf 0x0000000108300060
-Xmx4m
-Xbootclasspath/p:fix.jar
-Dinvokedviajava
-Djava.class.path=test
vfprintf
-Xdump

JVM Monitoring Interface (JVMMI)

------------------------
[not available]
...
...
------------------------------------------------------------------------
THREADS subcomponent dump routine
=================================

Current Thread Details

----------------------
"main" (TID:0x0000000108377D00, sys_thread_t:0x0000000108315E28, state:R, native ID:0x1112F10000000001)
prio=5
at gpf.action(Native Method)
at gpf.main(gpf.java:8)

File header (TITLE)

The top of the file shows general information about the dump. In this case, an
unhandled exception (″gpf″ event) occurred in the JVM and caused it to crash.
Looking down the dump at the THREADS section, you can see that it occurred in
a thread called ″main″ with TID (Thread Identifier) 0x0000000108377D00). This can
be cross-referenced against other parts of the file; for example, the stack traces and
locks.

Crash Summary (GPINFO)

This section is platform-specific. It contains the information that you get for a crash
on z/OS. It lists signal information and register contents at the time of the
problem.

System properties (ENVINFO)

This section of the file shows:
v SDK Version and Build Identifier: J2RE 1.4.2 IBM J9 2.2 z/OS s390x-64 build
20040819_1437_BHdSMr.
v The command-line argument that started the JVM: java -Xmx4m -classpath test
lock.
v The location from which the Runtime Environment executables were loaded:
/u/mcculls/sdk/J1.4_64..
v The bootclasspath: SYS Classpath
/u/mcculls/sdk/J1.4_64/lib/jclSC14/classes.zip;
/u/mcculls/sdk/J1.4_64/lib/core.jar;...

.
v Arguments supplied when initializing the JVM labeled under UserArgs. For
example: -Djava.class.path=test.

198 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 interpreting a Javadump (z/OS)

The main diagnostic function of this section is to determine exactly what native
executables and Java classes were being run when the dump occurred. This can
include the java executable, the Java API, IBM extensions, and user application
class files.

The UserArgs section shows arguments for the JVM, which might have been
supplied by the user or generated during JVM initialization. For example, the
-Djava.class.path=test property was generated by the user specifying the option
-classpath test on the command-line. However, -Duser.dir=/u/mcculls was
generated automatically.

The bootclasspath (classpath of the bootstrap class loader) contains the locations
from which the Java API is loaded. This takes the default bootclasspath and is then
modified by any supplied arguments. In this case, the -Xbootclasspath/p:\fix.jar
argument adds fix.jar to the start of the default bootclasspath. The effect of this
is that the JVM will attempt to load classes (including the Java API) from
D:\fix.jar before
/u/mcculls/sdk/J1.4_64/lib/jclSC14/classes.zip;
/u/mcculls/sdk/J1.4_64/lib/core.jar;...

The classpath (generally refers to the system or application class loader’s classpath)
takes a default value of ″.″ (the working directory of Java). This causes the
-Djava.class.path=. property to be set by default. In this example, it is then
overridden by -Djava.class.path=test (generated from the command-line option
-classpath test). This example shows the case where a later value of a system
property replaces an earlier value in the system property list.

Storage Management (MEMINFO)

See Chapter 2, “Understanding the Garbage Collector,” on page 7 for information
about how the memory manager component works. This part of the file gives
various storage management values, including:
v Free space in heap (5eff0) and size of current heap (a1010)
-----------------------------------------------------------
MEMINFO subcomponent dump routine
=================================
Bytes of Heap Space Free: 5eff0
Bytes of Heap Space Allocated: a1010

v A list of allocated memory segments

Threads and stack trace (THREADS)

This section shows a complete list of Java threads that are alive.

Chapter 21. Using Javadump 199

interpreting a Javadump (z/OS)

------------------------------------------------------------------------
THREADS subcomponent dump routine
=================================

Current Thread Details

----------------------
"main" (TID:0x0000000108377D00, sys_thread_t:0x0000000108315E28, state:R, native ID:0x1112F10000000001)
prio=5
at gpf.action(Native Method)
at gpf.main(gpf.java:8)

All Thread Details

------------------

Full thread dump J9SE VM (J2RE 1.4.2 IBM J9 2.2 z/OS s390x-64 build 20040819_1437_BHdSMr, native threads):
"main" (TID:0x0000000108377D00, sys_thread_t:0x0000000108315E28, state:R, native ID:0x1112F10000000001)
prio=5
at gpf.action(Native Method)
at gpf.main(gpf.java:8)
"[system]" (TID:0x0000000108378400, sys_thread_t:0x0000000108315F38, state:CW, native ID:0x11144B7000000003)
prio=11
"Signal Dispatcher" (TID:0x0000000119833A00, sys_thread_t:0x0000000108315FC0, state:R, native ID:0x11163C8000000004)
prio=5
at com/ibm/misc/SignalDispatcher.waitForSignal(Native Method)
at com/ibm/misc/SignalDispatcher.run(SignalDispatcher.java:78)
"Gc Slave Thread" (TID:0x0000000119834100, sys_thread_t:0x0000000108316158, state:CW, native ID:0x111669B000000007)
prio=5

A thread is alive if it has been started but not yet stopped. A Java thread is
implemented by a native thread of the operating system. Each thread is
represented by a line such as:
"main" (TID:0x0000000108377D00, sys_thread_t:0x0000000108315E28,
state:R, native ID:0x1112F10000000001) prio=5

The properties of a thread are name, identifier, JVM data structure address, current
state, native thread identifier, and priority. A large value for priority means that the
thread has a high priority. The values of state can be:
v R - Runnable - the thread is able to run when given the chance.
v CW - Condition Wait - the thread is waiting. For example, because:
– A sleep() call is made.
– The thread has been blocked for I/O.
– A synchronized method of an object locked by another thread has been called.
– The thread is synchronizing with another thread with a join() call.
v S – Suspended – the thread has been suspended by another thread.
v Z – Zombie – the thread has been killed.

Below each thread there is a stack trace for that thread. A stack trace is a
representation of the hierarchy of Java method calls made by the thread. For
example:

"AWT-Shutdown" (TID:0x000000013AE03900, sys_thread_t:0x0000000108316160, state:CW, native ID:0x111678C000000008) prio=5

at java/lang/Object.wait(Native Method)
at java/lang/Object.wait(Object.java:193)
at sun/awt/AWTAutoShutdown.run(AWTAutoShutdown.java:292)

The sun/awt/AWTAutoShutdown.run() method calls java/lang/Object.wait()

which calls java/lang/Object.wait() which is then waiting on some condition
(thread state is CW). To the right of each method name called is the source of the
code for the method. Examples of this are:
v at java/lang/Object.wait(Object.java:193) - The wait() method is at line 193
of a Java source file called Object.java.

200 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 interpreting a Javadump (z/OS)

v at java/lang/Object.wait(Bytecode PC:3) - The wait() method is at bytecode 3

of a Java method for which source information is not available (for example if it
was not compiled with -g).
v at java.lang.Object.wait(Native Method) - The wait() method is a native
method. This could be user application JNI code or (as in this case) Java API
implementation.

Classloaders and Classes (CLASSES)

See Chapter 3, “Understanding the class loader,” on page 25 for information about
the parent-delegation model. The classloader section includes:
v Classloader summaries. The defined class loaders and the relationship between
them
v Classoader loaded classes. The classes that are loaded by each classloader
In this example, there are the standard three classloaders:
v Application classloader (sun/misc/Launcher$AppClassLoader), which is a child
of the
v Extension classloader (sun/misc/Launcher$ExtClassLoader), which is a child of
the
v Bootstrap (sometimes called “system”) classloader (*System*).
As an example, take the application classloader with the full name
sun/misc/Launcher$AppClassLoader. Under ″Classloader summaries″, this has
flags -----ta-, which, from the key above, shows that the class loader is 6=trusted
and 7=application. It gives the number of loaded classes (1) and the parent
classloader sun/misc/Launcher$ExtClassLoader(0x000000010884EE70). The parent
address 0x102B4098 corresponds to the entry Shadow 0x000000010884EE70 for the
extension classloader entry below. Under the ″ClassLoader loaded classes″ heading,
you can see that the application classloader has loaded one class called gpf at
address 0x000000013AD6D068.

------------------------------------------------------------------------
CLASSES subcomponent dump routine
=================================
Classloader summaries
12345678: 1=primordial,2=extension,3=shareable,4=middleware,5=system,6=trusted,7=application,8=delegating
p---st-- Loader *System*(0x00000001088476C0)
Number of loaded classes 274
-x--st-- Loader sun/misc/Launcher$ExtClassLoader(0x000000010884EE70), Parent *none*(0x0000000000000000)
Number of loaded classes 0
-----ta- Loader sun/misc/Launcher$AppClassLoader(0x0000000108864A58), Parent sun/misc/Launcher$ExtClassLoader(0x000000010884EE70)
Number of loaded classes 1
ClassLoader loaded classes
Loader *System*(0x00000001088476C0)
java/security/UnresolvedPermission(0x000000013AD54018)
java/security/BasicPermissionCollection(0x000000013AD54A58)
java/security/Principal(0x000000013AD55218)
[Ljava/security/Principal;(0x000000013AD553C8)
....................left out to save space....................
java/lang/OutOfMemoryError(0x0000000108CB1C00)
java/util/Dictionary(0x0000000108CB1E60)
Loader sun/misc/Launcher$ExtClassLoader(0x000000010884EE70)
Loader sun/misc/Launcher$AppClassLoader(0x0000000108864A58)
gpf(0x000000013AD6D068)

Final section
This section gives confirmation that the file is complete.
------------------------------------------------------------------------
Javadump End section
Javadump Buffer Usage Information
=================================
Javadump buffer size (allocated): 2621440
Javadump buffer size (used) : 27752
---------------------- END OF DUMP -------------------------------------

Chapter 21. Using Javadump 201

Javadump sample output 2 (Linux)

Javadump sample output 2 (Linux)

This section describes the Linux-specific parts of the Javadump. The cross-platform
sections are covered in the z/OS section above; you should read them before this
section.

Crash summary (GPINFO)

------------------------------------------------------------------------
GPINFO subcomponent dump routine
================================
OS Level : Linux 2.4.21-231-smp
Processors -
Architecture : amd64
How Many : 4

Module: ./libgpf.so
Module_base_address: 0000007FBFFFDA50
Symbol: Java_gpf_action
Symbol_address: 0000000000000001

Registers:
rdi:000000004011E600
rsi:000000004038B068
rax:0000000000000000
rbx:0000000000000110
rcx:000000004015A5E2
rdx:000000004015A5D0
rbp:0000007FBFFFF170 r8:000000004015A5E7
r9:000000004015A5D0
r10:0000000000000000
r11:0000000000000000
r12:0000000000000002
r13:0000000001000109
r14:0000002A96090200
r15:0000002AB8ED1898
rip:0000002AB8ED18B0
rsp:0000007FBFFFF170
eflags:0000000000010246
cs:0000000000000033
fs:0000000000000000
gs:0000000000000000

VM flags:00000000

On Linux, this section gives more detailed information about the location of the
crash (for example the failing module) along with the register listing.

Javadump sample output 3 (Windows)

This section describes the Windows-specific parts of the Javadump. The
cross-platform sections are covered in the z/OS section above; you should read
them before this section.

Crash summary (GPINFO)

------------------------------------------------------------------------
GPINFO subcomponent dump routine
================================
OS Level : Windows Server 2003 5.2 build 3790 Service Pack 1, v.1159
Processors -
Architecture : amd64
How Many : 4

ExceptionCode: 00000000C0000005
ExceptionAddress: 000007FF8A89105A
ContextFlags: 000000000010001F

202 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 interpreting a Javadump (Windows)

Handler1: 00000000112027A0
Handler2: 0000000010026BB0

Module: E:\mcculls\test\gpf.dll
Module_base_address: 000007FF8A890000
Offset_in_DLL: 000000000000105A

Registers:
RDI:0000000000000000
RSI:0000000000000007
RAX:0000000000000000
RBX:0000000000000110
RCX:000007FFFFC00A00
RDX:000007FF8D040CF8
RBP:000007FFFFF7FCD0
R8:0000000000000000
R9:000007FFFFF7FCD0
R10:0000000000000000
R11:000007FF8D040DA0
R12:0000000001000109
R13:000007FF8A891005
R14:00000000100466A0
R15:000007FFFFF7F8C0
RIP:000007FF8A89105A
RSP:000007FFFFF7F8A0

VM flags:00000000

On Windows this section gives more detailed information about the location of the
crash (for example the failing module) along with the register listing.

Chapter 21. Using Javadump 203

interpreting a Javadump (Windows)

204 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 22. Using Heapdump
This chapter describes:
v “Summary of Heapdump”
v “Information for users of previous releases of Heapdump”
v “Enabling a Heapdump”
v “Location of the generated Heapdump” on page 207
v “Producing a Heapdump using jdmpview” on page 207
v “Available tools for processing Heapdumps” on page 208
v “Using VerboseGC to obtain heap information” on page 208

Summary of Heapdump
The term Heapdump is used to describe the IBM Virtual Machine for Java
mechanism that generates a dump of all the live objects that are on the Java heap;
that is, those that are being used by the running Java application. From Version
1.4.2 Service Refresh 2, this dump is stored in a file (using the phd format). You
can use various tools on the Heapdump output to analyze the composition of the
objects on the heap and (for example) help to find the objects that are controlling
large amounts of memory on the Java heap and the reason why the Garbage
Collector cannot collect them.

Information for users of previous releases of Heapdump

Heapdumps for the platforms described in this guide are different from previous
releases of the IBM Virtual Machine for Java. Heapdumps are now produced in
phd format and you can view them using a variety of tools. See “Available tools
for processing Heapdumps” on page 208. Before Version 1.4.2, Service Refresh 2,
Heapdump phd files were produced using the jdmpview tool from a combination
of full system dumps and the jextract post-processor tool. This technique is still
supported and described in Chapter 26, “Using the dump formatter,” on page 223.

Enabling a Heapdump
You can generate a Heapdump from a running JVM in either of two ways:
v Explicit generation
v JVM-triggered generation

When the Java heap is exhausted (that is, the OutOfMemory condition is
encountered and the resulting exception is not caught or handled by the
application), JVM-triggered generation is enabled by default, as are Heapdumps
that are generated by other programming methods. To enable signal-based
Heapdumps, set the IBM_HEAPDUMP or IBM_HEAP_DUMP environment
variable to any value,
export IBM_HEAPDUMP=<any_value>
export IBM_HEAP_DUMP=<any_value>

or set the appropriate JAVA_DUMP_OPTS before you start the Java process. You
can also use the -Xdump agent option to get more fine-grained control over
Heapdumps. See Chapter 24, “Using dump agents,” on page 213 for more
information.

© Copyright IBM Corp. 2003, 2006 205

Enabling a Heapdump

To display on JVM startup the conditions (if any) that will generate a Heapdump
(or javadump or systemdump), you can use -Xdump:what. See Chapter 24, “Using
dump agents,” on page 213 for more information.

To disable generation of a Heapdump, on platforms other than Windows use:

unset IBM_HEAPDUMP
unset IBM_HEAP_DUMP

On Windows, use:
set IBM_HEAPDUMP=
set IBM_HEAP_DUMP=

Explicit generation of a Heapdump

You can explicitly generate a Heapdump in the following ways:
v By sending a signal to the JVM from the operating system
v By using the HeapDump() method inside Java code that is being executed
v By using the JVMRI to request a Heapdump from a loaded agent

You can explicitly request a Heapdump in the same way as a Javadump. Before the
Heapdump starts, the heap is locked and remains locked until the whole
Heapdump file is written to disk. This operation can affect the behavior of your
Java application, and make it unresponsive while the dump is being produced.

For Linux, send the JVM the signal SIGQUIT (kill -3, or Ctrl+\ in the console
window).

For Windows, generate a SIGINT (press the Ctrl+Break keys simultaneously).

You can explicitly request a Heapdump from a Java method. The class
com.ibm.jvm.Dump contains a static HeapDump() method that causes Java code to
initiate a Heapdump.

Triggered generation of a Heapdump

The following events automatically trigger the JVM to produce a Heapdump (if
enabled):
v An OutOfMemory or heap exhaustion condition occurs and the resulting
exception is not caught or handled by the application
v If Heapdumps are enabled, they are normally produced immediately before a
Javadump.

The first option is enabled by default, and you can see it with -Xdump:what. It
gives a snapshot of the Java heap when no more memory is available. Usually, this
snapshot is the most useful output to help you determine the cause of an
OutOfMemory condition that is related to the Java heap. It works independently of
the IBM_HEAPDUMP environment variable. So, by default, you get Heapdumps
only when no more heap space is available; you do not get Heapdumps in crashes
or through a signal to the JVM. You can disable this feature, and a similar one for
Javadumps, by using IBM_HEAPDUMP_OUTOFMEMORY=FALSE and
IBM_JAVADUMP_OUTOFMEMORY=FALSE respectively.

Sometimes an application will catch and handle an OutOfMemory condition, in

which case no heapdumps will be taken. You can force heapdump generation for
any handled OutOfMemory conditions, by setting

206 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Enabling a Heapdump

IBM_HEAPDUMP_OUTOFMEMORY=TRUE and
IBM_JAVADUMP_OUTOFMEMORY=TRUE respectively.

Enabling text formatted (″classic″) Heapdumps

The generated Heapdump is by default in the binary, platform-independent, phd
format, which can be examined using the available tooling (see “Available tools for
processing Heapdumps” on page 208). However, it is sometimes useful to have an
immediately readable view of the heap. You can obtain this view by using the
opts= stanza with -Xdump:heap (see Chapter 24, “Using dump agents,” on page
213) or by the existence of an environment variable:
v IBM_JAVA_HEAPDUMP_TEST, which allows you to perform the equivalent of
opts=PHD+CLASSIC
v IBM_JAVA_HEAPDUMP_TEXT, which allows the equivalent of opts=CLASSIC

Location of the generated Heapdump

The JVM checks each of the following locations for existence and write-permission,
then stores the Heapdump in the first one that is available.
v The location that is specified using the file suboption on the triggered
-Xdump:heap agent.
v The location that is specified by the IBM_HEAPDUMPDIR environment
variable, if set.
v The current working directory of the JVM processes.
v The location that is specified by the TMPDIR environment variable, if set.
v The /tmp directory. On Windows, C:\temp.
Notes:
1. On z/OS, a Heapdump is always stored on TSO as a standard MVS data set.
2. Enough free disk space must be available for the Heapdump file to be written
correctly.

The generated Heapdump will have a name of the form:

heapdump.%Y%m%d.%H%M%S.%pid.phd

where:
v %pid is the process ID
v .%Y%m%d.%H%M%S is the date and time
Notes:
1. If ″Classic″ Heapdump is enabled, the name of the Heapdump will end in txt
rather than phd.
2. You can override the standard names by use of the label= parameter. See
Chapter 24, “Using dump agents,” on page 213 for more information.

Producing a Heapdump using jdmpview

Before the direct production of phd files using Heapdump, you could produce
them with the hd command from within jdmpview. This mechanism is still valid
and can be used with system dumps generated with request=exclusive+prepwalk
(see Chapter 24, “Using dump agents,” on page 213). For information on the use of
the hd command, see Chapter 26, “Using the dump formatter,” on page 223.

Chapter 22. Using Heapdump 207

Tools for processing Heapdumps

Available tools for processing Heapdumps

There are several tools available for Heapdump analysis, typically downloadable
from the Web. Further details of the range of available tools can be found at
http://www.ibm.com/support/docview.wss?uid=swg24009436

Using VerboseGC to obtain heap information

Use the VerboseGC utility to obtain information about the Java Object heap in real
time while running your Java applications. To activate this utility, run Java with
the -verbosegc option:

java -verbosegc

For more information see Chapter 2, “Understanding the Garbage Collector,” on

page 7.

208 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 23. JVM dump initiation
The JVM supports the ability to generate a native system dump. In addition, a
very simple scripting ability allows you to choose when and how a dump is
generated. The exact dump created is, by definition, platform-dependent. The way
you proceed with analysis of the dump depends on the platform tools that are
available. A short description of how to proceed with the JVM native dump is
provided.

This chapter contains the following sections:

v “Overview”
v “Settings” on page 210
v “Platform-specific variations” on page 211

Overview
The JVM might produce dump files in response to specific events, depending on
the setting of the environment variables JAVA_DUMP_OPTS and
JAVA_DUMP_TOOL, as well as any -Xdump agent options specified on the
command line; see Chapter 24, “Using dump agents,” on page 213 for more
information.

These events (or conditions) are grouped as follows:

EXCEPTION
Unexpected synchronous terminating signal; that is, an unrecoverable
storage violation.
ERROR
Controlled abort because an error is detected internally; for example, no
more memory is available.
INTERRUPT
Asynchronous terminating signal; for example, you pressed Ctrl-C.
DUMP
This can be caused if you press Ctrl-BREAK on Windows, Ctrl-V on z/OS,
or Ctrl-\ on Linux.
OUTOFMEMORY
The JVM cannot satisfy a request for storage.

The types of dump that can be produced (platform-specific variations are noted
below) are:
1. SYSDUMP. An unformatted dump that the operating system generated
(basically a core file).
2. HEAPDUMP. An internally generated dump of the objects that are on the Java
heap.
3. User specified. Whatever the JAVA_DUMP_TOOL variable specifies.
4. JAVADUMP. An internally generated and formatted analysis of the JVM.

If all types of dump are requested, they are produced in the above sequence
(JAVADUMP always being last). You can read system dumps by using native

© Copyright IBM Corp. 2003, 2006 209

Overview of JVM dump initiation

dump analysis tools (IPCS, dbx, and so on), although they can also be processed in
a platform-independent way using jextract and the JVM Dump Formatter.

SYSDUMP file names and locations vary with each platform and are detailed
below. For more information about JAVADUMP files, see Chapter 21, “Using
Javadump,” on page 191.

If any external dump exit routines have been registered, they are run before the
main JVM dump sequence (see above), and can optionally terminate all further
dump processing by returning RAS_DUMP_ABORT.

Settings
In the absence of overrides from the -Xdump option on the command line starting
the JVM (see Chapter 24, “Using dump agents,” on page 213), the dumps that are
produced for each condition are determined primarily by the JAVA_DUMP_OPTS
environment variable as follows:
JAVA_DUMP_OPTS="ONcondition(dumptype,dumptype),ONcondition(dumptype,...),...)

where:
v condition can be:
– ANYSIGNAL
– DUMP
– ERROR
– INTERRUPT
– EXCEPTION
– OUTOFMEMORY
v and dumptype can be:
– ALL
– NONE
– JAVADUMP
– SYSDUMP
– HEAPDUMP

The default, if JAVA_DUMP_OPTS is not set, is:

For platforms other than z/OS:
JAVA_DUMP_OPTS="ONDUMP(JAVADUMP),ONERROR(SYSDUMP,JAVADUMP),ONEXCEPTION(SYSDUMP,JAVADUMP)"
For z/OS:
JAVA_DUMP_OPTS="ONDUMP(HEAPDUMP,JAVADUMP),ONERROR(SYSDUMP,JAVADUMP),ONEXCEPTION(SYSDUMP,JAVADUMP)"
which indicates that system dumps and javadumps are to be taken on failing
conditions, and javadumps on user requests. JAVA_DUMP_OPTS is parsed by
taking the first (leftmost) occurrence of each condition, so duplicates are ignored.
That is,
ONERROR(SYSDUMP),ONERROR(JAVADUMP) creates system dumps for error conditions.
Also, the ONANYSIGNAL condition is parsed before all others, so
ONINTERRUPT(NONE),ONANYSIGNAL(SYSDUMP)
has the same effect as
ONANYSIGNAL(SYSDUMP),ONINTERRUPT(NONE).

If the JAVA_DUMP_TOOL environment variable is set, that variable is assumed to

specify a valid executable name and is parsed for replaceable fields, such as %pid.

210 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 JVM dump - settings

If %pid is detected in the string, the string is replaced with the JVM’s own process
ID. The tool specified by JAVA_DUMP_TOOL is run after any system or heap
dump has been taken, but before anything else.

If the OUTOFMEMORY condition is used, it overrides the

IBM_HEAPDUMP_OUTOFMEMORY and IBM_JAVADUMP_OUTOFMEMORY
settings and takes the prescribed dumps whenever an out-of-memory exception is
thrown (even if it is handled).

Platform-specific variations
Conditions can be mapped to different signals on different platforms, and some
signals are recognized on some platforms but not on others. Table 9 shows the
mapping across platforms. Note that if the JVM receives a signal that it does not
recognize (that is, it is not mapped to a condition as listed in the table), the default
operating system action for that signal is taken. Usually the signal is ignored.
Table 9. Signal mappings on different platforms
z/OS Windows Linux
EXCEPTION SIGTRAP SIGTRAP
SIGILL SIGILL SIGILL
SIGSEGV SIGSEGV SISEGV
SIGFPE SIGFPE SIGFPE
SIGBUS SIGBUS
SIGSYS
SIGXCPU SIGXCPU
SIGXFSZ SIGXFSZ

INTERRUPT SIGINT SIGINT SIGINT

SIGTERM SIGTERM SIGTERM
SIGHUP SIGHUP
ERROR SIGABRT SIGABRT SIGABRT
DUMP SIGQUIT SIGQUIT
SIGBREAK

If a signal is not handled by the JVM, the operating system takes its default action
for that signal. In the case of an EXCEPTION type signal, it is most likely to
produce a system dump.

z/OS
The full syntax for JAVA_DUMP_OPTS on z/OS is:
JAVA_DUMP_OPTS="ONcondition(dumptype,dumptype),ONcondition
(dumptype,...),...)"

where dumptype can be:

v ALL
v NONE
v JAVADUMP (see Chapter 21, “Using Javadump,” on page 191)
v SYSDUMP
Chapter 23. JVM dump initiation 211
JVM dump initiation - z/OS

v CEEDUMP
v HEAPDUMP (see Chapter 22, “Using Heapdump,” on page 205)

If CEEDUMP is specified, an LE CEEDUMP is produced for the relevant

conditions, after any SYSDUMP processing, but before a JAVADUMP is produced.
A CEEDUMP is a formatted summary system dump that shows stack traces for
each thread that is in the JVM process, together with register information and a
short dump of storage for each register.

Under z/OS, you can change the behavior of LE by setting the _CEE_RUNOPTS
environment variable (for details refer to the LE Programming Reference). In
particular, the TRAP option determines whether LE condition handling is enabled,
which, in turn, drives JVM signal handling, and the TERMTHDACT option
indicates the level of diagnostic information that LE should produce.

Dumps are produced in the following form:

v SYSDUMP: On TSO as a standard MVS data set, using the default name of the
form: %uid.JVM.TDUMP.%job.D%y%m%d.T%H%M%S, or as determined by the
setting of the JAVA_DUMP_TDUMP_PATTERN environment variable.
v CEEDUMP: In the current directory, or as determined by the setting of
_CEE_DMPTARG as: CEEDUMP.%Y%m%d.%H%M%S.%pid.
v HEAPDUMP: On TSO as a standard MVS data set, using the default name of
the form: heapdump.%y%m%d.T%H%M%S.phd, or as determined by the setting
of the JAVA_DUMP_HDUMP_PATTERN environment variable.
v JAVADUMP: In the same directory as CEEDUMP, or standard JAVADUMP
directory as: javacore.%Y%m%d.%H%M%S.%pid.txt.

Windows
Dumps are produced in the following form:
v SYSDUMP: Output is written to a file named core.%Y%m%d.%H%M%S.
%pid.dmp into the same directory that is used for JAVADUMP.
v JAVADUMP: Output is written to a file named a file named
javacore.%Y%m%d.%H%M%S.%pid.txt. See Chapter 21, “Using Javadump,” on
page 191 for more information.
v HEAPDUMP: The raw heap image is written to a file named
heapdump.%Y%m%d.%H%M%S.%pid.phd. See Chapter 22, “Using Heapdump,”
on page 205 for more information.

Linux
Dumps are produced in the following form:
v SYSDUMP: Output is written to a file named core.%Y%m%d.%H%M%S.
%pid.dmp into the same directory that is used for JAVADUMP.
v JAVADUMP: Output is written to a file named javacore.%Y%m%d.%H%M%S.
%pid.txt. See Chapter 21, “Using Javadump,” on page 191 for more information.
v HEAPDUMP: The raw heap image is written to a file named
heapdump.%Y%m%d.%H%M%S.%pid.phd . See Chapter 22, “Using
Heapdump,” on page 205for more information.

212 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 24. Using dump agents
Dump agents are set up during JVM initialization. They enable you to use events
occurring within the JVM, such as Garbage Collection, thread start, or JVM
termination, to initiate one of four types of dump or to launch an external tool.
There are default dump agents set up at JVM initialization, which will suffice in
most cases, but the use of the -Xdump option on the command line allows more
detailed configuration of dump agents. The total set of options and suboptions
available under -Xdump is very flexible and there are many examples presented in
this chapter to show this flexibility.

The -Xdump option allows you add and remove dump agents for various JVM
events, update default dump settings (such as the dump name), and limit the
number of dumps that are produced.

This chapter describes:

v “Help options”
v “Dump types and triggering” on page 214
v “Types of dump agents - examples” on page 215
v “Default dump agents” on page 217
v “Default settings for dumps” on page 218
v “Limiting dumps using filters” on page 218
v “Removing dump agents” on page 218

Help options
You can obtain help on the various usage aspects of -Xdump by using java
-Xdump:help.
Table 10. Usage from java -Xdump:help
Command Result
-Xdump:help Print general dump help
-Xdump:none Ignore all previous and default dump
options
-Xdump:events List available trigger events
-Xdump:request List additional VM requests
-Xdump:tokens List recognized label tokens
-Xdump:what Show registered agents on startup
-Xdump:<type>:help Print detailed dump help
-Xdump:<type>:none Ignore previous dump options of this type
-Xdump:<type>:defaults Print and update default settings for this
type
-Xdump:<type> Request this type of dump (using defaults)

Table 11. Types of dump

Valid types of dump Description

© Copyright IBM Corp. 2003, 2006 213

Using dump agents - help options

Table 11. Types of dump (continued)

-Xdump:console Basic thread dump to stderr
-Xdump:system Capture raw process image
-Xdump:tool Run command line program
-Xdump:java Write application summary
-Xdump:heap Capture raw heap image

As an example:
java -Xdump:heap:none -Xdump:heap:events=fullgc class [args...]

turns off default Heapdumps and then requests a Heapdump on every full GC.

As can be seen from Table 10 on page 213, further help is available for the assorted
suboptions under -Xdump. In particular, java -Xdump:events shows the available
keywords used to specify the events that can be used.

You must filter class events (such as load, throw, and uncaught) by class name. For
guidance, see “Limiting dumps using filters” on page 218.
Table 12. Keywords
Supported event keywords Event hook
gpf ON_GP_FAULT
user ON_USER_SIGNAL
abort ON_ABORT_SIGNAL
vmstart ON_VM_STARTUP
vmstop ON_VM_SHUTDOWN
load ON_CLASS_LOAD
unload ON_CLASS_UNLOAD
throw ON_EXCEPTION_THROW
brkpoint ON_BREAKPOINT
framepop ON_DEBUG_FRAME_POP
thrstart ON_THREAD_START
blocked ON_THREAD_BLOCKED
thrstop ON_THREAD_END
expand ON_HEAP_EXPAND
fullgc ON_GLOBAL_GC
uncaught ON_EXCEPTION_DESCRIBE
slow ON_SLOW_EXCLUSIVE_ENTER
any *

Dump types and triggering

The main purpose of the -Xdump stanza on the command line is to link events to a
dump type (-Xdump:tool is a little misleading, because it is a command, not a
dump). Thus, -Xdump:heap:events=vmstop is an instruction to JVM initialization
to create a dump agent that produces a Heapdump whenever the vmstop event

214 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Dump types and triggering

happens. The JVM is constructed to generate at the appropriate time the events
listed in Chapter 23, “JVM dump initiation,” on page 209.

You can have multiple -Xdump stanzas on the command line and also multiple
dump types driven by one or multiple events. Thus,
-Xdump:heap+java:events=vmstart+vmstop would create a dump agent that
would drive both heap and Java dump production when either a vmstart or
vmstop event was encountered.

Note that multiple -Xdump stanzas on the command line can be used to create
multiple agents at JVM initialization; these agents are chained together and all
evaluated whenever an event occurs. The dump agent processing ensures that
multiple -Xdump stanzas are optimized. You can use the -Xdump:what stanza to
clarify this optimization.

The keyword events is used as the prime trigger mechanism. However, there are a
number of additional keywords that you can use to further control the dump
produced (request and tokens, for example) or limit its production to a smaller
range of circumstances; use -Xdump<type>:help to find these.

Types of dump agents - examples

This section presents several examples of the use of -Xdump, based around each
dump type, to illustrate the style of syntax and the generated function. The
examples given are deliberately simplistic to limit the size of the output.

As you can see from using -Xdump:help, there are five dump types to consider.

Console dumps
Console dumps are simple dumps, in which the status of every Java thread is
written to stderr. Some output of this type is shown below. Note the use of the
range=1..1 suboption to control the amount of output to just one thread start and
stop (in this case, the start of the Signal Dispatcher thread).
java -Xdump:console:events=thrstart+thrstop,range=1..1

JVMDUMP006I Processing Dump Event "thrstart", detail "" - Please Wait.

-------- Console dump --------
Stack Traces of Threads:

ThreadName=Signal Dispatcher(00035B24)
Status=Running

ThreadName=main(00035A1C)
Status=Waiting
Monitor=00035128 (VM sig quit)
Count=0
Owner=(00000000)
^^^^^^^^ Console dump ^^^^^^^^

System dumps
System dumps involve dumping a whole frozen address space and as such are
generally very large. The bigger the footprint of an application the bigger its
dump. A dump of a major server-based application might take up many gigabytes
of file space and take several minutes to complete. Shown below is an example of
invoking a system dump on a Windows 32-bit machine. Note the use of
request=nodumps+exclusive+prepwalk in this example, to ensure that this dump
is not interrupted by other dumps and that the Java heap is walkable, enabling the

Chapter 24. Using dump agents 215

Types of dump agents - examples

objects within the heap to be processed under jextract or jdmpview (the equivalent
of -Xdump:heapdump in the previous release). Note also that the file name is
overridden from the default in this example.
java -Xdump:system:events=vmstop,request=nodumps+exclusive+prepwalk,file=my.dmp

::::::::: removed usage info :::::::::

JVMDUMP006I Processing Dump Event "vmstop", detail "#00000000" - Please Wait.

JVMDUMP007I JVM Requesting System Dump using ’C:\sdk142\sdk\jre\bin\my.dmp’
JVMDUMP010I System Dump written to C:\sdk142\sdk\jre\bin\my.dmp
JVMDUMP013I Processed Dump Event "vmstop", detail "#00000000".

Tool option
The tool option allows external processes to be spawned when an event occurs.
Consider the following simple example, which displays start of pid and end of pid
information (note the use of the token %pid). More realistic examples would
invoke a debugging tool, and that is the default taken if you use (for example)
-Xdump:tool:events=.....
java -Xdump:tool:events=vmstop,exec="cmd /c echo %pid has finished"
-Xdump:tool:events=vmstart,exec="cmd

/c echo %pid has started"

JVMDUMP006I Processing Dump Event "vmstart", detail "" - Please Wait.
JVMDUMP007I JVM Requesting Tool Dump using ’cmd /c echo 2184 has started’
JVMDUMP011I Tool Dump spawned process 2160
2184 has started
JVMDUMP013I Processed Dump Event "vmstart", detail "".

::::::::: removed usage info :::::::::

JVMDUMP006I Processing Dump Event "vmstop", detail "#00000000" - Please Wait.

JVMDUMP007I JVM Requesting Tool Dump using ’cmd /c echo 2184 has finished’
JVMDUMP011I Tool Dump spawned process 2204
2184 has finished
JVMDUMP013I Processed Dump Event "vmstop", detail "#00000000".

Javadumps
Java dumps are an internally generated and formatted analysis of the JVM, giving
information that includes the Java threads present, the classes loaded, and heap
statistics. An example (which also shows the use of the filter keyword) in which a
Javadump is produced on the loading of a class is shown below.
java -Xdump:java:events=load,filter=*ZipC

JVMDUMP006I Processing Dump Event "load", detail "java/util/zip/ZipConstants"

- Please Wait.
JVMDUMP007I JVM Requesting Java Dump using ’C:\sdk142\sdk\jre\bin\
javacore.20050323.114159.3728.txt’
JVMDUMP010I Java Dump written to C:\sdk142\sdk\jre\bin\
javacore.20050323.114159.3728.txt
JVMDUMP013I Processed Dump Event "load", detail "java/util/zip/ZipConstants".

Heapdumps
From Version 1.4.2, Service Refresh 2, Heapdumps now produce phd format files
by default (as described in Chapter 22, “Using Heapdump,” on page 205) unless
overridden. The example below shows the production of a Heapdump. Note that
in this case the normal production of a phd file only has been augmented by the
use of the opts= suboption to produce both phd and classic (.txt) heapdumps
(equivalent to using the environment variable IBM_JAVA_HEAPDUMP_TEST).

216 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Types of dump agents - examples

java -Xdump:none -Xdump:heap:events=vmstop,opts=PHD+CLASSIC

JVMDUMP006I Processing Dump Event "vmstop", detail "#00000000" - Please Wait.

JVMDUMP007I JVM Requesting Heap Dump using ’C:\sdk\jre\bin\
heapdump.20050323.142011.3272.phd’
JVMDUMP010I Heap Dump written to C:\sdk\jre\bin\heapdump.20050323.142011.3272.phd
JVMDUMP007I JVM Requesting Heap Dump using ’C:\sdk\jre\bin\
heapdump.20050323.142011.3272.txt’
JVMDUMP010I Heap Dump written to C:\sdk\jre\bin\
heapdump.20050323.142011.3272.txt
JVMDUMP013I Processed Dump Event "vmstop", detail "#00000000".

Default dump agents

The JVM adds a set of dump agents by default during its initialization. You can
override this set of dump agents using the JAVA_DUMP_OPTS environment
variable and further override the set by the use of -Xdump on the command line.
The -Xdump:what option on the command line is very useful for determining
which dump agents exist at the completion of startup and can help resolve issues
about what has overridden what. Below is sample output showing the default
dump agents that are in place when there have been no overrides by using
environment variables.
java -Xdump:what

Registered dump agents

----------------------
dumpFn=doSystemDump
events=gpf+abort
filter=
label=C:\sdk142\sdk\jre\bin\core.%Y%m%d.%H%M%S.%pid.dmp
range=1..0
priority=999
request=nodumps
----------------------
dumpFn=doHeapDump
events=uncaught
filter=java/lang/OutOfMemoryError
label=C:\sdk142\sdk\jre\bin\heapdump.%Y%m%d.%H%M%S.%pid.phd
range=1..4
priority=40
request=exclusive+prepwalk
----------------------
dumpFn=doJavaDump
events=gpf+user+abort
filter=java/lang/OutOfMemoryError
label=C:\sdk142\sdk\jre\bin\javacore.%Y%m%d.%H%M%S.%pid.txt
range=1..0
priority=10
request=exclusive
----------------------
dumpFn=doJavaDump
events=uncaught
filter=java/lang/OutOfMemoryError
label=C:\sdk142\sdk\jre\bin\javacore.%Y%m%d.%H%M%S.%pid.txt
range=1..4
priority=10
request=exclusive
----------------------

Chapter 24. Using dump agents 217

Default settings for dumps

Default settings for dumps

To view the default settings for a particular dump type, use:

-Xdump:<type>:defaults

You can change these defaults at runtime. For example, to guarantee unique files
you would use:

-Xdump:java:defaults:file=dumps/%pid/javacore-%seq.txt

Note that this option does not add a javadump agent; it updates the default
settings for dump agents. Further dump agents will then create dump files using
this specification for filenames, unless overridden.

Limiting dumps using filters

Some JVM events occur thousands of times during the lifetime of an application.
Dump agents can use filters and ranges to avoid excessive dumps being produced.

You can filter class events (such as load, throw, and uncaught) by class name:

-Xdump:java:events=throw,filter=java/lang/OutOfMem # prefix
-Xdump:java:events=throw,filter=java/lang/*Memory # substring

You can filter the JVM shutdown event by using one or more exit codes:

-Xdump:java:events=vmstop,filter=#129..192#-42#255

You can start and stop dump agents on a particular occurrence of a JVM event by
using the range suboption:

-Xdump:java:events=fullgc,range=100..200

Note that range=1..0 against an event means ″on every occurrence″.

Removing dump agents

You can remove all default dump agents and any preceding dump options by
using:

-Xdump:none

Use this option so that you can subsequently specify a completely new dump
configuration.

You can also remove dump agents of a particular type. For example,

-Xdump:java+heap:events=vmstop -Xdump:heap:none

turns off all heapdumps (including default agents) but leaves javadump enabled.

218 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 25. Using method trace
Method trace is a powerful and free tool that allows you to trace methods in any
Java code. You do not have to add any hooks or calls to existing code. Run the
JVM with method trace turned on and watch the data that is returned. Using
method trace provides a complete (and potentially large) diagnosis of code paths
inside your application and also inside the system classes. Use wild cards and
filtering to control method trace so that you can focus on the sections of code that
interest you.

Method trace can trace:

v Method entry
v Method exit

Use method trace to debug and trace application code and the system classes
provided with the JVM.

Method trace is part of the larger ’JVM trace’ package. JVM trace is described in
Chapter 30, “Tracing Java applications and the JVM,” on page 259.

This chapter describes the basic use of trace. Use this chapter to learn the basic use
of trace. When you feel comfortable using trace, see Chapter 30, “Tracing Java
applications and the JVM,” on page 259 for more detailed information.

Running with method trace

Control method trace by using the command-line option -Xtrace:<option>.

If you want method trace to be formatted, set two trace options:

v -Xtrace:print — set this option to ’mt’ to invoke method trace.
v -Xtrace:methods — set this option to decide what to trace.
The first property is only a constant: -Xtrace:print=mt

Use the methods parameter to control what is traced. To trace everything, set it to
methods=*.*. This is not recommended because you are certain to be overwhelmed
by the amount of output.

The methods parameter is formally defined as follows:

-Xtrace:methods=[[!]method_spec[,...]]

Where method_spec is formally defined as:

{*|[*]classname[*]}.{*|[*]methodname[*]}[()]

Note that
v The delimiter between parts of the package name is a forward slash, ’/’, even on
platforms like Windows that use a backward slash as a path delimiter.
v The ″!″ in the methods parameter is a NOT operator that allows you to tell the
JVM not to trace the specified method or methods. Use this with other methods
parameters to set up a trace of the form: ″trace methods of this type but not
methods of that type″.

© Copyright IBM Corp. 2003, 2006 219

Running with method trace

v The parentheses, (), that are in the method_spec define whether or not to trace
method parameters.

Examples of use
v Tracing entry and exit of all methods in a given class:
-Xtrace:methods=ReaderMain.*
methods=java/lang/String.*
v Tracing entry, exit and input parameters of all methods in a class:
-Xtrace:methods=ReaderMain.*()
v Tracing all methods in a given package:
-Xtrace:methods=com/ibm/socket/*.*()
v Multiple method trace:
-Xtrace:methods=Widget.*(),common/Gauge.*

This traces all method entry, exit, and parameters in the Widget class and all
method entry and exit in the Gauge package.
v Using the ! operator
-Xtrace:methods=ArticleUI.*,!ArticleUI.get*

This traces all methods in the class ArticleUI except those beginning with “get”.

Where does the output appear?

In this simple case, output appears on the ’stderr’. If you want to store your
output, redirect this stream to a file. You can also write method trace to a file
directly, as described in “Advanced options.”

Advanced options
The use of method trace described above forces a formatted version of the output,
however, it can be rather slow. To work around this, you can make the method
trace output appear in a compressed binary form and thus minimize its impact on
performance. You can then redirect this form of the output to an output file, rather
than only to the console as in the description above.

You use a tool, supplied with the IBM Virtual Machine for Java, to analyze and
dump the output binary file. You can even route trace to your own plug-in agent
and process it at will (see Chapter 30, “Tracing Java applications and the JVM,” on
page 259).

Real example
java-Xtracemethods=ReaderMain.*(),ConferenceUI.*() -Dibm.dg. trc.print=mt ReaderMain

Results:
java -Xtrace:methods=java/lang*.*,iprint=mt HW
15:33:44.786*0xa0900 040001 > java/lang/Class.initialize()V Compiled method, This = 0x1552b4
15:33:44.796 0xa0900 040001 > java/lang/Class.verify()V Compiled method, This = 0x15528c
15:33:44.796 0xa0900 040001 > java/lang/Class.verify()V Compiled method, This = 0x155260
15:33:44.796 0xa0900 040001 > java/lang/Class.setInitStatus(I)V Compiled method, This = 0x155234
15:33:44.796 0xa0900 040007 < java/lang/Class.setInitStatus(I)V Compiled method
15:33:44.796 0xa0900 040007 < java/lang/Class.verify()V Compiled method
15:33:44.796 0xa0900 040001 > java/lang/Class.setInitStatus(I)V Compiled method, This = 0x155260
15:33:44.796 0xa0900 040007 < java/lang/Class.setInitStatus(I)V Compiled method
15:33:44.796 0xa0900 040007 < java/lang/Class.verify()V Compiled method

220 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Method trace - examples of use

15:33:44.796 0xa0900 040001 > java/lang/Class.initialize()V Compiled method, This = 0x15528c

15:33:44.796 0xa0900 040001 > java/lang/Class.setInitStatus(I)V Compiled method, This = 0x155264
15:33:44.796 0xa0900 040007 < java/lang/Class.setInitStatus(I)V Compiled method
15:33:44.796 0xa0900 040007 < java/lang/Class.initialize()V Compiled method
15:33:44.796 0xa0900 040004 > java/lang/String.<clinit>()V Compiled static method
15:33:44.796 0xa0900 040001 > java/lang/Class.initialize()V Compiled method, This = 0x155224
15:33:44.796 0xa0900 040001 > java/lang/Class.verify()V Compiled method, This = 0x1551fc
15:33:44.796 0xa0900 040001 > java/lang/Class.verify()V Compiled method, This = 0x1551d0
15:33:44.796 0xa0900 040007 < java/lang/Class.verify()V Compiled method
15:33:44.796 0xa0900 040001 > java/lang/Class.setInitStatus(I)V Compiled method, This = 0x1551d0
15:33:44.796 0xa0900 040007 < java/lang/Class.setInitStatus(I)V Compiled method
15:33:44.796 0xa0900 040007 < java/lang/Class.verify()V Compiled method
15:33:44.796 0xa0900 040001 > java/lang/Class.initialize()V Compiled method, This = 0x1551fc
15:33:44.796 0xa0900 040007 < java/lang/Class.initialize()V Compiled method
15:33:44.796 0xa0900 040001 > java/lang/Class.setInitStatus(I)V Compiled method, This = 0x1551fc
15:33:44.796 0xa0900 040007 < java/lang/Class.setInitStatus(I)V Compiled method
15:33:44.796 0xa0900 040007 < java/lang/Class.initialize()V Compiled method
15:33:44.796 0xa0900 040001 > java/lang/String$CaseInsensitiveComparator.<init>
(Ljava/lang/String$1;)V Compiled method, This = 0x15524c
15:33:44.796 0xa0900 040001 > java/lang/Object.<init>()V Compiled method, This = 0x155238
15:33:44.796 0xa0900 040007 < java/lang/Object.<init>()V Compiled method
15:33:44.796 0xa0900 040007 < java/lang/String$CaseInsensitiveComparator.<init>
(Ljava/lang/String$1;)V Compiled method
15:33:44.796 0xa0900 04000A < java/lang/String.<clinit>()V Compiled static method
15:33:44.796 0xa0900 040001 > java/lang/Class.setInitStatus(I)V Compiled method, This = 0x15528c
15:33:44.796 0xa0900 040007 < java/lang/Class.setInitStatus(I)V Compiled method
15:33:44.796 0xa0900 040007 < java/lang/Class.initialize()V Compiled method
15:33:44.796 0xa0900 040001 > java/lang/Class.initialize()V Compiled method, This = 0x1552b4
15:33:44.796 0xa0900 040001 > java/lang/Class.verify()V Compiled method, This = 0x15528c
15:33:44.796 0xa0900 040001 > java/lang/Class.verify()V Compiled method, This = 0x155260
15:33:44.796 0xa0900 040007 < java/lang/Class.verify()V Compiled method
15:33:44.796 0xa0900 040001 > java/lang/Class.setInitStatus(I)V Compiled method, This = 0x155260
15:33:44.796 0xa0900 040007 < java/lang/Class.setInitStatus(I)V Compiled method
15:33:44.796 0xa0900 040007 < java/lang/Class.verify()V Compiled method
15:33:44.796 0xa0900 040001 > java/lang/Class.initialize()V Compiled method, This = 0x15528c
15:33:44.796 0xa0900 040007 < java/lang/Class.initialize()V Compiled method
15:33:44.796 0xa0900 040001 > java/lang/Class.setInitStatus(I)V Compiled method, This = 0x15528c
15:33:44.806 0xa0900 040007 < java/lang/Class.setInitStatus(I)V Compiled method
15:33:44.806 0xa0900 040007 < java/lang/Class.initialize()V Compiled method
15:33:44.806 0xa0900 040001 > java/lang/Class.initialize()V Compiled method, This = 0x1552b4
15:33:44.806 0xa0900 040001 > java/lang/Class.verify()V Compiled method, This = 0x15528c
15:33:44.806 0xa0900 040001 > java/lang/Class.verify()V Compiled method, This = 0x155260
15:33:44.806 0xa0900 040007 < java/lang/Class.verify()V Compiled method
15:33:44.806 0xa0900 040001 > java/lang/Class.setInitStatus(I)V Compiled method, This = 0x155260
15:33:44.806 0xa0900 040007 < java/lang/Class.setInitStatus(I)V Compiled method
15:33:44.806 0xa0900 040007 < java/lang/Class.verify()V Compiled method
15:33:44.806 0xa0900 040001 > java/lang/Class.initialize()V Compiled method, This = 0x15528c

The remaining lines comprise:

v 0xa0900, an internal JVM trace point used by some advanced diagnostics.
v 040001, the current execenv (execution environment). This data is fundamental
because every JVM thread has its own execenv. Hence, you can regard execenv
as a thread-id. All trace with the same execenv relates to a single thread.
v The remaining fields show whether a method is being entered (>) or left (<),
followed by details of the method.

Chapter 25. Using method trace 221

Method trace - examples of use

222 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 26. Using the dump formatter
v “What the dump formatter is”
v “Problems to tackle with the dump formatter” on page 224
v “Supported commands” on page 224
v “Example session” on page 229

What the dump formatter is

You can run the dump formatter on one platform from another platform and you
can use it to look at a dump that is taken on from any platform. For example, you
can look at z/OS dumps on a Windows platform.

The dump formatter consists of:

jextract
When a dump is available, you can invoke the jextract utility. jextract produces
an xml file that you can use together with the original dump to diagnose
problems. jextract is located in the directory sdk\jre\bin. To invoke jextract, at
a command prompt type:
jextract dumpfilename

jextract sends its output to a file called dumpfilename.xml. This file contains
details of useful JVM internal information. Note that the syntax of the xml is
subject to change so you should not design tools based on the contents of this
xml file. Preferably, you should run jextract on the same system as the one on
which the dump was produced. However, you can run jextract on a system
that has the same version of the JRE as the system on which the dump was
produced.
jdmpview
jdmpview is a launcher for the main method of the java class J9JVMConsole
that is contained in the sdk/jre/lib/ext/jdmpview.jar. To invoke jdmpview,
from a command prompt type:
jdmpview [-ddumpfilename] [-wworkdir] [-ooutput]

where
v dumpfilename is a dumpfile
v workdir is a writable directory
v output is an output file (typical format file:x:\myfile)
Typical usage is jdmpview my.dmp. The J9JVMConsole class opens and verifies
the my.dmp file (which it recognizes as a dump file) and the associated xml
file (my.dmp.xml).
After jdmpview processes the arguments with which it was launched, it
displays the message Ready.... This means that you can start invoking
commands on jdmpview. There is no limit on the number of jdmpview
sessions that you can run at the same time.
You can significantly improve the performance of jdmpview against larger
dumps by ensuring that there is enough memory available on your system to
avoid paging. On larger dumps (that is, ones with large numbers of objects on

© Copyright IBM Corp. 2003, 2006 223

What the dump formatter is

the heap) you might have to invoke jdmpview using the -Xmx option to
increase the maximum heap available to jdmpview:
jdmpview -J-Xmx<n>

For more information on using -Xmx, see Appendix F, “Command-line

options,” on page 329.

Problems to tackle with the dump formatter

Dumps including a JVM can arise either with the JVM in control (that is, when
you specify the -Xdump option on the command-line), from handled events (such
as an OutOfMemory exception), or when the JVM is not in control (such as
user-initiated dumps). The extent to which jextract can analyze the information in a
dump is affected by the kind of dump. You will get the most complete information
from heapdumps that were initiated under the control of a JVM started with the
option -Xdump:heap:events=. jextract works with other types of dumps but the
produced xml file might contain less information – in particular about the object
heaps within the JVM.

jdmpview is most useful in diagnosing customer-type problems and problems with

the J2SE class libraries. Typical scenarios are OutOfMemory errors in customer
applications.

For problems involving gprs, ABENDS, SIGSEVs, and similar problems, you can
get more information by using the system debugger (windbg, gdb) along with the
dump file. However, jdmpview can still provide useful information in conjunction
with the system debugger.

Supported commands
This section describes the commands available in jdmpview. Many of the
commands have short forms. For example, display, dis, and d are all considered
equivalent in the standard command syntax. The commands are split into common
subareas.

General commands
v Quit
Short form: q
Availability: always
Terminates the jdmpview session.
v cmds
Short form:
Availability: always
Displays the available commands at any point during the jdmpview session and
also indicates which class provides the support for that command. Note that the
range of available commands might change during the session; for example, the
DIS OS command is not available until after a dump has been identified.
v help and help <command>
Short form: h
Availability: always

224 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Supported commands

The help command (with no parameters) shows general help. With a parameter
it displays specific help on a command. For example, help dis os would
produce help information regarding the help dis os command.
v set
Short form: s
Availability: always
Some set commands (such as set dump) initiate specific processing within
jdmpview while others set and unset variables within the jdmpview
environment. The variations of set are covered below.
set without any parameters shows what jdmpview variables are defined and
what their values are. Similarly, set param shows the value of param. The
generic command set param=value sets up a key/value pair associating the value
with the key param. Parameters can be used for remembering discovered values
for later use.
v set dump
Short form: s du
Availability: always
Opens the specified dump. The syntax is:
set dump[=]<dumpname>

After the set dump command has executed successfully, several additional
commands (such as dis mem and dis mmap) become available. When set dump
has successfully run (i.e. it was a valid file and it was a dump), another use of
set dump does nothing. If you want to analyze another dump, you must start a
new jdmpview session.
v set metadata
Short form: s meta
Availability: after successful ″set dump″
Initiates the reading of the xml file produced by jextract, causes the xml file to
be parsed, and assorted details about the underlying nature of the dump stored
for use by other commands (such as dis os or dis cls). The syntax is
set metadata[=]<filename>

After set metadata has successfully run, subsequent uses of it will do nothing.
v set workdir
Short form:
Availability: always
Identifies a location to which jdmpview can write data. Some commands (such
as dis os or trace extract) create files as part of their function – usually these
files are created in the same location as the dumpfile; however sometimes it
might be convenient to keep the dumpfile (and the xml) in a read-only location.
Its syntax is:
set workdir[=]<location>

You can also use the –w option when launching jdmpview to set the working
directory.
v set output
Short form: s out
Availability: always

Chapter 26. Using the dump formatter 225

Supported commands

Redirects the output from jdmpview to a file rather than to the console
(System.out). Use it when large amounts of output are expected to be produced
from a command (for example, dis mem 10000,100000). Its syntax is:
set output[=]<location>

where <location> is either * (System.out) or file:filename (for example,

file:c:\myfile.out).
v add output
Short form:
Availability: always
Directs the output from a command to more than one location. Its syntax is:
add output[=]<location>

where <location> is either * (System.out) or file:filename (for example,

file:c:\myfile.out).

The following commands show details about the dump.

v dis t
Short form:
Availability: after set metadata has run
Gives information about threads within the dumped process. dis t * gives
information about all the known threads. dis t (with no parameters) just gives
information about the current thread.
v dis sys
Short form:
Availability: after set metadata has run
Gives information about the dump and the JVM.

Commands for analysing the memory

The major content of any dump is the image of memory associated with the
process that was dumped. Use the following commands to display and investigate
the memory. These commands do not function until after the set dump command
has been successfully issued.
v dis mmap
Short form:
Availability: after set dump has run
When a dump is opened (set dump), jdmpview establishes a mapping of virtual
memory ranges held within the dump to their location in the dump file. In
doing this, it creates an internal map which is then used by the rest of the
commands to access memory within the dump. Dis mmap allows this mapping to
be displayed and allow the user to see what valid memory ranges are contained
within the dump and their offsets within the dump file.
On z/OS memory ranges are also associated with an address space id (asid) and
dis mmap, besides showing the ranges and their offsets, also shows the asid to
which the memory range belongs. You need to be aware that areas of memory
that appear contiguous (or even overlap) according to the memory map will
almost certainly not be contiguous and will have different asids.
v dis mem
Short form:
Availability: after set dump has run

226 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Supported commands

Displays memory within the dump. The syntax is:

dis mem <address>[,<numbytes>]

where <address> is the hex address to display (it can be preceded by 0x) and
<numbytes> (defaults to 256) is the number of bytes to display.
v find
Short form: fn
Availability: after set dump has run
Looks for strings and hex values within the dump memory. The syntax is
Find pattern[,<start>][,<end>][,<boundary>][,<count>][,<limit>]

The <start> parameter controls where to start the search from, <end> where to
end it , <boundary> what byte boundary should be used, <count>how many
bytes of memory should be displayed when a hit is encountered, and <limit>
the limit of occurrences to display.

Commands for working with classes

Use the following commands to work with classes:
v dis cls and dis cls <classname>
Short form:
Availability: after set dump and set metadata have run
Without a classname specified, it produces a list of all the known classes together
with their instance size and (if dis os has run) a count of the instances
associated with that class. For array classes the instance size is always 0.
When <classname> is specified (and if dis os has run) the addresses of all the
object instances of that particular class are displayed. For classes such as [char,
where the ’[’ indicates that this is an array class, the number of instances can run
into many thousands!
v display class <classname>
Short form: dis cl <classname>
Availability: after set dump and set metadata have run
Displays information on the composition of the specified class. It displays the
methods, fields, and statics associated with the class along with other
information. DO NOT confuse it with dis cls <classname>.

Commands for working with objects

The following commands allow you to observe and analyze the objects that existed
when the dump was taken.
v dis os
Short form:
Availability: after set dump and set metadata have run
Scans the known heap segments (as identified in the incoming xml metadata)
and creates (if necessary) a ″jfod″ file with information about the object instances
found during the scan. It also creates some internal bitmaps that are linked to
each heap segment and that indicate the address points in each heap segment
that are the starting points of objects.
The output from dis os is an object summary that identifies all the classes and
gives a count of the number of object instances found and the byte count
associated with those instances.

Chapter 26. Using the dump formatter 227

Supported commands

Note: You can only run dis os once.

v dis obj address and dis obj classname
Short form:
Availability: after set dump, set metadata, and dis os have run
When you specify an <address>, it displays details about the object at that
address. When you specify a <classname>, it displays details about all objects of
that class. Use the second form with caution because if there are many instances
of the specified class, the output can be large (although you can direct it to an
output file for analysis using a file editor).
The output from dis os is an object summary that identifies all the classes and
gives a count of the number of object instances found and the byte count
associated with those instances. The information displayed about an object is
produced along with the stored details for its class

Commands for working with Heapdumps

Use the following commands to work with Heapdumps.
v set heapdump and set heapdump filename
Short form:
Availability: after successful dis os
Without a parameter, it displays the name of the file that was created by the hd
f command. When you specify the filename parameter (for example, set
heapdump c:\my.hd), the name of the file created by hd f is set to the filename
you specified. If filename ends in ″.gz″, the output is produced in gzip
compressed format.
The default value for the heapdump filename is dumpfilename.phd.gz. For
example, if the dump file name (as input to the set dump command) is
xyz.20041234.dmp, the default Heapdump output filename is
xyz.20041234.dmp.phd.gz.
v set heapdumpformat
Short form:
Availability: after successful dis os
Sets the format of the output produced. The two settings are classic and
portable. The classic option results in a readable text output file. The portable
option (the default) produces output in a compressed binary format, known as
phd.
v set hd_host and set hd_port
Short form:
Availability: after successful dis os
These two commands control the network host and port that are used for the hd
n command. The default settings for host and port are localhost and 21179
respectively.
v hd f
Short form:
Availability: after successful dis os
Generates heapdump output to a file. The location and format of the data
produced are controlled by using the set heapdump and set heapdumpformat
commands respectively.
v hd n
Short form:

228 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Supported commands

Availability: after successful dis os

Generates heapdump output to a network host. You must ensure that you have
a receiver running on the host and port specified in the HD_HOST and
HD_PORT options respectively. You must also ensure that any firewall software
is correctly set up to allow the connection between your machine and the host to
succeed.

Commands for working with trace

Use the following commands to work with trace.
v trace extract
Short form:
Availability: after successful dis os and set metadata
Uses the information in the metadata to extract the trace buffers from the dump
and write them to a file (called extracted.trc). If no buffers are present in the
dump, it displays an error message. The extracted buffers are available for
formatting by using the trace format command.
v trace format
Short form:
Availability: after successful dis os and set metadata
Formats the extracted trace buffers so that they can be viewed using the trace
display commands. If a trace extract has not been issued previously, it is
automatically issued by trace format.
v trace display
Short form:
Availability: after successful dis os and set metadata
Displays the trace output from the trace format command. It displays one page
at a time (you can control the page size using the page display=<size>
command) and allows scrolling throughout the file using the trace display +
and trace display – commands.

Example session
This example session is meant to illustrate some of the commands available and
their use. In the example session below some lines have been removed for clarity
(and shortness). Some comments (contained within braces) are included to explain
various aspects together with some comments on individual lines looking like
<< comment

User input is in bold italic

{first, invoke jdmpview with the name of a dump }

jdmpview c:\dl2.dmp

Command Console: " J9 Dump Analysis " << title

Please wait while I process inbound arguments

SET DUMP c:\dl2.dmp << command launched on basis of inbound argument

Recognised as a 32-bit little-endian windows dump. << dump exists and is supported
Trying to use "c:\dl2.dmp.xml" as metadata.....
Issuing "SET METADATA c:\dl2.dmp.xml" ..... << work with the xml
Parsing of xml started for file dl2.dmp.xml... be patient
Parsing ended

Chapter 26. Using the dump formatter 229

Example session

Ready....(’h’ shows help, ’cmds’ shows available commands) << jdmpview is ready
to accept user input

{ the output produced by h (or help) is illustrated below –

"help <command_name>" gives information on a specific command }

h
General Help
===============
To see what commands are available use the "cmds" command.
Note: The available command set can change as a result of some actions
- such as "set dump" or "set metadata".

The general form of a command is NOUN VERB PARM1 [,PARM2] ... [PARMn]
Note: some commands dont need a verb or parameters. The command parser
strips "=" characters and brackets from the input - this allows
alternative command formats like "set dump=c:\mydump.dmp" to work.

Use "help command" to obtain more help on a specific command

Ready....

help set dump

This command is usually one of the first commands entered. It requires a file
name as a parameter. The file identified (presuming it exists) is verified to
be a dump, its type established and the dump analysed to establish a memory map
(see "dis mmap" for more details).

Note: as an alternative to using set dump then starting jdmpview with a

parameter of "-ddumpname" (note no space between the -d and
filename) or with just the filename will open the dump before
the first "Ready...." appears.

As part of the processing when "set dump" is issued then if an xml file (as
produced out of jextract)is found matching the dump then a "set metadata"
command will be issued.

Ready....

{ The next command illustrated is "cmds" – this shows the syntax of the
currently recognised commands}

cmds

Known Commands
==============

SET DUMP (Identifies the dump to work with)

SET METADATA (Identifies xml metadata file - rarely needed))
QUIT (Terminates jdmpview session)
HELP * (Provides generic and specific help)
CMDS * (Shows available commands)
SYNONYMS * (Shows substitutions and short forms for commands)
SET OUT (Sets the output destination (terminal or file))
ADD OUT (Appends additional output destination)
REM OUT (Removes an output destination)
DIS OUT (Displays output destinations)
SET WORKDIR (Defines a writeable location for work files)
SET * (Generic command - use "help set" for more details)
EX * (Execute commands from a file)

"help command" shows details of each command

230 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Example session

Note: some supported commands may not be shown in the above

list as they only become available after successful issuance
of other commands (such as "set dump" or "dis os")

Ready....

{ The next command "dis os" is covered below – this command scans
the heap segments that were identified in the xml and produces a
names index file (.jfod) to allow subsequent
anlaysis of objects. For large dumps with several millions of
objects then this command could take a long time.
Following "dis os" is an alternative form of dis os where a
class name is specified. This allows the
addresses of all objects of a specific type to be displayed –
which could then be used as input to "dis obj" }

dis os

Names index file in use is: c:\dl2.dmp.jfod

Heap Summary
============

WARNING: It can take a long time to traverse the heaps!!!! - Please be patient
Starting scan of heap segment 0
start=0x13710000 end=0x1371070c object count= 45
Starting scan of heap segment 1
start=0x13710800 end=0x13712424 object count= 21

======== lines removed for terseness ========================

Starting scan of heap segment 23

start=0x137a7ab0 end=0x137a83c8 object count= 63
Starting scan of heap segment 24
start=0x137a8a2c end=0x137a8a7c object count= 2

Object Summary
Deadlock has 1 instances (total size= 72)
[java/lang/Object has 33 instances (total size= 1881)
[java/lang/Class has 10 instances (total size= 180)
[char has 2189 instances (total size= 199199)
[double has 10 instances (total size= 560)
[byte has 78 instances (total size= 257946)
[short has 3 instances (total size= 8784)
[int has 66 instances (total size= 8976)

================= lines removed for terseness ======================

sun/misc/Unsafe has 1 instances (total size= 12)

sun/net/www/MessageHeader has 1 instances (total size= 24)
sun/net/www/protocol/file/FileURLConnection has 1 instances (total size= 108)
sun/net/www/protocol/file/Handler has 1 instances (total size= 12)
sun/net/www/protocol/jar/Handler has 2 instances (total size= 24)
sun/nio/cs/StreamEncoder$ConverterSE has 4 instances (total size= 192)
sun/reflect/DelegatingMethodAccessorImpl has 1 instances (total size= 16)
sun/reflect/NativeMethodAccessorImpl has 1 instances (total size= 24)
sun/reflect/ReflectionFactory has 1 instances (total size= 12)
sun/reflect/ReflectionFactory$1 has 1 instances (total size= 12)

sun/security/action/GetPropertyAction has 28 instances (total size= 560)

sun/security/action/LoadLibraryAction has 3 instances (total size= 48)

Chapter 26. Using the dump formatter 231

Example session

Total number of objects = 5696

Total size of objects = 577320 bytes

Total locked objects = 4

Ready....

dis os java/lang/String
There are 2046 instances of java/lang/String ....

0x137a8a2c 0x137a83a0 0x137a8344 . . . . .

0x137a80f8 0x137a8080 . . . . .
0x137a7dbc 0x137a7d74 0x137a7d48 . . . . .

0x13712944 0x137128ec 0x13712898 . . . . .

0x13712600 0x137125c8 0x13712588 . . . . .
0x137113c0 0x13711360 . . . . .
0x13711088 0x13711050 0x13711010 . . . . .
0x13710e1c 0x13710ddc 0x13710d94 . . . . .
0x13710b38 0x13710ae0 0x13710a90 . . . . .

Ready....

{ "dis mmap" is used to show what memory ranges are available in the dump.}

dis mmap

Memory Map
==========
Addr: 0x00010000 Size: 4096 File Offset: 0x4054 (16468)
Addr: 0x00020000 Size: 4096 File Offset: 0x5054 (20564)
Addr: 0x00030000 Size: 49152 File Offset: 0x6054 (24660)
Addr: 0x0007d000 Size: 16384 File Offset: 0x12054 (73812)
Addr: 0x00090000 Size: 1073152 File Offset: 0x16054 (90196)
================= lines removed for terseness ======================
Addr: 0x13710000 Size: 4194304 File Offset: 0xdff054 (14676052)
================= lines removed for terseness ======================
Addr: 0x629c0000 Size: 32768 File Offset: 0x11ff054 (18870356)
Addr: 0x71aa0000 Size: 32768 File Offset: 0x1207054 (18903124)

Addr: 0x7ffd8000 Size: 16384 File Offset: 0x175e054 (24502356)

Addr: 0x7ffdd000 Size: 16384 File Offset: 0x1762054 (24518740)

Ready....

{ dis mem <address> is used to show memory contents, and +/- allows movement
forwards and backwards from that memory position}

dis mem 10000

00010000: 3D003A00 3A003D00 3A003A00 5C000000 | =.:.:.=.:.:.\... |
00010010: 3D004300 3A003D00 43003A00 5C006A00 | =.C.:.=.C.:.\.j. |
00010020: 39007300 64006B00 5C006A00 72006500 | 9.s.d.k.\.j.r.e. |
00010030: 5C006200 69006E00 00003D00 45007800 | \.b.i.n...=.E.x. |
00010040: 69007400 43006F00 64006500 3D003000 | i.t.C.o.d.e.=.0. |
00010050: 30003000 30003000 30003000 30000000 | 0.0.0.0.0.0.0... |
00010060: 41004C00 4C005500 53004500 52005300 | A.L.L.U.S.E.R.S. |
00010070: 50005200 4F004600 49004C00 45003D00 | P.R.O.F.I.L.E.=. |
00010080: 43003A00 5C004400 6F006300 75006D00 | C.:.\.D.o.c.u.m. |
00010090: 65006E00 74007300 20006100 6E006400 | e.n.t.s. .a.n.d. |
000100a0: 20005300 65007400 74006900 6E006700 | .S.e.t.t.i.n.g. |

232 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Example session

000100b0: 73005C00 41006C00 6C002000 55007300 | s.\.A.l.l. .U.s. |

000100c0: 65007200 73000000 41005000 50004400 | e.r.s...A.P.P.D. |
000100d0: 41005400 41003D00 43003A00 5C004400 | A.T.A.=.C.:.\.D. |
000100e0: 6F006300 75006D00 65006E00 74007300 | o.c.u.m.e.n.t.s. |
000100f0: 20006100 6E006400 20005300 65007400 | .a.n.d. .S.e.t. |

Ready....
+

00010100: 74006900 6E006700 73005C00 70006800 | t.i.n.g.s.\.p.h. |

00010110: 69006C00 72005C00 41007000 70006C00 | i.l.r.\.A.p.p.l. |
00010120: 69006300 61007400 69006F00 6E002000 | i.c.a.t.i.o.n. . |
00010130: 44006100 74006100 00004300 6F006D00 | D.a.t.a...C.o.m. |
00010140: 6D006F00 6E005000 72006F00 67007200 | m.o.n.P.r.o.g.r. |
00010150: 61006D00 46006900 6C006500 73003D00 | a.m.F.i.l.e.s.=. |
00010160: 43003A00 5C005000 72006F00 67007200 | C.:.\.P.r.o.g.r. |
00010170: 61006D00 20004600 69006C00 65007300 | a.m. .F.i.l.e.s. |
00010180: 5C004300 6F006D00 6D006F00 6E002000 | \.C.o.m.m.o.n. . |
00010190: 46006900 6C006500 73000000 43004F00 | F.i.l.e.s...C.O. |
000101a0: 4D005000 55005400 45005200 4E004100 | M.P.U.T.E.R.N.A. |
000101b0: 4D004500 3D005000 52004100 47004D00 | M.E.=.P.R.A.G.M. |
000101c0: 41005400 49005300 54000000 43006F00 | A.T.I.S.T...C.o. |
000101d0: 6D005300 70006500 63003D00 43003A00 | m.S.p.e.c.=.C.:. |
000101e0: 5C005700 49004E00 44004F00 57005300 | \.W.I.N.D.O.W.S. |
000101f0: 5C007300 79007300 74006500 6D003300 | \.s.y.s.t.e.m.3. |

Ready....

{ "find <text>" is used to find text in memory or (if the value starts with
0x then you can find byte sequences}

find java
Note: your search result limit was 1 ... there may be more results

00032441: 6A617661 002D2D58 64756D70 3A686561 | java.--Xdump:hea |

00032451: 703A6576 656E7473 3D766D73 746F7000 | p:events=vmstop. |
00032461: 44446561 646C6F63 6B006D6D 6F6E2004 | DDeadlock.mmon . |
00032471: 00090000 01080075 00730065 00720065 | .......u.s.e.r.e |
00032481: 006E0076 002E006C 006F0067 00000002 | .n.v...l.o.g.... |
00032491: 00040000 010800A0 F42C03B8 F42C0303 | .........’...’.. |
000324a1: 00020000 01080041 24030047 24030062 | .......A$..G$..b |
000324b1: 24030000 00000006 00030000 0108006A | $..............j |
000324c1: 61766120 2D586475 6D703A68 6561703A | ava -Xdump:heap: |
000324d1: 6576656E 74733D76 6D73746F 70204465 | events=vmstop De |
000324e1: 61646C6F 636B0009 00060000 010A0049 | adlock.........I |
000324f1: 424D5F4A 4156415F 434F4D4D 414E445F | BM_JAVA_COMMAND_ |
00032501: 4C494E45 3D6A6176 61202D58 64756D70 | LINE=java -Xdump |
00032511: 3A686561 703A6576 656E7473 3D766D73 | :heap:events=vms |
00032521: 746F7020 44656164 6C6F636B 004F4709 | top Deadlock.OG. |
00032531: 00090000 010A0049 424D5F4A 4156415F | .......IBM_JAVA_ |
Tip 1: Use FINDNEXT (FN) command to progress through them
Tip 2: Use "SET FINDMODE=V" to do automatic WHATIS

Find finished...

Ready....

{To display details of an object then the "dis obj" command is used, either
in the form shown below where only a class name is specified (there might be a
large number of different objects of this class) or in the form dis obj 0xhhhhhhh
for just one}

Chapter 26. Using the dump formatter 233

Example session

dis obj sun/net/www/protocol/file/FileURLConnection

sun/net/www/protocol/file/FileURLConnection@0x1379ece8

(20) fieldName: url sig: Ljava/net/URL; value= 0x13778df0

(24) fieldName: doInput sig: Z value= TRUE (0x1)
(28) fieldName: doOutput sig: Z value= FALSE (0x0)
(32) fieldName: allowUserInteraction sig: Z value= FALSE (0x0)
(36) fieldName: useCaches sig: Z value= TRUE (0x1)
(12) fieldName: ifModifiedSince sig: J value=0 (0x0)
(40) fieldName: connected sig: Z value= FALSE (0x0)
(44) fieldName: contentType sig: Ljava/lang/String;
(52) fieldName: contentLength sig: I value=-1 (0xffffffff)
(48) fieldName: properties sig: Lsun/net/www/MessageHeader; value= 0x1379ee30
(72) fieldName: contentType sig: Ljava/lang/String;
(76) fieldName: is sig: Ljava/io/InputStream; value= 0x0
(80) fieldName: file sig: Ljava/io/File; value= 0x1379e754
(84) fieldName: filename sig: Ljava/lang/String;
(96) fieldName: isDirectory sig: Z value= FALSE (0x0)
(100) fieldName: exists sig: Z value= FALSE (0x0)
(88) fieldName: files sig: Ljava/util/List; value= 0x0
(56) fieldName: length sig: J value=0 (0x0)
(64) fieldName: lastModified sig: J value=0 (0x0)
(104) fieldName: initializedHeaders sig: Z value= FALSE (0x0)
(92) fieldName: permission sig: Ljava/security/Permission; value= 0x1379f1fc

====================================================

Ready....

{To examine all the threads in the dump – "dis t *" is used. dis t can be used to
examine the current thread, and dis t 0cxhhhhh to examine a particular thread –
set t will show you the available threads}

dis t *
Info for thread - 0x9ff00
===============================
Name : main
Id : 0x9ff00
Obj : 0x1371036c (java/lang/Thread)
State : Blocked MonitorId: 0x35050
Stack:
methodId: 0x2ec19a8 pc: 0x2e99c65 arguments: 0x29830e4
===> Deadlock/main([Ljava/lang/String;)V

Info for thread - 0xa0700

===============================
Name : Signal Dispatcher
Id : 0xa0700
Obj : 0x1377c6a4 (java/lang/Thread)
State : Running
Stack:
methodId: 0x2b70f58 pc: 0x2ba41be arguments: 0x2b97818
===> com/ibm/misc/SignalDispatcher/waitForSignal()I
methodId: 0x2b70ec8 pc: 0x2b4e8c0 arguments: 0x2b9783c
===> com/ibm/misc/SignalDispatcher/run()V

Info for thread - 0x2ecb400

===============================
Name : Deadlock
Id : 0x2ecb400
Obj : 0x137a0c3c (java/lang/Thread)
State : Blocked MonitorId: 0x35018
Stack:

234 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Example session

methodId: 0x2ec1998 pc: 0x2e99b4c arguments: 0x2ecb2cc

===> Deadlock/run()V

Info for thread - 0x2ecb800

===============================
Name : SIGINT handler
Id : 0x2ecb800
Obj : 0x13783c2c (java/lang/Thread)
State : Running
Stack:
methodId: 0x2ecc634 pc: 0x7 arguments: 0x2af8700
===> java/lang/Shutdown/halt(I)V
methodId: 0x2ecc664 pc: 0x2e75346 arguments: 0x2af8720
===> java/lang/Shutdown/exit(I)V
methodId: 0x2b6e9dc pc: 0x2b48a5c arguments: 0x2af8734
===> java/lang/Terminator$1/handle(Lsun/misc/Signal;)V
methodId: 0x2e624fc pc: 0x2e74820 arguments: 0x2af8744
===> sun/misc/Signal$1/run()V
methodId: 0x2a4e4a4 pc: 0x2a32393 arguments: 0x2af8754
===> java/lang/Thread/run()V

Ready....

{If monitors (also known as locks) are of interest – then "dis ls" is used }

dis ls
Known Monitors....

Thread global
id= 0x342c0

NLS hash table

id= 0x342f8

MM_SublistPool
id= 0x34330

=================== Lines removed for clarity

Un-named Monitor@0x34fa8
id= 0x34fa8

Thread public flags mutex

id= 0x34fe0

&monitor id= 0x35018

object= 0x137a09fc

&monitor id= 0x35050

object= 0x137a0ac4

Thread public flags mutex

id= 0x35088

Locked Objects.....

java/lang/Object@0x137a09fc is Locked by thread with id 0x9ff00

java/lang/Integer@0x137a0ac4 is Locked by thread with id 0x2ecb400
java/lang/Long@0x137a0b80 is Locked by thread with id 0x2ecb400
java/lang/String@0x137a7d74 is Locked by thread with id 0x2ecb400

Chapter 26. Using the dump formatter 235

Example session

Finished lock summary

Ready....

{There is also a deadlock command which analyses the locks for deadlock
situations - see the example below. }

deadlock

Thread 0x9ff00 is participating in a deadlock with 1 other thread(s).

It is trying to obtain a lock on object 0x137a0ac4 using monitor 0x35050

The other participating thread(s):

0x2ecb400

Thread 0x9ff00 has locks on the following objects:

java/lang/Object@0x137a09fc
======================================================

Thread 0x2ecb400 is participating in a deadlock with 1 other thread(s).

It is trying to obtain a lock on object 0x137a09fc using monitor 0x35018

The other participating thread(s):

0x9ff00

Thread 0x2ecb400 has locks on the following objects:

java/lang/Integer@0x137a0ac4
java/lang/Long@0x137a0b80
java/lang/String@0x137a7d74

Ready....

{The heapdump (hd) command is used to produce output for the heapdump tool
(see Heapdump chapter. an example of the hd command is shown below, in this
case it is producing a file which contains details of all the objects found
in the dump and all their associated references).

hd f
Now dumping heapdump file c:\dl2.dmp.phd.gz

Writing Compressed Heapdump

Writing HeapDump in Portable Format

Heapdump complete, number of objects processed = 5696
Heapdump Successful

Ready....

{Finally the quit command is used to exit this jdmpview session

– shown below in its shortened form}

q
Quitting - bye

236 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 27. JIT problem determination
The Just-In-Time compiler (JIT) is tightly bound to the JVM, but is not part of it.
The JIT converts Java bytecodes, which are interpreted by the JVM at run time and
execute slowly, into native code, which is understood by the processor and
executes quickly.

Occasionally, valid bytecodes might compile into invalid native code, causing the
Java program to fail. By determining whether the JIT is faulty and, if so, where it is
faulty, you can provide valuable help to the Java service team.

This chapter describes how you can determine with reasonable certainty whether
your problem is JIT-related. This chapter also suggests some possible workarounds
and debugging techniques for solving JIT-related problems:
v “Disabling the JIT”
v “Selectively disabling the JIT”
v “Locating the failing method” on page 238
v “Identifying JIT compilation failures” on page 239
v “Performance of short-running applications” on page 240

Disabling the JIT

The JIT is enabled by default, but for efficiency reasons, not all methods in a Java
application are compiled. The JVM maintains a call count for each method in the
application; every time a method is called and interpreted, the call count for that
method is incremented. When the count reaches the JIT threshold, the method is
compiled and executed natively.

The call count mechanism spreads compilation of methods throughout the life of
an application, giving higher priority to methods that are used most frequently.
Some infrequently used methods might never be compiled at all. As a result, when
a Java program fails, the problem might be in the JIT, or it might be elsewhere in
the JVM. The first step in diagnosing the failure is to determine where the problem
is. To do this, you must first run your Java program in purely interpreted mode
(that is, with the JIT disabled): specify the -Xint option, and remove the -Xjit
option (and accompanying JIT parameters, if any) when you run the JVM. If the
failure still occurs, the problem is most likely in the JVM rather than the JIT. (Do
not use the -Xint and the -Xjit options together.)

Running the Java program with the JIT disabled leads to one of the following:
v The failure remains. The problem is, therefore, not in the JIT. Do not read further
in this chapter. In some cases, the program might start failing in a different
manner; nevertheless, the problem is not related to the JIT.
v The failure disappears. The problem is most likely, although not definitely, in the
JIT.

Selectively disabling the JIT

If the failure of your Java program appears to come from a problem within the JIT,
you can try to narrow down the problem further.

© Copyright IBM Corp. 2003, 2006 237

JIT problem determination

The JIT optimizes methods at various optimization levels; that is, different
selections of optimizations are applied to different methods, based on their call
counts. Methods that are called more frequently are optimized at higher levels. By
changing JIT parameters, you can control the optimization level at which methods
are optimized, and determine whether the optimizer is at fault and, if it is, which
optimization is problematic.

JIT parameters are specified as a comma-separated list, appended to the -Xjit

option. The syntax is -Xjit:param1,param2=value,.... For example,
java -Xjit:verbose,optLevel=noOpt HelloWorld

runs the HelloWorld program, while enabling verbose output from the JIT, and
making it generate native code without performing any of the optimizations listed
in “How the JIT optimizes code” on page 30.

The JIT parameters give you a powerful tool that enables you to determine the
location of a JIT problem; whether it is in the JIT itself or in a few lines of code
that cause the JIT to fail. In addition, when you have identified a problem area,
you are automatically given a workaround so that you can continue to develop or
deploy code while losing only a fraction of JVM performance.

The first JIT parameter to try is count=0, which sets the JIT threshold to zero and
effectively causes the Java program to be run in purely compiled mode.

If the failure still occurs, try disableInlining. With this parameter set, the JIT is
prohibited from generating larger and more complex code in an attempt to
perform aggressive optimizations.

If the failure persists, try decreasing JIT optimization levels. The various
optimization levels are:
1. scorching
2. veryHot
3. hot
4. warm
5. cold
6. noOpt

Run the Java program with:

-Xjit:count=0,disableInlining,optLevel=scorching

Try each of the optimization levels in turn, and record your observations. If one of
these settings causes your failure to disappear, you have a quick workaround that
you can use while the Java service team analyzes and fixes the JIT problem. If you
can remove disableInlining from the JIT parameter list (that is, if removing it does
not cause the failure to reappear), do so to improve performance.

Locating the failing method

When you have arrived at the lowest optimization level at which the JIT must
compile methods to trigger the failure, you can try to find out which part of the
Java program, when compiled, causes the failure. You can then instruct the JIT to
limit the workaround to a specific method, class, or package, allowing the JIT to
compile the rest of the program as it normally would. If the failure occurs with

238 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 JIT problem determination

optLevel=noOpt, you can also instruct the JIT to not compile the method or
methods that are causing the failure (thus avoiding it).

To locate the method that is causing the failure, follow these steps:
1. Run the Java program with the JIT parameters verbose and vlog=filename. With
these parameters, the JIT reports its progress, as it compiles methods, in a
verbose log file, also called a limit file. A typical limit file contains lines that
correspond to compiled methods, like:
+ (hot) java/lang/Math.max(II)I @ 0x10C11DA4-0x10C11DDD

Lines that do not start with the plus sign are ignored by the JIT in the steps
below, so you can edit them out of the file.
2. Make a backup of the limit file.
3. Delete some lines from the limit file, and run the program again with the JIT
parameter limitFile=filename, where filename is the path to the limit file. This
parameter causes the JIT to compile only the methods listed in the limit file.
Repeat this if the program still fails.
The recommended number of lines to delete from the limit file in each
repetition is half the file, so that this step is essentially a binary search for the
failing method. Since lines that do not start with a plus sign are ignored, you
can also comment out lines rather than delete them, by inserting a space or a
minus sign at the beginning of the lines you wish to remove.
4. If the program no longer fails, then one or more of the methods that you have
removed in the last iteration must have been the cause of the failure. Empty the
current limit file and restore the removed lines from the backup (or uncomment
them, and comment out everything else). Repeat the previous step to see if the
program starts to the fail again.
5. Repeat the last two steps, as many times as necessary, to find the minimum
number of methods that must be compiled to trigger the failure. Often, you can
reduce the file to a single line.

When you have obtained a workaround and located the failing method, you can
limit the workaround to the failing method. For example, if the method
java/lang/Math.max(II)I causes the program to fail when compiled with
optLevel=hot, you can run the program with:
-Xjit:{java/lang/Math.max(II)I}(optLevel=warm,count=0)

which tells the JIT to compile only the troublesome method at an optimization
level of ″warm″, but compile all other methods normally.

If a method fails when it is compiled at ″noOpt″, you can exclude it from

compilation altogether, using the exclude=<method> parameter:
-Xjit:exclude={java/lang/Math.max(II)I}

Identifying JIT compilation failures

If the JVM crashes, and you can see that the failure has occurred in the JIT library
(j9jit22.dll on Windows, or libj9jit22.so on other platforms), the JIT might have
failed during an attempt to compile a method.

To see if the JIT is crashing in the middle of a compilation, use the verbose option
with the following additional settings:
-Xjit:verbose={compileStart|compileEnd}

Chapter 27. JIT problem determination 239

JIT problem determination

These verbose settings report when the JIT starts to compile a method, and when
it ends. If the JIT fails on a particular method (that is, it starts compiling, but
crashes before it can end), use the exclude= parameter to exclude it from
compilation (refer to “Locating the failing method” on page 238). If excluding the
method prevents the crash, you have an excellent workaround that you can use
while the service team correct your problem.

Performance of short-running applications

The IBM JIT is tuned for long-running applications typically used on a server. So,
if the performance of short-running applications is worse than expected, try the
-Xquickstart command-line parameter (refer to the -Xquickstart option in
“Nonstandard command-line options” on page 331), especially for those
applications in which execution time is not concentrated into a small number of
methods.

Also try adjusting the JIT threshold (using trial and error) for short-running
applications to improve performance. Refer to “Selectively disabling the JIT” on
page 237.

240 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 28. Garbage Collector diagnostics
This chapter describes how to diagnose the garbage collection operation. The
topics that are discussed in this chapter are:
v “How does the Garbage Collector work?”
v “Common causes of perceived leaks”
v “Basic diagnostics (-verbosegc)” on page 242
v “Advanced diagnostics” on page 249
v “TGC tracing” on page 250
v “Heap and native memory use by the JVM” on page 254

How does the Garbage Collector work?

Read Chapter 2, “Understanding the Garbage Collector,” on page 7 to get a full
understanding of the Garbage Collector. A short introduction to the Garbage
Collector is given here.

The IBM Virtual Machine for Java includes a memory manager, which manages the
Java heap. The memory manager allocates space from the heap as objects are
instantiated, keeping a record of where the remaining free space in the heap is
located. When free space in the heap is low and an object allocation cannot be
satisfied, an allocation failure is triggered and a garbage collection cycle is started.
Garbage collection identifies and frees previously allocated storage that is no
longer in use. When this process is complete, the memory manager retries the
allocation that it could not previously satisfy.

An application can request a manual garbage collection at any time, but this action
is not recommended. See “How to coexist with the Garbage Collector” on page 18.

Common causes of perceived leaks

When a garbage collection cycle starts, the Garbage Collector must locate all
objects in the heap that are still in use or ″live″. When this has been done, any
objects that are not in the list of live objects are unreachable. They are garbage, and
can be collected.

The key here is the condition unreachable. The Garbage Collector traces all
references that an object makes to other objects. Any such reference automatically
means that an object is reachable and not garbage. So, if the objects of an
application make reference to other objects, those other objects are live and cannot
be collected. However, obscure references sometimes exist that the application
overlooks. These references are reported as memory leaks.

Listeners
By installing a listener, you effectively attach your object to a static reference that is
in the listener. Your object cannot be collected while the listener is active. You must
explicitly uninstall a listener when you have finished using the object to which you
attached it.

© Copyright IBM Corp. 2003, 2006 241

Garbage Collector - common causes of perceived leaks

Hash tables
Anything that is added to a hash table, either directly or indirectly, from an
instance of your object, creates a reference to your object from the hashed object.
Hashed objects cannot be collected unless they are explicitly removed from any
hash table to which they have been added.

Hash tables are common causes of perceived leaks. If an object is placed into a
hash table, that object and all the objects that it references are reachable.

Static data
This exists independently of instances of your object. Anything that it points to
cannot be collected even if no instances of your class are present that contain the
static data.

JNI references
Objects that are passed from the JVM to an application across the JNI interface
have a reference to them that is held in the JNI code of the JVM. Without this
reference, the Garbage Collector cannot trace live native objects. Such references
must be explicitly cleared by the native code application before they can be
collected. See the JNI documentation on the Sun website (java.sun.com) for more
information.

Premature expectation
You instantiate a class, finish with it, tidy up all listeners, and so on. You have a
finalizer in the class, and you use that finalizer to report that the finalizer has been
called. On all the later garbage collection cycles, your finalizer is not called. It
seems that your unused object is not being collected and that a memory leak has
occurred, but this is not so.

The IBM Garbage Collector does not collect garbage unless it needs to. It does not
necessarily collect all garbage when it does run. It might not collect garbage if you
manually invoke it (by using System.gc()). This is because running the Garbage
Collector is an intensive operation, and it is designed to run as infrequently as
possible for as a short time as possible.

Objects with finalizers

Objects that have finalizers cannot be collected until the finalizer has run.
Finalizers run on a separate thread, and thus their execution might be delayed, or
not occur at all. This is allowed. See “How to coexist with the Garbage Collector”
on page 18 for more details.

Basic diagnostics (-verbosegc)

Verbose logging is intended as the first tool to be used when attempting to
diagnose garbage collector problems; more detailed analysis can be performed by
invoking one or more TGC (trace garbage colector) traces. Note that the output
provided by -verbose:gc can and does change between releases. It is assumed that
you are familiar with details of the different collection strategies employed in the
v1.4.2 SDK.

Garbage collection triggered by System.gc()

Java programs can trigger garbage collections to occur manually by invoking the
method System.gc(). Verbose output produced by System.gc() calls is similar to:

242 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Basic diagnostics (-verbosegc)

<sys id="1" timestamp="Wed Jan 14 11:30:46 2004" intervalms="0.000">

<gc type="global" id="1" totalid="2" intervalms="0.000">
<compaction movecount="9138" movebytes="409188" />
<refs_cleared soft="0" weak="7" phantom="0" />
<finalization objectsqueued="0" />
<timesms mark="3.187" sweep="0.524" compact="4.629" total="8.392" />
<nursery freebytes="1790876" totalbytes="2097152" percent="85" />
<tenured freebytes="8278684" totalbytes="8388608" percent="98" />
</gc>
<time totalms="9.668" />
</sys>

<sys> Indicates that a System.gc() has occurred. The id attribute gives the
number of this System.gc() call; in this case, this is the first such call in the
life of this VM. timestamp gives the local timestamp when the System.gc()
call was made and intervalms gives the number of milliseconds that have
elapsed since the previous System.gc() call. In this case, because this is the
first such call, the number returned is zero.
<gc type=″global″>
Indicates that, as a result of the System.gc() call, a global garbage collection
was triggered. The contents of the <gc> tag for a global collection are
explained in detail in “Global collections” on page 245.
<time>
Shows the total amount of time taken to handle the System.gc call (in
milliseconds).

Allocation failures
When an attempt is made to allocate to the heap but insufficient memory is
available, an allocation failure is triggered. The output produced depends on the
area of the heap in which the allocation failure occurred.

Nursery allocation failures

<af type="nursery" id="3" timestamp="Wed Jan 14 11:30:47 2004" intervalms="88.219">
<minimum requested_bytes="60" />
<time exclusiveaccessms="0.010" />
<nursery freebytes="0" totalbytes="2097152" percent="0" />
<gc type="scavenger" id="3" totalid="5" intervalms="88.264">
<flipped objectcount="30350" bytes="1576372" />
<tenured objectcount="0" bytes="0" />
<finalization objectsqueued="0" />
<nursery freebytes="393216" totalbytes="2097152" percent="18" tenureage="9" />
<time totalms="12.538" />
</gc>
<time totalms="12.609" />
<nursery freebytes="391168" totalbytes="2097152" percent="18" />
<tenured freebytes="8261612" totalbytes="8388608" percent="98" />
</af>
<af type=″nursery″>
Indicates that an allocation failure has occurred when attempting to
allocate to the nursery. The id attribute shows the index of that type of
allocation failure that has occurred. timestamp shows a local timestamp at
the time of the allocation failure, and intervalms shows the number of
milliseconds elapsed since the previous allocation failure of that type.
<minimum>
Shows the number of bytes requested by the allocate that triggered the

Chapter 28. Garbage Collector diagnostics 243

Basic diagnostics (-verbosegc)

failure. Note that, following the garbage collection, freespace might drop
by more than this amount, because of a possible freelist discard or TLH
refresh.
<time exclusiveaccessms=>
Shows the amount of time taken to obtain exclusive VM access. Note that a
further optional line <warning details="exclusive access time includes
previous garbage collections" /> might occasionally appear, to inform
you that the following garbage collection was queued because the
allocation failure was triggered while another thread was already
performing a garbage collection. Normally, this first collection will have
freed enough heap space to satisfy both allocation requests (the original
one that triggered the garbage collection and the subsequently queued
allocation request). However, sometimes this is not the case and another
garbage collection is triggered almost immediately. This additional line
informs you that the pause time displayed might be slightly misleading
unless you are aware of the underlying threading used.
<nursery>
Shows the status of the nursery at the time of the failure, including the
percentage that was free.
<gc> Indicates that, as a result of the allocation failure, a garbage collection was
triggered. In this case, a scavenger collection occurred. The contents of this
tag are explained in detail in “Advanced diagnostics” on page 249.
<time>
Shows the total time taken to handle the allocation failure.
<nursery> and <tenured>
Show the status of the different heap areas following the handling of the
allocation failure, including the percentage of each area free.

Tenured allocation failures

Here is an example of an allocation occurring in the tenured area:
<af type="tenured" id="4" timestamp="Wed Jan 14 11:49:22 2004" intervalms="366.559">
<minimum requested_bytes="148" />
<time exclusiveaccessms="0.008" />
<tenured freebytes="0" totalbytes="11337472" percent="0" />
<gc type="global" id="6" totalid="6" intervalms="366.676">
<compaction movecount="139926" movebytes="9478888" />
<refs_cleared soft="0" weak="0" phantom="0" />
<finalization objectsqueued="0" />
<timesms mark="34.155" sweep="1.762" compact="65.891" total="101.849" />
<tenured freebytes="1784720" totalbytes="11337472" percent="15" />
</gc>
<expansion type="tenured" amount="2310400" newsize="13647872" timetaken="0.052"
reason="insufficient free space following gc" />
<time totalms="101.973" />
<tenured freebytes="4094548" totalbytes="13647872" percent="30" />
</af>

All the elements of this output have the same meanings as those for an allocation
failure occurring in the nursery. The only difference is the addition of an expansion
tag.
<expansion>
Indicates that during the handling of the allocation (but after the garbage
collection) a heap expansion was triggered. The area expanded, amount the
area was increased by (in bytes), its new size, the time taken to expand,
and the reason for the expansion are shown.

244 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Basic diagnostics (-verbosegc)

Global collections
An example of the output produced when a global collection is triggered is:
<gc type="global" id="6" totalid="6" intervalms="383.226">
<compaction movecount="139926" movebytes="9478888" />
<refs_cleared soft="0" weak="0" phantom="0" />
<finalization objectsqueued="0" />
<timesms mark="33.391" sweep="1.760" compact="65.958" total="101.149" />
<tenured freebytes="1784720" totalbytes="11337472" percent="15" />
</gc>
<gc> Indicates that a garbage collection was triggered on the heap.
Type=″global″ indicates that this was a global collection (mark, sweep,
possibly compact). The id attribute gives the occurrence number of this
global collection. The totalid indicates the total number of garbage
collections (of all types) that have taken place. Currently this is the sum of
the number of global collections and the number of scavenger collections.
intervalms gives the number of milliseconds since the previous global
collection.
<compaction>
Shows the number of objects that were moved during compaction, and the
total number of bytes these objects represented. This line appears only if
compaction occurred during the collection.
<refs_cleared>
Provides information relating to the number of Java reference objects that
were cleared during the collection. In this example, no references were
cleared.
<finalization>
Provides information detailing the number of objects containing finalizers
that were enqueued for VM finalization during the collection. Note that
this is not equal to the number of finalizers that were run during the
collection, because finalizers are scheduled by the VM.
<timems>
Provides information detailing, respectively, times taken for each of the
mark, sweep, and compact phases, as well as the total time taken. When
compaction was not triggered, the number returned is zero. Note that if
the VM being run is not compiled with compaction support, the compact
field will not be displayed.
<tenured>
Indicates the status of the tenured area following the collection. If running
in generational mode, there will also be a <nursery> line output, showing
the status of the active nursery area too.

Scavenger collections
An example of the output produced when a scavenger collection is triggered is:
<gc type="scavenger" id="17" totalid="19" intervalms="145.701">
<failed type="tenured" objectcount="355" bytes="32964" />
<flipped objectcount="4192" bytes="655164" />
<tenured objectcount="3858" bytes="650928" />
<finalization objectsqueued="0" />
<scavenger tiltratio="74" />
<nursery freebytes="1440760" totalbytes="2097152" percent="68" tenureage="1" />
<time totalms="6.451" />
</gc>
<gc> Indicates that a garbage collection has been triggered, and
type=″scavenger″ indicates that this is a scavenger collection. The id

Chapter 28. Garbage Collector diagnostics 245

Basic diagnostics (-verbosegc)

attribute shows the number of this type of collection that have taken place
and the totalid attribute shows the total number of garbage collections of
all types that have taken place (including this one). intervalms gives the
amount of time (in milliseconds) since the last collection of this type.
<failed type=″tenured″>
Indicates that the scavenger failed to tenure some objects when it tried to
during the collection. The number affected and the total bytes represented
by these objects is shown. Additionally or alternatively, <failed
type=″flipped″> could have been displayed, which would indicate that the
scavenger failed to flip certain objects into the survivor space.
<flipped>
Shows the number of objects that were flipped into the survivor space
during the scavenge, together with the total number of bytes flipped.
<scavenger tiltratio=″x″ />
Shows the amount that new-space is tilted by, following the post-scavenge
retilt. The scavenger can redistribute memory between the allocate and
survivor areas to maximize the time between scavenges and the number of
objects that ″die young″.
<tenured>
Shows the number of objects that were moved into the old area during the
scavenge, together with the total number of bytes tenured.
<nursery>
Shows the amount of free and total space in the nursery area following the
scavenge, along with the current number of flips an object must have
survived in order to be tenured.
<time>
Shows the total time taken to perform the scavenge, in milliseconds.

Note that there are a number of additional lines that can be output during a
scavenge. It is possible for a scavenge to fail (for example, if the nursery was
excessively tilted with a full old area, and certain objects could not be copied or
tenured). In this case, an additional <warning details="aborted collection" />
line is displayed.

During a scavenge, if it is not possible to tenure an object, an expansion of the

tenured area might be triggered. This will be shown as a separate line of
-verbosegc.

It is also possible for the entirety of new space to be resized following a scavenge.
Again, this is shown as a separate line of -verbosegc.

Concurrent mark
When running with concurrent mark, there are several additional -verbosegc
outputs which will be displayed.

Concurrent kickoff
When the concurrent mark process is triggered, the following output is produced:
<con event="kickoff" timestamp="Fri Nov 14 15:14:27 2003">
<stats tenurefreebytes="439416" tracetarget="4346357" kickoff="543294" tracerate="8" />
</con>

This output shows that concurrent mark was kicked off, and gives a local
timestamp for this. Statistics are produced showing the amount of free space in the

246 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Basic diagnostics (-verbosegc)

tenured area, the target amount of tracing to be performed by concurrent mark, the
kickoff threshold at which concurrent is triggered, and the initial trace rate. The
trace rate represents the amount of tracing each mutator thread should perform
relative to the amount of space it is attempting to allocate within the heap. In this
example, a mutator thread that allocates 20 bytes will be required to trace 20 * 8 =
160 bytes. If also running in generational mode, an additional nurseryfreebytes=
attribute is displayed, showing the status of the nursery as concurrent mark was
triggered.

Allocation failures during concurrent mark

When an allocation failure occurs during concurrent mark, either the tracing
performed so far will be discarded, or it will be used during the subsequent
collection. These two possibilities correspond to the ″aborted″ and ″halted″
concurrent mark events.

Concurrent aborted: Here is a sample output produced when concurrent mark is

aborted:
<af type="tenured" id="7" timestamp="Fri Nov 14 15:18:44 2003" intervalms="2108.509">
<minimum requested_bytes="36" />
<tenured freebytes="0" totalbytes="38263808" percent="0" />
<con event="aborted">
<stats tracetarget="2127800">
<traced total="528784" mutators="528784" helpers="0" percent="25" />
</stats>
</con>
<gc type="global" id="16" totalid="16" timestamp="Fri
Nov 14 15:18:44 2003" intervalms="1993.718">
<refs_cleared soft="2" weak="0" phantom="0" />
<finalization objectsqueued="5" />
<timesms mark="21.013" sweep="19.438" compact="0.000" total="40.533" />
<heap freebytes="31890452" totalbytes="38263808" percent="83" />
</gc>
<expansion type="tenured" amount="512" newsize="38264320" timetaken="0.052"
reason="excessive time being spent in gc" />
<time totalms="42.617" />
<tenured freebytes="31890964" totalbytes="38264320" percent="83" />
</af>
<con event=″aborted″>
Shows that, as a result of the allocation failure, concurrent mark tracing
was aborted. The statistics produced show the target amount of tracing to
be performed by concurrent mark, along with the amount actually traced
so far. Of the amount traced so far, the respective amount traced by
mutator threads and a background concurrent mark thread is shown. The
percentage of the trace target traced is shown.

Concurrent halted: Here is a sample output produced when concurrent mark is

halted:
<af type="tenured" id="1" timestamp="Wed Jan 14 12:48:50 2004" intervalms="0.000">
<minimum requested_bytes="28" />
<time exclusiveaccessms="0.007" />
<tenured freebytes="0" totalbytes="21676032" percent="0" />
<con event="halted" mode="clean trace">
<stats tracetarget="5204261">
<traced total="4058780" mutators="4058780" helpers="0" percent="78" />
<cards cleaned="5" estimateddirty="5" kickoff="443556" />
</stats>
</con>
<con event="final card cleaning">
<stats cardscleaned="6" traced="2432" durationms="0.156" />
</con>
<gc type="global" id="6" totalid="43" intervalms="1416.726">

Chapter 28. Garbage Collector diagnostics 247

Basic diagnostics (-verbosegc)

<compaction movecount="208435" movebytes="21087264" />

<refs_cleared soft="0" weak="0" phantom="0" />
<finalization objectsqueued="0" />
<timesms mark="1.058" sweep="6.905" compact="142.778" total="150.786" />
<nursery freebytes="903500" totalbytes="2097152" percent="43" />
<tenured freebytes="523328" totalbytes="21676032" percent="2" />
</gc>
<expansion>
<tenured increase="8542208" newsize="30218240" />
<time takenms="0.123" />
</expansion>
<time totalms="167.050" />
<tenured freebytes="9065508" totalbytes="30218240" percent="30" />
</af>
<con event=″halted″>
Shows that concurrent mark tracing was halted as a result of the allocation
failure. The tracing target is shown, together with the amount that was
actually performed, both by mutator threads and the concurrent mark
background thread. The percentage of the trace target traced is shown.
<cards cleaned=″5″>
Shows that five dirty cards in the card table were cleaned during
concurrent mark. Card cleaning occurs during concurrent mark after all
available tracing has been exhausted. An estimation of the number of dirty
cards is given.
<con event=″final card cleaning″>
Indicates that final card cleaning occurred before the garbage collection
was triggered. The number of cards cleaned during the process and the
number of bytes traced is shown, along with the total time taken by the
process.

Concurrent collection: If concurrent mark completes all tracing and card-cleaning,

a concurrent collection is triggered. The output produced by this is shown:
<con event="collection" id="5" timestamp="Wed Jan 14 12:48:59 2004" intervalms="7810.298">
<time exclusiveaccessms="0.009" />
<tenured freebytes="0" totalbytes="30218240" percent="0" />
<stats tracetarget="8060116">
<traced total="5509424" mutators="5509424" helpers="0" percent="68" />
<cards cleaned="110" kickoff="503757" />
</stats>
<con event="final card cleaning">
<stats cardscleaned="78" traced="4232" durationms="4.329" />
</con>
<gc type="global" id="9" totalid="95" intervalms="4364.041">
<compaction movecount="268267" movebytes="25955224" />
<refs_cleared soft="0" weak="0" phantom="0" />
<finalization objectsqueued="0" />
<timesms mark="9.330" sweep="35.326" compact="158.164" total="202.882" />
<nursery freebytes="1135852" totalbytes="2097152" percent="54" />
<tenured freebytes="4926264" totalbytes="30218240" percent="16" />
</gc>
<expansion type="tenured" amount="5913600" newsize="36131840" timetaken="0.120"
reason="insufficient free space following gc" />
<tenured freebytes="10839864" totalbytes="36131840" percent="30" />
<time totalms="209.324" />
</con>
<con event=″collection″
Shows that a concurrent collection has been triggered. The id attribute
shows the number of this concurrent collection, a local timestamp is
outputted, and the number of milliseconds since the previous concurrent
collection is displayed.

248 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Basic diagnostics (-verbosegc)

<stats>
Shows the tracing statistics for the concurrent tracing that has taken place
previously. The target amount of tracing is shown, together with that
which actually took place (both by mutators threads and helper threads).
Information is displayed showing the number of cards in the card table
that were cleaned during the concurrent mark process, and an estimation
of the total number of dirty cards.
<con event=″final card cleaning″>
Shows that final card cleaning has been triggered. The number of cards
cleaned is displayed, together with the number of milliseconds taken to do
so.

Following a concurrent collection, a normal global collection is triggered.

System.gc() calls during concurrent mark

<sys id="11" timestamp="Wed Jan 14 12:16:06 2004" intervalms="3988.669">
<con event="halted" mode="exhausted">
<stats tracetarget="35386516">
<traced total="19307804" mutators="19307804" helpers="0" percent="5" />
<cards cleaned="87" estimateddirty="87" kickoff="2211657" />
</stats>
</con>
<con event="final card cleaning">
<stats cardscleaned="345" traced="2543" durationms="5.062" />
</con>
<gc type="global" id="197" totalid="197" intervalms="3994.143">
<compaction movecount="719044" movebytes="90813316" />
<refs_cleared soft="0" weak="0" phantom="0" />
<finalization objectsqueued="0" />
<timesms mark="1.777" sweep="38.296" compact="565.708" total="605.824" />
<tenured freebytes="21631284" totalbytes="112857088" percent="19" />
</gc>
<time totalms="610.999" />
</sys>

This output shows that a system.gc() call was made after concurrent mark had
kicked off. In this case, enough tracing had been performed for the work to be
reused, so concurrent mark is halted rather than aborted. The results for final
card-cleaning are also shown.

Advanced diagnostics
The -verbosegc option is the main diagnostic that is available for runtime analysis
of the Garbage Collector. However, additional command-line options are available
that affect the behavior of the Garbage Collector and might aid diagnostics. These
options are:
-Xdisableexplicitgc
-Xgcthreads
-Xclassgc
-Xnoclassgc
-Xcompactgc
-Xnocompactgc
-Xcompactexplicitgc
-Xnocompactexplicitgc

-Xdisableexplicitgc
This option converts Java application calls to java.lang.System.gc() into no-ops.

Chapter 28. Garbage Collector diagnostics 249

Advanced diagnostics

Many applications still make an excessive number of explicit calls to System.gc to

request garbage collection. In many cases, these calls degrade performance through
premature garbage collection and compactions. However, it is not always possible
to remove the calls at source.

The -Xdisableexplicitgc parameter allows the JVM to ignore these garbage

collection suggestions. Typically, system administrators would use this parameter
in applications that show some benefit from its use. -Xdisableexplicitgc is a
nondefault setting.

-Xdisableexplicitgc should be used only when testing had shown it to be

beneficial; for example, from performance testing in conjunction with -verbose:gc
output.

-Xgcthreads
This option sets the number of helper threads that the Garbage Collector uses for
parallel operations. The number is set to (number of CPUs − 1). On single CPU
boxes, no helper threads run. The disabling of helper threads disables parallel
operations, at the cost of performance, and might expose problems in this area. No
advantage is gained if you increase the number of threads above the default
setting; you are recommended not to do so.

-Xclassgc
This option enables collection of class objects at every garbage collection.

-Xnoclassgc
This option disables collection of class objects.

-Xcompactgc
This option enables compaction at every garbage collection.

-Xnocompactgc
This option disables heap compaction.

-Xcompactexplicitgc
This option runs full compaction each time System.gc() is called.

-Xnocompactexplicitgc
This option means a compaction is never run when System.gc() is called.

TGC tracing
By enabling one or more TGC (trace garbage collector) traces, more detailed
garbage collection information than that displayed by -verbose:gc will be shown.
This section summarizes the different TGC traces available. The output is piped to
stdout. More than one trace can be enabled simultaneously by separating the
parameters with commas, for example -Xtgc:backtrace,compaction.

-Xtgc:backtrace
This trace shows information tracking which vmThread triggered the garbage
collection. For a System.gc() this might be similar to:
"main" (0x0003691C)

250 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Garbage Collector - TGC tracing

This shows that the GC was triggered by the thread with the name ″main″ and
osThread 0x0003691C.

One line is printed for each global or scavenger collection, showing the thread that
triggered the GC.

-Xtgc:compaction
This trace shows information relating to compaction, similar to:
Compact(3): reason = 7 (forced compaction)
Compact(3): Thread 0, setup stage: 8 ms.
Compact(3): Thread 0, move stage: handled 42842 objects in 13 ms, bytes moved 2258028.
Compact(3): Thread 0, fixup stage: handled 0 objects in 0 ms, root fixup time 1 ms.
Compact(3): Thread 1, setup stage: 0 ms.
Compact(3): Thread 1, move stage: handled 35011 objects in 8 ms, bytes moved 2178352.
Compact(3): Thread 1, fixup stage: handled 74246 objects in 13 ms, root fixup time 0 ms.
Compact(3): Thread 2, setup stage: 0 ms.
Compact(3): Thread 2, move stage: handled 44795 objects in 32 ms, bytes moved 2324172.
Compact(3): Thread 2, fixup stage: handled 6099 objects in 1 ms, root fixup time 0 ms.
Compact(3): Thread 3, setup stage: 8 ms.
Compact(3): Thread 3, move stage: handled 0 objects in 0 ms, bytes moved 0.
Compact(3): Thread 3, fixup stage: handled 44797 objects in 7 ms, root fixup time 0 ms.

This trace shows that compaction occurred during the third global GC, for reason
″7″. The compaction reasons are explained in detail in “Compaction phase” on
page 8. In this case, four threads are performing compaction. The trace shows the
work performed by each thread during setup, move, and fixup. The time for each
stage is shown together with the number of objects handled by each thread.

-Xtgc:concurrent
This trace displays basic extra information about the concurrent mark helper
thread.
<CONCURRENT GC BK thread 0x0002645F activated after GC(5)>

<CONCURRENT GC BK thread 0x0002645F (started after GC(5)) traced 25435>

This trace shows when the background thread was activated, and the amount of
tracing it performed (in bytes).

-Xtgc:dump
This trace shows extra information following the sweep phase of a global garbage
collection. This is an extremely large trace – a sample of one GC’s output is:
<GC(4) 13F9FE44 freelen=x000002C4 -- x00000038 spec/jbb/Stock>
<GC(4) 13FA0140 freelen=x00000010>
<GC(4) 13FA0150 freelen=x00000050 -- x0000001C java/lang/String>
<GC(4) 13FA0410 freelen=x000002C4 -- x00000024 spec/jbb/infra/Collections/longBTreeNode>
<GC(4) 13FA0788 freelen=x00000004 -- x00000050 java/lang/Object[]>
<GC(4) 13FA0864 freelen=x00000010>
<GC(4) 13FA0874 freelen=x0000005C -- x0000001C java/lang/String>
<GC(4) 13FA0B4C freelen=x000002C4 -- x00000038 spec/jbb/Stock>
<GC(4) 13FA0E48 freelen=x00000010>
<GC(4) 13FA0E58 freelen=x00000068 -- x0000001C java/lang/String>
<GC(4) 13FA1148 freelen=x000002C4 -- x00000038 spec/jbb/Stock>
<GC(4) 13FA1444 freelen=x00000010>
<GC(4) 13FA1454 freelen=x0000006C -- x0000001C java/lang/String>
<GC(4) 13FA174C freelen=x000002C4 -- x00000038 spec/jbb/Stock>
<GC(4) 13FA1A48 freelen=x00000010>
<GC(4) 13FA1A58 freelen=x00000054 -- x0000001C java/lang/String>
<GC(4) 13FA1D20 freelen=x000002C4 -- x00000038 spec/jbb/Stock>
<GC(4) 13FA201C freelen=x00000010>

Chapter 28. Garbage Collector diagnostics 251

 Garbage Collector - TGC tracing

<GC(4) 13FA202C freelen=x00000044 -- x0000001C java/lang/String>

<GC(4) 13FA22D4 freelen=x000002C4 -- x00000038 spec/jbb/Stock>
<GC(4) 13FA25D0 freelen=x00000010>
<GC(4) 13FA25E0 freelen=x00000048 -- x0000001C java/lang/String>
<GC(4) 13FA2890 freelen=x000002C4 -- x00000038 spec/jbb/Stock>
<GC(4) 13FA2B8C freelen=x00000010>
<GC(4) 13FA2B9C freelen=x00000068 -- x0000001C java/lang/String>
<GC(4) 13FA2E8C freelen=x000002C4 -- x00000038 spec/jbb/Stock>
<GC(4) 13FA3188 freelen=x00000010>

A line of output is printed for every free chunk in the system, including dark
matter (free chunks that are not on the free list for some reason, usually because
they are too small). Each line contains the base address and the size in bytes of the
chunk. If the chunk is followed in the heap by an object, the size and class name of
the object is also printed.

| -Xtgc:excessiveGC
| This trace shows statistics for garbage collection cycles.

| After a garbage collection cycle has completed, a trace entry is produced:

| excessiveGC: gcid="10" intimems="122.269" outtimems="1.721" \
| percent="98.61" averagepercent="37.89"

| This trace shows how much time was spent performing garbage collection and
| how much time was spent out of garbage collection. In this example, garbage
| collection cycle 10 took 122.269 ms to complete and 1.721 ms passed between
| collections 9 and 10. These statistics show that garbage collection accounted for
| 98.61% of the time from the end of collection 9 to the end of collection 10. The
| average time spent in garbage collection is 37.89%.

| When the average time in garbage collection reaches 95%, extra trace entries are
| produced:
| excessiveGC: gcid="65" percentreclaimed="1.70" freedelta="285728" \
| activesize="16777216" currentsize="16777216" maxiumumsize="16777216"

| This trace shows how much garbage was collected. In this example, 285728 bytes
| were reclaimed by garbage collection 65, which accounts for 1.7% of the total heap
| size. The example also shows that the heap has expanded to its maximum size (see
| -Xmx in “General Garbage Collection options” on page 334).

| When the average time in garbage collection reaches 95% and the percentage of
| free space reclaimed by a collection drops below 3%, another trace entry is
| produced:
| excessiveGC: gcid="65" percentreclaimed="1.70" minimum="3.00" excessive gc raised

| The JVM will then throw an OutOfMemoryError.

-Xtgc:freelist
Before a garbage collection, this trace prints information about the free list and
allocation statistics since the last GC. It prints the number of items on the free list,
including ″deferred″ entries (with the scavenger, the unused semispace is a
deferred free list entry). For TLH and non-TLH allocations, this prints the total
number of allocations, the average allocation size, and the total number of bytes

252 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Garbage Collector - TGC tracing

discarded in during allocation. For non-TLH allocations, also included is the

average number of entries that were searched before a sufficiently large entry was
found.
*8* free 0
*8* deferred 0
total 0
<Alloc TLH: count 3588, size 3107, discard 31>
< non-TLH: count 6219, search 0, size 183, discard 0>

-Xtgc:parallel
This trace shows statistics about the activity of the parallel threads during the
mark and sweep phases of a global GC.
Mark: busy stall tail acquire release
0: 30 30 0 0 3
1: 53 7 0 91 94
2: 29 31 0 37 37
3: 37 24 0 243 237
Sweep: busy idle sections 127 merge 0
0: 10 0 96
1: 8 1 0
2: 8 1 31
3: 8 1 0

This trace shows four threads (0-3) and the work done by each. For mark, the time
spent busy, stalled, and in tail is shown (in milliseconds) together with the number
of work packets each thread acquired and released during marking. For sweep, the
time spent busy and idle is shown (in milliseconds) together with the number of
sweep chunks processed by each thread and in total (127 above). The total merge
time is also shown (0ms above).

-Xtgc:references
This trace shows activity relating to reference handling during garbage collections.
enqueuing ref sun/misc/SoftCache$ValueCell@0x1564b5ac -> 0x1564b4c8
enqueuing ref sun/misc/SoftCache$ValueCell@0x1564b988 -> 0x1564b880
enqueuing ref sun/misc/SoftCache$ValueCell@0x15645578 -> 0x15645434

This trace shows three reference objects being enqueued. The location of the
reference object and the referent is displayed, along with the class name of the
object. Note that for finalizer objects this does not mean the finalizer has been run,
merely that it has been queued to the finalizer thread.

-Xtgc:scavenger
This trace prints a histogram following each scavenger collection. A graph is
shown of the different classes of objects remaining in the survivor space, together
with the number of occurrences of each class and the age of each object (the
number of times it has been flipped). A sample of the output from a single
scavenge is shown below:
{SCAV: tgcScavenger OBJECT HISTOGRAM}

{SCAV: | class | instances of age 0-14 in semi-space |

{SCAV: java/lang/ref/SoftReference 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0
{SCAV: java/io/FileOutputStream 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0
{SCAV: sun/nio/cs/StreamEncoder$ConverterSE 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0
{SCAV: java/io/FileInputStream 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0
{SCAV: char[][] 0 102 0 0 0 0 0 0 0 0 0 0 0 0 0
{SCAV: java/lang/ref/SoftReference[] 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0
{SCAV: java/io/BufferedOutputStream 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0
{SCAV: java/io/BufferedWriter 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0

Chapter 28. Garbage Collector diagnostics 253

Garbage Collector - TGC tracing

{SCAV: java/io/OutputStreamWriter 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0
{SCAV: java/io/PrintStream 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0
{SCAV: java/io/BufferedInputStream 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0
{SCAV: java/lang/Thread[] 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0
{SCAV: java/lang/ThreadGroup[] 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0
{SCAV: sun/io/ByteToCharCp1252 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0
{SCAV: sun/io/CharToByteCp1252 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0

-Xtgc:terse
This trace dumps the contents of the entire heap before and after a garbage
collection. As such, this is an extremely large trace. For each object or free chunk in
the heap, a line of trace output is produced. Each line contains the base address,
″a″ if it is an allocated object and ″f″ if it is a free chunk, the size of the chunk in
bytes, and if it is an object, its class name. A sample is shown below:
*DH(1)* 230AD778 a x0000001C java/lang/String
*DH(1)* 230AD794 a x00000048 char[]
*DH(1)* 230AD7DC a x00000018 java/lang/StringBuffer
*DH(1)* 230AD7F4 a x00000030 char[]
*DH(1)* 230AD824 a x00000054 char[]
*DH(1)* 230AD878 a x0000001C java/lang/String
*DH(1)* 230AD894 a x00000018 java/util/HashMapEntry
*DH(1)* 230AD8AC a x0000004C char[]
*DH(1)* 230AD8F8 a x0000001C java/lang/String
*DH(1)* 230AD914 a x0000004C char[]
*DH(1)* 230AD960 a x00000018 char[]
*DH(1)* 230AD978 a x0000001C java/lang/String
*DH(1)* 230AD994 a x00000018 char[]
*DH(1)* 230AD9AC a x00000018 java/lang/StringBuffer
*DH(1)* 230AD9C4 a x00000030 char[]
*DH(1)* 230AD9F4 a x00000054 char[]
*DH(1)* 230ADA48 a x0000001C java/lang/String
*DH(1)* 230ADA64 a x00000018 java/util/HashMapEntry
*DH(1)* 230ADA7C a x00000050 char[]
*DH(1)* 230ADACC a x0000001C java/lang/String
*DH(1)* 230ADAE8 a x00000050 char[]
*DH(1)* 230ADB38 a x00000018 char[]
*DH(1)* 230ADB50 a x0000001C java/lang/String
*DH(1)* 230ADB6C a x00000018 char[]
*DH(1)* 230ADB84 a x00000018 java/lang/StringBuffer
*DH(1)* 230ADB9C a x00000030 char[]
*DH(1)* 230ADBCC a x00000054 char[]
*DH(1)* 230ADC20 a x0000001C java/lang/String
*DH(1)* 230ADC3C a x00000018 java/util/HashMapEntry
*DH(1)* 230ADC54 a x0000004C char[]

Heap and native memory use by the JVM

The JVM itself makes little use of the heap except for class objects. Class objects
also use native memory.

The JVM does use native memory, but, for efficiency, does not use standard stack
frames. The JIT (see Chapter 4, “Understanding the JIT,” on page 29), the MMI (see
Chapter 27, “JIT problem determination,” on page 237), and the JVM all have their
own styles of stack frames. The only tool that can walk the stack is the dump
formatter (see Chapter 26, “Using the dump formatter,” on page 223). The only
other users of native memory are native code and some types of large native
objects.

254 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Garbage Collector - heap and native memory use by the JVM

Native code
The term “native code ” refers to native code (usually C or C++) that is compiled
into a library and accessed through the JNI. Alternatively, native code can load an
encapsulated JVM. Either way, the native code uses standard OS stack frames,
unless it manages the stack itself. The JVM keeps track of the portion of the stack
that it uses, because it needs this information to find a set of root objects for
garbage collection.

The JVM has no knowledge of and cannot control the native stack in this scenario.
Growth of the native stack is not normally due to JVM code.

Large native objects

On some platforms, the JVM can recognize large native objects (such as bitmaps)
and keep them in native memory. A small object is placed onto the heap, which
acts as an anchor for the native data (wherever it is). Clearly, such native memory
tends to consist of large chunks that can grow quickly unless the owning
application strictly controls the anchoring objects.

Chapter 28. Garbage Collector diagnostics 255

Garbage Collector - heap and native memory use by the JVM

256 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 29. Class-loader diagnostics
This chapter describes some diagnostics that are available for class-loading. The
topics that are discussed in this chapter are:
v “Class-loader command-line options”
v “Class-loader runtime diagnostics”
v “Loading from native code” on page 258

Class-loader command-line options

These extended command-line options are available:
-verbose:dynload
This option provides detailed information as each class is loaded by the
JVM, including:
v The class name and package
v For class files that were in a .jar file, the name and directory path of the
.jar
v Details of the size of the class and the time taken to load the class
The data is written out to stderr. An example of the output follows:
<Loaded java/lang/String from C:\sdk\jre\lib\vm.jar>
<Class size 17258; ROM size 21080; debug size 0>
<Read time 27368 usec; Load time 782 usec; Translate time 927 usec>
-Xverify
This option enforces strict class-loading checks on classes that are loaded
by way of the extensions and application class loaders. The default is that
strict checking is not performed.
-Xverify:none
This option disables strict class-loading checks on all class loaders. The
default is that strict checks are enforced except on the JVM internal class
loaders.
-Xverify:remote
This option enables strict class-loading checks on remotely loaded classes.

Class-loader runtime diagnostics

An extremely useful command-line definition is available that lets you trace the
way the class loaders find and load a given class. The command-line definition is
-Dibm.cl.verbose=<name>

For example:
C:\j9test>java -Dibm.cl.verbose=HelloWorld HelloWorld

might produce output that is similar to this:

ExtClassLoader attempting to find HelloWorld
ExtClassLoader using classpath C:\j9test\testjdk\sdk\jre\lib\ext\gskikm.jar;C:\j
9test\testjdk\sdk\jre\lib\ext\ibmjcefips.jar;C:\j9test\testjdk\sdk\jre\lib\ext\i
bmjceprovider.jar;C:\j9test\testjdk\sdk\jre\lib\ext\ibmjsseprovider2.jar;C:\j9te
st\testjdk\sdk\jre\lib\ext\ibmpkcs11.jar;C:\j9test\testjdk\sdk\jre\lib\ext\ibmpk
cs11impl.jar;C:\j9test\testjdk\sdk\jre\lib\ext\indicim.jar;C:\j9test\testjdk\sdk

© Copyright IBM Corp. 2003, 2006 257

\jre\lib\ext\jaccess.jar;C:\j9test\testjdk\sdk\jre\lib\ext\jdmpview.jar;C:\j9tes
t\testjdk\sdk\jre\lib\ext\ldapsec.jar;C:\j9test\testjdk\sdk\jre\lib\ext\oldcertp
ath.jar
ExtClassLoader could not find HelloWorld.class in C:\j9test\testjdk\sdk\jre\lib\
ext\gskikm.jar
ExtClassLoader could not find HelloWorld.class in C:\j9test\testjdk\sdk\jre\lib\
ext\ibmjcefips.jar
ExtClassLoader could not find HelloWorld.class in C:\j9test\testjdk\sdk\jre\lib\
ext\ibmjceprovider.jar
ExtClassLoader could not find HelloWorld.class in C:\j9test\testjdk\sdk\jre\lib\
ext\ibmjsseprovider2.jar
ExtClassLoader could not find HelloWorld.class in C:\j9test\testjdk\sdk\jre\lib\
ext\ibmpkcs11.jar
ExtClassLoader could not find HelloWorld.class in C:\j9test\testjdk\sdk\jre\lib\
ext\ibmpkcs11impl.jar
ExtClassLoader could not find HelloWorld.class in C:\j9test\testjdk\sdk\jre\lib\
ext\indicim.jar
ExtClassLoader could not find HelloWorld.class in C:\j9test\testjdk\sdk\jre\lib\
ext\jaccess.jar
ExtClassLoader could not find HelloWorld.class in C:\j9test\testjdk\sdk\jre\lib\
ext\jdmpview.jar
ExtClassLoader could not find HelloWorld.class in C:\j9test\testjdk\sdk\jre\lib\
ext\ldapsec.jar
ExtClassLoader could not find HelloWorld.class in C:\j9test\testjdk\sdk\jre\lib\
ext\oldcertpath.jar
ExtClassLoader could not find HelloWorld

AppClassLoader attempting to find HelloWorld

AppClassLoader using classpath C:\test\ras
AppClassLoader found HelloWorld.class in C:\test\ras
AppClassLoader found HelloWorld

The sequence of the loaders output is due to the ″delegate first″ convention of the
class loaders. In this convention, each loader checks its cache, then delegates to its
parent loader. Then, if the parent returns null, the loader checks the file system or
equivalent. This is the part of the process that is reported in the example above. In
the command-line definition, the classname can be given as any Java regular
expression. ″Dic*″ will produce output on all classes whose names begin with the
letters ″Dic″, and so on.

Loading from native code

When a native library is being loaded, how the class that makes the native call is
loaded determines where the loader looks to load the libraries.
v If the class that makes the native call is loaded by the Bootstrap Classloader, this
loader looks in the ’sun.boot.library.path’ to load the libraries.
v If the class that makes the native call is loaded by the Extensions Classloader,
this loader looks in the ’java.ext.dirs’ first, then ’sun.boot.library.path,’ and
finally the ’java.library.path’, to load the libraries.
v If the class that makes the native call is loaded by the Application Classloader,
this loader looks in the ’sun.boot.library.path’, then the ’java.library.path’, to load
the libraries.

258 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 30. Tracing Java applications and the JVM
JVM Trace is a low-overhead trace facility that is provided in all IBM-supplied
JVMs. In most cases, the trace data is kept in compact binary format, with
variable-length trace records from 8 to 64 KB. A cross-platform Java formatter is
supplied to format the trace. You can enable tracepoints at runtime by using levels,
components, group names, or individual tracepoint identifiers.

This chapter describes JVM trace in:

v “What can be traced?”
v “Where does the data go?” on page 260
v “Controlling the trace” on page 261
v “Determining the tracepoint ID of a tracepoint” on page 276
v “Application trace” on page 277

The trace tool provides an extremely powerful ability to diagnose the JVM.

What can be traced?

What can be traced depends on:
v Tracing methods
v Tracing applications
v Internal trace

Tracing methods
You can trace entry to and exit from methods for selected classes. Using the
methods trace option, you can select method trace by class, method name, or both.
Wildcards can be used, and a not operator is provided to allow for complex
selection criteria. Note that this option selects only the methods that are to be
traced. The MT trace component must be selected for a given trace destination. For
example:
-Xtrace:methods={*.*,!java/lang/*.*},print=mt

This routes method trace to stderr for all methods and for all classes except those
that start with java/lang.

Tracing applications
JVM trace contains an application trace facility that allows tracepoints to be placed
in Java code to provide trace data that will be combined with the other forms of
trace. API in the com.ibm.jvm.Trace class is provided to register a Java application
for trace and later to make trace entries. You can control the tracepoints at startup
or enable them dynamically by using Java or C API. When trace is not enabled,
little overhead is caused. Note that an instrumented Java application runs only on
an IBM-supplied JVM.

© Copyright IBM Corp. 2003, 2006 259

What can be traced?

Internal trace
The IBM Virtual Machine for Java is extensively instrumented for trace, as
described in this chapter. Interpretation of this trace data requires knowledge of the
internal operation of the JVM, and is provided for support personnel who
diagnose JVM problems.

Note: No guarantee is given that tracepoints will not vary from release to release
and from platform to platform.

Where does the data go?

Trace data can go into:
v In-storage buffers that can be dumped or snapped when a problem occurs
v One or more files that are using buffered I/O
v An external agent in real-time
v stderr in real time
v A combination of the above

Placing trace data into in-storage buffers

The use of in-storage buffers for trace is a very efficient method of running trace
because no explicit I/O is performed until either a problem is detected, or an API
is used to snap the buffers to a file. Buffers are allocated on a per-thread principle.
This principle removes contention between threads and prevents trace data for
individual threads from being swamped by other threads. For example, if one
particular thread is not being dispatched, its trace information is still available
when the buffers are dumped or snapped. Use -Xtrace:buffers=<size> to control
the size of the buffer that is allocated to each thread.

Note: On some computers, power management affects the timers that trace uses,
and gives misleading information. This problem affects mainly Intel-based
mobiles, but it can occur on other architectures. For reliable timing
information, disable power management.

To examine the trace data, you must snap or dump, then format the buffers.

Snapping buffers
Buffers are snapped when:
v An uncaught Java exception occurs
v An operating system signal or exception occurs
v The com/ibm/jvm/Trace.snap() Java API is called
v The JVMRI TraceSnap function is called
The resulting snap file is placed into the current working directory with a name of
the format Snapnnnn.yyyymmdd.hhmmssth.process.trc, where nnnn is a sequence
number starting at 0001 (at JVM startup), yyyymmdd is the current date, hhmmssth
is the current time, and process is the process identifier.

Dumping buffers
You can also dump the buffers by using the operating system dump services. You
can then extract the buffers from the dump by using the Dump Viewer.

260 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Where does the data go?

Placing trace data into a file

You can write trace data to a file continuously as an extension to the in-storage
trace, but, instead of one buffer per thread, at least two buffers per thread are
allocated. This allows the thread to continue to run while a full trace buffer is
written to disk. Depending on trace volume, buffer size, and the bandwidth of the
output device, multiple buffers might be allocated to a given thread to keep pace
with trace data that is being generated.

A thread is never stopped to allow trace buffers to be written. If the rate of trace
data generation greatly exceeds the speed of the output device, excessive memory
usage might occur and cause out-of-memory conditions. To prevent this, use the
nodynamic option of the buffers trace option. For long running trace runs, a wrap
option is available to limit the file to a given size. See the output option for details.
You must use the trace formatter to format trace data from the file.

Note: Because of the buffering of trace data, if the normal JVM termination is not
performed, residual trace buffers might not be flushed to the file. Snap
dumps do not occur, and the trace bytes are not flushed except when a fatal
operating-system signal is received. The buffers can, however, be extracted
from a system dump if that is available.

External tracing
You can route trace to an agent by using JVMRI TraceRegister. This allows a
callback routine to be invoked when any of the selected tracepoints is found in real
time; that is, no buffering is done. The trace data is in raw binary form.

Tracing to stderr
For lower volume or non-performance-critical tracing, the trace data can be
formatted and routed to stderr in real time. See Chapter 25, “Using method trace,”
on page 219.

Trace combinations
Most trace destinations can be combined, with the same or different trace data
going to different destinations. The exception to this is in-storage trace and trace to
a file, which are mutually exclusive.

Controlling the trace

You can control the trace in several ways:
v By using trace options when launching the JVM
v By using a trace properties file
v By dynamically using Java API
v By using trace trigger events
v By using the C API from inside the JVM
v From an external agent, by using JVMRI
Notes:
1. By default, trace is disabled and cannot be enabled later in the same run. To
use trace, you must specify at least one trace option at startup. If you have
done this, you can then control trace by using various mechanisms later in the
run. Note that by specifying unresettable event logging, you also enable trace.

Chapter 30. Tracing Java applications and the JVM 261

Controlling the trace

2. Whenever the JVM is run, it uses IBM_JAVA_OPTIONS if set.

IBM_JAVA_OPTIONS includes any Java utilities, such as the trace formatter,
the dump extractor, and the dump formatter. If the JVM uses
IBM_JAVA_OPTIONS, unwanted effects or loss of diagnostic data can occur.
For example, if you are Using IBM_JAVA_OPTIONS to trace to a file, that file
might be overwritten when the trace formatter is called. To avoid this problem,
add %d, %p, or %t into the filename to make it unique. Go to “Detailed
descriptions of trace options” on page 264 and see the appropriate trace option
description for more information.

Specifying trace options

The primary way to control trace is through trace options that you specify either
by using the -Xtrace option on the launcher command-line or the
IBM_JAVA_OPTIONS environment variable. Some trace options have the form
<name>, while others are of the form <name>=<value>, where <name> is
case-sensitive. Except where stated, <value> is case insensitive; the exceptions to
this rule are filenames on some platforms, class names, and method names.

If an option value contains commas, it must be enclosed in braces. For example,

methods={java/lang/*,com/ibm/*}

Note that this only applies to options specified on the command-line - not those
specified in a properties file.

The syntax for specifying trace options depends on the launcher. Usually, it is:
java -Xtrace:<name>,<another_name>=<value> HelloWorld

When you use the IBM_JAVA_OPTIONS environment variable, use this syntax:
set IBM_JAVA_OPTIONS=-Xtrace:<name>,<another_name>=<value>

or
export IBM_JAVA_OPTIONS=-Xtrace:<name>,<another_name>=<value>

Trace options summary

This section describes:
v “Options that control tracepoint selection”
v “Options that indirectly affect tracepoint selection” on page 263
v “Triggering and suspend or resume” on page 263
v “Options that specify output files” on page 263
v “MiscellaneousTrace control options” on page 264

Options that control tracepoint selection

These options enable and disable tracepoints. They also determine the destination
for the trace data. In some cases, you must use them with other options. For
example, if you specify maximal or minimal tracepoints, the trace data is put into
in-core buffers. If you are going to send the data to a file, you must use an output
option to specify the destination filename.

These properties have equivalents in the Java and JVMRI API that was mentioned
earlier.

262 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Controlling the trace

Table 13. Options that control tracepoint selection

minimal Trace selected tracepoints (identifier and
timestamp only) to in-core buffer. Associated
trace data is not recorded.
maximal Trace selected tracepoints (identifier and
timestamp and associated data) to in-core
buffer.
count Count the number of times selected
tracepoints are called in the life of the JVM.
print Trace selected tracepoints to stderr with no
indentation.
iprint Trace selected tracepoints to stderr with
indentation.
external Route selected tracepoints to a JVMRI
listener.
exception Trace selected tracepoints to an in-core
buffer reserved for exceptions.

Options that indirectly affect tracepoint selection

These options affect the availability of particular tracepoints but unless you specify
them with a tracepoint selection option, they have no effect other than possibly
degraded performance.
Table 14. Options that indirectly affect tracepoint selection
methods Select classes and methods to trace.
applids Select Java applications that are
instrumented for application trace.

Triggering and suspend or resume

These trace options provide mechanisms to tailor trace and trigger actions at
specified times
Table 15. Triggering and suspend or resume
trigger Trigger events by tracepoint, group or
method entry/exit.
suspend Suspend tracepoints globally (for all
threads).
resume Resume tracepoints globally (not really
useful, but here for completeness).
suspendcount Initial thread suspend count.
resumecount Initial thread resume count.

Options that specify output files

These options determine whether trace data is directed to a file. For the first two
options, you must activate tracepoints by using a tracepoint selection options or
through the various API that were mentioned earlier. If you specify the
state.output option, state trace is enabled automatically.

Chapter 30. Tracing Java applications and the JVM 263

Controlling the trace

Table 16. Options that specify output files

output Select output file name and options for trace
data from tracepoints that were selected
through the minimal and maximal
properties.
exception.output Select output file name and options for trace
data from tracepoints that were selected
through the exception property.
state.output Select output file name and options for state
trace.

MiscellaneousTrace control options

Table 17. MiscellaneousTrace control options
properties Specify a file containing options tor trace.
buffers Modify buffer size and allocation.

Detailed descriptions of trace options

The options are processed in the sequence in which they are described here.
properties[=properties_filespec]
This trace option allows you to specify in a file any of the other trace options,
thereby reducing the length of the invocation command-line. The format of the
file is a flat ASCII/EBCDIC file that contains trace options. If
properties_filespec is not specified, a default name of IBMTRACE.properties
is searched for in the current directory. Nesting is not supported; that is, the
file cannot contain a properties option. If any error is found when the file is
accessed, JVM initialization fails with an explanatory error message and return
code. All the options that are in the file are processed in the sequence in which
they appear in the file, before the next option that is obtained through the
normal mechanism is processed. Therefore, a command-line property always
overrides a property that is in the file.

Note: An existing restriction means that properties that take the form
<name>=<value> cannot be left to default if they are specified in the
property file; that is, you must specify a value, for example maximal=all.

You can make comments as follows:

// This is a comment. Note that it starts in column 1

Examples:
Use IBMTRACE.properties in the current directory:
-Xtrace:properties
Use trace.prop in the current directory:
-Xtrace:properties=trace.prop
Use c:\trc\gc\trace.props:
-Xtrace:properties=c:\trc\gc\trace.props

Here is an example property file:

264 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Controlling the trace

minimal=all
// maximal=st
maximal=cl
buffers=20k
output=c:\traces\classloader.trc
print=tpid(4002,4005)
buffers=nnnk|nnnm[,dynamic|nodynamic]
This option specifies the size of the buffer as nnn KB or MB. This buffer is
allocated for each thread that makes trace entries. If external trace is enabled,
this value is doubled; that is, each thread allocates two or more buffers. The
same buffer size is used for state and exception tracing, but, in this case,
buffers are allocated globally. The default is 8 KB per thread.
The dynamic and nodynamic options have meaning only when tracing to an
output file. If dynamic is specified, buffers are allocated as needed to match
the rate of trace data generation to the output media. Conversely, if nodynamic
is specified, a maximum of two buffers per thread is allocated. The default is
dynamic. The dynamic option is effective only when you are tracing to an
output file.

Important: If nodynamic is specified, you might lose trace data if the volume
of trace data that is produced exceeds the bandwidth of the trace
output file. Message UTE115 is issued when the first trace entry is
lost, and message UTE018 is issued at JVM termination.
Examples:
Dynamic buffering with 8 KB buffers:
-Xtrace:buffers=8k

or in a properties file:
buffers=8k

Trace buffers 2 MB per thread:

-Xtrace:buffers=2m

or in a properties file:
buffers=2m

Trace to only two buffers per thread, each of 128 KB:

-Xtrace:buffers={128k,nodynamic}

or in a properties file:
buffers=128k,nodynamic
applids=application_name[,...]
This option prepares for trace to be enabled for one or more Java applications
that have been instrumented for application trace. The identifier
application_name must match the name under which the application will
register itself. This name can later be used as a component name for tracepoint
selection.
minimal[=[[!]tracepoint_specification[,...]],
maximal[=[[!]tracepoint_specification[,...]], count[=[[!]tracepoint_specification[,...]],
print[=[[!]tracepoint_specification[,...]], iprint[=[[!]tracepoint_specification[,...]],
exception[=[[!]tracepoint_specification[,...]],
external[=[[!]tracepoint_specification[,...]]

Chapter 30. Tracing Java applications and the JVM 265

Controlling the trace

Summary
These options control which individual tracepoints are activated at runtime
and the implicit destination of the trace data. Minimal and maximal trace data
is placed into internal trace buffers that can then be written to a snap file or
written to the files that are specified in an output trace option.
Tracepoints that are activated with count are only counted. The totals are
written to dgTrcCounters in the current directory at JVM termination.
Tracepoints that are activated with print or iprint are routed to stderr.
When exception trace is enabled, the trace data is collected in internal buffers
that are separate from the normal buffers. These internal buffers can then be
written to a snap file or written to the file that is specified in an
exception.output system property.
External trace data is passed to a registered trace listener. Note that all these
properties are independent of each other and can be mixed and matched in
any way that you choose.
Multiple statements of each type of trace are allowed and their effect is
cumulative. Of course, you would have to use a trace properties file for
multiple trace options of the same name.
See “state.output” on page 271 for information about state trace, which is
enabled in a different way, independently of these options.
Types of trace
The minimal option records only the timestamp and tracepoint identifier.
When the trace is formatted, missing trace data is replaced with the characters
″???″ in the output file. The maximal option specifies that all associated data is
traced. If a tracepoint is activated by both trace options, maximal trace data is
produced. Note that these types of trace are completely independent from any
types that follow them. For example, if the minimal option is specified, it does
not affect a later option such as print.
The count option requests that a count of the selected tracepoints is kept. At
JVM termination, all non-zero totals of tracepoints (sorted by tracepoint id) are
written to a file, called utTrcCounters, in the current directory. This information
is useful if you want to determine the overhead of particular tracepoints, but
do not want to produce a large amount (GB) of trace data.
The print option causes the specified tracepoints to be routed to stderr in
real-time. The tracepoints are formatted by J9TraceFormat.dat, which must be
available at runtime. TraceFormat.dat is shipped in sdk/jre/lib and is
automatically found by the runtime.
The exception option allows low-volume tracing in buffers and files that are
distinct from the higher-volume information that minimal and maximal
tracing have provided. In most cases, this information is exception-type data,
but you can use this option to capture any trace data that you want.
This form of tracing is channeled through a single set of buffers, as opposed to
the buffer-per-thread approach for normal trace, and buffer contention might
occur if high volumes of trace data are collected. A difference exists in the
tracepoint_specification defaults for exception tracing; see “Tracepoint
selection” on page 267.

Note: When exception trace is entered for an active tracepoint, the current
thread id is checked against the previous caller’s thread id. If it is a

266 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Controlling the trace

different thread, or this is the first call to exception trace, a context

tracepoint is put into the trace buffer first. This context tracepoint
consists only of the current thread id. This is necessary because of the
single set of buffers for exception trace. (The formatter identifies all trace
entries as coming from the ″Exception trace pseudo thread″ when it
formats exception trace files.)
The external option channels trace data to registered trace listeners in
real-time. JVMRI is used to register or deregister as a trace listener. If no
listeners are registered, this form of trace does nothing except waste machine
cycles on each activated tracepoint.
Tracepoint selection:
If no qualifier parameters are entered, all tracepoints are enabled, except for
exception trace, where the default is all (exception).
The tracepoint_specification is as follows:
v [!]component[(type[,...])] or [!]tpid(tracepoint_id[,...]) where

! is a logical not. That is, the tracepoints that are specified

immediately following the ! are turned off.
component is one of:
v ALL.
v The JVM subcomponent (that is, dg, j9trc, j9vm, j9mm, j9bcu,
j9vrb, java,awt, awt_dnd_datatransfer, Audio, mt, fontmanager,
net, awt_java2d, awt_print, or nio).
v A group of tracepoints that have been specified by use of a group
name. For example, nativeMethods would select the group of
tracepoints in MT (Method Trace) that relate to native methods.
The following groups are supported:
compiledMethods
nativeMethods
staticMethods
type is the tracepoint type or all. The default is all, except for exception
tracing, where the default is exception. The following types are
supported:
v Entry
v Exit
v Event
v Exception
v Mem
tracepoint_id is the hexadecimal global tracepoint identifier. You can omit leading
zeroes. You can specify a range of tracepoints by using a hyphen
(dash, minus); for example, tpid(18007,c003-c01f).

Note: Some tracepoints can be both an exit and an exception; that is, the
function ended with an error. If you specify either exit or exception,
these tracepoints will be included.

Examples:
All tracepoints:
-Xtrace:maximal
All tracepoints except j9vrb and j9trc:

Chapter 30. Tracing Java applications and the JVM 267

Controlling the trace

-Xtrace:minimal={all,!j9vrb,!j9trc}
All entry and exit tracepoints in j9bcu:
-Xtrace:maximal={j9bcu(entry,exit)}
All tracepoints in j9mm except 4000, 4001, 4002, 4003:
-Xtrace:maximal={j9mm,!tpid(4000,4001,4002,4003)}
Tracepoints 18005 through 1801f and c003:
-Xtrace:print={tpid(18005-1801f,c003)}
All j9trc tracepoints:
-Xtrace:count=j9trc
All entry and exit tracepoints:
-Xtrace:external={all(entry,exit)}
All exception tracepoints:
-Xtrace:exception
All exception tracepoints:
-Xtrace:exception=all(exception)
All exception tracepoints in j9bcu:
-Xtrace:exception=j9bcu
Tracepoints c03e through c113:
-Xtrace:exception=tpid(c03e-c113)

Trace levels

Tracepoints have been assigned levels 0 through 9 that are based on the
importance of the tracepoint. A level 0 tracepoint is very important and is
reserved for extraordinary events and errors; a level 9 tracepoint is in-depth
component detail. To specify a given level of tracing, the level0 through level9
keywords are used. You can abbreviate these keywords to l0 through l9. For
example, if level5 is selected, all tracepoints that have levels 0 through 5 are
included. Level specifications do not apply to explicit tracepoint specifications
that use the TPID keyword.

The default is level 9.

You can use these keywords either before the tracepoint selection, or as a type
modifier. When a keyword is used before a tracepoint selection, that keyword
applies to all tracepoint selection criteria that follow it in the trace option. For
example:
-Xtrace:maximal={level5,j9mm,j9trc,j9bcu,level1,all}

or
-Xtrace:maximal={l5,j9mm,j9trc,j9bcu,l1,all}

In this example, tracepoints that have a level of 5 or below are enabled for the
j9mm, j9trc, and j9bcu components. Tracepoints that have a level of 1 or below
are enabled for all the other components. Note that the level applies only to
the current statement, so if multiple trace selection statements appear in a trace
properties file, the level is reset to the default for each new statement.

Alternatively, you can specify levels as a type modifier. In this case, the level
applies only to the component with which it is associated. The following
example is functionally equivalent to the global example that is shown above:
-Xtrace:maximal={j9mm(level5),j9trc(level5),j9bcu(level5),all(level1)}

268 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Controlling the trace

or
-Xtrace:maximal={j9mm(l5),j9trc(l5),j9bcu(l5),all(l1)}

Level specifications do not apply to explicit tracepoint specifications that use

the TPID keyword.

When the not operator is specified, the level is inverted; that is, !j9mm(level5)
disables all tracepoints of level 6 or above for the j9mm component. For
example:
-Xtrace:print={all,!j9trc(l5),!j9mm(l6)}

enables trace for all components at level 9 (the default), but disables level 6
and above for the locking component, and level 7 and above for the storage
component.

Examples:
Count all level zero and one tracepoints hit:
-Xtrace:count=all(l1)
Produce maximal trace of all components at level 5 and j9mm at level 9:
-Xtrace:maximal={LeVeL5,all,j9mm(L9)}
Trace all components at level 6, but do not trace j9vrb at all, and do not
trace level 0 through 3 entry and exit tracepoints in the j9trc component:
-Xtrace:minimal={all(l6),!j9vrb,!j9trc(level3,entry,exit)}
methods=method_specification[,...]
This trace option identifies which classes and methods are to be prepared to be
traced. You can then trace these methods by selecting the MT component
though the normal trace selection mechanism. When more than one
specification is made, it is cumulative, as if processed from left to right.
Although method trace works with the JIT on, input parameters cannot be
traced if the JIT is active.

Important: This trace option selects only the methods that are to be traced.
You must use one of the trace selection properties to select the
tracepoints that are in the MT component.
The method_specification is:
v [!][*]class[*][.[*]method[*]][()], where

! is a logical not. That is, the class or methods that are specified
immediately following the ! are deselected for method trace.
* is a wildcard that can appear at the beginning, end, or both, of the
class and method names.
class is the package or class name. Note that the delimiter between parts of
the package name is a forward slash, ’/’, even on platforms like
Windows that use a backward slash as a path delimiter.
. is the delimiter between the class and method.
method is the method name.

Examples:
Select all methods for all classes and print them with indentation:
-Xtrace:methods=*,iprint=mt

Chapter 30. Tracing Java applications and the JVM 269

Controlling the trace

All methods that are in java/lang/String. The trace data will be placed in
internal buffers:
-Xtrace:methods=java/lang/String.*,maximal=mt
All methods that contain a ″y″ in classes that start with com/ibm and print
them:
-Xtrace:methods=com/ibm*.*y*,print=mt
All methods that contain a ″y″ and do not start with an ″n″ in classes that
start with com/ibm and print them:
-Xtrace:methods={com/ibm*.*y*,!n*},iprint=mt
output=trace_filespec[,nnnm[,generations]]
This trace options indicates that minimal, or maximal trace data, or both, must
be sent to trace_filespec. If the file does not already exist, it is created
automatically. If it does already exist, it is overwritten.
Optionally:
v You can limit the file to nnn MB, at which point it wraps nondestructively to
the beginning. If you do not limit the file, it grows until all disk space has
been used.
v If you want the final trace filename to contain today’s date, the PID number
that produced the trace, or the time, do one of the following steps as
appropriate (see also the examples at the end of this section).
– To include today’s date (in ″yyyymmdd″ format) in the trace filename,
specify ″%d″ as part of the trace_filespec.
– To include the pidnumber of the process that is generating the tracefile,
specify ″%p″ as part of the trace_filespec.
– To include the time (in 24-hour hhmmss format) in the trace filename,
specify ″%t″ as part of the trace_filespec.
v You can specify generations as a value 2 through 36. These values cause up
to 36 files to be used in a round-robin way when each file reaches its size
threshold. When a file needs to be reused, it is overwritten. Therefore, if x
generations of n MB files are specified, the worst case is that only ((x - 1) * n
÷ x ) MB of trace data might be available. If generations is specified, the
filename must contain a ″#″ (hash, pound symbol), which will be substituted
with its generation identifier, the sequence of which is 0 through 9 followed
by A through Z.

Note: When tracing to a file, buffers for each thread are written when the
buffer is full or when the JVM terminates. If a thread has been inactive
for a period of time before JVM termination, what seems to be ’old’
trace data is written to the file. When formatted, it then seems that trace
data is missing from the other threads, but this is an unavoidable
side-effect of the buffer-per-thread design. This effect becomes especially
noticeable when you use the generation facility, and format individual
earlier generations.

Examples:
Trace output goes to /u/traces/gc.problem; no size limit:
-Xtrace:output=/u/traces/gc.problem
Output goes to trace and will wrap at 2 MB:
-Xtrace:output={trace,2m}
Output goes to gc0.trc, gc1.trc, gc2.trc, each 10 MB in size:
-Xtrace:output={gc#.trc,10m,3}

270 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Controlling the trace

Output filename contains today’s date in yyyymmdd format (for example,

traceout.20041025.trc):
-Xtrace:output=traceout.%d.trc
Output file contains the number of the process (the PID number) that
generated it (for example, tracefrompid2112.trc):
-Xtrace:output=tracefrompid%p.trc
Output filename contains the time in hhmmss format (for example,
traceout.080312.trc):
-Xtrace:output=traceout.%t.trc
exception.output=exception_trace_filespec[,nnnm]
This trace option indicates that exception trace data should be directed to
exception_trace_filespec. If the file does not already exist, it is created
automatically. If it does already exist, it is overwritten. Optionally, you can
limit the file to nnn MB, at which point it wraps nondestructively to the
beginning. If you do not limit the file, it grows until all disk space has been
used.
Optionally, if you want the final trace filename to contain today’s date, the PID
number that produced the trace, or the time, do one of the following steps as
appropriate (see also the examples at the end of this section).
v To include today’s date (in ″yyyymmdd″ format) in the trace filename,
specify ″%d″ as part of the exception_trace_filespec.
v To include the pidnumber of the process that is generating the tracefile,
specify ″%p″ as part of the exception_trace_filespec.
v To include the time (in 24-hour hhmmss format) in the trace filename,
specify ″%t″ as part of the exception_trace_filespec.

Examples:
Trace output goes to /u/traces/exception.trc. No size limit:
-Xtrace:exception.output=/u/traces/exception.trc
Output goes to except and wraps at 2 MB:
-Xtrace:exception.output={except,2m}
Output filename contains today’s date in yyyymmdd format (for example,
traceout.20041025.trc):
-Xtrace:exception.output=traceout.%d.trc
Output file contains the number of the process (the PID number) that
generated it (for example, tracefrompid2112.trc):
-Xtrace:exception.output=tracefrompid%p.trc
Output filename contains the time in hhmmss format (for example,
traceout.080312.trc):
-Xtrace:exception.output=traceout.%t.trc
state.output=state_trace_filespec[,nnnm]
This trace option indicates that “state” information should be captured in
state_trace_filespec. The state trace captures information about the JVM that
could be useful later, when the normal trace files or internal buffers have
wrapped many times.
Examples of state data might be:
v Interned string values and ids
v Classblock address or name correlation
v Methodblock address or name correlation

Chapter 30. Tracing Java applications and the JVM 271

Controlling the trace

A tracepoint is designated as a state-type tracepoint in the TDF at build time.

Note that you can also route the tracepoint to another trace destination, such
as print, if specified.

State trace differs from other forms of trace in that it is either totally on or off.
You cannot control which individual tracepoints are enabled at runtime. By
specifying this trace option, you turn it on. If state_trace_filespec does not
already exist, it is created automatically. If it does already exist, it is
overwritten. If nnn is not specified, the size of the file is not limited. If nnn is
specified, two files are created. The first file is named state_file_filespec with a
0 (zero) suffix and contains up to nnn MB of state information that is never
lost; that is, it never wraps. The second file is named state_file_filespec with a
1 (one) suffix and contains up to nnn MB of state information that wraps; that
is, state information might be lost. State trace captures all the startup state
information and all the latest state information. The file 0 and 1 filename
qualifiers position can optionally be controlled by the inclusion of a # (hash or
pound sign) in the filename; the # will be replaced by 0 or 1 respectively.
Under normal conditions, nnn should not be specified, but in the case of
long-running JVMs, its use might be unavoidable to limit the file size. In this
case, some useful state data could be lost.

Optionally, if you want the final trace filename to contain today’s date, the PID
number that produced the trace, or the time, do one of the following steps as
appropriate (see also the examples at the end of this section).
v To include today’s date (in ″yyyymmdd″ format) in the trace filename,
specify ″%d″ as part of the state_trace_filespec.
v To include the pidnumber of the process that is generating the tracefile,
specify ″%p″ as part of the state_trace_filespec.
v To include the time (in 24-hour hhmmss format) in the trace filename,
specify ″%t″ as part of the state_trace_filespec.

Examples:
Trace output goes to /u/traces/state; no size limit:
-Xtrace:state.output=/u/traces/state
Output goes to state0 for 4 MB, then state1, wrapping at 4 MB:
-Xtrace:output={state,4m}
Output goes to state0.trc for 4 MB, then state1.trc:
-Xtrace:state.output={state#.trc,4m}
Output filename contains today’s date in yyyymmdd format (for example,
traceout.20041025.trc):
-Xtrace:state.output=traceout.%d.trc
Output file contains the number of the process (the PID number) that
generated it (for example, tracefrompid2112.trc):
-Xtrace:state.output=tracefrompid%p.trc
Output filename contains the time in hhmmss format (for example,
traceout.080312.trc):
-Xtrace:state.output=traceout.%t.trc
suspend
Suspends tracing globally (for all threads and all forms of tracing) but leaves
tracepoints activated.
Example:

272 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Controlling the trace

Tracing suspended:
-Xtrace:suspend
resume
Resumes tracing globally. Note that suspend and resume are not recursive.
That is, two suspends that are followed by a single resume cause trace to be
resumed.
Example: Trace resumed (not much use as a startup option):
-Xtrace:resume
suspendcount=<count>
This trace option is for use with the trigger option (see “trigger”).
This suspendcount=<count> trace option determines whether tracing is
enabled for each thread. If <count> is greater than zero, each thread initially
has its tracing enabled and must receive <count> suspend this action before it
stops tracing.

Note: You cannot use resumecount and suspendcount together because they
both set the same internal counter.

Example:
Start with all tracing turned on. Each thread stops tracing when it has had
three suspendthis actions performed on it:
-Xtrace:suspendcount=3
resumecount=count
This system property is for use with the trigger property (see “trigger”).
This resumecount=<count> system property determines whether tracing is
enabled for each thread. If <count> is greater than zero, each thread initially
has its tracing disabled and must receive <count> resume this action before it
starts tracing.

Note: You cannot use resumecount and suspendcount together because they
both set the same internal counter.

Example:
Start with all tracing turned off. Each thread starts tracing when it has had
three resumethis actions performed on it:
-Xtrace:resumecount=3
trigger=clause[,clause][,clause]...
This trace option determines when various triggered trace actions should
occur. Supported actions include turning tracing on and off for all threads,
turning tracing on or off for the current thread, or producing a variety of
dumps.

Note: This trace option does not control what is traced. It controls only
whether what has been selected by the other trace options is produced
as normal or is blocked.
Each clause of the trigger option can be tpid(...), method(...), group(...), or
threshold(). You can specify multiple clauses of the same type if required, but
you do not need to specify all types. The clause types are:
method(methodspec,[entryAction],[exitAction][,delayCount][,matchcount])
On entering a method that matches methodspec, perform the specified

Chapter 30. Tracing Java applications and the JVM 273

Controlling the trace

entryAction. On leaving it, perform the specified exitAction. If you specify

a delayCount, the actions are performed only after a matching methodspec
has been entered that many times. If you specify a matchCount,
entryAction, and exitAction will be performed at most that many times.
group(groupname,action[,delayCount][,matchcount])
On finding any active tracepoint that is defined as being in trace group
groupname, perform the specified action. If you specify a delayCount, the
action is performed only after that many active tracepoints from group
groupname have been found. If you specify a matchCount, action will be
performed at most that many times.
tpid(tpid|tpidRange,action[,delayCount][,matchcount])
On finding the specified active tpid (tracepoint id) or a tpid that falls
inside the specified tpidRange, perform the specified action. If you specify
a delayCount, the action is performed only after the JVM finds such an
active tpid that many times. If you specify a matchCount, action will be
performed at most that many times.

Actions:

Wherever an action must be specified, you must select from the following
choices:
suspend
Suspend ALL tracing (except for special trace points).
resume
Resume ALL tracing (except for threads that are suspended by the
action of the resumecount property and Trace.suspendThis() calls).
suspendthis
Increment the suspend count for this thread. If the suspend-count is
greater than zero, all tracing for this thread is prevented.
resumethis
Decrement the suspend count for this thread. If the suspend-count is
zero or below, tracing for this thread is resumed.
coredump (or sysdymp)
Produce a coredump.
javadump
Produce a javadump or javacore.
heapdump
Produce a heap dump (see Chapter 22, “Using Heapdump,” on page
205).
snap Snap all active trace buffers to a file in the current working directory.
The name of the file is in the format
Snapnnnn.yyyymmdd.hhmmssth.ppppp.trc, where nnnn is the sequence
number of the snap file since JVM startup, yyyymmdd is the date,
hhmmssth is the time, and ppppp is the process id in decimal with
leading zeroes removed.
abort Halt the execution of the JVM.
segv Cause a segmentation violation. (Intended for use in debugging.)

Examples:

274 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Controlling the trace

v Start tracing this thread when it enters any method in java/lang/String and
stop tracing when it leaves it:
-Xtrace:resumecount=1
-Xtrace:trigger={method(java/lang/String.*,resumethis,suspendthis)}
v Resume all tracing when any thread enters a method in any class that starts
with “error”:
-Xtrace:trigger={method(*.error*,resume)}
v When you reach the 1000th and 1001st tracepoint from the ″jvmri″ trace
group, produce a core dump.

Note: Without matchcount there would be a risk of filling your disk with
coredump files.
-Xtrace:trigger={group(staticmethods,coredump,1000,2)}

If using the trigger option generates multiple dumps in rapid succession

(more than one per second), specify a dump option to guarantee unique
dump names. See Chapter 24, “Using dump agents,” on page 213 for more
information.
v Trace (all threads) while my application is active only; that is, not startup or
shutdown. (The application name is “HelloWorld”):
-Xtrace:suspend,trigger={method(HelloWorld.main,resume,suspend)}

Using the trace formatter

The trace formatter is a Java program that runs on any platform and can format a
trace file from any platform. The formatter, which is shipped with the SDK in
core.jar, also requires a file called TraceFormat.dat, which contains the formatting
templates. This file is shipped in jre/lib.

Invoking the trace formatter

Type:
java com.ibm.jvm.format.TraceFormat input_filespec [output_filespec] [options]

where com.ibm.jvm.format.TraceFormat is the traceformatter class, input_filespec

is the name of the binary trace file to be formatted, output_filespec is the optional
output filename. If it is not specified, the default output file name is
input_filespec.fmt.

The options are:

v summary specifies that a summary of the trace file is printed.
v entries:comp[,...] specifies that only trace entries for component comp are to
be formatted.
v thread:threadid,... specifies that only entries for threadid are to be formatted
(threadid is specified as 0xnnnnnnnn).
v indent specifies that the trace data is to be indented on entry type tracepoints
and outdented on exit type tracepoints. This might produce undesirable results
on selective traces where, for example, exits from a function are not traced, but
entries are.
v symbolic specifies that the symbolic name of the tracepoint is embedded in the
trace output. This is useful where the descriptive text for a particular tracepoint
does not make clear what is being traced.

Examples of formatting binary trace file trace1:

v Produce a summary of the trace file:

Chapter 30. Tracing Java applications and the JVM 275

Controlling the trace

java com.ibm.jvm.format.TraceFormat trace1 -summary

v Format trace1 using the formatting templates (TraceFormat.dat) that are in
d:\formats:
java -Xtrace:format=d:\formats com.ibm.jvm.format.TraceFormat trace1
v Format trace1 indenting for entry tracepoints and outdenting for exits:
java com.ibm.jvm.format.TraceFormat trace1 -indent
v Format only the trace information in trace1 that originated from the XE
component:
java com.ibm.jvm.format.TraceFormat trace1 -entries:xe
v Format only the trace information in trace1 that originated from the thread that
has an execenv address of 0x7ffee00:
java com.ibm.jvm.format.TraceFormat trace1 -thread:0x7ffee00 -indent

Trace properties file

You can use properties files to control trace; this saves typing, and, over time,
causes a library of these files to be created, with each file tailored to solving
problems in a particular area. You can remove unwanted tracepoints by using the
!TPID(xxxxxx) parameter.

What to trace
JVM trace can produce large amounts of data in a very short time. Before running
trace, think carefully about what information you need to solve the problem. In
many cases, you need only the trace information that is produced shortly before
the problem occurs; you should consider using the wrap option. Also, in many
cases, it is enough to use internal trace with an increased buffer size and snap the
trace when the problem occurs. If the problem results in a thread stack dump or
operating system signal or exception, trace buffers are snapped automatically to a
file that is in the current directory. The file is called:
Snapnnnn.yyyymmdd.hhmmssth.process.trc.

You must also think carefully about which components need to be traced and what
level of tracing is required. For example, if you are tracing a suspected garbage
collection problem, it might be enough to trace all components at level 1 or 3, and
ST at level 9, while maximal can be used to show parameters and other
information for the failing component.

Determining the tracepoint ID of a tracepoint

Each tracepoint has a unique 3-byte identifier (6 hex digits). This identifier relates
the tracepoint in the code to its entry in the format file (jre/lib/TraceFormat.dat).
You can use the identifier to select individual tracepoints at runtime by using the
TPID keyword. The tracepoint ID can be looked up in the format file, which has
the following format:

The first line is an internal version number.

Following the version number is a component name, followed by a line for each
tracepoint defined in that component, the format of which for this JVM is: nnnnnn
t o l e symbolic_name ″tracepoint_formatting_template″ where nnnnnn is the hex
tracepoint ID, t is the tracepoint type (0 through 11), o is the overhead (0 through
10) , l is the level of the tracepoint (0 through 9, or - if the tracepoint is obsolete) , e
is the explicit setting flag (Y/N), symbolic_name is the name of the tracepoint,
tracepoint_formatting_template is the template used to format the entry.

276 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Controlling the trace

Note that this is subject to change without notice, but the version number will be
different.

Application trace
Application trace allows you to trace Java applications using the JVM Trace
Facility.

You must register your Java application with application trace and add trace calls
where appropriate. When you run your application, the trace calls are treated as
no-operations. Unless you specified application trace at startup, the overhead is
very low. If you have started application trace, you can enable or disable
individual tracepoints at any time.

Note: If you are running your application with application trace or do not specify
the -Xtrace option, you might see the following error message:
JVMJ9VM034E jvmri requires trace engine: run with -Xtrace flag

You can either ignore this message or prevent it by using the -Xtrace option.

Implementing application trace

Application trace is in the package com.ibm.jvm.Trace. The application trace API is
described in this section.

Registering for trace

int registerApplication(String application_name, String[] format_template)

Use the registerApplication() method to specify the application to register with

application trace. The application_name argument is the name of the application
you want to trace; application_name must be the same as the one that you specify
at JVM startup – that is, the application that you want to trace. The
format_template argument is an array of printf-like format strings. You can specify
templates of up to 16 KB. The position in the array determines the tracepoint
identifier (starting at 0). You can use these identifiers to enable specific tracepoints
at runtime. The first character of each template identifies the type of tracepoint
(entry, exit, event, exception or exception exit) followed by a blank, followed by the
format string. The trace types are defined as statics in the Trace class:

public static final String EVENT= ″0 ″;

public static final String EXCEPTION= ″1 ″;
public static final String ENTRY= ″2 ″;
public static final String EXIT= ″4 ″;
public static final String EXCEPTION_EXIT= ″5 ″;

The registerApplication() method returns an integer that you must use on further
trace() calls. If tracing of the application is enabled, the integer returned is positive;
otherwise, it is -1.

Tracepoints
The following trace methods are implemented:

void trace(int handle, int traceId);

void trace(int handle, int traceId, String s1);
void trace(int handle, int traceId, String s1, String s2);
void trace(int handle, int traceId, String s1, String s2, String s3);

Chapter 30. Tracing Java applications and the JVM 277

Controlling the trace

void trace(int handle, int traceId, String s1, Object o1);

void trace(int handle, int traceId, Object o1, String s1);
void trace(int handle, int traceId, String s1, int i1);
void trace(int handle, int traceId, int i1, String s1);
void trace(int handle, int traceId, String s1, long l1);
void trace(int handle, int traceId, long l1, String s1);
void trace(int handle, int traceId, String s1, byte b1);
void trace(int handle, int traceId, byte b1, String s1);
void trace(int handle, int traceId, String s1, char c1);
void trace(int handle, int traceId, char c1, String s1);
void trace(int handle, int traceId, String s1, float f1);
void trace(int handle, int traceId, float f1, String s1);
void trace(int handle, int traceId, String s1, double d1);
void trace(int handle, int traceId, double d1, String s1);
void trace(int handle, int traceId, Object o1);
void trace(int handle, int traceId, Object o1, Object o2);
void trace(int handle, int traceId, int i1);
void trace(int handle, int traceId, int i1, int i2);
void trace(int handle, int traceId, int i1, int i2, int i3);
void trace(int handle, int traceId, long l1);
void trace(int handle, int traceId, long l1, long l2);
void trace(int handle, int traceId, long l1, long l2, long i3);
void trace(int handle, int traceId, byte b1);
void trace(int handle, int traceId, byte b1, byte b2);
void trace(int handle, int traceId, byte b1, byte b2, byte b3);
void trace(int handle, int traceId, char c1);
void trace(int handle, int traceId, char c1, char c2);
void trace(int handle, int traceId, char c1, char c2, char c3);
void trace(int handle, int traceId, float f1);
void trace(int handle, int traceId, float f1, float f2);
void trace(int handle, int traceId, float f1, float f2, float f3);
void trace(int handle, int traceId, double d1);
void trace(int handle, int traceId, double d1, double d2);
void trace(int handle, int traceId, double d1, double d2, double d3);
void trace(int handle, int traceId, String s1, Object o1, String s2);
void trace(int handle, int traceId, Object o1, String s1, Object o2);
void trace(int handle, int traceId, String s1, int i1, String s2);
void trace(int handle, int traceId, int i1, String s1, int i2);
void trace(int handle, int traceId, String s1, long l1, String s2);
void trace(int handle, int traceId, long l1, String s1, long l2);
void trace(int handle, int traceId, String s1, byte b1, String s2);
void trace(int handle, int traceId, byte b1, String s1, byte b2);
void trace(int handle, int traceId, String s1, char c1, String s2);
void trace(int handle, int traceId, char c1, String s1, char c2);
void trace(int handle, int traceId, String s1, float f1, String s2);
void trace(int handle, int traceId, float f1, String s1, float f2);
void trace(int handle, int traceId, String s1, double d1, String s2);
void trace(int handle, int traceId, double d1, String s1, double d2);

The handle argument is the value returned by the registerApplication() method.

The traceId argument is the number of the template entry starting at 0.

Example HelloWorld with application trace

The code below illustrates a “HelloWorld” application with application trace:

278 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Controlling the trace

import com.ibm.jvm.Trace;
public class HelloWorld
{
static int handle;
static String[] templates;
public static void main (String[] args)
{
templates = new String[8];
templates[0] = Trace.ENTRY + "Entering %s";
templates[1] = Trace.EXIT + "Exiting %s";
templates[2] = Trace.EVENT + "Event id %d, text = %s";
templates[3] = Trace.EXCEPTION + "Exception: %s";
templates[4] = Trace.EXCEPTION_EXIT + "Exception exit from %s";

// Register a trace application called HelloWorld

handle = Trace.registerApplication("HelloWorld", templates);
// Set any tracepoints requested on command line
for (int i = 0; i < args.length; i++) {
System.err.println("Trace setting: "+ args[i]);
Trace.set(args[i]);
}
// Trace something....
Trace.trace(handle, 2, 1, "Trace initialized");
// Call a few methods...
sayHello();
sayGoodbye();
}
private static void sayHello()
{
Trace.trace(handle, 0, "sayHello");
System.out.println("Hello");
Trace.trace(handle, 1, "sayHello");
}

private static void sayGoodbye()

{
Trace.trace(handle, 0, "sayGoodbye");
System.out.println("Bye");
Trace.trace(handle, 4, "sayGoodbye");
}
}

Using application trace at runtime

At runtime, you can enable one or more applications for application trace. For
example, in the case of the “HelloWorld” application described above:
java -Xtrace:applids=HelloWorld HelloWorld iprint=HelloWorld

The applids=HelloWorld option of –Xtrace specifies that an application called

HelloWorld is enabled for trace. Note that enabling the application for trace does
not turn on any tracepoints. The HelloWorld example uses the Trace.set() API to
pass any arguments to trace, enabling all of the HelloWorld tracepoints to be
routed to stderr. Invoking the HelloWorld application in this way outputs:
Trace setting: iprint=HelloWorld
09:50:29.417*0x2a08a00 084002 - Event id 1, text = Trace initialized
09:50:29.417 0x2a08a00 084000 > Entering sayHello
Hello
09:50:29.427 0x2a08a00 084001 < Exiting sayHello
09:50:29.427 0x2a08a00 084000 > Entering sayGoodbye
Bye
09:50:29.437 0x2a08a00 084004 * < Exception exit from sayGoodbye

You can obtain a similar result by specifying iprint on the command line:
java -Xtrace:applids=HelloWorld, iprint=HelloWorld HelloWorld

Chapter 30. Tracing Java applications and the JVM 279

Controlling the trace

You can enable Individual tracepoints like this:

java -Xtrace:applids=HelloWorld,iprint={HelloWorld(0,1)} HelloWorld

For details, see Table 13 on page 263.

Printf specifiers
Application trace supports the ANSI C printf specifiers. You must be careful when
you select the specifier; otherwise you might get unpredictable results, including
abnormal termination of the JVM.

For 64-bit integers, you must use the ll (lower case LL, meaning long long)
modifier. For example: %lld or %lli.

For pointer-sized integers use the z modifier. For example: %zx or %zd.

Using the Trace API

There are a number of ways that you can dynamically control trace from a Java
application by using the com.ibm.jvm.Trace class.

Activating and deactivating tracepoints:

int set(String cmd);

The Trace.set() method allows a Java application to select tracepoints dynamically.

For example:
Trace.set(“iprint=all”);

The syntax is the same as that used in a trace properties file for the print, iprint,
count, maximal, minimal and external trace options.

Obtaining snapshots of trace buffers:

void snap();

This method causes all active trace buffers to be written to a unique filename. You
must have activated trace previously with the maximal or minimal options and
without the out option.

Suspending or resuming trace:

void suspend();

The Trace.suspend() method suspends tracing for all the threads in the JVM. It is
not recursive.

void resume();

The Trace.resume() method resumes tracing for all threads in the JVM. It is not
recursive.

void suspendThis();

The Trace.suspendThis() method decrements the suspend and resume count for the
current thread and suspends tracing the thread if the result is negative.

void resumeThis();

280 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Controlling the trace

The Trace.resumeThis() method increments the suspend and resume count for the
current thread and resumes tracing the thread if the result is not negative.

Chapter 30. Tracing Java applications and the JVM 281

Controlling the trace

282 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 31. Using the Reliability, Availability, and
Serviceability Interface
The JVM Reliability, Availability, and Serviceability Interface (JVMRI) allows an
agent to access reliability, availability, and serviceability (RAS) functions by using a
structure of pointers to functions. You can use the interface to:
v Determine the trace capability that is present
v Set and intercept trace data
v Produce various dumps
v Inject errors

You need some programming skills to use the JVMRI. You must be able to build a
native library, add the code for JVMRI callbacks (described below), and interface
the code to the JVM through the JNI. This book provides the callback code but
does not provide the other programming information.

This chapter describes the JVMRI in:

v “Preparing to use JVMRI”
v “JVMRI functions” on page 286
v “API calls provided by JVMRI” on page 286
v “RasInfo structure” on page 292
v “RasInfo request types” on page 293
v “Intercepting trace data” on page 293
v “Calling external trace” on page 294
v “Formatting” on page 294

Preparing to use JVMRI

Before you can use the JVMRI, enable the trace engine using the -Xtrace option
along with any relevant options. See Appendix F, “Command-line options,” on
page 329 for more information.

Writing an agent
The following piece of code demonstrates how to write a very simple JVMRI
agent. When an agent is loaded by the JVM, the first thing that gets called is the
entry point routine JVM_OnLoad(). Therefore, your agent must have a routine
called JVM_OnLoad(). This routine then must obtain a pointer to the JVMRI
function table. This is done by making a call to the GetEnv() function.
/* jvmri - jvmri agent source file. */

#include "jni.h"
#include "jvmri.h"

DgRasInterface *jvmri_intf = NULL;

JNIEXPORT jint JNICALL

JVM_OnLoad(JavaVM *vm, char *options, void *reserved)
{
int rc;
JNIEnv *env;

© Copyright IBM Corp. 2003, 2006 283

Writing an agent

/*
* Get a pointer to the JNIEnv
*/

rc = (*vm)->GetEnv(vm, (void **)&env, JNI_VERSION_1_2);

if (rc != JNI_OK) {
fprintf(stderr, "RASplugin001 Return code %d obtaining JNIEnv\n", rc);
fflush(stderr);
return JNI_ERR;
}

/*
* Get a pointer to the JVMRI function table
*/

rc = (*vm)->GetEnv(vm, (void **)&jvmri_intf, JVMRAS_VERSION_1_3);

if (rc != JNI_OK) {
fprintf(stderr, "RASplugin002 Return code %d obtaining DgRasInterface\n", rc);
fflush(stderr);
return JNI_ERR;
}

/*
* Now a pointer to the function table has been obtained we can make calls to any
* of the functions in that table.
*/

.........................................................

return rc;
}

Registering a trace listener

Before you start using the trace listener, you must set the -Xtrace option with the
relevant external=tp_spec information to inform the object of the tracepoints for
which it should listen. See Appendix F, “Command-line options,” on page 329 for
more information.

An agent can register a function that is called back when the JVM makes a trace
point. The following example shows a trace listener that only increments a counter
each time a trace point is taken.
void JNICALL
listener(void *env, void ** tl, unsigned int traceId, const char * format,
va_list var)
{
int *counter;

if (*tl == NULL) {
fprintf(stderr, "RASplugin100 first tracepoint for thread %p\n", env);
*tl = (void *)malloc(4);
counter = (int *)*tl;
*counter = 0;
}

counter = (int *)*tl;

(*counter)++;

fprintf(stderr, "Trace point total = %d\n", *counter );

}

Add this code to the JVM_Onload() function or a function that it calls.

284 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Registering a trace listener

The following example is used to register the above trace listener.

/*
* Register the trace listener
*/

rc = jvmri_intf->TraceRegister(env, listener);
if (rc != JNI_OK) {
fprintf(stderr, "RASplugin003 Return code %d registering listener\n", rc);
fflush(stderr);
return JNI_ERR;
}

You can also do more difficult operation with a trace listener, including formatting
the trace point information yourself then displaying this or perhaps recording it in
a file or database

Changing trace options

This example uses the TraceSet() function to change the JVM trace setting. It makes
the assumption that the options string that is specified with the -Xrun option and
passed to JVM_Onload() is a trace setting.
/*
* If an option was supplied, assume it is a trace setting
*/

if (options != NULL && strlen(options) > 0) {

rc = jvmri_intf->TraceSet(env, options);
if (rc != JNI_OK) {
fprintf(stderr, "RASplugin004 Return code %d setting trace options\n", rc);
fflush(stderr);
return JNI_ERR;
}
}

To set Maximal tracing for ’j9mm’, use the following command when launching the
JVM and your agent:
java -Xrunjvmri:maximal=j9mm -Xtrace:external=j9mm App.class

Note: Trace must be enabled before the agent can be used. To do this, specify the
trace option on the command-line: -Xtrace:external=j9mm.

Launching the agent

To launch the agent when the JVM starts up, use the -Xrun option. For example if
your agent is called jvmri, specify -Xrunjvmri: <options> on the command-line.

Building the agent

Windows
Before you can build a JVMRI agent, ensure that:
v The agent is contained in a C file called myagent.c.
v You have Microsoft Visual C/C++ installed.
v The directories sdk\include\ and sdk\include\win32 have been added to the
environment variable INCLUDE.
To build a JVMRI agent, enter the command:
cl /MD /Femyagent.dll myagent.c /link /DLL

Chapter 31. Using the Reliability, Availability, and Serviceability Interface 285
Launching the agent

Linux
To build a JVMRI agent, write a shell script similar to this:
export SDK_BASE=<sdk directory>
export INCLUDE_DIRS="-I. -I$SDK_BASE/include"
export JVM_LIB=-L$SDK_BASE/jre/bin/classic
gcc $INCLUDE_DIRS $JVM_LIB -ljvm -o libmyagent.so -shared myagent.c

Where <sdk directory> is the directory where your SDK is installed.

z/OS
To build a JVMRI agent, write a shell script similar to this:
SDK_BASE= <sdk directory>
USER_DIR= <user agent’s source directory>
c++ -c -g -I$SDK_BASE/include -I$USER_DIR -W "c,float(ieee)"
-W "c,langlvl(extended)" -W "c,expo,dll" myagent.c
c++ -W "l,dll" -o libmyagent.so myagent.o
chmod 755 libmyagent.so

This builds a non-xplink library.

Agent design
The agent must reference the header files jni.h and jvmri.h, which are shipped with
the SDK and are in the sdk\include subdirectory. To launch the agent, use the
-Xrun command-line option. The JVM parses the -Xrunlibrary_name[:options]
switch and loads library_name if it exists. A check for an entry point that is called
JVM_OnLoad is then made. If the entry point exists, it is called to allow the library to
initialize. This processing occurs after the initialization of all JVM subcomponents.
The agent can then call the functions that have been initialized, by using the
JVMRI table.

JVMRI functions
At startup, the JVM initializes JVMRI. You access the JVMRI functions with the JNI
GetEnv() routine to obtain an interface pointer. For example:
JNIEXPORT jint JNICALL
JVM_OnLoad(JavaVM *vm, char *options, void *reserved)
{
DgRasInterface *ri;
......
(*vm)->GetEnv(vm, (void **)&ri, JVMRAS_VERSION_1_3)

rc = jvmras_intf->TraceRegister(env, listener);
......
}

API calls provided by JVMRI

The functions are not listed in the sequence in which they appear in the table. Note
that all calls must be made using a valid JNIEnv pointer as the first parameter.

CreateThread
int CreateThread( JNIEnv *env, void JNICALL (*startFunc)(void*),
void *args, int GCSuspend)
Description
Creates a thread. A thread can be created only after the JVM has been
initialized. However, calls to CreateThread can be made also before
initialization; the threads are created by a callback function after initialization.

286 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 JVMRI - API calls

Parameters
v A valid pointer to a JNIEnv.
v Pointer to start function for the new thread.
v Pointer to argument that is to be passed to start function.
v GCSuspend parameter is ignored.
Returns
JNI Return code JNI_OK if thread creation is successful; otherwise, JNI_ERR.

DumpDeregister
int DumpDeregister(JNIEnv *env, int (JNICALL *func)(JNIEnv *env2,
void **threadLocal, int reason))
Description
Deregisters a dump call back function that was previously registered by a call
to DumpRegister.
Parameters
v A valid pointer to a JNIEnv.
v Function pointer to trace function to register.
Returns
JNI return codes JNI_OK and JNI_EINVAL.

DumpRegister
int DumpRegister(JNIEnv *env, int (JNICALL *func)(JNIEnv *env2,
void **threadLocal, int reason))
Description
Registers a function that is called back when the JVM is about to generate a
JavaCore file.
Parameters
v A valid pointer to a JNIEnv.
v Function pointer to trace function to register.
Returns
JNI return codes JNI_OK and JNI_ENOMEM.

DynamicVerbosegc
void JNICALL *DynamicVerbosegc (JNIEnv *env, int vgc_switch,
int vgccon, char* file_path, int number_of_files,
int number_of_cycles);
Description
Not supported. Displays the message ″not supported″.
Parameters
v A valid pointer to a JNIEnv.
v Integer that indicates the direction of switch (JNI_TRUE = on, JNI_FALSE =
off)
v Integer that indicates the level of verbosegc (0 = -verbosegc, 1 =
-verbose:Xgccon)
v Pointer to string that indicates file name for file redirection
v Integer that indicates the number of files for redirection
v Integer that indicates the number of cycles of verbosegc per file

Chapter 31. Using the Reliability, Availability, and Serviceability Interface 287
JVMRI - API calls

Returns
None.

GenerateHeapdump
int GenerateHeapdump( JNIEnv *env )
Description
Generates a Heapdump file.
Parameters
v A valid pointer to a JNIEnv.
Returns
JNI Return code JNI_OK if running dump is successful; otherwise, JNI_ERR.

GenerateJavacore
int GenerateJavacore( JNIEnv *env )
Description
Generates a Javacore file.
Parameters
v A valid pointer to a JNIEnv.
Returns
JNI Return code JNI_OK if running dump is successful; otherwise, JNI_ERR.

GetComponentDataArea
int GetComponentDataArea( JNIEnv *env, char *componentName,
void **dataArea, int *dataSize )
Description
Not supported. Displays the message ″no data area for <requested
component>″
Parameters
v A valid pointer to a JNIEnv.
v Component name.
v Pointer to the component data area.
v Size of the data area.
Returns
JNI_ERR

GetRasInfo
int GetRasInfo(JNIEnv * env,
RasInfo * info_ptr)
Description
This function fills in the supplied RasInfo structure, based on the request type
that is initialized in the RasInfo structure. (See details of the RasInfo structure
in “RasInfo structure” on page 292.
Parameters
v A valid pointer to a JNIEnv. This parameter is reserved for future use.
v Pointer to a RasInfo structure. This should have the type field initialized to a
supported request.

288 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 JVMRI - API calls

Returns
JNI Return codes JNI_OK, JNI_EINVAL and JNI_ENOMEM.

InitiateSystemDump
int JNICALL InitiateSystemDump( JNIEnv *env )
Description
Initiates a system dump. The dumps and the output that are produced depend
on the settings for JAVA_DUMP_OPTS and JAVA_DUMP_TOOL and on the
support that is offered by each platform.
Parameters
v A valid pointer to a JNIEnv.
Returns
JNI Return code JNI_OK if dump initiation is successful; otherwise, JNI_ERR. If a
specific platform does not support a system-initiated dump, JNI_EINVAL is
returned.

InjectOutOfMemory
int InjectOutOfMemory( JNIEnv *env )
Description
Not supported. Displays the message ″not supported″.
Parameters
v A valid pointer to a JNIEnv.
Returns
JNI_ERR

InjectSigSegv
int InjectSigsegv( JNIEnv *env )
Description
Raises a SIGSEGV exception, or the equivalent for your platform.
Parameters
v A valid pointer to a JNIEnv.
Returns
JNI_ERR

NotifySignal
void NotifySignal(JNIEnv *env, int signal)
Description
Raises a signal in the JVM.
Parameters
v A valid pointer to a JNIEnv. This parameter is reserved for future use.
v Signal number to raise.
Returns
Nothing.

Chapter 31. Using the Reliability, Availability, and Serviceability Interface 289
JVMRI - API calls

ReleaseRasInfo
int ReleaseRasInfo(JNIEnv * env,
RasInfo * info_ptr)
Description
This function frees any areas to which the RasInfo structure might point after a
successful GetRasInfo call. The request interface never returns pointers to ’live’
JVM control blocks or variables.
Parameters
v A valid pointer to a JNIEnv. This parameter is reserved for future use.
v Pointer to a RasInfo structure. This should have previously been set up by a
call to GetRasInfo. An error occurs if the type field has not been initialized
to a supported request. (See details of the RasInfo structure in “RasInfo
structure” on page 292.)
Returns
JNI Return codes JNI_OK or JNI_EINVAL.

RunDumpRoutine
int RunDumpRoutine( JNIEnv *env, int componentID, int level, void (*printrtn)
(void *env, const char *tagName, const char *fmt, ...) )
Description
Runs one of the individual registered dump routines. Output is sent to the
supplied print routine.
Parameters
v A valid pointer to a JNIEnv.
v Id of component to dump.
v Detail level of dump.
v Print routine to which dump output is directed.
Returns
JNI Return code JNI_OK if running dump is successful; otherwise, JNI_ERR.

SetOutOfMemoryHook
int SetOutOfMemoryHook( JNIEnv *env, void (*rasOutOfMemoryHook)
(void) )
Description
Registers a callback function for an out-of-memory condition.
Parameters
v A valid pointer to a JNIEnv.
v Pointer to callback function.
Returns
JNI Return code JNI_OK if table is successfully updated; otherwise, JNI_ERR.

TraceDeregister
int TraceDeregister(JNIEnv *env, void (JNICALL *func)(JNIEnv *env2,
void **threadLocal, int traceId, const char *
format, va_list varargs))
Description
Deregisters an external trace listener.

290 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 JVMRI - API calls

Parameters
v A valid pointer to a JNIEnv.
v Function pointer to a previously-registered trace function.
Returns
JNI Return code JNI_OK or JNI_EINVAL.

TraceRegister
int TraceRegister(JNIEnv *env, void (JNICALL *func)(JNIEnv *env2,
void **threadLocal, int traceId, const char * format,
va_list var))
Description
Registers a trace listener.
Parameters
v A valid pointer to a JNIEnv.
v Function pointer to trace function to register.
Returns
JNI Return code JNI_OK or JNI_ENOMEM.

TraceResume
void TraceResume(JNIEnv *env)
Description
Resumes tracing.
Parameters
v A valid pointer to a JNIEnv. If MULTI_JVM; otherwise, it can be NULL.
Returns
Nothing.

TraceResumeThis
void TraceResumeThis(JNIEnv *env);
Description
Resume tracing from the current thread. This action decrements the
resumecount for this thread. When it reaches zero (or below) the thread starts
tracing (see Chapter 30, “Tracing Java applications and the JVM,” on page 259).
Parameters
v A valid pointer to a JNIEnv.
Returns
None.

TraceSet
int TraceSet(JNIEnv *env, const char *cmd)
Description
Sets the trace configuration options.
Parameters
v A valid pointer to a JNIEnv.
v Trace configuration command.

Chapter 31. Using the Reliability, Availability, and Serviceability Interface 291
JVMRI - API calls

Returns
JNI Return code JNI_OK, JNI_ERR, JNI_ENOMEM, JNI_EXIST and JNI_EINVAL.

TraceSnap
void TraceSnap(JNIEnv *env, char *buffer)
Description
Takes a snapshot of the current trace buffers.
Parameters
v A valid pointer to a JNIEnv; if set to NULL, current Execenv is used.
v The second parameter is no longer used, but still exists to prevent changing
the function interface. It can safely be set to NULL.
Returns
Nothing

TraceSuspend
void TraceSuspend(JNIEnv *env)
Description
Suspends tracing.
Parameters
v A valid pointer to a JNIEnv; if MULTI_JVM; otherwise, it can be NULL.
Returns
Nothing.

TraceSuspendThis
void TraceSuspendThis(JNIEnv *env);
Description
Suspend tracing from the current thread. This action decrements the
suspendcount for this thread. When it reaches zero (or below) the thread stops
tracing (see Chapter 30, “Tracing Java applications and the JVM,” on page 259).
Parameters
v A valid pointer to a JNIEnv.
Returns
None.

RasInfo structure
The RasInfo structure that is used by GetRasInfo() takes the following form.
(Fields that are initialized by GetRasInfo are underscored):
typedef struct RasInfo {
int type;
union {
struct {
int number;
char **names;
} query;
struct {
int number;
char **names;
} trace_components;
struct {
char *name

292 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 RasInfo, trace, and formatting

int first;
int last;
unsigned char *bitMap;
} trace_component;
} info;
} RasInfo;

RasInfo request types

The following request types are supported:
RASINFO_TYPES
Returns the number of request types that are supported and an array of
pointers to their names in the enumerated sequence. The names are in
codepage ISO8859-1.
RASINFO_TRACE_COMPONENTS
Returns the number of components that can be enabled for trace and an array
of pointers to their names in the enumerated sequence. The names are in
codepage ISO8859-1.
RASINFO_TRACE_COMPONENT
Returns the first and last tracepoint ids for the component name (code page
ISO8859-1) and a bitmap of those tracepoints, where a 1 signifies that the
tracepoint is in the build. The bitmap is big endian (tracepoint id first is the
most significant bit in the first byte) and is of length ((last-first)+7)/8 bytes.

Intercepting trace data

To receive trace information from the JVM, the TraceRegister() routine must
register a trace listener. In addition, you must specify the option
-Xtrace:external=<option> to route trace information to an external trace listener.

The -Xtrace:external=<option>
The format of this property is:
-Xtrace:external=[[!]tracepoint_specification[,...]]

This system property controls what is traced. Multiple statements are allowed and
their effect is cumulative.

The tracepoint_specification is as follows:

Component[(Class[,...])]
Where component is the JVM subcomponent or all. If no component is
specified, all is assumed.
class is the tracepoint type or all. If class is not specified, all is assumed.
TPID(tracepoint_id[,...])
Where tracepoint_id is the hexadecimal global tracepoint identifier.

If no qualifier parameters are entered, all tracepoints are enabled; that is, the
equivalent of specifying all.

The ! (exclamation mark) is a logical not. It allows complex tracepoint selection.

Chapter 31. Using the Reliability, Availability, and Serviceability Interface 293
RasInfo, trace, and formatting

Calling external trace

If an external trace routine has been registered and a tracepoint has been enabled
for external trace, it is called with the following parameters:
env
Pointer to the JNIEnv for the current thread.
traceid
Trace identifier
format
A zero-terminated string that describes the format of the variable argument list
that follows. Current possible values for each character position:

0x01 One character

0x02 Short
0x04 Int
0x08 Double or long long
0xff ASCII string pointer (may be NULL)
0x00 End of format string

If the format pointer is NULL, no trace data follows.

varargs
A va_list of zero or more arguments as defined in format argument.

Formatting
You can use J9TraceFormat.dat to format JVM-generated tracepoints that are
captured by the agent. J9TraceFormat.dat is shipped with the SDK. It consists of a
flat ASCII or EBCDIC file of the following format:
1.2
dg
000100 8 00 0 N DgTrcRecordsLost "***** %d records lost *****"
000101 0 00 0 N dgTraceLock_Event1 "dgTraceLock() Trace suspended and locked "
000102 0 00 0 N dgTraceUnlock_Event1 "dgTraceUnlock() Trace resumed and unlocked"

The first line contains the version number of the format file. A new version
number reflects changes to the layout of this file. The second line contains the
internal JVM component name. Following the component name are the tracepoint
formatting records for the named component. These formatting records continue
until another component name is found. (Component name entries can be
distinguished from format records, because they always contain only one field.)

The format of each tracepoint entry is as follows:

nnnnnn t o l e symbolic_name .tracepoint_formatting_template

where:
v nnnnnn is the hex tracepoint ID.
v t is the tracepoint type (0 through 11).
v o is the overhead (0 through 10).
v l is the level of the tracepoint (0 through 9, or - if the tracepoint is obsolete).
v e is the explicit setting flag (Y/N).
v symbolic_name is the name of the tracepoint.

294 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 RasInfo, trace, and formatting

v tracepoint_formatting_template is the template in double quotes that is used to

format the entry in double quotes.

Tracepoint types are as follows:

Type 0 Event
Type 1 Exception
Type 2 Entry
Type 4 Exit
Type 5 Exit-with-Exception
Type 6 Mem

Any other type is reserved for development use; you should not find any on a
retail build. Note that this condition is subject to change without notice. The
version number will be different for each change.

Chapter 31. Using the Reliability, Availability, and Serviceability Interface 295
RasInfo, trace, and formatting

296 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 32. Using the JVMPI
The JVMPI is a 2-way interface that allows communication between the JVM and a
profiler. JVMPI allows third parties to develop profiling tools based on this
interface. The interface contains mechanisms for the profiling agent to notify the
JVM about the kinds of information it wants to receive as well as a means of
receiving the relevant notifications. Several tools are based on this interface, such
as Jprobe, OptimizeIt, TrueTime, and Quantify. These are all third-party commercial
tools, so IBM cannot make any guarantees or recommendations with regard to
their use. IBM does provide a simple profiling agent, based on this interface, called
HPROF.

The HPROF profiler

HPROF is a profiler shipped with the IBM SDK that uses the JVMPI to collect and
record information about Java execution. Use it to work out which parts of a
program are using the most memory or processor time. To improve the efficiency
of your applications, you must know which parts of the code are using large
amounts of memory and CPU resources. HPROF is one of the nonstandard options
to java, and is invoked like this:
java -Xrunhprof[<option>=<value>,...] <classname>

When you run Java with HPROF, an output file is created at the end of program
execution. This file is placed in the current working directory and is called
java.hprof.txt (java.hprof if binary format is used) unless a different filename has
been given. This file contains a number of different sections, but the exact format
and content depend on the selected options.

The command java -Xrunhprof:help displays the options available:

heap=dump|sites|all
This option helps in the analysis of memory usage. It tells HPROF to generate
stack traces, from which you can see where memory was allocated. If you use
the heap=dump option, you get a dump of all live objects in the heap. With
heap=sites, you get a sorted list of sites with the most heavily allocated objects
at the top.
cpu=samples|times|old
The cpu option outputs information that is useful in determining where the
CPU spends most of its time. If cpu is set to ″samples″, the JVM pauses
execution and identifies which method call is active. If the sampling rate is
high enough, you get a good picture of where your program spends most of
its time. If cpu is set to ″timing″, you receive precise measurements of how
many times each method was called and how long each execution took.
Although this is more accurate, it slows down the program. If cpu is set to
″old″, the profiling data is output in the old hprof format. For more
information, go to http://java.sun.com/j2se/, and follow the appropriate links.
monitor=y|n
The monitor option can help you understand how synchronization affects the
performance of your application. Monitors are used to implement thread
synchronization, so getting information on monitors can tell you how much

© Copyright IBM Corp. 2003, 2006 297

JVMPI - HPROF profiler

time different threads are spending when trying to access resources that are
already locked. HPROF also gives you a snapshot of the monitors in use. This
is very useful for detecting deadlocks.
format=a|b
The default is for the output file to be in ASCII format. Set format to ’b’ if you
want to specify a binary format (which is required for some utilities such as
the Heap Analysis Tool).
file=<filename>
The file option lets you change the name of the output file. The default name
for an ASCII file is java.hprof.txt. The default name for a binary file is
java.hprof.
net=<host>:<port>
To send the output over the network rather than to a local file, use the net
option.
depth=<size>
The depth option indicates the number of method frames to display in a stack
trace (the default is 4).
thread=y|n
If you set the thread option to ″y″, the thread id is printed beside each trace.
This option is useful if it is not clear which thread is associated with which
trace (a problem that might occur in a multi-threaded application).
doe=y|n
The default behavior is to collect profile information when an application exits.
To collect the profiling data during execution, set doe (dump on exit) to ″n″.

Explanation of the HPROF output file

The top of the file contains general header information such as an explanation of
the options, copyright, and disclaimers. A summary of each thread appears next.

You can see the output after using HPROF with a simple program, as shown
below. This test program creates and runs two threads for a short time. From the
output, you can see that the two threads called respectively ″apples″ and ″oranges″
were created after the system-generated ″main″ thread. Both threads end before the
″main″ thread. For each thread its address, identifier, name, and thread group
name are displayed. You can see the order in which threads start and finish.
THREAD START (obj=11199050, id = 1, name="Signal dispatcher", group="system")
THREAD START (obj=111a2120, id = 2, name="Reference Handler", group="system")
THREAD START (obj=111ad910, id = 3, name="Finalizer", group="system")
THREAD START (obj=8b87a0, id = 4, name="main", group="main")
THREAD END (id = 4)
THREAD START (obj=11262d18, id = 5, name="Thread-0", group="main")
THREAD START (obj=112e9250, id = 6, name="apples", group="main")
THREAD START (obj=112e9998, id = 7, name="oranges", group="main")
THREAD END (id = 6)
THREAD END (id = 7)
THREAD END (id = 5)

The trace output section contains regular stack trace information. The depth of
each trace can be set and each trace has a unique id:
TRACE 5:
java/util/Locale.toLowerCase(Locale.java:1188)
java/util/Locale.convertOldISOCodes(Locale.java:1226)
java/util/Locale.<init>(Locale.java:273)
java/util/Locale.<clinit>(Locale.java:200)

298 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 JVMPI - HPROF profiler

A trace contains a number of frames, and each frame contains the class name,
method name, filename, and line number. In the example above you can see that
line number 1188 of Local.java (which is in the toLowerCase method) has been
called from the convertOldISOCodes() function in the same class. These traces are
useful in following the execution path of your program. If you set the monitor
option, a monitor dump is output that looks like this:
MONITOR DUMP BEGIN
THREAD 8, trace 1, status: R
THREAD 4, trace 5, status: CW
THREAD 2, trace 6, status: CW
THREAD 1, trace 1, status: R
MONITOR java/lang/ref/Reference$Lock(811bd50) unowned
waiting to be notified: thread 2
MONITOR java/lang/ref/ReferenceQueue$Lock(8134710) unowned
waiting to be notified: thread 4
RAW MONITOR "_hprof_dump_lock"(0x806d7d0)
owner: thread 8, entry count: 1
RAW MONITOR "Monitor Cache lock"(0x8058c50)
owner: thread 8, entry count: 1
RAW MONITOR "Monitor Registry lock"(0x8058d10)
owner: thread 8, entry count: 1
RAW MONITOR "Thread queue lock"(0x8058bc8)
owner: thread 8, entry count: 1
MONITOR DUMP END
MONITOR TIME BEGIN (total = 0 ms) Thu Aug 29 16:41:59 2002
MONITOR TIME END

The first part of the monitor dump contains a list of threads, including the trace
entry that identifies the code the thread executed. There is also a thread status for
each thread where:
v R — Runnable
v S — Suspended
v CW — Condition Wait
v MW — Monitor Wait
Next is a list of monitors along with their owners and an indication of whether
there are any threads waiting on them.

The Heapdump is the next section. This is a list of different areas of memory and
shows how they are allocated:
CLS 1123edb0 (name=java/lang/StringBuffer, trace=1318)
super 111504e8
constant[25] 8abd48
constant[32] 1123edb0
constant[33] 111504e8
constant[34] 8aad38
constant[115] 1118cdc8
CLS 111ecff8 (name=java/util/Locale, trace=1130)
super 111504e8
constant[2] 1117a5b0
constant[17] 1124d600
constant[24] 111fc338
constant[26] 8abd48
constant[30] 111fc2d0
constant[34] 111fc3a0
constant[59] 111ecff8
constant[74] 111504e8
constant[102] 1124d668
...
CLS 111504e8 (name=java/lang/Object, trace=1)
constant[18] 111504e8

Chapter 32. Using the JVMPI 299

JVMPI - HPROF profiler

CLS tells you that memory is being allocated for a class. The hexadecimal number
following it is the actual address where that memory is allocated.

Next is the class name followed by a trace reference. Use this to cross reference the
trace output and see when this is called. If you refer back to that particular trace,
you can get the actual line number of code that led to the creation of this object.
The addresses of the constants in this class are also displayed and, in the above
example, the address of the class definition for the superclass. Both classes are
children of the same superclass (with address 11504e8). Looking further through
the output you can see this class definition and name. It turns out to be the Object
class (a class that every class inherits from). The JVM loads the entire superclass
hierarchy before it can use a subclass. Thus, class definitions for all superclasses
are always present. There are also entries for Objects (OBJ) and Arrays (ARR):
OBJ 111a9e78 (sz=60, trace=1, class=java/lang/Thread@8b0c38)
name 111afbf8
group 111af978
contextClassLoader 1128fa50
inheritedAccessControlContext 111aa2f0
threadLocals 111bea08
inheritableThreadLocals 111bea08
ARR 8bb978 (sz=4, trace=2, nelems=0, elem type=java/io/ObjectStreamField@8bac80)

If you set the heap option to ″sites″ or ″all″ (″dump″ and ″sites″), you also get a list
of each area of storage allocated by your code. This list is ordered with the sites
that allocate the most memory at the top:
SITES BEGIN (ordered by live bytes) Thu Aug 29 16:30:31 2002
percent live alloc’ed stack class
rank self accum bytes objs bytes objs trace name
1 18.18% 18.18% 32776 2 32776 2 1332 [C
2 9.09% 27.27% 16392 2 16392 2 1330 [B
3 8.80% 36.08% 15864 92 15912 94 1 [C
4 4.48% 40.55% 8068 1 8068 1 31 [S
5 4.04% 44.59% 7288 4 7288 4 1130 [C
6 3.12% 47.71% 5616 36 5616 36 1 <Unknown>
7 2.51% 50.22% 4524 29 4524 29 1 java/lang/Class
8 2.05% 52.27% 3692 1 3692 1 806 [L<Unknown>;
9 2.01% 54.28% 3624 90 3832 94 77 [C
10 1.40% 55.68% 2532 1 2532 1 32 [I
11 1.37% 57.05% 2468 3 2468 3 1323 [C
12 1.31% 58.36% 2356 1 2356 1 1324 [C
13 1.14% 59.50% 2052 1 2052 1 95 [B
14 1.02% 60.52% 1840 92 1880 94 1 java/lang/String
15 1.00% 61.52% 1800 90 1880 94 77 java/lang/String
16 0.64% 62.15% 1152 10 1152 10 1390 [C
17 0.57% 62.72% 1028 1 1028 1 30 [B
18 0.52% 63.24% 936 6 936 6 4 <Unknown>
19 0.45% 63.70% 820 41 820 41 79 java/util/Hashtable$Entry

In this example, Trace 1332 allocated 18.18% of the total allocated memory. This
works out to be 32776 bytes.

The cpu option gives profiling information on the CPU. If cpu is set to samples,
you get an output containing the results of periodic samples during execution of
the code. At each sample, the code path being executed is recorded and a report
such as this is output:
CPU SAMPLES BEGIN (total = 714) Fri Aug 30 15:37:16 2002
rank self accum count trace method
1 76.28% 76.28% 501 77 MyThread2.bigMethod
2 6.92% 83.20% 47 75 MyThread2.smallMethod
...
CPU SAMPLES END

300 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 JVMPI - HPROF profiler

You can see that the bigMethod() was responsible for 76.28% of the CPU execution
time and was being executed 501 times out of the 714 samples. If you use the trace
IDs, you can see the exact route that led to this method being called.

Chapter 32. Using the JVMPI 301

JVMPI - HPROF profiler

302 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Chapter 33. Using DTFJ
The Diagnostic Tooling Framework for Java (DTFJ) is a Java application
programming interface (API) from IBM used to support the building of Java
diagnostics tools.

You process the dumps passed to DTFJ with the jextract tool; see “jextract” on
page 223. The jextract tool produces metadata from the dump, which allows the
internal structure of the JVM to be analyzed. jextract must be run on the system
that produced the dump.

The DTFJ API helps diagnostics tools access the following (and more) information:
v Memory locations stored in the dump
v Relationships between memory locations and Java internals
v Java threads running within the JVM
v Native threads held in the dump
v Java classes and objects that were present in the heap
v Details of the machine on which the dump was produced
v Details of the Java version that was being used
v The command line that launched the JVM

DTFJ is implemented in pure Java and tools written using DTFJ can be
cross-platform. Therefore, it is possible to analyze a dump taken from one machine
on another (remote and more convenient) machine. For example, a dump produced
on an AIX PPC machine can be analyzed on a Windows Thinkpad.

This chapter describes DTFJ in:

v “Overview of the DTFJ interface”
v “DTFJ example application” on page 307

The full details of the DTFJ Interface are provided with the SDK as Javadoc in
sdk/docs/dtfj.zip. DTFJ classes are accessible without modification to the class
path.

Overview of the DTFJ interface

To create applications that use DTFJ, you must use the DTFJ interface.
Implementations of this interface have been written that work with various JVMs.
All implementations support the same interface and therefore a diagnostic tool that
works against a Version 1.4.2 dump will generally work with a Version 5.0 dump
unless it is using knowledge of Version 1.4.2 specific internals in some way.

Figure 10 on page 306 illustrates the DTFJ interface. The starting point for working
with a dump is to obtain an Image instance by using the ImageFactory class
supplied with the concrete implementation of the API.

The following example shows how to work with a dump.

import java.io.File;
import java.util.Iterator;
import java.io.IOException;

© Copyright IBM Corp. 2003, 2006 303

Overview of the DTFJ interface

import com.ibm.dtfj.image.CorruptData;
import com.ibm.dtfj.image.Image;
import com.ibm.dtfj.image.ImageFactory;

public class DTFJEX1 {

public static void main(String[] args) {
Image image = null;
if (args.length > 0) {
File f = new File(args[0]);
try {
Class factoryClass = Class
.forName("com.ibm.dtfj.image.j9.ImageFactory");
ImageFactory factory = (ImageFactory) factoryClass
.newInstance();
image = factory.getImage(f);
} catch (ClassNotFoundException e) {
System.err.println("Could not find DTFJ factory class");
e.printStackTrace(System.err);
} catch (IllegalAccessException e) {
System.err.println("IllegalAccessException for DTFJ factory class");
e.printStackTrace(System.err);
} catch (InstantiationException e) {
System.err.println("Could not instantiate DTFJ factory class");
e.printStackTrace(System.err);
} catch (IOException e) {
System.err.println("Could not find/use required file(s)");
e.printStackTrace(System.err);
}
} else {
System.err.println("No filename specified");
}
if (image == null) {
return;
}

Iterator asIt = image.getAddressSpaces();

int count = 0;
while (asIt.hasNext()) {
Object tempObj = asIt.next();
if (tempObj instanceof CorruptData) {
System.err.println("Address Space object is corrupt: "
+ (CorruptData) tempObj);
} else {
count++;
}
}
System.out.println("The number of address spaces is: " + count);
}
}

In this example, the only section of code that ties the dump to a particular
implementation of DTFJ is the generation of the factory class – it would be a
straightforward task to amend this code to cope with handling different
implementations.

The getImage() methods in ImageFactory expect two files. The files must be the
dump itself and the .xml metadata file. If there is a problem with the file specified,
an IOException is thrown by getImage() and can be caught and (in the example
above) an appropriate message issued. If a missing file was passed to the above
example, the following output would be produced:

Could not find/use required file(s)

java.io.FileNotFoundException: core_file.xml (The system cannot find the file specified.)
at java.io.FileInputStream.open(Native Method)

304 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Overview of the DTFJ interface

at java.io.FileInputStream.<init>(FileInputStream.java:135)
at com.ibm.dtfj.image.j9.ImageFactory.getImage(ImageFactory.java:47)
at com.ibm.dtfj.image.j9.ImageFactory.getImage(ImageFactory.java:35)
at DTFJEX1.main(DTFJEX1.java:23)

In the case above, the DTFJ implementation is expecting a dump file to exist.
Different errors would be caught if the file existed but was not recognized as a
valid dump file.

After you have obtained an Image instance, you can begin analyzing the dump.
The Image instance is the second instance in the class hierarchy for DTFJ
illustrated by Figure 10 on page 306.

Chapter 33. Using DTFJ 305

Overview of the DTFJ interface

KEY
ImageFactory

S Returns Image Section

P Returns Image Pointer
Inheritance Image
Returns
Iterator
P P
ImageAddressSpace CorruptData
S
All iterators can return
CorruptData objects

ImageThread ImageProcess ImageModule

S S

P P
ImageStackFrame ImageRegister ImageSymbol

runtime Package ManagedRuntime

java Package
P
JavaRuntime

P
JavaThread JavaClassLoader JavaMonitor

JavaHeap
S

P P
JavaStackFrame JavaMethod JavaClass
S

P
JavaLocation JavaField

P
JavaMember JavaObject
S

Figure 10. Diagram of the DTFJ interface

The hierarchy displays some major points of DTFJ. Firstly, there is a separation
between the Image (the dump, a sequence of bytes with different contents on
different platforms) and the Java internal knowledge.

Some things to note from the diagram:

306 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Overview of the DTFJ interface

v The DTFJ interface is separated into two parts: classes with names that start with
Image and classes with names that start with Java.
v Image and Java classes are linked using a ManagedRuntime (which is extended
by JavaRuntime).
v An Image object contains one ImageAddressSpace object (or, on z/OS, possibly
more).
v An ImageAddressSpace object contains one ImageProcess object (or, on z/OS,
possibly more).
v Conceptually, you can apply the Image model to any program running with the
ImageProcess, although for the purposes of this document discussion is limited
to the IBM JVM implementations.

DTFJ example application

This example is a fully working DTFJ application. For clarity, it does not perform
full error checking when constructing the main Image object and does not perform
CorruptData handling in all of the iterators. In a production environment, you
would use the techniques illustrated in the example in the “Overview of the DTFJ
interface” on page 303.

In this example, the program iterates through every available Java thread and
checks whether it is equal to any of the available image threads. When they are
found to be equal, the program declares that it has, in this case, "Found a match".

The example demonstrates:

v How to iterate down through the class hierarchy.
v How to handle CorruptData objects from the iterators.
v The use of the .equals method for testing equality between objects.
import java.io.File;
import java.util.Iterator;

import com.ibm.dtfj.image.CorruptData;
import com.ibm.dtfj.image.CorruptDataException;
import com.ibm.dtfj.image.Image;
import com.ibm.dtfj.image.ImageAddressSpace;
import com.ibm.dtfj.image.ImageFactory;
import com.ibm.dtfj.image.ImageProcess;
import com.ibm.dtfj.java.JavaRuntime;
import com.ibm.dtfj.java.JavaThread;
import com.ibm.dtfj.image.ImageThread;

public class DTFJEX2 {

public static void main(String[] args) {
Image image = null;
if (args.length > 0) {
File f = new File(args[0]);
try {
Class factoryClass = Class
.forName("com.ibm.dtfj.image.j9.ImageFactory");
ImageFactory factory = (ImageFactory) factoryClass
.newInstance();
image = factory.getImage(f);
} catch (Exception ex) { /*
* Should use the error handling as
* shown in DTFJEX1.
*/
System.err.println("Error in DTFJEX2");
ex.printStackTrace(System.err);
}

Chapter 33. Using DTFJ 307

DTFJ example application

} else {
System.err.println("No filename specified");
}

if (null == image) {
return;
}

MatchingThreads(image);
}

public static void MatchingThreads(Image image) {

ImageThread imgThread = null;

Iterator asIt = image.getAddressSpaces();

while (asIt.hasNext()) {
System.out.println("Found ImageAddressSpace...");

ImageAddressSpace as = (ImageAddressSpace) asIt.next();

Iterator prIt = as.getProcesses();

while (prIt.hasNext()) {
System.out.println("Found ImageProcess...");

ImageProcess process = (ImageProcess) prIt.next();

Iterator runTimesIt = process.getRuntimes();

while (runTimesIt.hasNext()) {
System.out.println("Found Runtime...");
JavaRuntime javaRT = (JavaRuntime) runTimesIt.next();

Iterator javaThreadIt = javaRT.getThreads();

while (javaThreadIt.hasNext()) {
Object tempObj = javaThreadIt.next();
/* Should use CorruptData
* handling for all iterators
*/
if (tempObj instanceof CorruptData) {
System.out.println("We have some corrupt data");
} else {
JavaThread javaThread = (JavaThread) tempObj;
System.out.println("Found JavaThread...");
try {
imgThread = (ImageThread) javaThread
.getImageThread();

// Now we have a Java thread we can iterator

// through the image threads
Iterator imgThreadIt = process.getThreads();

while (imgThreadIt.hasNext()) {
ImageThread imgThread2 = (ImageThread) imgThreadIt
.next();
if (imgThread.equals(imgThread2)) {
System.out.println("Found a match:");
System.out.println("\tjavaThread "
+ javaThread.getName()
+ " is the same as "
+ imgThread2.getID());
}
}
} catch (CorruptDataException e) {
System.err.println("ImageThread was corrupt: " + e.getMessage());
}

308 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 DTFJ example application

}
}

}
}
}
}

Many DTFJ applications will follow much the same model.

Chapter 33. Using DTFJ 309

310 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Part 5. Appendixes

© Copyright IBM Corp. 2003, 2006 311

312 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Appendix A. Compatibility tables
Before release 5 of the IBM WebSphere Application Server, the SDK and the ORB
levels did not match. This table shows the version of the embedded JVM that ships
with the corresponding version of the WebSphere Application Server.

WebSphere Application SDK ORB

Server
3.5.4 1.2.2 1.2.2
3.5.5 1.2.2 1.2.2
4.0.1 1.3.0 1.3.0
4.0.2 1.3.0 1.3.0
4.0.3 1.3.1 1.3.0
4.0.4 1.3.1 1.3.0
4.0.5 1.3.1 1.3.1
5.0 1.3.1 1.3.1
5.0.1 1.3.1 1.3.1
5.0.2 1.3.1 1.3.1
5.1 1.4.1 1.4.1
5.1.0 1.4.1 Service Refresh 1 1.4.1
5.1.1 1.4.2 1.4.2

© Copyright IBM Corp. 2003, 2006 313

314 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Appendix B. ORB tracing for WebSphere Application Server
version 5
The diagnostic trace configuration settings for a server process determine the initial
trace state for that server process. The configuration settings are read at server
startup and are used to configure the trace service, either at server startup or while
the server is running. You can select whether to enable or disable ORB trace, and
you can change many of the trace service properties or settings while the server
process is running. This appendix describes:
v “Enabling trace at server startup”
v “Changing the trace on a running server” on page 316
v “Selecting ORB traces” on page 316

For more information, see:

ftp://ftp.software.ibm.com/software/webserver/appserv/library/wasv5base_pd.pdf
or
http://www-3.ibm.com/software/webservers/appserv/was/library/
or the WAS problem determination guide:
http://www.redbooks.ibm.com/abstracts/sg246798.html

Enabling trace at server startup

The diagnostic trace configuration settings for a server process determine the initial
trace state for that server process. The configuration settings are read at server
startup and are used to configure the trace service. To enable the trace:
1. Start the AdminConsole.
2. In the console navigation tree, click Troubleshooting > Logging and Tracing.
3. Click Server > Diagnostic Trace.
4. Click Configuration.
5. Select the Enable Trace check box to enable trace, or clear the check box to
disable trace.
6. Set the trace specification to the desired state by entering the correct
TraceString:
ORBRas=all=enabled
7. Select whether to send trace output to a file, or to an in-memory circular
buffer.
If you select a file, go to step 8.
If you select an in-memory circular buffer, go to step 11.
8. If you have selected a file for trace output, set the maximum size in MB to
which the file is allowed to grow. When the file reaches this size, the existing
file is closed, renamed, and a new file with the original name is opened. The
new name of the original file is the original name with a timestamp qualifier
added to it.
9. Specify how many history files you want to keep.
10. Go to step 12 on page 316.
11. If you have selected an in-memory circular buffer for the trace output, set the
size of the buffer. The size of the buffer determines the maximum number of

© Copyright IBM Corp. 2003, 2006 315

Enabling trace at server startup

entries that are to be kept in the buffer at any given time. Specify the size of
the buffer in thousands of entries. For example, if you want 1000 entries,
specify 1; if you want 3000 entries, specify 3.
12. Select the desired format for the generated trace.
13. Save the changed configuration.
14. Start the server.

Changing the trace on a running server

You can change the trace service state that determines which components are being
actively traced for a running server. To do this:
1. Start the AdminConsole.
2. In the console navigation tree, click Troubleshooting > Logging and Tracing.
3. Click Server > Diagnostic Trace.
4. Select the Runtime tab.
5. If you want to write your changes back to the server configuration, select the
Save Trace check box.
6. Change the existing trace state by changing the trace specification to the
desired state.
7. If you want to change from the existing trace output, configure a new one.
8. Click Apply.

Selecting ORB traces

You can select to enable or disable the ORB traces. To do this:
1. Start the AdminConsole.
2. In the console navigation tree, click Servers > Application Server.
3. Click Server.
4. Click Configuration
5. In the Additional Properties panel, click ORB Service.
6. Select the Enable Trace check box to enable ORB trace, or clear the check box
to disable ORB trace.
7. If you have chosen to disable ORB trace, go no further with these instructions.
If you have chosen to enable ORB trace, go to the next step.
8. In the Additional Properties panel, select Custom Properties.
9. Ensure that these two property names and values are present:
com.ibm.CORBA.Debug , true
com.ibm.CORBA.CommTrace , true
10. Add them if they are not present.

316 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Appendix C. CORBA GIOP message format
Table 18 shows:
v All types of messages
v The values of those messages as an integer number
v Whether those messages contain only a header or a header and a body
v Whether those messages are supported in GIOP versions 1.0, 1.1, and 1.2
Table 18. CORBA GIOP messages
Message Value Header Body 1.0 or 1.1 1.2 supported
supported
Request 0 X X X X
Reply 1 X X X X
Cancel 2 X X X
Request
Locate 3 X X X
request
Locate reply 4 X X X X
Close 5 X X
connection
Message error 6 X X
Fragment 7 X X X

Note: From now on in this chapter, each cell (table column) represents 1 byte
unless specified otherwise. Alignment of fields is not specified in the
following byte description. In a GIOP message, some fields must start at a 4-
or 8-byte boundary. Extra bytes of padding are present (always set to 0).

GIOP header
All types of messages that are described in Appendix C, “CORBA GIOP message
format” start with the GIOP Message Header:

47=G 49=I 4F=0 50=P Major; for Minor; for Flags Value (4 bytes)
example, example, length of
01 02 the rest of
the
message

In GIOP1.0, the least-significant bit of the Flags byte (that is, the first bit on the
right of the byte) indicates the byte sequence (big endian or little endian). In GIOP
1.1 and 1.2, the least-significant bit indicates the byte sequence that is used in later
elements of the message. A value of false (0) indicates a big-endian byte
sequencing; true (1) indicates little-endian byte sequencing.

The bit that is immediately to the left of the least-significant bit indicates whether
or not more fragments follow. A value of false (0) indicates that this message is the
last fragment; true (1) indicates that more fragments follow this message. The

© Copyright IBM Corp. 2003, 2006 317

CORBA GIOP message format

most-significant six bits are reserved; for GIOP 1.2 (1.1) they must be set to zero.
The Value field is the field that is indicated in Appendix C, “CORBA GIOP
message format,” on page 317.

Request header
Request id (4 bytes) Response Expected Reserved Reserved Reserved

The Response Expected flag indicates whether this request expects a reply from the
server. Values 1 = WITH_SERVER and 3 = WITH_TARGET correspond to a true
value. Therefore, the client expects a reply. A value of 0 = NONE or
WITH_TRANSPORT means that no reply is required. The reserved bytes are for
future use.

After this first 8 bytes, the header continues with the specification of the remote
reference. This specification, however, differs in different version of the GIOP. In
GIOP 1.0 and 1.1, the specification is:

Length of the Object Key (4 bytes) Object key (see previous length in bytes)

In GIOP 1.2, the specification is more complex. The next value of Addressing
Disposition index decides whether to insert an object key, a profile, or a full IOR
(one row corresponds to one value):

Addressing disposition (2 Object key length (4 bytes) Object key

bytes):
IOR profile ID (4 bytes) IOR profile length IOR profile data
0=object key
1=profile IOR profile index Full IOR
2=IOR

Then for all versions of GIOP the header continues with:

Length of operation name Operation name N= Number of service contexts

present (4 bytes)

and a sequence of N service contexts must come next. The following describes how
one of these service contexts is written. N of them are written consecutively.

Service context ID (4 bytes) Service context length (4 bytes) Service context data

Request body
Marshaled parameters (CORBA valuetype) Context pseudo object (for GIOP 1.0/1.1 only)

Reply header
For GIOP 1.2:

Request ID (4 bytes) Reply status (4 bytes)

318 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 CORBA GIOP message format

The reply status can be:

v 0 = NO_EXCEPTION
v 1 = USER_EXCEPTION
v 2 = SYSTEM_EXCEPTION
v 3 = LOCATE_FORWARD
v 4 = (deprecated)
v 5 = NEEDS_ADDRESSING_MODE (GIOP 1.2 only)

Request ID and reply status are then followed by:

Number of service contexts (4 bytes) < sequence of service contexts as before >

Note: In GIOP 1.0/1.1, the request ID and the reply status comes after the service
context list.

Reply body (based on reply status)

v NO_EXCEPTION:

Marshaled parameters

v USER_EXCEPTION: Varies (see CORBA specification)

v SYSTEM_EXCEPTION:

Exception ID length Exception ID Minor code (4 bytes) Completion status (4

(4 bytes) bytes)

v LOCATE_FORWARD:

IOR (starts with type ID)

v NEEDS_ADDRESSING_MODE (GIOP 1.2 only):

Addressing Disposition (2 bytes)

Cancel request header

This contains only the request ID coded in 4 bytes.

Locate request header

Request ID Addressing Object key Object key
disposition length (4 bytes)
(GIOP 1.2 only)
IOR Profile ID (4 IOR profile IOR profile data
bytes) length
IOR profile Full IOR
index

GIOP 1.0/1.1 supports only the object key version (first row only) and no
addressing disposition is specified.

Appendix C. CORBA GIOP message format 319

CORBA GIOP message format

Locate reply header

Request ID (4 bytes) Reply Status (4 bytes)

Locate reply body

v UNKNOWN_OBJECT = 0: No locate reply body
v OBJECT_HERE = 1: No locate reply body
v OBJECT_FORWARD = 2: IOR starting with the type ID
v Skip 3 (now deprecated)
v LOC_SYSTEM_EXCEPTION = 4: (Same as SYSTEM_EXCEPTION in reply body)
v NEEDS_ADDRESSING_MODE = 5: Addressing disposition index in two bytes
(short)

Fragment message
The fragment message observes these rules:
v The fragment length plus the GIOP header length (12 bytes) is a multiple of 8
for all but last message.
v All fragments must include at least the GIOP header and the request ID (total
length 16 bytes).
v In the GIOP header of the first fragment, the message type can be request,
reply, locate request, and locate reply. The fragment flag is set to 1.
v In the fragments that follow the first one, the message type is Fragment, and
the fragment flag is set to 1, except in the last fragment where the flag is set to
0.

Fragment header (GIOP 1.2 only)

The fragment header is made of only four bytes that represent the request ID.

320 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Appendix D. CORBA minor codes
This appendix gives definitions of the most common OMG- and IBM-defined
CORBA system exception minor codes that the IBM Java ORB uses. (See
“Completion status and minor codes” on page 169 for more information about
minor codes.)

When an error occurs, you might find additional details in the ORB FFDC log. By
default, the IBM Java ORB creates an FFDC log whose name is of the form
orbtrc.DDMMYYY.HHmm.SS.txt. If the IBM Java ORB is operating in the WebSphere
Application Server or other IBM product, see the publications for that product to
determine the location of the FFDC log.

org.omg.CORBA.COMM_FAILURE
CONNECT_FAILURE_1
User response: Investigate each request and reissue if
Explanation: The client attempted to open a
necessary. If the problem reoccurs, run with ORB,
connection with the server, but failed. The reasons for
network tracing, or both, active to determine the cause
the failure can be many; for example, the server might
of the failure.
not be up or it might not be listening on that port. If a
BindException is caught, it shows that the client could
not open a socket locally (that is, the local port was in CREATE_LISTENER_FAILED
use or the client has no local address).
Explanation: An exception occurred while a TCP/IP
Applicable CORBA exception class: listener was being created.
org.omg.CORBA.TRANSIENT
Applicable CORBA exception class:
User response: As with all TRANSIENT exceptions, a org.omg.CORBA.INTERNAL
retry or restart of the client or server might solve the
problem. Ensure that the port and server host names User response: The details of the caught exception are
are correct, and that the server is running and allowing written to the FFDC log. Review the details of the
connections. Also ensure that no firewall is blocking the exception, and take any further action that is necessary.
connection, and that a route is available between client
and server. LOCATE_UNKNOWN_OBJECT
Explanation: The server has no knowledge of the
CONN_CLOSE_REBIND object for which the client has asked in a locate request.
Explanation: An attempt has been made to write to a Applicable CORBA exception class:
TCP/IP connection that is closing. org.omg.CORBA.OBJECT_NOT_EXIST
Applicable CORBA exception class: User response: Ensure that the remote object that is
org.omg.CORBA.COMM_FAILURE requested resides in the specified server and that the
User response: Ensure that the completion status that remote reference is up-to-date.
is associated with the minor code is NO, then reissue
the request. NULL_PI_NAME
Explanation: One of the following methods has been
CONN_PURGE_ABORT called:
Explanation: An unrecoverable error occurred on a org.omg.PortableInterceptor.ORBInitInfoOperations.
TCP/IP connection. All outstanding requests are add_ior_interceptor
canceled. Errors include:
org.omg.PortableInterceptor.ORBInitInfoOperations.
v A GIOP MessageError or unknown message type add_client_request_interceptor
v An IOException that is received while data was
being read from the socket org.omg.PortableInterceptor.ORBInitInfoOperations
.add_server_request_interceptor
v An unexpected error or exception that occurs during
message processing
The name() method of the interceptor input parameter
Applicable CORBA exception class: returned a null string.

© Copyright IBM Corp. 2003, 2006 321

CORBA minor codses

Applicable CORBA exception class:

UNEXPECTED_CHECKED_EXCEPTION
org.omg.CORBA.BAD_PARAM
Explanation: An unexpected checked exception was
User response: Change the interceptor implementation
caught during the servant_preinvoke method. This
so that the name() method returns a non-null string.
method is called before a locally optimized operation
The name attribute can be an empty string if the
call is made to an object of type class. This exception
interceptor is anonymous, but it cannot be null.
does not occur if the ORB and any Portable Interceptor
implementations are correctly installed. It might occur
ORB_CONNECT_ERROR_6 if, for example, a checked exception is added to the
Request interceptor operations and these higher level
Explanation: A servant failed to connect to a interceptors are called from a back level ORB.
server-side ORB.
Applicable CORBA exception class:
Applicable CORBA exception class: org.omg.CORBA.UNKNOWN
org.omg.CORBA.OBJ_ADAPTER
User response: The details of the caught exception are
User response: See the FFDC log for the cause of the written to the FFDC log. Check whether the class from
problem, then try restarting the application. which it was thrown is at the expected level.

POA_DISCARDING UNSPECIFIED_MARSHAL_25
Explanation: The POA Manager at the server is in the Explanation: This error can occur at the server side
discarding state. When a POA manager is in the while the server is reading a request, or at the client
discarding state, the associated POAs discard all side while the client is reading a reply. Possible causes
incoming requests (whose processing has not yet are that the data on the wire is corrupted, or the server
begun). For more details, see the section that describes and client ORB are not communicating correctly.
the POAManager Interface in the http:// Communication problems can caused when one of the
www.omg.org/cgi-bin/doc?formal/99-10-07. ORBs has an incompatibility or bug that prevents it
Applicable CORBA exception class: from conforming to specifications.
org.omg.CORBA.TRANSIENT Applicable CORBA exception class:
User response: Put the POA Manager into the active org.omg.CORBA.MARSHAL
state if you want requests to be processed. User response: Check whether the IIOP levels and
CORBA versions of the client and server are
RESPONSE_INTERRUPTED compatible. Try disabling fragmentation (set
com.ibm.CORBA.FragmentationSize to zero) to
Explanation: The client has enabled the determine whether it is a fragmentation problem. In
AllowUserInterrupt property and has called for an this case, analysis of CommTraces
interrupt on a thread currently waiting for a reply from (com.ibm.CORBA.CommTrace) might give extra
a remote method call. information.
Applicable CORBA exception class:
org.omg.CORBA.NO_RESPONSE
User response: None.

TRANS_NC_LIST_GOT_EXC
Explanation: An exception was caught in the
NameService while the NamingContext.List() method
was executing.
Applicable CORBA exception class:
org.omg.CORBA.INTERNAL
User response: The details of the caught exception are
written to the FFDC log. Review the details of the
original exception, and any further action that is
necessary.

322 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Appendix E. Environment variables
This appendix provides the following information about environment variables:
v “Displaying the current environment”
v “Setting an environment variable”
v “Separating values in a list”
v “JVM environment settings”
v “z/OS environment variables” on page 326

Displaying the current environment

To show the current environment, run:
set (Windows)
env (Linux)
set (z/OS)

To show a particular environment variable, run:

echo %ENVNAME% (Windows)
echo $ENVNAME (Linux and z/OS)

Use values exactly as shown in the documentation. The names of environment

variables are case-sensitive in Unix but not in Windows.

Setting an environment variable

To set the environment variable LOGIN_NAME to Fred, run:
set LOGIN_NAME=Fred (Windows)
export LOGIN_NAME=Fred (Unix ksh or bash shells)

These variables are set only for the current shell or command-line session.

Separating values in a list

If the value of an environment variable is to be a list:
v On Unix the separator is normally a colon (:).
v On Windows the separator is usually a semicolon (;).

JVM environment settings

Table 19 on page 324 summarizes common environment settings. It is not a
definitive guide to all the settings. Also, the behavior of individual platforms might
vary. Refer to individual sections for a more complete description of behavior and
availability of these variables.

© Copyright IBM Corp. 2003, 2006 323

Environment variables

Table 19. JVM environment settings — general options

Variable Name Variable Values Notes
CLASSPATH A list of directories for the JVM to find user Any classpath that is set in this way is
class files, paths, or both to individual .jar completely replaced by the -cp or
or .zip files that contain class files; for -classpath Java argument if used.
example, /mycode:/utils.jar (Unix),
D:\mycode;D:\utils.jar (Windows)
IBM_JAVA_ Set by the JVM after it starts, to enable you This is not available if the JVM is
COMMAND_LINE to find the command-line parameters set invoked using JNI.
when the JVM started.
IBM_JAVA_OPTIONS This variable can be used to store default Any options are overridden by
Java options. These can include -X, -D or equivalent options that are specified
-verbose:gc style options; for example, when Java is started.
-Xms256m
-Dibm.jvm.trusted.middleware.class.path Does not support ’-showversion’.

Note that if you specify the name of a

trace output file either directly, or
indirectly, through a properties file, that
output file might be accidentally
overwritten if you run utilities such as
the trace formatter, dump extractor, or
dump formatter. For information about
how to avoid this problem, see
Controlling the trace, Note 2 on page
262.
JAVA_ASSISTIVE To prevent the JVM from loading Java
Accessibility support, set the
JAVA_ASSISTIVE environment variable to
OFF.
JAVA_FONTS Define the font directory.
JAVA_MMAP_MAXSIZE Specifies the maximum size of zip or jar Default=0. This default disables memory
files in MB for which the JVM will use mapping.
memory mapping to open those files. Files
below this size are opened with memory
mapping; files above this size with normal
I/O.
JAVA_PLUGIN_AGENT Specify the vesion of Mozilla Linux and z/OS only.
JAVA_PLUGIN_REDIRECT If this variable is set to a non-null value, Linux and z/OS only.
JVM output, while serving as a plug-in, is
redirected to files. The standard output and
error are redirected to files plugin.out and
plugin.err respectively.
JAVA_ZIP_DEBUG Setting this to any value displays memory
map information as it is created.
LANG Specify a locale to use by default. Linux and z/OS only.
LD_LIBRARY_PATH This variable contains a colon-separated list Linux only.
of the directories from where system and
user libraries are loaded. You can change
which versions of libraries are loaded, by
modifying this list.

324 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Environment variables

Table 19. JVM environment settings — general options (continued)

Variable Name Variable Values Notes
LIBPATH This variable contains a colon-separated list z/OS only.
of the directories from where system and
user libraries are loaded. You can change
which versions of libraries are loaded, by
modifying this list.
SYS_LIBRARY_PATH Specify the library path. Linux and z/OS only.

Table 20. Basic JIT options

Variable Name Variable Values Notes
IBM_MIXED_ Threshold for method compilation. This is The default values are around 500-2000,
MODE_THRESHOLD the number of times a method or loop is depending on the platform. Equivalent
executed before it is considered for to –Xjit:count=<value>.
compiling. A value of 0 means that the
compiler attempts to compile every method
on its first invocation.
JAVA_COMPILER The runtime Java compiler to use. Default The runtime Java compiler to use.
value is jitc, the Just-In-Time compiler. A Default value is j9jit22, the Just-In-Time
value of NONE causes the Java bytecode to compiler. A value of NONE causes the
be interpreted only, not compiled. Java bytecode to be interpreted only, not
compiled.

Table 21. Javadump and Heapdump options

Variable Name Variable Values Notes
DISABLE_JAVADUMP Disables Javadump creation by setting to Use command-line option
true (case-sensitive). -Xdisablejavadump instead. Avoid
using this environment variable because
it makes it more difficult to diagnose
failures.

On z/OS, JAVA_DUMP_OPTS should

be used in preference.
IBM_HEAPDUMP or If set to anything, heapdump is enabled. See Chapter 22, “Using Heapdump,” on
IBM_HEAP_DUMP page 205.
If unset, heapdump is not enabled.
IBM_HEAPDUMP_ When set to TRUE or 1 - generates a
OUTOFMEMORY heapdump each time an out-of-memory
exception is thrown, even if it is handled.

When set to FALSE or 0 - heapdumps are

not generated for out-of-memory exception.

When not set - generates a heapdump

when an out-of-memory exception is not
caught and handled by the application.
IBM_NOSIGHANDLER Disables Java dump creation by setting to Equivalent to -Xrs:all
true.
IBM_JAVACOREDIR Specify an alternative location for On z/OS _CEE_DMPTARG is used
Javadump files; for example, on Linux instead.
IBM_JAVACOREDIR=/dumps

Appendix E. Environment variables 325

Environment variables

Table 21. Javadump and Heapdump options (continued)

Variable Name Variable Values Notes
IBM_JAVADUMP When set to TRUE or 1 - generates a
_OUTOFMEMORY Javadump each time an out-of-memory
exception is thrown, even if it is handled.

When set to FALSE or 0 - Javadumps are

not generated for out-of-memory exception.

When not set - generates a Javadump when

an out-of-memory exception is not caught
and handled by the application.
JAVA_DUMP_OPTS Controls how diagnostic data are dumped. Recommended default value is
described in Chapter 12, “First steps in
problem determination,” on page 91.
_JVM_THREAD Specify maximum size of Javadump file in Default is 512 KB.
_DUMP_BUFFER_SIZE bytes.
TMPDIR Specify an alternative temporary directory. Defaults to /tmp on Unix and \tmp on
This is used only in the case when Windows.
Javadumps and Heapdumps can be written
only to a temporary directory.

Table 22. Diagnostics options

Variable Name Variable Values Notes
IBM_JVM_DEBUG_PROG Launches the JVM under the specified Linux only.
debugger.
IBM_MALLOCTRACE Setting this variable to a non-null value Equivalent to -memorycheck.
enables the tracing of memory allocation in
the JVM.
IBM_XE_COE_NAME This environment variable generates a
system dump when the specified exception
occurs. The value supplied is the package
description of the exception; for example,
java/lang/InternalError
JAVA_PLUGIN_TRACE To take a Java plug-in trace, set By default, this setting is disabled.
JAVA_PLUGIN_TRACE=1 in a session in
which the application will be run. This
setting produces traces from both the Java
and Native layer.
LD_ASSUME_ KERNEL Disables floating stacks (Redhat only). See On Redhat distros with kernel levels
SDK User Guide. 2.4.0 through 2.4.10, you are
recommended to set this variable to the
value 2.2.5.

z/OS environment variables

JAVA_DUMP_OPTS
See Chapter 23, “JVM dump initiation,” on page 209 for details.
JAVA_DUMP_TDUMP_PATTERN=string
Result: The specified string is passed to IEATDUMP to use as the data/set
name for the Transaction Dump. The default string is:
%s.JVM.TDUMP.&JOBNAME..D&YYMMDD..T&HHMMSS

326 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Environment variables

where the hlq is found from the following C code fragment:

pwd = getpwuid(getuid());
pwd->pw_name;
JAVA_LOCAL_TIME
The z/OS JVM does not look at the offset part of the TZ environment
variable and will therefore incorrectly show the local time. Where local
time is not GMT, you can set the environment variable
JAVA_LOCAL_TIME to display the correct local time as defined by TZ.
JAVA_THREAD_MODEL
JAVA_THREAD_MODEL can be defined as one of:
NATIVE
JVM uses the standard, POSIX-compliant thread model that is
provided by the JVM. All threads are created as _MEDIUM_WEIGHT
threads.
HEAVY
JVM uses the standard thread package, but all threads are created as
_HEAVY_WEIGHT threads.
MEDIUM
Same as NATIVE.
NULL
Default case: Same as NATIVE/MEDIUM.

Appendix E. Environment variables 327

Environment variables

328 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Appendix F. Command-line options
You can specify the options on the command-line while you are starting Java. They
override any relevant environment variables. For example, using -cp <dir1> with
the Java command completely overrides setting the environment variable
CLASSPATH=<dir2>.

This chapter provides the following information:

v “Specifying command-line options”
v “General command-line options”
v “System property command-line options” on page 330
v “Nonstandard command-line options” on page 331
v “JIT command-line options” on page 333
v “Garbage Collector command-line options” on page 334

Specifying command-line options

Although the command-line is the traditional way to specify command-line
options, there are other ways of passing options to the JVM. These are governed by
precedence rules (in descending order) in the following list:
1. command-line.
For example, java -X<option> MyClass
2. A file containing a list of options, specified using the –Xoptionsfile option on
the command-line. For example, java -Xoptionsfile=myoptionfile.txt
MyClass
In the options file, specify each option on a new line; you can use the ’\’
character as a continuation character if you want a single option to span
multiple lines. Use the ’#’ character to define comment lines. Note that you
cannot specify -classpath in an options file. Here is an example of an options
file:
#My options file
-X<option1>
-X<option2>=\
<value1>,\
<value2>
-D<sysprop1>=<value1>
3. IBM_JAVA_OPTIONS environment variable. You can set command-line
options using this environment variable. The options you specify with this
environment variable are added to the command-line when a JVM starts in that
environment.
For example, set IBM_JAVA_OPTIONS=-X<option1> -X<option2>=<value1>

General command-line options

-cp, -classpath<directories and zip or jar files separated by ;> (or : on Linux and
z/OS)
Sets search path for application classes and resources.
-help, -?
Prints a usage message.

© Copyright IBM Corp. 2003, 2006 329

General command-line options

-showversion
Prints product version and continues.
-memorycheck[:quick | nofree]
Identifies memory leaks inside the JVM.
v quick – less extensive checks
v nofree – does not free reused memory
-verbose[:class | gc | jni]
Enables verbose output.
-verbose:dynload
Provides detailed information as each class is loaded by the JVM, including:
v The class name and package
v For class files that were in a .jar file, the name and directory path of the .jar
v Details of the size of the class and the time taken to load the class
The data is written out to stderr. An example of the output follows:
<Loaded java/lang/String from C:\sdk\jre\lib\vm.jar>
<Class size 17258; ROM size 21080; debug size 0>
<Read time 27368 usec; Load time 782 usec; Translate time 927 usec>
-verbose:Xclassdep
Traces all the class loading and the method and classnames with line numbers.
-version
Prints product version.

System property command-line options

-D<name>=<value>
Sets a system property.
-Dcom.ibm.cacheLocalHost=true
Multiple calls to the java.net.InetAddress.getLocalHost() can impact JVM
performance. Set this property to enable caching of the local host name.
-Dibm.jvm.bootclasspath
The value of this property is used as an additional search path, which is
inserted between any value that is defined by -Xbootclasspath/p: and the
bootclass path. The bootclass path is either the default, or that which you
defined by using the -Xbootclasspath: option.
-Dibm.stream.nio={true|false}
From v1.4.1 onwards, by default the IO converters are used. This option
addresses the ordering of IO and NIO converters. When this option is set to
true, the NIO converters are used instead of the IO converters.
-Djava.compiler={ NONE | j9jit22 | j9mjit22}
Disable the Java compiler by setting to NONE. Enable JIT compilation by
setting to j9jit22 (Equivalent to –Xjit). Enable MicroJIT compilation by setting
to j9mjit22 (Equivalent to –Xmjit).
-Djava.net.connectiontimeout={n}
’n’ is the number of seconds to wait for the connection to be established with
the server. If this option is not specified in the command-line, the default value
of 0 (infinity) is used. The value can be used as a timeout limit when an
asynchronous java-net application is trying to establish a connection with its
server. If this value is not set, a java-net application waits until the default
connection timeout value is met. For instance, java

330 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 System property command-line options

-Djava.net.connectiontimeout=2 TestConnect causes the java-net client

application to wait only 2 seconds to establish a connection with its server.
-Dsun.net.client.defaultConnectTimeout=<value in milliseconds>
This property specifies the default value for the connect timeout for the
protocol handlers used by the java.net.URLConnection class. The default value
set by the protocol handlers is -1, which means there is no timeout set.
When a connection is made by an applet to a server and the server does not
respond properly, the applet might appear to hang and might also cause the
browser to hang. This apparent hang occurs because there is no network
connection timeout. To avoid this problem, the Java Plug-in has added a
default value to the network timeout of 2 minutes for all HTTP connections.
You can override the default by setting this property.
-Dsun.net.client.defaultReadTimeout=<value in milliseconds>
This property specifies the default value for the read timeout for the protocol
handlers used by the java.net.URLConnection class when reading from an
input stream when a connection is established to a resource. The default value
set by the protocol handlers is -1, which means there is no timeout set.
-Dsun.rmi.transport.tcp.connectionPool={true | any non-null value}
Enables thread pooling for the RMI ConnectionHandlers in the TCP transport
layer implementation.
-Dswing.useSystemFontSettings={false}
From v1.4.1 onwards, by default, Swing programs running with the Windows
Look and Feel render with the system font set by the user instead of a
Java-defined font. As a result, fonts for v1.4.1 differ from those in prior
releases. This option addresses compatibility problems like these for programs
that depend on the old behavior. By setting this option, v1.4.1 fonts and those
of prior releases will be the same for Swing programs running with the
Windows Look and Feel.
-Dsun.net.client.defaultConnectTimeout=<value in milliseconds>
This property specifies the default value for the connect timeout for the
protocol handlers used by the java.net.URLConnection class. The default value
set by the protocol handlers is -1, which means there is no timeout set. When a
connection is made by an applet to a server and the server does not respond
properly, the applet might appear to hang and might also cause the browser to
hang. This apparent hang occurs because there is no network connection
timeout. To avoid this problem, the Java Plug-in has added a default value to
the network timeout of 2 minutes for all HTTP connections. You can override
the default by setting this property.
-Dsun.net.client.defaultReadTimeout=<value in milliseconds>
This property specifies the default value for the read timeout for the protocol
handlers used by the java.net.URLConnection class when reading from an
input stream when a connection is established to a resource. The default value
set by the protocol handlers is -1, which means there is no timeout set.

Nonstandard command-line options

The -X options are nonstandard and subject to change without notice.

Options that relate to the Garbage Collector are listed under “Garbage Collector
command-line options” on page 334.
-X
Prints help on nonstandard options.

Appendix F. Command-line options 331

System property command-line options

-Xbootclasspath:<directories and zip or jar files separated by ;> (or : on Linux

and z/OS)
Sets search path for bootstrap classes and resources.
-Xbootclasspath/a:<directories and zip or jar files separated by ;> (or : on Linux
and z/OS)
Appends to the bootstrap class path.
-Xbootclasspath/p:<directories and zip or jar files separated by ;> (or : on Linux
and z/OS)
Prepends to the bootstrap class path.
-Xcheck:jni[:help][:<option>=<value>]
Performs additional checks for JNI functions. (Equivalent to -Xrunjnichk.)
-Xdbg:<options>
Loads debugging libraries to support remote debug applications. (Equivalent
to -Xrunjdwp.)
-Xdbginfo:<symbol file path>
Loads the debug info server with the symbol path specified.
-Xdebug
Enables remote debugging.
-Xdisablejavadump
Disables the Javadump facility.
-Xdump[:help] | [:<option>=<value>]
See Chapter 24, “Using dump agents,” on page 213.
-Xfuture
Enables strictest checks, anticipating future default.
-Xiss<size>
Sets the initial stack size for Java threads.
-Xmso<size>
Sets the default stack size for Operating System threads (format =
nn[K|M|G]).
-Xnoagent
Disables support for the oldjdb debugger.
-Xoptionsfile=<file>
Specify a file with command-line options to use.
-Xoss<size>
Recognized but DEPRECATED. Use -Xss and -Xmso. Sets maximum Java stack
size for any thread (format = nn[K|M|G]).
-Xrdbginfo:<host>:<port>
Loads the remote debug info server with the specified host and port.
-Xrs
Reduces the use of operating system signals. This prevents the JVM from
installing signal handlers for all but exception type signals (such as SIGSEGV,
SIGILL, SIGFPE.)

Note: Linux always uses SIGU3R1 and SIGU3R2.

-Xrunhprof[:help] | [:<option>=<value>, ...]
Performs heap, CPU, or monitor profiling.

332 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 System property command-line options

-Xrunjdwp[:help] | [:<option>=<value>, ...]

Loads debugging libraries to support remote debug applications.
-Xrunjnichk[:help] | [:<option>=<value>, ...]
Performs additional checks for JNI functions. (Equivalent to -Xcheck:jni.)
-Xss<size>
Sets maximum stack size for Java threads (format = nn[K|M|G]).
-Xnosigchain
Disables JVM signal handler chaining. The default is -Xnosigchain for z/OS,
-Xsigchain for all other platforms.
-Xsigchain
Enables JVM signal handler chaining.
-Xtrace[:help] | [:<option>=<value>, ...]
See page “Specifying trace options” on page 262.
-Xverify[:<option>=<value>, ...]
With no parameters, enables the verifier. Note that this is the default in all J2SE
JVMs; used on its own, this option has no effect. Optional parameters are:
v all – enable maximum verification
v none – disable the verifier

JIT command-line options

You might need to read Chapter 27, “JIT problem determination,” on page 237 to
understand some of the references that are given here. The following list contains
all the JIT command-line options that are available in this release.
-Xint
Interpreter only. Turns off the JIT and ahead-of-time compilation (AOT)
support.
-Xjit [:<option>=<value>, ...]
With no parameters, it enables the JIT. Note that the JIT is enabled by default
in all J2SE JVMs, so using this option on its own has no effect. Use this option
to control the behavior of the JIT. Useful parameters are:
v count=<n> – where <n> is the number of times a method is invoked before
it is compiled. For example, setting count=0 forces the JIT to compile
everything on first execution.
v optlevel=[noOpt | cold | warm | hot | veryHot | scorching] – forces the
JIT to compile all methods at a specific optimization level.
v verbose – displays information about the JIT configuration and method
compilation.
-Xmjit [:<option>=<value>, ...]
With no parameters, it enables the MicroJIT (if present). This option is
mutually exclusive with -Xjit so you can either enable the JIT only or the
MicroJIT only. Useful parameters are:
v count=<n> – where <n> is the number of times a method is invoked before
it is compiled. For example, setting count=0 forces the JIT to compile
everything on first execution.
v verbose – displays information about the JIT configuration and method
compilation.
-Xnojit
Turns off the JIT (AOT support still loads).

Appendix F. Command-line options 333

JIT command-line options

-Xnoaot
Turns off AOT support (JIT still loads).
-Xquickstart
Used for improving startup time of some Java applications. -Xquickstart causes
the JIT to run with a subset of optimizations; that is, a quick compile. This
quick compile allows for improved startup time. -Xquickstart is appropriate
for shorter running applications, especially those where execution time is not
concentrated into a small number of methods. -Xquickstart can degrade
performance if it is used on longer-running applications that contain hot
methods. The implementation of -Xquickstart is subject to change in future
releases.

Garbage Collector command-line options

You might need to read Chapter 2, “Understanding the Garbage Collector,” on
page 7 to understand some of the references that are given here. The following list
contains all the Garbage Collector command-line options that are available in this
release.

General Garbage Collection options

Key:
v <value> — integer value
v <percent> — integer value in the range of 0--100 (inclusive)
v <age> — integer value in the range of 1--14 (inclusive)
v <time> — integer value (in milliseconds)
-Xms<value>
Sets the initial memory size

scavenger enabled: minimum size 8KB (Xms >= Xmn + Xmo)

scavenger disabled: minimum size 8 KB (Xms >= Xmo)
-Xmx<value>
Sets the maximum memory size (Xmx >= Xms)

scavenger enabled: minimum size 8KB

scavenger disabled: minimum size 8 KB

Examples of the use of -Xms and -Xmx are:

-Xms2m -Xmx64m
Heap starts at 2 MB and grows to a maximum of 64 MB.
-Xms100m -Xmx100m
Heap starts at 100 MB and never grows.
-Xms20m -Xmx1024m
Heap starts at 20 MB and grows to a maximum of 1 GB.
-Xms50m
Heap starts at 50 MB and grows to the default maximum.
-Xmx256m
Heap starts at default initial value and grows to a maximum of 256 MB.
-Xmos<value>
Sets the initial size of the old space. The minimum size is 2048 bytes.

334 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Garbage Collector command-line options

-Xmox<value>
Sets the maximum size of the old space.
-Xmo<value>
Sets both –Xmos and –Xmox.
-Xmns<value>
Sets the initial size of the new space. The minimum size is 4096 bytes.

scavenger disabled: ignored

tiny enabled: ignored
-Xmnx<value>
Sets the maximum size of the new space.

scavenger disabled: ignored

tiny enabled: ignored
-Xmn<value>
Sets both –Xmns and –Xmnx.

scavenger disabled: ignored

tiny enabled: ignored
-Xmca<value>
Increments the RAM class segment by the specified <value>.
-Xmco<value>
Increments the ROM class segment by the specified <value>.
-Xmoi<value>
Increments the old space by the specified <value>. If you specify a <value> of
0, the old space cannot be expanded anymore.
-Xmine<value>
Sets the minimum size of heap expansion.
-Xmaxe<value>
Sets the maximum size of heap expansion.
-Xminf<value>
Sets the percentage of the minimum free heap size after a garbage collection.
-Xmaxf<value>
Sets the percentage of the maximum free heap size after a garbage collection.
-Xnoclassgc
Disables dynamic class unloading.
-Xclassgc
Enables dynamic class unloading checks during garbage collection.
-Xalwaysclassgc
Always perform dynamic class unloading checks during global collection.
-Xgcpolicy:<optthruput | optavgpause| gencon>
optthruput — flat heap (no scavenger, no concurrent mark). This is the default
setting.
optavgpause — concurrent mark
gencon — scavenger, concurrent mark

Appendix F. Command-line options 335

Garbage Collector command-line options

-Xdisableexplicitgc
Disables system garbage collection
-Xverbosegclog:<path to file><filename>
Causes verboseGC output to be written to the specified file. If the file cannot
be found, verboseGC tries to create the file, and then continues as normal if it
is successful. If it cannot create the file (for example, if an invalid filename is
passed into the command), it will redirect the output to stderr.
-Xverbosegclog:<path to file><filename, X, Y>
Filename must contain a ″#″ (hash symbol), which is substituted with a
generation identifier, starting at 1. X and Y are integers. This option works
similarly to -Xverbosegclog:<path to file><filename>, but, in addition, the
verboseGC output is redirected to X files, each containing verboseGC output
from Y GC cycles.

Scavenger options
-Xmr<value>
Sets the remembered size setting
-Xmrx<value>
Sets the remembered maximum size setting

Compact options
default (no compaction option specified)
Makes the GC compact based on a series of triggers that attempt to compact
only when it is beneficial to the future performance of the JVM. No forced
compactions
-Xnocompactexplicitgc
Never runs compaction on system garbage collections. Compaction takes place
on global garbage collections if you specify -Xcompactgc or if compaction
triggers are met.
-Xcompactexplicitgc
Compacts on all system garbage collections. Compaction takes place on global
garbage collections if you specify -Xcompactgc specified or if compaction
triggers are met.
-Xcompactgc
Compacts on all garbage collections (system and global).
-Xnocompactgc
Disables compaction on all garbage collections (system or global).

Concurrent options
-Xgcpolicy:optthruput
Disables concurrent mark.
-Xgcpolicy:optavgpause
Enables concurrent mark.

Trace GC options
java -Xtgc:<arguments> MyClass

where <arguments> is a comma-separated list containing one or more of the

following arguments:

336 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Garbage Collector command-line options

backtrace
Before a garbage collection, a single line is printed containing the name of the
master thread for garbage collection, as well as the value of the osThread slot
in its J9VMThread structure.
compaction
Prints extra information showing the relative time spent by threads in the
″move″ and ″fixup″ phases of compaction
concurrent
Prints extra information showing the activity of the concurrent mark
background thread
dump
Prints a line of output for every free chunk of memory in the system, including
″dark matter″ (free chunks that are not on the free list for some reason, usually
because they are too small). Each line contains the base address and the size in
bytes of the chunk. If the chunk is followed in the heap by an object, the size
and class name of the object is also printed. Similar to terse.
freeList
Before a garbage collection, prints information about the free list and allocation
statistics since the last GC. Prints the number of items on the free list,
including ″deferred″ entries (with the scavenger, the unused space is a deferred
free list entry). For TLH and non-TLH allocations, prints the total number of
allocations, the average allocation size, and the total number of bytes discarded
in during allocation. For non-TLH allocations, also included is the average
number of entries that were searched before a sufficiently large entry was
found.
parallel
Produces statistics on the activity of the parallel threads during the mark and
sweep phases of a global GC.
references
Prints extra information every time a reference object is enqueued for
finalisation, showing the reference type, reference address, and referent
address.
scavenger
Prints extra information following each scavenger collection. A histogram is
produced showing the number of instances of each class (and their relative
ages) present in the survivor space. The space is linearly walked to achieve
this.
terse
Dumps the contents of the entire heap before and after a garbage collection.
For each object or free chunk in the heap, a line of trace output is produced.
Each line contains the base address, ″a″ if it is an allocated object, and ″f″ if it
is a free chunk, the size of the chunk in bytes, and if it is an object, its class
name.

Appendix F. Command-line options 337

Garbage Collector command-line options

338 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Appendix G. Default settings for the JVM
This appendix shows the default settings that the JVM uses; that is, how the JVM
operates if you do not apply any changes to its environment. The tables show the
JVM operation and the default setting.

The last column shows how the operation setting is affected and is set as follows:
v e – setting controlled by environment variable only
v c – setting controlled by command-line parameter or the IBM_JAVA_OPTIONS
environment variable
v ec– setting controlled by both (command-line always takes precedence) All the
settings are described elsewhere in this document. These tables are only a quick
reference to the JVM vanilla state
For default GC settings, see Chapter 2, “Understanding the Garbage Collector,” on
page 7.
Table 23. Cross platform defaults
JVM setting Default Setting
affected by
Javadumps Enabled ec
Javadumps on out of memory Enabled ec
Heapdumps Disabled ec
Heapdumps on out of memory Enabled ec
Sysdumps Enabled ec
Where dump files appear Current directory ec
Verbose output Disabled c
Boot classpath search Disabled c
JNI checks Disabled c
Remote debugging Disabled c
Strict conformancy checks Disabled c
Quickstart Disabled c
Remote debug info server Disabled c
Reduced signalling Disabled c
Signal handler chaining Enabled c
Classpath Not set ec
Accessibility support Enabled e
JIT Enabled ec
AOT compiler Enabled c
JIT debug options Disabled c
Java2D max size of fonts with algorithmic bold 14 point e
Java2D use rendered bitmaps in scalable fonts Enabled e
Java2D freetype font rasterizing Enabled e
Java2D use AWT fonts Disabled e

© Copyright IBM Corp. 2003, 2006 339

Default settings for the JVM

Table 24. Platform specific defaults

JVM Setting AIX Linux Windows z/OS Setting
affected
by
Default locale None None N/A None e
Time to wait before starting N/A Zero N/A N/A e
plug-in
Temporary directory /tmp /tmp \tmp /tmp e
Plug-in redirection None None N/A None e
IM switching Disabled Disabled N/A Disabled e
IM modifiers Disabled Disabled N/A Disabled e
Thread model N/A N/A N/A Native e
Initial stack size for Java 2k 2k 2k 2k c
Threads 32-bit
Maximum stack size for Java 256k 256k 256k 256k c
Threads 32-bit
Stack size for OS Threads 32-bit 256k 256k 32k 32k c
Initial stack size for Java N/A 2k 2k 2k c
Threads 64-bit
Maximum stack size for Java 512k 512k 512k 512k c
Threads 64-bit
Stack size for OS Threads 64-bit N/A 256k 32k 32k c

340 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Appendix H. Notices
This information was developed for products and services offered in the U.S.A.
IBM may not offer the products, services, or features discussed in this document in
other countries. Consult your local IBM representative for information on the
products and services currently available in your area. Any reference to an IBM
product, program, or service is not intended to state or imply that only that IBM
product, program, or service may be used. Any functionally equivalent product,
program, or service that does not infringe any IBM intellectual property right may
be used instead. However, it is the user’s responsibility to evaluate and verify the
operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter
described in this document. The furnishing of this document does not give you
any license to these patents. You can send license inquiries, in writing, to:
IBM Director of Licensing
IBM Corporation
North Castle Drive
Armonk, NY 10504-1785
U.S.A.

For license inquiries regarding double-byte (DBCS) information, contact the IBM
Intellectual Property Department in your country or send inquiries, in writing, to:
IBM World Trade Asia Corporation
Licensing
2-31 Roppongi 3-chome, Minato-ku
Tokyo 106-0032, Japan

The following paragraph does not apply to the United Kingdom or any other
country where such provisions are inconsistent with local law:
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS
PUBLICATION “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER
EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS
FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or
implied warranties in certain transactions, therefore, this statement may not apply
to you.

This information could include technical inaccuracies or typographical errors.

Changes are periodically made to the information herein; these changes will be
incorporated in new editions of the publication. IBM may make improvements
and/or changes in the product(s) and/or the program(s) described in this
publication at any time without notice.

Any references in this information to non-IBM Web sites are provided for
convenience only and do not in any manner serve as an endorsement of those Web
sites. The materials at those Web sites are not part of the materials for this IBM
product and use of those Web sites is at your own risk.

IBM may use or distribute any of the information you supply in any way it
believes appropriate without incurring any obligation to you.

© Copyright IBM Corp. 2003, 2006 341

Notices

Licensees of this program who wish to have information about it for the purpose
of enabling: (i) the exchange of information between independently created
programs and other programs (including this one) and (ii) the mutual use of the
information which has been exchanged, should contact IBM United Kingdom
Laboratories, MP146, Hursley Park, Winchester, Hampshire, SO21 2JN, United
Kingdom. Such information may be available, subject to appropriate terms and
conditions, including in some cases, payment of a fee.

The licensed program described in this information and all licensed material
available for it are provided by IBM under terms of the IBM Customer Agreement,
IBM International Program License Agreement, or any equivalent agreement
between us.

Information concerning non-IBM products was obtained from the suppliers of

those products, their published announcements or other publicly available sources.
IBM has not tested those products and cannot confirm the accuracy of
performance, compatibility or any other claims related to non-IBM products.
Questions on the capabilities of non-IBM products should be addressed to the
suppliers of those products.

COPYRIGHT LICENSE:

This information contains sample application programs in source language, which

illustrate programming techniques on various operating platforms. You may copy,
modify, and distribute these sample programs in any form without payment to
IBM, for the purposes of developing, using, marketing or distributing application
programs conforming to the application programming interface for the operating
platform for which the sample programs are written. These examples have not
been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or
imply reliability, serviceability, or function of these programs. You may copy,
modify, and distribute these sample programs in any form without payment to
IBM for the purposes of developing, using, marketing, or distributing application
programs conforming to IBM’s application programming interfaces.

Each copy or any portion of these sample programs or any derivative work, must
include a copyright notice as follows:

© (your company name) (year). Portions of this code are derived from IBM Corp.
Sample Programs. © Copyright IBM Corp. _enter the year or years_. All rights
reserved.

If you are viewing this information softcopy, the photographs and color
illustrations may not appear.

Trademarks
The following terms are trademarks or registered trademarks of International
Business Machines Corporation in the United States, or other countries, or both.

IBM WebSphere

z/OS OS/390

Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the
United States, other countries, or both.

342 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
 Trademarks

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of
Microsoft Corporation in the United States, other countries, or both.

Linux is a trademark of Linus Torvalds in the United States, other countries, or

both.

Other company, product and service names may be trademarks or service marks of
others.

Appendix H. Notices 343

344 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Index
Special characters agent, JVMRI
building
-J-Djavac.dump.stack=1 167 Linux 286
-verbose:dynload 257 Windows 285
-verbosegc (garbage collection) 242 z/OS 286
-Xclassgc launching 285
garbage collection 250 writing 283
-Xcompactexplicitgc AIX
garbage collection 250 available disk space 97
-Xcompactgc checking environment 95
garbage collection 250 crashes 107
-Xdisableexplicitgc debugging commands 97
garbage collection 249 archon 102
-Xgcthreads band 103
garbage collection 250 bindprocessor 97
-Xnoclassgc bindprocessor –q 97
garbage collection 250 bootinfo 97
-Xnocompactexplicitgc cmd 102
garbage collection 250 cp 102
-Xnocompactgc dbx 97
garbage collection 250 dbx Plug-in 106
-Xtgc:backtrace Esid 104
garbage collection 250 f 102
-Xtgc:compaction iostat 98
garbage collection 251 lsattr 98
-Xtgc:concurrent netpmon 99
garbage collection 251 netstat 100
-Xtgc:dump nmon 101
garbage collection 251 pid 101
-Xtgc:excessiveGC ppid 101
garbage collection 252 pri 102
-Xtgc:freelist ps 101
garbage collection 252 sar 103
-Xtgc:parallel sc 102
garbage collection 253 st 102
-Xtgc:references stime 102
garbage collection 253 svmon 103
-Xtgc:scavenger tat 103
garbage collection 253 tid 102
-Xtgc:terse time 102
garbage collection 254 topas 105
-Xtrace 167 tprof 105
-Xverify 257 trace 105
*.nix platforms truss 105
font utilities 181 tty 102
–Xdump 224 Type 104
see also jdmpview 223 uid 101
user 102
vmstat 106
Numerics Vsid 104
32- and 64-bit JVMs debugging hangs 109
AIX 112 AIX deadlocks 109
32-bit AIX Virtual Memory Model, AIX 112 busy hangs 109
64-bit AIX Virtual Memory Model, AIX 113 poor performance 111
debugging memory leaks
32- and 64-bit JVMs 112
A 32-bit AIX Virtual Memory Model 112
64-bit AIX Virtual Memory Model 113
about this book xiii changing the Memory Model (32-bit JVM) 113
Addr Range, AIX segment type 104 fragmentation problems 117
agent design, JVMRI (launching) 286 Java heap exhaustion 117

© Copyright IBM Corp. 2003, 2006 345

AIX (continued) application trace (continued)
debugging memory leaks (continued) printf specifiers 280
Java or native heap exhausted 117 registering 277
Java2 32-Bit JVM default memory models 114 suspend or resume 280
monitoring the Java heap 116 trace api 280
monitoring the native heap 114 trace buffer snapshot 280
native and Java heaps 114 tracepoints 277
native heap exhaustion 117 using at runtime 279
native heap usage 115 applications trace, cross-platform tools 190
receiving OutOfMemory errors 116 archon, AIX 102
submitting a bug report 118
debugging performance problems 119
application profiling 124
collecting data from a fault condition 125
B
bad performance hangs, z/OS 161
CPU bottlenecks 120
BAD_OPERATION 168
finding the bottleneck 119
BAD_PARAM 168
getting AIX technical support 125
band, AIX 103
I/O bottlenecks 124
before you read this book xiii
JIT compilation 124
bindprocessor –q, AIX 97
JVM heap sizing 124
bindprocessor, AIX 97
memory bottlenecks 123
bootinfo , AIX 97
debugging techniques 97
bottlenecks, AIX
diagnosing crashes 107
CPU 120
documents to gather 107
finding 119
locating the point of failure 108
I/O 124
enabling full AIX core files 96
memory 123
Heapdumps 97
buffers
Java Virtual Machine settings 96
snapping 260
Javadumps 97
trace 260
MALLOCTYPE=watson 116
bug report
operating system settings 96
garbage collection 19
problem determination 95
busy hangs, AIX 109
setting up and checking AIX environment 95
stack trace 108
technical support 125
understanding memory usage 111 C
allocation failures 243 cache allocation (garbage collection) 10
analyzing deadlocks, Windows 147 cancel request header 319
API calls, JVMRI 286 categorizing problems 187
CreateThread 286 changing the Memory Model (32-bit JVM), AIX 113
DumpDeregister 287 changing the trace on a running server, ORB 316
DumpRegister 287 checking and setting up environment, Windows 143, 144
dynamic verbosegc 287 checklist for problem submission 79
GenerateHeapdump 288 before you submit 79
GenerateJavacore 288 data to include 79
GetComponentDataArea 288 factors that affect JVM performance 80
GetRasInfo 288 performance problem questions 81
InitiateSystemDump 289 test cases 80
InjectOutOfMemory 289 class loader 6
InjectSigsegv 289 how to write a custom class loader 27
NotifySignal 289 name spaces and the runtime package 26
ReleaseRasInfo 290 parent-delegation model 25
RunDumpRoutine 290 understanding 25
SetOutOfMemoryHook 290 why write your own class loader? 26
TraceDeregister 290 class loader diagnostics
TraceRegister 291 loading from native code 258
TraceResume 291 runtime 257
TraceResumeThis 291 class-loader diagnostics 257
TraceSet 291 command-line options 257
TraceSnap 292 classloaders and classes, Javadump
TraceSuspend 292 z/OS 201
TraceSuspendThis 292 client side interception points, ORB 52
application profiling, AIX 124 receive_exception (receiving reply) 52
application stack 4 receive_other (receiving reply) 52
application trace 277 receive_reply (receiving reply) 52
activating and deactivating tracepoints 280 send_poll (sending request) 52
example 278 send_request (sending request) 52

346 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
client side, ORB 43 compaction phase (garbage collection) 8
getting hold of the remote object 44 detailed description 13
bootstrap process 45 compatibility tables
ORB initialization 44 WebSphere Application Server and JVM/SDK levels 313
remote method invocation 46 compilation failures, JIT 239
delegation 46 COMPLETED_MAYBE 169
servant 46 COMPLETED_NO 169
stub creation 43 COMPLETED_YES 169
client, ORB 173 completion status, ORB 169
clnt , AIX segment type 104 component dump (LOCKS), Javadump 195
cmd, AIX 102 concurrent mark (garbage collection) 12
codes, minor (CORBA) 321 connection handlers 72
collecting additional diagnostic data, Linux 139 console dumps 215
collecting data from a fault condition control flow optimizations (JIT) 30
AIX 125 conventions and terminology in book xiv
Linux 138 CORBA 33
collecting additional diagnostic data 139 client side interception points 52
core files 138 receive_exception (receiving reply) 52
determining the operating environment 139 receive_other (receiving reply) 52
proc file system 139 receive_reply (receiving reply) 52
producing Javadumps 138 send_poll (sending request) 52
sending information to Java Support 139 send_request (sending request) 52
strace, ltrace, and mtrace 140 examples 34
using system logs 139 fragmentation 51
Windows 150 further reading 34
z/OS 164 interfaces 34
com.ibm.CORBA.AcceptTimeout 40 interoperable naming service (INS) 54
com.ibm.CORBA.AllowUserInterrupt 40 Java IDL or RMI-IIOP, choosing 34
com.ibm.CORBA.BootstrapHost 40 Linux 141
com.ibm.CORBA.BootstrapPort 40 minor codes 321
com.ibm.CORBA.BufferSize 40 portable interceptors 51
com.ibm.CORBA.CommTrace 167 portable object adapter 49
com.ibm.CORBA.ConnectTimeout 40 remote object implementation or servant 35
com.ibm.CORBA.Debug 167 RMI and RMI-IIOP 33
com.ibm.CORBA.Debug.Output 167 RMI-IIOP limitations 34
com.ibm.CORBA.enableLocateRequest 41 server code 36
com.ibm.CORBA.FragmentSize 41 differences between RMI (JRMP) and RMI-IIOP 39
com.ibm.CORBA.FragmentTimeout 41 summary of differences in client development 39
com.ibm.CORBA.GIOPAddressingDisposition 41 summary of differences in server development 39
com.ibm.CORBA.InitialReferencesURL 41 server side interception points 52
com.ibm.CORBA.ListenerPort 41 receive_request (receiving request) 52
com.ibm.CORBA.LocalHost 41 receive_request_service_contexts (receiving request) 52
com.ibm.CORBA.LocateRequestTimeout 41, 174 send_exception (sending reply) 52
com.ibm.CORBA.MaxOpenConnections 41 send_other (sending reply) 52
com.ibm.CORBA.MinOpenConnections 41 send_reply (sending reply) 52
com.ibm.CORBA.NoLocalInterceptors 42 stub and ties generation 35
com.ibm.CORBA.ORBCharEncoding 42 CORBA GIOP message format 317
com.ibm.CORBA.ORBWCharDefault 42 cancel request header 319
com.ibm.CORBA.RequestTimeout 42, 174 fragment header 320
com.ibm.CORBA.SendingContextRunTimeSupported 40 fragment message 320
com.ibm.CORBA.SendVersionIdentifier 42 GIOP header 317
com.ibm.CORBA.ServerSocketQueueDepth 42 locate reply body 320
com.ibm.CORBA.ShortExceptionDetails 42 locate reply header 320
com.ibm.tools.rmic.iiop.Debug 42 locate request header 319
com.ibm.tools.rmic.iiop.SkipImports 42 reply body 319
comm trace , ORB 172 reply header 318
COMM_FAILURE 168 request body 318
command-line options 329 request header 318
class-loader 257 core dumps
garbage collector 334 Linux 129
general 329 core files
JIT 333 Linux 127
nonstandard 331 core files, Linux 138
system property 330 cp, AIX 102
command-line parameters, JVM CPU bottlenecks, AIX 120
cross-platform tools 190 CPU usage, Linux 136
commands, (IPCS), z/OS 152

Index 347
crashes debugging memory leaks, AIX (continued)
AIX 107 32-bit AIX Virtual Memory Model 112
Linux 134 64-bit AIX Virtual Memory Model 113
Windows 146 changing the Memory Model (32-bit JVM) 113
z/OS 153 fragmentation problems 117
documents to gather 153 Java heap exhaustion 117
failing function 154 Java or native heap exhausted 117
crashes, diagnosing Java2 32-Bit JVM default memory models 114
Windows monitoring the Java heap 116
sending data to IBM 147 monitoring the native heap 114
CreateThread, JVMRI 286 native and Java heaps 114
cross-platform tools native heap exhaustion 117
application trace 190 native heap usage 115
command-line parameters, JVM 190 receiving OutOfMemory errors 116
dump formatter 188 submitting a bug report 118
Heapdump 188 debugging memory leaks, Windows
Javacore 188 memory model 148
Javadump 188 tracing leaks 148
JPDA tools 189 debugging performance problem, AIX
JVM environment variables 190 application profiling 124
JVM trace 189 collecting data from a fault condition 125
JVMPI tools 188 CPU bottlenecks 120
JVMRI 189 finding the bottleneck 119
method trace 190 getting AIX technical support 125
I/O bottlenecks 124
JIT compilation 124
D JVM heap sizing 124
memory bottlenecks 123
data submission with problem report 85
debugging performance problems, AIX 119
javaserv (IBM internal only) 85
debugging performance problems, Linux
sending files to IBM support 86
CPU usage 136
data to be collected, ORB 177
JIT 138
DATA_CONVERSION 168
JVM performance 137
dbx Plug-in, AIX 106
memory usage 136
dbx, AIX 97
network problems 137
deadlocked hangs, z/OS 160
system performance 135
deadlocks 109, 193
debugging performance problems, Windows 149
deadlocks, Windows
data for bug report 149
debugging 147
frequently reported problems 150
debug properties, ORB 167
debugging techniques, AIX 97
com.ibm.CORBA.CommTrace 167
bindprocessor –q 97
com.ibm.CORBA.Debug 167
bootinfo 97
com.ibm.CORBA.Debug.Output 167
dbx 97
debugging commands
dbx Plug-in 106
AIX 97
debugging commands 97
bindprocessor –q 97
iostat 98
bootinfo 97
lsattr 98
dbx 97
netpmon 99
dbx Plug-in 106
netstat 100
iostat 98
nmon 101
lsattr 98
sar 103
netpmon 99
starting Heapdumps 97
netstat 100
starting Javadumps 97
nmon 101
topas 105
sar 103
tprof 105
topas 105
trace 105
tprof 105
truss 105
trace 105
vmstat 106
truss 105
debugging techniques, Linux
vmstat 106
ps command 129
Linux 130
vmstat command
debugging hangs, AIX 109
processes section 130
AIX deadlocks 109
debugging techniques, Windows 145
busy hangs 109
Dump Extractor 145
poor performance 111
Heapdumps 145
debugging hangs, Windows 147
Javadumps 145
debugging memory leaks, AIX
default memory models, Java2 32-Bit JVM (AIX) 114
32- and 64-bit JVMs 112

348 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
default settings, JVM 339 dump agents (continued)
delegation, ORB client side 46 heapdumps 216
deprecated Sun properties 42 help options 213
description string, ORB 170 Java dumps 216
Description, AIX segment type 104 removing 218
determining the operating environment, Linux 139 system dumps 215
df command, Linux 139 tool option 216
diagnosing crashes, AIX 107 triggering 214
documents to gather 107 types 214
locating the point of failure 108 using 213
diagnostic tools, ORB dump extraction
-J-Djavac.dump.stack=1 167 Windows 144
-Xtrace 167 dump extractor
diagnostics 185 Linux 128
diagnostics component 5 Dump Extractor
diagnostics options, JVM environment 326 Windows 145
diagnostics, class loader dump formatter 223
loading from native code 258 analyzing dumps 229
runtime 257 cross-platform tools 188
diagnostics, class-loader 257 example session 229
command-line options 257 problems to tackle with 224
diagnostics, overview 187 dump, generated (Javadump) 191
categorizing problems 187 dump, system monitor (JVM) 195
cross-platform tools 187 DumpDeregister, JVMRI 287
applications trace 190 DumpRegister, JVMRI 287
command-line parameters, JVM 190 dumps, setting up (z/OS) 152
dump formatter 188 dynamic verbosegc, JVMRI 287
Heapdump 188
Javadump (or Javacore) 188
JPDA tools 189
JVM environment variables 190
E
enabling full AIX core files 96
JVM trace 189
enabling trace at server startup, ORB 315
JVMPI tools 188
environment
JVMRI 189
checking on AIX 95
method trace 190
displaying current 323
platforms 187
JVM settings 323
differences between RMI (JRMP) and RMI-IIOP, ORB 39
basic JIT options 325
disabling the JIT 237
diagnostics options 326
Distributed Garbage Collection (DGC)
general options 324
RMI 72
Javadump and Heapdump options 325
documents to gather
setting up and checking on Windows 143, 144
AIX 107
setting up on Windows
z/OS 153
dump extraction 144
DTFJ
native tools 144
counting threads example 307
environment variables 323
diagnostics 303
JVM
interface diagram 305
cross-platform tools 190
overview 303
separating values in a list 323
working with a dump 303
setting 323
dump
z/OS 151, 326
events 209
environment, determining
initiation 209
Linux 139
overview 209
df command 139
JVM
free command 139
settings 210
lsof command 139
Linux 212
ps-ef command 139
platform-specific variations 211
top command 139
types 209
uname -a command 139
Windows 212
vmstat command 139
z/OS 211
error message IDs
dump (LOCKS component), Javadump 195
z/OS 153
dump agents
errors (OutOfMemory), receiving (AIX) 116
console dumps 215
Esid, AIX 104
default 217
example of real method trace 220
default settings 218
examples of method trace 220
examples 215
exceptions
filters 218
JNI 64

Index 349
exceptions, ORB 168 free command, Linux 139
completion status and minor codes 169 frequently reported problems, Windows 150
nested 171 frequently-asked questions
system 168 JIT 31
BAD_OPERATION 168 functions (table), JVMRI 286
BAD_PARAM 168
COMM_FAILURE 168
DATA_CONVERSION 168
MARSHAL 168
G
garbage collection 8
NO_IMPLEMENT 168
advanced diagnostics 249
UNKNOWN 168
-Xclassgc 250
user 168
-Xcompactexplicitgc 250
exhaustion of Java heap, AIX 117
-Xcompactgc 250
exhaustion of native heap, AIX 117
-Xdisableexplicitgc 249
explicit generation of a Heapdump 206
-Xgcthreads 250
external trace, JVMRI 294
-Xnoclassgc 250
-Xnocompactexplicitgc 250
-Xnocompactgc 250
F -Xtgc:backtrace 250
f, AIX 102 -Xtgc:compaction 251
failing function, z/OS 154 -Xtgc:concurrent 251
fault condition in AIX -Xtgc:dump 251
collecting data from 125 -Xtgc:excessiveGC 252
features, ORB 49 -Xtgc:freelist 252
client side interception points 52 -Xtgc:parallel 253
receive_exception (receiving reply) 52 -Xtgc:references 253
receive_other (receiving reply) 52 -Xtgc:scavenger 253
receive_reply (receiving reply) 52 -Xtgc:terse 254
send_poll (sending request) 52 TGC tracing 250
send_request (sending request) 52 allocation failures 243
fragmentation 51 allocation failures during concurrent mark 247
interoperable naming service (INS) 54 basic diagnostics (verbosegc) 242
portable interceptors 51 cache allocation 10
portable object adapter 49 coexisting with the Garbage Collector 18
server side interception points 52 bug reports 19
receive_request (receiving request) 52 finalizers 19
receive_request_service_contexts (receiving request) 52 finalizers and the garbage collection contract 20
send_exception (sending reply) 52 finalizers, summary 20
send_other (sending reply) 52 how finalizers are run 20
send_reply (sending reply) 52 manual invocation 20
file header, Javadump nature of finalizers 19
z/OS 198 summary 21
final section, Javadump thread local heap 18
z/OS 201 command-line options 334
finalizers 242 common causes of perceived leaks 241
floating stacks limitations, Linux 140 hash tables 242
font limitations, Linux 141 JNI references 242
font.properties file listeners 241
Linux font 180 objects with finalizers 242
Windows font 181 premature expectation 242
fonts static data 242
properties file 180 compaction phase 8
Linux font 180 detailed description 13
Windows font 181 concurrent mark 12, 246
fonts, NLS 179 concurrent kickoff 246
common problems 182 detailed description 10
installed 179 fine tuning options 17
properties 179 frequently asked questions 21
utilities 181 global collections 245
*.nix platforms 181 heap and native memory use by the JVM 254
Windows systems 181 large native objects 255
formatting, JVMRI 294 heap expansion 15
fragment header 320 heap lock allocation 10
fragment message 320 heap shrinkage 15
fragmentation heap size 8
AIX 117 problems 9
ORB 51, 166 how does it work? 241

350 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
garbage collection (continued) hangs (continued)
how to do heap sizing 16 z/OS (continued)
initial and maximum heap sizes 16 looping 160
interaction with applications 18 hangs, debugging
interaction with JNI 58 AIX 109
object references 58 AIX deadlocks 109
retained garbage 59 poor performance 111
JNI weak reference 15 Windows 147
mark phase 8 hash tables 242
detailed description 10 heap
mark stack overflow 11 expansion 15
parallel mark 11 shrinkage 15
memory allocation 9 heap (Java) exhaustion, AIX 117
native code 255 heap and native memory use by the JVM
nursery allocation failures 243 garbage collection
object allocation 7 large native objects 255
output from a System.gc() 242 heap and native memory use by the JVM, garbage
overview 7 collection 254
parallel bitwise sweep 13 heap lock 10
phantom reference 14 heap size
reachable objects 8 garbage collection 8
reference objects 14 problems 9
scavenger collections 245 heap sizing, garbage collection 16
soft reference 14 heap, verbose GC 208
sweep phase 8 Heapdump 205
detailed description 12 AIX, starting 97
System.gc() calls during concurrent mark 249 cross-platform tools 188
tenured allocation failures 244 enabling 205
understanding the Garbage Collector 7 explicit generation of 206
using verbosegc 17 Linux, starting 128
verbose, heap information 208 location of 207
weak reference 14 phd format 205
garbage collector previous releases 205
interaction with JNI summary 205
global references 59 triggered generation of 206
general debugging techniques, z/OS 152 using jdmpview 207
IPCS commands 152 Windows
GenerateHeapdump, JVMRI 288 starting 145
GenerateJavacore, JVMRI 288 heapdumps 216
generating a user dump file in a hang condition, heaps, native and Java
Windows 144 AIX 114
generation of a Heapdump how to read this book xiii
explicit 206 HPROF 297
location 207 output file 298
triggered 206 hung JVM
GetComponentDataArea, JVMRI 288 getting a dump from
GetRasInfo, JVMRI 288 Windows 147
getting a dump from a hung JVM, Windows 147
getting AIX technical support 125
getting files from IBM support 86
GIOP header 317
I
I/O bottlenecks, AIX 124
glibc limitations, Linux 140
InitiateSystemDump, JVMRI 289
global optimizations (JIT) 31
InjectOutOfMemory, JVMRI 289
global references (JNI) 64
InjectSigsegv, JVMRI 289
capacity 64
inlining (JIT) 30
INS, ORB
See interoperable naming service
H interceptors (portable), ORB 51
hanging, ORB 174 Interface Definition Language (IDL) 34
com.ibm.CORBA.LocateRequestTimeout 174 interoperable naming service (INS), ORB 54
com.ibm.CORBA.RequestTimeout 174 interpreter 6
hangs interpreting the stack trace, AIX 108
AIX Inuse, AIX segment type 104
busy hangs 109 iostat, AIX 98
z/OS 160 IPCS commands, z/OS 152
bad performance 161
deadlocked 160

Index 351
J JIT
command-line options 333
Java dumps 216 control flow optimizations 30
Java duty manager 77 frequently-asked questions 31
Java heap, AIX 114 global optimizations 31
exhaustion 117 how the JIT optimizes code 30
monitoring 116 inlining 30
Java Native Interface local optimizations 30
understanding 57 native code generation 31
Java or native heap exhausted, AIX 117 overview 29
Java service understanding 29
overview 77 JIT compilation
IBM service 77 AIX 124
Java duty manager 77 JIT compilation failures, identifying 239
submitting problem report to IBM 77 JIT options
Java Virtual Machine 187 JVM environment 325
JAVA_DUMP_OPTS 91, 210, 326 JIT problem determination 237
JAVA_LOCAL_TIME 327 disabling 237
JAVA_TDUMP_PATTERN=string 326 locating the failing method 238
JAVA_THREAD_MODEL 327 selectively disabling 237
Java2 32-Bit JVM default memory models, AIX 114 short-running applications 240
Java2 security permissions for the ORB 169 JIT problem, ORB 166
Javacore (cross-platform tools) 188 JIT, Linux 138
Javadump 191 JNI 57
classes, z/OS 201 checklist 68
classloaders 201 copying and pinning 60
cross-platform tools 188 debugging 67
enabling 191 exceptions 64
file header, title 198 garbage collection 15
final section generic use of isCopy and mode flags 66
z/OS 201 global references 64
interpreting 192 interaction with Garbage Collector 58
JVM system monitor dump 195 global references 59
location of generated dump 191 object references 58
LOCKS component dump 195 retained garbage 59
locks, monitors, and deadlocks (LOCKS) 193 isCopy flag 65
sample output local references 60
Linux 202 mode flag 65
Windows 202 problem determination 67
z/OS 196 synchronization 66
stack trace 199 understanding 57
storage management weak reference 15
z/OS 199 JNI references 242
system properties JPDA tools, cross-platform tools 189
z/OS 198 JVM
tags 193 API 5
threads, z/OS 199 application stack 4
triggering 192 building blocks 3
XHPI section 198 class loader 6
z/OS 198 diagnostics component 5
Javadumps interpreter 6
AIX 97 memory management 5
Linux 128 platform port layer 6
Windows 145 subcomponents 4
Javadumps, producing (Linux) 138 JVM dump initiation 209
jdmpview 223 Linux 212
commands 224 overview 209
general 224 platform-specific variations 211
heapdump 228 settings 210
memory analysis 226 Windows 212
trace 229 z/OS 211
working with classes 227 JVM heap sizing
working with objects 227 AIX 124
example session 229 JVM performance, Linux 137
jextract 223 JVM settings
what it is 223 environment 323
jextract 223 basic JIT options 325

352 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
JVM settings (continued)
environment (continued)
L
diagnostics options 326 large native objects
general options 324 heap and native memory use by the JVM
Javadump and Heapdump options 325 garbage collection 255
JVM system monitor dump 195 LE HEAP, z/OS 161
JVM trace, cross-platform tools 189 LE settings, z/OS 151
JVMPI 297 limitations, Linux 140
HPROF profiler 297 CORBA limitations 141
output file 298 floating stacks limitations 140
JVMPI tools, cross-platform tools 188 font limitations 141
JVMRI 283 glibc limitations 140
agent design 286 threads as processes 140
API calls 286 Linux
CreateThread 286 checking the system environment 134
DumpDeregister 287 collecting data from a fault condition 138
DumpRegister 287 collecting additional diagnostic data 139
dynamic verbosegc 287 core files 138
GenerateHeapdump 288 determining the operating environment 139
GenerateJavacore 288 proc file system 139
GetComponentDataArea 288 producing Javadumps 138
GetRasInfo 288 sending information to Java Support 139
InitiateSystemDump 289 strace, ltrace, and mtrace 140
InjectOutOfMemory 289 using system logs 139
InjectSigsegv 289 CORBA 141
NotifySignal 289 core files 127
ReleaseRasInfo 290 crashes, diagnosing 134
RunDumpRoutine 290 debugging commands 130
SetOutOfMemoryHook 290 gdb 131
TraceDeregister 290 ltrace tool 131
TraceRegister 291 mtrace tool 131
TraceResume 291 ps 130
TraceResumeThis 291 strace tool 131
TraceSet 291 tracing 131
TraceSnap 292 debugging hangs 135
TraceSuspend 292 debugging memory leaks 135
TraceSuspendThis 292 debugging performance problems 135
building the agent CPU usage 136
Linux 286 JIT 138
Windows 285 JVM performance 137
z/OS 286 memory usage 136
changing trace options 285 network problems 137
cross-platform tools 189 system performance 135
external trace 294 debugging techniques 128
formatting 294 finding out about the Java environment 134
functions (table) 286 gathering process information 134
launching the agent 285 Javadump sample output 202
RasInfo JVM dump initiation 212
request types 293 known limitations 140
structure 292 CORBA limitations 141
registering a trace listener 284 floating stacks limitations 140
writing an agent 283 font limitations 141
glibc limitations 140
threads as processes 140
ltrace 140
K mtrace 140
kernel, AIX segment type 104 nm command 129
known limitations, Linux 140 objdump command 129
CORBA limitations 141 problem determination 127
floating stacks limitations 140 ps command 129
font limitations 141 setting up and checking the environment 127
glibc limitations 140 starting heapdumps 128
threads as processes 140 starting Javadumps 128
strace 140
threading libraries 128
top command 129
tracing 131
using core dumps 129

Index 353
Linux (continued) memory leaks (continued)
using system logs 129 z/OS 161
using the dump extractor 128 LE HEAP 161
vmstat command 130 OutOfMemoryErrors 162
io section 130 virtual storage 161
memory section 130 memory leaks, debugging
processes section 130 AIX
swap section 130 32- and 64-bit JVMs 112
system section 130 32-bit AIX Virtual Memory Model 112
working directory 127 64-bit AIX Virtual Memory Model 113
Linux problem determination changing the Memory Model (32-bit JVM) 113
collecting data from a fault condition 138 Java heap exhaustion 117
core files 138 Java or native heap exhausted 117
determining the operating environment 139 Java2 32-Bit JVM default memory models 114
producing Javadumps 138 monitoring the Java heap 116
using system logs 139 monitoring the native heap 114
debugging performance problems native and Java heaps 114
CPU usage 136 native heap exhaustion 117
JIT 138 native heap usage 115
JVM performance 137 receiving OutOfMemory errors 116
memory usage 136 memory leaks, Windows
network problems 137 tracing 148
system performance 135 memory management 5
debugging techniques Memory Model (32-bit JVM), changing, AIX 113
ps command 129 memory model, Windows 148
known limitations 140 memory models, Java2 32-Bit JVM default (AIX) 114
CORBA limitations 141 memory usage, Linux 136
floating stacks limitations 140 memory usage, understanding
font limitations 141 AIX 111
glibc limitations 140 message format, CORBA GIOP 317
threads as processes 140 cancel request header 319
listeners 241 fragment header 320
local optimizations (JIT) 30 fragment message 320
local references (JNI) 60 GIOP header 317
capacity 63 locate reply body 320
manually handling 63 locate reply header 320
scope 60 locate request header 319
summary 63 reply body 319
locate reply body 320 reply header 318
locate reply header 320 request body 318
locate request header 319 request header 318
locating the failing method 238 message trace , ORB 171
location of generated Heapdump 207 method trace 219
LOCKS component dump, Javadump 195 advanced options 220
locks, monitors, and deadlocks (LOCKS), Javadump 193 cross-platform tools 190
looping hangs, z/OS 160 examples 220
lsattr, AIX 98 real example 220
lsof command, Linux 139 running with 219
ltrace, Linux 140 where output appears 220
minor codes , CORBA 321
minor codes, ORB 169
M mmap, AIX segment type 104
monitoring the Java heap, AIX 116
maintenance, z/OS 151
monitoring the native heap, AIX 114
MALLOCTYPE=watson 116
monitors, Javadump 193
mark phase (garbage collection) 8
monitors, registered (JVM system monitor dump) 195
concurrent mark 12
mtrace, Linux 140
detailed description 10
MustGather
parallel mark 11
collecting the correct data to solve problems 79
MARSHAL 168
memory allocation 9
cache allocation 10
heap lock allocation 10 N
memory bottlenecks, AIX 123 native code
memory leaks garbage collection 255
Windows native code generation (JIT) 31
classifying 148 native heap, AIX 114
debugging 147 exhaustion 117

354 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
native heap, AIX (continued) ORB (continued)
monitoring 114 CORBA (continued)
usage 115 RMI-IIOP limitations 34
native tools server code 36
Windows 144 stub and ties generation 35
nested exceptions, ORB 171 summary of differences in client development 39
netpmon, AIX 99 summary of differences in server development 39
netstat, AIX 100 debug properties 167
network problems, Linux 137 com.ibm.CORBA.CommTrace 167
NLS com.ibm.CORBA.Debug 167
font properties 179 com.ibm.CORBA.Debug.Output 167
font.properties file 180 debugging 165
Linux font 180 diagnostic tools
Windows font 181 -J-Djavac.dump.stack=1 167
fonts 179 -Xtrace 167
installed fonts 179 exceptions 168
problem determination 179 features 49
nmon, AIX 101 client side interception points 52
NO_IMPLEMENT 168 fragmentation 51
NotifySignal, JVMRI 289 interoperable naming service (INS) 54
portable interceptors 51
portable object adapter 49
O server side interception points 52
how it works 43
object allocation 7
identifying a problem 165
objects
fragmentation 166
reachable 8
JIT problem 166
objects with finalizers 242
ORB versions 166
objects, reference (garbage collection)
packaging 166
Garbage Collector interaction with JNI 58
platform-dependent problem 166
options
what the ORB component contains 165
command-line 329
what the ORB component does not contain 166
general 329
properties 40
nonstandard 331
RMI and RMI-IIOP
system property 330
differences between RMI (JRMP) and RMI-IIOP 39
JVM environment
examples 34
basic JIT 325
further reading 34
diagnostics 326
interfaces 34
general 324
introduction 33
method trace, advanced 220
remote object implementation or servant 35
options that control tracepoint selection 262
server code 36
options that indirectly affect tracepoint selection 263
stub and ties generation 35
ORB 33
summary of differences in client development 39
choosing Java IDL or RMI-IIOP 34
summary of differences in server development 39
client side 43
RMI-IIOP limitations 34
bootstrap process 45
security permissions 169
delegation 46
server side 47
getting hold of the remote object 44
processing a request 48
ORB initialization 44
servant binding 47
remote method invocation 46
servant implementation 47
servant 46
tie generation 47
stub creation 43
service: collecting data 176
common problems 174
data to be collected 177
client and server running, not naming service 175
preliminary tests 176
com.ibm.CORBA.LocateRequestTimeout 174
stack trace 170
com.ibm.CORBA.RequestTimeout 174
description string 170
hanging 174
nested exceptions 171
running the client with client unplugged 176
system exceptions 168
running the client without server 175
BAD_OPERATION 168
completion status and minor codes 169
BAD_PARAM 168
CORBA
COMM_FAILURE 168
differences between RMI (JRMP) and RMI-IIOP 39
DATA_CONVERSION 168
examples 34
MARSHAL 168
further reading 34
NO_IMPLEMENT 168
interfaces 34
UNKNOWN 168
introduction 33
traces 171
Java IDL or RMI-IIOP? 34
client or server 173
remote object implementation or servant 35

Index 355
ORB (continued) overview of diagnostics (continued)
traces (continued) cross-platform tools (continued)
comm 172 Heapdump 188
message 171 Javadump (or Javacore) 188
service contexts 173 JPDA tools 189
understanding JVM environment variables 190
client side interception points 52 JVM trace 189
features 49 JVMPI tools 188
fragmentation 51 JVMRI 189
how it works 43 method trace 190
interoperable naming service (INS) 54 platforms 187
portable interceptors 51
portable object adapter 49
processing a request 48
servant binding 47
P
packaging, ORB 166
servant implementation 47
parallel mark (garbage collection) 11
server side interception points 52
parameters
the client side 43
command-line
the server side 47
cross-platform tools, JVM 190
tie generation 47
parent-delegation model (class loader) 25
using 40
performance
user exceptions 168
factors 80
versions 166
questions to ask 81
ORB and WebSphere Application Server
performance problems, debugging
selecting ORB traces 316
AIX 119
tracing 315
application profiling 124
changing on a running server 316
collecting data from a fault condition 125
enabling at server startup 315
CPU bottlenecks 120
ORB component
finding the bottleneck 119
what it contains 165
getting AIX technical support 125
what it does not contain 166
I/O bottlenecks 124
ORB properties
JIT compilation 124
com.ibm.CORBA.AcceptTimeout 40
JVM heap sizing 124
com.ibm.CORBA.AllowUserInterrupt 40
memory bottlenecks 123
com.ibm.CORBA.BootstrapHost 40
Linux
com.ibm.CORBA.BootstrapPort 40
CPU usage 136
com.ibm.CORBA.BufferSize 40
JIT 138
com.ibm.CORBA.ConnectTimeout 40
JVM performance 137
com.ibm.CORBA.enableLocateRequest 41
memory usage 136
com.ibm.CORBA.FragmentSize 41
network problems 137
com.ibm.CORBA.FragmentTimeout 41
system performance 135
com.ibm.CORBA.GIOPAddressingDisposition 41
Windows 149
com.ibm.CORBA.InitialReferencesURL 41
data for bug report 149
com.ibm.CORBA.ListenerPort 41
frequently reported problems 150
com.ibm.CORBA.LocalHost 41
performance problems, z/OS 163
com.ibm.CORBA.LocateRequestTimeout 41
pers, AIX segment type 104
com.ibm.CORBA.MaxOpenConnections 41
Pgsp, AIX segment type 104
com.ibm.CORBA.MinOpenConnections 41
pid, AIX 101
com.ibm.CORBA.NoLocalInterceptors 42
Pin, AIX segment type 104
com.ibm.CORBA.ORBCharEncoding 42
platform-dependent problem, ORB 166
com.ibm.CORBA.ORBWCharDefault 42
platform-specific variations, JVM dump initiation 211
com.ibm.CORBA.RequestTimeout 42
platforms supported in diagnostics 187
com.ibm.CORBA.SendingContextRunTimeSupported 40
poor performance, AIX 111
com.ibm.CORBA.SendVersionIdentifier 42
portable interceptors, ORB 51
com.ibm.CORBA.ServerSocketQueueDepth 42
portable object adapter
com.ibm.CORBA.ShortExceptionDetails 42
ORB 49
com.ibm.tools.rmic.iiop.Debug 42
power management 260
com.ibm.tools.rmic.iiop.SkipImports 42
ppid, AIX 101
other sources of information xiv
preliminary tests for collecting data, ORB 176
OutOfMemory errors, receiving (AIX) 116
premature expectation 242
OutOfMemoryErrors, z/OS 162
pri, AIX 102
overview of diagnostics 187
private storage usage, z/OS 151
categorizing problems 187
problem determination
cross-platform tools 187
first steps 91
applications trace 190
Linux
command-line parameters, JVM 190
collecting additional diagnostic data 139
dump formatter 188
collecting data from a fault condition 138

356 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
problem determination (continued) problem determination (continued)
Linux (continued) z/OS (continued)
CORBA limitations 141 documents to gather 153
core files 138 environment variables 151
CPU usage 136 environment, checking 151
determining the operating environment 139 general debugging techniques 152
floating stacks limitations 140 IPCS commands 152
font limitations 141 LE settings 151
glibc limitations 140 maintenance 151
JIT 138 OutOfMemoryErrors 162
JVM performance 137 private storage usage 151
known limitations 140 process deadlocked 160
memory usage 136 process looping 160
network problems 137 setting up dumps 152
proc file system 139 TDUMPs 155
producing Javadumps 138 virtual storage 161
ps command 129 problem determination, JIT 237
sending information to Java Support 139 disabling the JIT 237
strace, ltrace, and mtrace 140 locating the failing method 238
system performance 135 selectively disabling the JIT 237
threads as processes 140 short-running applications 240
using system logs 139 problem report
ORB 165 advice 83
collecting data 176 before you submit 79
common problems 174 checklist 79
debug properties 167 contents 83
fragmentation 166 data to include 79
identifying the problem 165 escalating problem severity 84
interpreting ORB traces 171 factors that affect JVM performance 80
interpreting the stack trace 170 getting files from IBM support 86
JIT problem 166 overview 77
ORB exceptions 168 IBM service 77
ORB versions 166 Java duty manager 77
packaging 166 performance problem questions 81
platform-dependent problem 166 problem severity ratings 83
what ORB contains 165 submitting data 85
what ORB does not contain 166 javaserv (IBM internal only) 85
Windows 143 sending files to IBM support 86
analyzing deadlocks 147 using your own ftp server 87
classifying leaks 148 submitting to IBM service 77
collecting data from a fault condition 150 test cases 80
data for bug report 149 when you will receive your fix 87
debugging hangs 147 problem severity ratings 83
debugging memory leaks 147 escalating 84
debugging performance problems 149 problem submission
debugging techniques 145 advice 83
diagnosing crashes 146 data 85
Dump Extractor 145 javaserv (IBM internal only) 85
frequently reported problems 150 sending files to IBM support 86
getting a dump from a hung JVM 147 using your own ftp server 87
Heapdumps 145 escalating problem severity 84
Javadumps 145 getting files from IBM support 86
memory model 148 overview 77
native tools 144 IBM service 77
sending data to IBM 147 Java duty manager 77
setting up and checking environment 143, 144 problem severity ratings 83
setting up for dump extraction 144 raising a report 83
tracing leaks 148 sending to IBM service 77
z/OS when you will receive your fix 87
allocations to LE HEAP 161 problems, frequently reported (Windows) 150
badly-performing process 161 problems, ORB 174
collecting data 164 hanging 174
debugging hangs 160 proc file system, Linux 139
debugging memory leaks 161 process private, AIX segment type 104
debugging performance problems 163 processes section, vmstat command 130
determining the failing function 154 producing Javadumps, Linux 138
diagnosing crashes 153 properties, ORB 40

Index 357
properties, ORB (continued) ReportEnv (continued)
com.ibm.CORBA.AcceptTimeout 40 Windows 143
com.ibm.CORBA.AllowUserInterrupt 40 reporting problems in the JVM, summary xiv
com.ibm.CORBA.BootstrapHost 40 request body 318
com.ibm.CORBA.BootstrapPort 40 request header 318
com.ibm.CORBA.BufferSize 40 request types, JVMRI (RasInfo) 293
com.ibm.CORBA.ConnectTimeout 40 RMI 71
com.ibm.CORBA.enableLocateRequest 41 debugging applications 73
com.ibm.CORBA.FragmentSize 41 Distributed Garbage Collection (DGC) 72
com.ibm.CORBA.FragmentTimeout 41 examples 34
com.ibm.CORBA.GIOPAddressingDisposition 41 further reading 34
com.ibm.CORBA.InitialReferencesURL 41 implementation 71
com.ibm.CORBA.ListenerPort 41 interfaces 34
com.ibm.CORBA.LocalHost 41 introduction 33
com.ibm.CORBA.LocateRequestTimeout 41 remote object implementation or servant 35
com.ibm.CORBA.MaxOpenConnections 41 server code 36
com.ibm.CORBA.MinOpenConnections 41 differences between RMI (JRMP) and RMI-IIOP 39
com.ibm.CORBA.NoLocalInterceptors 42 summary of differences in client development 39
com.ibm.CORBA.ORBCharEncoding 42 summary of differences in server development 39
com.ibm.CORBA.ORBWCharDefault 42 stub and ties generation 35
com.ibm.CORBA.RequestTimeout 42 thread pooling 72
com.ibm.CORBA.SendingContextRunTimeSupported 40 RMI-IIOP
com.ibm.CORBA.SendVersionIdentifier 42 choosing against Java IDL 34
com.ibm.CORBA.ServerSocketQueueDepth 42 examples 34
com.ibm.CORBA.ShortExceptionDetails 42 further reading 34
com.ibm.tools.rmic.iiop.Debug 42 interfaces 34
com.ibm.tools.rmic.iiop.SkipImports 42 introduction 33
properties, system (Javadump), z/OS 198 limitations 34
ps command remote object implementation or servant 35
AIX 101 server code 36
Linux 129 differences between RMI (JRMP) and RMI-IIOP 39
ps-ef command, Linux 139 summary of differences in client development 39
summary of differences in server development 39
stub and ties generation 35
R RunDumpRoutine, JVMRI 290
runtime diagnostics, class loader 257
raising a problem report for submission 77
contents 83
escalating problem severity 84
problem severity ratings 83 S
RAS interface (JVMRI) 283 sample output, Javadump
RasInfo, JVMRI Linux 202
request types 293 z/OS 196
structure 292 sar, AIX 103
receive_exception (receiving reply) 52 sc, AIX 102
receive_other (receiving reply) 52 selecting ORB traces 316
receive_reply (receiving reply) 52 selectively disabling the JIT 237
receive_request (receiving request) 52 send_exception (sending reply) 52
receive_request_service_contexts (receiving request) 52 send_other (sending reply) 52
receiving OutOfMemory errors, AIX 116 send_poll (sending request) 52
reference objects (garbage collection) 14 send_reply (sending reply) 52
registered monitors send_request (sending request) 52
JVM system monitor dump 195 sending data to IBM, Windows 147
ReleaseRasInfo, JVMRI 290 sending files to IBM support
reliability, availability, and serviceability interface IBM internal only 85
(JVMRI) 283 outside IBM 86
Remote Method Invocation using your own ftp server 87
See RMI 71 sending information to Java Support, Linux 139
remote object servant, ORB client side 46
ORB client side server code, ORB 36
bootstrap process 45 server side interception points, ORB 52
getting hold of 44 receive_request (receiving request) 52
remote method invocation 46 receive_request_service_contexts (receiving request) 52
remote object implementation or servant, ORB 35 send_exception (sending reply) 52
reply body 319 send_other (sending reply) 52
reply header 318 send_reply (sending reply) 52
ReportEnv server side, ORB 47
AIX 95, 127 processing a request 48

358 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
server side, ORB (continued) system exceptions, ORB (continued)
servant binding 47 BAD_PARAM 168
servant implementation 47 COMM_FAILURE 168
tie generation 47 DATA_CONVERSION 168
server, ORB 173 MARSHAL 168
service contexts, ORB 173 NO_IMPLEMENT 168
service: collecting data, ORB 176 UNKNOWN 168
data to be collected 177 system logs 129
preliminary tests 176 system logs, using (Linux) 139
SetOutOfMemoryHook, JVMRI 290 system monitor dump, JVM 195
setting up and checking AIX environment 95 system performance, Linux 135
setting up and checking environment, Windows 143, 144 system properties
setting up for dump extraction command-line options 330
Windows 144 system properties, Javadump
settings, default (JVM) 339 z/OS 198
settings, JVM System.gc() 242, 249
environment 323
basic JIT options 325
diagnostics options 326
general options 324
T
tags, Javadump 193
Javadump and Heapdump options 325
tat, AIX 103
settings, JVM dump initiation 210
TDUMPs
severity ratings for problems 83
z/OS 155
escalating 84
technical support for AIX 125
shared library, AIX segment type 104
terminology and conventions in this book xiv
shmat/mmap, AIX segment type 104
test cases 80
signal information, Javadump
TGC tracing
z/OS 198
garbage collection 250
skeletons, ORB 35
thread pooling
st, AIX 102
RMI 72
stack trace, interpreting (AIX) 108
threading libraries 128
stack trace, Javadump
threads and stack trace, Javadump
z/OS 199
z/OS 199
stack trace, ORB 170
threads as processes, Linux 140
description string 170
tid, AIX 102
nested exceptions 171
time, AIX 102
static data 242
tool option for dumps 216
stime, AIX 102
tools
storage management, Javadump
cross-platform 187
z/OS 199
platform-specific 190
storage usage, private (z/OS) 151
tools, native
storage, z/OS 161
Windows 144
strace, Linux 140
tools, ReportEnv
string (description), ORB 170
AIX 95, 127
stub and ties generation, ORB 35
Windows 143
submitting a bug report, AIX 118
top command, Linux 139
submitting data
topas, AIX 105
javaserv (IBM internal only) 85
tprof, AIX 105
sending files to IBM support 86
trace
submitting data with a problem report 85
AIX 105
javaserv (IBM internal only) 85
application trace 277
sending files to IBM support 86
applications 259
using your own ftp server 87
controlling 261
summary of differences in client development 39
internal 260
summary of differences in server development 39
Java applications and the JVM 259
Sun properties, deprecated 42
methods 259
svmon, AIX 103
MiscellaneousTrace control options 264
sweep phase (garbage collection) 8
option summary 262
detailed description 12
options
parallel bitwise sweep 13
applids 263, 265
synchronization
buffers 264, 265
JNI 66
count 265
system dump
detailed descriptions 264
Windows 145
exception 265
system dumps 215
exception.output 263, 271
system exceptions, ORB 168
external 265
BAD_OPERATION 168
iprint 265

Index 359
trace (continued) truss, AIX 105
options (continued) tty, AIX 102
maximal 262, 265 Type, AIX 104
methods 263, 269 clnt 104
minimal 262, 265 Description parameter 104
output 263, 270 mmap 104
print 265 pers 104
properties 264 work 104
resume 263, 273
resumecount 263, 273
specifying 262
state.output 263, 271
U
uid, AIX 101
summary 262
uname -a command, Linux 139
suspend 263, 272
understanding memory usage, AIX 111
suspendcount 263, 273
understanding the class loader 25
trigger 263, 273
UNKNOWN 168
options that control tracepoint selection 262
user dumps
options that specify output files 263
generating in hang condition 144
placing data into a file 261
user exceptions, ORB 168
external tracing 261
user, AIX 102
trace combinations 261
using dump agents 213
tracing to stderr 261
using system logs, Linux 139
placing data into in-storage buffers 260
utilities
snapping buffers 260
NLS fonts 181
power management effect on timers 260
*.nix platforms 181
properties file 276
Windows systems 181
trace formatter 275
invoking 275
tracepoint ID 276
triggering and suspend or resume 263 V
trace data, JVMRI versions, ORB 166
intercepting 293 virtual storage, z/OS 161
trace formatter 275 vmstat command, Linux 139
invoking 275 processes section 130
trace listener, registering (JVMRI) 284 vmstat, AIX 106
trace options Vsid, AIX 104
options that indirectly affect tracepoint selection 263
trace options, changing (JVMRI) 285
trace, external (JVMRI) 294
TraceDeregister, JVMRI 290
W
WebSphere Application Server
tracepoint specification 267
environment, working in 93
TraceRegister, JVMRI 291
WebSphere Application Server and ORB
TraceResume, JVMRI 291
selecting ORB traces 316
TraceResumeThis, JVMRI 291
tracing 315
traces, ORB 171
changing on a running server 316
client or server 173
enabling at server startup 315
comm 172
when you will receive your fix, problem report 87
message 171
who should read this book xiii
service contexts 173
Windows
TraceSet, JVMRI 291
analyzing deadlocks 147
TraceSnap, JVMRI 292
classifying leaks 148
TraceSuspend, JVMRI 292
collecting data 150
TraceSuspendThis, JVMRI 292
collecting data from a fault condition 150
tracing
deadlocks 147
Linux 131
debugging hangs 147
ltrace tool 131
analyzing deadlocks 147
mtrace tool 131
debugging memory leaks 147
strace tool 131
classifying leaks 148
ORB and WebSphere Application Server 315
memory model 148
changing on a running server 316
tracing leaks 148
enabling at server startup 315
debugging performance problems 149
selecting ORB traces 316
data for bug report 149
tracing leaks, Windows 148
frequently reported problems 150
transaction dumps
debugging techniques 145
z/OS 155
diagnosing crashes 146
triggered generation of a Heapdump 206
sending data to IBM 147
triggering dumps 214
Dump Extractor 145

360 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
Windows (continued)
frequently reported problems 150
generating a user dump file in a hang condition 144
getting a dump from a hung JVM 147
hangs
getting a dump 147
Heapdumps 145
Javadump sample output 202
Javadumps 145
JVM dump initiation 212
memory model 148
native tools 144
problem determination 143
sending data to IBM 147
setting up and checking environment 143, 144
setting up for data collection 144
dump extraction 144
native Windows tools 144
setting up for dump extraction 144
system dump 145
tracing leaks 148
Windows systems
font utilities 181
work, AIX segment type 104

X
XHPI section, Javadump
z/OS 198

Z
z/OS
collecting data 164
crashes 153
documents to gather 153
failing function 154
environment variables 151, 326
environment, checking 151
error message IDs 153
general debugging techniques 152
IPCS commands 152
hangs 160
bad performance 161
deadlocked 160
looping 160
Javadump sample output 196
JVM dump initiation 211
LE settings 151
maintenance 151
memory leaks 161
LE HEAP 161
OutOfMemoryErrors 162
virtual storage 161
performance problems 163
private storage usage 151
problem determination
environment, checking 151
setting up dumps 152
TDUMPs 155

Index 361
362 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms
򔻐򗗠򙳰