JDB - Debugging Concurrency issues in Java

Typically debugging concurrency issues using debugger is very difficult as it manifests only when multiple requests are processed concurrently. Also in a debugger tracking multiple threads is further difficult.
JDB is a free command line tool for Java. Please refer to my previous post for details about JDB ( http://rejeev.blogspot.com/2011/04/jdb-command-line-debugger-for-java.html ). We can debug concurrency issues much easily using JDB.
Typically concurrency issues are due to updation (or updaton and read) of a shared variable from two threads. Typically the challenge will be to identify which thread update the variable at which code segment. We can use JDB to identify that.

• run the application with debug enabled


• attach JDB to the application.


• set 'watch' on the variable at the suspect variable.


• set a 'monitor where all' to print the stack trace whenever the variable is accessed.


• 'run' the application.


• 'run' each time the application pauses at modification of the watch point.

Now whenever the application is modifying the variable, thread dump shall be printed. From the thread dump we will be able to identify which thread modifies the variable and at code segment.

Let us see an example now

Given below is a very simple example where a variable is shared between two threads and by the time, the thread 'main' prints out the value of the shared variable, the value has been modified by the other thread.
As this program is only few lines of code it is very trivial to go through the code and understand that the second thread is updating the variable at code Main2 (line:11). In a real application the shared variable may be updated from too many places with too many code paths to that. Eventhough you know that some thread updates the shared variable, it is very difficult to identify which thread updates from which code. JDB will help us to identify that.

Let us see how do we identify that

public class Main2 extends Thread {
 public static int count;
 public static void main(String[] args) throws Exception {
  new Main2().start();
  count = 5;
  Thread.sleep(10);
  System.out.println("value: " + count);
 }
 public void run(){
  try{ Thread.sleep(5); }catch(Exception e){}
  count = 10;
 }
}

Now compile the code

javac -g Main2.java

Now run the program with debug enabled

java -cp . -Xdebug -Xrunjdwp:transport=dt_socket,server=y,address=8000,suspend=y Main2
Listening for transport dt_socket at address: 8000

Now attach the debugger

jdb -sourcepath . -attach 8000
Set uncaught java.lang.Throwable
Set deferred uncaught java.lang.Throwable
Initializing jdb ...

VM Started: No frames on the current call stack
main[1]

Now set 'watch' on the shared variable count. This means whenever the variable 'count' is modified, program will stop execution (as if there is a break point).

watch Main2.count

Now set 'monitor where all'. This means that whenever the program stops (due to a breakpoint or watch) 'where all' command (thread dump) shall be executed.

monitor where all

Now let us start execution by giving the command 'run'. We will get the first thread dump when the varible count is modified first time.

run
> Set deferred watch modification of Main2.count

Field (Main2.count) is 0, will be 5: "thread=main", Main2.main(), line=5 bci=11

Signal Dispatcher:
Finalizer:
  [1] java.lang.Object.wait (native method)
  [2] java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:118)
  [3] java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:134)
  [4] java.lang.ref.Finalizer$FinalizerThread.run (Finalizer.java:159)
Reference Handler:
  [1] java.lang.Object.wait (native method)
  [2] java.lang.Object.wait (Object.java:485)
  [3] java.lang.ref.Reference$ReferenceHandler.run (Reference.java:116)
main:
  [1] Main2.main (Main2.java:5)
Thread-2:
  [1] Main2.run (Main2.java:10)
main[1]

'main[1]' as the command prompt indicates that current thread is 'main[1]'. First line of the thread 'main[1]' is Main2.java:5. That means the current line of code is at Main2.java:5.
run 'list' to see the current see the code at line number Main2.java:5

list
1    public class Main2 extends Thread {
2     public static int count;
3     public static void main(String[] args) throws Exception {
4      new Main2().start();
5 =>   count = 5;
6      Thread.sleep(10);
7      System.out.println("value: " + count);
8     }
9     public void run(){
10      try{ Thread.sleep(5); }catch(Exception e){}

Now resume execution again from there using 'run'. Now we get the thread dump again when the variable 'count' is modified.

run
Field (Main2.count) is 5, will be 10: "thread=Thread-2", Main2.run(), line=11 bci=12
11      count = 10;

Signal Dispatcher:
Finalizer:
  [1] java.lang.Object.wait (native method)
  [2] java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:118)
  [3] java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:134)
  [4] java.lang.ref.Finalizer$FinalizerThread.run (Finalizer.java:159)
Reference Handler:
  [1] java.lang.Object.wait (native method)
  [2] java.lang.Object.wait (Object.java:485)
  [3] java.lang.ref.Reference$ReferenceHandler.run (Reference.java:116)
main:
  [1] java.lang.Thread.sleep (native method)
  [2] Main2.main (Main2.java:6)
Thread-2:
  [1] Main2.run (Main2.java:11)
Thread-2[1]

Now current thread of execution is Thread-2 and current line of code is Main2.java:11.
To see the line of code at Main2.java:11

 list
7      System.out.println("value: " + count);
8     }
9     public void run(){
10      try{ Thread.sleep(5); }catch(Exception e){}
11 =>   count = 10;
12     }
13    }

This time 'count' is modified by the second thread. From this it can identified that 'Thread 2' is modifying the variable 'count' at Main2(line:11). Also the code path to this operation also can be infered from the thread dump.
Hence the culprit here is 'Thread-2' and code segment 'Main2.java:11' and the execution path as explained in the stack of Thread-2 in the thread dump.

JDB - Command line debugger for Java

Introduction

Debugger is an unavoidable tool in any Java developer’s toolkit. A Debugger helps when the application/module/function behaviors different than that of developers expectation. Typically developer will first try to understand difference in behavior by glancing through the code quickly. If he still not able to understand the reason for the unexpected behavior; next step would be step through the program execution using a debugger. I assume readers are familiar with the normal debugging procedures. If not please go through the appendix.
Most of the current debuggers are GUI based (either integrated with IDEs (IntelliJ, Eclipse) or stand alone debuggers (Yourkit).
However GUI based debuggers are not very useful when the application is running on a remote machine, especially when the connectivity between the machine where appication is running and the machine where debugger is running is poor.
In such scenarios a command line debugger which is also running on the same box as the application is running would be useful.

JDB – a free command line debugger

JDB is a free command line debugger. JDB is part of JDK and is supported by Sun JDK on various Operating systems like Windows, Linux, Solaris and Mac OS x.
How to use

1. We need to run the application with ‘debug enabled’. You need to provide the following JVM arguments ‘-Xdebug –Xrunjdwp:transport=st_socket,server=y,address=8000,suspend=y’ to start the application in debug mode. Please refer to http://download.oracle.com/javase/1.5.0/docs/guide/jpda/conninv.html#Invocation for further details.
2. Application should be compiled with –g option.
3. Attach JDB to the application ‘jdb –attach 8000’
4. You can see the list commands by typing ‘help’
5. Set breakpoint at some significant point using ‘stop at :’
6. Navigate through the code step by step using ‘step’ and ‘next’ and other commands.

Given below are a list of important commands

1. stop at : - to set breakpoint
2. run – to start execution or resume from a breakpoint
3. locals – list values of local variables
4. dump this – list variables of the instance ‘this’
5. where all – to see the thread dump
6. list – to see current code
7. step – execute one line (if it is function step into the function)
8. next – execute one line (if it is a function step over the function)
9. help – print list of available commands with description

To see the complete list of commands type ‘help’ command.
Let us see an example to illustrate this further.

Example – Debugging Jackson (JSON processing tool)

Jackson is tool for parsing JSON (from JSON to object and Object to JSON parsing).
You can download Jackson source from http://jackson.codehaus.org/1.7.6/jackson-src-1.7.6.zip
Unzip the contents to any convenient location (. Build the source using ‘ant’.
ant jars
All packages shall be generated as /build. We need jackson-mapper-asl-1.7.6.jar and jackson-core-asl-1.7.6.jar
Given below is a small code which converts a Java object to JSON string.
We will navigate through the Jackson processing using debugger.

import org.codehaus.jackson.map.ObjectMapper;
public class Main {
  public static void main(String[] args) throws Exception {
      System.out.println("entering the program....");
      MyEntity obj = new MyEntity();
      obj.setId(1);
      obj.setName("rejeev");
      obj.setAddress("c-48, sree nagar");
      ObjectMapper mapper = new ObjectMapper();
      String json = mapper.writeValueAsString(obj);
      System.out.println("json: " + json);
  }
}

public class MyEntity {
  private int id;
  private String name;
  private int age;
  private String address;
  public int getId() {
      return id;
  }
  public void setId(int id) {
      this.id = id;
  }
  public String getName() {
      return name;
  }
  public void setName(String name) {
      this.name = name;
  }
  public int getAge() {
      return age;
  }
  public void setAge(int age) {
      this.age = age;
  }
  public String getAddress() {
      return address;
  }
  public void setAddress(String address) {
      this.address = address;
  }
}

Save this code at
Compile the classes:

javac –g –cp ./build/jackson-mapper-asl-1.7.6.jar:./build/jackson-core-asl-1.7.6.jar:. –d . MyEntity.java Main.java

Run the application in debug mode.

java –Xbebug –Xrunjdwp:transport =st_socket,server=y,address=8000,suspend=y –cp ./build/Jackson-mapper-asl-1.7.6.jar:./build/Jackson-core-asl-1.7.6.jar:. Main

You will get the following output:

Listening for transport dt_socket at address: 8000

As the option ‘suspend=y’ is specified, the JVM is just started. Method exectution is not yet started. When we trigger ‘run’ command from debugger exectution will start.

Attach debugger to the application

jdb –sourcepath :/src/mapper/java –attach 8000

You will get the following output:

Set uncaught java.lang.Throwable
Set deferred uncaught java.lang.Throwable
Initializing jdb ...
>
VM Started: No frames on the current call stack
main[1]

To start the execution run the command ‘step’

main[1] step
>
Step completed: "thread=main", Main.main(), line=4 bci=0
4 System.out.println("entering the program....");

Now program is at the first line of the function ‘main’
Let us see the code around the current line of execution.

main[1] list
1    import org.codehaus.jackson.map.ObjectMapper;
2    public class Main {
3        public static void main(String[] args) throws Exception {
4 =>         System.out.println("entering the program....");
5            MyEntity obj = new MyEntity();
6            obj.setId(1);
7            obj.setName("rejeev");
8            obj.setAddress("c-48, sree nagar");
9            ObjectMapper mapper = new ObjectMapper();
10            String json = mapper.writeValueAsString(obj);

The arrow indicates the current line of execution.
Open the java source files (Main, MyEntity and Jackson source files) in a TextEditor or IDE.
Let us start from Main. Main:10 is the line which parse object to JSON string.
Let us set breakpoint at Main:10

main[1] stop at Main:10
Set breakpoint Main:10

To move the execution from the current line to next breakpoint (line 10)

main[1]
main[1] run
Breakpoint hit: "thread=main", Main.main(), line=10 bci=41
10 String json = mapper.writeValueAsString(obj);

To see the code around current line

main[1] list
6            obj.setId(1);
7            obj.setName("rejeev");
8            obj.setAddress("c-48, sree nagar");
9            ObjectMapper mapper = new ObjectMapper();
10 =>         String json = mapper.writeValueAsString(obj);
11            System.out.println("json: " + json);
12        }
13    }

We can see all local variables using ‘locals’

main[1] locals
Method arguments:
args = instance of java.lang.String[0] (id=867)
Local variables:
obj = instance of MyEntity(id=868)
mapper = instance of org.codehaus.jackson.map.ObjectMapper(id=869)

Now we can inspect the content of any object. Let us inspect ‘obj’.

main[1] dump obj
obj = {
  id: 1
  name: "rejeev"
  age: 0
  address: "c-48, sree nagar"
}

Now let us step into ‘writeValueAsString’ using the command ‘step’

main[1] step
>
Step completed: "thread=main", org.codehaus.jackson.map.ObjectMapper.writeValueAsString(), line=1,595 bci=0

Now see the current code again using the command ‘list’

main[1] list
1,591        public String writeValueAsString(Object value)
1,592            throws IOException, JsonGenerationException, JsonMappingException
1,593        {      
1,594            // alas, we have to pull the recycler directly here...
1,595 =>         SegmentedStringWriter sw = new SegmentedStringWriter(_jsonFactory._getBufferRecycler());
1,596            _configAndWriteValue(_jsonFactory.createJsonGenerator(sw), value);
1,597            return sw.getAndClear();
1,598        }

Let us go to the next line using the command ‘next’ and from line 1596, let us step into the function ‘_configAndWriteValue’

main[1] step
>
Step completed: "thread=main", org.codehaus.jackson.JsonFactory.createJsonGenerator(), line=489 bci=0
489            IOContext ctxt = _createContext(out, false);

Instead of stepping into _configAndWriteValue, we are now in JsonFactory.createGenerator. This is because first method parameters are calculated. To get into _configAndWriteValue, let us get out of createJsonGenerator using step up.

main[1] step up
>
Step completed: "thread=main", org.codehaus.jackson.map.ObjectMapper.writeValueAsString(), line=1,596 bci=24
1,596            _configAndWriteValue(_jsonFactory.createJsonGenerator(sw), value);

Now we are back at line no: 1596. Let us invoke step again.

main[1] step
>
Step completed: "thread=main", org.codehaus.jackson.map.ObjectMapper._configAndWriteValue(), line=1,982 bci=0
1,982            SerializationConfig cfg = copySerializationConfig();

Let us see the code around this

main[1] list
1,978         */
1,979        protected final void _configAndWriteValue(JsonGenerator jgen, Object value)
1,980            throws IOException, JsonGenerationException, JsonMappingException
1,981        {
1,982 =>         SerializationConfig cfg = copySerializationConfig();
1,983            // [JACKSON-96]: allow enabling pretty printing for ObjectMapper directly
1,984            if (cfg.isEnabled(SerializationConfig.Feature.INDENT_OUTPUT)) {
1,985                jgen.useDefaultPrettyPrinter();
1,986            }

Using this approach we can navigate around the code.
At any point we can see the thread dump using ‘where all’

main[1] where all
Signal Dispatcher:
Finalizer:
[1] java.lang.Object.wait (native method)
[2] java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:118)
[3] java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:134)
[4] java.lang.ref.Finalizer$FinalizerThread.run (Finalizer.java:159)
Reference Handler:
[1] java.lang.Object.wait (native method)
[2] java.lang.Object.wait (Object.java:485)
[3] java.lang.ref.Reference$ReferenceHandler.run (Reference.java:116)
main:
[1] org.codehaus.jackson.map.ObjectMapper._configAndWriteValue (ObjectMapper.java:1,982)
[2] org.codehaus.jackson.map.ObjectMapper.writeValueAsString (ObjectMapper.java:1,596)
[3] Main.main (Main.java:10)

Appendices:

Basic concept of debugging

A Debugger helps when the application/module/function behaviors different than that of developers expectation. Typically developer will first try to understand difference in behavior by glancing through the code quickly. If he still not able to understand the reason for the unexpected behavior; next step would be step through the program execution using a debugger.
Using a debugger one can put a mark (called breakpoint) on any line in the source code and the program will stop execution when the execution reaches that mark. Then we can do the following:
1. Inspect the state of the program at that point. Typically following states are valuable – (a) Local variables, (b) Instance variables of the current object (this), (b) Execution stack of the request
2. Execute the program from that point (breakpoint), step by step, one line at a time. The current line could be an invocation of a function. In that case we have two options – (a) execute the function as single step and process to next line in the current code segment (step over), (b) step into the function and go to the next line in the function (step in).
3. Continue the execution of program (resume) from the breakpoint (typically to next breakpoint).
Typically we put a breakpoint at a line where we think the beginning of the significant code. Then navigate through the code step by step using either step-over and step-in, also inspecting local variables and instance variables at relevant steps. Once we complete the analysis of one significant code segment, we will move to another significant code segment by putting a breakpoint at the beginning of that segment and using ‘rusume’ to move from old code segment to new code segment.
By this method most of the issues can be explained. However analyzing concurrency issues requires different approach.
Consider a scenario where your program execution is affected by the setting of a Boolean variable which is reset by some other thread which you don’t know.
This can be analyzed by putting a ‘watch’ on that variable. Whenever that variable is modified, execution is stopped at that point, by inspecting the stacktrace, we will be able understand which thread at which code modifies the variable.

IntelliJ IDEA

I have been using IntelliJ IDEA community edition ( http://www.jetbrains.com/idea/ ) lately and found that it is much better than Eclipse and NetBeans.
Addicted to IDEA!
Addicted toJava IDE with unparalleled java code
analyzer (600+ built-in inspections)

Glassbox

Glassbox is a good monitoring tool for Java web applications. It provides the response time statistics for key methods like servlets, database calls etc. You can extend methods you want to monitor by specifying the aspects (It uses Aspectj aop.xml for this). This is tool built using Aspectj AOP. This tool is particularly good monitoring performance of web applications in production.

String literals and Garbage Collection in Java

String literals are loaded during class loading. String literal objects are created in heap and a reference to each String literal object is stored in String literal pool. String literals are never garbage collected.
Can this cause OutOfMemory? Not really! The amount of memory required to load String laterals is not more than the size of the class it loaded from. You cannot dynamically create String literals. Unlike other memory leak issues where objects can be created dynamically.

Here is a interesting article on String literals http://www.javaranch.com/journal/200409/ScjpTipLine-StringsLiterally.html

Leaky Abstraction - Double Checked Locking

Some times our inferences based on the abstracted view fails due to the implementation dependencies. This is called leaky abstraction ( http://www.joelonsoftware.com/articles/LeakyAbstractions.html ).
Double Checked Locking idiom was an widely sited design pattern where instead of synchronizing the whole method, synchronize only the initialization part. See the following code snippents

private Helper helper = null;
public synchronized Helper getHelper() {
if (helper == null)
helper = new Helper();
return helper;
}
// other functions and members...
}

Using double checked idiom same code would be implemented as:

private Helper helper = null;
public Helper getHelper() {
if (helper == null)
synchronized(this) {
if (helper == null)
helper = new Helper();
}
return helper;
}
// other functions and members...
}

However it doesn't work. See the following link http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html
This is against our (or my) inuition based on the modern language threading model (that execution of method happens in the order of statements written in the code. However that is not correct. Language specifications provide lot of leniency on this part for sake of performance. Compiler optizers typically rewrite the code and final execution not necessarily strict oder as it written in the source code)

Javadoc tool with @exclude tag

The standard javadoc utility shipped with JDK doesn't provide an option to exclude some public methods from public API documentation. Currently we can filter only at file level (either all methods from all classes in the file or none).
Javadoc specification proposes a new tag called '@exclude' for this purpose - any method which @exclude tag shall be excluded from javadoc documentation. However this has not been implemented yet.
Here is a modified javadoc utility which can implements @exclude tag - any method with @exclude tag will be excluded from javadoc.
Usage:
1) download the javadoc.jar
2) create a file with list of source files which contain the methods to be documented
eg: inputs.txt
---------------------------------------------
c:\src\com\abc\spi\Abc.java
c:\src\com\abc\util\Xyz.java
----------------------------------------------
3) invoke javadoc as follows
javadoc -docletpath -d -doclet com.sun.tools.doclets.standard.Standard1 @
Please note that name of the file has been renamed to Standard1.

Method Tracing

Assume you have a complex application probably developed by somebody else. Sometime you have an application which runs on top of another platform like application server or portal. Sometime your application may be using some other third party component.
You may want to know trace the execution of specific requests or transactions in the system probably for debugging. This is also helpful to understand the architecture/design of the system.
You can do that using this tool. This tool traces the progress (execution) of each thread.

Steps:
0) Download and copy the following 3 jars to a convenient location. (all jars has to be in the same directory). (Click on the link to download files).
MethodTracer.jar
asm-3.1.jar
asm-commons-3.1.jar
1) Add the following JVM options during startup.
-javaagent:[path_to_MethodTracer.jar]
If you are using some startup script you may need to add it there. If you are using some exe to invoke java then there may be some file where you will be able to add this.

2) Extract trace.properties from the jar and edit it to change the configurations. Put it back. You can use the following commands for extracting the property file and put it back into jar.
jar -xvf MethodTracer.jar tracer.properties
jar -uvf MethodTracer.jar tracer.properties
Following are the options available.
ExcludedPackages – classes from this coma separated list of packages shall be excluded from instrumentation and hence execution of such methods shall not be traced.
IncludedPackages – If specified only classes from this packages shall be traced. You can use includePackages and excludePackages together to get fine control
allThreadsLogged – if true, all requests by threads shall be traced. You can trace the requests by specific threads by turn this flag false and use threadList
threadList – if allthreadsLogged is false, then only the threads specified in this list shall be traced.
This is useful in the case of systems like application server where there shall be lot of threads running always. Different kinds of requests are typically processed by separate threads.
loggingOnAtStartup – if true, tracing shall start from application startup. It may produce lot of verbose output. If you want to trace only certain specific transactions . You can turn it false and start/stop tracing from jconsole.
File/console – turn true/false appropriately to send the trace output to file (method_trace.log in the current directory of the application) or console window.

3) start the application. Name of all instrumented classes are printed out to the console. If classes from any of the packages of not interest are instrumented, you can modify the tracer.properties file (includePackages/excludePackages). (requires re-start of the application).

4) start jconsole and connect to this application. Jconsole is tool comes along with JDK. You can start jconsole by executing the jconsole.exe in the JDK/bin directory. While start up it will display the list of all available JVMs in the machine and ask to choose the one to connect. Choose the JVM of your application. Goto the ‘MBeans’ tab. Expand tracer >> custom >> configuration >> operations. Use enableLogging method to start/stop logging (as Boolean as parameter). Use addThread and removeThread to trace one (or list of) thread (name of the thread as parameter. You can see the list of all threads and it’s name from the ‘Threads’ tab. Copy the name from that tab and paste it here.



5) go to the jconsole > MBean tab, go to tracer >> custom >> configuration >> operations
a) select appropriate threads for tracing.
b) start logging (enableLogging method)

6) invoke the operation/transaction of your application, to be traced.

7) stop the logging from jconsole (enableLogging method in jconsole)

8) Open the method_trace.log file in the current directory.

How Method Tracing works?
It uses dynamic byte code instrumentation. It uses java.lang.instrument.* classes for bytecode instrumentation. It uses ASM Framework for modifying byte code. It modifies the byte code while loading the classes into JVM by Class Loader. The class files in the jar is not being modified.
It inserts tracing code at the beginning and end of each method.

Is it modifies your application binary?
No. It modifies the byte code while loading the classes into JVM by Class Loader.

Is it recommended to use in production?
No. This is only beta quality and there shall be considerable overhead because of method tracing.

Analyzing Memory Leak in Java Applications using VisualVM

Introduction
Memory Leak is a decease and OutOfMemoryError (OOM) is the symptom for that. However all OOM doesn’t necessarily implies Memory Leak. OOM can happen due to the generation of large number of local variables – particularly with large number of concurrent requests (in the case of server applications). On the other hand all memory leaks not necessarily manifest into OOM – especially in the case of desktop applications or client applications (those are not run for long without restart).
Memory leaks also can be in heap, perm space, or in native memory.
Following section talks only about memory leak in heap. How to verify that memory leak is the reason for OOM in your case is also out of scope (probably will discuss some other day). How to attack memory leaks in perm space is also out of scope (again another day’s work). I have absolutely no clue about debugging memory leak in native memory (Google search with error messages).

Memory leak in heap

1) Run the application with latest available JDK (JDK 1.6 as of now). New versions of java contain lot of improvements in debugging tools. Run the application with different operations repeatedly to identify the operation that causes memory leak. We don’t need to wait until OOM to identify memory leak. Monitoring the heap using any tool can identify memory leak. Run the application until it reaches a steady state. Monitor the heap size over a period of time. If the ‘heap size after full GC’ (full GC remember) increases each time, that implies a memory leak. We can use visualVM also for this purpose.


Now we have identified the operation that causes memory leak – means if we invoke this operation repeatedly we will get OOM. (What if we could not identify the operation that causes memory leak?). Now onwards we assume that this operation runs continuously.

2) Install VisualVM (https://visualvm.dev.java.net/download.html). You need to install VisualVM separately – unlike many other java debugging tools this one doesn’t come along with JDK [ Thanks to Jiri Sedlacek's comment! VisualVM is part of JDK from jdk1.6 update 7 onwards ]. Just unzip the file to any convenient location.

3) Run visualVM - you can run visualvm by executing ‘visualvm.exe in the bin directory. (You should have jdk 1.6 in path?)

4) Connect to the application – all the java processes running in the machine are listed in the left side pane (name of the main class shall be listed). You can double click the one you are interested. Right side there are four tabs – monitor and profiler are the ones we are interested in.


5) Go to the profiler tab. Check the settings checkbox it will display different options. Go to the memory settings tab. Check the ‘record allocations stack traces’. Uncheck the ‘settings’ checkbox; that will close the settings options display (all changes are automatically saved).

6) Now click on the ‘memory’ button – this will start memory profiling. Wait for few seconds – it will take a while. A tabular display of different classes, its instance count, total byte size etc are displayed.
7) Wait for a while to allow the application to be in stead state.
8) Take a snapshot of the objects by clicking the button on top of the tabular view (see below screen shot). A snapshot is generated and labeled in the left side pane as well as opened in the right side pane.

9) Wait for some more time to allow some memory leak. Take one more snapshot as explained in the previous step. Now we have two snapshots displayed at the left side pane (also opened in the right side pane).
10) Select both of these snapshots (in the left side pane, by using ‘ctrl’ button we can select multiple items), right click and select ‘compare’. A comparison tab shall be opened in the right side pane. That tab will display items that have been increased during the interval of first and second snapshot. The top item is the suspect of memory leak.

11) Go to the profiler tab again. Select the identified item in the previous step. Right click and select ‘Take snapshot and show allocation stack traces’. One more snapshot is generated. This time in the right side pane, an allocation stack traces sub tab also visible. It lists different places where this particular item is instantiated and it’s percentage of total count.

12) Wait for some more time to have more memory leak. Take one more snapshot with allocation stack traces.
13) Take a heap dump of the application. We can take heap dump by right clicking the application in the left side pane and select ‘heap dump’. Heap dump is also displayed in the left side pane as well as opened in the right side pane.
13) Now come back to the tow snapshots with stack traces. Compare these two snapshots with allocation stack traces. Identify the methods where there is large difference in contribution to total count. These are the places where leaked objects are instantiated.
14) Go to the heap dump tab in the right side pane. Go to the Classes view. Double click on the item identified in step 10. Instances of that class are displayed. Choose one instance which you feel could one of the leaked objects (just intuition). Look at the right bottom pane. Expand the view. You will find the objects that is holding referenced to the leaked objects. This is the reason for memory leak.

15) From the items identified in step 10, and clue we got from step 13 and 14 we could resolve the memory leak issues. We get the object which leaked, the place of instantiation, and the objects that hold reference to it.

Thread wise CPU monitoring for Java process

We come across many occations where java application is hung or slow with high CPU usage. We want to analyze which thread is causing this high CPU usage.
Java 1.5 provides a API "ThreadMXBean.getThreadCPUTime()"
We can use the following code in anywhere in the JVM to get the thread wise CPU usage of that JVM.
ThreadMXBean threads = ManagementFactory.getThreadMXBean();
ThreadInfo[] threadInfos = threads.getThreadInfo( threads.getAllThreadIds() );
for( int i=0; i < threadInfos.length; i++ )
{
long cpuTimeCumulative = threads.getThreadCpuTime( threadInfos[ i ].getThreadId() ); // in nano seconds
}
There is a utility called JTop available with JDK 1.6 (JDK_insall_dir/demo/management/JTop). This can be invoked as a JConsole plugin as well.
You can invoke the JConsole with plugin as follows:
/bin/jConsole.exe -pluginpath
eg: c:/jdk160/bin/jConsole.exe -pluginpath c:/jdk160/demo/management/JTop/JTop.jar
The default JTop shows only the cumulative CPU time (not the current CPU usage as if in Task manager).
I have modified the JTop little bit to include the current CPU usage as well.
You can download the modified JTop.jar from the following path.
http://rejeev.googlepages.com/JTop.jar
This will help us to identify the threads causing high CPU. Typically this will help us to resolve infinite loop issues.

MemoryLeak Analysis using Java SE 6

If you are having a memory leak you will get OutOfMemoryError (OOM). You may get OOM even if you are not having memory leak if the following cases
a) Huge amount of local variables (short living objects)
b) Large number of concurrent threads running
c) Complex application and insufficient amount of heap size (memory required for long living objects)
You can conclude if there is a memory leak or not by analyzing the jstat report (jstat -gc 30s) or by analyzing the jConsole graph.
If the bottom of the zig zag graph keep increasing there is a memory leak.
If the bottom of the zig zag graph is greater than the available heap then it is condition (c)
If the peak of zig zag graph is greater than the heap then it is condition (a)
OOM could in heap area, permanent generation or in native memory area
If OOM in heap then the message could be "OutOfMemoryError: Java heap space"
If OOM in permanent generation then "OutOfMemoryError: PermGen space"
If OOM in native area then "java.lang.OutOfMemoryError: request bytes for . Out of swap space?"
If you have memory leak in perm gen then verbose:class may help. You need to find out the class which is getting repeatedly loaded without getting unloaded.
For native area, no specific tools available.
If you have OOM in heap then you can use Jmap and jhat to analyze further.
Use the application for a while. Repeat the operations which you suspect causing the memory leak.
Take a heap dump of the JVM by jmap.exe -dump:live,format=b,file=heap1.dmp
Repeat the suspecting operations further
Take the heap dump again. Name the dump file heap2.dmp
Run Jhat over both dumps jhat -J-Xmx512m -baseline heap1.dmp heap2.dmp
In a browser type http://localhost:7000/histo/
From the histogram, find out the classes which is having highest total size. These are suspected classes for memory leak.
Click one class, another page opens where you can see all instances of that class. New Instances are marked new.
Click one instance which is new. Another page opens where you see reference chain from root (use exclude weak ref)
Click that link, which displays the reference from root to that instance, Study the path carefully. You may need do this analysis for couple of classes in the histogram, you will find some references which is otherwise not necessary.
Study the source code remove that references.
References
http://java.sun.com/developer/technicalArticles/J2SE/monitoring/
http://weblogs.java.net/blog/jfarcand/archive/2006/02/using_mustangs.html

Thread dump for javaw and java as windows service

I have been looking for a way to take thread dump for java started using javaw.exe and java installed as windows service. I have come across a wonderful tool called AdaptJ (www.adaptj.com). It has a wonderful GUI.
It lists all the running JVMs in the machine (including the ones inside browser, inbedded in other executables etc) with meaningful descriptions to identify the process
You can select any JVM and take thread dump
It also shows the system properties, and memory parameters
There is beanshell and a number of commands are available to debug the running JVM.
You can change the log level (java.util.logging, and log4j)
You can trace AWT events
You can monitor file sytem, network access by the JVM
And much more!!!
A cool tool

Java Obfuscator

We had a requirement to Obfuscate our client (We have Swing based think client) binaries. Most of the resources on Internet recommended that Java obfuscation is waste of time. I disagree with that. I would say that Obfuscation useful as security mechanism as well as it helps in protecting your Intellectual Property (IP). A determined skilled attacker may be able to overcome this hurdle, however lot of naive onlookers will be stopped at this blockade.
We were also looking for a good Java Obfuscating tool and found that not many available in the market. Finally we have decided for Zelix KlassMaster ( http://www.zelix.com/klassmaster ). I found it very good and priced reasonably.

ClassLoaders and class visibility

Assume you have one interface and one implemenation class

/* package */ interface com.abc.Xyz{

...

}

public class com.abc.XyzImpl implements Xyz {

...

}

Now if you keep Xyz in one classpath and XYZImpl in another classpath which is a child of former one you will get the following exception.

"java.lang.IllegalAccessError: class com.abc.XyzImpl cannot access its superinterface com.abc.Xyz" This is because eventhough both are in package, they are loaded by different classloaders and hence in different name space. If you define the package also public no issue else both class have to be loaded by the same classloader.

javax.mail.MessagingException and compatibility issues?

javax.mail.MessagingException has a constructor with parameters String and Exception. However if you instantiate do
MessagingException me = new MessgingException("abc", e);
and invoke me.getCause() and if you expect to get exception e which you imbedded in constructor you are wrong.
Because MessagingException constructor use the embedded exception to set setNextException(). It does not call super("abc", e);
Hence if you want to get root cause, use MessagingException.getNextException() instead of getCause()
Why this wiered api design? Probably MessagingException API might have been designed before JDK 1.4 where we didn't have feature for embedding one exception into another.
Now we are not in a position to change the API to ensure backward compatibility! Otherwise anybody who is using getNextException will fail.
Probably MessagingException.getCause() could override to return nextException!

Windows /3GB switch and Sun JDK

In windows application process's memory space is limited to 2GB. That means no application can use more memory than 2GB.
Windows has boot.ini parameter called /3GB switch using which we can increase the memory space for application processes to 3GB.
However even after using /3GB switch maximum heap for Sun JDK cannot be more than 2GB (to be precise not more than 1.6GB). This is because Sun JDK requires continuous block of memory for the entire heap. Whereas in Windows with /3GB it is not possible.
I have observed that JRockit JDK doesn't have this limitation.

LinkageError and some Class Loading issues

Consider the code:
setAbc(Abc obj){
this.abc = obj; //Abc abc;
}
Do we need the Class Abc in classpath while invoking this method? Not really! However when you invoke this method second time with an Object with a different ClassLoader (than the ClassLoader of the Object in the first invocation) you will get the following error:
LinkageError: loader constraints violated when linking Abc class
See the attached code for demonstration.
http://rejeev.googlepages.com/source_code_LinkageError.zip

ClassCastException

When do you get ClassCastExceptionn? When you try to cast an object to Class other than its class or its super class.
However you may get ClassCastException when you try to cast an Object whose class's ClassLoader and the ClassLoader of the Class to whom you are trying to cast are neither same nor later ClassLoader an ancestor (parent or grant parent etc) of former ClassLoader.
See the attached code demonstrating the same.
http://rejeev.googlepages.com/source_code_ClassCastExceptionExampl.zip

Serialization and ClassLoading

Do we need the Class in the classpath to Serialize an instance of the class? Not really, even if the Class is not accessible, we can get the Class by Object.getClass() method and get required information.
However if the Class referring (using) other classes which are not in classpath. then it will break.
It break when it try to get the SerialVersionUID which internally trying to getDeclaredFields
It would have better if we could have avoid this dependency.
You can download a sample program demonstrating the same here
http://rejeev.googlepages.com/source_code_Serialization.zip

NPE from JVM

NullPointerException from JVM. Good link
http://www.javaspecialists.eu/archive/Issue062.html