TDA - Thread Dump Analyzer

TDA - Thread Dump Analyzer

Published September, 2008 Copyright 2006-2008 Ingo Rockel <[email protected]>

Table of Contents
1. General ................................................................................................................................................ 1 1.1. Request Thread Dumps ................................................................................................................ 2 1.2. Thread Dump Parsing .................................................................................................................. 2 1.3. Regular Expression For Time Stamps .............................................................................................. 6 1.4. Using Filters ............................................................................................................................... 7 1.5. Using Categories ......................................................................................................................... 8 1.6. Deadlocks .................................................................................................................................. 8 1.7. Sessions .................................................................................................................................... 9 1.8. Long Running Threads ................................................................................................................. 9 1.9. Heavy Load Analysis ................................................................................................................. 10 1.10. JConsole Plugin ...................................................................................................................... 10 1.11. VisualVM Plugin ..................................................................................................................... 11 2. Analyzing production environments ........................................................................................................ 13 2.1. Dumping Information about running threads ................................................................................... 13 2.2. Long running thread detection on Solaris ....................................................................................... 15 2.3. Analyzing the heap profile .......................................................................................................... 15 3. Application Analysis using JConsole ....................................................................................................... 16 3.1. Remote application access ........................................................................................................... 16 3.2. Requesting Dumps .................................................................................................................... 16 3.3. Plugins For JConsole ................................................................................................................. 16 A. References ......................................................................................................................................... 17

iii

List of Figures
1.1. Welcome Screen ................................................................................................................................. 1 1.2. Information about the selected dump ....................................................................................................... 3 1.3. Thread Dump View ............................................................................................................................. 4 1.4. Monitors used by threads ...................................................................................................................... 5 1.5. Class Histogram of selected dump .......................................................................................................... 6 1.6. A filter filtering jgroups threads ............................................................................................................. 7 1.7. A Custom Category ............................................................................................................................. 8 1.8. Long running threads detection .............................................................................................................. 9 1.9. TDA JConsole Plugin ........................................................................................................................ 11 1.10. TDA VisualVM Plugin ..................................................................................................................... 12 2.1. TDA giving hints about selected dump .................................................................................................. 14 2.2. Long running threads detection ............................................................................................................ 15

iv

1
General
This chapter gives an introduction on how to use the TDA - Thread Dump Analyzer. TDA parses your log files and displays all found thread dumps and class histograms reported from a Sun JVM 1.4.x or better, SAP VM or HP-UX VM. Class Histograms are not included in the thread dumps by default but need a special JVM-Flag to be dumped with the thread dump (see below). As TDA does everything offline and thread dumps have very low impact on the VM (including the class histogram option), it can be used for production environments.

Figure 1.1. Welcome Screen

TDA can also be used as plugin in JConsole or VisualVM for requesting thread dumps from a remote virtual machine using JMX. See JConsole Plugin for further information on using it as JConsole Plugin and see VisualVM Plugin for information concerning VisualVM..

General

1.1. Request Thread Dumps

There are different methods to request a thread dump from a running applications, either using a kill Signal on UNIX, using a little tool on windows, since Java 1.5 using jstack or via Java Management Extensions (JMX). Using a kill-Signal. For requesting a thread dump, you need to send a kill-QUIT-Signal to the Java VM. For this you the PID of the VM is required and then do a kill -QUIT <PID> in a Unix Environment. On Windows there is a special tool for accomplishing this, e.g. SendSignal. See the References for a link to the homepage of this tool. It does the same like kill -QUIT on Unix. Note though, if you're using the remote connection on windows, you need to start the connection with mstsc.exe /console otherwise you won't have sufficient rights to use the tool. The JDK will write the thread-dump to standard-out. E.g. if your application is running in a tomcat, this would be catalina.out in a default tomcat installation. In other situations this might just be the console where the application was started from. Using Java Management Extensions (JMX). Since Sun's Java Version 1.5 JMX is bundled with the JDK. The JDK also includes a small JMX client application JConsole, which can be used to connect to a running application using JMX. TDA is available as a plugin for JConsole. JMX is also available with VisualVM, a new troubleshooting tool. included in recent Java 1.6 JVM's. See Application Analysis using JConsole for detailed information on how to use JConsole with the TDA Plugin. Using jstack. Also starting with Sun's JDK 1.5 you can also request a thread dump using the JVM utility jstack. jstack prints the thread dump into the shell window where you called it. To use the dump in TDA, pipe the output into a log file and open this file. To analyze multiple dumps just append them to the log file. jstack also allows you to get thread dumps from javacorefiles, either dumped on an OutOfMemoryError or using the JVM tool jmap. On the windows platform this tool is available starting with JDK 1.6. It is not available at all on the linux-ia64 platform.

1.2. Thread Dump Parsing

Open the log file you want to analyse, TDA will search for all thread dumps in this log file and displays them in a tree. Up to Sun's JDK 1.5 the VM doesn't log any date information when the dump was requested, you can provide a regular expression in the Preferences which is then used for parsing the lines before the thread dump to get a time stamp (see below for an example). In JDK 1.6 a time stamp is logged just before the dump, the default regular expression is set up to parse this time stamp. In Regular Expression For Time Stamps you will find information about how to set up your own regular expression for time stamp parsing.

General

Figure 1.2. Information about the selected dump

For each thread dump TDA sums up all found threads and all monitors found in the threads, it also groups threads waiting on, for and locking monitors. Clicking on the dump nodes itself will provide you with information about the dump. If there is something special about it, TDA will give you some hints about what it found and what to do next.

General

Figure 1.3. Thread Dump View

If you want to focus on a lock found in a thread you can just click on the monitor, TDA will then expand the Monitor node of the thread dump and focuses the clicked monitor. It will also give you information about the monitor, if there is something special about, e.g. it has no obvious thread locking the monitor but others waiting for it. Starting here you can easily see if a thread is hanging and holding a lock which a lot of other threads are waiting for. If there aren't any locking threads but only waiting threads it is very likely the garbage collector is locking the monitor currently. In this case you will get a hint telling you about this.

General

Figure 1.4. Monitors used by threads

If you added the -XX:+PrintClassHistogram to the VM-Parameters you will also see the class histogram for a thread dump, presented as node of the dump. Here you can examine all objects in the heap at the time of the thread dump. You can sort this view and filter it using the Filter Expression.

IMPORTANT
There is a bug in early 1.5.x releases preventing the class histogram to be printed if any other than the concurrent mark sweep garbage collector is used.

General

Figure 1.5. Class Histogram of selected dump

If you use the loggc option with your VM to log the garbage collection information into a different log file, the class histogram will go into this log file instead of standard out and the TDA will not find it by default. To add the class histogram from a loggc file you need to use Open loggc file... in the popup menu of the thread dump pane. TDA will then parse this file backward and adds the found class histograms to the thread dumps starting with the last dump. You can added multiple loggc-files, TDA will then continue after the last one where it added a class histogram with the last loggc-logfile or uses the dump you clicked on. If your log file contains a lot of thread dumps done during one session of the VM, you can use Find long running threads in the tools menu to extract long running threads from the thread dumps. Currently this does also show waiting threads, so you need to search for threads actually doing something. Use appropriate filters to filter out all idle and uninteresting threads from the result.

1.3. Regular Expression For Time Stamps

Because the Sun JDK (<=1.5.x) doesn't provide any time information when dumping threads, a regular expression time stamp parsing for each line in the log file is included. TDA matches every line to the regular expression given in the preferences and stores the first matching group if the regular expression matches the line. It then takes the last match found as time stamp for the next thread dump to get a time stamp nearest to the thread dump. An example for such a regular expression is (\d\d\/\d\d\/\d\d\s\d\d:\d\d:\d\d).* This expression matches lines like

General

06/02/14 14:54:04

at java.lang.Thread.run(Thread.java:534)

TDA takes the first capturing group of the expression as time stamp. In this example the \d is for digits and the capturing group is included in the brackets and matches <i>06/02/14 14:54:04</i> . This is stored as timestamp for the next thread dump. Starting with JDK 1.6 the SUN JDKs print out a time stamp in the line before the dump. There is a default regular expression which recognizes this time stamp. If the timestamp is stored in milliseconds since 1970 there is a checkbox in the preferences Parsed timestamp is a long representing msecs since 1970 to tell TDA to convert the parsed time stamp from milliseconds into a human readable time stamp. See References for detailed information about the regular expressions in java.

1.4. Using Filters

To filter out sleeping and uninteresting threads in your view you can use filters to just ignore these threads. For example if you use jgroups in your application and don't want to see als the STACKs and QUEUEs of jgroups you can specify something like that seen in the next screenshot.

Figure 1.6. A filter filtering jgroups threads

The status bar will provide information about the number of filtered threads and how many are remaining. Note that changes to the filters only apply after you changed a view. The filter changes will not be applied to the currently displayed thread view. Filters can be used to match different parts of the thread information. You can match or search in stack, title and you can check for the thread state the thread is in. Filters can be including or excluding. The example above is a excluding filter which checks 7

General

if org.jgroups is in the stack information of a thread. If a specified filter should be applied to all displayed categories, the Default checkbox needs to be checked. Filters which aren't applied to all categories can be used in custom categories.

1.5. Using Categories

If you have special groups of threads in your application or application environment, e.g. request pool threads, you can define custom thread dump categories for grouping these threads so you can easier access them if you want to check for specific data in these threads.

Figure 1.7. A Custom Category

The Screenshot shows a custom category grouping all pool threads in an application using Java NIO as API to parse input stream data. A category consist of at least one filter. For this category a filter JCSNioFilter was defined which filters all these threads and which is inactive as default filter so it isn't applied to the other categories (default and custom ones). If a category doesn't contain any threads it is automatically hidden.

1.6. Deadlocks
If the JVM finds a java level deadlock in your application it logs this into the thread dump. This information will be displayed in the deadlock node of the dump. There is also a hint concerning the found deadlock(s) in the dump summary. Note though, it doesn't have any deadlock detection for java level deadlocks itself, it relies on the deadlock detection of the VM. TDA also tries to find deadlocks which might be caused by some external resources or some remote communication and can't 8

General

be found by the Java VM. If there are indication for such a deadlock, TDA will display information concerning this in the dump summury and gives you hints what to do next.

1.7. Sessions
If you opened several logfiles and you want to quit TDA and go back to this logfiles later, you can store the open logfiles to a session file. The logfile tree then is dumped into a session file for later usage. You must keep the logfiles if you want to be able to browse them again. For the tree navigation they are not necessary. Note that session are only supported for one version of TDA, they are not exchangeable between different versions.

1.8. Long Running Threads

A common usecase for thread dump analysis is the long running thread detection. TDA offers some help for finding these long running threads. The "Find long running threads" option in the tools menu or from the popup menu in the main tree checks for long running threads in marked thread dumps in the main tree.

Figure 1.8. Long running threads detection

The analysis is quite simple though, it just compares the thread names and if they match adds the thread to the result. A regular expression can be specified for filtering just the interesting threads. The usual filters are also applied to the result view of the analysis to filter out uninteresting threads. For a useful result appropriate filters should be specified to filter out all sleeping and uninteresting threads.

General

HINT
The long running threads detection is a bit picky about the selected nodes for the detection. You should only select thread dump root nodes otherwise the detection might through an exception as it doesn't know what to do with the selected nodes. You also always should select nodes from one VM run, otherwise you might get quite weird results because the detection matched threads from different VM runs.

1.9. Heavy Load Analysis

If a system is under heavy load, several thread dumps should be taken (e.g. every two minutes over a short period). The threads all have a thread id which can be mapped to real process ID on Linux/Solaris Systems. TDA parses the ID and displays it as "Native ID" in the thread view. Using top in light-weight-process view on Linux and prstat on Solaris with the process id of the java process where the thread dumps came from you can match this native ID to the threads in the dump. This way it is easy to identify threads which produce heavy load as the thread view in top and prstat shows the cpu load of each single thread.

1.10. JConsole Plugin

TDA can be used as plugin in the JVM utility JConsole. Use the following to start JConsole with TDA as plugin: jconsole -pluginpath ./tda.jar The javahelp jar jhall.jar needs to be in the same directory as the tda.jar or in the subdirectory lib, otherwise clicking the help button would lead to an exception. The Plugin offers requesting thread dumps from a remote Virtual Machine through JMX. The Dumps areparsed and can be analyzed just like running as standalone-application. The upper-right-frame showing the dump tree has a popup menu offering the plugin features.

10

General

Figure 1.9. TDA JConsole Plugin

The toolbar is also available for convenience, it can be switched off using the popup menu. The requested Thread Dumps can also be stored as Logfile for later offline analysis. The plugin supports requesting thread dumps from 1.5 and 1.6 VMs. The extended informations from 1.6 are not parsed yet though (e.g. informations about locked synchronizers). But they are saved to the logfile also.

1.11. VisualVM Plugin

TDA can also be included into VisualVM . It is available as plugin from the VisualVM Plugin Center. Note though, if you use jvisualvm from recent Sun JDK's you wont find it in the plugin center but you can still download it manually from the TDA Website from the Document and Files section. All three files available for download need to be installed.
1

https://visualvm.dev.java.net

11

General

Figure 1.10. TDA VisualVM Plugin

The Plugin attaches itself to the thread dump feature in VVM and offers a TDA View of the application thread dumps and the navigation tree offers a Logfile section, where you can add logfiles containing thread dumps. The TDA help is only available as a limited subset of the full help because VisualVM doesn't include JavaHelp. To get the full help, either download the documentation from the TDA Website or start TDA stand-alone.

12

2
Analyzing production environments
Starting with Sun's JDK 1.5 it is quite easy to get information about a badly behaving application in a production environment using the JVM tools jmap and jstack. Jmap fetches a heap dump from the VM the application is running in and jstack fetches the thread dumps. These information can be taken from the server the application server is running on and analyzed offline. Even without remote access to the server you still can get these dumps from your customer. Many Monitoring tools usually need some kind of remote access. Imagine a customer's web application with a lot of concurrent users accessing the system and the system is under heavy load. The system is only accessed during office hours. But you have noticed the load of the system stays high in the night although nobody is accessing it. What can you do?

2.1. Dumping Information about running threads

First of all you can fetch several thread dumps either using jstack or using kill -QUIT if you're using an older VM than 1.5. Using these dumps you can easily detect long running (looping?) threads which might cause the load. You should fetch about five dumps and wait several seconds between fetching, lets say 20 seconds. The command line with jstack then looks something like this jstack <pid> > dump.log for the first dump and jstack >> dump.log

for the additional dumps. You then can use the TDA - Thread Dump Analyzer to analyze the thread dumps to get an idea what is currently happening in your application. TDA will try to give you some hints about what might be wrong in the dumps (e.g. a lot of threads are waiting for the garbage collector) like in the screenshot below.

13

Analyzing production environments

Figure 2.1. TDA giving hints about selected dump

High load without anybody accessing the application usually either means the garbage collector is running endlessly because the application is very low on memory and the garbage collector is unable to free enough memory (TDA will give you a hint on this) or there are some threads (or just one) looping and running endlessly. TDA tries to filter out all idle threads to make it easier for you to find really running threads which are either running or are waiting for some external resources (it will give you a hint regarding the last issue). You need to enable these filters in the filter settings. You might also have a look at the TDA help for additional information. For letting TDA search for long running threads mark the threads dumps you want to analyze and choose the "Find long running" threads option. TDA will then search for threads it finds in the dumps at least n-times, whereas you define n in the setting dialog for the detection. It will then present the result added to the dump tree. You should enable the idle-threads filter to filter out unimportant threads. In the screenshot below you can see a long running thread which is running in an endless loop.

14

Analyzing production environments

Figure 2.2. Long running threads detection

2.2. Long running thread detection on Solaris

If your application is running on Solaris and you have remote access to the machine you can even determine directly the thread which might be looping by using prstat -p <app-server-vm-pid>. prstat will show you all threads running in your application and how much CPU time each of them is consuming. The pids shown there (in decimal) can be matched to the nids (in hexadecimal) in the thread dumps.

2.3. Analyzing the heap profile

I use the YourKit Profiler (commercial but free eval-license for 15 days) to analyze the heap profiles. The tool has the quite handy "Find big objects" option which makes it very easy to find stuff which consumes a lot of memory and shouldn't be there or at least not that big. If you don't want to spend the money though, you can use the JVM tool jhat which starts a small web server, where you can connect to and analyze the heap profile. For a small overview you can also use the class histogram feature of the TDA tool. Use either jstack to fetch a class histogram from your application VM or use the VM-option -XX:+PrintClassHistogram to have it dumped when requesting a thread dump if you are on an 1.4.x VM. Refer to the TDA help for further information.

15

3
Application Analysis using JConsole
With JDK 1.5 Sun introduced JConsole, a swing based tool for instrumentalizing the new JMX MBeans for VM analysis in 1.5 and to provide easy remote access to all other JMX MBeans offered (e.g. by the application server or an application). With the release of JDK 1.6 Sun enhanced the utility to also support plugins and simplified access to the applications to analyze. With JConsole it is quite easy to do some application analysis without critical impact on the application.

3.1. Remote application access

For enabling remote access to an application using JConsole you need an open port from the machine the application is running on and start the application with this port. To connect to a remote application running with JDK 1.5 you need to specify something like this: -Dcom.sun.management.jmxremote.port=5555 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false I have done this for connecting to an Oracle Application Server 10.1.3.2 instance. Note: this is not the recommended setting for an production environment as is insecure. Read the jconsole documentation on how to set up a secure connection to a JVM.

3.2. Requesting Dumps

Since the 1.6 release it is also quite easy to request heap and thread dumps. These Dumps then can be parsed by tools like the SAP Memory Analyzer to analyze heap dumps and tools like Samurai or my TDA for thread dumps.

3.3. Plugins For JConsole

One interesting plugin currently available publically is an enhanced JTop Plugin (based on a demo plugin from the jdk samples). JConsole doesn't display any information about the cpu usage of the threads running in a JVM, it only displays a graph about the overall cpu usage. The JTop plugin shows enhanced information about the cpu utilization of all threads running in a JVM just like top does for processes on unix plattforms. The enhanced JTop plugin can be downloaded from Peter Doornbosch's Blog.

16

Appendix A. References
i. GC-Viewer ii. VisualVM
3 2 1

iii JConsole . 4 iv Java API Specification 1.4.2 (for regular expressions look for Pattern class.) . 5 v. SendSignal vi Dumpster (my blog) .
6

1 2

http://www.tagtraum.com/gcviewer.html http://visualvm.dev.java.net/ 3 http://java.sun.com/j2se/1.5.0/docs/guide/management/jconsole.html 4 http://java.sun.com/j2se/1.4.2/docs/api/index.html 5 http://www.latenighthacking.com/projects/2003/sendSignal/ 6 http://www.jroller.com/dumpster

17