Diagnosing Memory Leaks using

Graph Mining on Heap Dumps

Evan K. Maxwell Godmar Back Naren Ramakrishnan

[email protected] [email protected] [email protected]
Department of Computer Science
Virginia Tech, VA 24061

ABSTRACT downs and eventually the exhaustion of all available mem-

Memory leaks are caused by software programs that prevent ory, triggering out-of-memory conditions that usually lead
the reclamation of memory that is no longer in use. They can to application crashes. These crashes significantly affect
cause significant slowdowns, exhaustion of available storage availability, particularly of long-running server applications,
space and, eventually, application crashes. Detecting mem- which is why memory leaks are one of most frequently re-
ory leaks is challenging because real-world applications are ported types of bugs against server frameworks.
built on multiple layers of software frameworks, making it Memory leaks are challenging to identify and debug for
difficult for a developer to know whether observed references several reasons. First, the observed failure may be far re-
to objects are legitimate or the cause of a leak. We present moved from the error that caused it, requiring the use of
a graph mining solution to this problem wherein we ana- heap analysis tools that examine the state of the reachability
lyze heap dumps to automatically identify subgraphs which graph when a failure occurred. Second, real-world applica-
could represent potential memory leak sources. Although tions usually make heavy use of several layers of frameworks
heap dumps are commonly analyzed in existing heap pro- whose implementation details are unknown to the develop-
filing tools, our work is the first to apply a graph grammar ers debugging encountered memory leaks. Often, these de-
mining solution to this problem. Unlike classical graph min- velopers cannot distinguish whether an observed reference
ing work, we show that it suffices to mine the dominator chain is legitimate (such as when objects are kept in a cache
tree of the heap dump, which is significantly smaller than in anticipation of future uses), or represents a leak. Third,
the underlying graph. Our approach identifies not just leak- the sheer size of the heap — large-scale server applications
ing candidates and their structure, but also provides aggre- can easily contain tens of millions of objects — makes man-
gate information about the access path to the leaks. We ual inspection of even a small subset of objects difficult or
demonstrate several synthetic as well as real-world exam- impossible.
ples of heap dumps for which our approach provides more Existing diagnosis tools are either online or offline. On-
insight into the problem than state-of-the-art tools such as line tools monitor either the state of the heap or accesses
Eclipse’s MAT. to objects in it, or both. They analyze changes in the heap
over time to detect leak candidates, which are “stale” objects
Categories and Subject Descriptors: H.2.8 [Database that have not been accessed for some time. Online tools are
Management]: Data mining; D.2.5 [Software Engineering]: not widely used in production environments, in part because
Debugging aids; their overhead can make them too expensive, but also be-
General Terms: Algorithms, Experimentation, Reliabil- cause the need to debug memory leaks often occurs unex-
ity. pectedly after an upgrade or change to a framework compo-
Keywords: Memory leaks, heap profiling, graph mining, nent, and often when developers believe their code has been
graph grammars, dominator tree. sufficiently tested. Offline tools use heap snapshots, often
obtained post-mortem when the system runs out of memory.
These tools find leak candidates by analyzing the relation-
1. INTRODUCTION ships, types, and sizes of objects and reference chains. Most
Memory leaks are a frequent source of bugs in applica- existing heuristics, however, are based solely on the amount
tions that use dynamic memory allocation. They occur if of memory an object retains and ignore structural informa-
programmers’ mistakes prevent the deallocation of memory tion. Where structural information is taken into account,
that is no longer used. Undetected memory leaks cause slow- it often relies on priori knowledge of the application and its
libraries.
This paper presents a graph mining approach to identify-
ing leak candidate structures in heap dumps. Our approach
Permission to make digital or hard copies of all or part of this work for is based on the observation that leaks often involve container
personal or classroom use is granted without fee provided that copies are data structures from which programmers fail to remove un-
not made or distributed for profit or commercial advantage and that copies needed objects that were previously added. Consequently,
bear this notice and the full citation on the first page. To copy otherwise, to the heap dump involves many subgraphs of similar struc-
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
ture containing the leaked objects and their descendants. By
KDD’10, July 25–28, 2010, Washington, DC, USA. mining the dump, we can identify those recurring subgraphs,
Copyright 2010 ACM 978-1-4503-0055-1/10/07 ...$10.00.
 public class HiddenLeak
present the developer with statistics about their frequency {
and location within the graph. Our key contributions can static HashMap legitimateMap;
be summarized as follows:
static class Legitimate
1. Although analysis techniques are widely used in heap {
analysis [21–23], our work is the first to employ graph HashMap leakyMap = new HashMap();
mining for detecting leaking candidates. Specifically,
static class Leak
we demonstrate that graph grammar mining used in {
an offline manner can detect both seeded and known // This object is leaked
memory leaks in real applications. }

2. Compared to other offline analysis techniques, our ap- void leak()

proach does not require any a priori knowledge about {
which classes are containers, or about their internal // insert Leak instances into leakyMap
}
structure. It captures containers even when these are }
embedded into application classes, such as ad-hoc lists
or arrays. public static void main(String []av)
{
3. Our approach can identify leaks even if the leaks’ lo- // create N instances of Legitimate
cations within the graph do not share a common an- for (int i = 0; i < N; i++)
cestor node, or if the paths from that ancestor to the { Legitimate legit = new Legitimate();
instances are difficult to find by the manual examina- legit.leak();
legitimateMap.put(i, legit);
tion that is required in existing tools such as Eclipse }
Memory Analyzer (MAT). }
}
4. Graph grammar mining can find recursive structures,
giving a user insight into the data structures used in Figure 1: An example of a leak “hidden” underneath
a program. For instance, linked lists and trees can be legitimate objects.
identified by their distinct signatures.

5. Finally, the ability to combine subgraph frequency with

location information makes our algorithm robust to the container. Each Legitimate instance, by itself, represents
presence of object structures that occur naturally with data the program needs to maintain and thus it needs to re-
high frequency without constituting a leak. tain references to each instance. However, Legitimate also
references a container leakyMap that accrues, over time, ob-
jects of type Leak that should not be stored permanently.
2. ANATOMY OF A LEAK Figure 2 shows the resulting heap structure. The hash maps
Although memory leaks can occur in all languages that use exploit an open hashing scheme, which uses an array of buck-
dynamically allocated memory, they are particularly preva- ets. Each bucket maintains a separate chain of entries cor-
lent in type-safe languages such as Java, which rely on garbage responding to keys for which hash collisions occurred.
collection to reclaim memory. In these languages, dynam- Over time, the space taken up by the leaked object will
ically allocated objects are freed only if the garbage col- grow until all available heap space is exhausted. When this
lector can prove that no accesses to them are possible on limit is reached, the Java virtual machine throws a runtime
any future execution path, which is true if and only if there error (OutOfMemoryError). In many production environ-
is no path from a set of known roots to the object in the ments, the JVM is run with a flag that saves a snapshot
reachability graph. The reachability graph consists of nodes of the heap at this point, which is then fed to a heap ana-
that represent allocated objects and edges that correspond lyzer tool.
to inter-object references stored in instance variables (fields). Most existing tools compute and analyze the dominator
Roots, also known as garbage collection (GC) roots, are tree, which represents a relationship between nodes in the
nodes whose liveness does not depend on the liveness of other heap that expresses which objects a given object keeps alive.
heap objects, but on the execution semantics of the program. Object ‘a’ dominates ‘b’ if all paths from a root to ‘b’ go
For instance, in Java, local and global static variables repre- through ‘a’. If the object graph is a tree, as in this example,
sent roots because the objects referred by them must remain then it is identical to its dominator tree. In the example,
reachable throughout the execution of a method or program, the static (global) variable legitimateMap is the dominator
respectively. Hence, memory leaks form if objects remain tree root that keeps alive all leaked objects.
reachable from a GC root even though the program will no However, inspection of the dominator tree for this exam-
longer access them. Such leaks also occur in programming ple with existing tools does not readily point to the leak.
languages with explicit memory management, such as C++, For instance, when examining a heap dump with the Eclipse
and our work applies to them. We do not consider leaks Memory Analyzer tool (available at http://eclipse.org/mat),
arising from memory management errors in those languages the tool pointed at ‘legitimateMap’ as a likely leak candi-
(e.g., failing to deallocate unreachable objects). date, because it keeps alive a large fraction of the heap.
The Java program sketched in Figure 1 illustrates how None of its children stands out as a big consumer with re-
leaks in a program manifest themselves in the reachability spect to retained heap size. The leak is “hidden” under a
graph. In this example, a program maintains a number of blanket of legitimate objects. At this point, a developer
objects of type Legitimate, which are stored in a hash map would be required to “dig down,” and individually examine
 A1
0
0
1 A1
0
0
1
i. 0
1 ii. 0
1
HiddenLeak

legitimateMap
B1
0
0
1 C1
0
0
1
1D
0
0
1 B1
0
0
1 C1
0
0
1
1D
0
0
1
0
1 0
1 0
1 0
1 0
1 0
1

E1
0
0
1
1F
0
0
1
1G
0
0
1 E1
0
0
1
1F
0
0
1
1G
0
0
1
0
1 0
1 0
1 0
1 0
1 0
1

Legitimate Legitimate Legitimate

1
0H I1
0 1
0J 1
0K H1
0 1
0I 1
0J 1
0K
0
1 0
1 0
1 0
1 0
1 0
1 0
1 0
1
leakyMap leakyMap leakyMap 0
1 0
1 0
1 0
1 0
1 0
1 0
1 0
1

0
1
L1
0 0M
1
1
0 0
1
L1
0 0M
1
1
0
0
1 0
1 0
1 0
1

Figure 3: (i) An example graph and (ii) its domi-

L L L L L L L L L L L L nator tree. In practice, the dominator will have a
significantly reduced number of edges than the orig-
inal graph.

Figure 2: Heap graph after executing the program

shown in Figure 1. dominator of v in G. Figure 3 shows an example graph
and its dominator tree. The computation of D requires the
specification of an entry node into G. We therefore introduce
paths through each Legitimate instance, which is cumber- a pseudo-root node ρ to G, and add edges (ρ → GCi ) where
some without global information about the structure of the GCi ∈ GC, the set of Java garbage collection roots (see
subtrees emanating from these instances. Heap analyzers do Section 2).
support some global information, but is usually limited to The dominator tree computation on the heap dump graph
histogram statistics that shows how often objects of a certain provides us with several benefits. First, it significantly re-
type occur. This approach often leads to limited insight be- duces the number of edges in the graph to be mined. Second,
cause String objects and char arrays are the classes whose since the resulting graph is a tree, we can apply optimiza-
objects usually consume most memory in Java programs. tions in the mining algorithm specific to mining trees. Fi-
Consequently, there is a need to mine the graph to iden- nally, recall that our goal is not just to find the frequent
tify structures that are likely leak candidates, even if these subgraph representing the leak but also to characterize the
structures are hidden beneath legitimate live objects. More- source of the leak. Once the dominator tree has been com-
over, developers require aggregate information that describe puted, we can search the paths from the entry node of the
where in the object graph these leak candidates are located. leaking subgraph up to the root of the dominator tree. This
path is generally sufficient to identify the source of the leak.
3. ALGORITHMS Without the dominator tree computation, tracing all paths
As stated in the introduction, our approach is based on to garbage collection roots in the graph is much more ex-
the observation that a leak would manifest as a heap dump pensive and is full of noise.
containing many similar subgraphs. Rather than directly
mine the heap dump, we compute the dominator tree of the 3.2 Mining the Dominator Tree
heap dump and mine frequent graph grammars [15, 17] in In general, frequent subgraphs in the original heap dump
the dominator tree. We describe the rationale behind this need not necessarily be frequent in the dominator tree. To
approach and algorithmic design decisions in this section. understand this, consider the cases in Fig. 4 which shows
example graphs that are frequent in both the dump and the
3.1 Dominator Tree Computation dominator, frequent in the dump but not the dominator, as
The computation of dominators is a well studied topic well as the other two combinations. In particular, the (fre-
and we adopt the Lengauer-Tarjan algorithm [19] from the quent in dump, infrequent in dominator) combination occurs
Boost graph library implementation. This algorithm runs in due to the existence of different routes of entry into a fre-
O((|V | + |E|) log(|V | + |E|)) time, where |V | is the number quent subgraph S in graph G. This situates S into different
of vertices and |E| is the number of edges. subgraphs in D that may not be frequent individually. The
Formally, the dominator relationship is defined as follows. reverse combination, i.e., (infrequent in dump, frequent in
A node x dominates a node y in a directed graph iff all dominator), also happens, because frequent subgraphs in D
paths to node y must pass through node x. In Fig. 3 (left), may contain edges that summarize dissimilar paths in G.
all paths to node L must pass through node A, therefore A Nevertheless, practical heap dumps have specific degree
dominates L. A node x is said to be the unique immediate distribution properties that we can exploit. As we show
dominator of y iff x dominates y and there does not exist a later, a large majority of nodes in heap dumps have an in-
node z such that x dominates z and z dominates y. Note degree of either zero or one. This implies that cases as shown
that a node can have at most one immediate dominator, but in Fig. 4 (top right) are much fewer in number than cases
may be the immediate dominator of any number of nodes. in Fig. 4 (top left). This is a key distinction because we
The dominator tree D = (V D , E D ) is a tree induced from can guarantee that if a frequent subgraph S in G contains
the original directed graph G = (V G , E G ), where V D = only nodes having in-degree ≤ 1, all instances of S will be
V G , but an edge (u → v) ∈ E D iff u is the immediate completely conserved after the computation of the domi-
nator tree D and will retain their frequencies. Although and the result of compressing G using S (denoted by G|S).
it is unlikely that a frequent subgraph in the heap dump The first example shows how the graph grammar can de-
will comprise exclusively of nodes with in-degree ≤ 1, ex- scribe the same information as a frequent subgraph. The
perience shows that it will be composed predominately of second example demonstrates additional features of graph
such nodes. These observations justify our design decision grammars. The graph grammar mining algorithm works it-
to compute the dominator tree as a preprocessing step be- eratively, where in each iteration the top-ranked graph gram-
fore graph mining and to mine frequent structures in the mar is used to compress G. The next iteration repeats the
dominator. process on the newly compressed version of G with non-
terminal nodes. In practice, we run the algorithm for just
i. ii.
G D G D
1
0
A
1
0
A a few iterations (≤ 3) because we focus on the top-ranked
A A A A A A A A
1
0 1
0 1
0 1
0 1
0 1
0 1
0 1
0 grammars when trying to identify a memory leak. For scal-
B C B C
1
0 1
0 1
0 1
0 ability reasons, we use sampling at several key states in the
1
0 1
0
mining process.
B B
1D
0 1E
0 1D
0 1E
0
Frequent Frequent Frequent Not Frequent
1. Candidate generation: When we expand an evalu-
ated pattern with k edges to a candidate pattern with
iii.
G D
iv.
G D k+1 edges, we need not generate and explore all candi-
A A
1
0 1
0
1
0
A
A
1
0 1
0
A dates because the algorithm uses a beam search, which
1
0
X
0
Z
1 0
1U 1V
0
we limit to 15, to bound the number of patterns gen-
1Y
0
B C B C erated at each stage [17]. We take a random sample
1
0 1
0 1
0 1
0 1
0
1B
0 1B
0 B of < 1% of the instances of the extending pattern to
Not Frequent Frequent Not Frequent Not Frequent determine the best candidates to fill the beam. The
full set of instances will then be explored only for the
Figure 4: Examples of subgraphs that may be fre- best candidates determined by the sample.
quent or not frequent in either or both the graph G 2. Scoring candidates: When a new candidate graph
and its dominator tree D. (i-iv) show all 4 classes grammar is generated, we must calculate the size heuris-
of subgraphs defined by the different combinations. tic of the candidate. Since it will not have been checked
Dashed edges represent a “dominates” edge pro- for recursiveness yet, we must ensure that we do not
duced only for D. over-estimate the size in the case that instances over-
lap. Instead of checking all instances of the candidate,
we estimate the size heuristic by looking at a random
3.3 Inducing Graph Grammars sample of ∼ 5% of all instances to get an idea for how
To mine patterns in the dominator tree, we explore the much overlap occurs within the grammar’s population.
use of context-free graph grammars [10, 28] instead of mere This estimate is used to approximate the actual score.
subgraphs. Graph grammars are necessary because leak-
3. Recursive Opportunities: Similar to the sampling
ing objects are often recursive in nature and we require the
technique we use for scoring candidates, we sample the
expressiveness of graph grammars. Furthermore, it is not
candidate grammar’s instances to determine if and how
necessarily the number of instances of a subgraph that is
it is recursive. In this case, we use a sample of < 1%
important in debugging the leak, but rather the percent-
of the instances. We can afford to use a smaller sam-
age of the heap dump that is composed of instances of the
ple size than in the scoring function because scoring
subgraph. An algorithm that finds subgraphs could be mis-
requires a higher degree of accuracy.
leading because a simple count of the number of instances
could over-calculate the composition of the subgraph. Because the statistical significance of our sample would be
The concept of a graph grammar is akin to a formal lan- largely dependent on the prevalence and recursive nature
guage grammar, except that the productions generate sub- of a candidate graph grammar, we cannot generalize our
graphs rather than substrings. We are specifically interested method to all cases and we choose our sample sizes empiri-
in node-label controlled grammars, as these contain a single cally without claiming statistical significance. However, our
non-terminal labeled node which can be replaced with any small sample sizes worked well in all of our experiments.
subgraph. These grammars take the form of S → P , where We also note that once the top scored graph grammar is re-
S is a single non-terminal node, P can be any subgraph, and turned by the algorithm with sampling, we ensure that the
the arrow implies a production/replacement rule. recursion detection and compression of the input graph by
As described in [9, 15, 17], we build graph grammars in a the grammar are done exactly.
priori fashion where we find a production rule capturing a
significant portion of the input graph, replacing instances 4. EVALUATION
of the production by its non-terminal symbol, and contin-
Our evaluation contains three parts. First, we check whe-
uing. Candidate productions are evaluated for their ability
ther our algorithm finds seeded structures in a set of syn-
to compress the graph. We use the alternative size heuristic
thetically created dumps. Second, we examine its efficacy
from [17]:
on heap dumps we obtained from Java developers. Third,
size(G) we report its scalability and performance with respect to the
size(S) + size(G|S) sizes of the heap graphs considered.

where size(G) = |V | + |E|. Figure 5 shows two example

input graphs G, a top-scoring graph grammar S for each,
i. ii.
R R

R
(S) R
B C B A B A
B A

S S C S S S B S
C A B A C B C B A B C
A C
B C
Z Y S X Z
Z Y C A X Z Z Y X B X Z
Z Y X S X Z

G S G|S G S G|S

Figure 5: Two examples of an input graph G, a graph grammar S inferred from G, and the result of compress-
ing G on S, denoted G|S. (i) A non-recursive grammar which does not contain any embedded non-terminal
nodes or edges. (ii) Recursive grammar on node A containing an embedded non-terminal node that can
match either nodes of type B or C. The grey box containing the “(S)” label represents a recursive connection
instruction.

4.1 Preparing Heap Dumps node in the best grammar displayed in Figure 6, and edge
We obtained heap dumps from Sun’s Java VM (version labels (in parenthesis) stem from the explicit and implicit
1.6) using either the jmap tool or via the -XX:+HeapDump- variable names used in the source code from Figure 1:
OnOutOfMemoryError option, which triggers an automatic java.lang.Class
heap dump when the JVM runs out of memory. We use the | (legitimateMap)
+-+-> java.util.HashMap
com.sun.tools.hat.* API to process the dump and extract
| (table)
the reachability graph. Each node in the graph corresponds +-+-> [Ljava.util.HashMap$Entry;
to a Java object, which is labeled with its class. Each edge in | ($array$)
the graph corresponds to a reference, labeled with the name +-+-> java.util.HashMap$Entry
of the field containing the reference. We label edges from | (value)
arrays with $array$, ignoring the specific index in which a +-+-> HiddenLeak$Legitimate
| (leakyMap)
reference is stored. We remove all edges that correspond to
+-+-> java.util.HashMap
weak and soft references, because weak and soft references | (table)
to an object do not prevent that object from being reclaimed +-+-> [Ljava.util.HashMap$Entry;
if the garbage collector runs low on memory. | ($array$)
+-+-> java.util.HashMap$Entry
4.2 Synthetic Examples
The remaining paths contain an additional edge Entry-
.next, which represents the case in which a hash collision led
java.util.HashMap$Entry to chaining. This information immediately describes the lo-
cation of all instances of Leak objects in the graph, alerting
key value the developer that a large number of these structures has
accumulated underneath each Legitimate object. As dis-
cussed in Section 2, a size-based analysis of the dominator
java.lang.String HiddenLeak$Legitimate$Leak
tree as done in Eclipse’s Memory Analyzer would lead only
to the bucket array of the HashMap object referred to by ‘le-
value gitimateMap’ and require manual inspection of the subtree
emanating from it.
char[] Since programmers often choose container types depend-
ing on specific space/time trade-offs related to an expected
access pattern, we then investigated if the leaking struc-
Figure 6: Most frequent grammar mined from Hid- ture would be found if a different container type had been
denLeak example in Figure 2. The $ character de- used. We replaced both uses of HashMap with class TreeMap,
notes nested classes. For example, the node Hidden- which uses a red-black tree implementation. Our algorithm
Leak$Legitimate$Leak corresponds to Java class Hid- correctly identified a grammar consisting of TreeMap.Entry
denLeak.Legitimate.Leak in the source code. objects that refer to a (key, value) pair, near-identical to
the grammar shown in Figure 6. In addition, the aggre-
We first present the results for the motivating example gated path was expressed by the recursive grammar shown
presented in Section 2. Figure 6 shows the most frequently in Figure 7, which covers over 99% of observed paths from
occurring mined grammar, which represents a (key, value) a root to the grammar’s instances.
pair anchored by an instance of type HashMap.Entry. This This path grammar identifies the leak as hidden in a tree
information directs an expert’s attention immediately to a of trees and provides a global picture that would be nearly
HashMap mapping keys of type String to values of type Hid- impossible to obtain by visual inspection. The use of recur-
denLeak.Legitimate.Leak. We found that 70% of the paths sive productions enabled the algorithm to identify a classic
from the instances produced by this grammar to GC roots container data structure (a binary tree) without any a priori
in the original graph exhibit the following structure, where knowledge. The use of recursive grammars is also essential
java.lang.Class (top) is a root node of the dominator tree, for other recursive data structures, such as linked lists. The
java.util.HashMap$Entry (bottom) corresponds to the top following example demonstrates that our mining approach
+-+-> java.lang.Class
| (legitimateMap) (S)
+-+-> java.util.TreeMap
| (root)
{
+-+-> java.util.TreeMap$Entry OOML java.lang.String
| ( right | left )
+-+-> java.util.TreeMap$Entry
}*
| (value) next payload
+-+-> leaks.TreeMapLeaks$Leak
| (leakyMap)
+-+-> java.util.TreeMap
| (root)
{
OOML
+-+-> java.util.TreeMap$Entry
| ( right | left )
+-+-> java.util.TreeMap$Entry
}* Figure 9: Most frequent grammar in synthetic
linked list example.
Figure 7: Resulting root path grammar if a TreeMap
container is used for the example shown in Figure 2.
that makes heavy use of multiple frameworks [2] including
the Apache Tomcat 5.5 servlet container. These heap dumps
easily captures such data structures, even when they occur
were generated over a period of several months. When the
embedded in application classes (rather than in dedicated
server ran out of memory during intense testing, develop-
collection classes). Class OOML in Figure 8 embeds a link el-
ers would simply save a heap dump and restart the server,
ement next and an application-specific payload field. The
without further immediate investigation of the cause.
main method contains an infinite loop which will add el-
We obtained a total of 20 heapdumps, varying in sizes
ements to a list held in a local variable ‘root’ until heap
from 33 to 47MB. In all of these dumps, the grammar shown
memory is exhausted.
in Figure 10 percolated to the top. This grammar represents
public class OOML { instances of type BeanInfoManager that reference a HashMap
OOML next; // next element through their mPropertyByName field. 80% of the root paths
String payload; are expressed via the following grammar:
+-+-> org.apache.catalina.loader.StandardClassLoader
OOML(String payload, OOML next) {
| (classes)
this.payload = payload;
+-+-> java.util.Vector
this.next = next;
| (elementData)
}
+-+-> [Ljava.lang.Object;
| ($array$)
// add nodes to list until out of memory
+-+-> java.lang.Class
public static void main(String []av) {
| (mBeanInfoManagerByClass)
OOML root = new OOML("root", null);
+-+-> java.util.HashMap
for (int i = 0; ; i++)
| (table)
root = new OOML("content", root);
+-+-> [Ljava.util.HashMap$Entry;
}
| ($array$)
}
+-+-> java.util.HashMap$Entry
| (value)
Figure 8: A singly linked list embedded in an appli- +-+-> org.apache.commons.el.BeanInfoManager
cation class.
This grammar shows that the majority of these objects
Figure 9 shows the most frequent subgraph, which con- are kept alive via a field named mBeanInfoManagerByClass.
tains a recursive production OOM L →next OOM L, rep- Since the field is associated with a node of type Class, it
resenting a single linked list. The root path aggregation represents a static field. Examination of an actual path in-
showed the location of its instances in the graph: stance reveals that this static field belongs to class org-
+-+-> Java_Local .apache.commons.el.BeanInfoManager.
| (root) Similar to the HiddenLeak example, the Eclipse analyzer
{ reported the (legitimate!) HashMap.Entry array stored in
+-+-> OOML
| (next) the mBeanInfoManagerByClass class as accumulation point,
+-+-> OOML but could not provide insights into the structure of the ob-
}* jects kept in this table without tedious manual inspection.
We eventually found that this leak had already been re-
4.3 Web Application Heapdumps ported by another developer against the Tomcat Apache
server (Bug 38048: Classloader leak caused by EL evalu-
4.3.1 Apache Tomcat/J2EE ation). Interestingly, the original bug report had received
We obtained a series of heap dumps that resulted from re- little attention, likely because the bug reporter included only
curring out-of-memory situations during the development of a single trace to a leaked object reachable from the Bean-
the LibX Edition Builder, a complex J2EE web application InfoManager class.
 (S) as memory exhaustion due to a large object kept alive by a
long list of RegExMatch objects.
org.apache.commons.el.BeanInfoManager

mPropertyByName (S)

org.mvel.ast.RegExMatch
java.util.HashMap

nextASTNode stmt {name | pattern} patternStmt

table
org.mvel.ast.RegExMatch org.mvel.ExecutableAccessor [C org.mvel.CompiledExpression
[Ljava.util.HashMap$Entry;
node tokens

$array$
org.mvel.ast.ThisValDeepPropertyNode org.mvel.ASTLinkedList

java.util.HashMap$Entry name

next value [C

java.util.HashMap$Entry org.apache.commons.el.BeanInfoProperty

Figure 11: Most frequent grammar in MVEL heap

value
dump.
org.apache.commons.el.BeanInfoProperty

4.4 Scalability and Other Quantitative Aspects

Figure 10: Most frequent grammar in Tomcat 5.5 In this section, we study some quantitative aspects of our
heap dump. graph mining approach to illustrate its effectiveness at min-
ing heap dumps. First, we study the indegree distribution
of nodes from 24 real (i.e., not synthetic) heap dumps. As-
This leak was subsequently fixed in a 6.x release of Tom- sessing the % of nodes that have indegree ≤ 1 across these
cat. After updating the server, we periodically took heap dumps, we obtain statistics of a minimum of 84%, an av-
dumps via the jmat tool. We subjected these heap dumps, erage of 89%, and a maximum of 99.8% (from the MVEL
which contain no known leaks, to our analysis. Unsurpris- dumps). This suggests that assessing frequent subgraphs in
ingly, the subgraph anchored by HashMap.Entry rose to the the dominator tree should not cause significant loss of infor-
top, reflecting the ubiquitous use of hash maps. However, mation as compared to the original heap dump. At the same
path aggregation showed these hash maps were located in time, Table 1 illustrates the reduction gained in the number
very different regions of the object graph, thus making it of edges and the overall size of the graph by choosing to
less likely for them to be leaks. focus on the dominator tree.
Fig. 12 illustrates the time taken for diagnosing leaks as a
4.3.2 MVEL function of the size of the dominator. This time does not in-
clude loading and pre-processing (removal of weak and soft
In a separate project, one of the authors developed a rule-
references) and computation of the dominator. It does in-
based system for the application of software engineering pat-
clude the time to mine the top (best) graph grammar, for
terns to enhance code [29]. During the development of this
graph reduction, and for summarizing root paths. The lower
project, out of memory situations occurred when certain in-
cluster of points are drawn from the Tomcat/J2EE dumps.
put was fed to the rule engine, which was written using
These are processed faster because they do not involve re-
the Drools rule engine framework. In this system, rules can
cursive constructs and there are fewer instances of the mined
contain expressions written in the MVEL scripting language
grammar in the dump. We see that these are processed in
(mvel.codehaus.org).
Mining the resulting heap dump showed the grammar in
Figure 11, which contains a recursive production RegExM atch 1200
→nextAST N ode RegExM atch. This mined grammar mirrors MVEL dumps
the synthetic linked list discussed in Section 4.2. All RegEx- 1000
Runtime (seconds)

Match objects are contained in a single list held in a local

variable: 800
+-+-> Java_Local
| (??)
+-+-> org.mvel.ASTLinkedList 600

| (firstASTNode)
{ 400
+-+-> org.mvel.ast.RegExMatch
| (nextASTNode)
200
+-+-> org.mvel.ast.RegExMatch
}*
0
This path indicates that the cause of the memory exhaus- 0 500 1000 1500 2000 2500 3000 3500 4000

tion was the unbounded growth of a singly-linked list of

Size (D) = # nodes + # edges (in thousands)
RegExMatch objects, likely due to a bug in the MVEL parser.
Although this information does not directly lead to the un-
derlying bug, it rules out a number of other scenarios, such Figure 12: Runtime statistics on real heap dumps.
Table 1: Summary statistics of some of the heap dumps analyzed in this work. Each heap dump is named by
the type of leak it contains and the date it was created. Dumps labeled as ‘maintenance’ were taken before
the production server was shut down for maintenance; they appear to be healthy and leak-free.

Name Memory # nodes # edges # edges % edge % size Grammar

(MB) (G) (D) reduction reduction size (avg)
tomcat.jul03 39 713076 1243152 536628 57% 36% 10.3
tomcat.jul05 41 732089 1288883 555829 57% 36% 7.0
tomcat.jun02 33 603941 1035000 440201 57% 36% 11.7
tomcat.jun04 33 588559 974322 427347 56% 35% 9.7
tomcat.may02 41 745434 1442146 561377 61% 40% 31.0
tomcat.may15 40 788482 1545443 609702 61% 40% 31.0
tomcat.sep20 40 734027 1295724 561956 57% 36% 10.3
tomcat.oct20 36 655251 1133970 487573 57% 36% 14.3
tomcat.nov05 43 742729 1336582 559642 58% 37% 24.3
tomcat.oct24 41 707913 1218834 533087 56% 36% 11.3
tomcat.nov03 42 715032 1202226 522893 57% 35% 23.7
tomcat.nov06 42 710730 1250427 535371 57% 36% 25.7
tomcat.oct28 42 717464 1260223 540283 57% 36% 9.0
tomcat.oct06 38 700479 1212796 530758 56% 36% 10.3
tomcat.oct03 47 854365 1584715 674692 57% 37% 11.0
tomcat.oct14 40 722417 1278437 550516 57% 36% 10.3
maintenance.feb06 34 321786 464017 260367 44% 26% 15.0
maintenance.nov08 35 519435 635638 348284 45% 25% 11.0
maintenance.nov17 35 547759 689958 374595 46% 25% 11.0
maintenance.nov09 34 493408 577236 322739 44% 24% 14.7
mvel.feb12 69 1704284 3832262 1698297 56% 39% 15.0
mvel.feb13 69 1704286 3832264 1698296 56% 39% 13.0
mvel.feb14 69 1704810 3832897 1698405 56% 39% 13.0
mvel.feb19 69 1707258 3837542 1701496 56% 39% 15.0

about 2–3 minutes. Conversely, the MVEL dumps involve graph for tomcat.sep20 versus the runtime on a dominator
significant recursion and several hundreds of thousands of tree from a similar heap dump, tomcat.nov05, because the
instances. Furthermore, the path summarization for the discovered graph grammar from the dominator tree in tom-
MVEL constructs require greater work since we must tra- cat.nov05 was closer in size to that found in the full heap
verse up the linked list to observe the root path for even a dump graph for tomcat.sep20 and composed of the same
single instance. leak – but was actually larger and more frequent. We found
Finally, we compared the runtime of our algorithm when that in the dominator tree the runtime was 130 seconds ver-
used on the dominator tree versus the original heap dump sus 426 seconds in the full heap dump graph. Our results
graph. We chose one particular dump, tomcat.sep20 from show that we obtain ∼ 46% runtime reduction for identify-
Table 1, to make the comparison. We used this dump be- ing small graph grammars and ∼ 69% runtime reduction for
cause the Tomcat leak graph grammar does not display sig- large graph grammars when the percentage of size reduction
nificant recursion and this dump is one of many good rep- is only 36%. This suggests that mining the dominator tree
resentatives of the leak class. In order to compare the run- is not only quicker due to size reduction, but also because its
times, we excluded the path summarization step that was tree structure contains less noise and redundancy, thereby
included in the runtime plot in Fig. 12 because this step simplifying the mining process.
would not be comparable between graphs. We note that the
preprocessing step of removing weak and soft references will
differ between graphs as well (in fact it is less complex in the 5. RELATED WORK
dominator tree), but the resulting graphs are comparable. Our research combines ideas from software engineering
Further, we found that the most frequent graph grammar and graph mining. We discuss related work in each of these
in the dominator tree is a subgraph of the most frequent areas.
graph grammar in the full heap dump graph, thus requir-
ing more iterations of candidate generation and therefore 5.1 Memory Leak Detection Tools
more runtime for discovery. To enable a fair runtime com- One of the first systems to debug leaks exploited visual-
parison, we considered the time required to generate the ization, allowing a user to interactively focus on suspected
most frequent single-edge graph grammar from the domi- problem areas in the heap [26]. Most recent existing leak de-
nator tree versus from the full graph. We found that the tection tools use temporal information, including object age
dominator tree accomplished this task in 21 seconds versus and staleness, that is obtained by monitoring a program as
38 seconds in the full graph. We also compared the com- it runs. For instance, IBM’s Leakbot [21–23] acquires snap-
plete runtime for the graph grammar in the full heap dump shots at multiple times during the execution of a program,
applies heuristics to identify leak candidates, and monitors tences” being generated are connected graphs. Graph gram-
how they evolve over time. mars have an array of applications, but have generally been
Minimizing both the space and runtime overhead of dy- researched from a theoretical perspective for graph genera-
namic analyses have been the subject of intense study. Space tion [10,28] as opposed to inference problems as studied here.
overhead is incurred because object allocation sites and last The benefit of graph grammars is that they can capture
access times must be recorded; runtime overhead because richer information about the connectivity of subgraphs than
this information must be continuously updated. Bell and traditional frequent subgraphs. Although these graph gram-
Sleigh [3] use a novel encoding to minimize space overhead mars are primarily context-free and therefore lossy, they pro-
for allocation sites. To minimize the runtime cost, statisti- vide a more descriptive representation of a subgraph than
cal profiling approaches have been developed [13]. Cork [16] just a frequency count. Our work follows the graph gram-
combines low-overhead statistic profiling with type-slicing. mar philosophy but applies it toward the characterization of
Some profilers, notably the NetBeans profiler, use informa- dominator trees, which has not been studied before.
tion already kept by generational collectors to determine ob-
ject age. Lastly, hardware support for monitoring memory 5.3 Data Mining for Software Engineering
access events has been proposed in [30]. Data from programming projects (code, bug reports, doc-
By contrast, our approach explores mining information umentation, runtime snapshots, heap dumps) is now so plen-
from a single heap dump, which is often the only source of tiful that data mining approaches have been investigated
information available when out-of-memory errors occur un- toward software engineering goals (see [31] for a survey).
expectedly, which is the common case in production environ- Graph data, in particular, resurfaces in many guises such
ments in which dynamic tools are rarely deployed. Our work as call graphs, dependencies across subprojects, and heap
is complementary to dynamic approaches. Mined structural dumps. Graph mining techniques have been used mini-
information is likely to enhance information these tools can mally for program diagnosis. For instance, program behav-
provide, especially in the common scenario in which software ior graphs have been mined for frequent closed subgraphs
engineers diagnose suspected leaks in codes with which they that become features in a classifier to predict the existence
are not familiar. Moreover, the ability to identify data struc- of “noncrashing” bugs [20]. Behavior graphs were also mined
tures could be exploited to automatically infer which oper- with the LEAP algorithm [34] in [5] to identify discrimina-
ations are add/delete operations on containers, which could tive subgraphs signifying bug signatures. However, to the
benefit approaches that rely on monitoring the membership best of our knowledge, nobody has investigated the role of
of object containers to identify leaks [33]. mining heap dumps for detecting memory leaks or used a
In the context of languages with explicit memory man- graph grammar mining tool.
agement, several static analyses have been developed that
identify where a programmer failed to deallocate memory [8,
14, 25, 32]. Similarly, trace-based tools such as Purify [12] or 6. CONCLUSION
Valgrind [24] can identify unreachable objects in such envi- We have presented a general and expressive framework
ronments. By comparison, the garbage collected languages for diagnosing memory leaks using frequent grammar min-
at which our analysis aims do not employ explicit dealloca- ing. Our work extends the arsenal of memory leak diagnosis
tion; we aim to identify reachable objects that are unlikely tools available to software developers. For the KDD commu-
to be accessed in the future. Lastly, rather than eliminat- nity, we have introduced the notion of dominators and how
ing the source of leaks, some systems implement mitigation they possess sufficient statistics for mining certain types of
strategies such as swapping objects to disk [4]. frequent subgraphs. The experimental results are promising
in their potential to debug leaks when other state-of-the-
5.2 Graph Mining art tools cannot. We expect our algorithm to be used in
Graph mining is a well studied field that expands far in complement of tools like Eclipse MAT.
both breadth and depth. Initial works such as [18, 35] fo- Our future work revolves around three themes. First, we
cused on the problem of discovering frequent subgraphs and seek to embed our algorithm in a runtime infrastructure so
these algorithms have been expanded in several directions that it can track leaking subgraphs as they build up over
over the past decade [6, 7, 34, 36, 37]. Building upon FP- time. Second, we seek to investigate the theoretical proper-
tree data structures [11], fast data structures [1] have also ties of dominators and whether they can support a frequent
been developed. Similar to our work, graph mining algo- pattern growth [11] style of subgraph mining. This approach
rithms have been tailored toward specific application do- will allow us to process larger heap dumps than our current
mains where the structure of the desired subgrahs can be approach. Third, we plan to perform a quantitative evalua-
exploited in the discovery process. tion comparing the quality of our reports to existing tools.
Cook and Holder’s work [9] takes a different approach to Such quantitative comparisons require the definition of a
graph mining by finding a single, “best” subgraph as op- metric, which could be derived by approximating the num-
posed to all subgraphs frequent above some threshold. This ber of lines of code a user would have to investigate to verify
approach uses a scoring function based on the minimum de- the presence or absence of a bug, as proposed in [27].
scription length (MDL) principle and is therefore computa-
tionally more complex. This work has also been expanded
upon for the use of finding highly descriptive graph gram- Acknowledgements
mars [15, 17] which consider the recursive nature of the sub- This work is supported in part by US NSF grant CCF-
graphs and allow for variability in the edge and node labels. 0937133 and the Institute for Critical Technology and Ap-
Graph grammars are synonymous with other formal lan- plied Science (ICTAS) at Virginia Tech. Also, we would like
guage grammars, with the difference being that the “sen- to thank Jongsoo Park, the developer of the dominator tree
algorithm used in the Boost C++ libraries, for his ready finding dominators in a flowgraph. ACM Trans.
responses to our questions and comments. Program. Lang. Syst., 1(1):121–141, 1979.
[20] C. Liu, X. Yan, H. Yu, J. Han, and P. Yu. Mining
7. REFERENCES behavior graphs for ”backtrace” of noncrashing bugs.
In SDM ’05, pages 286–297.
[1] M. Akbar and R. Angryk. Frequent pattern-growth [21] N. Mitchell. The runtime structure of object
approach for document organization. In CIKM ’08, ownership. In D. Thomas, editor, ECOOP ’06, 2006.
pages 77–82, 2008.
[22] N. Mitchell and G. Sevitsky. LeakBot: An automated
[2] A. Bailey and G. Back. LibX–a Firefox extension for and lightweight tool for diagnosing memory leaks in
enhanced library access. Library Hi Tech, large java applications. In ECOOP ’03, 2003.
24(2):290–304, 2006.
[23] N. Mitchell and G. Sevitsky. The causes of bloat, the
[3] M. Bond and K. McKinley. Bell: bit-encoding online limits of health. In OOPSLA ’07, pages 245–260, 2007.
memory leak detection. In ASPLOS-XII ’06, pages
[24] N. Nethercote and J. Seward. Valgrind: a framework
61–72, 2006.
for heavyweight dynamic binary instrumentation. In
[4] M. Bond and K. McKinley. Tolerating memory leaks. PLDI ’07, pages 89–100, 2007.
In OOPSLA ’08, pages 109–126, 2008.
[25] M. Orlovich and R. Rugina. Memory leak analysis by
[5] H. Cheng, D. Lo, Y. Zhou, X. Wang, and X. Yan. contradiction. In Lecture Notes in Computer Science,
Identifying bug signatures using discriminative graph volume 4134, pages 405–424. Springer, 2006.
mining. In ISSTA ’09, pages 141–152, New York, NY,
[26] W. De Pauw and G. Sevitsky. Visualizing reference
USA, 2009.
patterns for solving memory leaks in java.
[6] H. Cheng, X. Yan, J. Han, and P. Yu. Direct Concurrency - Practice and Experience,
discriminative pattern mining for effective 12(14):1431–1454, 2000.
classification. In ICDE ’07, pages 169–178, 2008.
[27] M. Renieris and S.Reiss. Fault localization with
[7] C. Chent, X. Yan, F. Zhu, and J. Han. gApprox: nearest neighbor queries. In ASE ’03, pages 30–39,
Mining frequent approximate patterns from a massive 2003.
network. In ICDM ’07, pages 445–450, 2007.
[28] G. Rozenberg. Handbook of Graph Grammars and
[8] S. Cherem, L. Princehouse, and R. Rugina. Practical Computing by Graph Transformation, volume 1. 1997.
memory leak detection using guarded value-flow
[29] E. Tilevich and G. Back. Program, enhance thyself!
analysis. In PLDI ’07, pages 480–491, 2007.
demand-driven pattern-oriented program
[9] D. Cook and L. Holder. Substructure discovery using enhancement. In AOSD ’08, pages 13–24, April 2008.
minimum description length and background
[30] G. Venkataramani, B. Roemer, Y. Solihin, and
knowledge. JAIR, 1:231–255, 1994.
M. Prvulovic. Memtracker: Efficient and
[10] J. Engelfriet and G. Rozenberg. Graph grammars programmable support for memory access monitoring
based on node rewriting: an introduction to nlc graph and debugging. In HPCA ’07, pages 273–284, 2007.
grammars. In Graph grammars and their application
[31] T. Xie, S. Thummalapenta, D. Lo, and C. Liu. Data
to computer science: 4th Intl. Workshop, pages 12–23,
Mining for Software Engineering. IEEE Computer,
1991.
Vol. 42(8):35–42, Aug 2009.
[11] J. Han, J. Pei, and Y. Yin. Mining frequent patterns
[32] Y. Xie and A. Aiken. Context- and path-sensitive
without candidate generation. In SIGMOD ’00, pages
memory leak detection. In ESEC/FSE-13, pages
1–12, 2000.
115–125, 2005.
[12] R. Hastings and B. Joyce. Purify: A tool for detecting
[33] G. Xu and A. Rountev. Precise memory leak detection
memory leaks and access errors in c and c++
for Java software using container profiling. In ICSE
programs. In Winter USENIX Conference, pages
’08, pages 151–160, 2008.
125–138, 1992.
[34] X. Yan, H. Cheng, J. Han, and P. Yu. Mining
[13] M. Hauswirth and T. Chilimbi. Low-overhead memory
significant graph patterns by leap search. In SIGMOD
leak detection using adaptive statistical profiling. In
’08, pages 433–444, 2008.
ASPLOS-XI ’04, pages 156–164, 2004.
[35] X. Yan and J. Han. gSpan: graph-based substructure
[14] D. Heine and M. Lam. A practical flow-sensitive and
pattern mining. In ICDM ’02, pages 721–724, 2002.
context-sensitive C and C++ memory leak detector.
In PLDI ’03, pages 168–181, 2003. [36] X. Yan and J. Han. CloseGraph: mining closed
frequent graph patterns. In SIGKDD ’03, pages
[15] I. Jonyer, L. Holder, and D. Cook. MDL-based
286–295, 2003. 956784.
context-free graph grammar induction and
applications. IJAIT, 13(1):65–79, 2004. [37] Z. Zeng, J. Wang, L. Zhou, and G. Karypis.
Out-of-core coherent closed quasi-clique mining from
[16] M. Jump and K. McKinley. Cork: dynamic memory
large dense graph databases. ACM TODS, 32(2), 2007.
leak detection for garbage-collected languages. In
POPL ’07, pages 31–38, 2007.
[17] J. Kukluk, L. Holder, and D. Cook. Inference of node
and edge replacement graph grammars. In ICML
Grammar Induction Workshop ’07, 2007.
[18] M. Kuramochi and G. Karypis. Frequent subgraph
discovery. In ICDM ’01, pages 313–320, 2001.
[19] T. Lengauer and R. Tarjan. A fast algorithm for