Webapp classloader leaks problem with RiakNode.Builder or RiakClient [JIRA: CLIENTS-962]

[monday0rsunday]

I'm using Riak java client (2.0.4) for our jax-rs services and I've encountered permgen OOM leak when reloading services.

I've created an example, just using this Listener (and uncomment cluster/client code) for any webapps, and seeing permgen space increase when reloading services (at least 2 times).

public class ContextListener implements ServletContextListener {
    public static RiakCluster cluster = null;
    public static RiakClient client = null;

    @Override
    public void contextInitialized(ServletContextEvent sce) {
        try {
            // RiakNode.Builder have perm gen leak when initialize any instance
            RiakNode.Builder builder = null;
            // builder = new RiakNode.Builder();

            List<String> addresses = new LinkedList<String>();
            addresses.addAll(Arrays.asList("anyip:port".split(",")));
            // or memleak when call buildNodes also
            List<RiakNode> nodes = null;
            // nodes = RiakNode.Builder.buildNodes(builder, addresses);
            // cluster = new RiakCluster.Builder(nodes).build();
            // cluster.start();

            // client = RiakClient.newClient();
        } catch (Exception e) {

        }
    }

    @Override
    public void contextDestroyed(ServletContextEvent sce) {
        if (cluster != null)
            cluster.shutdown();
        if (client !=null)
            client.shutdown();
    }

}

Thanks,
Cong Nguyen

[alexmoore]

Hi Cong,

What happens if you run await() on the Future<Boolean> that cluster.shutdown() and client.shutdown() return?

[alexmoore]

@monday0rsunday Also, do you have a minimum reproducing java project you could share to help speed up bug hunting?

[monday0rsunday]

@alexmoore see https://github.com/monday0rsunday/riak_perm

[alexmoore]

So after running the refresh a few times, I see some extra WebAppClassLoaders sitting on the Heap:

If we dig down and "Merge Shortest Paths to GC Roots", it shows something on the Jetty Scanner TimerThread, but no big red flags.

Do you know if you can reproduce this with a console app, or another webapp container?

[alexmoore]

@monday0rsunday What version of Java are you running?

[alexmoore]

@monday0rsunday Aha! I think I found our culprit. It looks like the object that gets stuck in memory is one io.netty.util.internal.IternalThreadLocalMap object, which is ThreadLocal. It looks like this is a known issue that the Netty team has no control over. I'll see if there's a workaround...

[monday0rsunday]

@alexmoore I'm not sure about version exactly, but there are two major versions I tested: 7 and 8.

[alexmoore]

@monday0rsunday One workaround is to add the following to your contextDestroyed(..) method after you call client.shutdown.get();:

io.netty.util.concurrent.FastThreadLocal.removeAll();
io.netty.util.concurrent.FastThreadLocal.destroy();

This seems to allow the most PermGen/Metaspace space to be GC'd (Java 7 running here):

I'll need to do some more investigation to see if there's any more permanent fixes/workarounds for this Netty ThreadLocal issue, or if there's anything else not getting collected.

[monday0rsunday]

@alexmoore Thank you, I'll try it and feed back to you.

[alexmoore]

@monday0rsunday So looking into it more today, if we:

1. Add those two lines to cleanup netty
2. Start server
3. Take Heap Dump Import trifork's PBC Client, Map/Reduce filter support #1
4. Touch the xml file to force a redeploy
5. Force GC via VisualVM
6. Take Heap Dump add file extension so the README renders nicely on GitHub #2
7. Compare the dumps

After the refresh + GC, there are no additional instances of any io.netty or com.basho.riak classes in dump 2, which means that everything in those two libs are getting GC'd over the refresh.

There are some new additional objects in dump 2 that I can't find homes for, but I think they are related to the Jetty reload:

Digging further into the big java.lang.ref.Finalizer difference, this seems related to the Jar file reloading, same with the byte[], String, etc differences. Those objects seem to get GC'd later on after everything is reloaded or finalized.

Please try adding those two lines to your web service contextDestroyed(...) method and let me know if you see any more errant behavior.