Infinite loop in net.sourceforge.htmlunit.corejs.javascript.Interpreter interpretLoop(Context cx, CallFrame frame, Object throwable)

[vpostrigan]

I am opening different pages with HtmlUnit 2.55.0 and notices that 1 page loads CPU on 100%
This page has javascript protection.
Problem not in that HtmlUnit can't open protection.
Problem in infinite loop and stuck. javascript timeout helps, but until it will happen, it will load 100 % CPU.
All browsers open this page fine (old and new).

For me it's clear bug in HtmlUnit js processing.
I tried to investigate it and fix, but I can't handle it.
Below details that I found

Code
`

    WebClient webClient = null;
    try {
        final long javascriptTimeout = TimeUnit.SECONDS.toMillis(60);
        webClient = new WebClient();
        webClient.setAjaxController(new NicelyResynchronizingAjaxController());
        webClient.getOptions().setDownloadImages(true);
        webClient.getOptions().setCssEnabled(true);
        webClient.getOptions().setJavaScriptEnabled(true);
        webClient.setJavaScriptTimeout(javascriptTimeout);

        String loginURL = "http://www.pbc.gov.cn/diaochatongjisi/116219/116319/index.html";
        System.out.println("Connecting to " + loginURL + " (" + webClient.getBrowserVersion() + ")");

        HtmlPage page = webClient.getPage(loginURL);
        // never get here (will get JS timeout after 60 seconds)

        System.out.print("    Waiting for Javascript to complete...");
        webClient.waitForBackgroundJavaScript(javascriptTimeout);
        System.out.println(page.asXml());
    } catch (Exception e) {
        e.printStackTrace();
    } finally {
        if (webClient != null)
            webClient.close();
    }

`
this page saved to file
1.txt

so when I debug
net.sourceforge.htmlunit.corejs.javascript.Interpreter
method: interpretLoop
it write some other codes (not like below) and after it, I see this infinite pattern all time (again and again), until timeout:

frame.pc: 67 op: -45
Icode_REG_STR1: _0x432180
frame.pc: 69 op: 49
frame.pc: 70 op: -45
Icode_REG_STR1: _0x230f38
frame.pc: 72 op: 39
frame.pc: 73 op: -45
Icode_REG_STR1: _0x18d5c9
frame.pc: 75 op: 39
frame.pc: 76 op: 36
frame.pc: 77 op: -45
Icode_REG_STR1: _0x432180
frame.pc: 79 op: 8
frame.pc: 80 op: -4
frame.pc: 81 op: -42
frame.pc: 82 op: 49
frame.pc: 83 op: -42
frame.pc: 84 op: 39
frame.pc: 85 op: -45
Icode_REG_STR1: ;
frame.pc: 87 op: 41
frame.pc: 88 op: -45
Icode_REG_STR1: _0x432180
frame.pc: 90 op: 39
frame.pc: 91 op: 21
frame.pc: 92 op: 21
frame.pc: 93 op: -42
frame.pc: 94 op: 8
frame.pc: 95 op: -4
frame.pc: 96 op: -45
Icode_REG_STR1: _0x2ab90b
frame.pc: 98 op: 49
frame.pc: 99 op: -45
Icode_REG_STR1: _0x230f38
frame.pc: 101 op: 39
frame.pc: 102 op: -45
Icode_REG_STR1: _0x432180
frame.pc: 104 op: 39
frame.pc: 105 op: 36
frame.pc: 106 op: -45
Icode_REG_STR1: _0x2ab90b
frame.pc: 108 op: 8
frame.pc: 109 op: -4
frame.pc: 110 op: -45
Icode_REG_STR1: _0x230f38
frame.pc: 112 op: 39
frame.pc: 113 op: -45
Icode_REG_STR1: push
frame.pc: 115 op: 41
frame.pc: 116 op: -17
frame.pc: 117 op: -45
Icode_REG_STR1: _0x2ab90b
frame.pc: 119 op: 39
frame.pc: 120 op: -33
frame.pc: 121 op: 38
frame.pc: 122 op: -4
frame.pc: 123 op: -45
Icode_REG_STR1: _0x333808
frame.pc: 125 op: 49
frame.pc: 126 op: -45
Icode_REG_STR1: _0x230f38
frame.pc: 128 op: 39
frame.pc: 129 op: -45
Icode_REG_STR1: length
frame.pc: 131 op: 41
frame.pc: 132 op: 36
frame.pc: 133 op: -45
Icode_REG_STR1: _0x333808
frame.pc: 135 op: 8
frame.pc: 136 op: -4
frame.pc: 137 op: -45
Icode_REG_STR1: _0x2ab90b
frame.pc: 139 op: 39
frame.pc: 140 op: -32
frame.pc: 141 op: -29
frame.pc: 142 op: 66
frame.pc: 143 op: 26
frame.pc: 144 op: 26
frame.pc: 145 op: 47
frame.pc: 146 op: 7
frame.pc: 149 op: -42
frame.pc: 150 op: 49
frame.pc: 151 op: -42
frame.pc: 152 op: 39
frame.pc: 153 op: -44
frame.pc: 154 op: 41
frame.pc: 155 op: -45
Icode_REG_STR1: _0x2ab90b
frame.pc: 157 op: 39
frame.pc: 158 op: 21
frame.pc: 159 op: 21
frame.pc: 160 op: -42
frame.pc: 161 op: 8
frame.pc: 162 op: -4
frame.pc: 163 op: -45
Icode_REG_STR1: _0x18d5c9
frame.pc: 165 op: -8
frame.pc: 167 op: -4
frame.pc: 168 op: -45
Icode_REG_STR1: _0x18d5c9
frame.pc: 170 op: 39
frame.pc: 171 op: -45
Icode_REG_STR1: _0x333808
frame.pc: 173 op: 39
frame.pc: 174 op: 14
frame.pc: 175 op: 6

[comschmid]

I experience a similar problem, after HtmlUnit version: 2.58.0 maxes out one core after the other.
Since I'm crawling thousands of pages, I don't know the exact reason and with which JavaScript construct it happens.

I created a Thread Dump and attached the nearly 4000 lines long stack of the suspected Thread as attachment:
endless-loop-thread.txt

The repeating part is the following:
net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:481)
com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.callSecured(HtmlUnitContextFactory.java:349)
com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:883)
com.gargoylesoftware.htmlunit.javascript.host.dom.MutationObserver$2.execute(MutationObserver.java:211)
com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.doProcessPostponedActions(JavaScriptEngine.java:1004)
com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:951)
net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:582)

I found 4 more 'RenderedContent-p3' treads with the same stack, so in total there are currently 5 threads in a loop. This explains why I see about 500% CPU on a 8 core machine.

When looking at the Heap Dump that was analyzed with the Eclipse Memory Analyzer, there are 26'267 objects of type net.sourceforge.htmlunit.corejs.javascript.InterpreterData. After String and byte[] the 3rd biggest number of objects.

Thanks for looking into!

[rbri]

Hi @comschmid,

without a reproducible sample it is really hard to find the root of that problem and from your provided data a can't see more. Maybe you can enhance your logging a bit to get an idea what url causes the problem?.

And btw Since I'm crawling thousands of pages github stars are welcome and sponsoring is even more welcome. I'm doing all this in my spare time having a > 40h/week job and with more sponsoring i'm able to work more on HtmlUnit.

[comschmid]

Thank for your answer @rbri !

The problem is that I don't know for what I should look for and log while crawling. After some time I just see an increase in CPU. But @vpostrigan seems to have the same problem, so I guess his HTML file could be a good starting point.

I could follow and check the HtmlUnit stack myself, but I thought this would be much easier for the creator of the library.

I could also simply switch from HtmlUnit to headless Firefox or Chrome that I already use in another project.
Further, I may ask my sponsor, for sponsoring, but I had the impression that HtmlUnit was already sponsored by (Gargoyle Software) that has a list of reputable companies they are working with...

[sengsational]

without a reproducible sample it is really hard to find the root of that problem and from your provided data a can't see more. Maybe you can enhance your logging a bit to get an idea what url causes the problem?.

I was trying to work on this, but I'm at a loss as to how configure the build such that I can step into net.sourceforge.htmlunit.corejs classes. This isn't the right place to ask, but maybe a pointer here to where to ask about specifics associated with build configuration which would use source for core js instead of the "jar".

[comschmid]

@sengsational This depends on your development environment. For example, with Eclipse you can set the Java Source Attachement to the referenced library (htmlunit-core-js-2.61.1.jar in this case). Then set a break point in your code, run it and step into the code (F5).

@rbri I tried to find a problematic URL by writing after each URL request its current CPU consumption. Unfortunately there is not a jump, but a pretty steady grow in the load average. E.g. about a 7-8 (700-800% of the java process) on a 8 core machine, after about 20'000 URLs.
Maybe my configuration can be improved?

	private static HtmlPage requestHtmlPage(final String url, final int timeout) throws FailingHttpStatusCodeException, MalformedURLException, IOException
	{
		try (final WebClient webClient = new WebClient(BrowserVersion.BEST_SUPPORTED))
		{
			final WebClientOptions options = webClient.getOptions();
			options.setHomePage(url);
			options.setTimeout(timeout);
			options.setThrowExceptionOnScriptError(false);
			options.setThrowExceptionOnFailingStatusCode(false);
			options.setActiveXNative(false);
			options.setUseInsecureSSL(true);
			options.setJavaScriptEnabled(true);
			options.setCssEnabled(false);
			options.setWebSocketEnabled(false);
			options.setPopupBlockerEnabled(true);
			options.setPrintContentOnFailingStatusCode(false);
			options.setDownloadImages(false);
			options.setDoNotTrackEnabled(true);
			options.setGeolocationEnabled(false);
			options.setRedirectEnabled(true);

			// No caching
			options.setHistoryPageCacheLimit(0);
			options.setHistorySizeLimit(0);
			options.setMaxInMemory(-1);

			// webClient.getCache().setMaxSize(0) will not return rendered content
			webClient.setJavaScriptTimeout(timeout);
			webClient.waitForBackgroundJavaScript(timeout);
			webClient.setAjaxController(new NicelyResynchronizingAjaxController());
			webClient.setCssErrorHandler(new SilentCssErrorHandler());
			webClient.setJavaScriptErrorListener(new SilentJavaScriptErrorListener());
			return (HtmlPage) webClient.getPage(url);
		}
		catch (final Throwable e)
		{
			LOG.error("Error requesting rendered content for " + url);
			throw e;
		}
	}

After I retrieved the interesting content, I'm cleaning-up the HtmlPage response:

		finally
		{
			if (response != null)
			{
				response.cleanUp();
			}
		}

[rbri]

Have tested it today with this code

public static void main(String[] args) throws Exception {
    try (final WebClient webClient = new WebClient(BrowserVersion.FIREFOX)) {
        webClient.getOptions().setUseInsecureSSL(true);

        webClient.getOptions().setThrowExceptionOnScriptError(false);
        webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);

        HtmlPage page = webClient.getPage(url);
        webClient.waitForBackgroundJavaScriptStartingBefore(10_000);

        System.out.println("-------------------------------");
        System.out.println("-------------------------------");
        System.out.println(page.asXml());
        System.out.println("-------------------------------");
        System.out.println("-------------------------------");
    }
}

Looks like it works with 3.0.0 - at least the output looks similar to the one i got in the browser.
Please reopen or open a new one if you still have problems.