getPage returns the same old page

[niemand23oa]

Hi there,

I have a middleware platform which has jobs scheduled to scrap data from a website by calling the same Java service which uses HtmlUnit (by passing different search parameters to the web page). Therefore, it might happen that multiple separate threads are used with HtmlUnit.
The problem is that method getPage returns the same old page no matter what I set in the code. I tried to use below configs:

client.getCache().clear();
client.getCache().setMaxSize(0);
client.getOptions().setCssEnabled(true);
client.getOptions().setJavaScriptEnabled(true);

No matter what I do, below code always returns the same page:

page = client.getPage(webPageURL);

I also tried the method refresh() but it just won't work.
Can you please tell me what I am missing?

Best regards,
n23

[rbri]

Hi @niemand23oa ,

thanks for using HtmlUnit and the report.

I' not sure i got your point - so let me please ask some questions

You are using the latest version?

Calling

    page1 = client.getPage(webPageURL1);
    page1.asXML();
    page2 = client.getPage(webPageURL2);
    page2.asXML();

produces different pages - right?

Can you please log the webPageURL before and page.asXML() directly after this statement to make sure you really asking for different URL's and getting the same result.

    page = client.getPage(webPageURL);

[niemand23oa]

Hi @rbri
Thanks for your answer. Yes, I am using the latest version. I am actually not using a single thread from which I try to get different pages. I have multiple threads running (each having a new instance of the WebClient) and each thread tries to get a different page.

For instance, 1 thread tries to get:
page = client.getPage("https://website.com/?a=1&b=2");

The second one tries to get:
page = client.getPage("https://website.com/?a=2&b=3");

The third one tries to get:
page = client.getPage("https://website.com/?a=4&b=5");

and so on..

But when checking the returned page in the logs, I notice that all the threads are trying to get page "https://website.com/?a=1&b=2"

And if I now start a new thread for let's say page "https://website.com/?a=6&b=7", it will still get page "https://website.com/?a=1&b=2"

Best regards,
n23

[rbri]

Ah, ok now i got your point - paging.

Using one client per thread is the right idea - from my point of view.
And i can't imagine why HtmlUnit should return the wrong pages. Maybe there is something with the way you are using the application. Maybe because of a missing cookie or something else you always get the first page.

To dig deeper i like to suggest to write a simple client that only processes this page after page. Another option might be to enable wire logging (or use a web proxy like charles) to understand, if the correct request is sent by HtmlUnit.

[niemand23oa]

Ok will do.
To give you a context for my implementation, at the moment we are using java.net.urlconnection which simply reads different pages and retrieves the source which is then parsed (by tokenizing based on some expected values in the page source).
We have a source DB table from which we form the page URL string. Multiple processes can be triggered in the same time which read the source DB and parse and scrape the pages. The problem is that at the end of the month, the website which we are now using for webscrapping will be decommissioned and we will have to use another one. As the new one is different and it requires to wait for background JS jobs to finish in order to get the page source, I chose to use HtmlUnit. Unfortunately the current implementation does not allow using a single main thread from which we can navigate to multiple pages and changing the current process would entail some effort.

[rbri]

Another option you can check

• use a real browser - start a private tab (to have no cookie history) open the first url
• start a second private tab (to have no cookie history) open the second url

check if you really get different pages

[niemand23oa]

I did the check. The pages are different on each tab.

Am now every time I run the Java service using different pages I get the same page, as if it's somehow saved in a cache. I will try to restart the dev application server so see if afterwards I get a different page (I think that this will happen).

UPDATE:
After the restart, I am still getting the same page.
I also cloned the Java service and used a different library (jsoup) and I am having the same issue. I always get the same page no matter which page parameters I pass. Simply unbelievable :-)

Maybe it's because it's not waiting for the background JS jobs to finish?

This is what I have:
WebClient client = new WebClient(BrowserVersion.CHROME);
client.getCookieManager().clearCookies();
client.getCache().clear();
//client.getWebConnection().getState().clearCookies();
client.getCache().setMaxSize(0);
client.getOptions().setCssEnabled(true);
client.getOptions().setJavaScriptEnabled(true);
client.setAjaxController(new com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController());
client.getOptions().setThrowExceptionOnFailingStatusCode(false);
client.getOptions().setThrowExceptionOnScriptError(false);
client.waitForBackgroundJavaScript(10000);
//client.addRequestHeader("Cache-Control", "no-cache, no-store, must-revalidate");
//client.addRequestHeader("Pragma", "no-cache");
//client.addRequestHeader("Expires", "0"); //

[rbri]

Is the page public - to have a chance to reproduce it from here?
If you like you can also send the page url via mail..

[niemand23oa]

Yes, the page is public.
This is the URL

[rbri]

This code produces different output for Berlin & Köln

public static void main(String[] args) throws Exception {
    String uriBerlin = "https://www.benzinpreis-blitz.de/?land=de&suchfeld=Berlin&entfernung=5&sorte=super-e5&sortierung=preis";
    String uriKoeln = "https://www.benzinpreis-blitz.de/?land=de&suchfeld=Koeln&entfernung=5&sorte=super-e5&sortierung=preis";

    try (final WebClient webClient = new WebClient(BrowserVersion.CHROME)) {
        webClient.getOptions().setThrowExceptionOnScriptError(false);

        HtmlPage page = webClient.getPage(uriBerlin);
        System.out.println("------------");
        System.out.println("------------");
        System.out.println(page.asText());
        System.out.println("------------");
        System.out.println("------------");

        page = webClient.getPage(uriKoeln);
        System.out.println("------------");
        System.out.println("------------");
        System.out.println(page.asText());
        System.out.println("------------");
        System.out.println("------------");
    }
}

[niemand23oa]

Thanks for looking into it.
I think I found the root cause. We recently had a proxy change and it seems that after sending the requests without proxy it works without any issues. The proxy was messing everything up.

[rbri]

Great, thanks for using HtmlUnit