HtmlUnit returns UnexpectedPage when Content-Type header is missing

[antonireus]

We have a couple of system tests using Selenium/HtmlUnit in a legacy system using Java 7. Due to version restriction, in this tests I use HtmlUnit version 2.18 with Selenium 2.52.

Now we are migrating to Java 11, and we upgraded to HtmlUnit version 2.50 and Selenium 3.141.59, and we found an issue with one of the tests.

The problems seems to be that the response of one external service that we use doesn't contain the Content-Type header. Here is the log with the response headers:

DEBUG [pool-1-thread-1] (Wire.java 73) - http-outgoing-1 << "HTTP/1.1 200 OK[\r][\n]"
DEBUG [pool-1-thread-1] (Wire.java 73) - http-outgoing-1 << "Date: Fri, 11 Jun 2021 12:46:05 GMT[\r][\n]"
DEBUG [pool-1-thread-1] (Wire.java 73) - http-outgoing-1 << "Server: Apache[\r][\n]"
DEBUG [pool-1-thread-1] (Wire.java 73) - http-outgoing-1 << "X-XSS-Protection: 1; mode=block[\r][\n]"
DEBUG [pool-1-thread-1] (Wire.java 73) - http-outgoing-1 << "X-FRAME-OPTIONS: ALLOW-FROM http://intranet.pre.cdti.es[\r][\n]"
DEBUG [pool-1-thread-1] (Wire.java 73) - http-outgoing-1 << "Keep-Alive: timeout=3600, max=800[\r][\n]"
DEBUG [pool-1-thread-1] (Wire.java 73) - http-outgoing-1 << "Connection: Keep-Alive[\r][\n]"
DEBUG [pool-1-thread-1] (Wire.java 73) - http-outgoing-1 << "Transfer-Encoding: chunked[\r][\n]"

This leads to HtmlUnit creating an UnexpectedPage and unloading the DOM:

DEBUG [pool-1-thread-1] (HtmlPage.java 1254) - Firing Event unload (Current Target: com.gargoylesoftware.htmlunit.javascript.host.html.HTMLDocument@120eb694);
DEBUG [pool-1-thread-1] (WebWindowImpl.java 131) - setEnclosedPage: com.gargoylesoftware.htmlunit.UnexpectedPage@2bf01f1
DEBUG [pool-1-thread-1] (WebWindowImpl.java 213) - destroyChildren

It seems that with HtmlUnit 2.18, if Content-Type is missing, the default text/html is assumed. This is the same behaviour of the browsers I tested (Firefox, Chrome).

Is this an intended behaviour? Is there any workaround to assume text/html on missing Content-Type?

[antonireus]

Using a WebConntectionWrapper I could workaround the missing Content-Type header:

            new WebConnectionWrapper(webClient) {

                public WebResponse getResponse(WebRequest request) throws IOException {

                    WebResponse response = super.getResponse(request);

                    if (response.getStatusCode() == 200 && response.getContentType().isEmpty()) {

                        response = new WebResponseWrapper(response) {

                            @Override
                            public String getContentType() {
                                return "text/html";
                            }

                        };
                    }
                    return response;
                }
            };

[rbri]

Thanks for reporting this. HtmlUnit should work the same way as reals browsers, based on this i like to improve the impl. There is some kind of content sniffing if no content type found.
Can you please provide the page content (or at least the start sequence of the page your are facing this problem.

[rbri]

For details you can have a look at

htmlunit/src/main/java/com/gargoylesoftware/htmlunit/DefaultPageCreator.java

Line 154 in 73f3c4d

public static PageType determinePageType(final WebResponse webResponse) throws IOException {

[antonireus]

Hi, the page has nothing special. It's a valid html page, encoded in UTF-8.

I debugged the code and the problem is in the isBinary method in DefaultPageCreator

private static boolean isBinary(final byte[] bytes) {
    for (final byte b : bytes) {
        if (b < 0x08
            || b == 0x0B
            || (b >= 0x0E && b <= 0x1A)
            || (b >= 0x1C && b <= 0x1F)) {
            return true;
        }
    }
    return false;
}

The document I'm getting has the o-acute/ó letter in the <title> at the beginning of the document . This letter is encoded in UTF-8 with two bytes: 0xC3 0xB3

But, as the byte type is signed, the values in the array are -61 -77. So the first check in the if condition in the isBinary method: -61 < 0x08 evaluates to true. That check should be rewritten as:

`(b >= 0x00 && b < 0x08)`

[rbri]

Many thanks for all the details, will update the code.

[rbri]

Have written a test case for this, again many thanks for pointing this out.

[rbri]

As usual i will inform via twitter about new snapshots and releases

[antonireus]

Thanks to you for the quick update. Looking forward the next release!