Click javascript events not firing? dispatchEvent seems to not work

[skyhirider]

The javascript executor does not seem to launch the click event. But when I run the same javascript directly via dev tools it opens the new page correctly.

I would expect the "test" search results to show, but they do not.

I tried several approaches without success and started to wonder if its a bug or a limitation with HtmlUnit or the driver not correctly processing the javascript.

Not sure if I should open the issue in or in the webdriver repo, can re-open if it is not in the correct location.

package seleniumHtmlunit;

import org.openqa.selenium.JavascriptExecutor;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.htmlunit.HtmlUnitDriver;

import com.gargoylesoftware.htmlunit.BrowserVersion;
import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.javascript.SilentJavaScriptErrorListener;

public class Test {

	public static void main(String[] args) {

		WebDriver webDriver = new HtmlUnitDriver(BrowserVersion.FIREFOX, true) {
		    @Override
		    protected WebClient modifyWebClient(WebClient client) {
		        final WebClient webClient = super.modifyWebClient(client);
		        webClient.getOptions().setCssEnabled(false);
		        webClient.getOptions().setJavaScriptEnabled(true);
		        webClient.getOptions().setDownloadImages(false);
		        webClient.getOptions().setThrowExceptionOnScriptError(false);
		        webClient.setJavaScriptErrorListener(new SilentJavaScriptErrorListener());

		       return webClient;
		    }
		};

		duckDuckTest(webDriver);
		webDriver.close();
		
	}

	private static void duckDuckTest(WebDriver webDriver) {
		webDriver.get("https://duckduckgo.com/");
		JavascriptExecutor js = (JavascriptExecutor) webDriver;
		String script = "var search = document.evaluate(\"//input[@id='search_form_input_homepage']\", document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue;\r\n"
				+ "search.value=\"test\";\r\n"
				+ "\r\n"
				+ "const event = new MouseEvent('click', {\r\n"
				+ "    view: window,\r\n"
				+ "    bubbles: true,\r\n"
				+ "    cancelable: true\r\n"
				+ "  });\r\n"
				+ "var cb = document.evaluate(\"//input[@id='search_button_homepage']\", document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue;\r\n"
				+ "\r\n"
				+ "cb.dispatchEvent(event)";
		js.executeScript(script);

		
	}

}

This is the javascript:

var search = document.evaluate("//input[@id='search_form_input_homepage']", document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue;
search.value="test";

const event = new MouseEvent('click', {
    view: window,
    bubbles: true,
    cancelable: true
  });
var cb = document.evaluate("//input[@id='search_button_homepage']", document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue;

cb.dispatchEvent(event)

You can wait for a condition such as

(new WebDriverWait(webDriver,30)).until(ExpectedConditions.presenceOfElementLocated(By.xpath("//span[normalize-space()='The name test is reserved by the Internet Engineering Task Force in RFC 2606 as a domain name that is not intended to be installed as a top-level domain in the global Domain Name System of the Internet for production use.']")));

and it fails. It never reaches the next page, clicking the search button by firing off the click event.

[rbri]

HtmlUnit works like expected

public static void main(String[] args) throws Exception {
    String url = "https://duckduckgo.com/";

    // no special setup required for this sample
    WebClient webClient = new WebClient(BrowserVersion.FIREFOX);

    // open DuckDuckGo homepage
    HtmlPage page = webClient.getPage(url);

    // dump page content
    // System.out.println("#### ---- ");
    // System.out.println(page.asText());
    // System.out.println("#### ---- ");

    // fill the search field
    HtmlTextInput search = (HtmlTextInput) page.getElementById("search_form_input_homepage");
    search.type("HtmlUnit");

    // press the search button
    page.getElementById("search_button_homepage").click();

    // wait a bit for the js to do the async work
    webClient.waitForBackgroundJavaScript(4_000);

    // the window content (the whole page) gets replaced by js (maybe by setting the location)
    // we have to update our page reference to point to the new page (content)
    page = (HtmlPage) page.getEnclosingWindow().getEnclosedPage();

    // done
    System.out.println("#### ---- ");
    System.out.println(page.asText());
    System.out.println("#### ---- ");
}

Sorry i'm a bit short of time today - you have to do the same with the web driver. From my point of view there is no need to do sophisticated js magic to trigger events.

The key point here (as in many other cases) is the replacement of the page by another one - and your code has to get the current page content from the window.

Please try with HtmlUnit Driver and give me a sign if you need more support.

[skyhirider]

Thanks for the quick reply. I resorted to this javascript workaround as the page I am accessing, which I can not share, is not responding to Selenium clicks and I need to fire an event to make it work.

This has worked when sideloading a javascript via a proxy and triggering it in Firefox, but could not get it running via HtmlUnit and its web driver.

Do you see a way to make the javascript work, either in HtmlUnit or via its HtmlUnit driver and the selenium js executor?

I will try accessing the webclient directly, but I am afraid that might not work the same way a selenium click does not work. Plus it exposes the low level stuff where ideally one should only need to make calls via the WebDriver to be able to switch browsers later and keep the lower level domain of the browser hidden.

Is there a limitation in the javascript implementation in Rhino that it can't fire dispatchEvent correctly?

[rbri]

Ok, think a found at least one reason for this - will require some days to fix because i fear i have to write a bunch of tests for all our event implementations.

[rbri]

Hi @skyhirider, made the first commits for this problem. If the build is green, i will build a new snapshot. Please check at twitter.

Some background: The problem here is the line

event = new MouseEvent('click', 

The first parameter was ignored and as a result the event dispatcher has not forced the click handling.
So far i have done a bunch of fixes for the MouseEvent impl but i have to fix all the other event classes also.

Will be great if you can test the new snapshot (as soon as it is available :-) and please provide feedback.

[skyhirider]

I tried including htmlunit 2.45.0-SNAPSHOT and the snapshot driver 2.45 but my duckduckgo test fails still.

Took me a few minutes to set up as the instructions on your sourceforge page (https://htmlunit.sourceforge.io/) reference which according to what I was able to quickly find is used for building, not when downloading dependencies? Or can it be used for getting snapshots as well?

I used this to set up a repository so that I could get the library.

	<repositories>
		<repository>
			<id>sonatype-nexus-snapshots</id>
			<url>https://oss.sonatype.org/content/repositories/snapshots</url>
			<releases>
				<enabled>false</enabled>
			</releases>
			<snapshots>
				<enabled>true</enabled>
			</snapshots>
		</repository>
	</repositories>

[rbri]

Strange, this works here

public static void main(String[] args) {

    WebDriver webDriver = new HtmlUnitDriver(BrowserVersion.FIREFOX, true) {
        @Override
        protected WebClient modifyWebClient(WebClient client) {
            final WebClient webClient = super.modifyWebClient(client);
            webClient.getOptions().setCssEnabled(false);
            webClient.getOptions().setJavaScriptEnabled(true);
            webClient.getOptions().setDownloadImages(false);
            webClient.getOptions().setThrowExceptionOnScriptError(false);
            webClient.setJavaScriptErrorListener(new SilentJavaScriptErrorListener());

           return webClient;
        }
    };

    duckDuckTest(webDriver);

    (new WebDriverWait(webDriver,30))
        .until(ExpectedConditions.presenceOfElementLocated(By.xpath("//*[contains(text(),'GUI-Less browser for Java programs')]")));

    System.out.println("++++++++++++++++++");
    System.out.println(webDriver.findElement(By.tagName("body")).getText());
    System.out.println("++++++++++++++++++");

    webDriver.close();

}

private static void duckDuckTest(WebDriver webDriver) {
    webDriver.get("https://duckduckgo.com/");
    JavascriptExecutor js = (JavascriptExecutor) webDriver;
    String script = "var search = document.evaluate(\"//input[@id='search_form_input_homepage']\", document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue;\r\n"
            + "search.value=\"HtmlUnit\";\r\n"
            + "\r\n"
            + "const event = new MouseEvent('click', {\r\n"
            + "    view: window,\r\n"
            + "    bubbles: true,\r\n"
            + "    cancelable: true\r\n"
            + "  });\r\n"
            + "var cb = document.evaluate(\"//input[@id='search_button_homepage']\", document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue;\r\n"
            + "\r\n"
            + "cb.dispatchEvent(event)";
    js.executeScript(script);
}

[skyhirider]

You are right. I was waiting for text that was not showing when running the test via htmlunit, specifically the wikipedia snippet.

You were fast in fixing this.

[rbri]

Great - a new release is planned for the weekend....

[skyhirider]

I am curious how you implemented this. The private test case is still failing, but when running the javascript directly via a browser it works. And I just finished writing this as you closed it, great timing :)

[rbri]

check for the changes in the MouseEvent class here d8ef295