htmlunit-user Mailing List for HtmlUnit

Hi,


  I am trying to fetch resumes from a portal using HTMLUnit. The portal
folks
  use akamai bot detection service. The bot detection service is injecting a
   JS file in the html page before sending the response.

   The JS asynchronously sends sensor data to Bot detection service through
   CDN server. The request to view the profile is blocked as a BOT.

    Has anyone encountered such a problem with HTMLUnit?
     Any pointers or suggestions to bypass this bot service would help me.

    Looking forward to your reply

     The src value of the injected JS file in the HTML Page is given below

https://resdex.naukri.com/Z7gjQ2/wzrdB/DgfpQ/WA/7f3DSLcmSO/HxNednQ/cysrTTQ/RXRMB

   The sensor data computed by the JS as a JSON payload is sent in POST
request to the  above URL asynchronousluy

Regards
 Vasu

Thank you,

I have been using Htmlunit for 7 years and it has helped me develop many
web crawlers.

Em qui., 8 de dez. de 2022 18:34, Vasudevan Comandur <vco...@gm...>
escreveu:

> Opened an issue in github.
>
> Thanks
>  Vasu
>
> On Thu, 8 Dec 2022 at 23:43, Ronald Brill <rb...@rb...> wrote:
>
>> Can you please open an github issue for this.
>>
>> Am 7. Dezember 2022 17:35:48 MEZ schrieb Vasudevan Comandur <
>> vco...@gm...>:
>>>
>>> Hi,
>>>    Wanted to know if HTMLUnit will return page object when host response
>>>    is 204. The page object was null.
>>>
>>>     Is it the intended behaviour?
>>>     I was getting null pointer exception when I try to get the status
>>> code from
>>>     webResponse.
>>>
>>>    Looking forward to you reply
>>>
>>> regards
>>>  Vasu
>>>
>>> _______________________________________________
> Htmlunit-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>

Opened an issue in github.

Thanks
 Vasu

On Thu, 8 Dec 2022 at 23:43, Ronald Brill <rb...@rb...> wrote:

> Can you please open an github issue for this.
>
> Am 7. Dezember 2022 17:35:48 MEZ schrieb Vasudevan Comandur <
> vco...@gm...>:
>>
>> Hi,
>>    Wanted to know if HTMLUnit will return page object when host response
>>    is 204. The page object was null.
>>
>>     Is it the intended behaviour?
>>     I was getting null pointer exception when I try to get the status
>> code from
>>     webResponse.
>>
>>    Looking forward to you reply
>>
>> regards
>>  Vasu
>>
>>

Can you please open an github issue for this. 

Am 7. Dezember 2022 17:35:48 MEZ schrieb Vasudevan Comandur <vco...@gm...>:
>Hi,
>   Wanted to know if HTMLUnit will return page object when host response
>   is 204. The page object was null.
>
>    Is it the intended behaviour?
>    I was getting null pointer exception when I try to get the status code
>from
>    webResponse.
>
>   Looking forward to you reply
>
>regards
> Vasu

Hi,
Sorry,
This one's return is now 404. Possibly the url id has expired.
I need to get the object return page.

You can use the url
HtmlPage page = webClient.getPage("https://sso.acesso.gov.br");

it will redirect to
https://sso.acesso.gov.br/login?client_id=contas.acesso.gov.br&authorization_id={ID}

I disabled my browser's cookie to show what htmlunit displays.

thank you very much



[image: image.png]


Em qua., 7 de dez. de 2022 às 16:36, Vasudevan Comandur <vco...@gm...>
escreveu:

> Hi,
>    Wanted to know if HTMLUnit will return page object when host response
>    is 204. The page object was null.
>
>     Is it the intended behaviour?
>     I was getting null pointer exception when I try to get the status code
> from
>     webResponse.
>
>    Looking forward to you reply
>
> regards
>  Vasu
>
> _______________________________________________
> Htmlunit-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>

Hi,
   Wanted to know if HTMLUnit will return page object when host response
   is 204. The page object was null.

    Is it the intended behaviour?
    I was getting null pointer exception when I try to get the status code
from
    webResponse.

   Looking forward to you reply

regards
 Vasu

Hello,

I am try get page with htmlunit (all versions htmlunit, all BrowserVersion)
and the site return problem with cookie. As if it were disabled.

sample code:

WebClient webClient = new WebClient(BrowserVersion.BEST_SUPPORTED);

webClient.getCookieManager().setCookiesEnabled(true);

HtmlPage page = webClient.getPage("
https://sso.acesso.gov.br/logn?client_id=consumidor.gov.br&authorization_id=1848bfae21b
");

System.out.println(page.asNormalizedText());

show message
"Foi detectado que os cookies do seu browser podem estar
desabilitados. Por favor habilite os cookies e tente novamente."

Does it have a solution?

Thank you

Please open an issue on gitlab and if possible provide some info how to reproduce.

Am 7. Juli 2022 23:43:55 MESZ schrieb Kamal <kam...@gm...>:
>Hi Team,
>
>When I try to use the htmlunit and get a html page, the contents of the
>table(especially one row) is displayed as *[object ProgressEvent] *rather
>than actual message, though  it works fine in the browser.
>
>Thanks,
>K.

Hi Team,

When I try to use the htmlunit and get a html page, the contents of the
table(especially one row) is displayed as *[object ProgressEvent] *rather
than actual message, though  it works fine in the browser.

Thanks,
K.

Hi Jochen,
great to hear that HtmlUnit is useful for you. 
In general it is better to open an issue on github because otherwise i might loose sometimes the focus for some topics.

I have used HtmlUnit (as foundation of Wetator (www.wetator.org) for years to test many different jsf applications, so there is some hope to get it working.

Not sure what really happened behind your redirecting popup, i guess tis is some kind of async waiting for the server. The get method is synchronous. If the page is later on replaced by another one (changing the location via js) you will not notive this even if your code waits some time. You java code still looks at the old one. The sokutin her is to use the window, this is still there but the page inside gets replaced. You can try something like page.getEnclosingWindow().getEnclosedPage() after some wait. This will give you access to the current page.
There are other option but maybe we can start with this.

Please open an issue an we will see how we can proceed here.

  RBRi

Am 4. Juli 2022 13:19:56 MESZ schrieb Jochen van Waasen <jo...@va...>:
>Hi
>
>I am using HTMLUnit and got everything I needed working so far.
>Now I am stuck with an old JSF application that uses some JavaScript with a redirect popup to load content.
>
>When I load that page the only result I get is the redirect popup (HTML content) but not the updated page (in the browser it has a different URL).
>
>I have this:
>webClient.options.isCssEnabled = true
>webClient.options.isJavaScriptEnabled = true
>webClient.isRedirectEnabled = true
>webClient.cookieManager.isCookiesEnabled = true
>webClient.ajaxController = NicelyResynchronizingAjaxController()
>
>var thePage : HtmlPage = webClient.get(“someInternalUrl/…/entryPoint.xhtml?targetId=someId”)
>
>The redirect is working (as mentioned I get the content of the redirect HTML popup).
>
>In the browser I end up with an URL like this:
>https://someInternalUrl/…/resultOverview.xhtml
>
>How can I verify that this resultOverview.xhtml is loaded (it is in a browser)?
>How can I tell HTMLunit to update the page with it?
>
>Best Regards
>Jochen
>
>
>
>_______________________________________________
>Htmlunit-user mailing list
>Htm...@li...
>https://lists.sourceforge.net/lists/listinfo/htmlunit-user

Hi

I am using HTMLUnit and got everything I needed working so far.
Now I am stuck with an old JSF application that uses some JavaScript with a redirect popup to load content.

When I load that page the only result I get is the redirect popup (HTML content) but not the updated page (in the browser it has a different URL).

I have this:
webClient.options.isCssEnabled = true
webClient.options.isJavaScriptEnabled = true
webClient.isRedirectEnabled = true
webClient.cookieManager.isCookiesEnabled = true
webClient.ajaxController = NicelyResynchronizingAjaxController()

var thePage : HtmlPage = webClient.get(“someInternalUrl/…/entryPoint.xhtml?targetId=someId”)

The redirect is working (as mentioned I get the content of the redirect HTML popup).

In the browser I end up with an URL like this:
https://someInternalUrl/…/resultOverview.xhtml

How can I verify that this resultOverview.xhtml is loaded (it is in a browser)?
How can I tell HTMLunit to update the page with it?

Best Regards
Jochen

Hi, could someone help how do I start selenium-server with htmlunit? Just 
whatever command/configuration I try, I always get some error. 
Thanks for any help,

Adam

Say, as part of a JFX Browser kind of application you use the Wekit's
WebView which internally uses its own javascript engine.

Is there a way to avail yourself of WebView's:

https://webkit.org/blog/27/javascript-and-dom-compatibility/

instead of htmlunit's Rhino-based javascript engine?

Basically, if a page is fully or partially javascript generated (most
if not all pages nowadays) you would get its full html stream, tidy it
and tinker with it before it is handled by the browser's rendering
engine. This is how it would go:

 1) page request
 2) WebView component fields it, deals with cookies, https
connections, login credentials and such things
 3) fully parses its text stream and before it starts downloading
pictures, videos, ...
 4) *** here comes the code I have in mind to parse and scrape the
complete text stream of the html page request ***
 5) then, as determined by the user, the "sanitized" output of (4)
will be passed onto the browser

I am thinking of using htmlunit for (4)

How could you do that? Any code examples?

Thank you,
lbrtchx

Hi,

   Has any one tried getting the auth token using HTMLUnit for
accessing GMail?
   Google is removing the option "less-secure app" setting to read Mailbox
and it is
   recommended to use Oauth for authentication & authorization.

   How will I be able to use HTMLUnit to successfully get the auth token.?

   Any pointers/suggestions are welcome.

Regards
 Vasu

Hi, 

I have a dynamic page which has some srollable content in div tag. 
Scrollable content consists of rows - also divs.

There are about 100 rows, but visible are about only 20 rows, cca last 4 of 
them are not visible, they are 'behind'. 

The rest of the rows IS NOT generated at all. 

When i move the vertical scrollbar, the rows set is regenerated - old are 
gone, new are visible. 

My problem is this - I cannot find a technique which gives me ALL the rows. 

I would be nice to control the vertical scrollbar (and collect all the newly
found rows). 

Or make the last generated row 'scrollIntoView' (become visible). It would 
generate a few new rows 

and I will repeat this until reaching the last row.   

I don't know if is possible to implement any of these ideas - controlling 
the scrollbar or scrolling content to view

in HtmlUnit. Do you see any solution? Thanks.




If needed I can give the page url.




JN

All,

  Where is the class JavaScriptPage in 2.52 version of HTMLUnit located?
  I could not find that. Kindly help me.

Regards
 Vasu

Hi Vasu, can you please have a look at this... https://howchoo.com/chrome/stop-chrome-from-automatically-redirecting-https

Am 18. August 2021 14:49:12 MESZ schrieb Vasudevan Comandur <vco...@gm...>:
>All,
>
>  I am using HTMLUnit and when accessing a site, the server responded with
>303
>  error and had location value
>(*http://hire.timesjobs.com/employer/quickSearch.html?f=f&sf=q
><http://hire.timesjobs.com/employer/quickSearch.html?f=f&sf=q>*) for
>redirection.
> HTMLUnit redirected to the location URL which again got relocated to the
>login
>  page  whereas actual browser redirected to the*
>(https://hire.timesjobs.com/employer/quickSearch.html?f=f&sf=q
><https://hire.timesjobs.com/employer/quickSearch.html?f=f&sf=q>).*
>
>*What is the reason for this different behaviour? Any pointer to solve*
>*this issue is highly appreciated.*
>
>* Coped Below the RESPONSE Header for your reference*
>
>:status 303
>server nginx
>date Wed, 18 Aug 2021 11:56:36 GMT
>content-type text/html; charset=UTF-8
>content-length 0
>pragma No-cache
>expires Thu, 01 Jan 1970 00:00:00 GMT
>cache-control no-cache
>cache-control no-store
>location http://hire.timesjobs.com/employer/quickSearch.html?f=f&sf=q
>content-language en-GB
>x-cool 29.37
>x-xss-protection 1; mode=block
>x-content-type-options nosniff
>x-permitted-cross-domain-policies master-only
>set-cookie mblid="3O7V9Pa8f9auKEBxDEtLXA=="; Version=1; Domain=.
>timesjobs.com; Path=/; Secure; SameSite=None
>x-frame-options sameorigin
>strict-transport-security max-age=25920000; includeSubdomains
>x-xss-protection 1; mode=block
>x-content-type-options nosniff
>
>
>*Regards*
>* Vasu*

-- 
Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.

And maybe similar for Firefox https://stackoverflow.com/questions/30532471/firefox-redirects-to-https#comment96908820_54559584

Am 18. August 2021 14:49:12 MESZ schrieb Vasudevan Comandur <vco...@gm...>:
>All,
>
>  I am using HTMLUnit and when accessing a site, the server responded with
>303
>  error and had location value
>(*http://hire.timesjobs.com/employer/quickSearch.html?f=f&sf=q
><http://hire.timesjobs.com/employer/quickSearch.html?f=f&sf=q>*) for
>redirection.
> HTMLUnit redirected to the location URL which again got relocated to the
>login
>  page  whereas actual browser redirected to the*
>(https://hire.timesjobs.com/employer/quickSearch.html?f=f&sf=q
><https://hire.timesjobs.com/employer/quickSearch.html?f=f&sf=q>).*
>
>*What is the reason for this different behaviour? Any pointer to solve*
>*this issue is highly appreciated.*
>
>* Coped Below the RESPONSE Header for your reference*
>
>:status 303
>server nginx
>date Wed, 18 Aug 2021 11:56:36 GMT
>content-type text/html; charset=UTF-8
>content-length 0
>pragma No-cache
>expires Thu, 01 Jan 1970 00:00:00 GMT
>cache-control no-cache
>cache-control no-store
>location http://hire.timesjobs.com/employer/quickSearch.html?f=f&sf=q
>content-language en-GB
>x-cool 29.37
>x-xss-protection 1; mode=block
>x-content-type-options nosniff
>x-permitted-cross-domain-policies master-only
>set-cookie mblid="3O7V9Pa8f9auKEBxDEtLXA=="; Version=1; Domain=.
>timesjobs.com; Path=/; Secure; SameSite=None
>x-frame-options sameorigin
>strict-transport-security max-age=25920000; includeSubdomains
>x-xss-protection 1; mode=block
>x-content-type-options nosniff
>
>
>*Regards*
>* Vasu*

-- 
Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.

All,

  I am using HTMLUnit and when accessing a site, the server responded with
303
  error and had location value
(*http://hire.timesjobs.com/employer/quickSearch.html?f=f&sf=q
<http://hire.timesjobs.com/employer/quickSearch.html?f=f&sf=q>*) for
redirection.
 HTMLUnit redirected to the location URL which again got relocated to the
login
  page  whereas actual browser redirected to the*
(https://hire.timesjobs.com/employer/quickSearch.html?f=f&sf=q
<https://hire.timesjobs.com/employer/quickSearch.html?f=f&sf=q>).*

*What is the reason for this different behaviour? Any pointer to solve*
*this issue is highly appreciated.*

* Coped Below the RESPONSE Header for your reference*

:status 303
server nginx
date Wed, 18 Aug 2021 11:56:36 GMT
content-type text/html; charset=UTF-8
content-length 0
pragma No-cache
expires Thu, 01 Jan 1970 00:00:00 GMT
cache-control no-cache
cache-control no-store
location http://hire.timesjobs.com/employer/quickSearch.html?f=f&sf=q
content-language en-GB
x-cool 29.37
x-xss-protection 1; mode=block
x-content-type-options nosniff
x-permitted-cross-domain-policies master-only
set-cookie mblid="3O7V9Pa8f9auKEBxDEtLXA=="; Version=1; Domain=.
timesjobs.com; Path=/; Secure; SameSite=None
x-frame-options sameorigin
strict-transport-security max-age=25920000; includeSubdomains
x-xss-protection 1; mode=block
x-content-type-options nosniff


*Regards*
* Vasu*

Hello I would like to know if I can use htmlunit in an android
project, what packages should I import?

<html><head></head><body><div style="font-family: Verdana;font-size: 12.0px;"><div>Good day,</div>

<div>&nbsp;</div>

<div>&nbsp;</div>

<div>I have used HtmlUnit to&nbsp;open a web page from Java, and then call a Javascript function which returns a Promise. I am able to get a handle to the returned Promise (htmlunit.javascript.host.Promise) which has a .then(onFulfilled: Function, onRejected: Function) method&nbsp;(htmlunit.corejs.javascript.Function).&nbsp;But creating such a Function&nbsp;requires many methods to be implemented, and i am not sure if this is the way to go.</div>

<div>&nbsp;</div>

<div>&nbsp;</div>

<div>
<pre>val page = webClient.getPage(&quot;...&quot;).asInstanceOf[HtmlPage]</pre>

<div>
<pre>val res = page.executeJavaScript(&quot;myFunction(...)&quot;).getJavaScriptResult.asInstanceOf[Promise]</pre>

<div>res.then(...)</div>

<div>&nbsp;</div>

<div>&nbsp;</div>

<div>I have also looked through HtmlUnit tests and it seems there the Promise tests are using a DOM element on the page to which the result is written. I wanted to see if there is a way for Java to use the Promise result without the intermediary DOM node.</div>

<div>&nbsp;</div>

<div>&nbsp;</div>

<div>Thank you,</div>

<div>Alex</div>
</div>
</div></div></body></html>

Hi all,

it is a pleasure to announce the availability of HtmlUnit 2.49.1.

The full list of changes can be found in [1]
Thanks to all the contributors.

Happy Testing/Scraping!
   The HtmlUnit team

[1] http://htmlunit.sourceforge.net/changes-report.html#a2.49.1

Hi Vasu,
can you please open an issue for this.

Am 19. März 2021 16:50:08 MEZ schrieb Vasudevan Comandur <vco...@gm...>:
>Hi,
>
>   I am building an application using HTML UNit.
>   The flow of the application is as follows:
>    1. Instantiate a WebClient.
>    2. Send the URL to the webserver.
>   3. Validate the response
>   4. Store the received cookies gotten from the CookieManager of the
>       webclient in my application buffer.
>   5. Close the webclient
>
>    Another request to the same site.
>    1. Instantiate a Webclient
>    2 Populate the cookies from my app buffer to the cookie Manager
>       using AddCookie method.
>   3. Send the URL to the webserver
>   4. Validate the response.
>   5. Printed the ResponseHeader and I saw the webserver changing the
>      Expires Date to one of the existing cookies in Set-Cookie header.
>  6. I was fetching the cookies from the CookieManager method and found
>   out that HTMLUnit was not updating the Expires Date for that cookie.
>It
>      was returning the old Expires Date Value
>
>As you can see that I am instantiating the Webclient on a need basis
>but
>setting the cookies from my application buffer. I was expecting
>HTMLUnit
>   to return the updated Expires Date Value for that cookie.
>   Am I doing something wrong?
>   Looking forward to your reply.
>
>Regards
> Vasu

-- 
Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.

Hi,

   I am building an application using HTML UNit.
   The flow of the application is as follows:
    1. Instantiate a WebClient.
    2. Send the URL to the webserver.
   3. Validate the response
   4. Store the received cookies gotten from the CookieManager of the
       webclient in my application buffer.
   5. Close the webclient

    Another request to the same site.
    1. Instantiate a Webclient
    2 Populate the cookies from my app buffer to the cookie Manager
       using AddCookie method.
   3. Send the URL to the webserver
   4. Validate the response.
   5. Printed the ResponseHeader and I saw the webserver changing the
       Expires Date to one of the existing cookies in Set-Cookie header.
   6. I was fetching the cookies from the CookieManager method and found
      out that HTMLUnit was not updating the Expires Date for that cookie.
It
      was returning the old Expires Date Value

   As you can see that I am instantiating the Webclient on a need basis but
   setting the cookies from my application buffer. I was expecting HTMLUnit
   to return the updated Expires Date Value for that cookie.
   Am I doing something wrong?
   Looking forward to your reply.

Regards
 Vasu

Hi all,

it is a pleasure to announce the availability of HtmlUnit 2.47.1.

The full list of changes can be found in [1]
Thanks to all the contributors.

Happy Testing/Scraping!
   The HtmlUnit team

[1] http://htmlunit.sourceforge.net/changes-report.html#a2.47.1