Grabbing parameters from default requests

[lime-n]

scrapy_playwright tries to imitate the web-browser so it downloads all resources (images, scripts, stylesheets, etc). Therefore, given this information is downloaded, is it possible to grab the payload from specific requests sent from the network tab - specifically from the fetch/XHR tab?

For example (minimal reproducible code):

import scrapy
from scrapy_playwright.page import PageCoroutine

class testSpider(scrapy.Spider):
    name = 'test'

    def start_requests(self):
        yield scrapy.Request(
            url="http://quotes.toscrape.com/scroll",
            cookies={"foo": "bar", "asdf": "qwerty"},
            meta={
                "playwright": True,
                "playwright_page_coroutines": [
                    PageCoroutine("wait_for_selector", "div.quote"),
                    PageCoroutine("evaluate", "window.scrollBy(0, document.body.scrollHeight)"),
                    PageCoroutine("wait_for_selector", "div.quote:nth-child(11)"),  # 10 per page
                    PageCoroutine( "screenshot", path="scroll.png", full_page=True
                    ),
                ],
            },
        )
    def parse(self, response):
        pass

Produces the following output:

2022-02-21 11:50:39 [scrapy-playwright] DEBUG: [Context=default] Request: <GET http://quotes.toscrape.com/api/quotes?page=2> (resource type: xhr, referrer: http://quotes.toscrape.com/scroll)
2022-02-21 11:50:39 [scrapy-playwright] DEBUG: [Context=default] Request: <GET http://quotes.toscrape.com/api/quotes?page=3> (resource type: xhr, referrer: http://quotes.toscrape.com/scroll)
2022-02-21 11:50:39 [scrapy-playwright] DEBUG: [Context=default] Request: <GET http://quotes.toscrape.com/api/quotes?page=4> (resource type: xhr, referrer: http://quotes.toscrape.com/scroll)
...

Very similar to firefox which allows you to copy the url and url parameters. The aim is to store the url parameters into a list each time playwright downloads a specific request url. With a more complex website I would have to find specific requests urls and grab the url parameters.

Something like:

if response.meta['resource_type'] is 'xhr':
    print(parameters(response.meta['resource_type_urls'])) 

This is a pseduo-example to express what I want to get; parameters would be a function to grab the url parameters.

Or perhaps it works like this:

if response.meta['resource_type'] is 'xhr':
    print(response.meta['parameters']) 

However saving it into response.meta will likely overload the results if I have a large number of urls for resource types, and url parameters are fairly large dicts.

• I'm convinced this data is available as it's downloaded however I just do not know how to get it.

[lime-n]

For additional information, we have the following output from above -

 'playwright/context_count': 1,
 'playwright/page_count': 1,
 'playwright/page_count/closed': 1,
 'playwright/request_count': 15,
 'playwright/request_count/method/GET': 15,
 'playwright/request_count/navigation': 1,
 'playwright/request_count/resource_type/document': 1,
 'playwright/request_count/resource_type/font': 1,
 'playwright/request_count/resource_type/script': 1,
 'playwright/request_count/resource_type/stylesheet': 3,
 'playwright/request_count/resource_type/xhr': 9,

The output counts the resource_type, however I want to store or yield the specific resource types and their url parameters that are sent to the requests to imitate the web-browser.

How do I access these? I have tried looking at the source code but I'm marvelled at it's accomplishment.

[Edit]:

• It seems that I need request.resource_type as shown from the source code only when it's provided as xhr. Rather than incrementing the values, I need to grab this somehow whilst within the scraper as a lists of links instead. However, I do not see in the handler.py source code, where the url parameters are stored/sent.

Purpose:

1. More functionality when scraping sites especially javascript loaded sites with complex xhr to work with i.e. they have an id in the url parameter that loads on request and its unique. Therefore, if the request url is saved as well as the url parameters, this would optimise additional methods in integrating both playwright functionalities with scrapy.
2. It will prevent human interaction with the browser to collect information on the request urls when working with Json requests, and prevent interaction with the browser to collect payload. These can be completed via automation with scrapy playwright.

[elacuesta]

If I understand correctly, you should be able to get the info you want by attaching a page event handler for the request event. Note that the links in the readme seem to be broken, I'll try to update them soon.

[lime-n]

If I understand correctly, you should be able to get the info you want by attaching a page event handler for the request event. Note that the links in the readme seem to be broken, I'll try to update them soon.

Thank you for the response; I have managed to follow the links correctly. I have a question on the use of page event handler - I seem to want to grab the downloads which includes the urls from the resource types, and if also downloaded their url parameters. These should be available as mentioned in the docs. However I am not certain where the url parameters would be grabbed or how to get them.

I had thought these would be available via PageCoroutine - something like:

PageCoroutine('download', 'url')

However, there is no download available.

I have included the page_event_handler like-so:

'playwright_page_event_handlers':{
                    'Download':'handle_download'
                },

Here's the script I have using:

from playwright.async_api import Response as PlaywrightResponse
from playwright.async_api import Download
import scrapy
from scrapy_playwright.page import PageCoroutine
import logging

async def handle_download(download: Download) -> None:
    logging.info(f'Handled download message: {download.url}')
    await download.cancel
    return {'url':download.url}


class EventSpider(scrapy.Spider):
    name = "event"

    def start_requests(self):
        yield scrapy.Request(
            url="http://quotes.toscrape.com/scroll",
            cookies={"foo": "bar", "asdf": "qwerty"},
            meta=dict(
                playwright=True,
                playwright_page_coroutines = [
                    PageCoroutine("wait_for_selector", "div.quote"),
                    PageCoroutine("evaluate", "window.scrollBy(0, document.body.scrollHeight)"),
                    PageCoroutine("wait_for_selector", "div.quote:nth-child(11)"),  # 10 per page
                ],
                playwright_page_event_handlers={
                    "response": "handle_response",
                    "download": handle_download
                },
            ),
            callback = self.handle_response
        )

    async def handle_response(self, response: PlaywrightResponse) -> None:
        logging.info(f'received response {response.url}')
        yield {
            'dir':response.meta['playwright_page_event_handlers']
        }

[lime-n]

You might have to update the example script on the homepage for the playwright_page_event_handlers - when copying it as is there are quite a few errors which is misleading.

That said - I have updated my script using the example as a template however, how would I grab the downloads? It doesn't seem to make any reference to the first function. I couldn't get the example code to work neither and make reference to the Dialog given there were quite a few errors already.

Furthermore, I have also used the following response.request.resource_type and get the additional error:

AttributeError: 'Request' object has no attribute 'resource_type'

This does not make sense to me as the resource types were collected by the downloader. Please can you verify a working solution on how to grab the resource types and their url parameters.

[elacuesta]

The example in the Readme is a simplified version of the full script available in the examples directory.

download does not work as a PageCoroutine because there is no download coroutine in the Page class.

I think there is some confusion about what a "download" is, as the sample page that you're working with does not initiate any. In this context, downloads are things that get stored in your filesystem, for instance, files. http://quotes.toscrape.com/scroll only works with requests and responses.

Furthermore, in your last example you're using the handle_response method as both a Scrapy request callback and an event handler for the Playwright "response" event. That's the cause of the AttributeError you posted, because Scrapy requests do not have a resource_type attribute. Also, the code will break for Playwright responses because you're trying to access the meta attribute which is exclusive to Scrapy responses. It's better to use two separate methods for processing the two types of responses.

[lime-n]

That makes better sense, thank you for the clarification. I had not realised that page.on was not a coroutine when looking at the documents. Thanks for making this also clear to me.

I have successfully collected the urls as a result of your suggestion. Although, when you have the moment, where would the url parameters get stored? If the response stores the resource types, surely the url parameters must be stored somewhere also - as we get a response 200 for the resource type. For example, the quotes url should have the payload page:1 stored when it's sending requests from the XHR for the first page. I can grab it from the url in this case, but in other cases the payload is not appended to the request url.

My objective to want this information:

1. I can restrict the resource types as suggested by the following issue #26.
2. By sending requests/responses to only these resource types, I can then grab their url's using the event handlers
3. If possible - grab the url parameters sent for each response to the resource types

Why? I can prevent any human effort to grab payloads, and requests urls required especially when an API is involved. So that I can scrape multiple sites directly without having to collect any of this information (or even without entering the browser as everything is in the json)

Furthermore, when trying to grab the urls I have noticed not all the XHR's get a response (stops at page 3):

2022-02-27 10:14:02 [scrapy-playwright] DEBUG: [Context=default] Response: <200 http://quotes.toscrape.com/api/quotes?page=1> (referrer: None)
2022-02-27 10:14:02 [root] INFO: received response ('xhr', <Request url='http://quotes.toscrape.com/api/quotes?page=1' method='GET'>)
2022-02-27 10:14:02 [scrapy-playwright] DEBUG: [Context=default] Request: <GET http://quotes.toscrape.com/api/quotes?page=2> (resource type: xhr, referrer: http://quotes.toscrape.com/scroll)
2022-02-27 10:14:02 [scrapy-playwright] DEBUG: [Context=default] Request: <GET http://quotes.toscrape.com/api/quotes?page=3> (resource type: xhr, referrer: http://quotes.toscrape.com/scroll)
2022-02-27 10:14:02 [scrapy-playwright] DEBUG: [Context=default] Request: <GET http://quotes.toscrape.com/api/quotes?page=4> (resource type: xhr, referrer: http://quotes.toscrape.com/scroll)
2022-02-27 10:14:02 [scrapy-playwright] DEBUG: [Context=default] Request: <GET http://quotes.toscrape.com/api/quotes?page=5> (resource type: xhr, referrer: http://quotes.toscrape.com/scroll)
2022-02-27 10:14:02 [scrapy-playwright] DEBUG: [Context=default] Request: <GET http://quotes.toscrape.com/api/quotes?page=6> (resource type: xhr, referrer: http://quotes.toscrape.com/scroll)
2022-02-27 10:14:02 [scrapy-playwright] DEBUG: [Context=default] Request: <GET http://quotes.toscrape.com/api/quotes?page=7> (resource type: xhr, referrer: http://quotes.toscrape.com/scroll)
2022-02-27 10:14:02 [scrapy-playwright] DEBUG: [Context=default] Request: <GET http://quotes.toscrape.com/api/quotes?page=8> (resource type: xhr, referrer: http://quotes.toscrape.com/scroll)
2022-02-27 10:14:02 [scrapy-playwright] DEBUG: [Context=default] Request: <GET http://quotes.toscrape.com/api/quotes?page=9> (resource type: xhr, referrer: http://quotes.toscrape.com/scroll)
2022-02-27 10:14:02 [scrapy-playwright] DEBUG: [Context=default] Response: <200 http://quotes.toscrape.com/api/quotes?page=2> (referrer: None)
2022-02-27 10:14:02 [root] INFO: received response ('xhr', <Request url='http://quotes.toscrape.com/api/quotes?page=2' method='GET'>)
2022-02-27 10:14:02 [scrapy-playwright] DEBUG: [Context=default] Response: <200 http://quotes.toscrape.com/api/quotes?page=3> (referrer: None)
2022-02-27 10:14:02 [root] INFO: received response ('xhr', <Request url='http://quotes.toscrape.com/api/quotes?page=3' method='GET'>)

class EventSpider(Spider):
    name = "event"

    def start_requests(self):
        yield Request(
            url="http://quotes.toscrape.com/scroll",
            cookies={"foo": "bar", "asdf": "qwerty"},
            meta=dict(
                playwright=True,
                playwright_page_coroutines = [
                    PageCoroutine("wait_for_selector", "div.quote"),
                    PageCoroutine("evaluate", "window.scrollBy(0, document.body.scrollHeight)"),
                    PageCoroutine("wait_for_selector", "div.quote:nth-child(11)"),  # 10 per page
                ],
                playwright_page_event_handlers={
                    "response": "handle_response"
                },
            ),
        )

    async def handle_response(self, response: PlaywrightResponse) -> None:
        logging.info(f'received response {response.request.resource_type, response.request.url}')
        #return 

    def parse(self, response):
        return {"url": response.url}

[lime-n]

Issue solved.

I have uploaded a working script implementing the results discussed in such cases that others may want to know how to mostly automate with scrapy playwright, and limit human intervention on the web-browser.

script