How to use sha256 method in Playwright Python

Install these first

playwright

scrapy-playwright

Scrapy

And then run in the terminal

playwright install (This will install the driver required for playwright)

as the graph is being injected dynamically into the website via JavaScript, you have two ways of going about this task:

1. Emulate the script (unnecessary hard but fast)

2. WebScrape using a browser client (slower, but much easier) You can use Python libraries such as Selenium or Playwright (the second option being more modern) to handle this task. The documentation for playwright is pretty intuitive, and you can use css selectors in order to traverse the dom, which, I suppose, you already have knowledge of.

Let me know if that helps!

So with the help of Joran I was able to make this code work, tx!

I used his suggestion for using a future twice to get the data outside the main() function.

mainRespPromise = asyncio.Future()

async def main():
    async with async_playwright() as p:
        myRespPromise = asyncio.Future()
        async def handle_response(response): 
        # the endpoint we are insterested in 
            if ('eindpoint/name' in response.url): 
                json_data = await response.json()
                # "resolve the promise"
                myRespPromise.set_result(json_data)

        browser = await p.chromium.launch()
        page = await browser.new_page()
                
        # go to directly to searchpage
        await page.goto("website_url", wait_until='networkidle')
        
        page.on('response', handle_response)
        print("Made call, now await response...")

        await page.fill('input[id=zoeklocatie]', 'search_query')
        
        # Use two enters to first make button visible
        await page.keyboard.press("Enter")
        await page.keyboard.press("Enter")
        
        result_json = await myRespPromise
        print("GOT RESULT:",result_json)

        await page.wait_for_timeout(3000)
        
        await browser.close()
        mainRespPromise.set_result(result_json)

await main()

json_data = await mainRespPromise

The following is one way to get the images on that page - apparently only 398, not 600+ - it's even specified in the top breadcrumb - Trail Running Shoes(398)). Solution is based on Selenium, you're welcome to functionalize it, OOP it, whatever, I'm only giving you the actual method of obtaining the images.

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.keys import Keys
import time as t

chrome_options = Options()
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument('disable-notifications')
chrome_options.add_argument("--headless")
chrome_options.add_argument("window-size=1280,1080")

webdriver_service = Service("chromedriver/chromedriver") ## path to where you saved chromedriver binary
browser = webdriver.Chrome(service=webdriver_service, options=chrome_options)
wait = WebDriverWait(browser, 20)
actions = ActionChains(browser)


url = 'https://www.tradeinn.com/runnerinn/en/mens-shoes-trail-running-shoes/10005/s#fq=id_familia=10002&sort=v30_sum;desc@tm10;asc&fe=&pf=id_subfamilia=10005@&start=0'
browser.get(url)
pbody = wait.until(EC.presence_of_element_located((By.TAG_NAME, 'body')))
for x in range(14):
    pbody.send_keys(Keys.PAGE_DOWN)
    print('scrolled')
    t.sleep(1)
t.sleep(5)
images = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, 'img[class="imagen_buscador"]')))
print(len(images))
for i in images:
    print(i.get_attribute('src'))

Result in terminal:

scrolled
scrolled
scrolled
[...]
398
https://www.tradeinn.com/h/13855/138556150/nike-juniper-trail-running-shoes.jpg
https://www.tradeinn.com/h/13855/138552239/nike-react-pegasus-4-trail-running-shoes.jpg
https://www.tradeinn.com/h/13727/137276760/merrell-vapor-glove-3-shoes.jpg
https://www.tradeinn.com/h/13842/138429239/adidas-terrex-agravic-flow-2-goretex-trail-running-shoes.jpg
https://www.tradeinn.com/h/13855/138556425/nike-wildhorse-7-trail-running-shoes.jpg
https://www.tradeinn.com/h/13842/138429416/adidas-terrex-two-goretex-trail-running-shoes.jpg
https://www.tradeinn.com/h/13842/138429408/adidas-terrex-two-boa-trail-running-shoes.jpg
https://www.tradeinn.com/h/13855/138556423/nike-wildhorse-7-trail-running-shoes.jpg
https://www.tradeinn.com/h/13857/138574338/new-balance-410v7-all-terrain-trail-running-shoes.jpg
https://www.tradeinn.com/h/13857/138574792/new-balance-fresh-foam-x-hierro-v7-trail-running-shoes.jpg
https://www.tradeinn.com/h/13857/138574340/new-balance-410v7-all-terrain-trail-running-shoes.jpg
https://www.tradeinn.com/h/13789/137892027/adidas-terrex-two-boa-trail-running-shoes.jpg
https://www.tradeinn.com/h/13727/137276761/merrell-vapor-glove-3-shoes.jpg
https://www.tradeinn.com/h/13836/138368054/joma-trek-trail-running-shoes.jpg
https://www.tradeinn.com/h/13710/137107634/vibram-fivefingers-v-trail-2.0-trail-running-shoes.jpg
https://www.tradeinn.com/h/13789/137891690/adidas-terrex-agravic-flow-trail-running-shoes.jpg
https://www.tradeinn.com/h/13789/137892315/adidas-terrex-swift-r3-trail-running-shoes.jpg
https://www.tradeinn.com/h/13855/138552241/nike-react-pegasus-4-trail-running-shoes.jpg
https://www.tradeinn.com/h/13803/138030718/nike-wildhorse-7-trail-running-shoes.jpg
[..]

For Selenium docs, see https://www.selenium.dev/documentation/

With the node case, it's a bit different as you're passively subscribing to an event. In the python snippet, you are basically doing the equivalent of page.waitForResponse. Typically you'll do that in conjunction with some action that triggers the response (such as submitting a form).

Try using the python page.on API. Like this:

import asyncio
from playwright.async_api import async_playwright

def check_response(response):
    print(response.url)
    if 'getVerify' in response.url:
        print("Response URL: ", response.url)

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()
        page.on("response", check_response)
        await page.goto("http://playwright.dev")
        print(await page.title())
        await browser.close()

asyncio.run(main())

Until it's actually supported, a workaround is to use scripts in your Pipfile:

[scripts]
hooks = "playwright install"

And run pipenv install && pipenv run hooks.

Example with multiple hooks:

[scripts]
hooks = "bash -c 'pipenv run hook_1 && pipenv run hook_2'"
hook_1 = "some_command"
hook_2 = "another_command"

You can use a factory function here to pass the page, and the global_variable. Keep in mind that navigating away from the page from inside the callable will lead to an error. So make sure whatever you are clicking does not change the current URL of the page.

def wrapper(page, global_variable):
    def test(event):
        page.locator('//*[@id="n-currentevents"]/a').click()  # Action before boolean
        if event.url == 'https://en.wikipedia.org/':
            return True
    return test

Then, you can register the above function using page.wait_for_event like this:

page.wait_for_event('request', wrapper(page, global_variable))

Remember: You need to pass functions/callables (not their return values) to page.wait_for_event

It looks like you're trying to run python -m playwright install as python code rather than a shell command. Try running !python -m playwright install to signal to jupyter that this a shell command rather than python code. Use the ! prefix anytime you need to run a shell command on your environment in jupyter.

From what I understand, you have a page that has content within an iframe. You want to use Playwright to access elements within that frame.

The official docs to handle frames: Official docs: https://playwright.dev/docs/frames#frame-objects

You could then try something like this:

// Locate element inside frame
const iframeButton = await page.frameLocator('iFrame').locator("button:has-text(\"Add New User\")");
await iframeButton.click();

Notice that the example has the iFrame tag as locator, but you could and should use something more accurate like and id, name or url.