How to use test_network_events_response method in Playwright Python

With some small adjustments you can get it, lets try this:

from bs4 import BeautifulSoup as bs
from playwright.sync_api import sync_playwright
import time

url = 'https://franchisedisclosure.gov.au/Register'

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False, slow_mo=100)
    page = browser.new_page()
    page.goto(url)
    page.locator("text=I agree to the terms of use").click()
    page.locator("text=Continue").click()
    page.wait_for_load_state('domcontentloaded')
    names = []
    industry = []
    Locations = []
    # When you click to next page, an element with text "Loading" appears in the screen, so we save that element
    loading_icon = "//strong[text()='Loading...']"
    # This is the "next page" button
    next_page_locator = "//ul[@class='pagination']/li[3]"
    # We select the option of 50 elements per page
    page.select_option("#perPageCount", value="50")
    # We wait for the selector of loading icon to be visible and then to be hidden, which means the new list is fully loaded
    page.wait_for_selector(loading_icon, state="visible")
    page.wait_for_selector(loading_icon, state="hidden")
    time.sleep(1)
    # We make a loop until the button "Next page" is disabled, which means there are no more pages to paginate
    while "disabled" not in page.get_attribute(selector=next_page_locator, name="class"):
        # We get the info you wanted
        page.is_visible('tbody')
        html = page.inner_html('table.table.table-hover')
        soup = bs(html, 'html.parser')
        table = soup.find('tbody')
        rows = table.findAll('tr')
        for row in rows:
            info = row.findAll('td')
            name = info[0].text.strip()
            industry = info[1].text.strip()
            Locations = info[2].text.strip()
        # Once we get the info we click in next page and we wait for the loading element to be visible and then to be hidden.
        page.click(next_page_locator)
        page.wait_for_selector(loading_icon, state="visible")
        page.wait_for_selector(loading_icon, state="hidden")
        time.sleep(1)

I had a similar problem: I had PM2 run by an Apache Airflow Celery worker running a playwright task, hourly. The job hung.

I suspect it is the environment variables littered by Byobu and PM2. What worked for me was wrapping the Python script in a bash script and sanitizing the environment variable like the following:

#!/bin/bash
[ "$HOME" != "" ] && exec -c $0
# Your python script should come below:
/home/ubuntu/usr/venv-3.10-airflow/bin/airflow celery worker

The first line clears the env vars (as suggested in Sanitize environment with command or bash script?). Now you can run this script with pm2 start command. My worker ran fine for about 24 hours as of now. I hope this works for you too.

So I found a working solution.

What I did was to combine Javascript with python Playwright code.

I start the setInterval with a timer of 200ms to scroll down on the page with page.evaluate() and then I follow it up with a python loop that checks every second whether the total height of the page (scroll included) has changed. If it changes it continues to scroll and if it hasn't changed than the scroll is over.
This is what it looks like:

page.evaluate(
    """
    var intervalID = setInterval(function () {
        var scrollingElement = (document.scrollingElement || document.body);
        scrollingElement.scrollTop = scrollingElement.scrollHeight;
    }, 200);

    """
)
prev_height = None
while True:
    curr_height = page.evaluate('(window.innerHeight + window.scrollY)')
    if not prev_height:
        prev_height = curr_height
        time.sleep(1)
    elif prev_height == curr_height:
        page.evaluate('clearInterval(intervalID)')
        break
    else:
        prev_height = curr_height
        time.sleep(1)

EDIT

See the below answer using the new mouse.wheel(x, y) feature for an up to date way to scroll using playwright. Combine my answer with his to lessen the need to use JS.

This error means that python-playwright or selenium is not launching correctly. What I did is

1.Check the version of your chrome Right upper corner -> help -> About Google Chrome

2.Download the chromedriver from chromedriver

3.let your API find the driver

api = TikTokApi.get_instance(use_selenium=True, executablePath="/path/to/chromedriver")

Edit: Got the answer. The autothrottle was limited to default. Now I need to limit it to the number of website pages. The code looks like this CONCURRENT_REQUESTS = 3

Thanks for the question. I'll share what's possible today, and we would be happy to make changes if you could share your scenario as an issue on playwright-python.

playwright-python can work with environment variables like outlined in the linked doc. For example, you can set environment variable for PLAYWRIGHT_BROWSERS_PATH to specify a custom location for your browser binaries.

It is recommended to use Playwright against bundled browser binaries, since Playwright can often depend on browser-side changes that might not be available in older versions. These browser-side changes are done with the intent to improve execution reliability and breadth of Playwright capabilities.

In case you do want to use Playwright against a non-bundled browser binary, you can set the executablePath while launching a browser. For example, the script below launches Edge instead of the bundled Chromium binary.

Python 3.8.0 (default, Sep 21 2020, 12:25:40)
[Clang 12.0.0 (clang-1200.0.32.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from playwright import sync_playwright
>>> pw = sync_playwright().start()
>>> browser = pw.chromium.launch(headless=False, executablePath='/Applications/Microsoft Edge Dev.app/Contents/MacOS/Microsoft Edge Dev')

We do not provide browser specific packages like playwright-chromium for Python today, but we are happy to collect feedback on it and make changes.

You could use something called element_handle.is_visible() from here.

your code should look something like below.

href_element = page.query_selector("a")
href_element.is_visible()

You can ask for the class name of the element.

page.locator("div").evaluate("node => node.className")

Playwright does not support key filtering at the moment. But you can filter for the job.id which is part of the props:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://www.glassdoor.co.uk/Job/qa-engineer-jobs-SRCH_KO0,11.htm")
    page.locator("_react=q[job.id=1007630619432]").click()
    browser.close()