How to use test_click_offscreen_buttons method in Playwright Python

You can do:

hrefs_of_page = page.eval_on_selector_all("a[href^='/wiki/']", "elements => elements.map(element => element.href)")

which should work for your use-case. This will lookup for all the link tags which have a href attribute which starts with /wiki. Then on the browser side JavaScript gets evaluated which maps from an array of elements to the href attribute so a string array gets returned on the Python side.

You can scrape YouTube by:

• using requests-HTML, playwright or selenium libraries.
• using regular expression.
• using YouTube Search Engine Results API from SerpApi.

Code (it's really basic just to give an idea)

from requests_html import HTMLSession

session = HTMLSession()
url = "https://www.youtube.com/results?search_query=programming&sp=CAISBAgBEAE%253D"
response = session.get(url)
response.html.render(sleep=1, keep_page = True, scrolldown = 2)

for links in response.html.find('a#video-title'):
    link = next(iter(links.absolute_links))
    print(link)

Output:

https://www.youtube.com/watch?v=OUnxJk3Bphk
https://www.youtube.com/watch?v=vWvtt1ESNeY
https://www.youtube.com/watch?v=b8OIZu5y_Ak
https://www.youtube.com/watch?v=xp3fHaT2_VE
https://www.youtube.com/watch?v=e9toQAcjOrw
https://www.youtube.com/watch?v=em0Is0nyaXA
https://www.youtube.com/watch?v=N5JVTUAGmAM
https://www.youtube.com/watch?v=a0hQG-UdhYc
https://www.youtube.com/watch?v=SmQFxQ1fa2o
https://www.youtube.com/watch?v=uuMS1FYLgWQ
https://www.youtube.com/watch?v=8WJ-zSE32ZY
https://www.youtube.com/watch?v=c5MtH-xDspg
https://www.youtube.com/watch?v=5Xktqz6VUTU
https://www.youtube.com/watch?v=Wbo6j_iq2XY
https://www.youtube.com/watch?v=8eu9nliySO4
https://www.youtube.com/watch?v=j28PjOy_uk8
https://www.youtube.com/watch?v=fM2Ordt8Q9E
https://www.youtube.com/watch?v=tFSkaIVyNno
https://www.youtube.com/watch?v=1hDXlc2C3Rw
https://www.youtube.com/watch?v=vH9_Eo7VW3c

Using regex without a headless browser.

You need to reach the var ytInitialData element and then "commandMetadata" where you'll find a URL for the video {"url":"/https/test-backend.lambdatest.com/watch?v=Ae2TRkpjRCc",....

Here's a starting point where it grabs all inside var ytInitialData on regex101.

Alternatively, you can use YouTube Search Engine Results API from SerpApi. It's a paid API with a free plan. Check out the Playground.

Code to integrate:

from serpapi import GoogleSearch

params = {
  "engine": "youtube",
  "search_query": "programming",
  "sp": "CAISBAgBEAE%253D",
  "api_key": "your_secret_api_key"
}

search = GoogleSearch(params)
results = search.get_dict()

for link in results['video_results']:
    print(f"Title: {link['title']}\nLink: {link['link']}\n")

Output:

Title: CLASS VIII BASIC HTML TAGS AND PROGRAMMING 15 4 101`
Link: https://www.youtube.com/watch?v=KIPp63tXKpU

Title: For loop in c programming #bssdlectureclasses
Link: https://www.youtube.com/watch?v=nfRN0x9VvQc

Title: [C#] Programming NatsukiBot
Link: https://www.youtube.com/watch?v=chnigx-ezwg

Title: CS201 Short Lecture - 03 | VU Short Lecture | Introduction to Programming in (Urdu / Hindi)
Link: https://www.youtube.com/watch?v=qoxXJchd7N4

Title: Programming in C Language - While statement
Link: https://www.youtube.com/watch?v=cl0OpNCdF5I

Title: Introduction to html and Basic programming
Link: https://www.youtube.com/watch?v=A4We3NGqxuA

Title: Use of Printf & Scanf functions | Part 7 | C Programming | PadhoChalo
Link: https://www.youtube.com/watch?v=578xS-Ugc2c

Title: C++ course has started | Computer Programming | Aashu |
Link: https://www.youtube.com/watch?v=SjFgTK2HqbE

Title: Mitsubishi Outlander 2008 prox/twist transponder key programming tip
Link: https://www.youtube.com/watch?v=HlSJcBwxKFQ

Title: Computer Programming 1 -Introduction to the course
Link: https://www.youtube.com/watch?v=xdmPbhTT01g

Title: Programming, Data Structures and Algorithms in Python
Link: https://www.youtube.com/watch?v=0fUddu9cdAU

P.S - I wrote two blog posts about how to Scrape YouTube Search with Python (part 1) and Scrape YouTube Search with Python (part 2) that cover it more in-depth with visual representation.

Disclaimer, I work for SerpApi.

The misconception here seems to be that visible == clickable. That is not the case. In fact the docs explain what exact actionability checks are performed that determines whether the element is clickable

For example, for page.click(selector, **kwargs), Playwright will ensure that:

element is Attached to the DOM

element is Visible

element is Stable, as in not animating or completed animation

element Receives Events, as in not obscured by other elements

element is Enabled

Functionality to check whether all this is True for any element is currently not exposed in playwright (specifically, for checking whether the element is stable and reactive). The best you can do is change if locator.is_visible() to if locator.is_visible() and locator.is_enabled() to additionally check whether it is enabled as well (which would be done as part of the actionability checks later).

You can track the relevant issue here

afaik, playwright uses a virtual keyboard and does not exactly send the same signal as a real keyboard to the browser application.

consider going native and using something closer to kernel like pyautogui, after you've focused you window:

import pyautogui
pyautogui.press('f5')

I guess you are using codegen probably.

That page is a little bit strange for urls as I can see.

Anyways, I fixed your code:

from playwright.sync_api import Playwright, sync_playwright, expect


def run(playwright: Playwright) -> None:
    browser = playwright.chromium.launch(headless=False)
    context = browser.new_context()

    # Open new page
    page = context.new_page()

    # Go to https://www.elections.il.gov/CampaignDisclosure/ContributionSearchByCommittees.aspx?T=637994490317517425
    page.goto(
        "https://www.elections.il.gov/CampaignDisclosure/ContributionSearchByCommittees.aspx")

    # Fill input[name="ctl00\$ContentPlaceHolder1\$txtCmteID"]
    page.locator("input[name=\"ctl00\\$ContentPlaceHolder1\\$txtCmteID\"]").fill("34589")

    page.locator("input:has-text(\"Search\")").click()

    page.locator("text=Download This List").click()

    # Click text=CSV File
    with page.expect_download() as download_info:
        page.locator("text=CSV File").click()
    download = download_info.value
    download.save_as(download.suggested_filename)


    # ---------------------
    context.close()
    browser.close()


with sync_playwright() as playwright:
    run(playwright)

crawling_{201904307} is not a valid name for a function name in python. Method names can only contain alpha-numeric characters and underscores.

With some small adjustments you can get it, lets try this:

from bs4 import BeautifulSoup as bs
from playwright.sync_api import sync_playwright
import time

url = 'https://franchisedisclosure.gov.au/Register'

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False, slow_mo=100)
    page = browser.new_page()
    page.goto(url)
    page.locator("text=I agree to the terms of use").click()
    page.locator("text=Continue").click()
    page.wait_for_load_state('domcontentloaded')
    names = []
    industry = []
    Locations = []
    # When you click to next page, an element with text "Loading" appears in the screen, so we save that element
    loading_icon = "//strong[text()='Loading...']"
    # This is the "next page" button
    next_page_locator = "//ul[@class='pagination']/li[3]"
    # We select the option of 50 elements per page
    page.select_option("#perPageCount", value="50")
    # We wait for the selector of loading icon to be visible and then to be hidden, which means the new list is fully loaded
    page.wait_for_selector(loading_icon, state="visible")
    page.wait_for_selector(loading_icon, state="hidden")
    time.sleep(1)
    # We make a loop until the button "Next page" is disabled, which means there are no more pages to paginate
    while "disabled" not in page.get_attribute(selector=next_page_locator, name="class"):
        # We get the info you wanted
        page.is_visible('tbody')
        html = page.inner_html('table.table.table-hover')
        soup = bs(html, 'html.parser')
        table = soup.find('tbody')
        rows = table.findAll('tr')
        for row in rows:
            info = row.findAll('td')
            name = info[0].text.strip()
            industry = info[1].text.strip()
            Locations = info[2].text.strip()
        # Once we get the info we click in next page and we wait for the loading element to be visible and then to be hidden.
        page.click(next_page_locator)
        page.wait_for_selector(loading_icon, state="visible")
        page.wait_for_selector(loading_icon, state="hidden")
        time.sleep(1)

Playwright does not support key filtering at the moment. But you can filter for the job.id which is part of the props:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://www.glassdoor.co.uk/Job/qa-engineer-jobs-SRCH_KO0,11.htm")
    page.locator("_react=q[job.id=1007630619432]").click()
    browser.close()

You can ask for the class name of the element.

page.locator("div").evaluate("node => node.className")