How to use test_url_should_work method in Playwright Python

You can scrape YouTube by:

• using requests-HTML, playwright or selenium libraries.
• using regular expression.
• using YouTube Search Engine Results API from SerpApi.

Code (it's really basic just to give an idea)

from requests_html import HTMLSession

session = HTMLSession()
url = "https://www.youtube.com/results?search_query=programming&sp=CAISBAgBEAE%253D"
response = session.get(url)
response.html.render(sleep=1, keep_page = True, scrolldown = 2)

for links in response.html.find('a#video-title'):
    link = next(iter(links.absolute_links))
    print(link)

Output:

https://www.youtube.com/watch?v=OUnxJk3Bphk
https://www.youtube.com/watch?v=vWvtt1ESNeY
https://www.youtube.com/watch?v=b8OIZu5y_Ak
https://www.youtube.com/watch?v=xp3fHaT2_VE
https://www.youtube.com/watch?v=e9toQAcjOrw
https://www.youtube.com/watch?v=em0Is0nyaXA
https://www.youtube.com/watch?v=N5JVTUAGmAM
https://www.youtube.com/watch?v=a0hQG-UdhYc
https://www.youtube.com/watch?v=SmQFxQ1fa2o
https://www.youtube.com/watch?v=uuMS1FYLgWQ
https://www.youtube.com/watch?v=8WJ-zSE32ZY
https://www.youtube.com/watch?v=c5MtH-xDspg
https://www.youtube.com/watch?v=5Xktqz6VUTU
https://www.youtube.com/watch?v=Wbo6j_iq2XY
https://www.youtube.com/watch?v=8eu9nliySO4
https://www.youtube.com/watch?v=j28PjOy_uk8
https://www.youtube.com/watch?v=fM2Ordt8Q9E
https://www.youtube.com/watch?v=tFSkaIVyNno
https://www.youtube.com/watch?v=1hDXlc2C3Rw
https://www.youtube.com/watch?v=vH9_Eo7VW3c

Using regex without a headless browser.

You need to reach the var ytInitialData element and then "commandMetadata" where you'll find a URL for the video {"url":"/https/test-backend.lambdatest.com/watch?v=Ae2TRkpjRCc",....

Here's a starting point where it grabs all inside var ytInitialData on regex101.

Alternatively, you can use YouTube Search Engine Results API from SerpApi. It's a paid API with a free plan. Check out the Playground.

Code to integrate:

from serpapi import GoogleSearch

params = {
  "engine": "youtube",
  "search_query": "programming",
  "sp": "CAISBAgBEAE%253D",
  "api_key": "your_secret_api_key"
}

search = GoogleSearch(params)
results = search.get_dict()

for link in results['video_results']:
    print(f"Title: {link['title']}\nLink: {link['link']}\n")

Output:

Title: CLASS VIII BASIC HTML TAGS AND PROGRAMMING 15 4 101`
Link: https://www.youtube.com/watch?v=KIPp63tXKpU

Title: For loop in c programming #bssdlectureclasses
Link: https://www.youtube.com/watch?v=nfRN0x9VvQc

Title: [C#] Programming NatsukiBot
Link: https://www.youtube.com/watch?v=chnigx-ezwg

Title: CS201 Short Lecture - 03 | VU Short Lecture | Introduction to Programming in (Urdu / Hindi)
Link: https://www.youtube.com/watch?v=qoxXJchd7N4

Title: Programming in C Language - While statement
Link: https://www.youtube.com/watch?v=cl0OpNCdF5I

Title: Introduction to html and Basic programming
Link: https://www.youtube.com/watch?v=A4We3NGqxuA

Title: Use of Printf & Scanf functions | Part 7 | C Programming | PadhoChalo
Link: https://www.youtube.com/watch?v=578xS-Ugc2c

Title: C++ course has started | Computer Programming | Aashu |
Link: https://www.youtube.com/watch?v=SjFgTK2HqbE

Title: Mitsubishi Outlander 2008 prox/twist transponder key programming tip
Link: https://www.youtube.com/watch?v=HlSJcBwxKFQ

Title: Computer Programming 1 -Introduction to the course
Link: https://www.youtube.com/watch?v=xdmPbhTT01g

Title: Programming, Data Structures and Algorithms in Python
Link: https://www.youtube.com/watch?v=0fUddu9cdAU

P.S - I wrote two blog posts about how to Scrape YouTube Search with Python (part 1) and Scrape YouTube Search with Python (part 2) that cover it more in-depth with visual representation.

Disclaimer, I work for SerpApi.

I am not used to use async and I am not sure of your question, but I think this is what you want:

import asyncio
from playwright.async_api import async_playwright


async def main():
    async with async_playwright() as p:
        for browser_type in [p.chromium, p.firefox, p.webkit]:
            browser = await browser_type.launch(headless=False)
            page = await browser.new_page()


            # Now, we gonna expect a request, but that request we have to expect it after doing something, in my example is the navigation to google, in your example could be clicking a button or something else.
            async with page.expect_request("https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png") as first:
                await page.goto("https://google.com")
            first_request = await first.value
            print(f"The method of the request was: {first_request.method}")
            print(f"The url of the request was: {first_request.url}")
            print()
            print("And the headers:")
            # You can use this first_request.all_headers() as a Json
            print(await first_request.all_headers())

            await asyncio.sleep(5)

asyncio.run(main())

I did it with google, you should do it with your own page and knowing what should be the request url.

Here some doc: https://playwright.dev/python/docs/api/class-page#page-wait-for-request

You should be able to get that button by text:

page.locator('"Log in"').click()

You should install the module:

pip3 install greenlet

I just found the answer:

I need to set also the following and it works: browser.new_context(no_viewport=True)

Instead of:

page.once("dialog", lambda dialog: asyncio.create_task(dialog.accept()))

It should apparently be just:

page.once("dialog", lambda dialog: dialog.accept())

This was basically a lucky guess, so take it with a grain of salt. I can't find a good source to back it up.

At the time of writing, the documentation for the Python version of Playwright doesn't seem to mention page.once() at all.

In the original JavaScript version of Playwright, page.once() is documented as part of the standard Node.js EventEmitter class, from which Page is derived. Nothing like EventEmitter exists in the Python standard library, though.

I found that it's possible to use Page in context of exposed function, as long as:

1. You use async Playwright (as per @hardkoded suggestion)
2. Don't navigate away (as per @Charchit suggestion)

from playwright.async_api import  Page, async_playwright
import asyncio
from functools import partial
import time


async def handle_title(page: Page, title: str):
    # page = page_var.get()
    print(page)
    await page.locator("#searchInput").type(f"{title}")
    return "hello world"


async def main():
     async with async_playwright() as p:
        browser = await p.chromium.launch(headless=False)
        context = await browser.new_context(viewport={ 'width': 1920, 'height': 1024 })
        page = await context.new_page()
        await page.goto(f"https://wikipedia.org")
        await context.expose_function("handle_title", partial(handle_title, page))
        await page.evaluate("window.onload = () => handle_title(document.title)")
        while True:
            await asyncio.sleep(1)



asyncio.run(main())

I believe you want something like this:

for element in soup.find_all("a", {"class":"..."}):
    print(element['href'])

This should get you started, because you don't really need to do fancy image trickery.

All reddit posts have data-testid=post-container selectors which let you get the top level post. All comments have a .Comment class you can use, so you can enumerate all the comments on any given reddit post and capture them separately if that's what you want.

Some major caveats of this code:

• It does not respect comment nesting - I only took a brief glance at Reddit's page source but they don't seem to have easy ways of getting comment chains.

• Reddit could change their selector names/CSS classes any time, and they do have some inconsistencies in how they seem to name things anyway.

• You specify you want to capture AskReddit threads, which can be quite large, so you could easily end up with, say, 1000+ comment screenshots on popular threads, or even 10K+ screenshots if you just let it run with abandon. I set a limit of 10, modify code as needed.

• I wrote this as a demonstration, I have no idea how it will perform on giant posts or such, but hopefully it shows you how to actually pull the information you want. If you need to do more work, you will likely have to inspect Reddit page layouts with your browser's developer tools.

#!/usr/bin/env python3

import asyncio
from playwright.async_api import async_playwright, Browser

MAX_COMMENT_LIMIT = 10


async def capture(browser: Browser,  url: str) -> None:
    page = await browser.new_page()

    await page.goto(url)
    posts = page.locator("data-testid=post-container")

    # reddit loads differently sometimes, so you need to
    # find the first post on the page
    await posts.first.screenshot(path="post.png")

    # this does not respect comment-nesting
    comments = page.locator(".Comment")
    count = await comments.count()
    print(f"Found {count} comments")
      
    for i in range(count):
        await comments.nth(i).screenshot(path=f"comment-{i}.png")

        if i+1 == MAX_COMMENT_LIMIT:
            print(f"Screened {MAX_COMMENT_LIMIT} images, stopping.")
            break


async def main() -> None:
    reddit_url = "https://www.reddit.com/r/cpp/comments/w9xkmf/c_for_scientific_programming/"
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        await capture(browser, reddit_url)
        await browser.close()

asyncio.run(main())

The thread I picked generated some images like this...

Top-level post text

A comment