How to use test_reload_should_work_with_data_url method in Playwright Python

Doing something similar for a project of mine, this is how I would do it.

from playwright.sync_api import sync_playwright

posts = ['https://playwright.dev/','https://playwright.dev/python/',]

def scrap_post_info(context, post):
    page = context.new_page()
    page.goto(post)
    print(page.title())
    # do whatever scraping you need to
    page.close()

with sync_playwright() as p:
    browser = p.chromium.launch()
    context = browser.new_context()
    for post in posts:
        scrap_post_info(context, post)
        # some time delay

browser.close()

Thing is the code snippet from the playwright docs is more about opening a new page after clicking a link on an existing page. Since you already have the urls ready, you can just visit each page one by one, and do your scraping.

There are two things to notice:

1. Python executable
2. Alternatives priority

Executable

Running in the base image:

# which python
# which python3
/usr/bin/python3
# ls -l /usr/bin/python*
lrwxrwxrwx 1 root root       9 Mar 13  2020 /usr/bin/python3 -> python3.8
-rwxr-xr-x 1 root root 5230592 Jun 22 20:18 /usr/bin/python3.8

With the following Dockerfile (from the question);

FROM mcr.microsoft.com/playwright
RUN apt update
RUN apt install -y software-properties-common
RUN add-apt-repository -y ppa:deadsnakes/ppa
RUN apt update
RUN apt install -y python3.10

RUN update-alternatives --install /usr/bin/python python /usr/bin/python3.10 1
RUN update-alternatives --install /usr/bin/python python /usr/bin/python3.8 2

python is now added and controlled by alternatives, but python3 is untouched:

# ls -l /usr/bin/python*
lrwxrwxrwx 1 root root      24 Dec 16 04:10 /usr/bin/python -> /etc/alternatives/python
lrwxrwxrwx 1 root root       9 Mar 13  2020 /usr/bin/python3 -> python3.8
-rwxr-xr-x 1 root root 5565184 Dec  7 01:12 /usr/bin/python3.10
-rwxr-xr-x 1 root root 5230592 Jun 22 20:18 /usr/bin/python3.8
# ls -l /etc/alternatives/python
lrwxrwxrwx 1 root root 18 Dec 16 04:10 /etc/alternatives/python -> /usr/bin/python3.8

Priority

From alternative's manpage:

If the group is in automatic mode, and the newly added alternatives' priority is higher than any other installed alternatives for this group, the symlinks will be updated to point to the newly added alternatives.

This is numeric, so it's not that 1 is the winner and 2 is second-best, but rather 2 is higher than 1.

Resolution

By changing both the executable form python to python3, and the priority so that 3.10 will be 2 and 3.8 will be 1 (as such):

FROM mcr.microsoft.com/playwright
RUN apt update
RUN apt install -y software-properties-common
RUN add-apt-repository -y ppa:deadsnakes/ppa
RUN apt update
RUN apt install -y python3.10

RUN update-alternatives --install /usr/bin/python3 python /usr/bin/python3.10 2
RUN update-alternatives --install /usr/bin/python3 python /usr/bin/python3.8 1

the default python version should be 3.10:

# which python
# which python3
# ls -l /usr/bin/python*
lrwxrwxrwx 1 root root      24 Dec 16 04:16 /usr/bin/python3 -> /etc/alternatives/python
-rwxr-xr-x 1 root root 5565184 Dec  7 01:12 /usr/bin/python3.10
-rwxr-xr-x 1 root root 5230592 Jun 22 20:18 /usr/bin/python3.8
# ls -l /etc/alternatives/python
lrwxrwxrwx 1 root root 19 Dec 16 04:16 /etc/alternatives/python -> /usr/bin/python3.10
# python3 --version
Python 3.10.9

This should get you started, because you don't really need to do fancy image trickery.

All reddit posts have data-testid=post-container selectors which let you get the top level post. All comments have a .Comment class you can use, so you can enumerate all the comments on any given reddit post and capture them separately if that's what you want.

Some major caveats of this code:

• It does not respect comment nesting - I only took a brief glance at Reddit's page source but they don't seem to have easy ways of getting comment chains.

• Reddit could change their selector names/CSS classes any time, and they do have some inconsistencies in how they seem to name things anyway.

• You specify you want to capture AskReddit threads, which can be quite large, so you could easily end up with, say, 1000+ comment screenshots on popular threads, or even 10K+ screenshots if you just let it run with abandon. I set a limit of 10, modify code as needed.

• I wrote this as a demonstration, I have no idea how it will perform on giant posts or such, but hopefully it shows you how to actually pull the information you want. If you need to do more work, you will likely have to inspect Reddit page layouts with your browser's developer tools.

#!/usr/bin/env python3

import asyncio
from playwright.async_api import async_playwright, Browser

MAX_COMMENT_LIMIT = 10


async def capture(browser: Browser,  url: str) -> None:
    page = await browser.new_page()

    await page.goto(url)
    posts = page.locator("data-testid=post-container")

    # reddit loads differently sometimes, so you need to
    # find the first post on the page
    await posts.first.screenshot(path="post.png")

    # this does not respect comment-nesting
    comments = page.locator(".Comment")
    count = await comments.count()
    print(f"Found {count} comments")
      
    for i in range(count):
        await comments.nth(i).screenshot(path=f"comment-{i}.png")

        if i+1 == MAX_COMMENT_LIMIT:
            print(f"Screened {MAX_COMMENT_LIMIT} images, stopping.")
            break


async def main() -> None:
    reddit_url = "https://www.reddit.com/r/cpp/comments/w9xkmf/c_for_scientific_programming/"
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        await capture(browser, reddit_url)
        await browser.close()

asyncio.run(main())

The thread I picked generated some images like this...

Top-level post text

A comment

Use the .start() method:

>>> from playwright.sync_api import Playwright, sync_playwright, expect
>>> playwright = sync_playwright().start()
>>> browser = playwright.chromium.launch(headless=False)
>>> page = browser.new_page()

Alternatively, if you just want an interactive browser, and don't care about an interactive shell, you can also use the wait_for_timeout function instead (only applicable on Page objects) and set the timeout to a high value:

from playwright.sync_api import sync_playwright

with sync_playwright() as playwright:
    browser = playwright.chromium.launch(headless=False)
    page = browser.new_page()
    page.wait_for_timeout(10000)

The best way to do it is using playwright docker image as base image, then you won't need to install its dependencies. Take a look to the documentation here: https://playwright.dev/docs/docker

There is no outer_html out of the box. But it's not hard to implement it:

locator.evaluate("el => el.outerHTML") 

Looks like soup is unable to find anything and its returning None and Nonetype has no text property

The error looks here for soup in clothes: as you have overridden the soup parser

PS:

You can use get_text() to extract the text

price = soup.find('span', {'class' : 'price'}).get_text()

Sometimes you may have to use contents depending on the return type

price = soup.find('span', {'class' : 'price'}).contents[0]

The async context manager doesn't know anything about the asynchronous generator. Nothing in main knows about the asynchronous generator after you break, in fact. You've given yourself no way to wait for the generator's finalization.

If you want to wait for the generator to close, you need to handle closure explicitly:

async def main():
    async with async_context():
        gen = async_gen()
        try:
            async for _ in gen:
                break
        finally:
            await gen.aclose()

In Python 3.10, you'll be able to use contextlib.aclosing instead of the try/finally:

async def main():
    async with async_context():
        gen = async_gen()
        async with contextlib.aclosing(gen):
            async for _ in gen:
                break

I had the same issue as well. I ran this command, then restarted my command line and it started working.

playwright install-deps

I hope this helps.