How to use test_Interception_should_work_with_request_interception method in Playwright Python

I had the same issue and it was because of an unclosed page instance, however, I see in your code page is closed.

Please make sure you did the following steps:

1. Close all playwright instances
2. Close all BrowserContext objects
3. Close all opened Page objects

Opened Playwright instance looks like this:

playwright_instance = await async_playwright().start()
browser = await playwright_instance.firefox.launch(headless=headless, proxy=PROXIES[0])
context = await browser.new_context(java_script_enabled=True)

Here is my solution:

await page.close()
await browser.close()
await playwright_instance.stop()

The best way to do it is using playwright docker image as base image, then you won't need to install its dependencies. Take a look to the documentation here: https://playwright.dev/docs/docker

This should get you started, because you don't really need to do fancy image trickery.

All reddit posts have data-testid=post-container selectors which let you get the top level post. All comments have a .Comment class you can use, so you can enumerate all the comments on any given reddit post and capture them separately if that's what you want.

Some major caveats of this code:

• It does not respect comment nesting - I only took a brief glance at Reddit's page source but they don't seem to have easy ways of getting comment chains.

• Reddit could change their selector names/CSS classes any time, and they do have some inconsistencies in how they seem to name things anyway.

• You specify you want to capture AskReddit threads, which can be quite large, so you could easily end up with, say, 1000+ comment screenshots on popular threads, or even 10K+ screenshots if you just let it run with abandon. I set a limit of 10, modify code as needed.

• I wrote this as a demonstration, I have no idea how it will perform on giant posts or such, but hopefully it shows you how to actually pull the information you want. If you need to do more work, you will likely have to inspect Reddit page layouts with your browser's developer tools.

#!/usr/bin/env python3

import asyncio
from playwright.async_api import async_playwright, Browser

MAX_COMMENT_LIMIT = 10


async def capture(browser: Browser,  url: str) -> None:
    page = await browser.new_page()

    await page.goto(url)
    posts = page.locator("data-testid=post-container")

    # reddit loads differently sometimes, so you need to
    # find the first post on the page
    await posts.first.screenshot(path="post.png")

    # this does not respect comment-nesting
    comments = page.locator(".Comment")
    count = await comments.count()
    print(f"Found {count} comments")
      
    for i in range(count):
        await comments.nth(i).screenshot(path=f"comment-{i}.png")

        if i+1 == MAX_COMMENT_LIMIT:
            print(f"Screened {MAX_COMMENT_LIMIT} images, stopping.")
            break


async def main() -> None:
    reddit_url = "https://www.reddit.com/r/cpp/comments/w9xkmf/c_for_scientific_programming/"
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        await capture(browser, reddit_url)
        await browser.close()

asyncio.run(main())

The thread I picked generated some images like this...

Top-level post text

A comment

Looks like soup is unable to find anything and its returning None and Nonetype has no text property

The error looks here for soup in clothes: as you have overridden the soup parser

PS:

You can use get_text() to extract the text

price = soup.find('span', {'class' : 'price'}).get_text()

Sometimes you may have to use contents depending on the return type

price = soup.find('span', {'class' : 'price'}).contents[0]

The only way I found is wrap the goto inside a try-except block

Content of main.py

from playwright.sync_api import Playwright, sync_playwright, expect

with sync_playwright() as p:
    #We define the browser, the context and the page
    browser = p.chromium.launch(headless=False)
    context = browser.new_context()
    page = browser.new_page()
    with page.expect_download() as download_info:
        #If we navigate without try, it will throw an exception and it will stop our script, so, we wrap it inside a try except block
        try:
            page.goto("https://manueltgomes.com/wp-content/uploads/2021/09/ParseHTMLLInks_1_0_0_1.zip")
        except:
            pass
    download = download_info.value
    download.save_as(f"./{download.suggested_filename}")
    context.close()
    browser.close()

Then just run python3 main.py

The file will be saved in the same path of the main.py file

Doing something similar for a project of mine, this is how I would do it.

from playwright.sync_api import sync_playwright

posts = ['https://playwright.dev/','https://playwright.dev/python/',]

def scrap_post_info(context, post):
    page = context.new_page()
    page.goto(post)
    print(page.title())
    # do whatever scraping you need to
    page.close()

with sync_playwright() as p:
    browser = p.chromium.launch()
    context = browser.new_context()
    for post in posts:
        scrap_post_info(context, post)
        # some time delay

browser.close()

Thing is the code snippet from the playwright docs is more about opening a new page after clicking a link on an existing page. Since you already have the urls ready, you can just visit each page one by one, and do your scraping.

I managed to make a code that closes only a specific tab!

all_pages = page.context.pages
await all_pages[1].close()

No, the webpage isn't loaded content dynamically by JavaScript rather it's entirely static HTML DOM

from bs4 import BeautifulSoup
import requests

page = requests.get('https://order.mandarake.co.jp/order/listPage/list?categoryCode=07&keyword=naruto&lang=en')
soup = BeautifulSoup(page.content,'lxml')

data = []
for e in soup.select('div.title'):

    d = {
        'title':e.a.get_text(strip=True),
        
        }
    
    data.append(d)

print(data)

Output:

[{'title': 'NARUTO THE ANIMATION CHRONICLE\u3000genga made for sale'}, {'title': 'Plex DPCF Haruno Sakura Reboru ring of the eyes'}, {'title': 'Naruto: Shippuden\u3000(replica)  ナルト'}, {'title': 'Naruto: Shippuden\u3000(replica)  ナルト'}, {'title': 'Naruto: Shippuden\u3000(replica)  NARUTO -ナルト-'}, {'title': 'Naruto: Shippuden ナルト\u3000(replica)'}, {'title': 'Naruto Shippuuden\u3000(replica) NARUTO -ナルト-'}, {'title': 'NARUTO -ナルト- 疾風伝\u3000(複製セル)'}, {'title': 'MegaHouse    ちみ メガ Petit Chara Land NARUTO SHIPPUDEN ナルト blast-of-wind intermediary   Even [swirl ナルト special is a volume on ばよ.  
  All 6 types set] inner bag not opened/box damaged'}, {'title': 'NARUTO -ナルト- 疾風伝\u3000(複製セル)'}, {'title': 'NARUTO -ナルト- 疾風伝\u3000(複製セル)'}, {'title': 'NARUTO -ナルト- 疾風伝'}, {'title': 'NARUTO -ナルト- 疾風伝'}, {'title': 'NARUTO -ナルト-'}]

I believe you need to call your go_to_url function using the same recipe:

asyncio.get_event_loop().run_until_complete(go_to_url())