How to use test_goto_should_navigate_to_about_blank method in Playwright Python

Not sure exactly if by "browser to show up" you refer at the fact that you need the browser to not run in headless mode.

There are two possible ways:

1. Install Playwright Test for VSCode as suggest by @MingJie-MSFT. You'll have a Show browser checkbox there in case you want to run your tests headless or not
2. In the config file, for use, add:

headless: false

Example for both options

Doing something similar for a project of mine, this is how I would do it.

from playwright.sync_api import sync_playwright

posts = ['https://playwright.dev/','https://playwright.dev/python/',]

def scrap_post_info(context, post):
    page = context.new_page()
    page.goto(post)
    print(page.title())
    # do whatever scraping you need to
    page.close()

with sync_playwright() as p:
    browser = p.chromium.launch()
    context = browser.new_context()
    for post in posts:
        scrap_post_info(context, post)
        # some time delay

browser.close()

Thing is the code snippet from the playwright docs is more about opening a new page after clicking a link on an existing page. Since you already have the urls ready, you can just visit each page one by one, and do your scraping.

The only way I found is wrap the goto inside a try-except block

Content of main.py

from playwright.sync_api import Playwright, sync_playwright, expect

with sync_playwright() as p:
    #We define the browser, the context and the page
    browser = p.chromium.launch(headless=False)
    context = browser.new_context()
    page = browser.new_page()
    with page.expect_download() as download_info:
        #If we navigate without try, it will throw an exception and it will stop our script, so, we wrap it inside a try except block
        try:
            page.goto("https://manueltgomes.com/wp-content/uploads/2021/09/ParseHTMLLInks_1_0_0_1.zip")
        except:
            pass
    download = download_info.value
    download.save_as(f"./{download.suggested_filename}")
    context.close()
    browser.close()

Then just run python3 main.py

The file will be saved in the same path of the main.py file

Yes, according to your link, all proxies are "dead"

Before using proxies try checking them here is one possible solution:

import json
import requests
from pythonping import ping
from concurrent.futures import ThreadPoolExecutor


check_proxies_url = "https://httpbin.org/ip"
good_proxy = set()

# proxy_lst = requests.get("https://www.proxy-list.download/api/v1/get", params={"type": "https"})
# proxies = [proxy for proxy in proxy_lst.text.split('\r\n') if proxy]
proxy_lst = requests.get("http://proxylist.fatezero.org/proxy.list")
proxies = (f"{json.loads(data)['host']}:{json.loads(data)['port']}" for data in proxy_lst.text.split('\n') if data)

def get_proxies(proxy):
    proxies = {
        "https": proxy,
        "http": proxy
    }
    try:
        response = requests.get(url=check_proxies_url, proxies=proxies, timeout=2)
        response.raise_for_status()
        if ping(target=proxies["https"].split(':')[0], count=1, timeout=2).rtt_avg_ms < 150:
            good_proxy.add(proxies["https"])
            print(f"Good proxies: {proxies['https']}")
    except Exception:
        print(f"Bad proxy: {proxies['https']}")

with ThreadPoolExecutor() as executor:
    executor.map(get_proxies, proxies)

print(good_proxy)

Get a list of active proxies with ping up to 150ms.

Output:

{'209.166.175.201:8080', '170.39.194.156:3128', '20.111.54.16:80', '20.111.54.16:8123'}

But in any case, this is a shared proxy and their performance is not guaranteed. If you want to be sure that your parser will work, then it is better to buy a proxy.

I ran your code with received proxy '170.39.194.156:3128' and for now it works

You have to use inner_text for this:

email = page.locator.inner_text("#cphMain_hlOrgInboundEmail")
print(email)

You can use the select_options for selecting 30.

page.select_option('select#MainContent_ddlRecords', '30')

You can use the text selector and then click the checkboxes like this:

page.locator("text=Tabloul avocaţilor stagiari").click()

You can use the Page.innerText(".score") method to accomplish that, see here for an example:

// @ts-check
const playwright = require("playwright");

(async () => {
  const browser = await playwright.chromium.launch();
  const context = await browser.newContext();
  const page = await context.newPage();
  await page.setContent(`
    <span _ngcontent-fnp-c148="" class="score"> &nbsp;<!--bindings={
"ng-reflect-ng-if": "false"
}--> 96%, </span>
`);
  const content = await page.innerText(".score")
  console.log(content)
  await page.screenshot({ path: `example.png` });
  await browser.close();
})();

Or interactively here: https://try.playwright.tech/?s=bl1fed4

For Python usage see here: https://microsoft.github.io/playwright-python/sync_api.html#playwright.sync_api.Page.innerText

Closing actual popups

I think browser.once("targetcreated", ...) as described in this issue https://stackoverflow.com/a/57246544/2202112 could be something that you are looking for.

It allows you to setup a callback on newly created targets.

For reference, see puppeteer docs on "targetcreated"

Closing modals

1. Find the selector for the button that dismisses the modal, for example .button-close or .button-dismiss.
2. Use the page.click(selector, [options]) method to click the button before continuing further scraping

this worked for me in java:

Page page = browser.newPage();
Map<String, String> headers = new HashMap<>();
headers.put("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) `enter code here`AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36");
page.setExtraHTTPHeaders(headers );