How to use not_to_have_attribute method in Playwright Python

Try Firefox browser like this:

from playwright.sync_api import sync_playwright

chromium = playwright.firefox
browser = chromium.launch(headless=False)
page = browser.new_page()
page.goto(url)

Not sure about Colab, but in a normal Jupyter notebook you do:

import nest_asyncio
nest_asyncio.apply()

Install with pip install nest-asyncio, and then you can run async stuff in a notebook.

Edit: You are also trying to run a GUI instance of Chrome, with that headless=False - change that to headless=True, Colab doens't run with a GUI.

Not guaranteed for success. Change the parameters in function async_cf_retry if it tends to fail. I wrote this code by modifying my heavy modules, so it may not work elegantly.

import re
import asyncio
from playwright.async_api import async_playwright, Error, Page
from cf_clearance import stealth_async
import httpx

# local_client = httpx.AsyncClient(verify=False)


async def async_cf_retry(page: Page, tries=10) -> bool:
    # use tries=-1 for infinite retries
    # excerpted from `from cf_clearance import async_retry`
    success = False
    while tries != 0:
        try:
            title = await page.title()
        except Error:
            tries -= 1
            await asyncio.sleep(1)
        else:
            # print(title)
            if title == 'Please Wait... | Cloudflare':
                await page.close()
                raise NotImplementedError('Encountered recaptcha. Check whether your proxy is an elite proxy.')
            elif title == 'Just a moment...':
                tries -= 1
                await asyncio.sleep(5)
            elif "www." in title:
                await page.reload()
                tries -= 1
                await asyncio.sleep(5)
            else:
                success = True
                break
    return success


async def get_client_with_clearance(proxy: str = None):
    async def get_one_clearance(proxy=proxy, logs=False):
        # proxy = {"server": "socks5://localhost:7890"}
        if type(proxy) is str:
            proxy = {'server': proxy}
        async with async_playwright() as p:
            browser = await p.chromium.launch(headless=False, proxy=proxy,
                                              args=["--window-position=1000,800", "--disable-web-security",
                                                    "--disable-webgl"])
            page = await browser.new_page()#viewport={"width": 0, "height": 0})
            await stealth_async(page)
            
            # Download
            async def on_download(download):
                print('download', download)
                await download.save_as(download.suggested_filename)
            page.on("download", on_download)

            if logs:
                def log_response(intercepted_response):
                    print("a response was received:", intercepted_response.url)
                
                page.on("response", log_response)
            
            await page.goto("https://www.historique-meteo.net/site/export.php?ville_id=1067")
            res = await async_cf_retry(page)
            if res:
                cookies = await page.context.cookies()
                cookies_for_httpx = {cookie['name']: cookie['value'] for cookie in cookies}
                ua = await page.evaluate('() => {return navigator.userAgent}')
                # print(ua)
            else:
                await page.close()
                raise InterruptedError("cf challenge fail")
            await asyncio.sleep(10000)
            return ua, cookies_for_httpx
    
    ua, cookies_for_httpx = await get_one_clearance(logs=True)
    print(cookies_for_httpx)


print(asyncio.get_event_loop().run_until_complete(get_client_with_clearance(
    # proxy='http://localhost:8888'
    # use proxifier on windows as an elite proxy
)))

If download <Download url='https://www.historique-meteo.net/site/export.php?ville_id=1067' suggested_filename='export-aix-les-bains.csv'> is shown in the console, the browser must be downloading your file.
This code may keep trying to get a cloudflare clearance even if the file has been downloaded. Edit your strategies in async_cf_retry if this happens.
Use persisent context offered by playwright to save and reuse your cloudflare clearance. Consider downloading with httpx.

Absent seeing the actual site, a traditional selection and get_attribute should be sufficient:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.set_content("""
    <iframe title="IFRAME_NAME" src="https://www.data_I_want_TO_get.com"></iframe>
    """)
    src = page.get_attribute('iframe[title="IFRAME_NAME"]', "src")
    print(src)  # => https://www.data_I_want_TO_get.com
    browser.close()

If the frame isn't immediately visible you can wait for it by replacing the src = line with

src = (
    page.wait_for_selector('iframe[title="IFRAME_NAME"]')
        .get_attribute("src")
)

You can see from the documentation here that you can use pseudo-class of has-text

... the :has-text() pseudo-class

and maybe some some more advance matching selector with pseudo-class text-matches

the :text-matches() pseudo-class

but you need to remember that two ways is better if you combine the usage with some css selector.

Maybe you can add more details on where you want to search the string from is it literally in the entire document or not. (and I can update the answer with more detail code example according to your target)

What dialog.accept() does is to accept (click OK) the dialog. page.click("OK") only works with DOM (HTML) elements, not with browser dialogs.

So how do I scrape a website which has dynamic content?

there are a few options:

1. Use Selenium, which allows you to simulate opening a browser, letting the page render, then pull the html source code
2. Sometimes you can look at the XHR and see if you can fetch the data directly (like from an API)
3. Sometimes the data is within the <script> tags of the html source. You could search through those and use json.loads() once you manipulate the text into a json format

what exactly is the difference between dynamic and static content?

Dynamic means the data is generated from a request after the initial page request. Static means all the data is there at the original call to the site

How do I extract other information like price and image from the website? and how to get particular classes for example like a price?

Refer to your first question

how would I know that data is dynamically created?

You'll know it's dynamically created if you see it in the dev tools page source, but not in the html page source you first request. You can also see if the data is generated by additional requests in the dev tool and looking at Network -> XHR

Lastly

Amazon does offer an API to access the data. Try looking into that as well

You can use Playwright's codegen tool and open the webkit browser

playwright codegen -b webkit

Maybe if you put a break in each loop:

from playwright.sync_api import sync_playwright
from time import sleep

with sync_playwright() as p:
    browser = p.firefox.launch(headless=False)
    page = browser.new_page()
    page.goto("https://books.toscrape.com/")

    while True:
        try: 
            next = page.get_by_text("Next")  # next clicker
            next.click()
            sleep(2)
        except Exception:
            break