How to use test_hover_when_node_is_removed method in Playwright Python

For the 1st form:

def get_playwright():
    with sync_playwright() as playwright:
        yield playwright

print(get_playwright())  # <generator object get_playwright at 0x108aac580>

That is expected because get_playwright is a generator, which returns a generator iterator, which you have to call next(...) on to get each yielded value from the iterator.

Consider a simpler, non-playwright example:

In [14]: def generate_nums():
    ...:     for num in range(10):
    ...:         yield num
    ...: 

In [15]: nums = generate_nums()

In [16]: nums
Out[16]: <generator object generate_nums at 0x11115e6d0>

In [17]: next(nums)
Out[17]: 0

In [18]: next(nums)
Out[18]: 1

In [19]: next(nums)
Out[19]: 2

For more examples, see Understanding generators in Python.

Since your get_playwright returns an iterator, you need to call next() once to get the actual object:

from playwright.sync_api import sync_playwright


def get_playwright():
    with sync_playwright() as playwright:
        yield playwright

playwright_generator = get_playwright()
print(playwright_generator)  # <generator object get_playwright at 0x104031580>

playwright = next(playwright_generator)
print(playwright)  # <playwright._impl._playwright.Playwright object at 0x1041aabb0>

For the 2nd form:

@pytest.fixture()
def get_playwright():
    with sync_playwright() as playwright:
        yield playwright

def test(get_playwright):
    print(get_playwright)

It should be the same case, but it's just that pytest automatically calls next() on the fixture value if it's a generator. I could not find documentation for this behavior from the pytest docs, but it was mentioned by one of the pytest author's/maintainer's in a different answer:

Here's roughly the execution here

• pytest notices your fixture is used for the test function
• pytest calls the fixture function
  • since it is a generator, it returns immediately without executing code
• pytest notices it is a generator, calls next(...) on it
  • this causes the code to execute until the yield and then "pausing". you can think of it kind of as a co-routine ...
• pytest then executes your test function

...which is probably why the value passed to your test function is already the next-ed value, the playwright object.

You should be able to get that button by text:

page.locator('"Log in"').click()

So I found a working solution.

What I did was to combine Javascript with python Playwright code.

I start the setInterval with a timer of 200ms to scroll down on the page with page.evaluate() and then I follow it up with a python loop that checks every second whether the total height of the page (scroll included) has changed. If it changes it continues to scroll and if it hasn't changed than the scroll is over.
This is what it looks like:

page.evaluate(
    """
    var intervalID = setInterval(function () {
        var scrollingElement = (document.scrollingElement || document.body);
        scrollingElement.scrollTop = scrollingElement.scrollHeight;
    }, 200);

    """
)
prev_height = None
while True:
    curr_height = page.evaluate('(window.innerHeight + window.scrollY)')
    if not prev_height:
        prev_height = curr_height
        time.sleep(1)
    elif prev_height == curr_height:
        page.evaluate('clearInterval(intervalID)')
        break
    else:
        prev_height = curr_height
        time.sleep(1)

EDIT

See the below answer using the new mouse.wheel(x, y) feature for an up to date way to scroll using playwright. Combine my answer with his to lessen the need to use JS.

It looks like you're trying to run python -m playwright install as python code rather than a shell command. Try running !python -m playwright install to signal to jupyter that this a shell command rather than python code. Use the ! prefix anytime you need to run a shell command on your environment in jupyter.

So how do I scrape a website which has dynamic content?

there are a few options:

1. Use Selenium, which allows you to simulate opening a browser, letting the page render, then pull the html source code
2. Sometimes you can look at the XHR and see if you can fetch the data directly (like from an API)
3. Sometimes the data is within the <script> tags of the html source. You could search through those and use json.loads() once you manipulate the text into a json format

what exactly is the difference between dynamic and static content?

Dynamic means the data is generated from a request after the initial page request. Static means all the data is there at the original call to the site

How do I extract other information like price and image from the website? and how to get particular classes for example like a price?

Refer to your first question

how would I know that data is dynamically created?

You'll know it's dynamically created if you see it in the dev tools page source, but not in the html page source you first request. You can also see if the data is generated by additional requests in the dev tool and looking at Network -> XHR

Lastly

Amazon does offer an API to access the data. Try looking into that as well

in the docker, you don't need to go inside python. Just run from bash:

$ npx playwright open google.com

For more options, check out the doc: https://playwright.dev/docs/cli/

You can define your own function like this one:

def my_own_wait_for_selector(page, selector, time_out):
    try:
        page.wait_for_selector(selector, timeout=time_out)
        return True
    except:
        return False

That function will wait for a selector and if given some time (miliseconds) the element is not there it will return False instead of trowing an error.

Then in your code you could try something like:

while not my_own_wait_for_selector(page, "my_button_selector", 5000):
    page.reload()
page.locator("my_button_selector").click()

As you can see we are checking if our button is present, and we will wait only 5 seconds, if after 5 seconds the button is not there, we will reload, and again and again until our button is there, once our button is there code will continue and then we will click the button.

Be careful, because if your button is never there, your script will reload and your script will never finish, so you should add some other condition to your while

The main problem:

context.new_page() calls before variable declaration(context = ...)

Should works fine:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False, slow_mo=50)
    context = browser.new_context(storage_state="website1.json")
    page = context.new_page()
    page.goto('https://demo.opencart.com/admin')
    page.fill("input#input-username", "demo")
    page.fill("input#input-password", "demo")
    page.click("button[type=submit]")

Also you can use Persistent authentication:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    user_data_dir = 'FULL_PATH_TO_BROWSER_PROFILE'
    browser = p.chromium.launch_persistent_context(user_data_dir, headless=False)
    # login only 1 time...

JFYI: opencart uses user_token in GET parameters and validates it. playwright(storage_state, user_data_dir etc) should works fine. Example:

# run docker container with playwright
docker rm -fv example && docker run --name example -it mcr.microsoft.com/playwright/python:v1.27.1-focal bash
# create empty state
echo {} >> /tmp/state.json
# run python in interactive mode
python
# Python 3.8.10 (default, Jun 22 2022, 20:18:18) 
# [GCC 9.4.0] on linux
# Type "help", "copyright", "credits" or "license" for more information.

Paste the next script:

from time import sleep
from urllib.parse import parse_qs

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    # login
    browser = p.chromium.launch()
    context = browser.new_context(storage_state='/tmp/state.json')
    page = context.new_page()
    page.goto('https://demo.opencart.com/admin')
    page.fill('input#input-username', 'demo')
    page.fill('input#input-password', 'demo')
    page.click('button[type=submit]')
    sleep(5)  # just wait for redirect - you can wait for a element...
    page.context.storage_state(path='/tmp/state.json')
    page.screenshot(path='/tmp/after_state_init.png')
    # parse user_token for GET requests
    parsed_url = parse_qs(page.url)
    user_token = parsed_url['user_token'][0]
    print(f'user token {user_token}')
    page.close()
    browser.close()


with sync_playwright() as p:
    # open admin dashboard without login... 
    browser = p.chromium.launch()
    context = browser.new_context()
    page = browser.new_page(storage_state='/tmp/state.json')
    page.goto(f'https://demo.opencart.com/admin/index.php?route=common/dashboard&user_token={user_token}')
    sleep(5)
    page.screenshot(path='/tmp/open_using_saved_state.png')
    page.close()
    browser.close()

Now open a new terminal and copy screens from docker container:

docker cp example:/tmp/after_state_init.png ./
docker cp example:/tmp/open_using_saved_state.png ./

You'll see that they are the same - admin dashboard works without login.

Edit: Got the answer. The autothrottle was limited to default. Now I need to limit it to the number of website pages. The code looks like this CONCURRENT_REQUESTS = 3