How to use test_url_should_include_hashes method in Playwright Python

Methods in python are snake case. context.add_cookies should do the work.

For Typescript/Javascript there is playwright-dompath which you can install with:

npm install playwright-dompath

Basically you just call the xPath function with the element you selected, and you get back the full xPath. Under the hood, it uses the same functions from Chrome DevTools.

Playwright's default timer is 30 seconds. So in the try part, it tries for 30 seconds to get the series, but there is none, so after 30 seconds only, it goes to the exception.

You can set a custom timout when calling inner_text().

See: https://playwright.dev/python/docs/api/class-locator#locator-inner-text

You can access the href in an almost similar way to that of selenium.

element_handle = page.locator("//a")
link = element_handle.get_attribute('href')

To print the title, views and when text from the website you need to WebDriverWait for the visibility_of_element_located() and you can use the following Locator Strategies:

• Code Block:

  driver.get("https://www.youtube.com/c/JohnWatsonRooney/videos")
  titles = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a#video-title")))]
  views = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div#metadata-line > span:first-child")))]
  when = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div#metadata-line > span:nth-child(2)")))]
  for i, j, k in zip(titles, views, when):
  print(f"{i} had {j} since posted {k}")
  

• Console Output:

  The Python Package I Wish I'd Learned Earlier had 4.5K views since posted 4 days ago
  Rotate User Agents in Scrapy using custom Middleware had 1.2K views since posted 11 days ago
  GO for Beginners - Web Scraping with Golang Tutorial had 2K views since posted 13 days ago
  How I Scraped This HTML Table to a Python Dictionary had 3.6K views since posted 3 weeks ago
  Python TYPE HINTS Explained with Examples had 2.3K views since posted 3 weeks ago
  Use THIS Algorithm To Find KEYWORDS in Text - A Short Python Project had 4.2K views since posted 1 month ago
  SQLModel is the Pydantic inspired Python ORM we’ve been waiting for had 3.4K views since posted 1 month ago
  How to use Enumerate in Python to have a Counter in your loops had 4.6K views since posted 1 month ago
  How to HIDE Your API Keys in Python Projects had 6.6K views since posted 1 month ago
  How to Make 2500 HTTP Requests in 2 Seconds with Async & Await had 4.8K views since posted 2 months ago
  Are You Still Using Excel? AUTOMATE it with PYTHON had 10K views since posted 2 months ago
  How To Parse Data from HTML Tables Using Requests-HTML had 3.5K views since posted 2 months ago
  THIS is the most common ERROR when learning Web Scraping had 3.6K views since posted 2 months ago
  Learn Web Scraping With Python: Full Project - HTML, Save to CSV, Pagination had 10K views since posted 3 months ago
  Turn Websites into Real Time API's with ScrapyRT had 4.4K views since posted 3 months ago
  HTTPX is the ASYNC Requests I was Looking For had 5.8K views since posted 3 months ago
  Research Amazon Products by Extracting Review Data had 4K views since posted 4 months ago
  Web Scraping 101 - My (in)complete guide, methods, tools, how to had 5.4K views since posted 4 months ago
  How to Scrape JavaScript Websites with Scrapy and Playwright had 9.8K views since posted 5 months ago
  Web Scraping with Node js? Python Expert Opinion and demo had 3.3K views since posted 5 months ago
  Automate Buying online using Playwright’s Codegen feature had 4.6K views since posted 5 months ago
  Login and Scrape Data with Playwright and Python had 16K views since posted 5 months ago
  Parse HTML with BeautifulSoup AND Scrapy had 2.6K views since posted 5 months ago
  HIDING Data with JavaScript? Web Scraping Obfuscation had 3.9K views since posted 5 months ago
  Following LINKS Automatically with Scrapy CrawlSpider had 6.2K views since posted 6 months ago
  Web Scraping Weather Data with Python had 13K views since posted 6 months ago
  How I Try My CODE and Test My SELECTORS in SCRAPY had 1.8K views since posted 6 months ago
  How To Handle Errors & Exceptions with Requests and Python had 2.8K views since posted 6 months ago
  A Short and SIMPLE HTML Web Scraper in 6 lines of CODE had 1.9K views since posted 7 months ago
  Failed Requests? Try this RETRY Decorator for your Web Scraper had 3.5K views since posted 7 months ago
  

• Note : You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  

The misconception here seems to be that visible == clickable. That is not the case. In fact the docs explain what exact actionability checks are performed that determines whether the element is clickable

For example, for page.click(selector, **kwargs), Playwright will ensure that:

element is Attached to the DOM

element is Visible

element is Stable, as in not animating or completed animation

element Receives Events, as in not obscured by other elements

element is Enabled

Functionality to check whether all this is True for any element is currently not exposed in playwright (specifically, for checking whether the element is stable and reactive). The best you can do is change if locator.is_visible() to if locator.is_visible() and locator.is_enabled() to additionally check whether it is enabled as well (which would be done as part of the actionability checks later).

You can track the relevant issue here

The sync_api is simply a wrapper around the asyncio_api that abstracts asyncio usage away from you. As such, the capabilities are largely the same, but the async_api may afford some more flexibility in complex scenarios (for example, in previous playwright-python releases the only way to run instances in a multi-threaded fashion on Unix + Python 3.7 was to use the async_api for reasons I won't get into here).

Chances are you won't need that flexibility though, so I'd just suggest using whatever you're comfortable with.

So I found a working solution.

What I did was to combine Javascript with python Playwright code.

I start the setInterval with a timer of 200ms to scroll down on the page with page.evaluate() and then I follow it up with a python loop that checks every second whether the total height of the page (scroll included) has changed. If it changes it continues to scroll and if it hasn't changed than the scroll is over.
This is what it looks like:

page.evaluate(
    """
    var intervalID = setInterval(function () {
        var scrollingElement = (document.scrollingElement || document.body);
        scrollingElement.scrollTop = scrollingElement.scrollHeight;
    }, 200);

    """
)
prev_height = None
while True:
    curr_height = page.evaluate('(window.innerHeight + window.scrollY)')
    if not prev_height:
        prev_height = curr_height
        time.sleep(1)
    elif prev_height == curr_height:
        page.evaluate('clearInterval(intervalID)')
        break
    else:
        prev_height = curr_height
        time.sleep(1)

EDIT

See the below answer using the new mouse.wheel(x, y) feature for an up to date way to scroll using playwright. Combine my answer with his to lessen the need to use JS.

That happens because your webpage did not load totally, you can set a timeout like:

page.goto('https://www.zameen.com/', timeout=100000)

Or you can even don't wait till your page is loaded and just navigate without waiting the page to be fully loaded with timeout=0

page.goto('https://www.zameen.com/', timeout=0)