How to use test_should_work_with_main_frame method in Playwright Python

Version 1.22

iframe = page.frame_locator("YOUR_SELECTOR_HERE")

https://playwright.dev/python/docs/api/class-framelocator

for i in range(0, 5):

            html = response.text
            # print(html)
            soup = BeautifulSoup(html, "html.parser")
            search_date = soup.find_all("input")[-6]["value"]
            await page.click(
                "#yDmH0d > c-wiz.zQTmif.SSPGKf > div > div:nth-child(2) > c-wiz > div > c-wiz > div.PSZ8D.EA71Tc > div.Ep1EJd > div > div.rIZzse > div.bgJkKe.K0Tsu > div > div > div.dvO2xc.k0gFV > div > div > div:nth-child(1) > div > div.oSuIZ.YICvqf.kStSsc.ieVaIb > div > div.WViz0c.CKPWLe.U9gnhd.Xbfhhd > button"
            )

            yield {
                "search_date": search_date,
            }

This logic to extract all the dates in the range is incorrect. Understand the flow of

1. you make request to the flight page.
2. you get the response.
3. In parse method you are trying to extract the search date.

search_date = soup.find_all("input")[-6]["value"]

This line is only returning one date and not a list. Moreover, I don't understand the logic behind the for loop here. Because this piece of code is not making any further requests as you are trying to achieve with the extracted dates[mentioned in the question]

Important thing to note here is, you have to got the entire html response in one go. You can now use css selectors or soup to extract all the selected dates. Running the for loop for 5 times, doesn't solve any purpose, as you are executing the same information 5 times.

Use response.css('<<Path to the dates in the select>>').getall() to get all the dates you are looking for. Process the information further.

Improvisation on the logic: You can improvise on the logic, I am not sure why you are extracting range of dates, when you can just extract the departure and return dates for the page. And make a request using them. Or Just extract the departure date and increment it and make another request using that date for further information.

You can define your own function like this one:

def my_own_wait_for_selector(page, selector, time_out):
    try:
        page.wait_for_selector(selector, timeout=time_out)
        return True
    except:
        return False

That function will wait for a selector and if given some time (miliseconds) the element is not there it will return False instead of trowing an error.

Then in your code you could try something like:

while not my_own_wait_for_selector(page, "my_button_selector", 5000):
    page.reload()
page.locator("my_button_selector").click()

As you can see we are checking if our button is present, and we will wait only 5 seconds, if after 5 seconds the button is not there, we will reload, and again and again until our button is there, once our button is there code will continue and then we will click the button.

Be careful, because if your button is never there, your script will reload and your script will never finish, so you should add some other condition to your while

Chromium will not allow you to override the Host header. I presume this is because Host is something the browser manages internally. Source.

Why do you want to modify this header?

Just so you're aware:

# http://www.google.com/robots.txt

User-agent: *
Disallow: /search

I would like to preface my answer by saying that Google heavily relies on scripting. It's very possible that you're getting different results because the page you're requesting via reqeusts doesn't do anything with the scripts supplied in on the page, whereas loading the page in a web browser does.

Here's what i get when I request the url you supplied

The text I get back from requests.get(url).text doesn't contain 'imgurl' in it anywhere. Your script is looking for that as part of its criteria and it's not there.

I do however see a bunch of <img> tags with the src attribute set to an image url. If that's what you're after, than try this script:

import requests
from bs4 import BeautifulSoup

url = 'https://www.google.no/search?q=tower&client=opera&hs=cTQ&source=lnms&tbm=isch&sa=X&ved=0ahUKEwig3LOx4PzKAhWGFywKHZyZAAgQ_AUIBygB&biw=1920&bih=982'

# page = open('tower.html', 'r').read()
page = requests.get(url).text

soup = BeautifulSoup(page, 'html.parser')

for raw_img in soup.find_all('img'):
  link = raw_img.get('src')
  if link:
    print(link)

Which returns the following results:

https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcQyxRHrFw0NM-ZcygiHoVhY6B6dWwhwT4va727380n_IekkU9sC1XSddAg
https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcRfuhcCcOnC8DmOfweuWMKj3cTKXHS74XFh9GYAPhpD0OhGiCB7Z-gidkVk
https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcSOBZ9iFTXR8sGYkjWwPG41EO5Wlcv2rix0S9Ue1HFcts4VcWMrHkD5y10
https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcTEAZM3UoqqDCgcn48n8RlhBotSqvDLcE1z11y9n0yFYw4MrUFucPTbQ0Ma
https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcSJvthsICJuYCKfS1PaKGkhfjETL22gfaPxqUm0C2-LIH9HP58tNap7bwc
https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcQGNtqD1NOwCaEWXZgcY1pPxQsdB8Z2uLGmiIcLLou6F_1c55zylpMWvSo
https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcSdRxvQjm4KWaxhAnJx2GNwTybrtUYCcb_sPoQLyAde2KMBUhR-65cm55I
https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcQLVqQ7HLzD7C-mZYQyrwBIUjBRl8okRDcDoeQE-AZ2FR0zCPUfZwQ8Q20
https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcQHNByVCZzjSuMXMd-OV7RZI0Pj7fk93jVKSVs7YYgc_MsQqKu2v0EP1M0
https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcS_RUkfpGZ1xJ2_7DCGPommRiIZOcXRi-63KIE70BHOb6uRk232TZJdGzc
https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcSxv4ckWM6eg_BtQlSkFP9hjRB6yPNn1pRyThz3D8MMaLVoPbryrqiMBvlZ
https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcQWv_dHMr5ZQzOj8Ort1gItvLgVKLvgm9qaSOi4Uomy13-gWZNcfk8UNO8
https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcRRwzRc9BJpBQyqLNwR6HZ_oPfU1xKDh63mdfZZKV2lo1JWcztBluOrkt_o
https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcQdGCT2h_O16OptH7OofZHNvtUhDdGxOHz2n8mRp78Xk-Oy3rndZ88r7ZA
https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcRnmn9diX3Q08e_wpwOwn0N7L1QpnBep1DbUFXq0PbnkYXfO0wBy6fkpZY
https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcSaP9Ok5n6dL5K1yKXw0TtPd14taoQ0r3HDEwU5F9mOEGdvcIB0ajyqXGE
https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcTcyaCvbXLYRtFspKBe18Yy5WZ_1tzzeYD8Obb-r4x9Yi6YZw83SfdOF5fm
https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcTnS1qCjeYrbUtDSUNcRhkdO3fc3LTtN8KaQm-rFnbj_JagQEPJRGM-DnY0
https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcSiX_elwJQXGlToaEhFD5j2dBkP70PYDmA5stig29DC5maNhbfG76aDOyGh
https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcQb3ughdUcPUgWAF6SkPFnyiJhe9Eb-NLbEZl_r7Pvt4B3mZN1SVGv0J-s

There is another alternative package which works for scraping google play store data, below is the link

https://pypi.org/project/google-play-scraper/

In case, your looking straight for sample code follow the below steps

from google_play_scraper import app

result = app(
'com.nianticlabs.pokemongo',
lang='en', # defaults to 'en'
country='us' # defaults to 'us'
)
print(result)

If you want a Playwright image with Python instead of the Playwright image with Node you have currently set,

do

FROM mcr.microsoft.com/playwright/python:v1.23.0-focal

instead. You won't then need to install python3-pip (or playwright!) at all.

Additionally, you'll probably want to explicitly spell out python3 instead of python:

FROM mcr.microsoft.com/playwright/python:v1.23.0-focal
RUN pip3 install TikTokApi
RUN playwright install
COPY . .

You're missing some command line flags that tell chrome to simulate having a microphone. Give this sample a shot.

from playwright.sync_api import sync_playwright


def run(playwright):
    chromium = playwright.chromium
    browser = chromium.launch(headless=False, args=['--use-fake-device-for-media-stream', '--use-fake-ui-for-media-stream'])
    context = browser.new_context()
    context.grant_permissions(permissions=['microphone'])
    page = context.new_page()
    page.goto("https://permission.site/")
    page.click('#microphone')
    page.pause()
    # other actions...
    browser.close()


with sync_playwright() as playwright:
    run(playwright)

Heroku's filesystem is both ephemeral and dyno-local. Any changes made to it are lost when the dyno restarts. More importantly at the moment, when you run something like heroku run bash you don't connect to a running dyno. Instead, you get a separate one-off dyno.

Installing Chromium (or making any other filesystem changes) in such an environment will never affect what is available in your other dynos.

It looks like Playwright really wants to manage its own browser binaries. Your best bet might be to run playwright install near the beginning of your script, e.g. something like this:

import subprocess
from playwright.sync_api import sync_playwright

subprocess.run(["playwright", "install"])

with sync_playwright() as playwright:
    # ...

This should make sure that the required binaries are always available at runtime.