How to use test_should_click_background_tab method in Playwright Python

There are many things you can improve there.

• First, you can read the text area value using page.locator('#startText').value()
• Then, you click on the check button. Don't wait using wait_for_load_state, because the page is not navigating.
• After the check action there is an SPAN element inside the "#inside" div. You can do page.locator('#inside SPAN').inner_html(). The inner_html function will wait for the element for you.

Parameter slow_mo is misleading. What you should have done was passing 'networkilde' to your page.wait_for_load_state() like page.wait_for_load_state('networkilde') or use a locator

For anyone referring to this in the future, slow_mo slows down the rendering which gives more time for the DOM to load data and it is just a debugging tool from playwright

The accepted answer should not be used unless they are debugging their automation.

You should be able to get that button by text:

page.locator('"Log in"').click()

There is several good practices for creating locators/selectors. Using playwright there is official documentation for each common and unique selector on how-to and what-is doing.

More information in https://playwright.dev/docs/selectors#text-selector

About your case, i would suggest always to use an parent selector for locating an element. When there is a button, try to find its unique parent.

• By id
• By unique class
• Something else unique.

Example:

<dv id=test>
 <button id=continue-test>Continue</button>
</div>

In this case you can use the unique id of the button and not the text. Selector css: #continue-test But if you, don't have an unique identifier for the button you can use the parent and go down to the button. Selector css: #test > button

Matching text using css is not possible, but with XPATH can look like this:

//button[text()="Continue"] 

This selector MATCHES the text using "equals".

Using playwright:

button:has-text("Continue")

Using has-text and quotes - matches the text using equals. If you are using another selector for example text=Continue, this will match all elements that CONTAINS the text "Continue"

All this is explained with example in the official documentation for playwright selectors. That does not mean to not use XPATH to achieve the goals.

CSS selectors are fast but kind of restricted to work with text.

Xpath is quite slower but much more powerful to work in text/parent/child elements etc.

I would suggest always to use an parent element with unique identifier and go down to reach your actual element, which will receive the interaction.

The Download historical data button is inside an iframe that points to this URL.

https://www.macrotrends.net/assets/php/chart_iframe_comp.php?id=1476&url=copper-prices-historical-chart-data

But it needs Javascript to work, meaning, you have to load the page in a browser (that executes javascript) to click that button [^1].

Luckily, there are tools to automate browser interactions. Playwright is a library available in many languages, but we'll use it with Python.

1. Install the latest version of Python.
2. Update the pip (Python package manager) to the latest version:

   python -m pip install --user --upgrade pip
   

3. Install playright

   python -m pip install --user playwright
   

4. Use playwright to install a browser. It will be used by playwright only.

   python -m playwright install chromium
   

5. Open the URL with playwright inspector:

   python -m playwright open 'https://www.macrotrends.net/1476/copper-prices-historical-chart-data'
   

A browser window should pop up. Change the Target to python and click Record button in Playwright inspector window.

Then do what you need to do to click & download the file from the page.

Playwright will generate some code for us.

Stop recording and copy the script.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    with p.chromium.launch(headless=False) as browser:
        page = browser.new_page(accept_downloads=True)
        page.goto('https://www.macrotrends.net/1476/copper-prices-historical-chart-data')
        
        # paste the script here

In my runs, playwright generated this script for me:

# Click button:has-text("Download Historical Data")
with page.expect_download() as download_info:
    page.frame(name="chart_iframe").click("button:has-text(\"Download Historical Data\")")

Combining both and adding the code to save the file, we have:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    print('launching the browser')
    with p.chromium.launch(headless=False) as browser:
        page = browser.new_page(accept_downloads=True)
        print('visiting the page')
        page.goto('https://www.macrotrends.net/1476/copper-prices-historical-chart-data')
        
        print('clicking download button')
        # Click button:has-text("Download Historical Data")
        with page.expect_download() as download_info:
            page.frame(name="chart_iframe").click("button:has-text(\"Download Historical Data\")")
        download = download_info.value
        
        print('saving the file')
        download.save_as('prices.csv')

One important point you should look out for is that Python is a whitespace-sensitive language, meaning you have to align the code indentation properly to make it run.

Save this file as download_copper_prices.py, and when we run the script,

python download_copper_prices.py

A browser should pop up, visit the page, download and save the prices as prices.csv in the working directory.

[^1]: Actually, we can scrape the prices right off the page, as they're serialized as JSON. With a bit of regex, we can extract the JSON from the HTML. But that solution seemed to me a bit too job-specific, so I opted for a more hands-on solution that he can apply for other occasions. Scripting browser interactions is an important tool every power user should learn.

That said, I'm not knowledgeable in AutoIt, but assuming it has capable Regex tools, it's entirely possible to achieve the same result in AutoIt by downloading the HTML of the URL that the iframe points to (https://www.macrotrends.net/assets/php/chart_iframe_comp.php?id=1476&url=copper-prices-historical-chart-data) then using this regex to extract the JSON:

var originalData = (.*);

Running this code on the browser console returns an array of 15k+ entries:

JSON.parse(document.body.innerHTML.match('var originalData = (.*);')[1])

consisting of date and closing prices:

[{
    "date": "1959-07-02",
    "close": "0.2930"
}, {...}, ...]

Then it should be easy to convert this JSON to CSV.

References

• https://playwright.dev/python/docs/intro

There is no outer_html out of the box. But it's not hard to implement it:

locator.evaluate("el => el.outerHTML") 

Have you tried to download it as Binary package?

Maybe you can go over this libasound.so.2 package page and see if you can download it.

wget https://vault.centos.org/centos/8/AppStream/aarch64/os/Packages/alsa-lib-1.2.5-4.el8.aarch64.rpm

When I searched within apt, ubuntu package tool I don't see anything there

root@21be95fca6e6:/# apt search libasound.so.2
Sorting... Done
Full Text Search... Done

For instance, Vim package would appears like below

root@21be95fca6e6:/# apt search vim           
Sorting... Done
Full Text Search... Done
apvlv/jammy 0.4.0-2 arm64
  PDF viewer with Vim-like behaviour

Looks like soup is unable to find anything and its returning None and Nonetype has no text property

The error looks here for soup in clothes: as you have overridden the soup parser

PS:

You can use get_text() to extract the text

price = soup.find('span', {'class' : 'price'}).get_text()

Sometimes you may have to use contents depending on the return type

price = soup.find('span', {'class' : 'price'}).contents[0]

For future visitors, the playwright needs access to your /usr/bin and my initial approach of adding different bin directories was wrong. I should have used the following code to extend the PATH: Environment="PATH=$PATH:/usr/bin"

my server is working fine now.