3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps
Selenium Guide: Managing Cookies
Managing cookies is an important aspect of web automation, and Selenium provides methods to
interact with cookies in various ways.
In this comprehensive guide, we will delve into the world of Selenium and explore practical strategies for
handling cookies. We'll be covering:
Introduction
Understanding Cookies
Selenium and Cookies
Cookie Handling in Production
Conclusion
More Cool Articles
Handling Cookies With Selenium
Cookies are used to store information about your browser. They are used to remember your browser
when you visit a website, track you as you navigate a website, and to analyze the usage of a website.
In Selenium, we have a whole set of methods devoted to managing cookies:
https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 1/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps
driver.add_cookie() adds a cookie to our stored cookies
driver.get_cookie(name_of_cookie) returns an individual cookie
driver.get_cookies() returns all cookies that are currently being stored in the browser
driver.delete_cookie(name_of_cookie) will delete a specific cookie from the browser
driver.delete_all_cookies() (as you probably guessed) clears all cookies from the browser
The code below shows how all these methods can be used:
from selenium import webdriver
#open an instance of chrome
driver = webdriver.Chrome()
#navigate to the website
driver.get("https://quotes.toscrape.com")
#get cookies and save as a variable
cookies = driver.get_cookies()
#print the current list of cookies
print("Cookies:", cookies)
#add a cookie
driver.add_cookie({"name": "our new cookie!", "value": "some random text!"})
#update our cookie variable
cookies = driver.get_cookies()
#print our cookies
print("Cookies after adding:", cookies)
#iterate through our list of cookies
for cookie in cookies:
#if this line executes, we found our cookie
if cookie["name"] == "our new cookie!":
print("We have a match!")
cookie["value"] = "we changed our text"
#delete the old cookie
driver.delete_cookie("our new cookie!")
#replace it with the updated cookie
driver.add_cookie(cookie)
#update the cookies variable
cookies = driver.get_cookies()
#print the cookies
print("Cookies after modifying:", cookies
https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 2/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps
)
#add another cookie
driver.add_cookie({"name": "another_cookie", "value": "we added a second cookie!"})
#update our list
cookies = driver.get_cookies()
#print the cookies
print("There should be two cookies here:", cookies)
#delete all cookies
driver.delete_all_cookies()
#update our list
cookies = driver.get_cookies()
#print the list
print("Cookies after deleting all:", cookies)
driver.quit()
Understanding Cookies
Let's delve into the fundamentals of cookies to unravel their purpose, functionality, and the implications
they hold for both users and developers.
What are Cookies and What do They Do?
In the section above, we learned that all cookies have a name and a value . There are a number of
other fields that make up a cookie. Let's take a look at some of the information that makes up a cookie:
domain is the website you were at when the cookie was created
name is the name of a cookie
value is the value that the cookie is storing
httpOnly is a boolean. If httpOnly is set to true, client-side scripts cannot access this data
path is the portion of the domain that can access the cookie. If the path is / , this means that it is
on the root path (this cookie can be accessed from the entire domain)
sameSite is also used to restrict access to a cookie. sameSite can be set to None , Lax , or
Strict
secure is another boolean. If secure is set to True , this cookie can only be sent as an encrypted
request to the server using https
The cookies we created earlier used the attributes listed above. While cookies can contain more
information, there are a couple more attributes we need to pay attention to:
session is another boolean that tells the site whether or not our cookie is used to store a session
https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 3/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps
expirationDate or expiry is the UNIX timestamp when a cookie expires
hostOnly is another boolean value. If hostOnly is set to False , subdomains of this site can also
access the cookie.
Types of Cookies
Cookies serve various purposes and can be categorized based on their functions, lifespan, and origin.
Here are some common types of cookies:
Session Cookies are stored and used to manage a user session. When you close a browser, Session
Cookies are removed when you close the browser. These cookies are used specifically to store
information about your browsing session such as pages you visited.
Persistent Cookies are stored on a device even after the browser is closed. These cookies typically store
things like login information and user preferences on a site.
How are Cookies Stored and Accessed by the Browser?
As we did earlier, we can retrieve cookies using Selenium. You can also view cookies stored from right
inside your browser.
While not necessarily relevant to scraping, this is important for anyone who cares about their data when
browsing the web.
To view cookies in Chrome follow the steps below:
click the icon to the left of your url
After clicking this button, select Cookies and Site Data
https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 4/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps
Select Manage cookies and site data
Once you've selected to manage your cookies, you get a pop-up with a list of sites that can access
your data
https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 5/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps
To delete a cookie, simply click the trashcan icon
Why Manage Cookies in Selenium?
There are many reasons to manage our cookies in Selenium. You can use them for managing
credentials and for managing how your automated browser is tracked. Any type of stored browser
information typically gets stored in cookies.
With proper cookie management, we can keep logged into a site or we can clear all information from a
site and log out much more quickly and effectively. We can also remove cookies to make it more
difficult to track our browser.
While we use Selenium primarily for scraping, there are many developers who use it to automate testing
on their sites.
When using Selenium for automated site tests, the ability to quickly manage cookies makes the login
and logout process much more efficient (just like it does for us when scraping). Imagine writing unit
tests for a social media site and having to hard code user authentication every time you need to do
something different!
Cookie management is a critical skill when dealing with the following scenarios:
User Authentication: logging into a website
Session Handling: remembering specifics about a browsing session
Site with persistent login: sites that "remember" you when you revisit the site
Testing site permissions: when testing access points on a site, it is imperative that only permitted
users get to access sensitive data. Selenium allows developers to automate tests around this and
ensure that their sites are functioning properly.
Managing dynamic content: when viewing a site with dynamic content, the content generated by
the site can often be generated by information stored in the browser, such as a store's website
saving your location to display your nearest store
https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 6/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps
Selenium Query Commands for Cookies
In Selenium, you can use various commands and methods to interact with cookies. Here's a list of
common Selenium WebDriver commands for managing cookies in Python:
Selenium Query Commands Output
driver.get_cookies(); Return The List of all Cookies
Return specific cookie according
driver.get_cookie(name_of_cookie)
to name
driver.add_cookie({"name": name_of_cookie, "value":
Create and add the cookie
value_of_cookie})
driver.delete_cookie(name_of_cookie) Delete specific cookie
driver.delete_all_cookies() Delete all cookies
Methods for Handling With Cookies
Here are some common methods and operations for handling cookies in Selenium:
Retrieving cookies
The process of retrieving cookies involves obtaining information about the cookies currently stored for a
specific domain in the browser. This can include fetching all cookies or getting details about a particular
cookie by its name.
from selenium import webdriver
#open chrome
driver = webdriver.Chrome()
#navigate to the url
driver.get("https://quotes.toscrape.com")
#create a variable to hold our cookies
cookies = driver.get_cookies()
#print the cookies list
print("Cookies:", cookies)
https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 7/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps
#close the browser
driver.quit()
In the code above, we:
Open Chrome and navigate to the page with webdriver.Chrome() and driver.get()
Create a variable, cookies , using driver.get_cookies()
Print cookies to the terminal so we can see them
Adding cookies
Adding cookies is the action of setting new cookies in the browser for a specific domain.
from selenium import webdriver
#open chrome
driver = webdriver.Chrome()
#navigate to the url
driver.get("https://quotes.toscrape.com")
#add a new cookie
driver.add_cookie({"name": "new_cookie", "value": "added_cookie"})
#create a variable to hold our cookies
cookies = driver.get_cookies()
#print the cookies list
print("Cookies:", cookies)
#close the browser
driver.quit()
In this code, we:
Open a browser and navigate to the url with webdriver.Chrome() and driver.get()
Add a cookie with the name of new_cookie and the value of added_cookie
Get our list of cookies with driver.get_cookies()
Print the list so we can see the new cookie we just added
Deleting cookies
The deletion of cookies involves removing either a specific cookie or all cookies associated with a
particular domain from the browser. This action helps manage the state and session information stored
in cookies.
from selenium import webdriver
#open chrome
https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 8/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps
driver = webdriver.Chrome()
#navigate to the url
driver.get("https://quotes.toscrape.com")
#add a new cookie
driver.add_cookie({"name": "new_cookie", "value": "added_cookie"})
#create a variable to hold our cookies
cookies = driver.get_cookies()
#print the cookies list
print("Cookies after adding:", cookies)
#delete the new cookie
driver.delete_cookie("new_cookie")
#update the cookies variable
cookies = driver.get_cookies()
print("Cookies after deleting:", cookies)
#close the browser
driver.quit()
In this example, we:
Open a browser and navigate to the url with webdriver.Chrome() and driver.get()
Add a cookie with driver.add_cookie({"name": "new_cookie", "value": "added_cookie"})
Create a cookies variable and print our cookies list to the terminal
Delete the cookie with driver.delete_cookie("new_cookie")
Print the newly emptied cookies to the terminal
Modifying cookies (by changing key-value pairs)
Modifying cookies encompasses altering the values or properties of existing cookies, such as changing
the key-value pairs.
This process involves retrieving a specific cookie, creating a new cookie with the desired modifications,
and then adding the modified cookie back to the browser.
from selenium import webdriver
#open chrome
driver = webdriver.Chrome()
#navigate to the url
driver.get("https://quotes.toscrape.com")
#add a new cookie
driver.add_cookie({"name": "new_cookie", "value": "added_cookie"})
https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 9/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps
#create a variable to hold our cookies
cookies = driver.get_cookies()
#print the cookies list
print("Cookies after adding:", cookies)
#iterate through our cookies
for cookie in cookies:
#find the cookie we just created
if cookie["name"] == "new_cookie":
#change the value of the cookie in the list
cookie["value"] = "modified_cookie"
#delete the original cookie
driver.delete_cookie(cookie["name"])
#replace it with the modified one
driver.add_cookie(cookie)
#update the cookies variable
cookies = driver.get_cookies()
print("Cookies after deleting:", cookies)
#close the browser
driver.quit()
In the code above, we:
Open an instance of Chrome and navigate to the page with webdriver.Chrome() and
driver.get()
Add a cookie with driver.add_cookie()
Print the cookies list
Iterate through our cookies list and find the cookie with the name we're searching for
Once we've found our cookie in the list, we modify the value
Remove the original cookie with driver.delete_cookie() and replace it with the modified cookie
with driver.add_cookie()
Print the cookies to the terminal so we can view our modified cookie
Handling Cookie Attributes In Selenium
Anytime we wish to edit cookie attributes, we do this by changing the key-value pairs. Our cookies are
stored in a JSON object or as we would say in Python, a dictionary.
https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 10/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps
In the example above, when we changed the value of the cookie,could have changed other attributes
(keys) and their values.
To change the expiry date, instead of: cookie["value"] = "modified_cookie , we could use
cookie["expirationDate"] = some_new_timestamp
If we wanted to change the domain name and path, we could do cookie["domain"] =
some_new_domain
To change the path, we would use cookie["path"] = "/our/new/path/
To use the secure flag, we would use cookie["secure"] = True
To set httpOnly , we would use cookie["httpOnly"] = True
Working with Multiple Cookies In Selenium
As we've gone over in the sections above, we can use driver.get_all_cookies() to retrieve all of our
cookies. This method gets each cookie in the current browser and returns all the cookies in the form of a
list.
If we wish to filter cookies based on certain data, we iterate through our list and we looking for a certain
attribute to target. Take a look at the snippet below that we used in one of the previous sections of this
article:
for cookie in cookies:
#find the cookie we just created
if cookie["name"] == "new_cookie":
#change the value of the cookie in the list
cookie["value"] = "modified_cookie"
#delete the original cookie
driver.delete_cookie(cookie["name"])
#replace it with the modified one
driver.add_cookie(cookie)
We iterate through the list and use if cookie["name"] == "new_cookie" to find our target. If we wanted
all the cookies from quotes.toscrape.com, we could do this:
for cookie in cookies:
#find the cookie we just created
if cookie["domain"] == "quotes.toscrape.com":
https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 11/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps
#change the value of the cookie in the list
cookie["httpOnly"] = True
#delete the original cookie
driver.delete_cookie(cookie["name"])
#replace it with the modified one
driver.add_cookie(cookie)
We've now modified this snippet to take all cookies from quotes.toscrape.com and switch httpOnly to
True .
Advanced Cookie Operations
When using cookies for login and authentication, it is important to save them. If you wish to use them
longterm, you may even want to save them to a file. Once our cookies are loaded into the brower, we
can usually just refresh the page and we'll be logged in automatically.
While you need to push different buttons to get cookies using your normal browser. Selenium's builtin
cookie methods give you a universal way to create, read, update and delete your cookies no matter
which browser Selenium is controlling. To remove all cookies, we can simply use the
delete_all_cookies() method and this will clear the browser cache.
When managing cookies across multiple domains, we can simply filter through our cookie list and use
key-value pairs to target a specific domain. If we want only cookies from a certain site, we can use
get_cookies() to retrieve all our cookies and iterate through it with a for loop to target our cookies by
domain name.
Troubleshooting Cookie-Related Issues
Common challenges when handling cookies are typically related to expiration or adding cookies from a
previous into our current one.
In order to properly handle any exceptions thrown, it is best to use try to attempt to add a cookie into
the browser and a catch statement to handle any exceptions that may get thrown.
This combination of try and catch allows our script to continue execution should any errors occur. If
you are experiencing any performance issues, it is always a good idea clear your cookies.
If you wish to login to certain sites effectivley and efficiently, you may want to save your cookies to a file,
this way you can come back and read them any time you need to.
https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 12/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps
Cookie Handling in Production
In the example below, we're going to take an in-depth look at how we can use cookies for
authentication.
from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep
#open chrome
driver = webdriver.Chrome()
#navigate to the url
driver.get("https://www.linkedin.com")
#save username and password
username = "YOUR-USERNAME@SOME_EMAIL.com"
password = "YOUR-SUPER-SECRET-PASSWORD"
#find the username element
username_id = driver.find_element(By.ID, "session_key")
#find the password element
password_id = driver.find_element(By.ID, "session_password")
#fill in the username
username_id.send_keys(username)
#fill in the password
password_id.send_keys(password)
#xpath of the signin button
signin_button_xpath = '//*[@id="main-content"]/section[1]/div/div/form/div[2]/button'
#find the signin button by xpath
signin_button = driver.find_element(By.XPATH, signin_button_xpath)
#click the signin button
signin_button.click()
#create a variable to hold our cookies
cookies = driver.get_cookies()
#close the browser
driver.close()
#open a new browser
driver = webdriver.Chrome()
https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 13/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps
#go back to the site
driver.get("https://linkedin.com")
#add cookies from the previous section
for cookie in cookies:
print(cookie)
driver.add_cookie(cookie)
#sleep so user can see we're not logged in
sleep(5)
#refresh the page
driver.refresh()
#sleep so user can see we're logged in
sleep(5)
#close the browser
driver.quit()
In the example above, we:
Open Chrome and navigate to the page with webdriver.Chrome() and driver.get()
Next we find the username and password objects using their ID
We fill these objects with the username and password variables using the send_keys() method
Next we use XPATH to find and click() the login button
After signing in, we retrieve all of our cookies with driver.get_cookies() and save them as a
variable
Next, we close the browser with driver.close()
We then open up a new browser and head back to the page
We add our cookies from the previous session to the current session with driver.add_cookie()
We sleep() for 5 seconds just so you can see that we are not logged in yet
We then use driver.refresh() and you can see that we're logged in
Now let's use cookie management to handle a SPA (Single Page Application). Believe it or not, Github is
a single page application. Let's use cookies to interact with Github!
from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep
driver = webdriver.Chrome()
#navigate to the url
driver.get("https://www.github.com/login")
#save username input box as variable
username_input = driver.find_element(By.NAME, "login")
https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 14/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps
#save password input box as variable
password_input = driver.find_element(By.NAME, "password")
#send email to username_input
username_input.send_keys("
[email protected]")
#send password to password_input
password_input.send_keys("YOUR-PASSWORD-HERE")
#find the signin button
signin_button = driver.find_element(By.NAME, "commit")
#click the signin button
signin_button.click()
#create a bool to know if we're logged in
logged_in = False
#loop until the user says we're logged in
while not logged_in:
print("Are you logged in? y/n")
resp = input()
if resp == "y":
logged_in = True
#create a variable to hold our cookies
cookies = driver.get_cookies()
#close the browser
driver.close()
#open a new browser
driver = webdriver.Chrome()
#go back to the site
driver.get("https://github.com")
#add cookies from the previous section
for cookie in cookies:
print(cookie)
try:
driver.add_cookie(cookie)
except:
print("failed to add cookie")
#sleep so user can see we're not logged in
sleep(5)
#refresh the page
driver.refresh()
https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 15/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps
#sleep so user can see we're logged in
sleep(5)
#close the browser
driver.quit()
Github uses 2 Factor Authentication, when running the script above, make sure to login using your 2FA
before you input y to tell your script that you've logged in.
This script stops and pauses when you need to enter your 2FA. We input y to let the script know that
you've finished entering your 2FA and that the script is ready to continue its execution.
In the example above, we:
Open an instance of Chrome and navigate to the site
We use Selenium's driver.find_element() in order to find our username_input and
password_input by NAME and save them as variables
We then use the send_keys() method to fill the required input fields
After filling the inputs, we find the signin_button using find_element(By.NAME, "commit")
We click signin_button with signin_button.click()
After we click() to sign in, we create a loop that exits once we've told the bot that we're logged in
Once logged in, we save our cookies as a variable with the get_cookies() method
After we've saved our cookies, we navigate back to the page and try to add them all back in, we
use except to handle any exceptions
After adding the cookies back in, we refresh the page, and we're back into Github at lightning fast
speed
Our page loads so quickly because it's a Single Page Appliction and everything is handled
dynamically based on the the cookies stored inside the browser
Conclusion
You've reached the end of this article. You should have a solid grasp on adding, deleting, and even
modifying cookies. You now know how to save them as variables and use them in future sessions.
You've even logged into LinkedIn and Github using cookies! Now that you've harnessed the power of
cookies in Selenium, go use your knowledge to scrape something cool!
Want to know more about scraping in Python? Checkout the ScrapeOps Python Playbook
To know more about Selenium, take a look at the Selenium Documentation
More Selenium Articles
Want to learn more, but you're unsure where to start? Read one of the articles below!
Python Pyppeteer
Rotating Proxies in Python
https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 16/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps
Scraping With BeautifulSoup
https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 17/17
