3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps

Selenium Guide: Managing Cookies

Managing cookies is an important aspect of web automation, and Selenium provides methods to
interact with cookies in various ways.

In this comprehensive guide, we will delve into the world of Selenium and explore practical strategies for
handling cookies. We'll be covering:

Introduction
Understanding Cookies
Selenium and Cookies
Cookie Handling in Production
Conclusion
More Cool Articles

Handling Cookies With Selenium

Cookies are used to store information about your browser. They are used to remember your browser
when you visit a website, track you as you navigate a website, and to analyze the usage of a website.

In Selenium, we have a whole set of methods devoted to managing cookies:

https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 1/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps

driver.add_cookie() adds a cookie to our stored cookies

driver.get_cookie(name_of_cookie) returns an individual cookie
driver.get_cookies() returns all cookies that are currently being stored in the browser
driver.delete_cookie(name_of_cookie) will delete a specific cookie from the browser
driver.delete_all_cookies() (as you probably guessed) clears all cookies from the browser

The code below shows how all these methods can be used:

from selenium import webdriver

#open an instance of chrome

driver = webdriver.Chrome()

#navigate to the website

driver.get("https://quotes.toscrape.com")

#get cookies and save as a variable

cookies = driver.get_cookies()

#print the current list of cookies

print("Cookies:", cookies)

#add a cookie
driver.add_cookie({"name": "our new cookie!", "value": "some random text!"})

#update our cookie variable

cookies = driver.get_cookies()

#print our cookies

print("Cookies after adding:", cookies)

#iterate through our list of cookies

for cookie in cookies:
#if this line executes, we found our cookie
if cookie["name"] == "our new cookie!":

print("We have a match!")

cookie["value"] = "we changed our text"

#delete the old cookie

driver.delete_cookie("our new cookie!")

#replace it with the updated cookie

driver.add_cookie(cookie)

#update the cookies variable

cookies = driver.get_cookies()

#print the cookies

print("Cookies after modifying:", cookies

https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 2/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps

)
#add another cookie
driver.add_cookie({"name": "another_cookie", "value": "we added a second cookie!"})

#update our list

cookies = driver.get_cookies()

#print the cookies

print("There should be two cookies here:", cookies)

#delete all cookies

driver.delete_all_cookies()

#update our list

cookies = driver.get_cookies()

#print the list

print("Cookies after deleting all:", cookies)
driver.quit()

Understanding Cookies
Let's delve into the fundamentals of cookies to unravel their purpose, functionality, and the implications
they hold for both users and developers.

What are Cookies and What do They Do?

In the section above, we learned that all cookies have a name and a value . There are a number of
other fields that make up a cookie. Let's take a look at some of the information that makes up a cookie:

domain is the website you were at when the cookie was created
name is the name of a cookie
value is the value that the cookie is storing
httpOnly is a boolean. If httpOnly is set to true, client-side scripts cannot access this data
path is the portion of the domain that can access the cookie. If the path is / , this means that it is
on the root path (this cookie can be accessed from the entire domain)
sameSite is also used to restrict access to a cookie. sameSite can be set to None , Lax , or
Strict
secure is another boolean. If secure is set to True , this cookie can only be sent as an encrypted
request to the server using https

The cookies we created earlier used the attributes listed above. While cookies can contain more
information, there are a couple more attributes we need to pay attention to:

session is another boolean that tells the site whether or not our cookie is used to store a session

https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 3/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps

expirationDate or expiry is the UNIX timestamp when a cookie expires

hostOnly is another boolean value. If hostOnly is set to False , subdomains of this site can also
access the cookie.

Types of Cookies

Cookies serve various purposes and can be categorized based on their functions, lifespan, and origin.
Here are some common types of cookies:

Session Cookies are stored and used to manage a user session. When you close a browser, Session
Cookies are removed when you close the browser. These cookies are used specifically to store
information about your browsing session such as pages you visited.

Persistent Cookies are stored on a device even after the browser is closed. These cookies typically store
things like login information and user preferences on a site.

How are Cookies Stored and Accessed by the Browser?

As we did earlier, we can retrieve cookies using Selenium. You can also view cookies stored from right
inside your browser.

While not necessarily relevant to scraping, this is important for anyone who cares about their data when
browsing the web.

To view cookies in Chrome follow the steps below:

click the icon to the left of your url

After clicking this button, select Cookies and Site Data

https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 4/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps

Select Manage cookies and site data

Once you've selected to manage your cookies, you get a pop-up with a list of sites that can access
your data

https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 5/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps

To delete a cookie, simply click the trashcan icon

Why Manage Cookies in Selenium?

There are many reasons to manage our cookies in Selenium. You can use them for managing
credentials and for managing how your automated browser is tracked. Any type of stored browser
information typically gets stored in cookies.

With proper cookie management, we can keep logged into a site or we can clear all information from a
site and log out much more quickly and effectively. We can also remove cookies to make it more
difficult to track our browser.

While we use Selenium primarily for scraping, there are many developers who use it to automate testing
on their sites.

When using Selenium for automated site tests, the ability to quickly manage cookies makes the login
and logout process much more efficient (just like it does for us when scraping). Imagine writing unit
tests for a social media site and having to hard code user authentication every time you need to do
something different!

Cookie management is a critical skill when dealing with the following scenarios:

User Authentication: logging into a website

Session Handling: remembering specifics about a browsing session
Site with persistent login: sites that "remember" you when you revisit the site
Testing site permissions: when testing access points on a site, it is imperative that only permitted
users get to access sensitive data. Selenium allows developers to automate tests around this and
ensure that their sites are functioning properly.
Managing dynamic content: when viewing a site with dynamic content, the content generated by
the site can often be generated by information stored in the browser, such as a store's website
saving your location to display your nearest store

https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 6/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps

Selenium Query Commands for Cookies

In Selenium, you can use various commands and methods to interact with cookies. Here's a list of
common Selenium WebDriver commands for managing cookies in Python:

Selenium Query Commands Output

driver.get_cookies(); Return The List of all Cookies

Return specific cookie according

driver.get_cookie(name_of_cookie)
to name

driver.add_cookie({"name": name_of_cookie, "value":

Create and add the cookie
value_of_cookie})

driver.delete_cookie(name_of_cookie) Delete specific cookie

driver.delete_all_cookies() Delete all cookies

Methods for Handling With Cookies

Here are some common methods and operations for handling cookies in Selenium:

Retrieving cookies
The process of retrieving cookies involves obtaining information about the cookies currently stored for a
specific domain in the browser. This can include fetching all cookies or getting details about a particular
cookie by its name.

from selenium import webdriver

#open chrome
driver = webdriver.Chrome()

#navigate to the url

driver.get("https://quotes.toscrape.com")

#create a variable to hold our cookies

cookies = driver.get_cookies()

#print the cookies list

print("Cookies:", cookies)

https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 7/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps

#close the browser

driver.quit()

In the code above, we:

Open Chrome and navigate to the page with webdriver.Chrome() and driver.get()
Create a variable, cookies , using driver.get_cookies()
Print cookies to the terminal so we can see them

Adding cookies

Adding cookies is the action of setting new cookies in the browser for a specific domain.

from selenium import webdriver

#open chrome
driver = webdriver.Chrome()

#navigate to the url

driver.get("https://quotes.toscrape.com")

#add a new cookie

driver.add_cookie({"name": "new_cookie", "value": "added_cookie"})

#create a variable to hold our cookies

cookies = driver.get_cookies()

#print the cookies list

print("Cookies:", cookies)

#close the browser

driver.quit()

In this code, we:

Open a browser and navigate to the url with webdriver.Chrome() and driver.get()
Add a cookie with the name of new_cookie and the value of added_cookie
Get our list of cookies with driver.get_cookies()
Print the list so we can see the new cookie we just added

Deleting cookies

The deletion of cookies involves removing either a specific cookie or all cookies associated with a
particular domain from the browser. This action helps manage the state and session information stored
in cookies.

from selenium import webdriver

#open chrome
https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 8/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps

driver = webdriver.Chrome()

#navigate to the url

driver.get("https://quotes.toscrape.com")

#add a new cookie

driver.add_cookie({"name": "new_cookie", "value": "added_cookie"})

#create a variable to hold our cookies

cookies = driver.get_cookies()

#print the cookies list

print("Cookies after adding:", cookies)

#delete the new cookie

driver.delete_cookie("new_cookie")

#update the cookies variable

cookies = driver.get_cookies()
print("Cookies after deleting:", cookies)

#close the browser

driver.quit()

In this example, we:

Open a browser and navigate to the url with webdriver.Chrome() and driver.get()
Add a cookie with driver.add_cookie({"name": "new_cookie", "value": "added_cookie"})
Create a cookies variable and print our cookies list to the terminal
Delete the cookie with driver.delete_cookie("new_cookie")
Print the newly emptied cookies to the terminal

Modifying cookies (by changing key-value pairs)

Modifying cookies encompasses altering the values or properties of existing cookies, such as changing
the key-value pairs.

This process involves retrieving a specific cookie, creating a new cookie with the desired modifications,
and then adding the modified cookie back to the browser.

from selenium import webdriver

#open chrome
driver = webdriver.Chrome()

#navigate to the url

driver.get("https://quotes.toscrape.com")

#add a new cookie

driver.add_cookie({"name": "new_cookie", "value": "added_cookie"})

https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 9/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps

#create a variable to hold our cookies

cookies = driver.get_cookies()

#print the cookies list

print("Cookies after adding:", cookies)

#iterate through our cookies

for cookie in cookies:
#find the cookie we just created
if cookie["name"] == "new_cookie":

#change the value of the cookie in the list

cookie["value"] = "modified_cookie"

#delete the original cookie

driver.delete_cookie(cookie["name"])

#replace it with the modified one

driver.add_cookie(cookie)

#update the cookies variable

cookies = driver.get_cookies()
print("Cookies after deleting:", cookies)

#close the browser

driver.quit()

In the code above, we:

Open an instance of Chrome and navigate to the page with webdriver.Chrome() and
driver.get()
Add a cookie with driver.add_cookie()
Print the cookies list
Iterate through our cookies list and find the cookie with the name we're searching for
Once we've found our cookie in the list, we modify the value
Remove the original cookie with driver.delete_cookie() and replace it with the modified cookie
with driver.add_cookie()
Print the cookies to the terminal so we can view our modified cookie

Handling Cookie Attributes In Selenium

Anytime we wish to edit cookie attributes, we do this by changing the key-value pairs. Our cookies are
stored in a JSON object or as we would say in Python, a dictionary.

https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 10/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps

In the example above, when we changed the value of the cookie,could have changed other attributes
(keys) and their values.

To change the expiry date, instead of: cookie["value"] = "modified_cookie , we could use
cookie["expirationDate"] = some_new_timestamp

If we wanted to change the domain name and path, we could do cookie["domain"] =

some_new_domain

To change the path, we would use cookie["path"] = "/our/new/path/

To use the secure flag, we would use cookie["secure"] = True

To set httpOnly , we would use cookie["httpOnly"] = True

Working with Multiple Cookies In Selenium

As we've gone over in the sections above, we can use driver.get_all_cookies() to retrieve all of our
cookies. This method gets each cookie in the current browser and returns all the cookies in the form of a
list.

If we wish to filter cookies based on certain data, we iterate through our list and we looking for a certain
attribute to target. Take a look at the snippet below that we used in one of the previous sections of this
article:

for cookie in cookies:

#find the cookie we just created
if cookie["name"] == "new_cookie":

#change the value of the cookie in the list

cookie["value"] = "modified_cookie"

#delete the original cookie

driver.delete_cookie(cookie["name"])

#replace it with the modified one

driver.add_cookie(cookie)

We iterate through the list and use if cookie["name"] == "new_cookie" to find our target. If we wanted
all the cookies from quotes.toscrape.com, we could do this:

for cookie in cookies:

#find the cookie we just created
if cookie["domain"] == "quotes.toscrape.com":

https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 11/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps

#change the value of the cookie in the list

cookie["httpOnly"] = True

#delete the original cookie

driver.delete_cookie(cookie["name"])

#replace it with the modified one

driver.add_cookie(cookie)

We've now modified this snippet to take all cookies from quotes.toscrape.com and switch httpOnly to
True .

Advanced Cookie Operations

When using cookies for login and authentication, it is important to save them. If you wish to use them
longterm, you may even want to save them to a file. Once our cookies are loaded into the brower, we
can usually just refresh the page and we'll be logged in automatically.

While you need to push different buttons to get cookies using your normal browser. Selenium's builtin
cookie methods give you a universal way to create, read, update and delete your cookies no matter
which browser Selenium is controlling. To remove all cookies, we can simply use the
delete_all_cookies() method and this will clear the browser cache.

When managing cookies across multiple domains, we can simply filter through our cookie list and use
key-value pairs to target a specific domain. If we want only cookies from a certain site, we can use
get_cookies() to retrieve all our cookies and iterate through it with a for loop to target our cookies by
domain name.

Troubleshooting Cookie-Related Issues

Common challenges when handling cookies are typically related to expiration or adding cookies from a
previous into our current one.

In order to properly handle any exceptions thrown, it is best to use try to attempt to add a cookie into
the browser and a catch statement to handle any exceptions that may get thrown.

This combination of try and catch allows our script to continue execution should any errors occur. If
you are experiencing any performance issues, it is always a good idea clear your cookies.

If you wish to login to certain sites effectivley and efficiently, you may want to save your cookies to a file,
this way you can come back and read them any time you need to.

https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 12/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps

Cookie Handling in Production

In the example below, we're going to take an in-depth look at how we can use cookies for
authentication.

from selenium import webdriver

from selenium.webdriver.common.by import By
from time import sleep

#open chrome
driver = webdriver.Chrome()

#navigate to the url

driver.get("https://www.linkedin.com")

#save username and password

username = "YOUR-USERNAME@SOME_EMAIL.com"
password = "YOUR-SUPER-SECRET-PASSWORD"

#find the username element

username_id = driver.find_element(By.ID, "session_key")

#find the password element

password_id = driver.find_element(By.ID, "session_password")

#fill in the username

username_id.send_keys(username)

#fill in the password

password_id.send_keys(password)

#xpath of the signin button

signin_button_xpath = '//*[@id="main-content"]/section[1]/div/div/form/div[2]/button'

#find the signin button by xpath

signin_button = driver.find_element(By.XPATH, signin_button_xpath)

#click the signin button

signin_button.click()

#create a variable to hold our cookies

cookies = driver.get_cookies()

#close the browser

driver.close()

#open a new browser

driver = webdriver.Chrome()

https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 13/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps

#go back to the site

driver.get("https://linkedin.com")

#add cookies from the previous section

for cookie in cookies:
print(cookie)
driver.add_cookie(cookie)

#sleep so user can see we're not logged in

sleep(5)

#refresh the page

driver.refresh()

#sleep so user can see we're logged in

sleep(5)

#close the browser

driver.quit()

In the example above, we:

Open Chrome and navigate to the page with webdriver.Chrome() and driver.get()
Next we find the username and password objects using their ID
We fill these objects with the username and password variables using the send_keys() method
Next we use XPATH to find and click() the login button
After signing in, we retrieve all of our cookies with driver.get_cookies() and save them as a
variable
Next, we close the browser with driver.close()
We then open up a new browser and head back to the page
We add our cookies from the previous session to the current session with driver.add_cookie()
We sleep() for 5 seconds just so you can see that we are not logged in yet
We then use driver.refresh() and you can see that we're logged in

Now let's use cookie management to handle a SPA (Single Page Application). Believe it or not, Github is
a single page application. Let's use cookies to interact with Github!

from selenium import webdriver

from selenium.webdriver.common.by import By
from time import sleep

driver = webdriver.Chrome()

#navigate to the url

driver.get("https://www.github.com/login")

#save username input box as variable

username_input = driver.find_element(By.NAME, "login")

https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 14/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps

#save password input box as variable

password_input = driver.find_element(By.NAME, "password")

#send email to username_input

username_input.send_keys("[email protected]")

#send password to password_input

password_input.send_keys("YOUR-PASSWORD-HERE")

#find the signin button

signin_button = driver.find_element(By.NAME, "commit")

#click the signin button

signin_button.click()

#create a bool to know if we're logged in

logged_in = False

#loop until the user says we're logged in

while not logged_in:
print("Are you logged in? y/n")
resp = input()
if resp == "y":
logged_in = True

#create a variable to hold our cookies

cookies = driver.get_cookies()

#close the browser

driver.close()

#open a new browser

driver = webdriver.Chrome()

#go back to the site

driver.get("https://github.com")

#add cookies from the previous section

for cookie in cookies:
print(cookie)
try:
driver.add_cookie(cookie)
except:
print("failed to add cookie")

#sleep so user can see we're not logged in

sleep(5)

#refresh the page

driver.refresh()

https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 15/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps

#sleep so user can see we're logged in

sleep(5)

#close the browser

driver.quit()

Github uses 2 Factor Authentication, when running the script above, make sure to login using your 2FA
before you input y to tell your script that you've logged in.

This script stops and pauses when you need to enter your 2FA. We input y to let the script know that
you've finished entering your 2FA and that the script is ready to continue its execution.

In the example above, we:

Open an instance of Chrome and navigate to the site

We use Selenium's driver.find_element() in order to find our username_input and
password_input by NAME and save them as variables
We then use the send_keys() method to fill the required input fields
After filling the inputs, we find the signin_button using find_element(By.NAME, "commit")
We click signin_button with signin_button.click()
After we click() to sign in, we create a loop that exits once we've told the bot that we're logged in
Once logged in, we save our cookies as a variable with the get_cookies() method
After we've saved our cookies, we navigate back to the page and try to add them all back in, we
use except to handle any exceptions
After adding the cookies back in, we refresh the page, and we're back into Github at lightning fast
speed
Our page loads so quickly because it's a Single Page Appliction and everything is handled
dynamically based on the the cookies stored inside the browser

Conclusion
You've reached the end of this article. You should have a solid grasp on adding, deleting, and even
modifying cookies. You now know how to save them as variables and use them in future sessions.

You've even logged into LinkedIn and Github using cookies! Now that you've harnessed the power of
cookies in Selenium, go use your knowledge to scrape something cool!

Want to know more about scraping in Python? Checkout the ScrapeOps Python Playbook
To know more about Selenium, take a look at the Selenium Documentation

More Selenium Articles

Want to learn more, but you're unsure where to start? Read one of the articles below!

Python Pyppeteer
Rotating Proxies in Python

https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 16/17
3/31/24, 9:32 PM Python Selenium Guide - Managing Cookies | ScrapeOps

Scraping With BeautifulSoup

https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-managing-cookies/ 17/17