HANDS-ON
PYTHON
 

with Exercises, Projects, Assignments & Final

Exam
 
ADVANCED
 
Musa Arda
 
OceanofPDF.com
 
 
Hands-On Python with Exercises,
Projects, Assignments & Final Exam:
Advanced
 
By Musa Arda
 
Copyright © 2022 Musa Arda.
 
All rights reserved. No portion of this book
or its supplementary materials may be
reproduced in any form without
permission from the publisher, except as
permitted by U.S. copyright law.
 
For permissions contact:
[email protected]
 
OceanofPDF.com
 
Contents
 
Preface
1. Introduction
2. IDE - PyCharm Basics
3. Collections
4. Iterators
5. Generators
6. Date and Time
7. Decorators
8. Context Managers
9. Functional Programming in Python
10. Project 1 – Sending Emails with
Python
11. Assignment 1 – Sending Emails with
Python
12. Regular Expressions
13. Database Operations
14. Concurrency
15. Project 2 – Web Scraping with
Python
 16. Assignment 2 – Web Scraping with
Python
17. Project 3 – API Development with
Flask
18. Assignment 3 – API Development
with Flask
19. Final Exam
20. Conclusion
OceanofPDF.com
 Preface
 
OceanofPDF.com
 About This Book
 
This book is an in-depth and activity-
based introduction to the advanced level
topics of Python programming. It follows a
step-by-step practical approach by
combining the theory of the language with
the hands-on coding exercises including
quizzes, projects, assignments and exams.
We begin by introducing the Collections
module in Python. Then we cover
Iterators, Generators, Date and Time
Operations, Decorators and Context
Managers in Python. We move on with
Functional Programming in Python and we
have our first project which is about
sending emails using Python. Then we
learn Regular Expressions, Database
Operations and Concurrency in Python.
After these topics, we build two projects.
The second project in this book is Web
Scraping with Python and Scrapy. And the
third project is Developing Web APIs with
Flask. After each project, you have an
assignment to complete. And finally, we
have the Final Exam.
By the end of the book, you will learn
almost all of the advanced level concepts
of Python in great detail by writing
thousands of lines of code. All the
supplementary resources (code files,
quizzes, assignments, final exam etc.)
are available for download at the GitHub
repository. The link for the repository is
provided in the book.
This is the third book in Hands-On
Python Series. And here is what you will
find in this book:
Theory: In each topic, we will cover all
the Theoretical Details with example
coding.
Coding Exercises: At the end of each
chapter, we will have Coding Exercise,
Quizzes.
Projects: We will build projects in this
book. You will learn how to apply Python
concepts on real world problems.
Assignments: After each project, you
will have an Assignment. These
assignments will let you build the project
from scratch on your own.
Final Exam: At the end of this book,
you will have the Final Exam. It is a
multiple-choice exam with 20 questions
and a limited duration. The exam will let
you to test your Python level.
 
 
OceanofPDF.com
 About Hands-On Python Series
 
This is the third book in our Hands-On
Python Series. And it covers the advanced
level topics. So, we assume that you
already know the introductory and
intermediate concepts of Python
programming like; Variables, Functions,
Conditional Statements, Loops, Strings,
Lists, Dictionaries, Tuples, Sets and
Comprehensions, Exception Handling, File
Operations and OOP. If you don’t feel
comfortable with these topics, you are
strongly recommended to finish the first
two books in our series, which are the
Beginner and Intermediate levels. Here,
you can find them.[1]
 
 
OceanofPDF.com
 About the Author
 
Musa Arda has Bachelor’s degree from
Industrial Engineering in 2007, and he has
been working as a Software Developer for
more than 14 years. He  uses many
programming languages and platforms
like; Python, C#, Unity, Java, JavaScript,
SAP ABAP, SQL, React, Flutter and more.
He creates online learning content and
writes books to share his experience and
knowledge with his students. His main
purpose is to combine theory and hands-
on practice in his teaching resources.
That's why there are hundreds of
programming exercises, quizzes, tests,
projects, exams and assignments in all of
his courses and books. He is dedicated to
help his students to learn programming
concepts in depth via a practical and
exiting way.
 
 
OceanofPDF.com
 How to Contact Us
 
Please feel free to get in contact with
the author. To comment or ask technical
questions about this book, you can send
email to
[email protected].
 
OceanofPDF.com
1. Introduction
     

 
OceanofPDF.com
 Who Is This Book For?
 
The goal of this book is to help students
to learn Python programming language in
a hands-on and project-based manner.
With its unique style of combining
theory and practice, this book is for:
people who want to learn and
practice advanced concepts in
Python programming
people who are already working with
Python language
 
 
OceanofPDF.com
 What Can You Expect to Learn?
 
The purpose of this book is to provide
you a good introduction to the advanced
topics of Python programming. In general,
you will gain solid programming skills and
grasp the main idea of software
development. In particular, here are some
highlights on what you can you expect to
learn with this book.
You will:
learn & master advanced Python
topics in a hands-on approach
practice your Python knowledge with
Quizzes and Coding Exercises
build Real-World Project with Python
and do Assignments related to these
projects
take the Final Exam on Python with
20 questions to assess your learning
build Python applications with
PyCharm and master it
gain solid and profound Python
Programming skills needed for a
Python career
  
 
OceanofPDF.com
 Outline of This Book
 
In Chapter 1, you will get the basics of
the book. You will learn about this books
approach to Python programming and how
to get most out of it.
In Chapter 2, you will meet PyCharm,
our IDE (Integrated Development
Environment) for this book. You will learn
all the basics of PyCharm and how to do
debugging with it. You will also learn the
base interpreter and interpreter
configuration.
In Chapter 3, you will learn the
Collections module in Python. Collections
are specialized container datatypes
providing alternatives to Python’s general-
purpose containers, dict, list, set, and
tuple. You will learn; ChainMap, Counter,
Deque, DefaultDict, NamedTuple,
OrderedDict, UserDict, UserList, and
UserString.
In Chapter 4, you will learn Iterables and
Iterators in Python. You will see the details
of Iterator Protocol, how to loop through
an Iterator, how to define custom Iterators
and Infinite Iterators.
Chapter 5 is on Generators in Python.
Generators allow you to define Iterators
more easily and efficiently. You will define
custom Generators and learn the benefits
of using them.
In Chapter 6, you will learn all the
details of Date and Time operations in
Python, which are crucial for robust
application development. You will learn
the difference between Aware and Naive
Objects and get the details of the classes
in datetime module.
In Chapter 7, you will meet Decorators.
A very important concept in Python
programming. You will learn how to define
a decorator, how to chain them and how
to use class syntax for creating new ones.
In Chapter 8, you will learn the details of
Context Managers, which are very handy
tools when you need to deal with resource
management in your code. You will see
how to define and use a Context Manager
in both class form and function form.
 Chapter 9 is about Functional
Programming in Python. We will briefly
talk about Functional Programming
paradigm and then you will learn the basic
building blocks of this paradigm in Python.
You will learn how to use lambda, map(),
filter() and reduce() functions.
Chapter 10 will be the first Project in
this book. You will learn how to send
emails using Python. You will set a local
mail server to send emails on your local
machine. Then you will set a fake remote
mail server to send and test emails.
Finally, you will learn how to use a Gmail
account to send real emails.
In Chapter 11, you will have an
Assignment on the Sending Email Project.
You will have #TODO directives to develop
the same project. You will be able to write
your own code, to finish the tasks.
In Chapter 12, you will learn one of the
fundamental concepts in programming,
which is Regular Expressions (RE). You will
learn how to define RE patterns and how
to search a given text based on this
pattern. You will learn Python re module,
and see lots of use cases with coding
exercises.
Chapter 13 is dedicated to the Database
Operations in Python. You will learn how to
install a local MySQL Server, how to create
databases and tables, how to insert
records in these tables. You will also see
how to use some common SQL operations
like Select, Insert, Update and Delete in
Python code.
In Chapter 14, you will learn how to
handle Concurrency in Python. You will
learn about Multithreading and
Multiprocessing and the classes that
Python provides to support them. You will
see how to create and start threads to get
better performance in your applications.
Chapter 15, is the second project in this
book which is Web Scraping with Python.
You will learn how to scrap data out of web
pages using Python and Scrapy.
In Chapter 16, you will have an
Assignment on the Web Scraping Project.
You will have #TODO directives to develop
the same project. You will be able to write
your own code, to finish the tasks.
 In Chapter 17, we have the third project
which is API Development with Flask. You
will learn how to setup and a Flask project
from scratch, create API endpoints using
Blueprints, Views and Templates. We will
create a beautiful movie review web app
using Python and Flask.
In Chapter 18, you will have an
Assignment on the API Development with
Flask Project. You will have #TODO
directives to develop the same project.
You will be able to write your own code, to
finish the tasks.
Chapter 19 is your Final Exam. Now it’s
time to assess your Python level. You will
have an exam of 20 questions on the
topics you learned in this book. The Final
Exam will give you the chance to see what
you learned and which topics you should
repeat.
Chapter 20 is the Conclusion. You will
finalize this book and will see how you
should proceed for the next step.
 
 
OceanofPDF.com
 Conventions Used in This Book
 
The following typographical conventions
are used in this book:
 
Italic
Indicates new URLs, email addresses,
filenames, and file extensions.
 
Bold
Indicates new terms and important
concepts to pay attention.
 
Constant width
Used for string literals and programming
concepts within paragraphs.
 
Constant width bold
Used for program elements within
paragraphs, such as variable or function
names, data types, statements, and
keywords.
 
We will write our code in the code cells.
Here is an example code cell with the cell
number as 7:
 
 Figure 1-1: A code cell example used in this book
 
 
OceanofPDF.com
 Using Code Examples
 
You can find all the supplementary
resources for the book (code files, quizzes,
assignments, final exam etc.) available for
download at the GitHub Repository of the
book which is
https://github.com/musaarda/python-
hands-on-book-advanced.
 
 
OceanofPDF.com
 You & This Book
 
This book is designed in a way that, you
can learn and practice Python. At each
chapter you will learn the basic concepts
and how to use them with examples. Then
you will have a quiz the end of the
chapter. First you will try to solve the quiz
questions on your own, then I will provide
the solutions in detail. You will have
projects after each block of core concepts.
And after every project you will an
assignment to test your understanding.
This will be your path to learn real Python.
Before we deep dive into Python, I want
to give you some tips for how you can get
most out of this book:
Read the topics carefully before you
try to solve the quizzes
Try to code yourself while you are
reading the concepts in the chapters
Try to solve quizzes by yourself,
before checking the solutions
Read the quiz solutions and try to
replicate them
 Code the projects line by line with
the book
Do the assignments (seriously)
Do not start a new chapter before
finishing and solving quiz of the
previous one
Repeat the topics you fail in the Final
Exam
Learning takes time, so give yourself
enough time digest the concepts
 
OceanofPDF.com
 2. IDE - PyCharm Basics
     

 
OceanofPDF.com
 A Brief History of Python
 
Python is a widely used and general-
purpose, high-level programming
language[2]. Its design philosophy
emphasizes code readability and its
syntax allows programmers to express
concepts in fewer lines of code. Python is
dynamically-typed and garbage-collected.
It supports multiple programming
paradigms, including procedural, object-
oriented and functional programming.
It was initially designed by Guido van
Rossum  in the late 1980s and developed
by Python Software Foundation. Python
was named after the BBC TV show Monty
Python's Flying Circus.
In 1991, Van Rossum published the code
labeled version 0.9.0. Then in 1994,
Python reached version 1.0.
Python 2.0 was released in 2000, which
started the Python 2.x family that has
been used widely for almost a decade.
The final release, Python 2.7.18, occurred
on 2020 and marked the end-of-life of
Python 2. As of January 1st, 2020 no new
bug reports, fixes, or changes will be
made to Python 2, and Python 2 is no
longer supported.
Python 3.0 (also called "Python 3000" or
"Py3K") was released on December 3,
2008 and it was a major, backwards-
incompatible release. Since Python 3.0
broke backward compatibility, much
Python 2 code does not run unmodified on
Python 3. Although some old projects still
use Python 2, the preferred approach is
using Python 3. As of this writing, the
latest version is 3.10, but every code we
write in this book should run on Python 3.6
or any later version.
 
 
OceanofPDF.com
 Python Installation
 
To install Python, open the official Python
web site, python.org, in your web browser
and navigate to the Downloads tab.
 

Figure 2-1: Official Python web site

 
Here, you can download the current
version for your operating system. Once you
download the executable file then it is very
straightforward to install Python.
 
 Figure 2-2: The latest version on python.org
 
Run the executable installer to start
installation. You will see the initial screen for
setup. The recommended installation
includes pip, IDLE and documentation. Pip
is the package installer for Python. We will
see it later in this book. IDLE is the built-in
IDE for Python development, but we will not
be using it. We will be using PyCharm.
 
 Figure 2-3: Python setup wizard
 
Make sure you select the Install
launcher for all users and Add Python
3.10 to PATH checkboxes. After selecting
these checkboxes you can click Install Now
button. It will initialize the setup process. It
may take several minutes to finalize
depending on your computer.
 
 Figure 2-4: Python setup progress
When your computer finishes installation
you will see the final screen saying Setup
was successful.
 
 Figure 2-5: Final setup screen
 
There is an important option on the final
screen. It is about the path length limit. In
Windows, the pathnames are limited to 260
characters length. We don’t have that
limitation in macOS or Linux. Python has
introduced an option for disabling path
length limit to avoid path length related
problems. You can select disabling the path
length limit, which is recommended. It will
not affect any other system settings.
Disabling it will resolve potential path length
issues that may arise with projects
developed in Linux or macOS.
  
Verify Python installation:
After the last screen, your Python setup is
completed and you can verify if it was
installed successfully. To verify it, open the
command line on your machine (CMD on
Windows, Terminal on macOS and Linux)
and type python --version . You should see
the installed Python version on your
machine as below:
 

Figure 2-6: python --version command on Windows CMD

 
See where Python is installed:
We have installed Python and we verified
the installation by printing its version on
command line. Now let’s see where it is
installed. In other words, we want to see the
installation folder or folders for Python. You
might have different Python installations on
the same machine. To see the locations,
type where python on the command line as
below:
 

Figure 2-7: Locations of separate Python installations

 
As you see in Figure 2-7 there are multiple
versions of Python installed on my machine.
We can select any of them while creating
our projects in PyCharm. You will see how in
the next sections.
 
 
OceanofPDF.com
 Python Virtual Environments
 
Virtual Environment is an isolated
runtime environment that allows Python
users and applications to install and
upgrade Python packages without
interfering with other Python applications
running on the same system.
Python applications will often use
packages and modules that don’t come as
part of the standard library. Applications will
sometimes need a specific version of a
library, because the application may require
that a particular bug has been fixed or the
application may be written using an
obsolete version of the library’s interface.
This means it may not be possible for one
Python installation to meet the
requirements of every application. If
application A needs version 1.0 of a
particular module but application B needs
version 2.0, then the requirements are in
conflict and installing either version 1.0 or
2.0 will leave one application unable to run.
The solution for this problem is to create a
virtual environment, a self-contained
directory tree that contains a Python
installation for a particular version of
Python, plus a number of additional
packages.
Different applications can then use
different virtual environments. To resolve
the earlier example of conflicting
requirements, application A can have its
own virtual environment with version 1.0
installed while application B has another
virtual environment with version 2.0. If
application B requires a library be upgraded
to version 3.0, this will not affect application
A’s environment.[3]
 

Figure 2-8: Different Virtual Environments on the same OS,

completely isolated from each other
 
We will use separate virtual environments
(venv for short) for each project we create
in this book. That’s the recommended way
for Python development. Don’t worry, you
will see how to create virtual environments
in the next sections.
 
 
OceanofPDF.com
 About PyCharm
 
In this book we will be using PyCharm as
our development environment. PyCharm is
a dedicated Python Integrated
Development Environment (IDE) providing
a wide range of essential tools for Python
developers, tightly integrated to create a
convenient environment for productive
Python, web, and data science
development[4]. PyCharm is a cross-
platform IDE that works on Windows,
macOS, and Linux.
PyCharm is available in three editions:
Community (free and open-sourced):
for individual Python development,
including code assistance, refactorings,
visual debugging, and version control
integration.
Professional (paid): for professional
Python, web, and data science
development, including code assistance,
refactorings, visual debugging, version
control integration, remote configurations,
deployment, support for popular web
frameworks, such as Django and Flask,
database support, scientific tools
(including Jupyter notebook support), big
data tools.
Edu (free and open-sourced): for
learning programming languages and
related technologies with integrated
educational tools.
We will be using the Community Edition
in this book and it is more than enough for
small to medium Python projects. In the
next sections we will install PyCharm and
learn its basics.
 
 
OceanofPDF.com
 PyCharm Installation
 
To install PyCharm, open the official
PyCharm web site in your web browser and
navigate to the Downloads page. You can
select your operating system here. We will
download the Community edition which is
free.
 

Figure 2-9: PyCharm download page

 
It is very straight forward to install
PyCharm. Just click Next and start
installation. It is strongly recommended to
accept the default destination folder. Then
you can click Next and finalize the
installation.
 

Figure 2-10: PyCharm setup

 
When you open PyCharm for the first
time, you will see the default Welcome
Screen. Here you have 3 options:
You can create a New Project
You can Open an existing project
You can get an existing project from
a Version Control System (VCS) such
as Git.
 
 Figure 2-11: Welcome screen in PyCharm
 
On the Welcome Screen, you can
customize the color theme of your PyCharm
IDE. You can use the black theme for
example. The default black theme is called
Darcula in PyCharm. Here is how you can
change your theme:
 
 Figure 2-12: Changing the color theme in PyCharm
 
 
OceanofPDF.com
 First Project in PyCharm
 
Now that PyCharm is installed on our
machine, let’s create our first project. On
the Welcome Screen, click on New Project
button to start the project creation wizard.
You see the New Project wizard in Figure 2-
13. Here you select a location and a name
for your project. We name our first project
as My_Project which is the last part in the
location string. 
 

Figure 2-13: New Project wizard in PyCharm

 
 The second configuration is about the
interpreter.
 
Python Interpreter:
Python Interpreter is the application
that runs your Python code. More precisely,
it is the python.exe file which we installed
in the previous sections. We can either
create a new virtual environment or we can
use the base environment with the Base
Interpreter. As stated earlier, working with
isolated virtual environments is the
recommended way and that’s what we
generally do in this book. So, we will select
the New environment using Virtualenv
option.
 

Figure 2-14: New virtual environment option

 
When we select the New environment
using Virtualenv option, PyCharm creates
a default location path for our new virtual
environment. Which is a new folder in the
project as: My_Project\venv . PyCharm will
create this folder automatically and it will
put the python.exe file and other necessary
files into it. It will use the Base Interpreter to
copy the python.exe file. In other words,
PyCharm will copy the python.exe file in the
Base Interpreter and paste it into the venv
folder. You can select the Python version for
the Base Interpreter.
The third configuration is about the
main.py file. You can tell PyCharm that you
want a default main file in your project. In
general, main.py files are the starting point
of Python projects. If you select this option,
PyCharm will create it automatically. If you
do not select it, you can create it later in
your project. Let’s select it and click on the
Create button to initialize our first PyCharm
project.
It may take some time, for PyCharm to
create the new project with a new virtual
environment. And after it finishes creation,
you will see the main.py file and all the
folders in your project.
 
 Figure 2-15: New project in PyCharm
 
As you see in Figure 2-15, the venv folder
is your virtual environment. And inside venv
folder you see python.exe under the Scripts
folder. That’s our Python Interpreter.
You see the main.py file is open on the
right-hand side of PyCharm. This is the
default file, PyCharm creates for us.
Let’s run our first project in PyCharm. On
the top right of the PyCharm screen you see
the Run button with a green play icon. Click
on it, and you will see the output at the
bottom of the screen:
 

Figure 2-16: The run button and the output window in

PyCharm
 
In Figure 2-16 we run our project and it
prints “Hi, P yCharm” in the output window.
If you could run your first project without
getting any errors, this means your Python
and PyCharm installations are successful
and you can start development in PyCharm.
We will learn PyCharm basics in the next
section.
 
 
OceanofPDF.com
 PyCharm Basics
 
PyCharm user interface is composed of
four main parts: Main Menu, Project
Explorer, Code Editor and Output
Window. In this section, we will see the
default PyCharm layout. You can customize
it and change the layout as you want.

Figure 2-17: PyCharm User Interface

 
Main Menu:
 On the top you see the Main Menu, where
you have all the commands related to
PyCharm IDE, configuration settings and file
editing options. Here are the items in the
main menu and some of the important
functionalities they have:
File : Create a new project, open an
existing one, save options and IDE settings
Edit : Copy, paste, delete functions, find
and replace, line operations
View : Viewing options for tool windows
and appearance
Navigate : Backward and forward
navigation, navigate in files
Code : Code completion & inspection,
folding and formatting options
Refactor : Refactoring and extracting code
fragments
Run : Run and debug with configurations
Tools : Tasks and team configurations
VC S : Version Control System settings like
Github and Space
Window : Layout and Tabs configuration
Help : Keymap Reference, Demos and
Screencasts, Learn IDE Features
 
Project Explorer:
 Project Explorer is on the left side and it’s
where you manage project files, folders and
virtual environments. For the current project
you can switch between different views as
Project, Project Files and Open Files.

Figure 2-18: Project Explorer options in PyCharm

 
Code Editor:
On the right-hand side, we have Code
Editor. We use the Editor to read, write, and
explore the source code. You can have
multiple files open in the editor. On the top
right corner of the editor window, you see
the quick access buttons like Run, Debug,
Run Configurations, Search and
Settings.
 
 Figure 2-19: Quick Access buttons on top of the editor
window
 
Output Window:
On the bottom of the screen, we have the
Output Window. It displays the output
generated by our application and messages
from the Python Interpreter.
Here are the tabs in the Output Window:
Run: The output when you run the code.
You can rerun the code or edit run
configuration
 

Figure 2-20: Run tab in the output window

 
TODO: Displays the TODO lines in the
code. TODO’s are special comment lines
that starts as ‘# TODO’ . In the image below
you see two TODO lines in the code and
their listing in the TODO tab:
 

Figure 2-21: TODO statements in the code and in TODO tab

 
Problems: Displays the errors and
problems in the code and the entire project
 

Figure 2-22: Problems tab

  
Terminal: An integrated terminal in
PyCharm. You do not need to start a
standalone terminal for Python. This
terminal works the same as your operating
system terminal.
 

Figure 2-23: Integrated Terminal in PyCharm

 
Python Packages: This tab displays all
the packages you have in the current
project. You can view package
documentation and add new packages in
this tab.
 

Figure 2-24: Python Packages tab

 
 Python Console: Integrated Python
console. Python console is the text editor
where you can run Python code. See an
example in the screenshot below:
 

Figure 2-25: Python Console

 
You are now familiar with the PyCharm
user interface and I think it’s enough to
start development in PyCharm IDE. If you
want to learn more about PyCharm and how
to customize its look and feel, here is the
complete documentation for PyCharm user
interface on its official web site[5].
 
 
OceanofPDF.com
 Running Code in PyCharm
 
You have three options to run a project in
PyCharm:
1-          The Run button in the Context Menu by
right clicking in the editor.
 

Figure 2-26: Run button in the context menu of editor

 
2-      The Run button in quick access menu on
the top right of the editor.
 
 Figure 2-27: Run button in the quick access menu
 
3-          The Run button in the Context Menu by
right clicking on the file name in the
Project Explorer.
 

Figure 2-28: Run button in the context menu of project

explorer
 
 And also there is a keyboard shortcut for
Run command:
Windows: Ctrl + Shift + F10
macOS: ⌃ ⌥ R
 
When you press any of these Run buttons
or use the keyboard shortcuts, Python
Interpreter compiles and executes the code.
Keep in mind that, the interpreter does
compilation and execution line by line at run
time.
In our current project, My_Project , we only
have one Python file which is main.py . In
general, Python projects have a starting
point which is called the main file. Here, our
main file is the main.py file which is created
by PyCharm when we initialized the project
(Figure 2-13).
You can download the Project in the
Github Repository of this book. Here is the
content of the main.py file:
 
[1]: 1 # This is a sample Python script.
  2  
# Press Shift+F10 to execute it or replace it
  3
with your code.
  4 # Press Double Shift to search everywhere
for classes, files, tool windows, actions, and
 settings.
  5  
  6  
  7 def print_hi(name):
    # Use a breakpoint in the code line
  8
below to debug your script.
    print(f'Hi, {name}')  # Press Ctrl+F8 to
  9
toggle the breakpoint.
  10  
  11  
# Press the green button in the gutter to
  12
run the script.
  13 if __name__ == '__main__':
  14     print_hi('PyCharm')
  15  
# See PyCharm help at
  16
https://www.jetbrains.com/help/pycharm/
  17  
  18 # TODO - Refactor this code
  19  
  20 # TODO - Define a global variable
 
When we run the project, Python
Interpreter executes the code in cell 1
starting from the first line.
We have a function definition in line 7,
and the function name is print_hi . This
function takes a parameter as name and
prints it in an f-string.
In line 13, we have an if statement as: if
__name__ == '__main__' . It checks whether the
global variable __name__ is equal to text
'__main__' or not. If it is True , then it calls the
print_hi function as: print_hi('P yCharm') .
The __name__ is a special variable which is
defined by PyCharm when we first create
the project. You can see it in the Python
Console in the bottom of the screen. You
can set a new value for it if you want by
right clicking on it.
 

Figure 2-29: The special variable __name__ in the Python

Console
  
You don’t have to have a main.py file or a
variable called __name__ . You don’t have to
check its value as we did in line 13. Later on
we will change all of these, but for the time
being, I wanted to show you how the things
work when PyCharm initializes a project and
how it runs the main file.
 
 
OceanofPDF.com
 Interpreter Configuration
 
When you run the project, PyCharm looks
for the Python Interpreter and tells it that it
should run the code. So the interpreter
configuration is crucial. If PyCharm cannot
find it you will get an Interpreter Error. As
we stated earlier the interpreter is simply
the python.exe file and it resides in a virtual
environment. Now we will prove this.
We have a project named My_Project in
this chapter. And it has its own virtual
environment in the venv folder. Remember
Figure 2-15, where you can see the
python.exe file under the path:
\My_Project\venv\Scripts . You can navigate to
this path from PyCharm, by right clicking it
and Open In > Explorer .
 
 Figure 2-30: Open PyCharm files and folders in file explorer
 
All of your Python packages, libraries and
Python Interpreter reside in venv folder
which is your virtual environment. Now that
we know the Python Interpreter is in the
venv folder let’s delete it and see what
happens. To do this, navigate to the top
level folder of your project, namely the
project folder and delete the venv folder.
You may need to close the PyCharm first,
because the folder will be in use when you
try to delete it.
 

Figure 2-31: Delete venv folder in the project

 
Now reopen PyCharm after you deleted
the venv folder. You will see that, it will
display an error as Invalid P ython Interpreter
selected for the project . This means that you
don’t have a Python Interpreter for this
project.
 

Figure 2-32: Interpreter Error after deleting venv folder

 
Let’s fix this error and configure the
interpreter. There are multiple ways to
access the interpreter configuration screen.
The first way is to click on the Configure
P ython Interpreter button which is at the
right of the error text. The second way is to
click on the file name (which is main here)
at the quick access menu and then select
Edit Configurations . See the image below:
 

Figure 2-33: Edit Configurations button in the quick access

menu
 
And the third way is under File > Settings :
  

Figure 2-34: File > Settings

 
Let’s move on with File > Settings way.
When you click on the Settings button under
the File menu it will open the settings
window for PyCharm. This is the main
screen where we do almost any
customization in PyCharm.
 

Figure 2-35: Settings Window and Python Interpreter

configuration
  
On the Settings Window, under the
Project section, you see Python Interpreter.
And you will see the Invalid text in the
Python Interpreter field on the right (Figure
2-35). Now we will configure a new
interpreter.
To configure a new interpreter click on the
gear (settings) icon on the right and then
click on Add… button (Figure 2-36).
 

Figure 2-36: Add new Python Interpreter button

 
PyCharm will open the Add P ython
Interpreter window as below:
 
 Figure 2-37: Add Python Interpreter window
 
In Figure 2-37, the Virtual Environment
tab is selected. Here you can either create a
new virtual environment or use an existing
one. We will create a new virtual
environment. As you see, the default folder
name is venv . When we click the OK button,
PyCharm will create this folder and a new
virtual environment in it. We will do it in a
minute but before that, let me show you the
System Interpreter on this screen.
 
System Interpreter:
If you do not want to create a new virtual
environment and a new interpreter, you can
use the System Interpreter. It’s the base
Interpreter you have when you install
Python. To see it, click on the System
Interpreter tab on the left.
 

Figure 2-38: System Interpreter

 
When you click on the System Interpreter
tab, you will see all the Python Interpreters
on your machine (Figure 2-38). If you click
OK, your current project will be using the
Base Interpreter instead of creating a new
virtual environment and a new interpreter.
As we stated earlier we will create a new
virtual environment for this project, so click
on the Virtual Environment tab again and
then click the OK button (Figure 2-37).
PyCharm creates a new virtual environment
for us in the venv folder. Now you can see
that the Interpreter Error has been resolved
and the new venv folder created in the
project. See the final figure below:
 

Figure 2-39: New virtual environment (venv folder) in the

project
 
 
OceanofPDF.com
 Debugging in PyCharm
 
In computer programming, Debugging is
the process of detecting and removing of
existing and potential errors. These errors
are also called as ‘bugs’. The bugs in a
program can cause it to behave
unexpectedly or crash. In this section we
will see how to do debugging in PyCharm.
Debugging is done via breakpoints.
Breakpoints are the special places in the
code where you intentionally stop (pause)
the execution of the program. This is a
temporary stop and we do this to see what’s
going on in our program.
The breakpoints are only operational in
debug mode. They do not work in normal
run mode. In other words, to be able to
debug your code at breakpoints you need to
run the application in debug mode. We will
see it in a minute.
To place a breakpoint in PyCharm, you
simply click on the gutter once. The gutter
is the column where you see the line
numbers. In the image below we put a
breakpoint on line 9:
  

Figure 2-40: A breakpoint in the gutter

 
We have a breakpoint on line 9, which
means PyCharm will pause execution on this
line. And we will have look in our program
state.
Now let’s run the project in debug mode.
In Figure 2-40, you can see the debug run
button with a bug icon on the quick access
bar. Let’s click on this button and run our
application in debug mode.
 
 Figure 2-41: PyCharm in Debug Mode
 
When you run the application in debug
mode, you will see a new window in the
bottom, which is the Debug screen (Figure
2-41). There are two main tabs on the
Debug window, Debugger and Console.
Under the Debugger tab, you can see two
parts which are Frames and Variables.
Frames tab displays the execution order of
your program and the Variables part is
where you manage your variables. For
example, we only have one variable in the
print_hi function, which is name . The value
for this variable is ‘P yCharm’ .
Resume Program:
You can continue the execution with the
Resume Program button (Figure 2-42).
PyCharm will continue to execute and it will
finish execution if it does not encounter
another breakpoint. If it encounters another
breakpoint, it will pause again and wait.
Stop:
If you want to stop debugging and exit
the Debugger you can use the Stop button
(Figure 2-42). It will terminate the current
debug session and set the program to its
initial state.
 

Figure 2-42: Resume Program and Stop buttons in

Debugger
 
Now let’s see debugging in more detail.
We will first stop the current debug session.
Click on the stop button in the debugger
window. Then create a new Python file (.py)
in our project and name it as
debug_example.py . To create a new file, in the
Project Explorer pane, right click on the
project name and select New > File as in
Figure 2-43.  And name the file as
debug_example.py .
 

Figure 2-43: Create a new Python file in project

 
We will define a function in this new file.
Then we will call this function with some
arguments. Here is the complete code for
the debug_example.py file:
 
[2]: 1 # define a function to add numbers
  2 def summation(x_1, x_2):
  3     sum_of_nums = x_1 + x_2
  4     return sum_of_nums
  5  
  6 # define two variables
  7 a = 2
  8 b = 5
  9  
  10 # call the function
  11 summation(a, b)
 
We will set two breakpoints in this file, on
lines 3 and 11. See the image below:
 

Figure 2-44: Breakpoints in debug_example.py file

 
Now our debug_example.py file is ready
with breakpoints. Before running it in debug
mode, there is a configuration we should
change. The run configuration. Currently, it
is set to run the main.py . You can see it in
Figure 2-44. So if you click Run or Debug
buttons it will run the main.py file, not our
new file. To change this, right click in the
code editor of debug_example.py file and
select Debug ‘debug_example’ .
 

Figure 2-45: Run debug_example.py file

 
When you click on Debug ‘debug_example’ ,
PyCharm runs the debug_example.py file in
debug mode. As you see in Figure 2-46, the
run configuration is debug_example now.
PyCharm pauses the execution on line 11
and launch the Debugger at the bottom.
Line 11 is where we call the summation
function.
  

Figure 2-46: debug_example.py file is in debug mode

 
Now the code execution waits on line 11.
In Figure 2-46, you can see that we have to
variables a and b and their values. If you
click on the Resume Program button, which
is the play like green icon, , the code
execution will continue until it stops at line
3. Why? Because we have another
breakpoint on line 3. See the image below:
 
 Figure 2-47: Second breakpoint is in the summation
function
 
Now in the Debugger we see the
summation function on the left the frame and
on the right x_1 and x_2 are the variables in
this frame.
There is a set of important buttons on the
top bar of the Debugger window. These are
the stepping toolbar buttons for the
debugger. Here they are:
 
 Figure 2-48: Stepping toolbar buttons in Debugger
 
Here are the functionalities for the most
common stepping buttons:
 
Step Over: Executes the
current line.
Step Into: Steps into the
method called at the current
execution point.
Step Out: Steps out of the
current method, to the line
executed right after it.
Evaluate Expression:
Evaluates Python expressions.
 
Now let’s evaluate an expression.
Remember our debugger is waiting for
execution on line 3. And we have an
addition operation in this line as:
sum_of_nums = x_1 + x_2 . Before running this
line, let’s evaluate it. Click on the Evaluate
Expression button (calculator icon) in the
stepping toolbar. It will pop up the
evaluation window as follows:
  

Figure 2-49: Evaluate window

 
The Evaluate window is very useful and
you will be using it most of the time in your
projects. You can evaluate an expression,
you can do a function call, and you can
define variables on this window. It doesn’t
affect your project execution. It’s like a
separate Python environment which can use
the current project. In Figure 2-49 we
evaluate the expression of x_1 + x_2 and it
returns 7 .
Let’s click the Step Over (F8) button to run
the current line, which is line 3. PyCharm
runs the code in line 3 and it waits on line 4.
Because Step Over button only runs one
line of code. See the image below:
 

Figure 2-50: The new line of execution is 4

 
As you see in Figure 2-50, after we click
the Step Over button, PyCharm executes
line 3 and it creates a new variable which is
sum_of_nums . You can see it in the Variables
tab in Debugger. The current line is line 4
now. If we click on Step Over button one
more time, PyCharm will finish the
execution of the summation function and it
will return to the function call in line 11. And
if you click it one more time, PyCharm will
end the running because there is no more
lines to execute.
As a final example, you can remove the
breakpoint on line 3 and try the Step Into
button to be able to pause inside the
summation function. Here is the image for it:
 

Figure 2-51: Step Into the function

 
I think that’s enough for debugging in
PyCharm. You learned the basics of
debugging, how to set breakpoints and how
to run and evaluate the code line by line.
We are ready to start actual Python
development now. In the next chapter you
will learn Exception Handling in Python.
 
OceanofPDF.com
 3. Collections
     

 
OceanofPDF.com
 What are Collections
 
Collections are specialized container
datatypes providing alternatives to
Python’s general purpose built-in
containers, dict , list , set , and tuple . A
Container is a special-purpose object
which is used to store different objects. It
provides a way to access the contained
objects and iterate over them.
Python provides the collections module
which implements container datatypes. In
this chapter we will learn different classes
in this module. You can find the PyCharm
project for this chapter in the GitHub
Repository of this book.
 
Chapter Outline:
ChainMap
Counter
Deque
DefaultDict
NamedTuple
OrderedDict
UserDict
 UserList
UserString
QUIZ – Collections
SOLUTIONS - Collections
 
 
OceanofPDF.com
 ChainMap
 
A ChainMap class is provided for quickly
linking a number of mappings so they can
be treated as a single unit. It is often
much faster than creating a new
dictionary and running multiple update()
calls.
Syntax:
class collections.ChainMap(*maps)
A ChainMap groups multiple dicts or
other mappings together to create a
single, updateable view (list of
dictionaries). If no maps are specified, a
single empty dictionary is provided so that
a new chain always has at least one
mapping.
The underlying mappings are stored in a
list. That list is public and can be accessed
or updated using the maps attribute. There
is no other state in the ChainMap.
A ChainMap incorporates the underlying
mappings by reference. So, if one of the
underlying mappings gets updated, those
changes will be reflected in ChainMap.
 All of the usual dictionary methods are
supported. In addition, there is:
a maps attribute,
a method for creating new sub
contexts
a property for accessing all but the
first mapping
 
maps:
A user updateable list of mappings. The
list is ordered from first-searched to last-
searched. It is the only stored state and
can be modified to change which
mappings are searched. The list should
always contain at least one mapping.
 
# import ChainMap class from collections
[1]: 1
module
  2 from collections import ChainMap
  3  
  4 #--- Defining a ChainMap ---#
  5 numbers = {'one': 1, 'two': 2}
  6 letters = {'a': 'A', 'b': 'B'}
  7  
  8 # Define the ChainMap
  9 chain_map = ChainMap(numbers, letters)
  10  
  11 print(chain_map)
     
[1]:   ChainMap({'one': 1, 'two': 2}, {'a': 'A',
'b': 'B'})
 
In cell 1, we define a ChainMap object
( chain_map ) with two dictionaries. Then we
print the ChainMap. As you see in the
output, the result is a view of these dicts.
 
Accessing Keys and Values from
ChainMap:
We can access the keys and values of a
ChainMap by using the keys() and values()
methods.
 
#--- Accessing Keys and Values from
[2]: 1
ChainMap ---#
  2 print(chain_map.keys())
  3 print(chain_map.values())
     
KeysView(ChainMap({'one': 1, 'two': 2},
[2]:  
{'a': 'A', 'b': 'B'}))
ValuesView(ChainMap({'one': 1, 'two': 2},
   
{'a': 'A', 'b': 'B'}))
 
As you see in the output of cell 2, the
result of chain_map.keys() is a KeysView and
the result of chain_map.values() is a
ValuesView .
  
Accessing Individual Values with Key
Names:
We can access individual values from a
ChainMap by using the key name. This is
exactly the same way what we do with
regular dictionaries.
 
#--- Accessing Individual Values with Key
[3]: 1
Names ---#
  2 print(chain_map['one'])
  3 print(chain_map['b'])
     
[3]:   1
    B
 
In cell 3, we access the values of the
individual items in the underlying
dictionaries of the ChainMap by using the
key names as: chain_map['one'] .
 
Adding a New Dictionary to ChainMap:
ChainMap can contain any number of
dictionaries in it. We use the built-in
new_child() method to add new dictionaries
to the ChainMap. The new_child() method
returns a new ChainMap containing a new
map followed by all of the maps in the
current instance. One point to note here
is, the newly added dict will be placed at
the beginning of the ChainMap.
 
#--- Adding a New Dictionary to ChainMap
[4]: 1
---#
  2 variables = {'x': 0, 'y': 1}
new_chain_map =
  3
chain_map.new_child(variables)
  4 print('Old:', chain_map)
  5 print('New:', new_chain_map)
     
Old: ChainMap({'one': 1, 'two': 2}, {'a':
[4]:  
'A', 'b': 'B'})
New: ChainMap({'x': 0, 'y': 1}, {'one': 1,
   
'two': 2}, {'a': 'A', 'b': 'B'})
 
Get the List of Mappings in ChainMap:
We use the maps attribute the get the
list of all mappings in the ChainMap.
 
#--- Get the List of Mappings in ChainMap
[5]: 1
---#
  2 print(chain_map.maps)
     
[5]:   [{'one': 1, 'two': 2}, {'a': 'A', 'b': 'B'}]
 
In cell 5, we get all the mappings
(dictionaries) in the chain_map . As you see
in the output, the maps attribute returns a
list type object (a list of dictionaries).
 
 
OceanofPDF.com
 Counter
 
A Counter is a dict subclass for counting
hashable objects. It is a collection where
elements are stored as dictionary keys
and their counts are stored as dictionary
values. Counts are allowed to be any
integer value including zero or negative
counts. The Counter class is similar to
bags or multisets in other languages.
Elements are counted from an iterable
or initialized from another mapping (or
counter). Here are some ways we create
Counter objects in Python:
 
[6]: 1 from collections import Counter
  2  
  3 # a new, empty counter
  4 c1 = Counter()
  5 print(c1)
  6  
  7 # a new counter from an iterable
  8 c2 = Counter('aabbbcddeeee')
  9 print(c2)
  10  
  11 # a new counter from a mapping
c3 = Counter({'orange': 6, 'red': 3,
  12
'green': 5})
  13 print(c3)
  14  
  15 # a new counter from keyword args
  16 c4 = Counter(cats=4, dogs=8)
  17 print(c4)
     
[6]:   Counter()
Counter({'e': 4, 'b': 3, 'a': 2, 'd': 2, 'c':
   
1})
Counter({'orange': 6, 'green': 5, 'red':
   
3})
    Counter({'dogs': 8, 'cats': 4})
 
Counter objects have a dictionary
interface except that they return a zero
count for missing items instead of raising
a KeyError :
 
[7]: 1 # count of existing element
  2 c5 = Counter(['eggs', 'ham', 'jar', 'ham'])
  3 print(c5['ham'])
  4  
  5 # count of a missing element is zero
  6 print(c5['bacon'])
     
[7]:   2
    0
 
Delete Elements from a Counter:
To delete elements from a Counter, we
use the del keyword. Please keep in mind
that, setting a count to zero does not
remove an element from a counter.
 
[8]: 1 # --- Delete Elements from a Counter --- #
  2 # counter entry with a zero count
  3 c5['sausage'] = 0
  4 print(c5)
  5  
  6 # del actually removes the entry
  7 del c5['sausage']
  8 print(c5)
     
Counter({'ham': 2, 'eggs': 1, 'jar': 1,
[8]:  
'sausage': 0})
    Counter({'ham': 2, 'eggs': 1, 'jar': 1})
 
As you see in cell 8, we set zero to an
item which even doesn’t exist in the
Counter. And Python adds that item to the
Counter with zero value. In line 7, we
remove the item entirely with the del
keyword.
 
Counter Methods:
Counter objects support additional
methods beyond those available for all
dictionaries. Here are the most common
methods:
  
elements():
Return an iterator over elements
repeating each as many times as its
count. Elements are returned in the order
first encountered. If an element’s count is
less than one, elements() will ignore it.
 
[9]: 1 # --- Counter Methods --- #
  2 # elements()
counter = Counter(a=1, b=2, c=0, d=-2,
  3
e=4)
sorted_elements =
  4
sorted(counter.elements())
  5 print(sorted_elements)
     
[9]:   ['a', 'b', 'b', 'e', 'e', 'e', 'e']
 
most_common([n]):
Return a list of the n most common
elements and their counts from the most
common to the least. If n is omitted or
None, most_common() returns all elements
in the counter. Elements with equal counts
are ordered in the order first encountered:
 
[10]: 1 # most_common()
most_common_3 =
  2
Counter('abracadabra').most_common(3)
  3 print(most_common_3)
     
[10]:   [('a', 5), ('b', 2), ('r', 2)]
 
subtract([iterable-or-mapping]):
Elements are subtracted from an
iterable or from another mapping (or
counter). Like dict.update() but subtracts
counts instead of replacing them. Both
inputs and outputs may be zero or
negative.
 
[11]: 1 # subtract()
  2 c_1 = Counter(a=4, b=2, c=0, d=-2)
  3 c_2 = Counter(a=1, b=2, c=3, d=4)
  4 c_1.subtract(c_2)
  5 print(c_1)
     
[11]:   Counter({'a': 3, 'b': 0, 'c': -3, 'd': -6})
 
The usual dictionary methods are
available for  Counter  objects except for
two which work differently for counters:
fromkeys(iterable):
This class method is not implemented
for Counter objects.
 
update([iterable-or-mapping]):
 Elements are counted from
an  iterable  or added-in from
another  mapping  (or counter).
Like  dict.update()   but adds counts instead
of replacing them. Also, the  iterable  is
expected to be a sequence of elements,
not a sequence of (key, value) pairs.
 
[12]: 1 # update()
  2 d = Counter(a=3, b=1)
  3 d.update({'a': 5, 'c': 4})
  4 print(d)
     
[12]:   Counter({'a': 8, 'c': 4, 'b': 1})
 
 
OceanofPDF.com
 Deque
 
Deques are a generalization of stacks
and queues (the name is pronounced
“deck” and is short for “double-ended
queue”). Deques support thread-safe,
memory efficient appends and pops from
either side of the deque with
approximately the same O(1) performance
in either direction.
Though list objects support similar
operations, they are optimized for fast
fixed-length operations and incur O(n)
memory movement costs for pop(0) and
insert(0, v) operations which change both
the size and position of the underlying
data representation.
collections.deque([iterable[, maxlen]]) :
Returns a new deque object initialized left-
to-right (using append() ) with data from
iterable. If iterable is not specified, the
new deque is empty.
maxlen : Maximum size of a deque or
None if unbounded.
 If maxlen is not specified or is None ,
deques may grow to an arbitrary length.
Otherwise, the deque is bounded to the
specified maximum length. Once a
bounded length deque is full, when new
items are added, a corresponding number
of items are discarded from the opposite
end.
 
[13]: 1 ### Deque
  2  
  3 from collections import deque
  4  
  5 # Declaring the deque
  6 q = deque(['user', 'password', 'token'])
  7 print(q)
     
[13]:   deque(['user', 'password', 'token'])
 
In cell 13, we define a deque object by
passing a list as argument. Now let’s
create another one but this time we will
use a string:
 
[14]: 1 # make a new deque with three items
  2 d = deque('dqi')
  3 # iterate over the deque's elements
  4 for elem in d:
  5     print(elem.upper())
     
[14]:   D
    Q
    I
 
Now let’s see the contents in the deque
object:
 
[15]: 1 # list the contents of the deque
  2 deque_contents = list(d)
  3 print(deque_contents)
  4  
  5 # peek at leftmost item
  6 print(d[0])
  7  
  8 # peek at rightmost item
  9 print(d[-1])
     
[15]:   ['d', 'q', 'i']
    d
    i
 
Here are some methods that deque
objects support:
 
append(x) :
Add x to the right side of the deque.
 
appendleft(x) :
 Add x to the left side of the deque.
 
[16]: 1 # add a new entry to the right side
  2 d.append('j')
  3  
  4 # add a new entry to the left side
  5 d.appendleft('f')
  6  
  7 # show the representation of the deque
  8 print(d)
     
[16]:   deque(['f', 'd', 'q', 'i', 'j'])
 
pop() :
Remove and return an element from the
right side of the deque. If no elements are
present, raises an IndexError .
 
popleft() :
Remove and return an element from the
left side of the deque. If no elements are
present, raises an IndexError .
 
[17]: 1 # return and remove the rightmost item
  2 rightmost = d.pop()
  3 print(rightmost)
  4  
  5 # return and remove the leftmost item
  6 leftmost = d.popleft()
  7 print(leftmost)
     
[17]:   j
    f
 
clear() :
Remove all elements from the deque
leaving it with length 0.
 
copy() :
Create a shallow copy of the deque.
 
count(x) :
Count the number of deque elements
equal to x.
 
extend(iterable) :
Extend the right side of the deque by
appending elements from the iterable
argument.
 
[18]: 1 # add multiple elements at once
  2 d.extend('jkl')
  3 print(d)
     
[18]:   deque(['d', 'q', 'i', 'j', 'k', 'l'])
 
extendleft(iterable) :
 Extend the left side of the deque by
appending elements from iterable. Note,
the series of left appends results in
reversing the order of elements in the
iterable argument.
 
[19]: 1 # extendleft() reverses the input order
  2 d.extendleft('xyz')
  3 print(d)
     
[19]:   deque(['z', 'y', 'x', 'd', 'q', 'i', 'j', 'k', 'l'])
 
index(x[, start[, stop]]) :
Return the position of x in the deque (at
or after index start and before index stop).
Returns the first match or raises ValueError
if not found.
 
insert(i, x) :
Insert x into the deque at position i . If
the insertion would cause a bounded
deque to grow beyond maxlen , an
IndexError is raised.
 
remove(value) :
Remove the first occurrence of value. If
not found, raises a ValueError .
 
rotate(n=1) :
Rotate the deque n steps to the right. If
n is negative, rotate to the left.
 
[20]: 1 # deque at the beginning
  2 print(d)
  3  
  4 # right rotation
  5 d.rotate(1)
  6 print(d)
  7  
  8 # left rotation
  9 d.rotate(-1)
  10 print(d)
     
[20]:   deque(['z', 'y', 'x', 'd', 'q', 'i', 'j', 'k', 'l'])
    deque(['l', 'z', 'y', 'x', 'd', 'q', 'i', 'j', 'k'])
    deque(['z', 'y', 'x', 'd', 'q', 'i', 'j', 'k', 'l'])
 
reverse() :
Reverse the elements of the deque in-
place and then return None .
 
[21]: 1 # deque at the beginning
  2 print('old deque:', d)
  3  
  4 # reverse the elements in the deque
  5 new_deq = d.reverse()
  6 print('new deque:', new_deq)
  7  
  8 # original deque after reversed()
  9 print('old deque:', d)
     
old deque: deque(['z', 'y', 'x', 'd', 'q', 'i',
[21]:  
'j', 'k', 'l'])
    new deque: None
old deque: deque(['l', 'k', 'j', 'i', 'q', 'd',
   
'x', 'y', 'z'])
 
As you see in the output of cell 21, the
reverse() method reverses the elements of
the deque in-place, which means our
original deque object is modified. And it
returns None .
 
 
OceanofPDF.com
 DefaultDict
 
One of the common problems with the
Dictionary class in Python is the missing
keys. When you try to access a key that
does not exist in the dictionary you will
get a KeyError . So, you have to handle this
case whenever you need to access an
element in the dictionary. Fortunately, we
have DefaultDict class in Python. It is used
to provide some default values for a key
that does not exist and does not raise a
KeyError .
DefaultDict is a subclass of the built-in
dict class. It overrides one method and
adds one writable instance variable. The
remaining functionality is the same as for
the dict class and is not documented here.
collections.defaultdict(default_factory=Non
e, /[, ...]) :
Return a new dictionary-like
object, DefaultDict, which is a subclass of
the built-in dict class.
The first argument provides the initial
value for the default_factory attribute; it
defaults to None . All remaining arguments
are treated the same as if they were
passed to the dict constructor, including
keyword arguments.
DefaultDict objects support the
following method in addition to the
standard dict operations:
__missing__(key) :
If the default_factory attribute is None ,
this raises a KeyError exception with the
key as argument.
If default_factory is not None , it is called
without arguments to provide a default
value for the given key, this value is
inserted in the dictionary for the key, and
returned.
DefaultDict objects support the
following instance variable:
default_factory :
This attribute is used by the
__missing__() method; it is initialized from
the first argument to the constructor, if
present, or to None , if absent.
 
[22]: 1 from collections import defaultdict
  2  
s = [('yellow', 1), ('blue', 2), ('yellow', 3),
  3
('blue', 4), ('red', 1)]
  4 d = defaultdict(list)
  5 for k, v in s:
  6     d[k].append(v)
  7  
  8 sorted_items = sorted(d.items())
  9 print(sorted_items)
     
[('blue', [2, 4]), ('red', [1]), ('yellow', [1,
[22]:  
3])]
 
In cell 20, we use the list type as the
default_factory , to make it easy to group a
sequence of key-value pairs into a
dictionary of lists. When each key is
encountered for the first time, an entry is
automatically created using the
default_factory function which returns an
empty list. The list.append() operation
then attaches the value to the new list.
When keys are encountered again, the
look-up proceeds normally (returning the
list for that key) and the list.append()
operation adds another value to the list.
This technique is simpler and faster than
an equivalent technique using
dict.setdefault() .
 
[23]: 1 river = 'mississippi'
  2 dd = defaultdict(int)
  3 for r in river:
  4     dd[r] += 1
  5  
  6 s_items = sorted(dd.items())
  7 print(s_items)
     
[23]:   [('i', 4), ('m', 1), ('p', 2), ('s', 4)]
 
In cell 23, we set the default_factory to
int . This makes the DefaultDict useful for
counting (like a bag or multiset in other
languages). When a letter is first
encountered, it is missing from the
mapping, so the default_factory function
calls int() to supply a default count of
zero. The increment operation then builds
up the count for each letter.
 
 
OceanofPDF.com
 NamedTuple
 
NamedTuples assign meaning to each
position in a tuple and allow for more
readable, self-documenting code. They can
be used wherever regular tuples are used,
and they add the ability to access fields by
name instead of position index.
collections.namedtuple(typename,
field_names):
Returns a new tuple subclass named
typename . The new subclass is used to
create tuple-like objects that have fields
accessible by attribute lookup as well as
being indexable and iterable. Instances of
the subclass also have a helpful docstring
(with typename and field_names ) and a
helpful __repr__() method which lists the
tuple contents in a name=value format.
The field_names are a sequence of strings
such as ['x', 'y']. Alternatively, field_names
can be a single string with each fieldname
separated by whitespace and/or commas,
for example 'x y' or 'x, y'.
To understand how the NamedTuple
works, let’s assume we have an Employee
object. Employee has id, name and age
attributes.
 
[24]: 1 from collections import namedtuple
  2  
  3 # Declare the namedtuple
Employee = namedtuple('Employee', ['id',
  4
'name', 'age'])
  5  
  6 # Add some values to the tuple
  7 E_1 = Employee('111', 'Peter Parker', '18')
  8 E_2 = Employee('222', 'Clark Kent', '26')
  9  
  10 # Access using index
print("Employee name by index is : ",
  11
end="")
  12 print(E_1[1])
  13  
  14 # Access using keys
print("Employee name using key is : ",
  15
end="")
  16 print(E_2.name)
     
[24]:   Employee name by index is : Peter Parker
    Employee name using key is : Clark Kent
 
In addition to the methods inherited from
tuples, named tuples support three
additional methods and two attributes. To
prevent conflicts with field names, the
method and attribute names start with an
underscore.
  
_make(iterable) :
Class method that makes a new instance
from an existing sequence or iterable.
 
[25]: 1 # _make()
  2 # initialize an iterable
  3 bat_data = ['333', 'Batman', '28']
  4 batman = Employee._make(bat_data)
  5 print(batman)
     
Employee(id='333', name='Batman',
[25]:  
age='28')
 
_asdict() :
Return a new dict which maps field names
to their corresponding values:
 
[26]: 1 # _asdict()
  2 bat_dict = batman._asdict()
  3 print(bat_dict)
     
[26]:   {'id': '333', 'name': 'Batman', 'age': '28'}
 
_replace(**kwargs) :
Return a new instance of the named tuple
replacing specified fields with new values:
 
[27]: 1 # _replace()
  2 batman = batman._replace(id='777',
 age='34')
  3 print(batman)
     
Employee(id='777', name='Batman',
[27]:  
age='34')
 
_fields :
Tuple of strings listing the field names.
Useful for introspection and for creating new
named tuple types from existing named
tuples.
 
[28]: 1 # _fields
  2 print(batman._fields)
     
[28]:   ('id', 'name', 'age')
 
We can use the _fields attribute to create
new namedtuples from existing ones:
 
[29]: 1 # namedtuple fields from others
  2 Point = namedtuple('Point', ['x', 'y'])
  3 Color = namedtuple('Color', 'red green blue')
Pixel = namedtuple('Pixel', Point._fields +
  4
Color._fields)
  5 p = Pixel(5, 8, 128, 255, 0)
  6 print(p)
     
[29]:   Pixel(x=5, y=8, red=128, green=255, blue=0)
 
 
 
OceanofPDF.com
 OrderedDict
 
Ordered Dictionaries are just like regular
dictionaries but have some extra
capabilities relating to ordering
operations. OrderedDicts remember the
order in which the keys were inserted.
They have become less important now
that the built-in dict class gained the
ability to remember insertion order (this
new behavior became guaranteed in
Python 3.7).
collections.OrderedDict([items]) :
Return an instance of a dict subclass
that has methods specialized for
rearranging dictionary order.
 
popitem(last=True) :
The popitem() method for ordered
dictionaries returns and removes a (key,
value) pair. The pairs are returned in LIFO
(last in first out) order if last is true or FIFO
(first in first out) order if false.
 
move_to_end(key, last=True) :
 Move an existing key to either end of an
ordered dictionary. The item is moved to
the right end if last is true (the default) or
to the beginning if last is false. Raises
KeyError if the key does not exist:
 
[30]: 1 from collections import OrderedDict
  2  
  3 od = OrderedDict.fromkeys('abcde')
  4 od.move_to_end('b')
  5 print(''.join(od))
  6 # 'acdeb'
  7 od.move_to_end('b', last=False)
  8 print(''.join(od))
  9 # 'bacde'
     
[30]:   acdeb
    bacde
 
Let’s say we delete and re-insert the
same key to an OrderedDict. It will push
this key to the end to maintain the order
of insertion of the keys.
 
[31]: 1 # delete and re-insert same key
  2 d = OrderedDict()
  3 d['x'] = 'X'
  4 d['y'] = 'Y'
  5 d['z'] = 'Z'
  6  
  7 print('OrderedDict before deleting')
  8 for key, value in d.items():
  9     print(key, value)
  10  
  11 # delete the element
  12 d.pop('x')
  13  
  14 # re-insert the same key
  15 d['x'] = 'X'
  16  
  17 print('\nOrderedDict after insertion')
  18 for key, value in d.items():
  19     print(key, value)
     
[31]:   OrderedDict before deleting
    x X
    y Y
    z Z
     
    OrderedDict after insertion
    y Y
    z Z
    x X
 
 
OceanofPDF.com
 UserDict
 
The class, UserDict acts as a wrapper
around dictionary objects. The need for
this class has been partially supplanted by
the ability to subclass directly from dict;
however, this class can be easier to work
with because the underlying dictionary is
accessible as an attribute. You can use
UserDict when you want to create your
own dictionary with some modified or new
functionality.
collections.UserDict([initialdata]) :
Class that simulates a dictionary. The
instance’s contents are kept in a regular
dictionary, which is accessible via the data
attribute of UserDict instances. If
initialdata is provided, data is initialized
with its contents; note that a reference to
initialdata will not be kept, allowing it to be
used for other purposes.
In addition to supporting the methods
and operations of mappings, UserDict
instances provide the following attribute:
data :
 A real dictionary used to store the
contents of the UserDict class.
 
[32]: 1 from collections import UserDict
  2  
  3 us = {'name': 'John Doe', 'age': 24}
  4  
  5 # Create UserDict object
  6 ud = UserDict(us)
  7 print(ud.data)
     
[32]:   {'name': 'John Doe', 'age': 24}
 
Let’s say we want to define a custom
dictionary object which supports addition
operation. When we add two instances of
our custom dictionary, we want to get a
new dictionary with all of the elements in
both dictionaries. Keep in mind that, you
will get TypeError if you try to add to
regular dicts in Python. Let’s implement
this with the help of UserDict:
 
[33]: 1 # class for our custom dict
  2 # inherit from UserDict
  3 class AddEnabledDict(UserDict):
  4     # override the __add__ method
  5     def __add__(self, other):
  6         d = AddEnabledDict(self.data)
  7         d.update(other.data)
  8         return d
  9  
  10 # create custom objects
  11 d_1 = AddEnabledDict(x = 10)
  12 d_2 = AddEnabledDict(y = 20)
  13 total = d_1 + d_2
  14 print(total)
     
[33]:   {'x': 10, 'y': 20}
 
 
OceanofPDF.com
 UserList
 
The UserList class acts as a wrapper
around list objects. It is a useful base class
for your own list-like classes which can
inherit from them and override existing
methods or add new ones. In this way, one
can add new behaviors to lists in Python.
The need for this class has been
partially supplanted by the ability to
subclass directly from list; however, this
class can be easier to work with because
the underlying list is accessible as an
attribute.
collections.UserList([list]) :
Class that simulates a list. The
instance’s contents are kept in a regular
list, which is accessible via the data
attribute of UserList instances. The
instance’s contents are initially set to a
copy of list, defaulting to the empty list [].
list parameter can be any iterable, for
example a real Python list or a UserList
object.
 In addition to supporting the methods
and operations of mutable sequences,
UserList instances provide the following
attribute:
data :
A real list object used to store the
contents of the UserList class.
 
Let’s say we want to define a list which
doesn’t allow deleting the items in it. We
can easily define such a class by inheriting
UserList:
 
[34]: 1 from collections import UserList
  2  
  3 # define a custom class
  4 # this class will inherit from UserList
# it will not allow its items to be
  5
deleted
# List class in Python has to methods
  6
for delete:
  7 # remove() and pop()
class
  8
ListWithNoItemDelete(UserList):
  9     # override remove() method
  10     def remove(self, s=None):
  11         self.not_allowed()
  12  
  13     # override pop() method
  14     def pop(self, s=None):
  15         self.not_allowed()
  16  
  17     def not_allowed(self):
        raise RuntimeError("Deletion not
  18
allowed")
  19  
  20 # custom list object
custom_list = ListWithNoItemDelete(['a',
  21
'b', 'c'])
  22  
  23 # try to delete an item
  24 custom_list.remove('b')
     
[34]:   RuntimeError: Deletion not allowed
 
 
OceanofPDF.com
 UserString
 
The class, UserString acts as a wrapper
around string objects. The need for this
class has been partially supplanted by the
ability to subclass directly from str;
however, this class can be easier to work
with because the underlying string is
accessible as an attribute.
collections.UserString(seq) :
Class that simulates a string object. The
instance’s content is kept in a regular
string object, which is accessible via the
data attribute of UserString instances. The
instance’s contents are initially set to a
copy of seq . The seq argument can be any
object which can be converted into a
string using the built-in str() function.
In addition to supporting the methods
and operations of strings, UserString
instances provide the following attribute:
data :
A real str object used to store the
contents of the UserString class.
 
 Let’s say we want to define a custom str
class that have concatenate() method in it:
 
[35]: 1 # UserString
  2  
  3 from collections import UserString
  4  
  5 # define a custom class
  6 # this class will inherit from UserString
  7 class CustomStrClass(UserString):
  8     # define a new method
    def concatenate(self, other=None,
  9
delimiter=' '):
  10         self.data += delimiter + other
  11  
  12 # custom string object
custom_str = CustomStrClass('My
  13
Custom')
  14 custom_str.concatenate('String Class')
  15 print(custom_str)
     
[35]:   My Custom String Class
 
OceanofPDF.com
  
QUIZ - Collections
 
Now it’s time to solve the quiz for this
chapter. You can download the quiz file,
QUIZ_Collections.zip, from the GitHub
Repository of this book. You should put the
quiz file in the Collections project we
build in this chapter.
 
Here are the questions for this chapter:
 
Q1:
Here are three dictionaries:
letters = {'a': 'A', 'b': 'B', 'c': 'C'}
numbers = {1: 1000, 2: 2000, 3: 3000}
fruits = {'orange': 300, 'apple': 500}
 
Define a ChainMap by using letters and numbers
dictionaries.
The name of the ChainMap will be my_map.
Print the maps in my_map.
Print all of the keys and values in my_map as: "Key
: Value".
Add fruits dictionary to the my_map.
Print the maps in this new ChainMap.
Print all of the keys and values in this new
ChainMap as: "Key : Value".
 
Hints:
* new_child()
* new_child() will return a new ChainMap
 
[1]: 1 # Q 1:
  2  
  3 # import ChainMap
  4 # ---- your solution here ---
  5  
  6 # dictionaries
  7 # ---- your solution here ---
  8  
  9 # define the ChainMap
  10 # ---- your solution here ---
  11  
  12 # print maps
  13 # ---- your solution here ---
  14  
  15 # print key: value pairs
  16 # ---- your solution here ---
  17  
  18 # add fruits dictionary
  19 # ---- your solution here ---
  20  
  21 # print maps
  22 # ---- your solution here ---
  23  
  24 # print key: value pairs
  25 # ---- your solution here ---
     
[1]:   Mappings in my_map:
    [{'a': 'A', 'b': 'B'}, {1: 100, 2: 200}]
     
    Key:Value pairs in my_map:
    1: 100
    2: 200
    a: A
    b: B
     
    Mappings in new_map:
[{'orange': 33, 'apple': 55}, {'a': 'A', 'b':
   
'B'}, {1: 100, 2: 200}]
     
    Key:Value pairs in new_map:
    1: 100
    2: 200
    a: A
    b: B
    orange: 33
    apple: 55
 
Q2:
Here is some text:
'''Lorem ipsum dolor sit amet, consectetur
adipiscing elit.
Quisque bibendum sodales ipsum in lacinia.
Morbi metus dui, venenatis ut molestie a, porta sit
amet est.
Maecenas sagittis turpis nec nisl tempus
consequat.'''
 
Define a Counter which will store all the characters
and their respective counts in this text.
 
Print the most common 5 characters in the text.
 
Hints:
* Counter
* most_common()
* we don't want to count the space character
 
[2]: 1 # Q 2:
  2  
  3 # import Counter class
  4 from collections import Counter
  5  
  6 # define the text
  7 # ---- your solution here ---
  8  
  9 # remove the space character
  10 # ---- your solution here ---
  11  
  12 # define the Counter
  13 # ---- your solution here ---
  14  
  15 # print most common 5 characters
  16 # ---- your solution here ---
     
[('i', 21), ('e', 20), ('s', 20), ('t', 18), ('a',
[2]:  
13)]
 
Q3:
Create a deque object with name my_deque.
my_deque will have first four letters in the
alphabet: a, b, c, d.
Extend this deque from the left by adding three
more letters: e, f, g.
Print the items in my_deque.
Create a new deque (new_deque) using my_deque.
new_deque will be the reverse of my_deque.
Print the items in new_deque.
Please be careful that, my_deque object will not be
modified.
 
Hints:
* deque
* extendleft()
* reverse()
 
[3]: 1 # Q 3:
  2  
  3 # import Deque class
  4 # ---- your solution here ---
  5  
  6 # define a new deque
  7 # ---- your solution here ---
  8  
  9 # extend my_deque from left
  10 # ---- your solution here ---
  11  
  12 # print items in my_deque
  13 # ---- your solution here ---
  14  
  15 # copy my_deque
  16 # ---- your solution here ---
  17  
  18 # reverse my_deque
  19 # ---- your solution here ---
  20  
  21 # print items in new_deque
  22 # ---- your solution here ---
     
[3]:   ['g', 'f', 'e', 'a', 'b', 'c', 'd']
    ['d', 'c', 'b', 'a', 'e', 'f', 'g']
 
Q4:
We have a long word in English:
incomprehensibilities.
We want to count occurrences of each letter in this
word.
Print the most common three letters in this word.
 
Hints:
* use defaultdict
* default_factory
* do not use Counter
 
[4]: 1 # Q 4:
  2  
  3 # import defaultdict
  4 # ---- your solution here ---
  5  
  6 # define the text
  7 # ---- your solution here ---
  8  
  9 # define the defaultdict
  10 # ---- your solution here ---
  11  
  12 # fill the defaultdict
  13 # ---- your solution here ---
  14  
  15 # sort items in the defualtdict
# by number of occurrences in
 
16 decreasing order
  17 # ---- your solution here ---
  18  
  19 # print the most common 3 words
  20 # ---- your solution here ---
     
[4]:   [('i', 5), ('e', 3), ('n', 2)]
 
Q5:
Define a namedtuple as SuperHero.
SuperHero has three attributes:
* hero_name
* real_name
* age
Create two SuperHero instance as follows:
* Spiderman, Peter Parker, 18
* Superman, Clark Kent, 26
 
Print the real names of SuperHeroes using keys as
follows:
* Real name of Spiderman is: Peter Parker
* Real name of Superman is: Clark Kent
 
Hints:
* namedtuple
* attributes are keys as: namedtuple.key
 
[5]: 1 # Q 5:
  2  
  3 # import namedtuple
  4 # ---- your solution here ---
  5  
  6 # Declare the namedtuple
  7 # ---- your solution here ---
  8  
  9 # Create SuperHero instances
  10 # ---- your solution here ---
  11  
  12 # Access using key
  13 # ---- your solution here ---
     
 [5]:   Real name of Spiderman is: Peter Parker
    Real name of Superman is: Clark Kent
 

OceanofPDF.com
 
SOLUTIONS - Collections
 
Here are the solutions for the quiz for
this chapter.
 
S1:
 
[1]: 1 # S 1:
  2  
  3 # import ChainMap
  4 from collections import ChainMap
  5  
  6 # dictionaries
  7 letters = {'a': 'A', 'b': 'B'}
  8 numbers = {1: 100, 2: 200}
  9 fruits = {'orange': 33, 'apple': 55}
  10  
  11 # define the ChainMap
  12 my_map = ChainMap(letters, numbers)
  13  
  14 # print maps
print('Mappings in my_map:\n',
 
15 my_map.maps)
  16  
  17 # print key: value pairs
  18 print('\nKey:Value pairs in my_map:')
  19 for key, value in my_map.items():
  20     print(f'{key}: {value}')
  21  
  22 # add fruits dictionary
  23 new_map = my_map.new_child(fruits)
  24  
  25 # print maps
print('Mappings in new_map:\n',
 
26 new_map.maps)
  27  
  28 # print key: value pairs
  29 print('\nKey:Value pairs in new_map:')
  30 for key, value in new_map.items():
  31     print(f'{key}: {value}')
     
[1]:   Mappings in my_map:
    [{'a': 'A', 'b': 'B'}, {1: 100, 2: 200}]
     
    Key:Value pairs in my_map:
    1: 100
    2: 200
    a: A
    b: B
     
    Mappings in new_map:
[{'orange': 33, 'apple': 55}, {'a': 'A', 'b':
   
'B'}, {1: 100, 2: 200}]
     
    Key:Value pairs in new_map:
    1: 100
    2: 200
    a: A
    b: B
    orange: 33
    apple: 55
 
S2:
 
[2]: 1 # S 2:
  2  
  3 # import Counter class
  4 from collections import Counter
  5  
  6 # define the text
text = '''Lorem ipsum dolor sit amet,
 
7 consectetur adipiscing elit.
Quisque bibendum sodales ipsum in
 
8 lacinia.
Morbi metus dui, venenatis ut molestie
 
9 a, porta sit amet est.
Maecenas sagittis turpis nec nisl tempus
 
10 consequat.'''
  11  
  12 # remove the space character
  13 text = ''.join(text.split())
  14  
  15 # define the Counter
  16 text_counter = Counter(text)
  17  
  18 # print most common 5 characters
most_common_5 =
 
19 text_counter.most_common(5)
  20 print(most_common_5)
     
[('i', 21), ('e', 20), ('s', 20), ('t', 18), ('a',
[2]:  
13)]
 
S3:
 
[3]: 1 # S 3:
  2  
  3 # import Deque class
  4 from collections import deque
  5  
  6 # define a new deque
  7 my_deque = deque(['a', 'b', 'c', 'd'])
  8 print(list(my_deque))
  9  
  10 # extend my_deque from left
  11 my_deque.extendleft(['e', 'f', 'g'])
  12  
  13 # print items in my_deque
  14 print(list(my_deque))
  15  
  16 # copy my_deque
  17 new_deque = my_deque.copy()
  18  
  19 # reverse my_deque
  20 new_deque.reverse()
  21  
  22 # print items in new_deque
  23 print(list(new_deque))
     
[3]:   ['g', 'f', 'e', 'a', 'b', 'c', 'd']
    ['d', 'c', 'b', 'a', 'e', 'f', 'g']
 
S4:
 
[4]: 1 # S 4:
  2  
  3 # import defaultdict
  4 from collections import defaultdict
  5  
  6 # define the text
  7 word = "incomprehensibilities"
  8  
  9 # define the defaultdict
  10 def_dict = defaultdict(int)
  11  
  12 # fill the defaultdict
  13 for letter in word:
  14     def_dict[letter] += 1
  15  
  16 # sort items in the defualtdict
# by number of occurrences in
 
17 decreasing order
sorted_items = sorted(def_dict.items(),
 
18 key=lambda k_v: k_v[1], reverse=True)
  19  
  20 # print the most common 3 words
  21 print(sorted_items[:3])
     
[4]:   [('i', 5), ('e', 3), ('n', 2)]
 
 
S5:
 
[5]: 1 # S 5:
  2  
  3 # import namedtuple
  4 from collections import namedtuple
  5  
  6 # Declare the namedtuple
SuperHero = namedtuple('SuperHero',
 
7 ['hero_name', 'real_name', 'age'])
  8  
  9 # Add some values to the tuple
  10 S_1 = SuperHero('Spiderman', 'Peter
Parker', '18')
S_2 = SuperHero('Superman', 'Clark Kent',
 
11 '26')
  12  
  13 # Access using key
print(f"Real name of {S_1.hero_name} is:
 
14 {S_1.real_name}")
print(f"Real name of {S_2.hero_name} is:
 
15 {S_2.real_name}")
     
[5]:   Real name of Spiderman is: Peter Parker
    Real name of Superman is: Clark Kent
 
 
OceanofPDF.com
 4. Iterators
     

 
OceanofPDF.com
 Iterables and Iterators
 
Iterator :
An Iterator is an object representing a
stream of data and can be iterated upon.
In technical terms, a Python iterator is an
object that implements the iterator
protocol, which consist of two special
methods: __iter__() and __next__() .
 
Iterable :
An Iterable is an object capable of
returning its members one at a time.
Examples of iterables include all sequence
types (such as list, str, and tuple) and
some non-sequence types like dict, and
file objects. Technically, iterables are
objects of any classes with an __iter__()
method or with a __getitem__() method.
 
iter() :
The iter() function (which calls the
__iter__() method behind the scenes)
returns an iterator object. So, we can say
that; an iterable is an object which
returns an iterator.
 
In this chapter, we will learn how
iterators work in Python and how we can
define our own iterator classes. You can
find the PyCharm project for this chapter
in the GitHub Repository of this book.
 
Chapter Outline:
The Iterator Protocol
Looping Through an Iterator
Define a Custom Iterator
Infinite Iterators
Benefits of Iterators
QUIZ – Iterators
SOLUTIONS - Iterators
 
 
OceanofPDF.com
 The Iterator Protocol
 
In Python, Iterator objects are required
to support the following two methods,
which together form the Iterator Protocol :
__iter__() :
Returns the iterator object itself. This is
required to allow both containers and
iterators to be used with the for loops and
the in statements. You can use built-in
iter() function which calls the __iter__()
method.
__next__() :
Returns the next item from an iterator. If
there are no further items, raise the
StopIteration exception. You can use built-
in next() function which calls the __next__()
method.
 
As we learned in the previous section,
lists, tuples, dictionaries, and sets are all
iterable types. In other words, they are
the types which you can get an iterator
from. Let’s see some examples:
 
[1]: 1 # define a tuple (iterable)
  2 tup = ("A", "B", "C")
# get an iterator from the iterable ->
  3
iter()
  4 tup_iter = iter(tup)
  5  
  6 # call the next() function
  7 item_1 = next(tup_iter)
  8 print(item_1)
  9  
  10 # call the next() function
  11 item_2 = next(tup_iter)
  12 print(item_2)
  13  
  14 # call the next() function
  15 item_3 = next(tup_iter)
  16 print(item_3)
     
[1]:   A
    B
    C
 
In cell 1, we define a tuple which is an
iterable. Then in line 4, we call the iter()
function on this iterable. The iter()
function returns an iterator and we name
it as tup_iter . In lines 7 to 16 we call the
next() function several times. Each time
the next() function executes, it returns the
next item in the iterator.
 
[2]: 1 # define a string (iterable)
  2 pyt = 'python'
# get an iterator from the iterable ->
  3
iter()
  4 pyt_iter = pyt.__iter__()
  5  
  6 # call the next() function
  7 item_1 = pyt_iter.__next__()
  8 print(item_1)
  9  
  10 # call the next() function
  11 item_2 = pyt_iter.__next__()
  12 print(item_2)
     
[2]:   p
    y
 
In cell 2, we call the __ iter__() method
on a string object. Strings are iterable
objects that contain a sequence of
characters. The __ iter__() method returns
an iterator in line 4. And we print the
elements in this iterator one by one by
calling the __ next__() method.
 
 
OceanofPDF.com
 Looping Through an Iterator
 
As we see in the previous sections, we
use the next() function (or __next__()
method) to manually iterate over the
items of an iterator. When the next()
function reaches the end of the iterator,
then there is no more data to be returned
and you will get an StopIteration
exception.
 
[3]: 1 # define a list (iterable)
  2 a_list = [10, 20, 30]
  3  
# get an iterator from the iterable ->
  4
iter()
  5 list_iter = iter(a_list)
  6  
  7 # call the next() function
  8 print(next(list_iter))
  9 print(next(list_iter))
  10 print(next(list_iter))
  11  
# the following next() call will raise
  12
exception
  13 print(next(list_iter))
     
[3]:   10
    20
    30
    StopIteration
 
In cell 3, we call the next() function four
times which is more than the number of
items in the iterator. And in the last call
we get StopIteration exception.
The for loop in Python, is able to iterate
automatically through any object that can
return an iterator. In other words, the for
loop can iterate over any iterable object in
Python.
 
[4]: 1 # for loop
  2 for element in a_list:
  3     print(element)
     
[4]:   10
    20
    30
 
In cell 4, we use the for loop to iterate
over the list we defined in cell 3. As you
see, we do not use the next() function
manually or we don’t get any StopIteration
exception. That’s the beauty of the for
loop in Python. It handles all of these for
us behind the scenes.
 Now let’s define our own version of the
for loop. We will use the while loop and
copy the behavior of the for loop. At this
point, we have everything we need for this
implementation. Let’s do it:
 
[5]: 1 # custom for loop implementation
  2 # iterable
  3 my_list = [1, 2, 3]
  4  
  5 # iterator from this iterable
  6 list_iter = my_list.__iter__()
  7  
  8 # while loop
  9 while True:
  10     try:
  11         # next item
  12         element = list_iter.__next__()
  13         print(element)
  14     except StopIteration:
  15         break
     
[5]:   1
    2
    3
 
In cell 5, we implement our own version
of the for loop. We use an infinite while
loop as: while True . We set a try-except
block inside the loop. In the try block, we
get the next element by calling the
__next__() method on our iterator. If this call
is successful then we print the element. If
an error occurs, which is of type
StopIteration , then we catch that exception
in the except block. What we do inside the
except block is very simple. We simply
break the loop. Which means we have
already reached the end of our iterator.
 
 
OceanofPDF.com
 Define a Custom Iterator
 
Now that we know about iterators and
iterator protocol ( __iter__() and __next__()
methods) we can define our own iterator
classes from scratch.
Defining an iterator is quite easy in
Python. All we have to do is to implement
__iter__() and __next__() methods in our
class definition. Here are the general rules
that we must follow:
The __iter__() method must return
the iterator object itself.
The __next__() method must return
the next item in the sequence. Also,
it has to raise a StopIteration
exception when it reaches the end of
the sequence.
 
Let’s define an iterator object which will
generate a sequence of odd numbers like
1, 3, 5, 7, 9, … etc.
 
[6]: 1 # custom iterator
  2 class Odd:
  3     # implement __init__ method
  4     def __init__(self, limit):
  5         self.current = 1
  6         self.limit = limit
  7  
  8     # implement __iter__ method
  9     # simply return the object itself
  10     def __iter__(self):
  11         return self
  12  
  13     # implement __next__ method
  14     def __next__(self):
  15         # check if limit reached
  16         if self.current <= self.limit:
  17             # get the current value
  18             current_value = self.current
  19             # increase the current
  20             self.current += 2
  21             return current_value
        # limit is reached so raise
  22
exception
  23         else:
  24             raise StopIteration
 
In cell 6, we define our own iterator
class. It implements both __iter__() and
__next__() methods:
In the __iter__() method, it returns
the object itself as: return self .
In the __next__() method, it checks if
the current value is smaller than or
 equal to the limit . If this is True , then
it returns the existing value of
self.current and increases the
self.current value by 2 . If the limit is
exceeded than it will raise a
StopIteration exception.
 
Now let’s call this class and get some
odd numbers up to 20:
 
[7]: 1 # instantiate an Odd object
  2 odd_numbers = Odd(20)
  3  
  4 # get first 4 odd numbers
  5 print(odd_numbers.__next__())
  6 print(odd_numbers.__next__())
  7 print(next(odd_numbers))
  8 print(next(odd_numbers))
     
[7]:   1
    3
    5
    7
 
In cell 7, we instantiate an object from
our Odd class. This object will hold the
odd numbers from 1 to 20. And we print
 the first four odd numbers by calling the
next() method on this object four times.
 
Since our Odd class is an iterator, we can
easily set a for loop to iterate over it.
Let’s do it:
 
[8]: 1 # iterate over Odd class
  2 for n in Odd(8):
  3     print(n)
     
[8]:   1
    3
    5
    7
 
 
OceanofPDF.com
 Infinite Iterators
 
Infinite Iterators are special type objects
which has no terminating conditions in
their __next__() methods. They can be
useful when you need to set a counter
that you do not know where it will finalize.
Let’s define a custom infinite iterator that
keeps increasing one by one.
 
[9]: 1 class InfiniteCounter:
  2     def __init__(self):
  3         self.n = 1
  4  
  5     def __iter__(self):
  6         return self
  7  
  8     def __next__(self):
  9         current = self.n
  10         self.n += 1
  11         return current
 
Now let’s call our InfiniteCounter class
and get some numbers in ascending
order:
 
[10]: 1 # get first four numbers
  2 counter = InfiniteCounter()
  3 print(counter.__next__())
  4 print(counter.__next__())
  5 print(next(counter))
  6 print(next(counter))
     
[10]:   1
    2
    3
    4
 
 
OceanofPDF.com
 Benefits of Iterators
 
Use of an iterator simplifies the code
and makes it more efficient instead of
using a list. For small datasets, iterator
and list-based approaches have similar
performance. But for larger datasets,
iterators save both time and memory.
 
Here are some primary benefits of using
iterators:
Iterators provide cleaner code
Theoretically, iterators can work with
infinite sequences.
Iterators save resources. An Iterator
stores only one element in the
memory, while list (or tuple) stores
all the elements.
Iterator treats variables of all types,
sizes, and shapes uniformly, whether
they fit in memory or not.
Iterator makes recursion
unnecessary in handling arrays of
arbitrary dimensionality.
 Iterator supports iterating over
multiple variables concurrently,
because each variable's iteration
state is maintained in its own
iterator structure.
 
OceanofPDF.com
  
QUIZ - Iterators
 
Now it’s time to solve the quiz for this
chapter. You can download the quiz file,
QUIZ_Iterators.zip, from the GitHub
Repository of this book. You should put the
quiz file in the Iterators project we build
in this chapter.
 
Here are the questions for this chapter:
 
Q1:
Define an iterator class which will generate a
sequence of even numbers like 0, 2, 4, 6, 8, … etc.
Class name will be Even.
Instantiate an object of this class with name
even_numbers.
The even_numbers object will be an iterator that
provides even numbers from 0 to 20.
Print the even numbers using this object.
 
Hints:
* __init__()
* __iter__()
* __next__()
 
[1]: 1 # Q 1:
  2  
  3 # define the iterator class
  4 # ---- your solution here ---
  5  
  6 # instantiate an Odd object
  7 # ---- your solution here ---
  8  
  9 # print even numbers
  10 # ---- your solution here ---
     
[1]:   0
    2
    4
    6
    8
    10
    12
    14
    16
    18
    20
 
Q2:
Define an infinite iterator named Counter.
Counter has no terminating condition.
It starts from 10 and keeps increasing by 10 at each
iteration.
Instantiate an object of this class with name
counter.
The counter object will be an iterator that provides
numbers which are multiples of 10.
Print first 5 multiplies of 10 by using the counter
object.
 
Hints:
* __init__()
* __iter__()
* __next__()
 
[2]: 1 # Q 2:
  2  
  3 # define the Counter class
  4 # ---- your solution here ---
  5  
  6 # create an object
  7 # ---- your solution here ---
  8  
  9 # print first 5 numbers
  10 # ---- your solution here ---
     
[2]:   10
    20
    30
    40
    50
 
 
Q3:
We have a tuple of fruits:
fruits = ("orange", "apple", "cherry")
 
Convert this tuple to an iterator and print the fruits
in it.
Get the StopIteration exception by trying to call the
iterator.
 
Hints:
* iter()
* do not use loops
 
[3]: 1 # Q 3:
  2  
  3 # define the tuple
  4 # ---- your solution here ---
  5  
  6 # convert tuple to an iterator
  7 # ---- your solution here ---
  8  
  9 # print items in the iterator
  10 # ---- your solution here ---
  11  
  12 # get StopIteration exception
  13 # ---- your solution here ---
     
[3]:   orange
    apple
    cherry
    StopIteration
 
Q4:
We have a string object as:
movie = "Inception"
 
Convert this string to an iterator and print the
letters in it.
 
Hints:
* iter()
* use for loop
 
[4]: 1 # Q 4:
  2  
  3 # define the string
  4 # ---- your solution here ---
  5  
  6 # convert tuple to an iterator
  7 # ---- your solution here ---
  8  
  9 # print items in the iterator
  10 # ---- your solution here ---
     
[4]:   I
    n
    c
    e
    p
    t
    i
    o
    n
 
Q5:
Define a class named Numbers that implements
iterator protocol (__iter__ and __next__ methods).
It will return integers starting from 1 and will
increase by one at each iteration:
1, 2, 3, 4, 5, ...
Instantiate an object of this class with name
numbers.
Convert the numbers object to an iterator.
Print first 4 integers using the numbers object.
 
Hints:
* no __init__() method in the class
* iter()
* next()
* no for loops
 
[5]: 1 # Q 5:
  2  
  3 # define the Numbers class
  4 # ---- your solution here ---
  5  
  6 # instantiate the numbers object
  7 # ---- your solution here ---
  8  
# convert the numbers object to an
 
9 iterator
  10 # we have to do this explicitly because
  11 # the Numbers has no __init__() method
  12 # ---- your solution here ---
  13  
  14 # print first 4 integers
  15 # ---- your solution here ---
     
[5]:   1
    2
    3
    4

OceanofPDF.com
 
SOLUTIONS - Iterators
 
Here are the solutions for the quiz for
this chapter.
 
S1:
 
[1]: 1 # S 1:
  2  
  3 # define the iterator class
  4 class Even:
  5     # implement __init__ method
  6     def __init__(self, limit):
  7         self.current = 0
  8         self.limit = limit
  9  
  10     # implement __iter__ method
  11     # simply return the object itself
  12     def __iter__(self):
  13         return self
  14  
  15     # implement __next__ method
  16     def __next__(self):
  17         # check if limit reached
  18         if self.current <= self.limit:
  19             # get the current value
  20             current_value = self.current
  21             # increase the current
  22             self.current += 2
  23             return current_value
  24         # limit is reached so raise
exception
  25         else:
  26             raise StopIteration
  27  
  28 # instantiate an Odd object
  29 even_numbers = Even(20)
  30  
  31 # print even numbers
  32 for e in even_numbers:
  33     print(e)
     
[1]:   0
    2
    4
    6
    8
    10
    12
    14
    16
    18
    20
 
S2:
 
[2]: 1 # S 2:
  2  
  3 # define the Counter class
  4 class Counter:
  5     def __init__(self):
  6         self.n = 10
  7  
  8     def __iter__(self):
  9         return self
  10  
  11     def __next__(self):
  12         current = self.n
  13         self.n += 10
  14         return current
  15  
  16 # create an object
  17 counter = Counter()
  18  
  19 # print first 5 numbers
  20 for i in range(5):
  21     print(next(counter))
     
[2]:   10
    20
    30
    40
    50
 
S3:
 
[3]: 1 # S 3:
  2  
  3 # define the tuple
  4 fruits = ("orange", "apple", "cherry")
  5  
  6 # convert tuple to an iterator
  7 fuits_iter = iter(fruits)
  8  
  9 # print items in the iterator
  10 print(next(fuits_iter))
  11 print(next(fuits_iter))
  12 print(next(fuits_iter))
  13  
  14 # get StopIteration exception
  15 print(next(fuits_iter))
     
[3]:   orange
    apple
    cherry
    StopIteration
 
S4:
 
[4]: 1 # S 4:
  2  
  3 # define the string
  4 movie = "Inception"
  5  
  6 # convert tuple to an iterator
  7 movie_iter = iter(movie)
  8  
  9 # print items in the iterator
  10 for letter in movie_iter:
  11     print(letter)
     
[4]:   I
    n
    c
    e
    p
    t
    i
    o
    n
 
S5:
 
[5]: 1 # S 5:
  2  
  3 # define the Numbers class
  4 class Numbers:
  5     # no __init__() method
  6  
  7     def __iter__(self):
  8         self.current = 1
  9         return self
  10  
  11     def __next__(self):
  12         n = self.current
  13         self.current += 1
  14         return n
  15  
  16 # instantiate the numbers object
  17 numbers = Numbers()
  18  
# convert the numbers object to an
 
19 iterator
  20 # we have to do this explicitly because
  21 # the Numbers has no __init__() method
  22 numbers_iter = iter(numbers)
  23  
  24 # print first 4 integers
  25 print(next(numbers_iter))
  26 print(next(numbers_iter))
  27 print(next(numbers_iter))
  28 print(next(numbers_iter))
     
[5]:   1
    2
    3
    4
 
 
OceanofPDF.com
 5. Generators
     

 
OceanofPDF.com
 What is a Generator?
 
In the previous chapter we learned
about Iterators, which are great tools
especially when you need to deal with
large datasets. However, building an
iterator in Python is a bit cumbersome and
time consuming. You have to define a new
class which implements the iterator
protocol ( __iter__() and __next__()
methods). In this class, you need to
manage internal state of the variables and
update them. Moreover you need to raise
StopIteration exception when there is no
value to return back in the __next__()
method.
Fortunately, we have an elegant
solution for this in Python. Python provides
Generators to help you easily create
iterators. A Generator allows you to
declare a function that behaves like an
iterator, i.e., it can be used in a for loop.
In simple terms, a Generator is a function
which returns an iterator object. So it’s an
easy way of creating iterators. You don’t
need to think about all the work needed
when you create an iterator, because the
Generator will handle all of them.
In this chapter, we will learn how
generators work in Python and how we
can define them. You can find the
PyCharm project for this chapter in the
GitHub Repository of this book.
 
Chapter Outline:
Defining Generators
Generator Function vs. Normal
Function
Generator Expression
Benefits of Generators
QUIZ – Generators
SOLUTIONS – Generators
 
 
OceanofPDF.com
 Defining Generators
 
As stated in the first section, a
generator is a special type of function in
Python. This function does not return a
single value, instead, it returns an iterator
object. In the generator function, we use
the yield statement instead of the return
statement. Let’s define a simple generator
function:
 
[1]: 1 # define a generator function
  2 def first_generator():
  3     print('Yielding First item')
  4     yield 'A'
  5  
  6     print('Yielding Second item')
  7     yield 'B'
  8  
  9     print('Yielding Last item')
  10     yield 'C'
 
In cell 1, we define a generator function.
The function executes the yield statement
instead of the return keyword. The yield
statement is what makes this function a
generator. When we call this function it
will return (yield) an iterator object. Let’s
see it:
 
[2]: 1 # call the generator
  2 iter_obj = first_generator()
  3  
  4 # print first item
  5 first_item = next(iter_obj)
  6 print(first_item)
  7  
  8 # print second item
  9 second_item = next(iter_obj)
  10 print(second_item)
  11  
  12 # print third item
  13 third_item = next(iter_obj)
  14 print(third_item)
     
[2]:   Yielding First item
    A
    Yielding Second item
    B
    Yielding Last item
    C
 
In cell 2, we call the first_generator()
function which is a generator and returns
an iterator object. We name this iterator
as iter_obj . Then we call the next()
function on this iterator object. In each
next() call, the iterator executes the yield
statement in respective order and returns
an item.
As a general rule, the generator
function should not include the return
keyword. Because if it includes, then the
return statement will terminate the
function.
Now let’s define a more realistic
generator by the help of a for loop. In this
example, we want to define a generator
which will keep track of the sequence of
numbers starting from zero and up to a
given maximum limit.
 
[3]: 1 # generator for sequence of numbers
  2 def get_sequence_gen(max):
  3     for n in range(max):
  4         yield n
  5  
  6 # call the function and get iterator
  7 sequence_iter = get_sequence_gen(10)
  8 # call the next() method
  9 print(sequence_iter.__next__())
  10 print(sequence_iter.__next__())
  11 print(next(sequence_iter))
  12 print(next(sequence_iter))
     
[3]:   0
    1
    2
    3
 
In cell 3, we define a generator function
which yields the integers from zero up to a
given number. As you see, the yield
statement is inside the for loop. Please be
careful that, the value of n is stored
internally during successive next() calls to
the function.
 
 
OceanofPDF.com
 Generator Function vs. Normal Function
 
A function is a generator function if it
contains at least one yield statement. It
may contain other yield or return
statements if needed. Both yield and
return keywords will return “something”
from a function.
The difference between return and yield
keywords is very crucial for generators.
While the return statement terminates a
function entirely, yield statement pauses
the function saving all its states and later
continues from there on successive calls.
We call a generator function in the same
way we call a normal one. During its
execution, the generator pauses when it
encounters the yield keyword. It sends the
current value of the iterator stream to the
calling environment and wait for the next
call. Meanwhile, it saves the local
variables and their states internally.
 
Below are the key points where a
generator function differs from a normal
function:
A Generator function returns (yields)
an iterator object. You don’t need to
worry about creating this iterator
object explicitly, the yield keywords
does this for you.
A Generator function must contain at
least one yield statement. It may
include multiple yield keywords if
needed.
A Generator function implements the
iterator protocol ( iter() and next()
methods) internally.
A Generator function saves the local
variables and their states
automatically.
A Generator function pauses
execution at the yield keyword and
pass the control to the caller.
A Generator function raises the
StopIteration exception automatically
when the iterator stream has no
value to return.
 
Let’s consider a simple example to
demonstrate the difference between a
normal function and a generator function.
In this example, we want to calculate the
sum of first n positive integers. To do this,
we will define a function that gives us the
list of first n positive numbers. We will
implement this function in both ways, a
normal function and a generator.
Here is the code for the normal function:
 
[4]: 1 # import the time module
  2 from time import time
  3  
  4 # Normal Function
  5 def first_n_numbers(max):
  6     n, numbers = 1, []
  7     while n <= max:
  8         numbers.append(n)
  9         n += 1
  10     return numbers
  11  
  12 # call the function
  13 start = time()
  14 first_n_list = first_n_numbers(99999999)
  15 sum_of_first_numbers = sum(first_n_list)
  16 print(sum_of_first_numbers)
  17 # elapsed time
  18 end = time()
print("Elapsed Time in seconds:", end -
  19
start)
     
[4]:   4999999950000000
    Elapsed Time in seconds: 17.859
 
In cell 4, we define a normal function
that returns the list of first n positive
integers. When we call this function it
takes a while to complete execution
because the list it creates is huge. It also
uses a considerable amount of memory to
complete this task.
Now let’s define a generator function for
the same operation:
 
[5]: 1 # Generator Function
  2 def first_n_numbers_gen(max):
  3     n = 1
  4     while n <= max:
  5         yield n
  6         n += 1
  7  
  8 # call the function
  9 start = time()
first_n_list =
  10
first_n_numbers_gen(99999999)
  11 sum_of_first_numbers = sum(first_n_list)
  12 print(sum_of_first_numbers)
  13 # elapsed time
  14 end = time()
print("Elapsed Time in seconds:", end -
  15
start)
     
[5]:   4999999950000000
    Elapsed Time in seconds: 15.302
 
As you see in cell 5, the generator
finishes the same task in less time and it
uses less memory resources. Because the
generator yields items one by one instead
of returning the complete list.
The main reason for performance
improvement (when we use generators) is
the lazy generation of values. This on
demand value generation, results in lower
memory usage. One more advantage of
generators is, you do not need to wait
until all the elements have been
generated before you start to use them.
 
 
OceanofPDF.com
 Generator Expression
 
There are times that you need simple
generators for relatively simple tasks in
your code. This is where the Generator
Expression comes in. You can easily create
simple generators on the fly using
generator expressions.
Generator Expressions are similar to
lambda functions in Python. Remember
that, lambda’s are anonymous functions
which let us create one-line functions on
the fly. Just like a lambda function, a
generator expression creates an
anonymous generator function.
The syntax of a generator expression
looks like a list comprehension. The
difference is, we have parentheses instead
of square brackets in a generator
expression. Let’s see an example:
 
[6]: 1 # define a simple list
  2 nums = [1, 2, 3, 4, 5]
  3  
  4 # list comprehension
  5 num_cubes = [i**3 for i in nums]
  6  
  7 # generator expression
  8 cubes_gen = (i**3 for i in nums)
  9  
  10 # print both objects
  11 print(num_cubes)
  12 print(cubes_gen)
     
[6]:   [1, 8, 27, 64, 125]
<generator object <genexpr> at
   
0x103059d20>
 
In cell 6, line 8 we define a simple
generator with the help of the generator
expression. Here is the syntax: cubes_gen =
(i**3 for i in nums) . And you see the
generator object in the output. As we
already know, to be able to get the items
in a generator we either need to call the
next() method explicitly or use a for loop
to iterate over the generator. Let’s print
the items in the cubes_gen object:
 
[7]: 1 # loop over generator
  2 for item in cubes_gen:
  3     print(item)
     
[7]:   1
    8
    27
    64
    125
 
Let’s do another example. We will define
a generator that converts the letters of a
string to uppercase. Then we will call the
next() method to print first two letters.
 
[8]: 1 # generator for upper case
  2 text = 'machine learning'
  3 upper_gen = (l.upper() for l in text)
  4 print(upper_gen.__next__())
  5 print(next(upper_gen))
     
[8]:   M
    A
 
 
OceanofPDF.com
 Benefits of Generators
 
Benefits are great tools especially when
you need to deal with large data in
relatively limited memory. Here are some
key benefits of using generators in Python:
 
Memory Efficiency:
Let’s assume, you have a normal
function that returns a very large
sequence. A list with millions of items, for
example. You have to wait for this function
to finish all the execution and return you
the list as a whole. This is obviously not
efficient in terms of time and memory
resources. On the other hand, if you use a
generator function, it will return you the
items one by one, and you will have the
chance to continue to execute the next
lines of code. You don’t need to wait for all
of the items in the list to be executed by
the function. Because the generator will
give (yield) you one item at a time.
 
Lazy Evaluation:
 Generators provide the power of lazy
evaluation. Lazy evaluation is computing
a value when it is really needed, not when
it is instantiated. Let’s assume you have a
large dataset to compute; lazy evaluation
allows you to start using the data
immediately while the whole data set is
still being computed. Because you do not
need the whole data set if you are using a
generator.
 
Implement and Readability:
Generators are very easy to implement
and provide code readability. Remember
that, you do not need to worry about the
__iter__() and __next__() methods if you are
using a generator. All you need is a simple
yield statement in your function.
 
Dealing with Infinite Streams:
Generators are wonderful tools when
you need to represent an infinite stream of
data. An infinite counter, for example. In
theory, you cannot store an infinite stream
in the memory. You cannot be sure about
how much size you will need to store an
infinite stream. This is where a generator
really shines, since it produces only one
item at a time, it can represent an infinite
stream of data. And it doesn’t have to
store all the stream in the memory.
OceanofPDF.com
  
 
QUIZ - Generators
 
Now it’s time to solve the quiz for this
chapter. You can download the quiz file,
QUIZ_Generators.zip, from the GitHub
Repository of this book. You should put the
quiz file in the Generators project we
build in this chapter.
 
Here are the questions for this chapter:
 
Q1:
Define a generator function that returns (yields) the
list of first n positive and even integers.
Here is a sample list: 2, 4, 6, 8, 10, ...
Function name will be positive_evens and it will
take one parameter, n.
This will be the upper limit in the list.
 
Call this function and print positive and even
integers up to 20.
 
Hints:
* yield
* while loop
 
[1]: 1 # Q 1:
  2  
  3 # define the function
  4 # ---- your solution here ---
  5  
  6 # get positive and even numbers up to 20
  7 # ---- your solution here ---
  8  
  9 # convert the generator to a list and print
  10 # ---- your solution here ---
     
[1]:   [2, 4, 6, 8, 10, 12, 14, 16, 18, 20]
 
Q2:
The Fibonacci Sequence is the series of numbers:
1, 1, 2, 3, 5, 8, 13, 21, 34, ...
 
The next number is found by adding up the two
numbers before it:
* the 2 is found by adding the two numbers before
it (1+1),
* the 3 is found by adding the two numbers before
it (1+2),
* the 5 is (2+3),
* and so on!
 
Define a generator function which yields the
Fibonacci Sequence.
Function name will be fibonnaci_generator and it
takes one parameter, n.
This parameter is used to get the first n numbers in
the sequence.
For example the first 9 numbers are: 1, 1, 2, 3, 5, 8,
13, 21, 34
 
Print the first 9 numbers by using this function.
 
Hints:
* yield
* for loop
 
[2]: 1 # Q 2:
  2  
  3 # define the generator
  4 # ---- your solution here ---
  5  
  6 # get the first 9 numbers
  7 # ---- your solution here ---
  8  
  9 # print the numbers
  10 # ---- your solution here ---
     
[2]:   1
    1
    2
    3
    5
    8
    13
    21
    34
 
Q3:
Define a generator function named
get_squares_gen.
This function will take a list of integers as the
parameter.
And it will yield the squares of each element one by
one.
 
Call this generator function with the following list:
[1, 2, 3, 4, 5, 6]
 
Print the squares of items in this list.
To do this, first convert the iterator to a list.
 
Hints:
* yield
* for loop
* generator returns an iterator
 
[3]: 1 # Q 3:
  2  
  3 # define the generator function
  4 # ---- your solution here ---
  5  
  6 # define a list of numbers
  7 # ---- your solution here ---
  8  
  9 # call the generator function
  10 # ---- your solution here ---
  11  
  12 # convert the iterator to a list
  13 # ---- your solution here ---
  14  
  15 # print squares list
  16 # ---- your solution here ---
     
[3]:   [1, 4, 9, 16, 25, 36]
 
Q4:
Define a generator expression.
This is going to be a one-line generator.
It will return an iterator which contains the cubes of
odd elements in the numbers list.
Here is the numbers list:
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
 
Convert the resulting iterator to a list.
Then print the items in this list.
 
Hints:
* generator expressions are very similar to list
comprehensions
* generator expression returns an iterator object
 
[4]: 1 # Q 4:
  2  
  3 # define a list of numbers
  4 # ---- your solution here ---
  5  
  6 # define the generator expression
  7 # ---- your solution here ---
  8  
  9 # convert the iterator to a list
  10 # ---- your solution here ---
  11  
  12 # print the cubes of odd numbers
  13 # ---- your solution here ---
     
[4]:   [1, 27, 125, 343, 729]
 
Q5:
What is the difference between return and yield
statements?
 

OceanofPDF.com
 
SOLUTIONS - Generators
 
Here are the solutions for the quiz for
this chapter.
 
S1:
 
[1]: 1 # S 1:
  2  
  3 # define the function
  4 def positive_evens(n):
  5     counter = 1
  6     while counter <= n:
  7         if counter % 2 == 0:
  8             yield counter
  9         counter += 1
  10  
  11 # get positive and even numbers up to 20
  12 first_evens = positive_evens(20)
  13  
  14 # convert the generator to a list and print
  15 print(list(first_evens))
     
[1]:   [2, 4, 6, 8, 10, 12, 14, 16, 18, 20]
 
S2:
 
[2]: 1 # S 2:
  2  
  3 # define the generator
  4 def fibonnaci_generator(n):
  5     first = 1
  6     second = 1
  7     for i in range(n):
  8         yield first
        first, second = second, first +
 
9 second
  10  
  11 # get the first 9 numbers
  12 fibo_9 = fibonnaci_generator(9)
  13  
  14 # print the numbers
  15 for f in fibo_9:
  16     print(f)
     
[2]:   1
    1
    2
    3
    5
    8
    13
    21
    34
 
S3:
 
[3]: 1 # S 3:
  2  
  3 # define the generator function
  4 def get_squares_gen(nums):
  5     for num in nums:
  6         yield num**2
  7  
   8 # define a list of numbers
  9 numbers = [1, 2, 3, 4, 5, 6]
  10  
  11 # call the generator function
  12 squares = get_squares_gen(numbers)
  13  
  14 # convert the iterator to a list
  15 squares_list = list(squares)
  16  
  17 # print squares list
  18 print(squares_list)
     
[3]:   [1, 4, 9, 16, 25, 36]
 
S4:
 
[4]: 1 # S 4:
  2  
  3 # define a list of numbers
  4 numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
  5  
  6 # define the generator expression
odd_cubes_gen = (i**3 for i in numbers if
 
7 i % 2 == 1)
  8  
  9 # convert the iterator to a list
  10 odd_cubes_list = list(odd_cubes_gen)
  11  
  12 # print the cubes of odd numbers
  13 print(odd_cubes_list)
     
[4]:   [1, 27, 125, 343, 729]
 
S5:
 
The difference between return and yield
statements:
return:
The return statement, returns a value from a
function, and terminates the function entirely.
Which means, the function loses its local state.
So, the next time we call this function, it starts
over from its initial state.
 
yield:
The yield statement pauses the function, saving all
its states and later continues from there on
successive calls.
So, it maintains the state between function calls,
and resumes from where it left off when we call the
next() method again.
 
OceanofPDF.com
 6. Date and Time
     

 
OceanofPDF.com
 Date and Time in Python
 
Date and Time objects are very
important in programming. In Python, we
have a dedicated module for date and
time operations. The datetime module
provides classes for manipulating dates
and times. While date and time arithmetic
are supported, the focus of the
implementation of datetime module is on
efficient attribute extraction for output
formatting and manipulation.
 
In this chapter, we will learn how to use
datetime module in Python. You can find
the PyCharm project for this chapter in the
GitHub Repository of this book.
 
Chapter Outline:
Aware and Naive Objects
Constants and Main Classes
Determining if an Object is Aware or
Naive
Timedelta Class
Date Class
 Datetime Class
Time Class
Formatting Date and Time
QUIZ – Date and Time
SOLUTIONS – Date and Time
 
 
OceanofPDF.com
 Aware and Naive Objects
 
Date and time objects may be
categorized as “aware” or “naive”
depending on whether or not they include
timezone information.
With sufficient knowledge of applicable
algorithmic and political time adjustments,
such as time zone and daylight-saving
time information, an aware object can
locate itself relative to other aware
objects. An aware object represents a
specific moment in time that is not open
to interpretation.
A naive object does not contain enough
information to unambiguously locate itself
relative to other date/time objects.
Whether a naive object represents
Coordinated Universal Time (UTC), local
time, or time in some other timezone is
purely up to the program, just like it is up
to the program whether a particular
number represents meters, miles, or
mass. Naive objects are easy to
understand and to work with, at the cost
of ignoring some aspects of reality.
For applications requiring aware objects,
datetime and time objects have an
optional time zone information attribute,
tzinfo , that can be set to an instance of a
subclass of the abstract tzinfo class. These
tzinfo objects capture information about
the offset from UTC time, the time zone
name, and whether daylight-saving time is
in effect.
Only one concrete tzinfo class, the
timezone class, is supplied by the datetime
module. The timezone class can represent
simple timezones with fixed offsets from
UTC, such as UTC itself or North American
EST and EDT timezones. Supporting
timezones at deeper levels of detail is up
to the application. The rules for time
adjustment across the world are more
political than rational, change frequently,
and there is no standard suitable for every
application aside from UTC.
 
 
OceanofPDF.com
 Constants and Main Classes
 
Constants:
The datetime module exports the
following constants:
 
datetime.MINYEAR : The smallest year
number allowed in a date or datetime
object. MINYEAR is 1.
 
datetime.MAXYEAR : The largest year
number allowed in a date or datetime
object. MAXYEAR is 9999.
 
Available Types:
Here are the built-in types in datetime
module in Python. Keep in mind that,
objects of these types are immutable.
 
class datetime.date : An idealized naive
date, assuming the current Gregorian
calendar always was, and always will be,
in effect. Attributes: year, month, and day.
 
 class datetime.time : An idealized time,
independent of any particular day,
assuming that every day has exactly
24*60*60 seconds. (There is no notion of
“leap seconds” here.) Attributes: hour,
minute, second, microsecond, and tzinfo.
 
class datetime.datetime : A combination of
a date and a time. Attributes: year, month,
day, hour, minute, second, microsecond,
and tzinfo.
 
class datetime.timedelta : A duration
expressing the difference between two
date, time, or datetime instances to
microsecond resolution.
 
class datetime.tzinfo : An abstract base
class for time zone information objects.
These are used by the datetime and time
classes to provide a customizable notion
of time adjustment (for example, to
account for time zone and/or daylight-
saving time).
 
 class datetime.timezone : A class that
implements the tzinfo abstract base class
as a fixed offset from the UTC.
 
Common Properties:
The date , datetime , time , and timezone
types share these common features:
Objects of these types are
immutable.
Objects of these types are hashable,
meaning that they can be used as
dictionary keys.
Objects of these types support
efficient pickling via the pickle
module.
 
 
OceanofPDF.com
Determining if an Object is Aware or Naive
 
Naive and aware concepts are crucial
in data and time operations. Here are the
general rules:
Objects of the date type are always
naive.
An object of type time or datetime
may be aware or naive.
A datetime object d is aware if both
of the following hold:
d.tzinfo is not None
d.tzinfo.utcoffset(d) does not
return None
Otherwise, d is naive.
A time object t is aware if both of
the following hold:
t.tzinfo is not None
t.tzinfo.utcoffset(None) does not
return None .
Otherwise, t is naive.
The distinction between aware and
naive doesn’t apply to timedelta
objects.
 
 
OceanofPDF.com
 Timedelta Class
 
A timedelta object represents a duration,
the difference between two dates or
times.
 
class datetime.timedelta(days=0,
seconds=0, microseconds=0, milliseconds=0,
minutes=0, hours=0, weeks=0) :
All arguments are optional and default
to 0. Arguments may be integers or floats,
and may be positive or negative.
Only days, seconds and microseconds
are stored internally. Arguments are
converted to those units:
A millisecond is converted to 1000
microseconds.
A minute is converted to 60 seconds.
An hour is converted to 3600
seconds.
A week is converted to 7 days.
and days, seconds and microseconds
are then normalized so that the
representation is unique, with
0 <= microseconds < 1000000
 0 <= seconds < 3600*24 (the number of
seconds in one day)
-999999999 <= days <= 999999999
 
The following example illustrates how
any arguments besides days, seconds and
microseconds are “merged” and
normalized into those three resulting
attributes:
 
[1]: 1 from datetime import timedelta
  2  
  3 delta = timedelta(
  4      days=50,
  5      seconds=27,
  6      microseconds=10,
  7      milliseconds=29000,
  8      minutes=5,
  9      hours=8,
  10      weeks=2
  11 )
# Only days, seconds, and microseconds
 
12 remain
  13 print("Days:", delta.days)
  14 print("Seconds:", delta.seconds)
print("Microseconds:",
 
15 delta.microseconds)
     
[1]:   Days: 64
    Seconds: 29156
    Microseconds: 10
 
In the next example let’s add days two
timedelta objects:
 
[2]: 1 # Add two timedelta objects
delta1 = timedelta(minutes=10,
 
2 seconds=50)
delta2 = timedelta(hours=25,
 
3 seconds=20)
  4 delta_sum = delta1 + delta2
  5 print(delta_sum)
     
[2]:   1 day, 1:11:10
 
In cell 2, we add two timedelta objects
and print the result. Addition (+),
Subtraction (-), Multiplication (*), Division
(/), Floor Division (//) and Modulo (%)
operations are supported by the timedelta
class in Python.
 
 
OceanofPDF.com
 Date Class
 
A date object represents a date (year,
month and day) in an idealized calendar,
the current Gregorian calendar indefinitely
extended in both directions.
January 1 of year 1 is called day number
1, January 2 of year 1 is called day
number 2, and so on. 2
 
class datetime.date(year, month, day) :
All arguments are required. Arguments
must be integers, in the following ranges:
MINYEAR <= year <= MAXYEAR
1 <= month <= 12
1 <= day <= number of days in the given
month and year
If an argument outside those ranges is
given, ValueError is raised.
 
Class Attributes:
date.min : The earliest representable
date, date(MINYEAR, 1, 1) .
date.max : The latest representable date,
date(MAXYEAR, 12, 31) .
 date.resolution :
The smallest possible
difference between non-equal date
objects, timedelta(days=1) .
 
Instance Attributes (read-only):
date.year : Between MINYEAR and MAXYEAR
inclusive.
date.month : Between 1 and 12 inclusive.
date.day : Between 1 and the number of
days in the given month of the given year.
 
[3]: 1 from datetime import date
  2  
# instantiate a date object: year, month,
 
3 day
  4 a_valid_date = date(2021, 3, 26)
  5 print("This is a valid date:", a_valid_date)
  6  
  7 an_ivalid_date = date(2021, 3, 48)
print("This is an invalid date:",
 
8 an_ivalid_date)
     
[3]:   This is a valid date: 2021-03-26
    ValueError: day is out of range for month
 
date.today() :
Return the current local date. Let’s get
the current date:
 
[4]: 1 # current date
  2 current_date = date.today()
  3 print("Current date is:", current_date)
  4 print("Current year:", current_date.year)
print("Current month:",
 
5 current_date.month)
  6 print("Current day:", current_date.day)
     
[4]:   Current date is: 2022-03-25
    Current year: 2022
    Current month: 3
    Current day: 25
 
date.fromtimestamp(timestamp) :
Return the local date corresponding to
the POSIX timestamp.
POSIX timestamp is the time expressed
as the number of seconds that have
passed since January 1, 1970. That zero
moment, known as the epoch, is simply
the start of the decade in which the Unix
operating system (which first used this
time representation) was invented.
 
# fromtimestamp: get datetime from
[5]:
1 timestamp
date_time_from_timestamp =
 
2 date.fromtimestamp(1527635439)
print("Timestamp to datetime:",
 
3 date_time_from_timestamp)
     
[5]:   Timestamp to datetime: 2018-05-30
 
date.isoformat() :
Return a string representing the date in
ISO 8601 format, YYYY-MM-DD . It is
equivalent to date.__str__() .
 
[6]: 1 # isoformat() -> YYYY-MM-DD
iso_formatted_date = date(2002, 12,
 
2 4).isoformat()
  3 print(iso_formatted_date)
     
[6]:   2002-12-04
 
date.ctime() :
Return a string representing the date:
 
[7]: 1 # isoformat() -> YYYY-MM-DD
iso_formatted_date = date(2002, 12,
 
2 4).isoformat()
  3 print(iso_formatted_date)
     
[7]:   Wed Dec  4 00:00:00 2002
 
date.fromisoformat(date_string) :
Return a date corresponding to a
date_string given in the format YYYY-MM-
DD :
  
[8]: 1 # fromisoformat
date_from_iso_str =
 
2 date.fromisoformat('2019-12-04')
  3 print(date_from_iso_str)
     
[8]:   2019-12-04
 
date.fromordinal(ordinal) :
Return the date corresponding to the
proleptic Gregorian ordinal, where January
1 of year 1 has ordinal 1. ValueError is
raised unless 1 <= ordinal <=
date.max.toordinal() .
For any date d,
date.fromordinal(d.toordinal()) == d.
 
date.strftime(format) :
Return a string representing the date,
controlled by an explicit format string.
Format codes referring to hours, minutes
or seconds will see 0 values.
date.__format__(format) dunder (double
underscore) method also works the same.
 
[9]: 1 # fromordinal() and strftime()
  2 # 730920th day after 1. 1. 0001
  3 d = date.fromordinal(730920)
  4 print(d)
  5  
# Methods related to formatting string
 
6 output
  7 print(d.isoformat())
  8 print(d.strftime("%d/%m/%y"))
  9 print(d.strftime("%A %d. %B %Y"))
     
[9]:   2002-03-11
    2002-03-11
    11/03/02
    Monday 11. March 2002
 
date.isocalendar() :
Return a named tuple object with three
components: year , week and weekday . The
ISO calendar is a widely used variant of
the Gregorian calendar.
The ISO year consists of 52 or 53 full
weeks, and where a week starts on a
Monday and ends on a Sunday. The first
week of an ISO year is the first (Gregorian)
calendar week of a year containing a
Thursday. This is called week number 1,
and the ISO year of that Thursday is the
same as its Gregorian year.
For example, 2004 begins on a
Thursday, so the first week of ISO year
2004 begins on Monday, 29 Dec 2003 and
ends on Sunday, 4 Jan 2004:
 
[10]: 1 # isocalendar()
iso_date_year_end = date(2003, 12,
 
2 29).isocalendar()
  3 print(iso_date_year_end)
  4  
iso_date_year_start = date(2004, 1,
 
5 4).isocalendar()
  6 print(iso_date_year_start)
     
datetime.IsoCalendarDate(year=2004,
[10]:  
week=1, weekday=1)
datetime.IsoCalendarDate(year=2004,
   
week=1, weekday=7)
 
date.replace(year=self.year,
month=self.month, day=self.day) :
Return a date with the same value,
except for those parameters given new
values by whichever keyword arguments
are specified.
 
[11]: 1 # replace()
  2 d = date(2002, 12, 31)
  3 d_new = d.replace(day=26)
  4 print("d:", d)
  5 print("d_new:", d_new)
     
[11]:   d: 2002-12-31
    d_new: 2002-12-26
 
date.fromisocalendar(year, week, day) :
Return a date corresponding to the ISO
calendar date specified by year, week and
day. This is the inverse of the function
date.isocalendar() .
 
date.toordinal() :
Return the proleptic Gregorian ordinal of
the date, where January 1 of year 1 has
ordinal 1. For any date object d,
date.fromordinal(d.toordinal()) == d .
 
date.weekday() :
Return the day of the week as an
integer, where Monday is 0 and Sunday is
6. For example, date(2002, 12, 4).weekday()
== 2 , a Wednesday.
 
date.isoweekday() :
Return the day of the week as an
integer, where Monday is 1 and Sunday is
7. For example, date(2002, 12,
4).isoweekday() == 3 , a Wednesday.
  
date.timetuple() :
Return a time.struct_time such as
returned by time.localtime() , which we will
see later.
 
 
OceanofPDF.com
 Datetime Class
 
A datetime object is a single object
containing all the information from a date
object and a time object. Like a date
object, datetime assumes the current
Gregorian calendar extended in both
directions; like a time object, datetime
assumes there are exactly 3600*24
seconds in every day.
 
datetime.datetime(year, month, day,
hour=0, minute=0, second=0, microsecond=0,
tzinfo=None, *, fold=0) :
The year, month and day arguments are
required. tzinfo may be None , or an
instance of a tzinfo subclass. The
remaining arguments must be integers in
the following ranges:
MINYEAR <= year <= MAXYEAR,
1 <= month <= 12,
1 <= day <= number of days in the given
month and year,
0 <= hour < 24,
0 <= minute < 60,
0 <= second < 60,
0 <= microsecond < 1000000,
fold in [0, 1].
 If an argument outside those ranges is
given, ValueError is raised.
Let’s create a datetime object with
different parameter sets:
 
[12]: 1 from datetime import datetime
  2  
  3 # call the constructor
  4 datetime_obj = datetime(2020, 5, 29)
  5 print(datetime_obj)
  6  
# call the constructor with time
 
7 parameters
datetime_obj_with_time = datetime(2020,
 
8 5, 29, 8, 45, 52, 162420)
  9 print(datetime_obj_with_time)
     
[12]:   2020-05-29 00:00:00
    2020-05-29 08:45:52.162420
 
Now let’s get the year, month, hour,
minute, seconds, and timestamp
attributes from a datetime object:
 
[13]: 1 # get attributes
  2 dt = datetime(2019, 8, 17, 23, 38, 54)
  3 print("Year:", dt.year)
  4 print("Month:", dt.month)
  5 print("Day:", dt.day)
  6 print("Hour:", dt.hour)
  7 print("Minute:", dt.minute)
  8 print("Seconds:", dt.second)
  9 print("Timestamp:", dt.timestamp())
     
[13]:   Year: 2019
    Month: 8
    Day: 17
    Hour: 23
    Minute: 38
    Seconds: 54
    Timestamp: 1566074334.0
 
datetime.today() :
Return the current local datetime, with
tzinfo None .
 
datetime.now(tz=None) :
Return the current local date and time.
If optional argument tz is None or not
specified, this is like today() . If tz is not
None , it must be an instance of a tzinfo
subclass, and the current date and time
are converted to tz ’s time zone.
 
[14]: 1 # now()
  2 current_date_time = datetime.now()
print("Current date & time:",
 
3 current_date_time)
     
[14]:   Current date & time: 2022-03-26
12:51:39.045639
 
Class Attributes:
datetime.min : The earliest representable
datetime, datetime(MINYEAR, 1, 1,
tzinfo=None).
datetime.max : The latest representable
datetime, datetime(MAXYEAR, 12, 31, 23,
59, 59, 999999, tzinfo=None).
datetime.resolution : The smallest
possible difference between non-equal
datetime objects,
timedelta(microseconds=1).
 
Instance Attributes (read-only):
datetime.year : Between MINYEAR and
MAXYEAR inclusive.
datetime.month : Between 1 and 12
inclusive.
datetime.day : Between 1 and the number
of days in the given month of the given
year.
datetime.hour : In range(24).
datetime.minute : In range(60).
datetime.second : In range(60).
 datetime.microsecond : In
range(1000000).
datetime.tzinfo : The object passed as the
tzinfo argument to the datetime
constructor, or None if none was passed.
datetime.fold : In [0, 1]. Used to
disambiguate wall times during a repeated
interval. (A repeated interval occurs when
clocks are rolled back at the end of
daylight-saving time or when the UTC
offset for the current zone is decreased for
political reasons.) The value 0 (1)
represents the earlier (later) of the two
moments with the same wall time
representation.
 
Datetime class has the same methods
with date class. Here is the list of datetime
class methods:
astimezone() :
Returns the datetime object with
timezone (tz) information.
 
combine() :
Combines the date and time objects and
return a single datetime object.
  
ctime() :
Returns a string representation of date
and time.
 
date() :
Return date object with same year,
month and day.
 
fromisoformat() :
Returns a datetime object from the
string representation of the date and time.
 
fromordinal() :
Returns a date object from the proleptic
Gregorian ordinal, where January 1 of year
1 has ordinal 1. The hour, minute, second,
and microsecond are 0.
 
fromtimestamp() :
Returns the local date and time
corresponding to the POSIX timestamp.
 
isocalendar() :
Returns a named tuple with three
components: year, week and weekday.
 
 isoformat() :
Returns a string representing the date
and time in ISO 8601 format:
YYYY-MM-DDTHH:MM:SS.ffffff, if
microsecond is not 0
YYYY-MM-DDTHH:MM:SS, if microsecond is
0
 
isoweekday() :
Returns the day of the week as an
integer, where Monday is 1 and Sunday is
7.
 
replace() :
Returns a new datetime with the same
attributes, except for those attributes
given new values by whichever keyword
arguments are specified.
 
strftime() :
Returns a string representing the date
and time, controlled by an explicit format
string.
 
strptime() :
Returns a datetime object
corresponding to date string provided as
parameter.
 
time() :
Returns time object with same hour,
minute, second, microsecond and fold.
tzinfo is None.
 
timetuple() :
Returns an object of type
time.struct_time .
 
timetz() :
Returns time object with same hour,
minute, second, microsecond, fold, and
tzinfo attributes.
 
toordinal() :
Returns the proleptic Gregorian ordinal
of the date. The same as
self.date().toordinal() .
 
tzname() :
Returns the name of the timezone if
tzinfo is not None. If tzinfo is None, returns
None.
 
 utcfromtimestamp() :
Returns the UTC datetime corresponding
to the POSIX timestamp, with tzinfo None.
(The resulting object is naive.)
 
utcoffset() :
Returns the UTC offset if tzinfo is not
None. If tzinfo is None, returns None.
 
utcnow() :
Returns the current UTC date and time,
with tzinfo None. This is like now() , but
returns the current UTC date and time, as
a naive datetime object. An aware current
UTC datetime can be obtained by calling
datetime.now(timezone.utc) .
 
weekday() :
Returns the day of the week as an
integer, where Monday is 0 and Sunday is
6.
 
Let’s see some examples of working
with datetime objects:
 
[15]: 1 # Using datetime.combine()
  2 d = date(2005, 7, 14)
  3 t = time(12, 30)
  4 combined_dt = datetime.combine(d, t)
  5 print(combined_dt)
     
[15]:   2005-07-14 12:30:00
 
In cell 15, we instantiate two objects
with date() and time() constructors. Then
we combine them by using the
datetime.combine() method.
 
[16]: 1 # Using datetime.now()
  2 # GMT +1
  3 now = datetime.now()
  4 print(now)
  5  
  6 # with timezone info
  7 now_tz = datetime.now(timezone.utc)
  8 print(now_tz)
     
[16]:   2022-03-26 14:42:10.279281
    2022-03-26 11:42:10.279293+00:00
 
In cell 16, we call the datetime.now()
method with and without timezone
information.
 
[17]: 1 # Using datetime.strptime()
  2 dt = datetime.strptime("21/11/06 16:30",
 "%d/%m/%y %H:%M")
  3 print(dt)
     
[17]:   2006-11-21 16:30:00
 
In cell 17, we create a datetime object
by calling the datetime.strptime(date_string,
format) method. We pass the date_string
and the format parameters.
 
# Using datetime.timetuple() to get tuple
[18]:
1 of all attributes
  2 tt = dt.timetuple()
  3 for it in tt:
  4      print(it)
     
[18]:   2006    # year
    11      # month
    21      # day
    16      # hour
    30      # minute
    0       # second
    1       # weekday (0 = Monday)
325     # number of days since 1st
   
January
-1      # dst - method tzinfo.dst()
   
returned None
 
In cell 18, we print all of the attributes
of the dt object which we defined in cell
17.
 
[19]: 1 # Date in ISO format
  2 ic = dt.isocalendar()
  3 for it in ic:
  4      print(it)
     
[19]:   2006    # ISO year
    47      # ISO week
    2       # ISO weekday
 
In cell 19, we print the ISO calendar
information of our dt object. We get this
data by calling the isocalendar() method
on the datetime object.
 
[20]: 1 # Formatting a datetime
formatted_dt = dt.strftime("%A, %d. %B
 
2 %Y %I:%M%p")
  3 print(formatted_dt)
  4  
dt_str = 'The {1} is {0:%d}, the {2} is
{0:%B}, the {3} is
 
{0:%I:%M%p}.'.format(dt, "day", "month",
5 "time")
  6 print(dt_str)
     
[20]:   Tuesday, 21. November 2006 04:30PM
The day is 21, the month is November,
   
the time is 04:30PM.
 
 In cell 20, we print the formatted date in
two different ways. The first one uses the
datetime.strftime() method and the second
one is the str.format() method.
 
 
OceanofPDF.com
 Time Class
 
A time object represents a (local) time
of day, independent of any particular day,
and subject to adjustment via a tzinfo
object.
 
datetime.time(hour=0, minute=0, second=0,
microsecond=0, tzinfo=None, *, fold=0) :
All arguments are optional. tzinfo may
be None, or an instance of a tzinfo
subclass. The remaining arguments must
be integers in the following ranges:
0 <= hour < 24,
0 <= minute < 60,
0 <= second < 60,
0 <= microsecond < 1000000,
fold in [0, 1].
If an argument outside those ranges is
given, ValueError is raised. All default to 0
except tzinfo , which defaults to None.
 
Class Attributes:
time.min : The earliest representable
time, time(0, 0, 0, 0).
time.max :  The latest representable time,
time(23, 59, 59, 999999).
 time.resolution : The smallest possible
difference between non-equal time
objects, timedelta(microseconds=1),
although note that arithmetic on time
objects is not supported.
 
Instance Attributes (read-only):
time.hour : In range(24).
time.minute : In range(60).
time.second : In range(60).
time.microsecond : In range(1000000).
time.tzinfo : The object passed as the
tzinfo argument to the time constructor, or
None if none was passed.
time.fold : In [0, 1]. Used to disambiguate
wall times during a repeated interval. (A
repeated interval occurs when clocks are
rolled back at the end of daylight-saving
time or when the UTC offset for the
current zone is decreased for political
reasons.) The value 0 (1) represents the
earlier (later) of the two moments with the
same wall time representation.
 
time.fromisoformat(time_string) :
 Returns a time corresponding to a
time_string in one of the formats emitted
by time.isoformat() .
 
time.replace() :
Returns a time with the same value,
except for those attributes given new
values by whichever keyword arguments
are specified.
 
time.isoformat() :
Returns a string representing the time in
ISO 8601 format.
 
time.__str__() :
For a time t , str(t) is equivalent to
t.isoformat() .
 
time.strftime(format) :
Returns a string representing the time,
controlled by an explicit format string.
 
time.__format__(format) :
Same as time.strftime() . This makes it
possible to specify a format string for a
time object in formatted string literals and
when using str.format() .
 
time.utcoffset() :
If tzinfo is None, returns None, else
returns self.tzinfo.utcoffset(None) , and
raises an exception if the latter doesn’t
return None or a timedelta object with
magnitude less than one day.
 
time.dst() :
If tzinfo is None, returns None, else
returns self.tzinfo.dst(None) , and raises an
exception if the latter doesn’t return None,
or a timedelta object with magnitude less
than one day.
 
time.tzname() :
If tzinfo is None, returns None, else
returns self.tzinfo.tzname(None) , or raises
an exception if the latter doesn’t return
None or a string object.
 
Here are some examples of working
with a time object:
 
[21]: 1 from datetime import time, tzinfo,
timedelta
  2  
  3 # define a custom class
  4 class TZ1(tzinfo):
  5      def utcoffset(self, dt):
  6          return timedelta(hours=1)
  7      def dst(self, dt):
  8          return timedelta(0)
  9      def tzname(self,dt):
  10          return "+01:00"
  11      def  __repr__(self):
         return f"
 
12 {self.__class__.__name__}()"
 
In cell 21, we define a custom class,
TZ1 , which inherits from tzinfo class. And
we override some methods in its parent.
Let’s use this class for creating a time
object and print some attributes:
 
[22]: 1 # create a time object
  2 t = time(12, 10, 30, tzinfo=TZ1())
  3 print("t:",t)
  4 print("isoformat:", t.isoformat())
  5 print("dst:", t.dst())
  6 print("tzname:", t.tzname())
print("strftime:", t.strftime("%H:%M:%S
 
7 %Z"))
print("format:", 'The {} is
 
8 {:%H:%M}.'.format("time", t))
     
[22]:   t: 12:10:30+01:00
    isoformat: 12:10:30+01:00
    dst: 0:00:00
    tzname: +01:00
    strftime: 12:10:30 +01:00
    format: The time is 12:10.
 
 
OceanofPDF.com
 Formatting Date and Time
 
We already covered the strftime()
method that exist in the datetime class.
This is the main method which is used for
date and time formatting.
Here is the reference of all legal format
codes that you can use for date and time
formatting:
 
Directive Description Example
%a Weekday, short version Wed
%A Weekday, full version Wednesday
Weekday as a number 0-6,
%w 2
0 is Sunday
%d Day of month 01-31 28
%b Month name, short version Dec
%B Month name, full version December
%m Month as a number 01-12 12
Year, short version,
%y 18
without century
%Y Year, full version 2021
%H Hour 00-23 17
%I Hour 00-12 5
%p AM/PM PM
%M Minute 00-59 41
%S Second 00-59 8
Microsecond 000000-
%f 548513
999999
%z UTC offset 100
%Z Timezone CST
Day number of year 001-
%j 365
366
Week number of year,
%U Sunday as the first day of 52
week, 00-53
Week number of year,
%W Monday as the first day of 52
week, 00-53
Mon Dec
Local version of date and 31
%c
time 17:41:00
2018
%C Century 20
%x Local version of date 12/31/18
%X Local version of time 17:41:00
%% A % character %
%G ISO 8601 year 2018
%u ISO 8601 weekday (1-7) 1
ISO 8601 weeknumber (01-
%V 1
53)
 
Let’s see some examples for these
formats:
 
[23]: 1 # Formatting Examples
  2 from datetime import datetime
  3  
  4 # current date & time
  5 current = datetime.now()
  6 print("No formatting", current)
  7  
  8 # Weekday short version
weekday_short = current.strftime("%a
 
9 %-m %y")
print('Weekday Short Version:',
 
10 weekday_short)
  11  
  12 # Weekday
weekday = current.strftime("%A %m %-
 
13 Y")
  14 print('Weekday:', weekday)
  15  
  16 # Hour with PM
  17 pm = current.strftime("%-I %p %S")
  18 print('Hour with PM:', pm)
  19  
  20 # Time
common_time =
 
21 current.strftime("%H:%M:%S")
print('Common Time Representation:',
 
22 common_time)
     
No formatting 2022-03-26
[23]:  
18:17:54.351495
    Weekday Short Version: Sat 3 22
    Weekday: Saturday 03 2022
    Hour with PM: 6 PM 54
    Common Time Representation: 18:17:54
 
OceanofPDF.com
 
QUIZ – Date and Time
 
Now it’s time to solve the quiz for this
chapter. You can download the quiz file,
QUIZ_DateAndTime.zip, from the
GitHub Repository of this book. You should
put the quiz file in the DateAndTime
project we build in this chapter.
 
Here are the questions for this chapter:
 
Q1:
Define two timedelta objects with following
attributes:
* object 1: 23 days, 8 hours, 35 minutes
* object 2: 15 days, 6 hours, 50 minutes
 
Find the difference between these two objects
(object 1 - object 2).
Print the result.
 
[1]: 1 # Q 1:
  2  
  3 # import timedelta class
  4 # ---- your solution here ---
  5  
  6 # define objects
  7 # ---- your solution here ---
  8  
  9 # find the difference
  10 # ---- your solution here ---
  11  
  12 # print the result
  13 # ---- your solution here ---
     
[1]:   8 days, 1:45:00
 
Q2:
Instantiate a date object using the date()
constructor.
The date will be:
* year: 2017
* month: 6
* day: 5
 
Replace the day in this date with 24.
Print both dates (the old one and the new one).
 
Hints:
* date()
* replace()
* replace() will return a new date
 
[2]: 1 # Q 2:
  2  
  3 # import date class
  4 # ---- your solution here ---
  5  
  6 # create a date object
  7 # ---- your solution here ---
  8  
  9 # replace the day
  10 # ---- your solution here ---
  11  
  12 # print both dates
  13 # ---- your solution here ---
     
[2]:   2017-06-05
    2017-06-24
 
Q3:
Get the following two information from datetime
module:
* today's date (today)
* current date and time (now)
Assign them to two variables.
 
Print the today variable in formats below:
* dd/mm/YY
* mm/dd/y
* month (in text), day and year
* month (abbreviated), day and year
 
Hints:
* datetime (now)
* date (today)
* strftime()
 
[3]: 1 # Q 3:
  2  
  3 # import datetime and date classes
  4 # ---- your solution here ---
  5  
  6 # get today
  7 # ---- your solution here ---
  8  
  9 # get current date and time
  10 # ---- your solution here ---
  11  
  12 # formatting
  13 # dd/mm/YY
  14 # ---- your solution here ---
  15  
  16 # mm/dd/y
  17 # ---- your solution here ---
  18  
  19 # month (in text), day and year
  20 # ---- your solution here ---
  21  
  22 # month (abbreviated), day and year
  23 # ---- your solution here ---
     
[3]:   dd/mm/YY: 06/04/2022
    mm/dd/y: 04/06/22
month (in text), day and year: April 06,
   
2022
month (abbreviated), day and year: Apr
   
06, 2022
 
Q4:
Create a datetime object corresponding to date
string of "%d/%m/%y %H:%M".
Here are the date values:
* year: 2020
* month: 10
* day: 26
* hour: 18
* minutes: 47
 
Then print the following text by formatting this
datetime object:
"Monday, 26. October 2020 06:47PM"
 
Hints:
* strptime()
* strftime()
 
[4]: 1 # Q 4:
  2  
  3 # import datetime class
  4 # ---- your solution here ---
  5  
# create a datetime object with
 
6 strptime()
  7 # ---- your solution here ---
  8  
  9 # Format the datetime with strftime()
  10 # ---- your solution here ---
  11  
  12 # print the formatted date
  13 # ---- your solution here ---
     
[4]:   Monday, 26. October 2020 06:47PM
 
Q5:
Create a date object from ISO format.
The date will be: 2018-11-23
 
Then print the date in following formats:
* 2018-11-23
* 23/11/18
* Friday 23. November 201
 
Hints:
* fromisoformat()
* isoformat()
* strftime()
 
[5]: 1 # Q 5:
  2  
  3 # import date class
  4 # ---- your solution here ---
  5  
  6 # create a date from ISO format
  7 # ---- your solution here ---
  8  
  9 # formatted print
  10 # ---- your solution here ---
     
[5]:   2018-11-23
    23/11/18
    Friday 23. November 2018

OceanofPDF.com
 
SOLUTIONS – Date and Time
 
Here are the solutions for the quiz for
this chapter.
 
S1:
 
[1]: 1 # S 1:
  2  
  3 # import timedelta class
  4 from datetime import timedelta
  5  
  6 # define objects
object_1 = timedelta(days=23, hours=8,
 
7 minutes=35)
object_2 = timedelta(days=15, hours=6,
 
8 minutes=50)
  9  
  10 # find the difference
  11 difference = object_1 - object_2
  12  
  13 # print the result
  14 print(difference)
     
[1]:   8 days, 1:45:00
 
S2:
 
[2]: 1 # S 2:
  2  
  3 # import date class
  4 from datetime import date
  5  
  6 # create a date object
  7 date_1 = date(2017, 6, 5)
  8  
  9 # replace the day
  10 date_2 = date_1.replace(day=24)
  11  
  12 # print both dates
  13 print(date_1)
  14 print(date_2)
     
[2]:   2017-06-05
    2017-06-24
 
S3:
 
[3]: 1 # S 3:
  2  
  3 # import datetime and date classes
  4 from datetime import datetime, date
  5  
  6 # get today
  7 today = date.today()
  8  
  9 # get current date and time
  10 now = datetime.now()
  11  
  12 # formatting
  13 # dd/mm/YY
  14 format_1 = today.strftime("%d/%m/%Y")
  15 print("dd/mm/YY:", format_1)
  16  
  17 # mm/dd/y
  18 format_2 = today.strftime("%m/%d/%y")
  19 print("mm/dd/y:", format_2)
  20  
  21 # month (in text), day and year
  22 format_3 = today.strftime("%B %d, %Y")
print("month (in text), day and year:",
 
23 format_3)
  24  
  25 # month (abbreviated), day and year
  26 format_4 = today.strftime("%b %d, %Y")
print("month (abbreviated), day and
 
27 year:", format_4)
     
[3]:   dd/mm/YY: 06/04/2022
    mm/dd/y: 04/06/22
month (in text), day and year: April 06,
   
2022
month (abbreviated), day and year: Apr
   
06, 2022
 
S4:
 
[4]: 1 # S 4:
  2  
  3 # import datetime class
  4 from datetime import datetime
  5  
# create a datetime object with
 
6 strptime()
dt = datetime.strptime("26/10/20 18:47",
 
7 "%d/%m/%y %H:%M")
  8  
  9 # Format the datetime with strftime()
 formatted_dt = dt.strftime("%A, %d. %B
 
10 %Y %I:%M%p")
  11  
  12 # print the formatted date
  13 print(formatted_dt)
     
[4]:   Monday, 26. October 2020 06:47PM
 
S5:
 
[5]: 1 # S 5:
  2  
  3 # import date class
  4 from datetime import date
  5  
  6 # create a date from ISO format
  7 d = date.fromisoformat('2018-11-23')
  8  
  9 # formatted print
  10 print(d.isoformat())
  11 print(d.strftime("%d/%m/%y"))
  12 print(d.strftime("%A %d. %B %Y"))
     
[5]:   2018-11-23
    23/11/18
    Friday 23. November 2018
 
 
OceanofPDF.com
 7. Decorators
     

 
OceanofPDF.com
 Decorators in Python
 
Decorators are extremely useful tools in
Python. A decorator is a function that takes
another function as the parameter and
extends its functionality without explicitly
modifying it. It allows us to modify the
behavior of a function or a class without
touching its source code.
In other words, a decorator wraps a
function in order to extend its behaviors,
without permanently modifying it.
In this chapter, we will learn how
decorators work in Python. You can find
the PyCharm project for this chapter in the
GitHub Repository of this book.
 
Chapter Outline:
Functions are First-Class Citizens
Defining a Decorator
Decorators with Parameters
General Decorators
Decorator Changes the Function
Name
Chaining Decorators
 Class Decorators
QUIZ – Decorators
SOLUTIONS – Decorators
 
 
OceanofPDF.com
 Functions are First-Class Citizens
 
In order to understand how decorators
work, we need to revisit some important
concepts about functions in Python.
In Python, functions are first-class
citizens. By this, we mean:
Functions can be assigned as regular
variables
Functions can be passed as
arguments to other functions
Functions can return functions
Functions can have other functions
(inner functions or nested functions)
in their function body
 
Now let’s see some examples of these
points on functions:
 
Example 1: Functions can be assigned
as regular variables.
 
[1]: 1 # Example 1:
  2 # assign function to a variable
  3 def say_hi(user_name):
  4     return 'Hi ' + user_name
  5  
  6 # assign the function
  7 hi_name = say_hi
  8 print(hi_name('Bruce Wayne'))
     
[1]:   Hi Bruce Wayne
 
In cell 1, we define a function as say_hi .
Then we assign this function to a local
variable named hi_name . Now this hi_name
variable is a function too. And in line 8, we
call the hi_name as: hi_name('Bruce Wayne') .
 
Example 2: Functions can be passed as
arguments to other functions.
 
[2]: 1 # Example 2:
# Functions can be passed as arguments
 
2 to other functions.
  3 def print_hello(user):
  4     print('Hello', user)
  5  
  6 def hi_with_function(func, user_name):
  7     func(user_name)
  8  
  9 # call the function
  10 hi_with_function(print_hello, 'Clark Kent')
     
[2]:   Hello Clark Kent
 
 In cell 2, we define two functions;
print_hello and hi_with_function . The second
one takes a function as an argument:
hi_with_function(func, user_name) . And it
calls this function in its function body in
line 7 as: func(user_name) .
 
Example 3: Functions can return
functions.
 
[3]: 1 # Example 3:
  2 # Functions can return functions
  3 def return_hi_function():
  4     return say_hi
  5  
  6 # call the function
  7 hi = return_hi_function()
  8 print(hi('Spiderman'))
     
[3]:   Hi Spiderman
 
In cell 3, we define a function named
return_hi_function . This function simply
returns another function, which is the
say_hi function that we defined in cell 1. In
line 7, we assign the returning function to
a variable called hi . Now this hi variable is
also a function. Then we call it in line 8.
  
Example 4: Functions can have other
functions (inner functions) in their function
body.
 
[4]: 1 # Example 4:
# Functions can have other functions in
 
2 their body
  3 def outer_func(msg):
  4     """Outer function"""
  5  
  6     # define a nested function
  7     def inner_func():
  8         """Inner function"""
  9         print(msg, 'from nested function.')
  10  
  11     # call the nested function
  12     inner_func()
  13  
  14 # call the outer function
  15 outer_func('The Batman')
     
[4]:   The Batman from nested function.
 
In cell 4 we define an outer function as
outer_func . Inside this function we define a
nested function named inner_func . And in
line 12, we call the inner function.
When we call the outer the function in
line 15, we pass the text of ‘ The Batman’
for the value of the msg parameter. And
the output is ‘The Batman from nested
function.’ . This text is printed by the
inner_func . Be careful here, the inner_func
uses the msg variable which is not defined
in its own body. In other words, it uses a
variable which belongs to its parent’s
scope. This is the idea behind the Closures
in Python.
 
P ython Closures : A Closure is a function
object that remembers the values in the
parent’s scope even if they are not
present in memory.
 
 
OceanofPDF.com
 Defining a Decorator
 
A decorator takes in a function as an
argument, adds some functionality to it
and returns it back to the caller.
Sometimes, people call this action as
metaprogramming because a part of the
program tries to modify another part of
the program at compile time.
 
[5]: 1 # define a simple decorator
  2 def first_decorator(func):
  3     def wrapper():
        print("Before running {0}
 
4 function".format(func.__name__))
  5         func()
        print("After running {0}
 
6 function".format(func.__name__))
  7     return wrapper
  8  
  9 def greet():
  10     print("Hi there")
  11  
  12 # call the first_decorator function
  13 greet = first_decorator(greet)
  14  
  15 # call the greet function now
  16 greet()
     
[5]:   Before running greet function
    Hi there
    After running greet function
 
In cell 5, we define a simple decorator
function. The decorator’s name is
first_decorator and it has a nested function
in it. The nested function is called wrapper .
The first_decorator function simply returns
this wrapper function.
The wrapper function prints some text
then calls the function which is the
parameter value (func) as: func() . And
finally, it prints another text.
In line 13, we call the first_decorator
function by passing the greet() function as
the argument. And we re-assign the
returning value to the greet variable. Here
is the code: greet = first_decorator(greet) .
Remember that first_decorator returns a
function ( wrapper ).
In line 16, we call the greet() function.
And in the output you see that the
behavior of greet() function is modified. In
its original form, it was printing just one
line. But now, it prints three lines.
 To be sure about the final form of the
greet() function, let’s print its object data:
 
[6]: 1 # greet function object data
  2 print(greet)
     
<function first_decorator.
[6]:  
<locals>.wrapper at 0x100f569e0>
 
In cell 6, we print the greet() function
object data. And as you see in the output,
the function name is now wrapper in the
memory. Why? Because we re-assign it
from the returning value from the
first_decorator function, which is the
wrapper function.
Now we are sure that, we have
decorated our greet() function. In other
words, we didn’t modify its source code,
but we added some new functionality.
 
Decorator Syntax:
Line 13 in cell 5 is the way we decorate
the greet() function. It is: greet =
first_decorator(greet) . What we do is, we
pass the function to the decorator and re-
assign the resulting value to the same
function (actually a variable with the same
name).
In Python, we have a better and more
readable syntax for decorating functions.
Here it is:
 
[7]: 1 # Decorator Syntax:
  2  
  3 # Long Way
  4 # def greet():
  5 #     print("Hi there")
  6  
  7 # greet = first_decorator(greet)
  8 # greet()
  9  
  10 # Pythonic Way
  11 @first_decorator
  12 def greet():
  13     print("Hi there")
  14  
  15 greet()
     
[7]:   Before running greet function
    Hi there
    After running greet function
 
In cell 7 line 11, you see the syntax for
using a decorator. We simply put the
decorator’s name with an @ symbol on
the function definition line. Here it is:
  
[8]: 1 @first_decorator
  2 def greet():
  3      …
 
So, saying @first_decorator is the simple
way of saying greet = first_decorator(greet) .
This is how we apply a decorator to a
function.
 
 
OceanofPDF.com
 Decorators with Parameters
 
In the previous section, we learned how
we define and use a decorator function.
Now let’s see what happens if the function
which we want to decorate accepts some
parameters. How will we decorate such
functions?
Let’s assume we want to define a
function which takes two numbers,
performs the division operation and
returns the result. Here is the function:
 
[9]: 1 # division function
  2 def division(x, y):
  3     return x / y
  4  
  5 # call the function
  6 result_1 = division(20, 5)
  7 print(result_1)
  8  
  9 result_2 = division(8, 0)
  10 print(result_2)
     
[9]:   4.0
    ZeroDivisionError: division by zero
 
 In cell 9, you see the definition of the
division() function. It simply returns the
division result. And we call it with two
parameters (20, 5) and it returns 4.0 . But
there is a problem here. What if the
second number, the divisor, is 0? We will
get a ZeroDivisionError if the second
parameter is zero. We can implement a
try-except block inside our division function
to overcome this problem. However, we
don’t want to modify the code in the
function body. So, we need another way.
The solution is to decorate this function
with a decorator. The decorator will be
responsible of checking if the second
parameter is zero or not. Let’s define this
decorator function:
 
[10]: 1 # decorator
  2 def division_decorator(f):
  3     # define the wrapper function
  4     def wrapper(a, b):
  5         if b == 0:
            print("Division by zero is not
 
6 possible.")
  7             return
  8         else:
  9             return f(a, b)
  10  
  11     # return the wrapper function
  12     return wrapper
 
In cell 10, we define a decorator. The
name is division_decorator and it is
responsible for checking the
ZeroDivisionError . In the wrapper function,
it checks if b is equal to zero or not. If it is,
then it simply prints an error message and
returns. If b is not zero, then it calls the
function f and returns it: return f(a, b) .
Finally, the division_decorator function
returns the wrapper function. Remember
that this is the idea behind the decorators
in Python.
Now let’s decorate the division function
with the division_decorator :
 
[11]: 1 # decorate division function
  2 @division_decorator
  3 def division(x, y):
  4     return x / y
 
Now that we use the decorator for our
division() function let’s call it with zero for
the divisor value:
  
[12]: 1 # call the function now
  2 result_1 = division(20, 5)
  3 print(result_1)
  4  
  5 result_2 = division(8, 0)
  6 print(result_2)
     
[12]:   4.0
    Division by zero is not possible.
    None
 
As you see in the output of cell 12, we
handle the case where the caller may pass
a zero for the divisor. And we managed
this by the help of a decorator. The
division_decorator function implements all
the logic which is necessary for this case.
More importantly, we didn’t modify our
division() function body. It’s still the same
function, but decorated now.
 
 
OceanofPDF.com
 General Decorators
 
In the previous section we saw that the
parameters of the division() function and
the wrapper() function inside the decorator
must match. Why? Because we wrapper()
function will replace the division() function
after decoration. So, their parameters
should match. But this brings another
problem. What if we want to use the same
decorator with multiple functions? And
what if these functions have different
numbers of parameters? Let’s answer this
question now.
To start with, let’s assume we want to
print our users’ names in full capital
letters. Some users may have just their
first names, while the others may have
first names and last names. So, we will
define two separate functions as
first_name and full_name . And we will
define a decorator function which will
convert the names into upper case. Here
is the decorator:
 
[13]: 1 # define a general decorator
  2 def upper_decorator(func):
  3     # wrapper function
  4     def wrapper(*args):
  5         # modify the items in *args
  6         new_args = []
  7         for i, arg in enumerate(args):
  8             new_args.append(arg.upper())
  9         new_args = tuple(new_args)
  10  
  11         # return the call to the func
  12         return func(*new_args)
  13  
  14     # return wrapper function
  15     return wrapper
 
In cell 13, we define a general
decorator. Why is it general? Because, in
its wrapper function, the parameter is
*args . This enables the wrapper function to
take any number of parameters. In other
words, the wrapper function will be able to
present any function which is decorated
with this decorator.
In the wrapper function, it modifies the
items in the args tuple. It converts each
item to the upper case and appends it to a
list. In line 9, it converts this list to a tuple.
And finally, in line 12, it returns the call to
the func parameter by passing *new_args
tuple. Here is the line: return
func(*new_args) .
Now let’s define the first_name and
full_name functions. We want both of them
to be decorated with the upper_decorator .
Here they are:
 
[14]: 1 @upper_decorator
  2 def first_name(name):
  3     print(name)
  4  
  5 # call first_name function
  6 first_name('john')
  7  
  8 @upper_decorator
  9 def full_name(firt, last):
  10     print(firt, last)
  11  
  12 # call the full_name function
  13 full_name('john', 'doe')
     
[14]:   JOHN
    JOHN DOE
 
In cell 14, we define two functions,
first_name and full_name , and we decorate
both with the upper_decorator . Be careful
that, the functions have different number
of parameters. Then we call both functions
with different set of parameters. And in
the output, you see that the names have
been capitalized for both. That’s the way
we use the same generator for functions
with any number of parameters.
Here is a more general decorator syntax
with both *args and **kwargs :
 
[15]: 1 # generator with *args and **kwargs
  2 def most_general_decorator(func):
  3     def wrapper(*args, **kwargs):
  4         # implement some logic here
  5         return func(*args, **kwargs)
  6  
  7     return wrapper
 
 
OceanofPDF.com
 Decorator Changes the Function Name
 
There is an important point which you
should always keep in mind when you
work with decorators. The function name
will be changed after you decorate it. How
is that possible? Let’s see with an
example:
 
[16]: 1 # function without decorator
  2 def last_name(last):
  3     print(last)
  4  
  5 # print function name
print("Function name before
 
6 decorator:", last_name.__name__)
  7  
  8 # function with decoratoe
  9 @upper_decorator
  10 def last_name(last):
  11     print(last)
  12  
  13 # print function name
print("Function name after decorator:",
 
14 last_name.__name__)
     
Function name before decorator:
[16]:  
last_name
    Function name after decorator: wrapper
 
 In cell 16, we use the upper_decorator
which we defined in cell 13. We define a
new function as last_name .
In the first definition, we didn’t decorate
it. And we print its name in line 6. The
function name is ‘last_name’ as expected.
Then we redefine this function with a
decorator this time. And in line 14, we
print its name one more time. Surprisingly
this time its name is ‘wrapper ’ .
Why does the name changes from
‘last_name’ to ‘wrapper ’ ? Because when we
decorate it, the decorator returns the
wrapper function. Remember that
decorating is exactly the same as this line:
last_name = upper_decorator(last_name)
We reassign the returning value from
the decorator to our function. Since the
decorator simply returns the wrapper
function, the name of our original function
changes to ‘wrapper ’ now.
To fix this issue, Python provides a very
simple solution which is the functools
module. Let’s see how we can use this
module to keep the original function name
unchanged:
  
[17]: 1 # functools
  2 import functools
  3  
  4 # define a decorator with functools
  5 def upper_decorator(func):
  6     @functools.wraps(func)
  7     def wrapper(*args):
  8         # modify the items in *args
  9         new_args = []
  10         for i, arg in enumerate(args):
  11             new_args.append(arg.upper())
  12         new_args = tuple(new_args)
  13  
  14         # return the call to the func
  15         return func(*new_args)
  16  
  17     # return wrapper function
  18     return wrapper
 
In cell 17line 6, we use the
functools.wraps() method. This method
itself is a decorator. So, we use it on the
wrapper function with the @ symbol. Here
is the syntax: @functools.wraps(func) . The
original function ( func ) is the parameter to
this method. This method preserves
information about the original function.
Now let’s print the name of the
last_name function one more time:
  
[18]: 1 # function without decorator
  2 def last_name(last):
  3     print(last)
  4  
  5 # print function name
print("Function name before
 
6 decorator:", last_name.__name__)
  7  
  8 # function with decoratoe
  9 @upper_decorator
  10 def last_name(last):
  11     print(last)
  12  
  13 # print function name
print("Function name after decorator:",
 
14 last_name.__name__)
     
Function name before decorator:
[18]:  
last_name
Function name after decorator:
   
last_name
 
As you see in the output of cell 18, now
the name of our original function is the
same after we decorate it.
 
 
OceanofPDF.com
 Chaining Decorators
 
Most of the time you will need to use
more than one decorator for a function.
This is called chaining decorators on the
same function. Let’s see how we can use
multiple decorators (or the same
decorator multiple times).
 
[19]: 1 # function for printing symbols
  2 def print_symbol(symbol, times):
  3     print(symbol * times)
  4  
  5 # decorator 1
  6 def plus_sign(f):
  7     def wrapper(*args, **kwargs):
  8         print_symbol('+', 20)
  9         f(*args, **kwargs)
  10         print_symbol('+', 20)
  11     return wrapper
  12  
  13 # decorator 2
  14 def minus_sign(f):
  15     def wrapper(*args, **kwargs):
  16         print_symbol('-', 20)
  17         f(*args, **kwargs)
  18         print_symbol('-', 20)
  19     return wrapper
 
 In cell 19, we define two decorators.
Each one prints a different symbol before
and after calling the original function. Now
let’s use both of them on the same
function:
 
[20]: 1 # chain decorators
  2 @plus_sign
  3 @minus_sign
  4 def say_hi(msg):
  5     print(msg)
  6  
  7 # call the decorated function
  8 say_hi("Hi Python Developer")
     
[20]:   ++++++++++++++++++++
    --------------------
    Hi Python Developer
    --------------------
    ++++++++++++++++++++
 
In cell 20, we chain two decorators on
the say_hi() function. And in the output,
you see the order of execution of the
decorators. The one which is closer to the
function definition executes before the
others.
Now let’s reverse the order of
decorators and see the output one more
time:
 
[21]: 1 # change the order of chaining
  2 @minus_sign
  3 @plus_sign
  4 def say_hi(msg):
  5     print(msg)
     
[21]:   --------------------
    ++++++++++++++++++++
    Hi Python Developer
    ++++++++++++++++++++
    --------------------
 
As you see in the output of cell 21, the
order of execution changes when we
change the order of decorators.
 
 
OceanofPDF.com
 Class Decorators
 
Decorators can be either functions or
classes in Python. In the previous sections
we worked with function decorators. Now,
we will learn how to define class
decorators.
We will define custom classes that acts
as a decorator. When a function is
decorated with a class, that function
becomes an instance of the class. Let’s
see how:
 
[22]: 1 # define a class decorator
  2 class ClassDecorator:
  3     # init method takes the function
  4     def __init__(self, func):
  5         self.func = func
  6  
  7     # implement __call__ method
  8     def __call__(self):
  9         # some logic before func call
        print('__call__ method before
 
10 func')
  11         self.func()
  12         # Some logic after func call
  13         print('__call__ method after func')
 
 In cell 22, we have a simple class
decorator. For any class to be a decorator,
it needs to implement the __call__()
method. The __call__() method acts the
same way as the wrapper function in the
function decorators.
Now let’s use this class to decorate a
function:
 
[23]: 1 # add class decorator to func
  2 @ClassDecorator
  3 def say_hi():
  4     print("Hi Python")
  5  
  6 # call the decorated function
  7 say_hi()
     
[23]:   __call__ method before func
    Hi Python
    __call__ method after func
 
Class Decorator with *args and
**kwargs :
In order to use a class decorator with
*args and **kwargs arguments, we need to
implement the __call__() method with
these arguments and pass them to the
decorated function.
  
[24]: 1 # class decorator with *args & **kwargs
  2 class ClassDecorator:
  3     def __init__(self, func):
  4         self.func = func
  5  
  6     def __call__(self, *args, **kwargs):
  7         # some logic before func call
  8         self.func(*args, **kwargs)
  9         # Some logic after func call
 
In cell 24, the __call__() method of the
class decorator takes *args and **kwargs
arguments. And in line 8, it passes them
to the decorated function as:
self.func(*args, **kwargs) .
Let’s decorate a function with this class
decorator now:
 
[25]: 1 # add class decorator to func
  2 @ClassDecorator
  3 def say_hi(first, last, msg='Hi'):
    print("{0} {1} {2}".format(msg, first,
 
4 last))
  5  
  6 # call the decorated function
  7 say_hi("Bruce", "Wayne", "Hi")
     
[25]:   Hi Bruce Wayne
 
 Class Decorator with the return
statement:
Remember that, in the wrapper function
of a function decorator we use the return
keyword to return the decorated function.
We will do the same thing here, but inside
the __call__() method this time.
 
[26]: 1 # define a class decorator
  2 class UpperDecorator:
  3     def __init__(self, func):
  4         self.func = func
  5  
  6     def __call__(self, *args):
  7         # modify the items in *args
  8         new_args = []
  9         for i, arg in enumerate(args):
  10             new_args.append(arg.upper())
  11         new_args = tuple(new_args)
  12  
  13         # return the call to the func
  14         return self.func(*new_args)
  15  
  16 @UpperDecorator
  17 def full_name(first, last):
  18     print(first, last)
  19  
  20 # call decorated function
  21 full_name('jane', 'doe')
     
[26]:   JANE DOE
  
In cell 26 line 14, in the __call__()
method we return the decorated function
as: return self.func(*new_args) .
 
OceanofPDF.com
 
QUIZ – Decorators
 
Now it’s time to solve the quiz for this
chapter. You can download the quiz file,
QUIZ_Decorators.zip, from the GitHub
Repository of this book. You should put the
quiz file in the Decorators project we
build in this chapter.
 
Here are the questions for this chapter:
 
Q1:
We have a basic function which takes one argument
and simply returns it.
Here is this function:
def some_text(text):
    return text
 
We want this function to always return the text in
uppercase letters.
For example when we call it as:
some_text('this is a lowercase text...')
We want it to return 'THIS IS A LOWERCASE
TEXT...'.
But we do not want to modify the function body.
Function definition will not be changed.
What we need is a decorator to do the uppercase
conversion for our function.
 
You need to define this decorator.
Also make sure that, if the function parameter is
not a string (str) object, you shouldn't convert it to
upper case.
 
Hints:
* @decorator
* type()
 
[1]: 1 # Q 1:
  2  
  3 # define the decorator
  4 # ---- your solution here ---
  5  
  6 # decorate the function
  7 # ---- your solution here ---
  8  
  9 # call some_text function
  10 # ---- your solution here ---
  11  
  12 # print the final text
  13 # ---- your solution here ---
     
[1]:   THIS IS A LOWERCASE TEXT...
 
Q2:
We want to track the execution time for our
functions.
We will use the time module for time tracking.
Remember that time.time() returns the current time
in seconds.
So, if we get the start and end times of execution,
we can measure how long the function takes.
We want to define a decorator for this operation.
The decorator’s name will be time_tracker.
Since it is going to be a general-purpose decorator,
we don't know the number of parameters in
advance.
We want it to print something like this (after
executing the function):
"Function executed in (seconds):
2.0050909519195557"
 
To test your decorator, simply define a function
which sleeps for some seconds.
Here is the function:
def sleeper(seconds):
    time.sleep(seconds)
 
The sleeper() function just waits for the given
number of seconds.
Decorate this function with your decorator and print
its execution time.   
 
Hints:
* @decorator
* *args
* time.time()
 
[2]: 1 # Q 2:
  2  
  3 # import time module
  4 # ---- your solution here ---
  5  
  6 # define the decorator
  7 # ---- your solution here ---
  8  
  9 # decorate function
  10 # ---- your solution here ---
  11  
  12 # call the sleeper function to sleep 2
seconds
  13 # ---- your solution here ---
     
Function executed in (seconds):
[2]:  
2.005089044570923
 
Q3:
Define a class decorator named FunctionDetails.
This decorator will print the function details as
follows:
* Function Name
* Parameters
* Function Result
 
Here is the function which we want to decorate:
def full_name(first, last):
    print(first, last)
 
Call the function after you decorate it with
FunctionDetails.
 
Hints:
* __init__
* __call__
* *args
 
[3]: 1 # Q 3:
  2  
  3 # define a class decorator
  4 # ---- your solution here ---
  5  
  6 # decorate the function
  7 # ---- your solution here ---
  8  
  9 # call decorated function
  10 # ---- your solution here ---
     
[3]:   Function Name: full_name
    Parameters:
    (0, 'john')
    (1, 'doe')
    Function Result:
    john doe
 
Q4:
We have a function which simply adds two numbers:
def add_and_multiply(n_1, n_2):
    return n_1 + n_2
 
This function returns the summation of its
parameters.
We want to multiply the result of this function with
some numbers.
For this, we want to use a decorator.
 
Define a decorator which multiplies the function
result by the given factor.
Decorator name will be multiply_by and it will take
one parameter.
Here is the parameter:
* func: function to decorate
 
The wrapper (inner) function in the decorator will
take 3 arguments:
* num_1: first parameter from the function (n_1)
* num_2: second parameter from the function (n_2)
* factor: the number which we want to multiply with
 
[4]: 1 # Q 4:
  2  
  3 # define a decorator
  4 # ---- your solution here ---
  5  
  6 # decorate function
  7 # ---- your solution here ---
  8  
# call the function with n_1, n_2, and
 
9 factor
add_and_multiply_by_100 =
 
10 add_and_multiply(4, 7, 100)
  11  
  12 # print the result
  13 print(add_and_multiply_by_100)
     
[4]:   1100
 
Q5:
Here is a simple decorator:
 
1 # functools
2 import functools
3  
4 # define a decorator with functools
5 def upper_decorator(func):
6     @functools.wraps(func)
7     def wrapper(*args):
8         # modify the items in *args
9         new_args = []
10         for i, arg in enumerate(args):
11             new_args.append(arg.upper())
12         new_args = tuple(new_args)
13  
14         # return the call to the func
15         return func(*new_args)
16  
17     # return wrapper function
18     return wrapper
 
Why do we need to use @functools.wraps(func)
in line 6, to decorate the wrapper function?
What does it do?
 

OceanofPDF.com
 
SOLUTIONS – Decorators
 
Here are the solutions for the quiz for
this chapter.
 
S1:
 
[1]: 1 # S 1:
  2  
  3 # define the decorator
  4 def uppercase_convert(func):
  5     # define the wrapper function
  6     def wrapper(text):
  7         func_result = func(text)
  8         # check if func result is str type
  9         if type(func_result) == str:
  10             return func_result.upper()
  11         else:
  12             return func_result
  13  
  14     return wrapper
  15  
  16 # decorate the function
  17 @uppercase_convert
  18 def some_text(text):
  19     return text
  20  
  21 # call some_text function
upper_text = some_text("this is a
 
22 lowercase text...")
  23  
  24 # print the final text
  25 print(upper_text)
     
[1]:   THIS IS A LOWERCASE TEXT...
 
S2:
 
[2]: 1 # S 2:
  2  
  3 # import time module
  4 import time
  5  
  6 # define the decorator
  7 def time_tracker(func):
  8     # define the wrapper function
  9     def wrapper(*arg):
  10         # get start time
  11         t_start = time.time()
  12         # call the function
  13         func_result = func(*arg)
  14         # get end time
  15         t_end = time.time()
        print("Function executed in
 
16 (seconds):", str(t_end - t_start))
  17         return func_result
  18  
  19     return wrapper
  20  
  21 # decorate function
  22 @time_tracker
  23 def sleeper(seconds):
  24     time.sleep(seconds)
  25  
# call the sleeper function to sleep 2
 
26 seconds
  27 sleeper(2)
     
Function executed in (seconds):
[2]:  
2.005089044570923
 
S3:
 
[3]: 1 # S 3:
  2  
  3 # define a class decorator
  4 class FunctionDetails:
  5     def __init__(self, func):
  6         self.func = func
  7  
  8     def __call__(self, *args):
        print("Function Name:",
 
9 self.func.__name__)
  10         print("Parameters:")
  11         for arg in enumerate(args):
  12             print(arg)
  13  
  14         # execute the function
  15         print("Function Result:")
  16         return self.func(*args)
  17  
  18 # decorate the function
  19 @FunctionDetails
  20 def full_name(first, last):
  21     print(first, last)
  22  
  23 # call decorated function
  24 full_name('john', 'doe')
     
[3]:   Function Name: full_name
    Parameters:
    (0, 'john')
    (1, 'doe')
    Function Result:
    john doe
 
S4:
 
[4]: 1 # S 4:
  2  
  3 # define a decorator
  4 def multiply_by(func):
  5     def wrapper(num_1, num_2, factor):
  6         return func(num_1, num_2) * factor
  7     return wrapper
  8  
  9 # decorate function
  10 @multiply_by
  11 def add_and_multiply(n_1, n_2):
  12     return n_1 + n_2
  13  
# call the function with n_1, n_2, and
 
14 factor
add_and_multiply_by_100 =
 
15 add_and_multiply(4, 7, 100)
  16  
  17 # print the result
  18 print(add_and_multiply_by_100)
     
[4]:   1100
 
S5:
 
functools.wraps() function is a decorator which is
applied to the wrapper function of a decorator.
It updates the wrapper function to look like
decorated function.
So, it preserves information about the original
function, such as __name__, __doc__ etc.
 
 

OceanofPDF.com
 8. Context Managers
     

 
OceanofPDF.com
 What is a Context Manager?
 
A Context Manager is an object that
defines the runtime context to be
established when executing a with
statement. The context manager handles
the entry into, and the exit from, the
desired runtime context for the execution
of the block of code. Context managers
are normally invoked using the with
statement, but can also be used by
directly invoking their methods.
Typical uses of context managers
include saving and restoring various kinds
of global state, locking and unlocking
resources, closing opened files, etc...
In this chapter, we will learn how to
work with context managers in Python and
how to define custom context managers.
You can find the PyCharm project for this
chapter in the GitHub Repository of this
book.
 
Chapter Outline:
The with Statement
 Context Manager Protocol
Creating a Context Manager in Class
Form
Creating a Context Manager in
Function Form
QUIZ – Context Managers
SOLUTIONS – Context Managers
 
 
OceanofPDF.com
 The width Statement
The with statement is used to wrap the
execution of a block with methods defined
by a context manager. This allows
common try…except…finally usage patterns
to be encapsulated for convenient reuse.
Compared to traditional try…except…finally
blocks, the with statement provide a
shorter and reusable code.
In Python Standard Library many
classes support with statement. A very
common example is the built-in open()
function which provides tools to work with
the file objects using the with statement.
Here is the general syntax of the with
statement:
 
[1]: 1 with expression as target:
  2     # do something
  3     # using the target
 
Let’s see an example with the open()
function. We have a text file in the files
folder in our current project. The file name
is color_names.txt and it includes some
color names. We want to open and print
the content in this file by using the open()
function and with statement. Here is the
code.
 
[2]: 1 # define the file path
  2 path = 'files/color_names.txt'
  3  
  4 # with statement
  5 with open(path, mode='r') as file:
  6     # read the file content
  7     print(file.read())
     
[2]:   red
    orange
    yellow
    green
    blue
    white
    black
 
In cell 2, you see a common use case of
the with statement. We use the open()
function to open the file at the given path .
And the open() function returns the file
object in read mode. We use this file
object in line 7 to read and print its
content as: print(file.read()) .
 
 
OceanofPDF.com
 Context Manager Protocol
 
Python’s with statement supports the
concept of a runtime context defined by a
context manager. This is implemented
using a pair of methods that allow user-
defined classes to define a runtime
context that is entered before the
statement body is executed and exited
when the statement ends.
 
These methods are called the Context
Manager Protocol . Here they are:
__enter__(self) :
This method is called by the with
statement to enter the runtime context
related to current object. The with
statement will bind this method’s return
value to the target specified in the as
clause of the statement, if any. See cell 1.
An example of a context manager that
returns itself is a file object. File objects
return themselves from __enter__() to allow
open() to be used as the context
expression in a with statement. See cell 2.
  
__exit__(self, exc_type, exc_value,
traceback) :
This method is called when the
execution leaves the with code block. It
exits the runtime context related to this
object. The parameters describe the
exception that caused the context to be
exited. If the context was exited without
an exception, all three arguments will be
None .
If an exception is supplied, and the
method wishes to suppress the exception
(i.e., prevent it from being propagated), it
should return a true value. Otherwise, the
exception will be processed normally upon
exit from this method.
The __exit__() method returns a Boolean
value, either True or False .
 
The execution of the with statement
with the methods in context manager
protocol proceeds as follows:
 
[3]: 1 with EXPRESSION as TARGET:
  2     SUITE
 
1. The context expression is evaluated
to obtain a context manager.
2. The context manager’s __enter__() is
loaded for later use.
3. The context manager’s __exit__() is
loaded for later use.
4. The context manager’s __enter__()
method is invoked.
5. If a target was included in the with
statement, the return value from
__enter__() is assigned to it.
6. The suite (code block in the with
statement scope) is executed.
7. The context manager’s __exit__()
method is invoked. If an exception
caused the suite to be exited, its
type, value, and traceback are
passed as arguments to __exit__() .
Otherwise, three None arguments
are supplied.
If the suite was exited for any reason
other than an exception, the return
value from __exit__() is ignored, and
execution proceeds at the normal
location for the kind of exit that was
taken.
 
 
OceanofPDF.com
 Creating a Context Manager in Class Form
 
Now that we know the basic idea behind
the context manager protocol let’s
implement it in a class. This class will be
our context manager and we will use it
later with the with statement.
 
[4]: 1 # custom context manager class
  2 class CustomContextManager:
  3     # init method -> define variables
  4     def __init__(self, path, mode):
  5         self.path = path
  6         self.mode = mode
  7         self.file = None
  8  
  9     # __enter__ method -> open the file
  10     def __enter__(self):
        self.file = open(self.path,
 
11 self.mode)
  12         return self.file
  13  
  14     # exit method to close the file
    def __exit__(self, exc_type, exc_value,
 
15 exc_traceback):
  16         self.file.close()
 
Our CustomContextManager class
implements the necessary methods to
become a context manager: __enter__ and
__exit__ .
In its __init__ method, it defines three
instance variables to store the path , mode
and file objects.
In the __enter__ method it uses the built-
in open() function to open the file in the
specified path. Since the open() function
returns a file object, we assign it to the
self.file attribute.
In the __exit__ method we close the file
as: self.file.close() . The __exit__ method
accepts three arguments, which are
required by context manager protocol.
We can use our custom context
manager in a with statement now. Let’s do
it:
 
# custom context manager in with
[5]:
1 statement
  2 file_path = 'files/color_names.txt'
  3  
with
  CustomContextManager(path=file_path,
4 mode='r') as file:
  5     # print the file content
  6     print(file.read())
     
[5]:   red
    orange
    yellow
    green
    blue
    white
    black
 
In cell 5, we use our
CustomContextManager class in the with
statement. We read file content and print
it.
Here is what happens behind the
scenes:
1. The line 4 calls the __enter__ method
of the CustomContextManager class.
2. The __enter__ method opens the file
and returns it.
3. We name the opened file simply as
file .
4. In the suite of with statement, we
read the file content and print it.
5. The with statement calls the __exit__
method.
6. The __exit__ method closes the file.
 
 Let’s define another context manager
class. This time we want to print the list of
files in the specified folder.
 
[6]: 1 # context manager for listing files
  2 import os
  3  
  4 class ContentList:
  5     '''Returns the content of a directory'''
  6  
  7     def __init__(self, directory):
  8         self.directory = directory
  9  
  10     def __enter__(self):
  11         return os.listdir(self.directory)
  12  
    def __exit__(self, exc_type, exc_val,
 
13 exc_tb):
  14         if exc_type is not None:
            print("Error getting directory
 
15 list.")
  16         return True
  17  
# print the contents of the project
 
18 directory
  19 project_directory = '.'
with ContentList(project_directory) as
 
20 directory_list:
  21     print(directory_list)
     
['ClassContextManager.py',
[6]:  
'TheWithStatement.py', 'files', '.idea']
  
In cell 6, we define a new context
manager. The name of the class is
ContentList . Why is it a context manager?
Because it implements context manager
protocol ( __enter__ and __exit__ methods).
We take the directory path as the
parameter in the class constructor, which
is the __init__ method.
In the __enter__ method we get the list of
the contents in this directory simply by
calling the listdir() method in the os
module as: os.listdir(self.directory) . And we
return this list. Please be careful that, our
__enter__ method returns a list in this
context manager.
In the __exit__ method, we check if there
exist any errors. The exc_type , exc_val ,
exc_tb parameter values will not be None if
there is an error in our context manager.
So, we check if exc_type is not None to
print an error text.
In line 20, we use our context manager
in the with statement. Since it returns a
list object, we simple assign the returning
value to the directory_list variable. And in
the body of the with statement, we print
this list. In the cell output, you see the list
of the contents in the project directory.
Remember that, ‘.’ means the current
directory, which is the project directory in
our case.
 
 
OceanofPDF.com
 Creating a Context Manager in Function Form
 
In the previous section we learned how to define context
managers by using the class syntax. However, it is a bit
cumbersome and lengthy. You need to implement __enter__
and __exit__ methods explicitly and you need to handle
possible exceptions. Hopefully there is a better way of
creating context managers in Python: function-based
context managers.
Function-Based Context Managers are special functions
which use generators and contextlib.contextmanager
decorator. The contextlib.contextmanager decorator is
responsible for implementing the context manager protocol.
Let’s define a context manager function:
 
[7]: 1 from contextlib import contextmanager
  2  
  3 # define the context manager function
  4 @contextmanager
  5 def function_based_context_manager():
  6     print("Entering the context: __enter__")
  7     yield "This is a function based context manager"
  8     print("Leaving the context: __exit__")
  9  
  10 # use context manager function in with statement
  11 with function_based_context_manager() as yield_text:
  12     print(yield_text)
     
[7]:   Entering the context: __enter__
    This is a function based context manager
    Leaving the context: __exit__
 
In cell 7, we define a custom function which acts as a
context manager. The contextmanager decorator is what
turns a regular function into a full-stack context manager.
You don’t need to worry about implementing the __enter__
and __exit__ functions if you implement @contextmanager
decorator.
The yield statement in line 7, acts as the return statement
in the __enter__ method in a class-based context manager.
Since we have a yield statement, function-based context
managers are also generator functions.
Let’s define a new context manager. This time it will open
a file in write mode and append some text to it. Here it is:
 
[8]: 1 # context manager for write operations
  2 from contextlib import contextmanager
  3  
  4 @contextmanager
  5 def writer_context_manager(path, mode='w'):
  6     file_object = None
  7     try:
  8         file_object = open(path, mode=mode)
  9         yield file_object
  10     finally:
  11         if file_object:
  12             file_object.close()
  13  
  14 # context manager in with statement
with
  writer_context_manager("function_based_context_managers.txt")
15 as file:
  16     file.write("The contextlib.contextmanager decorator \n"
  17                "is responsible for implementing the\n"
  18                 "context manager protocol.")
 
In cell 8, we define a function-based context manager. In
the try block it tries to open the file in the specified path. If
it opens the file successfully, then it yields (returns) the
file_object . In the finally block we check if we have a
file_object to close. And we close the file_object if it is not
None .
In the with statement in line 15, we call our context
manager with a file name of
function_based_context_managers.txt . The context manager
opens this file in write mode and returns the file object
which we simply name as file . And in line 16, we write some
text in this file. Remember that ‘w’ mode will create an
empty file, if such a file doesn’t exist.
 
OceanofPDF.com
 
QUIZ – Context Managers
 
Now it’s time to solve the quiz for this
chapter. You can download the quiz file,
QUIZ_ContextManagers.zip, from the
GitHub Repository of this book. You should
put the quiz file in the ContextManagers
project we build in this chapter.
 
Here are the questions for this chapter:
 
Q1:
Define a custom context manager class as
MyContextManager.
Define the __init__, __enter__ and __exit__ methods.
Your context manager should open and return the
file object at specified path.
Also it should close the file on exit.
The parameters to the __init__ method should be:
* path: file path for the file to read
* mode: operation mode (read, write, append etc.)
 
Read and print the file content at
"files/color_names.txt".
 
Hints:
* __init__
* __enter__
* __exit__
 
[1]: 1 # Q 1:
  2  
  3 # define custom context manager class
  4 # ---- your solution here ---
  5  
# define the file path as
 
6 "files/color_names.txt"
  7 file_path = 'files/color_names.txt'
  8  
# read and print the content of the file at
 
9 this path
  10 # ---- your solution here ---
     
[1]:   red
    orange
    yellow
    green
    blue
    white
    black
 
Q2:
Define a context manager class to get the list of
folders in a directory.
The name will be FoldersInDirectory and it will
return only the folder names in a specified
directory.
The class constructor will take the directory path as
the parameter.
Make sure, you implement __enter__ and __exit__
methods properly.
 
Print the folder list by using a with statement.
 
Hints:
* __init__
* __enter__
* __exit__
 
[2]: 1 # Q 2:
  2  
  3 # import os module
  4 # ---- your solution here ---
  5  
  6 # context manager for listing folders
  7 # ---- your solution here ---
  8  
# define the directory path as current
 
9 directory
  10 # ---- your solution here ---
  11  
# print the folders in the project
 
12 directory
  13 # using a with statement
  14 # ---- your solution here ---
     
[2]:   files
    .idea
 
Q3:
Define a function based context manager which
creates a file and writes some text in it.
The name will be file_writer and it will take two
parameters:
* path: file path for the file to read
* mode: operation mode (read, write, append etc.)
 
Use this context manager in a with statement to
create a file and add the text below in this file.
File name should be: "quiz_text_file_for_writing.txt".
And the text is:
"The file is created by quiz question number 3.
It is a function-based context manager,
Decorated with contextlib.contextmanager."
 
You should see "quiz_text_file_for_writing.txt" file
after you run the code in this question.
 
Hints:
* contextlib.contextmanager
* try-finally
 
[3]: 1 # Q 3:
  2  
  3 # import contextmanager class
  4 # ---- your solution here ---
  5  
  6 # context manager for write operations
  7 # define and decorate
  8 # ---- your solution here ---
  9  
  10  
  11 # context manager in with statement
  12 # ---- your solution here ---
 
Q4:
Define a function-based context manager named
sleeper_context_manager.
It will print the following data:
* Start time: Time at start of its execution
* End time: Time at end of its execution
* Execution time in seconds: Difference between
end time and start time
 
Use this context manager in a with statement.
You should wait for 2 seconds in the with statement
body.
And you should print something similar to this:
"Start time: 1649345704.337794
End time: 1649345706.342916
Execution time in seconds: 2.005121946334839"
 
The start time and end time will change when you
run the code.
But the execution time should be around 2.00
seconds.
 
Hints:
* contextlib.contextmanager
* yield nothing
* time.sleep()
 
[4]: 1 # Q 4:
  2  
# import contextmanager class and time
 
3 module
  4 # ---- your solution here ---
  5  
# define and decorate the context
 
6 manager
  7 # ---- your solution here ---
  8  
# use the context manager in a with
 
9 statement
  10 # ---- your solution here ---
     
[4]:   Start time: 1649346091.427541
    End time: 1649346093.432682
Execution time in seconds:
   
2.005141019821167
 
Q5:
What are the methods in the Context Manager
Protocol?
And what are their functionalities?
 

OceanofPDF.com
 
SOLUTIONS – Context Managers
 
Here are the solutions for the quiz for
this chapter.
 
S1:
 
[1]: 1 # S 1:
  2  
  3 # define custom context manager class
  4 class MyContexManager:
  5     # init method -> define variables
  6     def __init__(self, path, mode):
  7         self.path = path
  8         self.mode = mode
  9         self.file = None
  10  
  11     # __enter__ method -> open the file
  12     def __enter__(self):
        self.file = open(self.path,
 
13 self.mode)
  14         return self.file
  15  
  16     # exit method to close the file
    def __exit__(self, exc_type, exc_value,
 
17 exc_traceback):
  18         self.file.close()
  19  
# define the file path as
 
20 "files/color_names.txt"
  21 file_path = 'files/color_names.txt'
  22  
# read and print the content of the file at
 
23 this path
with MyContexManager(path=file_path,
 
24 mode='r') as file:
  25     # print the file content
  26     print(file.read())
     
[1]:   red
    orange
    yellow
    green
    blue
    white
    black
 
S2:
 
[2]: 1 # S 2:
  2  
  3 # import os module
  4 import os
  5  
  6 # context manager for listing folders
  7 class FoldersInDirectory:
    '''Returns the folder list in the given
 
8 directory'''
  9  
  10     def __init__(self, directory):
  11         self.directory = directory
  12  
  13     def __enter__(self):
  14         # return folder names only (no file
 name)
        return [name for name in
  os.listdir(self.directory) if
15 os.path.isdir(name)]
  16  
    def __exit__(self, exc_type, exc_val,
 
17 exc_tb):
  18         if exc_type is not None:
            print("Error getting directory
 
19 list.")
  20         return True
  21  
# define the directory path as current
 
22 directory
  23 current_directory = '.'
  24  
# print the folders in the project
 
25 directory
  26 # using a with statement
with
  FoldersInDirectory(current_directory) as
27 folder_list:
  28     for folder in folder_list:
  29         print(folder)
     
[2]:   files
    .idea
 
S3:
 
[3]: 1 # S 3:
  2  
  3 # import contextmanager class
  4 from contextlib import contextmanager
  5  
  6 # context manager for write operations
  7 @contextmanager
  8 def file_writer(path, mode='w'):
  9     file_object = None
  10     try:
        file_object = open(path,
 
11 mode=mode)
  12         yield file_object
  13     finally:
  14         if file_object:
  15             file_object.close()
  16  
  17  
  18 # context manager in with statement
with
  file_writer("quiz_text_file_for_writing.txt")
19 as file:
    file.write("The file is created by quiz
 
20 question number 3.\n"
               "It is a function based context
 
21 manager,\n"
               "Decorated with
 
22 contextlib.contextmanager.")
 
S4:
 
[4]: 1 # S 4:
  2  
  3 # import contextmanager class
  4 from contextlib import contextmanager
  5 import time
  6  
  7 # define and decorate the context
 manager
  8 @contextmanager
  9 def sleeper_context_manager():
  10     start_time = time.time()
  11     print(f"Start time: {start_time}")
  12     yield
  13     end_time = time.time()
  14     print(f"End time: {end_time}")
    print(f"Execution time in seconds:
 
15 {end_time - start_time}")
  16  
# use the context manager in a with
 
17 statement
  18 with sleeper_context_manager():
  19     # sleep for 2 seconds
  20     time.sleep(2)
     
[4]:   Start time: 1649346091.427541
    End time: 1649346093.432682
Execution time in seconds:
   
2.005141019821167
 
S5:
The methods in the Context Manager Protocol are
__enter__ and __exit__.
__enter__(self):
This method is called by the with statement to
enter the runtime context related to current object.
The with statement will bind this method’s return
value to the target specified in the as clause of the
statement, if any.
__exit__(self, exc_type, exc_value, traceback):
This method is called when the execution leaves
the with code block.
It exits the runtime context related to this object.
The parameters describe the exception that caused
the context to be exited.
If the context was exited without an exception, all
three arguments will be None.
 
OceanofPDF.com
9. Functional Programming in
     

Python
 
OceanofPDF.com
 What is Functional Programming?
 
Functional Programming is a
programming paradigm where computer
programs are constructed by applying and
composing functions. It is a declarative
programming paradigm in which programs
are created by sequential functions rather
than statements. The main focus in
Functional Programming is “what to solve”
in contrast to an imperative style where
the main focus is “how to solve”.
Functional Programming uses expressions
instead of statements.
 
Declarative vs. Imperative:
Declarative programming expresses
the logic of a computation without
describing its control flow. It is like asking
your friend to fix your coffee machine. You
don’t care how your friend fixes it.
Imperative programming, on the
other hand, uses statements that change
a program’s state. It is like giving your
friend a list of steps that shows how to fix
the coffee machine.
 
Functional Programming is declarative.
So, the function definitions are trees of
expressions that map values to other
values, rather than a sequence of
imperative statements which update the
running state of the program.
In this chapter, we will learn the
concepts of Functional Programming in
Python. You can find the PyCharm project
for this chapter in the GitHub Repository
of this book.
 
Chapter Outline:
Core Concepts of Functional
Programming
Functional Programming in Python
Higher-Order Functions
lambda
map()
filter()
reduce()
QUIZ – Functional Programming in
Python
 SOLUTIONS – Functional
Programming in Python
 
 
OceanofPDF.com
 Core Concepts of Functional Programming
 
Functional Programming has some
concepts that any functional programming
language needs to follow. Here are some
of these concepts:
Functions are First-Class
Citizens:
In functional programming, functions
are treated as first-class citizens,
meaning that they can be bound to
names (including local identifiers),
passed as arguments, and returned
from other functions, just as any other
data type can. This allows programs to
be written in a declarative and
composable style, where small
functions are combined in a modular
manner.
Pure Functions:
In Functional Programming functions
need to be pure functions.
A pure function is a function that has
the following properties:
 The return values are identical for
identical arguments (no variation
with local static variables, non-
local variables, mutable reference
arguments or input streams).
The function has no side effects
(no mutation of local static
variables, non-local variables,
mutable reference arguments or
input/output streams).
Functions can be Higher-Order:
A higher-order function is a function
that has at least one of the following
properties:
takes one or more functions as
arguments
returns a function as its result
If a function has one of these two, then
we call it higher-order function and
Functional Programming is mainly
based on higher-order functions.
Immutability:
An immutable object is an object whose
state cannot be modified after it is
created. In other words, it is
unchangeable. In Functional
 Programming you are not allowed to
modify an object after it has been
initialized.
Immutability supports referential
integrity, which leads to function purity
and reduction of side effects.
Recursion:
Recursion is the case when a function
calls itself. Recursion allows us to break
down a problem into smaller pieces.
In theory, there shouldn’t exist any
loop structure ( for or while loops) in
Functional Languages. You should use
recursion for iteration instead of loops.
Recursion helps to remove some side
effects that might occur while writing
looping structures. And also, it makes
your code more expressive, readable
and maintainable.
 
There are more concepts in Functional
Programming but these will suffice for the
purpose of this chapter.
 
 
OceanofPDF.com
 Functional Programming in Python
 
Python is not primarily a functional
language. It is as a multi-paradigm
programming language which can be used
for both object-oriented and functional
programming. However, Python has many
constructs that enable a programmer to
follow functional programming approach
in application development.
In Python, everything is an object,
functions are first-class citizens and can
be higher-order. Python has built-in data
structures which support immutability.
In the next sections we will explore
built-in higher-order functions in Python
which are very important for Functional
Programming.
 
 
OceanofPDF.com
 lambda
 
lambda function allows us to create
function definitions in a declarative way. It
is a one-line, anonymous function. It can
take any number of arguments, but can
only have one expression. Here is the
syntax:
 
[1]: 1 lambda arguments : expression
 
Let’s say we want to split the given text
into its words. Python has built-in split()
function for this purpose. First, let’s see
how the split() function works:
 
[2]: 1 text = 'Python Hands- On Advanced'
  2 words = text.split()
  3 print(words)
     
[2]:   ['Python', 'Hands- On', 'Advanced']
 
In cell 2, we call the split() function on a
string and it returns a list of the words in
it. Actually, a list of items which are
separated with space character.
 Now let’s define a lambda function
which uses the split() function:
 
[3]: 1 # lambda function
  2 lambda x: x.split
 
In cell 3, we define a lambda function.
Here, lambda is the keyword, x is the
parameter and the code after the colon ( : )
is the function body. The function splits
the parameter x as: x.split() .
How will we call this lambda function? To
be able to call a lambda expression we
need to assign it to a variable. Let’s do it
now:
 
[4]: 1 # assign lambda function to a variable
  2 my_splitter_function = lambda x: x.split()
  3  
  4 # call the function variable
  5 words = my_splitter_function(text)
  6 print(words)
     
[4]:   ['Python', 'Hands- On', 'Advanced']
 
In cell 4, we assign our lambda function
to a variable named my_splitter_function . 
Since lambda is a function that takes an
argument named x, so is the
my_splitter_function .  In line 5, we call this
function by passing some text argument.
And as expected, the my_splitter_function
function returns a list of words in the given
text. Because behind the scenes, this is
what the lambda function returns.
Let’s define a new lambda function. This
time we want to define a function for
multiplying two numbers. So, the lambda
will have two parameters, x and y . And it
will return the multiplication result of
them.
 
[5]: 1 # multiplication with lambda function
  2 multiply = lambda x, y: x * y
  3 result = multiply(5, 8)
  4 print(result)
     
[5]:   40
 
In cell 5, we define a lambda function
and assign it to a variable called multiply .
This is a function and we call it in line 3
as: result = multiply(5, 8) . And we print the
result in the next line.
 Let’s define another lambda function.
This time for the power operation. The
name will be exponential and it will take
two parameters:
 
[6]: 1 # power function with lambda
exponential = lambda num, pow:
 
2 num**pow
  3 print(exponential(2, 7))
     
[6]:   128
 
Important Note: We do not need an
explicit return statement in the lambda
function. Python executes lambda function
body and returns the result automatically.
 
 
OceanofPDF.com
 map()
 
The map() function is one of the most
useful tools for Functional Programming in
Python. It is a higher-order function which
takes a function and an iterable as
arguments. Here is the syntax:
 
[7]: 1 map(function, iterable)
 
Parameters:
function : It is the given function to which
map() passes each element of the given
iterable.
iterable : It is an iterable which is to be
mapped.
The map() function returns an iterator
after applying the given function to each
item of a given iterable .
Let’s see an example on how to use it:
 
[8]: 1 # define a function
  2 def capitalize(text):
  3     return text.upper()
  4  
  5 # define a list
  6 heroes = ['batman', 'superman',
 'spiderman']
  7  
  8 # call the map() function
  9 capital_heroes = map(capitalize, heroes)
  10 print(capital_heroes)
     
[8]:   <map object at 0x1012fecb0>
 
In cell 8, we define a regular function
first. The function name is capitalize and it
takes some string as the argument. It
returns the capitalized form of this string.
This is the function which we will pass to
the map() function.
In line 6, we define a list of superheroes.
The items are all-in lower-case letters. This
is the iterable for the map() function.
In line 9, we call the map() function. It
takes two arguments; the capitalize
function and the heroes list. And we assign
the returning value to a variable called
capital_heroes . As you see in the cell
output, capital_heroes is a map object
(which is an iterator).
Now let’s print the items in this map
object iterator. We can easily convert it to
a list and print it:
  
[9]: 1 # print the items in map object
  2 print(list(capital_heroes))
     
[9]:   ['BATMAN', 'SUPERMAN', 'SPIDERMAN']
 
In the output of cell 9, you see the
resulting list after we apply the map()
function to the heroes list. All the letters
are in capital form now. The map() function
passes each item in the iterable( heroes )
one by one to the given
function( capitalize ).
Let’s see another example. In this
example we want to use a lambda function
in the map() function call.
 
[10]: 1 # map() function with lambda
  2 nums = [1, 2, 3, 4, 5]
  3  
  4 # call the map() function
  5 cubes = map(lambda x: x**3, nums)
  6  
  7 # print the result
  8 print(list(cubes))
     
[10]:   [1, 8, 27, 64, 125]
 
 In cell 10, line 5, we call the map()
function as: map(lambda x: x**3, nums) .
Here, the first parameter is a lambda
function. Lambda takes in a number and
returns its cube. The second parameter to
the map() function is the nums list. And as
you see in the output, the result is the
cubes of all items in this list.
 
Passing more than one iterable:
It is possible to pass multiple iterables
to the same map() function. Here is the
syntax for this:
 
map(function, iterable_1, iterable_2,
[11]:
1 iterable_3, ...)
 
The number of iterable arguments
passed to the map() must match the
number of arguments that function
parameter expects. Because the function
will act on all of the iterables at the same
time. Let’s see an example:
 
[12]: 1 # define two list
  2 list_1 = [1, 2, 3, 4, 5]
  3 list_2 = ['a', 'b', 'c', 'd', 'e']
  4  
  5 # define a concat function
  6 def concat(x, y):
  7     return str(x) + str(y)
  8  
  9 # call the map function on them
combined_list = map(concat, list_1,
 
10 list_2)
  11 print(list(combined_list))
     
[12]:   ['1a', '2b', '3c', '4d', '5e']
 
In cell 12, we call the map() function
with two iterables, list_1 and list_2 . And
the function we pass to the map is the
concat function which simply returns the
string concatenation of its parameters. As
you see in the cell output, the final list is
the concatenation of respective items
from both lists. Please keep in mind that,
the number of parameters of the concat
function is 2, which is the same as the
number of the lists we pass to the map .
 
 
OceanofPDF.com
 filter()
 
The filter() function is another
important tool for Functional Programming
in Python. Before the definition let’s see
its syntax first:
filter(function, iterable)
Parameters:
function : The function that tests each
element of the iterable. It either returns
True or False.
iterable : The sequence which needs to
be filtered. It can be a container of any
iterator type (list, tuple, set etc.).
Returns:
It returns an iterator which contains
filtered elements.
 
The filter() function constructs an
iterator from those elements of iterable for
which function returns True . iterable may
be either a sequence, a container which
supports iteration, or an iterator. If
function is None , the identity function is
assumed, that is, all elements of iterable
that are False are removed.
 Let’s see an example to understand it
better. In our example we will define a
function which checks if the given number
is an odd number. This function will return
True if the number is an odd number, False
otherwise. Then we will use this function
in the filter() function. The filter() function
will use it to filter the odd numbers from a
list of integers.
 
[13]: 1 # define a function
  2 def is_odd(number):
  3     """checks if number is odd"""
  4     if number % 2 == 1:
  5         return True
  6     else:
  7         return False
  8  
  9 # define a list to filter
  10 integers = [-4, -3, -2, -1, 0, 1, 2, 3, 4]
  11  
  12 # call the filter() function
  13 odd_integers = filter(is_odd, integers)
  14  
  15 # print the filter object
  16 print(odd_integers)
     
[13]:   <filter object at 0x103012c80>
 
 In cell 13, we define a function named
is_odd . This function returns True if the
given number is an odd number. In line 7,
we define a list of integer numbers. And in
line 10 we call the filter() function as:
filter(is_odd, integers) .
The parameters of the filter() function is
the is_odd (function) and the integers list
(iterable). It returns a filter object as you
see in the output.
We have to loop over this object or
convert to a list to see its items.
 
[14]: 1 # print the items
  2 print(list(odd_integers))
     
[14]:   [-3, -1, 1, 3]
 
In cell 14, we convert the odd_integers
object to a list and print it. And you see
the items in the cell output, which are the
odd numbers from the integers list.
Here is how the filter() function works in
Python:
1-          Each element in the integers list is
passed to the is_odd() function one by
one.
2-      If is_odd() returns True, that element is
selected, otherwise it will be
eliminated.
 
In the next example, let’s use a lambda
function to filter some words from a given
text. We want to filter the words which
contain the letter ‘a’ (either in lowercase
or uppercase).
 
[15]: 1 # some dummy text
text = 'Lorem ipsum dolor sit amet
 
2 consectetur adipiscing elit.'
  3  
  4 # filter() function with lambda
words_with_a = filter(lambda w: 'a' in
 
5 w.lower(), text.split())
  6  
  7 # print the words with a in it
  8 print(list(words_with_a))
     
[15]:   ['amet', 'adipiscing']
 
In cell 15, we use the filter() function
with the lambda function. The lambda ,
takes a word ( w ) as an argument and
returns True if this word (in lowercase)
contains the letter ‘a’ . The iterable to the
filter() function is text.split() , which is the
list of the words in the text.
 
 
OceanofPDF.com
 reduce()
 
The reduce() function is the last one
which we will cover in this chapter for
Functional Programming in Python. It
performs a rolling-computation on a given
sequence object. It is defined in functools
module. Here is the syntax:
functools.reduce(function, iterable[,
initializer]) :
Parameters:
function : the function to apply, also
known as the predicate of reduce function
iterable : The sequence which needs to
be reduced. It can be a container of any
iterator type (list, tuple, set etc.).
initializer : the value to start with.
Notice that the arguments in square
brackets [ ] are optional.
 
The reduce() function applies function of
two arguments cumulatively to the items
of iterable , from left to right, so as to
reduce the iterable to a single value.
For example, reduce(lambda x, y: x+y, [1,
2, 3, 4, 5]) calculates ((((1+2)+3)+4)+5) .
 The left argument, x, is the
accumulated value and the right
argument, y , is the update value from the
iterable. If the optional initializer is
present, it is placed before the items of
the iterable in the calculation, and serves
as a default when the iterable is empty. If
initializer is not given and iterable contains
only one item, the first item is returned.
Let’s use the reduce() function which
adds all the items in a list and returns the
final result:
 
[16]: 1 # import reduce() function
  2 from functools import reduce
  3  
  4 # define a summation function
  5 def custom_sum(n1, n2):
  6     return n1 + n2
  7  
  8 # define a list
  9 numbers = [1, 2, 3, 4, 5]
  10  
  11 # call reduce() function
summation_result = reduce(custom_sum,
 
12 numbers)
  13  
  14 # print the result
  15 print(summation_result)
     
[16]:   15
 
In cell 16, line 12, we call the reduce
function with custom_sum and numbers
arguments. The custom_sum function takes
two arguments and returns the sum of
them. Here is the mathematical
representation of the reduce() function in
this example: (((1 + 2) + 3) + 4) + 5) => 15
Here are the steps behind the scenes:
The reduce() function passes the first
two items ( 1 and 2 ) of the numbers
list to the custom_sum function.
The custom_sum function returns the
result of 1 + 2 as 3 .
The reduce() function takes this 3
and passes with the next item in the
list which is 3 .
The custom_sum function returns the
result of 3 + 3 as 6 .
The reduce() function takes this 6
and passes with the next item in the
list which is 4 .
The custom_sum function returns the
result of 6 + 4 as 10 .
 The reduce() function takes this 10
and passes with the next item in the
list which is 5 .
The custom_sum function returns the
result of 10 + 5 as 15 .
Since there is no items left in the
numbers list, the reduce() function
returns 15 as the final result.
 
Let’s do another example. This time we
want to find the largest number in a list.
We will define a custom function that
takes two parameters. It will return the
largest of these numbers. Then we will use
this function in the reduce() function. Let’s
see:
 
[17]: 1 # get the largest number in a list
  2 from functools import reduce
  3  
  4 nums = [5, 3, 12, 7, 15, 64, 19, 10]
  5  
  6 def get_max(n_1, n_2):
  7     if n_1 <= n_2:
  8         return n_2
  9     else:
  10         return n_1
  11  
  12 largest = reduce(get_max, nums)
  13 print(f"The Largest is: {largest}")
     
[16]:   The Largest is: 64
 
In cell 16, we define the get_max()
function which returns the maximum of
two parameters. Then in line 12, we pass
this function to the reduce() function. The
second argument of the reduce() function
is the nums list. The reduce() function will
pass the item pairs in the nums list to the
get_max() function and each time it will
return a new value for the next pair.
Finally, it will return the largest value in
the list, which is 64 in this case.
 
OceanofPDF.com
 
QUIZ – Functional Programming in Python
 
Now it’s time to solve the quiz for this
chapter. You can download the quiz file,
QUIZ_FunctionalProgrammingInPytho
n.zip, from the GitHub Repository of this
book. You should put the quiz file in the
FunctionalProgrammingInPython
project we build in this chapter.
 
Here are the questions for this chapter:
 
Q1:
Please explain the main concepts in Functional
Programming paradigm:
* Functions are First-Class Citizens
* Pure Functions
* Functions can be Higher- Order
* Immutability
* Recursion
 
Q2:
Define a lambda function which takes one numeric
parameter and checks if the given number is even
or not.
Assign it to a local variable.
Finally call this variable and print the results for 17
and 18.
 
[2]: 1 # Q 2:
  2  
  3 # define and assign the lambda function
  4 # ---- your solution here ---
  5  
  6 # call and print the result for 17 and 18.
  7 # ---- your solution here ---
     
[2]:   False
    True
 
Q3:
Define a map() function which takes a lambda
expression and a list as arguments.
This map() will use the lambda function which we
define in Q2.
We want it to return a map object that keeps True
and False values for each item in the list.
If the item is even, the respective value should be
True, otherwise it should be False.
 
The list is:
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
 
And the expected result (which should be a new
list):
[False, True, False, True, False, True, False, True,
False, True]
 
[3]: 1 # Q 3:
  2  
  3 # define the list
  4 # ---- your solution here ---
  5  
  6 # call the map() function
  7 # ---- your solution here ---
  8  
  9 # print the resulting list
  10 # ---- your solution here ---
     
[False, True, False, True, False, True,
[3]:  
False, True, False, True]
 
Q4:
We have a list of numbers which we want to filter
the negative ones.
Here is the list:
[4, -3, 0, -12, 9, 1, 2, -7, 8]
 
Define a filter() function which gives us the
negative numbers in this list.
It should take a lambda function and this list as the
parameters.
The filter function will return a filter object.
You should convert this filter object to a list.
Finally you should sort the list in ascending order
in-place.
 
The resulting list should be:
[-12, -7, -3]
 
Hints:
* lambda
* filter()
* list()
* sort()
 
[4]: 1 # Q 4:
  2  
  3 # define the list
  4 # ---- your solution here ---
  5  
  6 # define the filter function
  7 # ---- your solution here ---
  8  
  9 # convert filter object to list
  10 # ---- your solution here ---
  11  
  12 # sort the list
  13 # ---- your solution here ---
  14  
  15 # print the result
  16 # ---- your solution here ---
     
[4]:   [-12, -7, -3]
 
Q5:
Define a reduce() function which multiplies all the
items in the given list.
The initial value will be 1000, which means it will
start to multiply the first item with 1000.
 
Here is the list to multiply:
[1, 2, 3, 4]
 
The expected result is: 24000
 
Hints:
* functools.reduce()
* lambda
 
[5]: 1 # Q 5:
  2  
  3 # import reduce function
  4 # ---- your solution here ---
  5  
  6 # define the list
  7 # ---- your solution here ---
  8  
  9 # define the reduce function
  10 # ---- your solution here ---
  11  
  12 # print the final result
  13 # ---- your solution here ---
     
[5]:   24000
 
Q6:
Define a function named str_to_list.
It will take a list of strings as the parameter.
It will use the map() function to convert each string
item into a list.
And it will return a list of lists.
 
Expected Output:
List of strings:
['ABC', '123', 'xyz']
List of lists after function call:
[['A', 'B', 'C'], ['1', '2', '3'], ['x', 'y', 'z']]
 
[6]: 1 # Q 6:
  2  
  3 # define the function
  4 # ---- your solution here ---
  5  
  6 # define the list
  7 txt_list = ["ABC", "123", "xyz"]
  8  
  9 # print the string list
  10 print(f"List of strings:\n{txt_list}")
  11  
  12 # call the function
  13 list_of_lists = str_to_list(txt_list)
  14  
  15 # print list of lists
  16 print(f"List of lists:\n{list_of_lists}")
     
[6]:   List of strings:
    ['ABC', '123', 'xyz']
    List of lists:
    [['A', 'B', 'C'], ['1', '2', '3'], ['x', 'y', 'z']]
 
Q7:
We have to lists as:
list_1 = [1,2,3,4,5,6,7,10]
list_2 = [0,2,1,4,7,8,9,10]
 
From these lists, we want to filter the same
elements which are at the same indices.
We will define a function named
get_common_items_at_same_indices.
It will take two lists and return a new list that
contains the common elements.
Here is the expected output:
Common items at same indices: [2, 4, 10]
 
We will use two Python functions in our
get_common_items_at_same_indices function:
1- map()
2. itertools.compress()
 
itertools.compress(test_list, bool_list):
compress() function filters out all the elements from
the test_list based on the True values in the
bool_list.
 
[7]: 1 # Q 7:
  2  
  3 # import itertools.compress
  4 from itertools import compress
  5 # import eq from operator
  6 from operator import eq
  7  
  8 # define the function
  9 # ---- your solution here ---
  10  
  11 # define the lists
  12 list_1 = [1,2,3,4,5,6,7,10]
  13 list_2 = [0,2,1,4,7,8,9,10]
  14  
  15 # print the lists
  16 print(f"Lists:\n{list_1}\n{list_2}\n")
  17  
  18 # call function and get common items
common_items =
  get_common_items_at_same_indices(list_1,
19 list_2)
  20  
  21 # print common items
print(f"Common items at same
 
22 indices:\n{common_items}")
     
[7]:   Lists:
    [1, 2, 3, 4, 5, 6, 7, 10]
    [0, 2, 1, 4, 7, 8, 9, 10]
      
    Common items at same indices:
    [2, 4, 10]
 

OceanofPDF.com
 
SOLUTIONS – Functional Programming in
Python
 
Here are the solutions for the quiz for
this chapter.
 
S1:
 
The definitions of some core concepts in Functional
Programming:
Functions are First-Class Citizens:
In functional programming, functions are treated as
first-class citizens, meaning that they can be bound to
names (including local identifiers), passed as
arguments, and returned from other functions, just as
any other data type can. This allows programs to be
written in a declarative and composable style, where
small functions are combined in a modular manner.
Pure Functions:
In Functional Programming functions need to be pure
functions.
A pure function is a function that has the following
properties:
The return values are identical for identical
arguments (no variation with local static
variables, non-local variables, mutable
reference arguments or input streams).
The function has no side effects (no mutation of
local static variables, non-local variables,
mutable reference arguments or input/output
streams).
Functions can be Higher-Order:
 A higher-order function is a function that has at least
one of the following properties:
takes one or more functions as arguments
returns a function as its result
If a function has one of these two, then we call it higher-
order function and Functional Programming is mainly
based on higher-order functions.
Immutability:
An immutable object is an object whose state cannot be
modified after it is created. In other words, it is
unchangeable. In Functional Programming you are not
allowed to modify an object after it has been initialized.
Immutability supports referential integrity, which leads
to function purity and reduction of side effects.
Recursion:
Recursion is the case when a function calls itself.
Recursion allows us to break down a problem into
smaller pieces.
In theory, there shouldn’t exist any loop structure (for or
while loops) in Functional Languages. You should use
recursion for iteration instead of loops. Recursion helps
to remove some side effects that might occur while
writing looping structures. And also, it makes your code
more expressive, readable and maintainable.
 
S2:
 
[2]: 1 # S 2:
  2  
  3 # define and assign the lambda function
  4 is_even = lambda n: n % 2 == 0
  5  
  6 # call and print the result for 17 and 18.
  7 print(is_even(17))
  8 print(is_even(18))
     
[2]:   False
    True
 
S3:
 
[3]: 1 # S 3:
  2  
  3 # define the list
  4 a_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
  5  
  6 # call the map() function
evens = map(lambda n: n % 2 == 0,
 
7 a_list)
  8  
  9 # print the resulting list
  10 print(list(evens))
     
[False, True, False, True, False, True,
[3]:  
False, True, False, True]
 
S4:
 
[4]: 1 # S 4:
  2  
  3 # define the list
all_numbers = [4, -3, 0, -12, 9, 1, 2, -7,
 
4 8]
  5  
  6 # define the filter function
negative_numbers_filtered =
 
7 filter(lambda x: x < 0, all_numbers)
  8  
  9 # convert filter object to list
  10 negative_numbers =
list(negative_numbers_filtered)
  11  
  12 # sort the list
  13 negative_numbers.sort()
  14  
  15 # print the result
  16 print(negative_numbers)
     
[4]:   [-12, -7, -3]
 
S5:
 
[5]: 1 # S 5:
  2  
  3 # import reduce function
  4 from functools import reduce
  5  
  6 # define the list
  7 list_mult = [1, 2, 3, 4]
  8  
  9 # define the reduce function
multiplication = reduce(lambda x, y: x *
 
10 y, list_mult, 1000)
  11  
  12 # print the final result
  13 print(multiplication)
     
[5]:   24000
 
S6:
 
[6]: 1 # S 6:
  2  
  3 # define the function
  4 def str_to_list(str_list):
  5     result = map(list, str_list)
  6     return list(result)
  7  
  8 # define the list
  9 txt_list = ["ABC", "123", "xyz"]
  10  
  11 # print the string list
  12 print(f"List of strings:\n{txt_list}")
  13  
  14 # call the function
  15 list_of_lists = str_to_list(txt_list)
  16  
  17 # print list of lists
  18 print(f"List of lists:\n{list_of_lists}")
     
[6]:   List of strings:
    ['ABC', '123', 'xyz']
    List of lists:
    [['A', 'B', 'C'], ['1', '2', '3'], ['x', 'y', 'z']]
 
S7:
 
[7]: 1 # S 7:
  2  
  3 # import itertools.compress
  4 from itertools import compress
  5 # import eq from operator
  6 from operator import eq
  7  
  8 # define the function
  9 def
 get_common_items_at_same_indices(list1,
list2):
  10     result = map(eq, list1, list2)
  11     commons = compress(list1, result)
  12     return list(commons)
  13  
  14 # define the lists
  15 list_1 = [1,2,3,4,5,6,7,10]
  16 list_2 = [0,2,1,4,7,8,9,10]
  17  
  18 # print the lists
  19 print(f"Lists:\n{list_1}\n{list_2}\n")
  20  
  21 # call function and get common items
common_items =
  get_common_items_at_same_indices(list_1,
22 list_2)
  23  
  24 # print common items
print(f"Common items at same
 
25 indices:\n{common_items}")
     
[7]:   Lists:
    [1, 2, 3, 4, 5, 6, 7, 10]
    [0, 2, 1, 4, 7, 8, 9, 10]
     
    Common items at same indices:
    [2, 4, 10]
 
 
OceanofPDF.com
       Project 1 – Sending Emails with
Python
 
OceanofPDF.com
 Project Setup
 
In this project, we will learn how to send
emails using Python. You can find the
PyCharm project for this chapter in the
GitHub Repository of this book.
 
Chapter Outline:
What is SMTP?
The smtplib Module
Setting Up a Local SMTP Server
Configure a Fake Email Server
Sending HTML Emails
Sending Emails with Attachments
Sending Emails with Images
Sending Emails via Gmail
 
 
OceanofPDF.com
 What is SMTP?
 
Simple Mail Transfer Protocol (SMTP) is a
communication protocol for electronic mail
transmission. It is a universal standard,
which was first defined in 1982 by RFC
821, and updated in 2008 by RFC 5321 to
Extended SMTP additions. Mail servers
and other message transfer agents use
SMTP to send and receive mail messages.
 
 
OceanofPDF.com
 The smtplib Module
 
In Python, email sending operations are
mainly done by the help of the built-in
smtplib module. The smtplib module
defines an SMTP client session object that
can be used to send mail to any internet
machine with an SMTP or ESMTP listener
daemon.
We also have a separate package called
email in Python. The email package is a
library for managing email messages. It is
specifically not designed to do any
sending of email messages to SMTP (RFC
2821), NNTP, or other servers; those are
functions of the smtplib module. The email
package attempts to be as RFC-compliant
as possible, supporting RFC 5322 and RFC
6532 (Internet standards for SMTP).
 
 
OceanofPDF.com
 Setting Up a Local SMTP Server
 
To be able to work in our local
environment to send emails, we need to set
up the local SMTP debugging server. Python
offers the smtpd module for this purpose. It
has a DebuggingServer class, which will
discard messages you are sending out and
will print them to the console.
 
class smtpd.DebuggingServer(localaddr,
remoteaddr) :
Creates a new debugging server.
Arguments are as per SMTPServer .
Messages will be discarded, and printed on
stdout.
 
Now, let us set a local SMTP server on
localhost:1025 . PyCharm has a built-in
terminal in it. So, we will use this feature to
run Python commands without leaving
PyCharm. The command below will set an
SMTP server on port 1025 . Either use
“python” or “python3” at start depending on
your operating system.
 
  python -m smtpd -n -c DebuggingServer
localhost:1025
python3 -m smtpd -n -c DebuggingServer
 or localhost:1025
 
If you run this command successfully,
now you have a local SMTP server listening
on port 1025. Let’s test it:
 
LocalSMTPServer.py file:
 
[1]: 1 # import modules and packages
  2 import smtplib
from email.message import
 
3 EmailMessage
  4  
  5 # define email attributes
  6 subject = "Test Email Subject"
  7 sender = "[email protected]"
  8 receiver = "[email protected]"
  9  
  10 # define email object
  11 msg = EmailMessage()
  12  
  13 # set headers
  14 msg['Subject'] = subject
  15 msg['From'] = sender
  16 msg['To'] = receiver
  17  
  18 # set body text
msg.set_content("This is test email body
 
19 on local server...")
  20  
  21 # define port
  22 port = 1025
  23  
# call smtplib.SMTP in the with context
 
24 manager
with smtplib.SMTP('localhost', port) as
 
25 server:
  26     # send message
  27     server.send_message(msg)
  28     # print result to run console
  29     print("Test email sent successfully!")
     
[1]:   ---------- MESS AGE FOLLOWS ----------
    b'Subject: Test Email Subject'
    b'From: [email protected]'
    b'To: [email protected]'
    b'Content-Type: text/plain; charset="utf-8"'
    b'Content-Transfer-Encoding: 7bit'
    b'MIME-Version: 1.0'
    b'X-Peer: ::1'
    b''
    b'This is test email body on local server...'
    ------------ END MESS AGE ------------
 
In cell 1, we send a simple email using
the smtplib module, email package and a
local development server. Before getting
into the details of this code, here is an
image of PyCharm showing the result in the
terminal window:
 
 Figure 10-1: Email output on local SMTP Server
 
In cell 1, we import the necessary smtplib
module and the EmailMessage class from the
email package.
In lines 6 to 8, we define the subject ,
sender and receiver for our email.
In line 11, we instantiate an EmailMessage
object named msg . This object will contain
the email header and the body.
In lines 14 to 16 we set some attributes
of the msg object, which are ‘Subject’ ,
‘From’ and ‘To’ .
In line 19 we set the content of the email
as: msg.set_content("This is test email body on
local server...") . The set_content() method is
responsible for creating the email body.
Here, we just pass a simple text.
In line 22, we set the port for our local
server, which is 1025 in our case.
In line 25, we have a with context
manager. In which we call the SMTP class
constructor as: smtplib.SMTP('localhost',
port) . We pass two parameters: localhost
and port . The constructor returns an active
local server listening on the specified port.
 In line 27, inside the with context
manager scope, we call the send_message()
on our local server as:
server.send_message(msg) .
And as you see in the Terminal output,
our email is sent successfully on our local
server.
 
 
OceanofPDF.com
 Configure a Fake Email Server
 
An email server is an application which is
used to send and receive emails, by using the
SMTP (for Outgoing Messages) and POP3 (for
Incoming Messages). These servers are
provided globally by email providers.
Here are some public mail servers:
 
Google mail -
Server Authentication Port
Gmail
SMTP Server
(Outgoing smtp.gmail.com SSL 465
Messages)
  smtp.gmail.com StartTLS 587
POP3 Server
(Incoming pop.gmail.com SSL 995
Messages)
 
Outlook.com Server: Authentication: Port:
SMTP Server
(Outgoing smtp.live.com StartTLS 587
Messages)
POP3 Server
(Incoming pop3.live.com SSL 995
Messages)
 
Yahoo
Server: Authentication: Port:
Mail
SMTP
Server
smtp.mail.yahoo.com SSL 465
(Outgoing
Messages)
POP3 pop.mail.yahoo.com SSL 995
Server
(Incoming
Messages)
 
To set up a real mail server, there are some
security settings which you have to change in
your account to make them work. We will see
these settings for a Gmail account later in this
chapter. For the sake of simplicity, in this
project, we will mainly work with a fake SMTP
server.
 
Fake SMTP Server:
Fake SMTP server imitates the work of a real
3rd party web server. In this project, we will use
Mailtrap. Mailtrap is a platform for safe and
quick email testing using SMTP. Beyond testing
email sending, it will let us check how the email
will be rendered and displayed, review the
message raw data as well as will provide us
with a spam report. Mailtrap is very easy to set
up: all you need is just to copy the credentials
generated by the app and paste them into your
code.
Mailtrap offers a free plan which enables you
to send 500 mails per month with no cost. You
can use your Google, GitHub of Office 365
accounts to set up Mailtrap easily. Or you can
sign up with your email if you don’t have any of
these accounts. Once you signed up, you can
see the necessary credentials on your settings
page.
To copy the credentials, navigate to My Inbox .
And under the SMTP Settings tab, you will see a
dropdown as Integrations . Select smtplib under
P ython and you will see all the information you
need to send emails to Mailtrap. See the image
below, showing my settings:
 
 Figure 10-2: Login Credentials and sample email on Mailtrap
 
Once you set up your account on Mailtrap and
copy the smtplib integration details, it is very
easy to use it in PyCharm project. Create a new
Python file named Mailtrap.py and paste the
code you copied. Here it is:
 
Mailtrap.py file:
 
[2]: 1 # send email to Mailtrap fake server
  2  
  3 import smtplib
  4  
sender = "Private Person
 
5 <[email protected]>"
  6 receiver = "A Test User <[email protected]>"
  7  
  8 message = f"""\
  9 Subject: Hi from Hands- On Python Advanced
  10 To: {receiver}
  11 From: {sender}
  12  
  13 # This is a test e-mail message from PyCharm,
  14 # developed with smtplib with Mailtrap server.
  15 Python is really awesome :))
  16 """
  17  
with smtplib.SMTP("smtp.mailtrap.io", 2525) as
 
18 server:
  19     try:
        server.login("<your user_id>", "<your
 
20 password>")
        server.sendmail(sender, receiver,
 
21 message)
  22     except Exception as ex:
  23         print(ex)
  24     else:
        print("Message sent successfully to
 
25 Mailtrap.")
     
[2]:   Message sent successfully to Mailtrap.
 
In cell 2, we paste the code we copied from
the Mailtrap SMTP Settings. All we add is a
simple try-except-else block in the with context
manager. You have to replace your user_id and
password in line 20.
Here is what they are:
user_id : your login generated by Mailtrap
password : your password generated by
Mailtrap
 
When we run this code, a simple text email is
sent successfully to the Mailtrap server. See the
image below:
 
 Figure 10-3: A text email sent to the Mailtrap server
 
 
OceanofPDF.com
 Sending HTML Emails
 
In the previous section we sent simple
text emails. Which is not what you need in
real world. In an actual email, you need to
add some formatting, links, or images. The
common way of doing this is to put all of
them into an HTML structure. Python’s built-
in email package is the tool for any kind of
email formatting.
 
MIME: Multipurpose Internet Mail
Extensions (MIME) is an Internet standard
that extends the format of email messages
to support text in character sets other than
ASCII, as well as attachments of audio,
video, images, and application programs. It
is the way we combine HTML and plain text
in emails. In Python, we have email.mime
module for handling MIME-specific
operations.
The common approach is to write two
separate versions of the same email. A plain
text version and an HTML version. Then we
merge them with an instance of
MIMEMultipart class. In that way, our
message has two rendering options. In case
the HTML part isn’t be rendered successfully
for some reason, a text version will still be
available. You thank think of the text version
as a fallback option.
 
SendingHTMLEmail.py file:
 
[3]: 1 # Sending HTML Email
  2  
  3 # import smtplib and email.mime classes
  4 import smtplib
  5 from email.mime.text import MIMEText
from email.mime.multipart import
 
6 MIMEMultipart
  7  
  8 # define the port
  9 port = 2525
  10  
  11 # SMTP server
  12 smtp_server = "smtp.mailtrap.io"
  13  
  14 # login credentials
  15 # paste your login generated by Mailtrap
  16 login = "<your user_id>"
# paste your password generated by
 
17 Mailtrap
  18 password = "<your password>"
  19  
  20 # sender and receiver emails
  21 sender = "[email protected]"
  22 receiver = "[email protected]"
  23  
  24 # message object
  25 message = MIMEMultipart("alternative")
message["Subject"] = "HTML Mail Test with
 
26 multipart"
  27 message["From"] = sender
  28 message["To"] = receiver
  29  
  30 # --- PL AIN TEXT PART ---
with open("files/plain_text_part.txt") as
 
31 plain_text_part:
  32     text = plain_text_part.read()
  33  
  34 # --- HTML PART ---
with open("files/html_part.txt") as
 
35 html_part:
  36     html = html_part.read()
  37  
  38 # convert both parts to MIMEText objects
  39 part1 = MIMEText(text, "plain")
  40 part2 = MIMEText(html, "html")
  41  
# add the parts to the MIMEMultipart
 
42 message
  43 message.attach(part1)
  44 message.attach(part2)
  45  
  46 # send your email
with smtplib.SMTP(smtp_server, port) as
 
47 server:
  48     try:
  49         server.login(login, password)
        server.sendmail(sender, receiver,
 
50 message.as_string())
  51     except Exception as ex:
  52         print(ex)
  53     else:
        print("Multipart Message sent
 
54 successfully...")
     
[3]:   Multipart Message sent successfully...
 
For plain text and HTML options, we
create two separate .txt files under the
“files/” folder in our PyCharm project. Their
names are “plain_text_part.txt” and
“html_part.txt” . We read their contents and
assign them to local variables in lines 31
and 35.
In lines 39 and 40, we call the MIMEText
class to pass these two parts; “plain” and
“html” .
In lines 43 and 44, we attach the parts to
the MIMEMultipart message.
In line 47, we set a with context manager
to login and send the message to our SMTP
server.
Here is the mail which we send with this
code:
 
 Figure 10-4: An HTML email sent to the Mailtrap server
 
Here are the contents of
“plain_text_part.txt” and “html_part.txt” files:
 
plain_text_part.txt:
  
  Hello Developer,
   
  We are glad to see you here.
   
  https://www.amazon.com/dp/B09WVB49W8
Hands- On Python Advanced is great for learning
  Python in-depth.
  Feel free to contact us if you need any help!
   
  Musa Arda
 
html_part.txt:
 
  <html>
    <body>
      <p>Hello Developer,<br>
         We are glad to see you here.</p>
      <p>
        <a
  href="https://www.amazon.com/dp/B09WVB49W8">
              Hands- On Python Advanced
          </a>
          is great for learning Python in-depth.
      </p>
      <br>
    <p>Feel free to <strong>contact us</strong>
  if you need any help!</p>
      <br>
      <p><b>Musa Arda</b></p>
    </body>
  </html>
 
 
OceanofPDF.com
 Sending Emails with Attachments
 
Attachments are MIME objects but they
need to be encoded by using the base64
module before sending. You can attach text
files, images, audio files, and even
applications in Python. Every object type
has its own email class. For example audio
files are email.mime.audio.MIMEAudio or
images are email.mime.image.MIMEImage .
We will use a PDF attachment for the next
example. The PDF file which we will send is
the Keymap Reference of PyCharm IDE. To
view the keymap reference as PDF, select
Help | Keyboard Shortcuts PDF from the main
menu in PyCharm. You can also find the Mac
version under the “files” folder in the
current PyCharm project. File name is
ReferenceCardForMac.pdf .
Here is the complete code:
 
SedingAttachments.py file:
 
[4]: 1 # Sending Attachments with Email
  2 # import modules and classes
  3 import smtplib
  4 from email import encoders
  5 from email.mime.base import MIMEBase
  6 from email.mime.multipart import
MIMEMultipart
  7 from email.mime.text import MIMEText
  8  
  9 # server and port
  10 smtp_server = "smtp.mailtrap.io"
  11 port = 2525
  12  
# your login credentials generated by
 
13 Mailtrap
  14 login = "<your user_id>"
  15 password = "<your password>"
  16  
  17 # sender and receiver emails
  18 sender = "[email protected]"
  19 receiver = "[email protected]"
  20  
  21 # message object
  22 message = MIMEMultipart()
message["Subject"] = "PyCharm Keymap
 
23 Reference"
  24 message["From"] = sender
  25 message["To"] = receiver
  26  
  27 # email body
email_body = "You may find the Keymap
  Reference of PyCharm IDE in the
28 attachments."
message.attach(MIMEText(email_body,
 
29 "plain"))
  30  
  31 # file info
  32 filepath = "files/ReferenceCardForMac.pdf"
  33 filename = "ReferenceCardForMac.pdf"
  34  
  35 # open the PDF file in read-binary mode
  36 with open(filepath, "rb") as attachment:
    # content type "application/octet-
 
37 stream" means
  38     # a MIME attachment is a binary file
    part = MIMEBase("application", "octet-
 
39 stream")
  40     part.set_payload(attachment.read())
  41  
  42 # Encoding => to base64
  43 encoders.encode_base64(part)
  44  
  45 # Add mail header
  46 part.add_header(
  47     "Content-Disposition",
  48     f"attachment; filename= {filename}",
  49 )
  50  
# Add attachment to the message and
 
51 convert it to string
  52 message.attach(part)
  53 text = message.as_string()
  54  
  55 # send the email with attachment
with smtplib.SMTP(smtp_server, port) as
 
56 server:
  57     try:
  58         server.login(login, password)
        server.sendmail(sender, receiver,
 
59 text)
  60     except Exception as ex:
  61         print(ex)
  62     else:
        print("Mail with attachment sent
 
63 successfully...")
     
[4]:   Mail with attachment sent successfully...
 
In cell 4, line 39, we set the application
type as "octet-stream" which means binary
files, including PDF. Then in line 40, we set
the payload by reading the file in the
specified path.
In line 43, we encode the file content to
base64 with the help of the encoders
module in the email package.
In lines 46 to 48, we set the header.
In line 52, we attach the encoded
attachment to the message object as:
message.attach(part) . Then we convert the
message object to string as: text =
message.as_string() .
And finally, inside the with context
manager we login to the server and send
the email. Here is the mail output after we
run this code:
 
Figure 10-5: An email sent with attachment to the Mailtrap
server
 
 
OceanofPDF.com
 Sending Emails with Images
 
It is very easy to send emails with images in it.
All we need is some HTML document with some
image content. Keep in mind that, we need to
encode the images to base64. Here is a simple
example:
 
MailsWithImages.py file:
 
[5]: 1 # Sending Emails with Images
  2 # import the necessary components first
  3 import smtplib
  4 from email.mime.text import MIMEText
  5 from email.mime.multipart import MIMEMultipart
  6 import base64
  7  
  8 # server and port
  9 smtp_server = "smtp.mailtrap.io"
  10 port = 2525
  11  
  12 # your login credentials generated by Mailtrap
  13 login = "<your user_id>"
  14 password = "<your password>"
  15  
  16 # sender and receiver emails
  17 sender = "[email protected]"
  18 receiver = "[email protected]"
  19  
  20 # message object
  21 message = MIMEMultipart("alternative")
  22 message["Subject"] = "Python Logo in Email"
  23 message["From"] = sender
  24 message["To"] = receiver
  25  
  26 # read and encode the image file
  27 encoded_image =
base64.b64encode(open("images/python_logo.jpeg",
"rb").read()).decode()
  28  
  29 html = f"""\
  30 <html>
  31 <body>
    <h1>This is an email with image in it</h1>
 
32 <br>
   <img src="data:image/jpg;base64,
 
33 {encoded_image}">
  34 </body>
  35 </html>
  36 """
  37  
  38 # set MIME type and attach it to the message
  39 part = MIMEText(html, "html")
  40 message.attach(part)
  41  
  42 # send the email
  43 with smtplib.SMTP(smtp_server, port) as server:
  44     try:
  45         server.login(login, password)
        server.sendmail(sender, receiver,
 
46 message.as_string())
  47     except Exception as ex:
  48         print(ex)
  49     else:
  50         print("Email with Image sent successfully...")
     
[5]:   Email with Image sent successfully…
 
And here is the email on Mailtrap server after
we run the code in cell 5:
 
 Figure 10-6: An email sent with image in it to the Mailtrap server
 
 
OceanofPDF.com
 Sending Emails via Gmail
 
In this section, we will see how we can
send email with attachments via Gmail. We
choose Gmail, because it is one the most
popular mail servers and the process is very
similar to any other commercial server.
To be able to send emails via your Gmail
account, there are some settings you need
to do on Gmail side. First you need to
disable 2-Step Verification (which is not
recommend). Second you need to allow less
secure apps (which is not recommend
either).
 

Figure 10-7: Allowing less secure apps on Gmail Account

 
The best practice is to set up OAuth2
authorization protocol for Gmail API. OAuth2
is more difficult but recommended due to
obvious security reasons.
If you have finished the necessary
security settings, then you are ready to use
Gmail server for sending emails with
Python. In a test account, I allowed less
secure apps for the sake of simplicity in this
project.
Here is some basic information which you
should know about Gmail:
server name: smtp.gmail.com
port: 465 for SSL/TLS connection
(preferred)
or port: 587 for STARTTLS connection
username: your Gmail email address
password: your password
 
Let’s send a simple email using our Gmail
account:
 
SendViaGmail.py file:
 
[6]: 1 # Sending Emails via Gmail
  2 # import libraries
  3 import smtplib
from email.mime.multipart import
 
4 MIMEMultipart
  5 from email.mime.text import MIMEText
  6  
  7 # server and port
  8 smtp_server = "smtp.gmail.com"
  9 port = 587  # for STARTTLS connection
  10  
  11 content = '''Hi Gmail,
  12 This is a simple email created with Python.
  13 Sending emails with Python is very fun :)
  14  
  15 Best Regards,
  16 Musa Arda'''
  17  
  18 # mail addresses and password
sender =
 
19 '<your_gmail_address>@gmail.com'
  20 password = '<your_gmail_password>'
  21 receiver = '[email protected]'
  22  
  23 # set MIME data
  24 message = MIMEMultipart()
  25 message['From'] = sender
  26 message['To'] = receiver
message['Subject'] = 'Email sent from
 
27 Python to Gmail.'
  28  
  29 # set the body
  30 message.attach(MIMEText(content, 'plain'))
  31  
  32 # create SMTP session
session = smtplib.SMTP('smtp.gmail.com',
 
33 port)
  34  
  35 # enable security
  36 session.starttls()
  37  
  38 # login with credentials
  39 session.login(sender, password)
  40  
# convert message to string before
 
41 sending
  42 text = message.as_string()
  43  
  44 # send the email
  45 session.sendmail(sender, receiver, text)
  46  
  47 # close the session
  48 session.quit()
 
In cell 6, we send an email using Gmail
server.
In line 33, we start an SMTP session.
And in line 36, we enabled security by
calling the starttls() method. starttls() puts
the SMTP connection in TLS (Transport Layer
Security) mode. All SMTP commands that
follow will be encrypted.
In line 39, we login on the current session
as: session.login(sender, password) .
In line 45, we send the email and in line
48, we quit the session.
Here is the email in Gmail account inbox
after running this code:
 
 Figure 10-8: An email sent via Gmail server
 
This finalizes our project on sending emails
in Python. In the next chapter, you will
have an assignment related to project.
Good luck with your assignment.
 
 
OceanofPDF.com
        Assignment 1 – Sending Emails
with Python
 
OceanofPDF.com
 Assignment Setup
 
We finished our first project in this book
which is Project 1 - Sending Emails with
Python. Now it’s your turn to recreate the
same project. This chapter is the
assignment on the Project 1.
 
In this assignment you will start setting
up a local development server to send
emails. Then you will configure a fake
email server to be able to send emails to a
remote server. Then you will send html
emails, emails with attachments and
images. Finally, you will send emails using
a real email server like Gmail.
 
To complete the assignment, follow the
instructions where you see a #TODO
statement. You can find the solutions in
the previous chapter.
 
We will create a new PyCharm project.
The project name is Assignment_1 . You
should download the project from the
GitHub Repository of this book before
starting to code.
 
Chapter Outline:
What is SMTP?
The smtplib Module
Setting Up a Local SMTP Server
Configure a Fake Email Server
Sending HTML Emails
Sending Emails with Attachments
Sending Emails with Images
Sending Emails via Gmail
 
 
OceanofPDF.com
 What is SMTP?
 
Simple Mail Transfer Protocol (SMTP) is a
communication protocol for electronic mail
transmission. It is a universal standard,
which was first defined in 1982 by RFC
821, and updated in 2008 by RFC 5321 to
Extended SMTP additions. Mail servers
and other message transfer agents use
SMTP to send and receive mail messages.
 
 
OceanofPDF.com
 The smtplib Module
 
In Python, email sending operations are
mainly done by the help of the built-in
smtplib module. The smtplib module
defines an SMTP client session object that
can be used to send mail to any internet
machine with an SMTP or ESMTP listener
daemon.
We also have a separate package called
email in Python. The email package is a
library for managing email messages. It is
specifically not designed to do any
sending of email messages to SMTP (RFC
2821), NNTP, or other servers; those are
functions of the smtplib module. The email
package attempts to be as RFC-compliant
as possible, supporting RFC 5322 and RFC
6532 (Internet standards for SMTP).
 
 
OceanofPDF.com
 Setting Up a Local SMTP Server
 
To be able to work in our local
environment to send emails, we need to set
up the local SMTP debugging server. Python
offers the smtpd module for this purpose. It
has a DebuggingServer class, which will
discard messages you are sending out and
will print them to the console.
 
class smtpd.DebuggingServer(localaddr,
remoteaddr) :
Creates a new debugging server.
Arguments are as per SMTPServer .
Messages will be discarded, and printed on
stdout.
 
Now, let us set a local SMTP server on
localhost:1025 . PyCharm has a built-in
terminal in it. So, we will use this feature to
run Python commands. The command
below will set an SMTP server on port 1025 .
Either use “python” or “python3” at start
depending on your operating system.
 
python -m smtpd -n -c DebuggingServer
  localhost:1025
 or python3 -m smtpd -n -c DebuggingServer
localhost:1025
 
If you run this command successfully,
now you have a local SMTP server listening
on port 1025. Let’s test it.
 
LocalSMTPServer.py file:
 
[1]: 1 # import modules and packages
  2 # TODO - import smtplib module
# TODO - import EmailMessage class from
 
3 email.message
  4  
  5 # define email attributes
  6 # TODO - define subject variable
  7 # TODO - define sender variable
  8 # TODO - define receiver variable
  9  
  10 # define email object
# TODO - instantiate an EmailMessage
 
11 object
  12  
  13 # set headers
# TODO - set headers for EmailMessage
 
14 object
  15 # - Subject
  16 # - From
  17 # - receiver
  18  
  19 # set body text
  20 # TODO - set content for EmailMessage
 object
  21  
  22 # define port
  23 # TODO - define the port as 1025
  24  
# call smtplib.SMTP in the with context
 
25 manager
# TODO - set a with context manager for
 
26 local server
#      by calling the related class in the
 
27 smtplib module
  28 # TODO - send the message
  29 # TODO - print the final text for success
     
[1]:   ---------- MESS AGE FOLLOWS ----------
    b'Subject: Test Email Subject'
    b'From: [email protected]'
    b'To: [email protected]'
    b'Content-Type: text/plain; charset="utf-8"'
    b'Content-Transfer-Encoding: 7bit'
    b'MIME-Version: 1.0'
    b'X-Peer: ::1'
    b''
    b'This is test email body on local server...'
    ------------ END MESS AGE ------------
 
In cell 1, we send a simple email using
the smtplib module, email package and a
local development server. Before getting
into the details of this code, here is an
image of PyCharm showing the result in the
terminal window:
  

Figure 11-1: Email output on local SMTP Server

 
In cell 1, we import the necessary smtplib
module and the EmailMessage class from the
email package.
In lines 6 to 8, we define the subject ,
sender and receiver for our email.
In line 11, we instantiate an EmailMessage
object named msg . This object will contain
the email header and body.
In lines 14 to 16 we set some attributes
of the msg object, which are ‘Subject’ ,
‘From’ and ‘To’ .
In line 19 we set the content of the email
as: msg.set_content("This is test email body on
local server...") . The set_content() method is
responsible for setting the email body. Here,
we just pass a simple text.
In line 22, we set the port for our local
server, which is 1025 in our case.
In line 25, we have a with context
manager. In which we call the SMTP class
constructor as: smtplib.SMTP('localhost',
port) . We pass two parameters: localhost
and port . The constructor returns an active
local server listening on the specified port.
In line 27, inside the with context
manager scope, we call the send_message()
on our local server as:
server.send_message(msg) .
And as you see in the Terminal output,
our email is sent successfully on the local
server.
 
 
OceanofPDF.com
 Configure a Fake Email Server
 
An email server is an application which used
to send and receive emails, by using the SMTP
(for Outgoing Messages) and POP3 (for
Incoming Messages). These servers are
provided globally by email providers. For
example, here are some public mail servers:
 
Google mail -
Server Authentication Port
Gmail
SMTP Server
(Outgoing smtp.gmail.com SSL 465
Messages)
  smtp.gmail.com StartTLS 587
POP3 Server
(Incoming pop.gmail.com SSL 995
Messages)
 
Outlook.com Server: Authentication: Port:
SMTP Server
(Outgoing smtp.live.com StartTLS 587
Messages)
POP3 Server
(Incoming pop3.live.com SSL 995
Messages)
 
Yahoo
Server: Authentication: Port:
Mail
SMTP
Server
smtp.mail.yahoo.com SSL 465
(Outgoing
Messages)
POP3 pop.mail.yahoo.com SSL 995
Server
(Incoming
Messages)
 
To set up a real mail server, there are some
security settings which you have to change in
your account to make them work. We will see
these settings for a Gmail account later in this
chapter. For the sake of simplicity, in this
project, we will mainly work with a fake SMTP
server.
 
Fake SMTP Server:
Fake SMTP server imitates the work of a real
3rd party web server. In this project, we will use
Mailtrap. Mailtrap is a platform for safe and
quick email testing using SMTP. Beyond testing
email sending, it will let us check how the email
will be rendered and displayed, review the
message raw data as well as will provide us
with a spam report. Mailtrap is very easy to set
up: all you need is just to copy the credentials
generated by the app and paste them into your
code.
Mailtrap offers a free plan which enables you
to send 500 mails per month with no cost. You
can use your Google, GitHub of Office 365
accounts to set up Mailtrap easily. Or you can
sign up with your email if you don’t have any of
these accounts. Once you signed up, you can
see the necessary credentials on your settings
page.
To copy the credentials, navigate to My Inbox .
And under the SMTP Settings tab, you will see a
dropdown as Integrations . Select smtplib under
P ython and you will see all the information you
need to send emails to Mailtrap. See the image
below, showing my settings:
 
 Figure 11-2: Login Credentials and sample email on Mailtrap
 
Once you set up your account on Mailtrap and
copy the smtplib integration details, it is very
easy to use it in PyCharm project. Create a new
Python file named Mailtrap.py and paste the
code you copied. Here it is:
 
Mailtrap.py file:
 
[2]: 1 # send email to Mailtrap fake server
  2  
  3 # TODO - import smtplib module
  4  
  5 # TODO - define sender and receiver variables
  6  
# TODO - set the message object with an f-
 
7 string
  8  
# TODO - set a with context manager to call the
 
9 SMTP class
  10 #        with mailtrap server and port
  11 # TODO - write a try-except-else block,
  12 #        login to the server and send the email
  13 #        print a success message in the end
     
[2]:   Message sent successfully to Mailtrap.
 
In cell 2, we paste the code we copied from
the Mailtrap SMTP Settings. All we add is a
simple try-except-else block in the with context
manager. You have to replace your user_id and
password in line 20.
 Here is what they are:
user_id : your login generated by Mailtrap
password : your password generated by
Mailtrap
 
When we run this code, a simple text email is
sent successfully to the Mailtrap server. See the
image below:
 

Figure 11-3: A text email sent to the Mailtrap server

 
 
OceanofPDF.com
 Sending HTML Emails
 
In the previous section we sent simple
text emails. Which is not what you need in
real world. In an actual email, you need to
add some formatting, links, or images. The
common way of doing this is to put all of
them into an HTML structure. Python’s built-
in email package is the tool for any kind of
email formatting.
 
MIME: Multipurpose Internet Mail
Extensions (MIME) is an Internet standard
that extends the format of email messages
to support text in character sets other than
ASCII, as well as attachments of audio,
video, images, and application programs. It
is the way we combine HTML and plain text
in emails. In Python, we have email.mime
module for handling MIME-specific
operations.
The common approach is to write two
separate versions of the same email. A text
version and an HTML version. Then we
merge them with an instance of
MIMEMultipart class. In that way, our
message has two rendering options. In case
the HTML part isn’t be rendered successfully
for some reason, a text version will still be
available. You thank think of the text version
as a fallback option.
 
SendingHTMLEmail.py file:
 
[3]: 1 # Sending HTML Email
  2  
# TODO - import smtplib and email.mime
 
3 classes
  4 #        MIMEText and MIMEMultipart
  5  
  6 # TODO - define the port on 2525
  7  
  8 # TODO - define SMTP server for mailtrap
  9  
  10 # TODO - set login credentials
  11  
  12 # TODO - set sender and receiver emails
  13  
# TODO - create a message object from
 
14 MIMEMultipart
  15  
# TODO - set Subject, From, To attributes
 
16 of message object
  17  
  18 # --- PL AIN TEXT PART ---
# TODO - set a with context manager to
 
19 read "files/plain_text_part.txt"
  20 #        assign the content to a variable
 called text
  21  
  22 # --- HTML PART ---
# TODO - set a with context manager to
 
23 read "files/html_part.txt"
#        assign the content to a variable
 
24 called html
  25  
# TODO - convert both parts to MIMEText
 
26 objects
  27 #       "plain" and "html"
  28  
# TODO - add the parts to the
 
29 MIMEMultipart message
  30  
# TODO - set a with context manager to
 
31 call the SMTP class
  32 #        with mailtrap server and port
  33 # TODO - write a try-except-else block,
#        login to the server and send the
 
34 email
  35 #        print a success message in the end
     
[3]:   Multipart Message sent successfully...
 
For plain text and html options, we create
two separate txt files under the “files/”
folder in our PyCharm project. Their names
are “plain_text_part.txt” and “html_part.txt” .
We read their contents and assign them to
local variables in lines 31 and 35.
 In lines 39 and 40, we call the MIMEText
class to pass these two parts; “plain” and
“html” .
In lines 43 and 44, we attach the parts to
the MIMEMultipart message.
In line 47, we set a with context manager
to login and send the message to our SMTP
server.
Here is the mail which we send with this
code:
 
 Figure 11-4: An HTML email sent to the Mailtrap server
 
Here are the contents of
“plain_text_part.txt” and “html_part.txt” files:
 
plain_text_part.txt:
  
  Hello Developer,
   
  We are glad to see you here.
   
  https://www.amazon.com/dp/B09WVB49W8
Hands- On Python Advanced is great for learning
  Python in-depth.
  Feel free to contact us if you need any help!
   
  Musa Arda
 
html_part.txt:
 
  <html>
    <body>
      <p>Hello Developer,<br>
         We are glad to see you here.</p>
      <p>
        <a
  href="https://www.amazon.com/dp/B09WVB49W8">
              Hands- On Python Advanced
          </a>
          is great for learning Python in-depth.
      </p>
      <br>
    <p>Feel free to <strong>contact us</strong>
  if you need any help!</p>
      <br>
      <p><b>Musa Arda</b></p>
    </body>
  </html>
 
 
OceanofPDF.com
 Sending Emails with Attachments
 
Attachments are MIME objects but they
need to be encoded by using the base64
module before sending. You can attach text
files, images, audio files, and even
applications in Python. Every object type
has its own email class. For example audio
files are email.mime.audio.MIMEAudio or
images are email.mime.image.MIMEImage .
We will use a PDF attachment for the next
example. The PDF file which we will send is
the Keymap Reference of PyCharm IDE. To
view the keymap reference as PDF, select
Help | Keyboard Shortcuts PDF from the main
menu in PyCharm. You can also find the Mac
version under the “files” folder in the
current PyCharm project. File name is
ReferenceCardForMac.pdf .
Here is the complete code:
 
SedingAttachments.py file:
 
[4]: 1 # Sending Attachments with Email
  2 # TODO - import modules and classes
#       - smtplib, encoders, MIMEBase,
 
3 MIMEMultipart, MIMEText
  4  
  5 # TODO - set server and port
  6  
  7 # TODO - set login credentials by Mailtrap
  8  
  9 # TODO - set sender and receiver emails
  10  
# TODO - create a message object with
 
11 MIMEMultipart
  12  
# TODO - set Subject, From, To attributes
 
13 of message object
  14  
  15 # TODO - set email body
  16  
# TODO - attach email body to the
 
17 message
  18  
# TODO - set file info (file path and file
 
19 name)
  20  
# TODO - open the PDF file in read-binary
 
21 mode
# TODO - set application of the MIMEBase
 
22 class
# TODO - set payload to attachment
 
23 content
  24  
# TODO - encode the message part to
 
25 base64
  26  
  27 # TODO - Add mail header
  28 #       - Content-Disposition,
  29 #       - attachment; filename=
  30  
  31 # TODO - Add attachment to the message
  32  
  33 # TODO - convert message it to string
  34  
# TODO - set a with context manager to
 
35 call the SMTP class
  36 #        with mailtrap server and port
  37 # TODO - write a try-except-else block,
#        login to the server and send the
 
38 email
  39 #        print a success message in the end
     
[4]:   Mail with attachment sent successfully...
 
In cell 4, line 39, we read set the
application type as "octet-stream" which
means binary files, including PDF. Then in
line 40, we set the payload by reading the
file in the specified path.
In line 43, we encode the file content to
base64 with the help of the encoders
module in the email package.
In lines 46 to 48, we set the header.
In line 52, we attach the encoded
attachment to the message object as:
message.attach(part) . Then we convert the
message object to string as: text =
message.as_string() .
And finally, inside the with context
manager we login to the server and send
the email. Here is the mail output after we
run this code:
 

Figure 11-5: An email sent with attachment to the Mailtrap

server
 
 
OceanofPDF.com
 Sending Emails with Images
 
It is very easy to send emails with images
in it. All we need is some HTML document
with some image content. Keep in mind
that, we need to encode the images to
base64. Here is a simple example:
 
MailsWithImages.py file:
 
[5]: 1 # Sending Emails with Images
# TODO - import the necessary
 
2 components first
  3 #       - smtplib module
  4 #       - MIMEText, MIMEMultipart
  5 #       - base64
  6  
  7 # TODO - set server and port
  8  
  9 # TODO - set login credentials by Mailtrap
  10  
  11 # TODO - set sender and receiver emails
  12  
# TODO - create a message object with
 
13 MIMEMultipart
  14  
# TODO - set Subject, From, To attributes
 
15 of message object
  16  
# TODO - read and encode the image file
 
17 at "images/python_logo.jpeg"
  18  
# TODO - define html content with an f-
 
19 string
#       - having an img tag in it with src as
 
20 your image
  21  
# TODO - create a MIMEText part with
 
22 MIME type as html
  23  
  24 # TODO - attach the part to the message
  25  
# TODO - set a with context manager to
 
26 call the SMTP class
  27 #        with mailtrap server and port
  28 # TODO - write a try-except-else block,
#        login to the server and send the
 
29 email
  30 #        print a success message in the end
     
[5]:   Email with Image sent successfully…
 
And here is the email on Mailtrap server
after we run the code in cell 5:
 
 Figure 11-6: An email sent with image in it to the Mailtrap
server
 
 
OceanofPDF.com
 Sending Emails via Gmail
 
In this section, we will see how we can
send email with attachments via Gmail. We
choose Gmail, because it is one the most
popular mail servers and the process is very
similar to any other commercial server.
To be able to send emails via your Gmail
account, there are some settings you need
to do on Gmail side. First you need to
disable 2-Step Verification (which is not
recommend). Second you need to allow less
secure apps (which is not recommend
either).
 

Figure 11-7: Allowing less secure apps on Gmail Account

 
The best practice is to set up OAuth2
authorization protocol for Gmail API. OAuth2
is more difficult but recommended due to
obvious security reasons.
If you finished the necessary security
settings, then you are ready to use Gmail
server for sending emails with Python. In a
test account, I allowed less secure apps for
the sake of simplicity in this project.
Here is some basic information which you
should know about Gmail:
server name: smtp.gmail.com
port: 465 for SSL/TLS connection
(preferred)
or port: 587 for STARTTLS connection
username: your Gmail email address
password: your password
 
Let’s send a simple email using our Gmail
account:
 
SendViaGmail.py file:
 
[6]: 1 # Sending Emails via Gmail
  2 # TODO - import modules and classes
  3 #       - smtplib module
  4 #       - MIMEMultipart, MIMEText
  5  
  6 # TODO - set server and port
  7 #       - "smtp.gmail.com"
  8 #       - 587 (for STARTTLS connection)
  9  
  10 # TODO - set some content
  11  
  12 # TODO - set mail addresses and password
  13 #       - sender, password, receiver
  14  
# TODO - create a message object with
 
15 MIMEMultipart
  16  
# TODO - set Subject, From, To attributes
 
17 of message object
  18  
# TODO - set the body by attaching the
 
19 content to message
  20  
  21 # TODO - create SMTP session
  22  
  23 # TODO - enable security (starttls)
  24  
# TODO - login to the session with
 
25 credentials
  26  
# TODO - convert message to string before
 
27 sending
  28  
  29 # TODO - send the email on the session
  30  
  31 # TODO - quit the session
 
In cell 6, we send an email using Gmail
server.
In line 33, we start an SMTP session.
 And in line 36, we enabled security by
calling the starttls() method. starttls() puts
the SMTP connection in TLS (Transport Layer
Security) mode. All SMTP commands that
follow will be encrypted.
In line 39, we login on the current session
as: session.login(sender, password) .
In line 45, we send the email and in line
48, we quit the session.
Here is the email in Gmail account inbox
after running this code:
 

Figure 11-8: An email sent via Gmail server

 
This finalizes our project on sending emails
in Python.
 
OceanofPDF.com
 12. Regular Expressions
       

 
Regular Expressions (RE) are a powerful
language concepts for matching text
patterns. A Regular Expression is a
sequence of characters that forms a
search pattern. It specifies a set of strings
that matches it.
Python has the re module for handling
regular expressions. The functions in this
module let you check if a particular string
matches a given regular expression (or if a
given regular expression matches a
particular string, which comes down to the
same thing).
This chapter gives an introduction to
regular expressions and shows how they
work in Python. You can find the PyCharm
project for this chapter in the GitHub
Repository of this book.
 
Chapter Outline:
The re Module
Metacharacters in RE
Sequences in RE
 Sets in RE
RE Functions
Match Object
QUIZ – Regular Expressions
SOLUTIONS – Regular Expressions
 
 
OceanofPDF.com
 The re Module
 
The re module in Python, provides
regular expression matching operations.
Both patterns and strings to be searched
can be Unicode strings (str) as well as 8-
bit strings (bytes).
Regular expressions use the backslash
character ( '\' ) to indicate special forms or
to allow special characters to be used
without invoking their special meaning.
This collides with Python’s usage of the
same character for the same purpose in
string literals; for example, to match a
literal backslash, one might have to write
'\\\\' as the pattern string, because the
regular expression must be \\ , and each
backslash must be expressed as \\ inside
a regular Python string literal.
Raw String (r): The 'r' at the start of
the pattern string designates a Python
“raw string” which passes through
backslashes without change. This is very
handy when you are dealing with regular
expressions. So, for regular expressions, it
is recommended to write pattern strings
with the 'r' in front of them.
Let’s do a basic example to see a simple
regular expression. We want to search
some characters in the given text. The
code for this section is in reModule.py file.
Here is the first example:
 
[1]: 1 # re Module
  2  
  3 import re
  4  
text = 'Hands- On Python Advanced with
 
5 Coding Exercises'
  6  
  7 # call the search method
  8 match = re.search(r'on', text)
  9  
  10 # print start and end indices
  11 print('Start Index:', match.start())
  12 print('End Index:', match.end())
     
[1]:   Start Index: 13
    End Index: 15
 
In cell 1, we see a simple regular
expression search by using the re.search()
method. We will see this method in detail
later in this chapter. The parameters are
the pattern we want to search and the
text. The pattern here is r ’on’ , which is the
raw string of the ‘on’ text. And the result
of the search() method is a match object.
In lines 11 and 12, we print the start
and end indices of the match object. As
you see the start index is 13, because the
search is case-sensitive and it is the first
index which ‘on’ text resides. And the end
index is 15.
 
 
OceanofPDF.com
 Metacharacters in RE
 
Metacharacters are one of the main the
building blocks of regular expressions.
They are special-purpose characters which
define how the regular expressions are
interpreted. They indicate rules of regular
expressions. For example, the ^ (caret)
metacharacter is used to match a pattern
only at the beginning of the string.
Here is the table of metacharacters in
regular expressions:
 
Character Description Example
[] A set of characters "[c-y]"
Signals a special
sequence (can also be
\ "\d"
used to escape special
characters)
"py..on"
Any character (except python
.
newline character) py12on
py#_on
^ Starts with "^python"
$ Ends with "code$"
"tre*"
0 or more occurrences
tr
* (for the preceding
tre
character)
tree
 + 1 or more occurrences "tre+"
(for the preceding tre
character) tree
"colou?r"
0 or 1 occurrences (for
? color
the preceding character)
colour
Exactly the specified
number of occurrences
(for the preceding "py{2}"
{n}
character) pyy
{min, max} -> at least
min, at most max
| Either or "good|nice"
Capture and group.
Groups multiple tokens
() together and creates a  
capture group for
extracting a substring.
Table 12-1: Metacharacters in Regular Expressions
 
OceanofPDF.com
 Sequences in RE
 
Regular Expressions mainly work with
sequences. A sequence is a special-
purpose character following the backslash
character ( '\' ). For example, \A is a
sequence. Below is the list of sequences in
RE and their meanings:
 
Sequence Description Example
Returns a match if the
\A specified characters are at "\AThis"
the beginning of the string
Matches a word boundary
(the position between a
word and a space). Returns
a match where the
specified characters are at
\b "rt\b"
the beginning or at the end
of a word. For
example,  rt\b  matches
the  rt  in  start  but not
the  rt  in  party .
Matches a nonword
boundary. Returns a match
where the specified
\B "\Brt"
characters are present, but
NOT at the beginning (or
at the end) of a word.
\d Returns a match where the "\d"
string contains digits
 (numbers from 0-9)
Returns a match where the
\D string DOES NOT contain "\D"
digits
Returns a match where the
string contains a white
\s space character (spaces, "\s"
tabs, form-feed characters,
etc...)
Matches any non-white
\S "\S"
space character.
Returns a match where the
string contains any word
characters (from a to Z,
\w digits from 0-9, and the "\w"
underscore _). This
expression is equivalent
to [A-Za-z0-9_].
Matches any non-word
character. This expression
\W "\W"
is equivalent to [^A-Za-z0-
9_].
Matches only the end of a
\Z string, or before a newline "world\Z"
character at the end.
Table 12-2: Sequences in Regular Expressions
 
 
OceanofPDF.com
 Sets in RE
 
A set in Regular Expressions
terminology, is a combination of
characters inside a pair of square brackets
[] with denotes a special meaning. Here
are the sets we use in Regular Expressions
in Python:
 
Set Description
Returns a match where one of the
[adz] specified characters (a, d, or z) are
present
Returns a match for any lower-case
[a-z]
character, alphabetically between a and z
Returns a match for any character
[^bdh]
EXCEPT b, d, and h
Returns a match where any of the
[012]
specified digits (0, 1 or 2) are present
Returns a match for any digit
[0-9]
between 0 and 9
[2-7] Returns a match for any two -digit
[0-4] numbers from 20 and 74
Returns a match for any character
[a-zA-
alphabetically between a and z,
Z]
lower case OR upper case
In sets, +, *, ., |, (), $,{} has no special
meaning, so [+] means:
[+]
return a match for any + character in the
string
 Table 12-3: Sets in Regular Expressions
 
 
OceanofPDF.com
 RE Functions
 
The re module defines several
functions, constants, and an exception.
Some of the functions are simplified
versions of the full featured methods for
compiled regular expressions. Most non-
trivial applications always use the
compiled form.
 
re.compile(pattern, flags=0) :
Compile a regular expression pattern
into a regular expression object, which can
be used for matching using its match() ,
search() and other methods, described
below.
The expression’s behavior can be
modified by specifying a flags value.
Values can be any of the following
variables, combined using bitwise OR (the
| operator).
 
The sequence:
compiled_re = re.compile(pattern)
result = compiled_re.match(string)
 
 is equivalent to:
result = re.match(pattern, string)
 
But using re.compile() and saving the
resulting regular expression object for
reuse is more efficient when the
expression will be used several times in a
single program.
Let’s rewrite the code in cell 1, using
the compile() function this time:
 
[2]: 1 # compile()
  2 import re
  3  
text = 'Hands- On Python Advanced with
 
4 Coding Exercises'
  5  
  6 # define the pattern
  7 pattern = r'on'
  8  
  9 # compile the regular expression
  10 compiled_re = re.compile(pattern)
  11  
# call the search method on the
 
12 compiled pattern
  13 match = compiled_re.search(text)
  14  
  15 # print start and end indices
  16 print('Start Index:', match.start())
  17 print('End Index:', match.end())
     
[2]:   Start Index: 13
    End Index: 15
 
In cell 2, we first define the pattern
variable as: pattern = r'on' . Keep in mind
that, we use the ‘r ’ character in front of
the text to make it a raw string.
Then in line 10, we create a compiled
regular expression object by calling the
compile() function as: compiled_re =
re.compile(pattern) .
Now that we have a compiled RE object
we can do the actual search as: match =
compiled_re.search(text) . As you see in the
output, the result is exactly the same as
the result of cell 1.
 
re.search(pattern, string, flags=0) :
Scan through string looking for the first
location where the regular expression
pattern produces a match, and return a
corresponding match object. Return None
if no position in the string matches the
pattern.
 
 re.match(pattern, string, flags=0) :
If zero or more characters at the
beginning of string match the regular
expression pattern, return a corresponding
match object. Return None if the string
does not match the pattern. If you want to
locate a match anywhere in string, use
search() instead.
 
[3]: 1 # match()
text = 'Hands- On Python Advanced with
 
2 Coding Exercises'
  3  
  4 # define the pattern
  5 pattern = r'on'
  6  
  7 # compile the regular expression
  8 compiled_re = re.compile(pattern)
  9  
# call the search method on the compiled
 
10 pattern
  11 match = compiled_re.match(text)
  12  
  13 # print start and end indices
  14 try:
  15     print('Start Index:', match.start())
  16     print('End Index:', match.end())
  17 except:
    print(f"No match found at the
 
18 beginning of the string for {pattern}")
     
[3]:   No match found at the beginning of the
string for on
 
re.fullmatch(pattern, string, flags=0) :
If the whole string matches the regular
expression pattern, return a corresponding
match object. Return None if the string
does not match the pattern.
 
Difference between re.match() and
re.fullmatch() :
Both of the re.match() and re.fullmatch()
functions attempt to match at the
beginning of the string.  The difference is
that re.match() matches only at the
beginning but re.fullmatch() tries to match
at the end as well.
 
re.split(pattern, string, maxsplit=0,
flags=0) :
Split string by the occurrences of
pattern. If capturing parentheses are used
in pattern, then the text of all groups in
the pattern are also returned as part of
the resulting list. If maxsplit is nonzero, at
most maxsplit splits occur, and the
remainder of the string is returned as the
final element of the list.
 
[4]: 1 # split()
  2 text_to_split = 'Words, words, words.'
  3  
# split from non-word characters (space,
 
4 comma, period etc.)
  5 print(re.split(r'\W+', text_to_split))
  6  
# split from non-word characters with
 
7 maxsplit = 1
  8 print(re.split(r'\W+', text_to_split, 1))
  9 # ['Words', 'words, words.']
  10  
  11 # split from letters a to f and ignore case
print(re.split('[a-f]+', '0a3B9',
 
12 flags=re.IGNORECASE))
     
[4]:   ['Words', 'words', 'words', '']
    ['Words', 'words, words.']
    ['0', '3', '9']
 
re.findall(pattern, string, flags=0) :
Return all non-overlapping matches of
pattern in string, as a list of strings or
tuples. The string is scanned left-to-right,
and matches are returned in the order
found. Empty matches are included in the
result.
  
[5]: 1 # findall()
# find words starting with f and have any
 
2 number of letters (a to z)
print(re.findall(r'\bf[a-z]*', 'which foot or
 
3 hand fell fastest'))
     
[5]:   ['foot', 'fell', 'fastest']
 
In cell 5, we search for the pattern of
\bf[a-z]* . Here is how we read this pattern
from left to right:
\bf match the letter f which are at
the beginning of the words (word
boundary)
[a-z]* matches any lower-case
character, alphabetically between a
and z and any number of
occurrences (zero or many).
So, the expression of \bf[a-z]* actually
means the words which starts with
the letter f. And in the output, you see a
list of these words as: ['foot', 'fell',
'fastest'] .
 
# find groups with equal sign between
[6]:
1 (words and digits)
  2 print(re.findall(r'(\w+)=(\d+)', 'set
 width=20 and height=10'))
     
[6]:   [('width', '20'), ('height', '10')]
 
In cell 6, we search for the pattern of
(\w+)=(\d+) . Here is how we read this
pattern:
The parentheses, () , mean capturing
groups.
\w means any word characters
\d means any digits
= is the equal sign
+ means 1 or more occurrences of
the preceding character
So, the patter of (\w+)=(\d+) is looking
for something like this: width=20 . A word
on the left of equal sign and some digits
on the right. And in the output, you see a
list of tuples.
 
re.finditer(pattern, string, flags=0) :
Return an iterator yielding match
objects over all non-overlapping matches
for the RE pattern in string. The string is
scanned left-to-right, and matches are
returned in the order found. Empty
matches are included in the result.
 
[7]: 1 # finditer()
text = 'python machine leaning is
 
2 incredible.'
  3 pattern = r'(in)'
  4  
  5 # call finditer()
  6 matches = re.finditer(pattern, text)
  7  
  8 # print matches
  9 for match in matches:
  10     print(match)
     
<re.Match object; span=(11, 13),
[7]:  
match='in'>
<re.Match object; span=(19, 21),
   
match='in'>
<re.Match object; span=(26, 28),
   
match='in'>
 
re.sub(pattern, repl, string, count=0,
flags=0) :
Return the string obtained by replacing
the leftmost non-overlapping occurrences
of pattern in string by the replacement
repl . If the pattern isn’t found, string is
returned unchanged. repl can be a string
or a function; if it is a string, any
backslash escapes in it are processed.
That is, \n is converted to a single newline
character, \r is converted to a carriage
return, and so forth.
 
[8]: 1 # sub()
  2 text = "Data Science with Python"
  3 replaced_text = re.sub("\s", "_", text)
  4 print(replaced_text)
     
[8]:   Data_Science_with_Python
 
In cell 8, we replaced the white space
characters, “\s” , with the underscores. As
you see in the cell output.
 
 
OceanofPDF.com
 Match Object
 
In regular expressions, a Match object is
an object which contains information
about the RE search and the its result. If
there is no match after the RE search,
then Python Interpreter will return None ,
instead of a Match object.
 
[9]: 1 # Match Object
  2  
  3 import re
  4  
  5 text = "Python 3.0 Deep Learning"
  6 match_obj = re.search(r"\d", text)
  7 print(match_obj)
     
<re.Match object; span=(7, 8),
[9]:  
match='3'>
 
In cell 9, we search for any digit, \d , in
the given text. And you see the resulting
Match object in the cell output.
 
The Match object is very useful, when
you need information about the RE search
and the search result. Here are some of its
attributes:
 Match.group([group1, ...]) :
Returns one or more subgroups of the
match. If there is a single argument, the
result is a single string; if there are
multiple arguments, the result is a tuple
with one item per argument.
 
[10]: 1 # Match.group()
  2 text = "Isaac Newton, physicist"
  3 match_obj = re.search(r"\bN\w+", text)
  4 print(match_obj.group())
     
[10]:   Newton
 
In cell 10, we search for a pattern which
is "\bN\w+" . This pattern means the capital
letter N at the beginning of a word ( \b )
and any word character ( \w ) after it. The
match is letter N in the word “Newton” .
We print the group attribute of the Match
object and you see the whole word of the
search result.
 
Match.span([group]) :
For a match m, return the 2-tuple
( m.start , m.end ). So, it gives the start and
end indices of the match.
  
[11]: 1 # span()
  2 text = "Isaac Newton, physicist"
  3 match_obj = re.search(r"\bN\w+", text)
  4 print(match_obj.span())
     
[11]:   (6, 12)
 
Match.string :
This attribute gives the string passed to
match() or search() .
 
[12]: 1 # Match.string
  2 text = "Isaac Newton, physicist"
  3 match_obj = re.search(r"\bN\w+", text)
  4 print(match_obj.string)
     
[12]:   Isaac Newton, physicist
 
This finalizes the chapter on Regular
Expression in Python. Next section is the
quiz on the topics in this chapter.
 
OceanofPDF.com
 
QUIZ – Regular Expressions
 
Now it’s time to solve the quiz for this
chapter. You can download the quiz file,
QUIZ_RegularExpressions.zip, from the
GitHub Repository of this book. You should
put the quiz file in the
RegularExpressions project we build in
this chapter.
 
Here are the questions for this chapter:
 
Q1:
Define a regular expression pattern in the following
way.
It should:
* start with 3 digits
* first dash (-) after 3 digits
* 4 alphabetical lower case letters (a to z)
* second dash (-)
* one or more word characters after the second
dash
 
Here are some examples:
* "247-abcd-5555"
* "416-dfuy-t"
 
[1]: 1 # Q 1:
  2  
  3 # import re module
  4 # ---- your solution here ---
  5  
  6 # define the pattern
  7 # ---- your solution here ---
  8  
  9 # check match for both examples
  10 # ---- your solution here ---
  11  
  12 # print the groups of match objects
  13 # ---- your solution here ---
     
[1]:   247-abcd-5555
    416-dfuy-t
 
Q2:
Here is the general format for emails:
(user_name)@(domain_name).
(top_level_domain)
Write a function named email_validator.
The function will take one parameter which is the
email.
It will use a regular expression to validate the given
email.
It will return the match object (either an object or
None) depending on the email being valid.
 
Hints:
Some valid email addresses:
* [email protected]
* [email protected]
* [email protected]
 
Invalid examples:
* abc_example.com
* xyz123@yahoo
 
[2]: 1 # Q 2:
  2  
  3 # import re module
  4 # ---- your solution here ---
  5  
  6 # define the function
  7 # ---- your solution here ---
  8  
  9 # call the function with valid examples
  10 email = "[email protected]"
  11 print(email_validator(email))
  12  
  13 email = "[email protected]"
  14 print(email_validator(email))
  15  
  16 email = "[email protected]"
  17 print(email_validator(email))
  18  
  19 # call the function with invalid examples
  20 email = "abc_example.com"
  21 print(email_validator(email))
  22  
  23 email = "xyz123@yahoo"
  24 print(email_validator(email))
     
<re.Match object; span=(0, 26),
[2]:  
match='[email protected]'>
<re.Match object; span=(0, 25),
   
match='[email protected]'>
    <re.Match object; span=(0, 22),
 match='[email protected]'>
    None
    None
 
Q3:
Define a function named find_all_with_regex.
The function will take a regular expression and a
text as parameters.
Ant will return the list of all possible matches of
this regex in the text.
The regex will be: all words starting with 'a' or 'c' (a
word must include at least two characters).
The text is:
"Lorem ipsum dolor sit amet, a consectetur
adipiscing elit. Sed id cid tempor risus.
Quisque imperdiet, neque c pulvinar sollicitudin,
augue nisl varius nibh, a suscipit cerat non lectus."
 
[3]: 1 # Q 3:
  2  
  3 # import re module
  4 # ---- your solution here ---
  5  
  6 # define the function
  7 # ---- your solution here ---
  8  
  9 # define the text
  10 # ---- your solution here ---
  11  
# define regex: find all the words
 
12 starting with 'a' or 'c'
  13 # ---- your solution here ---
  14  
  15 # call the function
  16 # ---- your solution here ---
  17  
  18 # print result
  19 # ---- your solution here ---
     
['amet', 'consectetur', 'adipiscing', 'cid',
[3]:  
'augue', 'cerat']
 
Q4:
Define a function named make_snake_case.
It will take a text as the parameter and will return
the snake case form of this text.
Snake case means; replacing all occurrences of
space, comma, or dot with an underscore.
Normal text: this is, a text.
Snake case: this_is__a_text_
 
Hints:
* sub()
 
[4]: 1 # Q 4:
  2  
  3 # import re module
  4 # ---- your solution here ---
  5  
  6 # define the function
  7 # ---- your solution here ---
  8  
  9 # call the function
  10 # ---- your solution here ---
  11  
  12 # print the result
  13 # ---- your solution here ---
     
[4]:   this_is__a_text_
 
Q5:
Define a function named
remove_non_alphanumeric.
It will take a text as the parameter and will return a
new modified text.
The function will remove everything except
alphanumeric characters from the text.
 
[5]: 1 # Q 5:
  2  
  3 # import re module
  4 # ---- your solution here ---
  5  
  6 # define the function
  7 # ---- your solution here ---
  8  
  9 # call the function
  10 # ---- your solution here ---
  11  
  12 # print the result
  13 # ---- your solution here ---
     
[5]:   HandsOnPythonAdvanced333
 
 

OceanofPDF.com
 
SOLUTIONS – Regular Expressions
 
Here are the solutions for the quiz for
this chapter.
 
S1:
 
[1]: 1 # S 1:
  2  
  3 # import re module
  4 import re
  5  
  6 # define the pattern
pattern = re.compile(r"\d{3}-[a-z]{4}-
 
7 \w+")
  8  
  9 # check match for both examples
  10 m = pattern.search("247-abcd-5555")
  11 m_2 = pattern.search("416-dfuy-t")
  12  
  13 # print the groups of match objects
  14 print(m.group())
  15 print(m_2.group())
     
[1]:   247-abcd-5555
    416-dfuy-t
 
S2:
 
[2]: 1 # S 2:
  2  
  3 # import re module
  4 import re
  5  
  6 # define the function
  7 def email_validator(email):
  8     # define the regular expression
    regex = re.compile(r'[A-Za-z0-
 
9 9_\.-]+@[A-Za-z0-9-]+(\.[A-Z|a-z]{2,})+')
  10     return regex.search(email)
  11  
  12 # call the function with valid examples
  13 email = "[email protected]"
  14 print(email_validator(email))
  15  
  16 email = "[email protected]"
  17 print(email_validator(email))
  18  
  19 email = "[email protected]"
  20 print(email_validator(email))
  21  
  22 # call the function with invalid examples
  23 email = "abc_example.com"
  24 print(email_validator(email))
  25  
  26 email = "xyz123@yahoo"
  27 print(email_validator(email))
     
<re.Match object; span=(0, 26),
[2]:  
match='[email protected]'>
<re.Match object; span=(0, 25),
   
match='[email protected]'>
    <re.Match object; span=(0, 22),
 match='[email protected]'>
    None
    None
 
S3:
 
[3]: 1 # S 3:
  2  
  3 # import re module
  4 import re
  5  
  6 # define the function
  7 def find_all_with_regex(regex, text):
  8     return re.findall(regex, text)
  9  
  10 # define the text
text = """Lorem ipsum dolor sit amet, a
  consectetur adipiscing elit. Sed id cid
11 tempor risus.
Quisque imperdiet, neque c pulvinar
  sollicitudin, augue nisl varius nibh, a
12 suscipit cerat non lectus."""
  13  
# define regex: find all the words
 
14 starting with 'a' or 'c'
  15 regex = r"\b[ac]\w+"
  16  
  17 # call the function
match_list = find_all_with_regex(regex,
 
18 text)
  19  
  20 # print result
  21 print(match_list)
     
 ['amet', 'consectetur', 'adipiscing', 'cid',
[3]:  
'augue', 'cerat']
 
S4:
 
[4]: 1 # S 4:
  2  
  3 # import re module
  4 import re
  5  
  6 # define the function
  7 def make_snake_case(text):
  8     return re.sub("[ ,.]", "_", text)
  9  
  10 # call the function
  11 text = 'this is, a text.'
  12 snake_case = make_snake_case(text)
  13  
  14 # print the result
  15 print(snake_case)
     
[4]:   this_is__a_text_
 
S5:
 
[5]: 1 # S 5:
  2  
  3 # import re module
  4 import re
  5  
  6 # define the function
  7 def remove_non_alphanumeric(text):
  8     # define the pattern
  9     pattern = re.compile('[\W_]+')
  10     # call and return the sub() function
  11     return pattern.sub('', text)
  12  
  13 # call the function
s = '_*!/Hands- On Python -
 
14 Advanced\+++ 333...'
  15 s_new = remove_non_alphanumeric(s)
  16  
  17 # print the result
  18 print(s_new)
     
[5]:   HandsOnPythonAdvanced333
 
 
OceanofPDF.com
 13. Database Operations
       

 
Database Operations are one of the
fundamental parts in application
development. In this chapter you will learn
how to set up a local database server and
create a database instance with tables in
it. Then you will learn the basic SQL
commands and see how they work with
Python.
You can find the PyCharm project for
this chapter in the GitHub Repository of
this book.
 
Chapter Outline:
SQL
Install MySQL Server
Install MySQL Connector
Connect to MySQL Server
Create a Database
Creating Tables
Inserting Records
Querying Records
Updating Records
Deleting Records
 QUIZ – Database Operations
SOLUTIONS – Database Operations
 
 
OceanofPDF.com
 SQL
 
SQL (Structured Query Language) is a
standard language used in programming
and designed for managing data held in a
relational database management system
(RDBMS).
SQL is designed for a specific purpose:
to query and manage the data contained
in a relational database. It is a set-based,
declarative programming language, not an
imperative programming language like C
or BASIC. However, extensions to
Standard SQL add procedural
programming language functionality, such
as control-of-flow constructs.
Python has powerful features for
database operations. It supports various
databases like MySQL, Oracle, Microsoft
SQL Server, Sybase, PostgreSQL, IBM DB2,
etc. It also supports Data Definition
Language (DDL), Data Manipulation
Language (DML) and Data Query
Statements.
 This chapter is not about SQL language
syntax, so we assume that you are
familiar with SQL basic commands like
SELECT , INSERT , UPDATE , DELETE etc...
 
 
OceanofPDF.com
 Install MySQL Server
 
MySQL is a free and open-source
relational database management system
(RDBMS). A relational database, organizes
data into one or more data tables in which
data types may be related to each other;
these relations help structure the data.
MySQL implements a relational database in
a computer's storage system, manages
users, allows for network access and
facilitates testing database integrity and
creation of backups.
At the time of this writing, MySQL is used
by many of the major tech companies, like
IBM, Google, Facebook, Sony, LinkedIn,
Uber, Netflix, Twitter, and more.
In this chapter, we will use MySQL
Community Server as it is free and widely
used in the industry. You can skip this
section if you already have MySQL on your
development environment.
 
Here are the installation guides for major
operating systems:
 Windows:
https://dev.mysql.com/doc/refman/8.0/
en/windows-installation.html
macOS:
https://dev.mysql.com/doc/refman/8.0/
en/macos-installation.html
Linux:
https://dev.mysql.com/doc/refman/8.0/
en/linux-installation.html
 
For Microsoft Windows, the simplest and
recommended method is to download
MySQL Installer (for Windows) and let it
install and configure a specific version of
MySQL Server.
I am using macOS, so here are the steps
to install MySQL on macOS using the
package installer:
1. Download the disk image (.dmg) file
(the community version is available
here) that contains the MySQL
package installer. Double-click the file
to mount the disk image and see its
contents.
Double-click the MySQL installer package
from the disk. It is named according to
the version of MySQL you have
downloaded. For example, for MySQL
server 8.0.28 it might be named mysql-
8.0.28-macos-10.13-x86_64.pkg.
2. The initial wizard introduction screen
references the MySQL server version
to install. Click Continue to begin the
installation.
The MySQL community edition shows a
copy of the relevant GNU General Public
License. Click Continue and then Agree to
continue.
3. From the Installation Type page, you
can either click Install to execute the
installation wizard using all defaults,
click Customize to alter which
components to install (MySQL server,
MySQL Test, Preference Pane, Launchd
Support -- all but MySQL Test are
enabled by default).
Note: Although the Change Install
Location option is visible, the installation
location cannot be changed.
 
 Figure 13-1: MySQL Package Installer Wizard: Installation
Type
 
 Figure 13-2: MySQL Package Installer Wizard: Customize
 
4. Click Install to install MySQL Server.
The installation process ends here if
upgrading a current MySQL Server
installation, otherwise follow the
wizard's additional configuration steps
for your new MySQL Server
installation.
5. After a successful new MySQL Server
installation, complete the
configuration steps by choosing the
default encryption type for passwords,
 define the root password, and also
enable (or disable) MySQL server at
startup.
6. The default MySQL 8.0 password
mechanism is caching_sha2_password
(Strong), and this step allows you to
change it to mysql_native_password
(Legacy).
 

Figure 13-3: MySQL Package Installer Wizard: Choose a

Password Encryption Type
 
Choosing the legacy password
mechanism alters the generated launchd
file to set --
default_authentication_plugin=mysql_native_
password under ProgramArguments .
Choosing strong password encryption
does not set --
default_authentication_plugin because the
default MySQL Server value is used,
which is caching_sha2_password .
7. Define a password for the root user,
and also toggle whether MySQL Server
should start after the configuration
step is complete.
 
Figure 13-4: MySQL Package Installer Wizard: Define Root
Password
8. Summary is the final step and
references a successful and complete
MySQL Server installation. Close the
wizard.
 
 Figure 13-5: MySQL Package Installer Wizard: Summary
 
MySQL server is now installed on your
machine. If you chose to not start MySQL,
then use either launchctl from the command
line or start MySQL by clicking Start MySQL
Server button using the MySQL Preference
Pane. It is located under the System
Preferences. See the images below:
 
 Figure 13-6: MySQL in System Preferences
 

Figure 13-7: MySQL in Open Spotlight

 

Figure 13-8: MySQL Preference Pane

OceanofPDF.com
 When installing using the package
installer, the files are installed into a
directory within /usr/local matching the
name of the installation version and
platform. For example, the installer file
mysql-8.0.28-macos10.15-x86_64.dmg installs
MySQL into /usr/local/mysql-8.0.28-
macos10.15-x86_64/ with a symlink to
/usr/local/mysql .
 
 
OceanofPDF.com
 Install MySQL Connector
 
MySQL Connector is a self-contained
Python driver for communicating with
MySQL servers, and is used it to develop
database applications.
MySQL Connector enables Python
programs to access MySQL databases, using
an API that is compliant with the Python
Database API Specification v2.0 (PEP 249).
MySQL Connector/Python includes
support for:
Almost all features provided by MySQL
Server up to and including MySQL
Server version 8.0.
MySQL Connector 8.0 also supports X
DevAPI.
Converting parameter values back
and forth between Python and MySQL
data types, for example Python
datetime and MySQL DATETIME . You can
turn automatic conversion on for
convenience, or off for optimal
performance.
All MySQL extensions to standard SQL
syntax.
 Protocol compression, which enables
compressing the data stream between
the client and server.
Connections using TCP/IP sockets and
on Unix using Unix sockets.
Secure TCP/IP connections using SSL.
Self-contained driver. MySQL
Connector does not require the MySQL
client library or any Python modules
outside the standard library.
In this chapter, we will use MySQL
Connector as the driver to access the
MySQL database. Since we use, PyCharm
IDE, we will install MySQL Connector using
PyCharm package manager.
In PyCharm navigate to Preferences,
then under the Project section you will
Python Interpreter. This is where we
install Python packages. Click on the plus
(+) icon to search for the package. Official
MySQL Connector for Python is: mysql-
connector-python . Search for this name and
install the package which is from “Oracle
and/or its affiliates”. See the image below:
 
 Figure 13-9: Installing mysql-connector-python package in
PyCharm
 
To test MySQL Connector, simply try to
import it in the project as: import
mysql.connector . If you do not get any errors,
then you the connector is installed
successfully.
 
 
OceanofPDF.com
 Connect to MySQL Server
 
Now that we installed MySQL
Connector/Python we are ready for
database operations. Let’s start by
creating our first connection to MySQL
Server. Make sure the server is up and
running on your local machine. We will
start by creating two Python files. Here are
the files; Credentials.py and
MySQLConnector.py .
 
Credentials.py:
 
[1]: 1 # Credentials for local MySQL Server
  2  
  3 mysql_host = "localhost"
  4 mysql_user = "root"
mysql_password = "
 
5 <your_root_password>"
 
In cell 1, we have the credentials for
connection. We will import them whenever
needed in this chapter.
 
MySQLConnector.py:
 
[2]: 1 # Create a MySQL Connection
  2  
  3 # import MySQL Connector
  4 import mysql.connector
  5  
# import credentials from Credentials.py
 
6 file
from Credentials import mysql_host,
 
7 mysql_user, mysql_password
  8  
  9 # define a connection function
  10 def connect_to_mysql(host=mysql_host,
  11                      user=mysql_user,
                    
 
12 password=mysql_password):
  13     connection = None
  14     try:
  15         # start the connection
        connection =
 
16 mysql.connector.connect(
  17           host=host,
  18           user=user,
  19           password=password
  20         )
        print("Successfully connected to
 
21 MySQL Server.")
  22     except mysql.connector.Error as e:
  23         print(f"An Error occurred: '{e}'")
  24     # else:
  25     #     connection.close()
  26  
  27     return connection
  28  
  29 # call the function and get connection
  30 con = connect_to_mysql()
     
[2]:   Successfully connected to MySQL Server.
 
In cell 2, we establish our first
connection to the MySQL server on our
local machine. We import the
mysql.connector class first.
In line 7, we import the credentials
( mysql_host , mysql_user , mysql_password )
from Credentials.py file. This file will keep
the host, user name and password for this
chapter.
In line 10, we define the
connect_to_mysql() function. The
parameters are host, user and password .
The default values are our saved
credentials which we import here. In the
try block, we instantiate a connection by
calling the mysql.connector.connect()
function. We pass the host , user and
password arguments to it. In the except
block, we simply catch the
mysql.connector.Error and print it. In the
else block, we normally close the
connection by calling the close() method.
But we will need an open connection for
the next sections, so we comment this
out. Finally, in line 25, we return an active
connection object.
You should see the success message in
the output. Which shows that, your
connection is successful and you can
move on to the next section.
 
 
OceanofPDF.com
 Create a Database
 
In this section we will create a database
(db) for keeping the data for superheroes.
The database name will be HEROES and it
will have three tables in it; Superhero , Movie ,
and HeroMovie . Let’s start by creating the
db first:
 
[3]: 1 # Create a Database
  2  
  3 # import connector
  4 import mysql.connector
  5  
  6 # import custom connection function
from MySQLConnector import
 
7 connect_to_mysql
  8  
  9 # function for create the db
  10 def create_database(db_name):
  11     try:
  12         # connect to server
  13         connection = connect_to_mysql()
  14         if connection is not None:
  15             # set the cursor
  16             cursor = connection.cursor()
  17             # create the database
            cursor.execute(f"CREATE
 
18 DATABASE {db_name}")
  19     except mysql.connector.Error as e:
  20         print(f"An Error occurred in DB
Creation:\n'{e}'")
  21     else:
  22         if connection is not None:
            # close the cursor and the
 
23 connection
  24             cursor.close()
  25             connection.close()
            print("Database created
 
26 successfully.")
  27  
  28 # call the function and create the db
  29 create_database("heroes")
     
[3]:   Successfully connected to MySQL Server.
    Database created successfully.
 
In cell 3, we create a database named
heroes . We define a function for this
operation, create_database .
Inside the try block, we call the
connect_to_mysql() function which we
defined in the MySQLConnector.py file. This
function returns a connection object if it
can connects successfully. We check this
object in line 14, to see if it is not None.
In line 16, we set a cursor object by
calling the cursor() method on our
connection object. We execute DDL (Data
Definition Language) statements using a
structure known as the cursor.
In line 18, we call the execute() method
on the cursor object, by passing the SQL
statement to create a new database. Here
is the syntax: cursor.execute(f"CREATE
DATABASE {db_name}") .
In the else block, we again check for the
connection object. If it is not None, then we
close the cursor and the connection in
respective order.
When we run the code in cell 3, we will
have a new database instance on our local
MySQL server. Let’s check if it really exists
or not:
 
[4]: 1 # check database
  2 def print_databases():
  3     try:
  4         # connect to server
  5         connection = connect_to_mysql()
  6         if connection is not None:
  7             # set the cursor
  8             cursor = connection.cursor()
  9             # execute query to show dbs
            cursor.execute("SHOW
 
10 DATABASES")
  11             # print dbs
  12             for db in cursor:
  13                 print(db)
  14     except mysql.connector.Error as e:
        print(f"An Error occurred in DB
 
15 Creation:\n'{e}'")
  16     else:
  17         if connection is not None:
            # close the cursor and the
 
18 connection
  19             cursor.close()
  20             connection.close()
  21  
  22 print_databases()
     
[4]:   Successfully connected to MySQL Server.
    ('heroes',)
    ('information_schema',)
    ('mysql',)
    ('performance_schema',)
    ('sys',)
 
In cell 4, we define a function to print
database names on our MySQL server. The
SQL command for this is "SHOW DATABASES" .
Then we print the name of each database
by using a for loop on the cursor object.
As you see, we have a new database
called 'heroes' , which means we are ready
to create the some tables in it.
 
 
OceanofPDF.com
 Creating Tables
 
Now that we have an active database,
we can create our tables. We will create
three tables in our heroes db. The tables
will be:
Superhero : We will store superhero
details like; hero name, real name,
birth date and city.
Movie : We will store movie details
like; movie_name and release_year.
HeroMovie : This table will store the
relation between a superhero and a
movie. hero_id and movie_id will be
the foreign keys to the respective
tables.
 
Here are the tables which we will create
in this section:
 
Superhero   Movie   HeroMovie
hero_id   movie_id   hero_id
hero_name   movie_name   movie_id
real_name   release_year    
birth_date        
city        
Table 13-1: The tables in HEROES database
  
[5]: 1 # Creating Tables
  2  
  3 # import mysql classes
  4 import mysql.connector
from mysql.connector import Error,
 
5 errorcode
  6  
  7 # import custom connection function
from MySQLConnector import
 
8 connect_to_mysql
  9  
  10 # set db name
  11 db_name = 'heroes'
  12  
  13 # define a dictionary for tables
  14 tables = {}
  15  
  16 # define superhero table
  17 tables['superhero'] = (
  18     "CREATE TABLE `superhero` ("
    "  `hero_id` int(11) NOT NULL
 
19 AUTO_INCREMENT,"
    "  `hero_name` varchar(40) NOT
 
20 NULL,"
    "  `real_name` varchar(40) NOT
 
21 NULL,"
  22     "  `birth_date` date NOT NULL,"
  23     "  `city` varchar(20) NOT NULL,"
  24     "  PRIMARY KEY (`hero_id`)"
  25     ") ENGINE=InnoDB")
  26  
  27 # define movie table
  28 tables['movie'] = (
  29     "CREATE TABLE `movie` ("
    "  `movie_id` int(11) NOT NULL
 
30 AUTO_INCREMENT,"
    "  `movie_name` varchar(1000) NOT
 
31 NULL,"
  32     "  `release_year` int(4) NOT NULL,"
  33     "  PRIMARY KEY (`movie_id`)"
  34     ") ENGINE=InnoDB")
  35  
  36 # define heromovie table
  37 tables['heromovie'] = (
  38     "CREATE TABLE `heromovie` ("
  39     "  `hero_id` int(11) NOT NULL,"
  40     "  `movie_id` int(11) NOT NULL,"
  41     "  `release_year` int(4) NOT NULL,"
    "  PRIMARY KEY
 
42 (`hero_id`,`movie_id`),"
    "  CONSTRAINT `hero_id_fk` FOREIGN
 
43 KEY (`hero_id`) "
    "     REFERENCES `superhero`
 
44 (`hero_id`) ON DELETE CASCADE,"
    "  CONSTRAINT `movie_id_fk`
 
45 FOREIGN KEY (`movie_id`) "
    "     REFERENCES `movie`
 
46 (`movie_id`) ON DELETE CASCADE"
  47     ") ENGINE=InnoDB")
 
In cell 5, define a dictionary to keep our
tables data. The keys are table names and
the values are SQL commands which we
need to execute to create the tables. You
see MySQL syntax for creating tables,
table fields, primary keys and foreign keys
in the examples.
The final lines, which are
ENGINE=InnoDB" , are necessary to set the
storage engine for MySQL server. InnoDB is
a storage engine for the database
management system in MySQL.
Now that we have the syntax to create
our tables let’s define functions for this
operation. We will define two separate
functions. The first one will set the
connection to the MySQL server and the
second one will execute the SQL
commands to create the tables in our
database.
 
[6]: 1 # connect and start creating tables
  2 def start_tables_creation():
  3     try:
  4         # connect to server
  5         connection = connect_to_mysql()
  6         if connection is not None:
  7             # set the cursor
  8             cursor = connection.cursor()
  9             # set use db for cursor
            cursor.execute(f"USE
 
10 {db_name}")
  11             # create tables
            create_tables_in_db(connection,
 
12 cursor)
  13     except Error as e:
        print(f"An Error occurred in Table
 
14 Creation:\n'{e}'")
 
In cell 6,we have the
start_tables_creation() function. It is
responsible for starting an active
connection by calling the
connect_to_mysql() function and setting the
cursor. In line 10, we execute a special
syntax on the cursor to declare the
database which we want to work on. Here
is the syntax: USE heroes . This code
selects the heroes db for the next tasks
which we will execute using this cursor.
In line 12, we call another function for
actual creation operation. The function
name is create_tables_in_db() and we pass
the connection and the cursor objects to it.
 
[7]: 1 # define the create function
def create_tables_in_db(connection,
 
2 cursor):
  3     # create tables one by one
  4     for table in tables:
  5         table_creation_code = tables[table]
  6         try:
            print(f"Creation started for:
 
7 {table}")
           
 
8 cursor.execute(table_creation_code)
  9         except Error as e:
            if e.errno ==
 
10 errorcode.ER_TABLE_EXISTS_ERROR:
                print(f"Table {table} already
 
11 exists.")
  12             else:
  13                 print(e.msg)
  14         else:
            print(f"Table {table} created
 
15 successfully.")
  16  
  17     # close active connections
  18     cursor.close()
  19     connection.close()
  20  
# call the function to start creating
 
21 tables
  22 start_tables_creation()
     
[7]:   Successfully connected to MySQL Server.
    Creation started for: superhero
    Table superhero created successfully.
    Creation started for: movie
    Table movie created successfully.
    Creation started for: heromovie
    Table heromovie created successfully.
 
 In cell 7, we define the
create_tables_in_db() function. This function
uses a for loop to iterate over the tables
dictionary. In line 8, it executes the SQL
command to create the current table in
the loop. We also check the error codes to
be able handle possible exceptions.
Finally, we close the active connection in
line 18.
In line 22, we call start_tables_creation()
function to start creation. And in the
output, you can see that all the tables are
created successfully.
Let’s see if they really exist in the
database:
 
[8]: 1 # Display Tables
  2 # check tables in database
  3 def print_tables():
  4     try:
  5         # connect to server
  6         connection = connect_to_mysql()
  7         if connection is not None:
  8             # set the cursor
  9             cursor = connection.cursor()
  10             # set use db for cursor
  11             cursor.execute(f"USE heroes")
  12             # execute query to show dbs
  13             cursor.execute("SHOW TABLES")
  14             # print dbs
  15             for table in cursor:
  16                 print(table)
  17     except mysql.connector.Error as e:
        print(f"An Error occurred in
 
18 Displaying Tables:\n'{e}'")
  19     else:
  20         if connection is not None:
            # close the cursor and the
 
21 connection
  22             cursor.close()
  23             connection.close()
  24  
  25 # call the function to see tables
  26 print_tables()
     
[8]:   Successfully connected to MySQL Server.
    ('heromovie',)
    ('movie',)
    ('superhero',)
 
 
OceanofPDF.com
 Inserting Records
 
In this section we will insert some
records to our database tables. First, let’s
see the data which we will insert. Below
are the tables displaying the records:
 
Superhero
hero_name real_name birth_date city
Wonder Diana
22.03.1976 Themyscira
Woman Prince
Bruce
Batman 17.04.1915 Gotham
Wayne
Superman Clark Kent 18.04.1977 Metropolis
Peter New York
Spiderman 1.08.1962
Parker City
Table 13-2: Superhero data
 
Movie
movie_name release_year
Spider-Man: No Way Home 2021
The Dark Knight 2008
Wonder Woman 2017
Man of Steel 2013
Table 13-3: Movie data
 
We will create two lists of dictionaries
for these records. Then we will insert them
to MySQL server. We also have two CSV
files in the project; superhero.csv and
movie.csv . These files are under the “data”
folder and they contain the data in Tables
13-2 and 13-3. Let’s first start by reading
these files:
 
[9]: 1 # read csv files
  2 import csv
  3  
  4 superheroes = []
  5 movies = []
  6  
  7 def get_data_as_dict(path):
  8     file = open(path)
  9     reader = csv.DictReader(file)
  10     list_of_data = []
  11     for row in reader:
  12         list_of_data.append(row)
  13     file.close()
  14     return list_of_data
  15  
  16 # fill the lists
superheroes =
 
17 get_data_as_dict('data/superhero.csv')
movies =
 
18 get_data_as_dict('data/movie.csv')
  19  
  20 for hero in superheroes:
  21     print(hero)
  22  
  23 for movie in movies:
  24     print(movie)
     
{'hero_name': 'Wonder Woman',
[9]:   'real_name': 'Diana Prince', 'birth_date':
'22.03.1976', 'city': 'Themyscira'}
{'hero_name': 'Batman', 'real_name':
    'Bruce Wayne', 'birth_date': '17.04.1915',
'city': 'Gotham'}
{'hero_name': 'Superman', 'real_name':
    'Clark Kent', 'birth_date': '18.04.1977',
'city': 'Metropolis'}
{'hero_name': 'Spiderman', 'real_name':
    'Peter Parker', 'birth_date': '1.08.1962',
'city': 'New York City'}
{'movie_name': 'Spider-Man: No Way
   
Home', 'release_year': '2021'}
{'movie_name': 'The Dark Knight',
   
'release_year': '2008'}
{'movie_name': 'Wonder Woman',
   
'release_year': ‘2017’}
{'movie_name': 'Man of Steel',
   
'release_year': '2013'}
 
In cell 9, we read the data in files,
'data/superhero.csv' and 'data/movie.csv' .
And we fill a list for each file which are
superheroes and movies . Now we will insert
these lists to the related tables in our
database.
 
[10]: 1 # import mysql classes
  2 from mysql.connector import Error
  3  
  4 # import custom connection function
from MySQLConnector import
 
5 connect_to_mysql
  6  
  7 # set db name
  8 db_name = 'heroes'
  9  
  10 # connect and set cursor
  11 def start_record_insert():
  12     try:
  13         # connect to server
  14         connection = connect_to_mysql()
  15         if connection is not None:
  16             # set the cursor
  17             cursor = connection.cursor()
  18             # set use db for cursor
            cursor.execute(f"USE
 
19 {db_name}")
  20             # insert superheroes
  21             insert_superheroes(cursor)
  22             # insert movies
  23             insert_movies(cursor)
  24     except Error as e:
        print(f"An Error occurred in
 
25 Record Creation:\n'{e}'")
  26     else:
  27         # commit to the database
  28         connection.commit()
  29         cursor.close()
  30         connection.close()
 
 In cell 10, we have the definition of the
start_record_insert() function. It starts a
connection to MySQL server and sets a
cursor on our heroes database. Then in line
21, it calls insert_superheroes() function
and in line 23, it calls the insert_movies()
by passing the cursor as the argument.
These functions will be responsible for
formatting the columns and inserting
records.
In line 28, inside the else block, we
commit our changes to the database as:
connection.commit() . When you made a
modification in the database, you need to
call the commit() method on the
connection . This code will make sure that
your data is committed to db. Otherwise,
even if you don’t get any errors, the data
will not be saved to the database.
 
[11]: 1 # function for inserting superheroes
  2 def insert_superheroes(cursor):
  3     # convert str to date for birth_date
  4     str_to_date("birth_date")
  5  
    superhero_command = ("INSERT INTO
 
6 superhero "
  7                    "(hero_name, real_name,
birth_date, city) "
                   "VALUES (%(hero_name)s,
  %(real_name)s, %(birth_date)s, %
8 (city)s)")
  9     for superhero in superheroes:
       
  cursor.execute(superhero_command,
10 superhero)
  11     print("Superheroes inserted.")
  12  
  13 # function for converting str to date
  14 def str_to_date(column_name):
  15     from datetime import datetime
  16     for hero in superheroes:
  17         str_date = hero[column_name]
  18         date_format = "%d.%m.%Y"
        hero[column_name] =
  datetime.strptime(str_date,
19 date_format).date()
 
The type for the birth_date column is
date . So, we need to convert it for the
records in the superheroes list. That’s why
we have the str_to_date() function in cell
11, line 14.
Inside the insert_superheroes() function,
we first define the SQL command for
inserting. And we name it as
superhero_command . In line 9, we set a for
loop to execute each line one by one with
the cursor.
 
[12]: 1 # function for inserting movies
  2 def insert_movies(cursor):
    movie_command = ("INSERT INTO
 
3 movie "
                   "(movie_name,
 
4 release_year) "
                   "VALUES (%(movie_name)s,
 
5 %(release_year)s)")
  6     for movie in movies:
        cursor.execute(movie_command,
 
7 movie)
  8     print("Movies inserted.")
 
In cell 12, you see the definition of the
insert_movies() function. It first creates an
SQL command for insert and then loops
over the items of the movies list. For each
movie in the list, the cursor executes this
insert command.
Let’s run the start_record_insert()
function and do the actual insert
operation:
 
[13]: 1 # call the start function
  2 start_record_insert()
     
[13]:   Successfully connected to MySQL Server.
    Superheroes inserted.
    Movies inserted.
 
As you see in the output of cell 13,
records are inserted to the tables in our
database.
 
 
OceanofPDF.com
 Querying Records
 
In this section, we will read the records
which we inserted in the previous section.
We will start by creating an SQL query
statement. Let’s see how we set queries:
 
[14]: 1 # Querying Records
  2  
  3 # import mysql classes
  4 from mysql.connector import Error
  5  
  6 # import custom connection function
from MySQLConnector import
 
7 connect_to_mysql
  8  
  9 # select query to get superheroes
  10 query = ("SELECT * FROM superhero "
  11          "WHERE hero_name LIKE %s")
  12  
  13 name_start_with = 'B%'
 
In cell 14, you see a simple SQL query
statement with SELECT and WHERE clauses.
The query will filter the results in which
the hero_name starts with letter ‘B’ . In SQL
syntax, 'B%' means “start with b and the
rest does not matter ”. In the query variable,
we have %s placeholder for this
parameter.
 
[15]: 1 # connect and set cursor
  2 def query_records():
  3     try:
  4         # connect to server
  5         connection = connect_to_mysql()
  6         if connection is not None:
  7             # set the cursor
  8             cursor = connection.cursor()
            # execute query with
 
9 parameters
            cursor.execute(query,
 
10 (name_start_with,))
  11             # get the result
  12             result = cursor.fetchall()
  13             # print the results
  14             for row in result:
  15                 print(row)
  16     except Error as e:
        print(f"An Error occurred in
 
17 Querying Records:\n'{e}'")
  18     else:
  19         cursor.close()
  20         connection.close()
  21  
  22 # call the method to query
  23 query_records()
     
Successfully connected to MySQL
[15]:  
Server.
    (2, 'Batman', 'Bruce Wayne',
datetime.date(1915, 4, 17), 'Gotham')
 
In cell 15, we define the query_records()
function. In line 10, we execute our query
with parameters. The second argument to
the cursor.execute() function is a tuple
containing the parameters.
In line 12, we get the query results by
calling the cursor.fetchall() method. Then
we print the results. As you see in the
output, there is only one record which
starts with the letter ‘B’ .
We update the connect_to_mysql()
function, to connect to our heroes
database at the beginning. Here is the
updated form of it:
 
[16]: 1 # define a connection function
  2 def connect_to_mysql(host=mysql_host,
  3                      user=mysql_user,
                    
 
4 password=mysql_password,
  5                      database='heroes'):
  6     connection = None
  7     try:
  8         # start the connection
        connection =
 
9 mysql.connector.connect(
  10           host=host,
  11           user=user,
  12           password=password,
  13           database=database
  14         )
        print("Successfully connected to
 
15 MySQL Server.")
  16     except mysql.connector.Error as e:
  17         print(f"An Error occurred: '{e}'")
  18     # else:
  19     #     connection.close()
  20  
  21     return connection
 
In cell 16, you see the final form of the
connect_to_mysql() function. Since we are
working on the heroes database, it is
better to set it at the beginning of the
connection. So, we pass the database
parameter to the connect() function which
is set to ‘heroes’ by default.
 
 
OceanofPDF.com
 Updating Records
 
In this section we will update some records
in our database. Let’s say we want to change
the title and release year of one of the
movies. To be able to update a specific record
in our database, we need to know the unique
id of that record. And to get the id’s we will
query all the movies on our movie table.
 
[17]: 1 # Updating Records
  2  
  3 # import mysql classes
  4 from mysql.connector import Error
  5  
  6 # import custom connection function
from MySQLConnector import
 
7 connect_to_mysql
  8  
  9 #--- GET ALL MOVIES ---#
  10 # select query to get all movies
movie_select_query = ("SELECT * FROM
 
11 movie")
  12  
  13 # connect and set cursor
  14 def get_movies():
  15     try:
  16         # connect to server
  17         connection = connect_to_mysql()
  18         if connection is not None:
  19             # set the cursor
  20             cursor = connection.cursor()
  21             # execute query
  22             cursor.execute(movie_select_query)
  23             # get the result
  24             result = cursor.fetchall()
  25             # print the results
            for (movie_id, movie_name,
 
26 release_year) in result:
                print(f"
 
27 {movie_id}\t{movie_name}\t{release_year}")
  28     except Error as e:
        print(f"An Error occurred in Querying
 
29 Records:\n'{e}'")
  30     else:
  31         cursor.close()
  32         connection.close()
  33  
  34 # call the method to query
  35 get_movies()
     
[17]:   Successfully connected to MySQL Server.
    1 Spider-Man: No Way Home 2021
    2 The Dark Knight 2008
    3 Wonder Woman 2017
    4 Man of Steel 2013
 
In cell 17, we get all of the movies in the
movie table. In line 26, we can directly
address the names of the fields in the result
set in a tuple as: for (movie_id, movie_name,
release_year) in result .
In the output we see that the id of the
Wonder Woman movie is 3 . We want to update
this record in the database. We will change its
name to ‘Wonder Woman 1984’ and release
year to 2000 . Here is the code for update:
 
[18]: 1 #--- UPDATE A RECORD ---#
  2 # set the update query
movie_update_query = ("UPDATE movie SET
 
3 movie_name=%s, "
                      "release_year=2000 WHERE
 
4 movie_id=%s")
  5  
  6 # connect and update
  7 def update_movie(id, name):
  8     try:
  9         # connect to server
  10         connection = connect_to_mysql()
  11         if connection is not None:
  12             # set the cursor
  13             cursor = connection.cursor()
  14             # execute query
            cursor.execute(movie_update_query,
 
15 (name, id))
  16             # commit changes
  17             connection.commit()
  18     except Error as e:
        print(f"An Error occurred in Updating
 
19 Records:\n'{e}'")
  20     else:
  21         print(f"{name} updated successfully.")
  22         cursor.close()
  23         connection.close()
  24  
  25 # update the movie with id=3
  26 update_movie(3, 'Wonder Woman 1984')
     
[18]:   Successfully connected to MySQL Server.
    Wonder Woman 1984 updated successfully.
 
In cell 18, we update the movie with id = 3 .
We first define the update query in line 3.
Then in the update_movie() function we
execute this update query with parameters.
Finally, in line 17, we commit our changes to
the database by calling the
connection.commit() method.
Now if you get all the movies one more time
by calling the get_movies() function again, you
will see that Wonder Woman movie is updated
now:
 
[19]: 1 # get all movies again
  2 get_movies()
     
[19]:   Successfully connected to MySQL Server.
    1 Spider-Man: No Way Home 2021
    2 The Dark Knight 2008
    3 Wonder Woman 1984 2000
    4 Man of Steel 2013
 
 
OceanofPDF.com
 Deleting Records
 
Deleting records in a table in MySQL
server is very easy. All you need is to
define the unique id of the row which you
want to delete. Let’s say we want to
delete the movie with id = 4 , which is
“Man of Steel” .
 
[20]: 1 # Deleting Records
  2 # import mysql classes
  3 from mysql.connector import Error
  4 # import custom connection function
from MySQLConnector import
 
5 connect_to_mysql
  6 # import UpdatingRecords.py file
  7 import UpdatingRecords
  8  
  9 # set the delete query
movie_update_query = ("DELETE FROM
 
10 movie WHERE movie_id=%s")
  11  
  12 # connect and update
  13 def delete_movie(id):
  14     try:
  15         # connect to server
  16         connection = connect_to_mysql()
  17         if connection is not None:
  18             # set the cursor
  19             cursor = connection.cursor()
  20             # execute query
           
  cursor.execute(movie_update_query,
21 (id,))
  22             # commit changes
  23             connection.commit()
  24     except Error as e:
        print(f"An Error occurred in
 
25 Updating Records:\n'{e}'")
  26     else:
        print(f"Movie with id={id} deleted
 
27 successfully.")
  28         cursor.close()
  29         connection.close()
  30  
  31 # delete the movie with id=4
  32 delete_movie(4)
     
Successfully connected to MySQL
[20]:  
Server.
    Movie with id=4 deleted successfully.
 
In cell 20, we delete the movie record
with id = 4 . As you see the process is very
similar to update operation. If we call
get_movies() function from the
UpdatingRecords.py file (the file for the
previous section) again, you will see that
“Man of Steel” no longer exists.
 
[21]: 1 # get all movies again
  2 UpdatingRecords.get_movies()
     
[21]:   Successfully connected to MySQL Server.
    1 Spider-Man: No Way Home 2021
    2 The Dark Knight 2008
    3 Wonder Woman 1984 2000
 
OceanofPDF.com
 
QUIZ – Database Operations
 
Now it’s time to solve the quiz for this
chapter. You can download the quiz file,
QUIZ_DatabaseOperations.zip, from
the GitHub Repository of this book. You
should put the quiz file in the
DatabaseOperations project we build in
this chapter.
 
Here are the questions for this chapter:
 
Q1:
Define a function to connect to a MySQL Server and
print the connection result.
The function name will be mysql_connect.
It can be the local MySQL Server if you install it on
your machine.
If you get an exception, print the error description.
The function will return an active connection object.
 
[1]: 1 # Q 1:
  2  
  3 # import mysql
  4 # ---- your solution here ---
  5  
  6 # define the function
  7 # ---- your solution here ---
  8  
  9 # call the function
  10 # mysql_connect()
     
[1]:   Successfully connected to MySQL Server.
 
Q2:
Create a new database named quiz_db.
You should call the connection function which you
defined in Q1.
Print the result of the operation, either successful
or an exception.
 
Hints:
* close connection
* close cursor
 
[2]: 1 # Q 2:
  2  
  3 # import mysql
  4 # ---- your solution here ---
  5  
  6 # function for create the db
  7 # ---- your solution here ---
  8  
  9 # call the function and create the db
  10 create_quiz_db("quiz_db")
     
[2]:   Successfully connected to MySQL Server.
    Database created successfully.
 
Q3:
Define a function named create_student_table.
It will create a table in the quiz_db database.
The table name will be student.
You should call the connection function which you
defined in Q1.
 
Here are the fields for the student table:
* id: int(11), not null, auto increment and primary
key
* first_name: varchar(20)
* last_name: varchar(20)
* age: int(4)
 
Hints:
* set database as quiz_db on the cursor
* close connection
* close cursor
 
[3]: 1 # Q 3:
  2  
  3 # import mysql
  4 # ---- your solution here ---
  5  
  6 # define Student table
  7 # ---- your solution here ---
  8  
  9 # connect and create table
  10 # ---- your solution here ---
  11  
  12 # call the function
  13 create_student_table()
     
[3]:   Successfully connected to MySQL Server.
    Student table created successfully.
 
Q4:
Define a function name insert_student_records.
The function will take a list of students and insert
them to the student table in db.
We created the student table in quiz_db database in
Q3.
You should call the connection function which you
defined in Q1.
 
Here is the list:
students_to_insert = [
    ("Peter", "Parker", 24),
    ("Mary", "Jane", 23),
    ("Clark", "Kent", 32),
    ("Bruce", "Wayne", 29)]
 
Hints:
* commit()
* cursor.executemany() for executing sequences all
at once
 
[4]: 1 # Q 4:
  2  
  3 # import mysql
  4 # ---- your solution here ---
  5  
  6 # define insert query
  7 # ---- your solution here ---
  8  
  9 # define student list
  10 # ---- your solution here ---
  11  
  12 # connect and insert records
  13 # ---- your solution here ---
  14  
  15 # call the function
  16 # ---- your solution here ---
     
[4]:   Successfully connected to MySQL Server.
    Student records inserted.
 
Q5:
Define a function named query_students.
It will fetch and print all the student records in the
quiz_db database.
You should call the connection function which you
defined in Q1.
 
[5]: 1 # Q 5:
  2  
  3 # import mysql
  4 # ---- your solution here ---
  5  
  6 # query to get all students
  7 # ---- your solution here ---
  8  
  9 # connect and set cursor
  10 # ---- your solution here ---
  11  
  12 # call the function
  13 query_students()
     
[5]:   Successfully connected to MySQL Server.
    (1, 'Peter', 'Parker', 24)
    (2, 'Mary', 'Jane', 23)
    (3, 'Clark', 'Kent', 32)
    (4, 'Bruce', 'Wayne', 29)
 

OceanofPDF.com
 
SOLUTIONS – Database Operations
 
Here are the solutions for the quiz for
this chapter.
 
S1:
 
[1]: 1 # S 1:
  2  
  3 # import mysql
  4 import mysql.connector
  5  
  6 # define the function
  7 def mysql_connect():
  8     connection = None
  9     try:
  10         # start the connection
        connection =
 
11 mysql.connector.connect(
  12             host="localhost",
  13             user="root",
            password="
 
14 <your_root_password>"
  15         )
        print("Successfully connected to
 
16 MySQL Server.")
  17     except mysql.connector.Error as e:
  18         print(f"An Error occurred: '{e}'")
  19     return connection
  20  
  21 # call the function
  22 mysql_connect()
     
[1]:   Successfully connected to MySQL Server.
 
S2:
 
[2]: 1 # S 2:
  2  
  3 # import mysql
  4 import mysql.connector
  5  
  6 # function for create the db
  7 def create_quiz_db(db_name):
  8     try:
  9         # connect to server
  10         connection = mysql_connect()
  11         if connection is not None:
  12             # set the cursor
  13             cursor = connection.cursor()
  14             # create the database
            cursor.execute(f"CREATE
 
15 DATABASE {db_name}")
  16     except mysql.connector.Error as e:
        print(f"An Error occurred in DB
 
17 Creation:\n'{e}'")
  18     else:
  19         if connection is not None:
            # close the cursor and the
 
20 connection
  21             cursor.close()
  22             connection.close()
  23             print("Database created
 successfully.")
  24  
  25 # call the function and create the db
  26 create_quiz_db("quiz_db")
     
[2]:   Successfully connected to MySQL Server.
    Database created successfully.
 
S3:
 
[3]: 1 # S 3:
  2  
  3 # import mysql
  4 import mysql.connector
  5  
  6 # define Student table
  7 student_table = (
  8     "CREATE TABLE `student` ("
    "  `id` int(11) NOT NULL
 
9 AUTO_INCREMENT,"
  10     "  `first_name` varchar(20) NOT NULL,"
  11     "  `last_name` varchar(20) NOT NULL,"
  12     "  `age` int(4) NOT NULL,"
  13     "  PRIMARY KEY (`id`)"
  14     ") ENGINE=InnoDB")
  15  
  16 # connect and create table
  17 def create_student_table():
  18     try:
  19         # connect to server
  20         connection = mysql_connect()
  21         if connection is not None:
  22             # set the cursor
  23             cursor = connection.cursor()
  24             # set use db for cursor
  25             cursor.execute("USE quiz_db")
  26             # create table
  27             cursor.execute(student_table)
  28     except mysql.connector.Error as e:
        print(f"An Error occurred in Table
 
29 Creation:\n'{e}'")
  30     else:
        print("Student table created
 
31 successfully.")
  32         cursor.close()
  33         connection.close()
  34  
  35 # call the function
  36 create_student_table()
     
[3]:   Successfully connected to MySQL Server.
    Student table created successfully.
 
S4:
 
[4]: 1 # S 4:
  2  
  3 # import mysql
  4 import mysql.connector
  5  
  6 # define insert query
  7 insert_query = """
  8 INSERT INTO student
  9 (first_name, last_name, age)
  10 VALUES (%s, %s, %s)
  11 """
  12  
  13 # define student list
  14 students_to_insert = [
  15     ("Peter", "Parker", 24),
  16     ("Mary", "Jane", 23),
  17     ("Clark", "Kent", 32),
  18     ("Bruce", "Wayne", 29)]
  19  
  20 # connect and insert records
  21 def insert_student_records(students_list):
  22     try:
  23         # connect to server
  24         connection = mysql_connect()
  25         if connection is not None:
  26             # set the cursor
  27             cursor = connection.cursor()
  28             # set use db for cursor
  29             cursor.execute("USE quiz_db")
  30             # create table
            cursor.executemany(insert_query,
 
31 students_list)
  32             connection.commit()
  33     except mysql.connector.Error as e:
        print(f"An Error in insert
 
34 operation:\n'{e}'")
  35     else:
  36         print("Student records inserted.")
  37         cursor.close()
  38         connection.close()
  39  
  40 # call the function
  41 insert_student_records(students_to_insert)
     
[4]:   Successfully connected to MySQL Server.
    Student records inserted.
 
S5:
 
[5]: 1 # S 5:
  2  
  3 # import mysql
  4 import mysql.connector
  5  
  6 # query to get all students
  7 query = ("SELECT * FROM student")
  8  
  9 # connect and set cursor
  10 def query_students():
  11     try:
  12         # connect to server
  13         connection = mysql_connect()
  14         if connection is not None:
  15             # set the cursor
  16             cursor = connection.cursor()
  17             # set use db for cursor
  18             cursor.execute("USE quiz_db")
  19             # execute query
  20             cursor.execute(query)
  21             # get the result
  22             result = cursor.fetchall()
  23             # print the results
  24             for row in result:
  25                 print(row)
  26     except mysql.connector.Error as e:
        print(f"An Error occurred in
 
27 Querying Records:\n'{e}'")
  28     else:
  29         cursor.close()
  30         connection.close()
  31  
  32 # call the function
  33 query_students()
     
[5]:   Successfully connected to MySQL Server.
    (1, 'Peter', 'Parker', 24)
    (2, 'Mary', 'Jane', 23)
    (3, 'Clark', 'Kent', 32)
    (4, 'Bruce', 'Wayne', 29)
 
 
OceanofPDF.com
 14. Concurrency
       

 
In general, Concurrency is the fact of
two or more events or circumstances
happening or existing at the same time. In
Computer Science, it is the ability to
execute more than one program or task
simultaneously. Concurrent programming
may drastically improve the performance
of your applications if it’s used properly.
In this chapter, we will learn how Python
provides support for concurrent execution
of code. You can find the PyCharm project
for this chapter in the GitHub Repository
of this book.
 
Chapter Outline:
CPU Bound vs. IO Bound
Multithreading vs. Multiprocessing
Threading
ThreadPoolExecutor – submit()
ThreadPoolExecutor – map()
ProcessPoolExecutor – submit()
ProcessPoolExecutor – map()
QUIZ – Concurrency
 SOLUTIONS – Concurrency
 
 
OceanofPDF.com
 CPU Bound vs. IO Bound
 
The tasks that computers perform can
mainly be divided into two main
categories which are CPU Bound tasks and
IO Bound tasks.
CPU Bound means the rate of the
execution of a task is limited by the speed
of the CPU. Tasks as image resizing, large
dataset operations like matrix
multiplications, sorting of huge arrays,
rendering operations may be examples of
CPU Bound tasks.
IO Bound means the rate of the
execution of a task is limited by the speed
of the IO (input/output) subsystem. Tasks
like reading from and writing to the disk,
network or communication tasks can be
examples of IO Bound.
In real-life projects, the difference
between CPU Bound and IO Bound tasks is
not always obvious. You should be aware
that, most of the time it is not possible to
strictly classify a task as either IO Bound
or CPU Bound. You have to consider the
tools for both cases and do some trials to
decide based on your environment.
 
 
OceanofPDF.com
 Multithreading vs. Multiprocessing
 
In concurrent programming, there are
two concepts that you might encounter
often. These are Multithreading and
Multiprocessing. Both terms refer to
operations which increase the computing
power of a system. But there are some
important differences between them.
Multithreading is a technique which
enables a single process to be split into
multiple code segments (threads). A
thread, is a program execution context,
within a process. A process can contain
multiple threads. Threads run concurrently
(in parallel) within the context of their
parent process. Multithreaded applications
are the ones that have two or more
threads which run concurrently.
Multiprocessing refers to a system
which has more than two CPUs (central
processing units). In systems which have
Multiprocessing, every CPU added helps to
increase the computing power of the
entire system. Each CPU has a main
memory and can function independently.
Here are some key differences between
Multithreading and Multiprocessing:
To increase computing power:
In Multithreading, multiple
threads are created
In Multiprocessing, new CPUs
are added to the system
Multiprocessing executes multiple
processes on different CPUs, while
Multithreading executes multiple
threads in a single process.
Ease of implementation:
Multiprocessing needs
considerable amount of
investment (either time or
resources)
Multithreading is easy to
implement and requires less
resources compared to
Multiprocessing.
 
Python provides support for both
Multithreading and Multiprocessing. In
general, IO Bound tasks are handled with
Multithreading and CPU Bound tasks are
handling with Multiprocessing techniques.
However, this might not be the case in
every situation. It is up to the developer to
decide which one to use when needed.
 
 
OceanofPDF.com
 Threading
 
In this section we will learn how we create
and run threads in Python. We will download
some high-quality images from internet for
this task. These are the images of IMDB Top
20 movies. You can find the URLs of the
movies in the ‘data/imdb_top_20.csv’ file in
the PyCharm Project of this chapter. We will
download the images and save them to a
folder called ‘images’ . See the image below:
 

Figure 14-1:  imdb_top_20.csv file in the project

 
To be able to read the data in
‘data/imdb_top_20.csv’ file, we create a
Python file named Utils.py . Here is the
content in this file:
 
[1]: 1 # Utils.py file
  2  
  3 import csv
  4  
  5 # define a movies list
  6 movies = []
  7  
  8 # function to read movies
  9 def read_csv():
  10     """reads the csv file and returns
  11         the list of movies"""
  12     movie_path = 'data/imdb_top_20.csv'
  13     with open(movie_path, 'r') as file:
        movie_dict = csv.DictReader(file,
 
14 delimiter=';')
  15         for movie in movie_dict:
  16             movies.append(movie)
  17     # return the list
  18     return movies
 
In cell 1, we have Utils.py file and the
read_csv() function in it. This function reads
the movie data in ‘data/imdb_top_20.csv’ file
and returns a list of dictionaries called
movies .
Now let’s create another Python file
named WithoutThreading.py . In this file we
will download the movie images using the
URLs provided in the movies list.
 
[2]: 1 # WithoutThreading.py file
  2  
  3 import requests
  4 import shutil
  5 import time
  6 import Utils
  7  
  8 # call the function and fill movies
  9 movies = Utils.read_csv()
  10  
  11 # function to download a single image
  12 def download_image(movie):
  13     title = movie['Title']
  14     file_name = 'images/' + str(title) + '.jpg'
    response =
  requests.get(movie['ImageURL'],
15 stream=True)
  16     if response.status_code == 200:
  17         with open(file_name, 'wb') as file:
            shutil.copyfileobj(response.raw,
 
18 file)
  19         print(f"{title} downloaded.")
  20     else:
  21         print(f"Error in downloading {title}")
  22  
  23 # function to download images
  24 def downlaod_all_images():
  25     # set start time
  26     start = time.perf_counter()
  27     # call download_image in a loop
  28     for movie in movies:
  29         download_image(movie)
  30     # set end time
  31     end = time.perf_counter()
  32     # print elapsed time
  33     print(f"\nDownload took {end-start:.2f}
seconds without Threading.")
 
In cell 2, you see the content in the
WithoutThreading.py file. We first get the
movies list by calling the Utils.read_csv()
function in line 8.
In line12, we define a function called
download_image() . It downloads the image of
a single movie, which is the parameter to it.
It requests the image stream and saves it
under the ‘images’ folder with the movie
title.
In line 24, we have another function
called downlaod_all_images() . This function
loops over the movies list and call the
download_image() function for each movie.
We also keep track of the elapsed time in
this function. And finally, we print the total
time elapsed for downloading all of the
movies. Here is the result when we call this
function:
 
[3]: 1 # download all images
  2 downlaod_all_images()
     
[3]:   The Shawshank Redemption downloaded.
    The Godfather downloaded.
    The Godfather: Part II downloaded.
    The Dark Knight downloaded.
    12 Angry Men downloaded.
    Schindler's List downloaded.
The Lord of the Rings: The Return of the
   
King downloaded.
    Pulp Fiction downloaded.
    Fight Club downloaded.
The Lord of the Rings: The Fellowship of the
   
Ring downloaded.
    Forrest Gump downloaded.
Star Wars: Episode V - The Empire Strikes
   
Back downloaded.
    Inception downloaded.
The Lord of the Rings: The Two Towers
   
downloaded.
One Flew Over the Cuckoo's Nest
   
downloaded.
    Goodfellas downloaded.
    The Matrix downloaded.
    Se7en downloaded.
    It's a Wonderful Life downloaded.
    The Silence of the Lambs downloaded.
     
Download took 29.63 seconds without
   
Threading.
 
As you see in the output of cell 3, it took
about 29.63 seconds to download all movie
images when we do it with a for loop. This is
the result on my computer and my internet
connection. Your result will be different.
However, as you see, it took a bit long,
because it tries to download each image
one by one, one after another.
This example is a typical case, where IO
Bound is the limit. We should expect a
better result if we do the same operation
using threads. Let’s test it and see if it’s
true.
We will create a new Python file named
Threading.py . And here is the content of this
file:
 
[4]: 1 # Threading.py file
  2  
  3 import requests
  4 import shutil
  5 import time
  6 from threading import Thread
  7 import Utils
  8  
  9 # call the function and fill movies
  10 movies = Utils.read_csv()
  11  
  12 # create a threads list
  13 threads = []
  14  
  15 # function to download a single image
  16 def download_image(movie):
  17     title = movie['Title']
  18     file_name = 'images/' + str(title) + '.jpg'
  19     response =
requests.get(movie['ImageURL'],
 stream=True)
  20     if response.status_code == 200:
  21         with open(file_name, 'wb') as file:
            shutil.copyfileobj(response.raw,
 
22 file)
  23         print(f"{title} downloaded.")
  24     else:
  25         print(f"Error in downloading {title}")
  26  
  27 # function to download images
  28 def downlaod_all_images():
  29     # set start time
  30     start = time.perf_counter()
  31     # fill the threads list
  32     for movie in movies:
        thread =
  Thread(target=download_image, args=
33 (movie,))
  34         thread.start()
  35         threads.append(thread)
  36     # force for wait for finish
  37     for thread in threads:
  38         thread.join()
  39     # set end time
  40     end = time.perf_counter()
  41     # print elapsed time
    print(f"\nDownload took {end-start:.2f}
 
42 seconds with Threading.")
 
In cell 4, line 13, we create an empty list
for the threads which we will create in a
minute. The list name is threads .
 The definition of the download_image()
function is exactly the same with the one in
the previous section. It simply takes in a
movie as the parameter, downloads its
image and saves it to the ‘images’ folder.
We change the definition of the
downlaod_all_images() function. Now instead
of calling the download_image() function one
after another, we create Thread objects for
this. Here is the code for it:
thread = Thread(target=download_image,
args=(movie,))
The target here, is the function which we
want this thread to execute. And args is the
arguments it will pass to the given function.
We simply give the movie object. The
Thread() constructor returns a Thread object
which is ready to be started.
In line 34, we start this thread by calling
start() method, which starts the thread’s
activity.
In line 35, we append the current thread
object to our threads list.
In line 37, we set a for loop on the threads
list. Then inside the loop body, we call the
join() method on each thread object. The
join() method waits until the thread
terminates. We need this to force the code
execution to wait the thread to finish.
Otherwise, the code would move on and we
would not be able to track the end time.
And now, let’s see how long it takes to
finish downloading all images with threads.
Let’s call the function:
 
[5]: 1 # download all images
  2 downlaod_all_images()
     
[5]:   12 Angry Men downloaded.
    Forrest Gump downloaded.
The Lord of the Rings: The Return of the
   
King downloaded.
The Lord of the Rings: The Two Towers
   
downloaded.
    The Godfather downloaded.
    Pulp Fiction downloaded.
    It's a Wonderful Life downloaded.
One Flew Over the Cuckoo's Nest
   
downloaded.
    Goodfellas downloaded.
    Inception downloaded.
The Lord of the Rings: The Fellowship of the
   
Ring downloaded.
    The Silence of the Lambs downloaded.
    The Shawshank Redemption downloaded.
Star Wars: Episode V - The Empire Strikes
   
Back downloaded.
    The Godfather: Part II downloaded.
    Se7en downloaded.
    Schindler's List downloaded.
    The Matrix downloaded.
    The Dark Knight downloaded.
    Fight Club downloaded.
     
Download took 8.92 seconds with
   
Threading.
 
As you see in the output of cell 5, the
code execution takes 8.92 seconds with
threading. It was 29.63 seconds without
using threads. So, we have a considerable
improvement here. That’s because
downloading images via network is a good
candidate for IO bound tasks. And if the
images are not related to each other, like
we have here, we can easily assign separate
threads to them. None of them has to wait
another one to complete.
 
 
OceanofPDF.com
 ThreadPoolExecutor – submit()
 
In the previous section, we created each
Thread object and start them one by one.
Then in another loop, we call the join()
method on each thread. Python has a
better way to create and run threads. The
module for this is called concurrent.futures
and there are very handy classes in this
module. ThreadPoolExecutor is one of these
classes and it provides methods to
execute thread calls easily. It is the class
we use for Multithreading operations.
ThreadPoolExecutor has two important
methods as submit() and map() for
creating and managing threads.
 
submit(function, /, *args, **kwargs) :
Schedules the callable, function , to be
executed as function(*args, **kwargs) and
returns a Future object representing the
execution of the callable. Future instances
are created by Executor.submit() and
encapsulates the asynchronous execution
of a callable.
 We will create a new Python file named
ThreadPoolExecutor_Submit.py . We will do
the same image download operation but
with concurrent.futures module,
ThreadPoolExecutor class and the submit()
method this time.
 
[6]: 1 # ThreadPoolExecutor_Submit.py file
  2  
  3 import requests
  4 import shutil
  5 import time
from concurrent.futures import
 
6 ThreadPoolExecutor, as_completed
  7 import Utils
  8  
  9 # call the function and fill movies
  10 movies = Utils.read_csv()
  11  
  12 # create a threads list
  13 threads = []
  14  
  15 # function to download a single image
  16 def download_image(movie):
  17     title = movie['Title']
    file_name = 'images/' + str(title) +
 
18 '.jpg'
    response =
  requests.get(movie['ImageURL'],
19 stream=True)
  20     if response.status_code == 200:
  21         with open(file_name, 'wb') as file:
            shutil.copyfileobj(response.raw,
 
22 file)
  23         print(f"{title} downloaded.")
  24     else:
        print(f"Error in downloading
 
25 {title}")
  26  
  27 # function to download images
  28 def downlaod_all_images():
  29     # set start time
  30     start = time.perf_counter()
  31     # set ThreadPoolExecutor
    with
  ThreadPoolExecutor(max_workers=4) as
32 executor:
        # submit the threads and get a list
 
33 of futures
        future_list =
  [executor.submit(download_image,
34 movie)
  35                        for movie in movies]
        for future in
 
36 as_completed(future_list):
  37             future.result()
  38     # set end time
  39     end = time.perf_counter()
  40     # print elapsed time
    print(f"\nDownload took {end-
 
41 start:.2f} sec, ThreadPoolExecutor.")
 
In cell 6, you see the content of the
ThreadPoolExecutor_Submit.py file. We
import the ThreadPoolExecutor class and
the as_completed() function from the
concurrent.futures module.
In the downlaod_all_images() function, we
call the ThreadPoolExecutor(max_workers=4)
constructor and it gives us an executor
object.
In line 34, we set a list comprehension
to call the submit() method with each
movie. We pass movie objects one by one
to the submit() method as:
executor.submit(download_image, movie) . And
since the submit() method returns a Future
object, list comprehension gives us a list
of Future objects which we name it as
future_list .
In line 36, we pass this future_list to the
as_completed() function. This function
returns an iterator over the Future
instances as their related threads get
completed. In other words, as_completed()
function returns completed threads in
form of an iterator. And we loop over this
iterator with a for loop.
Inside the for loop, in line 37, we call
the result() method for each Future object.
Future.result(timeout=None) method returns
the value returned by the call (the return
value from the target function of the
thread). If the call hasn’t yet completed
then this method will wait up to timeout
seconds. In our example, the target
function is download_image() and it has no
return value. It simply prints the download
result. You can omit calling the result()
method here, since it’s not needed. I
added it to show you how you can get the
return values from the functions you pass
to the threads.
 
[7]: 1 # download all images
  2 downlaod_all_images()
     
[7]:   The Godfather downloaded.
    The Godfather: Part II downloaded.
    12 Angry Men downloaded.
    The Shawshank Redemption downloaded.
The Lord of the Rings: The Return of the
   
King downloaded.
    Pulp Fiction downloaded.
    Schindler's List downloaded.
    Forrest Gump downloaded.
The Lord of the Rings: The Fellowship of
   
the Ring downloaded.
    The Dark Knight downloaded.
    The Lord of the Rings: The Two Towers
downloaded.
    Inception downloaded.
    Fight Club downloaded.
    Goodfellas downloaded.
One Flew Over the Cuckoo's Nest
   
downloaded.
Star Wars: Episode V - The Empire Strikes
   
Back downloaded.
    It's a Wonderful Life downloaded.
    The Matrix downloaded.
    Se7en downloaded.
    The Silence of the Lambs downloaded.
     
Download took 9.89 seconds with
   
ThreadPoolExecutor.
 
As you see in cell 7, we got a similar
result by using ThreadPoolExecutor .
 
 
OceanofPDF.com
 ThreadPoolExecutor – map()
 
In the previous section we use the
submit() method on the ThreadPoolExecutor
objects to create and run threads. In this
section we will learn the map() method to
do the same operation.
 
map(func, *iterables, timeout=None,
chunksize=1) :
It is very similar to built-in map function
in Python which is map(func, *iterables) .
The map() function on the
ThreadPoolExecutor object returns an
iterator of Future objects. Let’s see it in
our download example:
 
[8]: 1 # ThreadPoolExecutor_Map.py file
  2  
  3 import requests
  4 import shutil
  5 import time
from concurrent.futures import
 
6 ThreadPoolExecutor, as_completed
  7 import Utils
  8  
  9 # call the function and fill movies
  10 movies = Utils.read_csv()
  11  
  12 # create a threads list
  13 threads = []
  14  
  15 # function to download a single image
  16 def download_image(movie):
  17     title = movie['Title']
    file_name = 'images/' + str(title) +
 
18 '.jpg'
    response =
  requests.get(movie['ImageURL'],
19 stream=True)
  20     if response.status_code == 200:
  21         with open(file_name, 'wb') as file:
            shutil.copyfileobj(response.raw,
 
22 file)
  23         print(f"{title} downloaded.")
  24     else:
        print(f"Error in downloading
 
25 {title}")
  26  
  27 # function to download images
  28 def downlaod_all_images():
  29     # set start time
  30     start = time.perf_counter()
  31     # set ThreadPoolExecutor
    with
  ThreadPoolExecutor(max_workers=4) as
32 executor:
  33         # map the threads
        executor.map(download_image,
 
34 movies)
  35     # set end time
  36     end = time.perf_counter()
  37     # print elapsed time
    print(f"\nDownload took {end-
 
38 start:.2f} sec, ThreadPoolExecutor.")
 
In cell 8,
line 34, inside the
downlaod_all_images() function we use the
map() function on the executor object. The
parameters are the target function
( download_image ) and the iterable ( movies ).
The map() function instantiates and starts
the threads all at once. And you will see a
similar result when you run this code:
 
[9]: 1 # download all images
  2 downlaod_all_images()
     
[9]:   The Godfather downloaded.
    The Godfather: Part II downloaded.
    12 Angry Men downloaded.
    The Shawshank Redemption downloaded.
The Lord of the Rings: The Return of the
   
King downloaded.
    Schindler's List downloaded.
    Pulp Fiction downloaded.
    Forrest Gump downloaded.
The Lord of the Rings: The Fellowship of
   
the Ring downloaded.
    The Dark Knight downloaded.
    The Lord of the Rings: The Two Towers
 downloaded.
    Inception downloaded.
Star Wars: Episode V - The Empire Strikes
   
Back downloaded.
    Fight Club downloaded.
    Goodfellas downloaded.
One Flew Over the Cuckoo's Nest
   
downloaded.
    Se7en downloaded.
    It's a Wonderful Life downloaded.
    The Silence of the Lambs downloaded.
    The Matrix downloaded.
     
Download took 11.29 sec,
   
ThreadPoolExecutor.
 
 
OceanofPDF.com
 ProcessPoolExecutor – submit()
 
In the
previous sections we learned
ThreadPoolExecutor , which is the class used for
Multithreading operations. In this section we
will learn ProcessPoolExecutor , which is used for
Multiprocessing tasks. The implementation of
ProcessPoolExecutor is almost the same as
ThreadPoolExecutor class. So, it is very easy to
switch between them to be able to check if a
task needs Multithreading or Multiprocessing.
For ProcessPoolExecutor , we will do a more
computation heavy task. The task will be to
decide if a given number is a Prime Number or
not.
We will create a new Python file as
ThreadPoolExecutor_Submit.py . Let’s start with a
single processing example:
 
[10]: 1 # ProcessPoolExecutor_Submit.py file
  2  
from concurrent.futures import
 
3 ProcessPoolExecutor
  4 import time
  5  
  6 # some numbers
primes = [7774777, 7778777, 111181111,
 
7 111191111, 777767777]
  8  
  9 # function to check for being prime
  10 def is_prime(num):
  11     if num > 1:
  12         for n in range(2, num):
  13             if (num % n) == 0:
  14                 return False
  15         return True
  16     else:
  17         return False
  18  
  19 def single_process_check():
  20     # set start time
  21     start = time.perf_counter()
  22     # check the primes list one by one
  23     for num in primes:
        print(f"Is Prime {num} -
 
24 {is_prime(num)}")
  25     # set end time
  26     end = time.perf_counter()
  27     # print elapsed time
    print(f"\nTook {end - start:.2f} sec, Single
 
28 Process.")
  29  
  30 # call the single process check
  31 single_process_check()
     
[10]:   Is Prime 7774777 - True
    Is Prime 7778777 - True
    Is Prime 111181111 - True
    Is Prime 111191111 - True
    Is Prime 777767777 - True
     
    Took 29.52 sec, Single Process.
 
In cell 10, we have a function which checks if
the given number is a prime number. Function
name is is_prime() and as you may easily guess
it is not suitable for high-performance
operations.
 We have another function named
single_process_check() . This function loops over
the primes list and checks each element one by
one by calling the is_prime() function. And as
you see in the cell output, it takes about 29.52
seconds to complete this task. Which is
obviously an overkill for such a simple task.
Now let’s implement multiprocessing for the
same task. We will use ProcessPoolExecutor and
the submit() method. Here it is:
 
[11]: 1 ### MULTIPROCESSING ###
  2 def multi_proces_check():
  3     # set start time
  4     start = time.perf_counter()
  5     # set ProcessPoolExecutor
  6     with ProcessPoolExecutor() as executor:
        # submit the threads and get a list of
 
7 futures
        future_list = [executor.submit(is_prime,
 
8 n)
  9                        for n in primes]
        for num, future in zip(primes,
 
10 as_completed(future_list)):
            print(f"Is Prime {num} -
 
11 {future.result()}")
  12     # set end time
  13     end = time.perf_counter()
  14     # print elapsed time
    print(f"\nTook {end - start:.2f} sec, Multi
 
15 Process.")
 
In cell 11, we define a new function named
multi_proces_check() . In the function body, we set
a with context manager to use
ProcessPoolExecutor . In line 8, call the submit()
method on the executor in a list comprehension.
As we already know, this comprehension will
return an iterator of Future objects. Then in line
10, we call as_completed() function on each
Future object, then we print the result.
To be able to run this function you need to
call it in a main.py file, especially on a macOS
computer. Otherwise, you might get an error
regarding subprocesses. That’s why we will
create a main.py file in our project and call this
function as follows:
 
[12]: # main.py file
1
  2
 
  3
import ProcessPoolExecutor_Submit
  4
 
  5
if __name__ == '__main__':
      # call multi process check
6
   
 
7 ProcessPoolExecutor_Submit.multi_proces_check()
     
[12]:   Is Prime 7774777 - True
    Is Prime 7778777 - True
    Is Prime 111181111 - True
    Is Prime 111191111 - True
    Is Prime 777767777 - True
     
    Took 22.26 sec, Multi Process.
 
 As you see in cell 12, we get an improvement
in the same operation when we use
Multiprocessing.
 
 
OceanofPDF.com
 ProcessPoolExecutor – map()
 
In the previous section we used the submit()
method for Multiprocessing. Now, let’s repeat the
same operation using map() function this time.
We will create a new Python file named
ProcessPoolExecutor_Map.py . And here is the
multi_proces_check() function it this file:
 
[13]: 1 ### MULTIPROCESSING ###
  2 def multi_proces_check():
  3     # set start time
  4     start = time.perf_counter()
  5     # set ProcessPoolExecutor
  6     with ProcessPoolExecutor() as executor:
  7         # map the threads
        future_list = executor.map(is_prime,
 
8 primes)
  9         for num, future in zip(primes, future_list):
  10             print(f"Is Prime {num} - {future}")
  11     # set end time
  12     end = time.perf_counter()
  13     # print elapsed time
    print(f"\nTook {end - start:.2f} sec, Multi
 
14 Process.")
 
Let’s call this function from the main.py file:
 
[14]: 1 # main.py file
  2  
  3 import ProcessPoolExecutor_Submit
  4 import ProcessPoolExecutor_Map
  5  
  6 if __name__ == '__main__':
  7     # call multi process check with submit()
    #
 
8 ProcessPoolExecutor_Submit.multi_proces_check()
  9  
  10     # call multi process check with map()
  11     ProcessPoolExecutor_Map.multi_proces_check()
     
[14]:   Is Prime 7774777 - True
    Is Prime 7778777 - True
    Is Prime 111181111 - True
    Is Prime 111191111 - True
    Is Prime 777767777 - True
     
    Took 22.40 sec, Multi Process.
 
Again, we see some improvement in the total
execution time of the operation. This is the end
of this chapter. Now it’s your turn to solve the
quiz questions on Concurrency.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
OceanofPDF.com
 
QUIZ – Concurrency
 
Now it’s time to solve the quiz for this
chapter. You can download the quiz file,
QUIZ_Concurrency.zip, from the GitHub
Repository of this book. You should put the
quiz file in the Concurrency project we
build in this chapter.
 
Here are the questions for this chapter:
 
Q1:
We want to resize the images in the "images" folder
in this project.
The initial size of this folder is around 27,5 MB with
20 images in it.
We want to create a "thumbnails" folder which
includes the thumbnails of these images.
Thumbnail dimensions (width * height) of each
image will 1/10 of the original image.
For example, the dimensions "Fight Club.jpg" file in
the “images” directory is 2021*3000 pixels, but its
thumbnail will be around 202*300 pixels.
And in terms of disk space, we expect the
"thumbnails" folder to be around 200 KB.
You should keep track of elapsed time (start & end).
 
You need to define four functions:
* get_images
* create_directory
* resize_all_images
* resize_image
 
Packages you need:
* os
* pathlib
* time
* Pillow (for PIL)
 
[1]: 1 # Q 1:
  2  
  3 # import packages
  4 import os
  5 from pathlib import Path
  6 import time
  7 # install Pillow package for PIL
  8 from PIL import Image
  9  
  10 # directory names
  11 images_directory = "images"
  12 thumbnails_directory = "thumbnails"
  13  
  14 # get images fn
  15 def get_images():
  16     """Returns a list of .jpg files
  17     in the images folder"""
  18     # ---- your solution here ---
  19  
  20 # create directory fn
  21 def create_directory(name):
  22     """Creates the directory with name.
  23     No errors if the directory exists."""
  24     # ---- your solution here ---
  25  
  26 # resize all images fn
  27 def resize_all_images():
  28     """Creates the directory.
    Gets images and call resize fn one by
 
29 one.
  30     Keeps track of elapsed time."""
  31     try:
  32         # set start time
  33         # ---- your solution here ---
  34         # create directory
  35         # ---- your solution here ---
  36         # get images
  37         # ---- your solution here ---
  38         # resize each image one by one
  39         # ---- your solution here ---
  40         # set end time
  41         # ---- your solution here ---
  42     except:
        print("An Error occurred in
 
43 resizing.")
  44     else:
  45         # print elapsed time
  46         # ---- your solution here ---
  47  
  48 # resize a single image fn
  49 def resize_image(image):
  50     # open the image
  51     # ---- your solution here ---
  52     # resize the image
  53     # ---- your solution here ---
  54     # Save the resized image
  55     # ---- your solution here ---
  56     # print result
  57     # ---- your solution here ---
  58  
  59 # call resize all function
  60 resize_all_images()
     
[1]:   …
    The Godfather: Part II.jpg resized.
    The Dark Knight.jpg resized.
    Inception.jpg resized.
    Images resized in 1.08 sec.
 
Q2:
"""
Redefine the resize_all_images() function you
defined in Q1.
This time it will use Threading for resize operation.
Create and start a thread for each resize_image()
function call.
Append these threads to a list called "threads".
Then join() the threads to force the code to wait
them to finish.
Keep track of time again and print the final result.
 
Hints:
* create each thread using the Thread class
* start() each thread you create
* join() them in a separate for loop
 
[2]: 1 # Q 2:
  2  
  3 from threading import Thread
  4  
  5 # create a threads list
  6 threads = []
  7  
  8 # redefine resize all images fn
  9 def resize_all_images():
  10     """Creates the directory.
        Gets images and call resize fn one
 
11 by one.
  12         Keeps track of elapsed time."""
  13     try:
  14         # set start time
  15         # ---- your solution here ---
  16         # create directory
  17         # ---- your solution here ---
  18         # get images
  19         # ---- your solution here ---
  20         # fill the threads list
  21         # ---- your solution here ---
  22         # force for wait for finish
  23         # ---- your solution here ---
  24         # set end time
  25         # ---- your solution here ---
  26     except:
        print("An Error occurred in
 
27 resizing.")
  28     else:
  29         # print elapsed time
  30         # ---- your solution here ---
  31  
  32 # call resize all function
  33 resize_all_images()
     
[2]:   …
    The Dark Knight.jpg resized.
    The Matrix.jpg resized.
    Fight Club.jpg resized.
    Images resized in 0.29 sec - Threading.
 
Q3:
Redefine the resize_all_images() function you
defined in Q1.
Use ThreadPoolExecutor in this question.
Set a with context manager using
ThreadPoolExecutor and use list comprehension to
submit() threads.
Use a for loop to be able to get the result when the
threads are completed.
Keep track of time again and print the final result.
 
Hints:
* ThreadPoolExecutor()
* [submit() in list comprehension]
* as_comleted()
* Future.result()
 
[3]: 1 # Q 3:
  2  
from concurrent.futures import
 
3 ThreadPoolExecutor, as_completed
  4  
  5 # create a threads list
  6 threads = []
  7  
  8 # redefine resize all images fn
  9 def resize_all_images():
  10     """Creates the directory.
        Gets images and call resize fn one
 
11 by one.
  12         Keeps track of elapsed time."""
  13     try:
  14         # set start time
  15         # ---- your solution here ---
  16         # create directory
  17         # ---- your solution here ---
  18         # get images
  19         # ---- your solution here ---
  20         # set ThreadPoolExecutor
  21         with # ---- your solution here ---:
            # submit the threads and get a
 
22 list of futures
  23             # ---- your solution here ---
            # call as_complete() method for
 
24 result
  25             # ---- your solution here ---
  26         # set end time
  27         # ---- your solution here ---
  28     except:
        print("An Error occurred in
 
29 resizing.")
  30     else:
  31         # print elapsed time
  32         # ---- your solution here ---
  33  
  34 # call resize all function
  35 resize_all_images()
     
[3]:   …
    The Dark Knight.jpg resized.
    The Matrix.jpg resized.
    Fight Club.jpg resized.
    Images resized in 0.33 sec -
 ThreadPoolExecutor.
 
Q4:
Redefine the resize_all_images() function you
defined in Q1.
Use ProcessPoolExecutor in this question.
Set a with context manager using
ProcessPoolExecutor and use list comprehension to
submit() threads.
Use a for loop to be able to get the result when the
threads are completed.
Keep track of time again and print the final result.
 
Hints:
* ProcessPoolExecutor()
* [submit() in list comprehension]
* as_comleted()
* Future.result()
 
[4]: 1 # Q 4:
  2  
from concurrent.futures import
 
3 ProcessPoolExecutor, as_completed
  4  
  5 # create a threads list
  6 threads = []
  7  
  8 # redefine resize all images fn
  9 def resize_all_images():
  10     """Creates the directory.
        Gets images and call resize fn one
 
11 by one.
  12         Keeps track of elapsed time."""
  13     try:
   14         # set start time
  15         # ---- your solution here ---
  16         # create directory
  17         # ---- your solution here ---
  18         # get images
  19         # ---- your solution here ---
  20         # set ThreadPoolExecutor
  21         with # ---- your solution here ---:
            # submit the threads and get a
 
22 list of futures
  23             # ---- your solution here ---
            # call as_complete() method for
 
24 result
  25             # ---- your solution here ---
  26         # set end time
  27         # ---- your solution here ---
  28     except:
        print("An Error occurred in
 
29 resizing.")
  30     else:
  31         # print elapsed time
  32         # ---- your solution here ---
  33  
  34 # call resize all function
  35 resize_all_images()
     
[4]:   …
    The Matrix.jpg resized.
    The Dark Knight.jpg resized.
    Fight Club.jpg resized.
Images resized in 0.63 sec -
   
ProcessPoolExecutor.
 
Q5:
Redefine the resize_all_images() function you
defined in Q1.
Use ProcessPoolExecutor with map() function in this
question.
Set a with context manager using
ProcessPoolExecutor and map() the threads.
Keep track of time again and print the final result.
 
Hints:
* ProcessPoolExecutor()
* map(function, iterable)
 
[5]: 1 # Q 5:
  2  
from concurrent.futures import
 
3 ProcessPoolExecutor
  4  
  5 # create a threads list
  6 threads = []
  7  
  8 # redefine resize all images fn
  9 def resize_all_images():
  10     """Creates the directory.
        Gets images and call resize fn one
 
11 by one.
  12         Keeps track of elapsed time."""
  13     try:
  14         # set start time
  15         # ---- your solution here ---
  16         # create directory
  17         # ---- your solution here ---
  18         # get images
  19         # ---- your solution here ---
   20         # set ThreadPoolExecutor
  21         with # ---- your solution here ---:
  22             # map the threads
  23             # ---- your solution here ---
  24         # set end time
  25         # ---- your solution here ---
  26     except:
        print("An Error occurred in
 
27 resizing.")
  28     else:
  29         # print elapsed time
  30         # ---- your solution here ---
  31  
  32 # call resize all function
  33 resize_all_images()
     
[5]:   …
    Goodfellas.jpg resized.
    The Dark Knight.jpg resized.
    Fight Club.jpg resized.
Images resized in 0.61 sec-
   
ProcessPoolExecutor-Map.
 

OceanofPDF.com
 
SOLUTIONS – Concurrency
 
Here are the solutions for the quiz for
this chapter.
 
S1:
 
[1]: 1 # S 1:
  2  
  3 # import packages
  4 import os
  5 from pathlib import Path
  6 import time
  7 # install Pillow package for PIL
  8 from PIL import Image
  9  
  10 # directory names
  11 images_directory = "images"
  12 thumbnails_directory = "thumbnails"
  13  
  14 # get images fn
  15 def get_images():
  16     """Returns a list of .jpg files
  17     in the images folder"""
  18     images = []
  19     # loop in the images folder
  20     for file in os.listdir(images_directory):
  21         # check if it is a .jpg file
  22         if file.endswith(".jpg"):
  23             images.append(file)
  24     return images
  25  
  26 # create directory fn
  27 def create_directory(name):
  28     """Creates the directory with name.
  29     No errors if the directory exists."""
  30     Path(name).mkdir(exist_ok=True)
  31  
  32 # resize all images fn
  33 def resize_all_images():
  34     """Creates the directory.
        Gets images and call resize fn one
 
35 by one.
  36         Keeps track of elapsed time."""
  37     try:
  38         # set start time
  39         start = time.perf_counter()
  40         # create directory
       
 
41 create_directory(thumbnails_directory)
  42         # get images
  43         images = get_images()
  44         # resize each image one by one
  45         for image in images:
  46             resize_image(image)
  47         # set end time
  48         end = time.perf_counter()
  49     except:
        print("An Error occurred in
 
50 resizing.")
  51     else:
  52         # print elapsed time
  53         print(f"Images resized in {end -
 start:.2f} sec.")
  54  
  55 # resize a single image fn
  56 def resize_image(image):
  57     # open the image
    im = Image.open(images_directory +
 
58 "/" + image)
  59     # resize the image
    resized_im =
  im.resize((round(im.size[0] * 0.1),
60 round(im.size[1] * 0.1)))
  61     # Save the resized image
    resized_im.save(thumbnails_directory
 
62 + "/" + image)
  63     # print result
  64     print(f"{image} resized.")
  65  
  66 # call resize all function
  67 resize_all_images()
     
[1]:   …
    The Godfather: Part II.jpg resized.
    The Dark Knight.jpg resized.
    Inception.jpg resized.
    Images resized in 1.08 sec.
 
S2:
 
[2]: 1 # S 2:
  2  
  3 from threading import Thread
  4  
  5 # create a threads list
  6 threads = []
  7  
  8 # redefine resize all images fn
  9 def resize_all_images():
  10     """Creates the directory.
        Gets images and call resize fn one
 
11 by one.
  12         Keeps track of elapsed time."""
  13     try:
  14         # set start time
  15         start = time.perf_counter()
  16         # create directory
       
 
17 create_directory(thumbnails_directory)
  18         # get images
  19         images = get_images()
  20         # fill the threads list
  21         for image in images:
            thread =
  Thread(target=resize_image, args=
22 (image,))
  23             thread.start()
  24             threads.append(thread)
  25         # force for wait for finish
  26         for thread in threads:
  27             thread.join()
  28         # set end time
  29         end = time.perf_counter()
  30     except:
        print("An Error occurred in
 
31 resizing.")
  32     else:
  33         # print elapsed time
  34         print(f"Images resized in {end -
start:.2f} sec - Threading.")
  35  
  36 # call resize all function
  37 resize_all_images()
     
[2]:   …
    The Dark Knight.jpg resized.
    The Matrix.jpg resized.
    Fight Club.jpg resized.
    Images resized in 0.29 sec - Threading.
 
S3:
 
[3]: 1 # S 3:
  2  
from concurrent.futures import
 
3 ThreadPoolExecutor, as_completed
  4  
  5 # create a threads list
  6 threads = []
  7  
  8 # redefine resize all images fn
  9 def resize_all_images():
  10     """Creates the directory.
        Gets images and call resize fn one
 
11 by one.
  12         Keeps track of elapsed time."""
  13     try:
  14         # set start time
  15         start = time.perf_counter()
  16         # create directory
  17        
 create_directory(thumbnails_directory)
  18         # get images
  19         images = get_images()
  20         # set ThreadPoolExecutor
        with ThreadPoolExecutor() as
 
21 executor:
            # submit the threads and get a
 
22 list of futures
            future_list =
 
23 [executor.submit(resize_image, image)
  24                            for image in images]
            # call as_complete() method for
 
25 result
            for future in
 
26 as_completed(future_list):
  27                 future.result()
  28         # set end time
  29         end = time.perf_counter()
  30     except:
        print("An Error occurred in
 
31 resizing.")
  32     else:
  33         # print elapsed time
        print(f"Images resized in {end -
 
34 start:.2f} sec - ThreadPoolExecutor.")
  35  
  36 # call resize all function
  37 resize_all_images()
     
[3]:   …
    The Dark Knight.jpg resized.
    The Matrix.jpg resized.
    Fight Club.jpg resized.
    Images resized in 0.33 sec -
ThreadPoolExecutor.
 
S4:
 
[4]: 1 # S 4:
  2  
from concurrent.futures import
 
3 ProcessPoolExecutor, as_completed
  4  
  5 # create a threads list
  6 threads = []
  7  
  8 # redefine resize all images fn
  9 def resize_all_images():
  10     """Creates the directory.
        Gets images and call resize fn one
 
11 by one.
  12         Keeps track of elapsed time."""
  13     try:
  14         # set start time
  15         start = time.perf_counter()
  16         # create directory
       
 
17 create_directory(thumbnails_directory)
  18         # get images
  19         images = get_images()
  20         # set ThreadPoolExecutor
        with ProcessPoolExecutor() as
 
21 executor:
            # submit the threads and get a
 
22 list of futures
            future_list =
 
23 [executor.submit(resize_image, image)
  24                            for image in images]
            # call as_complete() method for
 
25 result
            for future in
 
26 as_completed(future_list):
  27                 future.result()
  28         # set end time
  29         end = time.perf_counter()
  30     except:
        print("An Error occurred in
 
31 resizing.")
  32     else:
  33         # print elapsed time
        print(f"Images resized in {end -
 
34 start:.2f} sec - ProcessPoolExecutor.")
  35  
  36 # call resize all function
  37 resize_all_images()
     
[4]:   …
    The Matrix.jpg resized.
    The Dark Knight.jpg resized.
    Fight Club.jpg resized.
Images resized in 0.63 sec -
   
ProcessPoolExecutor.
 
S5:
 
[5]: 1 # S 5:
  2  
from concurrent.futures import
 
3 ProcessPoolExecutor
  4  
  5 # create a threads list
  6 threads = []
  7  
  8 # redefine resize all images fn
  9 def resize_all_images():
  10     """Creates the directory.
        Gets images and call resize fn one
 
11 by one.
  12         Keeps track of elapsed time."""
  13     try:
  14         # set start time
  15         start = time.perf_counter()
  16         # create directory
       
 
17 create_directory(thumbnails_directory)
  18         # get images
  19         images = get_images()
  20         # set ThreadPoolExecutor
        with ProcessPoolExecutor() as
 
21 executor:
  22             # map the threads
            executor.map(resize_image,
 
23 images)
  24         # set end time
  25         end = time.perf_counter()
  26     except:
        print("An Error occurred in
 
27 resizing.")
  28     else:
  29         # print elapsed time
        print(f"Images resized in {end-
 
30 start:.2f} sec-ProcessPoolExecutor-Map.")
  31  
  32 # call resize all function
   33 resize_all_images()
     
[5]:   …
    Goodfellas.jpg resized.
    The Dark Knight.jpg resized.
    Fight Club.jpg resized.
Images resized in 0.61 sec-
   
ProcessPoolExecutor-Map.
 
 

OceanofPDF.com
. Project 2 – Web Scraping with
       

Python
Project Setup
 
In this project, we will learn how to
scrap data out of web pages using Python.
You can find the PyCharm project for this
chapter in the GitHub Repository of this
book.
 
Chapter Outline:
Install Scrapy
Create a Scrapy Project
Define Our First Spider
Run Our Spider
Extracting Data
Extracting Data in Our Spider
Storing the Scraped Data
Following Links
Shortcut for Creating Requests
 
 
OceanofPDF.com
 Install Scrapy
 
In this project we will use Scrapy as our
primary tool for web crawling. Scrapy is an
open source and collaborative framework
built with Python, for extracting the data
you need from websites. It is an application
framework for crawling web sites and
extracting structured data which can be
used for a wide range of useful applications,
like data mining, information processing or
historical archival. Scrapy is one of the most
widely used tool for large web crawling
projects.
To install Scrapy we will simply use our
PyCharm IDE. Here you can find the
complete installation document. Let’s start
by creating a new PyCharm project named
WebScraping . Make sure you create your
project with a new virtual environment.
Inside PyCharm, navigate to Preferences ,
and under Project sub-section on the left,
click P ython Interpreter . Here we will install
Scrapy package. See the image below:
 
 Figure 15-1: Installing Scrapy in PyCharm IDE
 
Scrapy has some dependencies and
PyCharm will install them too. So, it might
take a bit long to install Scrapy and all the
dependencies.
Scrapy is written in pure Python and
depends on a few key Python packages
(among others):
lxml, an efficient XML and HTML
parser
parsel, an HTML/XML data extraction
library written on top of lxml,
 w3lib, a multi-purpose helper for
dealing with URLs and web page
encodings
twisted, an asynchronous networking
framework
cryptography and pyOpenSSL, to deal
with various network-level security
needs
 
 
OceanofPDF.com
 Create a Scrapy Project
 
Before we start using Scrapy we need to
set up a new Scrapy project. In our
WebScraping PyCharm project we will use
Python terminal to create and run Scrapy
project. You can have one or multiple
Scrapy projects in a single PyCharm
project. Let’s create the first one:
In PyCharm, right click on the project
name and select Open In -> Terminal :
 
 Figure 15-2: Open a Terminal Window in PyCharm
 
This will open a new Terminal Window in
PyCharm. Here is the command to create
a new Scrapy project:
 
  scrapy startproject firstscraping
 
You will see a new directory in your
PyCharm project named firstscraping when
you run this code.
  

Figure 15-3: Scrapy project inside PyCharm project

 
Here are the files in our Scrapy project:
 
  firstscraping/
    scrapy.cfg            # deploy configuration
  file
    firstscraping/        # project's Python
  module, you'll import your code from here
          __init__.py
        items.py          # project items definition
  file
        middlewares.py    # project middlewares
  file
          pipelines.py      # project pipelines file
          settings.py       # project settings file
        spiders/          # a directory where you'll
  later put your spiders
              __init__.py
 
Now that we successfully create a
Scrapy project, let’s start by defining our
first Spider class.
 
 
OceanofPDF.com
 Define Our First Spider
 
Spiders are classes which we define and
which Scrapy uses to scrape information
from a website (or a group of websites).
They must inherit from scrapy.Spider class
and define the initial requests to make,
optionally how to follow links in the pages,
and how to parse the downloaded page
content to extract data.
To learn the basic concepts of Scrapy we
will be working on a dedicated web site
which is https://quotes.toscrape.com. It is
a simple web site, which has a list of
famous quotes on different topics.
Create a new Python file named
quotes_spider.py . under the
firstscraping/spiders directory. And here is
the content of this file:
 
[1]: 1 # quotes_spider.py file
  2  
  3 from scrapy import Spider, Request
  4  
  5 class QuotesSpider(Spider):
  6     # unique identifier of this spider
  7     name = "quotes"
  8  
  9     # begin crawl and return an iterable
  10     def start_requests(self):
  11         urls = [
           
 
12 'https://quotes.toscrape.com/page/1/',
           
 
13 'https://quotes.toscrape.com/page/2/',
  14         ]
  15         for url in urls:
            yield Request(url=url,
 
16 callback=self.parse)
  17  
    # callback method to handle
 
18 responses
  19     def parse(self, response):
        # get the page number from
 
20 response
  21         page = response.url.split("/")[-2]
        # create a filename as quotes-
 
22 1.html
  23         filename = f'quotes-{page}.html'
  24         # write into this file
  25         with open(filename, 'wb') as f:
  26             f.write(response.body)
  27         # log the activity
  28         self.log(f'Saved file {filename}')
 
In cell 1, we define our first Spider
subclass. It inherits from scrapy.Spider and
defines some attributes and methods:
 name : identifies the Spider. It must be
unique within a project, that is, you cannot
set the same name for different Spiders.
start_requests() : must return an iterable
of Requests (you can return a list of
requests or write a generator function)
which the Spider will begin to crawl from.
Subsequent requests will be generated
successively from these initial requests.
parse() : a method that will be called to
handle the response downloaded for each
of the requests made. The response
parameter is an instance of TextResponse
that holds the page content and has
further helpful methods to handle it.
The parse() method usually parses the
response, extracting the scraped data as
dicts and also finding new URLs to follow
and creating new requests ( Request ) from
them.
 
Now that we have a Spider defined, let’s
run it and do our first web scraping.
 
 
OceanofPDF.com
 Run Our Spider
 
To run a spider in Scrapy, we need to go
to Scrapy project’s top-level directory
( firstscraping ) and run the following
command in the terminal:
 
  scrapy crawl quotes
 
This will run the spider and create two
.html files in the Scrapy project directory.
See the image below:
 

Figure 15-4: Running the Spider in firstscraping folder

  
You should see two html files as: quotes-
1.html and quotes-2.html . These are the page
contents of the respective URLs. Our parse
method gets and saves them.
The start_requests() method in line 10,
loops over each url in the urls list. And it
schedules an object of type scrapy.Request
for each one. When it receives the response
for each one, it instantiates a Response
object and calls the callback method, which
is the parse() method, associated with the
request.
The parse() method simply gets the
response, splits it and save its body into
local html files.
 
A shortcut to the start_requests() method:
 
The main idea of the start_requests()
method is to take a list of URLs list and loop
through each item to initiate a Request
object as: Request(url=url,
callback=self.parse) . However, we do not
need to implement this method. There is a
shortcut version, that can achieve the same
result. There is a built-in attribute called
start_urls which is the list of the URLs we
want to crawl. If you define this list, Scrapy
will create the respective Request objects for
each item in it. Just like we did with
implementing the start_requests() method.
Let’s redefine the QuotesSpider class by
using start_urls this time:
 
[2]: 1 # redefine the same spider with start_urls
  2 class QuotesSpider(Spider):
  3     # unique identifier of this spider
  4     name = "quotes"
    # start_urls for url list to crawl
 
5 automatically
  6     start_urls = [
  7         'https://quotes.toscrape.com/page/1/',
  8         'https://quotes.toscrape.com/page/2/',
  9     ]
  10  
  11     # callback method to handle responses
  12     def parse(self, response):
  13         # get the page number from response
  14         page = response.url.split("/")[-2]
  15         # create a filename as quotes-1.html
  16         filename = f'quotes-{page}.html'
  17         # write into this file
  18         with open(filename, 'wb') as f:
  19             f.write(response.body)
 
The code in cell 2 gives exactly the same
output with the cell 1. But with less lines of
code. The parse() method will be called to
handle each of the requests for those URLs
in the start_urls list, even though we haven’t
explicitly told Scrapy to do so. This happens
because parse() is the default callback
method, which is called for requests without
an explicitly assigned callback.
 
 
OceanofPDF.com
 Scrapy Shell
 
Before we start to extract data using Scrapy, there is an
important tool which you should learn, which is the Scrapy
shell. The Scrapy shell is an interactive shell where you
can try and debug your scraping code very quickly, without
having to run the spider. It’s meant to be used for testing
data extraction code, but you can actually use it for testing
any kind of code as it is also a regular Python shell.
The shell is used for testing XPath or CSS expressions and
see how they work and what data they extract from the web
pages you’re trying to scrape. It allows you to interactively
test your expressions while you’re writing your spider,
without having to run the spider to test every change.
Once you get familiarized with the Scrapy shell, you’ll see
that it’s an invaluable tool for developing and debugging
your spiders.
To launch the shell simply type the command below in
your terminal which you run Scrapy:
 
  scrapy shell <url>
 
So, to be able to inspect one of the URLs we used in the
previous section, we need to run the command below in the
firstscraping project:
 
  scrapy shell 'https://quotes.toscrape.com/page/1/'
 
Please keep in mind that, on Windows, use double quotes
instead of single ones. When we run this command
successfully, we will have an interactive Scrapy shell.
Here is what we can do with it:
shelp() - print a help with the list of available objects
and shortcuts
fetch(url[, redirect=True]) - fetch a new response from
the given URL and update all related objects
accordingly. You can optionally ask for HTTP 3xx
 redirections to not be followed by passing
redirect=False
fetch(request) - fetch a new response from the given
request and update all related objects accordingly.
view(response) - open the given response in your local
web browser, for inspection. This will add a <base>
tag to the response body in order for external links
(such as images and style sheets) to display properly.
Note, however, that this will create a temporary file in
your computer, which won’t be removed
automatically.
 
Let’s type view(response) command in the Scrapy shell.
When you type this command and press Enter, Scrapy shell
will open the response html file in your default web browser.
This is a temporary file.
The shell creates objects from the downloaded page, like
the Response object and the Selector objects (for both HTML
and XML content).
Those objects are:
crawler - the current Crawler object.
spider - the Spider which is known to handle the URL,
or a Spider object if there is no spider found for the
current URL
request - a Request object of the last fetched page.
You can modify this request using replace() or fetch a
new request (without leaving the shell) using the
fetch shortcut.
response - a Response object containing the last
fetched page
settings - the current Scrapy settings
 
 Figure 15-5: Scrapy shell (and the objects) in the terminal
 
Now that we have an active Scrapy shell, we can start
playing with objects. We will use XPath expressions here just
to get an idea of selecting objects in Scrapy:
 
Get the page title: '//title/text()'
 
  >>> response.xpath('//title/text()').get()
  'Quotes to Scrape'
 
Fetch another web site
 
  >>> fetch("https://www.amazon.com/Best-Sellers/zgbs")
2022-04-22 22:13:36 [scrapy.core.engine] DEBUG: Crawled (200)
  <GET https://www.amazon.com/robots.txt> (referer: None)
2022-04-22 22:13:37 [scrapy.core.engine] DEBUG: Crawled (200)
  <GET https://www.amazon.com/Best-Sellers/zgbs> (referer: None)
 
Get the page title of the last request (to Amazon best
sellers)
 
  >>> response.xpath('//title/text()').get()
  'Amazon.com Best Sellers: The most popular items on Amazon'
 
Print response headers
 
  >>> from pprint import pprint
  >>> pprint(response.headers)
{b'Accept-Ch': [b'ect,rtt,downlink,device-memory,sec-ch-device-
  memory,viewport'
                  b'-width,sec-ch-viewport-width,dpr,sec-ch-dpr' ],
  b'Accept-Ch-Lifetime': [b'86400'],
  b'Cache-Control': [b'no -cache'],
  b'Content-Language': [b'en-US'],
  b'Content-Type' : [b'text/html;charset=UTF-8'],
  b'Date': [b'Fri, 22 Apr 2022 19:13:37 GMT'],
  b'Expires': [b'-1'],
  b'Permissions-Policy': [b'interest-cohort=()'],
  b'Pragma': [b'no -cache'],
  b'Server': [b'Server'],
b'Strict-Transport-Security' : [b'max-age=47474747;
  includeSubDomains; preload'],
b'Vary': [b'Content-Type,Accept-Encoding,X-Amzn-CDN-Cache,X-Amzn-
  AX-Treatmen'
             b't,User-Agent' ],
b'Via': [b'1.1 9987fa8ab620895e83d1d8f10c40f6d2.cloudfront.net
  (CloudFront)'],
b'X-Amz-Cf-Id' :
  [b'bW1ADijyxFtP4Pa0iLGZ2_z_w_oURnvnrwnDcmDG_UGtfsFviKORXg=='],
  b'X-Amz-Cf-Pop' : [b'FRA56-P4'],
  b'X-Amz-Rid' : [b'Y0FBYQZ58BTDJWZEMPA3'],
  b'X-Cache': [b'Miss from cloudfront'],
  b'X-Content-Type- Options' : [b'nosniff'],
  b'X-Frame- Options': [b'S AMEORIGIN'],
  b'X-Xss-Protection' : [b'1;']}
 
 
OceanofPDF.com
 Extracting Data
 
Now that we have a basic
understanding of how Scrapy works, what
a Spider is and how to use Scrapy shell,
we can start data extraction. The ideal
approach is to see what we will extract.
We will run a shell on one of our URLs and
get some data using CSS selectors.
Open a new terminal, navigate to your
Scrapy project top-level folder and type
the shell command as:
 
scrapy shell
  'https://quotes.toscrape.com/page/1/'
 
Let’s use some CSS selectors to extract
data:
 
  >>> response.css("title")
[<Selector xpath='descendant- or-self::title'
  data='<title>Quotes to Scrape</title>'>]
 
We use response.css() method to use
CSS selectors and get data. Here we
simply ask for the title tag which is
<title> . And as you see in the result, we
get this tag. And the result is a list-like
object called SelectorList . SelectorList
represents a list of Selector objects that
wrap around XML/HTML elements and
allow us to run further queries to fine-
grain data extraction. Let’s extract just the
text from title tag above:
 
  >>> response.css('title::text').getall()
  ['Quotes to Scrape']
 
We added “ ::text ” to the CSS query
above. This means that we want to select
only the text elements directly inside the
<title> element. And getall() method is
used to extract all the elements satisfying
this query. If we want to get only the first
result, we can simply call the get() method
instead. Here is an example:
 
  >>> response.css('title::text')[0].get()
  'Quotes to Scrape'
 
C SS Selectors :
Selectors are patterns that match
against elements in a tree, and as such
form one of several technologies that can
be used to select nodes in a document.
Selectors have been optimized for use
with HTML and XML, and are designed to
be usable in performance-critical code.
They are a core component of CSS
(Cascading Style Sheets), which uses
Selectors to bind style properties to
elements in the document.
 
XPath :
Scrapy supports XPath expressions in
addition to CSS selectors. XPath
expressions are the foundation of Scrapy
Selectors. CSS selectors are converted to
XPath under-the-hood. You can see that if
you read closely the text representation of
the selector objects in the shell.
 
Now we are ready to complete our first
spider by adding some code to extract the
quotes from our example web site.
In the https://quotes.toscrape.com each
quote is represented by HTML elements
which look like the following:
 
  <div class="quote">
      <span class="text">“The world as we have
created it is a process of our
    thinking. It cannot be changed without
  changing our thinking.”</span>
      <span>
        by <small class="author">Albert
  Einstein</small>
        <a href="/author/Albert-Einstein">
  (about)</a>
      </span>
      <div class="tags">
          Tags:
        <a class="tag"
  href="/tag/change/page/1/">change</a>
        <a class="tag" href="/tag/deep-
  thoughts/page/1/">deep-thoughts</a>
        <a class="tag"
  href="/tag/thinking/page/1/">thinking</a>
        <a class="tag"
  href="/tag/world/page/1/">world</a>
      </div>
  </div>
 
Let’s start a Scrapy shell on this website
and see how we can extract data out of it:
 
  scrapy shell 'https://quotes.toscrape.com'
 
And let’s extract the div element which
has the class name as “quote” . The CSS
selector for this is: div.quote .
  
  >>> quote = response.css("div.quote")[0]
  >>> quote
<Selector xpath="descendant- or-
self::div[@class and contains(concat(' ',
normalize-space(@class), ' '), ' quote ')]"
data='<div class="quote" itemscope
  itemtype...'>
 
We get the first div element with class
name as “quote” and assign to a local
variable called quote . Then we simply
print it to the shell.
Let’s get more data. Let’s extract text,
author and the tags from that quote using
the quote object we created:
 
  >>> text = quote.css("span.text::text").get()
  >>> text
'“The world as we have created it is a process
of our thinking. It cannot be changed without
  changing our thinking.”'
>>> author =
  quote.css("small.author::text").get()
  >>> author
  'Albert Einstein'
>>> tags = quote.css("div.tags
  a.tag::text").getall()
  >>> tags
  ['change', 'deep-thoughts', 'thinking', 'world']
  
Now that, we know how a quote looks
like in the html response, we can set a
loop to print the data of each one:
 
  >>> for quote in response.css("div.quote"):
  ...     text = quote.css("span.text::text").get()
...     author =
  quote.css("small.author::text").get()
...     tags = quote.css("div.tags
  a.tag::text").getall()
...     print(dict(text=text, author=author,
  tags=tags))
  ...
{'text': '“The world as we have created it is a
process of our thinking. It cannot be changed
without changing our thinking.”', 'author':
'Albert Einstein', 'tags': ['change', 'deep-
  thoughts', 'thinking', 'world']}
{'text': '“It is our choices, Harry, that show what
we truly are, far more than our abilities.”',
'author': 'J.K. Rowling', 'tags': ['abilities',
  'choices']}
{'text': '“There are only two ways to live your
life. One is as though nothing is a miracle. The
other is as though everything is a miracle.”',
'author': 'Albert Einstein', 'tags': ['inspirational',
  'life', 'live', 'miracle', 'miracles']}
{'text': '“The person, be it gentleman or lady,
who has not pleasure in a good novel, must be
intolerably stupid.”', 'author': 'Jane Austen',
  'tags': ['aliteracy', 'books', 'classic', 'humor']}
  {'text': "“Imperfection is beauty, madness is
 genius and it's better to be absolutely ridiculous
than absolutely boring.”", 'author': 'Marilyn
Monroe', 'tags': ['be-yourself' , 'inspirational']}
{'text': '“Try not to become a man of success.
Rather become a man of value.”', 'author':
'Albert Einstein', 'tags': ['adulthood', 'success',
  'value']}
{'text': '“It is better to be hated for what you
are than to be loved for what you are not.”',
  'author': 'André Gide', 'tags': ['life', 'love']}
{'text': "“I have not failed. I've just found
10,000 ways that won't work.”", 'author':
'Thomas A. Edison', 'tags': ['edison', 'failure',
  'inspirational', 'paraphrased']}
{'text': "“A woman is like a tea bag; you never
know how strong it is until it's in hot water.”",
'author': 'Eleanor Roosevelt', 'tags':
  ['misattributed-eleanor-roosevelt']}
{'text': '“A day without sunshine is like, you
know, night.”', 'author': 'Steve Martin', 'tags':
  ['humor', 'obvious', 'simile']}
 
 
OceanofPDF.com
 Extracting Data in Our Spider
 
So far, we learned how to extract data
in Scrapy shell. Now, it’s time to do the
same thing in our spider. A Scrapy spider
can be used to generate dictionaries
containing the data extracted from the
page. To do that, we use the yield keyword
in the callback. Let’s define a new Spider
named QuoteSpiderExtractor inside a new
Python file named quote_spider_extractor.py
under the “spiders” folder.
 
[3]: 1 # quote_spider_extractor.py file
  2  
  3 from scrapy import Spider, Request
  4  
  5 class QuoteSpiderExtractor(Spider):
  6     # unique identifier of this spider
  7     name = "quotes_extractor"
    # start_urls for url list to crawl
 
8 automatically
  9     start_urls = [
       
 
10 'https://quotes.toscrape.com/page/1/',
       
 
11 'https://quotes.toscrape.com/page/2/',
  12     ]
  13  
  14     # callback method to handle
responses
  15     def parse(self, response):
        # get the div elements with class
 
16 name quote
        for quote in
 
17 response.css('div.quote'):
  18             # yield a quote dictionary
  19             yield {
                'text':
 
20 quote.css('span.text::text').get(),
                'author':
 
21 quote.css('small.author::text').get(),
                'tags': quote.css('div.tags
 
22 a.tag::text').getall()
  23             }
 
Let’s run this spider and see the output.
Data will be extracted in the logs in
terminal. And you will see a dictionary for
each quote.
 
  scrapy crawl quotes_extractor
 
  …
2022-04-23 10:43:25 [scrapy.core.scraper]
DEBUG: Scraped from <200
  https://quotes.toscrape.com/page/1/>
{'text': '“It is our choices, Harry, that show what
we truly are, far more than our abilities.”',
'author': 'J.K. Rowling', 'tags': ['abilities',
  'choices']}
  2022-04-23 10:43:25 [scrapy.core.scraper]
DEBUG: Scraped from <200
https://quotes.toscrape.com/page/1/>
{'text': '“There are only two ways to live your
life. One is as though nothing is a miracle. The
other is as though everything is a miracle.”',
'author': 'Albert Einstein', 'tags': ['inspirational',
  'life', 'live', 'miracle', 'miracles']}
2022-04-23 10:43:25 [scrapy.core.scraper]
DEBUG: Scraped from <200
  https://quotes.toscrape.com/page/1/>
{'text': '“The person, be it gentleman or lady,
who has not pleasure in a good novel, must be
intolerably stupid.”', 'author': 'Jane Austen',
  'tags': ['aliteracy', 'books', 'classic', 'humor']}
2022-04-23 10:43:25 [scrapy.core.scraper]
DEBUG: Scraped from <200
  https://quotes.toscrape.com/page/1/>
{'text': "“Imperfection is beauty, madness is
genius and it's better to be absolutely ridiculous
than absolutely boring.”", 'author': 'Marilyn
  Monroe', 'tags': ['be-yourself' , 'inspirational']}
  …
 
 
OceanofPDF.com
 Storing the Scraped Data
 
Now let’s assume we want to store the
scraped data instead of just printing them
to the log console. The simplest way to
store the scraped data is by using Feed
exports, with the following command:
 
scrapy crawl quotes_extractor -O
  quotes_extractor.json
 
This command will generate a
quotes_extractor.json file containing all
scraped items, serialized in JSON. The -O
(capital letter ‘ O ’) command-line switch
overwrites any existing file; use -o (lower
case letter ‘ o ’) instead to append new
content to any existing file.
 
You should see a new file in your Scrapy
project. See the image below:
 
 Figure 15-6: quotes_extractor.json file in Scrapy project
 
 
OceanofPDF.com
 Following Links
 
In the previous sections we mainly
scraped a single page. Now, let’s see how
we can scrape all the pages from
https://quotes.toscrape.com web site. To do
this, we need to learn how to follow the
links. In most cases, the links are just <a>
tags.
To extract a link in an HTML structure, we
need to examine it. Here is, how we can
examine the underlying HTML of a page in
Google Chrome. Right click on the element
you want to examine and click “Inspect”.
See the image below:
 
 Figure 15-7: Inspecting elements in Google Chrome
 
When you inspect an element in Google
Chrome, it will open the Chrome Developer
Tools extension. Inside the extension, under
the Elements tab, you can see the entire
HTML structure of the element.
The link which we want to inspect is the
“Next” button, which contains a link to the
second page. We need this element
because we want to get the next page in
our Spider. Here is the HTML code of this
element:
 
  <ul class="pager">
      <li class="next">
        <a href="/page/2/">Next <span aria-
  hidden="true">→</span></a>
      </li>
  </ul>
 
The link here is the href attribute of the
<a> tag and let’s extract it in our Spider
now. Create a new Spider named
QuoteSpiderFollowLinks inside a new Python
file named quote_spider_follow_links.py under
the “spiders” folder.
 
[4]: 1 # quote_spider_follow_links.py file
  2  
  3 from scrapy import Spider, Request
  4  
  5 class QuoteSpiderFollowLinks(Spider):
  6     # unique identifier of this spider
  7     name = "quotes_follow_links"
    # start_urls for url list to crawl
 
8 automatically
  9     start_urls = [
  10         'https://quotes.toscrape.com/page/1/',
  11     ]
  12  
  13     # callback method to handle responses
  14     def parse(self, response):
        # get the div elements with class
 
15 name quote
        for quote in
 
16 response.css('div.quote'):
  17             # yield a quote dictionary
  18             yield {
                'text':
 
19 quote.css('span.text::text').get(),
                'author':
 
20 quote.css('small.author::text').get(),
                'tags': quote.css('div.tags
 
21 a.tag::text').getall(),
  22             }
        # get next page element via the href
 
23 attribute
        next_page = response.css('li.next
 
24 a::attr(href)').get()
  25         if next_page is not None:
  26             # create the next page url
            next_page =
 
27 response.urljoin(next_page)
            # request the next page with
 
28 Scrapy Request
            yield Request(next_page,
 
29 callback=self.parse)
 
In cell 4, we define a Spider which follows
the links in a web site. After extracting the
data, the parse() method looks for the link
to the next page, builds a full absolute URL
using the urljoin() method (since the links
can be relative) and yields a new request to
the next page, registering itself as callback
to handle the data extraction for the next
page and to keep the crawling going
through all the pages.
 This shows us how Scrapy follows the
links. When it yields a Request in the
callback method, it will schedule that
request to be sent and register a callback
method to be executed when that request
finishes.
You can think this structure as some sort
of a loop, which follows all the links to the
next page until it doesn’t find one. This is
very useful especially with sites that have
pagination. You can implement a save
mechanism in the parse() method
depending on the page you visit.
 
 
OceanofPDF.com
 Shortcut for Creating Requests
 
In the previous section we used the
Request() constructor method to create
and yield a Request object in the parse()
method. However, we have a shortcut for
creating Request objects. The syntax is:
response.follow() . Let’s redefine the
QuoteSpiderFollowLinks class with this
shortcut this time:
 
[5]: 1 ####  response.follow()  ###
  2 class QuoteSpiderFollowLinks(Spider):
  3     # unique identifier of this spider
  4     name = "quotes_follow_links"
  5     # start_urls for url list to crawl
automatically
  6     start_urls = [
  7        
'https://quotes.toscrape.com/page/1/',
  8     ]
  9  
  10     # callback method to handle
responses
  11     def parse(self, response):
  12         # get the div elements with class
name quote
  13         for quote in
response.css('div.quote'):
  14             # yield a quote dictionary
  15             yield {
  16                 'text':
quote.css('span.text::text').get(),
  17                 'author':
quote.css('small.author::text').get(),
  18                 'tags': quote.css('div.tags
a.tag::text').getall(),
  19             }
  20         # get next page element via the
href attribute
  21         next_page = response.css('li.next
a::attr(href)').get()
  22         if next_page is not None:
  23             # yield a Request object with
response.follow()
  24             yield response.follow(next_page,
callback=self.parse)
 
While using scrapy.Request we need to
call the urljoin() method to create relative
URLs.  However, with response.follow()
method this is not necessary because it
supports relative URLs out of the box. In
line 24, we call response.follow() which
returns a Request instance. Then, we yield
this object.
It is also possible to pass a selector to
the response.follow() method, which makes
your code shorter:
 
[6]: 1 for href in response.css('ul.pager
a::attr(href)'):
    yield response.follow(href,
 
2 callback=self.parse)
 
One of the
best things about the
response.follow() method is that, it has a
built-in mechanism to follow the <a>
elements. It uses their href attributes
automatically for following. So, we do not
need to explicitly get the href attribute.
Here is how:
 
[7]: 1 for a in response.css('ul.pager a'):
    yield response.follow(a,
 
2 callback=self.parse)
 
In cell 7, we pass the <a> tag to the
response.follow() method and it uses the
href attributes automatically to follow.
 
response.follow_all(urls, callback) :
If you need to create multiple requests
from an iterable, there is a built-in method
which is response.follow_all() . It is a
generator that produces Request instances
to follow all links in urls parameter. Let’s
see how to use it:
 
[8]: 1 anchors = response.css('ul.pager a')
yield from response.follow_all(anchors,
 
2 callback=self.parse)
 
In cell 8, we first get all the anchor tags
( <a> ) in line 1. Then in the second line, we
call the response.follow_all() method by
passing these anchors. And since it is a
generator, we need to yield it as “yield
from” .
 
This finalizes our project on web
scraping with Python. Scrapy has lots of
features which will help you to build
complex and effective web crawlers.
However, for the sake of this project, we
believe it is enough to give you a basic
understating on how web scraping and
Scrapy works in Python.
 
In the next chapter, you will have an
assignment related to this project. Good
luck with your assignment.
  
 
OceanofPDF.com
. Assignment 2 – Web Scraping
       

with Python
 
OceanofPDF.com
 Assignment Setup
 
We finished the second project in this
book which is Project 2 – Web Scraping
with Python. Now it’s your turn to recreate
the same project. This chapter is the
assignment on the Project 2.
 
In this assignment you will start by
installing the Scrapy package. You will use
Scrapy Shell to inspect HTML structure of
web pages. You create Spider classes and
use them to extract and save scraped
data.
 
To complete the assignment, follow the
instructions where you see a #TODO
statement. You can find the solutions in
the previous chapter.
 
We will create a new PyCharm project.
The project name is WebScraping . You
should download the project from the
GitHub Repository of this book before
starting to code.
  
 
Chapter Outline:
Install Scrapy
Create a Scrapy Project
Define Our First Spider
Run Our Spider
Extracting Data
Extracting Data in Our Spider
Storing the Scraped Data
Following Links
Shortcut for Creating Requests
 
 
OceanofPDF.com
 Install Scrapy
 
In this project we will use Scrapy as our
primary tool for web crawling. Scrapy is an
open source and collaborative framework
built with Python, for extracting the data
you need from websites. It is an application
framework for crawling web sites and
extracting structured data which can be
used for a wide range of useful applications,
like data mining, information processing or
historical archival. Scrapy is one of the most
widely used tool for large web crawling
projects.
To install Scrapy we will simply use our
PyCharm IDE. Here you can find the
complete installation document. Let’s start
by creating a new PyCharm project named
WebScraping . Make sure you create your
project with a new virtual environment.
Inside PyCharm, navigate to Preferences ,
and under Project sub-section on the left,
click P ython Interpreter . Here we will install
Scrapy package. See the image below:
 
 Figure 15-1: Installing Scrapy in PyCharm IDE
 
Scrapy has some dependencies and
PyCharm will install them too. So, it might
take a bit long to install Scrapy and all the
dependencies.
Scrapy is written in pure Python and
depends on a few key Python packages
(among others):
lxml, an efficient XML and HTML
parser
parsel, an HTML/XML data extraction
library written on top of lxml,
 w3lib, a multi-purpose helper for
dealing with URLs and web page
encodings
twisted, an asynchronous networking
framework
cryptography and pyOpenSSL, to deal
with various network-level security
needs
 
 
OceanofPDF.com
 Create a Scrapy Project
 
Before we start using Scrapy we need to
set up a new Scrapy project. In our
WebScraping PyCharm project we will use
Python terminal to create and run Scrapy
project. You can have one or multiple
Scrapy projects in a single PyCharm
project. Let’s create the first one:
In PyCharm, right click on the project
name and select Open In -> Terminal :
 
 Figure 15-2: Open a Terminal Window in PyCharm
 
This will open a new Terminal Window in
PyCharm. Here is the command to create
a new Scrapy project:
 
  scrapy startproject firstscraping
 
You will see a new directory in your
PyCharm project named firstscraping when
you run this code.
  

Figure 15-3: Scrapy project inside PyCharm project

 
Here are the files in our Scrapy project:
 
  firstscraping/
    scrapy.cfg            # deploy configuration
  file
    firstscraping/        # project's Python
  module, you'll import your code from here
          __init__.py
        items.py          # project items definition
  file
        middlewares.py    # project middlewares
  file
          pipelines.py      # project pipelines file
          settings.py       # project settings file
        spiders/          # a directory where you'll
  later put your spiders
              __init__.py
 
Now that we successfully create a
Scrapy project, let’s start by defining our
first Spider class.
 
 
OceanofPDF.com
 Define Our First Spider
 
Spiders are classes which we define and
which Scrapy uses to scrape information
from a website (or a group of websites).
They must inherit from scrapy.Spider class
and define the initial requests to make,
optionally how to follow links in the pages,
and how to parse the downloaded page
content to extract data.
To learn the basic concepts of Scrapy we
will be working on a dedicated web site
which is https://quotes.toscrape.com. It is
a simple web site, which has a list of
famous quotes on different topics.
Create a new Python file named
quotes_spider.py . under the
firstscraping/spiders directory. And here is
the content of this file:
 
[1]: 1 # quotes_spider.py file
  2  
  3 from scrapy import Spider, Request
  4  
  5 class QuotesSpider(Spider):
  6     # unique identifier of this spider
  7     name = "quotes"
  8  
  9     # begin crawl and return an iterable
  10     def start_requests(self):
  11         urls = [
           
 
12 'https://quotes.toscrape.com/page/1/',
           
 
13 'https://quotes.toscrape.com/page/2/',
  14         ]
  15         for url in urls:
  16             # TODO - yield Request objects
  17  
    # callback method to handle
 
18 responses
  19     def parse(self, response):
        # get the page number from
 
20 response
        page = # TODO - get page number
 
21 from response.url
        # create a filename as quotes-
 
22 1.html
  23         filename = f'quotes-{page}.html'
  24         # write into this file
  25         with open(filename, 'wb') as f:
            # TODO - write the response
 
26 body
  27         # log the activity
  28         self.log(f'Saved file {filename}')
 
In cell 1, we define our first Spider
subclass. It inherits from scrapy.Spider and
defines some attributes and methods:
 name : identifies the Spider. It must be
unique within a project, that is, you can’t
set the same name for different Spiders.
start_requests() : must return an iterable
of Requests (you can return a list of
requests or write a generator function)
which the Spider will begin to crawl from.
Subsequent requests will be generated
successively from these initial requests.
parse() : a method that will be called to
handle the response downloaded for each
of the requests made. The response
parameter is an instance of TextResponse
that holds the page content and has
further helpful methods to handle it.
The parse() method usually parses the
response, extracting the scraped data as
dicts and also finding new URLs to follow
and creating new requests ( Request ) from
them.
Now that we have a Spider defined, let’s
run it and do our first web scraping.
 
 
OceanofPDF.com
 Run Our Spider
 
To run a spider in Scrapy, we need to go
to Scrapy project’s top-level directory
( firstscraping ) and run the following
command in the terminal:
 
  scrapy crawl quotes
 
This will run the spider and create two
.html files in the Scrapy project directory.
See the image below:
 

Figure 15-4: Running the Spider in firstscraping folder

  
You should see two html files as: quotes-
1.html and quotes-2.html . These are the page
contents of the respective URLs. Our parse
method gets and saves them.
The start_requests() method in line 10,
loops over each url in the urls list. And it
schedules an object of type scrapy.Request
for each one. When it receives the response
for each one, it instantiates a Response
object and calls the callback method, which
is the parse() method, associated with the
request.
The parse() method simply gets the
response, splits it and save its body into
local html files.
 
A shortcut to the start_requests() method:
 
The main idea of the start_requests()
method is to take a list of URLs list and
loops through each item to initiate a
Request object as: Request(url=url,
callback=self.parse) . However, we do not
need to implement this method. There is a
shortcut version, that can achieve the same
result. There is a built-in attribute called
start_urls which is the list of the URLs we
want to crawl. If you define this list, Scrapy
will create the respective Request objects for
each item in it. Just like we did with
implementing the start_requests() method.
Let’s redefine the QuotesSpider class by
using start_urls this time:
 
[2]: 1 # redefine the same spider with start_urls
  2 class QuotesSpider(Spider):
  3     # unique identifier of this spider
    name = # TODO - define a unique name
 
4 for spider
    # start_urls for url list to crawl
 
5 automatically
  6     start_urls = # TODO - define urls list
  7  
  8     # callback method to handle responses
  9     def parse(self, response):
        # get the page number from
 
10 response
        page = # TODO - get page number
 
11 from response.url
  12         # create a filename as quotes-1.html
        filename = # TODO - create a page
 
13 name
  14         # write into this file
        with # TODO - open filename in
 
15 write binary mode:
  16             # TODO - write the response body
 
 The code in cell 2 gives exactly the same
output with the cell 1. But with less lines of
code. The parse() method will be called to
handle each of the requests for those URLs
in the start_urls list, even though we haven’t
explicitly told Scrapy to do so. This happens
because parse() is the default callback
method, which is called for requests without
an explicitly assigned callback.
 
 
OceanofPDF.com
 Scrapy Shell
 
Before start to extract data using Scrapy, there is an
important tool which you should learn, which is the Scrapy
shell. The Scrapy shell is an interactive shell where you can
try and debug your scraping code very quickly, without
having to run the spider. It’s meant to be used for testing
data extraction code, but you can actually use it for testing
any kind of code as it is also a regular Python shell.
The shell is used for testing XPath or CSS expressions and
see how they work and what data they extract from the web
pages you’re trying to scrape. It allows you to interactively
test your expressions while you’re writing your spider,
without having to run the spider to test every change.
Once you get familiarized with the Scrapy shell, you’ll see
that it’s an invaluable tool for developing and debugging
your spiders.
To launch the shell simply type the command below in
your terminal which you run Scrapy:
 
  scrapy shell <url>
 
So, to be able to inspect one of the URLs we used in the
previous section, we need to run the command below in the
firstscraping project:
 
  scrapy shell 'https://quotes.toscrape.com/page/1/'
 
Please keep in mind that, on Windows, use double quotes
instead of single ones. When we run this command
successfully, we will have an interactive Scrapy shell.
Here is what we can do with it:
shelp() - print a help with the list of available objects
and shortcuts
fetch(url[, redirect=True]) - fetch a new response from
the given URL and update all related objects
accordingly. You can optionally ask for HTTP 3xx
 redirections to not be followed by passing
redirect=False
fetch(request) - fetch a new response from the given
request and update all related objects accordingly.
view(response) - open the given response in your local
web browser, for inspection. This will add a <base>
tag to the response body in order for external links
(such as images and style sheets) to display properly.
Note, however, that this will create a temporary file in
your computer, which won’t be removed
automatically.
 
Let’s type view(response) command in the Scrapy shell.
When you type this command and press Enter, Scrapy shell
will open the response html file in your default web browser.
This is a temporary file.
The shell creates objects from the downloaded page, like
the Response object and the Selector objects (for both HTML
and XML content).
Those objects are:
crawler - the current Crawler object.
spider - the Spider which is known to handle the URL,
or a Spider object if there is no spider found for the
current URL
request - a Request object of the last fetched page.
You can modify this request using replace() or fetch a
new request (without leaving the shell) using the
fetch shortcut.
response - a Response object containing the last
fetched page
settings - the current Scrapy settings
 
 Figure 15-5: Scrapy shell (and the objects) in the terminal
 
Now that we have an active Scrapy shell, we can start
playing with objects. We will use XPath expressions here just
to get an idea of selecting objects in Scrapy.
Get the page title: '//title/text()'
 
  >>> response.xpath('//title/text()').get()
  'Quotes to Scrape'
 
Fetch another web site
 
  >>> fetch("https://www.amazon.com/Best-Sellers/zgbs")
2022-04-22 22:13:36 [scrapy.core.engine] DEBUG: Crawled (200)
  <GET https://www.amazon.com/robots.txt> (referer: None)
2022-04-22 22:13:37 [scrapy.core.engine] DEBUG: Crawled (200)
  <GET https://www.amazon.com/Best-Sellers/zgbs> (referer: None)
 
Get the page title of the last request (to Amazon best
sellers)
 
  >>> response.xpath('//title/text()').get()
  'Amazon.com Best Sellers: The most popular items on Amazon'
 
Print response headers
  >>> from pprint import pprint
  >>> pprint(response.headers)
{b'Accept-Ch': [b'ect,rtt,downlink,device-memory,sec-ch-device-
  memory,viewport'
                  b'-width,sec-ch-viewport-width,dpr,sec-ch-dpr' ],
  b'Accept-Ch-Lifetime': [b'86400'],
  b'Cache-Control': [b'no -cache'],
  b'Content-Language': [b'en-US'],
  b'Content-Type' : [b'text/html;charset=UTF-8'],
  b'Date': [b'Fri, 22 Apr 2022 19:13:37 GMT'],
  b'Expires': [b'-1'],
  b'Permissions-Policy': [b'interest-cohort=()'],
  b'Pragma': [b'no -cache'],
  b'Server': [b'Server'],
b'Strict-Transport-Security' : [b'max-age=47474747;
  includeSubDomains; preload'],
b'Vary': [b'Content-Type,Accept-Encoding,X-Amzn-CDN-Cache,X-Amzn-
  AX-Treatmen'
             b't,User-Agent' ],
b'Via': [b'1.1 9987fa8ab620895e83d1d8f10c40f6d2.cloudfront.net
  (CloudFront)'],
b'X-Amz-Cf-Id' :
  [b'bW1ADijyxFtP4Pa0iLGZ2_z_w_oURnvnrwnDcmDG_UGtfsFviKORXg=='],
  b'X-Amz-Cf-Pop' : [b'FRA56-P4'],
  b'X-Amz-Rid' : [b'Y0FBYQZ58BTDJWZEMPA3'],
  b'X-Cache': [b'Miss from cloudfront'],
  b'X-Content-Type- Options' : [b'nosniff'],
  b'X-Frame- Options': [b'S AMEORIGIN'],
  b'X-Xss-Protection' : [b'1;']}
 
 
OceanofPDF.com
 Extracting Data
 
Now that we have a basic
understanding of how Scrapy works, what
a Spider is and how to use Scrapy shell,
we can start data extraction. The ideal
approach is to see what we will extract.
We will run a shell on one of our URLs and
get some data using CSS selectors.
Open a new terminal, navigate to your
Scrapy project top-level folder and type
the shell command as:
 
scrapy shell
  'https://quotes.toscrape.com/page/1/'
 
Let’s use some CSS selectors to extract
data:
 
  >>> response.css("title")
[<Selector xpath='descendant- or-self::title'
  data='<title>Quotes to Scrape</title>'>]
 
We use response.css() method to use
CSS selectors and get data. Here we
simply ask for the title tag which is
<title> . And as you see in the result, we
get his tag. And the result is a list-like
object called SelectorList . SelectorList
represents a list of Selector objects that
wrap around XML/HTML elements and
allow us to run further queries to fine-
grain data extraction. Let’s extract just the
text from title tag above:
 
  >>> response.css('title::text').getall()
  ['Quotes to Scrape']
 
We added ::text to the CSS query
above. This means that we want to select
only the text elements directly inside the
<title> element. And getall() method is
used to extract all the elements satisfying
this query. If we want to get only the first
result, we can simply call the get() method
instead. Here is an example:
 
  >>> response.css('title::text')[0].get()
  'Quotes to Scrape'
 
C SS Selectors :
Selectors are patterns that match
against elements in a tree, and as such
form one of several technologies that can
be used to select nodes in a document.
Selectors have been optimized for use
with HTML and XML, and are designed to
be usable in performance-critical code.
They are a core component of CSS
(Cascading Style Sheets), which uses
Selectors to bind style properties to
elements in the document.
 
XPath :
Scrapy supports XPath expressions in
addition to CSS selectors. XPath
expressions are the foundation of Scrapy
Selectors. CSS selectors are converted to
XPath under-the-hood. You can see that if
you read closely the text representation of
the selector objects in the shell.
 
Now we are ready to complete our first
spider by adding some code to extract the
quotes from our example web site.
In the https://quotes.toscrape.com each
quote is represented by HTML elements
which look like the following:
 
  <div class="quote">
      <span class="text">“The world as we have
created it is a process of our
    thinking. It cannot be changed without
  changing our thinking.”</span>
      <span>
        by <small class="author">Albert
  Einstein</small>
        <a href="/author/Albert-Einstein">
  (about)</a>
      </span>
      <div class="tags">
          Tags:
        <a class="tag"
  href="/tag/change/page/1/">change</a>
        <a class="tag" href="/tag/deep-
  thoughts/page/1/">deep-thoughts</a>
        <a class="tag"
  href="/tag/thinking/page/1/">thinking</a>
        <a class="tag"
  href="/tag/world/page/1/">world</a>
      </div>
  </div>
 
Let’s start a Scrapy shell on this website
and see how we can extract data out of it:
 
  scrapy shell 'https://quotes.toscrape.com'
 
And let’s extract the div element which
has the class name as “quote” . The CSS
selector for this is: div.quote .
  
  >>> quote = response.css("div.quote")[0]
  >>> quote
<Selector xpath="descendant- or-
self::div[@class and contains(concat(' ',
normalize-space(@class), ' '), ' quote ')]"
data='<div class="quote" itemscope
  itemtype...'>
 
We get the first div element with class
name as “quote” and assign to a local
variable called quote . Then we simply
print it to the shell.
Let’s get more data. Let’s extract text,
author and the tags from that quote using
the quote object we created:
 
  >>> text = quote.css("span.text::text").get()
  >>> text
'“The world as we have created it is a process
of our thinking. It cannot be changed without
  changing our thinking.”'
>>> author =
  quote.css("small.author::text").get()
  >>> author
  'Albert Einstein'
>>> tags = quote.css("div.tags
  a.tag::text").getall()
  >>> tags
  ['change', 'deep-thoughts', 'thinking', 'world']
  
Now that, we know how a quote looks
like in the html response, we can set a
loop to print the data of each one:
 
  >>> for quote in response.css("div.quote"):
  ...     text = quote.css("span.text::text").get()
...     author =
  quote.css("small.author::text").get()
...     tags = quote.css("div.tags
  a.tag::text").getall()
...     print(dict(text=text, author=author,
  tags=tags))
  ...
{'text': '“The world as we have created it is a
process of our thinking. It cannot be changed
without changing our thinking.”', 'author':
'Albert Einstein', 'tags': ['change', 'deep-
  thoughts', 'thinking', 'world']}
{'text': '“It is our choices, Harry, that show what
we truly are, far more than our abilities.”',
'author': 'J.K. Rowling', 'tags': ['abilities',
  'choices']}
{'text': '“There are only two ways to live your
life. One is as though nothing is a miracle. The
other is as though everything is a miracle.”',
'author': 'Albert Einstein', 'tags': ['inspirational',
  'life', 'live', 'miracle', 'miracles']}
{'text': '“The person, be it gentleman or lady,
who has not pleasure in a good novel, must be
intolerably stupid.”', 'author': 'Jane Austen',
  'tags': ['aliteracy', 'books', 'classic', 'humor']}
  {'text': "“Imperfection is beauty, madness is
 genius and it's better to be absolutely ridiculous
than absolutely boring.”", 'author': 'Marilyn
Monroe', 'tags': ['be-yourself' , 'inspirational']}
{'text': '“Try not to become a man of success.
Rather become a man of value.”', 'author':
'Albert Einstein', 'tags': ['adulthood', 'success',
  'value']}
{'text': '“It is better to be hated for what you
are than to be loved for what you are not.”',
  'author': 'André Gide', 'tags': ['life', 'love']}
{'text': "“I have not failed. I've just found
10,000 ways that won't work.”", 'author':
'Thomas A. Edison', 'tags': ['edison', 'failure',
  'inspirational', 'paraphrased']}
{'text': "“A woman is like a tea bag; you never
know how strong it is until it's in hot water.”",
'author': 'Eleanor Roosevelt', 'tags':
  ['misattributed-eleanor-roosevelt']}
{'text': '“A day without sunshine is like, you
know, night.”', 'author': 'Steve Martin', 'tags':
  ['humor', 'obvious', 'simile']}
 
 
OceanofPDF.com
 Extracting Data in Our Spider
 
So far, we learned how to extract data
in Scrapy shell. Now, it’s time to do the
same thing in our spider. A Scrapy spider
can be used to generate dictionaries
containing the data extracted from the
page. To do that, we use the yield keyword
in the callback. Let’s define a new Spider
named QuoteSpiderExtractor inside a new
Python file named quote_spider_extractor.py
under the “spiders” folder.
 
[3]: 1 # quote_spider_extractor.py file
  2  
# TODO - import Spider and Request
 
3 classes
  4 #  from scrapy package
  5  
  6 class QuoteSpiderExtractor(Spider):
  7     # unique identifier of this spider
    name =  # TODO - define a unique
 
8 name for spider
    # start_urls for url list to crawl
 
9 automatically
  10     start_urls = # TODO - define urls list
  11  
    # callback method to handle
 
12 responses
  13     def parse(self, response):
        # get the div elements with class
 
14 name quote
        for quote in # TODO - get div
 
15 elements with class 'quote' in response:
  16             # yield a quote dictionary
  17             yield {
                'text': # TODO - get the text
 
18 attribute of
                        # span elements with
 
19 class "text"
                'author': # TODO - get the
 
20 text attribute of
                          # small elements with
 
21 class "author"
                'tags': quote.css('div.tags
 
22 a.tag::text').getall()
  23             }
 
Let’s run this spider and see the output.
Data will be extracted in the logs in
terminal. And you will see a dictionary for
each quote.
 
  scrapy crawl quotes_extractor
 
  …
2022-04-23 10:43:25 [scrapy.core.scraper]
DEBUG: Scraped from <200
  https://quotes.toscrape.com/page/1/>
  {'text': '“It is our choices, Harry, that show what
we truly are, far more than our abilities.”',
 'author': 'J.K. Rowling', 'tags': ['abilities',
'choices']}
2022-04-23 10:43:25 [scrapy.core.scraper]
DEBUG: Scraped from <200
  https://quotes.toscrape.com/page/1/>
{'text': '“There are only two ways to live your
life. One is as though nothing is a miracle. The
other is as though everything is a miracle.”',
'author': 'Albert Einstein', 'tags': ['inspirational',
  'life', 'live', 'miracle', 'miracles']}
2022-04-23 10:43:25 [scrapy.core.scraper]
DEBUG: Scraped from <200
  https://quotes.toscrape.com/page/1/>
{'text': '“The person, be it gentleman or lady,
who has not pleasure in a good novel, must be
intolerably stupid.”', 'author': 'Jane Austen',
  'tags': ['aliteracy', 'books', 'classic', 'humor']}
2022-04-23 10:43:25 [scrapy.core.scraper]
DEBUG: Scraped from <200
  https://quotes.toscrape.com/page/1/>
{'text': "“Imperfection is beauty, madness is
genius and it's better to be absolutely ridiculous
than absolutely boring.”", 'author': 'Marilyn
  Monroe', 'tags': ['be-yourself' , 'inspirational']}
  …
 
 
OceanofPDF.com
 Storing the Scraped Data
 
Now let’s assume we want to store the
scraped data instead of just printing them
to the log console. The simplest way to
store the scraped data is by using Feed
exports, with the following command:
 
scrapy crawl quotes_extractor -O
  quotes_extractor.json
 
This command will generate a
quotes_extractor.json file containing all
scraped items, serialized in JSON. The -O
(capital letter ‘ O ’) command-line switch
overwrites any existing file; use -o (lower
case letter ‘ o ’) instead to append new
content to any existing file.
You should see a new file in your Scrapy
project. See the image below:
 
 Figure 15-6: quotes_extractor.json file in Scrapy project
 
 
OceanofPDF.com
 Following Links
 
In the previous sections we mainly
scraped a single page. Now, let’s see how
we can scrape all the pages from
https://quotes.toscrape.com web site. To do
this, we need to learn how to follow the
links. In most cases, the links are just <a>
tags.
To extract a link in an HTML structure, we
need to examine it. Here is, how we can
examine the underlying HTML of a page in
Google Chrome. Right click on the element
you want to examine and click “Inspect”.
See the image below:
 
 Figure 15-7: Inspecting elements in Google Chrome
 
When you inspect an element in Google
Chrome, it will open the Chrome Developer
Tools extension. Inside the extension, under
the Elements tab, you can see the entire
HTML structure of the element.
The link which we want to inspect is the
“Next” button, which contains a link to the
second page. We need this element
because we want to get the next page in
our Spider. Here is the HTML code of this
element:
 
  <ul class="pager">
      <li class="next">
        <a href="/page/2/">Next <span aria-
  hidden="true">→</span></a>
      </li>
  </ul>
 
The link here is the href attribute of the
<a> tag here and let’s extract it in our
Spider now. Create a new Spider named
QuoteSpiderFollowLinks inside a new Python
file named quote_spider_follow_links.py under
the “spiders” folder.
 
[4]: 1 # quote_spider_follow_links.py file
  2  
  3 from scrapy import Spider, Request
  4  
  5 class QuoteSpiderFollowLinks(Spider):
  6     # unique identifier of this spider
    name = # TODO - define a unique name
 
7 for spider
    # start_urls for url list to crawl
 
8 automatically
  9     start_urls = # TODO - define urls list
  10  
  11     # callback method to handle responses
  12     def parse(self, response):
        # get the div elements with class
 
13 name quote
        for quote in  # TODO - get div
 
14 elements with class 'quote' in response:
  15             # yield a quote dictionary
  16             yield {
                'text':  # TODO - get the text
 
17 attribute of
                # span elements with class
 
18 "text"
                    'author':  # TODO - get the
 
19 text attribute of
            # small elements with class
 
20 "author"
            'tags': quote.css('div.tags
 
21 a.tag::text').getall()
  22             }
  23  
        # get next page element via the href
 
24 attribute
        next_page = # TODO - get next page
 
25 element via the href attribute
                    # find li with class "next"
 
26 and the <a> tag in it
  27         if next_page is not None:
  28             # create the next page url
            next_page =
 
29 response.urljoin(next_page)
            # request the next page with
 
30 Scrapy Request
  31             # TODO - yield the Request object
 
In cell 4, we define a Spider which follows
the links in a web site. After extracting the
data, the parse() method looks for the link
to the next page, builds a full absolute URL
using the urljoin() method (since the links
can be relative) and yields a new request to
the next page, registering itself as callback
to handle the data extraction for the next
page and to keep the crawling going
through all the pages.
This shows us how Scrapy follows the
links. When it yields a Request in the
callback method, it will schedule that
request to be sent and register a callback
method to be executed when that request
finishes.
You can think this structure as some sort
of loop, which follows all the links to the
next page until it doesn’t find one. This is
very useful especially with sites that have
pagination. You can implement a save
mechanism in the parse() method
depending on the page you visit now.
 
 
OceanofPDF.com
 Shortcut for Creating Requests
 
In the previous section we used the
Request() constructor method to create
and yield a Request object in the parse()
method. However, we have a shortcut for
creating Request objects. The syntax is:
response.follow() . Let’s redefine the
QuoteSpiderFollowLinks class with this
shortcut this time:
 
[5]: 1 ####  response.follow()  ###
class
 
2 QuoteSpiderFollowLinks(Spider):
  3     # unique identifier of this spider
    name =  # TODO - define a unique
 
4 name for spider
    # start_urls for url list to crawl
 
5 automatically
  6     start_urls =  # TODO - define urls list
  7  
    # callback method to handle
 
8 responses
  9     def parse(self, response):
        # get the div elements with class
 
10 name quote
        for quote in
 
11 response.css('div.quote'):
  12             # yield a quote dictionary
  13             yield {
  14                 'text':  # TODO - get the text
attribute of
                # span elements with class
 
15 "text"
                    'author':  # TODO - get the
 
16 text attribute of
            # small elements with class
 
17 "author"
            'tags': quote.css('div.tags
 
18 a.tag::text').getall()
  19             }
        # get next page element via the
 
20 href attribute
        next_page = # TODO - get next
 
21 page element via the href attribute
  22         if next_page is not None:
            # TODO - yield a Request object
 
23 with response.follow()
 
While using scrapy.Request we need to
call the urljoin() method to create relative
URLs.  However, with response.follow()
method this is not necessary because it
supports relative URLs out of the box. In
line 24, we call response.follow() which
returns a Request instance. Then, we yield
this object.
It is also possible to pass a selector to
the response.follow() method, which makes
your code shorter:
  
for href in response.css('ul.pager
[6]:
1 a::attr(href)'):
    yield response.follow(href,
 
2 callback=self.parse)
 
One of the
best things about the
response.follow() method is that, it has a
built-in mechanism to follow the <a>
elements. It uses their href attributes
automatically for following. So, we do not
need to explicitly get the href attribute.
Here is how:
 
[7]: 1 for a in response.css('ul.pager a'):
    yield response.follow(a,
 
2 callback=self.parse)
 
In cell 7, we pass the <a> tag to the
response.follow() method and it uses the
href attributes automatically to follow.
 
response.follow_all(urls, callback) :
If you need to create multiple requests
from an iterable, there is a built-in method
which is response.follow_all() . It is a
generator that produces Request instances
to follow all links in urls parameter. Let’s
see how to use it:
 
[8]: 1 anchors = response.css('ul.pager a')
yield from response.follow_all(anchors,
 
2 callback=self.parse)
 
In cell 8, we first get all the anchor tags
( <a> ) in line 1. Then in the second line, we
call the response.follow_all() method by
passing these anchors. And since it is a
generator, we need to yield it as “yield
from” .
 
This finalizes our project on web
scraping with Python. Scrapy has lots of
features which will help you to build
complex and effective web crawlers.
However, for the sake of this project, we
believe it is enough to give you a basic
understating on how web scraping and
Scrapy works in Python.
 
OceanofPDF.com
7. Project 3 – API Development
       

with Flask
 
In this project, we will create a simple
movie review web application using Flask
and Python. You can find the PyCharm
project for this chapter in the GitHub
Repository of this book.
 
Chapter Outline:
Project Layout
Introduction to Flask
The Application Factory
Create the Database
Movies Data and The Utils Module
Views and Blueprints
Register Blueprint and View Function
Templates
The Base Template
Register Template
Login Blueprint & Template
Home Blueprint & Template
Review Blueprint & Template
Static Files
  
 
OceanofPDF.com
 Project Layout
 
Our project is a simple app which we will
use for reviewing some movies in the IMDB
Top 250 list. The movie data will be provided
as a CSV file in the project. We will use the
built-in SQLite database for this project.
Here are the pages in our application:
 
Home Page:
In the home page, we will see the list of
movies in a responsive grid. By clicking
any of the movies, you can either create a
review or update/delete it if it is already
created. The movies with a review will be
displayed with a green border around
them.
The first tile in the grid is the “All Reviews”
button, which will navigate you to a list of
all reviews.
We have a menu bar on top of the page.
On the menu bar on the left, we have the
app title which is also an anchor tag for
the Home Page. On the right, you see the
logged in user’s full name and a Logout
button.
  

Figure 17-1: Home Page

 
Register:
To be able to use the app, the user needs
to register and login in to the system. We
have two different pages for this. The first
page is the Register page. In which, user
will enter the full name, user name and a
password. All of which are required to fill.
 
 Figure 17-2: Register
 
Login:
After successful registration, user needs to
login in. We save user data in the user
table, so, the user can login any time with
correct user name and password. Inside
the app, we check if the credentials are
true or not. If not, we notify the user to try
again.
User can easily navigate to the Register
page from the Login screen.
 
 Figure 17-3: Login
Create a Review:
The logged in user can create a review for
any movie on the Home Page. By clicking
its tile in the Home Page, the user will be
navigated to the Create page. In Figure 17-
4, in the page url, you can see the Id of the
movie, for which the user will create a
review.
 
 Figure 17-4: Create a Review
 
Update the Review:
The user can update a review, by clicking
its tile in the Home Page. He/She will be
navigated to the Update page. In Figure
17-5, in the page url, you can see the Id of
the movie, which the user will update the
existing review.
Before leaving the Home Page, we check if
there is already a review for that movie. If
there is, then the Update page will render.
If not, then the user will see the Create
page.
 
 Figure 17-5: Update or Delete a Review
Reviews:
In the Home Page, we have a tile called
“All Reviews”. When clicked, it will
navigate the user to the Reviews Page.
This is where we see all the reviews in the
database. If the review is created by the
current user, then you will see an “Edit”
button on the right-hand side of that
review. So, the user can view all of the
reviews, but can edit only the ones which
he/she created.
 

Figure 17-6: All Reviews

 
 
OceanofPDF.com
 Introduction to Flask
 
We will use Flask for building our app.
Flask was created by Armin Ronacher in
2004. It is a microframework written in
Python. It is a lightweight WSGI (Web Server
Gateway Interface) web application
framework, which is designed to make
getting started quick and easy, with the
ability to scale up to complex applications.
Flask supports extensions that can add
application features like database
abstraction layer, form validation, etc.
A Flask application is an instance of the
Flask class. Everything about the
application, such as configuration and URLs,
will be registered with this class.
To understand Flask better, let’s start with
a small example. We will create a simple
web API, which simply renders “Hi from Flask
App” in the web browser.
We will create a new PyCharm project
named MovieReview with a new virtual
environment. And create a Python file
named hi.py .
 We have to install the Flask package first.
You can easily install it in PyCharm or you
can use pip to install it. Here is the official
documentation if you need a detailed
installation guideline.
 

Figure 17-7: Install Flask package in PyCharm IDE

 
[1]: 1 # hi.py file
  2  
  3 # import Flask class from flask package
  4 from flask import Flask
  5  
  6 # create a Flask app
  7 app = Flask(__name__)
  8  
  9 # define a root
  10 @app.route('/hi')
  11 def hi_flask():
  12     return "Hi from Flask App"
  
In cell 1, we have a very simple Flask app
in the hi.py file. We import the Flask class
first. Then in line 7, we create an instance of
the Flask class and name it simply as app .
This object will be our WSGI application. The
first argument is the name of the
application’s module or package. __name__ is
a convenient shortcut for this that is
appropriate for most cases. This is needed
so that Flask knows where to look for
resources such as templates and static files.
In line 10, we use the route() decorator to
tell Flask what URL should trigger the
function in the next line. Here, the URL will
be ‘/hi’ . Our hi_flask() function will return a
text message which will be displayed in the
user’s browser.
To run a Flask application, we need to use
the ‘flask’ command or ‘python -m flask’
command. In this project, we prefer to use
the ‘flask’ command. We need to make
some settings in the terminal before running
the ‘flask’ command. We need to tell the
terminal the application to work with by
exporting the FLASK_APP environment
variable. Here are the commands for this,
on different operating systems:
 
  Bash (macOS and Linux):
  ------------------------
  $ export FL ASK_APP=hi
  $ export FL ASK_ENV=development
  $ flask run
   * Running on http://127.0.0.1:5000/
   
   
  CMD (Windows):
  ------------------------
  > set FL ASK_APP=hi
  > set FL ASK_ENV=development
  > flask run
   * Running on http://127.0.0.1:5000/
   
   
  Powershell (Windows):
  ------------------------
  > $env:FL ASK_APP = "hi"
  > $env:FL ASK_ENV = "development"
  > flask run
   * Running on http://127.0.0.1:5000/
 
To be able to run these commands in your
terminal (either standalone or in PyCharm)
you need to be in the top-level directory of
the project. The top-level directory for this
project is the MovieReview project folder. In
PyCharm to start a terminal, right click on
the top-level folder and select Open In ->
Terminal . See the image below:
 

Figure 17-8: Open In Terminal in PyCharm

 
Now we can run the terminal command to
start our Flask app. See the image below on
macOS machine:
 
 Figure 17-9: Running a Flask App in PyCharm Terminal on
macOS
 
When you run these commands, your
Flask app will start on your local machine.
By default, the IP address for the localhost
will be http://127.0.0.1 and the port will be
5000 . So, the complete URL will become
http://127.0.0.1:5000 . Open this URL in a
browser window and add the ‘/hi’ route to
the end as: http://127.0.0.1:5000/hi . When
you call this URL in the browser, it will
render a string as “Hi from Flask App” . See
the image below:
 

Figure 17-10: Flask App running on the browser

 
If you can see a screen like the one in
Figure 17-10, then congratulations, your
first Flask app is up and running. You can
move to the next section.
 
 
OceanofPDF.com
 The Application Factory
 
In the previous section, we created a
Flask app instance inside the hi.py file. It
was just a simple example and in the real-
world project, there is a better way of
creating Flask apps. We have a special
function known as The Application
Factory. The Application Factory is
responsible for any configuration,
registration and other setup operations. And
it returns the configured app object, which is
an instance of the Flask class.
 
flaskapp directory:
In a Flask project the actual project has its
own directory. Keep in mind that, the Flask
project is not the same thing as the
PyCharm project. Flask project is a sub-
folder of the PyCharm project. So, inside our
MovieReview PyCharm project we will have
a Flask project as flaskapp . Here is the folder
structure:
 
 Figure 17-11: Flask App inside the PyCharm project
 
As you see in Figure 17-11, MovieReview is
our PyCharm Project, and the flaskapp is the
Flask project in it. flaskapp is the directory
where we will instantiate a Flask instance.
Actually, it will be a Python package
because it will contain the __init__.py file.
Now let’s create the flaskapp directory
(package) in our MovieReview project, if you
haven’t already. Make sure you have
__init__.py file in it. The __init__.py will contain
the application factory for our Flask app,
and it tells Python that the flaskapp
directory should be treated as a package.
Here is the content of this file:
 
[2]: 1 # __init__.py file
  2  
  3 import os
  4 from flask import Flask
  5  
  6 # application factory
  7 def create_app(test_config=None):
  8     # create a Flask app object
    app = Flask(__name__,
 
9 instance_relative_config=True)
  10  
  11     # configure the app
  12     app.config.from_mapping(
  13         SECRET_KEY='dev',
       
  DATABASE=os.path.join(app.instance_path,
14 'flaskapp.sqlite'),
  15     )
  16  
  17     # create the instance folder if not exists
  18     try:
  19         os.makedirs(app.instance_path)
  20     except OSError:
  21         pass
  22  
  23     # a simple test to see if it works
  24     @app.route('/test')
  25     def test():
        return 'Flaskapp init file works fine
 
26 :)'
  27  
  28     # return the configured Flask app object
  29     return app
 
In cell 2, you see __init__.py file content.
This is the initial form of this file and we will
modify it constantly. The create_app()
function is the application factory itself.
Make sure you give exactly the same name,
because “create_app” is the name Flask uses
for application factory by default.
In line 9 we create a Flask app instance
as: app = Flask(__name__,
instance_relative_config=True) . The
parameters are:
__name__ : name of the current Python
module. This tells the app where it’s
located to set up relative paths.
instance_relative_config=True : This
parameter tells the app that
 configuration files are relative to the
instance folder. The instance folder is a
special folder which is located outside
the flaskapp package and holds local
data like configuration secrets and
database files. It will automatically be
created when we run Flask in a
minute.
In line 12, we configure our Flask app
object. The app.config.from_mapping()
method sets some default configuration
that the app will use like SECRET_KEY and
DATABASE . We will use SQLite database for
this project.
SECRET_KEY is used by Flask to keep the
app data safe. During development
you can set it to 'dev' , but you should
override it with a random value before
deploying the app.
DATABASE is the path where we save
the SQLite database file. It’s under
app.instance_path , which is the path
that Flask has chosen for the instance
folder. Don’t worry, it will make sense
when we create our database.
In line 19, we create the instance folder
as: os.makedirs(app.instance_path) . This folder
will hold our local database file when we
initialize the db.
In line 24, we have a simple route for test
purposes. We want to see if our Flask app
works when we run this code. The url for
this route is simply ‘/test’ .
And finally, in line 29, we return the
configured Flask app object.
Now let’s run this code and test it. To run
it, start a new Terminal window in PyCharm
and set a new environment variable for our
flaskapp project. Make sure you are at the
top-level folder in the PyCharm project
which is MovieReview directory.
 
  Bash (macOS and Linux):
  ------------------------
  $ export FL ASK_APP=flaskapp
  $ export FL ASK_ENV=development
  $ flask run
   
   
  CMD (Windows):
  ------------------------
  > set FL ASK_APP=flaskapp
  > set FL ASK_ENV=development
  > flask run
   
   
  Powershell (Windows):
  ------------------------
  > $env:FL ASK_APP = "flaskapp"
  > $env:FL ASK_ENV = "development"
  > flask run
 
This time the environment variable name
is flaskapp , which is the same name with our
flask project. If you run this code in your
PyCharm terminal, then it should start a
new Flask app on the given local IP address
and port. You should see an output similar
to the one below when you run these
commands:
 
(venv) MovieReview % export
  FL ASK_APP=flaskapp  
(venv) MovieReview % export
  FL ASK_ENV=development
  (venv) MovieReview % flask run                  
  * Serving Flask app 'flaskapp' (lazy loading)
  * Environment: development
  * Debug mode: on
* Running on http://127.0.0.1:5000 (Press
  CTRL+C to quit)
  * Restarting with stat
  * Debugger is active!
  * Debugger PIN: 885-298-466
 
Now let’s check it on the web browser.
When you type http://127.0.0.1:5000/test in
the browser, you should see a text of
“Flaskapp init file works fine :)” . See the
images below:
 

Figure 17-12: Running flaskapp in PyCharm

 

Figure 17-13: The result in the browser for

http://127.0.0.1:5000/test
 
 If you see a similar result like the one in
Figure 17-13, then your application factory
is set up correctly. You can move on to the
next section now. Which is about creating a
local database.
 
 
OceanofPDF.com
 Create The Database
 
As we mentioned earlier, we will use
SQLite database to store our app data.
SQLite is a light-weight database server
which doesn’t require setting up a
separate db server and is built-in to
Python. It is convenient for small projects,
however for large applications it may not
serve well due to its limitations.
We will create a separate Python file in
our flaskapp folder, called db.py . This file
will setup and initialize the database. Here
is the code in this file:
 
[3]: 1 # db.py file
  2  
  3 import sqlite3
  4 import click
  5 from flask import current_app, g
  6 from flask.cli import with_appcontext
  7  
  8 # db connection
  9 def get_db():
  10     if 'db' not in g:
  11         g.db = sqlite3.connect(
  12             current_app.config['DATABASE'],
           
 
13 detect_types=sqlite3.PARSE_DECLTYPES
  14         )
  15         g.db.row_factory = sqlite3.Row
  16     return g.db
  17  
  18 # close connection
  19 def close_db(e=None):
  20     db = g.pop('db', None)
  21     # check if connection exists
  22     if db is not None:
  23         db.close()
  24  
  25 # initialize db
  26 def init_db():
  27     db = get_db()
    with
  current_app.open_resource('schema.sql')
28 as f:
       
 
29 db.executescript(f.read().decode('utf8'))
  30  
  31 # command line command
  32 @click.command('init-db')
  33 @with_appcontext
  34 def init_db_command():
    """Clear the existing data and create
 
35 new tables."""
  36     init_db()
    click.echo('Successfully initialized the
 
37 database.')
  38  
  39 # register functions with app instance
  40 def init_app(app):
  41     app.teardown_appcontext(close_db)
  42    
app.cli.add_command(init_db_command)
 
In cell 3, you see the code in our db.py
file. In line 5, we import current_app and g
objects from the flask package:
g is a special object which is unique
for each request. It stores the data
which is accessible for multiple
functions during the current request.
current_app is an object that points to
the current Flask application request.
sqlite3.connect() establishes a
connection to the file pointed at by
the DATABASE configuration key.
sqlite3.Row tells the connection to
return rows that behave like dicts,
which allows accessing the columns
by their names.
 
The functions are:
get_db() : creates a db connection and
sets it to the g.db attribute
close_db() : checks and closes the db
connection if it exists
 init_db() : initializes a database by
reading the 'schema.sql' file, which
contains the SQL commands to
create the database tables.
init_db_command() : it calls the init_db()
function and prints the result. It is
decorated with @click.command('init-
db') and @with_appcontext .
click.command() defines a command line
command called init-db which calls the
init_db() function and shows a success
message to the user.
with_appcontext decorator is in the
flask.cli module and wraps a callback to
guarantee it will be called with a
script's application context.
init_app() is used to register the
functions with the app instance,
which is the incoming parameter. We
will call this function from the
application factory.
app.teardown_appcontext() tells Flask to
call that function when cleaning up
after returning the response.
app.cli.add_command() adds a new
command that can be called with the
 flask command. This enables us to use
“flask init-db” command in the terminal.
 
Now let’s create the 'schema.sql' file
under the flaskapp directory. Here is the
code in this file:
 
[4]: 1 -- schema.sql file
  2  
  3 DROP TABLE IF EXISTS user;
  4 DROP TABLE IF EXISTS review;
  5  
  6 CREATE TABLE user (
  id INTEGER PRIMARY KEY
 
7 AUTOINCREMENT,
  8   username TEXT UNIQUE NOT NULL,
  9   password TEXT NOT NULL,
  10   fullname TEXT NOT NULL
  11 );
  12  
  13 CREATE TABLE review (
  14   id INTEGER PRIMARY KEY,
  15   author_id INTEGER NOT NULL,
  created TIMESTAMP NOT NULL
 
16 DEFAULT CURRENT_TIMESTAMP,
  17   title TEXT NOT NULL,
  18   body TEXT NOT NULL,
  FOREIGN KEY (author_id)
 
19 REFERENCES user (id)
  20 );
 
 In cell 4, you see the SQL commands to
create two database tables; user and
review . Each table has fields in their
definition commands. Before running
these commands, we need to call the
init_app() function from the application
factory. Here is updated __init__.py file
content:
 
[5]: 1 # __init__.py file
  2  
  3 import os
  4 from flask import Flask
  5 from . import db
  6  
  7 # application factory
  8 def create_app(test_config=None):
  9     # create a Flask app object
    app = Flask(__name__,
 
10 instance_relative_config=True)
  11  
  12     # configure the app
  13     app.config.from_mapping(
  14         SECRET_KEY='dev',
       
  DATABASE=os.path.join(app.instance_path,
15 'flaskapp.sqlite'),
  16     )
  17  
    # create the instance folder if not
 
18 exists
  19     try:
  20         os.makedirs(app.instance_path)
  21     except OSError:
  22         pass
  23  
  24     # db registration
  25     db.init_app(app)
  26  
  27     # a simple test to see if it works
  28     @app.route('/test')
  29     def test():
        return 'Flaskapp init file works fine
 
30 :)'
  31  
    # return the configured Flask app
 
32 object
  33     return app
 
In cell 5, line 35, we register the
database with our application by calling
the db.init_app() function and passing the
current app object as the argument.
Now we are ready to initialize the
database file. All we need to do is to call
the “flask init-db” command in the
terminal. Make sure you stop the Flask
server by pressing CTRL+C to quit. And set
the FLASK_APP and the FLASK_ENV
environment variables again before
running the “flask init-db” command. Here
is the output on a macOS computer:
 
(venv) MovieReview % export
 
FL ASK_APP=flaskapp  
(venv) MovieReview % export
 
FL ASK_ENV=development
  (venv) MovieReview % flask init-db
  Successfully initialized the database.
 
If you see the text of ‘Successfully
initialized the database.’ in the output, then
your database tables are created and the
database is initialized in the application. If
you check the “instance” folder in the
project now, you should see the
flaskapp.sqlite file in it. This is the file
where our database will be stored. You can
now run “flask run” command to start the
server again.
 
 
OceanofPDF.com
 Movies Data and The Utils Module
 
We will be using movies data which
belongs to the top 20 movies in the IMDB
Top 250 movies. The data will be in a CSV
file inside a folder called ‘data’ . File name
is imdb_top_20.csv and you can find it in
the project directory in GitHub repository
of this book.
In our project, we will also have a
module called utils.py . This module will
contain some common functions, like
reading CSV data and getting movies,
getting the user and review data from
database etc. Here is the content of this
file:
 
[6]: 1 # utils.py file
  2  
  3 from flask import (g, redirect, url_for)
  4 from flaskapp.db import get_db
  5 import csv
  6 import functools
  7  
  8 # function to read all movies
  9 def get_movies():
  10     """reads the csv file and returns
  11         the list of movies"""
  12     movies = []
  13     movie_path = 'data/imdb_top_20.csv'
  14     with open(movie_path, 'r') as file:
        movie_dict = csv.DictReader(file,
 
15 delimiter=';')
  16         for movie in movie_dict:
  17             movies.append(movie)
  18     # return the list
  19     return movies
  20  
  21 # function to read a single movie
  22 def get_movie(id):
  23     """returns a single movie"""
  24     movie = None
  25     movie_path = 'data/imdb_top_20.csv'
  26     with open(movie_path, 'r') as file:
        movie_dict = csv.DictReader(file,
 
27 delimiter=';')
  28         for m in movie_dict:
  29             if m['id'] == str(id):
  30                 movie = m
  31     return movie
  32  
  33 # get movie image
  34 def get_movie_img(id):
  35     return get_movie(id)['imageURL']
  36  
  37 # get all reviews
  38 def get_all_reviews():
  39     db = get_db()
  40     return db.execute(
        'SELECT r.id, title, body, created,
 
41 author_id, username, fullname'
  42         ' FROM review r JOIN user u ON
r.author_id = u.id'
  43         ' ORDER BY created DESC'
  44     ).fetchall()
  45  
  46 # get review from db
  47 def get_review_from_db(id):
  48     return get_db().execute(
        'SELECT r.id, title, body, created,
 
49 author_id, username'
        ' FROM review r JOIN user u ON
 
50 r.author_id = u.id'
  51         ' WHERE r.id = ?',
  52         (id,)
  53     ).fetchone()
  54  
  55 # get user
  56 def get_user(username):
  57     db = get_db()
  58     user = db.execute(
        'SELECT * FROM user WHERE
 
59 username = ?', (username,)
  60     ).fetchone()
  61     return user
  62  
  63 # Require Authentication
  64 def login_required(view):
  65     @functools.wraps(view)
  66     def wrapped_view(**kwargs):
  67         if g.user is None:
            return
 
68 redirect(url_for('login.login'))
  69         return view(**kwargs)
  70     return wrapped_view
 
In cell 6, we have the utils.py file
content. Here are the functions in it:
get_movies() : This function reads the
csv file and returns the list of
movies.
get_movie() : This function returns a
single movie based on the given id
parameter.
get_movie_img() : This function returns
the movie image URL based on the
given id.
get_all_reviews() : It returns the list of
all reviews in the database. It uses a
SQL join statement to join the review
and user tables. We call the fetchall()
method to get a list of results.
get_review_from_db() : It returns a
single review from the database. We
use fetchone() for returning a single
result.
get_user() : It returns the user data
from the database based on the
username provided.
 login_required() :We will require the
user to login to our application
before creating, editing and deleting
movie reviews. login_required is the
decorator which will check if the
user is logged in before any of these
operations. It returns a new view
function that wraps the original view
it’s applied to. The new function
simply checks if the user is logged in
as: if g.user is None . Remember that
the object g stores the data in the
current request. If g.user is not None ,
this means that the user is logged in.
You will see how we fill it later on. If
it is None , then we redirect the user
to the login page.
 
url_for(function, parameters) :
The url_for() function in Flask,
generates a URL to an endpoint using
the function passed in as an argument.
For example, the test() function that
was added to the application factory in
the previous section, has the name
'test' and can be linked to with
 url_for('test') .
You can think of it as
requesting the URL of
http://127.0.0.1:5000/test in your web
browser.
If it took an argument, which you’ll see
later, it would be linked to using
url_for('test', param_1=value_1,
param_2=value_2) .
In line 68, we use this function as
url_for('login.login') . The parameter is
‘login.login’ . The first ‘login’ is the
Blueprint name and the second one is
the endpoint name. So, this means
that, we redirect the user to an
endpoint (a function here) called ‘login’
inside a Blueprint named ‘login’ . Don’t
worry it will make sense when you
learn about the Blueprints.
 
 
OceanofPDF.com
 Views and Blueprints
 
In a Flask application the UI (user
interface) is mainly created by Templates
(HTML templates) and the back-end is
created by Views and Blueprints.
Views are the functions which respond
to the requests. Views handle the
incoming request based on the request
URL patterns. And they return data that
Flask turns into an outgoing response.
Blueprints are the way to organize a
group of related views. They can contain
any number of view functions. Views are
registered with a Blueprint and the
Blueprint is registered with the application
factory.
In our project we will have four
Blueprints which are: home , login , register ,
and review . Each one will have a separate
module and in each of them we will have
different View functions to handle
incoming requests.
 
 
OceanofPDF.com
 Register Blueprint and View Function
 
A Blueprint is a class in the flask
module, which is used to register the
Views (View functions). The name of our
first Blueprint will be Register and it will
have a View function called register() .
Create a new Python file in the flaskapp
directory called register.py . And here is the
code in this file:
 
[7]: 1 # register.py file
from flask import (Blueprint, flash,
 
2 redirect,
                   render_template, request,
 
3 session, url_for)
  4 from flaskapp.db import get_db
from werkzeug.security import
 
5 generate_password_hash
  6 from .utils import get_user
  7  
  8 # initialize the Blueprint object
bp = Blueprint('register', __name__,
 
9 url_prefix='/')
  10  
  11 # Register View Code
@bp.route('/register', methods=('GET',
 
12 'POST'))
  13 def register():
  14     if request.method == 'POST':
  15         # get input fields
        username, password, fullname =
 
16 get_input_fields()
  17         # get db
  18         db = get_db()
  19         # check input fields
        error =
  check_input_fields(username, password,
20 fullname)
  21         # check error
  22         if error is None:
  23             try:
  24                 # insert new user
                insert_into_db(db, username,
 
25 password, fullname)
  26             except db.IntegrityError:
                error = f"User {username} is
 
27 already registered."
  28             else:
  29                 session.clear()
                session['user_id'] =
 
30 get_user(username)['id']
                return
 
31 redirect(url_for('home.index'))
  32         flash(error)
    return
 
33 render_template('register/register.html')
  34  
  35 # fn for getting inputs
  36 def get_input_fields():
  37     username = request.form['username']
  38     password = request.form['password']
  39     fullname = request.form['fullname']
  40     return username, password, fullname
  41  
  42 # fn for checking input fields
def check_input_fields(username,
 
43 password, fullname):
    if not username or not password or
 
44 not fullname:
        return 'Full Name, Username or
 
45 Password can not be empty.'
  46     else:
  47         return None
  48  
  49 # fn for inserting a DB row
def insert_into_db(db, username,
 
50 password, fullname):
    db.execute("INSERT INTO user
 
51 (username, password, fullname) "
  52                "VALUES (?, ?, ?)",
        (username,
  generate_password_hash(password),
53 fullname),)
  54     db.commit()
 
In cell 7, line 9, we create a Blueprint
object as: bp = Blueprint('register', __name__,
url_prefix='/') . The first parameter is the
unique name of this Blueprint the second
one is the name of the current application
and the third parameter is the URL prefix.
In line 13 you see the definition of the
register() view function. Here is what this
function does:
@bp.route decorator associates the
URL “/register ” with the register()
view function. When Flask receives a
request to
http://127.0.0.1:5000/register, it will
call the register() view and use the
return value as the response. This
view function can handle both GET
and POST request types.
In line 14, we check the request
method being POST , which means
the user submits the form in this
view. If the method is POST, then we
have some validations for the form
fields like username , password and
fullname . We don’t want any of them
to be empty, that’s why we call the
check_input_fields() function.
If all of the fields are filled by the
user, then we can try to insert this
into the user table in db. That’s what
we do in line 25, by calling the
insert_into_db() function.
If db insert operation is successful,
then in the else block in line 29, we
clear the old session and add the
 user id to it. Then we redirect the
user to the home page by calling the
url_for('home.index') function. We will
define the home.index view function
later on.
In line 32, we call a special function
called flash() by passing the error as
the argument. This function stores
the error messages that can be
retrieved when rendering the
template. We will print the error
messages later on in our HTML
template.
Finally, in line 33, we call the
render_template() function by passing
the file location of the template. This
will render the HTML template for
the 'register/register.html' file in the
web browser. We will create this file
soon.
 
In line 50, we have the definition of the
insert_into_db() function. It simply inserts a
new row to the database. For security
reasons, we call the
generate_password_hash() function instead
of saving the password as it is. This will
store the hash value of the password.
To be able to use our Register Blueprint,
we need to register it inside our
application factory as follows:
 
[8]: 1 # __init__.py file
  2  
  3 import os
  4 from flask import Flask
  5 from . import db, register
  6  
  7 # application factory
  8 def create_app(test_config=None):
  9     # create a Flask app object
    app = Flask(__name__,
 
10 instance_relative_config=True)
  11  
  12     # configure the app
  13     app.config.from_mapping(
  14         SECRET_KEY='dev',
       
  DATABASE=os.path.join(app.instance_path,
15 'flaskapp.sqlite'),
  16     )
  17  
    # create the instance folder if not
 
18 exists
  19     try:
  20         os.makedirs(app.instance_path)
  21     except OSError:
  22         pass
  23  
  24     # db registration
  25     db.init_app(app)
  26  
  27     # register Blueprints
  28     app.register_blueprint(register.bp)
  29  
  30     # a simple test to see if it works
  31     @app.route('/test')
  32     def test():
        return 'Flaskapp init file works fine
 
33 :)'
  34  
    # return the configured Flask app
 
35 object
  36     return app
 
Now that we have our Register Blueprint
and a view function for it, let’s request its
URL in the web browser. If you type
http://127.0.0.1:5000/register in your
browser and press on Enter, you will see
an error page as TemplateNotFound . It will
say: jinja2.exceptions.TemplateNotFound:
register/register.html . This tells us that
Flask couldn’t find any Templates for
rendering. We will create HTML templates
in the succeeding sections.
 
 
OceanofPDF.com
 Templates
 
Templates are files that contain static
data as well as placeholders for dynamic
data. A template is rendered with specific
data to produce a final document. All the
template files in a Flask app need to be
located under a special folder called
“templates” . This folder should be directly
under the flaskapp directory and it is the
default location where Flask looks for
templates at run time.
Flask uses the Jinja template library to
render templates. In our application, we
will use templates to render HTML which
will display in the user’s browser.
A Jinja template is simply a text file.
Jinja can generate any text-based format
(HTML, XML, CSV, LaTeX, etc.). A Jinja
template doesn’t need to have a specific
extension: .html, .xml, or any other
extension is just fine.
A template contains variables and/or
expressions, which get replaced with
values when the template is rendered;
and tags, which control the logic of the
template.
The template syntax in Jinja is heavily
inspired by Python. Special delimiters are
used to distinguish Jinja syntax from the
static data in the template. Anything
between {{ and }} is an expression that
will be output to the final document. {%
and %} denotes a control flow statement
like if and for . Unlike Python, blocks are
denoted by start and end tags rather than
indentation since static text within a block
could change indentation. Here are a few
kinds of delimiters:
{% ... %} for Statements
{{ ... }} for Expressions to print to
the template output
{# ... #} for Comments not included
in the template output
 
 
OceanofPDF.com
 The Base Template
 
In our application, each page will have the
same basic layout around a different body. For
example, they will all include a navigation bar at
the top. Instead of writing the entire HTML
structure in each template, each template will
extend a base template and override specific
sections. You can think of the base templates as
the master pages.
In our flaskapp project directory, we will
create a new folder called templates , and inside
this folder, create a new HTML file named
base.html . Here is the code in this file:
 
[9]: 1 <!doctype html>
  2 <html lang="en" >
  3 <head>
  4   <meta charset="UTF-8">
  <link rel='stylesheet'
  href='https://fonts.googleapis.com/css?
5 family=Rubik:400,700'>
  6   <link rel="stylesheet" href="./style.css">
  <link rel="stylesheet" href="{{ url_for('static',
 
7 filename='style.css') }}">
  8   <title>Flask App</title>
  9 </head>
  10 <body>
  11  
  12 <div class="menu">
  <a class="title-anchor" href="{{ url_for('home.
 
13 }}"><h1>Movie Review with Flask</h1></a>
  14   <div class="menu-right">
  15     {% if g.user %}
  16       <h2>{{ g.user['fullname'] }}</h2>
      <a href="{{ url_for('login.logout') }}">Log
 
17 Out</a>
  18     {% else %}
      <a href="{{ url_for('register.register')
 
19 }}">Register</a>
  20       <a href="{{ url_for('login.login') }}">Log In<
  21     {% endif %}
  22   </div>
  23 </div>
  24 <div class="content">
  25   <header>
  26     {% block header %}{% endblock %}
  27   </header>
  28   {% for message in get_flashed_messages() %}
  29     <div class="flash">{{ message }}</div>
  30   {% endfor %}
  31   {% block content %}{% endblock %}
  32 </div>
  33 </body>
  34 </html>
 
In cell 9, we have the content of base.html file
in the templates folder. It is a regular HTML file
which includes Jinja commands in it.
In line 15, we check the user in the current
request as: {% if g.user %} . And if g.user is not
None , we are sure that the user is logged in and
we render a logout link button. Else, we render
Register and Log In buttons.
In line 16, inside the <h2> tag, we render the
user fullname as: {{ g.user['fullname'] }} .
 In line 26, we have a placeholder for the
header block as: {% block header %}{% endblock
%} .
There are two blocks defined in the base
template which will be overridden in the child
templates:
{% block header %} is where the unique
header of each page will replace.
{% block content %} is where the content of
each page goes, such as the register or
login forms.
In line 28, we check if there are any error
messages in the view functions. Remember that
we called the flash(error) function in the view
code for storing the error messages. And this is
where we get these messages by calling the
get_flashed_messages() function. We set a for
loop to iterate over the messages and render a
div element for each one. Here is the code: {%
for message in get_flashed_messages() %} .
And finally, in line 31, we have the content
block.
 
 
OceanofPDF.com
 Register Template
 
Now that we have the base template,
we can extend it and create a new
template called the Register Template. It
will contain the HTML code to create our
register page. Inside the templates folder
create a sub-folder called register . And
inside this folder create an HTML
document as register.html . Here is the
code in this file:
 
[10]: 1 {% extends 'base.html' %}
  2  
  3 {% block content %}
  4     <div class="login-form">
  5       <form method="post">
  6         <h1>Register</h1>
  7         <div class="content">
  8           <div class="input-field">
            <input class="input-text"
  id="fullname" name="fullname"
9 placeholder="Full Name">
  10           </div>
  11           <div class="input-field">
            <input class="input-text"
  id="username" name="username"
12 placeholder="User Name">
  13           </div>
  14           <div class="input-field">
  15             <input class="input-text"
type="password" id="password"
name="password"
placeholder="Password">
  16           </div>
  17         </div>
  18         <div class="action">
          <button type="submit"
 
19 class="actionbtn">Sign Up</button>
  20         </div>
  21       </form>
  22     </div>
  23 {% endblock %}
 
In cell 10, we have the register
template. In the first line you see that it
extends the base template as: {% extends
'base.html' %} .
In line 3, we have the structure for the
content block. This will replace the content
block in the base template. The register
Template is a basic HTML form element
with some input fields in it.
Remember that in cell 7 line 33, inside
the register() view function we had the
return statement as: return
render_template('register/register.html') . This
is where we tell Flask to find and render
the template when it matches the
specified route, which was:
@bp.route('/register', methods=('GET',
'POST')) .
Now if request the URL
http://127.0.0.1:5000/register in the
browser and press on Enter, Flask will try
to render the register template. You will
still get a BuildError , because we do not
have a template named home.index yet.
Don’t worry we will create it in a minute.
 
 
OceanofPDF.com
 Login Blueprint & Template
 
Before creating the Blueprint for Home
Page, we will create the Login Page. As we
did for the Register page, first we will
create a Blueprint with some View functions
in it. Then we will create a template as
login.html .
 
Login Blueprint and View Functions:
For the Blueprint, create a Python file
named login.py under the flaskapp
directory. Here is the code for this file:
 
[12]: 1 # login.py file
  2  
from flask import (Blueprint, flash, g,
 
3 redirect,
                   render_template, request,
 
4 session, url_for)
  5 from flaskapp.db import get_db
from werkzeug.security import
 
6 check_password_hash
  7  
  8 # initialize the Blueprint object
bp = Blueprint('login', __name__,
 
9 url_prefix='/')
  10  
  11 # Login View Code
  12 @bp.route('/login', methods=['GET',
 'POST'])
  13 def login():
  14     if request.method == 'POST':
        username =
 
15 request.form['username']
        password =
 
16 request.form['password']
  17         db = get_db()
  18         error = None
  19         user = db.execute(
            'SELECT * FROM user WHERE
 
20 username = ?', (username,)
  21         ).fetchone()
  22         # check user from db
  23         if user is None:
            error = "Incorrect username.
 
24 Please try again."
        elif not
  check_password_hash(user['password'],
25 password):
            error = "Incorrect password.
 
26 Please try again."
  27         # check error
  28         if error is None:
  29             session.clear()
  30             session['user_id'] = user['id']
            return
 
31 redirect(url_for('home.index'))
  32         # flash error
  33         flash(error)
  34     # render template
    return
 
35 render_template('login/login.html')
  36  
  37 # Run before all view functions
  38 @bp.before_app_request
  39 def load_logged_in_user():
  40     user_id = session.get('user_id')
  41     if user_id is None:
  42         g.user = None
  43     else:
  44         g.user = get_db().execute(
            'SELECT * FROM user WHERE id =
 
45 ?', (user_id,)
  46         ).fetchone()
  47  
  48 # logout route
  49 @bp.route('/logout')
  50 def logout():
  51     session.clear()
  52     return redirect(url_for('login.login'))
 
In cell 12, you see the code in the
login.py file. It creates a Blueprint called
‘login’ and define routes for '/login' and
'/logout' . Now we have to register this
Blueprint with the application factory. Here
it is:
 
[13]: 1 # __init__.py file
  2  
  3 import os
  4 from flask import Flask
  5 from . import db, register, login
  6  
  7 # application factory
  8 def create_app(test_config=None):
  9     # create a Flask app object
    app = Flask(__name__,
 
10 instance_relative_config=True)
  11  
  12     # configure the app
  13     app.config.from_mapping(
  14         SECRET_KEY='dev',
       
  DATABASE=os.path.join(app.instance_path,
15 'flaskapp.sqlite'),
  16     )
  17  
    # create the instance folder if not
 
18 exists
  19     try:
  20         os.makedirs(app.instance_path)
  21     except OSError:
  22         pass
  23  
  24     # db registration
  25     db.init_app(app)
  26  
  27     # register Blueprints
  28     app.register_blueprint(register.bp)
  29     app.register_blueprint(login.bp)
  30  
  31     # a simple test to see if it works
  32     @app.route('/test')
  33     def test():
        return 'Flaskapp init file works fine
 
34 :)'
  35  
  36     # return the configured Flask app
 object
  37     return app
 
Login Template:
Now that we have a Blueprint for the
login page, we can define a Template for it.
To do this, create a new folder called login
under the templates directory. And in that
folder, create a file called login.html . Here is
the code in this file:
 
[14]: 1 {% extends 'base.html' %}
  2  
  3 {% block content %}
  4     <div class="login-form">
  5       <form method="post">
  6         <h1>Login</h1>
  7         <div class="content">
  8           <div class="input-field">
            <input class="input-text"
  id="username" name="username"
9 placeholder="User Name">
  10           </div>
  11           <div class="input-field">
            <input class="input-text"
type="password" id="password"
 
name="password"
12 placeholder="Password">
  13           </div>
          <a href="#" class="link">Forgot
 
14 Your Password?</a>
  15         </div>
  16         <div class="action">
          <button class="actionbtn"
 
17 type="submit">Sign in</button>
          <a class="actionbtn" href="{{
  url_for('register.register')
18 }}">Register</a>
  19         </div>
  20       </form>
  21     </div>
  22 {% endblock %}
 
 
OceanofPDF.com
 Home Blueprint & Template
 
Now let’s create a Blueprint and a
Template for the home page which we will
call as index.html .
 
Home Blueprint and View Function:
For the Blueprint, create a Python file
named home.py under the flaskapp directory.
Here is the code for this file:
 
[15]: 1 # home.py file
from flask import (Blueprint, g,
 
2 render_template)
from .utils import login_required,
 
3 get_all_reviews, \
  4     get_movies, get_review_from_db
  5  
  6 # initialize the Blueprint object
bp = Blueprint('home', __name__,
 
7 url_prefix='/')
  8  
  9 # Index View Code
  10 @bp.route('/', methods=['GET'])
  11 @login_required
  12 def index():
  13     # get all movies
  14     movies = get_movies()
  15     # get all reviews
  16     all_reviews = get_all_reviews()
  17     # render index.html
  18     return
render_template('home/index.html',
movies=movies,
                          
 
19 all_reviews=all_reviews,
                          
 
20 get_if_reviewed=get_if_reviewed)
  21  
  22 def get_if_reviewed(movie):
    review =
 
23 get_review_from_db(movie['id'])
    if review is not None and g.user['id']
 
24 == review['author_id']:
  25         return True
  26     else:
  27         return False
 
In the Home Blueprint, we don’t have a
specific route for the index() view function.
The route is: @bp.route('/', methods=['GET']) .
Which means it will be the main page when
the user opens the application by
requesting http://127.0.0.1:5000/.
In line 15 and 16, it gets all the movies
and the reviews by calling the respective
functions in the utils file.
It renders a template in the
'home/index.html' location which we will
create next. Here, we also pass some
parameters while rendering the template.
Here is the code:
    return render_template('home/index.html',
movies=movies,
                           all_reviews=all_reviews,
                          
get_if_reviewed=get_if_reviewed)
The first two parameters are list type
objects and the third one is a function,
which we will call from the HTML template.
This is how we pass parameters to
templates from the view functions.
Now we have to register this Blueprint
with the application factory. Here it is:
 
[16]: 1 # __init__.py file
  2  
  3 import os
  4 from flask import Flask
  5 from . import db, register, login, home
  6  
  7 # application factory
  8 def create_app(test_config=None):
  9     # create a Flask app object
    app = Flask(__name__,
 
10 instance_relative_config=True)
  11  
  12     # configure the app
  13     app.config.from_mapping(
  14         SECRET_KEY='dev',
       
  DATABASE=os.path.join(app.instance_path,
15 'flaskapp.sqlite'),
  16     )
  17  
     # create the instance folder if not
 
18 exists
  19     try:
  20         os.makedirs(app.instance_path)
  21     except OSError:
  22         pass
  23  
  24     # db registration
  25     db.init_app(app)
  26  
  27     # register Blueprints
  28     app.register_blueprint(register.bp)
  29     app.register_blueprint(login.bp)
  30     app.register_blueprint(home.bp)
  31  
  32     # a simple test to see if it works
  33     @app.route('/test')
  34     def test():
        return 'Flaskapp init file works fine
 
35 :)'
  36  
  37     # return the configured Flask app object
  38     return app
 
 
Index Template:
Now that we have a Blueprint for the
home page, we can define a Template for it.
To do this, create a new folder called home
under the templates directory. And in that
folder, create a file called index.html . Here is
the code in this file:
  
[17]: 1 {% extends 'base.html' %}
  2  
  3 {% block content %}
  4     <div class="login-form">
  5       <form method="post">
  6         <h1>Login</h1>
  7         <div class="content">
  8           <div class="input-field">
            <input class="input-text"
  id="username" name="username"
9 placeholder="User Name">
  10           </div>
  11           <div class="input-field">
            <input class="input-text"
type="password" id="password"
 
name="password"
12 placeholder="Password">
  13           </div>
          <a href="#" class="link">Forgot
 
14 Your Password?</a>
  15         </div>
  16         <div class="action">
          <button class="actionbtn"
 
17 type="submit">Sign in</button>
          <a class="actionbtn" href="{{
  url_for('register.register')
18 }}">Register</a>
  19         </div>
  20       </form>
  21     </div>
  22 {% endblock %}
 
 At this point, if you run the app and type
http://127.0.0.1:5000/ in your web browser,
you should a screen like the one below:
 

Figure 17-14: Login page which is redirected from the Home

page
 
For the time being, the app doesn’t look
good because we have no formatting yet.
Don’t worry we will add some CSS and it will
look pretty.
 
 
OceanofPDF.com
 Review Blueprint & Template
 
The last Blueprint which we will add is
the Review Blueprint. It will be responsible
for creating new reviews, updating existing
ones, deleting reviews and displaying a list
of all reviews.
 
Review Blueprint and View Functions:
For the Blueprint, create a Python file
named review.py under the flaskapp
directory. Here is the code for this file:
 
[18]: 1 # review.py file
  2 from flask import (
  3     Blueprint, flash, g, redirect,
  4     render_template, request, url_for)
from werkzeug.exceptions import
 
5 abort
  6 from flaskapp.db import get_db
from .utils import login_required,
 
7 get_all_reviews, \
    get_movie, get_review_from_db,
 
8 get_movie_img
  9  
  10 bp = Blueprint('review', __name__)
  11  
  12 # reviews route
  13 @bp.route('/reviews')
  14 @login_required
  15 def reviews():
  16     all_reviews = get_all_reviews()
    return
 
17 render_template('review/reviews.html',
                          
 
18 all_reviews=all_reviews,
                          
 
19 get_movie_img=get_movie_img)
  20  
  21 # create a new review
@bp.route('/<int:id>/create', methods=
 
22 ('GET', 'POST'))
  23 @login_required
  24 def create(id):
  25     # get movie
  26     movie = get_movie(id)
  27     if request.method == 'POST':
  28         title = request.form['title']
  29         body = request.form['body']
  30         error = None
  31         if not body:
            error = 'You need to add a
 
32 review.'
  33         if error is not None:
  34             flash(error)
  35         else:
  36             db = get_db()
  37             db.execute(
                'INSERT INTO review (id, title,
 
38 body, author_id)'
  39                 ' VALUES (?, ?, ?, ?)',
  40                 (id, title, body, g.user['id'])
  41             )
  42             db.commit()
  43             return
redirect(url_for('review.reviews'))
    return
  render_template('review/create.html',
44 movie=movie)
  45  
  46 # get a single review
  47 def get_review(id, check_author=True):
  48     review = get_review_from_db(id)
  49     if review is None:
        abort(404, f"Post id {id} doesn't
 
50 exist.")
    if check_author and
 
51 review['author_id'] != g.user['id']:
  52         abort(403)
  53     return review
  54  
  55 # update review
@bp.route('/<int:id>/update', methods=
 
56 ('GET', 'POST'))
  57 @login_required
  58 def update(id):
  59     review = get_review(id)
  60     if request.method == 'POST':
  61         title = request.form['title']
  62         body = request.form['body']
  63         error = None
  64         if not title:
  65             error = 'Title is required.'
  66         if error is not None:
  67             flash(error)
  68         else:
  69             db = get_db()
  70             db.execute(
  71                 'UPDATE review SET title = ?,
body = ?'
  72                 ' WHERE id = ?',
  73                 (title, body, id)
  74             )
  75             db.commit()
            return
 
76 redirect(url_for('review.reviews'))
    return
  render_template('review/update.html',
77 review=review)
  78  
  79 # delete review
@bp.route('/<int:id>/delete', methods=
 
80 ('POST',))
  81 @login_required
  82 def delete(id):
  83     get_review(id)
  84     db = get_db()
    db.execute('DELETE FROM review
 
85 WHERE id = ?', (id,))
  86     db.commit()
    return
 
87 redirect(url_for('review.reviews'))
 
Review Templates:
We will have three separate templates
for review operations:
create.html
update.html
reviews.html
 To define these files, create a new folder
called review under the templates directory.
Here are the file contents:
 
review/create.html
 
[19]: 1 {% extends 'base.html' %}
  2  
  3 {% block content %}
  4   <div class="login-form">
  5       <form method="post">
  6         <h1>{{ movie['title'] }}</h1>
  7         <div class="content">
  8           <div class="input-field">
            <input class="input-text"
  id="title" name="title" value="Review
9 for: {{ movie['title'] }}">
  10           </div>
  11           <div class="input-field">
            <textarea class="input-text"
 
12 id="body"
                      name="body"
  placeholder="Review" rows="6">
13 </textarea>
  14           </div>
  15         </div>
  16         <div class="action">
          <button class="actionbtn"
  type="submit">Create
17 Review</button>
  18         </div>
  19       </form>
  20     </div>
  21 {% endblock %}
 
review/update.html:
 
[20]: 1 {% extends 'base.html' %}
  2  
  3 {% block content %}
  4   <div class="login-form">
  5       <form method="post">
  6         <h1>{{ review['title'] }}</h1>
  7         <div class="content">
  8           <div class="input-field">
            <input class="input-text"
 
9 id="title" name="title"
                   value="{{
  request.form['title'] or review['title']
10 }}">
  11           </div>
  12           <div class="input-field">
            <textarea class="input-text"
 
13 id="body"
  14                       name="body" rows="6">
              {{ request.form['body'] or
 
15 review['body'] }}
  16             </textarea>
  17           </div>
  18         </div>
  19         <div class="action">
          <button class="actionbtn"
 
20 type="submit">Save</button>
  21         </div>
  22       </form>
  23  
  24       <form class="login-form"
            action="{{
  url_for('review.delete', id=review['id'])
25 }}"
  26             method="post">
  27         <div class="action">
          <input class="actionbtn danger"
 
28 type="submit"
                 value="Delete"
  onclick="return confirm('Are you
29 sure?');">
  30         </div>
  31       </form>
  32     </div>
  33 {% endblock %}
 
review/reviews.html:
 
[21]: 1 {% extends 'base.html' %}
  2  
  3 {% block header %}
  4   <div class="pageheader">
    <h1>{% block title %}All Reviews{%
 
5 endblock %}</h1>
  6   </div>
  7 {% endblock %}
  8  
  9 {% block content %}
  10   {% for review in all_reviews %}
  11     <article class="review">
  12       <div class="review-left">
  13         <div class="article-img">
  14           <img src="{{
 get_movie_img(review['id']) }}"
style="width: 100px;" />
  15         </div>
  16         <div class="article-content">
  17           <header>
  18             <div>
              <h3>{{ review['title'] }}
 
19 </h3>
  20               <h5 class="about">
                {{ review['fullname'] }}, {{
  review['created'].strftime('%Y-%m-%d')
21 }}
  22               </h5>
  23             </div>
  24           </header>
          <p class="article-body">{{
 
25 review['body'] }}</p>
  26         </div>
  27       </div>
  28       <div class="review-right">
        {% if g.user['id'] ==
 
29 review['author_id'] %}
  30             <a class="edit-button"
               href="{{
  url_for('review.update', id=review['id'])
31 }}">Edit</a>
  32           {% endif %}
  33       </div>
  34     </article>
  35     {% if not loop.last %}
  36       <hr>
  37     {% endif %}
  38   {% endfor %}
  39 {% endblock %}
  
Finally, please do not forget to register
the Review Blueprint with the application
factory. Here is the final form of the
__init__.py file:
 
[22]: 1 # __init__.py file
  2  
  3 import os
  4 from flask import Flask
from . import db, register, login, home,
 
5 review
  6  
  7 # application factory
  8 def create_app(test_config=None):
  9     # create a Flask app object
    app = Flask(__name__,
 
10 instance_relative_config=True)
  11  
  12     # configure the app
  13     app.config.from_mapping(
  14         SECRET_KEY='dev',
       
  DATABASE=os.path.join(app.instance_path,
15 'flaskapp.sqlite'),
  16     )
  17  
    # create the instance folder if not
 
18 exists
  19     try:
  20         os.makedirs(app.instance_path)
  21     except OSError:
  22         pass
  23  
  24     # db registration
  25     db.init_app(app)
  26  
  27     # register Blueprints
  28     app.register_blueprint(register.bp)
  29     app.register_blueprint(login.bp)
  30     app.register_blueprint(home.bp)
  31     app.register_blueprint(review.bp)
  32  
  33     # a simple test to see if it works
  34     @app.route('/test')
  35     def test():
        return 'Flaskapp init file works fine
 
36 :)'
  37  
    # return the configured Flask app
 
38 object
  39     return app
 
 
OceanofPDF.com
 Static Files
 
Static files are CSS files and other types
of files with JavaScript functions, or a logo
image. They should be placed under the
flaskapp/static directory and referenced with
url_for('static', filename='...') .
We have already addressed the CSS file
named style.css in the base.html file as
follows:
<link rel="stylesheet" href="{{
url_for('static', filename='style.css') }}">
 
Now let’s create this file. You can find the
file style.css file in the project folder in the
GitHub repository of the book. Here is the
final image of the PyCharm project:
 
 Figure 17-15: PyCharm project structure finalized
 
After you add the CSS file, the app will
look really nice. Here is the image of the
home page:
 
 Figure 17-16: Movie Review app completed
 
This finalizes our project of API
Development with Flask. In the next
chapter, you will have an assignment on
this project. Good luck with your
assignment.
 
 
OceanofPDF.com
 18. Assignment 3 – API
       

Development with Flask

 
OceanofPDF.com
 Assignment Setup
 
We finished the third project in this book
which is Project 3 – API Development with
Flask. Now it’s your turn to recreate the
same project. This chapter is the
assignment on the Project 3.
 
To complete the assignment, follow the
instructions where you see a #TODO
statement. You can find the solutions in
the previous chapter.
 
We will create a new PyCharm project.
The project name is MovieReview . You
should download the project from the
GitHub Repository of this book before
starting to code.
 
Chapter Outline:
Project Layout
Introduction to Flask
The Application Factory
Create the Database
Movies Data and The Utils Module
 Views and Blueprints
Register Blueprint and View Function
Templates
The Base Template
Register Template
Login Blueprint & Template
Home Blueprint & Template
Review Blueprint & Template
Static Files
 
 
OceanofPDF.com
 Project Layout
 
Our project is a simple app which we will
use for reviewing some movies in the IMDB
Top 250 list. The movie data will be provided
as a CSV file in the project. We will use the
built-in SQLite database for this project.
Here are the pages in our application:
 
Home Page:
In the home page, we will see the list of
movies in a responsive grid. By clicking
any of the movies, you can either create a
review or update/delete it if it is already
created. The movies with a review will be
displayed with a green border around
them.
The first tile in the grid is the “All Reviews”
button, which will navigate you to a list of
all reviews.
We have a menu bar on top of the page.
On the menu bar on the left, we have the
app title which is also an anchor tag for
the Home Page. On the right, you see the
logged in user’s full name and a Logout
button.
  

Figure 17-1: Home Page

 
Register:
To be able to use the app, the user needs
to register and login in to the system. We
have two different pages for this. The first
page is the Register page. In which, user
will enter the full name, user name and a
password. All of which are required to fill.
 
 Figure 17-2: Register
 
Login:
After successful registration, user needs to
login in. We save user data in the user
table, so, the user can login any time with
correct user name and password. Inside
the app, we check if the credentials are
true or not. If not, we notify the user to try
again.
User can easily navigate to the Register
page from the Login screen.
 
 Figure 17-3: Login
Create a Review:
The logged in user can create a review for
any movie on the Home Page. By clicking
its tile in the Home Page, the user will be
navigated to the Create page. In Figure 17-
4, in the page url, you can see the Id of the
movie, for which the user will create a
review.
 
 Figure 17-4: Create a Review
 
Update the Review:
The user can update a review, by clicking
its tile in the Home Page. He/She will be
navigated to the Update page. In Figure
17-5, in the page url, you can see the Id of
the movie, which the user will update the
existing review.
Before leaving the Home Page, we check if
there is already a review for that movie. If
there is, then the Update page will render.
If not, then the user will see the Create
page.
 
 Figure 17-5: Update or Delete a Review
Reviews:
In the Home Page, we have a tile called
“All Reviews”. When clicked, it will
navigate the user to the Reviews Page.
This is where we see all the reviews in the
database. If the review is created by the
current user, then you will see an “Edit”
button on the right-hand side of that
review. So, the user can view all of the
reviews, but can edit only the ones which
he/she created.
 

Figure 17-6: All Reviews

 
 
OceanofPDF.com
 Introduction to Flask
 
We will use Flask for building our app.
Flask was created by Armin Ronacher in
2004. It is a microframework written in
Python. It is a lightweight WSGI (Web Server
Gateway Interface) web application
framework, which is designed to make
getting started quick and easy, with the
ability to scale up to complex applications.
Flask supports extensions that can add
application features like database
abstraction layer, form validation, etc.
A Flask application is an instance of the
Flask class. Everything about the
application, such as configuration and URLs,
will be registered with this class.
To understand Flask better, let’s start with
a small example. We will create a simple
web API, which simply renders “Hi from Flask
App” in the web browser.
We will create a new PyCharm project
named MovieReview with a new virtual
environment. And create a Python file
named hi.py .
 We have to install the Flask package first.
You can easily install it in PyCharm or you
can use pip to install it. Here is the official
documentation if you need a detailed
installation guideline.
 

Figure 17-7: Install Flask package in PyCharm IDE

 
[1]: 1 # hi.py file
  2  
# TODO - import Flask class from flask
 
3 package
  4  
  5 # TODO - create a Flask app
  6  
  7 # TODO - define a root for '/hi'
  8 def hi_flask():
  9     return "Hi from Flask App"
 
 In cell 1, we have a very simple Flask app
in the hi.py file. We import the Flask class
first. Then in line 7, we create an instance of
the Flask class and name it simply as app .
This object will be our WSGI application. The
first argument is the name of the
application’s module or package. __name__ is
a convenient shortcut for this that is
appropriate for most cases. This is needed
so that Flask knows where to look for
resources such as templates and static files.
In line 10, we use the route() decorator to
tell Flask what URL should trigger the
function in the next line. Here, the URL will
be ‘/hi’ . Our hi_flask() function will return a
text message which will be displayed in the
user’s browser.
To run a Flask application, we need to use
the ‘flask’ command or ‘python -m flask’
command. In this project, we prefer to use
the ‘flask’ command. We need to make
some settings in the terminal before running
the ‘flask’ command. We need to tell the
terminal the application to work with by
exporting the FLASK_APP environment
variable. Here are the commands for this,
on different operating systems:
  
  Bash (macOS and Linux):
  ------------------------
  $ export FL ASK_APP=hi
  $ export FL ASK_ENV=development
  $ flask run
   * Running on http://127.0.0.1:5000/
   
   
  CMD (Windows):
  ------------------------
  > set FL ASK_APP=hi
  > set FL ASK_ENV=development
  > flask run
   * Running on http://127.0.0.1:5000/
   
   
  Powershell (Windows):
  ------------------------
  > $env:FL ASK_APP = "hi"
  > $env:FL ASK_ENV = "development"
  > flask run
   * Running on http://127.0.0.1:5000/
 
To be able to run these commands in your
terminal (either standalone or in PyCharm)
you need to be in the top-level directory of
the project. The top-level directory for this
project is the MovieReview project folder. In
PyCharm to start a terminal, right click on
the top-level folder and select Open In ->
Terminal . See the image below:
 

Figure 17-8: Open In Terminal in PyCharm

 
Now we can run the terminal command to
start our Flask app. See the image below on
macOS machine:
 
 Figure 17-9: Running a Flask App in PyCharm Terminal on
macOS
 
When you run these commands, your
Flask app will start on your local machine.
By default, the IP address for the localhost
will be http://127.0.0.1 and the port will be
5000 . So, the complete URL will become
http://127.0.0.1:5000 . Open this URL in a
browser window and add the ‘/hi’ route to
the end as: http://127.0.0.1:5000/hi . When
you call this URL in the browser, it will
render a string as “Hi from Flask App” . See
the image below:
 

Figure 17-10: Flask App running on the browser

 
If you can see a screen like the one in
Figure 17-10, then congratulations, your
first Flask app is up and running. You can
move to the next section.
 
 
OceanofPDF.com
 The Application Factory
 
In the previous section, we created a
Flask app instance inside the hi.py file. It
was just a simple example and in the real-
world project, there is a better way of
creating Flask apps. We have a special
function known as The Application
Factory. The Application Factory is
responsible for any configuration,
registration and other setup operations. And
it returns the configured app object, which is
an instance of the Flask class.
 
flaskapp directory:
In a Flask project the actual project has its
own directory. Keep in mind that, the Flask
project is not the same thing as the
PyCharm project. Flask project is a sub-
folder of the PyCharm project. So, inside our
MovieReview PyCharm project we will have
a Flask project as flaskapp . Here is the folder
structure:
 
 Figure 17-11: Flask App inside the PyCharm project
 
As you see in Figure 17-11, MovieReview is
our PyCharm Project, and the flaskapp is the
Flask project in it. flaskapp is the directory
where we will instantiate a Flask instance.
Actually, it will be a Python package
because it will contain the __init__.py file.
Now let’s create the flaskapp directory
(package) in our MovieReview project, if you
haven’t already. Make sure you have
__init__.py file in it. The __init__.py will contain
the application factory for our Flask app,
and it tells Python that the flaskapp
directory should be treated as a package.
Here is the content of this file:
 
[2]: 1 # __init__.py file
  2  
  3 import os
  4 from flask import Flask
from . import db, register, login, home,
 
5 review
  6  
  7 # application factory
  8 def create_app(test_config=None):
  9     # TODO - create a Flask app object
    # with __name__ and
 
10 instance_relative_config=True
  11  
  12     # configure the app
  13     app.config.from_mapping(
  14         SECRET_KEY='dev',
       
  DATABASE=os.path.join(app.instance_path,
15 'flaskapp.sqlite'),
  16     )
  17  
  18     # create the instance folder if not exists
  19     try:
  20         os.makedirs(app.instance_path)
  21     except OSError:
  22         pass
  23  
  24     # db registration
  25     # TODO - register the database
  26  
  27     # register Blueprints
  28     # register
  29     # login
  30     # home
  31     # review
  32  
    # TODO - create a simple test to see if it
 
33 works
  34     # '/test'
  35  
    # TODO - return the configured Flask
 
36 app object
 
In cell 2, you see __init__.py file content.
This is the initial form of this file and we will
modify it constantly. The create_app()
function is the application factory itself.
Make sure you give exactly the same name,
because “create_app” is the name Flask uses
for application factory by default.
In line 9 we create a Flask app instance
as: app = Flask(__name__,
instance_relative_config=True) . The
parameters are:
__name__ : name of the current Python
module. This tells the app where it’s
located to set up relative paths.
instance_relative_config=True : This
parameter tells the app that
configuration files are relative to the
instance folder. The instance folder is a
special folder which is located outside
the flaskapp package and holds local
data like configuration secrets and
database files. It will automatically be
created when we run Flask in a
minute.
In line 12, we configure our Flask app
object. The app.config.from_mapping()
method sets some default configuration
that the app will use like SECRET_KEY and
DATABASE . We will use SQLite database for
this project.
SECRET_KEY is used by Flask to keep the
app data safe. During development
you can set it to 'dev' , but you should
override it with a random value before
deploying the app.
 DATABASE is the path where we save
the SQLite database file. It’s under
app.instance_path , which is the path
that Flask has chosen for the instance
folder. Don’t worry, it will make sense
when we create our database.
In line 19, we create the instance folder
as: os.makedirs(app.instance_path) . This folder
will hold our local database file when we
initialize the db.
In line 24, we have a simple route for test
purposes. We want to see if our Flask app
works when we run this code. The url for
this route is simply ‘/test’ .
And finally, in line 29, we return the
configured Flask app object.
Now let’s run this code and test it. To run
it, start a new Terminal window in PyCharm
and set a new environment variable for our
flaskapp project. Make sure you are at the
top-level folder in the PyCharm project
which is MovieReview directory.
 
  Bash (macOS and Linux):
  ------------------------
  $ export FL ASK_APP=flaskapp
  $ export FL ASK_ENV=development
  $ flask run
   
   
  CMD (Windows):
  ------------------------
  > set FL ASK_APP=flaskapp
  > set FL ASK_ENV=development
  > flask run
   
   
  Powershell (Windows):
  ------------------------
  > $env:FL ASK_APP = "flaskapp"
  > $env:FL ASK_ENV = "development"
  > flask run
 
This time the environment variable name
is flaskapp , which is the same name with our
flask project. If you run this code in your
PyCharm terminal, then it should start a
new Flask app on the given local IP address
and port. You should see an output similar
to the one below when you run these
commands:
 
(venv) MovieReview % export
  FL ASK_APP=flaskapp  
(venv) MovieReview % export
  FL ASK_ENV=development
  (venv) MovieReview % flask run                  
  * Serving Flask app 'flaskapp' (lazy loading)
  * Environment: development
  * Debug mode: on
  * Running on http://127.0.0.1:5000 (Press
CTRL+C to quit)
  * Restarting with stat
  * Debugger is active!
  * Debugger PIN: 885-298-466
 
Now let’s check it on the web browser.
When you type http://127.0.0.1:5000/test in
the browser, you should see a text of
“Flaskapp init file works fine :)” . See the
images below:
 

Figure 17-12: Running flaskapp in PyCharm

 
 Figure 17-13: The result in the browser for
http://127.0.0.1:5000/test
 
If you see a similar result like the one in
Figure 17-13, then your application factory
is set up correctly. You can move on to the
next section now. Which is about creating a
local database.
 
 
OceanofPDF.com
 Create The Database
 
As we mentioned earlier, we will use
SQLite database to store our app data.
SQLite is a light-weight database server
which doesn’t require setting up a
separate db server and is built-in to
Python. It is convenient for small projects,
however for large applications it may not
serve well due to its limitations.
We will create a separate Python file in
our flaskapp folder, called db.py . This file
will setup and initialize the database. Here
is the code in this file:
 
[3]: 1 # db.py file
  2  
  3 import sqlite3
  4 import click
  5 from flask import current_app, g
  6 from flask.cli import with_appcontext
  7  
  8 # db connection
  9 def get_db():
  10     if 'db' not in g:
        g.db = # TODO - create sqlite3
 
11 connection
  12         g.db.row_factory = sqlite3.Row
  13     return g.db
  14  
  15 # close connection
  16 def close_db(e=None):
  17     db = g.pop('db', None)
    # TODO - check if connection exists
 
18 and close
  19  
  20 # initialize db
  21 def init_db():
  22     db = # TODO - get db
    with
  current_app.open_resource('schema.sql')
23 as f:
  24         # TODO - execute script on db
  25  
  26 # command line command
# TODO - define a click command as 'init-
 
27 db'
  28 # TODO - call with_appcontext decorator
  29 def init_db_command():
    """Clear the existing data and create
 
30 new tables."""
    # TODO - initialize the db and print
 
31 the result
  32  
  33 # register functions with app instance
  34 def init_app(app):
    # TODO - call teardown_appcontext fn
 
35 to close db
  36     # TODO - add cli command to the app
 
In cell 3, you see the code in our db.py
file. In line 5, we import current_app and g
objects from the flask package:
g is a special object which is unique
for each request. It stores the data
which is accessible for multiple
functions during the current request.
current_app is an object that points to
the current Flask application request.
sqlite3.connect() establishes a
connection to the file pointed at by
the DATABASE configuration key.
sqlite3.Row tells the connection to
return rows that behave like dicts,
which allows accessing the columns
by their names.
 
The functions are:
get_db() : creates a db connection and
sets it to the g.db attribute
close_db() : checks and closes the db
connection if it exists
init_db() : initializes a database by
reading the 'schema.sql' file, which
contains the SQL commands to
create the database tables.
init_db_command() : it calls the init_db()
function and prints the result. It is
 decorated with @click.command('init-
db') and @with_appcontext .
click.command() defines a command line
command called init-db which calls the
init_db() function and shows a success
message to the user.
with_appcontext decorator is in the
flask.cli module and wraps a callback to
guarantee it will be called with a
script's application context.
init_app() is used to register the
functions with the app instance,
which is the incoming parameter. We
will call this function from the
application factory.
app.teardown_appcontext() tells Flask to
call that function when cleaning up
after returning the response.
app.cli.add_command() adds a new
command that can be called with the
flask command. This enables us to use
“flask init-db” command in the terminal.
 
Now let’s create the 'schema.sql' file
under the flaskapp directory. Here is the
code in this file:
  
[4]: 1 -- schema.sql file
  2  
  3 DROP TABLE IF EXISTS user;
  4 DROP TABLE IF EXISTS review;
  5  
  6 CREATE TABLE user (
  id INTEGER PRIMARY KEY
 
7 AUTOINCREMENT,
  8   username TEXT UNIQUE NOT NULL,
  9   password TEXT NOT NULL,
  10   fullname TEXT NOT NULL
  11 );
  12  
  13 CREATE TABLE review (
  14   id INTEGER PRIMARY KEY,
  15   author_id INTEGER NOT NULL,
  created TIMESTAMP NOT NULL
 
16 DEFAULT CURRENT_TIMESTAMP,
  17   title TEXT NOT NULL,
  18   body TEXT NOT NULL,
  FOREIGN KEY (author_id)
 
19 REFERENCES user (id)
  20 );
 
In cell 4, you see the SQL commands to
create two database tables; user and
review . Each table has fields in their
definition commands. Before running
these commands, we need to call the
init_app() function from the application
factory. Here is updated __init__.py file
content:
 
[5]: 1 # __init__.py file
  2  
  3 import os
  4 from flask import Flask
from . import db, register, login, home,
 
5 review
  6  
  7 # application factory
  8 def create_app(test_config=None):
  9     # TODO - create a Flask app object
    # with __name__ and
 
10 instance_relative_config=True
  11  
  12     # configure the app
  13     app.config.from_mapping(
  14         SECRET_KEY='dev',
       
  DATABASE=os.path.join(app.instance_path,
15 'flaskapp.sqlite'),
  16     )
  17  
    # create the instance folder if not
 
18 exists
  19     try:
  20         os.makedirs(app.instance_path)
  21     except OSError:
  22         pass
  23  
  24     # db registration
  25     # TODO - register the database
  26  
  27     # register Blueprints
  28     # register
  29     # login
  30     # home
  31     # review
  32  
    # TODO - create a simple test to see if
 
33 it works
  34     # '/test'
  35  
    # TODO - return the configured Flask
 
36 app object
 
In cell 5, line 35, we register the
database with our application by calling
the db.init_app() function and passing the
current app object as the argument.
Now we are ready to initialize the
database file. All we need to do is to call
the “flask init-db” command in the
terminal. Make sure you stop the Flask
server by pressing CTRL+C to quit. And set
the FLASK_APP and the FLASK_ENV
environment variables again before
running the “flask init-db” command. Here
is the output on a macOS computer:
 
  (venv) MovieReview % export
FL ASK_APP=flaskapp  
(venv) MovieReview % export
 
FL ASK_ENV=development
  (venv) MovieReview % flask init-db
  Successfully initialized the database.
 
If you see the text of ‘Successfully
initialized the database.’ in the output, then
your database tables are created and the
database is initialized in the application. If
you check the “instance” folder in the
project now, you should see the
flaskapp.sqlite file in it. This is the file
where our database will be stored. You can
now run “flask run” command to start the
server again.
 
 
OceanofPDF.com
 Movies Data and The Utils Module
 
We will be using movies data which
belongs to the top 20 movies in the IMDB
Top 250 movies. The data will be in a CSV
file inside a folder called ‘data’ . File name
is imdb_top_20.csv and you can find it in
the project directory in GitHub repository
of this book.
In our project, we will also have a
module called utils.py . This module will
contain some common functions, like
reading CSV data and getting movies,
getting the user and review data from
database etc. Here is the content of this
file:
 
[6]: 1 # utils.py file
  2  
  3 from flask import (g, redirect, url_for)
  4 from flaskapp.db import get_db
  5 import csv
  6 import functools
  7  
  8 # function to read all movies
  9 def get_movies():
  10     """reads the csv file and returns
  11         the list of movies"""
  12     movies = []
  13     movie_path = 'data/imdb_top_20.csv'
  14     # TODO - read movies at movie_path
  15     # and fill the movies list
  16  
  17     # return the list
  18     return movies
  19  
  20 # function to read a single movie
  21 def get_movie(id):
  22     """returns a single movie"""
  23     movie = None
  24     movie_path = 'data/imdb_top_20.csv'
  25     # TODO - get a single movie with id
  26  
  27     return movie
  28  
  29 # get movie image
  30 def get_movie_img(id):
    # TODO - return 'imageURL' of the
 
31 movie with id
  32  
  33 # get all reviews
  34 def get_all_reviews():
  35     db = # TODO - get db
  36     # TODO - get all reviews from DB
    # 'SELECT r.id, title, body, created,
 
37 author_id, username, fullname'
    #         ' FROM review r JOIN user u
 
38 ON r.author_id = u.id'
  39     #         ' ORDER BY created DESC'
  40  
  41 # get review from db
  42 def get_review_from_db(id):
    # TODO - get a single movie from db
 
43 with id
    # 'SELECT r.id, title, body, created,
 
44 author_id, username'
    #         ' FROM review r JOIN user u
 
45 ON r.author_id = u.id'
  46     #         ' WHERE r.id = ?'
  47  
  48 # get user
  49 def get_user(username):
  50     db = # TODO - get db
    # TODO - get as single user from db
 
51 with username
    # 'SELECT * FROM user WHERE
 
52 username = ?'
  53     return user
  54  
  55 # Require Authentication
  56 def login_required(view):
  57     @functools.wraps(view)
  58     def wrapped_view(**kwargs):
  59         if g.user is None:
            return
 
60 redirect(url_for('login.login'))
  61         return view(**kwargs)
  62     return wrapped_view
 
In cell 6, we have the utils.py file
content. Here are the functions in it:
get_movies() : This function reads the
csv file and returns the list of
movies.
get_movie() : This function returns a
single movie based on the given id
parameter.
get_movie_img() : This function returns
the movie image URL based on the
given id.
get_all_reviews() : It returns the list of
all reviews in the database. It uses a
SQL join statement to join the review
and user tables. We call the fetchall()
method to get a list of results.
get_review_from_db() : It returns a
single review from the database. We
use fetchone() for returning a single
result.
get_user() : It returns the user data
from the database based on the
username provided.
login_required() : We will require the
user to login to our application
before creating, editing and deleting
movie reviews. login_required is the
decorator which will check if the
user is logged in before any of these
operations. It returns a new view
 function that wraps the original view
it’s applied to. The new function
simply checks if the user is logged in
as: if g.user is None . Remember that
the object g stores the data in the
current request. If g.user is not None ,
this means that the user is logged in.
You will see how we fill it later on. If
it is None , then we redirect the user
to the login page.
 
url_for(function, parameters) :
The url_for() function in Flask,
generates a URL to an endpoint using
the function passed in as an argument.
For example, the test() function that
was added to the application factory in
the previous section, has the name
'test' and can be linked to with
url_for('test') . You can think of it as
requesting the URL of
http://127.0.0.1:5000/test in your web
browser.
If it took an argument, which you’ll see
later, it would be linked to using
 url_for('test', param_1=value_1,
param_2=value_2) .
In line 68, we use this function as
url_for('login.login') . The parameter is
‘login.login’ . The first ‘login’ is the
Blueprint name and the second one is
the endpoint name. So, this means
that, we redirect the user to an
endpoint (a function here) called ‘login’
inside a Blueprint named ‘login’ . Don’t
worry it will make sense when you
learn about the Blueprints.
 
 
OceanofPDF.com
 Views and Blueprints
 
In a Flask application the UI (user
interface) is mainly created by Templates
(HTML templates) and the back-end is
created by Views and Blueprints.
Views are the functions which respond
to the requests. Views handle the
incoming request based on the request
URL patterns. And they return data that
Flask turns into an outgoing response.
Blueprints are the way to organize a
group of related views. They can contain
any number of view functions. Views are
registered with a Blueprint and the
Blueprint is registered with the application
factory.
In our project we will have four
Blueprints which are: home , login , register ,
and review . Each one will have a separate
module and in each of them we will have
different View functions to handle
incoming requests.
 
 
OceanofPDF.com
 Register Blueprint and View Function
 
A Blueprint is a class in the flask
module, which is used to register the
Views (View functions). The name of our
first Blueprint will be Register and it will
have a View function called register() .
Create a new Python file in the flaskapp
directory called register.py . And here is the
code in this file:
 
[7]: 1 # register.py file
from flask import (Blueprint, flash,
 
2 redirect,
                   render_template, request,
 
3 session, url_for)
  4 from flaskapp.db import get_db
from werkzeug.security import
 
5 generate_password_hash
  6 from .utils import get_user
  7  
  8 # initialize the Blueprint object
bp = Blueprint('register', __name__,
 
9 url_prefix='/')
  10  
  11 # Register View Code
@bp.route('/register', methods=('GET',
 
12 'POST'))
  13 def register():
  14     if request.method == 'POST':
  15         # get input fields
        username, password, fullname =
 
16 get_input_fields()
  17         # get db
  18         db = # TODO - get db
  19         # check input fields
        error =
  check_input_fields(username, password,
20 fullname)
  21         # check error
  22         if error is None:
  23             try:
                # TODO - insert new user into
 
24 db
  25             except db.IntegrityError:
                error = f"User {username} is
 
26 already registered."
  27             else:
  28                 # TODO - clear session
  29                 session.clear()
                session['user_id'] =
 
30 get_user(username)['id']
                return # TODO - redirect to
 
31 url for 'home.index'
  32         flash(error)
    return # TODO - render template at
 
33 'register/register.html'
  34  
  35 # fn for getting inputs
  36 def get_input_fields():
  37     username = request.form['username']
  38     # TODO - get password from request
  39     # TODO - get fullname from request
  40     return username, password, fullname
  41  
  42 # fn for checking input fields
def check_input_fields(username,
 
43 password, fullname):
    if not username or not password or
 
44 not fullname:
        return 'Full Name, Username or
 
45 Password can not be empty.'
  46     else:
  47         return None
  48  
  49 # fn for inserting a DB row
def insert_into_db(db, username,
 
50 password, fullname):
  51     # TODO - insert new row to db
    # "INSERT INTO user (username,
 
52 password, fullname) "
  53     #                "VALUES (?, ?, ?)"
  54  
  55     # TODO - commit to db
 
In cell 7, line 9, we create a Blueprint
object as: bp = Blueprint('register', __name__,
url_prefix='/') . The first parameter is the
unique name of this Blueprint the second
one is the name of the current application
and the third parameter is the URL prefix.
In line 13 you see the definition of the
register() view function. Here is what this
function does:
@bp.route decorator associates the
URL “/register ” with the register()
view function. When Flask receives a
request to
http://127.0.0.1:5000/register, it will
call the register() view and use the
return value as the response. This
view function can handle both GET
and POST request types.
In line 14, we check the request
method being POST , which means
the user submits the form in this
view. If the method is POST, then we
have some validations for the form
fields like username , password and
fullname . We don’t want any of them
to be empty, that’s why we call the
check_input_fields() function.
If all of the fields are filled by the
user, then we can try to insert this
into the user table in db. That’s what
we do in line 25, by calling the
insert_into_db() function.
If db insert operation is successful,
then in the else block in line 29, we
clear the old session and add the
 user id to it. Then we redirect the
user to the home page by calling the
url_for('home.index') function. We will
define the home.index view function
later on.
In line 32, we call a special function
called flash() by passing the error as
the argument. This function stores
the error messages that can be
retrieved when rendering the
template. We will print the error
messages later on in our HTML
template.
Finally, in line 33, we call the
render_template() function by passing
the file location of the template. This
will render the HTML template for
the 'register/register.html' file in the
web browser. We will create this file
soon.
 
In line 50, we have the definition of the
insert_into_db() function. It simply inserts a
new row to the database. For security
reasons, we call the
generate_password_hash() function instead
of saving the password as it is. This will
store the hash value of the password.
To be able to use our Register Blueprint,
we need to register it inside our
application factory as follows:
 
[8]: 1 # __init__.py file
  2  
  3 import os
  4 from flask import Flask
from . import db, register, login, home,
 
5 review
  6  
  7 # application factory
  8 def create_app(test_config=None):
  9     # TODO - create a Flask app object
    # with __name__ and
 
10 instance_relative_config=True
  11  
  12     # configure the app
  13     app.config.from_mapping(
  14         SECRET_KEY='dev',
       
  DATABASE=os.path.join(app.instance_path,
15 'flaskapp.sqlite'),
  16     )
  17  
    # create the instance folder if not
 
18 exists
  19     try:
  20         os.makedirs(app.instance_path)
  21     except OSError:
  22         pass
  23  
  24     # db registration
  25     # TODO - register the database
  26  
  27     # register Blueprints
  28     # register
  29     # login
  30     # home
  31     # review
  32  
    # TODO - create a simple test to see if
 
33 it works
  34     # '/test'
  35  
    # TODO - return the configured Flask
 
36 app object
 
Now that we have our Register Blueprint
and a view function for it, let’s request its
URL in the web browser. If you type
http://127.0.0.1:5000/register in your
browser and press on Enter, you will see
an error page as TemplateNotFound . It will
say: jinja2.exceptions.TemplateNotFound:
register/register.html . This tells us that
Flask couldn’t find any Templates for
rendering. We will create HTML templates
in the succeeding sections.
 
 
OceanofPDF.com
 Templates
 
Templates are files that contain static
data as well as placeholders for dynamic
data. A template is rendered with specific
data to produce a final document. All the
template files in a Flask app need to be
located under a special folder called
“templates” . This folder should be directly
under the flaskapp directory and it is the
default location where Flask looks for
templates at run time.
Flask uses the Jinja template library to
render templates. In our application, we
will use templates to render HTML which
will display in the user’s browser.
A Jinja template is simply a text file.
Jinja can generate any text-based format
(HTML, XML, CSV, LaTeX, etc.). A Jinja
template doesn’t need to have a specific
extension: .html, .xml, or any other
extension is just fine.
A template contains variables and/or
expressions, which get replaced with
values when the template is rendered;
and tags, which control the logic of the
template.
The template syntax in Jinja is heavily
inspired by Python. Special delimiters are
used to distinguish Jinja syntax from the
static data in the template. Anything
between {{ and }} is an expression that
will be output to the final document. {%
and %} denotes a control flow statement
like if and for . Unlike Python, blocks are
denoted by start and end tags rather than
indentation since static text within a block
could change indentation. Here are a few
kinds of delimiters:
{% ... %} for Statements
{{ ... }} for Expressions to print to
the template output
{# ... #} for Comments not included
in the template output
 
 
OceanofPDF.com
 The Base Template
 
In our application, each page will have the
same basic layout around a different body. For
example, they will all include a navigation bar at
the top. Instead of writing the entire HTML
structure in each template, each template will
extend a base template and override specific
sections. You can think of the base templates as
the master pages.
In our flaskapp project directory, we will
create a new folder called templates , and inside
this folder, create a new HTML file named
base.html . Here is the code in this file:
 
[9]: 1 <!doctype html>
  2 <html lang="en" >
  3 <head>
  4   <meta charset="UTF-8">
  <link rel='stylesheet'
  href='https://fonts.googleapis.com/css?
5 family=Rubik:400,700'>
  6   <link rel="stylesheet" href="./style.css">
  <link rel="stylesheet" href="{{ url_for('static',
 
7 filename='style.css') }}">
  8   <title>Flask App</title>
  9 </head>
  10 <body>
  11  
  12 <div class="menu">
  <a class="title-anchor" href="{{ url_for('home.
 
13 }}"><h1>Movie Review with Flask</h1></a>
  14   <div class="menu-right">
  15     {% if g.user %}
  16       <h2>{{ g.user['fullname'] }}</h2>
      <a href="{{ url_for('login.logout') }}">Log
 
17 Out</a>
  18     {% else %}
      <a href="{{ url_for('register.register')
 
19 }}">Register</a>
  20       <a href="{{ url_for('login.login') }}">Log In<
  21     {% endif %}
  22   </div>
  23 </div>
  24 <div class="content">
  25   <header>
  26     {% block header %}{% endblock %}
  27   </header>
  28   {% for message in get_flashed_messages() %}
  29     <div class="flash">{{ message }}</div>
  30   {% endfor %}
  31   {% block content %}{% endblock %}
  32 </div>
  33 </body>
  34 </html>
 
In cell 9, we have the content of base.html file
in the templates folder. It is a regular HTML file
which includes Jinja commands in it.
In line 15, we check the user in the current
request as: {% if g.user %} . And if g.user is not
None , we are sure that the user is logged in and
we render a logout link button. Else, we render
Register and Log In buttons.
In line 16, inside the <h2> tag, we render the
user fullname as: {{ g.user['fullname'] }} .
 In line 26, we have a placeholder for the
header block as: {% block header %}{% endblock
%} .
There are two blocks defined in the base
template which will be overridden in the child
templates:
{% block header %} is where the unique
header of each page will replace.
{% block content %} is where the content of
each page goes, such as the register or
login forms.
In line 28, we check if there are any error
messages in the view functions. Remember that
we called the flash(error) function in the view
code for storing the error messages. And this is
where we get these messages by calling the
get_flashed_messages() function. We set a for
loop to iterate over the messages and render a
div element for each one. Here is the code: {%
for message in get_flashed_messages() %} .
And finally, in line 31, we have the content
block.
 
 
OceanofPDF.com
 Register Template
 
Now that we have the base template,
we can extend it and create a new
template called the Register Template. It
will contain the HTML code to create our
register page. Inside the templates folder
create a sub-folder called register . And
inside this folder create an HTML
document as register.html . Here is the
code in this file:
 
[10]: 1 {% extends 'base.html' %}
  2  
  3 {% block content %}
  4     <div class="login-form">
  5       <form method="post">
  6         <h1>Register</h1>
  7         <div class="content">
  8           <div class="input-field">
            <input class="input-text"
  id="fullname" name="fullname"
9 placeholder="Full Name">
  10           </div>
  11           <div class="input-field">
            <input class="input-text"
  id="username" name="username"
12 placeholder="User Name">
  13           </div>
  14           <div class="input-field">
  15             <input class="input-text"
type="password" id="password"
name="password"
placeholder="Password">
  16           </div>
  17         </div>
  18         <div class="action">
          <button type="submit"
 
19 class="actionbtn">Sign Up</button>
  20         </div>
  21       </form>
  22     </div>
  23 {% endblock %}
 
In cell 10, we have the register
template. In the first line you see that it
extends the base template as: {% extends
'base.html' %} .
In line 3, we have the structure for the
content block. This will replace the content
block in the base template. The register
Template is a basic HTML form element
with some input fields in it.
Remember that in cell 7 line 33, inside
the register() view function we had the
return statement as: return
render_template('register/register.html') . This
is where we tell Flask to find and render
the template when it matches the
specified route, which was:
@bp.route('/register', methods=('GET',
'POST')) .
Now if request the URL
http://127.0.0.1:5000/register in the
browser and press on Enter, Flask will try
to render the register template. You will
still get a BuildError , because we do not
have a template named home.index yet.
Don’t worry we will create it in a minute.
 
 
OceanofPDF.com
 Login Blueprint & Template
 
Before creating the Blueprint for Home
Page, we will create the Login Page. As we
did for the Register page, first we will
create a Blueprint with some View functions
in it. Then we will create a template as
login.html .
 
Login Blueprint and View Functions:
For the Blueprint, create a Python file
named login.py under the flaskapp
directory. Here is the code for this file:
 
[12]: 1 # login.py file
  2  
from flask import (Blueprint, flash, g,
 
3 redirect,
                   render_template, request,
 
4 session, url_for)
  5 from flaskapp.db import get_db
from werkzeug.security import
 
6 check_password_hash
  7  
  8 # initialize the Blueprint object
bp = Blueprint('login', __name__,
 
9 url_prefix='/')
  10  
  11 # Login View Code
  12 @bp.route('/login', methods=['GET',
 'POST'])
  13 def login():
  14     if request.method == 'POST':
        username =
 
15 request.form['username']
        password =
 
16 request.form['password']
  17         db = # TODO - get db
  18         error = None
        # TODO - get the user from eb with
 
19 username
  20  
  21         # check user from db
  22         if user is None:
            error = "Incorrect username.
 
23 Please try again."
        elif not
  check_password_hash(user['password'],
24 password):
            error = "Incorrect password.
 
25 Please try again."
  26         # check error
  27         if error is None:
  28             # TODO - clear session
            session['user_id'] = # TODO - set
 
29 user id
            # TODO - return redirect url for
 
30 'home.index'
  31  
  32         # TODO - flash the error
  33  
  34     # render template
    return # TODO - render template at
 
35 'login/login.html'
  36  
  37 # Run before all view functions
  38 @bp.before_app_request
  39 def load_logged_in_user():
    user_id = # TODO - get user id from
 
40 session
  41     if user_id is None:
  42         g.user = None
  43     else:
        g.user = # TODO - get user from db
 
44 with user_id
  45  
  46 # logout route
  47 @bp.route('/logout')
  48 def logout():
  49     # TODO - clear session
    # TODO - return redirect url for
 
50 'login.login'
 
In cell 12, you see the code in the
login.py file. It creates a Blueprint called
‘login’ and define routes for '/login' and
'/logout' . Now we have to register this
Blueprint with the application factory. Here
it is:
 
[13]: 1 # __init__.py file
  2  
  3 import os
  4 from flask import Flask
from . import db, register, login, home,
 
5 review
  6  
  7 # application factory
  8 def create_app(test_config=None):
  9     # TODO - create a Flask app object
    # with __name__ and
 
10 instance_relative_config=True
  11  
  12     # configure the app
  13     app.config.from_mapping(
  14         SECRET_KEY='dev',
       
  DATABASE=os.path.join(app.instance_path,
15 'flaskapp.sqlite'),
  16     )
  17  
    # create the instance folder if not
 
18 exists
  19     try:
  20         os.makedirs(app.instance_path)
  21     except OSError:
  22         pass
  23  
  24     # db registration
  25     # TODO - register the database
  26  
  27     # register Blueprints
  28     # register
  29     # login
  30     # home
  31     # review
  32  
    # TODO - create a simple test to see if
 
33 it works
  34     # '/test'
  35  
    # TODO - return the configured Flask
 
36 app object
 
Login Template:
Now that we have a Blueprint for the
login page, we can define a Template for it.
To do this, create a new folder called login
under the templates directory. And in that
folder, create a file called login.html . Here is
the code in this file:
 
[14]: 1 {% extends 'base.html' %}
  2  
  3 {% block content %}
  4     <div class="login-form">
  5       <form method="post">
  6         <h1>Login</h1>
  7         <div class="content">
  8           <div class="input-field">
            <input class="input-text"
  id="username" name="username"
9 placeholder="User Name">
  10           </div>
  11           <div class="input-field">
            <input class="input-text"
type="password" id="password"
 
name="password"
12 placeholder="Password">
  13           </div>
          <a href="#" class="link">Forgot
 
14 Your Password?</a>
  15         </div>
  16         <div class="action">
          <button class="actionbtn"
 
17 type="submit">Sign in</button>
          <a class="actionbtn" href="{{
  url_for('register.register')
18 }}">Register</a>
  19         </div>
  20       </form>
  21     </div>
  22 {% endblock %}
 
 
OceanofPDF.com
 Home Blueprint & Template
 
Now let’s create a Blueprint and a
Template for the home page which we will
call as index.html .
 
Home Blueprint and View Function:
For the Blueprint, create a Python file
named home.py under the flaskapp directory.
Here is the code for this file:
 
[15]: 1 # home.py file
from flask import (Blueprint, g,
 
2 render_template)
from .utils import login_required,
 
3 get_all_reviews, \
  4     get_movies, get_review_from_db
  5  
  6 # initialize the Blueprint object
bp = Blueprint('home', __name__,
 
7 url_prefix='/')
  8  
  9 # Index View Code
  10 @bp.route('/', methods=['GET'])
  11 @login_required
  12 def index():
  13     # get all movies
  14     movies = # TODO - get movies
  15     # get all reviews
  16     all_reviews = # TODO - get all reviews
  17     # render index.html
  18     return # TODO - render template at
'home/index.html'
    # with parameters as: movies,
 
19 all_reviews, get_if_reviewed
  20  
  21 def get_if_reviewed(movie):
    review = # TODO - get review from db
 
22 for the movie with 'id'
    if review is not None and g.user['id']
 
23 == review['author_id']:
  24         return True
  25     else:
  26         return False
 
In the Home Blueprint, we don’t have a
specific route for the index() view function.
The route is: @bp.route('/', methods=['GET']) .
Which means it will be the main page when
the user opens the application by
requesting http://127.0.0.1:5000/.
In line 15 and 16, it gets all the movies
and the reviews by calling the respective
functions in the utils file.
It renders a template in the
'home/index.html' location which we will
create next. Here, we also pass some
parameters while rendering the template.
Here is the code:
    return render_template('home/index.html',
movies=movies,
                           all_reviews=all_reviews,
                          
get_if_reviewed=get_if_reviewed)
The first two parameters are list type
objects and the third one is a function,
which we will call from the HTML template.
This is how we pass parameters to
templates from the view functions.
Now we have to register this Blueprint
with the application factory. Here it is:
 
[16]: 1 # __init__.py file
  2  
  3 import os
  4 from flask import Flask
from . import db, register, login, home,
 
5 review
  6  
  7 # application factory
  8 def create_app(test_config=None):
  9     # TODO - create a Flask app object
    # with __name__ and
 
10 instance_relative_config=True
  11  
  12     # configure the app
  13     app.config.from_mapping(
  14         SECRET_KEY='dev',
       
  DATABASE=os.path.join(app.instance_path,
15 'flaskapp.sqlite'),
  16     )
  17  
  18     # create the instance folder if not
 exists
  19     try:
  20         os.makedirs(app.instance_path)
  21     except OSError:
  22         pass
  23  
  24     # db registration
  25     # TODO - register the database
  26  
  27     # register Blueprints
  28     # register
  29     # login
  30     # home
  31     # review
  32  
    # TODO - create a simple test to see if
 
33 it works
  34     # '/test'
  35  
    # TODO - return the configured Flask
 
36 app object
 
Index Template:
Now that we have a Blueprint for the
home page, we can define a Template for it.
To do this, create a new folder called home
under the templates directory. And in that
folder, create a file called index.html . Here is
the code in this file:
 
[17]: 1 {% extends 'base.html' %}
  2  
  3 {% block content %}
  4     <div class="login-form">
  5       <form method="post">
  6         <h1>Login</h1>
  7         <div class="content">
  8           <div class="input-field">
            <input class="input-text"
  id="username" name="username"
9 placeholder="User Name">
  10           </div>
  11           <div class="input-field">
            <input class="input-text"
type="password" id="password"
 
name="password"
12 placeholder="Password">
  13           </div>
          <a href="#" class="link">Forgot
 
14 Your Password?</a>
  15         </div>
  16         <div class="action">
          <button class="actionbtn"
 
17 type="submit">Sign in</button>
          <a class="actionbtn" href="{{
  url_for('register.register')
18 }}">Register</a>
  19         </div>
  20       </form>
  21     </div>
  22 {% endblock %}
 
At this point, if you run the app and type
http://127.0.0.1:5000/ in your web browser,
you should a screen like the one below:
  

Figure 17-14: Login page which is redirected from the Home

page
 
For the time being, the app doesn’t look
good because we have no formatting yet.
Don’t worry we will add some CSS and it will
look pretty.
 
 
OceanofPDF.com
 Review Blueprint & Template
 
The last Blueprint which we will add is
the Review Blueprint. It will be responsible
for creating new reviews, updating existing
ones, deleting reviews and displaying a list
of all reviews.
 
Review Blueprint and View Functions:
For the Blueprint, create a Python file
named review.py under the flaskapp
directory. Here is the code for this file:
 
[18]: 1 # review.py file
  2 from flask import (
  3     Blueprint, flash, g, redirect,
  4     render_template, request, url_for)
from werkzeug.exceptions import
 
5 abort
  6 from flaskapp.db import get_db
from .utils import login_required,
 
7 get_all_reviews, \
    get_movie, get_review_from_db,
 
8 get_movie_img
  9  
  10 bp = Blueprint('review', __name__)
  11  
  12 # reviews route
  13 @bp.route('/reviews')
  14 @login_required
  15 def reviews():
  16     TODO - get all reviews
    # TODO - render template at
 
17 'review/reviews.html'
    # with parameters: all_reviews,
 
18 get_movie_img
  19  
  20 # create a new review
@bp.route('/<int:id>/create', methods=
 
21 ('GET', 'POST'))
  22 @login_required
  23 def create(id):
  24     # TODO - get movie with id
  25  
  26     if # TODO - check if method is POST
        # TODO - get title and body from
 
27 the form
  28         error = None
  29         if not body:
            error = 'You need to add a
 
30 review.'
  31         if error is not None:
  32             flash(error)
  33         else:
  34             # TODO - get db
            # TODO - insert a new review
 
35 into db
            # 'INSERT INTO review (id, title,
 
36 body, author_id)'
            #                 ' VALUES (?, ?, ?,
 
37 ?)',
  38  
  39             # TODO - commit to db
  40  
  41             # TODO - redirect to
'review.reviews'
  42  
    # TODO - render template at
 
43 'review/create.html'
  44     # with parameter: movie
  45  
  46 # get a single review
  47 def get_review(id, check_author=True):
  48     # TODO - get review from db with id
  49     if review is None:
        abort(404, f"Post id {id} doesn't
 
50 exist.")
    if check_author and
 
51 review['author_id'] != g.user['id']:
  52         abort(403)
  53     return review
  54  
  55 # update review
@bp.route('/<int:id>/update', methods=
 
56 ('GET', 'POST'))
  57 @login_required
  58 def update(id):
  59     # TODO - get review from db with id
  60     if request.method == 'POST':
        # TODO - get title and body from
 
61 the form
  62         error = None
  63         if not title:
  64             error = 'Title is required.'
  65         if error is not None:
  66             flash(error)
  67         else:
  68             # TODO - get db
  69             # TODO - update the review in
db
            # 'UPDATE review SET title = ?,
 
70 body = ?'
  71             #                 ' WHERE id = ?'
  72  
  73             # TODO - commit to db
  74  
            # TODO - redirect to
 
75 'review.reviews'
  76  
    # TODO - render template at
 
77 'review/update.html'
  78     # with parameter: review
  79  
  80 # delete review
@bp.route('/<int:id>/delete', methods=
 
81 ('POST',))
  82 @login_required
  83 def delete(id):
  84     # TODO - get review with id
  85  
  86     # TODO - get db
  87     # TODO - delete the review with id
    # 'DELETE FROM review WHERE id =
 
88 ?'
  89  
  90     # TODO - commit to db
  91  
  92     # TODO - redirect to 'review.reviews'
 
Review Templates:
 We will have three separate templates
for review operations:
create.html
update.html
reviews.html
To define these files, create a new folder
called review under the templates directory.
Here are the file contents:
 
review/create.html
 
[19]: 1 {% extends 'base.html' %}
  2  
  3 {% block content %}
  4   <div class="login-form">
  5       <form method="post">
  6         <h1>{{ movie['title'] }}</h1>
  7         <div class="content">
  8           <div class="input-field">
            <input class="input-text"
  id="title" name="title" value="Review
9 for: {{ movie['title'] }}">
  10           </div>
  11           <div class="input-field">
            <textarea class="input-text"
 
12 id="body"
                      name="body"
  placeholder="Review" rows="6">
13 </textarea>
  14           </div>
  15         </div>
  16         <div class="action">
          <button class="actionbtn"
  type="submit">Create
17 Review</button>
  18         </div>
  19       </form>
  20     </div>
  21 {% endblock %}
 
review/update.html:
 
[20]: 1 {% extends 'base.html' %}
  2  
  3 {% block content %}
  4   <div class="login-form">
  5       <form method="post">
  6         <h1>{{ review['title'] }}</h1>
  7         <div class="content">
  8           <div class="input-field">
            <input class="input-text"
 
9 id="title" name="title"
                   value="{{
  request.form['title'] or review['title']
10 }}">
  11           </div>
  12           <div class="input-field">
            <textarea class="input-text"
 
13 id="body"
  14                       name="body" rows="6">
              {{ request.form['body'] or
 
15 review['body'] }}
  16             </textarea>
  17           </div>
  18         </div>
  19         <div class="action">
          <button class="actionbtn"
 
20 type="submit">Save</button>
  21         </div>
  22       </form>
  23  
  24       <form class="login-form"
            action="{{
  url_for('review.delete', id=review['id'])
25 }}"
  26             method="post">
  27         <div class="action">
          <input class="actionbtn danger"
 
28 type="submit"
                 value="Delete"
  onclick="return confirm('Are you
29 sure?');">
  30         </div>
  31       </form>
  32     </div>
  33 {% endblock %}
 
review/reviews.html:
 
[21]: 1 {% extends 'base.html' %}
  2  
  3 {% block header %}
  4   <div class="pageheader">
    <h1>{% block title %}All Reviews{%
 
5 endblock %}</h1>
  6   </div>
  7 {% endblock %}
  8  
  9 {% block content %}
  10   {% for review in all_reviews %}
  11     <article class="review">
  12       <div class="review-left">
  13         <div class="article-img">
          <img src="{{
  get_movie_img(review['id']) }}"
14 style="width: 100px;" />
  15         </div>
  16         <div class="article-content">
  17           <header>
  18             <div>
              <h3>{{ review['title'] }}
 
19 </h3>
  20               <h5 class="about">
                {{ review['fullname'] }}, {{
  review['created'].strftime('%Y-%m-%d')
21 }}
  22               </h5>
  23             </div>
  24           </header>
          <p class="article-body">{{
 
25 review['body'] }}</p>
  26         </div>
  27       </div>
  28       <div class="review-right">
        {% if g.user['id'] ==
 
29 review['author_id'] %}
  30             <a class="edit-button"
               href="{{
  url_for('review.update', id=review['id'])
31 }}">Edit</a>
  32           {% endif %}
  33       </div>
  34     </article>
  35     {% if not loop.last %}
  36       <hr>
  37     {% endif %}
  38   {% endfor %}
  39 {% endblock %}
 
Finally, please do not forget to register
the Review Blueprint with the application
factory. Here is the final form of the
__init__.py file:
 
[22]: 1 # __init__.py file
  2  
  3 import os
  4 from flask import Flask
from . import db, register, login, home,
 
5 review
  6  
  7 # application factory
  8 def create_app(test_config=None):
  9     # create a Flask app object
    app = Flask(__name__,
 
10 instance_relative_config=True)
  11  
  12     # configure the app
  13     app.config.from_mapping(
  14         SECRET_KEY='dev',
       
  DATABASE=os.path.join(app.instance_path,
15 'flaskapp.sqlite'),
  16     )
  17  
    # create the instance folder if not
 
18 exists
  19     try:
  20         os.makedirs(app.instance_path)
  21     except OSError:
  22         pass
  23  
  24     # db registration
  25     db.init_app(app)
  26  
  27     # register Blueprints
  28     app.register_blueprint(register.bp)
  29     app.register_blueprint(login.bp)
  30     app.register_blueprint(home.bp)
  31     app.register_blueprint(review.bp)
  32  
  33     # a simple test to see if it works
  34     @app.route('/test')
  35     def test():
        return 'Flaskapp init file works fine
 
36 :)'
  37  
    # return the configured Flask app
 
38 object
  39     return app
 
 
OceanofPDF.com
 Static Files
 
Static files are CSS files and other types
of files with JavaScript functions, or a logo
image. They should be placed under the
flaskapp/static directory and referenced with
url_for('static', filename='...') .
We have already addressed the CSS file
named style.css in the base.html file as
follows:
<link rel="stylesheet" href="{{
url_for('static', filename='style.css') }}">
 
Now let’s create this file. You can find the
file style.css file in the project folder in the
GitHub repository of the book. Here is the
final image of the PyCharm project:
 
 Figure 17-15: PyCharm project structure finalized
 
After you add the CSS file, the app will
look really nice. Here is the image of the
home page:
 
 Figure 17-16: Movie Review app completed
 
This finalizes our project of API
Development with Flask.
 
 
OceanofPDF.com
 
 
19. Final Exam
       

 
The Final Exam includes the topics we have covered
in this book.
It is a multiple-choice exam with 20 questions.
Each question is 5 points, which makes 100 points
in total.
You should get at least 70 points to be successful.
The duration is 120 minutes.
 
Here is the number of questions in the respective
chapters:
 
Topic Questions
Collections 2
Iterators 2
Generators 2
Date and
2
Time
Decorators 2
Context
2
Managers
Functional
2
Programming
Regular 2
 Expressions
Database
2
Operations
Concurrency 2
Total 20
 
If you want to solve the final exam in Jupyter
environment, you can download the notebook file
for the final exam, Final_Exam.ipynb, from the
GitHub Repository of this book.
 
Here are the questions for the Final Exam:

OceanofPDF.com
 
QUESTIONS:
 
Q1:
We have some dummy text as: "lorem ipsum dolor sit
amet consectetur adipiscing elit".
We want to count occurrences of each letter in this text.
We will print the most common four letters in this text.
We don't want to count space characters.
Which function below can succeed this task?
 
A from collections import Counter
   
text = "lorem ipsum dolor sit amet consectetur
  adipiscing elit"
   
  def count_and_print_most_common(text):
      text_counter = Counter(text)
      return text_counter.most_common(4)
   
  most_5 = count_and_print_most_common(text)
  print(most_5)
 
B from collections import Counter
   
text = "lorem ipsum dolor sit amet consectetur
  adipiscing elit"
   
  def count_and_print_most_common(text):
      text = ''.join(text.split())
      text_counter = Counter(text)
      return text_counter.most_common(4)
   
  most_5 = count_and_print_most_common(text)
  print(most_5)
 
C from collections import Counter
    
text = "lorem ipsum dolor sit amet consectetur
  adipiscing elit"
   
  def count_and_print_most_common(text):
      text = ''.join(text.split())
      text_counter = Counter(text)
      return text_counter
   
  most_5 = count_and_print_most_common(text)
  print(most_5)
 
D from collections import Counter
   
text = "lorem ipsum dolor sit amet consectetur
  adipiscing elit"
   
  def count_and_print_most_common(text):
      text = ''.join(text.split())
      return text.most_common(4)
   
  most_5 = count_and_print_most_common(text)
  print(most_5)
 
Q2:
We want to define a namedtuple as SuperHero.
SuperHero has three attributes:
hero
name
age
Create two SuperHero instance as follows:
Batman, Bruce Wayne, 28
Superman, Clark Kent, 26
We will print the names of SuperHeroes using keys as
follows:
Bruce Wayne is Batman
Clark Kent is Superman
Which option can print this output?
 
A from collections import namedtuple
   
  SuperHero = namedtuple(['hero', 'name', 'age'])
   
  S_1 = SuperHero('Batman', 'Bruce Wayne', '28')
  S_2 = SuperHero('Superman', 'Clark Kent', '26')
   
  print(f"{S_1.name} is {S_1.hero}")
  print(f"{S_2.name} is {S_2.hero}")
 
B from collections import namedtuple
   
SuperHero = namedtuple('SuperHero', 'hero', 'name',
  'age')
   
  S_1 = SuperHero('Batman', 'Bruce Wayne', '28')
  S_2 = SuperHero('Superman', 'Clark Kent', '26')
   
  print(f"{S_1.name} is {S_1.hero}")
  print(f"{S_2.name} is {S_2.hero}")
 
C from collections import namedtuple
   
SuperHero = namedtuple('SuperHero', ['hero', 'name',
  'age'])
   
  S_1 = SuperHero('Batman', 'Bruce Wayne', '28')
  S_2 = SuperHero('Superman', 'Clark Kent', '26')
   
  print(f"{S_1.name} is {S_1.hero}")
  print(f"{S_2.name} is {S_2.hero}")
 
D from collections import namedtuple
   
SuperHero = namedtuple('SuperHero', ['hero', 'name',
  'age'])
   
  S_1 = SuperHero('Batman', 'Bruce Wayne', '28')
  S_2 = SuperHero('Superman', 'Clark Kent', '26')
   
  print(f"{S_1.hero} is {S_1.name}")
  print(f"{S_2.hero} is {S_2.name}")
 
Q3:
In Python, Iterator objects are required to support two
special methods, which together form the Iterator
Protocol.
What are these methods?
 
A * __init__()
  * __self__()
 
B * __init__()
  * __iter__()
 
C * __next__()
  * __call__()
 
D * __iter__()
  * __next__()
 
Q4:
Define an iterator class which will generate a sequence of
odd numbers like 1, 3, 5, 7, 9, … etc.
Class name will be OddsIterator.
Instantiate an object of this class with name
odd_numbers.
The odd_numbers object will be an iterator that provides
odd numbers from 1 to 19.
Print the odd numbers using this object inside a for loop.
Make sure it raises StopIteration exception if you want to
print a number bigger than 19.
Which one below is NOT a correct definition for this
iterator?
 
A class OddsIterator:
      def __init__(self, limit):
          self.current = 1
          self.limit = limit
   
      def __iter__(self):
          return self
   
      def __next__(self):
          if self.current <= self.limit:
              current_value = self.current
              self.current += 2
              return current_value
   
  odd_numbers = OddsIterator(19)
   
  for i in range(20):
      print(odd_numbers.__next__())
 
B class OddsIterator:
      def __init__(self, limit):
          self.current = 1
          self.limit = limit
   
      def __iter__(self):
          return self
   
      def __next__(self):
          if self.current <= self.limit:
              current_value = self.current
              self.current += 2
              return current_value
   
          raise StopIteration
   
   odd_numbers = OddsIterator(19)
   
  for i in range(20):
      print(next(odd_numbers))
 
C class OddsIterator:
      def __init__(self, limit):
          self.current = 1
          self.limit = limit
   
      def __iter__(self):
          return self
   
      def __next__(self):
          if self.current <= self.limit:
              current_value = self.current
              self.current += 2
              return current_value
          else:
              raise StopIteration
   
  odd_numbers = OddsIterator(19)
   
  for i in range(20):
      print(odd_numbers.__next__())
 
D class OddsIterator:
      counter = 0
      def __init__(self, limit):
          global counter
          self.current = 1
          counter = limit
   
      def __iter__(self):
          return self
   
      def __next__(self):
          if self.current <= counter:
              current_value = self.current
              self.current += 2
              return current_value
   
          raise StopIteration
   
  odd_numbers = OddsIterator(19)
   
  for i in range(20):
      print(next(odd_numbers))
 
Q5:
We want to define a generator function that gives the list
of first n positive integers which are multiples of m.
Function name will be positive_multiples and it will take
two parameters:
n: number of positive numbers
m: number which we will use to get multiples
For example, if n=7 and m=5 then we will get the
following list:
[5, 10, 15, 20, 25, 30, 35]
We will call this function and print the first 7 positive
multiples of 5.
Which one below is appropriate for this task?
 
A def positive_multiples(n, m):
      for i in range(n):
          yield i * m
   
  multiples = positive_multiples(7, 5)
   
  print(list(multiples))
 
B def positive_multiples(n, m):
      for i in range(n):
          return (i+1) * m
   
   multiples = positive_multiples(7, 5)
   
  print(list(multiples))
 
C def positive_multiples(n, m):
      for i in range(n):
          yield (i+1) * m
   
  multiples = positive_multiples(7, 5)
   
  print(list(multiples))
 
D def positive_multiples(n, m):
      counter = 1
      while counter <= n:
          return counter * m
          counter += 1
   
  multiples = positive_multiples(7, 5)
   
  print(list(multiples))
 
Q6:
We want to define a generator expression.
This is going to be a one-line generator.
It will return an iterator which contains the cubes of even
elements in the numbers list.
Here is the numbers list:
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
Convert the resulting iterator to a list.
Then print the items in this list.
Here is the expected output:
[8, 64, 216, 512, 1000]
Which cell below completes this operation successfully?
 
A numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
   
  even_cubes_gen = (i**3 for i in numbers if i % 2 ==
 1)
   
  even_cubes_list = list(even_cubes_gen)
   
  print(even_cubes_list)
 
B numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
   
even_cubes_gen = (i**2 for i in numbers if i % 2 ==
  0)
   
  even_cubes_list = list(even_cubes_gen)
   
  print(even_cubes_list)
 
C numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
   
even_cubes_gen = (i*3 for i in numbers if i % 2 ==
  1)
   
  even_cubes_list = list(even_cubes_gen)
   
  print(even_cubes_list)
 
D numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
   
even_cubes_gen = (i**3 for i in numbers if i % 2 ==
  0)
   
  even_cubes_list = list(even_cubes_gen)
   
  print(even_cubes_list)
 
Q7:
Create a datetime object corresponding to date string of
"%d/%m/%y %H:%M".
Here are the date values:
 year: 2022
month: 4
day: 19
hour: 16
minutes: 36
Then print the following text by formatting this datetime
object:
"Tuesday, 19. April 2022 04:36PM"
Which cell below completes this operation successfully?
 
A from datetime import datetime
   
dt = datetime.strptime("22/4/19 16:36", "%d/%m/%y
  %H:%M")
   
formatted_dt = dt.strftime("%A, %d. %B %Y
  %I:%M%p")
   
  print(formatted_dt)
 
B from datetime import datetime
   
dt = datetime.strptime("19/4/22 16:36", "%d/%m/%y
  %H:%M")
   
formatted_dt = dt.strftime("%A, %d. %B %Y
  %I:%M%p")
   
  print(formatted_dt)
 
C from datetime import datetime
   
dt = datetime.strptime("19/4/22 16:36",
  "%d/%m/%y")
   
formatted_dt = dt.strftime("%A, %d. %B %Y
  %I:%M%p")
   
   print(formatted_dt)
 
D from datetime import datetime
   
dt = datetime.strptime("19/4/22 16:36", "%d/%m/%Y
  %H:%M")
   
formatted_dt = dt.strftime("%A, %d. %B %Y
  %I:%M%p")
   
  print(formatted_dt)
 
Q8:
Create a date object from ISO format.
The date will be: 2021-10-25.
Then print the date in following format:
"Monday 25. October 2021"
Which cell below completes this operation successfully?
 
A from datetime import date
   
  d = date.fromisoformat('2021-10-25')
   
  print(d.isoformat())
 
B from datetime import date
   
  d = date.fromisoformat('2021-10-25')
   
  print(d.strftime("%d/%m/%y"))
 
C from datetime import date
   
  d = date.fromisoformat('2021-10-25')
   
  print(d.strftime("%A %d. %B %Y"))
 
D from datetime import date
   
  d = date.fromisoformat('2021-10-25')
   
  print(d.strftime("%d. %m %y"))
 
Q9:
We have a basic function which takes one argument and
simply returns it.
Here is this function:
def text_to_list(text):
    return text
We want this function to always return the characters of
the text in a list.
For example, when we call it as:
text_to_list('python advanced')
We want it to return:
['p', 'y', 't', 'h', 'o', 'n', ' ', 'a', 'd', 'v', 'a', 'n', 'c', 'e',
'd', '.']
But we do not want to modify the function body.
Function definition will not be changed.
What we need is a decorator to do the list conversion for
our function.
So, you need to define this decorator.
Which cell below completes this operation successfully?
 
A def list_decorator(func):
      def wrapper(text):
          func_result = func(text)
          return list(func_result)
      return wrapper
   
  @list_decorator
  def text_to_list(text):
      return text
   
  list_from_text = text_to_list('python advanced.')
   
   print(list_from_text)
 
B def list_decorator(func):
      def wrapper(text):
          func_result = func(text)
          yield list(func_result)
      return wrapper
   
  @list_decorator
  def text_to_list(text):
      return text
   
  list_from_text = text_to_list('python advanced.')
   
  print(list_from_text)
 
C def list_decorator(func):
      def wrapper(text):
          func_result = func(text)
          return func_result
      return wrapper
   
  @list_decorator
  def text_to_list(text):
      return text
   
  list_from_text = text_to_list('python advanced.')
   
  print(list_from_text)
 
D def list_decorator(func):
      def wrapper(text):
          return func(text)
      return wrapper
   
  @list_decorator
  def text_to_list(text):
      return text
   
  list_from_text = text_to_list('python advanced.')
   
  print(list_from_text)
 
Q10:
Define a class decorator named FunctionDetails.
This decorator will print the function arguments as
follows:
Parameters: (a tuple of parameters)
Here is the function which we want to decorate:
def some_function(first, last, age, a_list):
    return (first, last, age, a_list)
Call the function after you decorate it with
FunctionDetails.
When you call it, it should print the parameters as
following:
some_function('jane', 'doe', 28, [1,2,3])
-----
Parameters: ('jane', 'doe', 28, [1, 2, 3])
Which one below can achieve this?
 
A class FunctionDetails:
      def __call__(self, *args):
          print(f"Parameters: {args}")
   
  @FunctionDetails
  def some_function(first, last, age, a_list):
      return (first, last, age, a_list)
   
  some_function('jane', 'doe', 28, [1,2,3])
 
B class FunctionDetails:
      def __init__(self, func):
          self.func = func
   
      def __iter__(self, *args):
           print(f"Parameters: {args}")
   
  @FunctionDetails
  def some_function(first, last, age, a_list):
      return (first, last, age, a_list)
   
  some_function('jane', 'doe', 28, [1,2,3])
 
C class FunctionDetails:
      def __init__(self, func):
          self.func = func
   
      def __call__(self):
          print(f"Parameters: {args}")
   
  @FunctionDetails
  def some_function(first, last, age, a_list):
      return (first, last, age, a_list)
   
  some_function('jane', 'doe', 28, [1,2,3])
 
D class FunctionDetails:
      def __init__(self, func):
          self.func = func
   
      def __call__(self, *args):
          print(f"Parameters: {args}")
   
  @FunctionDetails
  def some_function(first, last, age, a_list):
      return (first, last, age, a_list)
   
  some_function('jane', 'doe', 28, [1,2,3])
 
Q11:
Define a custom context manager class as
CustomContextManager.
Define the necessary methods in it.
Your context manager should open and return the file
object at the specified path.
Also, it should close the file on exit.
The parameters to the class constructor should be:
path: file path for the file to read
mode: operation mode (read, write, append etc.)
Which one is the correct definition of this context
manager class?
 
A class CustomContexManager:
      def __init__(self, path, mode):
          self.path = path
          self.mode = mode
          self.file = None
   
      def __iter__(self):
          self.file = open(self.path, self.mode)
          return self.file
   
    def __exit__(self, exc_type, exc_value,
  exc_traceback):
          self.file.close()
 
B class CustomContexManager:
      def __init__(self, path, mode):
          self.path = path
          self.mode = mode
          self.file = None
   
      def __enter__(self):
          self.file = open(self.path, self.mode)
          return self.file
   
    def __exit__(self, exc_type, exc_value,
  exc_traceback):
          self.file.close()
 
C class CustomContexManager:
      def __init__(self, path, mode):
          self.path = path
          self.mode = mode
          self.file = None
   
      def __enter__(self):
          self.file = open(self.path, self.mode)
          return self.file
 
D class CustomContexManager:
      def __init__(self, path, mode):
          self.path = path
          self.mode = mode
          self.file = None
   
      def __enter__(self):
          self.file = open(self.path, self.mode)
   
    def __exit__(self, exc_type, exc_value,
  exc_traceback):
          self.file.close()
 
Q12:
Define a function-based context manager which creates a
file and writes some text in it.
The name will be file_writer and it will take two
parameters:
path: file path for the file to read
mode: operation mode (read, write, append etc.)
Use this context manager in a with statement to create a
file and add the text below in this file.
File name should be: "text_file_for_writing.txt".
And the text is:
"The file is created in the final exam.
It is a function-based context manager,
decorated with contextlib.contextmanager."
Which one below can achieve this?
 
A from contextlib import contextmanager
   
  @contextmanager
  def file_writer(path, mode='w'):
      file_object = None
      try:
          file_object = open(path, mode=mode)
          return file_object
      finally:
          if file_object:
              file_object.close()
   
  with file_writer("text_file_for_writing.txt") as file:
      file.write("The file is created in the final exam.\n"
                 "It is a function-based context manager,\n"
               "decorated with
  contextlib.contextmanager.")
 
B from contextlib import contextmanager
   
  @contextmanager
  def file_writer(path, mode='r'):
      file_object = None
      try:
          file_object = open(path, mode=mode)
          yield file_object
      finally:
          if file_object:
              file_object.close()
   
  with file_writer("text_file_for_writing.txt") as file:
      file.write("The file is created in the final exam.\n"
                 "It is a function-based context manager,\n"
               "decorated with
  contextlib.contextmanager.")
 
C from contextlib import contextmanager
   
  @contextmanager
  def file_writer(path, mode='w'):
      file_object = None
      try:
          file_object = open(path, mode=mode)
          yield file_object
      finally:
          if file_object:
              file_object.close()
   
  with file_writer("text_file_for_writing.txt") as file:
      file.write("The file is created in the final exam.\n"
                 "It is a function-based context manager,\n"
               "decorated with
  contextlib.contextmanager.")
 
D from contextlib import contextmanager
   
  def file_writer(path, mode='w'):
      file_object = None
      try:
          file_object = open(path, mode=mode)
          yield file_object
      finally:
          if file_object:
              file_object.close()
   
  with file_writer("text_file_for_writing.txt") as file:
      file.write("The file is created in the final exam.\n"
                 "It is a function-based context manager,\n"
               "decorated with
  contextlib.contextmanager.")
 
Q13:
We have a list of numbers which we want to filter only
the positive ones.

OceanofPDF.com
Here is the list:
[5, -16, 4, -3, 0, -12, 9, 1, 2, -7, 8]
Define a filter() function which gives us the positive
numbers in this list.
It should take a lambda function and this list as the
parameters.
The filter function will return a filter object.
You should convert this filter object to a list.
Finally, you should sort the list in ascending order in-
place.
The resulting list should be:
[1, 2, 4, 5, 8, 9]
Which one below can achieve this?
 
A all_numbers = [5, -16, 4, -3, 0, -12, 9, 1, 2, -7, 8]
   
numbers_filtered = filter(lambda x: x < 0,
  all_numbers)
   
  positives = list(numbers_filtered)
   
  positives.sort()
   
  print(positives)
 
B all_numbers = [5, -16, 4, -3, 0, -12, 9, 1, 2, -7, 8]
   
numbers_filtered = filter(lambda x: x > 0,
  all_numbers)
   
  positives = list(numbers_filtered)
   
  positives.sort()
   
  print(positives)
 
C all_numbers = [5, -16, 4, -3, 0, -12, 9, 1, 2, -7, 8]
   
   numbers_filtered = filter(lambda x: x, all_numbers)
   
  positives = list(numbers_filtered)
   
  positives.sort()
   
  print(positives)
 
D all_numbers = [5, -16, 4, -3, 0, -12, 9, 1, 2, -7, 8]
   
numbers_filtered = filter(all_numbers, lambda x: x >
  0)
   
  positives = list(numbers_filtered)
   
  positives.sort()
   
  print(positives)
 
Q14:
We want define a function which multiplies all the items
in the given list.
The initial value will be 100, which means it will start by
multiplying the first item with 100.
Here is the list to multiply:
[4, 5, 6, 7, 8]
The expected result is:
672000
Which one below can achieve this task?
 
A from functools import reduce
   
  list_mult = [4, 5, 6, 7, 8]
   
multiplication = reduce(lambda x, y: x * y, list_mult,
  100)
   
  print(multiplication)
 
B from functools import reduce
   
  list_mult = [4, 5, 6, 7, 8]
   
multiplication = map(lambda x, y: x * y, list_mult,
  100)
   
  print(multiplication)
 
C from functools import reduce
   
  list_mult = [4, 5, 6, 7, 8]
   
  multiplication = map(lambda x, y: x * y, list_mult)
   
  print(multiplication)
 
D from functools import reduce
   
  list_mult = [4, 5, 6, 7, 8]
   
  multiplication = reduce(lambda x, y: x * y, list_mult)
   
  print(multiplication)
 
Q15:
Define a regular expression pattern in the following way.
It should:
start with 2 digits
a dot (.) after 2 digits
3 alphabetical upper case letters (A to Z)
a dash (-)
zero or more word characters after the dash
Here are some examples:
"47.UPS-333"
"16.ARC-c"
"04.XYZ-"
What is the correct RE pattern for this?
 
A import re
   
  pattern = re.compile(r"\d{2}.[A-Z]{3}-\w+")
   
  m = pattern.search("47.UPS-333")
  m_2 = pattern.search("16.ARC-c")
  m_3 = pattern.search("04.XYZ-")
   
  print(m.group())
  print(m_2.group())
  print(m_3.group())
 
B import re
   
  pattern = re.compile(r"\d{2}.[A-Z]{3}-\w")
   
  m = pattern.search("47.UPS-333")
  m_2 = pattern.search("16.ARC-c")
  m_3 = pattern.search("04.XYZ-")
   
  print(m.group())
  print(m_2.group())
  print(m_3.group())
 
C import re
   
  pattern = re.compile(r"\d{2}.[A-Z]{3}-\w*")
   
  m = pattern.search("47.UPS-333")
  m_2 = pattern.search("16.ARC-c")
  m_3 = pattern.search("04.XYZ-")
   
  print(m.group())
  print(m_2.group())
  print(m_3.group())
 
D import re
   
  pattern = re.compile(r"\d{2}.[A-Z]-\w*")
   
  m = pattern.search("47.UPS-333")
  m_2 = pattern.search("16.ARC-c")
  m_3 = pattern.search("04.XYZ-")
   
  print(m.group())
  print(m_2.group())
  print(m_3.group())
 
Q16:
Define a function named find_all_with_re.
The function will take a regular expression and a text as
parameters.
Ant will return the list of all possible matches of this
regex in the text.
The regex will be: all words starting with 'c' or 'e' (a word
must include at least two characters).
The text is:
"Lorem ipsum dolor sit amet, a consectetur adipiscing
elit. Sed id cid tempor risus. Quisque imperdiet, neque c
pulvinar sollicitudin, augue nisl varius nibh, e suscipit
cerat non lectus."
Expected Result:
['consectetur', 'elit', 'cid', 'cerat']
Which one below can achieve this task?
 
A import re
   
  def find_all_with_re(regex, text):
      return re.find(regex, text)
   
  text = """Lorem ipsum dolor sit amet,
  a consectetur adipiscing elit.
  Sed id cid tempor risus.
   Quisque imperdiet, neque c pulvinar sollicitudin,
  augue nisl varius nibh, e suscipit cerat non lectus."""
   
  regex = r"\b[ce]\w+"
   
  match_list = find_all_with_re(regex, text)
   
  print(match_list)
 
B import re
   
  def find_all_with_re(regex, text):
      return re.findall(regex, text)
   
  text = """Lorem ipsum dolor sit amet,
  a consectetur adipiscing elit.
  Sed id cid tempor risus.
  Quisque imperdiet, neque c pulvinar sollicitudin,
  augue nisl varius nibh, e suscipit cerat non lectus."""
   
  regex = r"\b(ce)\w+"
   
  match_list = find_all_with_re(regex, text)
   
  print(match_list)
 
C import re
   
  def find_all_with_re(regex, text):
      return re.findall(regex, text)
   
  text = """Lorem ipsum dolor sit amet,
  a consectetur adipiscing elit.
  Sed id cid tempor risus.
  Quisque imperdiet, neque c pulvinar sollicitudin,
  augue nisl varius nibh, e suscipit cerat non lectus."""
   
   regex = r"\b[ce]\w"
   
  match_list = find_all_with_re(regex, text)
   
  print(match_list)
 
D import re
   
  def find_all_with_re(regex, text):
      return re.findall(regex, text)
   
  text = """Lorem ipsum dolor sit amet,
  a consectetur adipiscing elit.
  Sed id cid tempor risus.
  Quisque imperdiet, neque c pulvinar sollicitudin,
  augue nisl varius nibh, e suscipit cerat non lectus."""
   
  regex = r"\b[ce]\w+"
   
  match_list = find_all_with_re(regex, text)
   
  print(match_list)
 
Q17:
We will define a function to connect to a MySQL Server
and print the connection result.
The function name will be mysql_connector.
The server will be the local MySQL Server on your
machine (localhost).
The function has to complete these three tasks:
1- If it gets an exception, it should print the error
description.
2- If it successfully connects to the server it should print
it as:
"Successfully connected to MySQL Server."
3- It should return an active
mysql.connector.connection.MySQLConnection object.
Which one below can achieve this task?
 
A import mysql.connector
   
  def mysql_connector():
      connection = None
      try:
          connection = connect(
              host="localhost",
              user="root",
              password="<your_root_password>"
          )
        print("Successfully connected to MySQL
  Server.")
      except mysql.connector.Error as e:
          print(f"An Error occurred: '{e}'")
      return connection
   
  mysql_connector()
 
B import mysql.connector
   
  def mysql_connector():
      connection = None
      try:
          connection = mysql.connector.connect(
              host="localhost",
              user="root",
              password="<your_root_password>"
          )
        print("Successfully connected to MySQL
  Server.")
      except mysql.connector.Error as e:
          print(f"An Error occurred: '{e}'")
      return connection
   
  mysql_connector()
 
 C import mysql.connector
   
  # define the function
  def mysql_connector():
      try:
          yield mysql.connector.connect(
              host="localhost",
              user="root",
              password="<your_root_password>"
          )
        print("Successfully connected to MySQL
  Server.")
      except mysql.connector.Error as e:
          print(f"An Error occurred: '{e}'")
   
  mysql_connector()
 
D import mysql.connector
   
  # define the function
  def mysql_connector():
      try:
          connection = mysql.connector.connect(
              host="localhost",
              user="root",
              password="<your_root_password>"
          )
        print("Successfully connected to MySQL
  Server.")
          yield connection
      finally:
          pass
   
  mysql_connector()
 
Q18:
We want to define a function named create_a_database.
It will create a new database named my_local_db on the
local MySQL server.
To be able to create a database, you need to connect to
the server first.
The function should print the result of the operation,
either successful or an exception.
The function should also drop the database if it exists
with the same name.
What is the correct definition for this function?
 
from mysql.connector.connection import
A MySQLConnection
   
  def create_a_database(db_name):
      try:
          connection = MySQLConnection()
          if connection is not None:
              cursor = connection.cursor()
            cursor.execute(f"DROP DATABASE
  {db_name}")
            cursor.execute(f"CREATE DATABASE
  {db_name}")
      except mysql.connector.Error as e:
        print(f"An Error occurred in DB
  Creation:\n'{e}'")
      else:
          if connection is not None:
              cursor.close()
              connection.close()
              print("Database created successfully.")
   
  create_a_database("my_local_db")
 
B import mysql.connector
   
  def create_a_database(db_name):
      try:
           connection = mysql.connector.connect(
              user="root",
              password="<your_root_password>"
          )
          if connection is not None:
              cursor = connection.cursor()
            cursor.execute(f"DROP DATABASE
  {db_name}")
      except mysql.connector.Error as e:
        print(f"An Error occurred in DB
  Creation:\n'{e}'")
      else:
          if connection is not None:
              cursor.close()
              connection.close()
              print("Database created successfully.")
   
  create_a_database("my_local_db")
 
C import mysql.connector
   
  def create_a_database(db_name):
      try:
          connection = mysql.connector.connect(
              user="root",
              password="<your_root_password>"
          )
          if connection is not None:
              cursor = connection.cursor()
              cursor.execute(f"CREATE DATABASE {db_name}")
      except mysql.connector.Error as e:
          print(f"An Error occurred in DB Creation:\n'{e}'")
      else:
          if connection is not None:
              cursor.close()
              connection.close()
              print("Database created successfully.")
    
  create_a_database("my_local_db")
 
D import mysql.connector
   
  def create_a_database(db_name):
      try:
          connection = mysql.connector.connect(
              host="localhost",
              user="root",
              password="<your_root_password>"
          )
          if connection is not None:
              cursor = connection.cursor()
            cursor.execute(f"DROP DATABASE
  {db_name}")
            cursor.execute(f"CREATE DATABASE
  {db_name}")
      except mysql.connector.Error as e:
        print(f"An Error occurred in DB
  Creation:\n'{e}'")
      else:
          if connection is not None:
              cursor.close()
              connection.close()
              print("Database created successfully.")
   
  create_a_database("my_local_db")
 
Q19:
Let's assume we have a folder named "images" which
contains some image files, and an empty folder called
"thumbnails".
Let's also assume, the initial size of the "images" folder
is around 27,5 MB with 20 images in it.
The size is quite large, so we want to resize the images in
this folder.
We want to save the thumbnails of these images in the
"thumbnails" folder.
Thumbnail dimensions (width * height) of each image will
1/10 of the original image.
And in terms of disk space, we expect the "thumbnails"
folder to be around 200 KB.
We also want to keep track of elapsed time (start & end).
We want to use Threading for resize operation.
We have a function named get_images() which returns a
list of images in the "images" folder.
There is also a function named resize_image() which
resizes the given image based on width * height values.
Which function below is defined correctly for creating and
running threads for resize operation?
 
  import os
  from PIL import Image
   
  # directory names
  images_directory = "images"
  thumbnails_directory = "thumbnails"
   
  # get images fn
  def get_images():
      """Returns a list of .jpg files
      in the images folder"""
      images = []
      for file in os.listdir(images_directory):
          if file.endswith(".jpg"):
              images.append(file)
      return images
   
  # resize a single image fn
  def resize_image(image):
    """Resizes the image and saves to
  thumbnails_directory"""
      im = Image.open(images_directory + "/" + image)
      resized_im = im.resize((round(im.size[0] * 0.1),
 round(im.size[1] * 0.1)))
    resized_im.save(thumbnails_directory + "/" +
  image)
      print(f"{image} resized.")
 
A from threading import Thread
   
  threads = []
   
  def resize_all_images():
      try:
          start = time.perf_counter()
          images = get_images()
          for image in images:
            thread = Thread(target=resize_image, args=
  (image,))
              thread.start()
              threads.append(thread)
          for thread in threads:
              thread.join()
          end = time.perf_counter()
      except:
          print("An Error occurred in resizing.")
      else:
        print(f"Images resized in {end - start:.2f} sec -
  Threading.")
 
B from threading import Thread
   
  threads = []
   
  def resize_all_images():
      try:
          start = time.perf_counter()
          images = get_images()
          for image in images:
              thread = Thread(target=resize_image, args=
 (image,))
              thread.start()
          end = time.perf_counter()
      except:
          print("An Error occurred in resizing.")
      else:
        print(f"Images resized in {end - start:.2f} sec -
  Threading.")
 
C from threading import Thread
   
  threads = []
   
  def resize_all_images():
      try:
          start = time.perf_counter()
          images = get_images()
          for image in images:
            thread = Thread(target=resize_image, args=
  (image,))
              threads.append(thread)
          for thread in threads:
              thread.join()
          end = time.perf_counter()
      except:
          print("An Error occurred in resizing.")
      else:
        print(f"Images resized in {end - start:.2f} sec -
  Threading.")
 
D from threading import Thread
   
  threads = []
   
  def resize_all_images():
      try:
          start = time.perf_counter()
          images = get_images()
          for image in images:
            thread = Thread(target=resize_image, args=
  (image,))
              threads.append(thread)
          end = time.perf_counter()
      except:
          print("An Error occurred in resizing.")
      else:
        print(f"Images resized in {end - start:.2f} sec -
  Threading.")
 
Q20:
Let's assume we have a folder named "images" which
contains some image files, and an empty folder called
"thumbnails".
Let's also assume, the initial size of the "images" folder
is around 27,5 MB with 20 images in it.
The size is quite large, so we want to resize the images in
this folder.
We want to save the thumbnails of these images in the
"thumbnails" folder.
Thumbnail dimensions (width * height) of each image will
1/10 of the original image.
And in terms of disk space, we expect the "thumbnails"
folder to be around 200 KB.
We also want to keep track of elapsed time (start & end).
We want to use ThreadPoolExecutor for resize operation.
We have a function named get_images() which returns a
list of images in the "images" folder.
There is also a function named resize_image() which
resizes the given image based on width * height values.
Which function below is defined correctly for using
ThreadPoolExecutor and a list comprehension to submit()
threads.?
 
  import os
  from PIL import Image
    
  # directory names
  images_directory = "images"
  thumbnails_directory = "thumbnails"
   
  # get images fn
  def get_images():
      """Returns a list of .jpg files
      in the images folder"""
      images = []
      for file in os.listdir(images_directory):
          if file.endswith(".jpg"):
              images.append(file)
      return images
   
  # resize a single image fn
  def resize_image(image):
    """Resizes the image and saves to
  thumbnails_directory"""
      im = Image.open(images_directory + "/" + image)
    resized_im = im.resize((round(im.size[0] * 0.1),
  round(im.size[1] * 0.1)))
    resized_im.save(thumbnails_directory + "/" +
  image)
      print(f"{image} resized.")
 
from concurrent.futures import
A ThreadPoolExecutor, as_completed
   
  threads = []
   
  def resize_all_images():
      try:
          start = time.perf_counter()
          images = get_images()
          with ThreadPoolExecutor() as executor:
              executor.submit(resize_image, image)
               for future in as_completed(executor):
                  future.result()
          end = time.perf_counter()
      except:
          print("An Error occurred in resizing.")
      else:
        print(f"Images resized in {end - start:.2f} sec -
  ThreadPoolExecutor.")
 
from concurrent.futures import
B ThreadPoolExecutor, as_completed
   
  threads = []
   
  def resize_all_images():
      try:
          start = time.perf_counter()
          images = get_images()
          with ThreadPoolExecutor() as executor:
              future_list = [submit(resize_image, image)
                             for image in images]
          end = time.perf_counter()
      except:
          print("An Error occurred in resizing.")
      else:
        print(f"Images resized in {end - start:.2f} sec -
  ThreadPoolExecutor.")
 
from concurrent.futures import
C ThreadPoolExecutor, as_completed
   
  threads = []
   
  def resize_all_images():
      try:
          start = time.perf_counter()
          images = get_images()
           with ThreadPoolExecutor() as executor:
            future_list = [executor.submit(resize_image,
  image)
                             for image in images]
              for future in as_completed(future_list):
                  future.result()
          end = time.perf_counter()
      except:
          print("An Error occurred in resizing.")
      else:
        print(f"Images resized in {end - start:.2f} sec -
  ThreadPoolExecutor.")
 
from concurrent.futures import
D ThreadPoolExecutor, as_completed
   
  threads = []
   
  def resize_all_images():
      try:
          start = time.perf_counter()
          images = get_images()
          with ThreadPoolExecutor() as executor:
              executor.submit(resize_image, image)
          end = time.perf_counter()
      except:
          print("An Error occurred in resizing.")
      else:
        print(f"Images resized in {end - start:.2f} sec -
  ThreadPoolExecutor.")
 
OceanofPDF.com
 
 
ANSWERS OF THE FINAL EXAM QUESTIONS:
 
Question Answer
Q1 B
Q2 C
Q3 D
Q4 A
Q5 C
Q6 D
Q7 B
Q8 C
Q9 A
Q10 D
Q11 B
Q12 C
Q13 B
Q14 A
Q15 C
Q16 D
Q17 B
Q18 D
Q19 A
Q20 C
 

OceanofPDF.com
  
 
20. Conclusion
       

 
This is the last chapter in this book. We
have covered almost all of the advanced
level concepts in Python and we covered
them in great detail. We started with
setting our development environment, the
PyCharm IDE.
Then we learned Collections, Iterators,
Generators, Date and Time, Decorators,
Context Managers and Functional
Programming in Python.
  After these topics, we had our first
project which was Sending Emails with
Python. We learned how to use smtplib
module to send messages on both a test
server and an actual Gmail account.
Then we learned, Regular Expressions,
Database Operations and Concurrency in
Python.
We had our second project after these
topics, which was about Web Scraping.
You learned how to scrap data out of web
pages using Python and Scrapy.
Finally, we built our third project, which
was a Movie Review web application using
Python and Flask. We built the project step
by step from scratch and you learned all
of the fundamental concepts of building
API’s using Flask.
You also had assignments after each
project. And finally, you got the Final Exam
to test yourself.
 
The approach we followed in this book is
quite unique and intense. It aims to teach
you Python in a solid and unforgettable
way. That’s why we had more than 52
coding exercises, quizzes and
assignments. The idea is to make sure we
used all the possible ways to help you
learn Python programming. And I hope, I
achieved this.
 
This book is a part of the Hands-On
Python Series. There are three books in
this series: Beginner, Intermediate and
Advanced. This book is the third one, the
Advanced level. By completing this series,
I believe you can feel comfortable with
almost all of the fundamentals of Python.
The coding exercises, projects,
assignments and exam questions will give
you a great resource for your
programming life and job interviews on
Python.
 
 
 
Dear reader!
 
I want to thank you with all my heart,
for your interest in my book, your patience
and your desire to learn Python. You did a
great job finishing this intense book. And I
am very happy to be a part of this. I hope
we see each other again in another
programming book. Till then, I hope, you
have wonderful life and reach your
dreams.
 
 
Good bye.
 
 Musa Arda
 
 
[1] Hands-On Python Series:
https://www.amazon.com/gp/product/B09J
M26C3Z
[2] Python official website:
https://www.python.org/
[3] Python Virtual Environments:
https://docs.python.org/3/tutorial/venv.ht
ml
[4] PyCharm’s official website:
https://www.jetbrains.com/pycharm/
[5] PyCharm official documentation:
https://www.jetbrains.com/help/pycharm/q
uick-start-guide.html
OceanofPDF.com