Sold to

[email protected]

Python 201
Intermediate Python
Michael Driscoll
This book is for sale at http://leanpub.com/python201
This version was published on 2016-10-17

This is a Leanpub book. Leanpub empowers authors and publishers with the Lean Publishing
process. Lean Publishing is the act of publishing an in-progress ebook using lightweight tools and
many iterations to get reader feedback, pivot until you have the right book and build traction once
you do.
2016 Michael Driscoll

Tweet This Book!

Please help Michael Driscoll by spreading the word about this book on Twitter!
The suggested hashtag for this book is #python201.
Find out what other people are saying about the book by clicking on this link to search for this
hashtag on Twitter:
https://twitter.com/search?q=#python201

Also By Michael Driscoll

Python 101
wxPython Cookbook

Contents
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

.
.
.
.
.

2
2
3
3
3

Part I - The Intermediate Modules

. . . . . . . . . . . . . . . . .

Chapter 1 - An Intro to Argument Parsing using

argparse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

About the Author

Conventions . . .
Requirements . .
Reader Feedback .
Errata . . . . . . .

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

Getting Started . . . . . . . . . .
Adding Arguments . . . . . . .
Short Options and Long Options
Options that Conflict . . . . . .
Wrapping Up . . . . . . . . . . .

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

Chapter 2 - The collections module

ChainMap .
Counter . . .
defaultdict .
deque . . . .
namedtuple .
OrderedDict
Wrapping Up

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

Chapter 3 - Context Managers

Creating a Context Manager class . . . . . .
Creating a Context Manager using contextlib
contextlib.closing(thing) . . . . . . . . . . . .
contextlib.suppress(*exceptions) . . . . . . . .

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

. 10
. 10
. 12
. 13
. 17
. 19
. 20
. 22

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

. 23
. 23
. 24
. 25
. 26

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

5
6
7
8
9

CONTENTS

contextlib.redirect_stdout / redirect_stderr
ExitStack . . . . . . . . . . . . . . . . . .
Reentrant Context Managers . . . . . . .
Wrapping Up . . . . . . . . . . . . . . . .

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

Chapter 4 - The functools module

Caching with functools.lru_cache . . . . . . . . . . .
functool.partial . . . . . . . . . . . . . . . . . . . . .
Function Overloading with functools.singledispatch .
functools.wraps . . . . . . . . . . . . . . . . . . . .
Wraps to the Rescue! . . . . . . . . . . . . . . . . . .
Wrapping Up . . . . . . . . . . . . . . . . . . . . . .

Chapter 5 - All About Imports

Regular Imports . . . . . . . . . . . . .
Using from module import something
Relative Imports . . . . . . . . . . . . .
Optional Imports . . . . . . . . . . . . .
Local Imports . . . . . . . . . . . . . .
Import Pitfalls . . . . . . . . . . . . . .
Wrapping Up . . . . . . . . . . . . . . .

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

. 30
. 30
. 31
. 34
. 36
. 38
. 39

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

. 40
. 40
. 41
. 42
. 45
. 45
. 46
. 48

.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

. 49
. 49
. 50
. 52
. 53
. 53

.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

. 54
. 54
. 55
. 58
. 60

.
.
.
.
.

.
.
.
.
.

Chapter 7 - Iterators and Generators

Iterators . . . . . . . . . . .
Creating Your Own Iterators
Generators . . . . . . . . . .
Wrapping Up . . . . . . . . .

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

27
27
28
29

.
.
.
.
.
.

Chapter 6 - The importlib Module

Dynamic Imports . . . .
Module Import Checking
Import From Source File .
import_from_github_com
Wrapping Up . . . . . . .

.
.
.
.

.
.
.
.

Chapter 8 - The itertools Module

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

. . . . . . . . . . . . . . . . . . 61
The Infinite Iterators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Iterators That Terminate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

CONTENTS

The Combinatoric Generators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Wrapping Up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Chapter 9 - Regular Expressions

The Matching Characters . . .
Pattern Matching Using search
Escape Codes . . . . . . . . . .
Compiling . . . . . . . . . . .
Compilation Flags . . . . . . .
Finding Multiple Instances . .
Backslashes Are Complicated .
Wrapping Up . . . . . . . . . .

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

73
75

. 76
. 76
. 78
. 79
. 79
. 80
. 81
. 82
. 83

Chapter 10 - The typing Module

. . . . . . . . . . . . . . . . . . . 84
Type Hints and Overloaded Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
Wrapping Up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

Part II - Odds and Ends

. . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

Chapter 11 - Python Built-ins . . . . . . . . . . . . . . . . . . . . . .

any . . . . .
enumerate .
eval . . . . .
filter . . . .
map . . . . .
zip . . . . .
Wrapping Up

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

90
90
91
91
92
92
93
94

Chapter 12 - Unicode

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Encoding / Decoding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
Wrapping Up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

Chapter 13 - Benchmarking . . . . . . . . . . . . . . . . . . . . . . . . 101

timeit . . . . . . . . . . . . . . . .
Importing timeit for Testing . . . .
Use a decorator . . . . . . . . . .
Create a Timing Context Manager
cProfile . . . . . . . . . . . . . . .

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

101
102
103
104
105

CONTENTS

line_profiler . .
memory_profiler
profilehooks . .
Wrapping Up . .

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

106
108
110
112

Chapter 14 - Encryption and Cryptography . . . . . . . . 113

Hashing . . . . . . . . . .
Key Derivation . . . . . . .
PyCrypto . . . . . . . . . .
The cryptography Package
Wrapping Up . . . . . . . .

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

. 121
. 121
. 124
. 125
. 126
. 126
. 128
. 129
. 129

.
Python 2 vs Python 3 . . . . . . . . . . . . . . . . . .
Method Resolution Order (MRO) . . . . . . . . . . . .
Wrapping Up . . . . . . . . . . . . . . . . . . . . . . .

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

. 130
. 130
. 131
. 134

Chapter 15 - Databases
Basic SQL Syntax . . . . .
adodbapi . . . . . . . . . .
pyodbc . . . . . . . . . . .
pypyodbc . . . . . . . . . .
MySQL . . . . . . . . . . .
PostgreSQL . . . . . . . . .
Object Relational Mappers
Wrapping Up . . . . . . . .

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

Chapter 16 - The super built-in

113
114
115
119
120

Chapter 17 - Descriptors . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

The Descriptor Protocol
Calling a Descriptor . .
Descriptor Examples . .
Wrapping Up . . . . . .

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

135
136
136
140

Chapter 18 - Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

Local Scope . .
Global Scope .
nonlocal Scope
Wrapping Up .

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

141
143
145
146

CONTENTS

Part III - Working with the Web

Chapter 19 - Web Scraping
Tips for Scraping . .
Preparing to Scrape
BeautifulSoup . . .
Scrapy . . . . . . .
Wrapping Up . . . .

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

Chapter 20 - Web APIs

Twitter . . . . . . .
Reddit . . . . . . .
Wikipedia . . . . .
Other API Libraries
Wrapping Up . . . .

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

. . . . . . . . . . . . . . . . . . . 147

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

. 148
. 148
. 148
. 150
. 152
. 155

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

. 156
. 156
. 158
. 160
. 163
. 164

Chapter 21 - Working with FTP . . . . . . . . . . . . . . . . . . . . 165

Connecting to an FTP Server . . .
Navigating Directories with ftplib
Downloading a File via FTP . . . .
Uploading Files to an FTP Server .
Wrapping Up . . . . . . . . . . . .

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

Chapter 22 - The urllib Module

urllib.request . . .
urllib.parse . . . .
urllib.robotparser
Wrapping Up . . .

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

165
166
167
168
169

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

. 170
. 170
. 173
. 174
. 175

Part IV - Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176

Chapter 24 - The doctest Module
How it Works . . . . . . . . . .
Check Your Test With a Text File
Option Flags and Directives . . .
Wrapping Up . . . . . . . . . . .

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

. 177
. 179
. 180
. 182
. 182

CONTENTS

Chapter 24 - The unittest Module . . . . . . . . . . . . . . . . . . 183

A Simple Example . . . . . . .
Command-Line Interface . . .
Creating a More Complex Test
Creating Test Suites . . . . . .
How to Skip Tests . . . . . . .
Integrating with doctest . . . .
Wrapping Up . . . . . . . . . .

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

Chapter 25 - The mock Module

Simple Examples
Side Effects . . .
Autospeccing . .
The Patch . . . .
Wrapping Up . .

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

183
186
188
193
194
196
197

.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

. 198
. 198
. 200
. 200
. 201
. 202

Chapter 26 - An Intro to coverage.py

. . . . . . . . . . . . . . 203
Additional Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
Wrapping Up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209

Part V - Concurrency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210

Chapter 27 - The asyncio Module
Definitions . . . . . .
async and await . . .
A Coroutine Example
Scheduling Calls . . .
Tasks . . . . . . . . .
Wrapping Up . . . . .

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

. 211
. 211
. 212
. 213
. 214
. 216
. 217

Chapter 28 - The threading Module . . . . . . . . . . . . . . . . 218

Intro to threads . . . . . . .
Locks and Synchronization
Timers . . . . . . . . . . .
Other Thread Components
Thread Communication . .
Wrapping Up . . . . . . . .

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

218
222
227
228
228
230

CONTENTS

Chapter 29 - The multiprocessing Module

Getting Started With Multiprocessing
Locks . . . . . . . . . . . . . . . . . .
Logging . . . . . . . . . . . . . . . . .
The Pool Class . . . . . . . . . . . . .
Process Communication . . . . . . . .
Wrapping Up . . . . . . . . . . . . . .

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

. 231
. 231
. 233
. 234
. 236
. 237
. 238

Chapter 30 - The concurrent.futures Module . . . . . . . 239

Creating a Pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239

Deadlocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
Wrapping Up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244

Introduction
Welcome to Python 201! This book is a sequel to my first book, Python 101. I wrote this book to
help take you to the next level in your programming journey. It will also demonstrate to you some
of the many ways you can use Python in your day-to-day work. I will be covering intermediate and
some advanced level material in this book. This book is using Python 3 in all of its examples. Some
of the topics covered will only work in Python 3 as some of the modules covered contain updates
that were not back-ported to Python 2. Also note that this book will follow in the footsteps of my
first work in that I wont be covering the topics in minute detail, but just enough to allow you to
start using these building blocks yourself.
Python 201 will be split into four parts:

Part one will cover intermediate modules

Part two will be on a variety of topics, such as unicode, generators and iterators
Part three will cover web related tasks with Python (but not web frameworks)
Part four is going to cover testing code

Let me spend a few moments explaining what each part has to offer. In part one, we will cover the
following intermediate modules and topics:
argparse (Parsing command line arguments)
collections (Replacements for Pythons standard containers, such as tuple, list and
dict)
contextlib (Context Managers)
functools (Partials, function overloading, etc)
importing and importlib
iterators and generators
itertools (Creating your own iterators)
Regular Expressions
Type Hinting
In Part two, we will be learning what generators and iterators are as well as how they work. Well
also be looking at some of Pythons special functions, like map and filter. There will be a section on
Unicode, encoding and decoding strings, benchmarking and more in this section.

2
Part three will cover working with websites in Python. Well look at how to browse a website with
Python and scrape some data. We will work with various Python APIs to access major websites.
We will use Python to create a client and a server. Finally well play around with Pythons FTP
capabilities.
Part four is all about testing your code. We will look at Pythons doctest, unittest and mock modules.
Well also learn about some 3rd party packages that you can use for testing, such as tox, nose and
pytest.
Please note that the chapters and sections may not all be the same length. While every topic will be
covered well, not every topic will require the same page count.

About the Author

You may be wondering about who I am and why I might be knowledgeable enough about Python to
write about it, so I thought Id give you a little information about myself. I started programming in
Python in the Spring of 2006 for a job. My first assignment was to port Windows login scripts from
Kixtart to Python. My second project was to port VBA code (basically a GUI on top of Microsoft
Office products) to Python, which is how I first got started in wxPython. Ive been using Python
ever since, doing a variation of backend programming and desktop front end user interfaces.
I realized that one way for me to remember how to do certain things in Python was to write
about them and thats how my Python blog came about: http://www.blog.pythonlibrary.org/. As
I wrote, I would receive feedback from my readers and I ended up expanding the blog to include
tips, tutorials, Python news, and Python book reviews. I work regularly with Packt Publishing as
a technical reviewer, which means that I get to try to check for errors in the books before theyre
published. I also have written for the Developer Zone (DZone) and i-programmer websites as well as
the Python Software Foundation. In November 2013, DZone published The Essential Core Python
Cheat Sheet that I co-authored. I have also self-published Python 101, which came out in June 2014.

Conventions
As with most technical books, this one includes a few conventions that you need to be aware of. New
topics and terminology will be in bold. You will also see some examples that look like the following:
1

>>> myString = "Welcome to Python!"

The >>> is a Python prompt symbol. You will see this in the Python interpreter and in IDLE. Other
code examples will be shown in a similar manner, but without the >>>.

Requirements
You will need a working Python 3 installation. Most Linux and Mac machines come with Python
already installed, however they might not have Python 3 as their default. If you happen to find
yourself without Python, you can go download a copy from http://python.org/download/. There are
up-to-date installation instructions on their website, so I wont include any installation instructions
in this book. Any additional requirements will be explained later on in the book.

Reader Feedback
I welcome feedback about my writings. If youd like to let me know what you thought of the book,
you can send comments to the following address:
[email protected]

Errata
I try my best not to publish errors in my writings, but it happens from time to time. If you happen
to see an error in this book, feel free to let me know by emailing me at the following:
[email protected]
Now lets get started!
http://python.org/download/

Part I - The Intermediate Modules

Welcome to part I. In this section of the book, we will be looking at a sampling of some of Pythons
intermediate level modules and concepts. There are two big topics that will be covered plus eight
chapters devoted to specific modules from the Python Standard Library. For the first topical chapter,
you will learn about how iterators and generators work and what the differences are between the
two of them. In the second, you will learn all about how Pythons importing system works.
Heres a more specific run-down of whats covered in this section:

Chapter 1 - The argparse module

Chapter 2 - The collections module
Chapter 3 - The contextlib module (Context Managers)
Chapter 4 - The functools module (Function overloading, caching, etc)
Chapter 5 - All about imports
Chapter 6 - The importlib module
Chapter 7 - Iterators and Generators
Chapter 8 - The itertools module
Chapter 9 - The re module (An Intro to Regex in Python)
Chapter 10 - The typing module (Type Hinting)

Lets get started!