Opensource.

com

7 essential PyPI libraries

and how to use them
Opensource.com . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

About Opensource.com

What is Opensource.com?

Opensource.com publishes stories about creating,

adopting, and sharing open source
solutions. Visit Opensource.com to learn more about how the open source
way is improving technologies, education, business, government, health, law,
entertainment, humanitarian efforts, and more.

Submit a story idea: https://opensource.com/story

Email us: [email protected]

2 7 essential PyPI libraries and how to use them . CC BY-SA 4.0 . Opensource.com
 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ABOUT THE AUTHOR

MOSHE ZADKA

MOSHE HAS BEEN INVOLVED IN THE LINUX COMMUNITY

since 1998, helping
in Linux “installation parties”. He has been programming Python since 1999, and
has contributed to the core Python interpreter. Moshe has been a DevOps/SRE
since before those terms existed, caring deeply
about software reliability, build reproducibility and
other such things. He has worked in companies
as small as three people and as big as tens of
thousands — usually some place around where
software meets system administration.

FOLLOW MOSHE ZADKA

Twitter: https://twitter.com/moshezadka

7 ESSENTIAL PYPI LIBRARIES AND HOW TO USE THEM . CC BY-SA 4.0 . OPENSOURCE.COM 3
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Introduction

Introduction 5

Chapters

Write faster C extensions for Python with Cython 6

Format Python however you like with Black 7
Say goodbye to boilerplate in Python with attrs 8
Add methods retroactively in Python with singledispatch 9
Automate your Python code tests with tox 10
Ensure consistency in your Python code with flake8 12
Check type annotations in Python with mypy 13

Get Involved | Additional Resources

Write for Us 15

4 7 essential PyPI libraries and how to use them . CC BY-SA 4.0 . Opensource.com
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Introduction

Introduction
Python is one of the most popular programming languages in use
today—and for good reasons: it's open source, it has
a wide range of uses (such as web programming, business applications,
games, scientific programming, and much more), and it has a vibrant
and dedicated community supporting it. This community is the reason we
have such a large, diverse range of software packages available in the
Python Package Index (PyPI) to extend and improve Python and solve
the inevitable glitches that crop up.
In this series, we'll look at seven PyPI libraries that can help you solve
common Python problems.

7 essential PyPI libraries and how to use them . CC BY-SA 4.0 . Opensource.com 5
Write faster C extensions for Python with Cython . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Write faster C extensions

for Python with Cython
Cython is a language that simplifies writing C extensions for Python.

Python is fun to use, but sometimes,

programs written in it
can be slow. All the runtime dynamic dispatching comes
>>> import pyximport; pyximport.install()
>>> import fib
>>> fib.fib(36)
with a steep price: sometimes it’s up to 10-times slower
than equivalent code written in a systems language like Just using Cython with no code changes reduced the time
C or Rust. the algorithm takes on my laptop to around 2.5 seconds.
Moving pieces of code to a completely new language can That’s a reduction of almost 50% runtime with almost no ef-
have a big cost in both effort and reliability: All that manual fort; certainly, a scrumptious cake to eat and have!
rewrite work will inevitably introduce bugs. Can we have our Putting in a little more effort, we can make things even
cake and eat it too? faster.
To have something to optimize for this exercise, we need
something slow. What can be slower than an accidentally cpdef int fib(int n):
exponential implementation of the Fibonacci sequence? if n < 2:
return 1
def fib(n): return fib(n - 1) + fib(n - 2)
if n < 2:
return 1 We moved the code in fib to a function defined with cpdef
return fib(n-1) + fib(n-2) and added a couple of type annotations: it takes an integer
and returns an integer.
Since a call to fib results in two calls, this beautifully ineffi- This makes it much faster—around 0.05 seconds. It’s so
cient algorithm takes a long time to execute. For example, fast that I may start suspecting my measurement methods
on my new laptop, fib(36) takes about 4.5 seconds. These contain noise: previously, this noise was lost in the signal.
4.5 seconds will be our baseline as we explore how Python’s So, the next time some of your Python code spends too
Cython extension [1] can help. long on the CPU, maybe spinning up some fans in the pro-
The proper way to use Cython is to integrate it into setup. cess, why not see if Cython can fix things?
py. However, a quick and easy way to try things out is with
pyximport. Let’s put the fib code above in fib.pyx and run Links
it using Cython. [1] https://pypi.org/project/Cython/

6 7 essential PyPI libraries and how to use them . CC BY-SA 4.0 . Opensource.com
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Format Python however you like with Black

Format Python however

you like with Black
Black formats your Python code consistently for you.

Sometimes creativity can be a

wonderful
thing. Sometimes it is just a pain. I enjoy solving hard prob-
Black does offer the option of failing instead of fixing and
even outputting a diff-style edit. These options are great
in a continuous integration (CI) system that enforces run-
lems creatively, but I want my Python formatted as con- ning Black locally. In addition, if the diff output is logged
sistently as possible. Nobody has ever been impressed by to the CI output, you can directly paste it into patch in the
code that uses “interesting” indentation. rare case that you need to fix your output but cannot install
But even worse than inconsistent formatting is a code Black locally.
review that consists of nothing but formatting nits. It is an-
noying to the reviewer—and even more annoying to the $ black --check --diff bad
person whose code is reviewed. It’s also infuriating when --- math 2019-04-09 17:24:22.747815 +0000
your linter tells you that your code is indented incorrectly, +++ math 2019-04-09 17:26:04.269451 +0000
but gives no hint about the correct amount of indentation. @@ -1,7 +1,7 @@
Enter Black [1]. Instead of telling you what to do, Black is a -def add(a, b): return a + b
good, industrious robot: it will fix your code for you. +def add(a, b):
To see how it works, feel free to write something beautiful- + return a + b
ly inconsistent like:

def add(a, b): return a+b def mult(a, b):

- return \
def mult(a, b): - a * b
return \ + return a * b
a * b
would reformat math
Does Black complain? Goodness no, it just fixes it for you! All done!
1 file would be reformatted.
$ black math $ echo $?
reformatted math 1
All done!
1 file reformatted. Links
$ cat math [1] https://pypi.org/project/black/
def add(a, b):
return a + b

def mult(a, b):

return a * b

7 essential PyPI libraries and how to use them . CC BY-SA 4.0 . Opensource.com 7
Say goodbye to boilerplate in Python with attrs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Say goodbye to boilerplate

in Python with attrs
attrs is a Python package that helps you write concise, correct code quickly.

If you have been using Python for any length of time, you
are probably used to writing code like:
ISBNs have a specific format. What if we want to enforce
that format?

class Book(object): @attr.s(auto_attribs=True)

class Book(object):
def __init__(self, isbn, name, author): isbn: str = attr.ib()
self.isbn = isbn @isbn.validator
self.name = name def pattern_match(self, attribute, value):
self.author = author m = re.match(r"^(\d{3}-)\d{1,3}-\d{2,3}-\d{1,7}-\d$", value)
if not m:
Then you write a __repr__ function; otherwise, it would be raise ValueError("incorrect format for isbn", value)
hard to log instances of Book: name: str
author: str
def __repr__(self): published_year: int
return f"Book({self.isbn}, {self.name}, {self.author})" edition: int

Next, you write a nice docstring documenting the expected The attrs library also has great support for immutability-style
types. But you notice you forgot to add the edition and programming [2]. Changing the first line to @attr.s(auto_at-
published_year attributes, so you have to modify them in tribs=True, frozen=True) means that Book is now im-
five places. mutable: trying to modify an attribute will raise an exception.
What if you didn’t have to? Instead, we can get a new instance with modification using
attr.evolve(old_book, published_year=old_book.pub-
@attr.s(auto_attribs=True) lished_year+1), for example, if we need to push publication
class Book(object): forward by a year.
isbn: str
name: str Links
author: str [1] https://pypi.org/project/attrs/
published_year: int [2] https://opensource.com/article/18/10/functional-
edition: int programming-python-immutable-data-structures

Annotating the attributes with types using the new type anno-
tation syntax, attrs [1] detects the annotations and creates a
class.

8 7 essential PyPI libraries and how to use them . CC BY-SA 4.0 . Opensource.com
. . . . . . . . . . . . . . . . . . . . . . . . . . Add methods retroactively in Python with singledispatch

Add methods retroactively in

Python with singledispatch
singledispatch is a library that allows you to add methods to Python libraries retroactively.

Imagine you have a “shapes” library with a Cir-

cle class, a Square class, etc.
A Circle has a radius, a Square has a side, and a Rect-
One nice thing about doing things this way is that if some-
one writes a new shape that is intended to play well with our
code, they can implement get_area themselves.
angle has height and width. Our library already exists; we from area_calculator import get_area
do not want to change it.
However, we do want to add an area calculation to our @attr.s(auto_attribs=True, frozen=True)
library. If we didn’t share this library with anyone else, we class Ellipse:
could just add an area method so we could call shape.area() horizontal_axis: float
and not worry about what the shape is. vertical_axis: float
While it is possible to reach into a class and add a method,
this is a bad idea: nobody expects their class to grow new @get_area.register(Ellipse)
methods, and things might break in weird ways. def _get_area_ellipse(shape):
Instead, the singledispatch [1] function in functools can return math.pi * shape.horizontal_axis * shape.vertical_axis
come to our rescue.
Calling get_area is straightforward.
@singledispatch
def get_area(shape): print(get_area(shape))
raise NotImplementedError("cannot calculate area for
unknown shape", shape) This means we can change a function that has a long if
isintance()/elif isinstance() chain to work this way, without
The “base” implementation for the get_area function fails. changing the interface. The next time you are tempted to
This makes sure that if we get a new shape, we will fail check if isinstance, try using singledispatch!
cleanly instead of returning a nonsense result.
Links
@get_area.register(Square) [1] https://pypi.org/project/singledispatch/
def _get_area_square(shape):
return shape.side ** 2
@get_area.register(Circle)
def _get_area_circle(shape):
return math.pi * (shape.radius ** 2)

7 essential PyPI libraries and how to use them . CC BY-SA 4.0 . Opensource.com 9
Automate your Python code tests with tox . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Automate your Python code

tests with tox
tox is a tool for automating tests on Python code.

When writing Python code, it is good to

have automated checks.
While you could dump the rules for running the checks di-
commands =
pytest mylibrary
[testenv:docs]
rectly into the continuous integration (CI) environment, that’s changedir = docs
seldom the best place for it. Among other things, it is useful deps =
to run tests locally, using the same parameters the CI runs, sphinx
to save CI time.. commands =
The tox project [1] is designed to run different checks sphinx-build -W -b html -d {envtmpdir}/doctrees .
against different versions of Python and against different {envtmpdir}/html
versions of dependencies. Very quickly, we find the limiting basepython = python3.7
factor is not the flexibility of tox but the harsh realities of the
combinatorial explosions of options! This example uses Sphinx [2] to build documentation for
For example, a simple tox configuration can run the same the library. One nice thing is that the Sphinx library will be
tests against several versions of Python. installed only in the docs virtual environment. If mylibrary
imports on Sphinx but forgets to indicate an explicit depen-
[tox] dency, the tests will, correctly, fail.
envlist = py36,py37 We can also use tox to run the tests with different versions
[testenv] of the dependencies.
deps =
pytest [tox]
commands = envlist = {py36,py37}-{minimum,current}
pytest mylibrary [testenv]
Tox will automatically use the right version of the inter- deps =
preter, based on the version of the environment, to create minimum: thirdparty==1.0
the virtual environment. Tox will automatically rebuild the current: thirdparty
virtual environment if it is missing or if the dependencies pytest
change. commands =
It is possible to explicitly indicate the Python version in an pytest mylibrary
environment.
This will run four different test runs: py36-minimum,
[tox] py36-current, py37-minimum, and py37-current. This is
envlist = py36,py37,docs useful in the case where our library depends on thirdparty>=​
[testenv] 1.0: every test run makes sure we are still compatible with the
deps = 1.0 version while also making sure the latest version does not
pytest break us.

10 7 essential PyPI libraries and how to use them . CC BY-SA 4.0 . Opensource.com
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Automate your Python code tests with tox

It is also a good idea to run a linter in tox. For example, By default, tox will run all test environments. But you can run
running Black [3] will do the right thing. just one environment; for example, if you only want to run
Black, run tox -e py36-black.
[tox] If you have a Python library you care about, add tox.ini to
envlist = py36,py37,py36-black your workflow to keep its quality high.
[testenv]
deps = Links
pytest [1] https://opensource.com/article/19/5/python-tox
commands = [2]  http://www.sphinx-doc.org/en/master/
pytest mylibrary [3] https://opensource.com/article/19/5/python-black
[testenv:py36-black]
deps =
black
commands =
black --check --diff mylibrary

7 essential PyPI libraries and how to use them . CC BY-SA 4.0 . Opensource.com 11
Ensure consistency in your Python code with flake8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Ensure consistency in your

Python code with flake8
flake8 is a linter and linting platform that ensures consistency in Python code.

Python code is meant to be

easy
to read. For this reason, consistency matters. Consistency
If we install flake8 in a clean virtual environment and run it, it
will say nothing: this file looks fine.
If we install flake8-print and run flake8 spew.py, we get:
inside a project matters most of all. How can we enforce
such consistency? spew.py:2:1: T001 print found.
Flake8 is really two things: it is both a linter, enforcing
some basic rules. Even more important, it is a linting platform If we instead install flake8-eradicate, we get:
that allows plugins to add or change linting rules.
The best thing about flake8 [1] plugins is that you don’t spew.py:1:1: E800: Found commented out code:
need to do anything other than installing them in the virtual
environment where you want to run flake8. We can, of course, install both—and get both warnings.
Consider the following code: You can also write local, custom plugins. If your team has
local conventions that are constantly nit-picked in reviews,
# spew.py why not automate them with a custom flake8 plugin?
print("Hello world")
# print("Goodbye universe") Links
[1] https://pypi.org/project/flake8/

12 7 essential PyPI libraries and how to use them . CC BY-SA 4.0 . Opensource.com
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Check type annotations in Python with mypy

Check type annotations in

Python with mypy
mypy “a Python linter on steroids.”

Python is a “dynamically typed” language. How-

ever, sometimes it is nice to let other
beings, both robotic and human, know what types are ex-
We use type annotations to denote that add_one expects an
integer and returns an integer. This does not change what
the code does. However, now we can ask a safe robot to find
pected. Traditionally, humans have been prioritized: input problems for us.
and output types of functions were described in docstrings.
MyPy [1] allows you to put the robots on equal footing, letting $ mypy typed.py
them know what types are intended. typed.py:6: error: 
Argument 1 to "add_one" has incompatible
Let’s look at the following code: type "str"; expected "int"

def add_one(input): We have a nice, readable explanation of what we are doing

return input + 1 wrong. Let’s fix print_seven.

def print_seven(): def print_seven() -> None:

five = "5" five = 5
seven = add_one(add_one(five)) seven = add_one(add_one(five))
print(seven) print(seven)

Calling print_seven raises a TypeError informing us we If we run mypy on this, there will not be any complaints; we
cannot add a string and a number: we cannot add “5” fixed the bug. This also results, happily, in working code.
and 1. The Python type system can get pretty deep, of course. It
However, we cannot know this until we run the code. Run- is not uncommon to encounter signatures like:
ning the code, if it were correct, would have produced a print- from typing import Dict, List, Mapping, Sequence
out to the screen: a side-effect. A relatively harmless one, as
side-effects go, but still, a side-effect. Is it possible to do it def unify_results(
without risking any side-effects? results1: Mapping[str, Sequence[int]],
We just have to let the robots know what to expect. results2: Mapping[str, Sequence[int]]
) -> Dict[str, List[int]]:
def add_one(input: int) -> int: pass
return input + 1
In those cases, remember that everything is an object: yes,
def print_seven() -> None: even types.
five = "5"
seven = add_one(add_one(five)) ResultsType = Mapping[str, Sequence[int]]
print(seven) ConcreteResultsType = Dict[str, List[int]]

7 essential PyPI libraries and how to use them . CC BY-SA 4.0 . Opensource.com 13
Check type annotations in Python with mypy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

def unify_results(
results1: ResultsType, results2: ResultsType) guarantee less in order to allow future changes to change the
-> ConcreteResultsType: return type.
pass MyPy allows progressive annotation: not everything has to
be annotated at once. Functions without any annotations will
We defined the input types as abstract types (using Mapping not be type-checked.
and Sequence). This allows sending in, say, a defaultdict, Go forth and annotate!
which maps strings to tuples. This is usually the right choice.
We also chose to guarantee concrete return types in the sig- Links
nature. This is more controversial: sometimes it is useful to [1] https://pypi.org/project/mypy/

14 7 essential PyPI libraries and how to use them . CC BY-SA 4.0 . Opensource.com
 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Write for Us

Write for Us

In 2010, Red Hat CEO Jim Whitehurst announced the launch of Opensource.com
in a post titled Welcome to the conversation on Opensource.com. He explained,
“This site is one of the ways in which Red Hat gives something back to the open
source community. Our desire is to create a connection point for conversations
about the broader impact that open source can have—and is having—even beyond
the software world.” he wrote, adding, “All ideas are welcome, and all participants
are welcome. This will not be a site for Red Hat, about Red Hat. Instead, this will be
a site for open source, about the future.”

By 2013, Opensource.com was publishing an average of 46 articles per month,

and in March 2016, Opensource.com surpassed 1-million page views for the first
time. In 2019, Opensource.com averages more than 1.5 million page views and
90 articles per month.

More than 60% of our content is contributed by members of open source communities,
and additional articles are written by the editorial team and other Red Hat contributors.
A small, international team of staff editors and Community Moderators work closely
with contributors to curate, polish, publish, and promote open source stories from
around the world.

Would you like to write for us? Send pitches and inquiries to [email protected].

To learn more, read 7 big reasons to contribute to Opensource.com.

7 essential PyPI libraries and how to use them . CC BY-SA 4.0 . Opensource.com 15