Python Packaging User Guide

Python Packaging Authority

Jun 08, 2020

 CONTENTS

1 An Overview of Packaging for Python 1

1.1 Thinking about deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Packaging Python libraries and tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Packaging Python applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.4 What about. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.5 Wrap up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2 Tutorials 9
2.1 Installing Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2 Managing Application Dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.3 Packaging Python Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.4 Creating Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3 Guides 25
3.1 Tool recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.2 Installing packages using pip and virtual environments . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.3 Installing stand alone command line tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.4 Installing pip/setuptools/wheel with Linux Package Managers . . . . . . . . . . . . . . . . . . . . . 33
3.5 Installing scientific packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.6 Multi-version installs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.7 Packaging and distributing projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.8 Including files in source distributions with MANIFEST.in . . . . . . . . . . . . . . . . . . . . . . 51
3.9 Single-sourcing the package version . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.10 Supporting multiple Python versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.11 Dropping support for older Python versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.12 Packaging binary extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.13 Supporting Windows using Appveyor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.14 Packaging namespace packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.15 Creating and discovering plugins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.16 Analyzing PyPI package downloads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.17 Package index mirrors and caches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
3.18 Hosting your own simple repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
3.19 Migrating to PyPI.org . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
3.20 Using TestPyPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
3.21 Making a PyPI-friendly README . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
3.22 Publishing package distribution releases using GitHub Actions CI/CD workflows . . . . . . . . . . . 84

4 Discussions 87
4.1 Deploying Python applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.2 pip vs easy_install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

i
 4.3 install_requires vs requirements files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.4 Wheel vs Egg . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

5 PyPA specifications 93
5.1 Package Distribution Metadata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
5.2 Package Index Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

6 Project Summaries 111

6.1 PyPA Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
6.2 Non-PyPA Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
6.3 Standard Library Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

7 Glossary 119

8 How to Get Support 121

9 Contribute to this guide 123

9.1 Documentation types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
9.2 Building the guide locally . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
9.3 Where the guide is deployed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
9.4 Style guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

10 News 127
10.1 September 2019 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
10.2 August 2019 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
10.3 July 2019 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
10.4 June 2019 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
10.5 May 2019 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
10.6 April 2019 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
10.7 March 2019 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
10.8 February 2019 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
10.9 January 2019 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
10.10 December 2018 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
10.11 November 2018 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
10.12 October 2018 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
10.13 September 2018 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
10.14 August 2018 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
10.15 July 2018 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
10.16 June 2018 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
10.17 May 2018 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
10.18 April 2018 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
10.19 March 2018 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
10.20 February 2018 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
10.21 January 2018 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
10.22 December 2017 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
10.23 November 2017 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
10.24 October 2017 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
10.25 September 2017 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
10.26 August 2017 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
10.27 July 2017 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
10.28 June 2017 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
10.29 May 2017 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
10.30 April 2017 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
10.31 March 2017 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
10.32 February 2017 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132

ii
11 Get started 133

12 Learn more 135

Index 137

iii
iv
 CHAPTER

ONE

AN OVERVIEW OF PACKAGING FOR PYTHON

As a general-purpose programming language, Python is designed to be used in many ways. You can build web sites
or industrial robots or a game for your friends to play, and much more, all using the same core technology.
Python’s flexibility is why the first step in every Python project must be to think about the project’s audience and the
corresponding environment where the project will run. It might seem strange to think about packaging before writing
code, but this process does wonders for avoiding future headaches.
This overview provides a general-purpose decision tree for reasoning about Python’s plethora of packaging options.
Read on to choose the best technology for your next project.

Contents

• Thinking about deployment

• Packaging Python libraries and tools
– Python modules
– Python source distributions
– Python binary distributions
• Packaging Python applications
– Depending on a framework

* Service platforms
* Web browsers and mobile applications
– Depending on a pre-installed Python
– Depending on a separate software distribution ecosystem
– Bringing your own Python executable
– Bringing your own userspace
– Bringing your own kernel
– Bringing your own hardware
• What about. . .
– Operating system packages
– virtualenv
– Security

1
Python Packaging User Guide

• Wrap up

1.1 Thinking about deployment

Packages exist to be installed (or deployed), so before you package anything, you’ll want to have some answers to the
deployment questions below:
• Who are your software’s users? Will your software be installed by other developers doing software development,
operations people in a datacenter, or a less software-savvy group?
• Is your software intended to run on servers, desktops, mobile clients (phones, tablets, etc.), or embedded in
dedicated devices?
• Is your software installed individually, or in large deployment batches?
Packaging is all about target environment and deployment experience. There are many answers to the questions above
and each combination of circumstances has its own solutions. With this information, the following overview will guide
you to the packaging technologies best suited to your project.

1.2 Packaging Python libraries and tools

You may have heard about PyPI, setup.py, and wheel files. These are just a few of the tools Python’s ecosystem
provides for distributing Python code to developers, which you can read about in Packaging and distributing projects.
The following approaches to packaging are meant for libraries and tools used by technical audience in a development
setting. If you’re looking for ways to package Python for a non-technical audience and/or a production setting, skip
ahead to Packaging Python applications.

1.2.1 Python modules

A Python file, provided it only relies on the standard library, can be redistributed and reused. You will also need to
ensure it’s written for the right version of Python, and only relies on the standard library.
This is great for sharing simple scripts and snippets between people who both have compatible Python versions (such
as via email, StackOverflow, or GitHub gists). There are even some entire Python libraries that offer this as an option,
such as bottle.py and boltons.
However, this pattern won’t scale for projects that consist of multiple files, need additional libraries, or need a specific
version of Python, hence the options below.

1.2.2 Python source distributions

If your code consists of multiple Python files, it’s usually organized into a directory structure. Any directory containing
Python files can comprise an import package.
Because packages consist of multiple files, they are harder to distribute. Most protocols support transferring only one
file at a time (when was the last time you clicked a link and it downloaded multiple files?). It’s easier to get incomplete
transfers, and harder to guarantee code integrity at the destination.
So long as your code contains nothing but pure Python code, and you know your deployment environment supports
your version of Python, then you can use Python’s native packaging tools to create a source distribution package, or
sdist for short.

2 Chapter 1. An Overview of Packaging for Python

 Python Packaging User Guide

Python’s sdists are compressed archives (.tar.gz files) containing one or more packages or modules. If your code
is pure-Python, and you only depend on other Python packages, you can go here to learn more.
If you rely on any non-Python code, or non-Python packages (such as libxml2 in the case of lxml, or BLAS libraries
in the case of numpy), you will need to use the format detailed in the next section, which also has many advantages
for pure-Python libraries.

Note: Python and PyPI support multiple distributions providing different implementations of the same package. For
instance the unmaintained-but-seminal PIL distribution provides the PIL package, and so does Pillow, an actively-
maintained fork of PIL!
This Python packaging superpower makes it possible for Pillow to be a drop-in replacement for PIL, just by changing
your project’s install_requires or requirements.txt.

1.2.3 Python binary distributions

So much of Python’s practical power comes from its ability to integrate with the software ecosystem, in particular
libraries written in C, C++, Fortran, Rust, and other languages.
Not all developers have the right tools or experiences to build these components written in these compiled languages,
so Python created the wheel, a package format designed to ship libraries with compiled artifacts. In fact, Python’s
package installer, pip, always prefers wheels because installation is always faster, so even pure-Python packages
work better with wheels.
Binary distributions are best when they come with source distributions to match. Even if you don’t upload wheels of
your code for every operating system, by uploading the sdist, you’re enabling users of other platforms to still build it
for themselves. Default to publishing both sdist and wheel archives together, unless you’re creating artifacts for a very
specific use case where you know the recipient only needs one or the other.
Python and PyPI make it easy to upload both wheels and sdists together. Just follow the Packaging Python Projects
tutorial.

Fig. 1: Python’s recommended built-in library and tool packaging technologies. Excerpted from The Packaging
Gradient (2017).

1.3 Packaging Python applications

So far we’ve only discussed Python’s native distribution tools. Based on our introduction, you would be correct to
infer these built-in approaches only target environments which have Python, and an audience who knows how to install
Python packages.

1.3. Packaging Python applications 3

Python Packaging User Guide

With the variety of operating systems, configurations, and people out there, this assumption is only safe when targeting
a developer audience.
Python’s native packaging is mostly built for distributing reusable code, called libraries, between developers. You can
piggyback tools, or basic applications for developers, on top of Python’s library packaging, using technologies like
setuptools entry_points.
Libraries are building blocks, not complete applications. For distributing applications, there’s a whole new world of
technologies out there.
The next few sections organize these application packaging options according to their dependencies on the target
environment, so you can choose the right one for your project.

1.3.1 Depending on a framework

Some types of Python applications, like web site backends and other network services, are common enough that they
have frameworks to enable their development and packaging. Other types of applications, like dynamic web frontends
and mobile clients, are complex enough to target that a framework becomes more than a convenience.
In all these cases, it makes sense to work backwards, from the framework’s packaging and deployment story. Some
frameworks include a deployment system which wraps the technologies outlined in the rest of the guide. In these cases,
you’ll want to defer to your framework’s packaging guide for the easiest and most reliable production experience.
If you ever wonder how these platforms and frameworks work under the hood, you can always read the sections
beyond.

Service platforms

If you’re developing for a “Platform-as-a-Service” or “PaaS” like Heroku or Google App Engine, you are going to
want to follow their respective packaging guides.
• Heroku
• Google App Engine
• PythonAnywhere
• OpenShift
• “Serverless” frameworks like Zappa
In all these setups, the platform takes care of packaging and deployment, as long as you follow their patterns. Most
software does not fit one of these templates, hence the existence of all the other options below.
If you’re developing software that will be deployed to machines you own, users’ personal computers, or any other
arrangement, read on.

Web browsers and mobile applications

Python’s steady advances are leading it into new spaces. These days you can write a mobile app or web application
frontend in Python. While the language may be familiar, the packaging and deployment practices are brand new.
If you’re planning on releasing to these new frontiers, you’ll want to check out the following frameworks, and refer to
their packaging guides:
• Kivy
• Beeware
• Brython

4 Chapter 1. An Overview of Packaging for Python

 Python Packaging User Guide

• Flexx
If you are not interested in using a framework or platform, or just wonder about some of the technologies and tech-
niques utilized by the frameworks above, continue reading below.

1.3.2 Depending on a pre-installed Python

Pick an arbitrary computer, and depending on the context, there’s a very good chance Python is already installed.
Included by default in most Linux and Mac operating systems for many years now, you can reasonably depend on
Python preexisting in your data centers or on the personal machines of developers and data scientists.
Technologies which support this model:
• PEX (Python EXecutable)
• zipapp (does not help manage dependencies, requires Python 3.5+)
• shiv (requires Python 3)

Note: Of all the approaches here, depending on a pre-installed Python relies the most on the target environment. Of
course, this also makes for the smallest package, as small as single-digit megabytes, or even kilobytes.
In general, decreasing the dependency on the target system increases the size of our package, so the solutions here are
roughly arranged by increasing size of output.

1.3.3 Depending on a separate software distribution ecosystem

For a long time many operating systems, including Mac and Windows, lacked built-in package management. Only
recently did these OSes gain so-called “app stores”, but even those focus on consumer applications and offer little for
developers.
Developers long sought remedies, and in this struggle, emerged with their own package management solutions, such as
Homebrew. The most relevant alternative for Python developers is a package ecosystem called Anaconda. Anaconda
is built around Python and is increasingly common in academic, analytical, and other data-oriented environments, even
making its way into server-oriented environments.
Instructions on building and publishing for the Anaconda ecosystem:
• Building libraries and applications with conda
• Transitioning a native Python package to Anaconda
A similar model involves installing an alternative Python distribution, but does not support arbitrary operating system-
level packages:
• Enthought Canopy
• ActiveState ActivePython
• WinPython

1.3.4 Bringing your own Python executable

Computing as we know it is defined by the ability to execute programs. Every operating system natively supports one
or more formats of program they can natively execute.

1.3. Packaging Python applications 5

Python Packaging User Guide

There are many techniques and technologies which turn your Python program into one of these formats, most of which
involve embedding the Python interpreter and any other dependencies into a single executable file.
This approach, called freezing, offers wide compatiblity and seamless user experience, though often requires multiple
technologies, and a good amount of effort.
A selection of Python freezers:
• pyInstaller - Cross-platform
• cx_Freeze - Cross-platform
• constructor - For command-line installers
• py2exe - Windows only
• py2app - Mac only
• bbFreeze - Windows, Linux, Python 2 only
• osnap - Windows and Mac
• pynsist - Windows only
Most of the above imply single-user deployments. For multi-component server applications, see Chef Omnibus.

1.3.5 Bringing your own userspace

An increasing number of operating systems – including Linux, Mac OS, and Windows – can be set up to run appli-
cations packaged as lightweight images, using a relatively modern arrangement often referred to as operating-system-
level virtualization, or containerization.
These techniques are mostly Python agnostic, because they package whole OS filesystems, not just Python or Python
packages.
Adoption is most extensive among Linux servers, where the technology originated and where the technologies below
work best:
• AppImage
• Docker
• Flatpak
• Snapcraft

1.3.6 Bringing your own kernel

Most operating systems support some form of classical virtualization, running applications packaged as images con-
taining a full operating system of their own. Running these virtual machines, or VMs, is a mature approach, widespread
in data center environments.
These techniques are mostly reserved for larger scale deployments in data centers, though certain complex applications
can benefit from this packaging. Technologies are Python agnostic, and include:
• Vagrant
• VHD, AMI, and other formats
• OpenStack - A cloud management system in Python, with extensive VM support

6 Chapter 1. An Overview of Packaging for Python

 Python Packaging User Guide

1.3.7 Bringing your own hardware

The most all-encompassing way to ship your software would be to ship it already-installed on some hardware. This
way, your software’s user would require only electricity.
Whereas the virtual machines described above are primarily reserved for the tech-savvy, you can find hardware appli-
ances being used by everyone from the most advanced data centers to the youngest children.
Embed your code on an Adafruit, MicroPython, or more-powerful hardware running Python, then ship it to the data-
center or your users’ homes. They plug and play, and you can call it a day.

Fig. 2: The simplified gamut of technologies used to package Python applications.

1.4 What about. . .

The sections above can only summarize so much, and you might be wondering about some of the more conspicuous
gaps.

1.4.1 Operating system packages

As mentioned in Depending on a separate software distribution ecosystem above, some operating systems have pack-
age managers of their own. If you’re very sure of the operating system you’re targeting, you can depend directly on a
format like deb (for Debian, Ubuntu, etc.) or RPM (for Red Hat, Fedora, etc.), and use that built-in package manager
to take care of installation, and even deployment. You can even use FPM to generate both deb and RPMs from the
same source.
In most deployment pipelines, the OS package manager is just one piece of the puzzle.

1.4.2 virtualenv

Virtualenvs have been an indispensible tool for multiple generations of Python developer, but are slowly fading from
view, as they are being wrapped by higher-level tools. With packaging in particular, virtualenvs are used as a primitive
in the dh-virtualenv tool and osnap, both of which wrap virtualenvs in a self-contained way.
For production deployments, do not rely on running pip install from the Internet into a virtualenv, as one might
do in a development environment. The overview above is full of much better solutions.

1.4. What about. . . 7

Python Packaging User Guide

1.4.3 Security

The further down the gradient you come, the harder it gets to update components of your package. Everything is more
tightly bound together.
For example, if a kernel security issue emerges, and you’re deploying containers, the host system’s kernel can be
updated without requiring a new build on behalf of the application. If you deploy VM images, you’ll need a new
build. Whether or not this dynamic makes one option more secure is still a bit of an old debate, going back to the
still-unsettled matter of static versus dynamic linking.

1.5 Wrap up

Packaging in Python has a bit of a reputation for being a bumpy ride. This impression is mostly a byproduct of Python’s
versatility. Once you understand the natural boundaries between each packaging solution, you begin to realize that
the varied landscape is a small price Python programmers pay for using one of the most balanced, flexible language
available.

8 Chapter 1. An Overview of Packaging for Python

 CHAPTER

TWO

TUTORIALS

Tutorials are opinionated step-by-step guides to help you get familiar with packaging concepts. For more detailed
information on specific packaging topics, see Guides.

2.1 Installing Packages

This section covers the basics of how to install Python packages.

It’s important to note that the term “package” in this context is being used as a synonym for a distribution (i.e. a
bundle of software to be installed), not to refer to the kind of package that you import in your Python source code (i.e.
a container of modules). It is common in the Python community to refer to a distribution using the term “package”.
Using the term “distribution” is often not preferred, because it can easily be confused with a Linux distribution, or
another larger software distribution like Python itself.

Contents

• Requirements for Installing Packages

– Ensure you can run Python from the command line
– Ensure you can run pip from the command line
– Ensure pip, setuptools, and wheel are up to date
– Optionally, create a virtual environment
• Creating Virtual Environments
• Use pip for Installing
• Installing from PyPI
• Source Distributions vs Wheels
• Upgrading packages
• Installing to the User Site
• Requirements files
• Installing from VCS
• Installing from other Indexes
• Installing from a local src tree
• Installing from local archives

9
Python Packaging User Guide

• Installing from other sources

• Installing Prereleases
• Installing Setuptools “Extras”

2.1.1 Requirements for Installing Packages

This section describes the steps to follow before installing other Python packages.

Ensure you can run Python from the command line

Before you go any further, make sure you have Python and that the expected version is available from your command
line. You can check this by running:

python --version

You should get some output like Python 3.6.3. If you do not have Python, please install the latest 3.x version
from python.org or refer to the Installing Python section of the Hitchhiker’s Guide to Python.

Note: If you’re a newcomer and you get an error like this:

>>> python --version

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'python' is not defined

It’s because this command and other suggested commands in this tutorial are intended to be run in a shell (also called
a terminal or console). See the Python for Beginners getting started tutorial for an introduction to using your operating
system’s shell and interacting with Python.

Note: If you’re using an enhanced shell like IPython or the Jupyter notebook, you can run system commands like
those in this tutorial by prefacing them with a ! character:

In [1]: import sys

!{sys.executable} --version
Python 3.6.3

It’s recommended to write {sys.executable} rather than plain python in order to ensure that commands are
run in the Python installation matching the currently running notebook (which may not be the same Python installation
that the python command refers to).

Note: Due to the way most Linux distributions are handling the Python 3 migration, Linux users using the sys-
tem Python without creating a virtual environment first should replace the python command in this tutorial with
python3 and the pip command with pip3 --user. Do not run any of the commands in this tutorial with sudo:
if you get a permissions error, come back to the section on creating virtual environments, set one up, and then continue
with the tutorial as written.

10 Chapter 2. Tutorials
 Python Packaging User Guide

Ensure you can run pip from the command line

Additionally, you’ll need to make sure you have pip available. You can check this by running:

pip --version

If you installed Python from source, with an installer from python.org, or via Homebrew you should already have pip.
If you’re on Linux and installed using your OS package manager, you may have to install pip separately, see Installing
pip/setuptools/wheel with Linux Package Managers.
If pip isn’t already installed, then first try to bootstrap it from the standard library:

python -m ensurepip --default-pip

If that still doesn’t allow you to run pip:

• Securely Download get-pip.py1
• Run python get-pip.py.2 This will install or upgrade pip. Additionally, it will install setuptools and
wheel if they’re not installed already.

Warning: Be cautious if you’re using a Python install that’s managed by your operating system or another
package manager. get-pip.py does not coordinate with those tools, and may leave your system in an inconsis-
tent state. You can use python get-pip.py --prefix=/usr/local/ to install in /usr/local
which is designed for locally-installed software.

Ensure pip, setuptools, and wheel are up to date

While pip alone is sufficient to install from pre-built binary archives, up to date copies of the setuptools and
wheel projects are useful to ensure you can also install from source archives:

python -m pip install --upgrade pip setuptools wheel

Optionally, create a virtual environment

See section below for details, but here’s the basic venv3 command to use on a typical Linux system:

python3 -m venv tutorial_env

source tutorial_env/bin/activate

This will create a new virtual environment in the tutorial_env subdirectory, and configure the current shell to use
it as the default python environment.

2.1.2 Creating Virtual Environments

Python “Virtual Environments” allow Python packages to be installed in an isolated location for a particular applica-
tion, rather than being installed globally. If you are looking to safely install global command line tools, see Installing
stand alone command line tools.
1 “Secure” in this context means using a modern browser or a tool like curl that verifies SSL certificates when downloading from https URLs.
2 Depending on your platform, this may require root or Administrator access. pip is currently considering changing this by making user installs
the default behavior.
3 Beginning with Python 3.4, venv (a stdlib alternative to virtualenv) will create virtualenv environments with pip pre-installed, thereby

making it an equal alternative to virtualenv.

2.1. Installing Packages 11

Python Packaging User Guide

Imagine you have an application that needs version 1 of LibFoo, but another application requires version 2. How
can you use both these applications? If you install everything into /usr/lib/python3.6/site-packages (or whatever your
platform’s standard location is), it’s easy to end up in a situation where you unintentionally upgrade an application that
shouldn’t be upgraded.
Or more generally, what if you want to install an application and leave it be? If an application works, any change in
its libraries or the versions of those libraries can break the application.
Also, what if you can’t install packages into the global site-packages directory? For instance, on a shared host.
In all these cases, virtual environments can help you. They have their own installation directories and they don’t share
libraries with other virtual environments.
Currently, there are two common tools for creating Python virtual environments:
• venv is available by default in Python 3.3 and later, and installs pip and setuptools into created virtual environ-
ments in Python 3.4 and later.
• virtualenv needs to be installed separately, but supports Python 2.7+ and Python 3.3+, and pip, setuptools and
wheel are always installed into created virtual environments by default (regardless of Python version).
The basic usage is like so:
Using venv:

python3 -m venv <DIR>

source <DIR>/bin/activate

Using virtualenv:

virtualenv <DIR>
source <DIR>/bin/activate

For more information, see the venv docs or the virtualenv docs.
The use of source under Unix shells ensures that the virtual environment’s variables are set within the current shell,
and not in a subprocess (which then disappears, having no useful effect).
In both of the above cases, Windows users should _not_ use the source command, but should rather run the activate
script directly from the command shell like so:

<DIR>\Scripts\activate

Managing multiple virtual environments directly can become tedious, so the dependency management tutorial in-
troduces a higher level tool, Pipenv, that automatically manages a separate virtual environment for each project and
application that you work on.

2.1.3 Use pip for Installing

pip is the recommended installer. Below, we’ll cover the most common usage scenarios. For more detail, see the pip
docs, which includes a complete Reference Guide.

2.1.4 Installing from PyPI

The most common usage of pip is to install from the Python Package Index using a requirement specifier. Generally
speaking, a requirement specifier is composed of a project name followed by an optional version specifier. PEP 440
contains a full specification of the currently supported specifiers. Below are some examples.
To install the latest version of “SomeProject”:

12 Chapter 2. Tutorials
 Python Packaging User Guide

pip install "SomeProject"

To install a specific version:

pip install "SomeProject==1.4"

To install greater than or equal to one version and less than another:

pip install "SomeProject>=1,<2"

To install a version that’s “compatible” with a certain version:4

pip install "SomeProject~=1.4.2"

In this case, this means to install any version “==1.4.*” version that’s also “>=1.4.2”.

2.1.5 Source Distributions vs Wheels

pip can install from either Source Distributions (sdist) or Wheels, but if both are present on PyPI, pip will prefer a
compatible wheel.
Wheels are a pre-built distribution format that provides faster installation compared to Source Distributions (sdist),
especially when a project contains compiled extensions.
If pip does not find a wheel to install, it will locally build a wheel and cache it for future installs, instead of rebuilding
the source distribution in the future.

2.1.6 Upgrading packages

Upgrade an already installed SomeProject to the latest from PyPI.

pip install --upgrade SomeProject

2.1.7 Installing to the User Site

To install packages that are isolated to the current user, use the --user flag:

pip install --user SomeProject

For more information see the User Installs section from the pip docs.
Note that the --user flag has no effect when inside a virtual environment - all installation commands will affect the
virtual environment.
If SomeProject defines any command-line scripts or console entry points, --user will cause them to be installed
inside the user base’s binary directory, which may or may not already be present in your shell’s PATH. (Starting in
version 10, pip displays a warning when installing any scripts to a directory outside PATH.) If the scripts are not
available in your shell after installation, you’ll need to add the directory to your PATH:
• On Linux and macOS you can find the user base binary directory by running python -m site
--user-base and adding bin to the end. For example, this will typically print ~/.local (with ~ ex-
panded to the absolute path to your home directory) so you’ll need to add ~/.local/bin to your PATH. You
can set your PATH permanently by modifying ~/.profile.
4 The compatible release specifier was accepted in PEP 440 and support was released in setuptools v8.0 and pip v6.0

2.1. Installing Packages 13

Python Packaging User Guide

• On Windows you can find the user base binary directory by running py -m site
--user-site and replacing site-packages with Scripts. For example, this could return
C:\Users\Username\AppData\Roaming\Python36\site-packages so you would need to
set your PATH to include C:\Users\Username\AppData\Roaming\Python36\Scripts. You can
set your user PATH permanently in the Control Panel. You may need to log out for the PATH changes to take
effect.

2.1.8 Requirements files

Install a list of requirements specified in a Requirements File.

pip install -r requirements.txt

2.1.9 Installing from VCS

Install a project from VCS in “editable” mode. For a full breakdown of the syntax, see pip’s section on VCS Support.

pip install -e git+https://git.repo/some_pkg.git#egg=SomeProject # from git

pip install -e hg+https://hg.repo/some_pkg#egg=SomeProject # from
˓→mercurial

pip install -e svn+svn://svn.repo/some_pkg/trunk/#egg=SomeProject # from svn

pip install -e git+https://git.repo/some_pkg.git@feature#egg=SomeProject # from a
˓→branch

2.1.10 Installing from other Indexes

Install from an alternate index

pip install --index-url http://my.package.repo/simple/ SomeProject

Search an additional index during install, in addition to PyPI

pip install --extra-index-url http://my.package.repo/simple SomeProject

2.1.11 Installing from a local src tree

Installing from local src in Development Mode, i.e. in such a way that the project appears to be installed, but yet is
still editable from the src tree.

pip install -e <path>

You can also install normally from src

pip install <path>

2.1.12 Installing from local archives

Install a particular source archive file.

14 Chapter 2. Tutorials
 Python Packaging User Guide

pip install ./downloads/SomeProject-1.0.4.tar.gz

Install from a local directory containing archives (and don’t check PyPI)

pip install --no-index --find-links=file:///local/dir/ SomeProject

pip install --no-index --find-links=/local/dir/ SomeProject
pip install --no-index --find-links=relative/dir/ SomeProject

2.1.13 Installing from other sources

To install from other data sources (for example Amazon S3 storage) you can create a helper application that presents
the data in a PEP 503 compliant index format, and use the --extra-index-url flag to direct pip to use that index.

./s3helper --port=7777
pip install --extra-index-url http://localhost:7777 SomeProject

2.1.14 Installing Prereleases

Find pre-release and development versions, in addition to stable versions. By default, pip only finds stable versions.

pip install --pre SomeProject

2.1.15 Installing Setuptools “Extras”

Install setuptools extras.

$ pip install SomePackage[PDF]

$ pip install SomePackage[PDF]==3.0
$ pip install -e .[PDF]==3.0 # editable project in current directory

2.2 Managing Application Dependencies

The package installation tutorial covered the basics of getting set up to install and update Python packages.
However, running these commands interactively can get tedious even for your own personal projects, and things
get even more difficult when trying to set up development environments automatically for projects with multiple
contributors.
This tutorial walks you through the use of Pipenv to manage dependencies for an application. It will show you how to
install and use the necessary tools and make strong recommendations on best practices.
Keep in mind that Python is used for a great many different purposes, and precisely how you want to manage your
dependencies may change based on how you decide to publish your software. The guidance presented here is most
directly applicable to the development and deployment of network services (including web applications), but is also
very well suited to managing development and testing environments for any kind of project.
Developers of Python libraries, or of applications that support distribution as Python libraries, should also consider the
poetry project as an alternative dependency management solution.

2.2. Managing Application Dependencies 15

Python Packaging User Guide

2.2.1 Recommendation caveats (as of April 2020)

While this tutorial covers the pipenv project as a tool that focuses primarily on the needs of Python application
development rather than Python library development, the project itself is currently working through several process
and maintenance issues that are preventing bug fixes and new features from being published (with the entirety of 2019
passing without a new release).
This means that in the near term, pipenv still suffers from several quirks and performance problems without a clear
timeline for resolution of those isses.
While this remains the case, project maintainers are likely to want to investigate Other Tools for Application Depen-
dency Management for use instead of, or together with, pipenv.
Assuming the April 2020 pipenv release goes ahead as planned, and the release after that also remains on track,
then this caveat on the tutorial will be removed. If those releases don’t remain on track, then the tutorial itself will be
removed, and replaced with a discussion page on the available dependency management options.

2.2.2 Installing Pipenv

Pipenv is a dependency manager for Python projects. If you’re familiar with Node.js’ npm or Ruby’s bundler, it
is similar in spirit to those tools. While pip alone is often sufficient for personal use, Pipenv is recommended for
collaborative projects as it’s a higher-level tool that simplifies dependency management for common use cases.
Use pip to install Pipenv:

pip install --user pipenv

Note: This does a user installation to prevent breaking any system-wide packages. If pipenv isn’t available in your
shell after installation, you’ll need to add the user base’s binary directory to your PATH. See Installing to the User Site
for more information.

2.2.3 Installing packages for your project

Pipenv manages dependencies on a per-project basis. To install packages, change into your project’s directory (or just
an empty directory for this tutorial) and run:

cd myproject
pipenv install requests

Pipenv will install the Requests library and create a Pipfile for you in your project’s directory. The Pipfile is used to
track which dependencies your project needs in case you need to re-install them, such as when you share your project
with others. You should get output similar to this (although the exact paths shown will vary):

Creating a Pipfile for this project...

Creating a virtualenv for this project...
Using base prefix '/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/
˓→Versions/3.6'

New python executable in ~/.local/share/virtualenvs/tmp-agwWamBd/bin/python3.6

Also creating executable in ~/.local/share/virtualenvs/tmp-agwWamBd/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: ~/.local/share/virtualenvs/tmp-agwWamBd

Installing requests...
(continues on next page)

16 Chapter 2. Tutorials
 Python Packaging User Guide

(continued from previous page)

Collecting requests
Using cached requests-2.18.4-py2.py3-none-any.whl
Collecting idna<2.7,>=2.5 (from requests)
Using cached idna-2.6-py2.py3-none-any.whl
Collecting urllib3<1.23,>=1.21.1 (from requests)
Using cached urllib3-1.22-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
Using cached chardet-3.0.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
Using cached certifi-2017.7.27.1-py2.py3-none-any.whl
Installing collected packages: idna, urllib3, chardet, certifi, requests
Successfully installed certifi-2017.7.27.1 chardet-3.0.4 idna-2.6 requests-2.18.4
˓→urllib3-1.22

Adding requests to Pipfile's [packages]...

2.2.4 Using installed packages

Now that Requests is installed you can create a simple main.py file to use it:

import requests

response = requests.get('https://httpbin.org/ip')

print('Your IP is {0}'.format(response.json()['origin']))

Then you can run this script using pipenv run:

pipenv run python main.py

You should get output similar to this:

Your IP is 8.8.8.8

Using pipenv run ensures that your installed packages are available to your script. It’s also possible to spawn a
new shell that ensures all commands have access to your installed packages with pipenv shell.

2.2.5 Next steps

Congratulations, you now know how to effectively manage dependencies and development environments on a collab-
orative Python project!
If you’re interested in creating and distributing your own Python packages, see the tutorial on packaging and distribut-
ing packages.
Note that when your application includes definitions of Python source packages, they (and
their dependencies) can be added to your pipenv environment with pipenv install -e
<relative-path-to-source-directory> (e.g. pipenv install -e . or pipenv install
-e src).

2.2. Managing Application Dependencies 17

Python Packaging User Guide

2.2.6 Other Tools for Application Dependency Management

If you find this particular approach to managing application dependencies isn’t working well for you or your use case,
you may want to explore these other tools and techniques to see if one of them is a better fit:
• poetry for a tool comparable in scope to pipenv that focuses more directly on use cases where the repository
being managed is structured as a Python project with a valid pyproject.toml file (by contrast, pipenv
explicitly avoids making the assumption that the application being worked on that’s depending on components
from PyPI will itself support distribution as a pip-installable Python package).
• hatch for opinionated coverage of even more steps in the project management workflow (such as incrementing
versions, tagging releases, and creating new skeleton projects from project templates)
• pip-tools to build your own custom workflow from lower level pieces like pip-compile and pip-sync

2.3 Packaging Python Projects

This tutorial walks you through how to package a simple Python project. It will show you how to add the necessary
files and structure to create the package, how to build the package, and how to upload it to the Python Package Index.

2.3.1 A simple project

This tutorial uses a simple project named example_pkg. If you are unfamiliar with Python’s modules and import
packages, take a few minutes to read over the Python documentation for packages and modules. Even if you already
have a project that you want to package up, we recommend following this tutorial as-is using this example package
and then trying with your own package.
To create this project locally, create the following file structure:

packaging_tutorial
example_pkg
__init__.py

Once you create this structure, you’ll want to run all of the commands in this tutorial within the top-level folder - so
be sure to cd packaging_tutorial.
example_pkg/__init__.py is required to import the directory as a package, and can simply be an empty file.

2.3.2 Creating the package files

You will now create a handful of files to package up this project and prepare it for distribution. Create the new files
listed below and place them in the project’s root directory - you will add content to them in the following steps.

packaging_tutorial
LICENSE
README.md
example_pkg
__init__.py
setup.py
tests

18 Chapter 2. Tutorials
 Python Packaging User Guide

2.3.3 Creating a test folder

tests/ is a placeholder for unit test files. Leave it empty for now.

2.3.4 Creating setup.py

setup.py is the build script for setuptools. It tells setuptools about your package (such as the name and version) as
well as which code files to include.
Open setup.py and enter the following content. Update the package name to include your username (for example,
example-pkg-theacodes), this ensures that you have a unique package name and that your package doesn’t
conflict with packages uploaded by other people following this tutorial.

import setuptools

with open("README.md", "r") as fh:

long_description = fh.read()

setuptools.setup(
name="example-pkg-YOUR-USERNAME-HERE", # Replace with your own username
version="0.0.1",
author="Example Author",
author_email="[email protected]",
description="A small example package",
long_description=long_description,
long_description_content_type="text/markdown",
url="https://github.com/pypa/sampleproject",
packages=setuptools.find_packages(),
classifiers=[
"Programming Language :: Python :: 3",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
],
python_requires='>=3.6',
)

setup() takes several arguments. This example package uses a relatively minimal set:
• name is the distribution name of your package. This can be any name as long as only contains letters, numbers,
_ , and -. It also must not already be taken on pypi.org. Be sure to update this with your username, as this
ensures you won’t try to upload a package with the same name as one which already exists when you upload the
package.
• version is the package version see PEP 440 for more details on versions.
• author and author_email are used to identify the author of the package.
• description is a short, one-sentence summary of the package.
• long_description is a detailed description of the package. This is shown on the package detail package
on the Python Package Index. In this case, the long description is loaded from README.md which is a common
pattern.
• long_description_content_type tells the index what type of markup is used for the long description.
In this case, it’s Markdown.
• url is the URL for the homepage of the project. For many projects, this will just be a link to GitHub, GitLab,
Bitbucket, or similar code hosting service.

2.3. Packaging Python Projects 19

Python Packaging User Guide

• packages is a list of all Python import packages that should be included in the distribution package. Instead
of listing each package manually, we can use find_packages() to automatically discover all packages and
subpackages. In this case, the list of packages will be example_pkg as that’s the only package present.
• classifiers gives the index and pip some additional metadata about your package. In this case, the package
is only compatible with Python 3, is licensed under the MIT license, and is OS-independent. You should
always include at least which version(s) of Python your package works on, which license your package is
available under, and which operating systems your package will work on. For a complete list of classifiers, see
https://pypi.org/classifiers/.
There are many more than the ones mentioned here. See Packaging and distributing projects for more details.

2.3.5 Creating README.md

Open README.md and enter the following content. You can customize this if you’d like.

# Example Package

This is a simple example package. You can use

[Github-flavored Markdown](https://guides.github.com/features/mastering-markdown/)
to write your content.

2.3.6 Creating a LICENSE

It’s important for every package uploaded to the Python Package Index to include a license. This tells users who install
your package the terms under which they can use your package. For help picking a license, see https://choosealicense.
com/. Once you have chosen a license, open LICENSE and enter the license text. For example, if you had chosen the
MIT license:

Copyright (c) 2018 The Python Packaging Authority

Permission is hereby granted, free of charge, to any person obtaining a copy

of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

2.3.7 Generating distribution archives

The next step is to generate distribution packages for the package. These are archives that are uploaded to the Package
Index and can be installed by pip.

20 Chapter 2. Tutorials
 Python Packaging User Guide

Make sure you have the latest versions of setuptools and wheel installed:

python3 -m pip install --user --upgrade setuptools wheel

Tip: IF you have trouble installing these, see the Installing Packages tutorial.

Now run this command from the same directory where setup.py is located:

python3 setup.py sdist bdist_wheel

This command should output a lot of text and once completed should generate two files in the dist directory:

dist/
example_pkg_YOUR_USERNAME_HERE-0.0.1-py3-none-any.whl
example_pkg_YOUR_USERNAME_HERE-0.0.1.tar.gz

Note: If you run into trouble here, please copy the output and file an issue over on packaging problems and we’ll do
our best to help you!

The tar.gz file is a source archive whereas the .whl file is a built distribution. Newer pip versions preferentially
install built distributions, but will fall back to source archives if needed. You should always upload a source archive
and provide built archives for the platforms your project is compatible with. In this case, our example package is
compatible with Python on any platform so only one built distribution is needed.

2.3.8 Uploading the distribution archives

Finally, it’s time to upload your package to the Python Package Index!
The first thing you’ll need to do is register an account on Test PyPI. Test PyPI is a separate instance of the package
index intended for testing and experimentation. It’s great for things like this tutorial where we don’t necessarily want
to upload to the real index. To register an account, go to https://test.pypi.org/account/register/ and complete the steps
on that page. You will also need to verify your email address before you’re able to upload any packages. For more
details on Test PyPI, see Using TestPyPI.
Now you’ll create a PyPI API token so you will be able to securely upload your project.
Go to https://test.pypi.org/manage/account/#api-tokens and create a new API token; don’t limit its scope to a particular
project, since you are creating a new project.
Don’t close the page until you have copied and saved the token — you won’t see that token again.
Now that you are registered, you can use twine to upload the distribution packages. You’ll need to install Twine:

python3 -m pip install --user --upgrade twine

Once installed, run Twine to upload all of the archives under dist:

python3 -m twine upload --repository testpypi dist/*

You will be prompted for a username and password. For the username, use __token__. For the password, use the
token value, including the pypi- prefix.
After the command completes, you should see output similar to this:

2.3. Packaging Python Projects 21

Python Packaging User Guide

Uploading distributions to https://test.pypi.org/legacy/

Enter your username: [your username]
Enter your password:
Uploading example_pkg_YOUR_USERNAME_HERE-0.0.1-py3-none-any.whl
100%|| 4.65k/4.65k [00:01<00:00, 2.88kB/s]
Uploading example_pkg_YOUR_USERNAME_HERE-0.0.1.tar.gz
100%|| 4.25k/4.25k [00:01<00:00, 3.05kB/s]

Once uploaded your package should be viewable on TestPyPI, for example, https://test.pypi.org/project/
example-pkg-YOUR-USERNAME-HERE

2.3.9 Installing your newly uploaded package

You can use pip to install your package and verify that it works. Create a new virtualenv (see Installing Packages for
detailed instructions) and install your package from TestPyPI:

python3 -m pip install --index-url https://test.pypi.org/simple/ --no-deps example-

˓→pkg-YOUR-USERNAME-HERE

Make sure to specify your username in the package name!

pip should install the package from Test PyPI and the output should look something like this:

Collecting example-pkg-YOUR-USERNAME-HERE
Downloading https://test-files.pythonhosted.org/packages/.../example-pkg-YOUR-
˓→USERNAME-HERE-0.0.1-py3-none-any.whl

Installing collected packages: example-pkg-YOUR-USERNAME-HERE

Successfully installed example-pkg-YOUR-USERNAME-HERE-0.0.1

Note: This example uses --index-url flag to specify TestPyPI instead of live PyPI. Additionally, it specifies
--no-deps. Since TestPyPI doesn’t have the same packages as the live PyPI, it’s possible that attempting to install
dependencies may fail or install something unexpected. While our example package doesn’t have any dependencies,
it’s a good practice to avoid installing dependencies when using TestPyPI.

You can test that it was installed correctly by importing the package. Run the Python interpreter (make sure you’re
still in your virtualenv):

python

and from the interpreter shell import the package:

>>> import example_pkg

Note that the import package is example_pkg regardless of what name you gave your distribution package in
setup.py (in this case, example-pkg-YOUR-USERNAME-HERE).

2.3.10 Next steps

Congratulations, you’ve packaged and distributed a Python project!

Keep in mind that this tutorial showed you how to upload your package to Test PyPI, which isn’t a permanent storage.
The Test system occasionally deletes packages and accounts. It is best to use Test PyPI for testing and experiments
like this tutorial.

22 Chapter 2. Tutorials
 Python Packaging User Guide

When you are ready to upload a real package to the Python Package Index you can do much the same as you did in
this tutorial, but with these important differences:
• Choose a memorable and unique name for your package. You don’t have to append your username as you did
in the tutorial.
• Register an account on https://pypi.org - note that these are two separate servers and the login details from the
test server are not shared with the main server.
• Use twine upload dist/* to upload your package and enter your credentials for the account you reg-
istered on the real PyPI. Now that you’re uploading the package in production, you don’t need to specify
--repository; the package will upload to https://pypi.org/ by default.
• Install your package from the real PyPI using pip install [your-package].
At this point if you want to read more on packaging Python libraries here are some things you can do:
• Read more about using setuptools to package libraries in Packaging and distributing projects.
• Read about Packaging binary extensions.
• Consider alternatives to setuptools such as flit, hatch, and poetry.

2.4 Creating Documentation

This section covers the basics of how to create documentation using Sphinx and host the documentation for free in
Read The Docs.

2.4.1 Installing Sphinx

Use pip to install Sphinx:

pip install -U sphinx

For other installation methods, see this installation guide by Sphinx.

2.4.2 Getting Started With Sphinx

Create a doc directory inside your project to hold your documentation:

cd /path/to/project
mkdir docs

Run spinx-quickstart inside the docs directory:

cd docs
sphinx-quickstart

This sets up a source directory, walks you through some basic configurations, and creates an index.rst file as well
as a conf.py file.
You can add some information about your project in index.rst, then build them:

make html

For more details on the build process, see this guide by Read The Docs.

2.4. Creating Documentation 23

Python Packaging User Guide

2.4.3 Other Sources

For a more detailed guide on how to use Sphinx and reStructuredText, please see this documentation tutorial on
Hitchhiker’s Guide to Python.

24 Chapter 2. Tutorials
 CHAPTER

THREE

GUIDES

Guides are focused on accomplishing a specific task and assume that you are already familiar with the basics of Python
packaging. If you’re looking for an introduction to packaging, see Tutorials.

3.1 Tool recommendations

If you’re familiar with Python packaging and installation, and just want to know what tools are currently recommended,
then here it is.

3.1.1 Application dependency management

Use Pipenv to manage library dependencies when developing Python applications. See Managing Application Depen-
dencies for more details on using pipenv.
When pipenv does not meet your use case, consider other tools like:
• pip
• pip-tools
• Poetry

3.1.2 Installation tool recommendations

• Use pip to install Python packages from PyPI.12 Depending on how pip is installed, you may need to also install
wheel to get the benefit of wheel caching.3
• Use virtualenv, or venv to isolate application specific dependencies from a shared Python installation.4
• If you’re looking for management of fully integrated cross-platform software stacks, consider:
– buildout: primarily focused on the web development community
– Spack, Hashdist, or conda: primarily focused on the scientific community.
1 There are some cases where you might choose to use easy_install (from setuptools), e.g. if you need to install from Eggs (which pip

doesn’t support). For a detailed breakdown, see pip vs easy_install.

2 The acceptance of PEP 453 means that pip will be available by default in most installations of Python 3.4 or later. See the rationale section

from PEP 453 as for why pip was chosen.

3 get-pip.py and virtualenv install wheel, whereas ensurepip and venv do not currently. Also, the common “python-pip” package that’s found in

various linux distros, does not depend on “python-wheel” currently.

4 Beginning with Python 3.4, venv will create virtualenv environments with pip installed, thereby making it an equal alternative to virtualenv.

However, using virtualenv will still be recommended for users that need cross-version consistency.

25
Python Packaging User Guide

3.1.3 Packaging tool recommendations

• Use setuptools to define projects and create Source Distributions.56

• Use the bdist_wheel setuptools extension available from the wheel project to create wheels. This is espe-
cially beneficial, if your project contains binary extensions.
• Use twine for uploading distributions to PyPI.

3.1.4 Publishing platform migration

The original Python Package Index implementation (previously hosted at pypi.python.org) has been phased out in
favour of an updated implementation hosted at pypi.org.
See Migrating to PyPI.org for more information on the status of the migration, and what settings to change in your
clients.

3.2 Installing packages using pip and virtual environments

This guide discusses how to install packages using pip and a virtual environment manager: either venv for Python 3
or virtualenv for Python 2. These are the lowest-level tools for managing Python packages and are recommended if
higher-level tools do not suit your needs.

Note: This doc uses the term package to refer to a Distribution Package which is different from a Import Package
that which is used to import modules in your Python source code.

3.2.1 Installing pip

pip is the reference Python package manager. It’s used to install and update packages. You’ll need to make sure you
have the latest version of pip installed.

Windows

The Python installers for Windows include pip. You should be able to access pip using:

py -m pip --version
pip 9.0.1 from c:\python36\lib\site-packages (Python 3.6.1)

You can make sure that pip is up-to-date by running:

py -m pip install --upgrade pip

5 Although you can use pure distutils for many projects, it does not support defining dependencies on other projects and is missing several

convenience utilities for automatically populating distribution metadata correctly that are provided by setuptools. Being outside the standard
library, setuptools also offers a more consistent feature set across different versions of Python, and (unlike distutils), recent versions of
setuptools support all of the modern metadata fields described in Core metadata specifications.
Even for projects that do choose to use distutils, when pip installs such projects directly from source (rather than installing from a prebuilt
wheel file), it will actually build your project using setuptools instead.
6 distribute (a fork of setuptools) was merged back into setuptools in June 2013, thereby making setuptools the default choice for packaging.

26 Chapter 3. Guides
 Python Packaging User Guide

Linux and macOS

Debian and most other distributions include a python-pip package, if you want to use the Linux distribution-provided
versions of pip see Installing pip/setuptools/wheel with Linux Package Managers.
You can also install pip yourself to ensure you have the latest version. It’s recommended to use the system pip to
bootstrap a user installation of pip:

python3 -m pip install --user --upgrade pip

Afterwards, you should have the newest pip installed in your user site:

python3 -m pip --version

pip 9.0.1 from $HOME/.local/lib/python3.6/site-packages (python 3.6)

3.2.2 Installing virtualenv

Note: If you are using Python 3.3 or newer, the venv module is the preferred way to create and manage virtual
environments. venv is included in the Python standard library and requires no additional installation. If you are using
venv, you may skip this section.

virtualenv is used to manage Python packages for different projects. Using virtualenv allows you to avoid installing
Python packages globally which could break system tools or other projects. You can install virtualenv using pip.
On macOS and Linux:

python3 -m pip install --user virtualenv

On Windows:

py -m pip install --user virtualenv

3.2.3 Creating a virtual environment

venv (for Python 3) and virtualenv (for Python 2) allow you to manage separate package installations for different
projects. They essentially allow you to create a “virtual” isolated Python installation and install packages into that
virtual installation. When you switch projects, you can simply create a new virtual environment and not have to
worry about breaking the packages installed in the other environments. It is always recommended to use a virtual
environment while developing Python applications.
To create a virtual environment, go to your project’s directory and run venv. If you are using Python 2, replace venv
with virtualenv in the below commands.
On macOS and Linux:

python3 -m venv env

On Windows:

py -m venv env

The second argument is the location to create the virtual environment. Generally, you can just create this in your
project and call it env.

3.2. Installing packages using pip and virtual environments 27

Python Packaging User Guide

venv will create a virtual Python installation in the env folder.

Note: You should exclude your virtual environment directory from your version control system using .gitignore
or similar.

3.2.4 Activating a virtual environment

Before you can start installing or using packages in your virtual environment you’ll need to activate it. Activating a
virtual environment will put the virtual environment-specific python and pip executables into your shell’s PATH.
On macOS and Linux:

source env/bin/activate

On Windows:

.\env\Scripts\activate

You can confirm you’re in the virtual environment by checking the location of your Python interpreter, it should point
to the env directory.
On macOS and Linux:

which python
.../env/bin/python

On Windows:

where python
.../env/bin/python.exe

As long as your virtual environment is activated pip will install packages into that specific environment and you’ll be
able to import and use packages in your Python application.

3.2.5 Leaving the virtual environment

If you want to switch projects or otherwise leave your virtual environment, simply run:

deactivate

If you want to re-enter the virtual environment just follow the same instructions above about activating a virtual
environment. There’s no need to re-create the virtual environment.

3.2.6 Installing packages

Now that you’re in your virtual environment you can install packages. Let’s install the Requests library from the
Python Package Index (PyPI):

pip install requests

pip should download requests and all of its dependencies and install them:

28 Chapter 3. Guides
 Python Packaging User Guide

Collecting requests
Using cached requests-2.18.4-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
Using cached chardet-3.0.4-py2.py3-none-any.whl
Collecting urllib3<1.23,>=1.21.1 (from requests)
Using cached urllib3-1.22-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
Using cached certifi-2017.7.27.1-py2.py3-none-any.whl
Collecting idna<2.7,>=2.5 (from requests)
Using cached idna-2.6-py2.py3-none-any.whl
Installing collected packages: chardet, urllib3, certifi, idna, requests
Successfully installed certifi-2017.7.27.1 chardet-3.0.4 idna-2.6 requests-2.18.4
˓→urllib3-1.22

3.2.7 Installing specific versions

pip allows you to specify which version of a package to install using version specifiers. For example, to install a
specific version of requests:

pip install requests==2.18.4

To install the latest 2.x release of requests:

pip install requests>=2.0.0,<3.0.0

To install pre-release versions of packages, use the --pre flag:

pip install --pre requests

3.2.8 Installing extras

Some packages have optional extras. You can tell pip to install these by specifying the extra in brackets:

pip install requests[security]

3.2.9 Installing from source

pip can install a package directly from source, for example:

cd google-auth
pip install .

Additionally, pip can install packages from source in development mode, meaning that changes to the source directory
will immediately affect the installed package without needing to re-install:

pip install --editable .

3.2.10 Installing from version control systems

pip can install packages directly from their version control system. For example, you can install directly from a git
repository:

3.2. Installing packages using pip and virtual environments 29

Python Packaging User Guide

git+https://github.com/GoogleCloudPlatform/google-auth-library-python.git#egg=google-
˓→auth

For more information on supported version control systems and syntax, see pip’s documentation on VCS Support.

3.2.11 Installing from local archives

If you have a local copy of a Distribution Package’s archive (a zip, wheel, or tar file) you can install it directly with
pip:

pip install requests-2.18.4.tar.gz

If you have a directory containing archives of multiple packages, you can tell pip to look for packages there and not to
use the Python Package Index (PyPI) at all:

pip install --no-index --find-links=/local/dir/ requests

This is useful if you are installing packages on a system with limited connectivity or if you want to strictly control the
origin of distribution packages.

3.2.12 Using other package indexes

If you want to download packages from a different index than the Python Package Index (PyPI), you can use the
--index-url flag:

pip install --index-url http://index.example.com/simple/ SomeProject

If you want to allow packages from both the Python Package Index (PyPI) and a separate index, you can use the
--extra-index-url flag instead:

pip install --extra-index-url http://index.example.com/simple/ SomeProject

3.2.13 Upgrading packages

pip can upgrade packages in-place using the --upgrade flag. For example, to install the latest version of requests
and all of its dependencies:

pip install --upgrade requests

3.2.14 Using requirements files

Instead of installing packages individually, pip allows you to declare all dependencies in a Requirements File. For
example you could create a requirements.txt file containing:

requests==2.18.4
google-auth==1.1.0

And tell pip to install all of the packages in this file using the -r flag:

pip install -r requirements.txt

30 Chapter 3. Guides
 Python Packaging User Guide

3.2.15 Freezing dependencies

Pip can export a list of all installed packages and their versions using the freeze command:

pip freeze

Which will output a list of package specifiers such as:

cachetools==2.0.1
certifi==2017.7.27.1
chardet==3.0.4
google-auth==1.1.1
idna==2.6
pyasn1==0.3.6
pyasn1-modules==0.1.4
requests==2.18.4
rsa==3.4.2
six==1.11.0
urllib3==1.22

This is useful for creating Requirements Files that can re-create the exact versions of all packages installed in an
environment.

3.3 Installing stand alone command line tools

Many packages have command line entry points. Examples of this type of application are mypy, flake8, pipenv,and
black.
Usually you want to be able to access these from anywhere, but installing packages and their dependencies to the same
global environment can cause version conflicts and break dependencies the operating system has on Python packages.
pipx solves this by creating a virtual environment for each package, while also ensuring that package’s applications are
accessible through a directory that is on your $PATH. This allows each package to be upgraded or uninstalled without
causing conflicts with other packages, and allows you to safely run the program from anywhere.

Note: pipx only works with Python 3.6+.

pipx is installed with pip:

$ python3 -m pip install --user pipx

$ python3 -m pipx ensurepath # ensures the path of the CLI application directory is
˓→on your $PATH

Note: You may need to restart your terminal for the path updates to take effect.

Now you can install packages with pipx install and access the package’s entry point(s) from anywhere.

$ pipx install PACKAGE

$ ENTRYPOINT_OF_PACKAGE [ARGS]

For example

3.3. Installing stand alone command line tools 31

Python Packaging User Guide

$ pipx install cowsay

installed package cowsay 2.0, Python 3.6.2+
These binaries are now globally available
- cowsay
done!
$ cowsay moo
___
< moo >
===
\
\
^__^
(oo)\_______
(__)\ )\/ ||----w |
|| ||

To see a list of packages installed with pipx and which CLI applications are available, use pipx list.

$ pipx list
venvs are in /Users/user/.local/pipx/venvs
symlinks to binaries are in /Users/user/.local/bin
package black 18.9b0, Python 3.6.2+
- black
- blackd
package cowsay 2.0, Python 3.6.2+
- cowsay
package mypy 0.660, Python 3.6.2+
- dmypy
- mypy
- stubgen
package nox 2018.10.17, Python 3.6.2+
- nox
- tox-to-nox

To upgrade or uninstall the package

$ pipx upgrade PACKAGE

$ pipx uninstall PACKAGE

pipx can be upgraded or uninstalled with pip

$ pip install -U pipx

$ pip uninstall pipx

pipx also allows you to install and run the latest version of a cli tool in a temporary, ephemeral environment.

$ pipx run PACKAGE [ARGS]

For example

$ pipx run cowsay moooo

To see the full list of commands pipx offers, run

$ pipx --help

You can learn more about pipx at its homepage, https://github.com/pipxproject/pipx.

32 Chapter 3. Guides
 Python Packaging User Guide

3.4 Installing pip/setuptools/wheel with Linux Package Managers

Page Status Incomplete

Last Reviewed 2015-09-17
This section covers how to install pip, setuptools, and wheel using Linux package managers.
If you’re using a Python that was downloaded from python.org, then this section does not apply. See the Requirements
for Installing Packages section instead.
Note that it’s common for the versions of pip, setuptools, and wheel supported by a specific Linux Distribution to be
outdated by the time it’s released to the public, and updates generally only occur for security reasons, not for feature
updates. For certain Distributions, there are additional repositories that can be enabled to provide newer versions. The
repositories we know about are explained below.
Also note that it’s somewhat common for Distributions to apply patches for the sake of security and normalization to
their own standards. In some cases, this can lead to bugs or unexpected behaviors that vary from the original unpatched
versions. When this is known, we will make note of it below.

3.4.1 Fedora

• Fedora 21:
– Python 2:

sudo yum upgrade python-setuptools

sudo yum install python-pip python-wheel

– Python 3: sudo yum install python3 python3-wheel

• Fedora 22:
– Python 2:

sudo dnf upgrade python-setuptools

sudo dnf install python-pip python-wheel

– Python 3: sudo dnf install python3 python3-wheel

To get newer versions of pip, setuptools, and wheel for Python 2, you can enable the PyPA Copr Repo using the Copr
Repo instructions, and then run:

sudo yum|dnf upgrade python-setuptools

sudo yum|dnf install python-pip python-wheel

3.4.2 CentOS/RHEL

CentOS and RHEL don’t offer pip or wheel in their core repositories, although setuptools is installed by default.
To install pip and wheel for the system Python, there are two options:
1. Enable the EPEL repository using these instructions. On EPEL 6 and EPEL7, you can install pip like so:

sudo yum install python-pip

On EPEL 7 (but not EPEL 6), you can install wheel like so:

3.4. Installing pip/setuptools/wheel with Linux Package Managers 33

Python Packaging User Guide

sudo yum install python-wheel

Since EPEL only offers extra, non-conflicting packages, EPEL does not offer setuptools, since it’s in the core
repository.
2. Enable the PyPA Copr Repo using these instructions1 . You can install pip and wheel like so:

sudo yum install python-pip python-wheel

To additionally upgrade setuptools, run:

sudo yum upgrade python-setuptools

To install pip, wheel, and setuptools, in a parallel, non-system environment (using yum) then there are two options:
1. Use the “Sofware Collections” feature to enable a parallel collection that includes pip, setuptools, and wheel.
• For Redhat, see here: http://developers.redhat.com/products/softwarecollections/overview/
• For CentOS, see here: https://www.softwarecollections.org/en/
Be aware that collections may not contain the most recent versions.
2. Enable the IUS repository and install one of the parallel-installable Pythons, along with pip, setuptools, and
wheel, which are kept fairly up to date.
For example, for Python 3.4 on CentOS7/RHEL7:

sudo yum install python34u python34u-wheel

3.4.3 openSUSE

• Python 2:

sudo zypper install python-pip python-setuptools python-wheel

• Python 3:

sudo zypper install python3-pip python3-setuptools python3-wheel

3.4.4 Debian/Ubuntu

• Python 2:

sudo apt install python-pip

• Python 3:

sudo apt install python3-venv python3-pip

Warning: Recent Debian/Ubuntu versions have modified pip to use the “User Scheme” by default, which is a
significant behavior change that can be surprising to some users.

1 Currently, there is no “copr” yum plugin available for CentOS/RHEL, so the only option is to manually place the repo files as described.

34 Chapter 3. Guides
 Python Packaging User Guide

3.4.5 Arch Linux

• Python 2:
sudo pacman -S python2-pip

• Python 3:
sudo pacman -S python-pip

3.5 Installing scientific packages

Contents

• Building from source

• Linux distribution packages
• Windows installers
• macOS installers and package managers
• SciPy distributions
• Spack
• The conda cross-platform package manager

Scientific software tends to have more complex dependencies than most, and it will often have multiple build options
to take advantage of different kinds of hardware, or to interoperate with different pieces of external software.
In particular, NumPy, which provides the basis for most of the software in the scientific Python stack can be configured
to interoperate with different FORTRAN libraries, and can take advantage of different levels of vectorised instructions
available in modern CPUs.
Starting with version 1.10.4 of NumPy and version 1.0.0 of SciPy, pre-built 32-bit and 64-bit binaries in the wheel
format are available for all major operating systems (Windows, macOS, and Linux) on PyPI. Note, however, that
on Windows, NumPy binaries are linked against the ATLAS <http://www.netlib.org/atlas/> BLAS/LAPACK library,
restricted to SSE2 instructions, so they may not provide optimal linear algebra performance.
There are a number of alternative options for obtaining scientific Python libraries (or any other Python libraries that
require a compilation environment to install from source and don’t provide pre-built wheel files on PyPI).

3.5.1 Building from source

The same complexity which makes it difficult to distribute NumPy (and many of the projects that depend on it) as
wheel files also make them difficult to build from source yourself. However, for intrepid folks that are willing to spend
the time wrangling compilers and linkers for both C and FORTRAN, building from source is always an option.

3.5.2 Linux distribution packages

For Linux users, the system package manager will often have pre-compiled versions of various pieces of scientific
software, including NumPy and other parts of the scientific Python stack.

3.5. Installing scientific packages 35

Python Packaging User Guide

If using versions which may be several months old is acceptable, then this is likely to be a good option (just make sure
to allow access to distributions installed into the system Python when using virtual environments).

3.5.3 Windows installers

Many Python projects that don’t (or can’t) currently publish wheel files at least publish Windows installers, either
on PyPI or on their project download page. Using these installers allows users to avoid the need to set up a suitable
environment to build extensions locally.
The extensions provided in these installers are typically compatible with the CPython Windows installers published
on python.org.
For projects which don’t provide their own Windows installers (and even some which do), Christoph Gohlke at the
University of California provides a collection of Windows installers. Many Python users on Windows have reported a
positive experience with these prebuilt versions.
As with Linux system packages, the Windows installers will only install into a system Python installation - they do not
support installation in virtual environments. Allowing access to distributions installed into the system Python when
using virtual environments is a common approach to working around this limitation.
The wheel project also provides a wheel convert subcommand that can convert a Windows bdist_wininst installer to a
wheel.

3.5.4 macOS installers and package managers

Similar to the situation on Windows, many projects (including NumPy) publish macOS installers that are compatible
with the macOS CPython binaries published on python.org.
macOS users also have access to Linux distribution style package managers such as MacPorts. The SciPy site has
more details on using MacPorts to install the scientific Python stack

3.5.5 SciPy distributions

The SciPy site lists several distributions that provide the full SciPy stack to end users in an easy to use and update
format.
Some of these distributions may not be compatible with the standard pip and virtualenv based toolchain.

3.5.6 Spack

Spack is a flexible package manager designed to support multiple versions, configurations, platforms, and compilers.
It was built to support the needs of large supercomputing centers and scientific application teams, who must often build
software many different ways. Spack is not limited to Python; it can install packages for C, C++, Fortran, R, and
other languages. It is non-destructive; installing a new version of one package does not break existing installations, so
many configurations can coexist on the same system.
Spack offers a simple but powerful syntax that allows users to specify versions and configuration options concisely.
Package files are written in pure Python, and they are templated so that it is easy to swap compilers, dependency
implementations (like MPI), versions, and build options with a single package file. Spack also generates module files
so that packages can be loaded and unloaded from the user’s environment.

36 Chapter 3. Guides
 Python Packaging User Guide

3.5.7 The conda cross-platform package manager

Anaconda is a Python distribution published by Anaconda, Inc. It is a stable collection of Open Source packages for
big data and scientific use. As of the 5.0 release of Anaconda, about 200 packages are installed by default, and a total
of 400-500 can be installed and updated from the Anaconda repository.
conda is an open source (BSD licensed) package management system and environment management system included
in Anaconda that allows users to install multiple versions of binary software packages and their dependencies, and
easily switch between them. It is a cross-platform tool working on Windows, macOS, and Linux. Conda can be
used to package up and distribute all kinds of packages, it is not limited to just Python packages. It has full support
for native virtual environments. Conda makes environments first-class citizens, making it easy to create independent
environments even for C libraries. It is written in Python, but is Python-agnostic. Conda manages Python itself as a
package, so that conda update python is possible, in contrast to pip, which only manages Python packages. Conda is
available in Anaconda and Miniconda (an easy-to-install download with just Python and conda).

3.6 Multi-version installs

easy_install allows simultaneous installation of different versions of the same project into a single environment
shared by multiple programs which must require the appropriate version of the project at run time (using
pkg_resources).
For many use cases, virtual environments address this need without the complication of the require directive.
However, the advantage of parallel installations within the same environment is that it works for an environment
shared by multiple applications, such as the system Python in a Linux distribution.
The major limitation of pkg_resources based parallel installation is that as soon as you import pkg_resources
it locks in the default version of everything which is already available on sys.path. This can cause problems, since
setuptools created command line scripts use pkg_resources to find the entry point to execute. This means
that, for example, you can’t use require tests invoked through nose or a WSGI application invoked through
gunicorn if your application needs a non-default version of anything that is available on the standard sys.path
- the script wrapper for the main application will lock in the version that is available by default, so the subsequent
require call in your own code fails with a spurious version conflict.
This can be worked around by setting all dependencies in __main__.__requires__ before importing
pkg_resources for the first time, but that approach does mean that standard command line invocations of the
affected tools can’t be used - it’s necessary to write a custom wrapper script or use python -c '<commmand>'
to invoke the application’s main entry point directly.
Refer to the pkg_resources documentation for more details.

3.7 Packaging and distributing projects

This section covers the basics of how to configure, package and distribute your own Python projects. It assumes that
you are already familiar with the contents of the Installing Packages page.
The section does not aim to cover best practices for Python project development as a whole. For example, it does not
provide guidance or tool recommendations for version control, documentation, or testing.
For more reference material, see Building and Distributing Packages in the setuptools docs, but note that some advisory
content there may be outdated. In the event of conflicts, prefer the advice in the Python Packaging User Guide.

3.6. Multi-version installs 37

Python Packaging User Guide

Contents

• Requirements for packaging and distributing

• Configuring your project
– Initial files

* setup.py
* setup.cfg
* README.rst / README.md
* MANIFEST.in
* LICENSE.txt
* <your package>
– setup() args

* name
* version
* description
* url
* author
* license
* classifiers
* keywords
* project_urls
* packages
* py_modules
* install_requires
* python_requires
* package_data
* data_files
* scripts
* entry_points
· console_scripts
– Choosing a versioning scheme

* Standards compliance for interoperability

* Scheme choices
· Semantic versioning (preferred)
· Date based versioning
· Serial versioning

38 Chapter 3. Guides
 Python Packaging User Guide

· Hybrid schemes

* Pre-release versioning
* Local version identifiers
• Working in “development mode”
• Packaging your project
– Source distributions
– Wheels

* Universal Wheels
* Pure Python Wheels
* Platform Wheels
• Uploading your Project to PyPI
– Create an account
– Upload your distributions

3.7.1 Requirements for packaging and distributing

1. First, make sure you have already fulfilled the requirements for installing packages.
2. Install “twine”1 :

pip install twine

You’ll need this to upload your project distributions to PyPI (see below).

3.7.2 Configuring your project

Initial files

setup.py

The most important file is setup.py which exists at the root of your project directory. For an example, see the
setup.py in the PyPA sample project.
setup.py serves two primary functions:
1. It’s the file where various aspects of your project are configured. The primary feature of setup.py is that it
contains a global setup() function. The keyword arguments to this function are how specific details of your
project are defined. The most relevant arguments are explained in the section below.
2. It’s the command line interface for running various commands that relate to packaging tasks. To get a listing of
available commands, run python setup.py --help-commands.
1 Depending on your platform, this may require root or Administrator access. pip is currently considering changing this by making user installs
the default behavior.

3.7. Packaging and distributing projects 39

Python Packaging User Guide

setup.cfg

setup.cfg is an ini file that contains option defaults for setup.py commands. For an example, see the setup.cfg
in the PyPA sample project.

README.rst / README.md

All projects should contain a readme file that covers the goal of the project. The most common format is reStructured-
Text with an “rst” extension, although this is not a requirement; multiple variants of Markdown are supported as well
(look at setup()’s long_description_content_type argument).
For an example, see README.md from the PyPA sample project.

Note: Projects using setuptools 0.6.27+ have standard readme files (README.rst, README.txt, or README)
included in source distributions by default. The built-in distutils library adopts this behavior beginning in Python 3.7.
Additionally, setuptools 36.4.0+ will include a README.md if found. If you are using setuptools, you don’t need to
list your readme file in MANIFEST.in. Otherwise, include it to be explicit.

MANIFEST.in

A MANIFEST.in is needed when you need to package additional files that are not automatically included in a source
distribution. For details on writing a MANIFEST.in file, including a list of what’s included by default, see “Including
files in source distributions with MANIFEST.in”.
For an example, see the MANIFEST.in from the PyPA sample project.

Note: MANIFEST.in does not affect binary distributions such as wheels.

LICENSE.txt

Every package should include a license file detailing the terms of distribution. In many jurisdictions, packages without
an explicit license can not be legally used or distributed by anyone other than the copyright holder. If you’re unsure
which license to choose, you can use resources such as GitHub’s Choose a License or consult a lawyer.
For an example, see the LICENSE.txt from the PyPA sample project.

<your package>

Although it’s not required, the most common practice is to include your Python modules and packages under a single
top-level package that has the same name as your project, or something very close.
For an example, see the sample package that’s included in the PyPA sample project.

setup() args

As mentioned above, the primary feature of setup.py is that it contains a global setup() function. The keyword
arguments to this function are how specific details of your project are defined.

40 Chapter 3. Guides
 Python Packaging User Guide

The most relevant arguments are explained below. Most of the snippets given are taken from the setup.py contained in
the PyPA sample project.

name

name='sample',

This is the name of your project, determining how your project is listed on PyPI. Per PEP 508, valid project names
must:
• Consist only of ASCII letters, digits, underscores (_), hyphens (-), and/or periods (.), and
• Start & end with an ASCII letter or digit.
Comparison of project names is case insensitive and treats arbitrarily-long runs of underscores, hyphens, and/or periods
as equal. For example, if you register a project named cool-stuff, users will be able to download it or declare a
dependency on it using any of the following spellings:

Cool-Stuff
cool.stuff
COOL_STUFF
CoOl__-.-__sTuFF

version

version='1.2.0',

This is the current version of your project, allowing your users to determine whether or not they have the latest version,
and to indicate which specific versions they’ve tested their own software against.
Versions are displayed on PyPI for each release if you publish your project.
See Choosing a versioning scheme for more information on ways to use versions to convey compatibility information
to your users.
If the project code itself needs run-time access to the version, the simplest way is to keep the version in both setup.
py and your code. If you’d rather not duplicate the value, there are a few ways to manage this. See the “Single-sourcing
the package version” Advanced Topics section.

description

description='A sample Python project',

long_description=long_description,
long_description_content_type='text/x-rst',

Give a short and long description for your project.

These values will be displayed on PyPI if you publish your project. On pypi.org, the user interface displays
description in the grey banner and long_description in the section named “Project Description”.
description is also displayed in lists of projects. For example, it’s visible in the search results pages such as
https://pypi.org/search/?q=jupyter, the front-page lists of trending projects and new releases, and the list of projects
you maintain within your account profile (such as https://pypi.org/user/jaraco/).

3.7. Packaging and distributing projects 41

Python Packaging User Guide

A content type can be specified with the long_description_content_type argument, which can be one of
text/plain, text/x-rst, or text/markdown, corresponding to no formatting, reStructuredText (reST), and
the Github-flavored Markdown dialect of Markdown respectively.

url

url='https://github.com/pypa/sampleproject',

Give a homepage URL for your project.

author

author='A. Random Developer',

author_email='[email protected]',

Provide details about the author.

license

license='MIT',

The license argument doesn’t have to indicate the license under which your package is being released, although
you may optionally do so if you want. If you’re using a standard, well-known license, then your main indication can
and should be via the classifiers argument. Classifiers exist for all major open-source licenses.
The “license” argument is more typically used to indicate differences from well-known licenses, or to include your
own, unique license. As a general rule, it’s a good idea to use a standard, well-known license, both to avoid confusion
and because some organizations avoid software whose license is unapproved.
See “Classifier” for some examples of values for license.

classifiers

classifiers=[
# How mature is this project? Common values are
# 3 - Alpha
# 4 - Beta
# 5 - Production/Stable
'Development Status :: 3 - Alpha',

# Indicate who your project is intended for

'Intended Audience :: Developers',
'Topic :: Software Development :: Build Tools',

# Pick your license as you wish (should match "license" above)

'License :: OSI Approved :: MIT License',

# Specify the Python versions you support here. In particular, ensure

# that you indicate whether you support Python 2, Python 3 or both.
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 2.6',
(continues on next page)

42 Chapter 3. Guides
 Python Packaging User Guide

(continued from previous page)

'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.2',
'Programming Language :: Python :: 3.3',
'Programming Language :: Python :: 3.4',
],

Provide a list of classifiers that categorize your project. For a full listing, see https://pypi.org/classifiers/.
Although the list of classifiers is often used to declare what Python versions a project supports, this information is only
used for searching & browsing projects on PyPI, not for installing projects. To actually restrict what Python versions
a project can be installed on, use the python_requires argument.

keywords

keywords='sample setuptools development',

List keywords that describe your project.

project_urls

project_urls={
'Documentation': 'https://packaging.python.org/tutorials/distributing-packages/',
'Funding': 'https://donate.pypi.org',
'Say Thanks!': 'http://saythanks.io/to/example',
'Source': 'https://github.com/pypa/sampleproject/',
'Tracker': 'https://github.com/pypa/sampleproject/issues',
},

List additional relevant URLs about your project. This is the place to link to bug trackers, source repositories, or where
to support package development. The string of the key is the exact text that will be displayed on PyPI.

packages

packages=find_packages(include=['sample', 'sample.*']),

Set packages to a list of all packages in your project, including their subpackages, sub-subpackages, etc. Al-
though the packages can be listed manually, setuptools.find_packages() finds them automatically. Use the
include keyword argument to find only the given packages. Use the exclude keyword argument to omit packages
that are not intended to be released and installed.

py_modules

py_modules=["six"],

If your project contains any single-file Python modules that aren’t part of a package, set py_modules to a list of the
names of the modules (minus the .py extension) in order to make setuptools aware of them.

3.7. Packaging and distributing projects 43

Python Packaging User Guide

install_requires

install_requires=['peppercorn'],

“install_requires” should be used to specify what dependencies a project minimally needs to run. When the project is
installed by pip, this is the specification that is used to install its dependencies.
For more on using “install_requires” see install_requires vs requirements files.

python_requires

If your project only runs on certain Python versions, setting the python_requires argument to the appropriate
PEP 440 version specifier string will prevent pip from installing the project on other Python versions. For example, if
your package is for Python 3+ only, write:

python_requires='>=3',

If your package is for Python 3.3 and up but you’re not willing to commit to Python 4 support yet, write:

python_requires='~=3.3',

If your package is for Python 2.6, 2.7, and all versions of Python 3 starting with 3.3, write:

python_requires='>=2.6, !=3.0.*, !=3.1.*, !=3.2.*, <4',

And so on.

Note: Support for this feature is relatively recent. Your project’s source distributions and wheels (see Packaging your
project) must be built using at least version 24.2.0 of setuptools in order for the python_requires argument to be
recognized and the appropriate metadata generated.
In addition, only versions 9.0.0 and higher of pip recognize the python_requires metadata. Users with ear-
lier versions of pip will be able to download & install projects on any Python version regardless of the projects’
python_requires values.

package_data

package_data={
'sample': ['package_data.dat'],
},

Often, additional files need to be installed into a package. These files are often data that’s closely related to the
package’s implementation, or text files containing documentation that might be of interest to programmers using the
package. These files are called “package data”.
The value must be a mapping from package name to a list of relative path names that should be copied into the package.
The paths are interpreted as relative to the directory containing the package.
For more information, see Including Data Files from the setuptools docs.

44 Chapter 3. Guides
 Python Packaging User Guide

data_files

data_files=[('my_data', ['data/data_file'])],

Although configuring package_data is sufficient for most needs, in some cases you may need to place data files outside
of your packages. The data_files directive allows you to do that. It is mostly useful if you need to install files
which are used by other programs, which may be unaware of Python packages.
Each (directory, files) pair in the sequence specifies the installation directory and the files to install there.
The directory must be a relative path (although this may change in the future, see wheel Issue #92). and it is
interpreted relative to the installation prefix (Python’s sys.prefix for a default installation; site.USER_BASE
for a user installation). Each file name in files is interpreted relative to the setup.py script at the top of the
project source distribution.
For more information see the distutils section on Installing Additional Files.

Note: When installing packages as egg, data_files is not supported. So, if your project uses setuptools,
you must use pip to install it. Alternatively, if you must use python setup.py, then you need to pass the
--old-and-unmanageable option.

scripts

Although setup() supports a scripts keyword for pointing to pre-made scripts to install, the recommended approach
to achieve cross-platform compatibility is to use console_scripts entry points (see below).

entry_points

entry_points={
...
},

Use this keyword to specify any plugins that your project provides for any named entry points that may be defined by
your project or others that you depend on.
For more information, see the section on Dynamic Discovery of Services and Plugins from the setuptools docs.
The most commonly used entry point is “console_scripts” (see below).

console_scripts

entry_points={
'console_scripts': [
'sample=sample:main',
],
},

Use “console_script” entry points to register your script interfaces. You can then let the toolchain handle the work of
turning these interfaces into actual scripts2 . The scripts will be generated during the install of your distribution.
For more information, see Automatic Script Creation from the setuptools docs.
2 Specifically, the “console_script” approach generates .exe files on Windows, which are necessary because the OS special-cases .exe files.
Script-execution features like PATHEXT and the Python Launcher for Windows allow scripts to be used in many cases, but not all.

3.7. Packaging and distributing projects 45

Python Packaging User Guide

Choosing a versioning scheme

Standards compliance for interoperability

Different Python projects may use different versioning schemes based on the needs of that particular project, but all of
them are required to comply with the flexible public version scheme specified in PEP 440 in order to be supported in
tools and libraries like pip and setuptools.
Here are some examples of compliant version numbers:

1.2.0.dev1 # Development release

1.2.0a1 # Alpha Release
1.2.0b1 # Beta Release
1.2.0rc1 # Release Candidate
1.2.0 # Final Release
1.2.0.post1 # Post Release
15.10 # Date based release
23 # Serial release

To further accommodate historical variations in approaches to version numbering, PEP 440 also defines a comprehen-
sive technique for version normalisation that maps variant spellings of different version numbers to a standardised
canonical form.

Scheme choices

Semantic versioning (preferred)

For new projects, the recommended versioning scheme is based on Semantic Versioning, but adopts a different ap-
proach to handling pre-releases and build metadata.
The essence of semantic versioning is a 3-part MAJOR.MINOR.MAINTENANCE numbering scheme, where the
project author increments:
1. MAJOR version when they make incompatible API changes,
2. MINOR version when they add functionality in a backwards-compatible manner, and
3. MAINTENANCE version when they make backwards-compatible bug fixes.
Adopting this approach as a project author allows users to make use of “compatible release” specifiers, where name
~= X.Y requires at least release X.Y, but also allows any later release with a matching MAJOR version.
Python projects adopting semantic versioning should abide by clauses 1-8 of the Semantic Versioning 2.0.0 specifica-
tion.

Date based versioning

Semantic versioning is not a suitable choice for all projects, such as those with a regular time based release cadence
and a deprecation process that provides warnings for a number of releases prior to removal of a feature.
A key advantage of date based versioning is that it is straightforward to tell how old the base feature set of a particular
release is given just the version number.
Version numbers for date based projects typically take the form of YEAR.MONTH (for example, 12.04, 15.10).

46 Chapter 3. Guides
 Python Packaging User Guide

Serial versioning

This is the simplest possible versioning scheme, and consists of a single number which is incremented every release.
While serial versioning is very easy to manage as a developer, it is the hardest to track as an end user, as serial version
numbers convey little or no information regarding API backwards compatibility.

Hybrid schemes

Combinations of the above schemes are possible. For example, a project may combine date based versioning with
serial versioning to create a YEAR.SERIAL numbering scheme that readily conveys the approximate age of a release,
but doesn’t otherwise commit to a particular release cadence within the year.

Pre-release versioning

Regardless of the base versioning scheme, pre-releases for a given final release may be published as:
• zero or more dev releases (denoted with a “.devN” suffix)
• zero or more alpha releases (denoted with a “.aN” suffix)
• zero or more beta releases (denoted with a “.bN” suffix)
• zero or more release candidates (denoted with a “.rcN” suffix)
pip and other modern Python package installers ignore pre-releases by default when deciding which versions of
dependencies to install.

Local version identifiers

Public version identifiers are designed to support distribution via PyPI. Python’s software distribution tools also sup-
port the notion of a local version identifier, which can be used to identify local development builds not intended for
publication, or modified variants of a release maintained by a redistributor.
A local version identifier takes the form <public version identifier>+<local version label>.
For example:

1.2.0.dev1+hg.5.b11e5e6f0b0b # 5th VCS commmit since 1.2.0.dev1 release

1.2.1+fedora.4 # Package with downstream Fedora patches applied

3.7.3 Working in “development mode”

Although not required, it’s common to locally install your project in “editable” or “develop” mode while you’re
working on it. This allows your project to be both installed and editable in project form.
Assuming you’re in the root of your project directory, then run:

pip install -e .

Although somewhat cryptic, -e is short for --editable, and . refers to the current working directory, so together,
it means to install the current directory (i.e. your project) in editable mode. This will also install any dependencies
declared with “install_requires” and any scripts declared with “console_scripts”. Dependencies will be installed in the
usual, non-editable mode.

3.7. Packaging and distributing projects 47

Python Packaging User Guide

It’s fairly common to also want to install some of your dependencies in editable mode as well. For example, supposing
your project requires “foo” and “bar”, but you want “bar” installed from VCS in editable mode, then you could
construct a requirements file like so:

-e .
-e git+https://somerepo/bar.git#egg=bar

The first line says to install your project and any dependencies. The second line overrides the “bar” dependency, such
that it’s fulfilled from VCS, not PyPI.
If, however, you want “bar” installed from a local directory in editable mode, the requirements file should look like
this, with the local paths at the top of the file:

-e /path/to/project/bar
-e .

Otherwise, the dependency will be fulfilled from PyPI, due to the installation order of the requirements file. For more
on requirements files, see the Requirements File section in the pip docs. For more on VCS installs, see the VCS
Support section of the pip docs.
Lastly, if you don’t want to install any dependencies at all, you can run:

pip install -e . --no-deps

For more information, see the Development Mode section of the setuptools docs.

3.7.4 Packaging your project

To have your project installable from a Package Index like PyPI, you’ll need to create a Distribution (aka “Package”)
for your project.

Source distributions

Minimally, you should create a Source Distribution:

python setup.py sdist

A “source distribution” is unbuilt (i.e. it’s not a Built Distribution), and requires a build step when installed by pip.
Even if the distribution is pure Python (i.e. contains no extensions), it still involves a build step to build out the
installation metadata from setup.py.

Wheels

You should also create a wheel for your project. A wheel is a built package that can be installed without needing to
go through the “build” process. Installing wheels is substantially faster for the end user than installing from a source
distribution.
If your project is pure Python (i.e. contains no compiled extensions) and natively supports both Python 2 and 3, then
you’ll be creating what’s called a *Universal Wheel* (see section below).
If your project is pure Python but does not natively support both Python 2 and 3, then you’ll be creating a “Pure
Python Wheel” (see section below).
If your project contains compiled extensions, then you’ll be creating what’s called a *Platform Wheel* (see section
below).

48 Chapter 3. Guides
 Python Packaging User Guide

Before you can build wheels for your project, you’ll need to install the wheel package:

pip install wheel

Universal Wheels

Universal Wheels are wheels that are pure Python (i.e. contain no compiled extensions) and support Python 2 and 3.
This is a wheel that can be installed anywhere by pip.
To build the wheel:

python setup.py bdist_wheel --universal

You can also permanently set the --universal flag in setup.cfg:

[bdist_wheel]
universal=1

Only use the --universal setting, if:

1. Your project runs on Python 2 and 3 with no changes (i.e. it does not require 2to3).
2. Your project does not have any C extensions.
Beware that bdist_wheel does not currently have any checks to warn if you use the setting inappropriately.
If your project has optional C extensions, it is recommended not to publish a universal wheel, because pip will prefer
the wheel over a source installation, and prevent the possibility of building the extension.

Pure Python Wheels

Pure Python Wheels that are not “universal” are wheels that are pure Python (i.e. contain no compiled extensions), but
don’t natively support both Python 2 and 3.
To build the wheel:

python setup.py bdist_wheel

bdist_wheel will detect that the code is pure Python, and build a wheel that’s named such that it’s usable on any Python
installation with the same major version (Python 2 or Python 3) as the version you used to build the wheel. For details
on the naming of wheel files, see PEP 425.
If your code supports both Python 2 and 3, but with different code (e.g., you use “2to3”) you can run setup.py
bdist_wheel twice, once with Python 2 and once with Python 3. This will produce wheels for each version.

Platform Wheels

Platform Wheels are wheels that are specific to a certain platform like Linux, macOS, or Windows, usually due to
containing compiled extensions.
To build the wheel:

python setup.py bdist_wheel

3.7. Packaging and distributing projects 49

Python Packaging User Guide

bdist_wheel will detect that the code is not pure Python, and build a wheel that’s named such that it’s only usable on
the platform that it was built on. For details on the naming of wheel files, see PEP 425.

Note: PyPI currently supports uploads of platform wheels for Windows, macOS, and the multi-distro manylinux1
ABI. Details of the latter are defined in PEP 513.

3.7.5 Uploading your Project to PyPI

When you ran the command to create your distribution, a new directory dist/ was created under your project’s root
directory. That’s where you’ll find your distribution file(s) to upload.

Note: These files are only created when you run the command to create your distribution. This means that any time
you change the source of your project or the configuration in your setup.py file, you will need to rebuild these files
again before you can distribute the changes to PyPI.

Note: Before releasing on main PyPI repo, you might prefer training with the PyPI test site which is cleaned on a
semi regular basis. See Using TestPyPI on how to setup your configuration in order to use it.

Warning: In other resources you may encounter references to using python setup.py register and
python setup.py upload. These methods of registering and uploading a package are strongly discour-
aged as it may use a plaintext HTTP or unverified HTTPS connection on some Python versions, allowing your
username and password to be intercepted during transmission.

Tip: The reStructuredText parser used on PyPI is not Sphinx! Furthermore, to ensure safety of all users, certain kinds
of URLs and directives are forbidden or stripped out (e.g., the .. raw:: directive). Before trying to upload your
distribution, you should check to see if your brief / long descriptions provided in setup.py are valid. You can do
this by following the instructions for the pypa/readme_renderer tool.

Create an account

First, you need a PyPI user account. You can create an account using the form on the PyPI website.
Now you’ll create a PyPI API token so you will be able to securely upload your project.
Go to https://pypi.org/manage/account/#api-tokens and create a new API token; don’t limit its scope to a particular
project, since you are creating a new project.
Don’t close the page until you have copied and saved the token — you won’t see that token again.

Note: To avoid having to copy and paste the token every time you upload, you can create a $HOME/.pypirc file:

[pypi]
username = __token__
password = <the token value, including the `pypi-` prefix>

50 Chapter 3. Guides
 Python Packaging User Guide

Be aware that this stores your token in plaintext.

For more details, see the specification for .pypirc.

Upload your distributions

Once you have an account you can upload your distributions to PyPI using twine.
The process for uploading a release is the same regardless of whether or not the project already exists on PyPI - if it
doesn’t exist yet, it will be automatically created when the first release is uploaded.
For the second and subsequent releases, PyPI only requires that the version number of the new release differ from any
previous releases.

twine upload dist/*

You can see if your package has successfully uploaded by navigating to the URL https://pypi.org/project/
<sampleproject> where sampleproject is the name of your project that you uploaded. It may take a minute
or two for your project to appear on the site.

3.8 Including files in source distributions with MANIFEST.in

When building a source distribution for your package, by default only a minimal set of files are included. You may
find yourself wanting to include extra files in the source distribution, such as an authors/contributors file, a docs/
directory, or a directory of data files used for testing purposes. There may even be extra files that you need to include;
for example, if your setup.py computes your project’s long_description by reading from both a README
and a changelog file, you’ll need to include both those files in the sdist so that people that build or install from the sdist
get the correct results.
Adding & removing files to & from the source distribution is done by writing a MANIFEST.in file at the project root.

3.8.1 How files are included in an sdist

The following files are included in a source distribution by default:

• all Python source files implied by the py_modules and packages setup() arguments
• all C source files mentioned in the ext_modules or libraries setup() arguments
• scripts specified by the scripts setup() argument
• all files specified by the package_data and data_files setup() arguments
• the file specified by the license_file option in setup.cfg (setuptools 40.8.0+)
• all files specified by the license_files option in setup.cfg (setuptools 42.0.0+)
• all files matching the pattern test/test*.py
• setup.py (or whatever you called your setup script)
• setup.cfg
• README
• README.txt

3.8. Including files in source distributions with MANIFEST.in 51

Python Packaging User Guide

• README.rst (Python 3.7+ or setuptools 0.6.27+)

• README.md (setuptools 36.4.0+)
• pyproject.toml (setuptools 43.0.0+)
• MANIFEST.in
After adding the above files to the sdist, the commands in MANIFEST.in (if such a file exists) are executed in order to
add and remove further files to & from the sdist. Default files can even be removed from the sdist with the appropriate
MANIFEST.in command.
After processing the MANIFEST.in file, setuptools removes the build/ directory as well as any directories named
RCS, CVS, or .svn from the sdist, and it adds a PKG-INFO file and an *.egg-info directory. This behavior
cannot be changed with MANIFEST.in.

3.8.2 MANIFEST.in commands

A MANIFEST.in file consists of commands, one per line, instructing setuptools to add or remove some set of files
from the sdist. The commands are:

Command Description
include pat1 pat2 ... Include all files matching any of the listed patterns
exclude pat1 pat2 ... Exclude all files matching any of the listed patterns
recursive-include dir-pattern Include all files under directories matching dir-pattern that
pat1 pat2 ... match any of the listed patterns
recursive-exclude dir-pattern Exclude all files under directories matching dir-pattern that
pat1 pat2 ... match any of the listed patterns
global-include pat1 pat2 ... Include all files anywhere in the source tree matching any of the
listed patterns
global-exclude pat1 pat2 ... Exclude all files anywhere in the source tree matching any of the
listed patterns
graft dir-pattern Include all files under directories matching dir-pattern
prune dir-pattern Exclude all files under directories matching dir-pattern

The patterns here are glob-style patterns: * matches zero or more regular filename characters (on Unix, everything
except forward slash; on Windows, everything except backslash and colon); ? matches a single regular filename
character, and [chars] matches any one of the characters between the square brackets (which may contain character
ranges, e.g., [a-z] or [a-fA-F0-9]). Setuptools also has undocumented support for ** matching zero or more
characters including forward slash, backslash, and colon.
Directory patterns are relative to the root of the project directory; e.g., graft example* will include a directory
named examples in the project root but will not include docs/examples/.
File & directory names in MANIFEST.in should be /-separated; setuptools will automatically convert the slashes to
the local platform’s appropriate directory separator.
Commands are processed in the order they appear in the MANIFEST.in file. For example, given the commands:

graft tests
global-exclude *.py[cod]

the contents of the directory tree tests will first be added to the sdist, and then after that all files in the sdist with a
.pyc, .pyo, or .pyd extension will be removed from the sdist. If the commands were in the opposite order, then
*.pyc files etc. would be only be removed from what was already in the sdist before adding tests, and if tests
happened to contain any *.pyc files, they would end up included in the sdist because the exclusion happened before
they were included.

52 Chapter 3. Guides
 Python Packaging User Guide

3.9 Single-sourcing the package version

There are many techniques to maintain a single source of truth for the version number of your project:
1. Read the file in setup.py and get the version. Example (from pip setup.py):

import codecs
import os.path

def read(rel_path):
here = os.path.abspath(os.path.dirname(__file__))
with codecs.open(os.path.join(here, rel_path), 'r') as fp:
return fp.read()

def get_version(rel_path):
for line in read(rel_path).splitlines():
if line.startswith('__version__'):
delim = '"' if '"' in line else "'"
return line.split(delim)[1]
else:
raise RuntimeError("Unable to find version string.")

setup(
...
version=get_version("package/__init__.py")
...
)

2. Use an external build tool that either manages updating both locations, or offers an API that both locations can
use.
Few tools you could use, in no particular order, and not necessarily complete: bump2version, changes,
zest.releaser.
3. Set the value to a __version__ global variable in a dedicated module in your project (e.g. version.py),
then have setup.py read and exec the value into a variable.

version = {}
with open("...sample/version.py") as fp:
exec(fp.read(), version)
# later on we use: version['__version__']

Example using this technique: warehouse.

4. Place the value in a simple VERSION text file and have both setup.py and the project code read it.

with open(os.path.join(mypackage_root_dir, 'VERSION')) as version_file:

version = version_file.read().strip()

An advantage with this technique is that it’s not specific to Python. Any tool can read the version.

Warning: With this approach you must make sure that the VERSION file is included in all your source and
binary distributions (e.g. add include VERSION to your MANIFEST.in).

5. Set the value in setup.py, and have the project code use the importlib.metadata API to fetch the value
at runtime. (importlib.metadata was introduced in Python 3.8 and is available to older versions as the
importlib-metadata project.) An installed project’s version can be fetched with the API as follows:

3.9. Single-sourcing the package version 53

Python Packaging User Guide

try:
from importlib import metadata
except ImportError:
# Running on pre-3.8 Python; use importlib-metadata package
import importlib_metadata as metadata

assert metadata.version('pip') == '1.2.0'

Be aware that the importlib.metadata API only knows about what’s in the installation metadata, which
is not necessarily the code that’s currently imported.
If a project uses this method to fetch its version at runtime, then its install_requires value needs to be
edited to install importlib-metadata on pre-3.8 versions of Python like so:

setup(
...
install_requires=[
...
'importlib-metadata ~= 1.0 ; python_version < "3.8"',
...
],
...
)

An older (and less efficient) alternative to importlib.metadata is the pkg_resources API provided
by setuptools:

import pkg_resources
assert pkg_resources.get_distribution('pip').version == '1.2.0'

If a project uses pkg_resources to fetch its own version at runtime, then setuptools must be added to
the project’s install_requires list.
Example using this technique: setuptools.
6. Set the value to __version__ in sample/__init__.py and import sample in setup.py.

import sample
setup(
...
version=sample.__version__
...
)

Warning: Although this technique is common, beware that it will fail if sample/__init__.py im-
ports packages from install_requires dependencies, which will very likely not be installed yet when
setup.py is run.

7. Keep the version number in the tags of a version control system (Git, Mercurial, etc) instead of in the code, and
automatically extract it from there using setuptools_scm.

3.10 Supporting multiple Python versions

Page Status Incomplete

54 Chapter 3. Guides
 Python Packaging User Guide

Last Reviewed 2014-12-24

Contents

• Automated testing and continuous integration

• Tools for single-source Python packages
• What’s in which Python?

FIXME

Useful projects/resources to reference:

- DONE six
- DONE python-future (http://python-future.org)
- tox
- DONE Travis and Shining Panda CI (Shining Panda no longer available)
- DONE Appveyor
- DONE Ned Batchelder's "What's in Which Python"
- http://nedbatchelder.com/blog/201310/whats_in_which_python_3.html
- http://nedbatchelder.com/blog/201109/whats_in_which_python.html
- Lennart Regebro's "Porting to Python 3"
- Greg Hewgill's script to identify the minimum version of Python
required to run a particular script:
https://github.com/ghewgill/pyqver
- the Python 3 porting how to in the main docs
- cross reference to the stable ABI discussion
in the binary extensions topic (once that exists)
- mention version classifiers for distribution metadata

In addition to the work required to create a Python package, it is often necessary that the package must be made
available on different versions of Python. Different Python versions may contain different (or renamed) standard
library packages, and the changes between Python versions 2.x and 3.x include changes in the language syntax.
Performed manually, all the testing required to ensure that the package works correctly on all the target Python versions
(and OSs!) could be very time-consuming. Fortunately, several tools are available for dealing with this, and these will
briefly be discussed here.

3.10.1 Automated testing and continuous integration

Several hosted services for automated testing are available. These services will typically monitor your source code
repository (e.g. at Github or Bitbucket) and run your project’s test suite every time a new commit is made.
These services also offer facilities to run your project’s test suite on multiple versions of Python, giving rapid feedback
about whether the code will work, without the developer having to perform such tests themselves.
Wikipedia has an extensive comparison of many continuous-integration systems. There are two hosted services which
when used in conjunction provide automated testing across Linux, Mac and Windows:
• Travis CI provides both a Linux and a macOS environment. The Linux environment is Ubuntu 12.04 LTS Server
Edition 64 bit while the macOS is 10.9.2 at the time of writing.
• Appveyor provides a Windows environment (Windows Server 2012).

3.10. Supporting multiple Python versions 55

Python Packaging User Guide

TODO Either link to or provide example .yml files for these two
services.

TODO How do we keep the Travis Linux and macOS versions up-to-date in this
document?

Both Travis CI and Appveyor require a YAML-formatted file as specification for the instructions for testing. If any
tests fail, the output log for that specific configuration can be inspected.
For Python projects that are intended to be deployed on both Python 2 and 3 with a single-source strategy, there are a
number of options.

3.10.2 Tools for single-source Python packages

six is a tool developed by Benjamin Peterson for wrapping over the differences between Python 2 and Python 3. The
six package has enjoyed widespread use and may be regarded as a reliable way to write a single-source Python module
that can be use in both Python 2 and 3. The six module can be used from as early as Python 2.5. A tool called
modernize, developed by Armin Ronacher, can be used to automatically apply the code modifications provided by six.
Similar to six, python-future is a package that provides a compatibility layer between Python 2 and Python 3 source
code; however, unlike six, this package aims to provide interoperability between Python 2 and Python 3 with a lan-
guage syntax that matches one of the two Python versions: one may use
• a Python 2 (by syntax) module in a Python 3 project.
• a Python 3 (by syntax) module in a Python 2 project.
Because of the bi-directionality, python-future offers a pathway to converting a Python 2 package to Python 3 syntax
module-by-module. However, in contrast to six, python-future is supported only from Python 2.6. Similar to modern-
ize for six, python-future comes with two scripts called futurize and pasteurize that can be applied to either a
Python 2 module or a Python 3 module respectively.
Use of six or python-future adds an additional runtime dependency to your package: with python-future, the
futurize script can be called with the --stage1 option to apply only the changes that Python 2.6+ already pro-
vides for forward-compatibility to Python 3. Any remaining compatibility problems would require manual changes.

3.10.3 What’s in which Python?

Ned Batchelder provides a list of changes in each Python release for Python 2, Python 3.0-3.3 and Python 3.4-3.6.
These lists may be used to check whether any changes between Python versions may affect your package.

TODO These lists should be reproduced here (with permission).

TODO The py3 list should be updated to include 3.4

3.11 Dropping support for older Python versions

Dropping support for older Python versions is supported by the standard Core metadata specifications 1.2 specification
via a “Requires-Python” attribute.
Metadata 1.2+ clients, such as Pip 9.0+, will adhere to this specification by matching the current Python runtime and
comparing it with the required version in the package metadata. If they do not match, it will attempt to install the last
package distribution that supported that Python runtime.

56 Chapter 3. Guides
 Python Packaging User Guide

This mechanism can be used to drop support for older Python versions, by amending the “Requires-Python” attribute
in the package metadata.
This guide is specifically for users of setuptools, other packaging tools such as flit may offer similar functionality
but users will need to consult relevant documentation.

3.11.1 Requirements

This workflow requires that:

1. The publisher is using the latest version of setuptools,
2. The latest version of twine is used to upload the package,
3. The user installing the package has at least Pip 9.0, or a client that supports the Metadata 1.2 specification.

3.11.2 Defining the Python version required

1. Download the newest version of Setuptools

Ensure that before you generate source distributions or binary distributions, you update Setuptools and install twine.
Steps:
pip install--upgrade setuptools twine

setuptools version should be above 24.0.0.

2. Specify the version ranges for supported Python distributions

You can specify version ranges and exclusion rules, such as at least Python 3. Or, Python 2.7, 3.4 and beyond.
Examples:
Requires-Python: ">=3"
Requires-Python: ">2.7,!=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"

The way to set those values is within the call to setup within your setup.py script. This will insert the Requires-Python
metadata values based on the argument you provide in python_requires.
from setuptools import setup

setup(
# Your setup arguments
python_requires='>=2.7', # Your supported Python ranges
)

3. Validating the Metadata before publishing

Within a Python source package (the zip or the tar-gz file you download) is a text file called PKG-INFO.
This file is generated by Distutils or setuptools when it generates the source package. The file contains a set of keys
and values, the list of keys is part of the PyPa standard metadata format.
You can see the contents of the generated file like this:

3.11. Dropping support for older Python versions 57

Python Packaging User Guide

tar xfO dist/my-package-1.0.0.tar.gz my-package-1.0.0/PKG-INFO

Validate that the following is in place, before publishing the package:

• If you have upgraded correctly, the Metadata-Version value should be 1.2 or higher.
• The Requires-Python field is set and matches your specification in setup.py.

4. Using Twine to publish

Twine has a number of advantages, apart from being faster it is now the supported method for publishing packages.
Make sure you are using the newest version of Twine, at least 1.9.

3.11.3 Dropping a Python release

Once you have published a package with the Requires-Python metadata, you can then make a further update removing
that Python runtime from support.
It must be done in this order for the automated fail-back to work.
For example, you published the Requires-Python: “>=2.7” as version 1.0.0 of your package.
If you were then to update the version string to “>=3.5”, and publish a new version 2.0.0 of your package, any users
running Pip 9.0+ from version 2.7 will have version 1.0.0 of the package installed, and any >=3.5 users will receive
version 2.0.0.

3.12 Packaging binary extensions

Page Status Incomplete

Last Reviewed 2013-12-08
One of the features of the CPython reference interpreter is that, in addition to allowing the execution of Python code, it
also exposes a rich C API for use by other software. One of the most common uses of this C API is to create importable
C extensions that allow things which aren’t always easy to achieve in pure Python code.

Contents

• An overview of binary extensions

– Use cases
– Disadvantages
– Alternatives to handcoded accelerator modules
– Alternatives to handcoded wrapper modules
– Alternatives for low level system access
• Implementing binary extensions
• Building binary extensions
– Binary extensions for Windows

58 Chapter 3. Guides
 Python Packaging User Guide

– Binary extensions for Linux

– Binary extensions for macOS
• Publishing binary extensions
• Additional resources
– Cross-platform wheel generation with scikit-build
– Introduction to C/C++ extension modules

3.12.1 An overview of binary extensions

Use cases

The typical use cases for binary extensions break down into just three conventional categories:
• accelerator modules: these modules are completely self-contained, and are created solely to run faster than
the equivalent pure Python code runs in CPython. Ideally, accelerator modules will always have a pure Python
equivalent to use as a fallback if the accelerated version isn’t available on a given system. The CPython standard
library makes extensive use of accelerator modules. Example: When importing datetime, Python falls back
to the datetime.py module if the C implementation ( _datetimemodule.c) is not available.
• wrapper modules: these modules are created to expose existing C interfaces to Python code. They may either
expose the underlying C interface directly, or else expose a more “Pythonic” API that makes use of Python
language features to make the API easier to use. The CPython standard library makes extensive use of wrapper
modules. Example: functools.py is a Python module wrapper for _functoolsmodule.c.
• low-level system access: these modules are created to access lower level features of the CPython runtime, the
operating system, or the underlying hardware. Through platform specific code, extension modules may achieve
things that aren’t possible in pure Python code. A number of CPython standard library modules are written in
C in order to access interpreter internals that aren’t exposed at the language level. Example: sys, which comes
from sysmodule.c.
One particularly notable feature of C extensions is that, when they don’t need to call back into the interpreter run-
time, they can release CPython’s global interpreter lock around long-running operations (regardless of whether
those operations are CPU or IO bound).
Not all extension modules will fit neatly into the above categories. The extension modules included with NumPy, for
example, span all three use cases - they move inner loops to C for speed reasons, wrap external libraries written in C,
FORTRAN and other languages, and use low level system interfaces for both CPython and the underlying operation
system to support concurrent execution of vectorised operations and to tightly control the exact memory layout of
created objects.

Disadvantages

The main disadvantage of using binary extensions is the fact that it makes subsequent distribution of the software more
difficult. One of the advantages of using Python is that it is largely cross platform, and the languages used to write
extension modules (typically C or C++, but really any language that can bind to the CPython C API) typically require
that custom binaries be created for different platforms.
This means that binary extensions:
• require that end users be able to either build them from source, or else that someone publish pre-built binaries
for common platforms
• may not be compatible with different builds of the CPython reference interpreter

3.12. Packaging binary extensions 59

Python Packaging User Guide

• often will not work correctly with alternative interpreters such as PyPy, IronPython or Jython
• if handcoded, make maintenance more difficult by requiring that maintainers be familiar not only with Python,
but also with the language used to create the binary extension, as well as with the details of the CPython C API.
• if a pure Python fallback implementation is provided, make maintenance more difficult by requiring that changes
be implemented in two places, and introducing additional complexity in the test suite to ensure both versions
are always executed.
Another disadvantage of relying on binary extensions is that alternative import mechanisms (such as the ability to
import modules directly from zipfiles) often won’t work for extension modules (as the dynamic loading mechanisms
on most platforms can only load libraries from disk).

Alternatives to handcoded accelerator modules

When extension modules are just being used to make code run faster (after profiling has identified the code where the
speed increase is worth additional maintenance effort), a number of other alternatives should also be considered:
• look for existing optimised alternatives. The CPython standard libary includes a number of optimised data
structures and algorithms (especially in the builtins and the collections and itertools modules). The
Python Package Index also offers additional alternatives. Sometimes, the appropriate choice of standard library
or third party module can avoid the need to create your own accelerator module.
• for long running applications, the JIT compiled PyPy interpreter may offer a suitable alternative to the standard
CPython runtime. The main barrier to adopting PyPy is typically reliance on other binary extension modules
- while PyPy does emulate the CPython C API, modules that rely on that cause problems for the PyPy JIT,
and the emulation layer can often expose latent defects in extension modules that CPython currently tolerates
(frequently around reference counting errors - an object having one live reference instead of two often won’t
break anything, but no references instead of one is a major problem).
• Cython is a mature static compiler that can compile most Python code to C extension modules. The initial
compilation provides some speed increases (by bypassing the CPython interpreter layer), and Cython’s optional
static typing features can offer additional opportunities for speed increases. Using Cython still has the disad-
vantage of increasing the complexity of distributing the resulting application, but has the benefit of having a
reduced barrier to entry for Python programmers (relative to other languages like C or C++).
• Numba is a newer tool, created by members of the scientific Python community, that aims to leverage LLVM
to allow selective compilation of pieces of a Python application to native machine code at runtime. It requires
that LLVM be available on the system where the code is running, but can provide significant speed increases,
especially for operations that are amenable to vectorisation.

Alternatives to handcoded wrapper modules

The C ABI (Application Binary Interface) is a common standard for sharing functionality between multiple applica-
tions. One of the strengths of the CPython C API (Application Programming Interface) is allowing Python users to
tap into that functionality. However, wrapping modules by hand is quite tedious, so a number of other alternative
approaches should be considered.
The approaches described below don’t simplify the distribution case at all, but they can significantly reduce the main-
tenance burden of keeping wrapper modules up to date.
• In addition to being useful for the creation of accelerator modules, Cython is also useful for creating wrapper
modules. It still involves wrapping the interfaces by hand, however, so may not be a good choice for wrapping
large APIs.
• cffi is a project created by some of the PyPy developers to make it straightforward for developers that already
know both Python and C to expose their C modules to Python applications. It also makes it relatively straight-
forward to wrap a C module based on its header files, even if you don’t know C yourself.

60 Chapter 3. Guides
 Python Packaging User Guide

One of the key advantages of cffi is that it is compatible with the PyPy JIT, allowing CFFI wrapper modules
to participate fully in PyPy’s tracing JIT optimisations.
• SWIG is a wrapper interface generator that allows a variety of programming languages, including Python, to
interface with C and C++ code.
• The standard library’s ctypes module, while useful for getting access to C level interfaces when header infor-
mation isn’t available, suffers from the fact that it operates solely at the C ABI level, and thus has no automatic
consistency checking between the interface actually being exported by the library and the one declared in the
Python code. By contrast, the above alternatives are all able to operate at the C API level, using C header files
to ensure consistency between the interface exported by the library being wrapped and the one expected by the
Python wrapper module. While cffi can operate directly at the C ABI level, it suffers from the same interface
inconsistency problems as ctypes when it is used that way.

Alternatives for low level system access

For applications that need low level system access (regardless of the reason), a binary extension module often is
the best way to go about it. This is particularly true for low level access to the CPython runtime itself, since some
operations (like releasing the Global Interpreter Lock) are simply invalid when the interpreter is running code, even if
a module like ctypes or cffi is used to obtain access to the relevant C API interfaces.
For cases where the extension module is manipulating the underlying operating system or hardware (rather than the
CPython runtime), it may sometimes be better to just write an ordinary C library (or a library in another systems
programming language like C++ or Rust that can export a C compatible ABI), and then use one of the wrapping
techniques described above to make the interface available as an importable Python module.

3.12.2 Implementing binary extensions

The CPython Extending and Embedding guide includes an introduction to writing a custom extension module in C.

mention the stable ABI (3.2+, link to the CPython C API docs)
mention the module lifecycle
mention the challenges of shared static state and subinterpreters
mention the implications of the GIL for extension modules
mention the memory allocation APIs in 3.4+

mention again that all this is one of the reasons why you probably
*don't* want to handcode your extension modules :)

3.12.3 Building binary extensions

Binary extensions for Windows

Before it is possible to build a binary extension, it is necessary to ensure that you have a suitable compiler available.
On Windows, Visual C is used to build the official CPython interpreter, and should be used to build compatible binary
extensions.
Python 2.7 used Visual Studio 2008, Python 3.3 and 3.4 used Visual Studio 2010, and Python 3.5+ uses Visual Studio
2015 or later. Unfortunately, older versions of Visual Studio are no longer easily available from Microsoft, so for
versions of Python prior to 3.5, the compilers must be obtained differently if you do not already have a copy of the
relevant version of Visual Studio.
To set up a build environment for binary extensions, the steps are as follows:
For Python 2.7

3.12. Packaging binary extensions 61

Python Packaging User Guide

1. Install “Visual C++ Compiler Package for Python 2.7”, which is available from Microsoft’s website.
2. Use (a recent version of) setuptools in your setup.py (pip will do this for you, in any case).
3. Done.
For Python 3.4
1. Install “Windows SDK for Windows 7 and .NET Framework 4” (v7.1), which is available from
Microsoft’s website.
2. Work from an SDK command prompt (with the environment variables set, and the SDK on PATH).
3. Set DISTUTILS_USE_SDK=1
4. Done.
For Python 3.5
1. Install Visual Studio 2015 Community Edition (or any later version, when these are released).
2. Done.
Note that from Python 3.5 onwards, Visual Studio works in a backward compatible way, which means that any future
version of Visual Studio will be able to build Python extensions for all Python versions from 3.5 onwards.
Building with the recommended compiler on Windows ensures that a compatible C library is used throughout the
Python process.

Binary extensions for Linux

Linux binaries must use a sufficiently old glibc to be compatible with older distributions. The manylinux Docker
images provide a build environment with a glibc old enough to support most current Linux distributions on common
architectures.

Binary extensions for macOS

Binary compatibility on macOS is determined by the target minimum deployment system, e.g. 10.9, which is of-
ten specified with the MACOSX_DEPLOYMENT_TARGET environmental variable when building binaries on macOS.
When building with setuptools / distutils, the deployment target is specified with the flag --plat-name, e.g.
macosx-10.9-x86_64. For common deployment targets for macOS Python distributions, see the MacPython
Spinning Wheels wiki.

3.12.4 Publishing binary extensions

For interim guidance on this topic, see the discussion in this issue.

FIXME

cover publishing as wheel files on PyPI or a custom index server

cover creation of Windows and macOS installers
cover weak linking
mention the fact that Linux distros have a requirement to build from
source in their own build systems, so binary-only releases are strongly
discouraged

62 Chapter 3. Guides
 Python Packaging User Guide

3.12.5 Additional resources

Cross-platform development and distribution of extension modules is a complex topic, so this guide focuses primarily
on providing pointers to various tools that automate dealing with the underlying technical challenges. The additional
resources in this section are instead intended for developers looking to understand more about the underlying binary
interfaces that those systems rely on at runtime.

Cross-platform wheel generation with scikit-build

The scikit-build package helps abstract cross-platform build operations and provides additional capabilities when
creating binary extension packages. Additional documentation is also available on the C runtime, compiler, and build
system generator for Python binary extension modules.

Introduction to C/C++ extension modules

For a more in depth explanation of how extension modules are used by CPython on a Debian system, see the following
articles:
• What are (c)python extension modules?
• Releasing the gil
• Writing cpython extension modules using C++

3.13 Supporting Windows using Appveyor

Page Status Incomplete

Last Reviewed 2015-12-03
This section covers how to use the free Appveyor continuous integration service to provide Windows support for your
project. This includes testing the code on Windows, and building Windows-targeted binaries for projects that use C
extensions.

Contents

• Background
• Setting up
• Adding Appveyor support to your project
– appveyor.yml
– Support script
– Access to the built wheels
• Additional notes
– Testing with tox
– Automatically uploading wheels
– External dependencies
– Support scripts

3.13. Supporting Windows using Appveyor 63

 Python Packaging User Guide

3.13.1 Background

Many projects are developed on Unix by default, and providing Windows support can be a challenge, because setting
up a suitable Windows test environment is non-trivial, and may require buying software licenses.
The Appveyor service is a continuous integration service, much like the better-known Travis service that is commonly
used for testing by projects hosted on Github. However, unlike Travis, the build workers on Appveyor are Windows
hosts and have the necessary compilers installed to build Python extensions.
Windows users typically do not have access to a C compiler, and therefore are reliant on projects that use C extensions
distributing binary wheels on PyPI in order for the distribution to be installable via pip install <dist>. By
using Appveyor as a build service (even if not using it for testing) it is possible for projects without a dedicated
Windows environment to provide Windows-targeted binaries.

3.13.2 Setting up

In order to use Appveyor to build Windows wheels for your project, you must have an account on the service. Instruc-
tions on setting up an account are given in the Appveyor documentation. The free tier of account is perfectly adequate
for open source projects.
Appveyor provides integration with Github and Bitbucket, so as long as your project is hosted on one of those two
services, setting up Appveyor integration is straightforward.
Once you have set up your Appveyor account and added your project, Appveyor will automatically build your project
each time a commit occurs. This behaviour will be familiar to users of Travis.

3.13.3 Adding Appveyor support to your project

In order to define how Appveyor should build your project, you need to add an appveyor.yml file to your project.
The full details of what can be included in the file are covered in the Appveyor documentation. This guide will provide
the details necessary to set up wheel builds.
Appveyor includes by default all of the compiler toolchains needed to build extensions for Python. For Python 2.7,
3.5+ and 32-bit versions of 3.3 and 3.4, the tools work out of the box. But for 64-bit versions of Python 3.3 and 3.4,
there is a small amount of additional configuration needed to let distutils know where to find the 64-bit compilers.
(From 3.5 onwards, the version of Visual Studio used includes 64-bit compilers with no additional setup).

appveyor.yml

1 environment:
2

3 matrix:
4

5 # For Python versions available on Appveyor, see

6 # https://www.appveyor.com/docs/windows-images-software/#python
7 # The list here is complete (excluding Python 2.6, which
8 # isn't covered by this document) at the time of writing.
9

10 - PYTHON: "C:\\Python27"
11 - PYTHON: "C:\\Python33"
12 - PYTHON: "C:\\Python34"
13 - PYTHON: "C:\\Python35"
14 - PYTHON: "C:\\Python27-x64"
15 - PYTHON: "C:\\Python33-x64"
(continues on next page)

64 Chapter 3. Guides
 Python Packaging User Guide

(continued from previous page)

16 DISTUTILS_USE_SDK: "1"
17 - PYTHON: "C:\\Python34-x64"
18 DISTUTILS_USE_SDK: "1"
19 - PYTHON: "C:\\Python35-x64"
20 - PYTHON: "C:\\Python36-x64"
21

22 install:
23 # We need wheel installed to build wheels
24 - "%PYTHON%\\python.exe -m pip install wheel"
25

26 build: off
27

28 test_script:
29 # Put your test command here.
30 # If you don't need to build C extensions on 64-bit Python 3.3 or 3.4,
31 # you can remove "build.cmd" from the front of the command, as it's
32 # only needed to support those cases.
33 # Note that you must use the environment variable %PYTHON% to refer to
34 # the interpreter you're using - Appveyor does not do anything special
35 # to put the Python version you want to use on PATH.
36 - "build.cmd %PYTHON%\\python.exe setup.py test"
37

38 after_test:
39 # This step builds your wheels.
40 # Again, you only need build.cmd if you're building C extensions for
41 # 64-bit Python 3.3/3.4. And you need to use %PYTHON% to get the correct
42 # interpreter
43 - "build.cmd %PYTHON%\\python.exe setup.py bdist_wheel"
44

45 artifacts:
46 # bdist_wheel puts your built wheel in the dist directory
47 - path: dist\*
48

49 #on_success:
50 # You can use this step to upload your artifacts to a public website.
51 # See Appveyor's documentation for more details. Or you can simply
52 # access your wheels from the Appveyor "artifacts" tab for your build.

This file can be downloaded from here.

The appveyor.yml file must be located in the root directory of your project. It is in YAML format, and consists of
a number of sections.
The environment section is the key to defining the Python versions for which your wheels will be created. Ap-
pveyor comes with Python 2.6, 2.7, 3.3, 3.4 and 3.5 installed, in both 32-bit and 64-bit builds. The example file builds
for all of these environments except Python 2.6. Installing for Python 2.6 is more complex, as it does not come with
pip included. We don’t support 2.6 in this document (as Windows users still using Python 2 are generally able to move
to Python 2.7 without too much difficulty).
The install section uses pip to install any additional software that the project may require. The only requirement
for building wheels is the wheel project, but projects may wish to customise this code in certain circumstances (for
example, to install additional build packages such as Cython, or test tools such as tox).
The build section simply switches off builds - there is no build step needed for Python, unlike languages like C#.
The main sections that will need to be tailored to your project are test_script and after_test.
The test_script section is where you will run your project’s tests. The supplied file runs your test suite using
setup.py test. If you are only interested in building wheels, and not in running your tests on Windows, you can

3.13. Supporting Windows using Appveyor 65

Python Packaging User Guide

replace this section with a dummy command such as echo Skipped Tests. You may wish to use another test
tool, such as nose or py.test. Or you may wish to use a test driver like tox - however if you are using tox there
are some additional configuration changes you will need to consider, which are described below.
The after_test runs once your tests have completed, and so is where the wheels should be built. Assuming your
project uses the recommended tools (specifically, setuptools) then the setup.py bdist_wheel command
will build your wheels.
Note that wheels will only be built if your tests succeed. If you expect your tests to fail on Windows, you can skip
them as described above.

Support script

The appveyor.yml file relies on a single support script, which sets up the environment to use the SDK compiler
for 64-bit builds on Python 3.3 and 3.4. For projects which do not need a compiler, or which don’t support 3.3 or 3.4
on 64-bit Windows, only the appveyor.yml file is needed.
build.cmd is a Windows batch script that runs a single command in an environment with the appropriate compiler for
the selected Python version. All you need to do is to set the single environment variable DISTUTILS_USE_SDK to
a value of 1 and the script does the rest. It sets up the SDK needed for 64-bit builds of Python 3.3 or 3.4, so don’t set
the environment variable for any other builds.
You can simply download the batch file and include it in your project unchanged.

Access to the built wheels

When your build completes, the built wheels will be available from the Appveyor control panel for your project. They
can be found by going to the build status page for each build in turn. At the top of the build output there is a series
of links, one of which is “Artifacts”. That page will include a list of links to the wheels for that Python version /
architecture. You can download those wheels and upload them to PyPI as part of your release process.

3.13.4 Additional notes

Testing with tox

Many projects use the Tox tool to run their tests. It ensures that tests are run in an isolated environment using the exact
files that will be distributed by the project.
In order to use tox on Appveyor there are a couple of additional considerations (in actual fact, these issues are not
specific to Appveyor, and may well affect other CI systems).
1. By default, tox only passes a chosen subset of environment variables to the test processes. Because
distutils uses environment variables to control the compiler, this “test isolation” feature will cause the
tests to use the wrong compiler by default.
To force tox to pass the necessary environment variables to the subprocess, you need to set the tox configura-
tion option passenv to list the additional environment variables to be passed to the subprocess. For the SDK
compilers, you need
• DISTUTILS_USE_SDK
• MSSdk
• INCLUDE

66 Chapter 3. Guides
 Python Packaging User Guide

• LIB
The passenv option can be set in your tox.ini, or if you prefer to avoid adding Windows-
specific settings to your general project files, it can be set by setting the TOX_TESTENV_PASSENV
environment variable. The supplied build.cmd script does this by default whenever
DISTUTILS_USE_SDK is set.
2. When used interactively, tox allows you to run your tests against multiple environments (often, this means
multiple Python versions). This feature is not as useful in a CI environment like Travis or Appveyor, where
all tests are run in isolated environments for each configuration. As a result, projects often supply an argument
-e ENVNAME to tox to specify which environment to use (there are default environments for most versions of
Python).
However, this does not work well with a Windows CI system like Appveyor, where there are (for ex-
ample) two installations of Python 3.4 (32-bit and 64-bit) available, but only one py34 environment
in tox.
In order to run tests using tox, therefore, projects should probably use the default py environment
in tox, which uses the Python interpreter that was used to run tox. This will ensure that when
Appveyor runs the tests, they will be run with the configured interpreter.
In order to support running under the py environment, it is possible that projects with complex tox
configurations might need to modify their tox.ini file. Doing so is, however, outside the scope of
this document.

Automatically uploading wheels

It is possible to request Appveyor to automatically upload wheels. There is a deployment step available in
appveyor.yml that can be used to (for example) copy the built artifacts to a FTP site, or an Amazon S3 instance.
Documentation on how to do this is included in the Appveyor guides.
Alternatively, it would be possible to add a twine upload step to the build. The supplied appveyor.yml does
not do this, as it is not clear that uploading new wheels after every commit is desirable (although some projects may
wish to do this).

External dependencies

The supplied scripts will successfully build any distribution that does not rely on 3rd party external libraries for the
build.
It is possible to add steps to the appveyor.yml configuration (typically in the “install” section) to download and/or
build external libraries needed by the distribution. And if needed, it is possible to add extra configuration for the build
to supply the location of these libraries to the compiler. However, this level of configuration is beyond the scope of
this document.

Support scripts

For reference, the SDK setup support script is listed here:

appveyor-sample/build.cmd
1 @echo off
2 :: To build extensions for 64 bit Python 3, we need to configure environment
3 :: variables to use the MSVC 2010 C++ compilers from GRMSDKX_EN_DVD.iso of:
4 :: MS Windows SDK for Windows 7 and .NET Framework 4
5 ::
(continues on next page)

3.13. Supporting Windows using Appveyor 67

 Python Packaging User Guide

(continued from previous page)

6 :: More details at:
7 :: https://github.com/cython/cython/wiki/CythonExtensionsOnWindows
8

9 IF "%DISTUTILS_USE_SDK%"=="1" (
10 ECHO Configuring environment to build with MSVC on a 64bit architecture
11 ECHO Using Windows SDK 7.1
12 "C:\Program Files\Microsoft SDKs\Windows\v7.1\Setup\WindowsSdkVer.exe" -q -
˓→version:v7.1

13 CALL "C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin\SetEnv.cmd" /x64 /release

14 SET MSSdk=1
15 REM Need the following to allow tox to see the SDK compiler
16 SET TOX_TESTENV_PASSENV=DISTUTILS_USE_SDK MSSdk INCLUDE LIB
17 ) ELSE (
18 ECHO Using default MSVC build environment
19 )
20

21 CALL %*

3.14 Packaging namespace packages

Namespace packages allow you to split the sub-packages and modules within a single package across multiple, sepa-
rate distribution packages (referred to as distributions in this document to avoid ambiguity). For example, if you have
the following package structure:

mynamespace/
__init__.py
subpackage_a/
__init__.py
...
subpackage_b/
__init__.py
...
module_b.py
setup.py

And you use this package in your code like so:

from mynamespace import subpackage_a

from mynamespace import subpackage_b

Then you can break these sub-packages into two separate distributions:

mynamespace-subpackage-a/
setup.py
mynamespace/
subpackage_a/
__init__.py

mynamespace-subpackage-b/
setup.py
mynamespace/
subpackage_b/
__init__.py
module_b.py

68 Chapter 3. Guides
 Python Packaging User Guide

Each sub-package can now be separately installed, used, and versioned.

Namespace packages can be useful for a large collection of loosely-related packages (such as a large corpus of
client libraries for multiple products from a single company). However, namespace packages come with several
caveats and are not appropriate in all cases. A simple alternative is to use a prefix on all of your distributions such
as import mynamespace_subpackage_a (you could even use import mynamespace_subpackage_a
as subpackage_a to keep the import object short).

3.14.1 Creating a namespace package

There are currently three different approaches to creating namespace packages:

1. Use native namespace packages. This type of namespace package is defined in PEP 420 and is available in
Python 3.3 and later. This is recommended if packages in your namespace only ever need to support Python 3
and installation via pip.
2. Use pkgutil-style namespace packages. This is recommended for new packages that need to support Python 2
and 3 and installation via both pip and python setup.py install.
3. Use pkg_resources-style namespace packages. This method is recommended if you need compatibility with
packages already using this method or if your package needs to be zip-safe.

Warning: While native namespace packages and pkgutil-style namespace packages are largely compatible,
pkg_resources-style namespace packages are not compatible with the other methods. It’s inadvisable to use differ-
ent methods in different distributions that provide packages to the same namespace.

Native namespace packages

Python 3.3 added implicit namespace packages from PEP 420. All that is required to create a native namespace
package is that you just omit __init__.py from the namespace package directory. An example file structure:

setup.py
mynamespace/
# No __init__.py here.
subpackage_a/
# Sub-packages have __init__.py.
__init__.py
module.py

It is extremely important that every distribution that uses the namespace package omits the __init__.py or uses
a pkgutil-style __init__.py. If any distribution does not, it will cause the namespace logic to fail and the other
sub-packages will not be importable.
Because mynamespace doesn’t contain an __init__.py, setuptools.find_packages() won’t find the
sub-package. You must use setuptools.find_namespace_packages() instead or explicitly list all pack-
ages in your setup.py. For example:

from setuptools import setup, find_namespace_packages

setup(
name='mynamespace-subpackage-a',
...
packages=find_namespace_packages(include=['mynamespace.*'])
)

3.14. Packaging namespace packages 69

Python Packaging User Guide

A complete working example of two native namespace packages can be found in the native namespace package
example project.

Note: Because native and pkgutil-style namespace packages are largely compatible, you can use native namespace
packages in the distributions that only support Python 3 and pkgutil-style namespace packages in the distributions that
need to support Python 2 and 3.

pkgutil-style namespace packages

Python 2.3 introduced the pkgutil module and the extend_path function. This can be used to declare namespace
packages that need to be compatible with both Python 2.3+ and Python 3. This is the recommended approach for the
highest level of compatibility.
To create a pkgutil-style namespace package, you need to provide an __init__.py file for the namespace package:

setup.py
mynamespace/
__init__.py # Namespace package __init__.py
subpackage_a/
__init__.py # Sub-package __init__.py
module.py

The __init__.py file for the namespace package needs to contain only the following:

__path__ = __import__('pkgutil').extend_path(__path__, __name__)

Every distribution that uses the namespace package must include an identical __init__.py. If any distribution
does not, it will cause the namespace logic to fail and the other sub-packages will not be importable. Any additional
code in __init__.py will be inaccessible.
A complete working example of two pkgutil-style namespace packages can be found in the pkgutil namespace example
project.

pkg_resources-style namespace packages

Setuptools provides the pkg_resources.declare_namespace function and the namespace_packages argument to

setup(). Together these can be used to declare namespace packages. While this approach is no longer recom-
mended, it is widely present in most existing namespace packages. If you are creating a new distribution within
an existing namespace package that uses this method then it’s recommended to continue using this as the different
methods are not cross-compatible and it’s not advisable to try to migrate an existing package.
To create a pkg_resources-style namespace package, you need to provide an __init__.py file for the namespace
package:

setup.py
mynamespace/
__init__.py # Namespace package __init__.py
subpackage_a/
__init__.py # Sub-package __init__.py
module.py

The __init__.py file for the namespace package needs to contain only the following:

70 Chapter 3. Guides
 Python Packaging User Guide

__import__('pkg_resources').declare_namespace(__name__)

Every distribution that uses the namespace package must include an identical __init__.py. If any distribution
does not, it will cause the namespace logic to fail and the other sub-packages will not be importable. Any additional
code in __init__.py will be inaccessible.

Note: Some older recommendations advise the following in the namespace package __init__.py:

try:
__import__('pkg_resources').declare_namespace(__name__)
except ImportError:
__path__ = __import__('pkgutil').extend_path(__path__, __name__)

The idea behind this was that in the rare case that setuptools isn’t available packages would fall-back to the pkgutil-
style packages. This isn’t advisable because pkgutil and pkg_resources-style namespace packages are not cross-
compatible. If the presence of setuptools is a concern then the package should just explicitly depend on setuptools via
install_requires.

Finally, every distribution must provide the namespace_packages argument to setup() in setup.py. For
example:

from setuptools import find_packages, setup

setup(
name='mynamespace-subpackage-a',
...
packages=find_packages()
namespace_packages=['mynamespace']
)

A complete working example of two pkg_resources-style namespace packages can be found in the pkg_resources
namespace example project.

3.15 Creating and discovering plugins

Often when creating a Python application or library you’ll want the ability to provide customizations or extra fea-
tures via plugins. Because Python packages can be separately distributed, your application or library may want to
automatically discover all of the plugins available.
There are three major approaches to doing automatic plugin discovery:
1. Using naming convention.
2. Using namespace packages.
3. Using package metadata.

3.15.1 Using naming convention

If all of the plugins for your application follow the same naming convention, you can use pkgutil.
iter_modules() to discover all of the top-level modules that match the naming convention. For example, Flask
uses the naming convention flask_{plugin_name}. If you wanted to automatically discover all of the Flask
plugins installed:

3.15. Creating and discovering plugins 71

Python Packaging User Guide

import importlib
import pkgutil

discovered_plugins = {
name: importlib.import_module(name)
for finder, name, ispkg
in pkgutil.iter_modules()
if name.startswith('flask_')
}

If you had both the Flask-SQLAlchemy and Flask-Talisman plugins installed then discovered_plugins would
be:

{
'flask_sqlachemy': <module: 'flask_sqlalchemy'>,
'flask_talisman': <module: 'flask_talisman'>,
}

Using naming convention for plugins also allows you to query the Python Package Index’s simple API for all packages
that conform to your naming convention.

3.15.2 Using namespace packages

Namespace packages can be used to provide a convention for where to place plugins and also provides a way
to perform discovery. For example, if you make the sub-package myapp.plugins a namespace package then
other distributions can provide modules and packages to that namespace. Once installed, you can use pkgutil.
iter_modules() to discover all modules and packages installed under that namespace:

import importlib
import pkgutil

import myapp.plugins

def iter_namespace(ns_pkg):
# Specifying the second argument (prefix) to iter_modules makes the
# returned name an absolute name instead of a relative one. This allows
# import_module to work without having to do additional modification to
# the name.
return pkgutil.iter_modules(ns_pkg.__path__, ns_pkg.__name__ + ".")

discovered_plugins = {
name: importlib.import_module(name)
for finder, name, ispkg
in iter_namespace(myapp.plugins)
}

Specifying myapp.plugins.__path__ to iter_modules() causes it to only look for the modules directly
under that namespace. For example, if you have installed distributions that provide the modules myapp.plugins.
a and myapp.plugins.b then discovered_plugins in this case would be:

{
'a': <module: 'myapp.plugins.a'>,
'b': <module: 'myapp.plugins.b'>,
}

72 Chapter 3. Guides
 Python Packaging User Guide

This sample uses a sub-package as the namespace package (myapp.plugins), but it’s also possible to use a top-level
package for this purpose (such as myapp_plugins). How to pick the namespace to use is a matter of preference,
but it’s not recommended to make your project’s main top-level package (myapp in this case) a namespace package
for the purpose of plugins, as one bad plugin could cause the entire namespace to break which would in turn make
your project unimportable. For the “namespace sub-package” approach to work, the plugin packages must omit
the __init__.py for your top-level package directory (myapp in this case) and include the namespace-package
style __init__.py in the namespace sub-package directory (myapp/plugins). This also means that plugins
will need to explicitly pass a list of packages to setup()’s packages argument instead of using setuptools.
find_packages().

Warning: Namespace packages are a complex feature and there are several different ways to create them. It’s
highly recommended to read the Packaging namespace packages documentation and clearly document which
approach is preferred for plugins to your project.

3.15.3 Using package metadata

Setuptools provides special support for plugins. By providing the entry_points argument to setup() in
setup.py plugins can register themselves for discovery.
For example if you have a package named myapp-plugin-a and it includes in its setup.py:
setup(
...
entry_points={'myapp.plugins': 'a = myapp_plugin_a'},
...
)

Then you can discover and load all of the registered entry points by using pkg_resources.
iter_entry_points():
import pkg_resources

discovered_plugins = {
entry_point.name: entry_point.load()
for entry_point
in pkg_resources.iter_entry_points('myapp.plugins')
}

In this example, discovered_plugins would be:

{
'a': <module: 'myapp_plugin_a'>,
}

Note: The entry_point specification in setup.py is fairly flexible and has a lot of options. It’s recommended
to read over the entire section on entry points.

3.16 Analyzing PyPI package downloads

This section covers how to use the public PyPI download statistics dataset to learn more about downloads of a package
(or packages) hosted on PyPI. For example, you can use it to discover the distribution of Python versions used to

3.16. Analyzing PyPI package downloads 73

Python Packaging User Guide

download a package.

Contents

• Background
• Public dataset
– Getting set up
– Data schema
– Useful queries

* Counting package downloads

* Package downloads over time
* More queries
• Caveats
• Additional tools
– google-cloud-bigquery
– pypinfo
– pandas-gbq
• References

3.16.1 Background

PyPI does not display download statistics for a number of reasons:1

• Inefficient to make work with a Content Distribution Network (CDN): Download statistics change con-
stantly. Including them in project pages, which are heavily cached, would require invalidating the cache more
often, and reduce the overall effectiveness of the cache.
• Highly inaccurate: A number of things prevent the download counts from being accurate, some of which
include:
– pip’s download cache (lowers download counts)
– Internal or unofficial mirrors (can both raise or lower download counts)
– Packages not hosted on PyPI (for comparisons sake)
– Unofficial scripts or attempts at download count inflation (raises download counts)
– Known historical data quality issues (lowers download counts)
• Not particularly useful: Just because a project has been downloaded a lot doesn’t mean it’s good; Similarly
just because a project hasn’t been downloaded a lot doesn’t mean it’s bad!
In short, because it’s value is low for various reasons, and the tradeoffs required to make it work are high, it has been
not an effective use of limited resources.
1 PyPI Download Counts deprecation email

74 Chapter 3. Guides
 Python Packaging User Guide

3.16.2 Public dataset

As an alternative, the Linehaul project streams download logs from PyPI to Google BigQuery2 , where they are stored
as a public dataset.

Getting set up

In order to use Google BigQuery to query the public PyPI download statistics dataset, you’ll need a Google account
and to enable the BigQuery API on a Google Cloud Platform project. You can run the up to 1TB of queries per month
using the BigQuery free tier without a credit card
• Navigate to the BigQuery web UI.
• Create a new project.
• Enable the BigQuery API.
For more detailed instructions on how to get started with BigQuery, check out the BigQuery quickstart guide.

Data schema

Linehaul writes an entry in a the-psf.pypi.downloadsYYYYMMDD table for each download. The table contains
information about what file was downloaded and how it was downloaded. Some useful columns from the table schema
include:

Column Description Examples

file.project Project name pipenv, nose
file.version Package version 0.1.6, 1.4.2
details.installer.name Installer pip, bandersnatch
details.python Python version 2.7.12, 3.6.4

Useful queries

Run queries in the BigQuery web UI by clicking the “Compose query” button.
Note that the rows are stored in separate tables for each day, which helps limit the cost of queries. These exam-
ple queries analyze downloads from recent history by using wildcard tables to select all tables and then filter by
_TABLE_SUFFIX.

Counting package downloads

The following query counts the total number of downloads for the project “pytest”.

#standardSQL
SELECT COUNT(*) AS num_downloads
FROM `the-psf.pypi.downloads*`
WHERE file.project = 'pytest'
-- Only query the last 30 days of history
AND _TABLE_SUFFIX
BETWEEN FORMAT_DATE(
'%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY))
AND FORMAT_DATE('%Y%m%d', CURRENT_DATE())

2 PyPI BigQuery dataset announcement email

3.16. Analyzing PyPI package downloads 75

Python Packaging User Guide

num_downloads
2117807

To only count downloads from pip, filter on the details.installer.name column.

#standardSQL
SELECT COUNT(*) AS num_downloads
FROM `the-psf.pypi.downloads*`
WHERE file.project = 'pytest'
AND details.installer.name = 'pip'
-- Only query the last 30 days of history
AND _TABLE_SUFFIX
BETWEEN FORMAT_DATE(
'%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY))
AND FORMAT_DATE('%Y%m%d', CURRENT_DATE())

num_downloads
1829322

Package downloads over time

To group by monthly downloads, use the _TABLE_SUFFIX pseudo-column. Also use the pseudo-column to limit the
tables queried and the corresponding costs.
#standardSQL
SELECT
COUNT(*) AS num_downloads,
SUBSTR(_TABLE_SUFFIX, 1, 6) AS `month`
FROM `the-psf.pypi.downloads*`
WHERE
file.project = 'pytest'
-- Only query the last 6 months of history
AND _TABLE_SUFFIX
BETWEEN FORMAT_DATE(
'%Y%m01', DATE_SUB(CURRENT_DATE(), INTERVAL 6 MONTH))
AND FORMAT_DATE('%Y%m%d', CURRENT_DATE())
GROUP BY `month`
ORDER BY `month` DESC

num_downloads month
1956741 201801
2344692 201712
1730398 201711
2047310 201710
1744443 201709
1916952 201708

More queries

• Data driven decisions using PyPI download statistics

• PyPI queries gist

76 Chapter 3. Guides
 Python Packaging User Guide

• Python versions over time

• Non-Windows downloads, grouped by platform

3.16.3 Caveats

In addition to the caveats listed in the background above, Linehaul suffered from a bug which caused it to significantly
under-report download statistics prior to July 26, 2018. Downloads before this date are proportionally accurate (e.g.
the percentage of Python 2 vs. Python 3 downloads) but total numbers are lower than actual by an order of magnitude.

3.16.4 Additional tools

Besides using the BigQuery console, there are some additional tools which may be useful when analyzing download
statistics.

google-cloud-bigquery

You can also access the public PyPI download statistics dataset programmatically via the BigQuery API and the
google-cloud-bigquery project, the official Python client library for BigQuery.

from google.cloud import bigquery

# Note: depending on where this code is being run, you may require
# additional authentication. See:
# https://cloud.google.com/bigquery/docs/authentication/
client = bigquery.Client()

query_job = client.query("""
SELECT COUNT(*) AS num_downloads
FROM `the-psf.pypi.downloads*`
WHERE file.project = 'pytest'
-- Only query the last 30 days of history
AND _TABLE_SUFFIX
BETWEEN FORMAT_DATE(
'%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY))
AND FORMAT_DATE('%Y%m%d', CURRENT_DATE())""")

results = query_job.result() # Waits for job to complete.

for row in results:
print("{} downloads".format(row.num_downloads))

pypinfo

pypinfo is a command-line tool which provides access to the dataset and can generate several useful queries. For
example, you can query the total number of download for a package with the command pypinfo package_name.
Install pypinfo using pip.

pip install pypinfo

Usage:

3.16. Analyzing PyPI package downloads 77

Python Packaging User Guide

$ pypinfo requests
Served from cache: False
Data processed: 6.87 GiB
Data billed: 6.87 GiB
Estimated cost: $0.04

| download_count |
| -------------- |
| 9,316,415 |

pandas-gbq

The pandas-gbq project allows for accessing query results via Pandas.

3.16.5 References

3.17 Package index mirrors and caches

Page Status Incomplete

Last Reviewed 2014-12-24

Contents

• Caching with pip

• Caching with devpi
• Complete mirror with bandersnatch

Mirroring or caching of PyPI can be used to speed up local package installation, allow offline work, handle corporate
firewalls or just plain Internet flakiness.
Three options are available in this area:
1. pip provides local caching options,
2. devpi provides higher-level caching option, potentially shared amongst many users or machines, and
3. bandersnatch provides a local complete mirror of all PyPI packages.

3.17.1 Caching with pip

pip provides a number of facilities for speeding up installation by using local cached copies of packages:
1. Fast & local installs by downloading all the requirements for a project and then pointing pip at those downloaded
files instead of going to PyPI.
2. A variation on the above which pre-builds the installation files for the requirements using pip wheel:

$ pip wheel --wheel-dir=/tmp/wheelhouse SomeProject

$ pip install --no-index --find-links=/tmp/wheelhouse SomeProject

78 Chapter 3. Guides
 Python Packaging User Guide

3.17.2 Caching with devpi

devpi is a caching proxy server which you run on your laptop, or some other machine you know will always be
available to you. See the devpi documentation for getting started.

3.17.3 Complete mirror with bandersnatch

bandersnatch will set up a complete local mirror of all PyPI packages (externally-hosted packages are not mirrored).
See the bandersnatch documentation for getting that going.
A benefit of devpi is that it will create a mirror which includes packages that are external to PyPI, unlike bandersnatch
which will only cache packages hosted on PyPI.

3.18 Hosting your own simple repository

If you wish to host your own simple repository1 , you can either use a software package like devpi or you can use simply
create the proper directory structure and use any web server that can serve static files and generate an autoindex.
In either case, since you’ll be hosting a repository that is likely not in your user’s default repositories, you should
instruct them in your project’s description to configure their installer appropriately. For example with pip:
pip install --extra-index-url https://python.example.com/ foobar

In addition, it is highly recommended that you serve your repository with valid HTTPS. At this time, the security of
your user’s installations depends on all repositories using a valid HTTPS setup.

3.18.1 “Manual” repository

The directory layout is fairly simple, within a root directory you need to create a directory for each project. This
directory should be the normalized name (as defined by PEP 503) of the project. Within each of these directories
simply place each of the downloadable files. If you have the projects “Foo” (with the versions 1.0 and 2.0) and “bar”
(with the version 0.1) You should end up with a structure that looks like:
.
bar
bar-0.1.tar.gz
foo
Foo-1.0.tar.gz
Foo-2.0.tar.gz

Once you have this layout, simply configure your webserver to serve the root directory with autoindex enabled. For
an example using the built in Web server in Twisted, you would simply run twistd -n web --path . and then
instruct users to add the URL to their installer’s configuration.

3.19 Migrating to PyPI.org

PyPI.org is the new, rewritten version of PyPI that has replaced the legacy PyPI code base. It is the default version
of PyPI that people are expected to use. These are the tools and processes that people will need to interact with
PyPI.org.
1 For complete documentation of the simple repository protocol, see PEP 503.

3.18. Hosting your own simple repository 79

Python Packaging User Guide

3.19.1 Publishing releases

pypi.org is the default upload platform as of September 2016.

Uploads through pypi.python.org were switched off on July 3, 2017. As of April 13th, 2018, pypi.org is the
URL for PyPI.
The recommended way to migrate to PyPI.org for uploading is to ensure that you are using a new enough version of
your upload tool.
The default upload settings switched to pypi.org in the following versions:
• twine 1.8.0
• setuptools 27.0.0
• Python 2.7.13 (distutils update)
• Python 3.4.6 (distutils update)
• Python 3.5.3 (distutils update)
• Python 3.6.0 (distutils update)
In addition to ensuring you’re on a new enough version of the tool for the tool’s default to have switched, you must
also make sure that you have not configured the tool to override its default upload URL. Typically this is configured
in a file located at $HOME/.pypirc. If you see a file like:

[distutils]
index-servers =
pypi

[pypi]
repository = https://pypi.python.org/pypi
username = <your PyPI username>
password = <your PyPI username>

Then simply delete the line starting with repository and you will use your upload tool’s default URL.
If for some reason you’re unable to upgrade the version of your tool to a version that defaults to using PyPI.org,
then you may edit $HOME/.pypirc and include the repository: line, but use the value https://upload.
pypi.org/legacy/ instead:

[distutils]
index-servers =
pypi

[pypi]
repository = https://upload.pypi.org/legacy/
username = <your PyPI username>
password = <your PyPI password>

(legacy in this URL refers to the fact that this is the new server implementation’s emulation of the legacy server
implementation’s upload API.)
For more details, see the specification for .pypirc.

3.19.2 Registering package names & metadata

Explicit pre-registration of package names with the setup.py register command prior to the first upload is no
longer required, and is not currently supported by the legacy upload API emulation on PyPI.org.

80 Chapter 3. Guides
 Python Packaging User Guide

As a result, attempting explicit registration after switching to using PyPI.org for uploads will give the following error
message:

Server response (410): This API is no longer supported, instead simply upload the
˓→file.

The solution is to skip the registration step, and proceed directly to uploading artifacts.

3.19.3 Using TestPyPI

Legacy TestPyPI (testpypi.python.org) is no longer available; use test.pypi.org instead. If you use TestPyPI, you must
update your $HOME/.pypirc to handle TestPyPI’s new location, by replacing https://testpypi.python.
org/pypi with https://test.pypi.org/legacy/, for example:

[distutils]
index-servers=
pypi
testpypi

[testpypi]
repository = https://test.pypi.org/legacy/
username = <your TestPyPI username>
password = <your TestPyPI password>

For more details, see the specification for .pypirc.

3.19.4 Registering new user accounts

In order to help mitigate spam attacks against PyPI, new user registration through pypi.python.org was switched
off on February 20, 2018. New user registrations at pypi.org are open.

3.19.5 Browsing packages

While pypi.python.org is may still be used in links from other PyPA documentation, etc, the default interface
for browsing packages is pypi.org. The domain pypi.python.org now redirects to pypi.org, and may be disabled
sometime in the future.

3.19.6 Downloading packages

pypi.org is the default host for downloading packages.

3.19.7 Managing published packages and releases

pypi.org provides a fully functional interface for logged in users to manage their published packages and releases.

3.20 Using TestPyPI

TestPyPI is a separate instance of the Python Package Index (PyPI) that allows you to try out the distribution tools and
process without worrying about affecting the real index. TestPyPI is hosted at test.pypi.org

3.20. Using TestPyPI 81

Python Packaging User Guide

3.20.1 Registering your account

Because TestPyPI has a separate database from the live PyPI, you’ll need a separate user account for specifically for
TestPyPI. Go to https://test.pypi.org/account/register/ to register your account.

Note: The database for TestPyPI may be periodically pruned, so it is not unusual for user accounts to be deleted.

3.20.2 Using TestPyPI with Twine

You can upload your distributions to TestPyPI using twine by specifying the --repository flag

$ twine upload --repository testpypi dist/*

You can see if your package has successfully uploaded by navigating to the URL https://test.pypi.org/
project/<sampleproject> where sampleproject is the name of your project that you uploaded. It may
take a minute or two for your project to appear on the site.

3.20.3 Using TestPyPI with pip

You can tell pip to download packages from TestPyPI instead of PyPI by specifying the --index-url flag

$ pip install --index-url https://test.pypi.org/simple/ your-package

If you want to allow pip to also pull other packages from PyPI you can specify --extra-index-url to point to
PyPI. This is useful when the package you’re testing has dependencies:

pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.

˓→org/simple your-package

3.20.4 Setting up TestPyPI in .pypirc

If you want to avoid entering your username, you can configure TestPyPI in your $HOME/.pypirc:

[testpypi]
username = <your TestPyPI username>

For more details, see the specification for .pypirc.

3.21 Making a PyPI-friendly README

README files can help your users understand your project and can be used to set your project’s description on PyPI.
This guide helps you create a README in a PyPI-friendly format and include your README in your package so it
appears on PyPI.

3.21.1 Creating a README file

README files for Python projects are often named README, README.txt, README.rst, or README.md.

82 Chapter 3. Guides
 Python Packaging User Guide

For your README to display properly on PyPI, choose a markup language supported by PyPI. Formats supported by
PyPI’s README renderer are:
• plain text
• reStructuredText (without Sphinx extensions)
• Markdown (GitHub Flavored Markdown by default, or CommonMark)
It’s customary to save your README file in the root of your project, in the same directory as your setup.py file.

3.21.2 Including your README in your package’s metadata

To include your README’s contents as your package description, set your project’s Description and
Description-Content-Type metadata, typically in your project’s setup.py file.
See also:
• Description
• Description-Content-Type
For example, to set these values in a package’s setup.py file, use setup()’s long_description and
long_description_content_type.
Set the value of long_description to the contents (not the path) of the README file itself. Set the
long_description_content_type to an accepted Content-Type-style value for your README file’s
markup, such as text/plain, text/x-rst (for reStructuredText), or text/markdown.

Note: If you’re using GitHub-flavored Markdown to write a project’s description, ensure you upgrade the following
tools:
python3 -m pip install --user --upgrade setuptools wheel twine

The minimum required versions of the respective tools are:

• setuptools >= 38.6.0
• wheel >= 0.31.0
• twine >= 1.11.0
It’s recommended that you use twine to upload the project’s distribution packages:
twine upload dist/*

For example, see this setup.py file, which reads the contents of README.md as long_description and iden-
tifies the markup as GitHub-flavored Markdown:
from setuptools import setup

# read the contents of your README file

from os import path
this_directory = path.abspath(path.dirname(__file__))
with open(path.join(this_directory, 'README.md'), encoding='utf-8') as f:
long_description = f.read()

setup(
name='an_example_package',
(continues on next page)

3.21. Making a PyPI-friendly README 83

Python Packaging User Guide

(continued from previous page)

# other arguments omitted
long_description=long_description,
long_description_content_type='text/markdown'
)

3.21.3 Validating reStructuredText markup

If your README is written in reStructuredText, any invalid markup will prevent it from rendering, causing PyPI to
instead just show the README’s raw source.
Note that Sphinx extensions used in docstrings, such as directives and roles (e.g., “:py:func:`getattr`”
or “:ref:`my-reference-label`”), are not allowed here and will result in error messages like “Error:
Unknown interpreted text role "py:func".”.
You can check your README for markup errors before uploading as follows:
1. Install the latest version of twine; version 1.12.0 or higher is required:

pip install --upgrade twine

2. Build the sdist and wheel for your project as described under Packaging your project.
3. Run twine check on the sdist and wheel:

twine check dist/*

This command will report any problems rendering your README. If your markup renders fine, the command
will output Checking distribution FILENAME: Passed.

3.22 Publishing package distribution releases using GitHub Actions

CI/CD workflows

GitHub Actions CI/CD allows you to run a series of commands whenever an event occurs on the GitHub platform.
One popular choice is having a workflow that’s triggered by a push event. This guide shows you how to publish a
Python distribution whenever a tagged commit is pushed. It will use the pypa/gh-action-pypi-publish GitHub Action.

Attention: This guide assumes that you already have a project that you know how to build distributions for and it
lives on GitHub.

3.22.1 Saving credentials on GitHub

In this guide, we’ll demonstrate uploading to both PyPI and TestPyPI, meaning that we’ll have two separate sets of
credentials. And we’ll need to save them in the GitHub repository settings.
Let’s begin!
1. Go to https://pypi.org/manage/account/#api-tokens and create a new API token. If you have the project on PyPI
already, limit the token scope to just that project. You can call it something like GitHub Actions CI/CD
-- project-org/project-repo in order for it to be easily distinguishable in the token list. Don’t close
the page just yet — you won’t see that token again.

84 Chapter 3. Guides
 Python Packaging User Guide

2. In a separate browser tab or window, go to the Settings tab of your target repository and then click on Secrets
in the left sidebar.
3. Create a new secret called pypi_password and copy-paste the token from the fist step.
4. Now, go to https://test.pypi.org/manage/account/#api-tokens and repeat the steps. Save that TestPyPI token on
GitHub as test_pypi_password.

Attention: If you don’t have a TestPyPI account, you’ll need to create it. It’s not the same as a regular PyPI
account.

3.22.2 Creating a workflow definition

GitHub CI/CD workflows are declared in YAML files stored in the .github/workflows/ directory of your repos-
itory.
Let’s create a .github/workflows/publish-to-test-pypi.yml file.
Start it with a meaningful name and define the event that should make GitHub run this workflow:

name: Publish Python distributions to PyPI and TestPyPI

on: push

3.22.3 Defining a workflow job environment

Now, let’s add initial setup for our job. It’s a process that will execute commands that we’ll define later. In this guide,
we’ll use Ubuntu 18.04:

jobs:
build-n-publish:
name: Build and publish Python distributions to PyPI and TestPyPI
runs-on: ubuntu-18.04

3.22.4 Checking out the project and building distributions

Then, add the following under the build-n-publish section:

steps:
- uses: actions/checkout@master
- name: Set up Python 3.7
uses: actions/setup-python@v1
with:
python-version: 3.7

This will download your repository into the CI runner and then install and activate Python 3.7.
And now we can build dists from source. In this example, we’ll use pep517 package, assuming that your project has
a pyproject.toml properly set up (see PEP 517/PEP 518).

Tip: You can use any other method for building distributions as long as it produces ready-to-upload artifacts saved
into the dist/ folder.

3.22. Publishing package distribution releases using GitHub Actions CI/CD workflows 85
Python Packaging User Guide

So add this to the steps list:

- name: Install pep517

run: >-
python -m
pip install
pep517
--user
- name: Build a binary wheel and a source tarball
run: >-
python -m
pep517.build
--source
--binary
--out-dir dist/
.

3.22.5 Publishing the distribution to PyPI and TestPyPI

Finally, add the following steps at the end:

- name: Publish distribution to Test PyPI

uses: pypa/gh-action-pypi-publish@master
with:
password: ${{ secrets.test_pypi_password }}
repository_url: https://test.pypi.org/legacy/
- name: Publish distribution to PyPI
if: startsWith(github.event.ref, 'refs/tags')
uses: pypa/gh-action-pypi-publish@master
with:
password: ${{ secrets.pypi_password }}

These two steps use the pypa/gh-action-pypi-publish GitHub Action: the first one uploads contents of the dist/
folder into TestPyPI unconditionally and the second does that to PyPI, but only if the current commit is tagged.

3.22.6 That’s all, folks!

Now, whenever you push a tagged commit to your Git repository remote on GitHub, this workflow will publish it to
PyPI. And it’ll publish any push to TestPyPI which is useful for providing test builds to your alpha users as well as
making sure that your release pipeline remains healthy!

86 Chapter 3. Guides
 CHAPTER

FOUR

DISCUSSIONS

Discussions are focused on providing comprehensive information about a specific topic. If you’re just trying to get
stuff done, see Guides.

4.1 Deploying Python applications

Page Status Incomplete

Last Reviewed 2014-11-11

Contents

• Overview
– Supporting multiple hardware platforms
• OS packaging & installers
– Windows

* Pynsist
• Application bundles
• Configuration management

4.1.1 Overview

Supporting multiple hardware platforms

FIXME

Meaning: x86, x64, ARM, others?

For Python-only distributions, it *should* be straightforward to deploy on all

platforms where Python can run.

For distributions with binary extensions, deployment is major headache. Not only
must the extensions be built on all the combinations of operating system and
hardware platform, but they must also be tested, preferably on continuous
integration platforms. The issues are similar to the "multiple Python
(continues on next page)

87
Python Packaging User Guide

(continued from previous page)

versions" section above, not sure whether this should be a separate section.
Even on Windows x64, both the 32 bit and 64 bit versions of Python enjoy
significant usage.

4.1.2 OS packaging & installers

FIXME

- Building rpm/debs for projects

- Building rpms/debs for whole virtualenvs
- Building macOS installers for Python projects
- Building Android APKs with Kivy+P4A or P4A & Buildozer

Windows

FIXME

- Building Windows installers for Python projects

Pynsist

Pynsist is a tool that bundles Python programs together with the Python-interpreter into a single installer based on
NSIS. In most cases, packaging only requires the user to choose a version of the Python-interpreter and declare the
dependencies of the program. The tool downloads the specified Python-interpreter for Windows and packages it with
all the dependencies in a single Windows-executable installer.
The installed program can be started from a shortcut that the installer adds to the start-menu. It uses a Python interpreter
installed within its application directory, independent of any other Python installation on the computer.
A big advantage of Pynsist is that the Windows packages can be built on Linux. There are several examples for
different kinds of programs (console, GUI) in the documentation. The tool is released under the MIT-licence.

4.1.3 Application bundles

FIXME

- py2exe/py2app/PEX
- wheels kinda/sorta

4.1.4 Configuration management

FIXME

puppet
salt
chef
ansible
fabric

88 Chapter 4. Discussions
 Python Packaging User Guide

4.2 pip vs easy_install

easy_install was released in 2004, as part of setuptools. It was notable at the time for installing packages from PyPI
using requirement specifiers, and automatically installing dependencies.
pip came later in 2008, as alternative to easy_install, although still largely built on top of setuptools components. It
was notable at the time for not installing packages as Eggs or from Eggs (but rather simply as ‘flat’ packages from
sdists), and introducing the idea of Requirements Files, which gave users the power to easily replicate environments.
Here’s a breakdown of the important differences between pip and easy_install now:

pip easy_install
Installs from Wheels Yes No
Uninstall Packages Yes (pip uninstall) No
Dependency Overrides Yes (Requirements Files) No
List Installed Packages Yes (pip list and pip freeze) No
Yes No
PEP 438 Support
Installation format ‘Flat’ packages with egg-info metadata. Encapsulated Egg format
sys.path modification No Yes
Installs from Eggs No Yes
pylauncher support No Yes1
Multi-version installs No Yes
Exclude scripts during install No Yes
per project index Only in virtualenv Yes, via setup.cfg

4.3 install_requires vs requirements files

Contents

• install_requires
• Requirements files

4.3.1 install_requires

install_requires is a setuptools setup.py keyword that should be used to specify what a project minimally
needs to run correctly. When the project is installed by pip, this is the specification that is used to install its dependen-
cies.
For example, if the project requires A and B, your install_requires would be like so:

install_requires=[
'A',
'B'
]

1 https://setuptools.readthedocs.io/en/latest/easy_install.html#natural-script-launcher

4.2. pip vs easy_install 89

Python Packaging User Guide

Additionally, it’s best practice to indicate any known lower or upper bounds.
For example, it may be known, that your project requires at least v1 of ‘A’, and v2 of ‘B’, so it would be like so:

install_requires=[
'A>=1',
'B>=2'
]

It may also be known that project A follows semantic versioning, and that v2 of ‘A’ will indicate a break in compati-
bility, so it makes sense to not allow v2:

install_requires=[
'A>=1,<2',
'B>=2'
]

It is not considered best practice to use install_requires to pin dependencies to specific versions, or to specify
sub-dependencies (i.e. dependencies of your dependencies). This is overly-restrictive, and prevents the user from
gaining the benefit of dependency upgrades.
Lastly, it’s important to understand that install_requires is a listing of “Abstract” requirements, i.e just names
and version restrictions that don’t determine where the dependencies will be fulfilled from (i.e. from what index or
source). The where (i.e. how they are to be made “Concrete”) is to be determined at install time using pip options.1

4.3.2 Requirements files

Requirements Files described most simply, are just a list of pip install arguments placed into a file.
Whereas install_requires defines the dependencies for a single project, Requirements Files are often used to
define the requirements for a complete Python environment.
Whereas install_requires requirements are minimal, requirements files often contain an exhaustive listing of
pinned versions for the purpose of achieving repeatable installations of a complete environment.
Whereas install_requires requirements are “Abstract”, i.e. not associated with any particular index, require-
ments files often contain pip options like --index-url or --find-links to make requirements “Concrete”, i.e.
associated with a particular index or directory of packages.1
Whereas install_requires metadata is automatically analyzed by pip during an install, requirements files are
not, and only are used when a user specifically installs them using pip install -r.

4.4 Wheel vs Egg

Wheel and Egg are both packaging formats that aim to support the use case of needing an install artifact that doesn’t
require building or compilation, which can be costly in testing and production workflows.
The Egg format was introduced by setuptools in 2004, whereas the Wheel format was introduced by PEP 427 in 2012.
Wheel is currently considered the standard for built and binary packaging for Python.
Here’s a breakdown of the important differences between Wheel and Egg.
• Wheel has an official PEP. Egg did not.
1 For more on “Abstract” vs “Concrete” requirements, see https://caremad.io/2013/07/setup-vs-requirement/.

90 Chapter 4. Discussions
 Python Packaging User Guide

• Wheel is a distribution format, i.e a packaging format.1 Egg was both a distribution format and a runtime
installation format (if left zipped), and was designed to be importable.
• Wheel archives do not include .pyc files. Therefore, when the distribution only contains Python files (i.e. no
compiled extensions), and is compatible with Python 2 and 3, it’s possible for a wheel to be “universal”, similar
to an sdist.
• Wheel uses PEP376-compliant .dist-info directories. Egg used .egg-info.
• Wheel has a richer file naming convention. A single wheel archive can indicate its compatibility with a number
of Python language versions and implementations, ABIs, and system architectures.
• Wheel is versioned. Every wheel file contains the version of the wheel specification and the implementation that
packaged it.
• Wheel is internally organized by sysconfig path type, therefore making it easier to convert to other formats.

1 Circumstantially, in some cases, wheels can be used as an importable runtime format, although this is not officially supported at this time.

4.4. Wheel vs Egg 91

Python Packaging User Guide

92 Chapter 4. Discussions
 CHAPTER

FIVE

PYPA SPECIFICATIONS

This is a list of currently active interoperability specifications maintained by the Python Packaging Authority. The
process for updating these standards, and for proposing new ones, is documented on pypa.io.

5.1 Package Distribution Metadata

5.1.1 Core metadata specifications

The current core metadata file format, version 2.1, is specified in PEP 566. It defines the following specification as
the canonical source for the core metadata file format.
Fields defined in the following specification should be considered valid, complete and not subject to change. The
required fields are:
• Metadata-Version
• Name
• Version
All the other fields are optional.

Note: Interpreting old metadata: In PEP 566, the version specifier field format specification was relaxed to accept the
syntax used by popular publishing tools (namely to remove the requirement that version specifiers must be surrounded
by parentheses). Metadata consumers may want to use the more relaxed formatting rules even for metadata files that
are nominally less than version 2.1.

Contents

• Metadata-Version
• Name
• Version
• Platform (multiple use)
• Supported-Platform (multiple use)
• Summary
• Description

93
Python Packaging User Guide

• Description-Content-Type
• Keywords
• Home-page
• Download-URL
• Author
• Author-email
• Maintainer
• Maintainer-email
• License
• Classifier (multiple use)
• Requires-Dist (multiple use)
• Requires-Python
• Requires-External (multiple use)
• Project-URL (multiple-use)
• Provides-Extra (multiple use)
• Rarely Used Fields
– Provides-Dist (multiple use)
– Obsoletes-Dist (multiple use)

Metadata-Version

New in version 1.0.

Version of the file format; legal values are “1.0”, “1.1”, “1.2” and “2.1”.
Automated tools consuming metadata SHOULD warn if metadata_version is greater than the highest version
they support, and MUST fail if metadata_version has a greater major version than the highest version they
support (as described in PEP 440, the major version is the value before the first dot).
For broader compatibility, build tools MAY choose to produce distribution metadata using the lowest metadata version
that includes all of the needed fields.
Example:

Metadata-Version: 2.1

Name

New in version 1.0.

Changed in version 2.1: Added additional restrictions on format from PEP 508
The name of the distribution. The name field is the primary identifier for a distribution. A valid name consists only of
ASCII letters and numbers, period, underscore and hyphen. It must start and end with a letter or number. Distribution
names are limited to those which match the following regex (run with re.IGNORECASE):

94 Chapter 5. PyPA specifications

 Python Packaging User Guide

^([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])$

Example:

Name: BeagleVote

Version

New in version 1.0.

A string containing the distribution’s version number. This field must be in the format specified in PEP 440.
Example:

Version: 1.0a2

Platform (multiple use)

New in version 1.0.

A Platform specification describing an operating system supported by the distribution which is not listed in the “Op-
erating System” Trove classifiers. See “Classifier” below.
Examples:

Platform: ObscureUnix
Platform: RareDOS

Supported-Platform (multiple use)

New in version 1.1.

Binary distributions containing a PKG-INFO file will use the Supported-Platform field in their metadata to specify the
OS and CPU for which the binary distribution was compiled. The semantics of the Supported-Platform field are not
specified in this PEP.
Example:

Supported-Platform: RedHat 7.2

Supported-Platform: i386-win32-2791

Summary

New in version 1.0.

A one-line summary of what the distribution does.
Example:

Summary: A module for collecting votes from beagles.

5.1. Package Distribution Metadata 95

Python Packaging User Guide

Description

New in version 1.0.

Changed in version 2.1: This field may be specified in the message body instead.
A longer description of the distribution that can run to several paragraphs. Software that deals with metadata should not
assume any maximum size for this field, though people shouldn’t include their instruction manual as the description.
The contents of this field can be written using reStructuredText markup1 . For programs that work with the metadata,
supporting markup is optional; programs can also display the contents of the field as-is. This means that authors should
be conservative in the markup they use.
To support empty lines and lines with indentation with respect to the RFC 822 format, any CRLF character has to be
suffixed by 7 spaces followed by a pipe (“|”) char. As a result, the Description field is encoded into a folded field that
can be interpreted by RFC822 parser2 .
Example:

Description: This project provides powerful math functions

|For example, you can use `sum()` to sum numbers:
|
|Example::
|
| >>> sum(1, 2)
| 3
|

This encoding implies that any occurrences of a CRLF followed by 7 spaces and a pipe char have to be replaced by a
single CRLF when the field is unfolded using a RFC822 reader.
Alternatively, the distribution’s description may instead be provided in the message body (i.e., after a completely blank
line following the headers, with no indentation or other special formatting necessary).

Description-Content-Type

New in version 2.1.

A string stating the markup syntax (if any) used in the distribution’s description, so that tools can intelligently render
the description.
Historically, PyPI supported descriptions in plain text and reStructuredText (reST), and could render reST into HTML.
However, it is common for distribution authors to write the description in Markdown (RFC 7763) as many code hosting
sites render Markdown READMEs, and authors would reuse the file for the description. PyPI didn’t recognize the
format and so could not render the description correctly. This resulted in many packages on PyPI with poorly-rendered
descriptions when Markdown is left as plain text, or worse, was attempted to be rendered as reST. This field allows
the distribution author to specify the format of their description, opening up the possibility for PyPI and other tools to
be able to render Markdown and other formats.
The format of this field is the same as the Content-Type header in HTTP (i.e.: RFC 1341). Briefly, this means that
it has a type/subtype part and then it can optionally have a number of parameters:
Format:

Description-Content-Type: <type>/<subtype>; charset=<charset>[; <param_name>=<param

˓→value> ...]

1 reStructuredText markup: http://docutils.sourceforge.net/

2 RFC 822 Long Header Fields: http://www.freesoft.org/CIE/RFC/822/7.htm

96 Chapter 5. PyPA specifications

 Python Packaging User Guide

The type/subtype part has only a few legal values:

• text/plain
• text/x-rst
• text/markdown
The charset parameter can be used to specify the character encoding of the description. The only legal value is
UTF-8. If omitted, it is assumed to be UTF-8.
Other parameters might be specific to the chosen subtype. For example, for the markdown subtype, there is an
optional variant parameter that allows specifying the variant of Markdown in use (defaults to GFM if not specified).
Currently, two variants are recognized:
• GFM for Github-flavored Markdown
• CommonMark for CommonMark
Example:

Description-Content-Type: text/plain; charset=UTF-8

Example:

Description-Content-Type: text/x-rst; charset=UTF-8

Example:

Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM

Example:

Description-Content-Type: text/markdown

If a Description-Content-Type is not specified, then applications should attempt to render it as text/

x-rst; charset=UTF-8 and fall back to text/plain if it is not valid rst.
If a Description-Content-Type is an unrecognized value, then the assumed content type is text/plain
(Although PyPI will probably reject anything with an unrecognized value).
If the Description-Content-Type is text/markdown and variant is not specified or is set to an unrec-
ognized value, then the assumed variant is GFM.
So for the last example above, the charset defaults to UTF-8 and the variant defaults to GFM and thus it is
equivalent to the example before it.

Keywords

New in version 1.0.

A list of additional keywords, separated by commas, to be used to assist searching for the distribution in a larger
catalog.
Example:

Keywords: dog,puppy,voting,election

5.1. Package Distribution Metadata 97

Python Packaging User Guide

Note: The specification previously showed keywords separated by spaces, but distutils and setuptools implemented
it with commas. These tools have been very widely used for many years, so it was easier to update the specification to
match the de facto standard.

Home-page

New in version 1.0.

A string containing the URL for the distribution’s home page.
Example:

Home-page: http://www.example.com/~cschultz/bvote/

Download-URL

New in version 1.1.

A string containing the URL from which this version of the distribution can be downloaded. (This means that the URL
can’t be something like “. . . /BeagleVote-latest.tgz”, but instead must be “. . . /BeagleVote-0.45.tgz”.)

Author

New in version 1.0.

A string containing the author’s name at a minimum; additional contact information may be provided.
Example:

Author: C. Schultz, Universal Features Syndicate,

Los Angeles, CA <[email protected]>

Author-email

New in version 1.0.

A string containing the author’s e-mail address. It can contain a name and e-mail address in the legal forms for a
RFC-822 From: header.
Example:

Author-email: "C. Schultz" <[email protected]>

Per RFC-822, this field may contain multiple comma-separated e-mail addresses:

Author-email: [email protected], [email protected]

98 Chapter 5. PyPA specifications

 Python Packaging User Guide

Maintainer

New in version 1.2.

A string containing the maintainer’s name at a minimum; additional contact information may be provided.
Note that this field is intended for use when a project is being maintained by someone other than the original author:
it should be omitted if it is identical to Author.
Example:
Maintainer: C. Schultz, Universal Features Syndicate,
Los Angeles, CA <[email protected]>

Maintainer-email

New in version 1.2.

A string containing the maintainer’s e-mail address. It can contain a name and e-mail address in the legal forms for a
RFC-822 From: header.
Note that this field is intended for use when a project is being maintained by someone other than the original author:
it should be omitted if it is identical to Author-email.
Example:
Maintainer-email: "C. Schultz" <[email protected]>

Per RFC-822, this field may contain multiple comma-separated e-mail addresses:
Maintainer-email: [email protected], [email protected]

License

New in version 1.0.

Text indicating the license covering the distribution where the license is not a selection from the “License” Trove
classifiers. See “Classifier” below. This field may also be used to specify a particular version of a license which is
named via the Classifier field, or to indicate a variation or exception to such a license.
Examples:
License: This software may only be obtained by sending the
author a postcard, and then the user promises not
to redistribute it.

License: GPL version 3, excluding DRM provisions

Classifier (multiple use)

New in version 1.1.

Each entry is a string giving a single classification value for the distribution. Classifiers are described in PEP 301, and
the Python Package Index publishes a dynamic list of currently defined classifiers.
This field may be followed by an environment marker after a semicolon.
Examples:

5.1. Package Distribution Metadata 99

Python Packaging User Guide

Classifier: Development Status :: 4 - Beta

Classifier: Environment :: Console (Text Based)

Requires-Dist (multiple use)

New in version 1.2.

Changed in version 2.1: The field format specification was relaxed to accept the syntax used by popular publishing
tools.
Each entry contains a string naming some other distutils project required by this distribution.
The format of a requirement string contains from one to four parts:
• A project name, in the same format as the Name: field. The only mandatory part.
• A comma-separated list of ‘extra’ names. These are defined by the required project, referring to specific features
which may need extra dependencies.
• A version specifier. Tools parsing the format should accept optional parentheses around this, but tools generating
it should not use parentheses.
• An environment marker after a semicolon. This means that the requirement is only needed in the specified
conditions.
See PEP 508 for full details of the allowed format.
The project names should correspond to names as found on the Python Package Index.
Version specifiers must follow the rules described in Version specifiers.
Examples:

Requires-Dist: pkginfo
Requires-Dist: PasteDeploy
Requires-Dist: zope.interface (>3.5.0)
Requires-Dist: pywin32 >1.0; sys_platform == 'win32'

Requires-Python

New in version 1.2.

This field specifies the Python version(s) that the distribution is guaranteed to be compatible with. Installation tools
may look at this when picking which version of a project to install.
The value must be in the format specified in Version specifiers.
This field may be followed by an environment marker after a semicolon.
Examples:

Requires-Python: >=3
Requires-Python: >2.6,!=3.0.*,!=3.1.*
Requires-Python: ~=2.6
Requires-Python: >=3; sys_platform == 'win32'

100 Chapter 5. PyPA specifications

 Python Packaging User Guide

Requires-External (multiple use)

New in version 1.2.

Changed in version 2.1: The field format specification was relaxed to accept the syntax used by popular publishing
tools.
Each entry contains a string describing some dependency in the system that the distribution is to be used. This field
is intended to serve as a hint to downstream project maintainers, and has no semantics which are meaningful to the
distutils distribution.
The format of a requirement string is a name of an external dependency, optionally followed by a version declaration
within parentheses.
This field may be followed by an environment marker after a semicolon.
Because they refer to non-Python software releases, version numbers for this field are not required to conform to the
format specified in PEP 440: they should correspond to the version scheme used by the external dependency.
Notice that there is no particular rule on the strings to be used.
Examples:

Requires-External: C
Requires-External: libpng (>=1.5)
Requires-External: make; sys_platform != "win32"

Project-URL (multiple-use)

New in version 1.2.

A string containing a browsable URL for the project and a label for it, separated by a comma.
Example:

Project-URL: Bug Tracker, http://bitbucket.org/tarek/distribute/issues/

The label is a free text limited to 32 signs.

Provides-Extra (multiple use)

New in version 2.1.

A string containing the name of an optional feature. Must be a valid Python identifier. May be used to make a
dependency conditional on whether the optional feature has been requested.
Example:

Provides-Extra: pdf
Requires-Dist: reportlab; extra == 'pdf'

A second distribution requires an optional dependency by placing it inside square brackets, and can request multiple
features by separating them with a comma (,). The requirements are evaluated for each requested feature and added to
the set of requirements for the distribution.
Example:

Requires-Dist: beaglevote[pdf]
Requires-Dist: libexample[test, doc]

5.1. Package Distribution Metadata 101

Python Packaging User Guide

Two feature names test and doc are reserved to mark dependencies that are needed for running automated tests and
generating documentation, respectively.
It is legal to specify Provides-Extra: without referencing it in any Requires-Dist:.

Rarely Used Fields

The fields in this section are currently rarely used, as their design was inspired by comparable mechanisms in Linux
package management systems, and it isn’t at all clear how tools should interpret them in the context of an open index
server such as PyPI.
As a result, popular installation tools ignore them completely, which in turn means there is little incentive for package
publishers to set them appropriately. However, they’re retained in the metadata specification, as they’re still potentially
useful for informational purposes, and can also be used for their originally intended purpose in combination with a
curated package repository.

Provides-Dist (multiple use)

New in version 1.2.

Changed in version 2.1: The field format specification was relaxed to accept the syntax used by popular publishing
tools.
Each entry contains a string naming a Distutils project which is contained within this distribution. This field must
include the project identified in the Name field, followed by the version : Name (Version).
A distribution may provide additional names, e.g. to indicate that multiple projects have been bundled together. For
instance, source distributions of the ZODB project have historically included the transaction project, which is
now available as a separate distribution. Installing such a source distribution satisfies requirements for both ZODB and
transaction.
A distribution may also provide a “virtual” project name, which does not correspond to any separately-distributed
project: such a name might be used to indicate an abstract capability which could be supplied by one of multiple
projects. E.g., multiple projects might supply RDBMS bindings for use by a given ORM: each project might declare
that it provides ORM-bindings, allowing other projects to depend only on having at most one of them installed.
A version declaration may be supplied and must follow the rules described in Version specifiers. The distribution’s
version number will be implied if none is specified.
This field may be followed by an environment marker after a semicolon.
Examples:

Provides-Dist: OtherProject
Provides-Dist: AnotherProject (3.4)
Provides-Dist: virtual_package; python_version >= "3.4"

Obsoletes-Dist (multiple use)

New in version 1.2.

Changed in version 2.1: The field format specification was relaxed to accept the syntax used by popular publishing
tools.
Each entry contains a string describing a distutils project’s distribution which this distribution renders obsolete, mean-
ing that the two projects should not be installed at the same time.

102 Chapter 5. PyPA specifications

 Python Packaging User Guide

Version declarations can be supplied. Version numbers must be in the format specified in Version specifiers.
This field may be followed by an environment marker after a semicolon.
The most common use of this field will be in case a project name changes, e.g. Gorgon 2.3 gets subsumed into Torqued
Python 1.0. When you install Torqued Python, the Gorgon distribution should be removed.
Examples:

Obsoletes-Dist: Gorgon
Obsoletes-Dist: OtherProject (<3.0)
Obsoletes-Dist: Foo; os_name == "posix"

5.1.2 Version specifiers

Version numbering requirements and the semantics for specifying comparisons between versions are defined in PEP
440.
The version specifiers section in this PEP supersedes the version specifiers section in PEP 345.

5.1.3 Dependency specifiers

The dependency specifier format used to declare a dependency on another component is defined in PEP 508.
The environment markers section in this PEP supersedes the environment markers section in PEP 345.

5.1.4 Declaring build system dependencies

pyproject.toml is a build system independent file format defined in PEP 518 that projects may provide in order to
declare any Python level dependencies that must be installed in order to run the project’s build system successfully.

5.1.5 Distribution formats

Source distribution format

The accepted style of source distribution format based on pyproject.toml, defined in PEP 518 and adopted by
PEP 517 has not been implemented yet.
There is also the legacy source distribution format, implicitly defined by the behaviour of distutils module in the
standard library, when executing setup.py sdist.

Binary distribution format

The binary distribution format (wheel) is defined in PEP 427.

5.1.6 Platform compatibility tags

Platform compatibility tags allow build tools to mark distributions as being compatible with specific platforms, and
allows installers to understand which distributions are compatible with the system they are running on.
The platform compatibility tagging model used for the wheel distribution format is defined in PEP 425.

5.1. Package Distribution Metadata 103

Python Packaging User Guide

Platform tags for Windows

The scheme defined in PEP 425 covers public distribution of wheel files to systems running Windows.

Platform tags for macOS (Mac OS X)

The scheme defined in PEP 425 covers public distribution of wheel files to systems running macOS (previously known
as Mac OS X).

Platform tags for common Linux distributions

The scheme defined in PEP 425 is insufficient for public distribution of wheel files (and *nix wheel files in general)
to Linux platforms, due to the large ecosystem of Linux platforms and subtle differences between them.
Instead, PEP 513 defines the manylinux standard, which represents a common subset of Linux platforms, and
allows building wheels tagged with the manylinux platform tag which can be used across most common Linux
distributions.
There are multiple iterations of the manylinux specification, each representing the common subset of Linux plat-
forms at a given point in time:
• manylinux1 (PEP 513) supports x86_64 and i686 architectures, and is based on a compatible Linux
platform from 2007.
• manylinux2010 (PEP 571) supports x86_64 and i686 architectures. and updates the previous specifica-
tion to be based on a compatible Linux platform from 2010 instead.
• manylinux2014 (PEP 599) adds support for a number of additional architectures (aarch64, armv7l,
ppc64, ppc64le, and s390x) and updates the base platform to a compatible Linux platform from 2014.
In general, distributions built for older versions of the specification are forwards-compatible (meaning that
manylinux1 distributions should continue to work on modern systems) but not backwards-compatible (meaning
that manylinux2010 distributions are not expected to work on platforms that existed before 2010).
Package maintainers should attempt to target the most compatible specification possible, with the caveat that the pro-
vided build environment for manylinux1 has reached end-of-life, and the build environment for manylinux2010
will reach end-of-life in November 20201 , meaning that these images will no longer receive security updates.

Manylinux compatibility support

Note: The manylinux2014 specification is relatively new and is not yet widely recognised by install tools.

The following table shows the minimum versions of relevant projects to support the various manylinux standards:

Tool manylinux1 manylinux2010 manylinux2014

pip >=8.1.0 >=19.0 >=19.3
auditwheel >=1.0.0 >=2.0.0 >=3.0.0
1 https://wiki.centos.org/About/Product

104 Chapter 5. PyPA specifications

 Python Packaging User Guide

Platform tags for other *nix platforms

The scheme defined in PEP 425 is not generally sufficient for public distribution of wheel files to other *nix platforms.
Efforts are currently (albeit intermittently) under way to define improved compatibility tagging schemes for AIX and
for Alpine Linux.

5.1.7 Recording installed distributions

The format used to record installed packages and their contents is defined in PEP 376.
Note that only the dist-info directory and the RECORD file format from that PEP are currently implemented in the
default packaging toolchain.

5.1.8 Entry points specification

Entry points are a mechanism for an installed distribution to advertise components it provides to be discovered and
used by other code. For example:
• Distributions can specify console_scripts entry points, each referring to a function. When pip (or another
console_scripts aware installer) installs the distribution, it will create a command-line wrapper for each entry
point.
• Applications can use entry points to load plugins; e.g. Pygments (a syntax highlighting tool) can use additional
lexers and styles from separately installed packages. For more about this, see Creating and discovering plugins.
The entry point file format was originally developed to allow packages built with setuptools to provide integration
point metadata that would be read at runtime with pkg_resources. It is now defined as a PyPA interoperabil-
ity specification in order to allow build tools other than setuptools to publish pkg_resources compatible entry
point metadata, and runtime libraries other than pkg_resources to portably read published entry point metadata
(potentially with different caching and conflict resolution strategies).

Data model

Conceptually, an entry point is defined by three required properties:

• The group that an entry point belongs to indicates what sort of object it provides. For instance, the group
console_scripts is for entry points referring to functions which can be used as a command, while
pygments.styles is the group for classes defining pygments styles. The consumer typically defines the
expected interface. To avoid clashes, consumers defining a new group should use names starting with a PyPI
name owned by the consumer project, followed by .. Group names must be one or more groups of letters,
numbers and underscores, separated by dots (regex ^\w+(\.\w+)*$).
• The name identifies this entry point within its group. The precise meaning of this is up to the consumer. For
console scripts, the name of the entry point is the command that will be used to launch it. Within a distribution,
entry point names should be unique. If different distributions provide the same name, the consumer decides
how to handle such conflicts. The name may contain any characters except =, but it cannot start or end with
any whitespace character, or start with [. For new entry points, it is recommended to use only letters, numbers,
underscores, dots and dashes (regex [\w.-]+).
• The object reference points to a Python object. It is either in the form importable.module, or
importable.module:object.attr. Each of the parts delimited by dots and the colon is a valid Python
identifier. It is intended to be looked up like this:

5.1. Package Distribution Metadata 105

Python Packaging User Guide

import importlib
modname, qualname_separator, qualname = object_ref.partition(':')
obj = importlib.import_module(modname)
if qualname_separator:
for attr in qualname.split('.'):
obj = getattr(obj, attr)

Note: Some tools call this kind of object reference by itself an ‘entry point’, for want of a better term, especially
where it points to a function to launch a program.

There is also an optional property: the extras are a set of strings identifying optional features of the distribution
providing the entry point. If these are specified, the entry point requires the dependencies of those ‘extras’. See the
metadata field Provides-Extra (multiple use).
Using extras for an entry point is no longer recommended. Consumers should support parsing them from existing
distributions, but may then ignore them. New publishing tools need not support specifying extras. The functionality of
handling extras was tied to setuptools’ model of managing ‘egg’ packages, but newer tools such as pip and virtualenv
use a different model.

File format

Entry points are defined in a file called entry_points.txt in the *.dist-info directory of the distribution.
This is the directory described in PEP 376 for installed distributions, and in PEP 427 for wheels. The file uses the
UTF-8 character encoding.
The file contents are in INI format, as read by Python’s configparser module. However, configparser treats names
as case-insensitive by default, whereas entry point names are case sensitive. A case-sensitive config parser can be made
like this:

import configparser

class CaseSensitiveConfigParser(configparser.ConfigParser):
optionxform = staticmethod(str)

The entry points file must always use = to delimit names from values (whereas configparser also allows using :).
The sections of the config file represent entry point groups, the names are names, and the values encode both the object
reference and the optional extras. If extras are used, they are a comma-separated list inside square brackets.
Within a value, readers must accept and ignore spaces (including multiple consecutive spaces) before or after the
colon, between the object reference and the left square bracket, between the extra names and the square brackets and
colons delimiting them, and after the right square bracket. The syntax for extras is formally specified as part of PEP
508 (as extras). For tools writing the file, it is recommended only to insert a space between the object reference and
the left square bracket.
For example:

[console_scripts]
foo = foomod:main
# One which depends on extras:
foobar = foomod:main_bar [bar,baz]

# pytest plugins refer to a module, so there is no ':obj'

[pytest11]
nbval = nbval.plugin

106 Chapter 5. PyPA specifications

 Python Packaging User Guide

Use for scripts

Two groups of entry points have special significance in packaging: console_scripts and gui_scripts. In
both groups, the name of the entry point should be usable as a command in a system shell after the package is
installed. The object reference points to a function which will be called with no arguments when this command is run.
The function may return an integer to be used as a process exit code, and returning None is equivalent to returning 0.
For instance, the entry point mycmd = mymod:main would create a command mycmd launching a script like this:

import sys
from mymod import main
sys.exit(main())

The difference between console_scripts and gui_scripts only affects Windows systems.
console_scripts are wrapped in a console executable, so they are attached to a console and can use
sys.stdin, sys.stdout and sys.stderr for input and output. gui_scripts are wrapped in a GUI
executable, so they can be started without a console, but cannot use standard streams unless application code redirects
them. Other platforms do not have the same distinction.
Install tools are expected to set up wrappers for both console_scripts and gui_scripts in the scripts direc-
tory of the install scheme. They are not responsible for putting this directory in the PATH environment variable which
defines where command-line tools are found.
As files are created from the names, and some filesystems are case-insensitive, packages should avoid using names in
these groups which differ only in case. The behaviour of install tools when names differ only in case is undefined.

5.2 Package Index Interfaces

5.2.1 The .pypirc file

A .pypirc file allows you to define the configuration for package indexes (referred to here as “repositories”), so that
you don’t have to enter the URL, username, or password whenever you upload a package with twine or flit.
The format (originally defined by the distutils package) is:

[distutils]
index-servers =
first-repository
second-repository

[first-repository]
repository = <first-repository URL>
username = <first-repository username>
password = <first-repository password>

[second-repository]
repository = <second-repository URL>
username = <second-repository username>
password = <second-repository password>

The distutils section defines an index-servers field that lists the name of all sections describing a repository.
Each section describing a repository defines three fields:
• repository: The URL of the repository.
• username: The registered username on the repository.

5.2. Package Index Interfaces 107

Python Packaging User Guide

• password: The password that will used to authenticate the username.

Warning: Be aware that this stores your password in plain text. For better security, consider an alternative like
keyring, setting environment variables, or providing the password on the command line.
Otherwise, set the permissions on .pypirc so that only you can view or modify it. For example, on Linux or
macOS, run:
chmod 600 ~/.pypirc

Common configurations

Note: These examples apply to twine, and projects like hatch that use it under the hood. Other projects (e.g. flit) also
use .pypirc, but with different defaults. Please refer to each project’s documentation for more details and usage
instructions.

Twine’s default configuration mimics a .pypirc with repository sections for PyPI and TestPyPI:

[distutils]
index-servers =
pypi
testpypi

[pypi]
repository = https://upload.pypi.org/legacy/

[testpypi]
repository = https://test.pypi.org/legacy/

Twine will add additional configuration from $HOME/.pypirc, the command line, and environment variables to
this default configuration.

Using a PyPI token

To set your API token for PyPI, you can create a $HOME/.pypirc similar to:

[pypi]
username = __token__
password = <PyPI token>

For TestPyPI, add a [testpypi] section, using the API token from your TestPyPI account.

Using another package index

To configure an additional repository, you’ll need to redefine the index-servers field to include the repository
name. Here is a complete example of a $HOME/.pypirc for PyPI, TestPyPI, and a private repository:

[distutils]
index-servers =
pypi
(continues on next page)

108 Chapter 5. PyPA specifications

 Python Packaging User Guide

(continued from previous page)

testpypi
private-repository

[pypi]
username = __token__
password = <PyPI token>

[testpypi]
username = __token__
password = <TestPyPI token>

[private-repository]
repository = <private-repository URL>
username = <private-repository username>
password = <private-repository password>

Warning: Instead of using the password field, consider saving your API tokens and passwords securely using
keyring (which is installed by Twine):
keyring set https://upload.pypi.org/legacy/ __token__
keyring set https://test.pypi.org/legacy/ __token__
keyring set <private-repository URL> <private-repository username>

5.2.2 Simple repository API

The current interface for querying available package versions and retrieving packages from an index server is defined
in PEP 503, with the addition of “yank” support (allowing a kind of file deletion) as defined in PEP 592.

5.2. Package Index Interfaces 109

Python Packaging User Guide

110 Chapter 5. PyPA specifications

 CHAPTER

SIX

PROJECT SUMMARIES

Summaries and links for the most relevant projects in the space of Python installation and packaging.

6.1 PyPA Projects

6.1.1 bandersnatch

Mailing list2 | Issues | GitHub | PyPI | Dev IRC:#bandersnatch

bandersnatch is a PyPI mirroring client designed to efficiently create a complete mirror of the contents of PyPI.
Organizations thus save bandwidth and latency on package downloads (especially in the context of automated tests)
and to prevent heavily loading PyPI’s Content Delivery Network (CDN).

6.1.2 distlib

Docs | Mailing list2 | Issues | Bitbucket | PyPI

distlib is a library which implements low-level functions that relate to packaging and distribution of Python soft-
ware. distlib implements several relevant PEPs (Python Enhancement Proposal standards) and is useful for de-
velopers of third-party packaging tools to make and upload binary and source distributions, achieve interoperability,
resolve dependencies, manage package resources, and do other similar functions.
Unlike the stricter packaging project (below), which specifically implements modern Python packaging interoper-
ability standards, distlib also attempts to provide reasonable fallback behaviours when asked to handle legacy
packages and metadata that predate the modern interoperability standards and fall into the subset of packages that are
incompatible with those standards.

6.1.3 packaging

Docs | Dev list | Issues | GitHub | PyPI | User IRC:#pypa | Dev IRC:#pypa-dev
Core utilities for Python packaging used by pip and setuptools.
The core utilities in the packaging library handle version handling, specifiers, markers, requirements, tags, and similar
attributes and tasks for Python packages. Most Python users rely on this library without needing to explicitly call it;
developers of the other Python packaging, distribution, and installation tools listed here often use its functionality to
parse, discover, and otherwise handle dependency attributes.
This project specifically focuses on implementing the modern Python packaging interoperability standards defined
at PyPA specifications, and will report errors for sufficiently old legacy packages that are incompatible with those
2 Multiple projects reuse the distutils-sig mailing list as their user list.

111
Python Packaging User Guide

standards. In contrast, the distlib project is a more permissive library that attempts to provide a plausible reading of
ambiguous metadata in cases where packaging will instead report on error.

6.1.4 pip

Docs | User list1 | Dev list | Issues | GitHub | PyPI | User IRC:#pypa | Dev IRC:#pypa-dev
The most popular tool for installing Python packages, and the one included with modern versions of Python.
It provides the essential core features for finding, downloading, and installing packages from PyPI and other Python
package indexes, and can be incorporated into a wide range of development workflows via its command-line interface
(CLI).

6.1.5 Pipenv

Docs | Source | Issues | PyPI

Pipenv is a project that aims to bring the best of all packaging worlds to the Python world. It harnesses Pipfile, pip,
and virtualenv into one single toolchain. It features very pretty terminal colors.
Pipenv aims to help users manage environments, dependencies, and imported packages on the command line. It also
works well on Windows (which other tools often underserve), makes and checkes file hashes, to ensure compliance
with hash-locked dependency specifiers, and eases uninstallation of packages and dependencies. It is used by Python
users and system administrators, but has been less maintained since late 2018.

6.1.6 Pipfile

Source
Pipfile and its sister Pipfile.lock are a higher-level application-centric alternative to pip’s lower-level
requirements.txt file.

6.1.7 Python Packaging User Guide

Docs | Mailing list | Issues | GitHub | User IRC:#pypa | Dev IRC:#pypa-dev

This guide!

6.1.8 readme_renderer

GitHub and docs | PyPI

readme_renderer is a library that package developers use to render their user documentation (README) files
into HTML from markup languages such as Markdown or reStructuredText. Developers call it on its own or via twine,
as part of their release management process, to check that their package descriptions will properly display on PyPI.
1 pip was created by the same developer as virtualenv, and early on adopted the virtualenv mailing list, and it’s stuck ever since.

112 Chapter 6. Project Summaries

 Python Packaging User Guide

6.1.9 setuptools

Docs | User list2 | Dev list | Issues | GitHub | PyPI | User IRC:#pypa | Dev IRC:#pypa-dev
setuptools (which includes easy_install) is a collection of enhancements to the Python distutils that allow you to
more easily build and distribute Python distributions, especially ones that have dependencies on other packages.
distribute was a fork of setuptools that was merged back into setuptools (in v0.7), thereby making setuptools the
primary choice for Python packaging.

6.1.10 trove-classifiers

Issues | GitHub | PyPI

trove-classifiers is the canonical source for classifiers on PyPI, which project maintainers use to systematically describe
their projects so that users can better find projects that match their needs on the PyPI.
The trove-classifiers package contains a list of valid classifiers and deprecated classifiers (which are paired with the
classifiers that replace them). Use this package to validate classifiers used in packages intended for uploading to PyPI.
As this list of classifiers is published as code, you can install and import it, giving you a more convenient workflow
compared to referring to the list published on PyPI. The issue tracker for the project hosts discussions on proposed
classifiers and requests for new classifiers.

6.1.11 twine

Docs | Mailing list2 | Issues | GitHub | PyPI

Twine is the primary tool developers use to upload packages to the Python Package Index or other Python package
indexes. It is a command-line program that passes program files and metadata to a web API. Developers use it because
it’s the official PyPI upload tool, it’s fast and secure, it’s maintained, and it reliably works.

6.1.12 virtualenv

Docs | User list | Dev list | Issues | GitHub | PyPI | User IRC:#pypa | Dev IRC:#pypa-dev
virtualenv is a tool which uses the command-line path environment variable to create isolated Python Virtual Envi-
ronments, much as venv does. virtualenv provides additional functionality, compared to venv, by supporting Python
2.7 and by providing convenient features for configuring, maintaining, duplicating, and troubleshooting the virtual
environments. For more information, see the section on Creating Virtual Environments.

6.1.13 Warehouse

Docs | Mailing list2 | Issues | GitHub | Dev IRC:#pypa-dev

The current codebase powering the Python Package Index (PyPI). It is hosted at pypi.org. The default source for pip
downloads.

6.1.14 wheel

Docs | Mailing list2 | Issues | GitHub | PyPI | User IRC:#pypa | Dev IRC:#pypa-dev
Primarily, the wheel project offers the bdist_wheel setuptools extension for creating wheel distributions. Addi-
tionally, it offers its own command line utility for creating and installing wheels.

6.1. PyPA Projects 113

Python Packaging User Guide

See also auditwheel, a tool that package developers use to check and fix Python packages they are making in the binary
wheel format. It provides functionality to discover dependencies, check metadata for compliance, and repair the wheel
and metadata to properly link and include external shared libraries in a package.

6.2 Non-PyPA Projects

6.2.1 bento

Docs | Mailing list | Issues | GitHub | PyPI

Bento is a packaging tool solution for Python software, targeted as an alternative to distutils, setuptools, etc. . . . Bento’s
philosophy is reproducibility, extensibility and simplicity (in that order).

6.2.2 buildout

Docs | Mailing list2 | Issues | PyPI | GitHub | IRC:#buildout

Buildout is a Python-based build system for creating, assembling and deploying applications from multiple parts, some
of which may be non-Python-based. It lets you create a buildout configuration and reproduce the same software later.

6.2.3 conda

Docs
conda is the package management tool for Anaconda Python installations. Anaconda Python is a distribution from
Anaconda, Inc specifically aimed at the scientific community, and in particular on Windows where the installation of
binary extensions is often difficult.
Conda is a completely separate tool from pip, virtualenv and wheel, but provides many of their combined features in
terms of package management, virtual environment management and deployment of binary extensions.
Conda does not install packages from PyPI and can install only from the official Anaconda repositories, or ana-
conda.org (a place for user-contributed conda packages), or a local (e.g. intranet) package server. However, note
that pip can be installed into, and work side-by-side with conda for managing distributions from PyPI. Also, conda
skeleton is a tool to make Python packages installable by conda by first fetching them from PyPI and modifying their
metadata.

6.2.4 devpi

Docs | Mailing List | Issues | PyPI

devpi features a powerful PyPI-compatible server and PyPI proxy cache with a complementary command line tool
to drive packaging, testing and release activities with Python. devpi also provides a browsable and searchable web
interface.

6.2.5 flit

Docs | Issues | PyPI

Flit provides a simple way to upload pure Python packages and modules to PyPI. It focuses on making the easy things
easy for packaging. Flit can generate a configuration file to quickly set up a simple project, build source distributions
and wheels, and upload them to PyPI.

114 Chapter 6. Project Summaries

 Python Packaging User Guide

Flit uses pyproject.toml to configure a project. Flit does not rely on tools such as setuptools to build distributions,
or twine to upload them to PyPI. Flit requires Python 3, but you can use it to distribute modules for Python 2, so long
as they can be imported on Python 3.

6.2.6 enscons

Source | Issues | PyPI

Enscons is a Python packaging tool based on SCons. It builds pip-compatible source distributions and wheels without
using distutils or setuptools, including distributions with C extensions. Enscons has a different architecture and philos-
ophy than distutils. Rather than adding build features to a Python packaging system, enscons adds Python packaging
to a general purpose build system. Enscons helps you to build sdists that can be automatically built by pip, and wheels
that are independent of enscons.

6.2.7 Hashdist

Docs | GitHub
Hashdist is a library for building non-root software distributions. Hashdist is trying to be “the Debian of choice for
cases where Debian technology doesn’t work”. The best way for Pythonistas to think about Hashdist may be a more
powerful hybrid of virtualenv and buildout. It is aimed at solving the problem of installing scientific software, and
making package distribution stateless, cached, and branchable. It is used by some researchers but has been lacking in
maintenance since 2016.

6.2.8 hatch

GitHub and Docs | PyPI

Hatch is a unified command-line tool meant to conveniently manage dependencies and environment isolation for
Python developers. Python package developers use Hatch to configure, version, specify dependencies for, and publish
packages to PyPI. Under the hood, it uses twine to upload packages to PyPI, and pip to download and install packages.

6.2.9 pex

Docs | GitHub | PyPI

pex is both a library and tool for generating .pex (Python EXecutable) files, standalone Python environments in
the spirit of virtualenv. .pex files are just carefully constructed zip files with a #!/usr/bin/env python and
special __main__.py, and are designed to make deployment of Python applications as simple as cp.

6.2.10 pipx

Docs | GitHub | PyPI

pipx is a tool to safely install and run Python CLI applications globally.

6.2.11 pip-tools

GitHub and Docs | PyPI

pip-tools is a suite of tools meant for Python system administrators and release managers who particularly want to
keep their builds deterministic yet stay up to date with new versions of their dependencies. Users can specify particular

6.2. Non-PyPA Projects 115

Python Packaging User Guide

release of their dependencies via hash, conveniently make a properly formatted list of requirements from information
in other parts of their program, update all dependencies (a feature pip currently does not provide), and create layers of
constraints for the program to obey.

6.2.12 piwheels

Website | Docs | GitHub

piwheels is a website, and software underpinning it, that fetches source code distribution packages from PyPI and
compiles them into binary wheels that are optimized for installation onto Raspberry Pi computers. pip in Raspbian is
pre-configured to use piwheels.org as an additional index to PyPI.

Warning: Note that piwheels does not yet fully support PEP 503 and thus some users have trouble
installing certain wheels; this is in progress.

6.2.13 poetry

Docs | GitHub | PyPI

poetry is a command-line tool to handle dependency installation and isolation as well as building and packaging of
Python packages. It uses pyproject.toml and provides its own dependency resolver, and, instead of depending
on the resolver functionality within pip, provides its own dependency resolver. It attempts to speed users’ experience
of installation and dependency resolution by locally caching metadata about dependencies.

6.2.14 pypiserver

Docs | GitHub | PyPI

pypiserver is a minimalist application that serves as a private Python package index within organizations, implementing
a simple API and browser interface. You can upload private packages using standard upload tools, and users can
download and install them with pip, without publishing them publicly. Organizations who use pypiserver usually
download packages both from pypiserver and from PyPI.

6.2.15 scikit-build

Docs | Mailing list | GitHub | PyPI

Scikit-build is an improved build system generator for CPython C/C++/Fortran/Cython extensions that integrates with
setuptools, wheel and pip. It internally uses cmake (available on PyPI) to provide better support for additional compil-
ers, build systems, cross compilation, and locating dependencies and their associated build requirements. To speed up
and parallelize the build of large projects, the user can install ninja (also available on PyPI).

6.2.16 shiv

Docs | GitHub | PyPI

shiv is a command line utility for building fully self contained Python zipapps as outlined in PEP 441, but with all
their dependencies included. Its primary goal is making distributing Python applications and command line tools fast
& easy.

116 Chapter 6. Project Summaries

 Python Packaging User Guide

6.2.17 Spack

Docs | GitHub | Paper | Slides

A flexible package manager designed to support multiple versions, configurations, platforms, and compilers. Spack is
like Homebrew, but packages are written in Python and parameterized to allow easy swapping of compilers, library
versions, build options, etc. Arbitrarily many versions of packages can coexist on the same system. Spack was
designed for rapidly building high performance scientific applications on clusters and supercomputers.
Spack is not in PyPI (yet), but it requires no installation and can be used immediately after cloning from GitHub.

6.2.18 zest.releaser

Docs | GitHub | PyPI

zest.releaser is a Python package release tool providing an abstraction layer on top of twine. Python developers
use zest.releaser to automate incrementing package version numbers, updating changelogs, tagging releases in
source control, and uploading new packages to PyPI.

6.3 Standard Library Projects

6.3.1 ensurepip

Docs | Issues
A package in the Python Standard Library that provides support for bootstrapping pip into an existing Python installa-
tion or virtual environment. In most cases, end users won’t use this module, but rather it will be used during the build
of the Python distribution.

6.3.2 distutils

Docs | User list2 | Issues | User IRC:#pypa | Dev IRC:#pypa-dev

The original Python packaging system, added to the standard library in Python 2.0.
Due to the challenges of maintaining a packaging system where feature updates are tightly coupled to language run-
time updates, direct usage of distutils is now actively discouraged, with setuptools being the preferred replacement.
setuptools not only provides features that plain distutils doesn’t offer (such as dependency declarations and entry point
declarations), it also provides a consistent build interface and feature set across all supported Python versions.

6.3.3 venv

Docs | Issues
A package in the Python Standard Library (starting with Python 3.3) for creating Virtual Environments. For more
information, see the section on Creating Virtual Environments.

6.3. Standard Library Projects 117

Python Packaging User Guide

118 Chapter 6. Project Summaries

 CHAPTER

SEVEN

GLOSSARY

Binary Distribution A specific kind of Built Distribution that contains compiled extensions.
Built Distribution A Distribution format containing files and metadata that only need to be moved to the correct
location on the target system, to be installed. Wheel is such a format, whereas distutil’s Source Distribution is
not, in that it requires a build step before it can be installed. This format does not imply that Python files have
to be precompiled (Wheel intentionally does not include compiled Python files).
Distribution Package A versioned archive file that contains Python packages, modules, and other resource files that
are used to distribute a Release. The archive file is what an end-user will download from the internet and install.
A distribution package is more commonly referred to with the single words “package” or “distribution”, but this
guide may use the expanded term when more clarity is needed to prevent confusion with an Import Package
(which is also commonly called a “package”) or another kind of distribution (e.g. a Linux distribution or the
Python language distribution), which are often referred to with the single term “distribution”.
Egg A Built Distribution format introduced by setuptools, which is being replaced by Wheel. For details, see The
Internal Structure of Python Eggs and Python Eggs
Extension Module A module written in the low-level language of the Python implementation: C/C++ for Python,
Java for Jython. Typically contained in a single dynamically loadable pre-compiled file, e.g. a shared object
(.so) file for Python extensions on Unix, a DLL (given the .pyd extension) for Python extensions on Windows,
or a Java class file for Jython extensions.
Known Good Set (KGS) A set of distributions at specified versions which are compatible with each other. Typically
a test suite will be run which passes all tests before a specific set of packages is declared a known good set. This
term is commonly used by frameworks and toolkits which are comprised of multiple individual distributions.
Import Package A Python module which can contain other modules or recursively, other packages.
An import package is more commonly referred to with the single word “package”, but this guide will use the
expanded term when more clarity is needed to prevent confusion with a Distribution Package which is also
commonly called a “package”.
Module The basic unit of code reusability in Python, existing in one of two types: Pure Module, or Extension Module.
Package Index A repository of distributions with a web interface to automate package discovery and consumption.
Per Project Index A private or other non-canonical Package Index indicated by a specific Project as the index pre-
ferred or required to resolve dependencies of that project.
Project A library, framework, script, plugin, application, or collection of data or other resources, or some combination
thereof that is intended to be packaged into a Distribution.
Since most projects create Distributions using distutils or setuptools, another practical way to define projects
currently is something that contains a setup.py at the root of the project src directory, where “setup.py” is the
project specification filename used by distutils and setuptools.

119
Python Packaging User Guide

Python projects must have unique names, which are registered on PyPI. Each project will then contain one or
more Releases, and each release may comprise one or more distributions.
Note that there is a strong convention to name a project after the name of the package that is imported to run
that project. However, this doesn’t have to hold true. It’s possible to install a distribution from the project ‘foo’
and have it provide a package importable only as ‘bar’.
Pure Module A module written in Python and contained in a single .py file (and possibly associated .pyc and/or .pyo
files).
Python Packaging Authority (PyPA) PyPA is a working group that maintains many of the relevant projects in
Python packaging. They maintain a site at https://www.pypa.io, host projects on GitHub and Bitbucket, and
discuss issues on the distutils-sig mailing list and the Python Discourse forum.
Python Package Index (PyPI) PyPI is the default Package Index for the Python community. It is open to all Python
developers to consume and distribute their distributions.
pypi.org pypi.org is the domain name for the Python Package Index (PyPI). It replaced the legacy index domain
name, pypi.python.org, in 2017. It is powered by Warehouse.
Release A snapshot of a Project at a particular point in time, denoted by a version identifier.
Making a release may entail the publishing of multiple Distributions. For example, if version 1.0 of a project
was released, it could be available in both a source distribution format and a Windows installer distribution
format.
Requirement A specification for a package to be installed. pip, the PYPA recommended installer, allows various
forms of specification that can all be considered a “requirement”. For more information, see the pip install
reference.
Requirement Specifier A format used by pip to install packages from a Package Index. For an EBNF diagram of
the format, see the pkg_resources.Requirement entry in the setuptools docs. For example, “foo>=1.3” is a
requirement specifier, where “foo” is the project name, and the “>=1.3” portion is the Version Specifier
Requirements File A file containing a list of Requirements that can be installed using pip. For more information, see
the pip docs on Requirements Files.
setup.py The project specification file for distutils and setuptools.
Source Archive An archive containing the raw source code for a Release, prior to creation of a Source Distribution
or Built Distribution.
Source Distribution (or “sdist”) A distribution format (usually generated using python setup.py sdist)
that provides metadata and the essential source files needed for installing by a tool like pip, or for generating a
Built Distribution.
System Package A package provided in a format native to the operating system, e.g. an rpm or dpkg file.
Version Specifier The version component of a Requirement Specifier. For example, the “>=1.3” portion of
“foo>=1.3”. PEP 440 contains a full specification of the specifiers that Python packaging currently supports.
Support for PEP440 was implemented in setuptools v8.0 and pip v6.0.
Virtual Environment An isolated Python environment that allows packages to be installed for use by a particular
application, rather than being installed system wide. For more information, see the section on Creating Virtual
Environments.
Wheel A Built Distribution format introduced by PEP 427, which is intended to replace the Egg format. Wheel is
currently supported by pip.
Working Set A collection of distributions available for importing. These are the distributions that are on the sys.path
variable. At most, one Distribution for a project is possible in a working set.

120 Chapter 7. Glossary

 CHAPTER

EIGHT

HOW TO GET SUPPORT

For support related to a specific project, see the links on the Projects page.
For something more general, or when you’re just not sure, use the distutils-sig list.

121
Python Packaging User Guide

122 Chapter 8. How to Get Support

 CHAPTER

NINE

CONTRIBUTE TO THIS GUIDE

The Python Packaging User Guide welcomes contributors! There are lots of ways to help out, including:
• Reading the guide and giving feedback
• Reviewing new contributions
• Revising existing content
• Writing new content
Most of the work on the Python Packaging User Guide takes place on the project’s GitHub repository. To get started,
check out the list of open issues and pull requests. If you’re planning to write or edit the guide, please read the style
guide.
By contributing to the Python Packaging User Guide, you’re expected to follow the Python Packaging Authority’s
Contributor Code of Conduct. Harassment, personal attacks, and other unprofessional conduct is not acceptable.

9.1 Documentation types

This project consists of four distinct documentation types with specific purposes. When proposing new additions to
the project please pick the appropriate documentation type.

9.1.1 Tutorials

Tutorials are focused on teaching the reader new concepts by accomplishing a goal. They are opinionated step-by-step
guides. They do not include extraneous warnings or information. example tutorial-style document.

9.1.2 Guides

Guides are focused on accomplishing a specific task and can assume some level of pre-requisite knowledge. These
are similar to tutorials, but have a narrow and clear focus and can provide lots of caveats and additional information as
needed. They may also discuss multiple approaches to accomplishing the task. example guide-style document.

9.1.3 Discussions

Discussions are focused on understanding and information. These explore a specific topic without a specific goal in
mind. example discussion-style document.

123
Python Packaging User Guide

9.1.4 Specifications

Specifications are reference documention focused on comprehensively documenting an agreed-upon iterface for inter-
operability between packaging tools. example specification-style document.

9.2 Building the guide locally

Though not required to contribute, it may be useful to build this guide locally in order to test your changes. In order
to build this guide locally, you’ll need:
1. Nox. You can install or upgrade nox using pip:

pip install --user nox

2. Python 3.6. Our build scripts are designed to work with Python 3.6 only. See the Hitchhiker’s Guide to Python
installation instructions to install Python 3.6 on your operating system.
To build the guide, run the following bash command in the source folder:

nox -s build

After the process has completed you can find the HTML output in the ./build/html directory. You can open the
index.html file to view the guide in web browser, but it’s recommended to serve the guide using an HTTP server.
You can build the guide and serve it via an HTTP server using the following command:

nox -s preview

The guide will be browsable via http://localhost:8000.

9.3 Where the guide is deployed

The guide is deployed via ReadTheDocs and the configuration lives at https://readthedocs.org/projects/
python-packaging-user-guide/. It’s served from a custom domain and fronted by Fast.ly.

9.4 Style guide

This style guide has recommendations for how you should write the Python Packaging User Guide. Before you start
writing, please review it. By following the style guide, your contributions will help add to a cohesive whole and make
it easier for your contributions to be accepted into the project.

9.4.1 Purpose

The purpose of the Python Packaging User Guide is to be the authoritative resource on how to package, publish, and
install Python projects using current tools.

124 Chapter 9. Contribute to this guide

 Python Packaging User Guide

9.4.2 Scope

The guide is meant to answer questions and solve problems with accurate and focused recommendations.
The guide isn’t meant to be comprehensive and it’s not meant to replace individual projects’ documentation. For
example, pip has dozens of commands, options, and settings. The pip documentation describes each of them in detail,
while this guide describes only the parts of pip that are needed to complete the specific tasks described in this guide.

9.4.3 Audience

The audience of this guide is anyone who uses Python with packages.
Don’t forget that the Python community is big and welcoming. Readers may not share your age, gender, education,
culture, and more, but they deserve to learn about packaging just as much as you do.
In particular, keep in mind that not all people who use Python see themselves as programmers. The audience of this
guide includes astronomers or painters or students as well as professional software developers.

9.4.4 Voice and tone

When writing this guide, strive to write with a voice that’s approachable and humble, even if you have all the answers.
Imagine you’re working on a Python project with someone you know to be smart and skilled. You like working with
them and they like working with you. That person has asked you a question and you know the answer. How do you
respond? That is how you should write this guide.
Here’s a quick check: try reading aloud to get a sense for your writing’s voice and tone. Does it sound like something
you would say or does it sound like you’re acting out a part or giving a speech? Feel free to use contractions and don’t
worry about sticking to fussy grammar rules. You are hereby granted permission to end a sentence in a preposition, if
that’s what you want to end it with.
When writing the guide, adjust your tone for the seriousness and difficulty of the topic. If you’re writing an introductory
tutorial, it’s OK to make a joke, but if you’re covering a sensitive security recommendation, you might want to avoid
jokes altogether.

9.4.5 Conventions and mechanics

Write to the reader When giving recommendations or steps to take, address the reader as you or use the imperative
mood.

Wrong: To install it, the user runs. . .

Right: You can install it by running. . .
Right: To install it, run. . .

State assumptions Avoid making unstated assumptions. Reading on the web means that any page of the guide may
be the first page of the guide that the reader ever sees. If you’re going to make assumptions, then say what
assumptions that you’re going to make.
Cross-reference generously The first time you mention a tool or practice, link to the part of the guide that covers it,
or link to a relevant document elsewhere. Save the reader a search.
Respect naming practices When naming tools, sites, people, and other proper nouns, use their preferred capitaliza-
tion.

9.4. Style guide 125

Python Packaging User Guide

Wrong: Pip uses. . .

Right: pip uses. . .

Wrong: . . . hosted on github.

Right: . . . hosted on GitHub.

Use a gender-neutral style Often, you’ll address the reader directly with you, your and yours. Otherwise, use gender-
neutral pronouns they, their, and theirs or avoid pronouns entirely.

Wrong: A maintainer uploads the file. Then he. . .

Right: A maintainer uploads the file. Then they. . .
Right: A maintainer uploads the file. Then the maintainer. . .

Headings Write headings that use words the reader is searching for. A good way to do this is to have your heading
complete an implied question. For example, a reader might want to know How do I install MyLibrary? so a
good heading might be Install MyLibrary.
In section headings, use sentence case. In other words, write headings as you would write a typical sentence.

Wrong: Things You Should Know About Python

Right: Things you should know about Python

Numbers In body text, write numbers one through nine as words. For other numbers or numbers in tables, use
numerals.

126 Chapter 9. Contribute to this guide

 CHAPTER

TEN

NEWS

10.1 September 2019

• Added a guide about publishing dists via GitHub Actions. (#647)

10.2 August 2019

• Updated to use python3 -m when installing pipx. (#631)

10.3 July 2019

• Marked all PEP numbers with the :pep: role. (#629)

• Upgraded Sphinx version and removed pypa.io intersphinx. (#625)
• Mentioned find_namespace_packages. (#622)
• Updated directory layout examples for consistency. (#611)
• Updated Bandersnatch link to GitHub. (#623)

10.4 June 2019

• Fixed some typos. (#620)

10.5 May 2019

• Added python_requires usage to packaging tutorial. (#613)

• Added a MANIFEST.in guide page. (#609)

10.6 April 2019

• Added a mention for shiv in the key projects section. (#608)

• Reduced emphasis on virtualenv. (#606)

127
Python Packaging User Guide

10.7 March 2019

• Moved single-sourcing guide version option to Python 3. (#605)

• Covered RTD details for contributing. (#600)

10.8 February 2019

• Elaborate upon the differences between the tutorial and the real packaging process. (#602)
• Added instructions to install Python CLI applications. (#594)

10.9 January 2019

• Added --no-deps to the packaging tutorial. (#593)

• Updated Sphinx and Nox. (#591)
• Referenced Twine from Python3. (#581)

10.10 December 2018

• No programmers in the office!

10.11 November 2018

• Removed landing page link to PyPI migration guide. (#575)

• Changed bumpversion to bump2version. (#572)
• Added single-sourcing package version example. (#573)
• Added a guide for creating documentation. (#568)

10.12 October 2018

• Updated Nox package name. (#566)

• Mentioned Sphinx extensions in guides. (#562)

10.13 September 2018

• Added a section on checking RST markup. (#554)

• Updated user installs page. (#558)
• Updated Google BigQuery urls. (#556)
• Replaced tar command with working command. (#552)

128 Chapter 10. News

 Python Packaging User Guide

• Changed to double quotes in the pip install SomeProject==1.4. (#550)

10.14 August 2018

• Removed the recommendation to store passwords in cleartext. (#546)

• Moved the Overview to a task based lead in along with the others. (#540)
• Updated Python version supported by virtualenv. (#538)
• Added outline/rough draft of new Overview page. (#519)

10.15 July 2018

• Improved binary extension docs. (#531)

• Added scikit-build to key projects. (#530)

10.16 June 2018

• Fixed categories of interop PEP for pypa.io. (#527)

• Updated Markdown descriptions explanation. (#522)

10.17 May 2018

• Noted issues with Provides-Dist and Obsoletes-Dist. (#513)

• Removed outdated warning about Python version mixing with Pipenv. (#501)
• Simplified packaging tutorial. (#498)
• Updated Windows users instructions for clarity. (#493)
• Updated the license section description for completeness. (#492)
• Added specification-style document to contributing section. (#489)
• Added documentation types to contributing guide. (#485)

10.18 April 2018

• Added README guide. (#461)

• Updated instructions and status for PyPI launch. (#475)
• Added instructions for Warehouse. (#471)
• Removed GPG references from publishing tutorial. (#466)
• Added ‘What’s in which Python 3.4–3.6?’. (#468)
• Added a guide for phasing out Python versions. (#459)

10.14. August 2018 129

Python Packaging User Guide

• Made default Description-Content-Type variant GFM. (#462)

10.19 March 2018

• Updated “installing scientific packages”. (#455)

• Added long_description_content_type to follow PEP 556. (#457)
• Clarified a long description classifier on pypi.org. (#456)
• Updated Core Metadata spec to follw PEP 556. (#412)

10.20 February 2018

• Added python3-venv and python3-pip to Debian installation instructions. (#445)

• Updated PyPI migration info. (#439)
• Added a warning about managing multiple versions with pipenv. (#430)
• Added example of multiple emails to Core Metadata. (#429)
• Added explanation of “legacy” in test.pypi.org/legacy. (#426)

10.21 January 2018

• Added a link to PyPI’s list of classifiers. (#425)

• Updated README.rst explanation. (#419)

10.22 December 2017

• Replaced ~ with $HOME in guides and tutorials. (#418)

• Noted which fields can be used with environment markers. (#416)
• Updated Requires-Python section. (#414)
• Added news page. (#404)

10.23 November 2017

• Introduced a new dependency management tutorial based on Pipenv. (#402)

• Updated the Single Sourcing Package Version tutorial to reflect pip’s current strategy. (#400)
• Added documentation about the py_modules argument to setup. (#398)
• Simplified the wording for the manifest.in section. (#395)

130 Chapter 10. News

 Python Packaging User Guide

10.24 October 2017

• Added a specification for the entry_points.txt file. (#398)

• Created a new guide for managing packages using pip and virtualenv. (#385)
• Split the specifications page into multiple pages. (#386)

10.25 September 2017

• Encouraged using readme_renderer to validate README.rst. (#379)

• Recommended using the –user-base option. (#374)

10.26 August 2017

• Added a new, experimental tutorial on installing packages using Pipenv. (#369)

• Added a new guide on how to use TestPyPI. (#366)
• Added pypi.org as a term. (#365)

10.27 July 2017

• Added flit to the key projects list. (#358)

• Added enscons to the list of key projects. (#357)
• Updated this guide’s readme with instructions on how to build the guide locally. (#356)
• Made the new TestPyPI URL more visible, adding note to homepage about pypi.org. (#354)
• Added a note about the removal of the explicit registration API. (#347)

10.28 June 2017

• Added a document on migrating uploads to PyPI.org. (#339)

• Added documentation for python_requires. (#338)
• Added a note about PyPI migration in the Tool Recommendations tutorial. (#335)
• Added a note that manifest.in does not affect wheels. (#332)
• Added a license section to the distributing guide. (#331)
• Expanded the section on the name argument. (#329)
• Adjusted the landing page. (#327, #326, #324)
• Updated to Sphinx 1.6.2. (#323)
• Switched to the PyPA theme. (#305)
• Re-organized the documentation into the new structure. (#318)

10.24. October 2017 131

Python Packaging User Guide

10.29 May 2017

• Added documentation for the Description-Content-Type field. (#258)

• Added contributor and style guide. (#307)
• Documented pip and easy_install’s differences for per-project indexes. (#233)

10.30 April 2017

• Added travis configuration for testing pull requests. (#300)

• Mentioned the requirement of the wheel package for creating wheels (#299)
• Removed the twine register reference in the Distributing Packages tutorial. (#271)
• Added a topic on plugin discovery. (#294, #296)
• Added a topic on namespace packages. (#290)
• Added documentation explaining prominently how to install pip in /usr/local. (#230)
• Updated development mode documentation to mention that order of local packages matters. (#208)
• Convert readthedocs link for their .org -> .io migration for hosted projects (#239)
• Swaped order of setup.py arguments for the upload command, as order is significant. (#260)
• Explained how to install from unsupported sources using a helper application. (#289)

10.31 March 2017

• Covered manylinux1 in Platform Wheels. (#283)

10.32 February 2017

• Added PEP 518. (#281)

Welcome to the Python Packaging User Guide, a collection of tutorials and references to help you distribute and install
Python packages with modern tools.
This guide is maintained on GitHub by the Python Packaging Authority. We happily accept any contributions and
feedback.

132 Chapter 10. News

 CHAPTER

ELEVEN

GET STARTED

Essential tools and concepts for working within the Python development ecosystem are covered in our Tutorials
section:
• to learn how to install packages, see the tutorial on installing packages.
• to learn how to manage dependencies in a version controlled project, see the tutorial on managing application
dependencies.
• to learn how to package and distribute your projects, see the tutorial on packaging and distributing
• to get an overview of packaging options for Python libraries and applications, see the Overview of Python
Packaging.

133
Python Packaging User Guide

134 Chapter 11. Get started

 CHAPTER

TWELVE

LEARN MORE

Beyond our Tutorials, this guide has several other resources:

• the Guides section for walk throughs, such as Installing pip/setuptools/wheel with Linux Package Managers or
Packaging binary extensions
• the Discussions section for in-depth references on topics such as Deploying Python applications or pip vs
easy_install
• the PyPA specifications section for packaging interoperability specifications
Additionally, there is a list of other projects maintained by members of the Python Packaging Authority.

135
Python Packaging User Guide

136 Chapter 12. Learn more

 INDEX

B PEP 440#compatible-release, 13, 46

Binary Distribution, 119 PEP 440#local-version-identifiers,
Built Distribution, 119 47
PEP 440#normalization, 46
D PEP 440#public-version-identifiers,
Distribution Package, 119 46
PEP 440#version-specifiers, 12, 120
E PEP 441, 116
Egg, 119 PEP 453, 25
environment variable PEP 453#rationale, 25
PATH, 13 PEP 503, 15, 109, 116
Extension Module, 119 PEP 508, 41, 94, 100, 103, 106
PEP 513, 50, 104
I PEP 517, 85, 103
PEP 518, 85, 103, 132
Import Package, 119
PEP 566, 93
K PEP 571, 104
PEP 592, 109
Known Good Set (KGS), 119
PEP 599, 104
M Python Package Index (PyPI), 120
Python Packaging Authority (PyPA), 120
Module, 119

P R
Release, 120
Package Index, 119
Requirement, 120
PATH, 13
Requirement Specifier, 120
Per Project Index, 119
Requirements File, 120
Project, 119
Pure Module, 120
pypi.org, 120
S
Python Enhancement Proposals setup.py, 120
PEP 438, 89 Source Archive, 120
Python Enhancement Proposals Source Distribution (or "sdist"), 120
PEP 301, 99 System Package, 120
PEP 345, 103
PEP 376, 91, 105, 106
V
PEP 397, 45 Version Specifier, 120
PEP 420, 69 Virtual Environment, 120
PEP 425, 49, 50, 91, 103–105
PEP 427, 90, 103, 106, 120
W
Wheel, 120
PEP 427#is-it-possible-to-import-python-code-directly-from-a-wheel-file,
91 Working Set, 120
PEP 440, 12, 13, 19, 44, 46, 94, 95, 101, 103, 120

137