DevOps in Python: Infrastructure as Python 2nd

Edition Moshe Zadka install download

https://ebookmeta.com/product/devops-in-python-infrastructure-as-
python-2nd-edition-moshe-zadka-2/

Download more ebook from https://ebookmeta.com

We believe these products will be a great fit for you. Click
the link to download now, or visit ebookmeta.com
to discover even more!

DevOps in Python: Infrastructure as Python 2nd Edition

Moshe Zadka

https://ebookmeta.com/product/devops-in-python-infrastructure-as-
python-2nd-edition-moshe-zadka-2/

Patterns and Practices for Infrastructure as Code: With

examples in Python and Terraform (MEAP V08) Rosemary
Wang

https://ebookmeta.com/product/patterns-and-practices-for-
infrastructure-as-code-with-examples-in-python-and-terraform-
meap-v08-rosemary-wang/

Mastering Python Networking Your one stop solution to

using Python for network automation programmability and
DevOps 3rd Edition Eric Chou

https://ebookmeta.com/product/mastering-python-networking-your-
one-stop-solution-to-using-python-for-network-automation-
programmability-and-devops-3rd-edition-eric-chou/

Patches the Cat Lisa Mullarkey

https://ebookmeta.com/product/patches-the-cat-lisa-mullarkey/
You Shall Not Kill The Prohibition of Killing in
Ancient Religions and Cultures Journal of Ancient
Judaism Supplements 27 J. Cornelis De Vos (Editor)

https://ebookmeta.com/product/you-shall-not-kill-the-prohibition-
of-killing-in-ancient-religions-and-cultures-journal-of-ancient-
judaism-supplements-27-j-cornelis-de-vos-editor/

Hodder GCSE (9-1) History for Pearson Edexcel

Foundation Edition: Medicine Through Time c.
1250-Present Slater

https://ebookmeta.com/product/hodder-gcse-9-1-history-for-
pearson-edexcel-foundation-edition-medicine-through-
time-c-1250-present-slater/

Saved By The Mountain Man Steamy BBW Instalove Romance

Mountain Men Love Curves 1st Edition Kelsie Calloway

https://ebookmeta.com/product/saved-by-the-mountain-man-steamy-
bbw-instalove-romance-mountain-men-love-curves-1st-edition-
kelsie-calloway/

When Good Government Meant Big Government The Quest to

Expand Federal Power 1913 1933 Jesse Tarbert

https://ebookmeta.com/product/when-good-government-meant-big-
government-the-quest-to-expand-federal-power-1913-1933-jesse-
tarbert/

Injunctions In Patent Law: Trans-Atlantic Dialogues On

Flexibility And Tailoring 1st Edition Jorge L.
Contreras

https://ebookmeta.com/product/injunctions-in-patent-law-trans-
atlantic-dialogues-on-flexibility-and-tailoring-1st-edition-
jorge-l-contreras/
Data-Driven SEO with Python: Solve SEO Challenges with
Data Science Using Python 1st Edition Andreas Voniatis

https://ebookmeta.com/product/data-driven-seo-with-python-solve-
seo-challenges-with-data-science-using-python-1st-edition-
andreas-voniatis/
Moshe Zadka
DevOps in Python
Infrastructure as Python
2nd ed.
Moshe Zadka
Belmont, CA, USA

ISBN 978-1-4842-7995-3 e-ISBN 978-1-4842-7996-0

https://doi.org/10.1007/978-1-4842-7996-0

© Moshe Zadka 2022

Apress Standard

The use of general descriptive names, registered names, trademarks,

service marks, etc. in this publication does not imply, even in the
absence of a specific statement, that such names are exempt from the
relevant protective laws and regulations and therefore free for general
use.

The publisher, the authors and the editors are safe to assume that the
advice and information in this book are believed to be true and accurate
at the date of publication. Neither the publisher nor the authors or the
editors give a warranty, expressed or implied, with respect to the
material contained herein or for any errors or omissions that may have
been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.

This Apress imprint is published by the registered company APress

Media, LLC, part of Springer Nature.
The registered company address is: 1 New York Plaza, New York, NY
10004, U.S.A.
Dedicated to A and N, my favorite two projects
Introduction
Python began as a language to automate an operating system: the
Amoeba. A typical Unix shell would be ill-suited since it had an API, not
just textual file representations. The Amoeba OS is a relic now.
However, Python continues to be a useful tool for automation of
operations—the heart of typical DevOps work.
It is easy to learn and easy to write readable code is a necessity
when a critical part of the work is responding to a 4 a.m. alert and
modifying some misbehaving program.
It has powerful bindings to C and C++, the universal languages of
the operating system—and yet is natively memory-safe, leading to few
crashes at the automation layer.
Finally, although not true when it was created, Python is one of the
most popular languages. This means that it is relatively easy to hire
people with Python experience and easy to get training materials and
courses for people who need to learn on the job.
This book guides you through how to take advantage of Python to
automate operations.
To get the most out of the book, you need to be somewhat familiar
with Python. If you are new to Python, there are many great resources
to learn it, including the official Python tutorial at docs.python.org. You
also need to be somewhat familiar with Unix-like operating systems like
Linux, especially how to use the command line.
Any source code or other supplementary material referenced by the
author in this book is available to readers on GitHub
(github.com/apress). For more detailed information, please visit
http://www.apress.com; https://github.com/Apress/DevOps-in-
Python-2nd-ed-
Acknowledgments
Thanks to my wife, Jennifer Zadka, without whose support I could not
have written this book.
Thanks to my parents, Yaacov and Pnina Zadka, who taught me how
to learn.
Thanks to my advisor, Yael Karshon, who taught me how to write.
Thanks to Mahmoud Hashemi for inspiration and encouragement.
Thanks to Mark Williams for being there for me.
Thanks to Glyph Lefkowitz for teaching me about Python,
programming, and being a good person.
Thanks to Brennon Church and Andrea Ross, who supported my
personal growth and learning journey.
Table of Contents
Chapter 1:​Installing Python
1.​1 OS Packages
1.​2 Using pyenv
1.​3 Building Python from Source
1.​4 PyPy
1.​5 Anaconda
1.​6 Summary
Chapter 2:​Packaging
2.​1 Virtual Environments
2.​2 pip
2.​3 Setup and Wheels
2.​4 Binary Wheels
2.5 manylinux Wheels
2.​5.​1 Self-Contained Wheels
2.​5.​2 Portable Wheels
2.​5.​3 manylinux Containers
2.​5.​4 Installing manylinux Wheels
2.​6 tox
2.​6.​1 One Environment
2.​6.​2 Multiple Environments
2.​6.​3 Multiple Differently Configured Environments
2.​7 Pip Tools
2.​8 Poetry
2.​8.​1 Installing
2.​8.​2 Creating
 2.​8.​3 Dependencies
2.​8.​4 Developing
2.​8.​5 Building
2.​9 Pipenv
2.​10 DevPI
2.​11 pex and shiv
2.​11.​1 pex
2.​11.​2 shiv
2.​12 Summary
Chapter 3:​Interactive Usage
3.​1 Native Console
3.​2 The Code Module
3.​3 ptpython
3.​4 IPython
3.​5 JupyterLab
3.​6 Summary
Chapter 4:​OS Automation
4.​1 Files
4.​2 Processes
4.​3 Networking
4.​4 Summary
Chapter 5:​Testing
5.​1 Unit Testing
5.​2 Mocks, Stubs, and Fakes
5.​3 Testing Files
5.​3.​1 Testing with Subdirectories
 5.​3.​2 Accelerating Tests with eatmydata
5.​3.​3 Accelerating Tests with tmpfs
5.​4 Testing Processes
5.​5 Testing Networking
5.​6 Testing HTTP Clients
Chapter 6:​Text Manipulation
6.​1 Bytes, Strings, and Unicode
6.​2 Strings
6.​3 Regular Expressions
6.​4 JSON
6.​5 CSV
6.​6 Summary
Chapter 7:​HTTPX
7.​1 Clients
7.​2 REST
7.​3 Security
7.​4 Authentication
7.​5 Async client
7.​6 Summary
Chapter 8:​Cryptography
8.​1 Fernet
8.​2 PyNaCl
8.​3 Passlib
8.​4 TLS Certificates
8.​5 Summary
Chapter 9:​Paramiko
 9.​1 SSH Security
9.​2 Client Keys
9.​3 Host Identity
9.​4 Connecting
9.​5 Running Commands
9.​6 Remote Files
9.​6.​1 Metadata Management
9.​6.​2 Upload
9.​6.​3 Download
9.​7 Summary
Chapter 10:​SaltStack
10.​1 Salt Basics
10.​2 Salt Concepts
10.​3 Salt Formats
10.​4 Salt Extensions
10.​4.​1 States
10.​4.​2 Execution
10.​4.​3 Utility
10.​4.​4 Extra Third-Party Dependencies
10.​5 Summary
Chapter 11:​Ansible
11.​1 Ansible Basics
11.​2 Ansible Concepts
11.​3 Ansible Extensions
11.​4 Summary
Chapter 12:​Containers
 12.​1 Choosing a Base Image
12.​1.​1 GNU C Library Support
12.​1.​2 Long-Term Support
12.​1.​3 Avoiding Unexpected Changes
12.​2 Installing the Python Interpreter
12.​2.​1 conda
12.​2.​2 Third-Party Repositories
12.​2.​3 Building Python in the Container
12.​2.​4 Python Base Image
12.​3 Installing Python Applications
12.​4 Optimizing Container Build Cache
12.​5 Rebuilding Containers
12.​6 Container Security
12.​7 Summary
Chapter 13:​Amazon Web Services
13.​1 Security
13.​1.​1 Configuring Access Keys
13.​1.​2 Creating Short-Term Tokens
13.​2 Elastic Computing Cloud (EC2)
13.​2.​1 Regions
13.​2.​2 Amazon Machine Images
13.​2.​3 SSH Keys
13.​2.​4 Bringing up Machines
13.​2.​5 Securely Logging In
13.​2.​6 Building Images
13.​3 Simple Storage Service (S3)
 13.​3.​1 Managing Buckets
13.​4 Summary
Chapter 14:​Kubernetes
14.​1 Pods
14.​1.​1 Liveness and Readiness
14.​1.​2 Configuration
14.​1.​3 Python Sidecars
14.​2 REST API
14.​3 Operators
14.​3.​1 Permissions
14.​3.​2 Custom Types
14.​3.​3 Retrieval
14.​3.​4 Goal State
14.​3.​5 Comparison
14.​3.​6 Reconciliation
14.​3.​7 Combining the Pieces
14.​4 Summary
Chapter 15:​Terraform
15.​1 JSON Syntax
15.​2 Generating Terraform Configurations
15.​3 Summary
Index
About the Author
Moshe Zadka
has been involved in the Linux
community since 1998, helping in Linux
“installation parties.” He has been
programming Python since 1999 and has
contributed to the core Python
interpreter. Moshe has been a
DevOps/SRE since before those terms
existed, caring deeply about software
reliability, build reproducibility, and
more. He has worked in companies as
small as three people and as big as tens
of thousands—and usually in a position
where software meets system
administration.
About the Technical Reviewer
Martyn Bristow
is a software developer in the United
Kingdom. He began using Python as a
researcher but is now an experienced
broad user of Python for data analysis,
software test automation, and DevOps.
He currently builds data analysis web
apps in Python deployed to Kubernetes.
You can find Martyn on GitHub
@martynbristow.
© The Author(s), under exclusive license to APress Media, LLC, part of Springer
Nature 2022
M. Zadka, DevOps in Python
https://doi.org/10.1007/978-1-4842-7996-0_1
1. Installing Python
Moshe Zadka1
(1) Belmont, CA, USA

Before you can use Python, you need to install it. Some operating
systems, such as macOS and some Linux variants, have Python
preinstalled. Those versions of Python, colloquially called system
Python, often make poor defaults for people who want to develop in
Python.
The version of Python installed is often behind the latest practices.
System integrators often patch Python in ways that can lead to
surprises. For example, Debian-based Python is often missing modules
like venv and ensurepip. macOS Python links against a Mac shim
around its native SSL library. Those things mean, especially when
starting and consuming FAQs and web resources, it is better to install
Python from scratch.
This chapter covers a few ways to do so and the pros and cons of
each.
1.1 OS Packages
Volunteers have built ready-to-install packages for some of the more
popular operating systems.
The most famous is the deadsnakes PPA (Personal Package
Archives). The dead in the name refers to the fact that those packages
are already built—with the metaphor that sources are “alive.” Those
packages are built for Ubuntu and usually support all the versions of
Ubuntu that are still supported upstream. Getting those packages is
done with a simple

$ sudo add-apt-repository ppa:deadsnakes/ppa

$ sudo apt update

On macOS, the Homebrew third-party package manager has up-to-

date Python packages. An introduction to Homebrew is beyond the
scope of this book. Homebrew being a rolling release, the Python
version is upgraded from time to time. While this means that it is a
useful way to get an up-to-date Python, it makes it a poor target for
reliably distributing tools.
It also has some downsides for doing day-to-day development. Since
it upgrades quickly after a new Python release, development
environments can break quickly and without warning. It also means
that sometimes code can stop working; even if you are careful to watch
upcoming Python versions for breaking changes, not all packages will.
Homebrew is a good fit when needing a well-built up-to-date Python
interpreter for a one-off task. Writing a quick script to analyze data, or
automate some APIs, is a good use of Homebrew Python.
Finally, for Windows, it is possible to download an installer from
Python.org for any version of Python.
1.2 Using pyenv
pyenv tends to offer the highest return on investment for installing
Python for local development. The initial setup does have some
subtleties. However, it allows installing many side-by-side Python
versions as needed. It also allows you to manage how each is accessed
—either using a per-user default or a per-directory default.
Installing pyenv itself depends on the operating system. On a
macOS, the easiest way is to install it via Homebrew. Note that in this
case, pyenv itself might need to be upgraded to install new versions of
Python.
On a Unix-based operating system, such as Linux or FreeBSD, the
easiest way to install pyenv is by using the curl|bash command.

$ PROJECT=https://github.com/pyenv/pyenv-installer
\
PATH=raw/master/bin/pyenv-installer \
curl -L $PROJECT/PATH | bash

Of course, this comes with its own security issues and could be
replaced with a two-step process where you can inspect the shell script
before running or even use git-checkout to pin to a specific revision.

$ git clone https://github.com/pyenv/pyenv-

installer
$ cd pyenv-installer
$ bash pyenv-installer

Unfortunately, pyenv does not work on Windows.

After installing pyenv, it is useful to integrate it with the running
shell. You do this by adding the following to the shell initialization file
(e.g., .bash_profile).

export PATH="~/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
 This allows pyenv to properly intercept all the necessary
commands.
pyenv separates the notion of installed interpreters from available
interpreters. Enter the following to install a version.

$ pyenv install <version>

For CPython, <version> is the version number, such as 3.6.6 or

3.7.0rc1.
An installed version is distinct from an available version. Versions
are available globally (for a user) by using

$ pyenv global 3.7.0

or locally by using

$ pyenv local 3.7.0

Local means they are available in a given directory. This is done by

putting a python-version.txt file in this directory. This is important for
version-controlled repositories, but a few different strategies are used
to manage those. One strategy is to add this file to the ignored list. This
is useful for heterogeneous teams or open source projects. Another
strategy is to check in the file so that the same version of Python is used
in the repository.
Note that since pyenv is designed to install side-by-side versions of
Python, it has no concept of upgrading Python. A newer Python version
needs to be installed with pyenv and then set as the default.
By default, pyenv installs non-optimized versions of Python. If
optimized versions are needed, enter the following.

env PYTHON_CONFIGURE_OPTS="--enable-shared
--enable-optimizations
--with-computed-gotos
--with-lto
--enable-ipv6" pyenv
install
 Let’s build a version that is pretty similar to binary versions from
python.org.
1.3 Building Python from Source
The main challenge in building Python from source is that, in some
sense, it is too forgiving. It is all too easy to build it with one of the built-
in modules disabled because its dependency was not detected. This is
why it is important to know which dependencies are fragile and how to
make sure a local installation is good.
The first fragile dependency is SSL. It is disabled by default and
must be enabled in Modules/Setup.dist. Carefully follow the
instructions there about the location of the OpenSSL library. If you have
installed OpenSSL via system packages, it is usually in /usr/. If you
have installed it from source, it is usually in /usr/local.
The most important thing is to know how to test for it. When
Python is done building, run ./python.exe -c 'import _ssl'.
That .exe is not a mistake; this is how the build process calls the newly
built executable, which is renamed to Python during installation. If this
succeeds, the SSL module was built correctly.
Another extension that can fail to build is SQLite. Since it is a built-
in, many third-party packages depend on it, even if you are not using it
yourself. This means a Python installation without the SQLite module is
pretty broken. Test it by running ./python.exe -c 'import
sqlite3'.
In a Debian-based system (such as Ubuntu), libsqlite3-dev is
required for this to succeed. In a Red Hat-based system (such as Fedora
or CentOS), libsqlite3-dev is required for this to succeed.
Next, check for _ctypes with ./python.exe -c 'import
_ctypes'. If this fails, likely, the libffi headers are not installed.
Finally, remember to run the built-in regression test suite after
building from source. This ensures that there have been no silly
mistakes while building the package.
1.4 PyPy
The usual implementation of Python is sometimes known as CPython to
distinguish it from the language proper. The most popular alternative
implementation is PyPy, a Python-based JIT implementation of Python
in Python. Because it has a dynamic JIT (just-in-time) compilation to
assembly, it can sometimes achieve phenomenal speed-ups (three times
or even ten times) over regular Python.
There are sometimes challenges in using PyPy. Many tools and
packages are tested only with CPython. However, sometimes spending
the effort to check if PyPy is compatible with the environment is worth
it if performance matters.
There are a few subtleties in installing Python from source. While it
is theoretically possible to translate using CPython, in practice, the
optimizations in PyPy mean that translating using PyPy works on more
reasonable machines. Even when installing from source, it is better to
first install a binary version to bootstrap.
The bootstrapping version should be PyPy, not PyPy3. PyPy is
written in the Python 2 dialect, which is one of the only cases where
worrying about the deprecation is irrelevant since PyPy is a Python 2
dialect interpreter. PyPy3 is the Python 3 dialect implementation,
which is usually better in production as most packages are slowly
dropping support for Python 2.
The latest PyPy3 supports 3.5 features of Python, as well as f-
strings. However, the latest async features, added in Python 3.6, do not
work.
1.5 Anaconda
The closest to a system Python that is still reasonable for use as a
development platform is Anaconda, a metadistribution. It is, in essence,
an operating system on top of the operating system. Anaconda has its
grounding in the scientific computing community, and so its Python
comes with easy-to-install modules for many scientific applications.
Many of these modules are non-trivial to install from PyPI, requiring a
complicated build environment.
It is possible to install multiple Anaconda environments on the
same machine. This is handy when needing different Python versions
or different versions of PyPI modules.
To bootstrap Anaconda, you can use the bash installer available at
https://conda.io/miniconda.html. The installer also modifies
~/.bash_profile to add the path to conda, the installer.
conda environments are created using conda create --name
<name> and activated using source conda activate <name>.
There is no easy way to use inactivated environments. It is possible to
create a conda environment while installing packages: conda create
--name some-name python. You can specify the version using = –
conda create --name some-name python=3.5. It is also
possible to install more packages into a conda environment, using
conda install package[=version], after the environment has
been activated. Anaconda has a lot of prebuilt Python packages,
especially ones that are non-trivial to build locally. This makes it a good
choice if those packages are important to your use case.
1.6 Summary
Running a Python program requires an interpreter installed on the
system. Depending on the operating system and the versions, there are
several different ways to install Python. Using the system Python is a
problematic option. On macOS and Unix systems, using pyenv is almost
always the preferred option. On Windows, using the prepackaged
installers from Python.org is often a good idea.
© The Author(s), under exclusive license to APress Media, LLC, part of Springer
Nature 2022
M. Zadka, DevOps in Python
https://doi.org/10.1007/978-1-4842-7996-0_2
2. Packaging
Moshe Zadka1
(1) Belmont, CA, USA

One of the main strengths of Python is the ecosystem, the third-party

packages on PyPI. There are packages to do anything from running
computations in parallel on GPUs for machine learning to reducing the
boilerplate needed for writing classes. This means that a lot of the practical
work with Python is handling the third-party dependencies.
The current packaging tooling is pretty good, but things have not
always been this way. It is important to understand which best practices
are antiquated rituals based on faulty assumptions but have some merit
and are actually good ideas.
When dealing with packaging, there are two ways to interact. One is to
be a consumer wanting to use the functionality of a package. Another is to
be the producer, publishing a package. These describe, usually, different
development tasks, not different people.
It is important to have a solid understanding of the consumer side of
packages before moving to production. If the goal of a package publisher is
to be useful to the package user, it is crucial to imagine the last mile before
starting to write a single line of code.
2.1 Virtual Environments
Virtual environments are often misunderstood because the concept of
environments is not clear. A Python environment refers to the root of the
Python installation. The reason an environment is important is because of
the lib/site-packages subdirectory of that root. The lib/site-
packages subdirectory is where third-party packages are installed.
The most popular tool to add packages to an environment is pip, which
is covered in the next section. Before using pip, it is important to
understand how virtual environments work.
A real environment is based on Python installation, which means that to
get a new real environment, a new Python must be installed and often
rebuilt. This is sometimes an expensive proposition.
The advantage of a virtual environment is that it is cheap to set up and
tear down. Some modern Python tooling takes advantage of that, setting up
and tearing down virtual environments as a normal part of their operation.
Setting up and tearing down virtual environments, being cheap and fast, is
also a common part of Python developer workflow.
A virtual environment copies the minimum necessary out of the real
environment to mislead Python into thinking it has a new root. The precise
file structure is less important than remembering that the command to
create a virtual environment is simple and fast.
Here, simple means that all the command does is copy some files and
perhaps make a few symbolic links. Because of that, there are a few failure
modes—mostly when file creation fails because of permission issues or a
full disk.
There are two ways to use virtual environments: activated and
inactivated. To use an inactivated virtual environment, which is most
common in scripts and automated procedures, you explicitly call Python
from the virtual environment.
This means that a virtual environment in /home/name/venvs/my-
special-env calling /home/name/venvs/my-special-
env/bin/python has a Python process that uses this environment. For
example, /home/name/venvs/my-special-env/bin/python -m
pip runs pip but installs in the virtual environment.
Note that entrypoint–based scripts are installed alongside Python,
so running /home/name/venvs/my-special-env/bin/pip also
installs packages in the virtual environment.
The other way to use a virtual environment is to activate it. Activating a
virtual environment in a bash-like shell means sourcing its activated script.

$ source /home/name/venvs/my-special-env/bin/activate

The sourcing sets a few environment variables, only one of which is

important. The important variable is PATH, which gets prefixed by
/home/name/venvs/my-special-env/bin. This means that
commands like python or pip are found there first. Two cosmetic
variables are set. $VIRTUAL_ENV points to the root of the environment.
This is useful in management scripts that want to be aware of virtual
environments. PS1 is prefixed with (my-special-env), which is useful
for visualizing the virtual environment while working interactively in the
console.
It is generally a good practice to only install third-party packages inside
a virtual environment. Combined with the fact that virtual environments
are cheap, if one gets into a bad state, it is best to remove the whole
directory and start from scratch.
For example, imagine a bad package install that causes the Python start-
up to fail. Even running pip uninstall is impossible since pip fails on
start-up. However, the cheapness means you can remove the whole virtual
environment and re-create it with a good set of packages.
A modern practice is to move increasingly toward treating virtual
environments as semi-immutable. After creating them, there is a single
stage for installing all required packages. Instead of modifying the virtual
environment if an upgrade is required, destroy the environment, re-create,
and reinstall.
The modern way to create virtual environments is to use the venv
standard library module. This only works on Python 3. Since Python 2 has
been strongly deprecated since the beginning of 2020, it is best avoided in
any case.
venv is used as a command with python -m venv <directory>,
as there is no dedicated entrypoint. It creates the directory for the
environment.
It is best if this directory does not exist before that. A best practice is to
remove it before creating the environment. There are also two options for
creating the environment: which interpreter to use and what initial
packages to install.
2.2 pip
The packaging tool for Python is pip. There have been other tools that
have mostly been abandoned by the community and should not be used.
Installations of Python used to not come with pip out of the box. This
has changed in recent versions, but many versions which are still
supported do not have it. When running on such a version, python -m
ensurepip installs it.
Some Python installations, especially system ones, disable ensurepip.
When lacking ensurepip, there is a way of manually getting it: get-
pip.py. This is a single downloadable file that, when executed, unpacks
pip.
Luckily, pip is the only package that needs these weird gyrations to
install. All other packages can, and should, be installed using pip.
For example, if sample-environment is a virtual environment,
installing the glom package can be done with the following code.

$ ./sample-environment/bin/python -m pip install glom

...
$ ./sample-environment/bin/python -m glom
{}

The last command tests that glom has been properly installed. Glom is
a package to handle deeply-nested data, and called with no arguments,
outputs an empty Python dictionary. This makes it handy for quickly
testing whether a new virtual environment can install new packages
properly.
Internally, pip is also treated as a third-party package. Upgrading pip
itself is done with pip install --upgrade pip.
Depending on how Python was installed, its real environment might or
might not be modifiable by the user. Many instructions in various README
files and blogs might encourage using sudo pip install. This is
almost always the wrong thing to do; it installs the packages in the global
environment.
The pip install command downloads and installs all dependencies.
However, it can fail to downgrade incompatible packages. It is always
possible to install explicit versions: pip install package-name==
<version> installs this precise version. This is also a good way for local
testing to get explicitly non-general-availability packages, such as release
candidates, beta, or similar.
If wheel is installed, pip builds, and usually caches, wheels for
packages. This is especially useful when dealing with a high virtual
environment churn since installing a cached wheel is a fast operation. This
is also highly useful when dealing with native or binary packages that need
to be compiled with a C compiler. A wheel cache eliminates the need to
build it again.
pip does allow uninstalling with pip uninstall <package>. This
command, by default, requires manual confirmation. Except for exotic
circumstances, this command is not used. If an unintended package has
snuck in, the usual response is to destroy the environment and rebuild it.
For similar reasons, pip install --upgrade <package> is not
often needed; the common response is to destroy and re-create the
environment. There is one situation where it is a good idea.
pip install supports a requirements file: pip install --
requirements or pip install -r. The requirements file simply has
one package per line. This is no different from specifying packages on the
command line. However, requirement files often specify strict
dependencies. A requirements file can be generated from an environment
with pip freeze.
Like most individual packages or wheels, installing anything that is not
strict and closed under requirements requires pip to decide which
dependencies to install. The general problem of dependency resolution
does not have an efficient and complete solution. Different strategies are
possible to approach such a solution.
The way pip resolves dependencies is by using backtracking. This
means that it optimistically tries to download the latest possible
requirements recursively. If a dependency conflict is found, it backtracks;
try a different option.
As an example, consider three packages.
top
middle
base
There are two base versions: 1.0 and 2.0. The package dependencies
are setup.cfg files.
 The following is for the top.

[metadata]
name = top
version = 1.0
[options]
install_requires =
base
middle

The following is for the middle.

[metadata]
name = middle
version = 1.0
[options]
install_requires =
base<2.0

The base package has two versions: 1.0 and 2.0. It does not have any
dependencies.
Because top depends directly on base, pre-backtracking versions of
pip get the latest and then have a failed resolution.

$ pip install top

Looking in links: .
Collecting top
Collecting middle (from top)
Collecting base (from top)
middle 1.0 has requirement base<2.0, but you'll have
base 2.0 which is incompatible.
Installing collected packages: base, middle, top
Successfully installed base-2.0 middle-1.0 top-1.0

The backtracking algorithm discards the base 2.0 version.

$ pip install top

Looking in links: .
Processing ./top-1.0-py3-none-any.whl
Processing ./base-2.0-py3-none-any.whl
Processing ./middle-1.0-py3-none-any.whl
Processing ./base-1.0-py3-none-any.whl
Installing collected packages: base, middle, top
Successfully installed base-1.0 middle-1.0 top-1.0
This solution has the advantage that it is complete, but it can take
unfeasible amounts of time in certain cases. This is rare, but merely taking
a long time is not.
One way to increase the speed is to include >= dependencies in the
loose requirements. This is usually a good idea since packages are better at
guaranteeing backward compatibility than forward compatibility. As a side
benefit, this can dramatically reduce the solution space that pip needs to
backtrack in.
In most scenarios, it is better to use strict requirements for day-to-day
development and regenerate the strict requirements from the loose
requirements (which can take a while) on a cadence that balances keeping
up to date with churn.
2.3 Setup and Wheels
The term third party (as in third-party packages) refers to someone other
than the Python core developers (first-party) or the local developers
(second-party). I have covered how to install first-party packages in the
installation section. You used pip and virtualenv to install third-party
packages. It is time to finally turn your attention to the missing link: local
development and installing local packages or second-party packages.
Note that the word package here means something different from post-
installation. In Python, a package is an importable directory, a way to keep
multiple modules together. The pedantic way to call installable things is
distribution. A distribution can correspond to no packages (it can be a top-
level single-module distribution) or multiple packages.
It is good to keep a 1-1-1 relationship when packaging things: a single
distribution corresponding to one package and named the same. Even if
there is only one file, put it as an __init__.py file under a directory.
Packaging is an area that has seen a lot of changes. Copying and pasting
from existing packages is not a good idea; good packages are, for the most
part, mature packages. Following the latest best practices means making
changes to an existing working process.
Starting with setuptools version 61.0.0, it is possible to create a
package with only two files besides the code files.
pyproject.toml
README.rst
The README is not strictly necessary. However, most source code
management systems display it rendered, so it is best to break it out into its
own file.
Even an empty pyproject.toml generates a package. However,
almost all packages need at least a few more details.
The build-system is the one mandatory section in a non-empty
pyproject.toml file. It is usually the first.

[build-system]
requires = [
"setuptools"
]
build-backend = "setuptools.build_meta"
 Many systems can be used to build valid distributions. The
setuptools system, which used to be the only possibility, is now one of
several. However, it is still the most popular one.
Most of the rest of the data can be found in the project section.

[project]
name = "awesome_package"
version = "0.0.3"
description = "A pretty awesome package"
readme = "README.rst"
authors = [{name = "My Name",
email = "[email protected]"}]
dependencies = ["httpx"]

For most popular code organizations, this is enough for the

setuptools systems to find the code and create a correct package.
There are ways to have setuptools treat the version as dynamic and
take it from a file or an attribute. An alternative is to take advantage of
pyproject.toml in a structured format and manipulate it directly.
For example, the following code uses a CalVer (calendar versioning)
scheme of YEAR.MONTH.release in a month. It uses the built-in
zoneinfo module, which requires Python 3.9 or above, and the tomlkit
library, which supports roundtrip-preserving TOML parsing and
serialization.

import tomlkit
import datetime
import os
import pathlib
import zoneinfo

now =
datetime.datetime.now(tz=zoneinfo.ZoneInfo("UTC"))
prefix=f"{now.year}.{now.month}."
pyproject = pathlib.Path("pyproject.toml")
data = tomlkit.loads(pyproject.read_text())
current = data["project"].get("version", "")
if current.startswith(prefix):
serial = int(current.split(".")[-1]) + 1
else:
serial = 0
version = prefix + str(serial)
data["project"]["version"] = version
pyproject.write_text(tomlkit.dumps(data))
Some utilities keep the version synchronized between several files; for
example, pyproject.toml and example_package/__init__.py.
The best way to use these utilities is by not needing to do it.
If example_package/__init__.py wants to expose the version
number, the best way is to calculate it using importlib.metadata.

# example_package/__init__.py
from importlib import metada
__version__ =
metadata.distribution("example_package").version
del metadata # Keep top-level namespace clean

This avoids needing to keep more than one place in sync.

The field dependencies in pyproject.toml is present on almost
every package. This is how to mark other distributions that the code needs.
It is a good practice to put loose dependencies in pyproject.toml. This
is in contrast to exact dependencies, which specify a specific version. A
loose dependency looks like Twisted>=17.5, specifying a minimum
version but no maximum. Exact dependencies, like Twisted==18.1, are
usually a bad idea in pyproject. toml. They should only be used in
rare cases, for example, when using significant chunks of a private API
package.
The pyproject.toml file also allows defining entrypoints. Some
frameworks, like Pyramid, allow using entrypoints to add plugin-like
features.
It also allows you to define scripts. These used to be
console_scripts entrypoints but now have their own section.

[project.scripts]
example-command = "example_package.commands:main"

The syntax is package.....module:function. This function is

called with no arguments when the script is being run.
 Usually, this includes command-line parsing, but the following is a short
example.

# example_package/commands.py
def main():
print("an example")

In this example, running example-command causes the string to

print.

$ example-command
an example

You can build a distribution with pyproject.toml, a README.rst,

and some Python code. There are several formats a distribution can take,
but the one covered here is the wheel.
After installing build using pip install build, run

python -m build --wheel

This creates a wheel under dist. If the wheel needs to be in a different

directory, add --outdir <output directory> to the command.
You can do several things with the wheel, but it is important to note that
one thing you can do is pip install <wheel file>. Doing this as
part of continuous integration makes sure the wheel, as built by the current
directory, is functional.
It is possible to use python -m build to create a source distribution.
This is usually a good idea to accommodate use cases that prefer to install
from source. These use cases are esoteric, but generating the source
distribution is easy enough to be worth it.

$ python -m build --sdist

It is possible to combine the --sdist and --wheel arguments into

one run of python -m build. This is also what python -m build
does by default: create both a source distribution and a wheel.
By default, python -m build installs any packages it needs to build
the package in a fresh virtual environment. When running python -m
build in a tight edit-debug loop, perhaps to debug a setup.cfg, this can
get tedious. In those cases, create and activate a virtual environment, and
then run

$ python -m build --no-isolation

This installs its dependencies in the current environment. While this is
not a good fit for production use, this is a faster way to debug packaging
issues.
2.4 Binary Wheels
Python is well known and often used as an integration language. One of the
ways this integration happens is by linking to native code.
This is usually done using the C Application Binary Interface (C ABI).
The C ABI is used not only for integrating with C libraries but with other
languages, such as C++, Rust, or Swift, which can generate C ABI-
compatible interfaces.
There needs to be some glue code bridging the C ABI to Python to
integrate Python with such code. It is possible to write this code by hand.
This is a tedious and error-prone process, so code generators are often
used. Cython is a popular generator that uses a Python-compatible
language. Although Cython is often used to interface to C ABI libraries, it
can be used to generate extensions without such integration. This makes
examples slightly simpler, so the following Cython code is used as a running
example.

#cython: language_level=3

def add(x, y):

return x + y

This code is in the binary_module.pyx file. It is short and does just

enough to be clear if it works correctly.
To build code with native integration, the files that describe the build
are slightly more complicated.
The pyproject.toml file is no longer empty. It now has two lines.

[build-system]
requires = ["setuptools", "cython"]

This makes sure the cython package is installed before trying to build
a wheel.
The setup.py is no longer minimal. It contains enough code to
integrate with Cython.

import setuptools
from Cython import Build
setuptools.setup(
ext_modules=Build.cythonize("binary_module.pyx"),
)
The Cython.Build.cythonize function does two things.
Creates (or re-creates) binary_module.c from
binary_module.pyx.
Returns an Extension object.
Since *.pyx files are not included by default, it needs to be enabled
explicitly in the MANIFEST.in file.

include *.pyx

Since, in this example, there are no regular Python files, the

setup.cfg does not need to specify any.

[metadata]
name = binary_example
version = 1.0

With these files, running python -m build --wheel builds a

binary wheel in dist named something like binary_example-1.0-
cp39-cp39-linux_x86_64.whl. Details of the name depend on the
platform, the architecture, and the Python version.
After installing this wheel, it can be used as follows.

$ pip install dist/binary_example*.whl

$ python -c 'import
binary_module;print(binary_module.add(1, 2))'
3

This is a simple example that demonstrates the mechanics of binary

packaging. It is designed to show how all the pieces fit together in a small
example.
Realistic binary packages are usually more complicated, implementing
subtle algorithms that can take advantage of the optimizations Cython
gives or wrapping a native-code library.
2.5 manylinux Wheels
A binary wheel is not a pure Python wheel because at least one of its files
contains native code. On a Linux system, this native code is a shared library,
a file with the .so suffix for a shared object.
This shared library links against other libraries. For a library designed
to wrap a specific native library, as with pygtk wrapping gtk, it links with
the wrapped library.
In almost all cases, whether it is designed to wrap a specific library or
not, it links against the standard C library. This is the library that has C
functions like printf. Few things can be done in native code without
linking against it.
On most modern Linux systems, this linking is usually dynamic. This
means that the binary wheel does not contain the library it is linked with; it
expects to load it at runtime.
If a wheel is built on a different system than the one it is installed on, a
library that is binary compatible with the one it is linked with has to be
installed on the system. If a binary compatible library is not installed, this
leads to a failure at import time.

2.5.1 Self-Contained Wheels

The auditwheel tool takes binary wheels and patches them to make
them more portable. One of its functions is to grab the pieces from dynamic
libraries and put them in the wheel. This allows the wheels to be installed
without requiring a different library.
For auditwheel to work correctly, the patchelf utility needs to be
installed. Older versions might produce wheels that break in strange ways.
The safest way to have the right version of patchelf is to download the
latest source distribution and build it.
To make a self-contained wheel, first, build a regular build. This might
require careful reading of the instructions for building the package from
source. This results in the regular binary wheel in dist/. This was the
case in the example before with the binary_example module.
After this is done, run

$ auditwheel repair --plat linux_x86_64 dist/*.whl

 By default, auditwheel creates the self-contained wheel in a
wheelhouse subdirectory. The wheel created is self-contained but
expects to be installed on a compatible version of Linux.

2.5.2 Portable Wheels

The --plat flag in auditwheel is the platform tag. If it is linux_<cpu
architecture>, the wheel makes no guarantees about which GNU C
Library it is compatible with.
Wheels like that should only be installed on a compatible Linux system.
To avoid mistakes, most Python package index systems, including PyPI, do
not let these wheels be uploaded.
Uploadable Python wheels must be tagged with a proper platform tag,
which shows which versions of the GNU C Library they are compatible
with. Historically, those tags relied on the CentOS release year: manylinux1
corresponded to CentOS 5, manylinux20210 corresponded to CentOS 6,
and manylinux20214 corresponded to CentOS 7.
At the time of writing, manylinux_2_24 and manylinux_2_27 are the
only post–CentOS 7 versions. These correspond to Debian 9 and Ubuntu
18.04, respectively.
After deciding on the oldest supported platform tag, build on the
newest system which supports it. For example, if no deployment target
uses GNU C Library < 2.24, build the wheel until Debian 9. Especially for
binary wheels with complicated build dependencies, a newer system
makes it easier to follow the documentation and reduces the chances of
running into unexpected issues.

2.5.3 manylinux Containers

Making sure that the patchelf tool is correctly installed and Python is
built with the correct version of the C library is a subtle and error-prone
process. One way to avoid this is to use the official manylinux container
images.
These container images are available at
quay/pypa/manylinux_<version>. There are versions available for
manylinux_2_24, manylinux2014, manylinux2010, and manylinux1. These
images contain all officially supported versions of Python and the rest of
the tooling necessary.
 Note that specialized build dependencies need to be installed on those
systems; for example, when using manylinux_2_24 (a Debian-based
container).

$ docker run --rm -it quay/pypa/manylinux_2_24

# apt-get update
# apt-get install -y <dependencies>

2.5.4 Installing manylinux Wheels

By default, pip uses manylinux wheels with compatible platform tags.
Wheels can be uploaded to a package index or added to a directory passed
to pip with --find-links.
In some situations, it is better if pip fails quickly when no prebuilt
wheel is available. The --only-binary :all: option can be given to
disable installing from source distributions.
Other documents randomly have
different content
 buffalo hunting.

The buffalo, more properly called the bison, is the great object of
Indian hunting in the west. These animals abound in the prairies;
and they are often seen coursing over the plains in immense herds.
Thousands of them appear under the direction of one of their
number, who acts as leader. This propensity to follow a leader
affords a ready means to the Indians of destroying them. The
manner in which this is accomplished is graphically described in the
following extract from the account of a late writer. It affords a wild
picture of the scenes which present themselves to the notice of the
traveller as he passes through the great prairies of the west.
We passed a precipice of about one hundred and twenty feet high,
under which lay scattered the fragments of at least one hundred
carcases of buffaloes, although the water, which had washed away
the lower part of the hill, must have carried off many of the dead.
These buffaloes had been chased down the precipice, in a way very
common on the Missouri, and by which vast herds are destroyed in a
moment. The mode of hunting is, to select one of the most active
and fleet young men, who is disguised, by a buffalo skin around his
body, the skin of the head, with the ears and the horns, fastened on
his own head, in such a way as to deceive the buffalo. Thus dressed,
he fixes himself at a convenient distance, between a herd of
buffaloes and any of the river precipices, which sometimes extend
for some miles. His companions, in the meantime, get into the rear,
and on the side of the herd, and, at a given signal, show
themselves, and advance towards the buffalo: they instantly take the
alarm; and, finding the hunters beside them, they run towards the
disguised Indian or decoy, who leads them on at full speed toward
the river, when, suddenly securing himself in some crevice of the cliff
which he had previously fixed on, the herd is left on the brink of the
precipice. It is then in vain for the foremost to retreat, or even to
stop—they are pressed on by the hindmost rank, who, seeing no
danger, but from the hunters, goad on those before them, till the
whole are precipitated, and the shore is strewed with their dead
bodies. Sometimes, in this perilous seduction, the Indian is himself
either trodden under foot, by the rapid movements of the buffaloes
or missing his footing in the cliff, is urged down the precipice along
with the falling herd.
The Indians now select as much meat as they choose, and the rest
is abandoned to the wolves, and creates a most dreadful stench.
The wolves who had been feasting on these carcases were very fat,
and so gentle, that one of them was killed with an espontoon.
 religion of the indians.

The earliest visiters of the New World, on seeing among the Indians
neither priests, temples, idols, nor sacrifices, represented them as a
people wholly destitute of religious opinions. Closer inquiry, however,
showed that a belief in the spiritual world, however imperfect, had a
commanding influence over almost all their actions. Their creed
includes even some lofty and pure conceptions. Under the title of the
Great Spirit, the Master of Life, the Maker of heaven and earth, they
distinctly recognise a supreme ruler of the universe and an arbiter of
their destiny. A party of them, when informed by the missionaries of
the existence of a being of infinite power, who had created the
heavens and the earth, with one consent exclaimed, “Atahocan!
Atahocan!” that being the name of their principal deity. According to
Long, the Indians among whom he resided ascribe every event,
propitious or unfortunate, to the favour or anger of the Master of
Life. They address him for their daily subsistence; they believe him
to convey to them presence of mind in battle; and amid tortures
they thank him for inspiring them with courage. Yet though this one
elevated and just conception is deeply graven on their minds, it is
combined with others which show all the imperfection of unassisted
reason in attempting to think rightly on this great subject. It may
even be observed, that the term, rendered into our language “great
spirit,” does not really convey the idea of an immaterial nature. It
imports with them merely some being possessed of lofty and
mysterious powers, and in this sense is applied to men, and even to
animals. The brute creation, which occupies a prominent place in all
their ideas, is often viewed by them as invested, to a great extent,
with supernatural powers; an extreme absurdity, which, however,
they share with the civilized creeds of Egypt and India.
When the missionaries, on their first arrival, attempted to form an
idea of the Indian mythology, it appeared to them extremely
complicated, more especially because those who attempted to
explain it had no fixed opinions. Each man differed from his
neighbour, and at another time from himself; and when the
discrepancies were pointed out, no attempt was made to reconcile
them. The southern tribes, who had a more settled faith, are
described by Adair as intoxicated with spiritual pride, and
denouncing even their European allies as “the accursed people.” The
native Canadian, on the contrary, is said to have been so little
tenacious, that he would at any time renounce all his theological
errors for a pipe of tobacco, though, as soon as it was smoked, he
immediately relapsed. An idea was found prevalent respecting a
certain mystical animal, called Mesou or Messessagen, who, when
the earth was buried in water, had drawn it up and restored it.
Others spoke of a contest between the hare, the fox, the beaver,
and the seal, for the empire of the world. Among the principal
nations of Canada, the hare is thought to have attained a decided
preeminence; and hence the Great Spirit and the Great Hare are
sometimes used as synonymous terms. What should have raised this
creature to such distinction seems rather unaccountable; unless it
were that its extreme swiftness might appear something
supernatural. Among the Ottowas alone the heavenly bodies become
an object of veneration; the sun appears to rank as their supreme
deity.
To dive into the abyss of futurity has always been a favourite object
of superstition. It has been attempted by various means; but the
Indian seeks it chiefly through his dreams, which always bear with
him a sacred character. Before engaging in any high undertaking,
especially in hunting or war, the dreams of the principal chiefs are
carefully watched and studiously examined; and according to the
interpretation their conduct is guided. A whole nation has been set
in motion by the sleeping fancies of a single man. Sometimes a
person imagines in his sleep that he has been presented with an
article of value by another, who then cannot, without impropriety,
leave the omen unfulfilled. When Sir William Johnson, during the
American war, was negotiating an alliance with a friendly tribe, the
chief confidentially disclosed that, during his slumbers, he had been
favoured with a vision of Sir William bestowing upon him the rich
laced coat which formed his full dress. The fulfilment of this
revelation was very inconvenient; yet, on being assured that it
positively occurred, the English commander found it advisable to
resign his uniform. Soon after, however, he unfolded to the Indian a
dream with which he had himself been favored, and in which the
former was seen presenting him with a large tract of fertile land
most commodiously situated. The native ruler admitted that, since
the vision had been vouchsafed, it must be realized, yet earnestly
proposed to cease this mutual dreaming, which he found had turned
much to his own disadvantage.
The manitou is an object of peculiar veneration; and the fixing upon
this guardian power is not only the most important event in the
history of a youth, but even constitutes his initiation into active life.
As a preliminary, his face is painted black, and he undergoes a
severe fast, which is, if possible, prolonged for eight days. This is
preparatory to the dream in which he is to behold the idol destined
ever after to afford him aid and protection. In this state of excited
expectation, and while every nocturnal vision is carefully watched,
there seldom fails to occur to his mind something which, as it makes
a deep impression, is pronounced his manitou. Most commonly it is a
trifling and even fantastic article; the head, beak, or claw of a bird,
the hoof of a cow, or even a piece of wood. However, having
undergone a thorough perspiration in one of their vapour-baths, he
is laid on his back, and a picture of it is drawn upon his breast by
needles of fish-bone dipped in vermilion. A good specimen of the
original being procured, it is carefully treasured up; and to it he
applies in every emergency, hoping that it will inspire his dreams,
and secure to him every kind of good fortune. When, however,
notwithstanding every means of propitiating its favour, misfortunes
befall him, the manitou is considered as having exposed itself to just
and serious reproach. He begins with remonstrances, representing
all that has been done for it, the disgrace it incurs by not protecting
its votary, and, finally, the danger that, in case of repeated neglect,
it may be discarded for another. Nor is this considered merely as an
empty threat; for if the manitou is judged incorrigible, it is thrown
away; and by means of a fresh course of fasting, dreaming,
sweating, and painting, another is installed, from whom better
success may be hoped.
The absence of temples, worship, sacrifices, and all the observances
to which superstition prompts the untutored mind, is a remarkable
circumstance, and, as we have already remarked, led the early
visiters to believe that the Indians were strangers to all religious
ideas. Yet the missionaries found room to suspect that some of their
great feasts, in which every thing presented must be eaten, bore an
idolatrous character, and were held in honour of the Great Hare. The
Ottawas, whose mythological system seems to have been the most
complicated, were wont to keep a regular festival to celebrate the
beneficence of the sun; on which occasion the luminary was told
that this service was in return for the good hunting he had procured
for his people, and as an encouragement to persevere in his friendly
cares. They were also observed to erect an idol in the middle of their
town, and sacrifice to it; but such ceremonies were by no means
general. On first witnessing Christian worship, the only idea
suggested by it was that of their asking some temporal good, which
was either granted or refused. The missionaries mention two
Hurons, who arrived from the woods soon after the congregation
had assembled. Standing without, they began to speculate what it
was the white men were asking, and then whether they were getting
it. As the service continued beyond expectation, it was concluded
they were not getting it; and as the devotional duties still proceeded,
they admired the perseverance with which this rejected suit was
urged. At length, when the vesper hymn began, one of the savages
observed to the other: “Listen to them now in despair, crying with all
their might.”
The grand doctrine of a life beyond the grave was, among all the
tribes of America, most deeply cherished and most sincerely
believed. They had even formed a distinct idea of the region whither
they hoped to be transported, and of the new and happier mode of
existence, free from those wars, tortures, and cruelties which throw
so dark a shade over their lot upon earth. Yet their conceptions on
this subject were by no means either exalted or spiritualized. They
expected simply a prolongation of their present life and enjoyments,
under more favourable circumstances, and with the same objects
furnished in greater choice and abundance. In that brighter land the
sun ever shines unclouded, the forests abound with deer, the lakes
and rivers with fish; benefits which are farther enhanced in their
imagination by a faithful wife and dutiful children. They do not reach
it, however, till after a journey of several months, and encountering
various obstacles; a broad river, a chain of lofty mountains, and the
attack of a furious dog. This favoured country lies far in the west, at
the remotest boundary of the earth, which is supposed to terminate
in a steep precipice, with the ocean rolling beneath. Sometimes, in
the too eager pursuit of game, the spirits fall over, and are converted
into fishes. The local position of their paradise appears connected
with certain obscure intimations received from their wandering
neighbours of the Mississippi, the Rocky Mountains, and the distant
shores of the Pacific. This system of belief labours under a great
defect, inasmuch as it scarcely connects felicity in the future world
with virtuous conduct in the present. The one is held to be simply a
continuation of the other; and under this impression, the arms,
ornaments, and everything that had contributed to the welfare of
the deceased, are interred along with him. This supposed assurance
of a future life, so conformable to their gross habits and conceptions,
was found by the missionaries a serious obstacle when they
attempted to allure them by the hope of a destiny, purer and higher
indeed, but less accordant with their untutored conceptions. Upon
being told that in the promised world they would neither hunt, eat,
drink, nor marry, many of them declared that, far from endeavouring
to reach such an abode, they would consider their arrival there as
the greatest calamity. Mention is made of a Huron girl whom one of
the Christian ministers was endeavouring to instruct, and whose first
question was what she would find to eat. The answer being
“Nothing,” she then asked what she would see; and being informed
that she would see the Maker of heaven and earth, she expressed
herself much at a loss how she should address him.
 indian funerals.

Another sentiment, congenial with that now described, is most

deeply rooted in the mind of the Indians. This is reverence for the
dead, with which Chateaubriand, though somewhat hastily,
considers them more deeply imbued than any other people. During
life they are by no means lavish in their expressions of tenderness,
but on the hour of final separation it is displayed with extraordinary
force. When any member of a family becomes seriously ill, all the
resources of magic and medicine are exhausted in order to procure
his recovery. When the fatal moment arrives, all the kindred burst
into loud lamentations, which continue till some person possessing
the requisite authority desires them to cease. These expressions of
grief, however, are renewed for a considerable time at sunrise and
sunset. After three days the funeral takes place, when all the
provisions which the family can procure are expended in a feast, to
which the neighbours are generally invited; and, although on all
solemn occasions it is required that every thing should be eaten, the
relations do not partake. These last cut off their hair, cover their
heads, paint their faces of a black colour, and continue long to deny
themselves every species of amusement. The deceased is then
interred with his arms and ornaments, his face painted, and his
person attired in the richest robes which they can furnish. It was the
opinion of one of the early missionaries, that the chief object of the
Hurons in their traffic with the French was to procure materials for
honouring their dead; and, as a proof of this, many of them have
been seen shivering half naked in the cold, while their hut contained
rich robes to be wrapped round them after their decease. The body
is placed in the tomb in an upright posture, and skins are carefully
spread round it, so that no part may touch the earth. This, however,
is by no means the final ceremony, being followed by another still
more solemn and singular. Every eighth, tenth, or twelfth year,
according to the custom of the different nations, is celebrated the
festival of the dead; and, till then, the souls are supposed to hover
round their former tenement, and not to depart for their final abode
in the west. On this occasion the people march in procession to the
places of interment, open the tombs, and, on beholding the mortal
remains of their friends, continue some time fixed in mournful
silence. The women then break out into loud cries, and the party
begin to collect the bones, removing every remnant of flesh. The
remains are then wrapped in fresh and valuable robes, and conveyed
amid continual lamentation to the family cabin. A feast is then given,
followed during several days by dances, games, and prize-combats,
to which strangers often repair from a great distance. This mode of
celebration certainly accords very ill with the sad occasion; yet the
Greek and Roman obsequies were solemnized in a similar manner;
nay, in many parts of Scotland, till very recently, they were
accompanied by festival, and often by revelry. The relics are then
carried to the council-house of the nation, where they are hung for
exhibition along the walls, with fresh presents destined to be
interred along with them. Sometimes they are even displayed from
village to village. At length, being deposited in a pit previously dug in
the earth, and lined with the richest furs, they are finally entombed.
Tears and lamentations are again lavished; and during a few days
food is brought to the place. The bones of their fathers are
considered by the Indians the strongest ties to their native soil; and
when calamity forces them to quit it, these mouldering fragments
are, if possible, conveyed along with them.
 indian cannibalism.

It has been made a question whether the Indians can be justly

charged with cannibalism. It is certain that all the terms by which
they designate their inhuman mode of putting a prisoner to death
bear reference to this horrid practice. The expressions are to throw
him into the caldron, to devour him, to eat soup made of his flesh. It
has hence been plausibly inferred that this enormity really prevailed
in early times, but was changed, we can scarcely say mitigated, into
the present system of torture. Yet, as every action is described by
them in terms highly figurative, those now quoted may have been
used as expressing most fully the complete gratification of their
revenge. Of this charge they cannot now be either condemned or
wholly acquitted. In the excited fury of their passions, portions of
the flesh are often seized, roasted, and eaten, and draughts taken of
the blood. To eat an enemy’s heart is considered a peculiar
enjoyment. Long mentions a gentleman who came upon a party who
were busy broiling a human heart, when he with difficulty prevailed
on them to desist. There is little hesitation among them, in periods
of scarcity, to relieve hunger with the flesh of their captives; and
during one war, this fate is said to have befallen many French
soldiers who fell into the hands of the Five Nations. Colonel Schuyler
told Colden, that, having entered the cabin of a chief who had some
rich soup before him, he was invited to partake. Being hungry and
tired, he readily agreed, till the ladle, being put into the great
caldron, brought up a human hand, the sight of which put an
immediate end to his appetite and meal.
 indian dances.

The dances of the Indians, even those at common festivals, are on

an extensive scale, requiring to a complete performance forty or fifty
persons, who execute their evolutions by following each other round
a great fire kindled in the centre. Their movements, monotonous but
violent, consist in stamping furiously on the ground, and often
brandishing their arms in a manner compared by an able writer to a
baker converting flour into dough. They keep good time; but the
music is so exceedingly simple that this implies little merit. They
conclude with a loud shout or howl, which echoes frightfully through
the woods. The dances in celebration of particular events are of a
more varied character, and often form a very expressive pantomime.
The war-dance is the most favourite and frequent. In this
extraordinary performance, a complete image is given of the terrible
reality; the war-whoop is sounded with the most frightful yells; the
tomahawk is wildly brandished; and the enemy are surprised, seized,
and scalped, or carried off for torture. The calumet-dance, which
celebrates peace between nations, and the marriage-dance, which
represents domestic life, are much more pleasing. Some mention is
made of a mystic dance, carried on by the jugglers or doctors, with
strange superstitious ceremonies, and in which a supernatural
personage, termed by some the devil, rises and performs; but it
does not seem to have been witnessed by any European, and is said
to be now in a great measure disused.
 indian games.

There are games to which the Indians are fondly attached, which,
though they be only ranked under the head of amusement, are yet
constructed in the same serious manner as their other transactions.
Their great parties are said to be collected by supernatural authority,
communicated by the jugglers; and they are preceded, like their
wars and hunts, by a course of fasting, dreaming, and other means
of propitiating fortune. The favourite game is that of the bone, in
which small pieces of that substance, resembling dice, and painted
of different colours, are thrown in the air, and according to the
manner in which they fall, the game is decided. Only two persons
can play; but a numerous party, and sometimes whole villages,
embrace one side or the other, and look on with intense interest. At
each throw, especially if it be decisive, tremendous shouts are
raised; the players and spectators equally resemble persons
possessed; the air rings with invocations to the bones and to the
manitous. Their eagerness sometimes leads to quarreling and even
fighting, which on no other occasion ever disturb the interior of
these societies. To such a pitch are they occasionally worked up, that
they stake successively all they possess, and even their personal
liberty; but this description must apply only to the more southern
nations, as slavery was unknown among the Canadian Indians.
A temporary interval of wild license, of emancipation from all the
restraints of dignity and decorum, seems to afford an enjoyment
highly prized in all rude societies. Corresponding with the saturnalia
and bacchanals of antiquity, the Indians have their festivals of
dreams, which, during fifteen days, enlivens the inaction of the
coldest season. Laying aside all their usual order and gravity, they
run about, frightfully disguised, and committing every imaginable
extravagance. He who meets another demands an explanation of his
visions, and if not satisfied, imposes some fantastic penalty. He
throws upon him cold water, hot ashes, or filth; sometimes, rushing
into his cabin, he breaks and destroys the furniture. Although
everything appears wild and unpremeditated, it is alleged that
opportunities are often taken to give vent to old and secret
resentments. The period having elapsed, a feast is given, order is
restored, and the damages done are carefully repaired.
 beautiful trait of character.

One M’Dougal, a native of Argyleshire, having emigrated to Upper

Canada, from anxiety to make the most of his scanty capital, or
some other motive, he purchased a location, where the price of land
is merely nominal, in a country thinly peopled, and on the extreme
verge of civilization. His first care was to construct a house, and
plant in the wild. This task finished, he spent his whole time, early
and late, in the garden and the fields. By vigorous exertions, and
occasional assistance, he brought a few acres of ground under crop;
acquired a stock of cattle, sheep and hogs; made additional inroads
on the glade and the forest, and, though his toils were hard,
gradually and imperceptibly became, in a rough way, “well enough
to live,” as compared with the poverty he had abandoned at home.
His greatest discomforts were, distance from his neighbours, the
church, markets, and even the mill; and, along with these, the
suspension (or rather, the enjoyment) after long intervals of time, of
those endearing charities and friendly offices, which lend such a
charm to social life.
On one occasion, M’Dougal had a melder of corn to grind, and as the
distance was considerable, and the roads none of the smoothest,
this important part of his duty could only be performed by starting
with the sun, and returning with the going down of the same. In his
absence, the care of the cattle devolved on his spouse, and as they
did not return at the usual hour, the careful matron went out in
quest of them. Beyond its mere outskirts, the forest was, to her,
terra incognita, in the most emphatic sense of the term; and with no
compass, or notched trees to guide, it is not to be wondered at that
she wandered long and wearily to very little purpose. Like alps on
alps, tall trees arose on every side—a boundless continuity of shade
—and, fatigued with the search, she deemed it prudent to retrace
her steps, while it was yet time. But this resolution was much easier
formed than executed; returning was as dangerous as “going o’er,”
and, after wandering for hours, she sunk on the ground, her eyes
swollen and filled with tears, and her mind agitated almost to
distraction. But here she had not rested many minutes, before she
was startled by the sound of approaching footsteps, and, anon, an
Indian hunter stood before her, “a stoic of the woods, a man without
fear.” Mrs. M‘Dougal knew that Indians lived at no great distance,
but as she had never seen a member of the tribe, her emotions were
those of terror—quickening, it may be said, every pulse, and yet
paralyzing every limb. The Indian’s views were more comprehensive;
constantly on the look out, in search of the quarry, and accustomed
to make circuits, comprising the superficies of many a highland
mountain and glen, he had observed her, without being observed
himself, knew her home, recognized her person, comprehended her
mishap, divined her errand, and immediately beckoned her to follow
him. The unfortunate woman understood his signal, and obeyed it,
as far as terror left her power; and, after a lengthened sweep, which
added not a little to her previous fatigue, they arrived at the door of
an Indian wigwam.
Her conductor invited her to enter, by signs: but this she sternly
refused to do, dreading the consequence, and preferring death in
the open air to the tender mercies of cannibals within. Perceiving her
reluctance, and scanning her feelings, the hospitable Indian darted
into the wigwam, and communed with his wife, who, in a few
minutes, also appeared: and, by certain signs and sympathies,
known only to females, calmed the stranger’s fears, and induced her
to enter their lowly abode. Venison was instantly prepared for
supper, and Mrs. M’Dougal—though still alarmed at the novelty of
her situation, found the viands delicious, and had rarely, if ever,
partaken of so savoury a meal. Aware that she was wearied, the
Indians removed from their place near the roof, two beautiful deer
skins, and, by stretching and fixing them across, divided the wigwam
into two apartments. Mats were also spread in both, and next, the
stranger was given to understand, that the further dormitory was
expressly designed for her accommodation. But here again her
courage failed her, and to the most pressing intreaties, she replied
by signs, as well as she could, that she would prefer to sit and sleep
by the fire. This determination seemed to puzzle the Indian and his
squaw sadly. Often they looked at each other, and conversed softly
in their own language: and, at last, the Red took the White woman
by the hand, led her to her couch, and became her bedfellow. In the
morning she awoke greatly refreshed, and anxious to depart,
without further delay—but the Indian would on no account permit it.
Breakfast was prepared—another savoury and well-cooked meal—
and then the Indian accompanied his guest, and conducted her to
the very spot where the cattle were grazing. These he kindly drove
from the wood, on the verge of which Mrs. M‘Dougal descried her
husband, running about every where, hallooing and seeking for her,
in a state of absolute distraction. Great was his joy, and great his
gratitude to her Indian benefactor, who was invited to the house,
and treated to the best the larder afforded, and presented, on his
departure, with a suit of clothes.
In about three days he returned, and endeavoured, by every wile, to
induce Mr. M’Dougal to follow him into the forest. But this invitation
the other positively declined—and the poor Indian went on his way,
obviously grieved and disappointed. But again he returned; and,
though words were wanting, renewed his intreaties—but still vainly,
and without effect: and then, as a last desperate effort, he hit upon
an expedient, which none, save an Indian hunter, would have
thought of. Mrs. M’Dougal had a nursling only a few months old—a
fact the Indian failed not to notice. After his pantomimic eloquence
had been thrown away, he approached the cradle, seized the child,
and darted out of the house with the speed of an antelope. The
alarmed parents instantly followed, supplicating and imploring, at
the top of their voices. But the Indian’s resolves were as fixed as
fate—and away he went, slow enough to encourage his pursuers,
but still in the van by a good many paces, and far enough ahead to
achieve the secret purpose he had formed—like the parent-bird,
skimming the ground, when she wishes to wile the enemy from her
nest. Again and again Mr. M’Dougal wished to continue the chase
alone—but maternal anxiety baffled every remonstrance; and this
anxiety was, if possible, increased, when she saw the painted savage
enter the wood, and steer, as she thought, his course towards his
own cabin. The Indian, however, was in no hurry;—occasionally, he
cast a glance behind, poised the child almost like a feather, treading
his way with admirable dexterity, and kept the swaddling clothes so
closely drawn around it, that not even the winds of heaven were
permitted to visit it roughly. It is, of course, needless to go into all
the details of this singular journey, further than to say, that the
Indian, at length, called a halt on the margin of a most beautiful
prairie, teeming with the richest vegetation, and comprising many
thousands of acres. In a moment the child was restored to its
parents—who, wondering what so strange a procedure could mean,
stood, for some minutes, panting for breath, and eyeing one another
in silent and speechless astonishment.
The Indian, on the other hand, appeared overjoyed at the success of
his manœuvre—and never did a human being frisk about and
gesticulate with greater animation. We have heard, or read, of a
professor of signs: and supposing such a character were wanted, the
selection could not—or, at least should not—be a matter of difficulty,
so long as even a remnant remains of the aborigines of North
America. All travellers agree in describing their gestures as highly
dignified, eloquent, and intelligent: and we have the authority of Mr.
M’Dougal for saying, that the hero of the present strictly authentic
tale, proved himself to be a perfect master of the art. The
restoration of the child—the beauty and wide extent of the prairies,
and various other circumstances combined—flashed across our
countryman’s mind—operating conviction where jealously and
distrust had lurked before. Mr. M’Dougal, in a trice, examined the
soil, and immediately saw the propriety of the advice given by the
untutored one. By a sort of tacit agreement, a day was fixed for the
removal of the materials of our countryman’s cabin, goods and
chattels;—and the Indian, true to his word, brought a detachment of
his tribe to assist in one of the most romantic “flittings” that ever
was undertaken either in the old or new world. In a few days a
roomy log-house was fashioned, and a garden formed in a
convenient section of the beautiful prairie, from which the smoke
was seen curling, and the woodpecker tapping at no great distance.
M‘Dougal was greatly pleased at the change—and no wonder, seeing
that he could almost boast of a body-guard as bold as the bowmen
of Robin Hood. His Indian friend speedily became a sort of foster
brother, and his tribe as faithful as the most attached Tail of Gillies
that ever surrounded a Highland chieftain. Even the stupid kine
lowed, on finding themselves suddenly transferred to a boundless
range of richest pasture:—and, up to the date of the last advices,
were improving rapidly in condition, and increasing in numbers.
The little garden was smiling like a rose in the desert-grass,
overabundant, was gradually giving way to thriving crops, and the
kine so well satisfied with their gang, that the herds and enclosures
were like unheeded to keep them from the corn. The Indians
continued friendly and faithful—occasionally bringing presents of
venison and other game, and were uniformly rewarded from the
stores of a dairy, overflowing with milk, butter, and cheese.
Attached as the Red man was to his own mode of life, he was
induced at length to form a part of the establishment, in the capacity
of grieve, or head shepherd—a duty he undertook most cheerfully,
as it still left him opportunities of meeting and communing with his
friends, and reconnoitering the altering denizens of the forest. Let us
hope, therefore, that no untoward accident will occur to mar this
beautiful picture of sylvan life; that the M’Dougal colony will wax
stronger, till every section of the prairie is forced to yield tribute to
the spade and the plough.
 the reformed indian.

Some of the Indians believe, that the “Evil Spirit” is the maker of
spirituous liquors, from which, notwithstanding, hardly one of them
can refrain. An Indian near the Delaware Water Gap, told Mr.
Heckewelder, a missionary, that he had once, when under the
influence of strong liquor, killed the best Indian friend he had,
fancying him to be his worst avowed enemy. He said that the
deception was complete; and that while intoxicated, the face of his
friend presented to his eyes all the features of the man with whom
he was in a state of hostility. It is impossible to express the horror
which struck him, when he awoke from that delusion. He was so
shocked, that from that moment, he resolved never more to taste of
the maddening potion, of which he was convinced the devil was the
inventor; for that it could only be the “Evil Spirit” who made him see
his enemy when his friend was before him, and produced so strong
a delusion on his bewildered senses, that he actually killed him.
From that time until his death, which happened thirty years
afterwards, he never drank a drop of ardent spirits, which he always
called “the devil’s blood;” and was firmly persuaded that the devil, or
some of his infernal spirits, had a hand in preparing it.
 fidelity.

Among the North American Indians, one of the first lessons they
inculcate on their children, is duty to their parents, and respect for
old age; and there is not among the most civilized nations, any
people who more strictly observe the duty of filial obedience. A
father need only to say, in the presence of his children, “I want such
a thing done”—“I want one of my children to go upon such an
errand”—“Let me see who is the good child that will do it.” The word
good operates as it were by magic, and the children immediately vie
with each other to comply with the parent’s wishes. If a father sees
an old decrepid man or woman pass by, led along by a child, he will
draw the attention of his own children to the object, by saying,
“What a good child that must be, which pays such attention to the
aged! That child, indeed, looks forward to the time when it will
likewise be old, and need its children’s help.” Or he will say, “May the
Great Spirit, who looks upon him, grant this good child a long life!”