Experience Seamless Full Ebook Downloads for Every Genre at textbookfull.

com

Using Asyncio in Python Understanding Python s

Asynchronous Programming Features Caleb Hattingh

https://textbookfull.com/product/using-asyncio-in-python-
understanding-python-s-asynchronous-programming-features-
caleb-hattingh/

OR CLICK BUTTON

DOWNLOAD NOW

Explore and download more ebook at https://textbookfull.com

 Recommended digital products (PDF, EPUB, MOBI) that
you can download immediately if you are interested.

Using Asyncio in Python 3 Caleb Hattingh

https://textbookfull.com/product/using-asyncio-in-python-3-caleb-
hattingh/

textboxfull.com

Introduction to Computation and Programming Using Python

with Application to Understanding Data Guttag

https://textbookfull.com/product/introduction-to-computation-and-
programming-using-python-with-application-to-understanding-data-
guttag/
textboxfull.com

Expert Twisted: Event-Driven and Asynchronous Programming

with Python 1st Edition Mark Williams

https://textbookfull.com/product/expert-twisted-event-driven-and-
asynchronous-programming-with-python-1st-edition-mark-williams/

textboxfull.com

Expert Twisted: Event-Driven and Asynchronous Programming

with Python 1st Edition Mark Williams

https://textbookfull.com/product/expert-twisted-event-driven-and-
asynchronous-programming-with-python-1st-edition-mark-williams-2/

textboxfull.com
Python Programming Using Problem Solving Approach Thareja
Reema

https://textbookfull.com/product/python-programming-using-problem-
solving-approach-thareja-reema/

textboxfull.com

Introduction to Computation and Programming Using Python

With Application to Understanding Data Second Edition John
V. Guttag
https://textbookfull.com/product/introduction-to-computation-and-
programming-using-python-with-application-to-understanding-data-
second-edition-john-v-guttag/
textboxfull.com

Powerful Python: the most impactful patterns, features and

development strategies modern Python provides Maxwell

https://textbookfull.com/product/powerful-python-the-most-impactful-
patterns-features-and-development-strategies-modern-python-provides-
maxwell/
textboxfull.com

Python Descriptors: Understanding and Using the Descriptor

Protocol 1st Edition Jacob Zimmerman

https://textbookfull.com/product/python-descriptors-understanding-and-
using-the-descriptor-protocol-1st-edition-jacob-zimmerman/

textboxfull.com

Introduction to Computation and Programming Using Python

3rd Edition John V. Guttag

https://textbookfull.com/product/introduction-to-computation-and-
programming-using-python-3rd-edition-john-v-guttag/

textboxfull.com
 Using Asyncio in Python
Understanding Python’s Asynchronous Programming
Features

Caleb Hattingh
Using Asyncio in Python
by Caleb Hattingh
Copyright © 2020 Tekmoji Pty Ltd. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North,
Sebastopol, CA 95472.
O’Reilly books may be purchased for educational, business, or sales
promotional use. Online editions are also available for most titles
(http://oreilly.com). For more information, contact our
corporate/institutional sales department: 800-998-9938 or
[email protected].

Acquisitions Editor: Jessica Haberman

Developmental Editor: Corbin Collins

Production Editor: Beth Kelly

Copyeditor: Rachel Head

Proofreader: Sharon Wilkey

Indexer: Ellen Troutman-Zaig

Interior Designer: David Futato

Cover Designer: Karen Montgomery

Illustrator: Rebecca Demarest

February 2020: First Edition

Revision History for the First Edition
2020-01-30: First Release

See http://oreilly.com/catalog/errata.csp?isbn=9781492075332 for

release details.
The O’Reilly logo is a registered trademark of O’Reilly Media, Inc.
Using Asyncio in Python, the cover image, and related trade dress
are trademarks of O’Reilly Media, Inc.
The views expressed in this work are those of the author, and do not
represent the publisher’s views. While the publisher and the author
have used good faith efforts to ensure that the information and
instructions contained in this work are accurate, the publisher and
the author disclaim all responsibility for errors or omissions,
including without limitation responsibility for damages resulting from
the use of or reliance on this work. Use of the information and
instructions contained in this work is at your own risk. If any code
samples or other technology this work contains or describes is
subject to open source licenses or the intellectual property rights of
others, it is your responsibility to ensure that your use thereof
complies with such licenses and/or rights.
978-1-492-07533-2
[LSI]
To my partner, Gina: I deeply appreciate all your encouragement and
support while writing this book; it has made all the difference.
—Caleb
Preface

Python 3.4 introduced the asyncio library, and Python 3.5 produced
the async and await keywords to use it palatably. These new
additions allow so-called asynchronous programming.
All of these new features, which I’ll refer to under the single name
Asyncio, have been received by the Python community somewhat
warily; a segment of the community seems to see them as complex
and difficult to understand. This view is not limited to beginners:
several high-profile contributors to the Python community have
expressed doubts about the complexity of the Asyncio API in Python,
and educators in the community have expressed concern about how
best to teach Asyncio to students.
Most people with a few years’ experience with Python have used
threads before, and even if you haven’t, you are still likely to have
experienced blocking. For example, if you’ve written programs using
the wonderful requests library, you will surely have noticed that your
program pauses for a bit while it does requests.get(url); this is
blocking behavior.
For one-off tasks, this is fine; but if you want to fetch ten thousand
URLs simultaneously, it’s going to be difficult to use requests. Large-
scale concurrency is one big reason to learn and use Asyncio, but
the other big attraction of Asyncio over preemptive threading is
safety: it will be much easier for you to avoid race condition bugs
with Asyncio.
My goal with this book is to give you a basic understanding of why
these new features have been introduced and how to use them in
your own projects. More specifically, I aim to provide the following:
 A critical comparison of asyncio and threading for
concurrent network programming
An understanding of the new async/await language syntax
A general overview of the new asyncio standard library
features in Python
Detailed, extended case studies with code, showing how to
use a few of the more popular Asyncio-compatible third-
party libraries
We’ll begin with a story that illustrates the shift in thinking that must
accompany a transition from threaded to async programming. Then,
we’ll take a look at the changes that were made in the Python
language itself to accommodate async programming. Finally, we’ll
explore some of the ways in which these new features can be used
most effectively.
The new Asyncio features are not going to radically change the way
you write programs. They provide specific tools that make sense
only for specific situations; but in the right situations, asyncio is
exceptionally useful. In this book, we’re going to explore those
situations and how you can best approach them by using the new
Asyncio features.

Conventions Used in This Book

The following typographical conventions are used in this book:
Italic
Indicates new terms, URLs, email addresses, filenames, and file
extensions.

Constant width
 Used for program listings, as well as within paragraphs to refer to
program elements such as variable or function names, databases,
datatypes, environment variables, statements, and keywords.

Constant width bold

Shows commands or other text that should be typed literally by
the user.

Constant width italic

Shows text that should be replaced with user-supplied values or
by values determined by context.

TIP
This element signifies a tip or suggestion.

NOTE
This element signifies a general note.

WARNING
This element indicates a warning or caution.

O’Reilly Online Learning

NOTE
For more than 40 years, O’Reilly Media has provided technology and
business training, knowledge, and insight to help companies succeed.
Our unique network of experts and innovators share their knowledge
and expertise through books, articles, conferences, and our online
learning platform. O’Reilly’s online learning platform gives you on-
demand access to live training courses, in-depth learning paths,
interactive coding environments, and a vast collection of text and
video from O’Reilly and 200+ other publishers. For more information,
please visit http://oreilly.com.

How to Contact Us
Please address comments and questions concerning this book to the
publisher:

O’Reilly Media, Inc.

1005 Gravenstein Highway North

Sebastopol, CA 95472

800-998-9938 (in the United States or Canada)

707-829-0515 (international or local)

707-829-0104 (fax)

We have a web page for this book, where we list errata, examples,
and any additional information. It can be accessed at
https://oreil.ly/using-asyncio-in-python.
Email [email protected] to comment or ask technical
questions about this book.
For more information about our books, courses, conferences, and
news, see our website at http://www.oreilly.com.
Find us on Facebook: http://facebook.com/oreilly
Follow us on Twitter: http://twitter.com/oreillymedia
Watch us on YouTube: http://www.youtube.com/oreillymedia

Acknowledgments
Many thanks to Ashwini Balnaves and Kevin Baker for working
through very early drafts of this book and providing invaluable
feedback. I am deeply grateful to Yury Selivanov for making precious
time available to review an earlier incarnation of this book, when it
was first published as an O’Reilly report. Finally, I would also like to
thank the team at O’Reilly for their excellent editorial support.
Chapter 1. Introducing
Asyncio

My story is a lot like yours, only more interesting ’cause it involves

robots.
—Bender, Futurama episode “30% Iron Chef”
The most common question I receive about Asyncio in Python 3 is
this: “What is it, and what do I do with it?” The answer you’ll hear
most frequently is likely something about being able to execute
multiple concurrent HTTP requests in a single program. But there is
more to it than that—much more. Asyncio requires changing how
you think about structuring programs.
The following story provides a backdrop for gaining this
understanding. The central focus of Asyncio is on how best to best
perform multiple tasks at the same time—and not just any tasks, but
specifically tasks that involve waiting periods. The key insight
required with this style of programming is that while you wait for this
task to complete, work on other tasks can be performed.

The Restaurant of ThreadBots

The year is 2051, and you find yourself in the restaurant business.
Automation, largely by robot workers, powers most of the economy,
but it turns out that humans still enjoy going out to eat once in a
while. In your restaurant, all the employees are robots—humanoid,
of course, but unmistakably robots. The most successful
manufacturer of robots is Threading Inc., and robot workers from
this company have come to be called “ThreadBots.”
Except for this small robotic detail, your restaurant looks and
operates like one of those old-time establishments from, say, 2020.
Your guests will be looking for that vintage experience. They want
fresh food prepared from scratch. They want to sit at tables. They
want to wait for their meals—but only a little. They want to pay at
the end, and they sometimes even want to leave a tip, for old times’
sake.
Being new to the robotic restaurant business, you do what every
other restaurateur does and hire a small fleet of robots: one to greet
diners at the front desk (GreetBot), one to wait tables and take
orders (WaitBot), one to do the cooking (ChefBot), and one to
manage the bar (WineBot).
Hungry diners arrive at the front desk and are welcomed by
GreetBot, your front-of-house ThreadBot. They are then directed to
a table, and once they are seated, WaitBot takes their order. Then
WaitBot brings that order to the kitchen on a slip of paper (because
you want to preserve that old-time experience, remember?). ChefBot
looks at the order on the slip and begins preparing the food. WaitBot
will periodically check whether the food is ready, and when it is, will
immediately take the dishes to the customers’ table. When the
guests are ready to leave, they return to GreetBot, who calculates
the bill, takes their payment, and graciously wishes them a pleasant
evening.
Your restaurant is a hit, and you soon grow to have a large customer
base. Your robot employees do exactly what they’re told, and they
are perfectly good at the tasks you assign them. Everything is going
really well, and you couldn’t be happier.
Over time, however, you do begin to notice some problems. Oh, it’s
nothing truly serious; just a few things that seem to go wrong. Every
other robotic restaurant owner seems to have similar niggling
glitches. It is a little worrying that these problems seem to get worse
the more successful you become.
Though rare, there are the occasional collisions that are very
unsettling: sometimes, when a plate of food is ready in the kitchen,
WaitBot will grab it before ChefBot has even let go of the plate. This
usually ends up with the plate shattering and leaves a big mess.
ChefBot cleans it up, of course, but still, you’d think that these top-
notch robots would know how to be a bit more synchronized with
each other. This happens at the bar too: sometimes WineBot will
place a drink order on the bar, and WaitBot will grab it before
WineBot has let go, resulting in broken glass and spilled Nederburg
Cabernet Sauvignon.
Also, sometimes GreetBot will seat new diners at exactly the same
moment that WaitBot has decided to clean what it thought was an
empty table. It’s pretty awkward for the diners. You’ve tried adding
delay logic to WaitBot’s cleaning function, or delays to GreetBot’s
seating function, but these don’t really help, and the collisions still
occur. But at least these events are rare.
Well, they used to be. Your restaurant has become so popular that
you’ve had to hire a few more ThreadBots. For very busy Friday and
Saturday evenings, you’ve had to add a second GreetBot and two
extra WaitBots. Unfortunately, the hiring contracts for ThreadBots
mean that you have to hire them for the whole week, so this
effectively means that for most of the quiet part of the week, you’re
carrying three extra ThreadBots that you don’t really need.
The other resource problem, in addition to the extra cost, is that it’s
more work for you to deal with these extra ThreadBots. It was fine
to keep tabs on just four bots, but now you’re up to seven. Keeping
track of seven ThreadBots is a lot more work, and because your
restaurant keeps getting more and more famous, you become
worried about taking on even more ThreadBots. It’s going to
become a full-time job just to keep track of what each ThreadBot is
doing. And another thing: these extra ThreadBots are using up a lot
more space inside your restaurant. It’s becoming a tight squeeze for
your customers, what with all these robots zipping around. You’re
worried that if you need to add even more bots, this space problem
is going to get even worse. You want to use the space in your
restaurant for customers, not ThreadBots.
The collisions have also become worse since you added more
ThreadBots. Now, sometimes two WaitBots take the exact same
order from the same table at the same time. It’s as if they both
noticed that the table was ready to order and moved in to take it,
without noticing that the other WaitBot was doing the exact same
thing. As you can imagine, this results in duplicated food orders,
which causes extra load on the kitchen and increases the chance of
collisions when picking up the ready plates. You’re concerned that if
you add more WaitBots, this problem might get worse.
Time passes.
Then, during one very, very busy Friday night service, you have a
singular moment of clarity: time slows, lucidity overwhelms you, and
you see a snapshot of your restaurant frozen in time. My ThreadBots
are doing nothing! Not really nothing, to be fair, but they’re just…
waiting.
Each of your three WaitBots at different tables is waiting for one of
the diners at their table to give their order. The WineBot has already
prepared 17 drinks, which are now waiting to be collected (it took
only a few seconds), and is waiting for a new drink order. One of the
GreetBots has greeted a new party of guests and told them they
need to wait a minute to be seated, and is waiting for the guests to
respond. The other GreetBot, now processing a credit card payment
for another guest that is leaving, is waiting for confirmation on the
payment gateway device. Even the ChefBot, who is currently
cooking 35 meals, is not actually doing anything at this moment, but
is simply waiting for one of the meals to complete cooking so that it
can be plated up and handed over to a WaitBot.
You realize that even though your restaurant is now full of
ThreadBots, and you’re even considering getting more (with all the
problems that entails), the ones that you currently have are not
being fully utilized.
The moment passes, but not the realization. On Sunday, you add a
data collection module to your ThreadBots. For each ThreadBot,
you’re measuring how much time is spent waiting and how much is
spent actively doing work. Over the course of the following week,
the data is collected. Then on Sunday evening, you analyze the
results. It turns out that even when your restaurant is at full
capacity, the most hardworking ThreadBot is idle about 98% of the
time. The ThreadBots are so enormously efficient that they can
perform any task in fractions of a second.
As an entrepreneur, this inefficiency really bugs you. You know that
every other robotic restaurant owner is running their business the
same as you, with many of the same problems. But, you think,
slamming your fist on your desk, “There must be a better way!”
So the very next day, which is a quiet Monday, you try something
bold: you program a single ThreadBot to do all the tasks. Every time
it begins to wait, even for a second, the ThreadBot switches to the
next task to be done in the restaurant, whatever it may be, instead
of waiting. It sounds incredible—only one ThreadBot doing the work
of all the others—but you’re confident that your calculations are
correct. And besides, Monday is a quiet day; even if something goes
wrong, the impact will be small. For this new project, you call the
bot “LoopBot” because it will loop over all the jobs in the restaurant.
The programming was more difficult than usual. It isn’t just that you
had to program one ThreadBot with all the different tasks; you also
had to program some of the logic of when to switch between tasks.
But by this stage, you’ve had a lot of experience with programming
these ThreadBots, so you manage to get it done.
You watch your LoopBot like a hawk. It moves between stations in
fractions of a second, checking whether there is work to be done.
Not long after opening, the first guest arrives at the front desk. The
LoopBot shows up almost immediately, and asks whether the guest
would like a table near the window or near the bar. But then, as the
LoopBot begins to wait, its programming tells it to switch to the next
task, and it whizzes off. This seems like a dreadful error, but then
you see that as the guest begins to say “Window please,” the
LoopBot is back. It receives the answer and directs the guest to
table 42. And off it goes again, checking for drink orders, food
orders, table cleanup, and arriving guests, over and over again.
Late Monday evening, you congratulate yourself on a remarkable
success. You check the data collection module on the LoopBot, and
it confirms that even with a single ThreadBot doing the work of
seven, the idle time was still around 97%. This result gives you the
confidence to continue the experiment all through the rest of the
week.
As the busy Friday service approaches, you reflect on the great
success of your experiment. For service during a normal working
week, you can easily manage the workload with a single LoopBot.
And you’ve noticed another thing: you don’t see any more collisions.
This makes sense; since there is only one LoopBot, it cannot get
confused with itself. No more duplicate orders going to the kitchen,
and no more confusion about when to grab a plate or drink.
Friday evening service begins, and as you had hoped, the single
ThreadBot keeps up with all the customers and tasks, and service is
proceeding even better than before. You imagine that you can take
on even more customers now, and you don’t have to worry about
having to bring on more ThreadBots. You think of all the money
you’re going to save.
Then, unfortunately, something goes wrong: one of the meals, an
intricate soufflé, has flopped. This has never happened before in
your restaurant. You begin to study the LoopBot more closely. It
turns out that at one of your tables, there is a very chatty guest.
This guest has come to your restaurant alone and keeps trying to
make conversation with the LoopBot, even sometimes holding your
LoopBot by the hand. When this happens, your LoopBot is unable to
dash off and attend to the ever-growing list of tasks elsewhere in
your restaurant. This is why the kitchen produced its first flopped
soufflé: your LoopBot was unable to make it back to the kitchen to
remove the dish from the oven because it was held up by a guest.
Friday service finishes, and you head home to reflect on what you
have learned. It’s true that the LoopBot was able to do all the work
that was required in the busy Friday service; but on the other hand,
your kitchen produced its very first spoiled meal, something that has
never happened before. Chatty guests used to keep WaitBots busy
all the time, but that never affected the kitchen service at all.
All things considered, you decide it is still better to continue using a
single LoopBot. Those worrying collisions no longer occur, and there
is much more space in your restaurant—space that you can use for
more customers. But you realize something profound about the
LoopBot: it can be effective only if every task is short, or at least can
be performed in a short period of time. If any activity keeps the
LoopBot busy for too long, other tasks will begin to suffer neglect.
It is difficult to know in advance which tasks may take too much
time. What if a guest orders a cocktail that requires intricate
preparation, taking much more time than usual? What if a guest
wants to complain about a meal at the front desk, refuses to pay,
and grabs the LoopBot by the arm, preventing it from task
switching? You decide that instead of figuring out all of these issues
up front, it is better to continue with the LoopBot, record as much
information as possible, and deal with any problems later as they
arise.
More time passes.
Gradually, other restaurant owners notice your operation, and
eventually they figure out that they too can get by, and even thrive,
with only a single ThreadBot. Word spreads. Soon every single
restaurant operates in this way, and it becomes difficult to remember
that robotic restaurants ever operated with multiple ThreadBots at
all.
Epilogue
In our story, each robot worker in the restaurant is a single thread.
The key observation in the story is that the nature of the work in the
restaurant involves a great deal of waiting, just as requests.get() is
waiting for a response from a server.
In a restaurant, the worker time spent waiting isn’t huge when slow
humans are doing manual work, but when super-efficient and quick
robots are doing the work, nearly all their time is spent waiting. In
computer programming, the same is true when network
programming is involved. CPUs do work and wait on network I/O.
CPUs in modern computers are extremely fast—hundreds of
thousands of times faster than network traffic. Thus, CPUs running
networking programs spend a great deal of time waiting.
The insight in the story is that programs can be written to explicitly
direct the CPU to move between work tasks as necessary. Although
there is an improvement in economy (using fewer CPUs for the same
work), the real advantage, compared to a threading (multi-CPU)
approach, is the elimination of race conditions.
It’s not all roses, however: as we found in the story, there are
benefits and drawbacks to most technology solutions. The
introduction of the LoopBot solved a certain class of problems but
also introduced new problems—not the least of which is that the
restaurant owner had to learn a slightly different way of
programming.

What Problem Is Asyncio Trying to Solve?

For I/O-bound workloads, there are exactly (only!) two reasons to
use async-based concurrency over thread-based concurrency:
 Asyncio offers a safer alternative to preemptive multitasking
(i.e., using threads), thereby avoiding the bugs, race
conditions, and other nondeterministic dangers that
frequently occur in nontrivial threaded applications.
Asyncio offers a simple way to support many thousands of
simultaneous socket connections, including being able to
handle many long-lived connections for newer technologies
like WebSockets, or MQTT for Internet of Things (IoT)
applications.

That’s it.
Threading—as a programming model—is best suited to certain kinds
of computational tasks that are best executed with multiple CPUs
and shared memory for efficient communication between the
threads. In such tasks, the use of multicore processing with shared
memory is a necessary evil because the problem domain requires it.
Network programming is not one of those domains. The key insight
is that network programming involves a great deal of “waiting for
things to happen,” and because of this, we don’t need the operating
system to efficiently distribute our tasks over multiple CPUs.
Furthermore, we don’t need the risks that preemptive multitasking
brings, such as race conditions when working with shared memory.
However, there is a great deal of misinformation about other
supposed benefits of event-based programming models. Here are a
few of the things that just ain’t so:
Asyncio will make my code blazing fast.
Unfortunately, no. In fact, most benchmarks seem to show that
threading solutions are slightly faster than their comparable
Asyncio solutions. If the extent of concurrency itself is considered
a performance metric, Asyncio does make it a bit easier to create
very large numbers of concurrent socket connections, though.
Operating systems often have limits on how many threads can be
 created, and this number is significantly lower than the number
of socket connections that can be made. The OS limits can be
changed, but it is certainly easier to do with Asyncio. And while
we expect that having many thousands of threads should incur
extra context-switching costs that coroutines avoid, it turns out
to be difficult to benchmark this in practice.1 No, speed is not the
benefit of Asyncio in Python; if that’s what you’re after, try
Cython instead!
Asyncio makes threading redundant.
Definitely not! The true value of threading lies in being able to
write multi-CPU programs, in which different computational tasks
can share memory. The numerical library numpy, for instance,
already makes use of this by speeding up certain matrix
calculations through the use of multiple CPUs, even though all
the memory is shared. For sheer performance, there is no
competitor to this programming model for CPU-bound
computation.

Asyncio removes the problems with the GIL.

Again, no. It is true that Asyncio is not affected by the GIL,2 but
this is only because the GIL affects multithreaded programs. The
“problems” with the GIL that people refer to occur because it
prevents true multicore parallelism when using threads. Since
Asyncio is single-threaded (almost by definition), it is unaffected
by the GIL, but it cannot benefit from multiple CPU cores either.3
It is also worth pointing out that in multithreaded code, the
Python GIL can cause additional performance problems beyond
what has already been mentioned in other points: Dave Beazley
presented a talk on this called “Understanding the Python GIL” at
PyCon 2010, and much of what is discussed in that talk remains
true today.

Asyncio prevents all race conditions.

 False. The possibility of race conditions is always present with
any concurrent programming, regardless of whether threading or
event-based programming is used. It is true that Asyncio can
virtually eliminate a certain class of race conditions common in
multithreaded programs, such as intra-process shared memory
access. However, it doesn’t eliminate the possibility of other kinds
of race conditions, such as the interprocess races with shared
resources common in distributed microservices architectures. You
must still pay attention to how shared resources are being used.
The main advantage of Asyncio over threaded code is that the
points at which control of execution is transferred between
coroutines are visible (because of the presence of await
keywords), and thus it is much easier to reason about how
shared resources are being accessed.

Asyncio makes concurrent programming easy.

Ahem, where do I even begin?
The last myth is the most dangerous one. Dealing with concurrency
is always complex, regardless of whether you’re using threading or
Asyncio. When experts say “Asyncio makes concurrency easier,” what
they really mean is that Asyncio makes it a little easier to avoid
certain kinds of truly nightmarish race condition bugs—the kind that
keep you up at night and that you tell other programmers about in
hushed tones over campfires, wolves howling in the distance.
Even with Asyncio, there is still a great deal of complexity to deal
with. How will your application support health checks? How will you
communicate with a database that may allow only a few connections
—much fewer than your five thousand socket connections to clients?
How will your program terminate connections gracefully when you
receive a signal to shut down? How will you handle (blocking!) disk
access and logging? These are just a few of the many complex
design decisions that you will have to answer.
Application design will still be difficult, but the hope is that you will
have an easier time reasoning about your application logic when you
have only one thread to deal with.

1 Research in this area seems hard to find, but the numbers seem to be
around 50 microseconds per threaded context switch on Linux on modern
hardware. To give a (very) rough idea: one thousand threads implies 50 ms
total cost just for the context switching. It does add up, but it isn’t going to
wreck your application either.
2 The global interpreter lock (GIL) makes the Python interpreter code (not
your code!) thread-safe by locking the processing of each opcode; it has the
unfortunate side effect of effectively pinning the execution of the interpreter
to a single CPU, and thus preventing multicore parallelism.
3 This is similar to why JavaScript lacks a GIL “problem”: there is only one
thread.
Chapter 2. The Truth About
Threads

Let’s be frank for a moment—you really don’t want to use Curio.

All things equal, you should probably be programming with
threads. Yes, threads. THOSE threads. Seriously. I’m not kidding.
—Dave Beazley, “Developing with Curio”
If you’ve never heard of threads before, here’s a basic description:
threads are a feature provided by an operating system (OS), made
available to software developers so that they may indicate to the OS
which parts of their program may be run in parallel. The OS decides
how to share CPU resources with each of the parts, much as the OS
decides to share CPU resources with all the other different programs
(processes) running at the same time.
Since you’re reading an Asyncio book, this must be the part where I
tell you, “Threads are terrible, and you should never use them,”
right? Unfortunately, the situation is not so simple. We need to
weigh the benefits and risks of using threads, just like with any
technology choice.
This book is not supposed to be about threads at all. But there are
two problems here: Asyncio is offered as an alternative to threading,
so it’s hard to understand the value proposition without some
comparison; and even when using Asyncio, you will still likely have
to deal with threads and processes, so you need to know something
about threading.
 WARNING
The context of this discussion is exclusively concurrency in network
programming applications. Preemptive multithreading is also used in
other domains, where the trade-offs are entirely different.

Benefits of Threading
These are the main benefits of threading:
Ease of reading code
Your code can run concurrently, but still be set out in a very
simple, top-down linear sequence of commands to the point
where—and this is key—you can pretend, within the body of your
functions, that no concurrency is happening.

Parallelism with shared memory

Your code can exploit multiple CPUs while still having threads
share memory. This is important in many workloads where it
would be too costly to move large amounts of data between the
separate memory spaces of different processes, for example.

Know-how and existing code

There is a large body of knowledge and best practices available
for writing threaded applications. There is also a huge amount of
existing “blocking” code that depends on multithreading for
concurrent operation.
Now, with Python, the point about parallelism is questionable
because the Python interpreter uses a global lock, called the global
interpreter lock (GIL), to protect the internal state of the interpreter
itself. That is, it provides protection from the potential catastrophic
effects of race conditions between multiple threads. A side effect of
the lock is that it ends up pinning all threads in your program to a
single CPU. As you might imagine, this negates any parallelism
performance benefits (unless you use tools like Cython or Numba to
maneuver around the limitation).
The first point regarding perceived simplicity, however, is significant:
threading in Python feels exceptionally simple, and if you haven’t
been burned before by impossibly hard race condition bugs,
threading offers a very attractive concurrency model. Even if you
have been burned in the past, threading remains a compelling option
because you will likely have learned (the hard way) how to keep
your code both simple and safe.
I don’t have space to get into safer threaded programming here, but
generally speaking, the best practice for using threads is to use the
ThreadPoolExecutor class from the concurrent.futures module,
passing all required data in through the submit() method.
Example 2-1 shows a basic example.
Example 2-1. Best practice for threading
from concurrent.futures import ThreadPoolExecutor as Executor

def worker(data):
<process the data>
with Executor(max_workers=10) as exe:
future = exe.submit(worker, data)

The ThreadPoolExecutor offers an extremely simple interface for

running functions in a thread—and the best part is that, if needed,
you can convert the pool of threads into a pool of subprocesses
simply by using ProcessPoolExecutor instead. It has the same API
as ThreadPoolExecutor, which means that your code will be little
affected by the change. The executor API is also used in asyncio
and is described in the next chapter (see Example 3-3).
In general, you’ll prefer your tasks to be somewhat short-lived, so
that when your program needs to shut down, you can simply call
Executor.shutdown(wait=True) and wait a second or two to allow
the executor to complete.
Most importantly: if at all possible, you should try to prevent your
threaded code (in the preceding example, the worker() function)
from accessing or writing to any global variables!

TIP
Raymond Hettinger presented several great guidelines for safer
threaded code at PyCon Russia 2016 and PyBay 2017. I strongly urge
you to add these videos to your watch list.

Drawbacks of Threading
[N]ontrivial multithreaded programs are incomprehensible to
humans. It is true that the programming model can be improved
through the use of design patterns, better granularity of atomicity
(e.g., transactions), improved languages, and formal methods.
However, these techniques merely chip away at the unnecessarily
enormous non-determinism of the threading model. The model
remains intrinsically intractable.
—Edward A. Lee “The Problem with Threads”
The drawbacks of threading have been mentioned in a few other
places already, but for completeness let’s collect them here anyway:
Threading is difficult
Threading bugs and race conditions in threaded programs are the
hardest kinds of bugs to fix. With experience, it is possible to
design new software that is less prone to these problems, but in
nontrivial, naively designed software, they can be nearly
impossible to fix, even by experts. Really!

Threads are resource-intensive

Threads require extra operating system resources to create, such
as preallocated, per-thread stack space that consumes process
virtual memory up front. This is a big problem with 32-bit
operating systems, because the address space per process is
limited to 3 GB.1 Nowadays, with the widespread availability of
64-bit operating systems, virtual memory isn’t as precious as it
used to be (addressable space for virtual memory is typically 48
bits; i.e., 256 TiB). On modern desktop operating systems, the
physical memory required for stack space for each thread isn’t
even allocated by the OS until it is required, including stack space
per thread. For example, on a modern, 64-bit Fedora 29 Linux
with 8 GB memory, creating 10,000 do-nothing threads with this
short snippet:

# threadmem.py
import os
from time import sleep
from threading import Thread
threads = [
Thread(target=lambda: sleep(60)) for i in range(10000)
]
[t.start() for t in threads]
print(f'PID = {os.getpid()}')
[t.join() for t in threads]

leads to the following information in top:

MiB Mem : 7858.199 total, 1063.844 free, 4900.477 used

MiB Swap: 7935.996 total, 4780.934 free, 3155.062 used

PID USER PR NI VIRT RES SHR COMMAND

15166 caleb 20 0 80.291g 131.1m 4.8m python3

Preallocated virtual memory is a staggering ~80 GB (due to 8 MB

stack space per thread!), but resident memory is only ~130 MB.
On a 32-bit Linux system, I would be unable to create this many
because of the 3 GB user-space address-space limit, regardless
of actual consumption of physical memory. To get around this
problem on 32-bit systems, it is sometimes necessary to
 decrease the preconfigured stack size, which you can still do in
Python today, with threading.stack_size([size]). Obviously,
decreasing stack size has implications for runtime safety with
respect to the degree to which function calls may be nested,
including recursion. Single-threaded coroutines have none of
these problems and are a far superior alternative for concurrent
I/O.
Threading can affect throughput
At very high concurrency levels (say, >5,000 threads), there can
also be an impact on throughput due to context-switching costs,
assuming you can figure out how to configure your operating
system to even allow you to create that many threads! It has
become so tedious on recent macOS versions, for example, to
test the preceding 10,000 do-nothing-threads example, that I
gave up trying to raise the limits at all.
Threading is inflexible
The operating system will continually share CPU time with all
threads regardless of whether a thread is ready to do work or
not. For instance, a thread may be waiting for data on a socket,
but the OS scheduler may still switch to and from that thread
thousands of times before any actual work needs to be done. (In
the async world, the select() system call is used to check
whether a socket-awaiting coroutine needs a turn; if not, that
coroutine isn’t even woken up, avoiding any switching costs
completely.)
None of this information is new, and the problems with threading as
a programming model are not platform-specific either. For example,
this is what the Microsoft Visual C++ documentation says about
threading:
Discovering Diverse Content Through
Random Scribd Documents
the wits of the men who are paid high salaries for detecting them. They
belong to quite different spheres. If we are looking for analogies we
should look up and not down.
“You wanted to know,” he said, “what was the difference between stealing
and check-raising. Now, let me ask you a question. What’s the difference
between check-raising and some of those big financial operations we’ve
all been reading about? Suppose I have a check for five dollars, and I put
my brains into it and I manipulate it so that I can pass it off for five
hundred. I shove it in to the cashier and he takes it. Before he finds out
his mistake, I have made myself scarce. What’s the difference between
that transaction and what the ‘big fellows’ on the street are doing?” He
mentioned several names that I had not thought of in that connection.
“The difference,” said I, “is—” Then it occurred to me that it was a subject
to which I should give further thought. So we postponed the conversation
till he should call again—which he never did.

* * * * * *
I may be doing injustice to my friend the forger, but he gave me the
impression that he considered himself to be, on the whole, a rather
admirable character. His proposed change of business seemed to be
rather a concession to the prejudices of the legal profession than the
result of any personal scruple. As he saw himself he was a man of
idealistic temper whose ideals conflicted with social usage. Society was
all the time getting into his way, and in the inevitable collisions he had
usually the worst of it. He regretted this, but he bore no malice. By the
time a man has reached middle life and accumulated a good deal of
experience he takes the world as he finds it.
He had encased himself in a moral system which was self-consistent and
which explained to his own satisfaction all that had happened to him. One
thing fitted into another, and there was no room for self-reproach.

* * * * * *
Many attempts have been made to depict the character of an
accomplished scamp. But Gil Blas and Roderick Random and Jonathan
Wild the Great are after all seen from the outside. The author may attempt
to do them justice, but there is a vein of irony that reveals a judgment of
his own. They lack the essential element of incorrigibility, which is that the
scamp does not suspect himself, has not found himself out.
No novelist has ever been able to give such a portraiture of a complacent
criminal as was given a century ago in the autobiography of Stephen
Burroughs.
Burroughs was the son of a worthy clergyman of Hanover, New
Hampshire, and from the outset was looked upon as a black sheep. As a
mere boy he ran away from home and joined the army, and then with
equal irresponsibility deserted. He became a ship’s surgeon, a
privateersman, then a self-ordained minister, a counterfeiter, a teacher of
youth, a founder of libraries, and a miscellaneous philanthropist. He was a
patriot and an optimist and an enthusiastic worker in the cause of general
education. He was chock-full of fine sentiment and had a gift for its
expression. He enjoyed doing good, though in his own way, and never
neglected any opportunity to rebuke those who he felt were in the wrong.
He had a desire to reform the world, and had no doubt of the plans which
he elaborated. He was capable on occasions of acts of magnanimity,
which, while not appreciated by the public, gave him great pleasure in the
retrospect. The intervals between his various enterprises were spent in
New England jails. These experiences only deepened his love of liberty,
which was one of the passions of his life.
Burroughs had a happy disposition that enabled him to get a measure of
satisfaction out of all the vicissitudes of his life. He had learned neither to
worry nor to repine. He was not troubled by the harsh judgments of his
fellow men, for he had learned to find his happiness in the approbation of
his own conscience.
He writes: “I possess an uncommon share of sensibility, and at the same
time maintain an equality of mind that is uncommon, particularly in the
midst of those occurrences which are calculated to wound the feelings. I
have learned fortitude in the school of adversity. In draining the cup of
bitterness to its dregs, I have been taught to despise the occurrences of
misfortune. This one thing I fully believe, that our happiness is more in our
power than is generally thought, or at least we have the ability of
preventing that misery which is so common to unfortunate situations. No
state or condition in life, but from which we may (if we exercise that
reason which the God of Nature has given us) draw comfort and
happiness. We are too apt to be governed by the opinions of others, and if
they think our circumstances unhappy, to consider them so ourselves, and
of course make them so. The state of mind is the only criterion of
happiness or misery.”
It was from this lofty point of view that Stephen Burroughs wrote the
history of his own life. His tendency to didacticism interferes with the
limpid flow of the narrative. Sometimes a whole chapter will be given over
to moralizings, but the observations are never painful. They all reveal the
author’s cheerful acquiescence in the inevitability of his own actions.
Along with this there is the air of chastened surprise over the fact that he
was made the object of persecution.
At the very beginning of the narrative one recognizes an independence
which would do credit to a better man. In New England, clergymen have
always been looked upon as making good ancestors, and Burroughs
might have been pardoned if he had shown some family pride. From this
weakness he was free. “I am,” he says, “the only son of a clergyman,
living in Hanover, in the State of New Hampshire; and were any to expect
merit from their parentage, I might justly look for that merit. But I am so far
a Republican that I consider a man’s merit to rest entirely with himself,
without any regard to family, blood, or connection.”
The accounts of the escapades of his boyhood are intermingled with
dissertations on the education of youth. “I have been in the habit of
educating youth for seven years, constantly; in the course of my business
I have endeavored to study the operations of the human heart, that I
might be able to afford that instruction which would be salutary; and in this
I find one truth clearly established, viz.: a child will endeavor to be what
you make him think mankind in general are.”
The neglect of this truth on the part of his parents and teachers was the
cause of much annoyance to Burroughs. Throughout his life he was the
innocent victim of an educational mistake. Though after a while he
learned to forgive the early injustice, one can see that it rankled. He
endeavored to think well of mankind in general, but it was more difficult
than if he had been habituated to the exercise in infancy.
At Dartmouth young Burroughs was peculiarly unfortunate; he fell into bad
company. As an unkind fate would have it, his room-mate was an
exemplary young man who was studying for the ministry. It appears that
this misguided youth attempted to entice him into what he describes as “a
sour, morose, and misanthropic line of conduct.” Nothing could have been
more disastrous. “To be an inmate with such a character, you will readily
conceive, no way comported with a disposition like mine, and
consequently we never enjoyed that union and harmony of feeling in our
intercourse as room-mates which was necessary for the enjoyment of
social life.”
To the malign influence of his priggish room-mate several misfortunes
were attributed. In endeavoring to restore the moral equilibrium which had
been disturbed by the too great scrupulosity of his chum, he exerted too
much strength in the other direction. The result was that “a powerful
triumvirate” was formed against him in the Faculty. The triumvirate
triumphed and his connection with Dartmouth ended suddenly.
This gave occasion to a chapter on the failure of the institutions of
learning to prepare for real life. The author declares “more than one half
of the time spent in the universities, according to their present
establishment on this continent, is thrown away, and that my position is
founded in fact I will endeavor to prove.”
I do not see how his argument is affected by the fact to which the editor
calls attention in a carping footnote. “It is not strange that the author
should reason in this manner. He was expelled from college in the second
quarter of his second year, and in fact he studied but little while he was a
member.” The editor, I fear, had a narrow mind and judged according to
an academic standard which Burroughs would have despised.
From the uncongenial limitations of a college town it was a satisfaction to
escape to sea. Here Burroughs’s versatility stood him in good stead.
“Having no doctor engaged, I undertook to act in that capacity; and after
obtaining the assistance, advice, and direction of an old practitioner,
together with marks set on each parcel of medicine, I thought myself
tolerably well qualified to perform the office of a physician on board the
ship.”
From his seafaring life Burroughs returned with his reputation under a
cloud. There were ugly rumors afloat which were readily believed by a
censorious world. For once he confesses that his philosophy failed him. “I
returned to my father’s house sunken and discouraged; the world
appeared a gloomy chaos; the sun arose to cast a sickly glimmer on
surrounding objects; the flowers of the field insulted my feelings with their
gayety and splendor; the frolicsome lamb, the playful kitten, and the antic
colt were beheld with those painful emotions which are beyond
description. Shall all nature, shall the brute creation break out into
irregular transports, by the overflowing of pleasing sensations, whilst I am
shut out from even the dim rays of hope?”
Certainly not. To a mind constituted as was his there was an absurdity in
the very suggestion. The brute creation should not have any monopoly of
comfortable sensations, so he cheered up immediately and spent the next
year loafing around his father’s house.
He had been on the coast of Africa and had taken part in some strange
scenes, but his moral sense had not been blunted to such an extent that
he could not grieve over some infractions of the moral law which he
observed in peaceful Hanover. He regretted that he had been led
inadvertently by a young man named Huntington to join a party which
robbed a farmer’s beehive.
“For some unaccountable reason or other, youth are carried away with
false notions of right and wrong. I know, for instance, that Huntington
possessed those principles of integrity that no consideration would have
induced him to deprive another of any species of property, except fruit,
bees, pigs, and poultry. And why it is considered by youth that depriving
another of these articles is less criminal than stealing any other kind of
property, I cannot tell.”
Burroughs himself was inclined to take a harsher view of these
transgressions than he did of some others; for example, of counterfeiting,
in which he was afterwards for a time engaged during one of his brief
pastorates.
The argument by which his scruples in this particular were overcome are
worth repeating. The law was indeed violated in its letter, but might not a
justification be found by one who interpreted it in a large spirit of charity?
“Money is of itself of consequence only as we annex to it a nominal value
as the representation of property. Therefore we find the only thing
necessary to make a matter valuable is to induce the world to deem it so;
and let that esteem be raised by any means whatever, yet the value is the
same, and no one becomes injured by receiving it at the valuation.”
The principle of fiat money having been established, the only question
that remained was whether the circumstances of the times were such as
to justify him in issuing the fiat. The answer was in the affirmative. “That
an undue scarcity of cash now prevails is a truth too obvious for me to
attempt to prove. Hence whoever contributes to increase the quantity of
cash does not only himself but likewise the community an essential
benefit.”
It was in his attempt to benefit the community in this way that he first
experienced the ingratitude of republics, being landed in the Northampton
jail.
But to see Burroughs at his best one must enter into his thoughts at that
crisis in his life when he determined that his true vocation was preaching.
He lingers fondly on his emotions at that period. It was at a time when he
had been driven out of Hanover for conduct which had outraged the
feelings of that long-suffering community.
“One pistareen was all the ready cash I had on hand, and the suddenness
with which I departed deprived me of the chance to raise more. Traveling
on leisurely I had time for reflection.”
As was usually the case when he reflected, he grew more serene and
enjoyed a frame of mind that bordered on the heroic.
“I began to look about me to see what was to be done in my present
situation and to what business I could turn my attention. The practice of
law, which would have been most to my mind, I could not undertake until I
had spent some time in the study, which would have been attended with
expense far beyond my abilities; therefore this object must be laid aside.
Physic was under the same embarrassments; business in the mercantile
line I could not pursue for want of capital. ... What can be done? There is
one thing, said contrivance, that you can do, and it will answer your
purpose—preach.”
The idea came to him as an inspiration, but immediately there was
suggested an objection which to a less resourceful mind would have
seemed insuperable. “What an appearance should I make in my present
dress? which consisted of a light gray coat, with silver-plated buttons,
green vest, and red velvet breeches.”
Down the Connecticut valley he trudged, calling to mind his father’s old
sermons and gradually working himself into a state of pious rapture. The
heart of no young pulpiteer beat with more appropriate emotions than his,
when on the next Sunday, under an assumed name, he preached his first
sermon in the village of Ludlow. “I awoke with anxious palpitation for the
issue of the day. I considered this as the most important scene of my life
—that, in a great measure, my future happiness or wretchedness
depended on my conduct this day. The time for assembling approached! I
saw the people come together. My feelings were up in arms against me,
my heart would almost leap into my mouth. What a strange thing, said I, is
man! Why am I thus perturbated by these whimsical feelings!”
The moment he began the service these perturbations came to an end.
Words came in a steady flow, and he felt sure that he had found his true
calling in life. “No monarch when seated on a throne had more sensible
feelings of prosperity than what I experienced at this time.”
The neighboring town of Pelham being without a minister, Burroughs
presented himself as a candidate, and was enthusiastically accepted. He
made a specialty of funeral sermons, and was soon in demand in all the
surrounding country. It was at this time also that he became acquainted
with the coiner who showed him how he might surreptitiously increase the
amount of cash in circulation. All went well till an enemy appeared who
called him by name and revealed his antecedents. All Pelham was in an
uproar, for the Pelhamites were “a people generally possessing violent
passions, which, once disturbed, raged uncontrolled by the dictates of
reason, unpolished in their manners, possessing a jealous disposition,
and either very friendly or very inimical, not knowing a medium between
these extremes.”
In this case they suddenly became very inimical, and Burroughs was
again compelled to depart under cover of darkness. His night thoughts
were always among his very best.
“Journeying on, I had time for reflection. At the dead of night—all alone—
reflection would have its operation. A very singular scene have I now
passed through, said I, and to what does it amount? Have I acted with
propriety as a man, or have I deviated from the path of rectitude? I have
had an unheard-of, disagreeable part to act; I do not feel entirely satisfied
with myself in this business, and yet I do not know how I could have done
otherwise, and have made the matter better. My situation has been such
that I have violated the principle of veracity which we implicitly pledge
ourselves to maintain towards each other, as a general thing, in society.
Whether my peculiar circumstances would warrant such a line of
procedure is the question. I know many things will be said in favor of it as
well as against it.”
From this difficult question of casuistry he found relief in reverting to the
one instance in which he had been clearly wrong, viz., joining the young
men in Hanover in their raid on the farmer’s beehive. “My giving
countenance to an open breach of the laws of the land in the case of the
bees was a matter in which I was justly reprehensible; but that matter is
now past. I must take things as they are, and under these circumstances
do the best I can. I know the world will blame me, but I wish to justify my
conduct to myself, let the world think what it may.”
In this endeavor he was highly successful; and as he walked on, his
spirits rose. He contrasted his own clear views with the muddled ideas of
his late parishioners. “They understand the matter in the gross, that I have
preached under a fictitious name and character, and consequently have
roused many ideas in the minds of the people not founded on fact.
Therefore they concluded from this general view the whole to be founded
on wrong. The name impostor is therefore easily fixed on my character.
An impostor, we generally conceive, puts on feigned appearances in
order to enrich or aggrandize himself to the damage of others. That this is
not the case with me in this transaction, I think is clear. That I have aimed
at nothing but the bare necessaries of life, is a fact.”
Having thus cleared himself of the charge of imposture, he determined to
rest his case on the broad ground of religious liberty. “That I have a good
and equitable right to preach, if I choose and others choose to hear me, is
a truth of which I entertain no doubt.”
When he was pursued into the borders of the town of Rutland, it was too
much for his patience. “I turned and ran about twenty rods down a small
hill, and the Pelhamites all after me, hallooing with all their might, ‘Stop
him! stop him!’ To be pursued like a thief, an object of universal
speculation to the inhabitants of Rutland, gave me very disagreeable
sensations, which I determined not to bear. I therefore stopped, took up a
stone, and declared that the first who should approach me I would kill on
the spot. To hear such language and to see such a state of determined
defiance in one whom they had lately reverenced as a clergyman struck
even the people of Pelham with astonishment and fear.”
By the way, there follows a scene which makes us suspect that parts of
Massachusetts in the good old days may have had a touch of “the wild
West.” The two deacons who were leaders of the mob drew attention to
the fact that besides having come to them under false pretenses
Burroughs had absconded with five dollars that had been advanced on
his salary. He owed them one sermon which was theirs of right. In the
present excited state of public opinion it was obviously impossible for
Burroughs to deliver the sermon, but it was suggested that he might give
an equivalent. A peacemaker intervened, saying, “Wood keeps an
excellent tavern hard by; I propose for all to move up there.” This proposal
was accepted by all. “I therefore came down, and we all went up towards
the tavern. I called for drink, according to the orator’s advice, to the
satisfaction of all.”
After that the career of Burroughs went on from bad to worse, but never
was he without the inner consolations that belong to those who are
misunderstood by the world. Even when he unsuccessfully sought to set
fire to the jail he was full of fine sentiments borrowed from Young’s “Night
Thoughts.” He quotes the whole passage beginning

Night, sable goddess! from her ebon throne.

This he seems to consider to be in some way a justification for his action.

He is ever of the opinion that a man’s heart can not be wrong so long as
he is able to quote poetry.
The various incarcerations to which he was subjected might only have
imbittered a less magnanimous mind. They rather instilled into Burroughs
a missionary spirit. He felt that he ought to take more pains to enlighten
the ignorance of the world in regard to his excellent qualities. “I have
many times lamented my want of patient perseverance in endeavoring to
convince my persecutors of their wrong by the cool dictates of reason.
Error once seen ought to be corrected. The pruning hook should never be
laid aside; then we should live up to the condition of our nature, which
requires a state of improving and progressing in knowledge till time shall
cease.”
But even Burroughs was human. It is easier to bear great misfortunes
than to meet the petty annoyances of every-day life. To one who plans his
life in such a way as to depend largely on the casual gifts of strangers,
their dilatoriness is often a cause of real anxiety.
Here is a painful incident which happened to him in Philadelphia. He
applied to a member of Congress for a small sum of money. The
gentleman was not all that he should have been. “The most striking
features of his character were his great fondness for close metaphysical
reasoning and a habit of great economy in his domestic concerns, and he
had so long practiced upon this system that any variation from it in a
person’s conduct or any want of success in a person’s undertakings were,
in his view, perfectly wrong. This was the man to whom I applied as my
ultimatum.”
We can see at a glance that such a man was likely to be disappointing.
“I described my circumstances to him in as clear terms as possible, and
afterwards told him of the request I wished to make. Without giving me an
answer either in the affirmative or the negative, he went on with a lengthy
discourse to prove that my system of economy had been wrong, drawing
a comparison between his prosperity and my adversity, and then pointed
out a certain line of conduct that I ought then to take up and observe, and
offered to assist me in prosecuting such; but as his plan had many things
in it which I could not reconcile my mind to, I took the liberty of reasoning
with him upon a better plan which I had marked out in my own mind.”
Upon this, the congressman became obstinate and would do nothing. His
depravity came to Burroughs as a sudden shock.
“When I took a view of the world, of the pomp and splendor which
surrounded crowds which perpetually passed before my eyes, to see
them roll in affluence and luxury, inhabiting lofty houses, with superb
equipages, and feasting upon all the delicacies of life, under these affluent
circumstances withholding from me what would never have been missed
from their superfluity, this brought to my mind a train of ideas that were
desperate and horrid.... My eyes lighted up with indignation, my
countenance was fortified with despair, my heart was swollen to that
bigness which was almost too large for my breast to contain. Under this
situation I arose with a tranquil horror, composedly took my hat, and
politely bid Mr. Niles farewell. I believe the desperate emotions of my
heart were apparently manifested to his view by my countenance; his
apparent immovability relaxed, he put his hand in his pocket, and handed
me three dollars. This act of kindness in a moment melted the ferocious
feelings of my heart, all those desperate sensations vanished, and I found
myself a man.”
Dear reader, have you not often taken a part in such a scene? When
instead of handing out your dollars at once you conditioned them upon
adherence to some “line of conduct,”—your conscience accuses you that
you might have pointed even to the buck-saw,—do you realize what a
pitiful spectacle you made of yourself?

* * * * * *
Stephen Burroughs does not at all fulfill our preconceived notion of an
habitual criminal. He did not love evil for its own sake. His crimes were
incidental, and he mentions them only as the unfortunate results of
circumstances beyond his own control. His life was rather spent in the
contemplation of virtue. There were some virtues which came easy to
him, and he made the most of them. Like an expert prestidigitator, he kept
the attention fixed on what was irrelevant, so that what was really going
on passed unnoticed. He had eliminated personal responsibility from his
scheme of things, and then proceeded as if nothing were lacking. He had
one invariable measure for right and wrong. That was right which
ministered to his own peace of body and of mind; that was wrong which
did otherwise.
We are coming to see that that imperturbable egotism is the characteristic
of the “criminal mind” that is least susceptible to treatment. Sins of
passion are often repented of as soon as they are committed. Sins of
ignorance are cured by letting in the light. Sins of weakness yield to an
improved environment. But what are you going to do with the man who is
incapable of seeing that he is in the wrong? Treat him with compassion,
and he accepts the kindness as a tribute to his own merits; attempt to
punish him, and he is a martyr; reason with him, and his controversial
ardor is aroused in defense of his favorite thesis.
Sometimes the lover of humanity, after he has tried everything which he
can think of to make an impression on such a character and to bring him
to a realizing sense of social responsibility, becomes utterly discouraged.
He feels tempted to give up trying any longer. In this he is wrong. He
should not allow himself to be discouraged. Something must be done,
even though nobody knows what it is.
But if the lover of humanity should give up for a time and take a rest by
turning his attention to a more hopeful case, I should not be too hard on
him. My Pardoner, I am sure, must have some indulgence for such a
weakness.
 A MAN UNDER ENCHANTMENT

I SAT down by the wayside of life like a man under enchantment.”

So Nathaniel Hawthorne wrote of his own visionary youth, and,
truth to tell, the spell lasted through life.
The wayside itself was not conducive to dreams. It was a busy
thoroughfare. Eager traffickers jostled one another, and there was
much crying up of new wares. Many important personages went
noisily along. There was a fresh interest in all sorts of good works
and many improvements on the roadway. There were not many
priests or Levites passing by on the other side, for ecclesiasticism
was not in fashion, but there were multitudes of Good Samaritans,
each one intent on his own brand-new device for universal
helpfulness. There were so many of them that the poor man who fell
among philanthropists often sighed for the tender mercies of the
thieves. The thieves, at least, when they had done their work would
let him alone. From time to time there would come groups of eager
reformers, advance agents of the millennium. At last there came
down the road troops hurrying to the front, and there was the distant
sound of battle.
It was a stirring time, the noon of the nineteenth century; and the stir
was nowhere more felt than in New England. It was a ferment of
speculation, a whirl of passion, a time of great aspiration and of no
mean achievement.
But if you would get a sense of all this, do not turn to the pages of
Nathaniel Hawthorne. The ardor of Transcendentalism, the new spirit
of reform, the war between the States,—these were noted, but they
made no very vivid impression on the man who sat under
enchantment. There was an interval between these happenings and
his consciousness that made them seem scarcely
contemporaneous.
It is a fashion in literary criticism to explain an author by his
environment. With Hawthorne this method is not successful. It is not
that his environment was not interesting in itself. His genius was
essentially aloof. It was a plant that drew its nourishment from the air
rather than from the soil. There are some men who have the happy
faculty of making themselves at home wherever they happen to be.
Hawthorne, wherever he had been born, would have looked upon
the scene with something of a stranger’s eye. Indeed, when we think
about it, the wonder is that most of us are able to take the world in
such a matter-of-fact way. One would suppose that we had always
been here, instead of being transient guests who cannot even
engage our rooms a day in advance.
It is perhaps a happy limitation which makes us to forget our slight
tenure, and to feel an absolute ownership in the present moment.
We are satisfied with the passing experience because it appears to
us as permanent.
To the man who sat by the wayside the present moment did not
stand in the sunshine sufficient unto itself. It did not appear, as it did
to the man of affairs, an ultimate and satisfying reality. He was not
unobservant. He saw the persons passing by. But each one, in the
present moment, seemed but a fugitive escaping from the past into
the future. Futile flight! unavailing freedom! for in the Future the Past
stands waiting for it. As he looked at each successive action it was
as one who watches the moving shadow of an old deed, which now
for some creature has become doom.
Did I say that Hawthorne was little influenced by his environment? It
would be truer to say that the environment to which he responded
was that to which most men are so strangely oblivious. He felt what
another Salem mystic has expressed:

Around us ever lies the enchanted land

In marvels rich to thine own sons displayed.

The true-born Yankee has always persisted, in spite of the purists, in

using “I guess” as equivalent to “I think.” To his shrewd good-
humored curiosity, all thinking resolves itself into a kind of
guesswork; and one man has as good a right to his guess as
another.
It is a far cry from the talk of the village store to Emerson and
Hawthorne, but to these New Englanders thinking was still a kind of
guessing. The observer looks at the outward show of things, which
has such an air of finality, and says, “I guess there’s something
behind all this. I guess it’s worth while to look into it.”
Such a mind is not deterred by the warnings of formal logic that there
is “no thoroughfare.” When it leaves the public road and sees the
sign “Private way, dangerous passing,” it says, “that looks
interesting. I guess I’ll take that.”
And from our streets and shops and newspapers, from our
laboratories and lecture rooms and bureaus of statistics, it is, after
all, such a little way to the border-land of mystery, where all minds
are on an equality and where the wisest can but dimly guess the
riddles that are propounded.
Hawthorne belonged to no school or party. To the men of his
generation he was like the minister of whom he writes who preached
with a veil over his face.
Nor is his relation in thought to his ancestry more intimate than that
to his contemporaries. Born to the family of New England Puritanism,
we think we recognize the family likeness—and yet we are not quite
sure. There are traits that suggest a spiritual changeling.
When we enter into the realm of Hawthorne’s imagination we are
conscious of sombre realities.
Is not this a survival of the puritanic spirit, with its brooding
mysticism, its retributive predestination, its sense of the judgment to
come? It was said of Carlyle that he was a Calvinist who had lost his
creed. May not the same be said of Hawthorne? The old New
England theology had in him become attenuated to a mere film, but
through it all may we not see the old New England conscience?
Doubtless there is much of this transmitted influence. Hawthorne
himself insisted upon it. Speaking of “the stern and black-browed
Puritan ancestors,” he said, “Let them scorn me as they will, strong
traits of their nature have intertwined themselves with mine.”
But it is possible to exaggerate such likenesses. In Hawthorne’s
case there is danger of argument in a circle. We say that there is
something in Hawthorne’s imagination, in its sombre mysticism, in its
brooding sense of destiny, which is like that of the spirit of the
inhabitants of Salem and Boston in the old days when they walked
through the narrow streets and through the shadowy woodland ways
pondering the fatal sequences of life.
But how do we see these old Puritans? We see them through
Hawthorne’s eyes. His imagination peoples for us the old houses.
Was Hawthorne’s genius tinged with Puritanism, or are our
conceptions of the Puritan character largely Hawthornesque? It is
not necessary to argue this matter; it might be better to answer “Yes”
to both questions.
It is the privilege of a creative genius to imprint his own features
upon his forbears. It is difficult here to determine which is cause and
which is effect. How marvelously Rembrandt gets the spirit of the
Dutch Burgomeisters! It was fortunate for him that he had such
subjects,—stalwart men with faces that caught the light so
marvelously. Yes, but had it not been for Rembrandt, who would
have told us that these Dutch gentlemen were so picturesque?
The subject of a good artist is accurately figured; the subject of a
great artist is transfigured. We cannot separate the historic reality
from the transfiguring light.
But however Hawthorne may have been influenced by his Puritan
inheritance, it would be hard to find one whose habitual point of view
was further removed from what we are accustomed to call the “New
England conscience.” It is the characteristic of that type of
conscience that it has an ever-present and sometimes oppressive
sense of personal responsibility. It is militant and practical rather than
mystical. To it evil is not something to be endured but something to
be resisted. If there is a wrong it must be righted, and with as little
delay as possible.
The highest praise a Puritan could give his pastor was that he was “a
painful preacher.” Jonathan Mitchell, writing of the beginnings of the
church in Cambridge, says that the people of Cambridge “were a
gracious, savory-spirited people, principled by Mr. Sheperd, liking an
humbling, heart-breaking ministry and spirit.”
The Puritan theology was based on predestination, but the Puritan
temper was not fatalistic. When that latter-day Puritan, Lyman
Beecher, was expounding the doctrines of the divine decrees, one of
his sons asked him, “Father, what if we are decreed to be lost?” The
answer was, “Fight the decrees, my boy!”
The Calvinistic spirit was exactly opposite to the fatalistic
acquiescence which shifts the responsibility from the creature to the
Creator. To be sure the fall of man took place a long time ago, but we
cannot say that it was none of our business. It was not an hereditary
misfortune to be borne with fortitude; it was to be assumed as our
personal guilt. “Original sin” means real sin. Adam sinned as the
typical and representative man, and every man became a sinner. No
individual could plead an alibi. The “conviction of sin” was not the
acquiescence in a penalty,—it was the heartbreaking consciousness
of the “exceeding sinfulness of sin.”
“In Adam’s fall we sinned all.” When they said that, they were
thinking not of Adam, but of themselves. They did it; it was the guilt
that was imputed to them. Sensitive consciences were tortured in the
attempt fully to realize their guilt.
The real inheritors of this type of conscience were to be found
among many of the radical reformers and agitators who were
Hawthorne’s contemporaries and with whom he had little in common.
When their formal creed had fallen off, there remained the sense of
personal guilt for original sin. The sin of the nation and of the whole
social order weighed heavily upon them and tortured them, and they
found relief only in action.
All this was foreign to Hawthorne’s mind. In his treatment of sin there
is always a sense of moral detachment. We are not made to see, as
George Eliot makes us see, the struggle with temptation,—the soul,
like a wild thing, seeing the tempting bait and drawing nearer to the
trap. Hawthorne begins after the deed is done. He shows us the

wild thing taken in a trap

Which sees the trapper coming thro’ the wood.

Of what is the trap made? It is made of a deed already done.

Whence comes the ghostly trapper? He is no stranger in the wood.
There is no staying his advance as he makes his fatal rounds.
In the preface to the “House of the Seven Gables” the author gives
the argument of the story,—“the truth, namely, that the wrong-doing
of one generation lives into the successive ones, and, divesting itself
of every temporary advantage, becomes a pure and uncontrollable
mischief.”
This is the theme of the Greek tragedy—Nemesis. The deed is done
and cannot be undone; the inevitable consequences must be
endured.
In the “Scarlet Letter,” when Hester and Roger Chillingworth review
the past and peer into the future, Hester says, “I said but now that
there can be no good event for him or thee or me who are wandering
together in this gloomy maze of evil, and stumbling at every step
over the guilt wherewith we have strewn our path.”
But is the present stumbling guilt or is it merely misery? The old man
replies, “By the first slip awry thou didst plant the germ of evil, but
since that moment it has been a dark necessity. Ye that have
wronged me are not sinful, save in a kind of typical illusion, neither
am I fiend-like who have snatched a fiend’s office from his hands. It
is our fate. Let the black flower blossom as it may.”
Strange words to come from one who had sat in a Puritan meeting-
house! It is such comment as the Greek Chorus might make
watching the unfolding of the doom of the house of Agamemnon.
And when the tale of the “Scarlet Letter” has been told, how does the
author himself look upon it? How does he distribute praise and
blame?
“To all these shadowy beings so long our near acquaintances—as
well Roger Chillingworth as his companions—we would fain be
merciful. It is a curious subject of observation and inquiry whether
love and hatred be not the same thing at bottom. Each in its utmost
development supposes a high degree of intimacy and heart-
knowledge; each renders one individual dependent for his spiritual
life on another; each leaves the passionate lover or the no less
passionate hater forlorn and desolate by the withdrawal of its
subject. Philosophically considered, therefore, the passions seem
essentially the same except that one happens to be seen in celestial
radiance and the other in a dusky lurid glow.” This is not the Puritan
Conscience uttering itself. It is an illusive and questioning spirit.
If in his attitude toward human destiny Hawthorne was in some
essential respects un-Puritan, so also was he un-modern. There is a
characteristic difference between antique and modern symbols for
those necessary processes, beyond the sphere of our own wills, by
which our lives are determined. The ancients pictured it with austere
simplicity. Life is a simple thread. The Fates spin it. It is drawn out on
the distaff and cut off by the fatal shears.
Compare this with the phrase Carlyle loved to quote, “the roaring
loom of Time.” Life is not a spinning-wheel, but a loom. A million
shuttles fly; a million threads are inextricably interwoven. You cannot
long trace the single thread; you can discern only the growing
pattern. There is inevitable causation, but it is not simple but
complex. The situation at the present moment is the result not of one
cause but of innumerable causes, and it is in turn the cause of
results that are equally incalculable. We are a part of

the web of being blindly wove

By man and beast and air and sea.

Men of science show us how the whole acts upon each part and
each part acts upon the whole. Modern novelists attempt, not always
successfully, to give the impression of the amazing complexity of
actual life, where all sorts of things are going on at the same time.
Whether we look upon it as his limitation or as his good fortune,
Hawthorne adhered to the spinning-wheel rather than the loom. We
see the antique Fates drawing out the thread. A long series of events
follow one another from a single cause.
A part of the power of Hawthorne over our imagination lies in his
singleness of purpose. In “The Marble Faun” we are told, “The
stream of Miriam’s trouble kept its way through this flood of human
life, and neither mingled with it nor was turned aside.”
We are made to see the dark streams that do not mingle nor turn
aside, and we watch their fatal flow.
But is this real, normal life? In such life do not the streams mingle?
Are not evil influences quickly neutralized, as noxious germs die in
the sunshine? No one would more readily acknowledge this than
Hawthorne. He says: “It is not, I apprehend, a healthy kind of mental
occupation to devote ourselves too exclusively to the study of
individual men and women. If the person under examination be one’s
self, the result is pretty certain to be diseased action of the heart
almost before we can snatch a second glance. Or if we take the
freedom to put a friend under the microscope, we thereby insulate
him from many of his true relations, magnify his peculiarities,
inevitably tear him into parts, and of course patch him clumsily
together again. What wonder, then, that we be frightened at such a
monster, which, after all—though we can point to every feature of his
deformity in the real personage—may be said to have been created
mainly by ourselves.”
The critic of Hawthorne could not describe better the limitation of his
stories as pictures of real life. His characters, however clearly
conceived, are insulated from many of their real relations, and their
peculiarities are magnified.
In the preface to “The Scarlet Letter” he says that the tale “wears to
my eye a stern and sombre aspect, too much ungladdened by the
tender and familiar influences which soften almost every scene of
Nature and real life, and which undoubtedly should soften every
picture of them.”
One who would defend Hawthorne the Author against Hawthorne the
Critic must point out the kind of literature to which his work belongs.
When we judge it by the rule of the romance or of the realistic novel,
we fail to do justice to its essential quality. The romancer, the story-
teller pure and simple, is attracted by the swift sequence of events.
His nimble fancy follows a plot as a kitten follows a string. Now it
happens that in a world constituted as ours is the sequence of
events follows a moral order. A good story has always in it an
element of poetic justice. But the romancer does not tell his story for
the sake of the moral. He professes to be as much surprised when it
is discovered as is the most innocent reader. In like manner the
realistic novel, in proportion as it is a faithful portrayal of life, has an
ethical lesson. But the writer disclaims any purpose of teaching it.
His business is to tell what the world is like. He leaves the rest to
your intelligence.
But there is another kind of literature; it is essentially allegory. The
allegorist takes a naked truth and clothes it with the garments of the
imagination. Frequently the clothes do not fit and the poor truth
wanders about awkwardly, self-conscious to the last degree. But if
the artist be a genius the abstract thought becomes a person.
Hawthorne’s work is something more than allegory, but his mind
worked allegorically. His characters were abstract before they
became concrete. He was not a realist aiming to give a
comprehensive survey of the actual world. He consciously selected
the incidents and scenes which would illustrate his theme.
In his conclusion of “The Marble Faun,” when the actors have
withdrawn, the Author comes before the curtain and says that he
designed “the story and the characters to bear, of course, a certain
relation to human nature and human life, but still to be so artfully and
airily removed from our mundane sphere that some laws and
proprieties of their own should be implicitly and insensibly
acknowledged. The idea of the modern Faun, for example, loses all
the poetry and beauty which the Author fancied in it and becomes
nothing better than a grotesque absurdity if we bring it into the actual
light of day.” This is not realism.
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

textbookfull.com