Download Full Version ebook - Visit ebookmeta.

com

Introduction to Data Engineering Daniel Beach

https://ebookmeta.com/product/introduction-to-data-
engineering-daniel-beach/

OR CLICK HERE

DOWLOAD NOW

Discover More Ebook - Explore Now at ebookmeta.com

 Instant digital products (PDF, ePub, MOBI) ready for you
Download now and discover formats that fit your needs...

Start reading on any device today!

Introduction to Environmental Data Science Hsieh

https://ebookmeta.com/product/introduction-to-environmental-data-
science-hsieh/

ebookmeta.com

An Introduction to Thermal Physics 1st Edition Daniel

Schroeder

https://ebookmeta.com/product/an-introduction-to-thermal-physics-1st-
edition-daniel-schroeder/

ebookmeta.com

Engineering Fundamentals: An Introduction to Engineering,

6th Edition Moaveni Saeed

https://ebookmeta.com/product/engineering-fundamentals-an-
introduction-to-engineering-6th-edition-moaveni-saeed/

ebookmeta.com

Microsoft® Project For Dummies® Cynthia Snyder Dionisio

https://ebookmeta.com/product/microsoft-project-for-dummies-cynthia-
snyder-dionisio/

ebookmeta.com
Debt Sustainability A Global Perspective Schuknecht

https://ebookmeta.com/product/debt-sustainability-a-global-
perspective-schuknecht/

ebookmeta.com

Divided we stand the battle over women s rights and family

values that polarized American politics Marjorie Julian
Spruill
https://ebookmeta.com/product/divided-we-stand-the-battle-over-women-
s-rights-and-family-values-that-polarized-american-politics-marjorie-
julian-spruill/
ebookmeta.com

The Oslo Accords 1993 2013 A Critical Assessment 1st

Edition Petter Bauck

https://ebookmeta.com/product/the-oslo-accords-1993-2013-a-critical-
assessment-1st-edition-petter-bauck/

ebookmeta.com

Royal Icing 1st Edition Pinky Cockburn

https://ebookmeta.com/product/royal-icing-1st-edition-pinky-cockburn/

ebookmeta.com

Python 3 Object oriented Programming Building robust and

maintainable software with object oriented design patterns
in Python 2nd Edition Phillips
https://ebookmeta.com/product/python-3-object-oriented-programming-
building-robust-and-maintainable-software-with-object-oriented-design-
patterns-in-python-2nd-edition-phillips/
ebookmeta.com
Love Practically Van Nichole

https://ebookmeta.com/product/love-practically-van-nichole/

ebookmeta.com
Introduction to Data Engineering
Learn the skills needed to break into Data Engineering.

Daniel Beach
This book is for sale at http://leanpub.com/dataengineeringwithpython

This version was published on 2022-02-11

This is a Leanpub book. Leanpub empowers authors and publishers with the Lean Publishing
process. Lean Publishing is the act of publishing an in-progress ebook using lightweight tools and
many iterations to get reader feedback, pivot until you have the right book and build traction once
you do.

© 2019 - 2022 Daniel Beach

Contents

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
What is a Data Engineer? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
What To Expect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
The Focus of This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Knowledge and Experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
What are the topics we will cover? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Chapter 1 - The Theory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

What Is a Data Pipeline? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Data Pipelines built with Passion and Creativity . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Storage and File Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Repeatable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Resilient . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Scalable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
In Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Chapter 2 - Data Pipeline Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Project Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Data Pipeline Code Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Code Readability and Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Tests. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Containerzation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Architecture First . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

Chapter 3 - Pipeline Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

Architecture Applied to Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Data Size and Velocity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Calculating Compute Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Calculating Storage Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Understanding the End Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
CONTENTS

Understanding Cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Code Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Batch vs Streaming Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Puzzle Pieces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

Chapter 4 - Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Access Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
SQL/NoSQL Databases vs files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
File Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Row vs Columnar Storage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Common file types in data engineering. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Parquet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Avro. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
Orc. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
CSV / Flat-file. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
JSON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
Compression. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Storage location. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
Partitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

Chapter 5 - Compute and Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
RAM/Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
CPU/Cores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
Cluster/Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

Chapter 6 - Mastering SQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

Introduction To SQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
Does the type of database matter? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
The fundamentals of SQL/Databases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
OLTP vs. OLAP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
Table design/layout. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
Table Design in Real Life. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
Understanding Indexing Basics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
How to write fast/tune queries. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
Where to look for common problems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
SQL Fundementals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
SQL Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

Chapter 7 - Data Warehousing / Data Lakes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

Data Warehouse vs Data Lake vs Lake House . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
Facts and Dimensions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
CONTENTS

Constraints and Schema. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

Data Types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
Column Names. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
The Role of ID’s in a Data Warehouses or Data Lake. . . . . . . . . . . . . . . . . . . . . . . . 103
CDC / History Tracking. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

Chapter 8 - Data Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

Data Types and Schema. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
Data Types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
Data Size. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
Constraints. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
Data Definitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
Modeling Data Logically. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
Logical data models lead to physical relationships. . . . . . . . . . . . . . . . . . . . . . . . . 111
Grain of Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
Uniqueness of Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
Access Patterns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
Talking to the Business. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
Normal Forms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
De-Duplication of Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
Join Integrity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Keys - Primary and Foreign. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
The Idea Behind Keys. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Relational Databases (SQL) vs Data Lake (File Based) Modeling. . . . . . . . . . . . . . . . . 116
The number of Fact tables and Dimensions and normalization. . . . . . . . . . . . . . . . . . 116
File size and table size matter in the new File-Based Data Lakes. . . . . . . . . . . . . . . . . 117
Partitions vs Indexes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
Walking the data model line between old and new. . . . . . . . . . . . . . . . . . . . . . . . . 118

Chapter 9 - Data Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

What is Data Quality. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
Reasoning about Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
Double meanings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
Data Value Quality. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
Measures of Data Quality. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
Correct Header or Column Names. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
Correct Data/File Formatting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122

Chapter 10 - DevOps for Data Engineers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

Dockerfiles and Docker-compose. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Unit Testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
CI/CD. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
CONTENTS

Automation is the name of the game. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
Introduction
This book is all about the movement of data, specifically developing data pipelines and how to
become an awesome Data Engineer.
With the rise of Business Intelligence, Data Science, Machine Learning, and the general propensity
for companies to gather as much data as possible, the ability to design data pipelines has become a
valuable skill.
Data engineering is an interesting combination of technical and non-technical skills, and varies
from many classic software engineering disciplines. In this book I want to cover the basic topics and
discuss at a high level what are the most important skills to a Data Engineer.

What is a Data Engineer?

What is a Data Engineer? That has changed a lot and will continue to change as technology is
ever-changing, but there are many things that remain constant.
Data Engineers facilitate the movement of data, and enable businesses to consume that data.

• Facilitate data movement.

• Enable data consumption.

The Data Engineer has become a sought-after position and unfortunately, it has not become easier
to find those people with the requisite skills to do the job. Learning those skills as an individual is
not exactly an easy task either. It seems the training and classes are still lagging behind the demand
for real-world Data Engineering knowledge.
This is the gap I’m attempting to fill with the topics in this book. I rewound myself to my first days
as a new data developer and thought about how hard it is to even know what topics to learn.

What To Expect
In this book, I want to give you the skills and knowledge, especially the underlying theory, to write
beautiful, fast, scalable data pipelines. It’s impossible to teach everything and cover every topic, but
I at least want you to know, what you should focus on. Hopefully, you discover many topics that
you can dive into at your leisure.
This book isn’t about how to write code.
Introduction 2

Data pipelines are so different and varied in their structure, based on technology stacks being used,
but most of the concepts are the same. Some people wrongly assume that they should learn how to
be a great coder, especially in the beginning, sure, that is helpful. But, as you grow in your career
you will quickly realize that it’s other skills that enable you to be a good Data Engineer.

• Knowledge and concepts first.

• Writing code second.

What I don’t want to teach you is how to write code. You will see me using Python in my examples,
and that is just for the ease of code readability. I expect you are a smart and savvy person, you
reading this book after all.
The theory and ideas behind many data engineering topics are more important than how well you
write code, which comes with time and experience.

Chapters
Here are the chapters and topics you can expect to encounter.

• The Theory Of Data Engineering and Pipelines

• Data Pipeline Basics
• Pipeline Architecture
• Storage - Files
• Compute and Resources
• SQL and Databases
• Data Warehousing and Data Lakes
• Data Modeling
• Data Quality
• DevOps

The Focus of This Book

This book is going to focus on theory, rather than diving into every detail of writing code for Data
Engineering. Programming skills are built over time, if you’re looking to sharpen your skills in some
language, then, by all means, take some courses or classes.
Many times the harder skills to develop are those that are less obvious, the skills that come with
experience.
It’s hard to know what you don’t know when you are starting out. I will insert code snippets and
examples where I feel it would make a point or concept clear.

• The book is about the underlying concepts and theories.

Introduction 3

• Try to learn lessons before you learn them the hard way.
• Data Engineering is a journey, to fail is to succeed.

I’m going to give you the headstart you need to help you surpass all your contemporaries and learn
the skills that are central to becoming a successful data engineer. The best part is, you can do all
this with Python, in which most of our examples will be written, but the choice of language doesn’t
matter as much as the skill sets and thought processes.
I’ve personally built a successful career as a Senior Data Engineer, never have taken a Computer
Science class in my life, and used Python for 90%+ of my professional life.

Knowledge and Experience

Building data pipelines require a unique set of knowledge that crosses many disciplines and isn’t
easy to come by without specific experience. What makes it even harder is that many of the skills,
like data modeling, for example, are somewhat esoteric, half art and half science. The good news is
that I can help speed you down the path to success by giving you the 20,000-foot view of the topics
and problems you will encounter in the real world.

• Data Engineering covers a wide variety of topics and technology.

• Data Engineering is both art and science.

I want to share those experiences, tips, and tricks in this book to jump-start you into building reliable,
scalable data pipelines.

What are the topics we will cover?

Theory and Basics

First, we will discuss the theory of data pipelines, I encourage you not to skip this section. It’s
important to understand where you going before you go on a trip.
Next, we will dive into the basic components of every data pipeline regardless of the complexity, I
call these the fundamental skills and thought processes.

Architecture and Storage/Files

These two topics will be quickly followed by architecture, the high-level choices we make in the
beginning will affect every step we take from that point forward.
Of course, we will cover the basics of storage options (files). File types play a big part in data
engineering, that should be no surprise.
Introduction 4

Compute and Resources

In the age of the Cloud, we must covert compute (ram, cpu), and how to think and work with these
resources. Big Data processing requires utilizing all available compute resources available.
Cost eventually becomes a question in our data pipelines, being able to calculate resource usage is
a very useful skill.

SQL and Databases

No data engineering book would be complete without a quick overview of SQL and relational
databases. Although their importance is waning today you will still find them used for meta-data
and highly transactional storage systems.
A data engineer who doesn’t know their way around SQL queries and tuning is going to run into
serious problems. Popular tools like Spark have made SQL even more popular with SparkSQL, so
fundamental knowledge about querying datasets will serve you for a long time to come.

Data Warehouses / Data Lakes

Closely related to SQL and relational databases is the topic of Data Warehousing and Data Lakes.
Although the storage layer can range from SQL Server to Parquet files, much of the methodology
remains the same.
Being able to provide usable analytics is at the crux of solving and providing most businesses with
value.

Data Modeling
Another topic near and dear to my heart is Data Modeling. It’s half art and half science, easily one
of the most important topics in the book.
What good is a data pipeline if the model fails to provide the needed value?

Data Quality
Probably a less popular topic, but one of great importance to the longevity and usability of data
output by engineers is Data Quality. It’s still a fairly new topic even in the data engineering world,
with not many good tools to pick from, so I will do my best to give a good overview.

DevOps
Things just wouldn’t be complete without taking a look at DevOps-CI/CD and the role it plays in
data pipelines. It’s an often overlooked and ignored part of data engineering that has a cult-like
following in the great software engineering world.
Introduction 5

Summary
My goal is to prove to you that anyone who learns the topics in this book can easily build robust
data pipelines like a seasoned data engineer. I will show you the tips and tricks you can use for every
data pipeline project that will have everyone coming for your help as the expert. Let’s dive in!
Chapter 1 - The Theory.
Why start a book about data engineering and data pipelines with a chapter on theory? I encourage
you not to skip ahead to the next chapters without reading this. Building a solid understanding of
why we need data pipelines and what they should accomplish will end up driving every decision
made along the way.
Every adventure requires a plan and some foresight and your journey into the world of data
engineering will be no different.

• Building a solid foundation reduces downstream headaches.

• The foundational principles directly impact code and technology decisions.

Theory can be dry and I will do my best to be to the point. Having a high-level understanding of
what we are trying to accomplish with our data pipelines should make writing them a little bit more
manageable.

What Is a Data Pipeline?

I believe it’s important to stop and ask simple questions sometimes, it’s easy to miss the forest
through the trees. What is a data pipeline? Simply put, moving data from point A to point C, with
a transformation B in the middle is a data pipeline at its core.

You might be tempted to think that it’s simply the movement of data, but it’s more complicated than
that. I would rather say that a good data pipeline is about …
Chapter 1 - The Theory. 7

“facilitating the movement, storage, and access to data in a repeatable, resilient, and scalable
manner.”
These are the fundamentals you should think about when it comes to data engineering and data
pipelines.

• Movement
• Storage
• Access
• Repeatable
• Resilient
• Scalable

A little more than you thought? Maybe you think I’m nitpicking and throwing out jargon? We will
dig into these 6 fundamentals shortly.
I would argue the difference in understanding a data pipeline as “just the movement data” vs
“facilitating the movement, storage, and access to data in a repeatable, resilient, and scalable manner”
is the difference between an amateur, broken, and unworthy pipeline and one built by a professional
that will be in place for years.

Data Pipelines built with Passion and Creativity

Being passionate about what you working on vs doing something for a paycheck is large, we all
know this. When you were in middle school and your mother told you to clean your room or not go
out with your friends, I’m sure you “cleaned” it. But did you really clean it? Your heart wasn’t in it.

• Data engineering and pipelines are complex.

• The business relies heavily on functioning pipelines.
• Heart and passion for data goes a long ways.
• Creativity produces better results than rigidity.

All this can lead to data engineering jobs being very stressful and all-time consuming. If you have
a passion for working with data, this will help offset the burdens that can come with the job.
Passion for data engineering should lead to excellence in the code we write and help us design more
creative long-term solutions.
Chapter 1 - The Theory. 8

Creativity is the spice in life, both in coding and leisure. Data engineering is a hard profession to
master, not thinking outside the box and being rigid will curtail your career and results. Complex
problems require creative solutions, some of the best data engineers come from non-software
backgrounds.
Truly good data pipelines are built with heart and love. You have to see it as more than just another
task that is going to make your boss happy. If we see the movement of data in a system as a simple,
meaningless, and overhead task, the obvious result is going to be lackluster and the codebase will
fail at a critical moment.

• Passion for data leads to better data engineering solutions.

• It’s hard work, and requires dedication inside and outside work to learning.
• Once trust in data is lost, it’s impossible to replace.

Worse, it will compromise trust from business units and lead to incorrect results. Let’s examine our
theory of what a data pipeline is and how it will affect the code we will write later.
 Chapter 1 - The Theory. 9

Data Engineering Standards

• Data Engineering is not “beneath” Software Engineering.
• Pipelines conform to the same level or higher expectations as normal SE code.

There can be an attitude in tech that Data Engineering is somehow beneath Software Engineering
and doesn’t ascribe to the same standards. This is a crucial mistake. Data engineers should hold
themselves and the complex pipelines they design to the same high level of standards.
Let’s dive into more of what this means and looks like day-to-day.

Movement
The movement of data is basic to the data pipeline. We are by definition picking up data from
somewhere and dropping it off at “home”, with some number of stops in between. Therefore we
can’t brush past the topic of data movement. In Data Engineering some movements will be complex
and some will be simple.
For example, reading a CSV and doing something every row is about as basic as it gets. A very small
but simple data pipeline.

1 import csv
2
3 def open_csv_file(file_location: str) -> object:
4 with open(file_location) as f:
5 csv_reader = csv.reader(f)
6 for row in csv_reader:
7 print(row)

Usually, the life of a Data Engineer isn’t so easy though.

There are a few obvious topics that come to mind when thinking about moving data from point A
to B. Some of the most common, you are probably are familiar with or have heard of …

• Streaming data (real-time).

• Batch processing.
• Micro batch processing (near-real-time).
• ETL / ELT

When data engineers think about the movement of data, one of the first thoughts should be … “Is
this data pipeline streaming, batch, or something in between?”
Chapter 1 - The Theory. 10

This type of basic approach to data movement is critical because all downstream actions are most
likely very different based upon the answer of streaming vs batch.
Don’t be fooled, the basic choice at the foundation of building data pipelines will have a huge impact
on how you design the system. Your code will take shape around the technology choice you make
here. It’s something you have to think about, “How will my data move point A to B?”

• Technology choices affect code and architecture.

If you choose a messaging service like Pub/Sub, Kafka, Pulsar, etc, that code is going to look
completely different and have a different setup requirement than if you decide to push CSV files
to Parquet files with Spark. Those options couldn’t be more different.

Complexity in Data Pipelines

When considering what the movement of the data should look like, remember complexity is a killer.
Chapter 1 - The Theory. 11

Do you already have numerous custom complex transformations that need to take place? Will adding
a complex data movement or storage option make the codebase nearly unapproachable? What are
the speed requirements of the project? Is some nominal data loss acceptable?

• Data Engineers should reduce complexity.

• Don’t over-engineer solutions.

If your dealing with financial data, sure a messaging system where it becomes hard to “lose” data
might be a good choice. If you’re dealing with terabytes of satellite imagery, missing one image
won’t have a reasonable effect on the outcome.
Sounds like a lot of jargon just for deciding the data movement stack you will choose for your
pipelines?
The point I’m trying to make is that your code is probably a pyramid of complexity. The foundational
decisions you make upfront will direct the type and scope of the code that is needed to implement
the rest of the project. Don’t take it lightly.
Again, the movement of data by code is fundamentally what Data Engineering is all about. We know
we have to move and act upon the data, it’s the details of how to implement that data movement is
where the Data Engineer provides value.

Storage and File Types

Don’t forget our definition of data pipelines. “Facilitating the movement, storage, and access to data
in a repeatable, resilient, and scalable manner.”
Very closely related to data movement that we just covered is storage, or data at rest. Obviously,
with the rise of cloud storage options like s3, gs, azure blob, it’s been much easier to ignore choices
of file and data storage. But, just because it’s easy doesn’t mean that data pipelines won’t depend in
many ways on the type of storage chosen.

• Storage affects both time and money.

• Data rest (file) storage affects technology and code access patterns.

The difference between a CSV file and parquet is enormous (we will explore details later), especially
as data grows.
Again, many of the file storage choices can be hard to back out of once baked into the software
solution. Understanding the data that is flowing and how it is used will always give clues to the
storage option needed.
Chapter 1 - The Theory. 12

Storage and Files affect Technology and Code

I’ve seen tens of millions of individual JSON files stored in cloud buckets, and then a year later
analytics need to be done. Of course, at this point, even a massive AWS EMR cluster was having
trouble reading tens of millions of individual JSON files to retrieve needed business analytics.
Obviously, for that project, someone chose JSON because it was easy, and the consequences were
large and required a sizable project to regain insights into the data that could have been avoided by
proper storage upfront.
Storage can be seen as just an in-consequential decision when building data pipelines, but this is
rarely the case once the project is popular and starts to scale.

Access
Continuing on with our definition of data pipeline, we come to data access.
What does access have to do with data pipelines? Everything. The data output from a pipeline or
intermediate steps should be easily and quickly accessible. Why is this so important?
Chapter 1 - The Theory. 13

Well, what happens when the inevitable bug or business requirements change and you need to
troubleshoot? What happens when the pipeline has scaled to hundreds or even terabytes of data?
You will care about data access then.
Simply put, how do people or machines consume the data in question? Data is no good if it cannot
be used.

I’ve seen complex projects completed by some very smart people, with code that looks like it should
be admired. And those codebases solved very hard problems. But they failed in the end with no
adoption even within the same engineering group.
Why? The data access layer was impossible to understand and interact with. It made the whole
project useless. Don’t let your project or pipeline fall into that hole.

• Data is of no use without reasonable accessibility and usability.

When you design a data pipeline never forgot about how the data will be accessed and explored. This
comes down to how and where the data is stored at rest, and then presented for use by consuming
users or applications.
Moral of the story? Choosing storage and file types is one of the most important topics and with
far-reaching consequences when designing data pipelines.
Chapter 1 - The Theory. 14

Repeatable
Repeatability is arguably one of the most important pieces of any data pipeline. Every pipeline
codebase must meet these criteria, it has to be repeatable. What good is the code if the author is the
only one who understands, can troubleshoot, or even worse, run the pipeline? This is more common
than you think.

• Anyone must be able to run a data pipeline.

• Anyone must be able to troubleshoot a pipeline.
• Pipelines break, they will have to be re-run.
• Pipelines should produce the same results (idempotent).

Some pipeline breaks, everyone is running around like a chicken with their head cut off because that
“one person” is on vacation today, and no one else knows what to do.

People and Assumptions Affect Pipelines

Engineers will many times design into their code assumptions about the world that data lives in.
It makes the code unreadable, and anyone should be able to clone the code repository, look at the
README, and with a few lines at the console, kick off the pipeline.
Anything more inherently means the pipeline is not repeatable. It shouldn’t take more than one
button click or command line argument issued to re-start or run any data pipeline.
There should not be a myriad of configurations, flags, and files that must be staged in certain
ambiguous places for the data pipeline to run.
A good run of thumb is this… if you can’t schedule a CRON job to run a single Python file with a
main() function and a few arguments, which are noted in the README, then the code is probably
not repeatable.
If no one but the person who wrote the code can run it… that is a problem. This is another important
subject and we will explore it more later.

Resilient
This might seem obvious to some, but to others, it isn’t something they’ve had experience with.
What do I mean when I say a data pipeline must be resilient? I mean the codebase must be written
in such a way that it isn’t fragile.
There should not be many instances of “hardcoded” values, like dates or numbers in the code. This
inevitably means that an engineer will have to make a code change just to run a pipeline for some
historic data. Is there no try: except: blocks? This probably means the author didn’t think about the
pipeline in a resilient way.
Chapter 1 - The Theory. 15

• When data pipelines break every day, it signifies fundamental problems.

• Resilient pipelines require attention to code and architecture.

Becoming ready for the unknown is important in life and code!

A good data pipeline should be specific enough to get the job done efficiently, but not so specific
that minor changes in requirements don’t lead to codebase re-writes and refactors at every turn.
Having a resilient pipeline means thinking ahead, without over-engineering a solution. This can end
up being half art and half science but it’s an easy trap to fall into.
Thinking outside the box and not making assumptions like “I will assume this file always has records
in it and is never empty,” always make a pipeline less resilient.

Scalable
Last but not least, and probably the most important, scalability is the name of the game. A data
pipeline that isn’t scalable isn’t a pipeline at all,
it’s just a one-time script that is doomed from the beginning.
One of the most common mistakes and time wastes I’ve seen is someone taking a few weeks to
write a codebase to solve a problem, testing and testing, releasing the code. Only coming to realize
in production when scaled up it either crashes or is so slow as to be useless.

• Data pipelines should be built to handle more data … and more.

• Scalability issues leading to poor performance are common problems.

You can rarely write a pipeline to deal with one piece of code and have it scale up without significant
changes. Scalability has to be one of the main tenants driving the creation of the pipeline from the
beginning.
When your thinking about a piece of data working its way through the pipeline, ask yourself, what if
this record has 10 million friends? Scalability affects file storage options, the processing framework,
and everything in between.

In Summary
“Facilitating the movement, storage, and access to data in a repeatable, resilient, and scalable
manner.”
It’s easy to get caught up in the mundane and day-to-day data engineering work of writing the next
ETL script. But, I suggest you take a different approach, looking at problems and resolving to make
good decisions, not to rush design, and to think through problems.
Chapter 1 - The Theory. 16

• Movement
• Storage
• Access
• Repeatable
• Resilient
• Scalable

You should always consider the above 6 points when approaching a new problem or pipeline.
Grasping the importance of these theories and them putting them into practice will put you ahead
of most engineers.
Reminding yourself daily that your pipelines need to move data efficiently, and what file types
are chosen do matter is will save you headaches down the road. Ensuring every pipeline is easily
repeatable and resilient from the start will come in handy when something breaks at night or on the
weekend.
Of course, when the business starts to scale and your data needs to keep up, you won’t be left holding
the bag! It’s very tempting to get caught up in this and that new technology, chasing the shiny new
toy. It’s easy to say “been there, done that,” when it comes to the next pipeline.
I encourage you to remember these basic ideas we discussed and use them to drive all the decisions
you make when tackling the next problem.
Chapter 2 - Data Pipeline Basics
Finally, it’s time to dive into the inside workings of data pipelines, what we’ve all been waiting for!
Let’s start by talking about the basics of building data pipelines. There are a few tips and tricks that
will help you generalize all the pipeline projects you work on.
I consider these things basic 101 level requirements. Chances are if you don’t follow these approaches
or some variation of them, you’re going to struggle to produce reliable and repeatable pipelines that
you can be proud of.

The Best Pipelines Start with a Solid Foundation

I have no intention that this book teaches you how to write code, I will leave that for others. What
I do want to teach you is how to best write data pipelines possible using best practices, I will use
Python in my examples, but these practices can be applied to any language.
It’s more about the exercise and thought process than it is about what your language of choice
is. Don’t let people tell you that X or Y language is better because it’s “faster” or some other
reason. The more you write code and work on software projects you will realize it’s more about
the implementation of the code, than the language chosen.
I want to review these data engineering coding best practices that you will run into out in the real
world. Practices that will make your life, and the data engineers’ lives that come after you more
bearable.

• Project Structure.
• Testing.
• Documentation.
• Containerization.
• Architecture First.

These are just some basics that will assist us going forward, as they are the core of what will drive
the design and layout of all your data pipeline projects. I know you might be getting tired of the
theory, but this is where the theory starts to meet the reality of writing code to move data.
This isn’t a book about how to write code, so write code as you like, but adopting a few well-accepted
styles can have a positive impact on your success at delivering highly scalable and efficient pipelines.
Chapter 2 - Data Pipeline Basics 18

Project Structure
When starting our data pipelines in Python, or any language, we should follow a clear and concise
design pattern that shows any reader the flow of the data through the code. Just because something
is complex doesn’t mean it’s acceptable to have our program and data flow is hidden into obscurity.
Be short and to the point when writing code, the data should flow through the code in an obvious
and human-readable way. That’s why we are using Python!
We will dive into DevOps later but anytime you start a new codebase/pipeline, or work on an old
one, you should ensure a few basics. Always start by creating the following resources.

• README.md - a few well-written sentences can save someone hours.

• Dockerfile - they are easy to use and level the playing field.
• Tests folder - as a gentle reminder that tests are important.
• Requirements.txt - project dependencies should be captured somewhere.
• Folder structure to divide the codebase logically.

Let’s see what a very basic project structure might look like.
 Chapter 2 - Data Pipeline Basics 19

It’s a contrived and simple example in the above picture, but it’s obvious the main.py is where we
should look for the code, test probably has our tests, there is a Dockerfile. More than anything we
are looking for a project to be clear.
Here is an example of what you don’t want to see when looking at a new pipeline codebase.

1 run_database.py
2 general_functions.py
3 data_file.csv
4 read_methods.py
5 configurations.json

You don’t know where to start, what to look at, what is important or what isn’t. It’s just messy and
not obvious, be clean and organized in your code as well the project structure itself.
What would a better project structure look like applied above?

1 README.md
2 Dockerfile
3 requirements.txt
4 utilties/general_functions.py
5 configs/configurations.json
6 sample_data/data_file.csv
7 database/run_database.py
8 read_methods.py

Not perfect but I’m sure you get the idea.

Chapter 2 - Data Pipeline Basics 20

Build On a Solid Foundation

You might not think this has that much to do with data engineering pipelines, but it does. A solid
foundation is a key to the future success of any project. Taking the guessing game out of your
pipelines is one of the best things you can do.
There is nothing worse than going to a repository in GitHub and finding random files and folders
littered everywhere with no apparent consistency. Remember what your mom always said … “clean
your room!”

• You can’t have good pipelines without a solid base.

• You have to do the small parts well, to make the whole better.

Before writing the so-called guts of our codebase, large or small, it’s important to get a few things
right. I do understand that there are many different styles of writing code, but there is no excuse for
sloppy and dirty code and organization.
Pipeline code that is thrown together haphazardly is probably code that was also written poorly, not
tested, and will break often. Taking ownership and pride in a codebase is important.
Let’s dive a little deeper into structuring the actual codebase of a pipeline.

Data Pipeline Code Structure

If you’ve never written a data pipeline before, or are just starting, where do you even start? There
are so many steps to writing a complete data pipeline end-to-end, how do you accomplish the task?
One small step at a time, starting with the basics.
It all starts with main()
Every pipeline should start with a main.py file and a main() function.
This might sound picky but it’s essential to be clear from the beginning, and that starts with a
main.py file with a main() function that anyone can go to for a start. There is nothing worse than
opening a file with 20 functions and having no idea where to look.

• Always include a main() entrypoint for your data pipelines.

Or even worse than that, just a bunch of files all in the same directory, each with a different name,
and you have no idea where the entry point of the codebase is!
You should always start with something along these lines.
 Chapter 2 - Data Pipeline Basics 21

1 >> main.py
2 if __name__ == '__main__':
3 main()

I mean call it entry_point() if you have to. Just give the obvious indication of where in the world
to start.

Good Coding Practices make Good Data Pipelines

I can’t stress this enough and I want to make a point here. This is a book about coding practices,
but you can’t be a successful Data Engineer without embracing a mindset of clean and concise code
structures that are easy to read and use.
Your data pipeline is only going to be as good as the code you write. If it’s hard to read and follow,
then it will leave a bad taste and impression, regardless of how good a job you think you did.

The Code Should Flow with the Data

Simple, but important. We should continue this pattern of obvious code and data flow. Your pipeline
code should follow the data flow in a sense. The methods and functions should be named to indicate
what they are doing and should be called inside your main() method in an obvious pattern.
As the data flows, the code flows, and vice versa.
 Chapter 2 - Data Pipeline Basics 22

Simply reading a main() function should tell you what’s happening to the data and where it is
coming from and going too.

Example
Let’s say we have a very simple pipeline to read CSV files, get some metrics from the files, and store
those results into a SQL database.

• Read CSV files.

• Get metrics from files.
• Store results in SQL.

Don’t Focus on the Details Right Away

It can be tempting to start thinking about the details or writing code right away. We should shy away
from this practice. Let’s sketch our code the way we would sketch a drawing. It helps to understand
the work that lies ahead of us and will help keep us on the straight and narrow as we develop the
pipeline.
It doesn’t matter very much if you write Object Oriented Programming (OOP) or Functional
programming with lots of little methods. If you are writing ETL or data pipelines, your code should
show the flow of the data when read.
In the above example, we talked about a simple program to open CSV files, get metrics, and put
those metrics into the database. The code should be framed in such a way this is obvious with a
cursory glance.

1 >> main.py
2 def main():
3 calculate_workload()
4 download_csv_files_in_parallel()
5 stream_files_to_memory()
6 calculate_metrics()
7 save_metrics_to_database()

Why write code like this? Because it’s obvious how the data flows through the program. It also
gives a high-level understanding of what the pipeline is going to look like before every bit of code
is written.

• Data flows through a data pipeline.

• Human readable data pipelines increase productivity and reduce bugs.
• Don’t work on small details, do the big picture first.
 Chapter 2 - Data Pipeline Basics 23

This is a great advantage because it will help us think about how scalable, efficient, and extensible
this pipeline will be. I can’t emphasize this enough. If you just started by working on calculate_-
workload() and not thinking about what comes after or before, you’re on the road to a life of
hardship.
You get caught up in “how am I going to read this data”, and writing the code straight away, your
missing details about what is coming after that could change how you write the code you working
on!

Code Readability and Organization

Again, every main() function should show the flow of the data through the pipeline in an obvious
fashion. Also, if the different steps you will be designing throughout the pipeline require 5 or more
functions, you should consider breaking those out into a separate Python Class or file if you writing
only functions.
This makes your code more reusable, maintainable, and easier to debug. It gives context and doesn’t
overwhelm someone else, or you when debugging or feature addition happens.
So perhaps in the above example, streaming files into memory could be a few methods/functions.

1 >> stream_csv_files.py
2 import csv
3
4 def open_csv_file(file_location: str) -> object:
5 with open(file_location) as f:
6 csv_reader = csv.reader(f)
7 for row in csv_reader:
8 yield row
9
10 def stream_csv_rows(rows: iter, transform_etl: object) -> iter:
11 for row in rows:
12 transformed_row = transform_etl(row)
13 yield transformed_row

This is a simple but powerful idea. Don’t have the bad habit of writing data pipelines all in a single
file. It isn’t helpful and will most certainly leave yourself and others confused as the code grows,
group your code in logical units.

• Break up functions into the smallest units possible.

• Write data transformations that are re-usable.
• Avoid long functions and methods.
 Chapter 2 - Data Pipeline Basics 24

You should not write functions that are 50 lines long. This means your not breaking up your code
into small enough logical units.
Functions and methods that are working on data and contain too many lines of business or
transformation logic are going to be prone to error and impossible to maintain or troubleshoot.

Tests.
We will talk more about testing in the DevOps chapter, but this topic is so important it’s worth a
cursory overview now.
You need to have tests for every data pipeline. Why? Because if want to be cut above the rest you
will follow best practices every step of the way, even when you don’t have to.

• Tests are basic to data pipelines.

• Tests protect pipelines from bugs and errors.
• Tests increase development productivity.

Think about it, if you change code how are you supposed to know it works and that you have not
broken something you don’t know about, especially if you are new to a codebase?

Unit Testing
Unit tests are the first line of defense. No one should have to run a pipeline to be able to know if it’s
working or not. Changing the business or transformation logic acting upon data should be tested
in such a way that anyone can attempt the change and run tests with a reasonable assumption that
unit tests will catch any problems.
Unit tests will not catch every bug, they never will, but they will catch some, and they will protect
you from those “silly” mistakes we are all prone to making.
In our above example, we would have at least …

1 tests()
2 test_download_csv_files_in_parallel()
3 test_stream_files_to_memory()
4 test_calculate_metrics()
5 test_save_metrics_to_database()

Every language has its own popular testing frameworks, for Python that would be pytest. Any
Google search or YouTube video can introduce you to the basics of testing for any language and
have you writing your own in an hour or less.
Discovering Diverse Content Through
Random Scribd Documents
 ‘I heard the Countess give a throttled cry, as she struggled like
one caught in a fire; but the Queen kissed her again before she
could free herself. When at last she had flung away, with crying and
a blenched face—she who had been so hard before was now in a
state of wild alarm, warning off our lady with her fighting hands.
“No, no, no! Touch me not—defile not yourself. Oh, never that—I
dare not suffer you!”
‘“What, am I so vile?” says the poor Queen, misunderstanding her
in this new mood. The Countess burst out into passionate weeping,
which hurt her so much (for she was no tearful woman by nature)
that she writhed under the affliction as if the grief within was tearing
at her vitals. She shrieked, “Ah, no! Not you—not you—but I. Oh,
you torture me, brand me with fire!” I could not guess what she
meant, save that she was beaten, and her wicked passion with her.
‘She sat up and stared at our Mistress, her face all writhen with
grief. “Listen, listen—this is the truth as God knows it. That man who
stands between us two and Heaven is your ruin and mine. For I love
him not at all, and have consented to him now, degrading myself for
hatred’s sake. And for you, who have loved him so well, he has no
care at all—but only for your crown and royal seat; for he loves me
only—and so it has always been.”
‘The Queen could only nod her head. Mary Sempill said sternly:
“Woman, you do well to lash yourself at last; for none can hurt you
beside yourself. Now, may God forgive you, for I never will.”
‘“Oh, Mary,” says the Queen, “what have you or I to do with
forgiveness of sins? Alas, we need it for ourselves. And she is in as
bad a case as I am.” Then, “Come to me now, Jeannie,” she said;
and most humbly that wicked, beaten woman crept up to her late
enemy. The Queen embraced and comforted her. “Farewell, Jeannie,”
said she, “and think as well of me as you can. For I go on to I know
not what—only I do think it will be unhappiness—and we shall never
meet again.” With sublime calm she turned to us, weeping behind
her. “Come, my children, let us go our ways.”
 ‘This is the most terrible secret sorrow which broke her heart, and
ends my plea for pity upon her who loved so fondly. My breath and
strength are done; for I had them from her alone, and with her high
heart’s death dies my book.’
Honest, ingenuous, loyal Des-Essars! seeing, maybe, but in a glass
darkly; seeing, certainly, not more than half—thou wert right there.
If thy mistress beat the woman at last, it was with her fading breath.
She knew herself beaten to the dust by the man.

[9] Here I am bound to agree with Bothwell; for if Huntly

wished to keep him from blood-guiltiness and knew that he could,
why not have kept him and his kegs away altogether? One
answer may be, of course, that Morton and his friends would
never have stood in had Bothwell and his been ruled out.
[10] Des-Essars, plainly, was at work during the Queen’s
captivity in England; and, as I judge, while the inquiry was being
held in Westminster Hall in 1568.

CHAPTER IX
THE BRIDE’S TRAGEDY

The heart being an organ of which we have opinions more gallant

than practical, Des-Essars should perhaps have judged wiselier that
his Secret of Secrets was what broke the Queen’s spirit. There he
had been right, for from this day onwards to the end of her throned
life the tragedy is pure pity: she drifts, she suffers, but she scarcely
acts—unless the struggles of birds in nets can be called acts. After
her spirit went rapidly her animal courage; after that her womanly
habit. She was like to become a mere tortured beast. And as I have
no taste for vivisection, nor can credit you with any, I shall be as
short as I can.
 Silent all the long way home from wooded Crichton to the sea, it
might seem as if she had been hardening herself by silent
meditation for what she knew must take place. She saw nothing of
Bothwell that night—she was not yet ready for him; but she did what
had to be done with Mary Sempill.
When that loyal soul came late into the bedchamber to bid her
good-night, she found her mistress in bed, calm and clear in mind.
Forewarned in some measure, as she stooped over to kiss her, the
Queen did not as usual put out her arms to draw her friend nearer,
but lay waiting for the kiss, which hovered, as it were, above her;
and before it could come she said, ‘Do you kiss me, Mary? Wait
while I tell you something. I am to be married to my lord come the
day after to-morrow.’
Sempill, prepared or not, started back, on fire. ‘You’ll never do it.
You’ll never dare to do it.’
‘I shall dare to do it, if I dare avouch it.’
Sempill was trembling. ‘I cannot endure it, cannot face it—most
wicked! Oh, my dear love and my friend, you that have been all the
world to me in times bygone, never go so far from me that I cannot
follow you!’
The Queen bit her lip, and wrinkled her eyes where the tears were
brimming, drowning her sight. ‘I must, I must—I cannot go back.
Oh, have mercy upon me! Oh, Mary——’
Sempill hid her face. ‘I cannot see it done. I cannot know of it. I
am—I do my best to be—an honest woman. These things be far
from me—unholy things. As Christ is my Saviour, I believe He will
pardon you and me all our sins of the hot blood. But not of the cold
blood—not of the dry!’ She changed suddenly, as if struck chill. ‘Why,
you will be an harlot!’ she said.
The Queen turned over in her bed and faced the wall.
Sempill went down on her knees. ‘I conjure you—I beseech you!
Madam, I implore you! By your mother’s bliss and your father’s
crown imperial, by the great calling of your birth! By Christ’s dear
blood shed for you and all, by the sorrows of Our Lady—the swords
in her heart—the tears that she shed; by her swooning at the Cross
—I implore, I implore!—make not all these woes to be in vain. By
your young child I conjure you—by my own upon earth and the
other in my womb—by all calm and innocent things—oh, put it from
you: suffer all things—even death, even death!’
There was no response. She rose and stood over the bed. ‘We
have loved much, and had sweet commerce, you and I. Many have
had sweetness of you and left you: Beaton is gone, Fleming is
alienate. You drive me to go their way, you drive me from you. For if
you do this, go I must. Honour is above all—and yon man, by my
soul, is as foul as hell. Turn to me, my Mary, look at me once, and I
shall never leave you till I die.’
She did not stir nor utter a sound; she lay like a log. Mary Sempill,
with a sob that shook her to pieces, and a gesture of drowning
hands, went out of the room, and at midnight left the palace. Those
two, who had been lovers once and friends always, never met again
in this world.
What the Queen’s motives may have been I know not, whether of
desperate conviction that retreat was not possible, or of desperate
effort to entice the man to her even at this last hour: let them go.
[11] She held to her resolve next day; she faced the remnant of her
friends, all she had left; lastly, she faced the strong man himself, and
like a doll in his arms suffered his lying kisses upon her lips. And she
never reproached him, being paralysed by the knowledge of what he
would have done if she had. To see him throw up the head, expose
the hairy throat, to see him laugh! She could not bear that.
On this day, the eve of her wedding, she found out that her
courage had ebbed. Things frightened her now which before she
would have scoffed at. A May marriage—hers was to be that: and
they who feared ill-luck from such gave her fears. A Highland
woman became possessed in the street, and prophesied to a crowd
of people. She said that the Queen would be a famous wife, for she
would have five husbands, and in the time of the fifth would be
burned. ‘Name them, mother—name them!’ they cried; and the mad
creature peered about with her sly eyes. ‘I dinna see him here, but
the third is in this town, and the fourth likewise!’ ‘The fourth! Who is
he?’ ‘He’s a Hamilton, I ken that fine, and dwells by Arbroath. I
doubt his name will be Jock.’
Lord John! The Lord of Arbroath—why, yes, she had given him a
great horse. They rehearse this tale at dinner, and see Bothwell grow
red, and hear the Queen talk to herself: ‘Will they burn me? Yes,
yes, that is the punishment of light women. Poor souls, they burn for
ever!’
She carried the thought about with her all day, and at dusk was
much agitated when they lit the candles. About supper-time Father
Roche, asking to speak with her, was admitted. He told her that his
conscience would not permit him to be any longer in her service.
Bothwell had refused to be married with the mass: in Father Roche’s
eyes this would be no marriage at all. She was angry for a second in
her old royal way—her Tudor way; moved towards him swiftly as if
she would have quelled him with a forked word; but stopped mid-
road and let her hands unclench themselves. ‘Yes, yes, go your ways
—you will find a well-trodden road. Why should you stop? I need
you no more.’ He would have kissed her hands, but she put them
behind her and stood still till he had gone. Then to bed, without
prayers.
At ten o’clock of the morning she was married to him without
state, without religion. There was no banquet: the city acted as if
unaware of anything done; and after dinner she rode away with him
to Borthwick. Melvill, Des-Essars, Lethington went with her, Mary
Seton and Carwood. Bothwell had his own friends, the Ormistons
and others of mean degree.
With tears they put her to bed; but she had none. ‘I would that I
might die within the next hour,’ she said to Des-Essars; and he,
grown older and drier suddenly—‘By my soul, ma’am, it should be
within less time, to do you service.’
 She shook her head. ‘No, you are wrong. He needs me not. You
will see.’ She sent him away to his misery, and remained alone in
hers.
It cannot be known when the Earl went up. He stayed on in the
parlour below, drinking with his friends so long as they remained
above-board, talking loudly, boasting of what he had done and of
what he should do yet. He took her back to Edinburgh within a few
days, moved thereto by the urgency of public affairs.
Those who had not seen her go, but now saw her return, did not
like her looks—so leaden-coloured, so listless and dejected, so thin
she seemed. The French Ambassador—Du Croc, an old friend and a
sage—waiting for audience, heard a quarrel in her cabinet, heard
Bothwell mock and gibe, depart with little ceremony; and then the
Queen in hysterics, calling for friends who had gone—for
Livingstone, for Fleming.
Carwood came in. ‘O madam, what do you lack?’
‘My courage, my courage.’
Carwood, with a scream—‘God’s sake, ma’am, put down that
knife!’
‘The knife is well enough,’ says she, ‘but the hand is numb. Feel
me, Carwood: I am dead in the hand.’
Du Croc heard Carwood grunt as she tussled. ‘Leave it—leave it—
give it me! But you shall. You are Queen, but my God to me. Leave
it, I say——’ The Queen began to whimper and coax for the knife—
called it her lover. Carwood flung open the window and threw it on
to the grass.

No doubt the worst was to be feared, no doubt Bothwell had

reason to be nervous. At the council-board, to which he ordered her
to come, he told her what was before her. The lords were in league,
clustered about the Prince: he was not ashamed to tell her in the
hearing of all that she was useless without the child. Dejected,
almost abject as she was become, she quailed—shrinking back, with
wide eyes upon him—at this monstrous insult, as if she herself had
been a child struck to the soul by something more brutish than your
whips. Lord Herries rose in his place—‘By the living God, my lord, I
cannot hear such talk——’ Bothwell was driven to extenuate. ‘My
meaning, madam, is that your Majesty can have no force in your
arm, nor can your loyal friends have any force, without the Prince
your son be with you. You know very well how your late consort
desired to have him; and no man can say he was not wise. Believe
me, madam—and these lords will bear me out—he is every whit as
necessary to your Majesty and me.’
Huntly, on the Queen’s left, leaned behind her chair and spoke in a
fierce whisper: ‘You forget, I think, that you speak to the Queen,
and of the Queen. The Prince hath nothing but through her.’
‘By God, Geordie,’ he said, whispering back, but heard
everywhere, ‘and what have I but through her? I tell you fairly we
have lost the main unless we can put up that cockerel.’
The Queen tried to justify herself to her tyrant. ‘You know that I
have tried—you know that my brother worked against me——’
‘And he was wise. But now he is from home; we must try again.’
She let her head sink. ‘I am weary—I am weary. Whom have we
to send? Do you trust Lethington?’
This was not heard; but Lethington saw Bothwell’s eye gleam red
upon him.
‘Him? I would as soon go myself. If he wormed in there, do you
suppose we could ever draw him out again?’
‘No,’ she said aloud, ‘I am of your mind. Send we Melvill, then.’
He would not have Melvill: he chose Herries.
They sent out Lord Herries on a fruitless errand; fruitless in the
main sense, but fruitful in another, since he brought back a waverer.
This was the Earl of Argyll, head of a great name, but with no head
of his own worth speaking about. He might have been welcome but
for the news that came with him. All access to the Prince had been
refused to Herries the moment it was known on whose behalf he
asked it. The Countess of Mar mounted guard over the door, and
would not leave until the Queen’s emissary was out of the house.
There was more than statecraft here, as Herries had to confess:
witchcraft from the Queen was in question, from the mother upon
the child. The last time she had been to see him, they said, she had
given him an apple, which he played with and presently cast down.
A dog picked it up, ran under the table with it and began to mumble
it. The dog, foaming and snaping, jerked away its life. ‘Treason and
lies!’ roared Bothwell, who was present; ‘treason heaped on lies!
Why, when was your Majesty last at Stirling?’ He had forgotten,
though she had not.
‘It was the night before you took me at Almond Brig,’ she said;
and, when he chuckled, broke out with vehemence of pain, ‘You
laugh at it! You laugh still, O Christ! Will you laugh at my graveside,
Bothwell?’ She hid her head in her arm and wept miserably. It was
grievous to see her and not weep too. Yet these were no times in
which to weep.

On the same day in which Lord Lindsay departed, to join the Lords
at Stirling, Huntly also, most unhappily, asked leave to go to his
lands. The Queen used him bitterly. She could be gentle with any
other and move their pity: with him she must always be girding. ‘Do
you turn traitor like your father? Have you too kept a dagger for my
last hours?’ He did not break into reproaches, nor seek to justify
himself, as he might have done—for no one had tried to serve her at
more peril to himself. He said, ‘Madam, I have tried to repair my
faults committed against you,’ and turned away with a black look of
despair. He went north, as she thought, lost to her: it was Bothwell
who afterwards told her that he had gone to summon his kindred
against the war which he saw could not be far off. So scornful are
women to those who love them in vain—that should surely have
touched her, but did not. Lord John Hamilton took Huntly’s empty
place, too powerful an ally to be despised.
The Earl of Argyll came and went between Stirling and Edinburgh,
very diligent to accommodate the two cities, if that might be. He
dared—or was fool enough—to tell the Queen that all would be well
if she would give up the King’s murderers. She replied: ‘Go back to
Stirling, then, and take them. I do give them up. It is there you shall
find them.’ Whether he knew this to be truth or not, for certain he
did not report the message to the Earl of Morton. It would have
fared ill with him if he had.
Before he could come back, a baffled but honest intermediary,
Lethington had fled the Court and taken his wife with him. He went
out, as he said, to ride in the meadows; he did ride there, but did
not return. His wife slipt away separately, and joined her man at
Callander; thence, when Lord Livingstone sent them word that he
could not harbour the Queen’s enemies, they went on to Lord
Fleming’s, Mary’s father’s house, and finally to Stirling. It was a bad
sign that the gentle girl, flying like a thief at her husband’s bidding,
should write no word, nor send any message to the Queen; it was a
worse to the last few faithful that the Queen took no notice. All she
was heard to say was that Fleming could not be blamed for paying
her merchet.
Mercheta Mulicrum, Market of Women—the money-fee exacted by
the lord of the soil before a girl could be wed, clean, to the man who
chose her! Livingstone had paid it, Beaton had paid it; she, Queen
Mary, God knows! had paid it deep. She shook her head—and was
Fleming to escape? ‘No! but Love—that exorbitant lord—will have it
of all of us women. And now’s for you, Seton!’
She looked strangely at the glowing, golden-haired girl before her;
the green-eyed, the sharp-tongued Mary Seton, last of her co-
adventurers of six years agone. Fair Seton made no promises; but all
the world knows that she alone stayed by her lady to the long and
very end.
 Returned from Stirling, my Lord of Argyll, with perturbed face,
disorderly dress, and entire absence of manners, broke in upon the
Queen’s privacy, claiming secret words. The lords were prepared for
the field. They intended an attack upon the lower town by land and
water; they would surround Holyroodhouse, seize her person.
She flamed. ‘You mean my husband’s. It is him they seek.’
He did not affect to deny it. She sent for Bothwell and told him all.
Bothwell said: ‘You are right. They want me. Well, they shall not
have me so easily. You and I will away this night to Borthwick.
Arbroath will be half way to us by now, and the Gordons not far
behind. Let Adam go and hasten his brother. Madam, we should be
speedy.’
She took Seton with her—having no other left; she took Des-
Essars. Arthur Erskine was to captain Holyroodhouse. Bothwell had,
perhaps, half a dozen of his dependents. They went after dark, but
in safety.
There, at Borthwick, they stayed quietly through the 8th and 9th
of June: close weather, with thunder brewing.
No news of Huntly, none of the Hamiltons. Bothwell was out each
day for long spells, spying and judging. He opened communication
with Dunbar, got in touch with his own country. At home sat the
Queen with her two friends, very silent.
What was there to say? Who could nurse her broken heart save
this one man, who had no thought to do it, nor any heart of his
own, either, to spare for her? Spited had he been by Fortune,
without doubt. He had had the Crown and Mantle of Scotland in his
pair of hands; having schemed for six years to get them, he had had
them, and felt their goodly weight: and here he was now in hiding,
trusting for bare life to the help of men who had no reason to love
him. Where, then, were his friends? He had none, nor ever had but
one—this fair, frail woman, whom he had desired for her store, and
had emptied, and would now be rid of.
 If his was a sorry case, what was hers? Alas, the heart sickens to
think of it. With how high a head came she in, she and her cohort of
maids, to win wild Scotland! Where were they? They had received
their crowns, but she had besoiled and bedrabbled hers. They had
lovers, they had children, they had troops of friends; but she, who
had sought with panting mouth for very love, had had husbands
who made love stink, and a child denied her, and no friend in
Scotland but a girl and a poor boy. You say she had sought wrongly.
I say she had overmastering need to seek. Love she must; and if she
loved amiss it was that she loved too well. You say that she misused
her friends. I deny that a girl set up where she was could have any
friends at all. She was a well of sweet profit—the Honeypot; and
they swarmed about her for their meat like house-flies; and when
that was got, and she drained dry, they departed by the window in
clouds, to settle and fasten about the nearest provand they could
meet with: carrion or honeycomb, man’s flesh, dog’s flesh or maid’s
flesh, what was it to them? In those days of dreadful silent waiting
at Borthwick, less than a month after marriage, I tell you very plainly
that she was beggared of all she had in the world, and knew it. The
glutted flies had gone by the window, the gorged rats had
scampered by the doors. So she remained alone with the man she
had risked all to get, who was scheming to be rid of her. Her heart
was broken, her love was murdered, her spirit was gone: what more
could she suffer? One more thing—bodily terror, bodily fear.

[11] I am unwilling to intrude myself and my opinions, but feel

drawn to suggest that the latter was her motive. If she had
beaten the Countess at the eleventh hour, could she not beat the
Earl? Was she not Huntress to the utterance? Let God (Who made
her) pity her: I do believe it.

CHAPTER X
THE KNOCKING AT BORTHWICK
 The 10th of June had been a thunderous day, and was followed by
a stifling night. In the lower parlour where the Queen lay the
candles seemed to be clogged, the air charged with steam. Mary
Seton sat on the floor by the couch, Des-Essars, bathed in sweat,
leaned against the window-sill. In the hall beyond could be heard
Bothwell’s voice, grating querulously to young Crookstone and Paris
about his ruined chances. He was not laughing any more—was not
one, it was found, to bear misfortunes gaily. His tongue had
mastered him of late, and his hand too. He had nearly killed Paris
that morning with one smashing blow.
There came a puff of wind, with branches sweeping the window,
the pattering, swishing sound as of heavy rain. ‘Thank God for rain!
Baptist, the window, lest I suffocate. The rain will cool the air.’ He set
it wide open, and leaned out. There was no rain at all; but the sky
was a vaporous vault, through which, in every part, the veiled moon
diffused her light. He saw a man standing on the grass as plainly as
you see this paper, who presently, after considering him, went away
towards the woods. It might have been one of their own sentries, it
might have been any one: but why did it make his heart beat? He
stayed where he was, watching intently, considering with himself
whether he should tell the Queen, or by some ruse let my lord have
warning without her knowledge. Then, while he was hammering it
out, she got up and came to the window, and leaned over him, her
hand on his shoulder.
‘Poor prisoners, you and I, my Baptist.’
He turned to her with burning eyes. ‘Madam, there can be no
prison for me where you are; but my heart walks with yours through
all space.’
‘My heart,’ she said, ‘limps, and soon will be bedridden; and then
yours will stop. You are tied to me, and I to him. The world has gone
awry with us, my dear.’
Very nervous, on account of what he had seen, he had no answer
ready. Thought, feeling, passion, desire, were all boiling and stirring
together in his brain. The blood drummed at his ears, like a call to
arms.
Suddenly—it all came with a leap—there was hasty knocking at
the hall doors, and at the same instant a bench was overturned out
there, and Bothwell went trampling towards the sound. Des-Essars,
tensely moved, shut the windows and barred the shutters over
them. The Queen watched him—her hands held her bosom. ‘What is
it? Oh, what is it?’
‘Hush, for God’s sake! Let me listen.’
Mary Seton opened the parlour door, as calm as she had ever
been. They listened all.
They heard a clamour of voices outside. ‘Bothwell! Bothwell! Let
us in.’
‘Who are ye?’
‘We are hunted men—friends. We are here for our lives.’
Bothwell put his ear close to the door; his mouth worked fearfully,
all his features were distorted. Heavens! how he listened.
‘Who are ye? Tell me that.’
‘Friends—friends—friends!’
He laughed horribly—with a hollow, barking noise, like a leopard’s
cough. ‘By my God, Lindsay, I know ye now for a fine false friend.
You shall never take me here.’
For answer, the knocking was doubled; men rained blows upon
the door; and some ran round to the windows and jumped up at
them, crying, ‘Let us in—let us in!’ Some glass was broken; but the
shutter held. Mary Seton held the Queen close in her arms, Des-
Essars stood in the doorway with a drawn sword. Bothwell came up
to him for a moment. ‘By God, man, we’re rats in a drain—damned
rats, by my soul! Ha!’ he turned as Paris came down from the turret,
where he had been sent to spy.
 The house, Paris said, was certainly surrounded. The torches
made it plain that these were enemies. He had seen my lord of
Morton on a white horse, my Lords Hume and Sempill and some
more.
They all looked at each other, a poor ten that they were.
‘Hark to them now, master,’ says Paris. ‘They have a new cry.’
Bothwell listened, biting his tongue.
‘Murderer, murderer, come out! Come out, adulterous thief!’ This
was Lindsay again. There was no sound of Morton’s voice, the thick,
the rich and mellow note he had. But who was Morton, to call for the
murderer?
Paris, after spying again, said that they were going to fire the
doors; and added, ‘Master, it is hot enough without a fire. We had
best be off.’
Bothwell looked at the Queen. ‘My dear, I must go.’
She barely turned her eyes upon him; but she said, ‘Do you leave
me here?’ Scathing question from a bride, had a man been able to
observe such things.
He said, ‘Ay, I do. It is me they want, these dogs. You will be safe
if they know that I am away—and I will take care they do know it. I
go to Dunbar, whence you shall hear from me by some means.
Crookstone, come you with me, and come you, Hobbie. Paris, you
stay here.’
‘Pardon, master,’ says Paris, ‘I go with your lordship.’
Pale Paris was measured with his eye. ‘I’ll kill you if you do, my
fine man.’
‘That is your lordship’s affair,’ says Paris with deference; ‘but first I
will show you the way out. There are horses in the undercroft.’
Bothwell lifted up his wife, held her in his arms and kissed her
twice. ‘Fie, you are cold!’ he said, and put her down. She had lain
listless against him, without kissing.
 He turned at once and followed Paris; young Crookstone followed
him. It seems that he got clear off in the way he intended, for the
noises outside the house ceased; and in the grey of the morning,
before three o’clock, all was quiet about the policies. They must
have been within an ace of capturing him: in fact, Paris admitted
afterwards that they were but a bowshot away at one time.

The Queen sent Seton for Des-Essars at about four o’clock in the
morning. Neither mistress nor maid had been to bed.
He found her in a high fever; her eyes glowing like jet, her face
white and pinched; the stroke of her certain fate drawing down her
mouth. She said, ‘I have been a false woman, a coward, and a
shame to my race.’
‘God knows your Majesty is none of these.’
‘Baptist, I am going to my lord.’
‘Oh, madam, God forbid you!’
‘God will forbid me presently if I do not. It should have been last
night—I may be too late. But make haste.’
They procured a guide of a sort, a wretched poltroon of a fellow,
who twice tried to run for it and leave them in Yester woods. Des-
Essars, after the second attempt, rode beside him with a cocked
pistol in his hand. From Yester they went north by Haddington, for
fear of Whittingehame and the Douglases. As it was, they had to
skirt Lethington, and the Secretary’s fine grey house there in the
park; but the place was close-barred—nothing hindered them. They
passed unknown through Haddington, the Queen desperately tired.
Sixteen hours in the saddle, a cold welcome at the end.
Bothwell received them without cheer. ‘You would have been wiser
to have stayed. Here you are in the midst of war.’
‘My place was by your side.’
 The mockery of the thing struck him all at once. This schemed-for
life of his—a vast, empty shell of a house!
‘Oh, God, I sicken of this folly!’ He turned from her.
She had nothing to say, could hardly stand on her feet. Seton took
her to bed.
A message next day from Huntly in Edinburgh. Balfour held the
castle; all the rest of the town was Grange’s. Morton, Atholl, and
Lethington were rulers. Atholl had Holyroodhouse; Lethington and
his wife were with Morton. He himself, said Huntly, would move out
in a day or two and join the Hamiltons at Dalkeith. Let Bothwell raise
the Merse and meet them. He named Gladsmuir for rendezvous, on
the straight road from Haddington to the city, five miles by west of
Haddington.
Bothwell read all this to the Queen, who said nothing. She was
thinking of a business of her own, as appeared when she was alone.
She beckoned up Baptist.
‘There’s not a moment to be lost. Find me a messenger, a trusty
one, who will get speech with Mary Fleming.’
‘Madam,’ says Baptist, ‘let me go.’
‘No, no: I need you. Try Paris—no! my lord would never spare
him. And he would deny me again. Do you choose somebody.’
‘What is he to say to her, ma’am?’
‘He shall speak to her in private. She knows where my coffer is—
my casket.’
Ah! this was a grave affair. Des-Essars made up his mind at once.
‘Madam,’ he said, ‘let me advise your Majesty. Either send me, or
send no one. If you send me I will bring the casket back. That I
promise. If you send no one—if you do not remind her—it will slip
her memory.’
The Queen’s eyes showed her fears. ‘Remember you, Baptist, of
my casket. If Fleming were to betray me to Lethington——’ No need
to end.
‘Again I say, madam, send me.’
She thought; but even so her eyes filled with tears, which began
to fall fast.
‘Dearest madam, do you weep?’
‘I cannot let you go. Do not ask me—I need you here.’
He leaned to her. ‘Alas, what can I do to help your Majesty?’
She took his hand. ‘Stay. You are my only friend. The end is not
far. Have a little patience—stay.’
‘But your casket——’
She shook her head. ‘Let all go now. Stay you with me.’
‘Certainly I will stay with you,’ he said. ‘It will be to see you
triumph over your enemies.’
And again she shook her head. ‘Not with a broken heart!’ Then in
a frightened whisper she began to tell him her fears. ‘Do you know
what they make ready for me? The stake, and the faggot, and the
fire! Fire for the wife that slew her husband. Baptist, you will never
forsake me now! This is my secret knowledge. Never forsake me!’
She hid her face on his shoulder and cried there, as one lost.
Bothwell burst into the room: they sprang apart. He was eager,
flush with news. ‘We march to-morrow with the light. My men are
coming in—in good order. Be of good cheer, madam, for with God’s
help we shall pound these knaves properly.’
‘How shall God help us, my lord,’ said she, ‘who have helped not
Him?’
‘Why, then, my dear,’ cries he with a laugh, ‘why, then, we will
help ourselves.’
 CHAPTER XI
APPASSIONATA

Grange, that fine commander, got his back to the sun and gave
the lords the morning advantage. ‘We shall want no more than that,’
he told Morton; ‘by ten o’clock they will be here, and by noon we
shall be through with it.’
‘Shall we out banner, think you?’ says Morton.
‘Nay, my lord, nay. Keep her back the now.’ Grange was fighting
with his head, disposing his host according to the lie of the ground,
and his reserves also. He took the field before dawn, and had every
man at his post by seven o’clock. There was a ground mist, and the
sea all blotted out: everything promised great heat.
They were to be seen, a waiting host, when the Queen crested
Carbery Hill and watched her men creep round about; with Erskine
beside her she could make them out—arquebusiers, pikemen, and
Murrays from Atholl on the lowest ground (Tillibardine leading
them), on either wing horsemen with spears. They had a couple of
brass field-pieces in front. One could see the chiefs walking their
horses up and down the lines, or pricking forward to confer, or
clustering together, looking to where one pointed with his staff.
There was Morton on his white horse, himself, portly man, in black
with a steel breastplate—white sash across it—in his steel bonnet a
favour of white. White was their badge, then; for, looking at them in
the mass, the host was seen to be spattered with it, as if in a
neglected field of poppies and corncockles there grew white daisies
interspersed. The stout square man in leather jerkin and buff boots
was Grange—on a chestnut horse; with him to their right rode Atholl
on a black—Atholl in a red surtout, and the end of his fine beard lost
in the white sash which he too had. Who is the slim rider in black—
haunting Atholl like a shadow? Who but careful Mr. Secretary
Lethington could have those obsequious shoulders, that attentive
cock of the head? Lethington was there, then! Ah! and there, by
one’s soul, was Archie Douglas’s grey young head, and his white
minister’s ruff, where a red thread of blood ought to be. Glencairn
was there, Lindsay, Sempill, Rothes—all those strong tradesmen,
who had lied for their profit, and were now come to claim wages: all
of them but the trader of traders, the white-handed prayerful man,
the good Earl of Moray, safe in France, waiting his turn.
So prompt as they stood down there in the grey haze, all rippling
in the heat; without sound of trumpet or any noise but the
whinnying of a horse; without any motion save now and then, when
some trooper plunged out of line and must pull back—that thing of
all significant things about them was marked by the Queen, who
stood shading her eyes from the sun atop of Carbery Hill. ‘Oh,
Erskine!’ she said, ‘oh, Bothwell! they have no standard. Against
whom, then, do we fight?’
Bothwell, exasperated by anxiety, made short answer: ‘It is plain
enough to see what and who they are. They are men—desperate
men. They are men for whom loss means infamous death. For, mark
you well, madam, if Morton lose this day he loses his head.’
‘Ay,’ she gloomed, ‘and many more shall lose theirs. I will have
Lindsay’s and Archie’s—and you shall have Lethington’s.’
‘I would have had that long ago, if you had listened to me. And
now you see whether I was right or wrong. But when women take to
ruling men——’
She touched his arm. ‘Dear friend, for whom I have suffered many
things, do not reproach me at this hour.’ The tears were in her eyes
—she was always quick at self-pity.
But he had turned his head. ‘Ha! they need me, I see. Forgive me,
madam, I must have a word with Ormiston.’ He saluted and rode
down to meet his allies. Monsieur Du Croc, the French Ambassador,
approached her, hat in hand. He was full of sympathy; but, with his
own theories of how to end this business, could not give advice.
Sir James Melvill, watching the men come up, shook his head at
the look of them. ‘No heart in their chance—no heart at all,’ he was
heard to say.
 The Queen’s forces deployed across the eastern face of Carbery
Hill in a long line which, it was clear, was not of equal strength with
the lords’. It became less so as the day wore; for had you looked to
its right you would have seen a continual trickle of stooping, running
men crossing over to the enemy. These were deserters at the
eleventh hour; Bothwell rode one of them down, chased him, and
when he fell drove his horse over him and over in a blind fury of
rage, trampling him out of semblance to his kind. It stayed the leak
for a while; but it began again, and he had neither heart nor time to
deal with it. Where were the Hamiltons who should have been with
her? Where, alas, were the Gordons? In place of them the Borderers
and Foresters looked shaggy thieves—gypsies, hill-robbers, savage
men, red-haired, glum-faced, many without shoes and some without
breeches. The tressured Lion of Scotland was in Arthur Erskine’s
hold: at near ten o’clock Bothwell bade him display it. It unfurled
itself lazily its full length; but there was no breath of air. It clung
about the staff like so much water-weed; and they never saw the
Lion. No matter; it would be a sign to that watchful host in the plain:
now let us see what flag they dare to fly. They waited tensely for it,
a group of them together—the Queen with her wild tawny hair fallen
loose, her bare thin neck, her short red petticoat and blue scarf;
Bothwell biting his tongue; Ormiston, Des-Essars, sage Monsieur Du
Croc.
They saw two men come out of the line bearing two spears close
together. At a word they separated, backing from each other: a great
white sheet was displayed, having some picture upon it—green, a
blot like blood, a wavy legend above. One could make out a tree;
but what was the red stain? They talked—the Queen very fast and
excitedly. She must know what this was—she would go down and
find out—it was some insult, she expected. Was that red a fire? Who
would go? Des-Essars offered, but she refused him. She chose Lord
Livingstone for the service, and he went, gallantly enough—and
returned, a scared old optimist indeed. However, she would have it,
so she learned that they had the King lying dead under a tree, and
the Prince his son praying at his feet—with the legend, ‘Judge and
avenge my cause, O Lord!’ The red was not a fire, but the Prince’s
robe. The Queen cried out: ‘Infamy! Infamy! They carry their own
condemnation—do you not see it?’ If anybody did, he did not say so.

Monsieur Du Croc had his way at last, and was allowed to carry
messages between the hosts. The burden of all that he brought back
was that the lords would obey the Queen if she would give up the
murderers, whom they named. The offer was ludicrous, coming from
Morton—but when she ordered Du Croc back to expose it, he fairly
told her to read below the words. They had come for Lord Bothwell.
‘I will die sooner than let him be touched,’ said she. ‘Let some one—
Hob Ormiston, go you—fetch Grange to speak with me.’ Hob went
off, with a white scarf in his held-up hand; and the Queen rode half-
way down the hill for the parley. The great banner dazzled her: it
was noticed that she bent her head down, as one rides against the
sun.
Grange came leisurely up towards her—a rusty man of war,
shrewd, terse, and weathered. He could only report what his
masters bade him: they called for the surrender of the murderers.
She flamed and faced him with her royal anger. ‘And I, your
sovereign lady, bid you, Grange, go over there and bring the
murderers to me. Look, there goes one on his white horse! And
there shirk two after him, hiding behind him—the one with a grey
head, and the other with a grey face. Fetch you me those.’
‘Bah!’ snarled Bothwell, ‘we talk for ever. Let me shoot down this
dog.’ A Hepburn—quiet and sinewy—stepped out of the ranks with a
horse-pistol. Grange watched him without moving a muscle; but
‘Oh!’ cried the Queen, ‘what villainy are you about?’ She struck down
the pistol-arm,—as once before she had struck down Fawdonsyde’s.
Bothwell, red in the face, said, ‘Let us end this folly. Let him who
calls for me come and fetch me. I will fight with him here and now.
Go you, Grange, and bring my Lord Morton hither.’
 ‘No need for his lordship, if I will serve your turn, Earl of Bothwell,’
says Grange.
But Bothwell said, ‘Damn your soul, I fight with my equals. None
knows it better than you.’ He would have no one below an Earl’s
rank—himself being now, you must recollect, Duke of Orkney and
Zetland—and it should be Morton for choice.
Grange, instructed by the Queen, rode back. They saw Morton
accost him, listen, look over the valley. He called a conference—they
talked vehemently: then Morton and Lindsay pricked forward up the
hill, and stopped within hailing distance.
‘You, Bothwell,’ cried Morton, ‘come you down, then; and have at
you here.’
The Queen’s high voice called clearly back. ‘He shall never fight
with you, murderer.’
Lindsay bared his head. ‘Then let him take me, madam; for I am
nothing of that sort.’
‘No, no, Lindsay,’ said Bothwell; ‘I have no quarrel with you.’
The Earl of Morton had been looking at Bothwell in his heavy,
ruminating way, as if making up his mind. While the others were
bandying their cries, the Queen’s voice flashing and shrieking above
the rest, he still looked and turned his thoughts over. Presently—in
his time—he gave Lindsay his sword and walked his horse up the hill
to the Queen’s party. He saluted her gravely. ‘With your gracious
leave, madam, I seek to put two words into my Lord Bothwell’s ear.
You see I have no sword.’
The Queen looked at once to her husband. He nodded, gave his
sword to Huntly, and said, ‘I am ready for you.’ They moved ten
yards apart; Morton talked and the other listened.
‘Bothwell, my man,’ he said, ‘there’s no a muckle to pick between
us, I doubt—I played one card and you another; but I have the
advantage of ye just now, and am no that minded to take it up.
Man!’ he chuckled, ‘ye stumbled sorely when ye let them find for the
powder!’
‘Get on, get on,’ says Bothwell, drawing a great breath.
‘I will,’ Morton said. ‘I am here to advise ye to make off while you
can. Go your ways to Dunbar, and avoid the country for a while. I’ll
warrant you you’ll not be followed oversea. All my people will serve
the Queen—have no fear for her. Now, take my advice; ’tis fairly
given. I’ve no wish to work you a mischief—though, mind you, I
have the power—for you and I have been open dealers with each
other this long time. And you brought me home—I’m not one to
forget it. But—Lord of Hosts! what chance have you against Grange?’
He waited. ‘Come now, come! what say you?’
Lord Bothwell considered it, working his strong jaw from side to
side: a fair proffer, an honourable proffer. He looked at the forces
against him—though he had no need; he knew them better men
than his, because Grange was a better man than he. That banner of
murder—the cry behind it—the Prince behind the cry, up on the rock
of Stirling: in his heart he knew that he had lost the game. No way
to Stirling—no way! But the other way was the sea-way—the old free
life, the chances of the open water. Eh, damn them, he was not to
be King of Scots, then! But he had known that for a week. He turned
his head and saw the sea like molten gold, and far off, dipped in it, a
little ship with still sails—Ho! the sea-way!
‘By God, Morton,’ he said, ‘you may be serving me. I’ll do it.’
‘Go and tell her,’ says Morton; and they both went back to the
Queen.
Both took off their bonnets. Bothwell said: ‘Madam, we must avoid
blood-shedding if we may, and I have talked with my lord of Morton.
He makes an offer of fair dealing, which I have taken. I have a clear
road to Dunbar, thence where I will. All these hosts will follow you if
I am not there. They pay me the compliment of high distrust, you
perceive. After a little, I doubt not but you shall see me back again
where I would always be. Madam, get the Prince in your own hands:
all depends upon him. And now, kiss me, sweetheart, for I must be
away.’
She heard him—she understood him—she believed him. She was
curious to observe that she felt so little. Her voice when she
answered him had no spring in it—it was worn and thin, with a little
grating rasp in it—an older voice.
‘It may be better so. I hate to shed good blood. Whither shall I
write to you? At Dunbar? In England? Flanders?’ There had been a
woman in Dunkirk—she remembered that.
He was looking away, answering at random, searching whom he
should take with him, or on whom he could reckon to follow him if
he asked. ‘I will send you word. Yes, yes, you will write to me. You
shall know full soon. But now I cannot stay.’
Morton had returned to his friends.
‘Paris, come you with me. Ormiston, are you for the sea? No? Stay
and be hanged, then. Hob? What, man, afraid? Where is Michael
Elliott? Where is Crookstone? What Hepburn have I?’ He collected six
or eight—both the Ormistons decided for him—Powrie and Wilson,
Dalgleish, one or two more.
He took the Queen’s hand gaily. ‘Farewell, fair Queen!’ he said;
and she, ‘Adieu, my lord.’ He leaned towards her: ‘One kiss, my
wife!’ but she drew back. ‘Your lips are foul—you have kissed too
many—no, no.’ ‘I must have it—you must kiss me’—he pressed
against her. For a while she was agitated, defending herself; but
then, with a sob, ‘Ay, take what you will of me,’ she said—‘it is little
worth.’ He got his cold kiss, and rode fast through his scattering
host. This going of his was the Parthian shot. He had beaten her.
Desire was dead.
The Queen sat still—with a face like a rock. ‘Has he gone?’ she
asked Des-Essars in a whisper.
‘Yes, thank God,’ said he.
 She shook herself into action, gathered up the reins, and turned to
Erskine. ‘Come,’ she said, ‘we will go down to them now.’
She surrendered to the Earl of Atholl, who, with Sempill and
Lindsay, came up to fetch her. Followed by one or two of her friends
—Des-Essars, Melvill, Du Croc, and Livingstone—she rode down the
hill from her host and joined the other. Grange cantered up,
bareheaded, to meet her, reined up short, took her hand and kissed
it. Many followed him—Glencairn, Glamis, young Ruthven. Each had
his kiss; but then came Archie Douglas smelling and smiling for his—
and got nothing. She drew back from him shuddering: he might
have been a snake, he said. Lethington was not to be seen. The host
stood at ease awaiting her; the white banner wagged and dipped, as
if mocking her presence. ‘Take that down,’ she said, with a crack in
her dry throat; but no one answered her. She had to go close by the
hateful thing—a daub of red and green and yellow—crowned
Darnley crudely lying under a tree, a crowned child kneeling at his
feet, spewing the legend out of his mouth. She averted her eyes and
blinked as she passed it: an ominous silence greeted her, sullen
looks; one or two steady starers showed scornful familiarity with ‘a
woman in trouble’; one said ‘Losh!’ and spat as she passed.
She was led through the Murrays, Humes, and Lindsays; murmurs
gathered about her; all eyes were on her now, some passionate,
some vindictive, some fanatic. On a sudden a pikeman ran out of his
ranks and pointed at her—his face was burnt almost black, his eyes
showed white upon it. ‘Burn the hure!’ he raved, and when she
caught her breath and gazed at him, he was answered, ‘Ay, ay, man.
Let her burn herself clean. To the fire with her!’
Her fine heart stood still. ‘Oh!’ she said, shocked into childish
utterance, ‘oh, Baptist, they speak of me. They will burn me—did
you hear them?’ Her head was thrown back, her arm across her
face. She broke into wild sobbing—‘Not the fire! Not the fire! Oh,
pity me! Oh, keep me from them!’
‘Quick, man,’ said Atholl, ‘let us get her in.’ Orders were shortly
given, lieutenants galloped left and right to carry the words. The
companies formed; the monstrous banner turned about. Morton
bade sound the advance; between him and Atholl she was led
towards Edinburgh. ‘If Erskine is a man he will try a rescue,’ thought
Des-Essars, and looked over his shoulder to Carbery Hill—now a
bare brae. The Queen’s army had vanished like the smoke.
So towards evening they came to town, heralded by scampering
messengers, and met by the creatures of the suburb, horrible
women and the men who lived upon them—dancing about her,
mocking obscenely, hailing her as a spectacle. She bowed her head,
swaying about in the saddle. Way was driven through; they passed
under the gates, and began to climb the long street, packed from
wall to wall with raving, cursing people. They shook their fists at her,
threw their bonnets; stones flew about—she might have been killed
outright. The cries were terrible—‘Burn her, burn her! Nay, let her
drown, the witch!’ Dust, heat, turmoil, a brown fetid air, hatred and
clamour—the houses seemed to whirl and dizzy about her. The earth
rocked; the people, glued in masses of black and white, surged
stiffly, like great sea waves. Pale as death, with shut eyes and
moving, dumb lips, she wavered on her seat, held up on either side
by a man’s arm. Des-Essars prayed aloud that a stone might strike
her dead.
They took her to a house by the Tron Church, a house in the High
Street, and shut her in an upper room, setting a guard about the
door. The white banner was planted before the windows, and the
crowd swarmed all about it, shrieking her name, calling her to come
out and dance before them. Her dancing was notorious, poor soul;
many a mad bout had she had in her careless days. ‘Show your legs,
my bonnie wife!’ cried some hoarse shoemaker. ‘You had no shame
to do it syne.’ This lasted till near midnight—for when it grew dark
torches were kindled from end to end of the street, drums and pipes
were set going, and many a couple danced. The Queen during this
hellish night was crouched upon the floor, hiding her face upon Mary
Seton’s bosom. Des-Essars knelt by her, screening her from the
windows. She neither spoke nor wept—seemed in a stupor. Food
was brought her, but she would not move to take it; nor would she
open her mouth when the cup was held at her lips.
Next morning, having had a few hours’ peace, the tumult began
betimes—by six o’clock the din was deafening. She had had a sop in
wine, and was calmer; talked a little, even peeped through the
curtain at the gathering crowd. She watched it for, perhaps, an hour,
until they brought the mermaid picture into action—herself naked to
the waist, with a fish-tail—confronted it with the murder flag, and
jigged it up against it. This angered her; colour burned in her white
cheeks. ‘Infamous! Swine that they are! I will brave them all.’
Before they could stop her she had thrown open the window, and
stood outside on the balcony, proudly surveying and surveyed.
At first there was a hush—‘Whisht! She will likely speak till us,’
they told each other. But she said nothing, and gave them time to
mark her tumbled bodice and short kirtle, her wild hair and stained
face. They howled at her, mocking and gibing at her—the two
banners flacked like tailless kites. Presently a horseman came at a
foot’s pace through the press. The rider when he saw her pulled his
hat down over his eyes—but it was too late. She had seen
Lethington. ‘Ha, traitor, whose rat-life I saved once,’ she called out,
in a voice desperately clear and cold, ‘are you come to join your
friends against me? Stay, Mr. Secretary, and greet your Queen in the
way they will teach you. Or go, fetch your wife, that she may thank
her benefactress with you. Do you go, Mr. Secretary?’
He was, in fact, going; for the crowd had turned against him and
was bidding him fetch his wife. ‘Give us the Popish Maries together,
sir, and we’ll redd Scotland of them a’.’
‘Rid Scotland of this fellow, good people,’ cried the Queen, ‘and
there will be room for one honest man.’
They jeered at her for her pains. ‘Who shall be honest where ye
are, woman? Hide yourself—pray to your idols—that they keep ye
from the fire.’
 ‘Oh, men, you do me wrong,’ she began to moan. ‘Oh, sirs, be
pitiful to a woman. Have I ever harmed any?’
They shrieked her down, cursing her for a witch and a husband-
killer. The flags were jigged together again—a stone broke the
window over her head. Des-Essars then got her back by force.
It is amazing that she could have a thought in such a riot of fiends
—yet the sight of Lethington had given her one. She feared his grey,
rat’s face. She whispered it to Des-Essars. ‘Baptist, you can save me.
Quick, for the love of Christ! The coffer! the coffer!’
He knew what she meant. That coffer contained her letters to
Bothwell, her sonnets—therefore, her life. He understood her, and
went away without a word. He took his sword, put a hood over his
head, got out of the backside of the house, over a wall, into the
wynd. Hence, being perfectly unknown, he entered the crowd in the
High Street and worked his way down the Canongate. He intended
to get into Holyroodhouse by the wall and the kitchen window, as he
had done many a time, and notably on the night of David’s
slaughter.[12]

Des-Essars had gone to save her life; but whether he did it or no,
he did not come back. She wore herself to thread, padding up and
down the room, wondering and fretting about him. This new anxiety
made her forget the street; but towards evening, when her nerves
were frayed and raw, it began to infuriate her—as an incessant cry
always will. She suddenly began panting, and stood holding her
breasts, staring, moving her lips, her bosom heaving in spite of her
hands. ‘God! Mother of God! Aid me: I go mad,’ she cried,
strangling, and ‘Air! I suffocate!’ and once more threw open the
windows and let in the hubbub.
She was really tormented for air and breath. She tore at her
bodice, split it open and showed herself naked to the middle.
‘Yes—yes—you shall look upon me as I was made. You shall see
that I am a woman—loved once—loved much. See, see, my flesh!’
Horrible scandal!—but the poor soul was mad.
Soon after this some of the lords came to her—Lindsay, Morton,
and Atholl. The windows, they said, must be closed at once; they
feared a riot. They would take her back to Holyroodhouse if she
would be patient. But she must be rendered decent: Atholl gave her
his cloak. She had quieted immediately they came, and thanked
them meekly.
They took her away at once. Mary Seton followed close, but was
gently pushed back by Lord Morton. ‘No, no: she must come alone.
You shall see her after a little. You cannot come now.’ For the first
time in her life, as I believe, Mary Seton shed tears.
A very strong guard, with pikes presented, hedged her in. She
reached Holyrood on foot, and was shut into her own cabinet. It was
empty and dark but for the candle they had left with her. She
snatched it up, and began a mad, fruitless hunt for her casket. It
was not in its place—it was nowhere. She hunted until she dropped.
She began to tear at herself and to shriek. Doom! Doom! She must
be burned. They had taken her coffer. She was alone—condemned
and alone.
Then Des-Essars crawled out of the dark on his hands and one
knee, dragging a broken leg after him, and fell close beside her, and
kissed the hem of her petticoat.
 [12] The casket, which was not at Holyrood, is supposed to
have been secured by Bothwell in the Castle, where it was to be
found in due time. But Des-Essars did not know that. Nor is it
clear to me how Bothwell had found opportunity to get it there.

CHAPTER XII
ADDOLORATA

She sat on the floor, and had his head at rest on her lap. Her
hands were upon him, and so he rested. The great tears fell fast and
wetted his hair.
Her grief was silent and altogether gentle. Still as she sat there,
looking before her with wide unwinking eyes and lips a little parted,
she was unconscious of what she was suffering or had suffered: all
about her was the blankness of dark, and without her knowledge the
night fell; the dusk like a vast cloak gathered round about her, fold
over fold; and still she sat and looked at nothing with her wide
unwinking eyes. Slowly they filled and brimmed, and slowly the
great tears, as they ripened, fell. There were no other forms of grief,
none of grief’s high acts: only their bitter symbol—lamentation
embodied in tears, and nakedly there.
‘Nay, move not your hands—nay, touch my brows: my head aches
—I am blind.’ The lad supine in her lap pleaded in whispers.
Gentle-voiced she answered him. ‘There is no work left for my
hands to do but to tend thee, my dear.’
He lay dumb for a while; then said he: ‘You shall not blame me. It
is not here—not in the house. I know not where it is. They are
seeking it now. He came here with two archers. He snarled like a fox
to find me.’