Download the Full Version of textbook for Fast Typing at textbookfull.

com

Machine Learning Engineering in Action MEAP V04

Ben T Wilson

https://textbookfull.com/product/machine-learning-
engineering-in-action-meap-v04-ben-t-wilson/

OR CLICK BUTTON

DOWNLOAD NOW

Download More textbook Instantly Today - Get Yours Now at textbookfull.com

 Recommended digital products (PDF, EPUB, MOBI) that
you can download immediately if you are interested.

Apache Pulsar in Action MEAP V04 David Kjerrumgaard

https://textbookfull.com/product/apache-pulsar-in-action-
meap-v04-david-kjerrumgaard/

textboxfull.com

Spring in Action - Sixth Edition (MEAP V04) Craig Walls

https://textbookfull.com/product/spring-in-action-sixth-edition-
meap-v04-craig-walls/

textboxfull.com

grokking Machine Learning MEAP v07 Luis G Serrano

https://textbookfull.com/product/grokking-machine-learning-
meap-v07-luis-g-serrano/

textboxfull.com

Five Lines of Code MEAP V04 Christian Clausen

https://textbookfull.com/product/five-lines-of-code-
meap-v04-christian-clausen/

textboxfull.com
Microservices in NET Core Second Edition MEAP V04
Christian Horsdal Gammelgaard

https://textbookfull.com/product/microservices-in-net-core-second-
edition-meap-v04-christian-horsdal-gammelgaard/

textboxfull.com

Domain Specific Languages Made Easy MEAP V04 Meinte

Boersma

https://textbookfull.com/product/domain-specific-languages-made-easy-
meap-v04-meinte-boersma/

textboxfull.com

Interpretable AI Building explainable machine learning

systems MEAP V02 Ajay Thampi

https://textbookfull.com/product/interpretable-ai-building-
explainable-machine-learning-systems-meap-v02-ajay-thampi/

textboxfull.com

Quantum Computing In Action MEAP v09 Johan Vos

https://textbookfull.com/product/quantum-computing-in-action-
meap-v09-johan-vos/

textboxfull.com

Kubernetes in Action, Second Edition (MEAP V05) Marko

Luksa

https://textbookfull.com/product/kubernetes-in-action-second-edition-
meap-v05-marko-luksa/

textboxfull.com
 MEAP Edition
Manning Early Access Program
Machine Learning Engineering in Action
Version 4

Copyright 2021 Manning Publications

For more information on this and other Manning titles go to

manning.com

©Manning Publications Co. To comment go to liveBook

 welcome
Thank you for taking the time to take a look at Machine Learning Engineering in Action. I
sincerely hope that all of the lessons that I’ve learned the hard way throughout my career can
be of a benefit to you!
Like any ML project that I’ve ever worked on, I’ve always made the solution better by listening
very carefully to what other people have to say and learning from their thoughts and opinions.
This MEAP is no different. I’m excited to read what you have to say about what you’re seeing in
this book and working to make it better through your feedback.

To get the most out of most of the lessons and topics in this book it will help to have at least a
moderate knowledge of the applications of ML (algorithmic theory will be discussed in places,
but there are no requirements to understand the nuances of these libraries) and have the ability
to build an ML solution in either Python or Scala.

Throughout the book we’ll be covering the enormous elephant in the room of nearly all
companies who are working towards getting benefits out of Data Science and ML work: why so
many projects fail. Whether it be failure-to-launch scenarios, overly complex solutions to
problems that could be solved in simpler manners, fragile code, expensive solutions, or poorly
trained implementations that provide really poor results, the end result in many projects across
so many industries are failing to live up to the promises of what is expected in predictive
modeling. As we go through scenarios and code implementation demonstrations, we’ll be
covering not only the ‘how’ of building resilient ML solutions, but the ‘why’ as well.

The goal of this book is not to show how to implement certain solutions through the application
of specific algorithms, explain the theory behind how the algorithms work, nor is it focused on
one particular sub-genre of ML. It instead is a focus on the broad topic of successful ML work,
how to apply Agile fundamentals to ML, and demonstrably production-ready code bases
that will, if adhered to, ensure that you have a maintainable and robust solution for projects.

With the dozens of ML theory and model-centric books that I’ve read and collected throughout
my career, I really wish that I had a copy of this book when I was getting started in this field
many years ago. The crippling failures that I’ve endured as I’ve learned these paradigms have
been both painful and formative, and to be able to provide them for the next generation of
ML practitioners out there is truly a gift for me.

Thank you once again for taking the time to look at this work, provide feedback in liveBook
discussion forum (pointing out things that I might have missed or that you’d like me to cover

©Manning Publications Co. To comment go to liveBook

will be most welcome!), and joining in on (hopefully pleasant) conversations about what you
don’t and do like about the work.

Kindest Regards,
Ben Wilson

©Manning Publications Co. To comment go to liveBook

 brief contents
PART 1 INTRODUCTION TO MACHINE LEARNING ENGINEERING
1 An Introduction to Machine Learning Engineering
2 Your Data Science could use some Engineering
3 Before you model: Planning and Scoping
4 Before you model: Communication and Logistics of projects
5 Experimentation in Action: planning and researching an ML project
6 Experimentation in Action: testing and evaluating a project
7 Experimentation in Action: moving from prototype to MVP
8 Experimentation in Action: finalizing an MVP with MLFlow and runtime
optimization
PART 2 PREPARING FOR PRODUCTION: CREATING MAINTAINABLE ML
9 Modularity for ML: writing testable and legible code
10 Standards of coding and creating maintainable ML code
11 Big O(no) and how to think about runtime performance
12 Model measurement and why it’s so important
13 ML Development hubris
PART 3 DEVELOPING PRODUCTION MACHINE LEARNING CODE
14 Writing Production Code
15 Qualifying and Acceptance Testing
16 Reworking

©Manning Publications Co. To comment go to liveBook

PART 4 INFERENCE AND AUTOMATION
17 Supporting your model and your code
18 Automation tooling

©Manning Publications Co. To comment go to liveBook

 1

1
An Introduction to Machine Learning
Engineering

This chapter covers

• Defining Machine Learning Engineering and why it is important to increase the chances of
successful ML project work
• Explaining why ML Engineering and the processes, tools, and paradigms surrounding it can
minimize the chances of ML project abandonment
• Discussing the six primary tenets of ML Engineering and how failing to follow them causes
project failure
Machine learning (ML) is exciting. To the layperson, it brings with it the promise of
seemingly magical abilities of soothsaying; uncovering mysterious and miraculous answers to
difficult problems. ML makes money for companies, it autonomously tackles overwhelmingly
large tasks, and relieves people from the burden of monotonous work involved in analyzing
data to draw conclusions from. To state the obvious, though, it’s challenging. From
thousands of algorithms, a diverse skill set ranging from Data Engineering (DE) skills to
advanced statistical analysis and visualization, the work required of a professional
practitioner of ML is truly intimidating and filled with complexity.
ML Engineering is the concept of applying a system around this staggering level of
complexity. It is a set of standards, tools, processes, and methodology that aims to minimize
the chances of abandoned, misguided, or irrelevant work being done in an effort to solve a
business problem or need. It, in essence, is the roadmap to creating ML-based systems that
can not only be deployed to production, but can be maintained and updated for years in the
future, allowing businesses to reap the rewards in efficiency, profitability, and accuracy that
ML in general has proven to provide (when done correctly).

©Manning Publications Co. To comment go to liveBook

 2

This book is, at its essence, a roadmap to guide you through this system, as shown in
figure 1.1 below. It gives a proven set of processes (mostly a ‘lessons learned’ of things I’ve
screwed up in my career) about the planning phase of project work, navigating the difficult
and confusing translation of business needs into the language of ML work. From that, it
covers a standard methodology of experimentation work, focusing on the tools and coding
standards for creating an MVP that will be comprehensive and maintainable. Finally, it will
cover the various tools, techniques, and nuances involved in crafting production-grade
maintainable code that is both extensible and easy to troubleshoot.

Figure 1.1 The ML Engineering Roadmap, showing the proven stages of work involved in creating successful
ML solutions. While some projects may require additional steps (particularly if working with additional
Engineering teams), these are the fundamental steps that should be involved in any ML-backed project.

ML Engineering is not exclusively about the path shown in figure 1.1. It is also the
methodology within each of these stages that can make or break a project. It is the way in
which a Data Science team talks to the business about the problem, the manner in which
research is done, the details of experimentation, the way that code is written, and the
multitude of tools and technology that are employed while traveling along the roadmap path
that can greatly reduce the worst outcome that a project might have – abandonment.
The end goal of ML work is, after all, about solving a problem. By embracing the concepts
of ML Engineering and following the road of effective project work, the end goal of getting a

©Manning Publications Co. To comment go to liveBook

 3

useful modeling solution can be shorter, far cheaper, and have a much higher probability of
succeeding than if you just ‘wing it’ and hope for the best.

1.1 Why ML Engineering?

To put it most simply, ML is hard. It’s even harder to do correctly in the sense of serving
relevant predictions, at scale, with reliable frequency. With so many specialties existing in
the field (NLP, forecasting, Deep Learning, traditional linear and tree-based modeling, et al),
an enormous focus on active research, and so many algorithms that have been built to solve
specific problems, it’s remarkably challenging to learn even slightly more than an
insignificant fraction of all there is to learn. Coupling that complexity with the fact that one
can develop a model on everything from a RapsberryPi to an enormous NVIDIA GPU cluster,
the very platform complexities that are out there is an entirely new set of information that no
one person could have enough time in their life to learn.
There are also additional realms of competency that a Data Scientist is expected to be
familiar with. From mid-level Data Engineering skills (you have to get your data for your
Data Science from somewhere, right?), software development skills, project management
skills, visualization skills, and presentation skills, the list grows ever longer and the volumes
of experience that need to be gained become rather daunting. It’s not much of a surprise,
considering all of this, as to why ‘just figuring it out’ in reference to all of the required skills
to create production-grade ML solutions is untenable.
The aim of ML Engineering is not to iterate through the lists of skills just mentioned and
require that a DS master each of them. Instead, it’s a collection of certain aspects of those
skills, carefully crafted to be relevant to Data Scientists, all with the goal of increasing the
chances of getting an ML project into production and to make sure that it’s not a solution
that needs constant maintenance and intervention to keep running.
An ML Engineer, after all, doesn’t need to be able to create applications and software
frameworks for generic algorithmic use cases. They’re also not likely to be writing their own
large-scale streaming ingestion ETL pipelines. They similarly don’t need to be able to create
detailed and animated front-end visualizations in JavaScript.
An ML Engineer needs to know just enough software development skills to be able to
write modular code and to implement unit tests. The don’t need to know about the intricacies
of non-blocking asynchronous messaging brokering. They need just enough Data
Engineering skills to build (and schedule the ETL for) feature datasets for their models, but
not how to construct a PB-scale streaming ingestion framework. They need just enough
visualization skills to create plots and charts that communicate clearly what their research
and models are doing, but not how to develop dynamic web apps that have complex UX
components. They also need just enough project management experience to know how
to properly define, scope, and control a project to solve a problem, but need not go through
a PMP certification.
If you’ve come here hoping for discussions on the finer points of Data Science,
algorithms, or to get implementation ideas for a specific use case, this book isn’t going to
focus on any of that in incredible detail (although the code examples in the ML sections are
going to be real code and not some simple hand-waving pseudo-code). The books that

©Manning Publications Co. To comment go to liveBook

 4

delve into the specifics of these topics have already been (and continue to be) written in
great detail.
We’re here instead to talk about the giant elephant in the room when it comes to ML.
We’re going to be talking about why, with so many companies going all-in on ML, hiring
massive teams of highly compensated Data Scientists, devoting massive amounts of financial
and temporal resources to projects, these projects end up failing at an incredibly high rate.
We’ll be covering the 6 major parts of project failure, as showing in figure 1.1 below,
discussing process around how to identify the reasons why these each cause so many
projects to fail, be abandoned, or take far longer than they should to reach production. In
each section throughout the first part of this book we will be discussing the solutions to each
of these common failures and covering the processes that can make the chances of these
derailing your projects very low.

Hubris: 5 %

Cost: 10 %

Planning: 30 %

Fragility: 15 %

Technology: 15 %

Scoping: 25 %

Planning Scoping Technology Fragility Cost Hubris

Figure 1.2 Primary reasons for ML project failures from not only my own early work, but many others that I’ve
been involved in a reclamation capacity since those early years.

Figure 1.2 shows some rough estimates of what I’ve come to see as the primary reasons
why projects fail (and the rates of these failures in any given industry, from my experience,
are truly surprising). Generally, this is due to a DS team that is either inexperienced with
solving a large-scale production-grade model to solve a particular need or has simply failed
to understand what the desired outcome from the business is.

©Manning Publications Co. To comment go to liveBook

 5

At no point in any project have I ever seen any of these issues arise due to malicious
intent; rather, these elements are due in large part to the fact that most ML projects are
incredibly challenging, complex, and are comprised of algorithmic software tooling that is
hard to explain to a layperson (hence the breakdowns in communication with business units
that most projects endure). With such complex solutions at play, so many moving parts, and
a world of corporations trying to win in this new data-focused arms race of profiting off of ML
as quickly as possible, it’s no wonder that the perilous journey of taking a solution to a point
of stability in production fails so frequently.
This book isn’t focused as a doom-riddled treatise on how challenging ML is; rather, it’s
meant to show how these elements can be a risk for projects and to teach the tools that help
minimize the risk of each. Through focusing on each of these areas in a conscientious and
deliberate manner, many of these risks can be entirely mitigated, if not eliminated entirely.
Below, in figure 1.3, we can see the representation of the path that all of us are moving
towards when we employ ML to solve a problem. From the outset of a project to its planned
successful, long running and maintainable state, the journey is fraught with detours that can
spell the termination of our hard work. However, by focusing on building up the knowledge,
skills, and utilization of processes and tooling, each of these 6 major areas can be generally
avoided (or, at the very least, addressed in a way that won’t cause a complete failure of the
project).

©Manning Publications Co. To comment go to liveBook

 6

Figure 1.3 The branching paths of failure for the vast majority of ML projects. Nearly all ML solutions that don’t
plan to focus on these 6 core areas have a much higher chance of being abandoned either before production,
or shortly after running in production.

©Manning Publications Co. To comment go to liveBook

 7

The framework that is ML Engineering is exactly dedicated to address each of these

primary failure modes shown above in figure 1.3. Eliminating these chances of failure is at
the heart of this methodology. It is done by providing the processes to make better
decisions, ease communication with internal customers, eliminate rework during the
experimentation and development phases, create code bases that can be easily maintained,
and to bring a best-practices focused approach to any project work that is heavily influenced
by DS work. Just as software engineers decades ago refined their processes from large-scale
waterfall implementations to a more flexible and productive agile process, ML Engineering
seeks to define a new set of practices and tools that will optimize the wholly unique realm of
software development for Data Scientists.

1.2 The core tenets of ML Engineering

Now that we have a general idea of ‘what’ ML Engineering is, we can focus in a bit to the
key elements that make up those incredibly broad categories from section 1.1. Each of these
topics will be a focus of entire chapter-length in-depth discussions later in this book, but for
now we’re going to look at them in a holistic sense by way of potentially painfully familiar
scenarios to elucidate why they’re so important.

1.2.1 Planning
By far the largest cause of project failures, failing to plan out a project thoroughly is one
of the most demoralizing ways for a project to be cancelled. Imagine for a moment that
you’re the first-hired DS for a company. On your first week, an executive from marketing
approaches you, explaining (in their terms) a serious business issue that they are having.
They need to figure out an efficient means of communicating to customers through email to
let them know of upcoming sales that they might be interested in. With very little additional
detail provided to you, the executive merely says, “I want to see the click and open rates go
up.”
If this was the only information supplied and repeated queries to members of the
marketing team simply state the same end-goal of ‘increasing the clicking and opening rate’,
there seems to be a limitless number of avenues to pursue. Left to your own devices, do
you:
• Focus on content recommendation and craft custom emails for each user?
• Provide predictions with an NLP-backed system that will craft relevant subject lines for
each user?
• Attempt to predict a list of products most relevant to the customer base to put on sale
each day?
With so many options of varying complexity and approaches, with very little guidance,
the possibility of creating a solution that is aligned with the expectations of the executive is
highly unlikely.
If a proper planning discussion were had that goes into the correct amount of detail,
avoiding the complexity of the ML side of things, the true expectation might be revealed,
letting you know that the only thing that they are expecting is a prediction for each user for
when they would be most likely open to reading emails. They simply want to know when

©Manning Publications Co. To comment go to liveBook

 8

someone is most likely to not be at work, commuting, or sleeping so that they can send
batches of emails throughout the day to different cohorts of customers.
The sad reality is that many ML projects start off in this way. There is frequently very
little communication with regards to project initiation and the general expectation is that ‘the
DS team will figure it out’. However, without the proper guidance on what needs to be built,
how it needs to function, and what the end goal of the predictions is, the project is almost
doomed to failure.
After all, what would have happened if an entire content recommendation system were
built for that use case, with months of development and effort wasted when a very simple
analytics query based on IP geolocation was what was really needed? The project would not
only be cancelled, but there would likely be many questions from on-high as to why this
system was built and why the cost of development was so high.
If we were to look at a very simplified planning discussion, at an initial phase of
discussion, as shown in figure 1.4 below, we can see how just a few careful questions and
clear answers can give the one thing that every Data Scientist should be looking for in this
situation (of being the first DS at a company working on the first problem): a quick win.

©Manning Publications Co. To comment go to liveBook

 9

Figure 1.4 A simplified planning discussion to get to the root of what an internal customer (in this case, the
marketing executive who wants high open rates on their emails) actually needs for a solution.

©Manning Publications Co. To comment go to liveBook

 10

As figure 1.4 showed at the right (the DS internal monologue), the problem at hand here
is not at all in the list of original assumptions that were made. There is no talk of content of
the emails, relevancy to the subject line or the items in the email. It’s a simple analytical
query to figure out which time zone customers are in and to analyze historic opening in local
times for each customer. By taking a few minutes to plan and understand the use case fully,
weeks (if not months) of wasted effort, time, and money were saved. The arguably more
important result from lines of questioning such as this is that it gives both parties (the DS
team and the business sponsor) and understanding of what will be built and why it’s
being built. Notice the complete lack of how it will be built. The business sponsor (usually)
won’t care and doesn’t need to know those details.
We will be covering the processes of planning, having project expectation discussions
with internal business customers, and general communications about ML work with a non-
technical audience at length and in much greater depth throughout Chapter 2.

1.2.2 Scoping & Research

The focus of scoping and research needs to answer the two biggest questions that
internal customers (the business) have about the project.

FIRST QUESTION Is this going to solve my problem?

SECOND QUESTION How long is this going to take?

Let’s take a look at another potentially quite familiar scenario to discuss polar opposite
ways that this stage of ML project development can go awry. For this example, there are two
separate DS teams at a company (team ‘A’ in figure 1.5, team ‘B’ in figure 1.6), each being
pitted against one another for developing a solution to an escalating incidence of fraud being
conducted with the company’s billing system.
Team A’s research and scoping process is illustrated in figure 1.5 below.

©Manning Publications Co. To comment go to liveBook

 11

Figure 1.5 Research and scoping of a fraud detection problem for a junior team of well-intentioned but
inexperienced Data Scientists.

For team ‘A’, comprised of mostly junior Data Scientists, all of whom entered the workforce
without an extensive period in academia. Their actions, upon getting the details of the
project and what is expected of them, is to immediately go to blog posts. They search the
internet for ‘detecting payment fraud’ and ‘fraud algorithms’, finding hundreds of results
from consultancy companies, a few extremely high-level blog posts from similar junior Data
Scientists who have likely never put a model into production, and some open-source data
examples with very rudimentary examples.
Team B’s research and scoping, in contrast, is shown below in figure 1.6.

©Manning Publications Co. To comment go to liveBook

 12

Figure 1.6 Research and scoping for an academia-focused group of researchers for the fraud detection
problem.

Team ‘B’, on the other hand, is filled with a group of PhD academic researchers. With
their studious approach to research and vetting of ideas, their first actions are to dig into
published papers on the topic of fraud modeling. Spending several days reading through
journals and papers, they are now armed with a large collection of theory encompassing
some of the most cutting-edge research being done on detecting fraudulent activity.
If we were to ask either of these teams what the level of effort is to produce a solution,
we would get wildly divergent answers. Team ‘A’ would likely state that it would take about 2
weeks to build their XGBoost binary classification model (they mistakenly believe that they
already have the code, after all, from the blog post that they found).
Team ‘B’ would tell a vastly different tale. They would estimate that it would take several
months to implement, train, and evaluate the novel deep learning structure that they found

©Manning Publications Co. To comment go to liveBook

 13

in a highly regarded white paper whose proven accuracy for the research was significantly
better than any perforce implemented algorithm for this use case.
The problem here with scoping and research is that these two polar opposites would both
have their projects fail for two completely different reasons. Team ‘A’ would have a project
failure due to the fact that the solution to the problem is significantly more complex than
the example shown in the blog post (the class imbalance issue alone is too challenging of
a topic to effectively document in the short space of a blog post). Team ‘B’ would, even
though their solution would likely be extremely accurate, would never be allocated
resources to build the risky solution that they came up with as an initial fraud detection
service at the company (although it would be a great candidate for a version 2.0
implementation).
Project scoping for ML is incredibly challenging. Even for the most seasoned of ML
veterans, making a conjecture about how long a project will take, which approach is going to
be most successful, and the amount of resources that will need to be involved is a futile and
frustrating exercise. The risk associated with making erroneous claims is fairly high, but
there are means of structuring proper scoping and solution research that can help minimize
the chances of being wildly off on estimation.
Most companies have a mix of the types of people in the hyperbolic scenario above.
There are academics whose sole goal is to further the advancement of knowledge and
research into algorithms, paving the way for future discoveries from within industry. There
are also ‘Applications of ML’ Engineers who just want to use ML as a tool to solve a business
problem. It’s very important to embrace and balance both aspects of these philosophies
toward ML work, strike a compromise during the research and scoping phase of a project,
and know that the middle ground here is the best path to trod upon to ensure that a project
actually makes it to production.

1.2.3 Experimentation
In the experimentation phase, the largest causes of project failure is either due to the
experimentation taking too long (testing too many things or spending too long fine-tuning an
approach) or in an under-developed prototype that is so abysmally bad that the business
decides to move on to something else.
Let’s use a similar example from section 1.2.2 to illustrate how these two approaches
might play out at a company that is looking to build an image classifier for detecting
products on retail store shelves. The experimentation paths that the two groups take
(showing the extreme polar opposites of experimentation) are shown in figures 1.7 and 1.8
below.

©Manning Publications Co. To comment go to liveBook

 14

Figure 1.7 A rushed experimentation phase by a team of inexperienced Data Scientists.

Team ‘A’ in figure 1.7 is an exceedingly hyperbolic caricature of an exceptionally

inexperienced DS team, performing only the most cursory of research. Using the single
example blog post that they found regarding image classification tasks, they copy the
example code, use the exact pre-trained TensorFlow-Keras model that the blog used,
retrained the model on only a few hundred images of two of the many thousands of products
from their corpus of images, and demonstrated a fairly solid result in classification for the
holdout images from these two classes.
Due to the fact that they didn’t do thorough research, they were unable to understand
the limitations that were in the model that was chosen. With their rushed approach to

©Manning Publications Co. To comment go to liveBook

 15

creating a demo to show how good they could classify their own images, they chose a too-
simplistic test of only two classes. With cherry-picked results and a woefully inadequate
evaluation of the approach, this project would likely fail early in the development process (if
someone on their leadership team was checking in on their progress), or late into the final
delivery phases before production scheduling (when the business unit internal customer
could see just how badly the approach was performing). Either way, using this rushed and
lazy approach to testing of approaches will nearly always end in a project that is either
abandoned or cancelled.
Team ‘B’ and their approach to this problem is shown below in figure 1.8.

Figure 1.8 An overly thorough experimentation phase that effectively became the build-out of 3 separate
MVP’s for the project.

©Manning Publications Co. To comment go to liveBook

 16

Team ‘B’ in figure 1.8, on the other hand, is the polar opposite of team ‘A’. They’re an
example of the ‘pure researchers’; people who, even though they currently work for a
company, still behave as though they are conducting research in a University Their approach
to solving this problem is to spend weeks searching for an devouring cutting edge papers,
reading journals, and getting a solid understanding of the nuances of both the theory and
construction around various convolutional neural network (CNN) approaches that might work
best for this project. They’ve settled on 3 broad approaches, each consisting of several tests
that need to run and be evaluated against the entire collection of their training image
dataset.
It isn’t the depth of research that failed them in this case. The research was appropriate,
but the problem that they got themselves into was that they were trying too many things.
Varying the structure and depth of a custom-built CNN requires dozens (if not hundreds) of
iterations to ‘get right’ for the use case that they’re trying to solve. This is work that should
be scoped into the development stage of the project, when they have no other distractions
other than developing this single approach. Instead of doing an abbreviated adjudication of
the custom CNN, they decided to test out transfer learning of 3 large pre-trained CNN’s, as
well as building a Generative Adversarial Network (GAN) to get semi-supervised learning to
work on the extremely large corpus of classes that are needed to be classified.
Team B quite simply took on too much work for an experimentation phase. What they’re
left with at the point that they need to show demonstrations of their approaches is nothing
more than decision paralysis and a truly staggering cloud services GPU VM bill. With no real
conclusion on the best approach and such a large amount of money already spent on the
project, the chances that the entire project will be scrapped is incredibly high.
While not the leading cause of project failure, an effective experimentation phase can, if
done too incorrectly, stall or cancel an otherwise great project. There are patterns of
experimentation that have proven to work remarkably well for ML project work, though, the
details of which lie somewhere between the paths that these two teams took. We will be
covering this series of patterns and ways in which they can be adapted to any ML-based
project at length in the remainder of Part 1 of this book.

1.2.4 Development
While not precisely a major factor for getting a project cancelled directly, having a poor
development practice for ML projects can manifest itself in a multitude of ways that can
completely kill a project. It’s usually not as directly visible as some of the other leading
causes but having a fragile and poorly designed code base and poor development practices
can actually make a project harder to work on, easier to break in production, and far harder
to improve as time goes on.
For instance, let’s look at a rather simple and frequent modification situation that comes
up during the development of a modeling solution: changes to the feature engineering. In
figure 1.9 below, we see two Data Scientists attempting to make a set of changes in a
monolithic code base. In this development paradigm, all of the logic for the entire job is
written in a single notebook through scripted variable declarations and functions.

©Manning Publications Co. To comment go to liveBook

 17

Figure 1.9 Editing a monolithic code base (a script) for ML project work.

©Manning Publications Co. To comment go to liveBook

 18

Julie, in the monolithic code base, is likely going to have a lot of searching and scrolling
to do, finding each individual location where the feature vector is defined and adding her new
fields to collections. Her encoding work will need to be correct and carried throughout the
script in the correct places as well. It’s a daunting amount of work for any sufficiently
complex ML code base (where the number of lines of code for feature engineering and
modeling combined can reach to the thousands if developed in a scripting paradigm) and is
prone to frustrating errors in the form of omissions, typos, and other transcription errors.
Joe, meanwhile, has far fewer edits to do, but is still subject to the act of searching
through the long code base and relying on editing the hard-coded values correctly.
The real problem with the monolithic approach comes when they try to incorporate each
of their changes into a single copy of the script. As they both have mutual dependencies on
one another’s work, they will both have to update their code and select one of their copies to
serve as a ‘master’ for the project, copying in the changes from the other’s work. It will be a
long and arduous process, wasting precious development time and likely requiring a
great deal of debugging to get correct.
Figure 1.10 below shows a different approach to maintaining an ML project’s code base,
utilizing modularized code architecture to separate the tight coupling that is present within
the large script from figure 1.9.

©Manning Publications Co. To comment go to liveBook

 19

Figure 1.10 Updating of a modular ML code base to prevent rework and merge conflicts.

©Manning Publications Co. To comment go to liveBook

 20

Figure 1.10 shows a modularized code base written in an Integrated Development

Environment (IDE). While both changes that are being made by the two DS’s are identical in
their nature to those being made in figure 1.9 (Julie is adding a few fields to the feature
vector and updating encodings for these new fields, while Joe is updating the scaler used on
the feature vector), the amount of effort and time spent getting these changes working in
concert with one another is dramatically different.
With a fully modularized code base registered in git, the both of them can each check out
a feature branch from master, make their small edits to the modules that are part of their
features, write some new tests (if needed), run their tests, and submit a pull request. Once
their work is complete, due to the configuration-based code and the fact that the methods in
each of the modules classes can act upon the data for their project through leveraging the
job configuration, each of their feature branches will not impact one another’s and should
just work as designed. They can cut a release branch of both of their changes in a single
build, run a full integration test, and safely merge to master, confident in the fact that their
work is correct.
Writing code in this manner (modular design, written in an IDE) is a large departure for
many Data Scientists. We’ve learned in interactive notebooks, and many of us still use
notebooks quite frequently for prototyping ideas, experimentation, and for analysis of our
work (myself included). However, by adopting this alternate way of writing ML code (porting
prototype scripts and functions into object-oriented or functional programming paradigms),
projects can support many users developing new features for them simultaneously, as well
as ensuring that each new idea and bit of functionality is tested fully to eliminate difficult to
track down bugs. The overhead in time and effort associated with creating an ML code
framework based in these long-ago proven paradigms of software development will be
thoroughly worth it once even the 2nd change to the code base is needed to be done.
We will be covering all of these topics regarding development paradigms, methodology,
and tools in depth from Chapter 5 onwards throughout the book.

1.2.5 Deployment
Perhaps the most confusing and complex part of ML project work comes at the point long
after a powerfully accurate model is built. The path between the model creation and serving
of the predictions to a point that they can be used is nearly as difficult and its possible
implementations as varied as there are models to serve prediction needs.
Let’s take a company that provides analysis services to the fast food industry as an
example for this section. They’ve been fairly successful in serving predictions for inventory
management at region-level groupings for years, running large batch predictions for the per-
day demands of expected customer counts at a weekly level, submitting their forecasts as
bulk extracts each week.
The DS team up until this point has been accustomed to an ML architecture that
effectively looks like figure 1.11 below.

©Manning Publications Co. To comment go to liveBook

 21

Figure 1.11 The relatively simple scheduled batch internal-facing prediction serving architecture.

This relatively standard architecture for serving up scheduled batch predictions, shown
above in figure 1.11, solely focused on exposing inference results to internal analytics
personnel, isn’t particularly complex and is a paradigm that they are very familiar with. With
the scheduled synchronous nature of the design, as well as the large amounts of time
between subsequent retraining and inference, the general sophistication of their technology
stack doesn’t have to be particularly high (which is a good thing; see sidenote below).

©Manning Publications Co. To comment go to liveBook

 22

A brief note on simplistic architecture

In the world of ML, one should always strive to use the simplest design possible when building an architecture. If the
project requires a periodicity of inference of 1 week, then use a batch process (not real-time streaming). If the data
volumes are in the megabytes, then use a database and a simple VM (not a 25-node Spark cluster). If the runtime of
training is measured in minutes, stick to CPUs (not GPUs).
Using complex architecture, platforms, and technology simply for the sake of using them will create a condition that
you will inevitably regret, as it introduces unnecessary complexity to an already complex solution. With each new
complexity that is introduced, the chances rise that something is going to break (usually in a spectacularly complex
manner). Keeping the technology, the stack, and the architecture as simple as is needed to solve the imminent
business needs of the project is always a recommended best practice in order to deliver a consistent, reliable, and
effective solution to a business unit.

As the company has realized the benefits of predictive modeling over time, when a new
business segment opens up, the business unit approaches the DS team to build a new
prediction system for them. This new service is one requiring an approach to inventory
forecasting at a per-store level, with a requirement that the predictions respond in near-real-
time throughout the day. Realizing that they need to not only build a completely different
ensemble of models to solve this use case, the DS team focuses most of their time and
energy on the ML portion of the project. They didn’t realize that the serving component of
this solution would need to rely on not only a REST API to serve the data to individual store
owners through an application, but that they would have to be frequently updating the per-
store forecasts fairly frequently throughout the day.
After coming up with an architecture that supports the business need (months after the
start of the project, well after the modeling portion of the project had been finished), they
proceed to build it with the assistance of some Java software engineers. It wasn’t until after
the first week of going live that the business realized that the implementation’s in cloud
computing costs are more than an order of magnitude higher than the revenue they are
getting for the service. The new architecture that is needed to serve the business need is
shown in figure 1.12 below.

©Manning Publications Co. To comment go to liveBook

 23

Figure 1.12 The far more complex pseudo-real time serving architecture required to meet the business needs
for the project.

It doesn’t take long for the project to get cancelled and a complete redesign of the
architecture and modeling approach to be commissioned to keep the costs down.

©Manning Publications Co. To comment go to liveBook

Random documents with unrelated
content Scribd suggests to you:
 "The question is—does he care for her now?" Bertha
went on.

"Bertha, I almost think he does."

"You don't imagine that he has taken to somebody

else?"

"No."

Bertha saw that a certain conjecture in her own mind

had not so much as penetrated into Lettice's thoughts; the
idea of Mr. Kelly in connection with herself was non-
existent.

"No," she repeated; "certainly nobody that I have come

across. I don't know most of Mr. Kelly's friends, of course.
But now that I think of it, I have noticed how he talks of
Prue. I thought it was because she is such a friend of mine,
only—"

"Men are not so accommodating. They don't understand

the force of female friendships."

"But if he does care, why should he not speak?"

"Ah, there it is! He may not be sure of his own mind—

he may still less be sure of Prue's mind. That self-restrained
manner of hers is not easy to read. At all events, you know
now how things stand."

"I wish he would! It would be only too delightful!"

"Well, you and I, of course, can do nothing. He must

just please himself. Perhaps I feel a little more hopeful,
after speaking to you—and I know it is safe! You will never
breathe a word to anybody—least of all to Prue? . . . Tell me
now all about yourself. I must not stay many more
minutes."

After Bertha's departure, Lettice stood again, gazing out

of the window, busied in consideration. She did not quite
see why Bertha had said so much. It seemed that the
conversation could lead to no particular result. As Bertha
had truly remarked, they could take absolutely no steps in
the matter. If Mr. Kelly did not come forward of his own free
will, no living person had power to induce him to do so.

"And, after all, it may be a mere fancy," decided Lettice.

"Mr. Kelly may have forgotten his old liking; and Prue may
not really care any longer. Prue always seems contented."

Then, to her surprise, she saw Mr. Kelly himself in the

street, apparently steering a straight course for the front
door—Mr. Kelly, with downward-bent head and intent
visage, evidently much occupied with some subject
mentally viewed. Lettice did not open the front door this
time.

"How odd that he should come now, just after Bertha's

call! I have a great mind to make him talk about Prue, just
for the sake of watching how he does it . . . Did Bertha
think that perhaps I was getting to like him a little too
well?" This idea flashed up unbidden, and Lettice burst into
a soft fit of laughter.

"O how absurd! Then she really came in Prue's interest.

Dear Prue! Why, he is old enough to be my father!"

"All alone, Miss Anderson?" Mr. Kelly surprised the

laugh, only half completed; and he was not sure whether to
be disconcerted, since it might be that she was laughing at
him. "You seem very cheerful," he hazarded.
 "O yes: I am as cheerful as possible," Lettice answered,
composing her face with all speed. "Please sit down. I have
a good part of the day alone, of course. And sometimes my
own thoughts amuse me."

"Your mind to you, in fact, a kingdom is."

"Not always, I am afraid."

Mr. Kelly sank into profound silence. This was not his
usual mode; and Lettice became speedily aware that he had
something on his mind. He had been very friendly and
pleasant of late, and she enjoyed a call from him; but the
abstraction to-day became somewhat heavy. Lettice tried to
get up a conversation, and there was no response. She
spoke of Prue, and he only said, "Yes."

"I don't think Prue is looking at all well?" hazarded

Lettice.

"No? Indeed?"

"For a long time past. Ever since I came."

"Ah, yes!"

"And Bertha thinks the same. Bertha has just been to

see me."

"Yes, indeed," murmured Mr. Kelly.

Lettice gave up, and imitated his silence. For three

minutes, at least, the noisy little clock had things his own
way, undisturbed.

"Dear me, I'm afraid this is very unsociable," said Mr.

Kelly at length. "I had no intention—I am sure—"
 "I am afraid something must have gone wrong in the
Parish."

"No, thanks; nothing at all. Nothing in connection with

the Parish."

"Then, is it anything you would like me to do? Any work

—?"

Another prolonged pause.

"Miss Anderson, you once came to me years ago, to ask

a question—was it not to ask advice?" His memory was
slightly at fault, but Lettice made no attempt to set it right.
"I am come to-day, on something of the same errand—that
is to say—to ask a question, for a particular reason."

"My advice would not be worth much."

"That may depend upon certain circumstances. If, for

instance, you had better means of judging—"

Did he mean—Prue? The thought whirled through

Lettice, producing an inward turmoil. Could it be? Yet, why
should he come to her?

"Better means of judging—" reflectively repeated Mr.

Kelly.

"If I have. Please ask me anything you like!" Lettice's

cheeks were bright.

"It is, of course, in confidence. I may depend upon you

—should there be no result."

"Quite!"
 "Then, could you tell me this? Is there the slightest
hope, that—say, under any circumstances—Miss Valentine
might marry?"

"I don't see why not!"

"I imagined—that perhaps—she seems so entirely the

home-daughter—"

"If it were for her happiness, how could they not be

glad?" asked Lettice. "And they all love Prue so much!
Everybody does, who knows her."

"As you know her!"

"Yes. I have never seen any one else like her."

A smile stole over Mr. Kelly's serious face, and vanished.

"She was once, I believe, engaged."

"Was she?"

"It was years ago, long before I first saw you. I knew
her then, well. In fact—though I had not meant to reveal
any personal interest in this question—I do not mind saying
that she made a very strong impression on me then. But I
was told that she was engaged; and I at once left the
place."

"Fled! Without making sure if the report were true!"

"Perhaps I acted too precipitately. My information

seemed reliable."

"It was not!" Lettice spoke decisively. "You were told

what was untrue."
 Mr. Kelly showed some slight agitation. "And all these
years since—"

"All these years since, she has never been engaged." It

seemed very strange to Lettice that Bertha's visit should
have been just in time to prepare her for this interview. She
could not but speak out, however, having the requisite
knowledge.

"You are sure? It was a long while ago."

"I am sure: because I have heard particulars. Prue was

not engaged at that time; and she has never been engaged
since. I know so much, positively. There cannot be any
mistake."

Mr. Kelly kept profound silence, and Lettice's heart beat

fast. Would he ask any more? Had she said enough?

"Thanks!" came at length, and he stood up. "I must not

pay a longer visit to-day."

"And this is all you wanted to know?"

"Yes, thanks. Of course you will not repeat this—this

little talk of ours?"

"No, indeed. And I suppose I must not ask you one

question in return. Would it be wrong? That impression—the
impression Prue made upon you, all those years ago—I
wonder if it has gone quite off? Do people change?"

"Some do!"

"I could not, if once I cared for anybody very very

much."
 "Nor I!" said Mr. Kelly.

She lifted her face to his, and said markedly—"Nor

Prue!"

The light that broke over his face was very singular.
Lettice had never seen anything exactly like it.

"Thank you very much," he said at length. "You have

given me courage."

Then he was gone; and Lettice saw him striding off, at

"double-quick-march" in the direction of the Valentines'
house, that house which they had only reentered two days
before. He would find things in some confusion; but what
did that matter.

"I'm so glad I could say a word. If only he will come to

the point!" murmured Lettice.

Three hours later, a note was left at the door, and

Lettice read:

"DEAR LETTICE,—You must be the first to

hear. Mr. Kelly has asked me to be his wife, and
I have consented. He says it is your doing.
Thanks, dear little 'outside sister.'—

"Ever your loving,

"PRUE."

"Lettice, I say, here's a telegram for you."

 Felix had overtaken the telegraph boy at the door, on
arrival at the close of his day's work. Lettice looked up with
dreamy eyes.

"A what! O Felix, I have such news! Guess—good news!

What do you think is going to happen?"

"Can't tell."

"Mr. Kelly and Prue are engaged to be married."

"Time they should, if they ever mean to do it. I thought

they would have dangled on to the end of the chapter.
When did he speak?"

"Only to-day. Prue wrote at once to tell me. I am so

delighted. Nothing could be nicer. And she will have a home
so near to them all. But what do you mean? Did you expect
it? How came you to think of such a thing?"

"How could anybody avoid seeing it? Here, what is

this?"

"A telegram!" Lettice tore open the envelope and lifted

scared eyes to her brother.

"Don't stare! What is it? Anything wrong?"

"Felix, I must go!"

"Go where?"

"Keith is dying." She sat down, trembling.

He looked over and read aloud. "Come at once; late

train. Keith dying, wants to see you. Bring companion if
needful. Will pay all expenses. No delay. Maurice Bryant."
 "I must go directly. The very first train. Poor little
Keith."

"Now, Lettice, be sensible! Don't upset yourself. It's no

manner of use, and you'll only lose time. I will look-out
trains, while you run upstairs and put your things together.
You may have to stay a few days. There's a train, I fancy,
about nine or ten. The boy is waiting, and a reply has been
pre-paid. I'll say we are both coming. Yes, I shall go too.
You can't travel alone at midnight."

"How good you are."

"I don't see much goodness in the question. There's

nothing else to be done. That woman is having her deserts."

Lettice gave him a look, and rushed away. When she

came back, he had settled everything, and was seated in
the arm-chair.

"Supper will be up directly, but there is plenty of time. I

declare it would serve the Bryants right if you did not go."

"I couldn't do such a thing."

"No; I dare say not. Only, it would serve them right."

"Felix, do you think you ought to come?"

"Why 'ought'?"

"If you feel so—"

"What! You think I might give Mrs. Bryant a piece of my

mind?"

"You could not. You couldn't say anything unkind, when

she is in trouble. And Felix, dear, you would not. I am sure
of that. If you knew how she cares for that boy. It will
almost kill her to lose him. O no; I am not afraid."

"I don't think you need be,—" in rather an odd voice.

"But I do wonder what is the matter. I do wonder whether
he may not, after all, get well!"

"Telegrams don't give very abundant information. You'll

like to leave a message for the Valentines?"

"I had written to Prue, before you came in. I'll just add
a word of postscript . . . What a day this has been. I am so
glad for Prue! But oh, that poor little Keith! Felix, if you
knew how loving he was to me before I came away: and
how he said he would miss me. Oh, I hope we shall be in
time!"

"Here comes supper. Now, mind, if you don't eat, you

don't go!" Felix spoke with a determined air of authority.
 CHAPTER XIX.
RETRIBUTION.

THE next few hours were to Lettice one prolonged whirl.

She did what had to be done, with some measure of
outward quietness; but her mind was in a dizzy tumult of
feeling. After all that had passed, it seemed such a fearful
reverse for Theodosia! Lettice forgot injuries to herself, and
her whole heart went out in passionate pity to that unhappy
woman, who in losing Keith might almost be said to lose her
all.

Felix said little during the journey, but he saw to

everything, looked well after Lettice, and made her lie down
on the seat, while nobody except themselves occupied the
compartment. She found herself involuntarily tracing and
retracing the course of late events. Things had turned out
utterly unlike all previous expectations. But for Theodosia's
determined antagonism, she might not have come to
London at this time, and she might never have lived again
with Felix! This new phase of affairs had sprung from a
chain of circumstances over which she had small control;
not one of which, separately viewed, could have been
deemed likely to produce any such result.

"I suppose things often are so," she meditated, reposing

in the lamplight, with the rush of the train in her ears. "And
after all, it has been worth going through the trouble, to
bring about my home with Felix. Yes; worth the whole. I
would not undo a single step, if it must mean undoing that
too . . . I never fancied Felix could be what he is now to me.
Three months ago, if I could have looked ahead, how little I
should have cared about Mrs. Bryant's unkindness . . . At
least it would have been a different feeling altogether—not
desolate! I do care now: but I shall be cleared some day,
quite cleared. I am sure of it. And meantime life is so
happy! How can I help being contented? . . . That poor little
Keith? Ought I to be glad about anything, when he is
dying?"

So the hours passed, in alternations of feeling; and

after midnight, the journey ended, in a dark drive to
Quarrington Cottage.

Lettice sprang out, to be met at the door by Dr. Bryant.

He looked grave and tried; but at the first glimpse of

Lettice, something of the old light came into his eyes. "My
child, I have missed you!" he said.

"Have you? I did wonder if, perhaps, you might!"

"Hardly possible that I should not!"

"And Keith?"

"You are in time. Not much more than in time, I fear.

Your brother was right to come too," as he held out a hand
to Felix.

"I could not let Lettice travel alone at night."

"No: you are perfectly right."

"Uncle, what is the illness?"

"A chill first. Bronchitis, with complications. He cannot

last many hours now. At this moment he is not conscious."

"Will he know me?"

 "I hope so. He has asked for you incessantly, poor little
lad. Theodosia would not consent, until yesterday
afternoon, to have you sent for. I feared then that it might
be too late. The boy was so distressed that she had to give
way."

Lettice shivered at the thought of Theodosia; not with

reference to herself. "It is terrible for Mrs. Bryant!" she
murmured.

"Yes; and the boy has seemed to turn from her in this
illness. It has been most painful. My dear, will you take off
your wraps; and then you must have supper. After that you
shall see him."

"O let me go to him first. We had supper at home."

"You must take something more now."

She had no choice as to submission, and obedience was

the quicker mode. Fatigue could not be thought of; and she
had no desire for sleep. Felix at least was glad to avail
himself of the food provided; and Lettice did her best to
satisfy Dr. Bryant.

"You are looking well in health. Not the worse for

London!" he said, after an earnest scrutiny.

"Felix takes such care of me."

Felix grunted slightly, as if in token that he desired no

compliments.

"Anderson will be no loser in the end."

Then came a summons. "Master Keith was sensible, and

he wanted Miss Lettice. He had asked if she were in the
house."

"Come, both of you," the doctor said unexpectedly. He

could hardly have explained why he extended his invitation
to Felix. The words sprang from a momentary impulse; and
Felix followed without hesitation.

Keith was sitting up in his little bed, supported by

pillows. Beside him stood Theodosia; haggard, wretched
beyond description. After one glance, as Lettice entered,
her eyes were averted: while the boy extended eager
hands, and gasped, "Dear old Lettice! Come!"

She bent over him, and he held her with his thin arms,
until they dropped through weakness. The breathing was
sorely oppressed. He seemed striving to say something, and
unable to bring it out.

Theodosia drew nearer; but with one hand, he tried to

push her away—a hasty childish motion.

"No, no, Keith!" the doctor said in a gentle voice.

"I can't! Lettice! Not Mamsie! It was she—did it!" The

boy could with difficulty speak, and he looked towards Dr.
Bryant, panting. "I want—want—to tell you! . . . Mamsie did
it! . . . I saw her! . . . Lettice knows! . . . Ask Lettice!"

"What does he mean?" asked Dr. Bryant, in a low voice.

Lettice shook her head. How could she reply?

"Lettice knows!" repeated the boy brokenly. "Hold me,

Lettice! Don't go! . . . She put the bank-note . . . her own
self . . . into Lettice's box! . . . I saw her . . . Mamsie knows
quite well . . . And Lettice knows!"
 Theodosia's face grew stony; and Dr. Bryant's changed
to a grey whiteness.

"Who put the bank-note into Lettice's box, Keith?"

"Mamsie! Her own self! I saw her! . . . I can't think—

how—she could!"

An oppressive silence followed, broken only by the boy's

gasps.

Lettice was the first to speak.

"Keith, darling, don't think of all that now. Try to forget

about the money. You have not to do with it, you know.
Think of the stories I used to tell you, up in the playroom on
Sunday afternoon. About JESUS, dying on the Cross, and
how He loved the children always."

"Yes, I know! Will He take me—right away—up there?"

"I am sure He will. He loves you so; and He died for

you, Keith. Ask Him now to forgive you all that you have
done wrong; and to take care of you."

"I can't speak—it hurts! You ask?"

There was an instant's shy reluctance, but at such a

moment she could not think of self.

Others simply stood around, waiting. Lettice looked at

none of them. She knew that Theodosia, Dr. Bryant, Felix,
Susanna, were present; yet as she knelt, with bowed head,
clasping the boy's hand, she had a sense of being alone
with Keith before the Heavenly Throne.
 In soft tones she uttered the words of her own old
evening prayer, learnt long ago from Cecilia, and taught
later to Keith, only changing the pronouns to suit the needs
of the present occasion, and bringing in Keith's name. He
joined brokenly in the familiar sentences: and then a fit of
choking came on. Theodosia interfered to give help: her
own face ghastly. When the attack passed off, it left the boy
unconscious.

"Is this true?" the doctor asked, in a voice of

suppressed pain.

He had no immediate answer. Theodosia's gaze was

riveted on her boy's face, over which a marked change was
creeping. It had come without warning: and it deepened
fast. Again he struggled for breath: and once more, with
returning sense, his eyes sought Lettice's face. A smile
glimmered: and one word passed his lips, that word of
"hope for the dying." He said distinctly—"JESUS!"

"Keith, darling, look at mamsie! Give poor mamsie one

smile."

He made an effort to obey: but in the act, his eyes

closed.

There was another slight struggle: and the child was

gone.

Silence again fell upon them all; and Lettice trembled,

holding the small lifeless hand. Theodosia stood up white
and stern.

"Yes: it is true," she said, "I did it! I wanted to get rid of
Lettice,—for Keith's sake! I could not have done such a
thing for anybody else's sake,—only for Keith! . . . Only for
Keith! . . . I know what I am saying! Don't stare at me—all
of you! I tell you—Lettice stole my boy's heart from me!
And she would have robbed him—robbed him of his rights! I
saw it all—and so—for his sake—my boy's sake—"

"Come out of the room, Theodosia!" Dr. Bryant spoke

with infinite pity.

"Why should I go? Nothing is left to me now. Oh, I know

—it was wrong, of course—but this is an awful punishment!"
And she sank over the child's body, uttering a wail of such
misery, that Lettice burst into weeping at the sound. Who
could comfort the unhappy woman? She had sacrificed her
truth, her sense of right, deliberately wronging an innocent
girl, as a gratification of her own jealousy, and for the
supposed advantage of the single being whom she loved:
and now, not only was he taken from her, but before he
went, she had the additional anguish of knowing that she
had forfeited his esteem, if not also his love.

There was a bitter irony in the fact that her own idolised
boy should have been the one, of all others, to make known
her wrong-doing. Either, his resolution not to speak of it had
broken down under the weakness of suffering, or his
childish conscience had refused to let him pass away
without clearing Lettice from unjust accusation. Whichever
way it might be, Theodosia's cup was thereby filled to the
brim.

"Go away! Go and leave me!" she cried. "I want Keith!
Nobody but Keith! Lettice may have all your money now.
Now Keith is gone."

To Lettice the scene was heartrending: to Felix it was a

revelation. If the love of money could lead to this—love of
money, not for self even, but for another—who might count
himself free from peril! He said nothing; but that which he
saw sank deeply, and was not forgotten. Theodosia's
jealousy of her boy's affection for Lettice was an additional
motive which he could not so well see or appreciate. One
side of the matter was clear enough: and it carried its own
lesson.

"Take Lettice into another room," Dr. Bryant said quietly,

and Felix obeyed. Half-an-hour later Dr. Bryant appeared
alone, entering the study where the brother and sister had
taken refuge. Lettice, not a little shaken by the past scene,
was sobbing still: but at the sound of his step, she stood up.

"Is Mrs. Bryant—!" she tried to say.

Dr. Bryant was still ashen pale, with the look of a man
who has received a severe blow. He came in front of Lettice,
and said, "My child, forgive me!"

Lettice clung to him, without a word.

"Forgive me!" he repeated. "You have been cruelly

wronged."

"You couldn't know! You couldn't tell. It was not your

fault. I have nothing to forgive."

"You have much! I ought to have been sure! The marvel

to me now is—how I could ever have thought it possible. My
own credulity amazes me."

His grieved look wandered to the brother, and Felix said

promptly, "The other would have seemed much more
impossible."

"True!" Dr. Bryant sighed heavily. "The least of the pain

is that our little lad is safe at Home!"
 "Don't mind about me! Don't think of that again,"
entreated Lettice. "You have been so good, so dear and
good to me always. And if I had spoken out at first—"

"That was always a mystery! I never could understand

your silence. Why not have denied it from the beginning?"

Lettice was dumb, and Felix spoke for her. "She thought
she might divert suspicion upon me. I was in the room,
also, with the bank-note."

"I see!—" with a flash of understanding. "But, no! How

could that affect you? You were not in the room alone?"

"Yes,—alone!"

"Ah!" The sound was almost a groan.

"It is all right now—I mean, you will never doubt me

again—and I shall always be your child!" murmured Lettice,
longing to comfort him.

His lips touched her brow, as he echoed,—

"Always my child!"

"Is she any better, uncle?"

A negative mournful movement came in answer.

"She has insisted on being alone for a time. After all—

what can one do? What can any one say to comfort her? . .
. Now you must both go to bed? And do not hurry up in the
morning, either of you. Breakfast can be made ready at any
time."

"And you, uncle?"

 "I must have a little time to myself. Good-night, dear
child. Try to sleep . . . There is need to sorrow more for the
living than the dead; and that touches me more closely
than you."

"I think—what touches you touches me!" she tried to

say.

"Thanks, my comfort!" and with a stirred face he was

gone, able to bear no more.

"I am glad I know him," Felix observed abruptly.

"There's something noble about that man—something
unlike other men!"

Four days later, Keith was laid to rest in the little village
churchyard; and some who knew Theodosia well, said
plainly that it was a merciful stroke which had taken the boy
thus early away from her influence. Dr. Bryant, whatever he
might have felt, passed no such judgment. He uttered no
reproaches, and showed to his wife only a steadfast
compassion.

She bore up sullenly till after the funeral, only keeping

to her own room, and refusing to see Lettice. Then she
broke down, and for many weeks she was laid low with
brain fever. Out of this illness, she emerged a permanent
invalid, shattered in body and mind, childishly sorry for her
past conduct, so far as she was capable of recalling it, yet
apt to be amused with the veriest trifles.

Lettice could be no help to Dr. Bryant, so far as

Theodosia was concerned: since her presence proved
always in a measure harmful, by exciting more vivid
recollections. A good nurse was in constant attendance: and
Dr. Bryant devoted himself, with patient and forgiving
assiduity, to lightening, so far as was in his power, the
burden of his wife's existence. Hers truly was a spoilt life. In
less than a year, she was laid beside her boy.

Dr. Bryant had been thoroughly shaken out of his love

for Quarrington Cottage by these painful events. Without
consulting anybody, he put the place into the hands of
agents, to be let or sold: and six weeks after his wife's
death, he betook himself to a London hotel, within five
minutes' walk of the Andersons' lodgings.

Nobody expected him. Lettice had been greatly

exercised of late, wondering whether her first duty lay with
Felix or with Dr. Bryant, now that the latter was left alone in
the world.

"If only I could live with them both!" she said often to
herself. But Felix was tied to the neighbourhood of London:
and that Dr. Bryant should be willing to quit his old home
was a notion which never so much as occurred to her
imagination. Everybody looked upon him as a fixture there.

The day on which he arrived happened to be the day of

Prue Valentine's wedding: a quiet affair, with only Bertha
and Nan, Lettice and one other girl, for bride's-maids. The
ceremony took place in Mr. Kelly's own Church.

Prue looked calmly happy, in her neat white dress: not

only happier but younger than for years past. The bride's-
maids wore mauve and white, in consideration of Lettice's
mourning, which was of course slight in degree.

It was a morning of much excitement for Lettice, who

had never been a bride's-maid before: and for about the
first time during six weeks past, her mind did not dwell
continuously on the thought of Dr. Bryant in his solitude.
 Coming out of Church, after the happy pair, a vague
sense took possession of Lettice that somebody familiar and
unexpected was present. She could not at once give shape
to the notion, and she did not exactly see the "somebody."
It was rather a dim consciousness that such an one might
be seen—and her eyes roved about anxiously. Had she
caught a glimpse, or had she not—almost without knowing?
An answer came in the porch, when a kind finely-outlined
face, framed in grey hair, was suddenly close at hand; and
then she was aware that she had before detected him in
Church, though her brain had not fully translated what her
eyes had seen.

"Uncle Maurice!" burst from her lips.

"Presently!" he said, as his hand clasped hers. "Go and

enjoy yourself, my child. I shall see you by-and-by."

The next two hours were passed in pleasurable

suspense. When at length bride and bridegroom had taken
their departure, Lettice was free to go home,—and there,
according to hope, she found Dr. Bryant comfortably
established.

"So this is where you live!" he said.

"It isn't a palace, uncle. But big rooms don't make

happiness."

"That's an aphorism worthy of a copybook. Wedding

gone off all right?"

"O yes. But how did you know?"

"I called here, and learnt where you were gone."

"Didn't you think Prue looked nice?"

 "I thought somebody else did . . . Lettice, are you and
Felix wedded to this neighbourhood?"

"To London? Felix has his work."

"Business men often live outside London: within easy

distance."

"But the expense of going in and out every day! Of

course I should like it, and so would Felix: only it can't be
thought of. You see, Felix ought to lay by, if only a little,
every year, and I cost him a great deal. But I do try to be
economical, and to save in every possible way. And Felix is
so glad now to have me with him."

"My dear, I want you also."

"If only Quarrington Cottage were not all that great

distance—!" Lettice spoke distressfully.

"I am not going back to Quarrington Cottage. It holds

too many sad memories. The place is to be sold. I think of
finding a new home outside London,—perhaps in the
direction of Reigate or Dorking. And I must have you to live
with me."

"But—Felix!"

"Felix too! I would not on any account separate you. I

want you both to make my home your home. Why should
you not? All that I have will belong to you by-and-by: and
my will is already made, to that effect. No,—I shall leave
something to Felix—" as she uttered her brother's name
—"but the bulk will be yours. Felix can make his own way:
and he will make it the faster, if by this plan, he is able to
lay by present earnings. I should undertake all his current
expenses, as with a son of my own. Will Felix object?—And
would you mind?"

"Mind! If Felix consents, I shall be only too glad!! To be

with you always:—and to have Felix:—and to live in the
country! I didn't know life could be so lovely!" cried Lettice,
enraptured.

No difficulties arose in the path of the plan. Felix was

both touched and grateful.

"Nonsense!" he said to Lettice, when she expressed

privately her regrets as to Dr. Bryant's intentions for the
future, which he had plainly stated to the young man. "Of
course he is free to leave his money where he likes. Neither
of us has the smallest claim upon him, but you deserve it, if
anybody does. He is uncommonly good to me, I can tell
you: and this will be a capital lift. To have a home provided,
and expenses paid, just when I am making my way! Don't
you see what a lot I shall be able to save?"

"I hope we shall not both get to care too much for
money," Lettice said seriously.

"Not much danger for you. That isn't your sort. And
mind—if you think I'm getting into an avaricious groove—
just speak and warn me, Lettice."

Quarrington Cottage was sold, and a pretty home was

chosen within forty minutes of the City by rail. There the
brother and sister lived with Dr. Bryant, finding in him a
true father. Neither of the two had after cause to regret the
arrangement, and Dr. Bryant's old age was the happiest
period of his life.

Although the desire to "get on" was no longer the ruling

passion of Felix Anderson's mind, to the exclusion of higher
aims, he did get on, and gave promise in time of becoming
a thoroughly prosperous man. But in after years, he never
lost sight of one main fact, that what he possessed was
literally not his own, but only "held in trust."

THE END.

Printed by BALLANTYNE, HANSON & CO.

Edinburgh & London

THE PILGRIM SERIES.

With Illustrations. Crown 8vo. 2s. With gilt edges, 2s. 6d.

SUITABLE FOR PRIZES AND REWARDS.

——————————————————
 1. BUNYAN'S PILGRIM'S PROGRESS.
2. BUNYAN'S HOLY WAR.
3. FOXE'S BOOK OF MARTYRS.
4. BEN-HUR. By LEW WALLACE.
5. THE LAMPLIGHTER. By MARIA S. CUMMINS.
6. UNCLE TOM'S CABIN. By H. B. STOWE.
7. ROBINSON CRUSOE. By DANIEL DEFOE.
8. MY DESIRE. By SUSAN WARNER.
9. NOBODY. By SUSAN WARNER.
10. THE FAIRCHILD FAMILY. By Mrs. SHERWOOD.
11. THE SWISS FAMILY ROBINSON.
12. ROMANCE OF NATL. HISTORY. By P. H. GOSSE.
13. GREAT MEN. By the late Rev. F. MYERS.
14. LITTLE WOMEN AND GOOD WIVES. By L. M.
ALCOTT.
15. DRAYTON HALL. By JULIA MATHEWS.
16. THE END OF A COIL. By SUSAN WARNER.
17. GLEN LUNA. By ANNA WARNER.
18. DIANA. By SUSAN WARNER.
19. STEPHEN, M.D. By SUSAN WARNER.
20. MELBOURNE HOUSE. By SUSAN WARNER.
21. BIBLE WARNINGS. By the Rev. Dr. NEWTON.
22. THE PHYSICIAN'S DAUGHTERS. By LUCY NELSON.
23. THE WIDE, WIDE WORLD. By E. WETHERELL.
24. DAISY. By SUSAN WARNER.
25. DAISY IN THE FIELD. By SUSAN WARNER.
26. NOR'ARD OF THE DOGGER. By E. J. MATHER.
27. SOLDIERS AND SERVANTS OF CHRIST. By ANNA
LEHRER.
28. QUEECHY. By SUSAN WARNER.
29. DARE TO DO RIGHT. By JULIA MATHEWS.
30. NETTIE'S MISSION. By JULIA MATHEWS.
31. YOKED TOGETHER. By ELLEN DAVIS.
32. OPENING A CHESTNUT BURR. By E. P. ROE.
33. ST. ELMO. By A. J. E. WILSON.
34. NAOMI. By Mrs. J. B. WEBB.
35. BARRIERS BURNED AWAY. By E. P. ROE.
36. WYCH HAZEL. By S. and A. WARNER.
37. THE GOLD OF CHICKAREE. By S. WARNER.
38. THE OLD HELMET. By SUSAN WARNER.
39. A LETTER OF CREDIT. By SUSAN WARNER.
40. GENERAL GORDON. By Major SETON CHURCHILL.
41. A KNIGHT OF THE NINETEENTH CENTURY. By E. P.
ROE.
42. IN THE DAYS OF BRUCE. By GRACE AGUILAR.
43. HOME INFLUENCE. By GRACE AGUILAR.
44. A MOTHER'S RECOMPENSE. By GRACE AGUILAR.
45. THE VALE OF CEDARS. By GRACE AGUILAR.
46. THE GOLDEN LADDER. By SUSAN and ANNA
WARNER.
47. INFELICE. By A. J. E. WILSON.
48. AT THE MERCY OF TIBERIUS. By A. J. E. WILSON.
49. DORRINCOURT. By the Author of "Expelled."
50. WESTWARD HO! By CHARLES KINGSLEY.
51. HOLDEN WITH THE CORDS. By W. L. M. JAY.
52. A RED WALLFLOWER. By SUSAN WARNER.
53. JOHN HALIFAX, GENTLEMAN. By Mrs. CRAIK.
54. ARABIAN NIGHTS' ENTERTAINMENTS.
55. CRANFORD. By Mrs. GASKELL.
56. WAVERLEY. By Sir WALTER SCOTT.
57. HYPATIA. By CHARLES KINGSLEY.
58. IT IS NEVER TOO LATE TO MEND. By CHARLES
READE.
59. ADAM BEDE. By GEORGE ELIOT.
60. WAGES. By L. T. MEADE.
61. BETWIXT TWO FIRES. By J. JACKSON WRAY.

——————————————————

JAMES NISBET & CO., LTD., 21, BERNERS STREET, W.

 *** END OF THE PROJECT GUTENBERG EBOOK THE ANDERSONS
***

Updated editions will replace the previous one—the old editions will
be renamed.

Creating the works from print editions not protected by U.S.

copyright law means that no one owns a United States copyright in
these works, so the Foundation (and you!) can copy and distribute it
in the United States without permission and without paying
copyright royalties. Special rules, set forth in the General Terms of
Use part of this license, apply to copying and distributing Project
Gutenberg™ electronic works to protect the PROJECT GUTENBERG™
concept and trademark. Project Gutenberg is a registered trademark,
and may not be used if you charge for an eBook, except by following
the terms of the trademark license, including paying royalties for use
of the Project Gutenberg trademark. If you do not charge anything
for copies of this eBook, complying with the trademark license is
very easy. You may use this eBook for nearly any purpose such as
creation of derivative works, reports, performances and research.
Project Gutenberg eBooks may be modified and printed and given
away—you may do practically ANYTHING in the United States with
eBooks not protected by U.S. copyright law. Redistribution is subject
to the trademark license, especially commercial redistribution.

START: FULL LICENSE

THE FULL PROJECT GUTENBERG LICENSE
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

textbookfull.com