ML Projects For Final Year

The goal of this document is to provide a common framework for approaching machine learning
projects that can be referenced by practitioners. If you build ML models, this post is for you. If
you collaborate with people who build ML models, I hope that this guide provides you with a
good perspective on the common project workflow. Knowledge of machine learning is assumed.

Overview
This overview intends to serve as a project "checklist" for machine learning practitioners.
Subsequent sections will provide more detail.

Project lifecycle
Machine learning projects are highly iterative; as you progress through the ML lifecycle, you’ll
find yourself iterating on a section until reaching a satisfactory level of performance, then
proceeding forward to the next task (which may be circling back to an even earlier step).
Moreover, a project isn’t complete after you ship the first version; you get feedback from real-
world interactions and redefine the goals for the next iteration of deployment.

1. Planning and project setup

• Define the task and scope out requirements

• Determine project feasibility

• Discuss general model tradeoffs (accuracy vs speed)

• Set up project codebase

2. Data collection and labeling

• Define ground truth (create labeling documentation)

• Build data ingestion pipeline

• Validate quality of data

• Label data and ensure ground truth is well-definend

• Revisit Step 1 and ensure data is sufficient for the task

3. Model exploration
• Establish baselines for model performance

• Start with a simple model using initial data pipeline

• Overfit simple model to training data

• Stay nimble and try many parallel (isolated) ideas during early stages

• Find SoTA model for your problem domain (if available) and reproduce results, then apply to
your dataset as a second baseline

• Revisit Step 1 and ensure feasibility

• Revisit Step 2 and ensure data quality is sufficient

4. Model refinement
• Perform model-specific optimizations (ie. hyperparameter tuning)

• Iteratively debug model as complexity is added

• Perform error analysis to uncover common failure modes

• Revisit Step 2 for targeted data collection and labeling of observed failure modes

5. Testing and evaluation

• Evaluate model on test distribution; understand differences between train and test set
distributions (how is “data in the wild” different than what you trained on)

• Revisit model evaluation metric; ensure that this metric drives desirable downstream user
behavior

• Write tests for:

• Input data pipeline

• Model inference functionality

• Model inference performance on validation data

• Explicit scenarios expected in production (model is evaluated on a curated set of observations)

6. Model deployment
• Expose model via a REST API
• Deploy new model to small subset of users to ensure everything goes smoothly, then roll out to
all users

• Maintain the ability to roll back model to previous versions

• Monitor live data and model prediction distributions

7. Ongoing model maintenance

• Understand that changes can affect the system in unexpected ways

• Periodically retrain model to prevent model staleness

• If there is a transfer in model ownership, educate the new team

Team roles
A typical team is composed of:

• data engineer (builds the data ingestion pipelines)

• machine learning engineer (train and iterate models to perform the task)
• software engineer (aids with integrating machine learning model with the rest of the product)
• project manager (main point of contact with the client)
•

Planning and project setup

It may be tempting to skip this section and dive right in to "just see what the models can do".
Don't skip this section. All too often, you'll end up wasting time by delaying discussions
surrounding the project goals and model evaluation criteria. Everyone should be working
toward a common goal from the start of the project.

It's worth noting that defining the model task is not always straightforward. There's often many
different approaches you can take towards solving a problem and it's not always immediately
evident which is optimal. If your problem is vague and the modeling task is not clear, jump over
to my post on defining requirements for machine learning projects before proceeding.

Prioritizing projects
Ideal: project has high impact and high feasibility.
Mental models for evaluating project impact:

• Look for places where cheap prediction drives large value

• Look for complicated rule-based software where we can learn rules instead of programming
them
When evaluating projects, it can be useful to have a common language and understanding of the
differences between traditional software and machine learning software. Andrej
Karparthy's Software 2.0 is recommended reading for this topic.
Software 1.0
• Explicit instructions for a computer written by a programmer using a programming
language such as Python or C++. A human writes the logic such that when the system is
provided with data it will output the desired behavior.

Software 2.0
• Implicit instructions by providing data, "written" by an optimization algorithm
using parameters of specified model architecture. The system logic is learned from a provided
collection of data examples and their corresponding desired behaviour.

See this talk for more detail.

A quick note on Software 1.0 and Software 2.0 - these two paradigms are not mutually exclusive.
Software 2.0 is usually used to scale the logic component of traditional software systems by
leveraging large amounts of data to enable more complex or nuanced decision logic.
 For example, Takeoff Projects about how the code for Google Translate used to be a very
complicated system consisting of ~500k lines of code. Google was able to simplify this product
by leveraging a machine learning model to perform the core logical task of translating text to a
different language, requiring only ~500 lines of code to describe the model. However, this
model still requires some "Software 1.0" code to process the user's query, invoke the machine
learning model, and return the desired information to the user.
In summary, machine learning can drive large value in applications where decision logic is
difficult or complicated for humans to write, but relatively easy for machines to learn. On that
note, we'll continue to the next section to discuss how to evaluate whether a task is "relatively
easy" for machines to learn.

Determining feasibility
Some useful questions to ask when determining the feasibility of a project:

• Cost of data acquisition

• How hard is it to acquire data?

• How expensive is data labeling?

• How much data will be needed?

• Cost of wrong predictions

• How frequently does the system need to be right to be useful?

• Are there scenarios where a wrong prediction incurs a large cost?

• Availability of good published work about similar problems

• Has the problem been reduced to practice?

• Is there sufficient literature on the problem?

• Are there pre-trained models we can leverage?

• Computational resources available both for training and inference

• Will the model be deployed in a resource-constrained environment?

• What are the latency requirements for the model?

Specifying project requirements

Establish a single value optimization metric for the project. Can also include several
other satisficing metrics (ie. performance thresholds) to evaluate models, but can
only optimize a single metric.
Example:
• Optimize for accuracy

• Prediction latency under 10 ms

• Model requires no more than 1gb of memory

• 90% coverage (model confidence exceeds required threshold to consider a prediction as valid)
The optimization metric may be a weighted sum of many things which we care about. Revisit
this metric as performance improves.

Some teams may choose to ignore a certain requirement at the start of the project, with the goal
of revising their solution (to meet the ignored requirements) after they have discovered a
promising general approach.

Decide at what point you will ship your first model.

Some teams aim for a “neutral” first launch: a first launch that explicitly deprioritizes machine
learning gains, to avoid getting distracted.

The motivation behind this approach is that the first deployment should involve a simple model
with focus spent on building the proper machine learning pipeline required for prediction. This
allows you to deliver value quickly and avoid the trap of spending too much of your time trying
to “squeeze the juice”

Setting up a ML codebase
A well-organized machine learning codebase should modularize data processing, model
definition, model training, and experiment management.

Example codebase organization:

configs/
baseline.yaml
latest.yaml
data/
docker/
project_name/
api/
app.py
models/
base.py
simple_baseline.py
cnn.py
datasets.py
train.py
experiment.py
scripts/

data/ provides a place to store raw and processed data for your project. You can also include
a data/README.md file which describes the data for your project.

docker/ is a place to specify one or many Dockerfiles for the project. Docker (and other
container solutions) help ensure consistent behavior across multiple machines and
deployments.

api/app.py exposes the model through a REST client for predictions. You will likely choose to
load the (trained) model from a model registry rather than importing directly from your library.

models/ defines a collection of machine learning models for the task, unified by a common API
defined in base.py. These models include code for any necessary data preprocessing and output
normalization.

datasets.py manages construction of the dataset. Handles data pipelining/staging areas,

shuffling, reading from disk.

experiment.py manages the experiment process of evaluating multiple models/ideas. This

constructs the dataset and models for a given experiment.

train.py defines the actual training loop for the model. This code interacts with the optimizer
and handles logging during training.