DEEP LEARNING : LEARNING

DIAGNOSE PERFORMANCE CURVES TO

HEMANT THAPA
Learning curves are a widely used diagnostic tool in machine learning for algorithms that
learn from a training dataset incrementally. The model can be evaluated on the training
dataset and on a holdout validation dataset after each update during training, and plots
of the measured performance can be created to show learning curves.
Reviewing learning curves of models during training can be used to diagnose problems
with learning, such as an underfit or overfit model, as well as whether the training and
validation datasets are suitably representative.
In this document, you will discover learning curves and how they can be used to
diagnose the learning and generalization behavior of machine learning models, with
example plots showing common learning problems.
After reading this post, you will know:
Learning curves are plots that show changes in learning performance over time in terms
of experience. Learning curves of model performance on the training and validation
datasets can be used to diagnose an underfit, overfit, or well-fit model. Learning curves
of model performance can be used to diagnose whether the training or validation
datasets are not relatively representative of the problem domain.
1. Understanding Learning Curves in Machine Learning
A learning curve is like a friendly graph that captures how you're getting better at
something over time. Imagine the x-axis as time or experience, and the y-axis as your
progress or improvement.
Learning curves (LCs) are deemed effective tools for monitoring the performance of
workers exposed to a new task. LCs provide a mathematical representation of the
learning process that takes place as task repetition occurs.
— Learning curve models and applications: Literature review and research directions,
2011.
Here's a real-world
instrument, someoneexample: if youyouwere
could give learning
a score everytoweek
play for
a musical
a year.
Plotting those scores over the 52 weeks would create a learning curve.
This curve would help you see how you're doing on the instrument as time
goes by.
So, what exactly is a learning curve? It's just a line graph that tells you how much you're
learning (that's on the up-and-down side) as you gather more experience (that's on the
left-to-right side).
In the world of machine learning, we use learning curves to keep an eye on algorithms
that learn and improve gradually, like those fancy neural networks in deep learning.
The way we measure learning might be like trying to get the highest score possible,
where bigger numbers mean more learning. Think of it as a game of maximizing.
But sometimes, we use a score where smaller numbers are better, like errors or losses.
Here, lower numbers mean you're getting better. If the score hits 0.0, you've aced the
training and made zero mistakes.
When we're training a machine learning model, we can check how well it's doing at each
step. We test it on the training dataset to see how much it's "learning." Then we test it
on a separate validation dataset that wasn't part of the training. This tells us how well
the model is "generalizing," or applying its learning to new stuff.
There are two types of learning curves we often make:
Train Learning Curve: This one uses the training data to show how well the model is
learning. Validation Learning Curve: Here, we use the validation data to see how well the
model is doing on new things. We usually make both curves while the model is learning,
using both the training and validation datasets.
Sometimes, we even make curves for more than one thing, like in problems where we
predict categories. Imagine tuning a model based on both how wrong it is (loss) and how
many things it gets right (accuracy). You'd end up with two plots, each showing two
learning curves – one for training and one for validation.
So, we've got:
Optimization Learning Curves: These show how the model's key parameters are getting
better over time (using a measure like loss). Performance Learning Curves: These tell us
how the model is doing, based on the evaluation criteria we care about (like accuracy).
Learning curves are like our learning buddies, helping us watch how well our models are
catching on and how good they are becoming.
2. Understanding Model Behavior Through Learning Curves
The structure and patterns within a learning curve provide valuable insights for
diagnosing the behavior of a machine learning model. This, in turn, can guide
recommendations for potential configuration adjustments to enhance both learning and
performance.
Three prevalent dynamics often manifest in learning curves. They are as follows:
1. Underfitting
2. Overfitting
3. Optimal Fit

3. Identifying Underfitting Through Learning Curves

Underfitting refers to a model's inability to grasp the training dataset.
This issue arises when the model struggles to achieve a sufficiently low error value on
the training set.
— Cited from "Deep Learning," 2016, Page 111.
Recognition of an underfit model typically revolves around the learning curve related to
training loss.
This curve might display a flat line or exhibit fluctuating values that represent relatively
high loss. Such behavior indicates the model's failure to comprehend the intricacies of
the training dataset.
Here's an example that illustrates this phenomenon. It's a common occurrence when the
model lacks the requisite complexity to handle the intricacies inherent in the dataset.m
An underfit model can also reveal itself through a specific pattern in the training loss
curve. If the training loss keeps decreasing and continues to do so at the plot's end, it's
a sign.
This pattern suggests that the model has more room to learn and improve, and that the
training process might have been stopped prematurely.
A plot displaying learning curves indicates underfitting when:
1. The training loss remains stagnant despite ongoing training.
2. The training loss keeps decreasing steadily until the conclusion of the training.

4. Recognizing Overfitting Using Learning Curves

Overfitting occurs when a model becomes too attuned to the training dataset, even
capturing its statistical noise and random fluctuations.
fitting a more flexible model requires estimating a greater number of
parameters. These more complex models can lead to a phenomenon
known as overfitting the data, which essentially means they follow the
errors, or noise, too closely.
— Extract from "An Introduction to Statistical Learning: with Applications in R," 2013,
Page 22.
The challenge with overfitting is that as the model becomes increasingly specialized to
the training data, its capacity to generalize to new data diminishes. Consequently,
generalization error rises. The magnitude of this increase in generalization error can be
assessed through the model's performance on the validation dataset.
This is an example of overfitting the data,. It is an undesirable situation
because the fit obtained will not yield accurate estimates of the response
on new observations that were not part of the original training data set.
— Extract from "An Introduction to Statistical Learning: with Applications in R," 2013,
Page 24.
Overfitting often arises when the model possesses more capacity than required for the
task, resulting in excessive flexibility. It can also manifest due to excessive training.
Learning curve plots indicate overfitting when:
The training loss curve continues to decline alongside accumulated experience. The
validation loss curve initially drops but subsequently starts ascending. The inflection
point in the validation loss marks the juncture where training should potentially cease, as
subsequent experience showcases overfitting tendencies. The following example plot
aptly illustrates a scenario of overfitting.

5. Identifying Good Fit Through Learning Curves

Achieving a good fit represents the objective of the learning algorithm, positioned
between the extremes of overfitting and underfitting.
A good fit is recognized by both training and validation losses descending to a state of
stability, accompanied by a minimal disparity between their respective concluding loss
values.
Typically, a model's loss on the training dataset will be lower than that on the validation
dataset. Consequently, a discernible gap tends to exist between the learning curves of
training and validation losses—a gap termed the "generalization gap."
Learning curve plots illustrate a good fit when:
The training loss curve descends to a state of stability. The validation loss curve similarly
attains stability and maintains a narrow gap with the training loss. Further training of a
model displaying a good fit is liable to lead to overfitting.

6. Diagnosing Unrepresentative Datasets through Learning

Curves
Learning curves extend their utility to diagnosing dataset attributes and assessing their
relative representativeness.
An unrepresentative dataset signifies a dataset that might fail to encompass the
statistical traits in comparison to another dataset sourced from the same domain. This
discrepancy often arises between a training and a validation dataset and could stem
from an insufficient number of samples in one dataset when contrasted with the other.
Two prevailing scenarios warrant consideration:
1. Relatively Unrepresentative Training Dataset
2. Relatively Unrepresentative Validation Dataset

7. Detecting Unrepresentative Training Datasets

An unrepresentative training dataset signifies that the provided training data lacks
adequate information to effectively grasp the problem, especially when juxtaposed with
the validation dataset utilized for assessment.
This circumstance may arise due to a scarcity of training examples when compared to
the validation dataset.
Such a scenario becomes evident through learning curves: the training loss curve
demonstrates enhancement, while the validation loss curve also displays improvement.
However, a substantial gap persists between the two curves.

8. Identifying Unrepresentative Validation Datasets

An unrepresentative validation dataset denotes a situation where the validation data
lacks the necessary information to assess the model's capacity to generalize.
Such instances can arise if the validation dataset contains an insufficient number of
examples compared to the training dataset.
This scenario can be recognized through learning curves: the training loss curve adopts
a pattern similar to a good fit (or other fits), while the validation loss curve exhibits
erratic fluctuations around the training loss curve.
It can also be discerned by a validation loss that registers lower than the training loss. In
such instances, this points towards the possibility that the model finds the validation
dataset comparatively easier to predict than the training dataset.

Summary
 1. Learning curves manifest changes in learning performance over time, corresponding
to experience.
2. Learning curves of model performance across both training and validation datasets
serve as diagnostic tools for identifying underfitting, overfitting, or optimal fitting
models.
3. Learning curves of model performance also help in ascertaining the potential
mismatch between the train or validation datasets and the problem domain's
representation.

References:
Machine Learning Mastery. (n.d.). Learning Curves for Diagnosing Machine Learning
Model Performance. Retrieved from https://machinelearningmastery.com/learning-
curves-for-diagnosing-machine-learning-model-performance/
Dimleve. (n.d.). Back Propagation Explained. Retrieved from
https://dimleve.medium.com/back-propagation-explained-9720c2d4a566
Stanford University. (n.d.). MultiLayer Neural Networks. Deep Learning Tutorial.
Retrieved from
http://deeplearning.stanford.edu/tutorial/supervised/MultiLayerNeuralNetworks/