Copyright Notice

These slides are distributed under the Creative Commons License.

DeepLearning.AI makes these slides available for educational purposes. You may not use or distribute
these slides for commercial purposes. You may make copies of these slides and use or distribute them for
educational purposes as long as you cite DeepLearning.AI as the source of the slides.

For the rest of the details of the license, see https://creativecommons.org/licenses/by-sa/2.0/legalcode

 Error Analysis

Carrying out error

deeplearning.ai
analysis
Look at dev examples to evaluate ideas

Should you try to make your cat classifier do better on dogs?

Error analysis:
• Get ~100 mislabeled dev set examples.
• Count up how many are dogs.

Andrew Ng
Evaluate multiple ideas in parallel
Ideas for cat detection:
• Fix pictures of dogs being recognized as cats
• Fix great cats (lions, panthers, etc..) being misrecognized
• Improve performance on blurry images
Image
1
2
3
..
.
% of total
Andrew Ng
 Error Analysis

Cleaning up
deeplearning.ai Incorrectly labeled
data
Incorrectly labeled examples

y 1 0 1 1 0 1 1

DL algorithms are quite robust to random errors in the

training set.

Andrew Ng
Error analysis
Incorrectly
Image Dog Great Cat Blurry Comments
labeled
…
Labeler missed cat
98 in background

99
Drawing of a cat;
100 Not a real cat.

% of total 8% 43% 61% 6%

Overall dev set error

Errors due incorrect labels

Errors due to other causes

Goal of dev set is to help you select between two classifiers A & B.
Andrew Ng
Correcting incorrect dev/test set examples

• Apply same process to your dev and test sets to

make sure they continue to come from the same
distribution
• Consider examining examples your algorithm got
right as well as ones it got wrong.
• Train and dev/test data may now come from
slightly different distributions.
Andrew Ng
 Error Analysis

Build your first system

deeplearning.ai
quickly, then iterate
Speech recognition example

• Noisy background • Set up dev/test set

• Café noise and metric
• Car noise • Build initial
• Accented speech system quickly
Guideline:
• Far from microphone • Use Bias/Variance
Build your
• Young children’s first
speech analysis & Error
system quickly,
• Stuttering analysis to
• … prioritize next
then iterate steps.
Andrew Ng
 Mismatched training
and dev/test data
Training and testing
deeplearning.ai on different
distributions
Cat app example
Data from webpages Data from mobile app

Andrew Ng
Speech recognition example

Training Dev/test

Purchased data Speech activated

rearview mirror
Smart speaker control

Voice keyboard

Andrew Ng
 Mismatched training
and dev/test data

Bias and Variance with

deeplearning.ai mismatched data
distributions
Cat classifier example
Assume humans get ≈ 0% error.

Training error
Dev error

Training-dev set: Same

distribution as training
set, but not used for
training
Andrew Ng
Bias/variance on mismatched training and
dev/test sets

Andrew Ng
More general formulation

Andrew Ng
 Mismatched training
and dev/test data

Addressing data
deeplearning.ai
mismatch
Addressing data mismatch
• Carry out manual error analysis to try to understand difference
between training and dev/test sets

• Make training data more similar; or collect more data similar to

dev/test sets

Andrew Ng
Artificial data synthesis

+ =

“The quick brown Car noise Synthesized

fox jumps in-car audio
over the lazy dog.”

Andrew Ng
Artificial data synthesis

Car recognition:

Andrew Ng
 Learning from
multiple tasks

Transfer learning
deeplearning.ai
Transfer learning

x !"

x !"

Andrew Ng
When transfer learning makes sense

• Task A and B have the same input x.

• You have a lot more data for Task A than Task B.

• Low level features from A could be helpful for learning B.

Andrew Ng
 Learning from
multiple tasks

Multi-task
deeplearning.ai
learning
Simplified autonomous driving example

Andrew Ng
Neural network architecture

x !"

Andrew Ng
When multi-task learning makes sense
• Training on a set of tasks that could benefit from having
shared lower-level features.
• Usually: Amount of data you have for each task is quite
similar.

• Can train a big enough neural network to do well on all

the tasks.

Andrew Ng
 End-to-end deep
learning
What is
deeplearning.ai end-to-end
deep learning
What is end-to-end learning?
Speech recognition example

Andrew Ng
Face recognition

[Image courtesy of Baidu]

Andrew Ng
More examples

Machine translation

Estimating child’s age:

Andrew Ng
 End-to-end deep
learning

Whether to use
deeplearning.ai
end-to-end learning
Pros and cons of end-to-end deep learning
Pros:
• Let the data speak
• Less hand-designing of components needed

Cons:
• May need large amount of data
• Excludes potentially useful hand-designed
components
Andrew Ng
Applying end-to-end deep learning
Key question: Do you have sufficient data to learn
a function of the complexity needed to map x to y?

Andrew Ng