4/28/2019 Simple guide to confusion matrix terminology

March 25, 2014 · MACHINE LEARNING

Simple guide to
confusion matrix
Launch a data science career!
terminology

 

A confusion matrix is a table that is often used to

Name:
describe the performance of a classi cation
model (or "classi er") on a set of test data for which
Email address:
the true values are known. The confusion matrix
itself is relatively simple to understand, but the
Join the Newsletter related terminology can be confusing.

I wanted to create a "quick reference guide" for

New? Start here!
confusion matrix terminology because I couldn't
Machine Learning course nd an existing resource that suited my
requirements: compact in presentation, using
Join my 80,000+ YouTube
numbers instead of arbitrary variables, and
subscribers
explained both in terms of formulas and sentences.

Join Data School Insiders Let's start with an example confusion matrix for

Private forum for Insiders a binary classi er (though it can easily be

extended to the case of more than two classes):
About

What can we learn from this matrix?

There are two possible predicted classes:

"yes" and "no". If we were predicting the
https://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/ 1/14
4/28/2019 Simple guide to confusion matrix terminology

presence of a disease, for example, "yes"

would mean they have the disease, and "no"
would mean they don't have the disease.
The classi er made a total of 165 predictions
(e.g., 165 patients were being tested for the
presence of that disease).
Out of those 165 cases, the classi er
Launch a data science career!
predicted "yes" 110 times, and "no" 55 times.
 
In reality, 105 patients in the sample have the
Name: disease, and 60 patients do not.

Let's now de ne the most basic terms, which are

Email address: whole numbers (not rates):

true positives (TP): These are cases in which

Join the Newsletter we predicted yes (they have the disease), and
they do have the disease.
New? Start here!
true negatives (TN): We predicted no, and

Machine Learning course they don't have the disease.

false positives (FP): We predicted yes, but
Join my 80,000+ YouTube they don't actually have the disease. (Also
subscribers known as a "Type I error.")
false negatives (FN): We predicted no, but
Join Data School Insiders
they actually do have the disease. (Also known
Private forum for Insiders as a "Type II error.")

About I've added these terms to the confusion matrix, and

also added the row and column totals:

https://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/ 2/14
4/28/2019 Simple guide to confusion matrix terminology

This is a list of rates that are often computed from a

confusion matrix for a binary classi er:

Accuracy: Overall, how often is the classi er

correct?
(TP+TN)/total = (100+50)/165 = 0.91
Misclassi cation Rate: Overall, how often is
Launch a data science career! it wrong?

  (FP+FN)/total = (10+5)/165 = 0.09
equivalent to 1 minus Accuracy
Name:
also known as "Error Rate"
True Positive Rate: When it's actually yes,
Email address: how often does it predict yes?
TP/actual yes = 100/105 = 0.95
also known as "Sensitivity" or "Recall"
Join the Newsletter
False Positive Rate: When it's actually no,
New? Start here! how often does it predict yes?
FP/actual no = 10/60 = 0.17
Machine Learning course True Negative Rate: When it's actually no,
how often does it predict no?
Join my 80,000+ YouTube
TN/actual no = 50/60 = 0.83
subscribers
equivalent to 1 minus False Positive
Join Data School Insiders Rate
also known as "Speci city"
Private forum for Insiders
Precision: When it predicts yes, how often is

About it correct?
TP/predicted yes = 100/110 = 0.91
Prevalence: How often does the yes
condition actually occur in our sample?
actual yes/total = 105/165 = 0.64

A couple other terms are also worth mentioning:

Null Error Rate: This is how often you would

be wrong if you always predicted the majority
class. (In our example, the null error rate
would be 60/165=0.36 because if you always
predicted yes, you would only be wrong for
the 60 "no" cases.) This can be a useful
https://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/ 3/14
4/28/2019 Simple guide to confusion matrix terminology

baseline metric to compare your classi er

against. However, the best classi er for a
particular application will sometimes have a
higher error rate than the null error rate, as
demonstrated by the Accuracy Paradox.
Cohen's Kappa: This is essentially a measure
of how well the classi er performed as
Launch a data science career!
compared to how well it would have
 
performed simply by chance. In other words,
Name: a model will have a high Kappa score if there
is a big di erence between the accuracy and

Email address:
the null error rate. (More details about
Cohen's Kappa.)
F Score: This is a weighted average of the
Join the Newsletter true positive rate (recall) and precision. (More
details about the F Score.)
New? Start here! ROC Curve: This is a commonly used graph
that summarizes the performance of a
Machine Learning course
classi er over all possible thresholds. It is
Join my 80,000+ YouTube generated by plotting the True Positive Rate
subscribers (y-axis) against the False Positive Rate (x-axis)
as you vary the threshold for assigning
Join Data School Insiders
observations to a given class. (More details
Private forum for Insiders about ROC Curves.)

About And nally, for those of you from the world of

Bayesian statistics, here's a quick summary of these
terms from Applied Predictive Modeling:

In relation to Bayesian statistics, the

sensitivity and speci city are the
conditional probabilities, the prevalence
is the prior, and the positive/negative
predicted values are the posterior
probabilities.

Want to learn more?

https://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/ 4/14
4/28/2019 Simple guide to confusion matrix terminology

In my new 35-minute video, Making sense of the

confusion matrix, I explain these concepts in more
depth and cover more advanced topics:

How to calculate precision and recall for

multi-class problems
How to analyze a 10-class confusion matrix
Launch a data science career! How to choose the right evaluation metric for

  your problem
Why accuracy is often a misleading metric
Name:

EMAIL FACEBOOK
TWITTERLINKEDIN
TUMBLRREDDIT GOOGLE+
POCKET
Email address:

Join the Newsletter

Data School Comment Policy


New? Start here! All comments are moderated, and will usually be
approved by Kevin within a few hours. Thanks for
Machine Learning course your patience!

Join my 80,000+ YouTube

subscribers Comments Community 
1 Login

Join Data School Insiders t Tweet f Share

 Recommend 46

Private forum for Insiders Sort by Best

About Join the discussion…

LOG IN WITH

OR SIGN UP WITH DISQUS ?

Name

Engr Ali Raza • 2 years ago

Dear Kevin,
can you please tell me the relationship between
Misclassifcations and split value or split value
index??also explain that how we can calculate
https://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/
th lit l i i l d i i l ith ? 5/14