Machine Learning Course - CS-433

Classification

Oct 8, 2019

changes by Rüdiger Urbanke 2019,2018,2017,2016; Mohammad

c Emtiyaz Khan 2015
Last updated on: October 8, 2019
Classification
Similar to regression, classification relates the input variable
x to the output variable y, but now y can only take on
discrete values. We say that y is a categorical variable.

Binary classification
When y can only take on two values, it is called binary clas-
sification. Sometimes we refer to the two discrete values
abstractly as y ∈ {C1, C2}. The Ci are called class labels or
simply classes. Other times it is more conveniently to as-
sume that y ∈ {−1, +1} or y ∈ {0, 1}. Note that even if
the class labels are real values, there is typically no ordering
implied between the two classes.

Multi-class classification
In a multi-class classification, y can take on more than two
values, i.e., y ∈ {C0, C1, . . . , CK−1} for a K-class problem.
Again, even though there is in general no ordering among
these classes, we sometimes use the labels y ∈ {0, 1, 2, . . . , K−
1}.

Examples of classification problems

A credit card service must be able to determine whether or
not a requested transaction is fraudulent. They might have
at their disposal the user’s IP address (if the transaction
happens on the web), past transaction history, and perhaps
some other features.
An example from the book Elements of Statistical Learning
by Hastie, Tibshirani, and Friedman is shown below. We
have at our disposal the annual incomes and monthly credit
card balances of a number of individuals. The individuals
who defaulted on their credit card payments are shown in
orange, and those who did not default are shown in blue.
60000
Income
40000
20000
0

0 500 1000 1500 2000 2500

Balance

To consider another example, a person arrives at the emer-

gency room with a set of symptoms that could possibly be
attributed to one of three medical conditions. Which of the
three conditions does the individual have?

Classifier
A classifier will divide the input space into a collection of
regions belonging to each class. The boundaries of these re-
gions are called decision boundaries. A classifier can be linear
or nonlinear. This distinction is less strict than it might seem
at first. E.g., if you look at the classifier in the right-hand
side of “Figure 4.1” you will see that the decision regions
are non-linear (not straight lines). But in fact the classifier
that led to these region is linear – we added some non-linear
features to the original feature vector (think polynomial ba-
sis) before performing the linear classification. The decision
boundaries appear non-linear since the plot is made in the
original feature space and not the extended feature space.

What is the aim of classification?

In some situations we are interested in classification in it-
self. This means, we are constructing a predictor based on
a training set and are interested in applying this predictor
to “new” data. Consider e.g. the example of the credit card
company that wants to predict if a customer will default or
not.
But in some instances we are interested in more. We would
like to “understand” the cause. Think e.g. of a disease
prediction. Not only are we interested to know who is at risk
but we would like to understand why somebody is at risk. So
we are interested in the “interpretation” of the prediction. In
particular for this second task it is important to have simple
models and to have means of eliminating features that do
not contribute significantly to the prediction.
Consider “Figure 4.12”. In this data set various risk factors
are given for a particular heart disease. These risk factors are
blood pressure (sbp), tobacco usage, family history, obesity,
 Elements of Statistical Learning (2nd Ed.) Hastie,
c Tibshirani & Friedman 2009 Chap 4

1 1

2 1 2 1
1 1
2 1 2 1
22 3 22 3
2 22 2 1 2 22 2 1
2 22 3 2 22 3
2 22 2 33 3 3 3 2 22 2 33 3 3 3
2 12 33 3 3 2 12 33 3 3
2 2 22 2 22 2 3 3 3 2 2 22 2 22 2 3 3 3
2 222 2 1 2 222 2 1
2
2 22 2 2 2 1 3 2
2 22 2 2 2 1 3
22 2 2 2 22 22 22 2 3 3 22 2 2 2 22 22 22 2 3 3
2 2222 2222 22222 222 22222 2 3 33333 3 33 2 2222 2222 22222 222 22222 2 3 33333 3 33
2
2 2222 2 2222 2 2 1 33 3 2
2 2222 2 2222 2 2 1 33 3
22 2 222 2
22 22 222 2222 2 2 3 333 33 22 2 222 2
22 22 222 2222 2 2 3 333 33
2 2 3 3 3 2 2 3 3 3
2 2 2222 22 22 22 22222222 22 2 13 333
33 33 2 2 2222 22 22 22 22222222 22 2 13 333
33 33
2 22 2
1 2
2
1 222 2 2222 22222 2 2 2 3 3 3 33 2 22 2
1 2
2
1 222 2 2222 22222 2 2 2 3 3 3 33
1 333333 3 1 333333 3
2 2 1 222
2 12 12 1 1 1 1 1 3 33 2 2 1 222
2 12 12 1 1 1 1 1 3 33
2 122122 22 2 2222 1 1 1 1 1 3
33 3 2 122122 22 2 2222 1 1 1 1 1 3
33 3
11
1 2
2 2 2 2
2211 22 1 1 11 11 11 1 1 1 1 11333333 33 3333
1 3 11
1 2
2 2 2 2
2211 22 1 1 11 11 11 1 1 1 1 11333333 33 3333
1 3
21221 2 1 1111 1 1333333 3 3 21221 2 1 1111 1 1333333 3 3
11 2 1 2212
2 1
1 1 1 11 1
1 1 3 33 3 11 2 1 2212
2 1
1 1 1 11 1
1 1 3 33 3
2 111 12 1 11 1 1 31333333333 3 2 111 12 1 11 1 1 31333333333 3
1 1 11 2 111 1 1 1 1111 1 11 1 3333333
11 1 1 1 3333333
3 1 1 11 2 111 1 1 1 1111 1 11 1 3333333
11 1 1 1 3333333
3
1 1 1 1
1 1 11 1 11 111 111 1 1 11 11 1 111 3 1 3
13133333 3 1 1 1 1
1 1 11 1 11 111 111 1 1 11 11 1 111 3 1 3
13133333 3
111 1 1 3 3
1 11 1 1 1 3333333 3 33 1 11 1 111 1 1 1 1 3333333 3 33
1 1 1 11 1 31 33 3 3 1 1 1 11 1 31 33 3 3
11 1 1 1 1 111111 11 111 1 1 1 1 1 333333333333 11 1 1 1 1 111111 11 111 1 1 1 1 1 333333333333
1 11 1 1 3 3 333333
11 3 33 1 11 1 1 3 3 333333
11 3 33
1 1 1 1
1 1 11 1
3 33333 3 1 1 11 1
3 33333 3
1 1 11 3 33 1 1 11 3 33
1 1 3 1 1 3
333 3 333 3
3 3
3 3 3 3

FIGURE 4.1. The left plot shows some data from

three classes, with linear decision boundaries found
by linear discriminant analysis. The right plot shows
quadratic decision boundaries. These were obtained by
finding linear boundaries in the five-dimensional space
X1 , X2 , X1 X2 , X12 , X22 . Linear inequalities in this space
are quadratic inequalities in the original space.

alcohol consumption, and age. For each pair of these risk

factors the figure shows how the cases (people who have this
disease) separate from the controls (people who do not have
the desease). Such plots can help to decide which risk factors
should be included in a model and which might have little
predictive power.
 Elements of Statistical Learning (2nd Ed.) Hastie,
c Tibshirani & Friedman 2009 Chap 4

0 10 20 30 0.0 0.4 0.8 0 50 100

220
o o o o oo o o oo oooo o o o o oo
oo ooo o
o oo o o
o o ooo
o o
o
ooo
o o o o o ooo
ooo o oo o ooo o
o o o
ooooooooo ooo
oo oo ooo
ooo oo oo ooo ooo
ooo ooooooo
o o o o o
o o ooo
o
o
o
o o o
o ooooooooo oooooo o oo o o oo o o
o ooooo oo ooo
ooo oooooooooooooo oooooooooo o
ooooooooooooo oo ooo oo o o o o oooooooooooo o ooooooo
oooo oooooooooooo o o oo
oo

160
sbp o oooooo o o o oo o o o
oooooooo o o o o ooo oooooo oo oo oooo
oooo o oo
oooooooooo o
oo o oo o
ooo o o
ooo ooo
oo o o o o
o
o oo oo
o oo
ooo ooooo
o oooooooooo o o ooo oooo ooooooooooooo oo o ooo ooooooooo oooooooooooooo ooo
o
oooooo
o o
ooo ooo o
o
o oo
ooo
ooooo
oooooooo o
o
oooooo o ooooooooo
o o
oo
oo o
oooo
o
oooooooooooooo
ooo o
o
ooo
o
o o
o
o o
o
oooo
o oooooooooooo
o o ooooo
ooooo oooooo o
ooo o
o o o
o o
oo
ooo o o
oo
ooo oooooooo o
ooooooooo o o ooo
o o oo
ooooo
oo oo ooo
oooooooooooooooo oo o
oo
o
oo o
oooooooooooooo oo
ooo
oo
o
o o oo
oooo
oo
o
oo o
oo
o
ooooooooooo oo
oo
ooo
oo
oo o oo
oooo
oooo
ooo
oooo
o oooo
oo
oo
oo
oo o
ooo oo
o
o o o
ooo o o
oo
o o o
o
oo
oo
o
oo oo
o ooo o
oo
ooo o
ooo
oo
o o
oooo
ooooo
o o
o
ooo
o o oo
ooooo oooooooo
oo o o
oo
oo
o
oooooooo
oo
oo oo
ooooo
oooooo
oooo
oooooooo o o oooooo
ooo ooo o
ooooooooo
oo oo
ooooooooooo ooo
oooo
o oooooo ooooooo oooooooo
ooo
oo
o
o
oo
ooo
ooo oooooooo ooooo oo o oooooo
oo
o oo
oooooooo
oooooo oooo
oo oooooooooo ooo o o
oooo oo
o o ooo ooooo oooo
oo o
o oooooo oo
oo o o ooooo o o o o oo o
o oo o
oo oooooooooooooo
oo o o o oo o o
oo oooo ooooooo o o o o
oo
o
ooo
o o o
o
o
oo
o o oo ooo o
o
oo
o
o
oo
o
o
oooo
o o
ooooo o o
oooooooo o o o o o o
ooooo o o o
ooo
oo
o
o o oo
oooooooooooooo oo oo o o oooooooooooo o o o oo o o o
ooo oooooo o

100
o
ooo o o
o oo o ooo ooooo ooo o o o o
o o o
30

o o o
o o o o o
o o o
o o o o
20

oo oo o o oo oo o ooo o ooooo o o o o o o o
o o
o o o
o ooooo oo o tobacco ooooo
o
oooooo oo
o o
ooo oo ooo o
oooooo ooo
o ooooo ooo o
oo o oo ooooo o
o oooooooooo o oo
o o o ooooo ooo o ooo ooooooo o oo ooo oo
oooooooooooooooooooo o o ooo oooooooo o o
oooo
o o oo o o o oooooooooo
10

o o oooo oo o o
o o ooooo oooooooooooo oooooo o o o
o oo
o ooo oooooooo o oooooo ooo o
ooooooooooooo oooooooo ooo oo o o oo ooo o
oooo
ooooooo
oooo
oo
oooooo
ooooo o o o
oooo
oooo o o
o o
ooooooooo
o o ooooo o ooo
oooo oooooo oooooooooo
ooooooo oooo o oooo
o oo o o
o oo oo o
o o o o oooo oooo o
o o ooo oo oooo ooooo ooooo
o oooooo oo oo o oooo oo
oo
oooo oo
oo oooo ooo
oooo oooooo o oooooo
o ooo
oo
oo
o
oo
o
oooooo
o oo oooooo
ooo o oo o o
ooo
o
oo o
ooo
o
oo
o
oo
ooooo
o
ooo
o o
ooooooo
ooo
o
oo
ooo
o
oooooo
o o oooooo ooooo o
oooooo o
oooo
oooooooooo ooo o
o o
o o o
oooooooooooo ooo
o o o
ooooo
oo oo ooo
oo
ooooooooo
ooo
o
o
oooooo oo o o
oooo
ooo o o
oo
oo
oo
oooooo ooo oo
o
oooooooooooooo
oo
oooo o
o
o
ooooo
oo
ooo
oo
o o
oooooo
o
oo
o
ooo
o
ooooo
oo
ooooo
oo o
o oo o
oo o oo
o o
o
oooooooo o ooo o
o o o
oooooo
oo o
o
oooo
oo o o o o o
oo
ooooooooooooooooooooooooooo
oo o o o o ooo o
o oo
oo o o oo o
o o
oo
ooo
oo
o
ooo o ooooooooooo
o o oo o ooo o
o
ooooooo o ooooooooooooooo o
o o
o o ooo o
o o
ooo
o
oo
oo
oo oooooooo ooooooooo oooo o ooooooooooooo
oooooooooo ooooooooooo ooo ooooo
oo oo ooooooooooooooo o
ooo
0

oo oo o o o o
o o o o o o
o

10 14
o o o o o
ooo o o oooo
o o oooo o oo oo o o o ooo o o
oo oooo oo oo
o o ooo o oooo oooo o o
o oo o oo
o oooooo oo o
oooooo o o o o o o oooooooo o
o ooooooooo o o o o o
ooooo ooooo oooo ooooo oo
oo ooo oo
oo
ooo
oooo ldl ooo ooo o oooo
oooooooooo o ooooo oooooo o o oo oo o oo oooooooooooooooooooooooo
oooo o o oo ooo ooo oo o oo oooo
ooooooo ooooo ooooooo
oooo oooo
o oo oo ooooo ooooooooo o o
oo oo ooooo ooooooo o o oo ooooooooooo ooooooo o
ooooooooooooooo
oo
ooo
o
o ooo
ooo oo
ooooooo oo oo o oooooo
oo o
o oo o
ooo
o o o ooo ooooo
ooo oooooo
oo oooo o o o
oo
o o oooooooo oo oo oo oooooooooooooooo

6
oo ooo oo o o o oo oo o o o o o
oo o o o oooo oo o oo o o o o o o o o o
oo oo
ooo ooo ooooo ooo ooo
ooo oooooooooooooo o o o ooooooo
ooo ooo o o o
ooooooooooooo o o
o o o o
ooooo o ooo
o o ooo o o o
ooooo o oo oooooo oo oo o
oooo o o o
oooo o
oo oo o o
ooooooooo o o oooo o ooooooooo oooooooo oo o
oooooooooo o o o o o
ooooooooooooo
oo o oo
oo
o oo o
oooooooo oo
oo oo
ooo
ooooo ooo
oooo oo
oooo
ooooo
oo oo
oo
oooooo oooo
oo ooo o o oo ooo
oooo ooo
oooo oooo o oooooooooo oo o o o
oo o
oo oo
oo o ooo
ooo oo ooo
oo o
o
oooo o oooooo
o ooo oo oo
ooooooooooooooo
o o o oo
o ooo oo
o
o oo o
oooooooooooooooo ooooooooo ooo o o
ooo oooo oooooooooooo o oooo
o oooooooooo ooo oo
ooo
ooo
ooo o oooo o
oooooo
o o oo
ooo
ooooooooooooo
o ooooooo oo ooo oo
ooo
oooo ooooo oooo o
o o
ooo
o
o ooo o ooooo
o
oooooooo o oooo o
oooooo
ooo
oo o o ooooooo o o
oo ooo
o oooo o
o o o o o ooo
o o oo o
oooo oooo o ooo
o o o o
o
o oo
o o o o o o o oo
o ooo
o oo o
ooooo o ooo
o

2
o o o
o o oooooo ooo ooo o
ooooo
ooo
ooooo
oo
ooo
oo
oo oo
ooooooooooo
oo oo
ooooooooooooooooooo oo
o oooooooooooooooooooo
oo
o
oo
oo
oo oooooooooooooo o o ooooooooo
o oooooo
oooo
ooo o oo
ooo
oooooooooo oooo
oo oooo
o oo oo
ooooo
oo oo
oo oo
oo
oo oo
ooo
oo oooooo
oo oooooooo ooooooooo ooooooooo
ooo
o ooooooooooo ooooo oo oo
ooooooo ooooooooooooooo
ooooo oooooo
oooooooo
oooooooo
oo ooo
o
o
oooooo
oooo
oo
0.8

famhist
0.4
0.0

oooooo
ooooo o ooo
oo
oo oo
ooooooooooooo
oo
oo
oo
o
oo
ooo
oo ooooooooooooooooooooooo oooo
oo ooooo
ooo
oo
oo
ooo oo oooooooooooooooooo
ooo
oo oo ooo oooo
oooooo
oooo
ooooooooooo ooo
oo
oooo oooooooooo o o o
oooooo o
oooooooo
oo
o o
oo
oo oo
ooo
ooo
oo oo
oo
o ooooo
oo
oo oooooo
oooooo o o
oo
o
ooo
o
o oo oooo
o
oooo
ooo
oo
oo ooo
ooooooooo
oooooooooo oooooooooo
ooo ooooooooooooo
o
ooo ooooooo ooo
ooooooooo ooooooooooooooooooooooooo
o o o o o o oo oo o o

45
o o oo o oo oo o oo
o o oo oo ooo ooo o o oo oo o
o oooo o
oo ooooooo o
oo oooooooo oo oooo oo o oo

35
ooo oooooooooooo o oo
ooooooo ooo o oooooooooooooooo o o o
ooooo o ooo oooooooooo oo oo oooooooooo oooooooo
oo o o o o oooooooo ooooooo o o ooo o oooo ooooo o o ooooo oooo o
oo o
oooooooo
oo
ooo
oooo o
oo
o
ooooo ooooo
oooooo o o
ooooo ooo ooooooo o o
ooo
oooo o oooooooooooo
oo
ooooo
oooooooo oo o o
o
o
ooooo o
oo
ooooooooooo
o o
o oo
o
oooooooo o
ooo
o
o oo o o o
o
ooo
o
o
o o
oo
o o o
o
o obesity o
o
o
oo
ooo
oo
o o
oooo oo o o o o
ooooooooooo
oo o
oo o oo ooo o
o
o o o
oooooooooo ooo
o
o
o o oo oooooo
o
oo
o oo o ooooo
oo
o
oooooo
ooooo
o
ooooo oo ooooo oooooo
oo oo ooo o oo o oo oooo o
ooo ooooo ooooooooooooooooo ooooo oo oo oo
ooooooooo ooooooooo oo o o o
oo
oo oo
o
o oo
oo o oooo ooo
oooooo oooooo
oooo o
ooooooo
oooo
ooooo o
ooooooooo
ooooooooooo oooo
ooooo o ooo o oooooooooo
o
oooooooo oooo oooooo oo
ooo o oooooo oo oo oo o ooo oo oooo oo oooooo

25
ooo ooooo
ooo
o
ooo oo oo
oooo
ooooo oooooooooooo
o
o ooooooo o o o
oooooo
o oo
oooo oo ooooooo
oo oo ooo oooooo
oooo
o o
oo
ooo
oo oooo
oooooooo
oo oo
o oooo o o oo
o
oooo o
oo oo ooo
o
o
ooooooooooooooo ooooo o oo ooo
ooo oooooooooooooooooooo o
oo oo
ooooooo oooooo
o oo oo
ooooooo o ooo o ooo ooooo ooooooooooooo o oooooooooooooooo
o o
o o o o
o o o o o ooo o
o oooooooooooooooooooooo o
oo o
o
oo
oo o o
oo o oo o
oooooo
o
o o ooo
ooooo ooo
o o o o
oo
oooo
o
o o
o o oo ooo o ooo o
o o o o o o o
ooooooo
o
oo
o oooo oo o o oo oo oooo o ooo
o ooooooo oooo

15
o o o o o o
ooo o oo o oo o oo o o o
o o o oo o o
oo o o oo o
100

o
oo ooooo o o
o o
o ooo ooo o o o
o ooo oooo o ooo ooo oo ooo o
o oo o o oo o oooo
ooooo o oo o oo ooooooo oooo ooooooooo o ooo ooooo alcohol oo ooo oooooo ooooo
ooooooooo o o ooo oo o o oooooo o
oo o oo
ooo oo o o o ooooo o oooo oo o o
o o o ooooo oo oo oo oo ooooooooooooo
50

oo o
oooo ooooo ooooo oooooooo oo oo
oooo ooooooooooo ooo oo ooooo o ooooo o o o
o o
o ooo
oo ooooo oo
ooo
o
oo ooo ooooo
o
o o oooooooooooooooooooo
ooo ooo o
o o o oo o o o o o o o oo o ooooooo oooooooo oooo
oooooo
oooooo o o o o o o o
oooo o
o o ooo ooo o o ooooo
o o oo o oo ooo
o oo oooooo oo oooo oooooo ooo oooo
ooooooooooo
oooo
oooo
oooo
oooo ooo oo o
ooo oooo ooooooo o oooo o
oooooooooooo o
o o o o o oo
oo oooo
o ooooooo ooooo
oooo oo
ooo o
o
ooooooo
ooooooo oooo
ooo
o oooo
oooo oo o
ooo
ooo oo
o oooo ooo
o
o o o ooo oooo
ooooo
oo ooooo
oo oooooo
oooo ooo ooo
ooo ooooooooooooo o o ooo o ooo
ooo oo oooooo oo
oo oooooo o oo
ooooo
ooooo oooooo
ooooo ooooooo oo
oo oo
o ooo
ooo
o
o
oooooooooooo o
oooooooooooooo ooo
o
ooooooo ooo
oooo o oooo
oo oooooooooooo ooooooooo o oooo
ooo
oo
oo
oooo
ooooooo
o
ooooooooooooo oo
oooooooooooooooooo o
oo oo
o
o
ooo o
o oooo o oooo
o
o
o
o
o o
o
o
o
o
oo
oo
o o ooo
oo
o oo
o o
o ooo
o
o
oooo oo o o o oo
oo oo
oo
oo oo
ooo
oo
ooooo oo
oooooo
oooo
oo oooo
ooo o
ooo
ooo
ooooo
oo ooooo oooooooooo
oooo
oooo
oooooooo ooo
o
ooo
oo
oo
0

o o o o o o o o o o o o oo
ooooo oo ooo ooo ooo o
oooo o
oooo o o oooooooo oooo oo oooo oooooo oo ooo ooo
o ooo oo oo o o o o o ooooooooo ooo o o o o oo ooooo oo o oo o oooo o
oooooo oo
ooo ooooo o o oooo
o o ooooo oo
ooo ooo ooooooooooo oooo ooo ooo oo ooooo
oooo o o

60
o o oo o o o o o o o o o o o oo o oo oo ooo o o oo o oo ooo oo
oooooooo oooo ooooooooooooo o ooo oooo o oooooooo oooooooooooo oooo o
ooo
oo oo ooooo ooo oo
ooooooooooo o
o o
o oooo
o o o oooooooo oo
oooooo oooooooo oooo ooooo oo
oo oooooooo oo oo
ooooooooo
oo
oooo ooo oo oo ooo oooooooooooooooo o o oooooo ooooo
ooo oooooo ooo ooo o oooo ooooooooo
o
oooooooooooo o
o
oooooo oo o o oooo o o oooo o
oo o
oooo o
oooooooooo
oo oooo o
oo o ooo oo ooooooooo o o o oo o
o o oooo o ooooooooo o o oooo oo o oooo
ooo
o o o
o oooooooooooooooooo
o o o o o o o oooo oo
o oo oo o oooo
oooo o
ooooooo ooo o
ooooooo o ooo o
ooo o o
oooooooo o o o oo
oooo o o oo oooooo
o o o
o o
oooooooooooo
o ooooooo o
oooooo oo oo o oo ooooooooooooooooo ooooo o oo
oooo o o
oo o ooooooooo ooo o
o o
o o ooooooooooo oo oooooooo o oooo oooo o oooo ooo
o oo oooooooo
o oooo
oo oo o o o
o o
o o oooo oooooo o ooooo oo

40
ooooooooo oo o ooo oooo o o o oooo o oooooooooooo o oo age
oooooo oooooooooo
o oooo
ooo ooooooo
oooo o
oo
oooo ooo
ooo oo ooooooooooo o
o o oooo
o
o oo
oo
o o
oooo
oo o
oooo
o o ooo
oooooooooo o o oooooo oo o
oooo o o ooo o
ooooo
ooooooooooo
o
ooo oo ooo
ooo oooooo
o ooooo oooo
o o ooooo
oooo
o
ooo ooo ooooo o
oo ooo ooo ooo
oooo
oooooo
oooooooooooo
o
ooooooo oo oooo
o oooooooooooooo ooooo
ooooo
ooo
ooooo o o o ooooo o o ooooooo o o o o o o o ooo ooooo o o
ooooooo o o ooo oo o o
oooo
oooooooo o o oo oo
o o ooooooooo o oo
o o o
20
oooo
oooooo oo oo
ooooo oo o
oo oooo ooooooooo ooo oo o
ooo oo oo o oooooo
oooooo
o o
ooooo oooo o
oooo ooo o o oo ooooooooooooo o o o
o ooooo oo
oooo o o oo o
100 160 220 2 6 10 14 15 25 35 45 20 40 60

FIGURE 4.12. A scatterplot matrix of the South

African heart disease data. Each plot shows a pair of
risk factors, and the cases and controls are color coded
(red is a case). The variable family history of heart
disease (famhist) is binary (yes or no).

Classification as a special case of regression

From the very definition we see that classification is a special
case of regression. It is a special case since the output is
restricted to a small discrete set. So it might appear that
there is not much new and that we should simply apply our
standard regression techniques to this special case.
E.g., we can assign y = 0 for C1 and y = 1 for C2 and,
given a training set Strain, we can use (regularized) least-
squares to learn a prediction function fStrain for this regression
problem. To convert the regression into a classification it is
then natural to decide on class C1 if fStrain (x) < 0.5 and C2
if fStrain (x) > 0.5.
In the figure below this approach is applied to the credit-card
default problem. To keep things simple we use as feature only
the balance (as we have seen in a previous plot the second
feature (the income) does not contain much information).
We think of y = 0 as “no default” and y = 1 as “default”.
The dots we see corresponds to the various data points and
all dots are either on the line y = 0 or y = 1. The horizontal
axis corresponds to the input x, the balance.
In the figure the output y is labeled as probability. This is
just a convenient way of interpretation. Since the desired
label y is either 0 or 1 we can think of y as the probability
of a default.
So let us now run a regression on the training data Strain. To
keep things simple we run a linear regression and learn the
linear function fStrain (x). The result is the blue curve that is
indicated.
We might want to interpret the value fStrain (x) as the prob-
ability of a default and then assign a label depending on
whether this “probability” is smaller or larger than 0.5. Of
course, this “probability” can be negative or be larger than
1 so such an interpretation has to be taken with a grain of
salt.
 1.0

1.0
| | | | || | ||||| || ||| ||||||||||| ||||| |||||||||||||||||| |||||||| ||||||||||| |||||||||||||||||||||||||||| ||||||||||||||||||||||||| | |||| | | | | | | | | | | || | |||

Probability of Default

Probability of Default
0.8

0.8
0.6

0.6
0.4

0.4
0.2

0.2
0.0

0.0
||||||||||||||||||||||||||||||||||||||||||||||||||||||
||||||||||||||
|||||||||||||||||||||||||||||
|||||||||||||
||||||||||||||||||||||||||||||||||
|||||||
|||||
|||||||||||||||||||
||||||||||||
||||||||||||||||
|||||||||||||
|||||||||||||||||||
|||||
||||
||||||||||||
|||||||||
||||
||||||||||||||
|||||||||||||||||||||||||||||||||||||||||||||||||||||
|||||||||||||||||||||||||
|||||||||||||||||||||||
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| |||||| || || ||||| | | || ||||||||||||||||||||||||||||||||||||||||||||||||||||||
||||||||||||||||
||||||||||||||||||||||||||||
|||||||||||||
|||||||||||||||
||||||||||||||||||
||||
|||||||||||
|||||
|||||||||||||||||||
||||||||||||||
||||||||||||||||
||
|||||||||||
|||||||||||||||||||
|||||
||||
||||||||||||||
|||||||||
||||
||||||||||||||
||||||||||||||||||||

0 500 1000 1500 2000 2500 0 500 1000

Balance B

Why classification as not just a special case

of regression
It is not hard to see that this method can lead to questionable
results. Even if the cases and controls are well separated, the
general “position” of the line will depend crucially on how
many points are in each class and where these points lie. E.g.,
if we add a few points with y = 1 and a very large balance
this will shift the curve significantly even though only a few
points changed. This is clearly not a desirable property.
Why does this happen? The squared loss function that we
used for regression is not a good match to our objective. We
would like that the fraction of misclassified cases is small.
But the mean-squared error is only very loosely related to
this objective. In particular, the mean-squared error counts
positive and negative deviations from the class label equally
bad, although only one of them can potentially lead to a
misclassification. If we do have a very small mean-squared
error then indeed we can guarantee a small classification error
but the opposite is not true – a regression function can have
arbitrarily large mean-squared error even though the fraction
of misclassified cases is arbitrarily small. We therefore might
have to work “much harder” than we should in bringing down
the mean-squared error and so to have a guarantee on the
misclassification error.
Based on the above observation, we see that classification
is not just a special form of regression with a simple loss
function like the mean-squared error.

Some basic ideas of how to perform

classification
Many different approaches have been developed over the
years of how to efficiently perform classification. Our aim
right now is not to give an exhaustive account of all possible
techniques. Rather, let us quickly discuss some basic ideas.
We will come back and discuss those in more detail in later
lectures.

Nearest Neighbor
In some cases it is reasonable to postulate that inputs that
are “close” are also likely to have the same label attached.
Here “close” might e.g. be measured by the Euclidean dis-
tance. If we believe that this assumption is good, then, given
an input x and a training set Strain, we can look for that point
x∗ which is closest to x and an element of Strain and then
output y ∗, the label attached to x∗.
The good point about such a classifier is that it might work
well even in cases where the decision boundaries are very ir-
regular (see the Figure 2.3 below). But, as we will discuss
in a later lecture, such a scheme fails miserably in high di-
mensions since in this case the geometry renders the notion
Elements of Statistical Learning (2nd Ed.) Hastie,
c Tibshirani & Friedman 2009 Chap 2

of “close by” meaningless.

1-Nearest Neighbor Classifier

.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. o
.. .. .. .. ..
.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
.. ..
... ... o ..... ..... ..... .....o..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... o..... ..... ..... .....o..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... .....
..
..
..
.. ..
... ...
.. ..
... ...
.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. o .. .. .. ..
. . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
... ... .. .. .. .. o .. .. .. .. ..o.. .. .. .. .. .. .. ..o
..
.. o.. .. ... ...o
... ... .. .. .. .. .. .. .. .. .. .. .. .. .. o ... ... .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
.. .. .. .. .. .. .. .. ... ... ... ... ... ... ... ... ... ... ...o
.. .. .. ... ... ...o... ... ... ...o
.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
.. ..
o .... o.... .... .... .... .... o
.. .. .. .. .. .. .. .. .. .. .. .. .. .. ... ...o .. .. .. .. .. .. .. .. .. ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
.. ..
o
... ... .. .. ..o.. .. .. .. .. .. .. .. .. .. ..o .. .. .. .. ..o
... ... ... ...
.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
o o... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...o... ... ... o... ... ...o... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
.. ..
. . . . . . .. .. .. .. ..
... ... . . . . . . . . . . .o. . . . . . . . . . . . . . . . . . . . . .. ..o.. o
..
... ... ... ... .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
o .... .... .... .... .... .... .... .... .... .... .... .... .... o.... .... .... .... .... ....o.... .... .... .... ....oo.... .... .... .... .... .... .... ....o
.. ..
..
.. .. .. .. .. o.. ..o.. .... .... .... .... .... .... .... .... .... .... .... .... ....o.... .... .... .... .... .... .... .... .... .... .... .... .... ....
.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..o.. .. .. .. ..o.. .. o
.. .. .. . . .. .. .. ... ... ... o
.. ..
... ... .. .. .. .. .. .. .. .. .. .. .. ..o.. .. .. .. o .. .. .. o .. .. .. .. .. .. .. .. .. ... ... o
... ... .o.. .. .. o
... ... . .. ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
.. .. .. .. .. .. o . . ........o ... ... ... ... ... ... ... ... ... ... ...o... ... ... ... ... ... ...
.. .. .. .. .. .. ... ... ... ... ... ... ... ... ... o . . . . . . . . . . .. .. .. ... ... ... ... ... ... ... ...o
.. .. . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. o
.. .. .. .. ..
. .. .. .. o
. .. .. .. .. ...o . .o. .. .. .. .. ..o.. .. .. ..o..o
.. . .. .. .. o
... ... .. .. .. .. .. .. .. o . . .o .. .. .. ... o ... ... ... ... .. .. .. .. ... ... ... o .. .. .. .. .. .. ..o .. .. .. .. .. .. .. ..
.. .. .. .. ... ...o... ... ... ... ... ... ... ... ... ... ... o
.. ... ... ... ... ... ... ... o
.. .. .. .. .. .. .. .. .. ... ... ... ...o .. .oo. o. ...oo... o .. .. .. .. . . . .o . . . .. .. .. .. .. .. .. ... ... ... ... ... ... ...o...
o .. .. .. .. .. .. .. .. .. .. .. .. .. ... ... ... o .. . .. .. .. .. .. .. .. .. .. .. .. .. o
.. .. ..o .. o ..o
.. .. .. ... ... ... ... ...o ... ... ... ...o... ... ... ... ... ... ... ... ... ... ... ... o .o .......
... ..... .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..o .. . . .. .. o
. ..o.. o .. o .. .. .. .. .. .. ..o
... ... .. .. .. .. .. ..o
... ... . .. .. .. .. .. .. o .o .. .. .. .. o .. .. .. .. .. .. .. .. ... ... ... ... ... ... ... ...
..o . . . . . .o . . . . . . . . . . . . .o. . . . . .o
.. o .. .. .. .. .. .. ..o o . .
. . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. ..
.. .. .. ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...o .. .. .o. . .o.. ..o .. .. .. ... ... ... ... ...o... ... ... ...o... ... ... ... ... ... o
.. .. .. .. .. .. .. .. .. .. .. ... ... ... ... o .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ...o
.. . .. .. .. .. .. .. .. .. .. .. .. .. ..
... ..... .. .. .. .. .. .. .. .. .. .. .. .. .. ..o .. ... ... .. .. o... ... .. ..o.. .. .. .. .. .. .. .. .. .. ..
.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..o .o . .o . . . .. o . . . . . . . .. .. .. .. .. .. .. o . . . . . . . . . . . . . . . . .o
. .. .. .. ... ... ... ... o
.. .. .. .. .. .. .. .......
.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ... ... ... ... ...o... o . . . . .. .. .. .. ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
.. ... ... ... ... o
.. .. .. .. .. . . .. o
.. ..
.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. o . . . .o .o . ... ... ... ... ... ... ... ... o
... ...
.. .... .. .. .. .. .. .. .. .. .. .. .. .. o
..
.. .. .. .. .. .. ... ... ... ... o ... ...
.. o
.. .... .. .. .. ... ... ... ... ...o
. . . .. ... o
.
... ...
. ... o
.. ..
o.. .. .. o.. .. .. ..o.... .... o.... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... ....
..o
.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. o . .o. . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
. . .. ... ... ... o
..
.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..o
... ... . . . . . .o .o..o.. ..o
... ... . .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..o
... ...
. .. o
.. ..............
.. .... .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ... ... ... ... ... ... ... o
.. . . . .. .. .. o
.. .. . .o . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. ..
.. ..
.. .... .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ... ...o... ... ... ... ...o... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
.. .. .. .. ..
.. .. .. .. .. .. .. .. .. .. .. .. .. .. ..o.. .. .. .. .. .. .. .. .. .. .. .. .. .. ..o.. ..o
... ... .. ... ... o .. .. .. o
... ... . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
..
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...o .. .. .. ..o.. .. .. ... ... ... ... ... ... ... ... o
. o .. .. .. .. .. .. .. ..o .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
. . . .. .. .. .. .. .. .. o
.. .. .. .. .. ..
.. .. .. .. .. .. .. .. .. .. .. .. o
... ... . . . . . .o
..
.. ... ... . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
... ...
.. .. .. .. .. .. .. .. .. .. .. .. ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..o
.. .. .. .. .. .. .. .. .. .. ..o.. .. .. .. .. .. o
.. .. .. ..
..o.. .. .. .. .. o .. .. .. ..o.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
.. ... ... ... ... ... ... ... ... ... ...o... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
.. .. .. .. .. .. ..
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... o .. o
.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ... ... o
... ... .. ... ... ... ...
. . . . .. .. o
. . .. ... ... ... ...o
.. . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
............................o
.. .. .. .. . . .. .. ... ... o .. ..
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...o... ... ... ... ... ...o
.. .....................
... ... .. .. .. . . .. .. .. o ... ... . . . . .. .. ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
.. ... ...
.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ... ... ... ... ... ... ... ... ... ... ... o
.. .. .. .. . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ... ...o
.. .. .. .. .. .. .. o. . . . . . . . . . . . . . . . .
.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
... ... .. ... ... ... ...
.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..o.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. o
.. .. .. .. .. .. .. ................
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
..
... ... .. .. .. ... ... .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
.. .. .. ... ... .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
.. . .. .. o

FIGURE 2.3. The same classification example in two

dimensions as in Figure 2.1. The classes are coded as
a binary variable (BLUE = 0, ORANGE = 1), and then
predicted by 1-nearest-neighbor classification.

There are many natural generalizations of this concept. In-

stead of using a single neighbor we can use lets say the k
nearest neighbors or we can take a weighted linear combina-
tion of elements in our neighborhood. The latter idea leads
to to smoothing kernels.

Linear decision boundaries

One starting point is to assume that decision boundaries are
linear (hyperplanes). Consider e.g. a binary classification
problem and look at schemes where the boundary between
the two classes is a hyperplane. We can ask how to pick this
boundary. To keep things simple, assume that there exists a
“separating hyperplane” i.e., a hyperplane so that no point
in the training set is misclassified.
In general, there might be many hyperplanes that do the
trick (assuming there is at least one). So which one should
we pick?
One idea is to pick a hyperplane so that the decision has
as much “robustness/margin” with respect to the training
set as is possible. I.e., if we slightly change the training set
by “wiggling” the inputs we would like that the number of
misclassifications stays low.
In the figure below (taken from Wikipedia) we see that H1
does not separate the data, but H2 and H3 do. Between
H2 and H3, H3 is preferable, since it has a larger “margin.”
This idea will lead us to support vector machines (SVM) as
well as logistic regression.
Non-linear decision boundaries
In many cases linear decision boundaries will not allow us
to separate the data and non-linearities are needed. One
option is to augment the feature vector with some non-linear
functions. (The kernel trick is a method of doing this in an
efficient way. We will learn about this at a later stage.)
Another way is to find an appropriate non-linear transform
of the input so that the transformed input is then linearly
separable. This is what is done when we are using neural
networks.

Optimal classification for known generating model

It is instructive to think about how one could classify in an
optimal fashion, i.e., how one could minimize the probabil-
ity of misclassification, if the distribution of the generating
model was known. To be concrete, assume that we know the
joint distribution

p(x, y)

and that y takes on elements in a discret set Y.

Given the “observation” (input x), let ŷ(x) be our estimate
of the class label. What is the optimum choice for this func-
tion? Note that our estimate is only a function of the input
x. Further, for a given input x, the probability that the ”cor-
rect” label is y is p(y | x), according to our model. So if our
estimate is ŷ(x) then we will be correct a fraction p(ŷ(x) | x)
of the time. We conclude that if we want to maximize the
probability of guessing the correct label then we should chose
the decision rule

ŷ(x) = argmaxy∈Y p(y | x).

This is called the maximum a-posteriori (MAP) criterion

since we maximize the posterior probability (it is called pos-
terior probability, since it is the probability of a class label
after we have observed the input x.)
The probability of a correct guess is then the average (over
all inputs x) of this probability, i.e.,
Z
P{ŷ(x) = y} = p(x)p(ŷ(x) | x)dx.

In practice we do not know the joint distribution p(x, y).

But we could use such an approach by using the data itself
to learn the distribution (perhaps by assuming that the dis-
tribution is Gaussian and then just fitting the parameters
from the data).