LEARNING

FROM
DATA

The book website AMLbook. com

contains supporting material for
instructors and readers.

LEARNING FROM DATA

A SHORT COURSE

Yaser S . Abu-Mostafa

California Institute of Technology

Malik Magdon-Ismail

Rensselaer Polytechnic Institute

Hsuan-Tien Lin

National Taiwan University

AMLbook.com

Yaser S. Abu 1/fostafa

Departments of Electrical Engineering
and Computer Science
California Institute of Technology
Pasadena, CA 9 1 125, USA

yasercaltech.edu

Malik Magdon Ismail

Department of Computer Science
Rensselaer Polytechnic Institute
Troy, NY 12180, USA

[email protected]

Hsuan Tien Lin

Department of Computer Science
and Information Engineering
National Taiwan University
Taipei, 106, Taiwan

htlincsie.ntu.edu.tw

ISBN 1 0: 1 60049 006 9

ISBN 13:978 1 60049 006 4

@2012 Yaser S. Abu Mostafa, Malik Magdon Ismail, Hsuan Tien Lin.

1.10

All rights reserved. This work may not be translated or copied in whole or in part
without the written permission of the authors. No part of this publication may
be reproduced, stored in a retrieval system, or transmitted in any form or by any
means-electronic, mechanical, photocopying, scanning, or otherwise-without prior
written permission of the authors, except as permitted under Section 107 or 108 of
the 1976 United States Copyright Act.
Limit of Liability/Disclaimer of Warranty: While the authors have used their best
efforts in preparing this book, they make no representation or warranties with re
spect to the accuracy or completeness of the contents of this book and specifically
disclaim any implied warranties of merchantability or fitness for a particular purpose.
No warranty may be created or extended by sales representatives or written sales
materials. The advice and strategies contained herein may not be suitable for your
situation. You should consult with a professional where appropriate. The authors
shall not be liable for any loss of profit or any other commercial damages, including
but not limited to special, incidental, consequential, or other damages.
The use in this publication of tradenames, trademarks, service marks, and similar
terms, even if they are not identified as such, is not to be taken as an expression of
opinion as to whether or not they are subject to proprietary rights.
This book was typeset by the authors and was printed and bound in the United
States of America.

To our teachers) and to our students

P reface
This book is designed for a short course on machine learning. It is a short
course, not a hurried course. From over a decade of teaching this material, we
have distilled what we believe to be the core topics that every student of the
subject should know. We chose the title 'learning from data' that faithfully
describes what the subject is about, and made it a point to cover the topics in
a story-like fashion. Our hope is that the reader can learn all the fundamentals
of the subject by reading the book cover to cover.
Learning from data has distinct theoretical and practical tracks. If you
read two books that focus on one track or the other, you may feel that you
are reading about two different subjects altogether. In this book, we balance
the theoretical and the practical, the mathematical and the heuristic. Our
criterion for inclusion is relevance. Theory that establishes the conceptual
framework for learning is included, and so are heuristics that impact the per
formance of real learning systems. Strengths and weaknesses of the different
parts are spelled out . Our philosophy is to say it like it is: what we know,
what we don't know, and what we partially know.
The book can be taught in exactly the order it is presented. The notable
exception may be Chapter 2, which is the most theoretical chapter of the book.
The theory of generalization that this chapter covers is central to learning
from data, and we made an effort to make it accessible to a wide readership.
However, instructors who are more interested in the practical side may skim
over it, or delay it until after the practical methods of Chapter 3 are taught.
You will notice that we included exercises (in gray boxes) throughout the
text. The main purpose of these exercises is to engage the reader and enhance
understanding of a particular topic being covered. Our reason for separating
the exercises out is that they are not crucial to the logical flow. Nevertheless,
they contain useful information, and we strongly encourage you to read them,
even if you don't do them to completion. Instructors may find some of the
exercises appropriate as 'easy' homework problems, and we also provide ad
ditional problems of varying difficulty in the Problems section at the end of
each chapter.
To help instructors with preparing their lectures based on the book, we
provide supporting material on the book's website ( AMLbook. corn) . There is
also a forum that covers additional topics in learning from data. We will
vii

PREFACE

discuss these further in the Epilogue of this book.

Acknowledgment ( in alphabetical order for each group ) : We would like to
express our gratitude to the alumni of our Learning Systems Group at Caltech
who gave us detailed expert feedback: Zehra Cataltepe, Ling Li, Amrit Pratap,
and Joseph Sill. We thank the many students and colleagues who gave us useful
feedback during the development of this book, especially Chun-Wei Liu. The
Caltech Library staff, especially Kristin Buxton and David McCaslin, have
given us excellent advice and help in our self-publishing effort. We also thank
Lucinda Acosta for her help throughout the writing of this book.
Last, but not least, we would like to thank our families for their encourage
ment, their support, and most of all their patience as they endured the time
demands that writing a book has imposed on us.
Yaser S. Abu-Mostafa, Pasadena, California.
Malik Magdon-Ismail, Troy, New York.
Hsuan-Tien Lin, Taipei, Taiwan.

March, 2012.

viii

Contents
Prefa e

vii

1 The Learning Problem

1.1

Problem Setup
1.1.1

1.2

1.4

1.5

Components of Learning . . . . . . . . . . . . . . . . . .

1.1.2

A Simple Learning Model . . . . . . . . . . . . . . . . .

1.1.3

Learning versus Design

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . .

11

Supervised Learning . . . . . . . . . . . . . . . . . . . .

11

Types of Learning
1.2.1

1.3

. . . . . . . . . . . . . . . . . . . . . . . . . . .

1.2.2

Reinfor ement Learning . . . . . . . . . . . . . . . . . .

12

1.2.3

Unsupervised Learning . . . . . . . . . . . . . . . . . . .

13

1.2.4

Other Views of Learning . . . . . . . . . . . . . . . . . .

14

Is Learning Feasible? . . . . . . . . . . . . . . . . . . . . . . . .

15

1.3.1

Outside the Data Set . . . . . . . . . . . . . . . . . . . .

16

1.3.2

Probability to the Res ue

18

1.3.3

Feasibility of Learning . . . . . . . . . . . . . . . . . . .

24

Error and Noise . . . . . . . . . . . . . . . . . . . . . . . . . . .

27

. . . . . . . . . . . . . . . . .

1.4.1

Error Measures . . . . . . . . . . . . . . . . . . . . . . .

28

1.4.2

Noisy Targets . . . . . . . . . . . . . . . . . . . . . . . .

30

Problems

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2 Training versus Testing

2.1

2.2

2.3

33

39

Theory of Generalization . . . . . . . . . . . . . . . . . . . . . .

39

2.1.1

41

Ee tive Number of Hypotheses

. . . . . . . . . . . . .

2.1.2

Bounding the Growth Fun tion . . . . . . . . . . . . . .

46

2.1.3

The VC Dimension . . . . . . . . . . . . . . . . . . . . .

50

2.1.4

The VC Generalization Bound

. . . . . . . . . . . . . .

53

Interpreting the Generalization Bound . . . . . . . . . . . . . .

55

2.2.1

Sample Complexity . . . . . . . . . . . . . . . . . . . . .

57

2.2.2

Penalty for Model Complexity

58

2.2.3

The Test Set

2.2.4

Other Target Types

. . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Approximation-Generalization Tradeo
ix

. . . . . . . . . . . . .

59
61
62

Contents

2.4

2.3.1

Bias and Varian e

2.3.2

The Learning Curve

Problems

. . . . . . . . . . . . . . . . . . . . .

66

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

69

3 The Linear Model

3.1

3.2

3.3

3.4

3.5

77

Linear Classi ation

3.1.1

. . . . . . . . . . . . . . . . . . . . . . . .

Non-Separable Data

4.2

4.3

79

Linear Regression . . . . . . . . . . . . . . . . . . . . . . . . . .

82

3.2.1

The Algorithm

. . . . . . . . . . . . . . . . . . . . . . .

84

3.2.2

Generalization Issues . . . . . . . . . . . . . . . . . . . .

87

Logisti Regression . . . . . . . . . . . . . . . . . . . . . . . . .

88

3.3.1

Predi ting a Probability . . . . . . . . . . . . . . . . . .

89

3.3.2

Gradient Des ent . . . . . . . . . . . . . . . . . . . . . .

93

Nonlinear Transformation . . . . . . . . . . . . . . . . . . . . .

99

3.4.1

The

3.4.2

Computation and Generalization . . . . . . . . . . . . . 104

Problems

Spa e

. . . . . . . . . . . . . . . . . . . . . . . .

119
. . . . . . . . . . . . . . . . . . 119

4.1.1

A Case Study: Overtting with Polynomials

4.1.2

Catalysts for Overtting . . . . . . . . . . . . . . . . . . 123

Regularization

. . . . . . 120

. . . . . . . . . . . . . . . . . . . . . . . . . . . 126

4.2.1

A Soft Order Constraint . . . . . . . . . . . . . . . . . . 128

4.2.2

Weight De ay and Augmented Error . . . . . . . . . . . 132

4.2.3

Choosing a Regularizer: Pill or Poison?

. . . . . . . . . 134

Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
The Validation Set . . . . . . . . . . . . . . . . . . . . . 138

4.3.2

Model Sele tion . . . . . . . . . . . . . . . . . . . . . . . 141

4.3.3

Cross Validation

4.3.4

Theory Versus Pra ti e

Problems

. . . . . . . . . . . . . . . . . . . . . . 145
. . . . . . . . . . . . . . . . . . 151

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154

5 Three Learning Prin iples

5.1

99

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

When Does Overtting O ur?

4.3.1

4.4

77

. . . . . . . . . . . . . . . . . . . .

4 Overtting
4.1

62

. . . . . . . . . . . . . . . . . . . .

O am's Razor

167

. . . . . . . . . . . . . . . . . . . . . . . . . . . 167

5.2

Sampling Bias . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171

5.3

Data Snooping

5.4

Problems

. . . . . . . . . . . . . . . . . . . . . . . . . . . 173

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178

Epilogue

181

Further Reading

183
x

Contents

Appendix Proof of the VC Bound

187

A.1

Relating Generalization Error to In-Sample Deviations . . . . . 188

A.2

Bounding Worst Case Deviation Using the Growth Fun tion . . 190

A.3

Bounding the Deviation between In-Sample Errors

. . . . . . . 191

Notation

193

Index

197

xi

NOTATION

A complete table of the notation used in

this book is included on page 193, right
before the index of terms. We suggest
referring to it as needed.

xii

Chapter

The Learning P ro bl em
If you show a picture to a three-year-old and ask if there is a tree in it, you will
likely get the correct answer. If you ask a thirty-year-old what the definition
of a tree is, you will likely get an inconclusive answer. We didn't learn what
a tree is by studying the mathematical definition of trees. We learned it by
looking at trees. In other words, we learned from 'data'.
Learning from data is used in situations where we don't have an analytic
solution, but we do have data that we can use to construct an empirical solu
tion. This premise covers a lot of territory, and indeed learning from data is
one of the most widely used techniques in science, engineering, and economics,
among other fields.
In this chapter, we present examples of learning from data and formalize
the learning problem. We also discuss the main concepts associated with
learning, and the different paradigms of learning that have been developed.
1.1

Problem Setup

What do financial forecasting, medical diagnosis, computer vision, and search

engines have in common? They all have successfully utilized learning from
data. The repertoire of such applications is quite impressive. Let us open the
discussion with a real-life application to see how learning from data works.
Consider the problem of predicting how a movie viewer would rate the
various movies out there. This is an important problem if you are a company
that rents out movies, since you want to recommend to different viewers the
movies they will like. Good recommender systems are so important to business
that the movie rental company Netflix offered a prize of one million dollars to
anyone who could improve their recommendations by a mere 103.
The main difficulty in this problem is that the criteria that viewers use to
rate movies are quite complex. Trying to model those explicitly is no easy task,
so it may not be possible to come up with an analytic solution. However, we
1

1 . 1 . PROBLEM SETUP

1 . THE LEARNING PROBLEM

viewer

add contributions
from each factor

:t\fatch movie and

viewer factors

movie

Figure

1.1: A model for how a viewer rates a movie

know that the historical rating data reveal a lot about how people rate movies,
so we may be able to construct a good empirical solution. There is a great
deal of data available to movie rental companies, since they often ask their
viewers to rate the movies that they have already seen.
Figure 1.1 illustrates a specific approach that was widely used in the
million-dollar competition. Here is how it works. You describe a movie as
a long array of different factors, e.g. , how much comedy is in it, how com
plicated is the plot, how handsome is the lead actor, etc. Now, you describe
each viewer with corresponding factors; how much do they like comedy, do
they prefer simple or complicated plots, how important are the looks of the
lead actor, and so on. How this viewer will rate that movie is now estimated
based on the match/mismatch of these factors. For example, if the movie is
pure comedy and the viewer hates comedies, the chances are he won't like it.
If you take dozens of these factors describing many facets of a movie's content
and a viewer's taste, the conclusion based on matching all the factors will be
a good predictor of how the viewer will rate the movie.
The power of learning from data is that this entire process can be auto
mated, without any need for analyzing movie content or viewer taste. To do
so, the learning algorithm 'reverse-engineers' these factors based solely on pre2

1 . THE LEARNING PROBLEM

1 . 1 . PROBLEM SETUP

vious ratings. It starts with random factors, then tunes these factors to make
them more and more aligned with how viewers have rated movies before, until
they are ultimately able to predict how viewers rate movies in general. The
factors we end up with may not be as intuitive as 'comedy content', and in
fact can be quite subtle or even incomprehensible. After all, the algorithm is
only trying to find the best way to predict how a viewer would rate a movie,
not necessarily explain to us how it is done. This algorithm was part of the
winning solution in the million-dollar competition.
1. 1. 1

Components of Learning

The movie rating application captures the essence of learning from data, and
so do many other applications from vastly different fields. In order to abstract
the common core of the learning problem, we will pick one application and
use it as a metaphor for the different components of the problem. Let us take
credit approval as our metaphor.
Suppose that a bank receives thousands of credit card applications every
day, and it wants to automate the process of evaluating them. Just as in the
case of movie ratings, the bank knows of no magical formula that can pinpoint
when credit should be approved, but it has a lot of data. This calls for learning
from data, so the bank uses historical records of previous customers to figure
out a good formula for credit approval.
Each customer record has personal information related to credit , such as
annual salary, years in residence, outstanding loans, etc. The record also keeps
track of whether approving credit for that customer was a good idea, i.e . , did
the bank make money on that customer. This data guides the construction of
a successful formula for credit approval that can be used on future applicants.
Let us give names and symbols to the main components of this learning
problem. There is the input x (customer information that is used to make
a credit decision) , the unknown target function f: X -- Y (ideal formula for
credit approval) , where X is the input space ( set of all possible inputs x) , and Y
is the output space (set of all possible outputs, in this case just a yes/no deci
sion) . There is a data set D of input-output examples (x1 , Y1 ) ,
, (xN , YN ) ,
where Yn = f (xn ) for n = 1, . . . , N (inputs corresponding to previous customers
and the correct credit decision for them in hindsight). The examples are often
referred to as data points. Finally, there is the learning algorithm that uses the
data set D to pick a formula g: X -- Y that approximates f. The algorithm
chooses g from a set of candidate formulas under consideration, which we call
the hypothesis set 1-l . For instance, 1-l could be the set of all linear formulas
from which the algorithm would choose the best linear fit to the data, as we
will introduce later in this section.
When a new customer applies for credit, the bank will base its decision
on g (the hypothesis that the learning algorithm produced) , not on f (the
ideal target function which remains unknown) . The decision will be good only
to the extent that g faithfully replicates f. To achieve that , the algorithm

1 . 1 . PROBLEM SETUP

1 . THE LEARNING PROBLEM

UNKNOWN TARGET FUNCTION

(ideal

f :X

cred'il

approval forrn'Ulo)

TRAINING EXAMPLES

(xN, YN)

FINAL
HYPOTHESIS
g f

(learned credit approval forrn'Ula)

HYPOTHESIS SET
1-

(set of cand,idate form'alas)

Figure

1.2:

Basic setup of the learning problem

chooses g that best matches f on the training examples of previous customers,

with the hope that it will continue to match f on new customers. Whether
or not this hope is justified remains to be seen. Figure 1.2 illustrates the
components of the learning problem.
Exercise 1.1
Express each o f the following tasks i n t h e framework o f learning from d ata by
specifying the i nput space X, output space Y, target function f:
Y.
a n d the specifics of the data set that we will learn from.
(a) Med ica l diagnosis: A patient wal ks i n with a medical h istory and some
symptoms, a n d you want to identify the problem.
(b) H andwritten digit recognition (for example postal zip code recognition
for m a i l sorting) .
( c) Determi ning if a n email is spam or not.
( d) P redicting how an electric load varies with price, temperature, and
day of the week.
( e) A problem of i nterest to you for which there is no a n alytic sol ution,
but you have data from which to construct an empirica l sol ution .

1 . 1 . PROBLEM SETUP

1 . THE LEARNING PROBLEM

We will use the setup in Figure 1.2 as our definition of the learning problem.
Later on, we will consider a number of refinements and variations to this basic
setup as needed. However, the essence of the problem will remain the same.
There is a target to be learned. It is unknown to us. We have a set of examples
generated by the target. The learning algorithm uses these examples to look
for a hypothesis that approximates the target.
1. 1.2

A Simple Learning Model

Let us consider the different components of Figure 1.2. Given a specific learn
ing problem, the target function and training examples are dictated by the
problem. However, the learning algorithm and hypothesis set are not. These
are solution tools that we get to choose. The hypothesis set and learning
algorithm are referred to informally as the learning model.
Here is a simple model. Let X =]Rd be the input space, where JRd is the
d-dimensional Euclidean space, and let Y = { + 1, - 1 } be the output space,
denoting a binary (yes/no) decision. In our credit example, different coor
dinates of the input vector x E JRd correspond to salary, years in residence,
outstanding debt, and the other data fields in a credit application. The bi
nary output y corresponds to approving or denying credit. We specify the
hypothesis set 1{ through a functional form that all the hypotheses h E 1{
share. The functional form h(x) that we choose here gives different weights to
the different coordinates of x, reflecting their relative importance in the credit
decision. The weighted coordinates are then combined to form a 'credit score'
and the result is compared to a threshold value. If the applicant passes the
threshold, credit is approved; if not, credit is denied:

i=I:l
i=I:l
d

Approve credit if

Deny credit if

WiXi

> threshold,

WiXi

<

threshold.

This formula can be written more compactly as

(1.1)
where x i , , x d are the components of the vector x; h(x) = + 1 means 'ap
prove credit' and h(x) = - 1 means 'deny credit'; sign(s) = + 1 if s > 0 and
sign(s) = - 1 if s < 0. 1 The weights are w1, , wd , and the threshold is
determined by the bias term b since in Equation (1.1) , credit is approved if
WiXi > - b.
This model of 1{ is called the perceptron, a name that it got in the context
of artificial intelligence. The learning algorithm will search 1{ by looking for

I::=l

1 The value of sign (s) whens

0 is a simple technicality that we ignore for the moment.

1 . THE LEARNING PROBLEM

1 . 1 . PROBLEM SETUP

( a) Misclassified data

( b) Perfectly classified data

Figure 1 .3: Perceptron classification of linearly separable data in a two

dimensional input space ( a) Some training examples will be misclassified
( blue points in red region and vice versa) for certain values of the weight
parameters which define the separating line. ( b) A final hypothesis that
classifies all training examples correctly.
is + 1 and
is - 1 . )
weights and bias that perform well o n the data set. Some o f the weights
, Wd may end up being negative, corresponding to an adverse effect on
credit approval. For instance, the weight of the 'outstanding debt' field should
come out negative since more debt is not good for credit. The bias value b
may end up being large or small, reflecting how lenient or stringent the bank
should be in extending credit. The optimal choices of weights and bias define
the final hypothesis g E 1-l that the algorithm produces.
w1,

Exercise 1. 2
S uppose that we use a perceptron to detect spam messages. Let's say
that each email message is represented by the frequency of occurrence of
if the message is considered spa m .
keywords, a nd the output is

( a ) Can you t h i n k o f some keywords that wil l e n d u p with a large positive

weight in the perceptron?

( b ) H ow a bout keywords that wil l get a negative weight?

( c) What parameter in the perceptron d i rectly affects how many border
line messages end up being classified as spam ?

Figure 1.3 illustrates what a perceptron does i n a two-dimensional case (d = 2) .

The plane is split by a line into two regions, the + 1 decision region and the - 1
decision region. Different values for the parameters w1, w2, b correspond to
different lines w1x1 + w 2 x 2 + b = 0. If the data set is linearly separable, there
will be a choice for these parameters that classifies all the training examples
correctly.
6

1 . THE LEARNING PROBLEM

1 . 1 . PROBLEM S ETUP

To simplify the notation of the perceptron formula, we will treat the bias b
as a weight wo = b and merge it with the other weights into one vector
w = [w0, w 1 ,
, wd]T, where T denotes the transpose of a vector, so w is a
column vector. We also treat x as a column vector and modify it to become x =
[x0, xi, , xd]T, where the added coordinate x0 is fixed at x0 = 1 . Formally
speaking, the input space is now

With this convention, wTx =

ten in vector form as

=O
h (x)

WiXi,

and so Equation

sign(wTx) .

(1.1) can be rewrit

(1.2)

We now introduce the perceptron learning algorithm (PLA) . The algorithm

will determine what w should be, based on the data. Let us assume that the
data set is linearly separable, which means that there is a vector w that
makes (1.2) achieve the correct decision h (xn ) = Yn on all the training exam
ples, as shown in Figure 1.3.
Our learning algorithm will find this w using a simple iterative method.
Here is how it works. At iteration t, where t = 0, 1, 2, . . . , there is a current
value of the weight vector, call it w(t) . The algorithm picks an example
from (x1 , Y1 )
(xN , YN) that is currently misclassified, call it (x(t) , y (t) ) , and
uses it to update w(t) . Since the example is misclassified, we have y ( t ) #
sign(wT(t)x(t) ) . The update rule is

w(t +

1) = w(t) + y (t)x(t) .

(1.3)

This rule moves the boundary in the direction of classifying x(t) correctly, as
depicted in the figure above. The algorithm continues with further iterations
until there are no longer misclassified examples in the data set .
7

1 . THE LEARNING PROBLEM

Exercise

1 . 1 . PROBLEM SETUP

1.3

The weight u pdate rule i n {1.3) has the n ice interpretation that it moves
in the direction of classifying x(t) correctly.
(a) Show that y(t)wT(t)x(t)
(b) S how that y(t)wT(t

<

0. [Hint: x(t) is misclassified by w(t).]

l)x(t)

>

y(t)wT(t)x(t). [Hint: Use (1.3).]

( c) As far as classifying x(t) is concerned, argue that the move from w(t)
to w(t + 1) is a move ' i n the right direction ' .

Although the update rule in ( 1 . 3) considers only one training example at a

time and may 'mess up' the classification of the other examples that are not
involved in the current iteration, it turns out that the algorithm is guaranteed
to arrive at the right solution in the end. The proof is the subject of Prob
lem 1.3. The result holds regardless of which example we choose from among
the misclassified examples in (x1, Y1 ) (xN, YN) at each iteration, and re
gardless of how we initialize the weight vector to start the algorithm. For
simplicity, we can pick one of the misclassified examples at random ( or cycle
through the examples and always choose the first misclassified one) , and we
can initialize w(O) to the zero vector.
Within the infinite space of all weight vectors, the perceptron algorithm
manages to find a weight vector that works, using a simple iterative process.
This illustrates how a learning algorithm can effectively search an infinite
hypothesis set using a finite number of simple steps. This feature is character
istic of many techniques that are used in learning, some of which are far more
sophisticated than the perceptron learning algorithm.
Exercise 1 .4
Let us create our own target function f a nd data set 1) a n d see how the
perceptron learning a lgorithm works. Take d = 2 so you can visua lize the
problem , a nd choose a random l i ne i n the plane as you r target function ,
where o ne side of the line m a ps to 1 a nd the other m a ps to - 1. Choose
the i n puts Xn of the data set as random points in the pla ne, a n d eval u ate
the target function on each Xn to get the corresponding output Yn

Now, generate a data set of size 20. Try the perceptron learning a lgorithm
on you r data set a n d see how long it takes to converge a n d how wel l the
fin a l hypothesis g matches you r target f. You can find other ways to play
with this experiment in Problem 1.4.

The perceptron learning algorithm succeeds in achieving its goal; finding a hy

pothesis that classifies all the points in the data set V = { (x1, y1) (xN, yN) }
correctly. Does this mean that this hypothesis will also be successful in classi
fying new data points that are not in V? This turns out to be the key question
in the theory of learning, a question that will be thoroughly examined in this
book.

1 . 1 . PROBLEM S ETUP

1 . THE LEARNING PROBLEM

Size

Size

( b) Learned classifier

(a ) Coin data

Figure 1 .4: The learning approach to coin classification ( a) Training data of

pennies, nickels, dimes, and quarters ( 1 , 5, 10, and 25 cents) are represented
in a size mass space where they fall into clusters. (b) A classification rule is
learned from the data set by separating the four clusters. A new coin will
be classified according to the region in the size mass plane that it falls into.

1. 1. 3

Learning versus Design

So far, we have discussed what learning is. Now, we discuss what it is not. The
goal is to distinguish between learning and a related approach that is used for
similar problems. While learning is based on data, this other approach does
not use data. It is a 'design' approach based on specifications, and is often
discussed alongside the learning approach in pattern recognition literature.
Consider the problem of recognizing coins of different denominations, which
is relevant to vending machines , for example. We want the machine to recog
nize quarters, dimes, nickels and pennies. We will contrast the 'learning from
data' approach and the 'design from specifications' approach for this prob
lem. We assume that each coin will be represented by its size and mass, a
two-dimensional input.
In the learning approach, we are given a sample of coins from each of
the four denominations and we use these coins as our data set . We treat
the size and mass as the input vector, and the denomination as the output.
Figure 1 .4( a) shows what the data set may look like in the input space. There
is some variation of size and mass within each class, but by and large coins
of the same denomination cluster together. The learning algorithm searches
for a hypothesis that classifies the data set well. If we want to classify a new
coin, the machine measures its size and mass, and then classifies it according
to the learned hypothesis in Figure l .4(b) .
In the design approach, we call the United States Mint and ask them about
the specifications of different coins. We also ask them about the number
9

1 . THE LEARNING P ROBLEM

1 . 1 . P ROBLEM SETUP

Size
(a) Probabilistic model of data

Size
(b) Inferred classifier

Figure 1 .5: The design approach to coin classification (a) A probabilistic

model for the size, mass, and denomination of coins is derived from known
specifications. The figure shows the high probability region for each denom
ination ( 1 , 5, 10, and 25 cents) according to the model. (b) A classification
rule is derived analytically to minimize the probability of error in classifying
a coin based on size and mass. The resulting regions for each denomination
are shown.

of coins of each denomination in circulation, in order to get an estimate of

the relative frequency of each coin. Finally, we make a physical model of
the variations in size and mass due to exposure to the elements and due to
errors in measurement. We put all of this information together and compute
the full joint probability distribution of size, mass, and coin denomination
(Figure 1 . 5 ( a) ) . Once we have that joint distribution, we can construct the
optimal decision rule to classify coins based on size and mass (Figure 1 . 5 (b) ) .
The rule chooses the denomination that has the highest probability for a given
size and mass, thus achieving the smallest possible probability of error. 2
The main difference between the learning approach and the design ap
proach is the role that data plays. In the design approach, the problem is well
specified and one can analytically derive f without the need to see any data.
In the learning approach, the problem is much less specified, and one needs
data to pin down what f is.
Both approaches may be viable in some applications, but only the learning
approach is possible in many applications where the target function is un
known. We are not trying to compare the utility or the performance of the
two approaches. We are just making the point that the design approach is
distinct from learning. This book is about learning.
2 This is called Bayes optimal decision theory. Some learning models are based on the
same theory by estimating the probability from data.

10

1 . 2 . TYPES OF LEARNING

1 . THE LEARNING PROBLEM

Exercise 1. 5
Which of the following problems a re more suited for the learning a pproach
and which a re more suited for the d esign approach?
(a) Determining the a ge at which a particular med ica l test should be
performed
(b) Classifying n u m bers into primes a n d non-primes
( c) Detecting potentia l fraud i n credit card charges
( d) Determi ning the time it wou ld ta ke a fal l i ng object to h it the ground
(e) Determining the optima l cycle for traffic lights i n a busy intersection

1. 2

Types of Learning

The basic premise of learning from data is the use of a set of observations to
uncover an underlying process. It is a very broad premise, and difficult to fit
into a single framework. As a result, different learning paradigms have arisen
to deal with different situations and different assumptions. In this section, we
introduce some of these paradigms.
The learning paradigm that we have discussed so far is called supervised
learning. It is the most studied and most utilized type of learning, but it is
not the only one. Some variations of supervised learning are simple enough
to be accommodated within the same framework. Other variations are more
profound and lead to new concepts and techniques that take on lives of their
own. The most important variations have to do with the nature of the data
set.
1.2. 1

Supervised Learning

When the training data contains explicit examples of what the correct output
should be for given inputs, then we are within the supervised learning set
ting that we have covered so far. Consider the hand-written digit recognition
problem ( task (b ) of Exercise 1 . 1) . A reasonable data set for this problem is
a collection of images of hand-written digits, and for each image, what the
digit actually is. We thus have a set of examples of the form ( image , digit ) .
The learning is supervised in the sense that some 'supervisor' has taken the
trouble to look at each input, in this case an image, and determine the correct
output, in this case one of the ten categories {O, 1 , 2, 3, 4, 5, 6 , 7, 8, 9}.
While we are on the subject of variations, there is more than one way that
a data set can be presented to the learning process. Data sets are typically cre
ated and presented to us in their entirety at the outset of the learning process.
For instance, historical records of customers in the credit-card application,
and previous movie ratings of customers in the movie rating application, are
already there for us to use. This protocol of a 'ready' data set is the most
11

1 . THE LEARNING PROBLEM

1 .2. TYPES OF LEARNING

common in practice, and it is what we will focus on in this book. However, it

is worth noting that two variations of this protocol have attracted a significant
body of work.
One is active learning, where the data set is acquired through queries that
we make. Thus, we get to choose a point x in the input space, and the
supervisor reports to us the target value for x. As you can see, this opens
the possibility for strategic choice of the point x to maximize its information
value, similar to asking a strategic question in a game of 20 questions.
Another variation is called online learning, where the data set is given to
the algorithm one example at a time. This happens when we have stream
ing data that the algorithm has to process 'on the run'. For instance, when
the movie recommendation system discussed in Section 1 . 1 is deployed, on
line learning can process new ratings from current users and movies. Online
learning is also useful when we have limitations on computing and storage
that preclude us from processing the whole data as a batch. We should note
that online learning can be used in different paradigms of learning, not just in
supervised learning.
1. 2. 2

Reinforcement Learning

When the training data does not explicitly contain the correct output for each
input, we are no longer in a supervised learning setting. Consider a toddler
learning not to touch a hot cup of tea. The experience of such a toddler
would typically comprise a set of occasions when the toddler confronted a hot
cup of tea and was faced with the decision of touching it or not touching it.
Presumably, every time she touched it, the result was a high level of pain, and
every time she didn't touch it, a much lower level of pain resulted ( that of an
unsatisfied curiosity) . Eventually, the toddler learns that she is better off not
touching the hot cup.
The training examples did not spell out what the toddler should have done,
but they instead graded different actions that she has taken. Nevertheless , she
uses the examples to reinforce the better actions, eventually learning what she
should do in similar situations. This characterizes reinforcement learning,
where the training example does not contain the target output, but instead
contains some possible output together with a measure of how good that out
put is. In contrast to supervised learning where the training examples were of
the form ( input , correct output ) , the examples in reinforcement learning are
of the form
( input , some output , grade for this output ) .
Importantly, the example does not say how good other outputs would have
been for this particular input.
Reinforcement learning is especially useful for learning how to play a game.
Imagine a situation in backgammon where you have a choice between different
actions and you want to identify the best action. It is not a trivial task to
ascertain what the best action is at a given stage of the game, so we cannot
12

1 . THE LEARNING PROBLEM

1. 2. TYPES OF LEARNING

Size

Size

( a) Unlabeled Coin data

( b) Unsupervised learning

Figure 1 .6: Unsupervised learning of coin classification ( a) The same data

set of coins in Figure 1.4 ( a) is again represented in the size mass space, but
without being labeled. They still f all into clusters. ( b) An unsupervised
classification rule treats the four clusters as different types. The rule may
be somewhat ambiguous, as type 1 and type 2 could be viewed as one cluster

easily create supervised learning examples. If you use reinforcement learning

instead, all you need to do is to take some action and report how well things
went, and you have a training example. The reinforcement learning algorithm
is left with the task of sorting out the information coming from different ex
amples to find the best line of play.
1.2 . 3

Unsupervised Learning

In the unsupervised setting, the training data does not contain any output
information at all. We are just given input examples xi, , XN . You may
wonder how we could possibly learn anything from mere inputs. Consider the
coin classification problem that we discussed earlier in Figure 1 .4. Suppose
that we didn't know the denomination of any of the coins in the data set. This
unlabeled data is shown in Figure l .6(a) . We still get similar clusters , but they
are now unlabeled so all points have the same 'color' . The decision regions
in unsupervised learning may be identical to those in supervised learning, but
without the labels (Figure 1 . 6 (b) ) . However, the correct clustering is less
obvious now, and even the number of clusters may be ambiguous.
Nonetheless, this example shows that we can learn something from the
inputs by themselves. Unsupervised learning can be viewed as the task of
spontaneously finding patterns and structure in input data. For instance, if
our task is to categorize a set of books into topics, and we only use general
properties of the various books, we can identify books that have similar prop
erties and put them together in one category, without naming that category.
13

1 . THE LEARNING P ROBLEM

1 . 2. TYPES OF LEARNING

Unsupervised learning can also be viewed as a way to create a higher

level representation of the data. Imagine that you don't speak a word of
Spanish, but your company will relocate you to Spain next month. They
will arrange for Spanish lessons once you are there, but you would like to
prepare yourself a bit before you go. All you have access to is a Spanish radio
station. For a full month, you continuously bombard yourself with Spanish;
this is an unsupervised learning experience since you don't know the meaning
of the words. However, you gradually develop a better representation of the
language in your brain by becoming more tuned to its common sounds and
structures. When you arrive in Spain, you will be in a better position to start
your Spanish lessons. Indeed, unsupervised learning can be a precursor to
supervised learning. In other cases, it is a stand-alone technique.
Exercise

1.6

For each of the following tasks, identify which type of learning is involved
(supervised , reinforcement, or u nsupervised) and the tra in ing data to be
used . I f a task can fit more tha n one type, explain how a nd describe the
tra i n i n g data for each type.
(a) Recommending a book to a user in an online bookstore
(b) Playing tic tac toe
( c) Categorizing movies i nto d ifferent types
( d) Learning to play m usic
( e) Credit l i m it: Deciding the m axi m u m a llowed debt for each ban k cus
tome r

Our main focus in this book will be supervised learning, which is the most
popular form of learning from data.
1. 2 .4

Other Views of Learning

The study of learning has evolved somewhat independently in a number of

fields that started historically at different times and in different domains, and
these fields have developed different emphases and even different jargons. As a
result, learning from data is a diverse subject with many aliases in the scientific
literature. The main field dedicated to the subject is called machine learning,
a name that distinguishes it from human learning. We briefly mention two
other important fields that approach learning from data in their own ways.
Statistics shares the basic premise of learning from data, namely the use
of a set of observations to uncover an underlying process. In this case, the
process is a probability distribution and the observations are samples from that
distribution. Because statistics is a mathematical field, emphasis is given to
situations where most of the questions can be answered with rigorous proofs.
As a result, statistics focuses on somewhat idealized models and analyzes them
in great detail. This is the main difference between the statistical approach
14

1. 3. Is LEARNING FEASIBLE?

1. THE LEARNING PROBLEM

-1

+1

Figure 1 .7: A visual learning problem. The first two rows show the training
examples ( each input x is a 9 bit vector represented visually as a 3 x 3 black
and white array ) . The inputs in the first row have f(x) = - 1 , and the inputs
in the second row have f(x) = + 1. Your task is to learn from this data set
what f is, then apply f to the test input at the bottom. Do you get - 1
o r +1?
to learning and how we approach the subject here. We make less restrictive
assumptions and deal with more general models than in statistics. Therefore,
we end up with weaker results that are nonetheless broadly applicable.
Data mining is a practical field that focuses on finding patterns, correla
tions, or anomalies in large relational databases. For example, we could be
looking at medical records of patients and trying to detect a cause-effect re
lationship between a particular drug and long-term effects. We could also be
looking at credit card spending patterns and trying to detect potential fraud.
Technically, data mining is the same as learning from data, with more empha
sis on data analysis than on prediction. Because databases are usually huge,
computational issues are often critical in data mining. Recommender systems,
which were illustrated in Section 1 . 1 with the movie rating example, are also
considered part of data mining.
1. 3

Is Learning Feasible?

The target function f is the object of learning. The most important assertion
about the target function is that it is unknown. We really mean unknown.
This raises a natural question. How could a limited data set reveal enough
information to pin down the entire target function? Figure 1 . 7 illustrates this
15

1 . 3. Is LEARNING FEASIBLE?

1 . THE LEARNING P ROBLEM

difficulty. A simple learning task with 6 training examples of a 1 target

function is shown. Try to learn what the function is then apply it to the test
input given. Do you get - 1 or + 1? Now, show the problem to your friends
and see if they get the same answer.
The chances are the answers were not unanimous, and for good reason.
There is simply more than one function that fits the 6 training examples, and
some of these functions have a value of - 1 on the test point and others have a
value of + 1 . For instance, if the true f is + 1 when the pattern is symmetric,
the value for the test point would be + 1 . If the true f is + 1 when the top left
square of the pattern is white, the value for the test point would be - 1 . Both
functions agree with all the examples in the data set, so there isn't enough
information to tell us which would be the correct answer.
This does not bode well for the feasibility of learning. To make matters
worse, we will now see that the difficulty we experienced in this simple problem
is the rule, not the exception.
1. 3 . 1

Outside the Data Set

When we get the training data V, e.g. , the first two rows of Figure 1 . 7, we
know the value of f on all the points in V. This doesn't mean that we have
learned f, since it doesn't guarantee that we know anything about f outside
of V. We know what we have already seen, but that's not learning. That 's
memorizing.
Does the data set V tell us anything outside of V that we didn't know
before? If the answer is yes, then we have learned something. If the answer is
no, we can conclude that learning is not feasible.
Since we maintain that f is an unknown function, we can prove that f
remains unknown outside of V. Instead of going through a formal proof for
the general case, we will illustrate the idea in a concrete case. Consider a
Boolean target function over a three-dimensional input space X = {O, 1 } 3 .
We are given a data set V of five examples represented in the table below. We
denote the binary output by o / for visual clarity,
0
0
0
0
1

Xn

0
0
1
1
0

0
1
0
1
0

Yn
0

where Yn = f (xn) for n = 1 , 2, 3, 4, 5. The advantage of this simple Boolean

case is that we can enumerate the entire input space (since there are only 2 3 = 8
distinct input vectors) , and we can enumerate the set of all possible target
functions (since f is a Boolean function on 3 Boolean inputs, and there are
3
only 22 = 256 distinct Boolean functions on 3 Boolean inputs) .
16

1 . 3. ls LEARNING FEASIBLE?

1 . THE LEARNING PROBLEM

Let us look at the problem of learning i. Since i is unknown except

inside D, any function that agrees with D could conceivably be i. The table
below shows all such functions Ji, , is. It also shows the data set D (in
blue) and what the final hypothesis g may look like.

f4

f5

f6

fs

The final hypothesis g is chosen based on the five examples in D. The table
shows the case where g is chosen to match i on these examples.
If we remain true to the notion of unknown target, we cannot exclude any
of Ji, , is from being the true i Now, we have a dilemma. The whole
purpose of learning i is to be able to predict the value of f on points that we
haven't seen before. The quality of the learning will be determined by how
close our prediction is to the true value. Regardless of what g predicts on
the three points we haven't seen before (those outside of D, denoted by red
question marks) , it can agree or disagree with the target, depending on which
of Ji , , is turns out to be the true target. It is easy to verify that any 3
bits that replace the red question marks are as good as any other 3 bits.
Exercise 1. 7
For each of the following learning scenarios in the a bove problem, eval uate
outside V. To measure the
the performa nce of g on the three points in
performa nce, compute how m a ny of the 8 possible target fun ctions agree
with g on a l l three points, on two of them, on one of them, a nd on none
of them .

(a )

1-l has on ly two hypotheses, one that a lways returns ' ' a nd one that
a lways returns 'o'. The learn ing a lgorithm picks the hypothesis that
m atches the data set the most.

( b ) The same

1-l, but the learni ng a lgorith m now picks the hypothesis

that matches the data set the least.

( c) 1-l = {XOR} (only one hypothesis which is a lways picked ) , where

XOR is defined by XOR(x) = if the n um ber of l's in x is odd a nd
XOR(x) = o if the n um ber is even .
( d ) 1-l contai ns a l l possible hypotheses ( a l l Boolean functions on th ree
varia bles) , a nd the lea rn i ng a lgorith m picks the hypothesis that agrees
with a l l tra i n i ng exa mples, but otherwise disagrees the most with the

XOR.

17

1 . 3. Is LEARNING FEASIBLE?

1 . THE LEARNING PROBLEM

BIN

SAMPLE

=probability of red marbles

Figure 1 .8: A random sample is picked from a bin ofred and green marbles.
The probability of red marbles in the bin is unknown. What does the
fraction v of red marbles in the sample tell us about ?
It doesn't matter what the algorithm does or what hypothesis set 1-l is used.
Whether 1-l has a hypothesis that perfectly agrees with V (as depicted in the
table) or not, and whether the learning algorithm picks that hypothesis or
picks another one that disagrees with V (different green bits) , it makes no
difference whatsoever as far as the performance outside of V is concerned. Yet
the performance outside V is all that matters in learning!
This dilemma is not restricted to Boolean functions, but extends to the
general learning problem. As long as f is an unknown function, knowing V
cannot exclude any pattern of values for f outside of V. Therefore, the pre
dictions of g outside of V are meaningless.
Does this mean that learning from data is doomed? If so, this will be a
very short book @. Fortunately, learning is alive and well, and we will see
why. We won't have to change our basic assumption to do that. The target
function will continue to be unknown, and we still mean unknown.
1. 3 .2

Probability to the Rescue

We will show that we can indeed infer something outside V using only V, but
in a probabilistic way. What we infer may not be much compared to learning
a full target function, but it will establish the principle that we can reach
outside V. Once we establish that, we will take it to the general learning
problem and pin down what we can and cannot learn.
Let's take the simplest case of picking a sample, and see when we can say
something about the objects outside the sample. Consider a bin that contains
red and green marbles, possibly infinitely many. The proportion of red and
green marbles in the bin is such that if we pick a marble at random, the
probability that it will be red is and the probability that it will be green
is 1 . We assume that the value of is unknown to us.
-

18

1 . 3. Is LEARNING FEASIBLE?

1 . THE LEARNING PROBLEM

We pick a random sample of N independent marbles (with replacement)

from this bin, and observe the fraction v of red marbles within the sample
(Figure 1.8). What does the value of v tell us about the value of ?
One answer is that regardless of the colors of the N marbles that we picked,
we still don't know the color of any marble that we didn't pick. We can get
mostly green marbles in the sample while the bin has mostly red marbles.
Although this is certainly possible, it is by no means probable.
Exercise 1.8
If

= 0 .9, what is the probability that a sam ple of 10 marbles wil l h ave
:: 0 . 1 ? [Hints: 1. Use binomial distribution. 2. The answer is a very

small number.]

The situation is similar to taking a poll. A random sample from a population

tends to agree with the views of the population at large. The probability
distribution of the random variable v in terms of the parameter is well
understood, and when the sample size is big, v tends to be close to .
To quantify the relationship between v and , we use a simple bound called
the Hoeffding Inequality . It states that for any sample size N,
for any

E > 0.

(1 .4)

Here, JP> [ ] denotes the probability of an event, in this case with respect to
the random sample we pick, and E is any positive value we choose. Putting
Inequality (1.4) in words, it says that as the sample size N grows, it becomes
exponentially unlikely that v will deviate from by more than our 'tolerance' E.
The only quantity that is random in ( 1 .4) is v which depends on the random
sample. By contrast, is not random. It is just a constant, albeit unknown to
us. There is a subtle point here. The utility of (1 .4) is to infer the value of
using the value of v, although it is that affects v, not vice versa. However,
since the effect is that v tends to be close to , we infer that 'tends' to be
close to v .
Although JP> [I v I > E] depends on , as appears in the argument and
also affects the distribution of v, we are able to bound the probability by 2e- 2E2 N
which does not depend on . Notice that only the size N of the sample affects
the bound, not the size of the bin. The bin can be large or small, finite or
infinite, and we still get the same bound when we use the same sample size.
Exercise 1. 9
If = 0 .9, use the Hoeffding I neq uality to bound the probabil ity that a
sample of 10 marbles will have v :: 0 . 1 a nd compare the a nswer to the
previous exercise.

If we choose E to be very small in order to make v a good approximation of ,

we need a larger sample size N to make the RHS of lnequality (1 .4) small. We

19

1 . 3 . Is LEARNING FEASIBLE?

1 . THE LEARNING PROBLEM

can then assert that it is likely that v will indeed be a good approximation of .
Although this assertion does not give us the exact value of , and doesn't even
guarantee that the approximate value holds, knowing that we are within E
of most of the time is a significant improvement over not knowing anything
at all.
The fact that the sample was randomly selected from the bin is the reason
we are able to make any kind of statement about being close to v. If the
sample was not randomly selected but picked in a particular way, we would
lose the benefit of the probabilistic analysis and we would again be in the dark
outside of the sample.
How does the bin model relate to the learning problem? It seems that the
unknown here was just the value of while the unknown in learning is an entire
function f : X -+ Y. The two situations can be connected. Take any single
hypothesis h E 'H and compare it to f on each point x E X . If h (x) = f (x) ,
color the point x green. If h(x) =/- f (x) , color the point x red. The color
that each point gets is not known to us, since f is unknown. However, if we
pick x at random according to some probability distribution P over the input
space X, we know that x will be red with some probability, call it , and green
with probability 1 - . Regardless of the value of , the space X now behaves
like the bin in Figure 1 .8.
The training examples play the role of a sample from the bin. If the
inputs xi ,
, XN in V are picked independently according to P, we will get
a random sample of red (h(xn) =/- J(xn ) ) and green (h(xn) = f (xn)) points.
Each point will be red with probability and green with probability 1 - . The
color of each point will be known to us since both h(xn) and f (xn) are known
for n = 1 ,
, N ( the function h is our hypothesis so we can evaluate it on
any point, and f (xn ) = Yn is given to us for all points in the data set V). The
learning problem is now reduced to a bin problem, under the assumption that
the inputs in V are picked independently according to some distribution P
on X . Any P will translate to some in the equivalent bin. Since is
allowed to be unknown, P can be unknown to us as well. Figure 1 . 9 adds this
probabilistic component to the basic learning setup depicted in Figure 1 .2 .
With this equivalence, the Hoeffding Inequality can b e applied to the learn
ing problem, allowing us to make a prediction outside of V. Using v to pre
dict tells us something about f, although it doesn't tell us what f is. What
tells us is the error rate h makes in approximating f. If v happens to be close
to zero, we can predict that h will approximate f well over the entire input
space. If not , we are out of luck.
Unfortunately, we have no control over v in our current situation, since v
is based on a particular hypothesis h. In real learning, we explore an entire
hypothesis set 'H, looking for some h E 'H that has a small error rate. If we
have only one hypothesis to begin with, we are not really learning, but rather
'verifying' whether that particular hypothesis is good or bad. Let us see if we
can extend the bin equivalence to the case where we have multiple hypotheses
in order to capture real learning.

20

1 . 3. Is LEARNING FEASIBLE?

1 . THE LEARNING PROBLEM

UNKNOWN TARGET FUNCTION

f : ,Y r-t Y

TRAINING EXAMPLES

FINAL
HYPOTHESIS
g

HYPOTHESIS SET
H

Figure 1.9: Probability added to the basic learning setup

To do that, we start by introducing more descriptive names for the dif

ferent components that we will use. The error rate within the sample, which
corresponds to v in the bin model, will be called the in-sample error,

( fraction of 'D where

1

n= l

f and h disagree )

[h(xn) f f(xn)] ,

where [statement] = 1 if the statement is true, and = 0 if the statement is

false. We have made explicit the dependency of Ein on the particular h that
we are considering. In the same way, we define the out-of-sample error

Eout (h) = JPl [h(x) f f (x)] ,

which corresponds to in the bin model. The probability is based on the
distribution P over X which is used to sample the data points x.
21

1. 3. Is LEARNING FEASIBLE?

1 . THE LEARNING PROBLEM

Figure 1 . 10: Multiple bins depict the learning problem with M hypotheses

Substituting the new notation Ein for

Inequality ( 1.4) can be rewritten as

and

Eout

for , the Hoeffding

for any E > 0,

( 1 . 5)

where N is the number of training examples. The in-sample error Ein, just
like v , is a random variable that depends on the sample. The out-of-sample
error Eout, j ust like , is unknown but not random.
Let us consider an entire hypothesis set H instead of just one hypothesis h,
and assume for the moment that H has a finite number of hypotheses
We can construct a bin equivalent in this case by having M bins as shown in
Figure 1 . 10 . Each bin still represents the input space X , with the red marbles
in the mth bin corresponding to the points x E X where hm (x) -f f (x). The
probability of red marbles in the mth bin is Eout (hm) and the fraction of
red marbles in the mth sample is Ein(hm), for m = 1 ,
, M. Although the
Hoeffding Inequality ( 1 . 5) still applies to each bin individually, the situation
becomes more complicated when we consider all the bins simultaneously. Why
is that? The inequality stated that

for any E > 0,

where the hypothesis h is fixed before you generate the data set, and the
probability is with respect to random data sets V; we emphasize that the
assumption "h is fixed before you generate the data set" is critical to the
validity of this bound. If you are allowed to change h after you generate the
data set, the assumptions that are needed to prove the Hoeffding Inequality
no longer hold. With multiple hypotheses in H, the learning algorithm picks
22

1 . 3. Is LEARNING FEASIBLE?

1 . THE LEARNING PROBLEM

the final hypothesis g based on D, i.e. after generating the data set. The
statement we would like to make is not

" JP> [IEin(hm) - Eout(hm) I > E] is small"

( for any particular, fixed hm E 1-l) , but rather
" JP>[IEin(g) - Eout (g) I > E] is small" for the final hypothesis g .
The hypothesis g is not fixed ahead o f time before generating the data, because
which hypothesis is selected to be g depends on the data. So, we cannot just
plug in g for h in the Hoeffding inequality. The next exercise considers a simple
coin experiment that further illustrates the difference between a fixed h and
the final hypothesis g selected by the learning algorithm.
Exercise 1 . 10
Here is a n experiment that i l lustrates the d ifference between a single bin
a n d m u ltiple bins. R u n a computer sim u lation for flipping 1, 000 fair coins.
Flip each coi n independently
times. Let's focus on 3 coins as follows:
c 1 is the first coin flipped; Crand is a coin you choose at random; Cmin is the
coi n that had the m i n i m u m frequency of heads ( pick the earlier one in case
of a tie) . Let v1 , Vrand a n d Vmin be the fraction of heads you obtai n for the
respective three coi ns.

( a ) What is for the th ree coins selected?

( b) Repeat this entire experiment a large n um ber of times ( e.g. , 100, 000
runs of the entire experiment) to get several instances of v1 , Vrand
a nd plot the histograms of the distributions of v1 , Vrand a nd
Notice that which coins end u p being Crand a n d Cmin may differ
from one run to a n other.
and

Vmin

Vmin

(c) Using (b), plot estimates for JP [ j v - j > E] as a function of E , together

with the Hoeffd i ng bound 2e- 2 c:2N ( on the same graph ) .
(d) Which coins obey the Hoeffding bound, a n d which ones do not? Ex
plain why.

( e) Relate part ( d ) to the m u ltiple bins in Figure

1. 10.

The way to get around this is to try to bound JP> [ IEin(g) - Eout(g) I > E] in
a way that does not depend on which g the learning algorithm picks. There
is a simple but crude way of doing that. Since g has to be one of the hm 's
regardless of the algorithm and the sample, it is always true that

" IEin(g) - Eout(g) I > E"

==

"

or

23

IEin( h1) - Eout( h1 ) I > E

IEin(h2 ) - Eout (h2 ) I > E

1 . 3. Is LEARNING FEASIBLE?

1 . THE LEARNING PROBLE!VI

where B1 ==:;:. B2 means that event B1 implies event B2 . Although the events
on the RHS cover a lot more than the LHS, the RHS has the property we want ;
the hypotheses hm are fixed. We now apply two basic rules in probability;
and, if B1 , B2 , , BM are any events, then
The second rule is known as the union bound. Putting the two rules together,
we get

IP' [ IEin(g) - Eout (g) I > E ] < JP' [

IEin (h 1 ) - Eout ( h1 ) I > E

or IEin(h2 ) - Eout (h2 ) I > E

or IEin(hM) - Eout(hM ) I > E ]

M
< L IP' [IEin(hm) Eout(hm) I > E] .
m= l
Applying the Hoeffding Inequality ( 1 .5) to the M terms one at a time, we can
2
bound each term in the sum by 2 e-2 E N . Substituting, we get

(1 .6)
Mathematically, this is a 'uniform' version of (1 .5) . We are trying to simul
taneously approximate all Eout (hm)'s by the corresponding Ein(hm) 's. This
allows the learning algorithm to choose any hypothesis based on Ein and ex
pect that the corresponding Eout will uniformly follow suit, regardless of which
hypothesis is chosen.
2
The downside for uniform estimates is that the probability bound 21\lfe- 2 E N
is a factor of ]\If looser than the bound for a single hypothesis, and will only
be meaningful if ]\If is finite. We will improve on that in Chapter 2.
1. 3 . 3

Feasibility of Learning

We have introduced two apparently conflicting arguments about the feasibility

of learning. One argument says that we cannot learn anything outside of V,
and the other says that we can. We would like to reconcile these two arguments
and pinpoint the sense in which learning is feasible:

1 . Let us reconcile the two arguments. The question of whether V tells us

anything outside of V that we didn't know before has two different answers.
If we insist on a deterministic answer, which means that V tells us something
certain about f outside of V, then the answer is no. If we accept a probabilistic
answer, which means that V tells us something likely about f outside of V,
then the answer is yes.

24

1 . 3. ls LEARNING FEASIBLE?

1 . THE LEARNING PROBLEM

Exercise 1 . 1 1
We a re given a data set 'D o f 2 5 t ra i ning exam ples from a n u nknown target
fun ction j :
= JR a n d = {-1, +1}.
Y, where
learn f, we use
a simple hypothesis set = {h1 , h2 } where h1 is the constant
function
a n d h2 is the constant -1.

We consider two learning a lgorithms, S (smart) a n d ( crazy) . S chooses

and chooses the other hy
the hypothesis that agrees the most with
pothesis deliberately. Let us see how these a lgorithms perform out of sam
ple from the deterministic a n d probabilistic points of view. Assume i n
t h e probabilistic view that there i s a probability distribution on X , a n d let

= p.
JID[f(x) =
( a ) Can S produce a hypothesis that is guaranteed to perform better than
random on a ny point outside

'D?

( b ) Assume for the rest of the exercise that a l l the exam ples in

have
Is it possible that the hypothesis that produces turns out
to be better than the hypothesis that S produces?

Yn =

1.

( c ) If p = 0.9, what is the probability that

pothesis than

C?

wil l produce a better hy

( d) Is there any val ue of p for which it is more likely than not that C wil l
produce a better hypothesis than

S?

By adopting the probabilistic view, we get a positive answer to the feasibility

question without paying too much of a price. The only assumption we make
in the probabilistic framework is that the examples in V are generated inde
pendently. We don't insist on using any particular probability distribution,
or even on knowing what distribution is used. However, whatever distribu
tion we use for generating the examples, we must also use when we evaluate
how well g approximates f (Figure 1 .9) . That's what makes the Hoeffding
Inequality applicable. Of course this ideal situation may not always happen
in practice, and some variations of it have been explored in the literature.
2 . Let us pin down what we mean by the feasibility of learning. Learning pro
duces a hypothesis g to approximate the unknown target function f. If learning
is successful, then g should approximate f well, which means Eout(g) Rj 0 .
However, this i s not what we get from the probabilistic analysis. What we
get instead is Eout (g) Rj Ein (g). We still have to make Ein (g) Rj 0 in order to
conclude that Eout (g) Rj 0.
We cannot guarantee that we will find a hypothesis that achieves Ein (g) Rj 0 ,
but at least we will know if we find it. Remember that Eout (g) is an unknown
quantity, since f is unknown, but Ein (g) is a quantity that we can evaluate.
We have thus traded the condition Eout (g) Rj 0 , one that we cannot ascertain,
for the condition Ein (g) Rj 0, which we can ascertain. What enabled this is
the Hoeffding Inequality (1.6) :
lP[JEin(g) Eout (g) J > E] :S 2Me 2 E2 N
25

1 . 3. Is LEARNING FEASIBLE?

1 . THE LEARNING PROBLEM

that assures us that Eout (g)

Ein (g) so we can use Ein as a proxy for Eout .

Exercise 1.12
friend comes to you with a l earning problem . She says the target func
is completely u nknown , but she has 4, 000 data points. She is
tion
wil ling to pay you to solve her problem a n d produce for her a g which
a pproximates f. What is the best that you can promise her a mong the
following:

(a ) After learning you wil l provide her with a g that you wil l guarantee
a pproximates

wel l out

of

sample.

( b) After learn i ng you wil l provide her with a g , and with h igh probabil ity
the

which you produce will a pproximate

wel l out of sample.

(c ) One of two things wil l h a ppen.

( i ) You wil l produce a hypothesis g;
(i i ) You wil l decla re that you failed .

I f you d o return a hypothesis g , then with h igh proba bility the g which
you produce wil l a pproxim ate wel l out of sample.

One should note that there are cases where we won't insist that Ein (g) 0.
Financial forecasting is an example where market unpredictability makes it
impossible to get a forecast that has anywhere near zero error. All we hope
for is a forecast that gets it right more often than not. If we get that, our
bets will win in the long run. This means that a hypothesis that has Ein (g)
somewhat below 0.5 will work, provided of course that Eout (g) is close enough
to Ein (g) .
The feasibility of learning is thus split into two questions:

1 . Can we make sure that Eout (g) is close enough to Ein (g) ?
2. Can we make Ein (g) small enough?
The Hoeffding Inequality (1 .6) addresses the first question only. The second
question is answered after we run the learning algorithm on the actual data
and see how small we can get Ein to be.
Breaking down the feasibility of learning into these two questions provides
further insight into the role that different components of the learning problem
play. One such insight has to do with the 'complexity' of these components.
The complexity of }{. If the number of hypotheses ]VJ goes up, we run
more risk that Ein (g) will be a poor estimator of Eout (g) according to In
equality (1.6). ]VJ can be thought of as a measure of the 'complexity' of the

26

1 . THE LEARNING PROBLEM

1 . 4. ERROR AND NOISE

hypothesis set 1{ that we use. If we want an affirmative answer to the first

question, we need to keep the complexity of 1{ in check. However, if we want
an affirmative answer to the second question, we stand a better chance if 1{
is more complex, since g has to come from 1{. So, a more complex 1{ gives us
more flexibility in finding some g that fits the data well, leading to small Ein (g) .
This tradeoff in the complexity of 1{ is a major theme in learning theory that
we will study in detail in Chapter 2.
The complexity of f. Intuitively, a complex target function f should be
harder to learn than a simple f . Let us examine if this can be inferred from
the two questions above. A close look at Inequality (1 .6) reveals that the
complexity of f does not affect how well Ein ( g ) approximates Eout (g) . If
we fix the hypothesis set and the number of training examples, the inequality
provides the same bound whether we are trying to learn a simple f (for instance
a constant function) or a complex f (for instance a highly nonlinear function) .
However, this doesn't mean that we can learn complex functions as easily as
we learn simple functions. Remember that (1.6) affects the first question only.
If the target function is complex, the second question comes into play since
the data from a complex f are harder to fit than the data from a simple f .
This means that we will get a worse value for Ein (g) when f i s complex. We
might try to get around that by making our hypothesis set more complex so
that we can fit the data better and get a lower Ein (g) , but then Eout won't be
as close to Ein per (1.6) . Either way we look at it, a complex f is harder to
learn as we expected. In the extreme case, if f is too complex, we may not be
able to learn it at all.

Fortunately, most target functions in real life are not too complex; we can
learn them from a reasonable V using a reasonable H. This is obviously a
practical observation, not a mathematical statement. Even when we cannot
learn a particular f, we will at least be able to tell that we can't. As long as
we make sure that the complexity of 1{ gives us a good Hoeffding bound, our
success or failure in learning f can be determined by our success or failure in
fitting the training data.

1 .4

Error and Noise

We close this chapter by revisiting two notions in the learning problem in order
to bring them closer to the real world. The first notion is what approximation
means when we say that our hypothesis approximates the target function
well. The second notion is about the nature of the target function. In many
situations, there is noise that makes the output of f not uniquely determined
by the input . What are the ramifications of having such a 'noisy' target on
the learning problem?
27

1 .4. ERROR AND NOISE

1 . THE LEARNING PROBLEM

1.4 . 1

Error Measures

Learning is not expected to replicate the target function perfectly. The final
hypothesis g is only an approximation of f . To quantify how well g approxi
mates f , we need to define an error measure 3 that quantifies how far we are
from the target.
The choice of an error measure affects the outcome of the learning process.
Different error measures may lead to different choices of the final hypothesis,
even if the target and the data are the same, since the value of a particular error
measure may be small while the value of another error measure in the same
situation is large. Therefore, which error measure we use has consequences
for what we learn. What are the criteria for choosing one error measure over
another? We address this question here.
First, let's formalize this notion a bit. An error measure quantifies how
well each hypothesis h in the model approximates the target function f ,
Error

E(h, f ) .

While E(h, f ) is based on the entirety of h and f , it is almost universally de

fined based on the errors on individual input points x. If we define a pointwise
error measure e(h(x) , f (x) ) , the overall error will be the average value of this
pointwise error. So far, we have been working with the classification error
e(h(x) , f (x) ) = [h(x) f- J (x)] .
In an ideal world, E(h, J) should be user-specified. The same learning task
in different contexts may warrant the use of different error measures. One may
view E(h, J) as the 'cost' of using h when you should use f . This cost depends
on what h is used for, and cannot be dictated just by our learning techniques.
Here is a case in point.
Example 1 . 1 (Fingerprint verification) . Consider the problem of verifying
that a fingerprint belongs to a particular person. What is the appropriate
error measure?

1+
{ -1

you

The target function takes as input a fingerprint, and returns

to the right person, and -1 if it belongs to an intruder.

+1

if it belongs

3 This measure is also called an error function in the literature, and sometimes the error
is referred to as cost, objective, or risk.

28

1 . THE LEARNING PROBLEM

1 .4 . ERROR AND NOISE

There are two types of error that our hypothesis h can make here. If the
correct person is rejected (h = -1 but f = +1) , it is called false reject , and if
an incorrect person is accepted (h = +1 but f = -1), it is called false accept .

+1
-1

+1

no error
false reject

-1
false accept
no error

How should the error measure be defined in this problem? If the right person
is accepted or an intruder is rejected, the error is clearly zero. We need to
specify the error values for a false accept and for a false reject. The right
values depend on the application.
Consider two potential clients of this fingerprint system. One is a super
market who will use it at the checkout counter to verify that you are a member
of a discount program. The other is the CIA who will use it at the entrance
to a secure facility to verify that you are authorized to enter that facility.
For the supermarket, a false reject is costly because if a customer gets
wrongly rejected, she may be discouraged from patronizing the supermarket
in the future. All future revenue from this annoyed customer is lost. On the
other hand, the cost of a false accept is minor. You just gave away a discount
to someone who didn't deserve it, and that person left their fingerprint in your
system they must be bold indeed.
For the CIA, a false accept is a disaster. An unauthorized person will gain
access to a highly sensitive facility. This should be reflected in a much higher
cost for the false accept. False rejects, on the other hand, can be tolerated
since authorized persons are employees (rather than customers as with the
supermarket) . The inconvenience of retrying when rejected is just part of the
job , and they must deal with it .
The costs of the different types of errors can be tabulated in a matrix. For
our examples, the matrices might look like:
f

+1 -1
+1 0 1
-1 10 0

+1 -1
+1 0 1000
-1 1
0
CIA

Supermarket

These matrices should be used to weight the different types of errors when
we compute the total error. When the learning algorithm minimizes a cost
weighted error measure, it automatically takes into consideration the utility
of the hypothesis that it will produce. In the supermarket and CIA scenarios,
D
this could lead to two completely different final hypotheses.
The moral of this example is that the choice of the error measure depends
on how the system is going to be used, rather than on any inherent criterion
29

1 . THE LEARNING PROBLEM

1 . 4 . ERROR AND NOISE

I x)

UNKNOWN
INPUT DISTRIBUTION
TRAINING EXAMPLES

HYPOTHESIS SET

Figure

1.11:

The general (supervised) learning problem

that we can independently determine during the learning process. However,

this ideal choice may not be possible in practice for two reasons. One is
that the user may not provide an error specification, which is not uncommon.
The other is that the weighted cost may be a difficult objective function for
optimizers to work with. Therefore, we often look for other ways to define the
error measure, sometimes with purely practical or analytic considerations in
mind. We have already seen an example of this with the simple binary error
used in this chapter, and we will see other error measures in later chapters.
1.4.2

Noisy Targets

In many practical applications, the data we learn from are not generated by
a deterministic target function. Instead, they are generated in a noisy way
such that the output is not uniquely determined by the input. For instance,
in the credit-card example we presented in Section 1.1, two customers may
have identical salaries, outstanding loans, etc. , but end up with different credit
behavior. Therefore, the credit 'function' is not really a deterministic function,

30

1 . THE LEARNING PROBLEM

1 . 4 . ERROR AND NOISE

but a noisy one.

This situation can be readily modeled within the same framework that we
have. Instead of y = f(x) , we can take the output y to be a random variable
that is affected by, rather than determined by, the input x. Formally, we have
a target distribution P(y I x) instead of a target function y = f (x) . A data
point (x, y) is now generated by the joint distribution P (x, y) = P(x)P (y I x) .
One can think of a noisy target as a deterministic target plus added noise.
If y is real-valued for example, one can take the expected value of y given x to
be the deterministic f (x) , and consider y - f (x) as pure noise that is added
to f.
This view suggests that a deterministic target function can be considered
a special case of a noisy target, just with zero noise. Indeed, we can formally
express any function f as a distribution P(y I x) by choosing P(y I x) to be
zero for all y except y = f (x) . Therefore, there is no loss of generality if we
consider the target to be a distribution rather than a function. Figure 1 . 1 1
modifies the previous Figures 1 .2 and 1 . 9 to illustrate the general learning
problem, covering both deterministic and noisy targets.
Exercise 1 . 13
Consider the bin model for a hypothesis h that makes a n error with prob
a b i lity in a pproximating a deterministic target function ( both h a nd
ar binary fu nctions). If we use the same h to a pproximate a noisy version
of f given by
P(y I x)

1 - .A

( a ) What i s t h e probability o f error that

(b) At what val ue of

y
y
h

f(x),
f(x).

makes i n a pproxim ating y?

wil l the performance of

be independent of ?

[Hint: The noisy target will look completely random.]

There is a difference between the role of P(y I x) and the role of P (x) in
the learning problem. While both distributions model probabilistic aspects
of x and y, the target distribution P(y I x) is what we are trying to learn,
while the input distribution P (x) only quantifies the relative importance of
the point x in gauging how well we have learned.
Our entire analysis of the feasibility of learning applies to noisy target
functions as well. Intuitively, this is because the Hoeffding Inequality (1 .6)
applies to an arbitrary, unknown target function. Assume we randomly picked
all the y's according to the distribution P(y I x) over the entire input space X .
This realization of P(y I x ) i s effectively a target function. Therefore, the
inequality will be valid no matter which particular random realization the
'target function' happens to be.
This does not mean that learning a noisy target is as easy as learning a
deterministic one. Remember the two questions of learning? With the same
learning model, Eout may be as close to Ein in the noisy case as it is in the
31

1 . 4 . ERROR AND NOISE

1 . THE LEARNING PROBLEM

deterministic case, but Ein itself will likely be worse in the noisy case since it
is hard to fit the noise.
In Chapter 2, where we prove a stronger version of ( 1 . 6) , we will assume
the target to be a probability distribution P(y I x), thus covering the general
case.

32

1 . THE LEARNING PROBLEM

1.5

1 . 5 . PROBLEMS

Problems

Problem 1 . 1

We have 2 opaque bags, each containing 2 ba l ls. One bag

has 2 black ba l ls and the other has a black and a white ba l l . You pick a bag
at ra ndom a nd then pick one of the ba lls in that bag at random. When you
look at the ba l l it is black. You now pick the second ba l l from that same bag.
What is the pro bability that this ba l l is also black? {Hint: Use Bayes ' Theorem:
JID[A and B] = JID[A I B] JID [BJ = JID[B I A] JID [A] .]

Problem 1 . 2
Consider the perceptron in two dimensions: h(x) =
sign(wTx) where w = [wo , w1 , w2 r and x = [1, x1 , x 2 r . Technical ly, x has
three coordi nates, but we cal l this perceptron two-dimensional beca use the fi rst
coord inate is fixed at 1 .
(a) Show that the regions o n the plane where h(x) = + 1 a nd h(x) = - 1 are
separated by a l ine. If we express t h is line by the eq uation x 2 = ax1 + b,
what are the slope a a nd intercept b in terms of wo , w1 , w2 ?
(b) Draw a pictu re for the cases w

[1 , 2, 3r and w

- [1 , 2, 3r .

I n more tha n two d i mensions, the +1 and - 1 regions are separated by a hy

perplane, the genera l ization of a line.

Problem 1.3

P rove that the P LA eventua lly converges to a l inear

separator for separa ble data . The fol lowing steps wil l guide you through the
proof. Let w* be a n optim a l set of weights (one which separates the data ) .
T h e essenti a l idea i n this proof i s t o show that t h e P LA weights w (t) get "more
a ligned" with w* with every iteration . For simplicity, assume that w(O) = 0.
(a) Let p = min1::n ::N Yn (wnxn ) . Show that p > 0.
(b) Show that wT (t)w* wT (t- l)w* +p, and conclude that wT (t)w* tp.

[Hint: Use induction. ]

(c) Show that ll w (t) ll 2 :: ll w(t - 1) 11 2 + ll x(t - 1 ) 11 2 .

{Hint: y(t - 1) (wT (t - l)x(t - 1 ) ) :: 0 because x(t - 1) was misclas

sified by w (t - 1 ) .j

(d) Show by induction that ll w(t) ll 2 :: tR2 , where R = max1::n :: N ll xn ll

(continued on next page)

33

1 . THE LEARNING PROBLEM

1 . 5 . PROBLEMS

( e ) Using ( b ) a nd ( d ) , show that

Vt p
WT (t)
t
w

R'
ll w(t) ll
*

and hence prove that

[ Hint:

ll w (t) l/ ll w * ll

:: 1 . Why?

In practice, PLA converges more q uickly tha n the bound

suggests.
p
Nevertheless, beca use we do not know p in advance, we ca n 't determine the
n u m ber of iterations to convergence, wh ich does pose a problem if the data is
non-separable.

Problem 1 .4

I n Exercise 1 .4, we use a n artificial data set to study the

perceptron learning algorith m . This problem leads you to explore the algorith m
fu rther with data sets of d ifferent sizes a n d dimensions.

( a ) Generate a linearly separa ble data set of size 20 as indicated in Exer

cise 1.4. Plot the exa m ples { (xn , Yn ) } as wel l as the target function f on
a plane. Be sure to mark the exa m ples from different classes d ifferently,
and add la bels to the axes of the plot.

( b ) Run the perceptron lea rning a lgorith m on the data set a bove. Report the
n u m ber of u pdates that the a lgorith m ta kes before converging. P lot the
exa mples { (xn , Yn) } , the target fu nction f, and the fin a l hypothesis g in
the same figu re. Com ment on whether f is close to g.

( c ) Repeat everyth i ng in ( b ) with a nother ra ndomly generated data set of

size 20. Compare you r resu lts with ( b ) .
( d ) Repeat everythi ng i n ( b ) with a nother randomly generated data set of
size 100. Compare you r results with ( b ) .
( e ) Repeat everyth ing in ( b ) with a nother ra ndomly generated data set of
size 1 , 000. Com pare you r resu lts with ( b ) .
2
10
( f ) Mod ify the a lgorith m such that it takes Xn E JR instead of JR . Ra n
10
dom ly generate a linea rly separa ble data set of size 1, 000 with Xn E JR
and feed the data set to the a lgorithm. How many u pdates does the
a lgorithm ta ke to converge?

( g) Repeat the a lgorithm on the same data set as ( f ) for 100 experi ments. I n

t h e iterations of each experiment, pick x(t) ra ndomly instead of determ i n

istica lly. Plot a histogra m for the n u m ber of u pdates that the a lgorith m
takes to converge.

( h ) S u m ma rize your concl usions with respect to accu racy a nd run n ing time
as a fu nction of N a n d d.
34

1 . THE LEARNING PROBLEM

1 . 5 . PROBLEMS

Problem 1.5 The perceptron learning a l gorithm works l i ke this: In each it

eration t, pick a ran dom (x(t), y(t)) a n d compute the 'signa l ' s(t) = wT(t)x(t).
If y(t) s(t) ::=:; 0, u pdate w by

w(t + 1) +- w(t) + y(t) x(t) ;

One may a rgue that this algorithm does not ta ke the 'closeness' between s(t)
and y(t) into consideratio n . Let's look at a nother perceptron learning algo
rithm: I n each iteration, pick a ra ndom (x(t) , y(t)) a nd com pute s(t). If
y(t) s(t) ::; 1, update w by

w (t + 1) +- w (t) + 'T/ (y(t)

s(t)) x(t) ,

where 'T/ is a constant. That is, if s(t) agrees with y(t) wel l (their prod uct
is > 1 ) , the a lgorithm does nothing. On the other hand, if s(t) is further
from y(t), the a lgorithm cha nges w(t) more. In this problem , you a re asked to
im plement this algorithm a n d study its performa nce.
(a) Generate a tra in i ng data set of size 100 similar to that used in Exercise 1 .4.
Generate a test data set of size 10, 000 from the same process. To get g,
run the a lgorith m a bove with 'T/ = 100 on the training data set, u nti l
a maximum of 1 , 000 u pdates has been reached . Plot the training data
set, the target function f, and the final hypothesis g on the same figu re.
Report the error on the test set.
(b) Use the data set in (a) and redo everything with
( c ) Use t h e data set in (a) and redo everything with
( d ) Use the data set in (a) and redo everything with

'T/

'T/

'T/

= 1.
= 0.01.
= 0. 0001.

(e) Com pare the resu lts that you get from (a) to (d ) .
T h e algorithm a bove i s a variant of the so ca l led Adaline (Adaptive Linear
Neuron) a lgorithm for perceptron learn ing.

Consider a sa m ple of 10 marbles d rawn i ndependently from

a bin that holds red a nd green marbles. The probability of a red marble is .
For = 0.05, = 0.5, and = 0.8, com pute the probability of getting no red
marbles ( v = 0) in the fol lowing cases.

Problem 1.6

(a) We d raw only one such sample. Com pute the proba bility that

= 0.

( b) We d raw 1 , 000 independent sa mples. Com pute the proba bility that ( at
least) one of the sa m ples has v = 0.
( c) Repeat ( b) for 1 , 000, 000 independent sam ples.

35

1 . 5 . PROBLEMS

1 . THE LEARNING PROBLEM

Problem 1. 7

A sample of heads a nd tails is created by tossing a coin

a n u m ber of times independently. Assume we have a n u mber of coins that
generate different sa m ples independently. For a given coin , let the probability
of heads ( proba bility of error) be . The proba bility of obtaining k heads in N
tosses of this coin is given by the binomial distribution :

Remem ber that the training error v is

(a) Assume the sam ple size ( N ) is 10. I f a l l the coins have = 0.05 compute
the proba bility that at least one coin wil l have v = 0 for the case of 1
coi n , 1 , 000 coi ns, 1 , 000, 000 coins. Repeat for = 0.8.
(b) For the case N = 6 and 2 coins with = 0.5 for both coins, plot the
probability
P[mx I Vi - i i > E]
i

for E in the range [O, 1 ] (the max is over coins) . On the same plot show the
bound that wou ld be obtained usi ng the Hoeffding I neq u a lity . Remember
that for a single coin , the Hoeffd i n g bound is

[Hint: Use P[A or B] = P[A] + P[B] P[A and BJ = P[A] + P[B] P[A] P[B] , where the last equality follows by independence, to evaluate
P[max . . . ]}

Problem 1 . 8
The Hoeffd i ng I nequ a l ity is one form of the law of large
numbers. One of the sim plest forms of that law is the Chebyshev Inequality,
which you wil l prove here.
(a) If t is a non negative random varia ble, prove that for a ny a > 0,

JP' [t a] :S; JE(t)/a.

( b) If u is a ny ra ndom variable with mean a nd variance

2
a ny a > 0, JP' [ (u ) 2: a] :S;
[Hint: Use (a)]

2
0" ,

prove that for

(c) If u1 ,
, UN are iid random varia bles, each with mean and varia nce 0"
and u = tr l.:=l Un , prove that for any a > 0,

JP' [ (u

(]"

) 2 2: a] :S; Na .

Notice that the RHS of this Chebyshev I nequality goes down linearly in N,
while the cou nterpart in Hoeffding's I neq uality goes down exponenti a lly. In
P roblem 1 .9, we develop an exponential bound using a similar a pproach.

36

1.

1.5.

THE LEARNING PROBLEM

PROBLEMS

Problem 1 . 9

In this problem , we derive a form of the law of large n u mbers

that has a n exponential bound, cal led the Chernoff bound. We focus on the
simple case of flipping a fair coin , a nd use an a pproach similar to P roblem 1 .8.
(a) Let t be a (fin ite) ra ndom variable, a be a positive consta nt, a nd
positive para meter. If T(s) = E ( est) , prove that

s be a

[Hint: est is monotonically increasing in t.]

u1 , , uN be iid random varia bles, and let

U(s) = lEun (es un ) (for any n ) , prove that

(b) Let

= if L::= l Un . If

(c) S uppose lP'[un = O] = IP[un = 1] = (fa i r coin ) . Eval u ate U(s) as

a fun ction of s, and minim ize e-sa u(s) with respect to s for fixed a,

O < a < l.

(d) Conclude in (c) that, for 0 < E <

JP'[u JE(u) + E] :: Tf3 N ,

where (3 = 1 + ( + E) log2 ( + E) + ( - E) log2 ( - E) a n d E (u) =
Show that (3 > 0, hence the bound is exponentia l ly decreasing in N.

Problem 1 . 10
Assume that X = {x 1 , x2 , . . . , xN , XN+ 1 , . . . , xN +M }
a nd Y = { - 1, + 1 } with an u nknown target function f : X --+ Y. The tra i n i ng
data set V is (x1 , y1 ) , . . , (xN , YN ) . Define the off-training-set error of a
hypothesis h with respect to f by

1 M
I: [h (xN+m ) -I f(XN +m ) ] .
M =
m l
(a) Say f (x) = + 1 for a l l x a nd
1 , for x = X k a nd k is odd and 1 :: k :: M + N
h(x) = +
otherwise
-1,
What is Eoff (h, f)?
(b) We say that a target function f can 'generate' V in a noiseless setting
if Yn = f (xn ) for a l l (xn , Yn ) E D. For a fixed V of size N, how m a ny
possible f : X --+ Y can generate V in a noiseless setting?
Eoff (h, f) =

( c) For a given hypothesis h a nd a n i nteger k between 0 a nd M, how many

of those f i n (b) satisfy Eoff (h, f) = -it ?
( d) For a given hypothesis h, if a l l those f that generate V in a noiseless
setting are equ a l ly l i kely in proba bility, what is the expected off training
set error E1 [Eoff (h, ! )]?
(continued on next page)

37

1 . THE LEARNING PROBLEM

1 . 5 . PROBLEMS

( e) A d eterministic a l gorithm A is defined as a procedu re that takes V as

an i nput, and outputs a hypothesis h = A(V) . Argue that for a ny two
deterministic a lgorithms Ai a nd A2 ,

You have now proved that i n a noiseless setting, for a fixed V, if a l l possible f
a re equ a l l y likely, any two deterministic algorithms a re eq u iva lent in terms of the
expected off tra ining set error. Similar results can be proved for more genera l
settings.

Problem 1 . 1 1

The matrix which tab u lates the cost of various errors for
the C I A a nd Supermarket a pplications in Exa mple 1 . 1 is ca l led a risk or loss

matrix.
For the two risk matrices in Exa mple 1 . 1 , explicitly write down the in sa m ple
error Ein that one shou ld minimize to obta in g . This in-sa mple error should
weight the different types of errors based on the risk matrix. [Hint: Consider
Yn = + 1 and Yn = - 1 separately.]

Problem 1 . 12

This problem i nvestigates how changing the error measu re

ca n cha nge the result of the learning process. You have N data points y1 ::
:: YN and wish to estimate a ' representative' val ue.
(a) If you r a lgorith m is to find the hypothesis
sum of sq uared deviations,

h that m i n i m izes the in sa mple

Ein (h) = L (h - Yn ) 2 ,
n=l
then show that you r estimate wil l be the in sa mple mea n ,

hmea n

L Yn
N
1

n=l

(b) If your a lgorith m is to find the hypothesis

su m of absol ute deviations,

h that m i n i m izes the in sa mple

Ein (h) = L l h - Yn l ,
n= l
then show that you r estimate will be the in sa mple median hmed . which
is any va lue for which half the data points are at most hmed and h a lf the
data points are at least hmed

(c) S u ppose Y N is pertu rbed to YN + E, where E -+ oo . So, the single data

point YN becomes a n outl ier. What happens to you r two estimators hmean
and hmed?

38

Chapter

Training versus Testing

Before the final exam, a professor may hand out some practice problems and
solutions to the class. Although these problems are not the exact ones that
will appear on the exam, studying them will help you do better. They are the
'training set' in your learning.
If the professor's goal is to help you do better in the exam, why not give
out the exam problems themselves? Well, nice try @. Doing well in the
exam is not the goal in and of itself. The goal is for you to learn the course
material. The exam is merely a way to gauge how well you have learned the
material. If the exam problems are known ahead of time, your performance
on them will no longer accurately gauge how well you have learned.
The same distinction between training and testing happens in learning from
data. In this chapter, we will develop a mathematical theory that characterizes
this distinction. We will also discuss the conceptual and practical implications
of the contrast between training and testing.
2.1

Theory of Generalization

The out-of-sample error Eout measures how well our training on D has gener
alized to data that we have not seen before. Eout is based on the performance
over the entire input space X . Intuitively, if we want to estimate the value
of Eout using a sample of data points, these points must be 'fresh' test points
that have not been used for training, similar to the questions on the final exam
that have not been used for practice.
The in sample error Ein, by contrast, is based on data points that have
been used for training. It expressly measures training performance, similar to
your performance on the practice problems that you got before the final exam.
Such performance has the benefit of looking at the solutions and adjusting
accordingly, and may not reflect the ultimate performance in a real test . We
began the analysis of in-sample error in Chapter 1 , and we will extend this
39

2 . 1 . THEORY OF GENERALIZATION

2 . TRAINING VERSUS TESTING

analysis to the general case in this chapter. We will also make the contrast
between a training set and a test set more precise.
A word of warning: this chapter is the heaviest in this book in terms of
mathematical abstraction. To make it easier on the not-so-mathematically
inclined, we will tell you which part you can safely skip without 'losing the
plot' . The mathematical results provide fundamental insights into learning
from data, and we will interpret these results in practical terms.
Generalization error. We have already discussed how the value of Ein
does not always generalize to a similar value of Eout . Generalization is a key
issue in learning. One can define the generalization error as the discrepancy
between Ein and Eout 1 The Hoeffding Inequality
provides a way to
characterize the generalization error with a probabilistic bound,

(1.6)

for any E > 0. This can be rephrased as follows. Pick a tolerance level
example o = 0.05 , and assert with probability at least
o that

(2.1 )
(1.6 )

1
2M .
2N o

8,

for

(2.1 )

We refer to the type of inequality in

as a generalization bound because
it bounds Eout in terms of Ein. To see that the Hoeffding Inequality implies
this generalization bound, we rewrite
as follows: with probability at least
IEout Ein l ::; E, which implies Eout ::; Ein + E. We may now
follows.
from which E =
ln
and
identify o =
Notice that the other side of IEout Ein l ::; E also holds, that is, Eout 2:
Ein - E for all h E 1-l. This is important for learning, but in a more subtle way.
Not only do we want to know that the hypothesis g that we choose (say the
one with the best training error) will continue to do well out of sample (i.e. ,
Eout ::; Ein + E ) , but we also want to be sure that we did the best we could
with our 1-l (no other hypothesis h E 1-l has Eout h) significantly better than
Eout (g)). The Eout (h) 2: Ein h) - E direction of the bound assures us that
we couldn't do much better because every hypothesis with a higher Ein than
the g we have chosen will have a comparably higher Eout .
or 'error bar' if you will, depends
The error bound
ln
in
on IV!, the size of the hypothesis set 1-l. If 1-l is an infinite set, the bound goes
to infinity and becomes meaningless. Unfortunately, almost all interesting
learning models have infinite 1-l, including the simple perceptron which we
discussed in Chapter
In order t o study generalization in such models, we need t o derive a coun
terpart to
that deals with infinite 1-l. We would like to replace
with

1 21Vle 2NE2 ,
2Me 2NE2 ,

(2. 1 )

(2 . 1 ) ,

(2.1 )

1.

1 Sometimes 'generalization error' is used

as

another name for

40

Eout, but not in this book.

2. TRAINING VERSUS TESTING

2. 1 . THEORY OF GENERALIZATION

something finite, so that the bound is meaningful. To do this, we notice that

the way we got the M factor in the first place was by taking the disjunction
of events:

" J Ein (h1) Eout (h1 ) J

" JEin (h2 ) Eout (h2 ) J

>
>

"

"
E

or
or

(2.2)
which is guaranteed to include the event " JEin (g) Eout (g) J > E" since g is al
ways one of the hypotheses in 1-l. We then over-estimated the probability using
the union bound. Let Bm be the (Bad) event that " J Ein(hm) Eout(hm ) J > E" .
Then,

If the events B1 , B2 ,
, BM are strongly
overlapping, the union bound becomes par
ticularly loose as illustrated in the figure to
the right for an example with 3 hypotheses;
the areas of different events correspond to
their probabilities. The union bound says
that the total area covered by 81 , B2 , or Bs
is smaller than the sum of the individual ar
eas, which is true but is a gross overestimate
when the areas overlap heavily as in this ex
ample. The events " JEin(hm) Eout (hm) J >
"
E ; m = 1,
, JV[, are often strongly overlap
ping. If h1 is very similar to h2 for instance,
the two events " JEin(h1) Eout (h1 ) J > E" and " JEin (h2 ) Eout (h2 ) J > E" are
likely to coincide for most data sets. In a typical learning model, many hy
potheses are indeed very similar. If you take the perceptron model for instance,
as you slowly vary the weight vector w , you get infinitely many hypotheses
that differ from each other only infinitesimally.
The mathematical theory of generalization hinges on this observation.
Once we properly account for the overlaps of the different hypotheses, we
will be able to replace the number of hypotheses M in (2. 1 ) by an effective
number which is finite even when ]\If is infinite, and establish a more useful
condition under which Eout is close to Ein.

2. 1. 1

Effective Number o f Hypotheses

We now introduce the growth function, the quantity that will formalize the
effective number of hypotheses. The growth function is what will replace 11/f
41

2. TRAINING VERSUS TESTING

2 . 1 . THEORY OF GENERALIZATION

in the generalization bound (2. 1 ) . It is a combinatorial quantity that cap

tures how different the hypotheses in 1-l are, and hence how much overlap the
different events in (2. 2) have.
We will start by defining the growth function and studying its basic prop
erties. Next, we will show how we can bound the value of the growth function.
Finally, we will show that we can replace M in the generalization bound with
the growth function. These three steps will yield the generalization bound that
we need, which applies to infinite 1-l. We will focus on binary target functions
for the purpose of this analysis, so each h E 1-l maps X to { - 1 , + 1 } .
The definition o f the growth function i s based on the number o f different
hypotheses that 1-l can implement, but only over a finite sample of points
rather than over the entire input space X. If h E 1-l is applied to a finite sample
x1 , . . . , xN E X , we get an N-tuple h(x1 ) , . . . , h(xN) of l's. Such an N-tuple
is called a dichotomy since it splits x1 ,
, XN into two groups: those points for
which h is - 1 and those for which h is + 1 . Each h E 1-l generates a dichotomy
on x1 ,
, XN , but two different h's may generate the same dichotomy if they
happen to give the same pattern of 1 's on this particular sample.

Definition 2 . 1 . Let x1 ,

these points are defined by

, XN E X . The dichotomies generated by 1-l on

1-l (x1 , . . , XN ) = { (h(x1 ) , . . , h(xN )) I h E 1-l} .

(2.3)

One can think of the dichotomies 1-l(xi ,

, XN) as a set of hypotheses just
like 1-l is, except that the hypotheses are seen through the eyes of N points
only. A larger 1-l(x1 ,
, XN ) means 1-l is more 'diverse' generating more
dichotomies on x1 ,
, XN . The growth function is based on the number of
dichotomies.

Definition 2.2. The growth function is defined for a hypothesis set 1-l by

where I I denotes the cardinality (number of elements) of a set.

In words, mti ( N) is the maximum number of dichotomies that can be gen

erated by 1-l on any N points. To compute mH (N) , we consider all possible
choices of N points x1 ,
, XN from X and pick the one that gives us the
most dichotomies. Like ]\![, mH (N) is a measure of the number of hypotheses
in 1-l, except that a hypothesis is now considered on N points instead of the
entire X. For any 1-l , since 1-l (x1 ,
, xN ) { - 1 , +l} N (the set of all possible
dichotomies on any N points) , the value of mH (N) is at most l { - 1 , + l } N I ,
hence
mH (N) ::; 2 N .

If 1-l is capable of generating all possible dichotomies on x1 ,

1-l (x1 ,
, XN) = { - 1 , + 1 } N and we say that 1-l can shatter x1 ,
signifies that 1-l is as diverse as can be on this particular sample.

42

, XN , then
, XN . This

2 . 1. THEORY OF GENERALIZATION

2. TRAINING VERSUS TESTING

( a)

(b)

( c)

Figure 2 . 1 : Illustration of the growth function for a two dimensional per

ceptron. The dichotomy of red versus blue on the 3 colinear points in part
( a) cannot be generated by a perceptron, but all 8 dichotomies on the 3
points in part (b ) can. By contrast, the dichotomy of red versus blue on
the 4 points in part ( c) cannot be generated by a perceptron. At most 14
out of the possible 16 dichotomies on any 4 points can be generated.

Example 2 . 1 . If X is a Euclidean plane and 1-l is a two-dimensional percep

tron, what are m1-l (3) and m1-l (4)? Figure 2 . l ( a) shows a dichotomy on 3 points
that the perceptron cannot generate, while Figure 2 . l ( b ) shows another 3
points that the perceptron can shatter, generating all 2 3 = 8 dichotomies.
Because the definition of m1-l ( N) is based on the maximum number of di
chotomies, m1-l (3) = 8 in spite of the case in Figure 2.l ( a) .
In the case of 4 points, Figure 2 . 1 ( c ) shows a dichotomy that the perceptron
cannot generate. One can verify that there are no 4 points that the perceptron
can shatter. The most a perceptron can do on any 4 points is 14 dichotomies
out of the possible 16, where the 2 missing dichotomies are as depicted in
Figure 2 . l ( c ) with blue and red corresponding to - 1 , +1 or to + 1 , - 1 . Hence,
D
m1-l (4) = 14.

Let us now illustrate how to compute mH (N) for some simple hypothesis
sets. These examples will confirm the intuition that m1-l ( N) grows faster
when the hypothesis set 1-l becomes more complex. This is what we expect of
a quantity that is meant to replace l\!f in the generalization bound ( 2 . 1 ) .
Example 2 . 2 . Let us find a formula for mH (N) in each of the following cases.
1. Positive rays: 1-l consists of all hypotheses h : R -7 { - 1 , + 1} of the form
h ( x ) = sign ( x - a ) , i.e. , the hypotheses are defined in a one-dimensional
input space, and they return -1 to the left of some value a and + 1 to
the right of a.

43

2. 1 . THEORY OF GENERALIZATION

2 . TRAINING VERSUS TESTING

N(N),+

To compute m1-l
we notice that given points, the line is split by
the points into
1 regions. The dichotomy we get on the
points
is decided by which region contains the value a . As we vary a, we will
get N + 1 different dichotomies. Since this is the most we can get for
any points, the growth function is

N
(N)

N+

Notice that if we picked

points where some of the points coincided
(which is allowed) , we will get less than
1 dichotomies. This does
not affect the value of m1-l
since it is defined based on the maximum
number of dichotomies.
2. Positive intervals : 1-l consists of all hypotheses in one dimension that
return + 1 within some interval and -1 otherwise. Each hypothesis is
specified by the two end values of that interval.

(N), N +

To compute m1-l
we notice that given
points, the line is again
split by the points into
1 regions. The dichotomy we get is decided
by which two regions contain the end values of the interval, resulting
in
different dichotomies. If both end values fall in the same
region, the resulting hypothesis is the constant -1 regardless of which
region it is. Adding up these possibilities, we get

(Nil )

(N) (N+l) + N2 + N +
N,
(N) (
m1-l

Notice that m1-l N ) grows as the square of

ear m1-l
of the 'simpler' positive ray case.

1.

faster than the lin

JR2

3. Convex sets : 1-l consists of all hypotheses in two dimensions h :

-+
{ - 1 , 1} that are positive inside some convex set and negative elsewhere
( a set is convex if the line segment connecting any two points in the set
lies entirely within the set ) .
To compute m1-l
in this case, we need to choose the points care
fully. Per the next :figure, choose N points on the perimeter of a circle.
Now consider any dichotomy on these points, assigning an arbitrary pat
tern of 1 's to the points. If you connect the 1 points with a polygon,
the hypothesis made up of the closed interior of the polygon (which has
to be convex since its vertices are on the perimeter of a circle ) agrees
with the dichotomy on all points. For the dichotomies that have less
than three 1 points, the convex set will be a line segment, a point, or
an empty set.

(N)
N

44

2.

2.1.

TRAINING VERSUS TESTING

THEORY OF GENERALIZATION

This means that any dichotomy on these N points can be realized using a
convex hypothesis, so 1-l manages to shatter these points and the growth
function has the maximum possible value

Notice that if the N points were chosen at random in the plane rather
than on the perimeter of a circle, many of the points would be 'internal'
and we wouldn't be able to shatter all the points with convex hypotheses
as we did for the perimeter points. However, this doesn't matter as far
as mH (N) is concerned, since it is defined based on the maximum (2 N
D
in this case) .
It is not practical to try to compute m11 ( N) for every hypothesis set we use.
Fortunately, we don't have to. Since mH (N) is meant to replace ]\If in (2. 1 ) ,
we can use an upper bound o n m 1l (N ) instead of the exact value, and the
inequality in (2. 1) will still hold. Getting a good bound on mH (N) will prove
much easier than computing m1l ( N) itself, thanks to the notion of a break
point.
Definition 2 . 3. If no data set of size k can be shattered by 1-l, then k is said

to be a break point for 1-l .

If k is a break point, then mH (k) < 2k . Example 2 . 1 shows that k = 4 is a

break point for two-dimensional perceptrons. In general, it is easier to find a
break point for 1-l than to compute the full growth function for that 1-l .
Exercise 2 . 1
B y i nspection , find a break point k for each hypothesis set in Example 2 . 2

( if there i s one ) . Verify that m11, (k) < 2 k using the form u las derived i n

that Example.

We now use the break point k to derive a bound on the growth function m11 (N)
for all values of N. For example, the fact that no 4 points can be shattered by
45

2. TRAINING VERSUS TESTING

2. 1 . THEORY OF GENERALIZATION

the two-dimensional perceptron puts a significant constraint on the number of

dichotomies that can be realized by the perceptron on 5 or more points. We
will exploit this idea to get a significant bound on m1-l ( N) in general.

2. 1.2

Bounding the Growth Function

The most important fact about growth functions is that if the condition
m1-l ( N) = 2 N breaks at any point, we can bound m1-l ( N) for all values of N
by a simple polynomial based on this break point. The fact that the bound
is polynomial is crucial. Absent a break point (as is the case in the convex
hypothesis example) , m1-l ( N ) = 2N for all N. If m1-l ( N ) replaced M in Equaln
on the generalization error would not go to
tion ( 2 . 1 ) , the bound
zero regardless of how many training examples N we have. However, if m1-l ( N)
can be bounded by a polynomial any polynomial , the generalization error
will go to zero as N -- oo . This means that we will generalize well given a
sufficient number of examples.
safe skip: If you trust our math, you can
the following part without compromising the
sequence. A similar green box will tell you when
rejoin.

To prove the polynomial bound, we will introduce a combinatorial quantity

that counts the maximum number of dichotomies given that there is a break
point, without having to assume any particular form of 1-l. This bound will
therefore apply to any 1-l.
Definition 2.4. B (N, k) is the maximum number of dichotomies on N points
such that no subset of size k of the N points can be shattered by these di

chotomies.

The definition of B (N, k} assumes a break point k, then tries to find the
most dichotomies on N points without imposing any further restrictions.
Since B ( N, k) is defined as a maximum, it will serve as an upper bound for
any m1-l (N ) that has a break point k;

m1-l (N) ::; B (N, k) if k is a break point for 1-l .

The notation B comes from ' Binomial' and the reason will become clear
shortly. To evaluate B (N, k) , we start with the two boundary conditions
k = 1 and N = 1 .
B (N, 1)
B ( l , k)

1
2 for k > 1 .
46

2.

2.1.

TRAINING VERSUS TESTING

THEORY OF GENERALIZATION

B ( N, 1) = 1 for all N since if no subset of size 1 can be shattered, then only

one dichotomy can be allowed. A second different dichotomy must differ on at
least one point and then that subset of size 1 would be shattered. B ( l , k) = 2
for k > 1 since in this case there do not even exist subsets of size k; the
constraint is vacuously true and we have 2 possible dichotomies ( + 1 and - 1 )
o n the one point .
We now assume N 2: 2 and k 2: 2 and try t o develop a recursion. Consider
the B (N, k) dichotomies in definition 2 .4, where no k points can be shattered.
We list these dichotomies in the following table,
# of rows

X1 X2

XN - 1 XN
+1
+1

+1
-1

-1
+1
-1
-1

-1
-1
+1
+1

-1
+1

+1
-1
+1
-1

-1
-1
-1
-1

+1
-1
+1
+1

+1
-1

-1
-1

+1
-1

+1
-1

+1
+1

+1
-1
+1
-1

S1

(3

S2
(3

where x1 , , XN in the table are labels for the N points of the dichotomy.
We have chosen a convenient order in which to list the dichotomies, as follows.
Consider the dichotomies on xi , , XN -l Some dichotomies on these N
points appear only once (with either + 1 or - 1 in the X N column, but not
both) . We collect these dichotomies in the set S1 . The remaining dichotomies
on the first N 1 points appear twice, once with + 1 and once with - 1 in
the X N column. We collect these dichotomies in the set S2 which can be
divided into two equal parts, St and S-;; (with + 1 and - 1 in the XN column,
respectively) . Let S1 have a rows, and let st and s-;; have (3 rows each. Since
the total number of rows in the table is B (N, k) by construction, we have
-

B (N, k) = a + 2(3.

(2.4)

The total number of different dichotomies on the first N 1 points is given

by a + (3; since st and S2 are identical on these N 1 points, their di
chotomies are redundant. Since no subset of k of these first N 1 points can
-

47

2.

2. 1 .

TRAINING VERSUS TESTING

THEORY OF GENERALIZATION

be shattered (since no k-subset of all N points can be shattered) , we deduce

that
a + ,B ::; B (N 1 , k )
(2. 5)

by definition of B . Further, no subset of size k 1 of the first N 1 points can

be shattered by the dichotomies in st. If there existed such a subset, then
taking the corresponding set of dichotomies in 82 and adding XN to the data
points yields a subset of size k that is shattered, which we know cannot exist
in this table by definition of B ( N, k) . Therefore,
,8 ::; B (N

- 1, k

1) .

(2.6)

Substituting the two Inequalities (2.5) and (2.6) into (2.4) , we get
B(N, k) ::; B (N

- 1 , k) + B (N

1, k

1).

(2.7)

We can use (2 .7) to recursively compute a bound on B (N, k) , as shown in the

following table.

1
2
3

1
1
1
1

2
2
3
4

3
2
4
7

\i

k
4
2
4
8

5
2
4
8

6
2
4
8

11

where the first row (N = 1) and the first column (k = 1) are the bound
ary conditions that we already calculated. We can also use the recursion to
bound B ( N, k) analytically.
Lemma 2 . 3 (Sauer's Lemma) .

B (N, k) '.O

( )

Proof. The statement is true whenever k = 1 or N = 1 , by inspection. The

proof is by induction on N. Assume the statement is true for all N ::; N0
and all k . We need to prove the statement for N = N0 + 1 and all k . Since
the statement is already true when k = 1 (for all values of N) by the initial
condition, we only need to worry about k 2: 2. By (2. 7) ,
B(No + 1 , k) ::; B (No , k)
48

+ B (No , k

- 1) .

2.

2.1 .

TRAINING VERSUS TESTING

THEORY OF GENERALIZATION

Applying the induction hypothesis to each term on the RHS , we get

B (No + l , k)

<

( ) ( )
( ) c )
[( ) ( )]
0

1+

1+

"

( Nt 1 ) ( 1:0 ) ( i1!_01 )

+
where the combinatorial identity
has been used.
This identity can be proved by noticing that to calculate the number of ways
to pick i objects from N0 + 1 distinct objects, either the first object is included,
ways, or the first object is not included, in
ways. We have
in
thus proved the induction step, so the statement is true for all N and k. II

( 1:0 )

( i1!_01 )

( )

It turns out that B (N, k) in fact equals 2=7:

(see Problem 2.4) , but
we only need the inequality of Lemma 2.3 to bound the growth function. For
a given break point k, the bound 2=7:
is polynomial in N, as each term
in the sum is polynomial (of degree i :: k 1 ) . Since B ( N, k) is an upper
bound on any mH (N) that has a break point k, we have proved

( )

End safe skip: Those who skipped are now rejoining

us. The next theorem states that any growth function
m1l ( N) with a break point is bounded by a polyno
mial.
Theorem 2 . 4. If m1l (k)

< 2k

for some value k, then

(2 .8)

for all N. The RHS is polynomial in N of degree k

1.

The implication of Theorem 2.4 is that if H has a break point, we have

what we want to ensure good generalization; a polynomial bound on mH (N) .

49

2.

2. 1 .

TRAINING VERSUS TESTING

Exercise

THEORY OF GENERALIZATION

2.2

(a) Verify the bound of Theorem 2.4 i n the three cases of Exa mple 2.2:

(i) Positive rays: 1-l consists of all hypotheses i n one dimension of

the form h(x) = sign(x a) .
( ii) Positive i ntervals: 1-l consists of a l l hypotheses in one dim ension
that a re positive withi n some i nterval a nd negative elsewhere.
( iii) Convex sets: 1-l consists of a l l hypotheses in two dimensions that
a re positive i nside some convex set a nd negative elsewhere.
-

( Note: you can use the break points you found in Exercise 2 . 1. )
(b) Does there exist a hypothesis set fo r which m1i (N)
(whe re LN/2j is the largest integer N/2)?

2. 1. 3

= N

2LN/ 2J

The VC Dimension

Theorem 2. 4 bounds the entire growth function in terms of any break point.
The smaller the break point, the better the bound. This leads us to the fol
lowing definition of a single parameter that characterizes the growth function.
Definition 2 . 5 . The Vapnik- Chervonenkis dimension of a hypothesis set ti,
denoted by dvc (ti) or simply dvc , is the largest value of N for which mH ( N ) =
2N . If mH ( N ) = 2 N for all N, then dvc (ti) = oo.

If dvc i s the VC dimension o f ti, then k = dvc + 1 i s a break point for m1-l
since m1-l ( N ) cannot equal 2 N for any N > dvc by definition. It is easy to see
that no smaller break point exists since ti can shatter dvc points, hence it can
also shatter any subset of these points.
Exercise

2.3

Compute the VC dimension of 1-l for the hypothesis sets in parts (i), (ii),
(iii) of Exercise 2.2(a) .

Since k = dvc + 1 is a break point for m1-l , Theorem 2.4 can be rewritten in
terms of the VC dimension:
mH ( N )

dvc

()
N
i

(2.9)

Therefore, the VC dimension is the order of the polynomial bound on m1-l ( N ) .

It is also the best we can do using this line of reasoning, because no smaller
break point than k = dvc + 1 exists. The form of the polynomial bound can
be further simplified to make the dependency on dvc more salient. We state a
useful form here, which can be proved by induction ( Problem 2.5) .

(2. 10)
50

2.

2. 1.

TRAINING VERSUS TESTING

THEORY OF GENERALIZATION

Now that the growth function has been bounded in terms of the VC dimen
sion, we have only one more step left in our analysis, which is to replace the
number of hypotheses JV[ in the generalization bound (2.1) with the growth
function m1-l (N) . If we manage to do that, the VC dimension will play a
pivotal role in the generalization question. If we were to directly replace M
by mH (N) in (2. 1 ) , we would get a bound of the form

Unless dvc (H) = oo, we know that mH (N) is bounded by a polynomial in N;

thus, ln m1-l (N) grows logarithmically in N regardless of the order of the poly
nomial, and so it will be crushed by the -k factor. Therefore, for any fixed
tolerance 8, the bound on Eout will be arbitrarily close to Ein for sufficiently
large N.
Only if dvc (H) = oo will this argument fail, as the growth function in this
case is exponential in N. For any finite value of dvc , the error bar will converge
to zero at a speed determined by dvc , since dvc is the order of the polynomial.
The smaller dvc is, the faster the convergence to zero.
It turns out that we cannot just replace M with m1-l (N) in the generaliza
tion bound (2. 1) , but rather we need to make other adjustments as we will see
shortly. However, the general idea above is correct, and dvc will still play the
role that we discussed here. One implication of this discussion is that there
is a division of models into two classes. The 'good models' have finite dvc ,
and for sufficiently large N, Ein will be close to Eout; for good models, the
in-sample performance generalizes to out of sample. The 'bad models' have
infinite dvc . With a bad model, no matter how large the data set is, we cannot
make generalization conclusions from Ein to Eout based on the VC analysis.
Because of its significant role, it is worthwhile to try to gain some insight
about the VC dimension before we proceed to the formalities of deriving the
new generalization bound. One way to gain insight about dvc is to try to
compute it for learning models that we are familiar with. Perceptrons are one
case where we can compute dvc exactly. This is done in two steps. First,
we show that dvc is at least a certain value, then we show that it is at most
the same value. There is a logical difference in arguing that dvc is at least a
certain value, as opposed to at most a certain value. This is because

dvc 2. N there exists

D of size N such that }{ shatters D,

hence we have different conclusions in the following cases.

1 . There is a set of N points that can be shattered by }{ . In this case, we
can conclude that dvc 2. N.
2 In some cases with infinite dvc , such as the convex sets that w e discussed, alternative
analysis based on an ' average' growth function can establish good generalization behavior.

51

2.

2.1.

TRAINING VERSUS TESTING

THEORY OF GENERALIZATION

2. Any set of N points can be shattered by 1-l. In this case, we have more
than enough information to conclude that dvc N.
3. There is a set of N points that cannot be shattered by 1-l. Based only on
this information, we cannot conclude anything about the value of dvc
4. No set of N points can be shattered by 1-l . In this case, we can conclude
that dvc < N.
Exercise 2.4

Consider the i n put space

x ]Rd ( including the constant coordinate
dimension of the perceptron (with d 1
= 1). Show that the
para m eters, counting wo ) is exactly
1
1 by showing that it is at lea st d
and at most d 1, a s follows.
xo

(a )

( b)

show that dvc

1, find
1 points i n
can shatter. [Hint: Construct a nonsingular

To

that the perceptron

1) x
1) matrix
whose rows represent the d 1 points, then use the nonsingu/arity to
argue that the perceptron can shatter these points.]
To show that dvc
d 1, show that no set of d 2 points i n
can be shattered by the perceptron. [Hint: Represent each point
as a vector of length d 1, then use the fact that any d 2
in

vectors of length d 1 have to be linearly dependent. This means

that some vector is a linear combination of all the other vectors.
Now, if you choose the class of these other vectors carefully, then the
classification of the dependent vector will be dictated. Conclude that
there is some dichotomy that cannot be implemented, and therefore
that for N d 2, m1-l (N) < 2N.J

The VC dimension of a d-dimensional perceptron3 is indeed d + 1 . This is

consistent with Figure 2 . 1 for the case d = 2 , which shows a VC dimension
of 3. The perceptron case provides a nice intuition about the VC dimension,
since d + 1 is also the number of parameters in this model. One can view
the VC dimension as measuring the 'effective' number of parameters. The
more parameters a model has, the more diverse its hypothesis set is, which
is reflected in a larger value of the growth function mH ( N) . In the case
of perceptrons, the effective parameters correspond to explicit parameters in
the model, namely wo, wi,
, Wd In other models, the effective parameters
may be less obvious or implicit. The VC dimension measures these effective
parameters or 'degrees of freedom' that enable the model to express a diverse
set of hypotheses.
Diversity is not necessarily a good thing in the context of generalization.
For example, the set of all possible hypotheses is as diverse as can be, so
mH (N) = 2 N for all N and dvc ( H ) = oo. In this case, no generalization at all
is to be expected, as the final version of the generalization bound will show.

3X

{1}

JRd is considered d dimensional since the first coordinate x o

52

1 is fixed.

2.

2.1 .

TRAINING VERSUS TESTING

2. 1.4

THEORY OF GENERALIZATION

The VC Generalization Bound

If we treated the growth function as an effective number of hypotheses, and

replaced M in the generalization bound (2. 1) with m1-l (N) , the resulting bound
would be
2 m1-l (N)
?
1
l
n
(2. 1 1 )
Eout (g) :s; Ein (g)
2N
8 .
It turns out that this i s not exactly the form that will hold. The quantities in
red need to be technically modified to make (2. 1 1 ) true. The correct bound,
which is called the VC generalization bound, is given in the following theorem;
it holds for any binary target function f, any hypothesis set 1-l, any learning
algorithm A, and any input probability distribution P.

Theorem 2.5 (VC generalization bound) . For any tolerance

Eout (g)
with probability 2 1

:s;

Ein(g)

8
4m1-l (2N)
ln
N
8

8 > 0,
(2. 12)

8.

If you compare the blue items in (2. 12) to their red counterparts in (2. 1 1 ) , you
notice that all the blue items move the bound in the weaker direction. How
ever, as long as the VC dimension is finite, the error bar still converges to zero
(albeit at a slower rate) , since m1-l (2N) is also polynomial of order dvc in N,
just like m1-l (N) . This means that, with enough data, each and every hypoth
esis in an infinite 1-l with a finite VC dimension will generalize well from Ein
to Eout. The key is that the effective number of hypotheses, represented by
the finite growth function, has replaced the actual number of hypotheses in
the bound.
The VC generalization bound is the most important mathematical result
in the theory of learning. It establishes the feasibility of learning with infinite
hypothesis sets. Since the formal proof is somewhat lengthy and technical, we
illustrate the main ideas in a sketch of the proof, and include the formal proof
as an appendix. There are two parts to the proof; the justification that the
growth function can replace the number of hypotheses in the first place, and
the reason why we had to change the red items in (2. 1 1 ) into the blue items
in (2. 12) .
Sketch of the proof. The data set V is the source of randomization in the
original Hoeffding Inequality. Consider the space of all possible data sets. Let
us think of this space as a 'canvas' (Figure 2 . 2(a) ) . Each V is a point on that
canvas. The probability of a point is determined by which Xn 's in X happen to
be in that particular V, and is calculated based on the distribution P over X .
Let 's think of probabilities of different events as areas on that canvas, s o the
total area of the canvas is 1 .
53

2.

TRAINING VERSUS TESTING

2. 1 . THEORY OF GENERALIZATION

space of
data sets

( a) Hoeffding Inequality

( b) Union Bound

( c) VC Bound

Figure 2.2: Illustration of the proof of the VC bound, where the 'canvas'
represents the space of all data sets, with areas corresponding to probabili
ties. ( a) For a given hypothesis, the colored points correspond to data sets
where Ein does not generalize well to Eout The Hoeffding Inequality guar
antees a small colored area. ( b) For several hypotheses, the union bound
assumes no overlaps, so the total colored area is large. ( c) The VC bound
keeps track of overlaps, so it estimates the total area of bad generalization
to be relatively small.

For a given hypothesis h E 1-i , the event " IEin(h) Eout(h) I > E" consists
of all points V for which the statement is true. For a particular h, let us paint
all these 'bad' points using one color. What the basic Hoeffding Inequality
tells us is that the colored area on the canvas will be small (Figure 2.2 ( a)) .
Now, if we take another h E 1-i , the event " IEin(h) Eout(h) I > E" may
contain different points, since the event depends on h. Let us paint these points
with a different color. The area covered by all the points we colored will be
at most the sum of the two individual areas, which is the case only if the two
areas have no points in common. This is the worst case that the union bound
considers. If we keep throwing in a new colored area for each h E 1-i, and never
overlap with previous colors, the canvas will soon be mostly covered in color
( Figure 2.2 (b)) . Even if each h contributed very little, the sheer number of
hypotheses will eventually make the colored area cover the whole canvas. This
was the problem with using the union bound in the Hoeffding Inequality ( 1 . 6 ) ,
and not taking the overlaps o f the colored areas into consideration.
The bulk of the VC proof deals with how to account for the overlaps. Here
is the idea. If you were told that the hypotheses in 1-i are such that each
point on the canvas that is colored will be colored 100 times (because of 100
different h's ) , then the total colored area is now 1/100 of what it would have
been if the colored points had not overlapped at all. This is the essence of
the VC bound as illustrated in (Figure 2.2 ( c )) . The argument goes as follows.
54

2.

2.2.

TRAINING VERSUS TESTING

INTERPRETING THE B OUND

Many hypotheses share the same dichotomy on a given D, since there are
finitely many dichotomies even with an infinite number of hypotheses. Any
statement based on D alone will be simultaneously true or simultaneously
false for all the hypotheses that look the same on that particular D. What
the growth function enables us to do is to account for this kind of hypothesis
redundancy in a precise way, so we can get a factor similar to the ' 100' in the
above example.
When 1-l is infinite, the redundancy factor will also be infinite since the
hypotheses will be divided among a finite number of dichotomies. Therefore,
the reduction in the total colored area when we take the redundancy into
consideration will be dramatic. If it happens that the number of dichotomies
is only a polynomial, the reduction will be so dramatic as to bring the total
probability down to a very small value. This is the essence of the proof of
Theorem 2.5.
The reason m 1-l ( 2N) appears in the VC bound instead of m 1-l (N) is that
the proof uses a sample of 2N points instead of N points. Why do we need 2N
points? The event " IEin(h) Eout (h) J > E" depends not only on D, but also on
the entire X b ecause Eout ( h) is based on X. This breaks the main premise of
grouping h's based on their behavior on D, since aspects of each h outside of D
affect the truth of " JEin(h) Eout (h) J > E." To remedy that , we consider the
artificial event "IEin(h) E{n (h) J > E" instead, where Ein and E{n are based
on two samples D and D' each of size N. This is where the 2N comes from.
It accounts for the total size of the two samples D and D'. Now, the truth of
the statement " IEin(h) E{n (h) J > E" depends exclusively on the total sample
of size 2N, and the above redundancy argument will hold.
Of course we have to justify why the two-sample condition "JEin ( h)
E{n (h) J > E" can replace the original condition " JEin(h) Eout (h) J > E." In
doing so, we end up having to shrink the E's by a factor of 4, and also end up
with a factor of 2 in the estimate of the overall probability. This accounts for
the instead of
in the VC bound and for having 4 instead of 2 as the
multiplicative factor of the growth function. When you put all this together,
you get the formula in (2.12).
D

2.2

Int erpreting t he Generalizat ion Bound

The VC generalization bound (2. 12) is a universal result in the sense that
it applies to all hypothesis sets, learning algorithms, input spaces, probability
distributions, and binary target functions. It can be extended to other types of
target functions as well. Given the generality of the result, one would suspect
that the bound it provides may not be particularly tight in any given case,
since the same bound has to cover a lot of different cases. Indeed, the bound
is quite loose.
55

2 . TRAINING VERSUS TESTING

2 . 2 . INTERPRETING THE BOUND

Exercise 2.5

S uppose we h ave a simple learning m o d e l whose growth function i s

Use the VC bound (2.12) to esti
m1l (N) = N 1 , hence dvc =
mate the probability that Eout wil l be within 0 . 1 of Ein given 100 tra in i ng
exam ples. [Hint: The estimate will be ridiculous.}

Why is the VC bound so loose? The slack in the bound can be attributed to
a number of technical factors. Among them,
1 . The basic Hoeffding Inequality used in the proof already has a slack.
The inequality gives the same bound whether Eout is close to 0.5 or
close to zero. However, the variance of Ein is quite different in these
two cases. Therefore, having one bound capture both cases will result
in some slack.
2. Using mH (N) to quantify the number of dichotomies on N points, re
gardless of which N points are in the data set, gives us a worst-case
estimate. This does allow the bound to be independent of the prob
ability distribution P over X. However, we would get a more tuned
bound if we considered specific x1 ,
, XN and used I H ( x1 ,
, XN ) I or
its expected value instead of the upper bound mH (N) . For instance, in
the case of convex sets in two dimensions, which we examined in Exam
ple 2.2, if you pick N points at random in the plane, they will likely have
far fewer dichotomies than 2 N , while mH ( N) = 2 N .

3. Bounding mH (N) by a simple polynomial of order dvc, as given in (2. 10) ,

will contribute further slack t o the V C bound.

Some effort could be put into tightening the VC bound, but many highly
technical attempts in the literature have resulted in only diminishing returns.
The reality is that the VC line of analysis leads to a very loose bound. Why
did we bother to go through the analysis then? Two reasons. First, the VC
analysis is what establishes the feasibility of learning for infinite hypothesis
sets, the only kind we use in practice. Second, although the bound is loose,
it tends to be equally loose for different learning models, and hence is useful
for comparing the generalization performance of these models. This is an
observation from practical experience, not a mathematical statement . In real
applications, learning models with lower dvc tend to generalize better than
those with higher dvc Because of this observation, the VC analysis proves
useful in practice, and some rules of thumb have emerged in terms of the VC
dimension. For instance, requiring that N be at least 10 x dvc to get decent
generalization is a popular rule of thumb.
Thus, the VC bound can be used as a guideline for generalization, relatively
if not absolutely. With this understanding, let us look at the different ways
the bound is used in practice.
56

2 . TRAINING VERSUS TESTING

2.2. 1

2 . 2 . INTERPRETING THE BOUND

Sample Complexity

The sample complexity denotes how many training examples N are needed
to achieve a certain generalization performance. The performance is specified
by two parameters, E and 8. The error tolerance E determines the allowed
generalization error, and the confidence parameter 8 determines how often the
error tolerance E is violated. How fast N grows as E and 8 become smaller4
indicates how much data is needed to get good generalization.
We can use the VC bound to estimate the sample complexity for a given
learning model. Fix 8 > 0, and suppose we want the generalization error to
be at most E. From Equation (2.12), the generalization error is bounded by
ln
::; E. It follows that
ln
and so it suffices to make

m1-l (2N) )
ln ( 4
N >-
E2
8

suffices to obtain generalization error at most E (with probability at least 1 - 8).

This gives an implicit bound for the sample complexity N, since N appears on
both sides of the inequality. If we replace m1-l (2N) in (2. 12) by its polynomial
upper bound in (2.10) which is based on the the VC dimension, we get a
similar bound
dvc
(2.13)
N 8 ln (4 ((2N) + l) ) '

8
E2
which is again implicit in N. We can obtain a numerical value for N using

simple iterative methods.

Example 2 . 6 . Suppose that we have a learning model with dvc = 3 and

would like the generalization error to be at most 0.1 with confidence 90% (so
E = 0.1 and 8 = 0.1). How big a data set do we need? Using (2.13), we need

+ .
N -> 0.12 1n
0.1
Trying an initial guess of N = 1, 000 in the RHS, we get
3
1000)
+
x
N -> 0.12 ln
21 ' 193.
0.1
We then try the new value N = 21, 193 in the RHS and continue this iterative
process, rapidly converging to an estimate of N 30, 000. If dvc were 4 , a
similar calculation will find that N 40, 000. For dvc = 5, we get N 50, 000.

You can see that the inequality suggests that the number of examples needed
is approximately proportional to the VC dimension, as has been observed in
practice. The constant of proportionality it suggests is 10,000, which is a gross
overestimate; a more practical constant of proportionality is closer to 10. D
4 The term 'complexity' comes from a similar metaphor in computational complexity.

57

2 . 2 . INTERPRETING THE BOUND

2 . TRAINING VERSUS TESTING

2.2.2

Penalty for Model Complexity

Sample complexity fixes the performance parameters E (generalization error)

and 8 ( confidence parameter ) and estimates how many examples N are needed.
In most practical situations, however, we are given a fixed data set V, so N
is also fixed. In this case, the relevant question is what performance can we
expect given this particular N. The bound in (2.12) answers this question:
with probability at least 1 8,
-

Eout (g ) Ein(g ) + N8 ln

If we use the polynomial bound based on dvc instead of m1-l ( 2N) , we get
another valid bound on the out-of-sample error,

( 4 ((2N)dvc + 1) )
Eout (g ) Ein (g ) +
N ln
8

(2.1 4 )

Example 2 .7. Suppose that

N = 100 and we have a 903 confidence require

ment (8 = 0.1). We could ask what error bar can we offer with this confidence,
if 1{ has dvc = 1. Using (2.1 4 ), we have
Eout (g )

Ein(g ) +

8 ln ( 4 (201) )
Q:-1

lOO

Ein ( g ) + 0.848

(2. 15)

with confidence 903. This is a pretty poor bound on Eout Even if Ein = 0,
Eout may still be close to 1. If N = 1, 000, then we get Eaut(g ) Ein (g ) + 0.301,
D
a somewhat more respectable bound.
Let us look more closely at the two parts that make up the bound on Eout
in (2. 12) . The first part is Ein, and the second part is a term that increases
as the VC dimension of 1{ increases.
where

Eout (g ) Ein (g ) + fl (N, 1-l, 8) ,

(2.16)

rl(N, 1-l , 8)
<

N ln

One way to think of rl(N, 1{, 8) is that it is a penalty for model complexity. It
penalizes us by worsening the bound on Eout when we use a more complex 1{
(larger dv0 ) . If someone manages to fit a simpler model with the same training

58

2 . 2 . INTERPRETING THE B OUND

2 . TRAINING VERSUS TESTING

VC dimension, dvc

dc

Figure 2 . 3 : When we use a more complex learning model, one that has
higher VC dimension dvc , we are likely to fit the training data better re
sulting in a lower in sample error, but we pay a higher penalty for model
complexity. A combination of the two, which estimates the out of sample
error, thus attains a minimum at some intermediate d0
error, they will get a more favorable estimate for Eout The penalty O(N, 1-i , o)
gets worse if we insist on higher confidence (lower o) , and it gets better when
we have more training examples, as we would expect.
Although O(N, 1-i, o) goes up when 1i has a higher VC dimension, Ein is
likely to go down with a higher VC dimension as we have more choices within 1{
to fit the data. Therefore, we have a tradeoff: more complex models help Ein
and hurt O(N, 1-i, o) . The optimal model is a compromise that minimizes a
combination of the two terms, as illustrated informally in Figure 2.3.
2.2. 3

The Test Set

As we p.ave seen, the generalization bound gives us a loose estimate of the

out-of-sample error Eout based on Ein. While the estimate can be useful as
a guideline for the training process, it is next to useless if the goal is to get
an accurate forecast of Eout . If you are developing a system for a customer,
you need a more accurate estimate so that your customer knows how well the
system is expected to perform.
An alternative approach that we alluded to in the beginning of this chapter
is to estimate Eout by using a test set, a data set that was not involved in the
training process. The final hypothesis g is evaluated on the test set, and the
result is taken as an estimate of Eout We would like to now take a closer look
at this approach.
Let us call the error we get on the test set Etest. When we report Etest as
our estimate of Eout, we are in fact asserting that Etest generalizes very well
to Eout. After all, Etest is just a sample estimate like Ein. How do we know
59

2 . TRAINING VERSUS TESTING

2 . 2 . INTERPRETING THE BOUND

that Etest generalizes well? We can answer this question with authority now
that we have developed the theory of generalization in concrete mathematical
terms.
The effective number of hypotheses that matters in the generalization be
havior of Etest is 1 . There is only one hypothesis as far as the test set is
concerned, and that's the final hypothesis g that the training phase produced.
This hypothesis would not change if we used a different test set as it would if
we used a different training set. Therefore, the simple Hoeffding Inequality is
valid in the case of a test set. Had the choice of g been affected by the test
set in any shape or form, it wouldn't be considered a test set any more and
the simple Hoeffding Inequality would not apply.
Therefore, the generalization bound that applies to Etest is the simple
Hoeffding Inequality with one hypothesis. This is a much tighter bound than
the VC bound. For example, if you have 1 , 000 data points in the test set, Etest
will be within 53 of Eout with probability 983. The bigger the test set
you use, the more accurate Etest will be as an estimate of Eout.
Exercise

2.6

A d ata set has 600 exam ples. To properly test the performa nce of the
fin a l hypothesis, you set aside a randomly selected subset of 200 exa mples
which are never used in the tra in i ng phase; these form a test set. You use
a learning model with 1, 000 hypotheses a n d select the fin a l hypothesis g
based on the 400 tra i n ing exam ples. We wish to estimate Eout (g) . We have
access to two estimates: Ein (g ) , the i n sample error on the 400 t raining
exa mples; and, Etest (g ) , the test error on the 200 test exam ples that were
set aside.

( a ) Using a 53 error tolera nce (8 = 0.05), which estimate has the h igher
'error bar' ?

( b ) Is there a ny reason why you shouldn 't reserve even more exam ples for
testing?

Another aspect that distinguishes the test set from the training set is that the
test set is not biased. Both sets are finite samples that are bound to have
some variance due to sample size, but the test set doesn't have an optimistic
or pessimistic bias in its estimate of Eout. The training set has an optimistic
bias, since it was used to choose a hypothesis that looked good on it. The VC
generalization bound implicitly takes that bias into consideration, and that's
why it gives a huge error bar. The test set just has straight finite-sample
variance, but no bias. When you report the value of Etest to your customer
and they try your system on new data, they are as likely to be pleasantly
surprised as unpleasantly surprised, though quite likely not to be surprised at
all.
There is a price to be paid for having a test set. The test set does not
affect the outcome of our learning process, which only uses the training set.
The test set just tells us how well we did. Therefore, if we set aside some
60

2 . TRAINING VERSUS TESTING

2 . 2 . INTERPRETING THE B OUND

of the data points provided by the customer as a test set, we end up using
fewer examples for training. Since the training set is used to select one of the
hypotheses in 1-l, training examples are essential to finding a good hypothesis.
If we take a big chunk of the data for testing and end up with too few examples
for training, we may not get a good hypothesis from the training part even if
we can reliably evaluate it in the testing part. We may end up reporting to
the customer, with high confidence mind you, that the g we are delivering is
terrible . There is thus a tradeoff to setting aside test examples. We will
address that tradeoff in more detail and learn some clever tricks to get around
it in Chapter 4.
In some of the learning literature, Etest is used as synonymous with Eout.
When we report experimental results in this book, we will often treat Etest
based on a large test set as if it was Eout because of the closeness of the two
quantities.
2.2.4

Other Target Types

Although the VC analysis was based on binary target functions, it can be

extended to real-valued functions, as well as to other types of functions. The
proofs in those cases are quite technical, and they do not add to the insight
that the VC analysis of binary functions provides. Therefore, we will introduce
an alternative approach that covers real-valued functions and provides new
insights into generalization. The approach is based on bias-variance analysis,
and will be discussed in the next section.
In order to deal with real-valued functions, we need to adapt the definitions
of Ein and Eout that have so far been based on binary functions. We defined Ein
and Eout in terms of binary error; either h(x) = f (x) or else h(x) -/= f (x) . If f
and h are real-valued, a more appropriate error measure would gauge how far
f (x) and h(x) are from each other, rather than just whether their values are
exactly the same.
An error measure that is commonly used in this case is the squared error
e(h(x) , J (x)) = (h(x) - f(x)) 2 . We can define in-sample and out-of-sample
versions of this error measure. The out-of-sample error is based on the ex
pected value of the error measure over the entire input space X ,

Eout (h) = lE [ (h(x) - J (x)) 2 ] ,

while the in-sample error is based on averaging the error measure over the
data set,
N
1 L

Ein (h) = N (h(xn ) - f(xn )) 2 .

n=l
These definitions make Ein a sample estimate of Eout just as it was in the case

of binary functions. In fact, the error measure used for binary functions can
also be expressed as a squared error.

61

2 . TRAINING VERSUS TESTING

2 . 3 . APPROXIMATION GENERALIZATION

Exercise 2. 7
For binary target functions, show that JP>[h(x) f(x)] can be written as a n
expected val ue of a mean sq u a red error measure in the following cases.
( a ) The convention used for the binary fu nction is 0 or

( b ) The convention used for the binary function is 1.

[Hint: The difference between (a) and (b) is just a scale.}

Just as the sample frequency of error converges to the overall probability of

error per Hoeffding's Inequality, the sample average of squared error converges
to the expected value of that error (assuming finite variance) . This is a man
ifestation of what is referred to as the 'law of large numbers' and Hoeffding's
Inequality is just one form of that law. The same issues of the data set size
and the hypothesis set complexity come into play just as they did in the VC
analysis.
2.3

Approximat ion- Generalization Tradeoff

The VC analysis showed us that the choice of 1-l needs to strike a balance
between approximating f on the training data and generalizing on new data.
The ideal 1-l is a singleton hypothesis set containing only the target function.
Unfortunately, we are better off buying a lottery ticket than hoping to have
this 1-l . Since we do not know the target function, we resort to a larger model
hoping that it will contain a good hypothesis, and hoping that the data will
pin down that hypothesis. When you select your hypothesis set, you should
balance these two conflicting goals; to have some hypothesis in 1-l that can
approximate f, and to enable the data to zoom in on the right hypothesis.
The VC generalization bound is one way to look at this tradeoff. If 1-l is
too simple, we may fail to approximate f well and end up with a large in
sample error term. If 1-l is too complex, we may fail to generalize well because
of the large model complexity term. There is another way to look at the
approximation-generalization tradeoff which we will present in this section. It
is particularly suited for squared error measures, rather than the binary error
used in the VC analysis. The new way provides a different angle; instead of
bounding Eout by Ein plus a penalty term 0, we will decompose Eout into two
different error terms.
2.3. 1

Bias and Variance

The bias-variance decomposition of out-of-sample error is based on squared

error measures. The out-of-sample error is
(2. 1 7)
62

2 . 3 . APPROXIMATION GENERALIZATION

2 . TRAINING VERSUS TESTING

where lEx denotes the expected value with respect to x ( based on the probabil
ity distribution on the input space X) . We have made explicit the dependence
of the final hypothesis g on the data V, as this will play a key role in the cur
rent analysis. We can rid Equation ( 2 .17) of the dependence on a particular
data set by taking the expectation with respect to all data sets. We then get
the expected out-of-sample error for our learning model, independent of any
particular realization of the data set,

lEv [lEx [ (g (D ) (x) - f (x)) 2 J]

lEx [lEv [ (g (D) (x) - f (x)) 2 J]
lEx [lEv [g (D ) (x) 2 ] - 2 lEv [g (D ) (x)] f (x) + f (x) 2

J.

The term lEv [g (D ) (x)] gives an 'average function', which we denote by g(x).
One can interpret g(x) in the following operational way. Generate many data
sets V1 , . . . , V K and apply the learning algorithm to each data set to produce
final hypotheses 91 , . . . , 9K . We can then estimate the average function for
any x by g(x) -k 1== l gk (x) . Essentially, we are viewing g(x) as a random
variable, with the randomness coming from the randomness in the data set;
g(x) is the expected value of this random variable ( for a particular x) , and g
is a function, the average function, composed of these expected values. The
function g is a little counterintuitive; for one thing, g need not be in the
model's hypothesis set, even though it is the average of functions that are.

Exercise 2.8
(a) Show that if 1-l i s closed u nder l inear combination (any l inear combi
n ation of hypotheses i n 1-l is a lso a hypothesis in 1-l), then g E 1-l .

( b) Give a model for which the average function g is not i n the model's
hypothesis set. [Hint: Use a very simple model.]
(c) For binary classification, do you expect g to be a binary function?

We can now rewrite the expected out-of-sample error in terms of g:

lEv [Eout (g ( V) )]
lEx [lEv [gCD) (x) 2 ] - 2g(x) f (x) + f (x) 2
lEx [ lEv [gCD ) (x) 2 ] - g(x) 2 + g(x) 2 - 2g(x) f (x) + f (x) 2
(g(x) - f (x) ) 2
lEv [ (g ( D) (x) - g(x) ) 2 ]
where the last reduction follows since g(x) is constant with respect to V.
The term (g(x ) - f (x)) 2 measures how much the average function that we

J,

would learn using different data sets V deviates from the target function that
generated these data sets. This term is appropriately called the bias:

bias(x) = (g(x) - f (x) ) 2 ,

63

2 . 3 . APPROXIMATION GENERALIZATION

2 . TRAINING VERSUS TESTING

as it measures how much our learning model is biased away from the target
function. 5 This is because g has the benefit of learning from an unlimited
number of data sets, so it is only limited in its ability to approximate f by
the limitation in the learning model itself. The term 1Ev [ (g(V ) (x) g(x) ) 2]
is the variance of the random variable g( V ) (x),

var(x) = 1Ev [ (g ('.D) (x) - g(x)) 2 ] ,

which measures the variation in the final hypothesis, depending on the data
set. We thus arrive at the bias-variance decomposition of out-of-sample error,

1Ex[bias(x) + var(x)]
bias + var,
where bias = 1Ex [ bias(x)] and var = 1Ex[var(x)]. Our derivation assumed that

the data was noiseless. A similar derivation with noise in the data would lead
to an additional noise term in the out-of-sample error (Problem 2.22) . The
noise term is unavoidable no matter what we do, so the terms we are interested
in are really the bias and var.
The approximation-generalization tradeoff is captured in the bias-variance
decomposition. To illustrate, let's consider two extreme cases: a very small
model (with one hypothesis) and a very large one with all hypotheses.

Very large model. The target

function is in 1-i. Different data sets
will lead to different hypotheses that
agree with f on the data set, and are
spread around f in the red region.
Thus, bias 0 because g is likely
to be close to f. The var is large
(heuristically represented by the size
of the red region in the figure) .

Very small model. Since there is

only one hypothesis, both the av
erage function g and the f nal hy
pothesis g(D) will be the same, for
any data set. Thus, var = 0. The
bias will depend solely on how well
this single hypothesis approximates
the target f, and unless we are ex
tremely lucky, we expect a large
bias.

One can also view the variance as a measure of 'instability' in the learning
model. Instability manifests in wild reactions to small variations or idiosyn
crasies in the data, resulting in vastly different hypotheses.
5 What we call bias is sometimes called bias2 in the literature.

64

2 . TRAINING VERSUS TESTING

2 . 3 . APPROXIMATION GENERALIZATION

Example 2 . 8 . Consider a target function

f (x) = sin(nx) and a data set

of size N = 2. We sample x uniformly in [- 1, 1) to generate a data set
(x1 , Y1) , (x 2 , Y2 ); and fit the data using one of two models:
Ho :

H1 :

Set of all lines of the form h(x) = b;

Set of all lines of the form h(x) = ax + b.

For Ho , we choose the constant hypothesis that best fits the data (the hori
For H1 , we choose the line that passes
zontal line at the midpoint, b =
through the two data points (x1 , Y1) and (x 2 , y2 ) . Repeating this process with
many data sets, we can estimate the bias and the variance. The figures which
follow show the resulting fits on the same (random) data sets for both models.

1-l o

With H1 , the learned hypothesis is wilder and varies extensively depending

on the data set. The bias-var analysis is summarized in the next figures.

1-l 1
bias = 0 . 2 1 ;
var = 1 .69.

1-l o
bias = 0.50;
var = 0 . 25.

Average hypothesis g ( red) with var(x) indicated by the gray shaded

region that is g(x)

For Hi , the average hypothesis g (red line) is a reasonable fit with a fairly
small bias of 0.21. However, the large variability leads to a high var of 1 .69
resulting in a large expected out-of-sample error of 1 .90. With the simpler
65

2 . 3 . APPROXIMATION GENERALIZATION

2 . TRAINING VERSUS TESTING

model 1-lo , the fits are much less volatile and we have a significantly lower var
of 0.25, as indicated by the shaded region. However, the average fit is now
the zero function, resulting in a higher bias of 0.50. The total out-of-sample
error has a much smaller expected value of 0 .75 . The simpler model wins by
significantly decreasing the var at the expense of a smaller increase in bias.
Notice that we are not comparing how well the red curves (the average hy
potheses ) fit the sine. These curves are only conceptual, since in real learning
we do not have access to the multitude of data sets needed to generate them.
We have one data set, and the simpler model results in a better out-of-sample
error on average as we fit our model to just this one data. However, the var
term decreases as N increases, so if we get a bigger and bigger data set, the
D
bias term will be the dominant part of Eout , and 1-l 1 will win.
The learning algorithm plays a role in the bias-variance analysis that it did
not play in the VC analysis. Two points are worth noting.
1. By design, the VC analysis is based purely on the hypothesis set 1-l , in
dependently of the learning algorithm A. In the bias-variance analysis,
both 1-l and the algorithm A matter. With the same 1-l, using a differ
ent learning algorithm can produce a different g(V) . Since g (V) is the
building block of the bias-variance analysis, this may result in different
bias and var terms.
2. Although the bias-variance analysis is based on squared-error measure,

the learning algorithm itself does not have to be based on minimizing

the squared error. It can use any criterion to produce g(V) based on V.
However, once the algorithm produces g CTJ ) , we measure its bias and
variance using squared error.
Unfortunately, the bias and variance cannot be computed in practice, since
they depend on the target function and the input probability distribution (both
unknown) . Thus, the bias-variance decomposition is a conceptual tool which
is helpful when it comes to developing a model. There are two typical goals
when we consider bias and variance. The first is to try to lower the variance
without significantly increasing the bias, and the second is to lower the bias
without significantly increasing the variance. These goals are achieved by
different techniques, some principled and some heuristic. Regularization is
one of these techniques that we will discuss in Chapter 4 . Reducing the bias
without increasing the variance requires some prior information regarding the
target function to steer the selection of 1-l in the direction of f, and this task is
largely application-specific. On the other hand, reducing the variance without
compromising the bias can be done through general techniques.
2.3.2

The Learning Curve

We close this chapter with an important plot that illustrates the tradeoffs
that we have seen so far. The learning curves summarize the behavior of the

66

2 . 3 . APPROXIMATION GENERALIZATION

2 . TRAINING VERSUS TESTING

in-sample and out-of-sample errors as we vary the size of the training set.
After learning with a particular data set ]) of size N, the final hypothe
sis g CD ) has in-sample error Ein (g (TJ) ) and out-of-sample error Eout (g ( TJ) ) , both
of which depend on JJ . As we saw in the bias-variance analysis, the expectation
with respect to all data sets of size N gives the expected errors: 1Ev [Ein(g ( TJ) )]
and 1Ev [Eout(g ( 'D) )] . These expected errors are functions of N, and are called
the learning curves of the model. We illustrate the learning curves for a simple
learning model and a complex one, based on actual experiments.

H
0

t:
:i
'"O
<!)

t)

<!)

Number of Data Points, N

Number of Data Points, N

Complex Model

Simple Model

Notice that for the simple model, the learning curves converge more quickly
but to worse ultimate performance than for the complex model. This behavior
is typical in practice. For both simple and complex models, the out-of-sample
learning curve is decreasing in N, while the in-sample learning curve is in
creasing in N. Let us take a closer look at these curves and interpret them in
terms of the different approaches to generalization that we have discussed.
In the VC analysis, Eout was expressed as the sum of Ein and a generaliza
tion error that was bounded by n, the penalty for model complexity. In the
bias-variance analysis, Eaut was expressed as the sum of a bias and a variance.
The following learning curves illustrate these two approaches side by side.

Number of Data Points, N

Number of Data Points, N

Bias-Variance Analysis

VC Analysis

67

2 . 3 . APPROXIMATION GENERALIZATION

2 . TRAINING VERSUS TESTING

The VC analysis bounds the generalization error which is illustrated on the

left.6 The bias-variance analysis is illustrated on the right. The bias-variance
illustration is somewhat idealized, since it assumes that, for every N, the aver
age learned hypothesis g has the same performance as the best approximation
to f in the learning model.
When the number of data points increases, we move to the right on the
learning curves and both the generalization error and the variance term shrink,
as expected. The learning curve also illustrates an important point about Ein
As N increases, Ein edges toward the smallest error that the learning model
can achieve in approximating f. For small N, the value of Ein is actually
smaller than that 'smallest possible' error. This is because the learning model
has an easier task for smaller N; it only needs to approximate f on the N
points regardless of what happens outside those points. Therefore, it can
achieve a superior fit on those points, albeit at the expense of an inferior fit
on the rest of the points as shown by the corresponding value of Eaut .

6 For the learning curve, we take the expected values of all quantities with respect to 'D
of size N.

68

2 . 4 . PROBLEMS

2 . TRAINING VERSUS TESTING

2.4

P roblems

Problem 2 . 1

(a) For
(b) For

M =

M =

( c) For M

I n Equ ation (2. 1 ) , set 8

0.03 a nd let

1 , how m a ny exam ples do we need to m a ke

100, how m a ny exa m ples do we need to m a ke

0.05?
E

0.05?

10, 000, how many exam ples do we need to m a ke

0.05?

Problem 2.2

Show that for the learning model of positive rectangles

(aligned horizonta l ly or vertical ly) , mH (4) = 24 a n d mH (5) < 25 . Hence, give
a bound for mH (N) .

Problem 2 . 3
Compute the maxi m u m n um ber of dichotomies, mH (N) ,
for these learni ng models, a nd consequently com pute dvc , the VC d i mensio n .
( a ) Positive or negative ray: 1-l contai ns the functions which are + 1 on [a, oo )
(for some a) together with those that are +1 on ( - oo , a] (for som e a).

(b) Positive or negative i nterval : 1-l contains the functions which a re + 1 on

a n i nterval [a, b] a n d - 1 elsewhere or -1 on a n i nterval [a, b] a nd + 1
elsewhere.
(c) Two concentric spheres in
a xf + . . . + x b.

JRd :

1-l contains the functions which are +1 for

Problem 2.4
Show that B (N, k)
d irection to Lemma 2.3, namely that

B (N k )

I::==-i ( )

by showing the other

( )

To do so, construct a specific set o f I::==-i ( ) dichotomies that does not

shatter any subset of k varia bles. [Hint: Try limiting the number of - 1 's in

each dichotomy.]

Problem 2 . 5

D
P rove by induction that 'I: ( ) ND + 1 , hence
i=O

m'H (N) N dvc + 1 .

69

2 . TRAINING VERSUS TESTING

2 . 4 . PROBLEMS

P rove that fo r N ;: d,

Problem 2 . 6

We suggest you first show the following i ntermediate steps.

(a )
(b)

t ( ) t ( 1: ) ( Jt) d

i=O

i=O

N
I:
( ) (1J f

Hence, a rgue that

i =O

ed . {Hints: Binomial theorem; (1 + r e for

) dvc .
m11, (N) (
.

i=O

i ( Jt) d t ( 1: ) ( 1J) i .

>

O.j

Problem 2 . 7
Plot the bou nds for m11, (N) given in Problems 2.5 and 2.6
= 2 a nd dvc = 5. When do you prefer one bound over the other?

for dva

Which of the following a re possible growth functions m11, (N)

for some hypothesis set:

Problem 2.8
l + N , 1 + N +

N(N - 1) N l v'N J . L N/ 2 J .
N(N - l)(N - 2)
1 + N+
2
.
; 2 ' 2
2
'
'
6

[hard] For t h e perceptron in d d imensions, show that

Problem 2.9

m11, (N)

d
t; ( N 1 ) .

{Hint: Cover(1965) in Further Reading.}

Use this formu l a to verify that dvc = d + 1 by eva luating m11, (d + 1) a n d

m11, (d + 2) . Plot m11, (N)/2 N for d = 10 and N E [1, 40] . If you gen erate a
random d ichotomy on N points i n 10 dimensions, give a n u pper bound on the
probability that the dichotomy wil l be separable for N = 10, 20, 40.

Show that m11, (2N)

m11, (N) 2 , and hence obtain a
genera I ization bound which o n ly i nvolves m11, ( N) .

Problem 2.10

Problem 2 . 1 1
S uppose m11, (N) = N + 1 , so dva = 1 . You have 100
tra ining exam ples. Use the gen era lization bound to give a bound for Eaut with
confidence 90%. Repeat for N = 10, 000.

70

2 . TRAINING VERSUS TESTING

2 . 4 . PROBLEMS

Problem 2. 12
For an 1-l with dvc = 10, what sample size do you need
( as prescri bed by the genera lization bound ) to have a 95% confidence that you r
genera l ization error i s a t most 0.05?

Problem 2.13
( a ) Let 1-l

log2 M.

{h1 , h2 , . . . , hM} with some fin ite M. Prove that dvc (1-l) ::;

( b ) For hypothesis sets 1-l 1, 1-l2 , , 1-lK with fin ite V C dimensions dvc (1-l k) ,
derive and prove the tightest u pper a n d lower bound that you can get
on dvc (n1 1-l k)

( c ) For hypothesis sets 1-l1 , 1-l2 , , 1-lK with fin ite VC dimensions dvc(1-lk ) ,
derive a n d prove t h e tightest u pper a n d lower bounds that you c a n get
on dvc (uf;= 1 1-l k)

Problem 2 . 14
dimension

Let 1-l1 , 1-l2 , . . . , 1-lK be K hypothesis sets with fin ite VC

dvc Let 1-l = 1-l1 U 1-l2 U U 1-lK be the u n ion of these models.

( a ) Show that dvc(1-l) < K(dvc + 1 ) .

( b ) S u ppose that f satisfies 2 > 2Kfdvc . Show that dvc(1-l) ::;
( c ) Hence, show that

dvc (1-l)
That is,

min ( K(dvc + 1), 7(dvc + K) log2 (dvcK) ) .

dvc (1-l) = O (max(dvc , K) log2 max(dvc, K) ) is not too bad .

Problem 2. 15

where

=S;

The monotonica l ly increasing hypothesis set is

x1 ;: x2 if a nd only if the ineq u a l ity is satisfied for every com ponent.

( a ) G ive an example of a monotonic classifier in two dimensions, clearly show

ing the +1 a nd - 1 regions.
( b ) Compute m1-1. (N) a nd hence the VC dimension. {Hint: Consider a set
of N points generated by first choosing one point, and then generating the
next point by increasing the first component and decreasing the second
component until N points are obtained.}

71

2 . 4 . PROBLEMS

2 . TRAINING VERSUS TESTING

Problem 2 . 16

I n this problem , we wil l consider X

is a one d imensional variable. For a hypothesis set

=R

That is, x

prove that the VC d imension of 1-l is exactly (D + 1) by showing that

(a) There a re (D + 1) points which are shattered by 1-l.

(b) There a re no (D + 2) points which are shattered by 1-l.

Problem 2 . 1 7

The VC d imension depends on the in put space as wel l

a s 1-l. For a fixed 1-l, consider two i n put spaces X1 s:;:; X2 . Show that the VC
dimension of 1-l with respect to i n put space X1 is at most the VC dimension
of 1-l with respect to i nput space X2 .
How can the result of this problem be used to a nswer part (b) i n Problem 2 . 16?

[Hint: How is Problem 2. 16 related to a perceptron in D dimensions?}

Problem 2 . 18

The VC d imension of the perceptron hypothesis set

corresponds to the n u m ber of para meters (w0 , w1 ,
, wd ) of the set, and this
observation is ' usua l ly' true for other hypothesis sets. However, we wil l present
a cou nter exam ple here. Prove that the fol lowing hypothesis set for x E IR has
an infinite VC d imension :

1-l

{ ha I ha (x)

a
(-l) L xJ , where

E IR ,

where LAJ is the biggest integer A (the floor function ) . This hypothesis
has o n ly one para meter a but 'enjoys' a n infi n ite VC dimensio n . [Hint: Con
10n , and show how to implement an arbitrary
sider x1 , . . . , x N , where X n

dichotomy Y1 , . . . , YN .J

Problem 2 . 1 9

This problem derives a boun d for the VC dimension of a

com plex hypothesis set that is built from sim pler hypothesis sets via com posi
tio n . Let 1-l1 , . . , 1-LK be hypothesis sets with VC d imension d1 , . . . , dK . Fix
hi , . . . , hK , where hi E 1-Li . Define a vector z obtained from x to have com
ponents hi (x) . Note that x E JRd , but z E { - 1 , + l } K . Let fl be a hypothesis
set of functions that ta ke i nputs in IRK . So
.

E fl:

E IRK 1- {+l , - 1} ,

a n d suppose that il has V C dimension J.

72

2 . TRAINING VERSUS TESTING

2 .4 . PROBLEMS

We can a pply a hypothesis in iL to the z constructed from (hi , . . . , hK) . This

is the composition of the hypothesis set iL with (Hi , . . . , 1-LK ) . More formal ly,
the com posed hypothesis set 1l = iL o (Hi , . . . , 1-LK) is defi ned by h E 1l if
h(x)

h(hi (x) , . . . , hK (x) ) ,

( a ) Show that

K
m1i (N) :: mi{ (N) IT m1ii (N) .
(2. 18)
i=i
{Hint: Fix N points xi , . . . , X N and fix hi , . . . , hK . This generates N
transformed points zi , . . . , Z N . These z i , . . . , Z N can be dichotomized
in at most mi{ (N) ways, hence for fixed (hi , . . . , hK), (xi , . . . , xN )
can be dichotomized in at most mi{ (N) ways. Through the eyes of
xi , . . . , XN , at most how many hypotheses are there (effectively) in 1-Li ?
Use this bound to bound the effective number of K-tuples (hi , . . . , hK)

that need to be considered. Finally, argue that you can bound the number
of dichotomies that can be implemented by the product of the number
of possible K-tuples (hi , . . . , hK ) and the number of dichotomies per
K-tuple.j

( b ) Use the bound m(N)

d, di , . . . , dK .

:: rvc

to get a bound for m1i (N) i n terms of

( c ) Let D = d + 2= i di , a nd assume that D

>

2 e log2 D. Show that

( d ) If 1-Li a nd iL are all perceptron hypothesis sets, show that

dvc (H)

O (dK log(dK) ) .

I n t h e next cha pter, we w i l l further develop t h e sim ple linear mode l . Th is l inear
model is the build ing block of many other models, such as neu ra l networks.
The resu lts of this problem show how to bound the VC d i mension of the more
com plex models built in this manner.

Problem 2 . 20
There are a n u mber of bounds on the general ization
error E , a l l hold i ng with proba bility at least 1 8.
-

( a ) Origin a l VC-bound :

<

( b ) Rademacher Penalty Bound:

!}_ 1 4m1i (2N)

8
.

(continued o n next page)

73

2.4.

2 . TRAINING VERSUS TESTING

PROBLEMS

( c) Parrondo a nd Van den B roek:

<
E_

(2 E

11

6m1-l (2N)
b

( d) Devroye:

Note that ( c) and ( d) are implicit bounds in E. Fix dvc

plot these bou nds as a function of N. Which is best?

Problem 2.21
Theorem
JP>

50 and

b=

0.05 and

Assume t h e fol lowing theorem t o hold

Eout ( g) - Ein( g) > E ::; c . m1-l (2N) exp - E2 N

4

)'

where c is a constant that is a little bigger than 6.

This bound is usefu l because sometimes what we care a bout is not the a bsolute
genera l ization error but instead a relative genera l ization error (one ca n imagine
that a genera lization error of 0.01 is more sign ifica nt when Eout = 0.01 than
when Eout = 0.5). Convert this to a genera lization bound by showing that
with probability at least 1 - b ,

Eout ( g ) ::; Ein ( g ) + 2 1 +

where

ft log

4Ein (g)

'

(2N) .

When there is noise in the data , Eout (g(D) ) =

= J(x) + E. If E is a zero mean noise
random variable with variance o-2 , show that the bias varia nce decom position
becomes
lEv [Eout ( /D) )] = o- 2 + bias + var.

Problem 2.22

lEx,y [(g(D) (x) - y(x)) 2 ] , where y(x)

Problem 2.23

Consider the lea rning problem i n Exam ple 2.8, where the
i n put space is X = [-1, + 1] , the target fu nction is f (x) = sin(?rx) , and the
i n put probability distribution is u n iform on X . Assu me that the training set V
has only two data poi nts ( picked i ndependently) , a n d that the learning a lgorith m
picks the hypothesis that m i n i m izes t h e i n sa mple m e a n squared error. I n this
problem, we wil l d ig deeper i nto this case.

74

2 . TRAINING VERSUS TESTING

2 .4 . PROBLEMS

For each of the following learn i ng models, find ( a n alytica l ly or n umerical ly )

( i ) the best hypothesis that a pproximates f i n the mea n sq uared error sense
( assume t h at f is known for this part ) , ( ii ) the expected va l ue ( with respect
to 'D) of the hypothesis that the learn ing a l gorith m produces, and ( i i i ) the
expected out of sample error a n d its bias and var com ponents.

( a ) The learn ing model consists of a l l hypotheses of the form h(x) = ax + b

( if you need to dea l with the infi n itesima l proba bility case of two identica l
data points, choose the hypothesis ta ngentia l to

f) .

( b ) The learn ing model consists of a l l hypotheses of the form h(x) = ax.
This case was not covered in Exa m ple 2 . 8 .

( c ) The learning model consists of a l l hypotheses of the form h(x) = b.

Problem 2.24
Consider a simplified learn ing scenario. Assume that
the in put d imension is one. Assume that the input varia ble x is u n iform ly
distributed in the interva l [- 1 , 1] . The data set consists of 2 points { x 1 , x 2 }
and assume that the target fu nction is f (x) = x 2 . Th us, the fu ll data set is
'D = { (x 1 , xt) , (x 2 , x)}. The lea rning a lgorith m returns the line fitting these
two points as g (1-l consists of functions of the form h(x) = ax + b). We are
interested in the test performa nce (Bout) of our learn ing system with respect
to the sq uared error measu re, the bias and the var.

( a ) Give the a n a lytic expression for the average function g(x) .

( b ) Describe a n experiment that you cou ld ru n to determ ine ( n u merical ly)
g(x) , Bout , bias, a n d var.
( c ) Run you r experiment and report the resu lts. Com pare Bout with bias+var.
P rovide a plot of you r g(x) and f(x) ( on the same plot ) .
( d ) Compute ana lytica l ly what Bout . bias and var should be.

75

76

Chapter

The L inear Mo del

We often wonder how to draw a line between two categories; right versus
wrong, personal versus professional life, useful email versus spam, to name a
few. A line is intuitively our first choice for a decision boundary. In learning,
as in life, a line is also a good first choice.
In Chapter 1, we ( and the machine @) ) learned a procedure to 'draw a line'
between two categories based on data ( the perceptron learning algorithm) . We
started by taking the hypothesis set 1{ that included all possible lines ( actually
hyperplanes ) . The algorithm then searched for a good line in 1{ by iteratively
correcting the errors made by the current candidate line, in an attempt to
improve Ein . As we saw in Chapter 2 , the linear model set of lines has a
small VC dimension and so is able to generalize well from Ein to Eout .
The aim of this chapter is to further develop the basic linear model into a
powerful tool for learning from data. We branch into three important prob
lems: the classification problem that we have seen and two other important
problems called regression and probability estimation. The three problems
come with different but related algorithms, and cover a lot of territory in
learning from data. As a rule of thumb, when faced with learning problems,
it is generally a winning strategy to try a linear model first.
3. 1

Linear C lassificat ion

The linear model for classifying data into two classes uses a hypothesis set of
linear classifiers, where each h has the form
h (x ) = sign (wTx) ,
for some column vector w E JR.d+ l , where d is the dimensionality of the input
space, and the added coordinate x0 = 1 corresponds to the bias 'weight' w0
( recall that the input space X = { 1 } x JR.d is considered d-dimensional since
the added coordinate x0 = 1 is fixed) . We will use h and w interchangeably

77

3 . THE LINEAR MODEL

3 . 1 . LINEAR C LASSIFICATION

to refer to the hypothesis when the context is clear. When we left Chapter 1 ,
we had two basic criteria for learning:
1 . Can we make sure that Eout (g) is close to Ein (g) ? This ensures that what

we have learned in sample will generalize out of sample.

2. Can we make Ein (g ) small? This ensures that what we have learned in

sample is a good hypothesis.

The first criterion was studied in Chapter 2. Specifically, the VC dimension

of the linear model is only d + 1 (Exercise 2 .4) . Using the VC generalization
bound (2. 12), and the bound (2. 10) on the growth function in terms of the
VC dimension, we conclude that with high probability,
Eout ( g )

= E;n (9 ) + 0

(3. 1)

Thus, when N is sufficiently large, Ein and Eout will be close to each other
( see the definition of 0 ( - ) in the Notation table ) , and the first criterion for
learning is fulfilled.
The second criterion, making sure that Ein is small, requires first and
foremost that there is some linear hypothesis that has small Ein . If there
isn't such a linear hypothesis, then learning certainly can't find one. So, let's
suppose for the moment that there is a linear hypothesis with small Ein . In
fact, let's suppose that the data is linearly separable, which means there is
some hypothesis w* with Ein (w*) = 0. We will deal with the case when this
is not true shortly.
In Chapter 1 , we introduced the perceptron learning algorithm (PLA) .
Start with an arbitrary weight vector w ( O ) . Then, at every time step t 2: 0,
select any misclassified data point (x(t) , y (t) ) , and update w(t) as follows:
w(t + 1 )

= w(t) + y (t)x(t).

The intuition is that the update is attempting to correct the error in classify
ing x(t) . The remarkable thing is that this incremental approach of learning
based on one data point at a time works. As discussed in Problem 1 . 3 , it can be
proved that the PLA will eventually stop updating, ending at a solution wPLA
with Ein (wPLA ) = 0. Although this result applies to a restricted setting (lin
early separable data) , it is a significant step. The PLA is clever it doesn't
na1vely test every linear hypothesis to see if it (the hypothesis ) separates the
data; that would take infinitely long. Using an iterative approach, the PLA
manages to search an infinite hypothesis set and output a linear separator in
( provably) finite time.
As far as PLA is concerned, linear separability is a property of the data,
not the target. A linearly separable V could have been generated either from
a linearly separable target, or ( by chance ) from a target that is not linearly
separable. The convergence proof of PLA guarantees that the algorithm will
78

3 . THE LINEAR MODEL

3 . 1 . LINEAR CLASSIFICATION

(a) Few noisy data.

(b) Nonlinearly separable.

Figure 3.1:

Data sets that are not linearly separable but are (a) linearly
separable after discarding a few examples, or (b) separable by a more so
phisticated curve.

work in both these cases, and produce a hypothesis with Ein = 0 . Further,
in both cases, you can be confident that this performance will generalize well
out of sample, according to the VC bound.
Exercise 3 . 1
Wil l

3 . 1. 1

P LA

ever stop u pdating i f t h e data i s n ot l inearly separable?

Non-Separable Data

We now address the case where the data is not linearly separable. Figure 3.1
shows two data sets that are not linearly separable. In Figure 3. l (a) , the data
becomes linearly separable after the removal of just two examples, which could
be considered noisy examples or outliers. In Figure 3.l(b) , the data can be
separated by a circle rather than a line. In both cases, there will always be
a misclassified training example if we insist on using a linear hypothesis, and
hence PLA will never terminate. In fact, its behavior becomes quite unstable,
and can jump from a good perceptron to a very bad one within one update; the
quality of the resulting Ein cannot be guaranteed. In Figure 3.l(a) , it seems
appropriate to stick with a line, but to somehow tolerate noise and output a
hypothesis with a small Ein , not necessarily Ein = 0. In Figure 3 . l (b) , the
linear model does not seem to be the correct model in the first place, and we
will discuss a technique called nonlinear transformation for this situation in
Section 3.4.
79

3 . THE LINEAR MODEL

3 . 1 . LINEAR CLASSIFICATION

The situation in Figure 3.l (a) is actually encountered very often: even
though a linear classifier seems appropriate, the data may not be linearly sep
arable because of outliers or noise. To find a hypothesis with the minimum Ein ,
we need to solve the combinatorial optimization problem:
min

w Ed+1

N [sign (wTxn ) # Yn ] .

(3.2)

n=l

The difficulty in solving this problem arises from the discrete nature of both
sign() and [-] . In fact, minimizing Ein (w) in (3.2) in the general case is known
to be NP-hard, which means there is no known efficient algorithm for it, and
if you discovered one, you would become really, really famous . Thus, one
has to resort to approximately minimizing Ein .
One approach for getting an approximate solution is to extend PLA through
a simple modification into what is called the pocket algorithm. Essentially, the
pocket algorithm keeps 'in its pocket' the best weight vector encountered up
to iteration t in PLA. At the end, the best weight vector will be reported as
the final hypothesis. This simple algorithm is shown below.
The pocket algorithm:
1:

2:
3:

4:

5:
6:

Set the pocket weight vector w to w(O) of PLA.

= 0, . . . , T
1 do
Run PLA for one update to obtain w(t + 1) .
Evaluate Ein (w(t + 1 ) ) .
If w(t + 1) is better than w in terms o f Ein , set
w(t + 1).
Return w .
for t

to

The original PLA only checks some of the examples using w(t) to identify
(x(t) , y (t) ) in each iteration, while the pocket algorithm needs an additional
step that evaluates all examples using w(t + 1) to get Ein (w(t + 1)) . The
additional step makes the pocket algorithm much slower than PLA. In addi
tion, there is no guarantee for how fast the pocket algorithm can converge to a
good Ein . Nevertheless, it is a useful algorithm to have on hand because of its
simplicity. Other, more efficient approaches for obtaining good approximate
solutions have been developed based on different optimization techniques, as
shown later in this chapter.
Exercise 3.2
Take d = 2 a nd create a data set 'D of size N = 100 that is not linearly
separab le. You can do so by first choosing a random line in the plane as
you r target function and the i n p uts Xn of the data set as random points
in the pla ne. Then, eval uate the target function on each Xn to get the
corresponding output Yn Fin a lly, fli p the la bels of ft random ly selected
Yn 's a n d the data set will l i kely become non separable.

80

3 . THE LINEAR MODEL

3 . 1 . LINEAR CLASSIFICATION

Now, try the pocket a lgorith m on you r data set using = 1 , 000 iterations.
Repeat the experiment 20 times. Then, plot the average Ein (w(t)) and the
average Ein (w) ( which is a lso a function of t) on the same figure a nd see
how they behave when t i ncreases. Similarly, use a test set of size 1, 000
and plot a figure to show how Eout (w(t)) a nd Eout (w) behave.

Example 3.1 ( Handwritten digit recognition ) . We sample some digits from

the US Postal Service Zip Code Database. These 16 x 16 pixel images are
preprocessed from the scanned handwritten zip codes. The goal is to recognize
the digit in each image. We alluded to this task in part (b ) of Exercise 1 . 1 .
A quick look at the images reveals that this is a non-trivial task ( even for a
human) , and typical human Eout is about 2.5 % . Common confusion occurs
between the digits { 4, 9} and {2, 7} . A machine-learned hypothesis which can
achieve such an error rate would be highly desirable.

ITl

Let's first decompose the big task of separating ten digits into smaller tasks of
separating two of the digits. Such a decomposition approach from multiclass
to binary classification is commonly used in many learning algorithms. We will
focus on digits { 1 , 5} for now. A human approach to determining the digit
corresponding to an image is to look at the shape ( or other properties ) of the
black pixels. Thus, rather than carrying all the information in the 256 pixels,
it makes sense to summarize the information contained in the image into a few
features . Let's look at two important features here: intensity and symmetry.
Digit 5 usually occupies more black pixels than digit 1 , and hence the average
pixel intensity of digit 5 is higher. On the other hand, digit 1 is symmetric
while digit 5 is not. Therefore, if we define asymmetry as the average absolute
difference between an image and its flipped versions, and symmetry as the
negation of asymmetry, digit 1 would result in a higher symmetry value. A
scatter plot for these intensity and symmetry features for some of the digits is
shown next.
81

3 . THE LINEAR MODEL

3 . 2 . LINEAR REGRESSION

While the digits can be roughly separated by a line in the plane representing
these two features, there are poorly written digits (such as the '5' depicted in
the top-left corner) that prevent a perfect linear separation.
We now run PLA and pocket on the data set and see what happens. Since
the data set is not linearly separable, PLA will not stop updating. In fact,
as can be seen in Figure 3.2(a) , its behavior can be quite unstable. When
it is forcibly terminated at iteration 1 , 000, PLA gives a line that has a poor
Ein = 2.243 and Eout = 6.373. On the other hand, if the pocket algorithm is
applied to the same data set, as shown in Figure 3.2(b) , we can obtain a line
that has a better Ein = 0.453 and a better Eout = 1 .893.
D

3.2

Linear Regression

Linear regression is another useful linear model that applies to real-valued

target functions.1 It has a long history in statistics, where it has been studied
in great detail, and has various applications in social and behavioral sciences.
Here, we discuss linear regression from a learning perspective, where we derive
the main results with minimal assumptions.
Let us revisit our application in credit approval, this time considering a
regression problem rather than a classification problem. Recall that the bank
has customer records that contain information fields related to personal credit,
such as annual salary, years in residence, outstanding loans, etc. Such variables
can be used to learn a linear classifier to decide on credit approval. Instead of
just making a binary decision (approve or not) , the bank also wants to set a
proper credit limit for each approved customer. Credit limits are traditionally
determined by human experts. The bank wants to automate this task, as it
did with credit approval.
1 Regression, a term inherited from earlier work in statistics, means y is real valued.
82

3 . THE LINEAR MODEL

3 . 2 . LINEAR REGRESSION

50%

50%

250

500

750

Iteration Number, t

1000

250

500

750

Iteration Number, t

Average Intensity

1000

Average Intensity

( a) PLA

( b) Pocket

Figure 3 . 2 :

Comparison of two linear classification algorithms for sep

arating digits 1 and 5 . Ein and Bout are plotted versus iteration number
and below that is the learned hypothesis g . ( a) A version of the PLA which
selects a random training example and updates w if that example is misclas
sified ( hence the fiat regions when no update is made ) . This version avoids
searching all the data at every iteration. ( b) The pocket algorithm.

This is a regression learning problem. The bank uses historical records to

construct a data set 'D of examples (xi , Y1 ) , (x2 , Y2 ) , . . . , (xN , YN ) , where Xn
is customer information and Yn is the credit limit set by one of the human
experts in the bank. Note that Yn is now a real number (positive in this case)
instead of just a binary value 1. The bank wants to use learning to find a
hypothesis g that replicates how human experts determine credit limits.
Since there is more than one human expert, and since each expert may
not be perfectly consistent, our target will not be a deterministic function
y = f (x) . Instead, it will be a noisy target formalized as a distribution of the
random variable y that comes from the different views of different experts as
well as the variation within the views of each expert. That is, the label Yn
comes from some distribution P(y I x) instead of a deterministic function f (x) .
Nonetheless, as we discussed in previous chapters, the nature of the problem
is not changed. We have an unknown distribution P(x, y) that generates
83

3 . 2 . LINEAR REGRESSION

3 . THE LINEAR MODEL

each ( Xn, Yn ) , and we want to find a hypothesis g that minimizes the error
between g (x) and y with respect to that distribution.
The choice of a linear model for this problem presumes that there is a linear
combination of the customer information fields that would properly approx
imate the credit limit as determined by human experts. If this assumption
does not hold, we cannot achieve a small error with a linear model. We will
deal with this situation when we discuss nonlinear transformation later in the
chapter.
3 .2. 1

The Algorithm

The linear regression algorithm is based on minimizing the squared error be

tween h(x) and y. 2
Eout (h)

lE

[(h(x)

y) 2 ,

where the expected value is taken with respect to the joint probability distri
bution P(x, y) . The goal is to find a hypothesis that achieves a small Eout (h) .
Since the distribution P(x, y) is unknown, Eout (h) cannot be computed. Sim
ilar to what we did in classification, we resort to the in-sample version instead,
Ein ( h)

N
= 1 L (h(xn ) Yn ) 2 .
N n=
l

In linear regression, h takes the form of a linear combination of the components

of x. That is,
h (x)

d
= L Wi X i = wT x ,
i =O

where x0 = 1 and x E { 1 } x .!Rd as usual, and w E JRd + 1 . For the special case
of linear h , it is very useful to have a matrix representation of Ein ( h) . First,
define the data matrix X E JRN x ( d+ l ) to be the N x (d + 1) matrix whose rows
are the inputs Xn as row vectors, and define the target vector y E JRN to be
tlie column vector whose components are the target values Yn The in-sample
error is a function of w and the data X , y:

nLN= (
1

N JJ

T n yn) 2

w X

Xw - y ll 2

(wTXT Xw - 2wTXTy + yTy) ,

(3.3)
(3.4)

where II II is the Euclidean norm of a vector, and (3.3) follows because the nth
component of the vector Xw - y is exactly wTXn Yn. The linear regression

2 The term 'linear regression' has been historically confined to squared error measures.

84

3 . THE LINEAR MODEL

3 . 2 . LINEAR REGRESSION

(a) one dimension (line)

(b) two dimensions (hyperplane)

Figure 3 .3:

The solution hypothesis (in blue) of the linear regression algo

rithm in one and two dimensions. The sum of squared errors is minimized.

algorithm is derived by minimizing Ein (w) over all possible w E JRd+ l , as

formalized by the following optimization problem:
WHn = argmin Ein (w) .
w EJRd+1

(3.5)

Figure 3.3 illustrates the solution in one and two dimensions. Since Equa
tion ( 3.4) implies that Ein (w ) is differentiable, we can use standard matrix
calculus to find the w that minimizes Ein (w ) by requiring that the gradient
of Ein with respect to w is the zero vector, i.e. , '\! Ei11 (w ) = 0 . The gradient is
a ( column) vector whose ith component is [ '\!Ein ( w ) ] i =
B y explicitly computing
the reader can verify the following gradient identities,
These identities are the matrix analog of ordinary differentiation of quadratic
and linear functions. To obtain the gradient of Ein , we take the gradient of
each term in (3.4) to obtain

Note that both w and '\!Ei11 ( w ) are column vectors. Finally, to get '\!Ei11 (w )
to be 0, one should solve for w that satisfies
If XTX is invertible, w = xt y where xt = (XTx) - 1 XT is the pseudo-inverse
of X. The resulting w is the unique optimal solution to (3.5) . If XTX is not
85

3 . 2 . LINEAR REGRESSION

3 . THE LINEAR MODEL

invertible, a pseudo-inverse can still be defined, but the solution will not be
unique (see Problem 3 . 15) . In practice, XTX is invertible in most of the cases
since N is often much bigger than d + 1 , so there will likely be d + 1 linearly
independent vectors Xn . We have thus derived the following linear regression
algorithm.

Linear regression algorithm:

1:

Construct the matrix X and the vector y from the data set
(x1 , Y1 ) ,
, (xN , YN ) , where each x includes the x o = 1
bias coordinate, as follows

X=

3:

'-

target vector
Compute the pseudo-inverse xt of the matrix x. If XTX
input data matrix

2:

l' [ :t l
y=

is invertible,

Return Wlin = xty.

This algorithm is sometimes referred to as ordinary least squares ( OLS) . It may

seem that, compared with the perceptron learning algorithm, linear regression
doesn't really look like 'learning', in the sense that the hypothesis Wiin comes
from an analytic solution (matrix inversion and multiplications) rather than
from iterative learning steps. Well, as long as the hypothesis Wlin has a decent
out-of-sample error, then learning has occurred. Linear regression is a rare
case where we have an analytic formula for learning that is easy to evaluate.
This is one of the reasons why the technique is so widely used. It should
be noted that there are methods for computing the pseudo-inverse directly
without inverting a matrix, and that these methods are numerically more
stable than matrix inversion.
Linear regression has been analyzed in great detail in statistics. We would
like to mention one of the analysis tools here since it relates to in-sample and
out-of-sample errors, and that is the hat matrix H. Here is how H is defined.
The linear regression weight vector W!in is an attempt to map the inputs X
to the outputs y. However, wlin does not produce y exactly, but produces an
estimate
y = XW!in

which differs from y due to in-sample error. Substituting the expression

for Wiin (assuming XTX is invertible) , we get
y- = x(xTx ) - 1 XTy .

86

3 . THE LINEAR MODEL

3 . 2 . LINEAR REGRESSION

Therefore the estimate y is a linear transformation of the actual y through

matrix multiplication with H, where
(3.6)
Since y = Hy, the matrix H 'puts a hat' on y, hence the name. The hat
matrix is a very special matrix. For one thing, H2 = H, which can be verified
using the above expression for H. This and other properties of H will facilitate
the analysis of in-sample and out-of-sample errors of linear regression.
Exercise 3 . 3
Consider t h e h a t matrix H =
matrix, a n d XTX is i nvertible.

X(XTX) 1 XT, where

is an

by

( a ) S how that H is sym metric.

(b) Show that HK = H for a ny positive i nteger K.
(c) If I is the identity matrix of size N , show that (I - H)K = I - H for
a n y positive i nteger

K.

( d ) Show that trace(H) = d

elements. {Hint: trace(AB)

3 .2.2

1, where the trace is the sum of diagonal

= trace(BA) .J

Generalization Issues

Linear regression looks for the optimal weight vector in terms of the in-sample
error Ein, which leads to the usual generalization question: Does this guarantee
decent out-of-sample error Eout? The short answer is yes. There is a regression
version of the VC generalization bound (3.1) that similarly bounds Eout In
the case of linear regression in particular, there are also exact formulas for
the expected Eout and Ein that can be derived under simplifying assumptions.
The general form of the result is

Eout (g) = E,n (g) +

o( ) ,

where Eout (g) and Ein (g) are the expected values. This is comparable to the
classification bound in ( 3 . 1 ) .
Exercise 3 . 4
Consider a noisy target y = w *Tx + E fo r generating the data , where E is
a noise term with zero mean and 0" 2 variance, independently generated for
every exam ple (x, y) . The expected error of the best possible linear fit to
this target is thus 0"2 .
For the d ata 'D = {(x1 , y1 ), . . . , (xN , YN )}, denote the noise in Yn as En
and let E = [E1 , E 2 , . . . , E N r; assu me that XT X is i nvertible. By following
(continued o n next page)

87

3 . THE LINEAR 1\!IODEL

3 . 3 . LOGISTIC REGRESSION

the steps below, show that the expected i n sam ple error of l i near regression
with respect to 'D is given by

lEv [Ein (Wiin)] = 0"2

(a) Show that the i n sa mple estimate of

(b) Show that the i n sa m ple error vector
matrix times E. What is the matrix?

is given by

Xw * + HE .

can be expressed by a

( c) Express Ein(W!in) i n terms of E using (b ), a n d simplify the expression

usi n g Exercise 3.3( c) .

( d) Prove that JEv(Ein (WHn ) ] 0" 2 (1

dence of E1 ,
, EN . [Hint: The sum
-

using ( c) and the indepen

of the

diagonal elements
matrix (the trace) will play a role. See Exercise 3.3{d).J

of a

For the expected out of sample error, we take a specia l case which is easy to
a n alyze. Consider a test data set 'Dtest = {(x1 , yi) , . . . , (xN, y)}. which
but with a d ifferent real ization of
shares the same input vectors Xn with
the n oise terms. Denote the noise i n y as
a nd let E1 = [Ei ' E '
' E r.
Define Etest (W!in) to be the average squared error on 'Dtest

(e) Prove that lE v ,e1 [Etest (Wiin)]

= 0"2 ( 1

).

The special test error Etest is a very restricted case of the genera l out
of sam ple error. Some detai led a n a lysis shows that similar results can be
obtai n ed for the general case, as shown in Problem 3 . 11 .

Figure 3.4 illustrates the learning curve of linear regression under the assump
tions of Exercise 3.4. The best possible linear fit has expected error a2 The
for N d + 1. The
expected in-sample error is smaller, equal to a2 (1
learned linear fit has eaten into the in-sample noise as much as it could with
the d + 1 degrees of freedom that it has at its disposal. This occurs because
the fitting cannot distinguish the noise from the 'signal. ' On the other hand,
the expected out-of-sample error is a2 (1 +
) , which is more than the un
avoidable error of a2. The additional error reflects the drift in Wun due to
fitting the in-sample noise.
-

3.3

Logistic Regression

The core of the linear model is the 'signal' s = wTx that combines the input
variables linearly. v. have seen two models based on this signal, and we are
now going to introduce a third. In linear regression, the signal itself is taken
as the output, which is appropriate if you are trying to predict a real response
that could be unbounded. In linear classification, the signal is thresholded
at zero to produce a 1 output, appropriate for binary decisions. A third
possibility, which has wide application in practice, is to output a probability,

88

3 . THE LINEAR MODEL

3 . 3 . LOGISTIC REGRESSION

Number of Data Points, N

Figure 3.4:

The learning curve for linear regression.

a value between 0 and 1. Our new model is called logistic regression. It has
similarities to both previous models, as the output is real ( like regression) but
bounded ( like classification) .
Example 3 . 2 (Prediction of heart attacks) . Suppose we want to predict the
occurrence of heart attacks based on a person's cholesterol level, blood pres
sure, age, weight, and other factors. Obviously, we cannot predict a heart
attack with any certainty, but we may be able to predict how likely it is to
occur given these factors. Therefore, an output that varies continuously be
tween 0 and 1 would be a more suitable model than a binary decision. The
closer y is to 1 , the more likely that the person will have a heart attack. D
3.3.1

Predicting a Probability

Linear classification uses a hard threshold on the signal s = w Tx,

h ( x) = sign (wTx) ,
while linear regression uses no threshold at all,

In our new model, we need something in between these two cases that smoothly
restricts the output to the probability range [O, l ] . One choice that accom
plishes this goal is the logistic regression model,

where 8 is the so-called logistic function B(s) =

and 1 .
89

whose output is between 0

3 . THE LINEAR IVIODEL

3 . 3 . LOGISTIC REGRESSION

The output can be interpreted as a probabil

ity for a binary event (heart attack or no heart
1
attack, digit 'l ' versus digit '5', etc. ) . Linear
classification also deals with a binary event, but
the difference is that the 'classification' in logis
tic regression is allowed to be uncertain, with
intermediate values between 0 and 1 reflecting
this uncertainty. The logistic function B is referred to as a soft threshold, in
contrast to the hard threshold in classification. It is also called a sigmoid
because its shape looks like a flattened out 's' .
Exercise 3.5
Another pop u la r soft threshold i s the hyperbolic tangent
tanh(s)

es - e s
es + e s

( a ) How is tanh related to the logistic function ()? [Hint: shift and scale]
( b ) Show that tanh(s) converges to a h a rd th reshold for l a rge j s j , a nd

converges to no threshold for sma l l I s l [Hint: Formalize the figure

below.]

The specific formula of B ( s ) will allow us to define an error measure for learning
that has analytical and computational advantages, as we will see shortly. Let
us first look at the target that logistic regression is trying to learn. The target
is a probability, say of a patient being at risk for heart attack, that depends
on the input x ( the characteristics of the patient ) . Formally, we are trying to
learn the target function

(x) = JP[y = +1

I x) .

The data does not give us the value of explicitly. Rather, it gives us samples
generated by this probability, e.g. , patients who had heart attacks and patients
who didn't. Therefore, the data is in fact generated by a noisy target P(y I x) ,

P(y I x) =

{f f
f
(x)

1 - (x)

for y = +1;
for y = - 1 .

(3.7)

To learn from such data, we need to define a proper error measure that gauges
how close a given hypothesis h is to in terms of these noisy 1 examples.
90

3 . 3 . LOGISTIC REGRESSION

3 . THE LINEAR MODEL

Error measure. The standard error measure e(h(x) , y) used in logistic re

gression is based on the notion of likelihood ; how 'likely' is it that we would get
this output y from the input x if the target distribution P(y I x) was indeed
captured by our hypothesis h(x)? Based on (3.7) , that likelihood would be
p

(y I x

) - { h(x)
1 h(x)

for y =
for y =

+1;
-1.

We substitute for h(x) by its value B(wTx) , and use the fact that
e( s) (easy to verify) to get

P(y I x) = B(y wT x) .

B(s) =

1+

(3.8)

One of our reasons for choosing the mathematical form e ( s) = es I ( es ) is

that it leads to this simple expression for P(y I x) .
Since the data points (x1 , Y1 ), . . . , (x N , YN ) are independently generated,
the probability of getting all the Yn 's in the data set from the correspond
ing Xn 's would be the product

IT P(yn I Xn)

n=l

The method of maximum likelihood selects the hypothesis h which maximizes

this probability.3 We can equivalently minimize a more convenient quantity,
-

1 ln (gN P(yn I Xn) ) 1 N ln ( 1

=

P(yn I Xn)

),

since ' - -ft ln( ) ' is a monotonically decreasing function. Substituting with
Equation (3.8) , we would be minimizing

t in ( e(Ynw1 Txn) )

N n=l

with respect to the weight vector w. The fact that we are minimizing this
quantity allows us to treat it as an 'error measure. ' Substituting the func
tional form for B(yn WTXn) produces the in-sample error measure for logistic
regression,

l n Tn

(3.9)

The implied pointwise error measure is e(h(xn), Yn) = ln ( + e - Y w x ). Notice

that this error measure is small when Yn wTxn is large and positive, which
would imply that sign (wTxn) Yn Therefore, as our intuition would expect,
the error measure encourages w to 'classify' each Xn correctly.

3 Although the method of maximum likelihood is intuitively plausible, its rigorous justi
fication as an inference tool continues to be discussed in the statistics community.

3 . 3 . LOGISTIC REGRESSION

3 . THE LINEAR MODEL

Exercise 3.6 [Cross-entropy error measure]

(a) M ore genera l ly, if we are learning from 1 data to predict a noisy
target P(y I x) with candidate hypothesis h, show that the maxi m u m
likelihood method reduces t o t h e task o f finding h that minimizes

Ein (w)

[Yn

+l] ln h
(xn)

[ yn

- 1] ln

- h (xn )

(b) For the case h(x) = B(wTx) , argue that m i n imizing the i n sa m ple
error i n part (a) is equ ivalent to minimizing the one i n (3. 9) .
For two probability d istributions {p, 1 - p} a nd {q, 1 q} with binary out
comes, the cross entropy (from i nformation theory) is

1
1
p log - + (1 - p) log
.
q

-q

The i n sa m ple error i n part (a) corresponds to a cross entropy error measure
on the data point (xn , Yn ) , with p = [Yn = +1] a n d q = h(xn) .

For linear classification, we saw that minimizing Ein for the perceptron is a
combinatorial optimization problem; to solve it, we introduced a number of al
gorithms such as the perceptron learning algorithm and the pocket algorithm.
For linear regression, we saw that training can be done using the analytic
pseudo-inverse algorithm for minimizing Ein by setting \7 Ein ( w ) = 0 . These
algorithms were developed based on the specific form of linear classification or
linear regression, so none of them would apply to logistic regression.
To train logistic regression, we will take an approach similar to linear re
gression in that we will try to set \7 Ein (w) = 0. Unfortunately, unlike the case
of linear regression, the mathematical form of the gradient of Ein for logistic
regression is not easy to manipulate, so an analytic solution is not feasible.
Exercise 3. 7
For logistic regression , show that

\7 Ein (w)

Argue that a ' misclassified ' example contributes more to the gradient tha n
a correctly classified one.

Instead of analytically setting the gradient to zero, we will iteratively set it to

zero. To do so, we will introduce a new algorithm, gradient descent. Gradient
92

3 . 3 . LOGISTIC REGRESSION

3 . THE LINEAR MODEL

descent is a very general algorithm that can be used to train many other
learning models with smooth error measures. For logistic regression, gradient
descent has particularly nice properties.
3 . 3 .2

Gradient Descent

Gradient descent is a general technique for

minimizing a twice-differentiable function, such
as Ein ( w) in logistic regression. A useful phys
ical analogy of gradient descent is a ball rolling
down a hilly surface. If the ball is placed on
a hill, it will roll down, coming to rest at the
bottom of a valley. The same basic idea under
lies gradient descent. Ein(w) is a 'surface' in
a high-dimensional space. At step 0, we start
somewhere on this surface, at w(O) , and try to
roll down this surface, thereby decreasing Ein One thing which you imme
diately notice from the physical analogy is that the ball will not necessarily
come to rest in the lowest valley of the entire surface. Depending on where
you start the ball rolling, you will end up at the bottom of one of the valleys
a local minimum. In general, the same applies to gradient descent. Depending
on your starting weights, the path of descent will take you to a local minimum
in the error surface.
A particular advantage for logistic regression
with the cross-entropy error is that the picture
looks much nicer. There is only one valley! So,
it does not matter where you start your ball
rolling, it will always roll down to the same
( unique ) global minimum. This is a consequence
of the fact that Ein ( w) is a convex function
of w , a mathematical property that implies a
single 'valley' as shown to the right. This means
Weights, w
that gradient descent will not be trapped in lo
cal minima when minimizing such convex error measures. 4
Let's now determine how to 'roll' down the Bin-surface. We would like to
take a step in the direction of steepest descent, to gain the biggest bang for
our buck. Suppose that we take a small step of size T/ in the direction of a unit
vector v. The new weights are w(O) + TJV. Since T/ is small, using the Taylor
expansion to first order, we compute the change in Ein as

Ein

Ein(w(O) + TJV) Ein(w(O))

TJ V7 Ein(w(O)) Tv + 0(TJ2 )
> TJll V7 Ein (w(O)) ll ,

4 In fact, the squared in-sample error in linear regression is also convex, which is why the
analytic solution found by the pseudo-inverse is guaranteed to have optimal in-sample error.

93

3 . THE LINEAR MODEL

3 . 3 . LOGISTIC REGRESSION

where we have ignored the small term 0( TJ 2 ) . Since v is a unit vector, equality
holds if and only if
v=

\7 Ein (w(O) )
J J V Ein (w(O)) JI '

(3. 10)

This direction, specified by v, leads to the largest decrease in Ein for a given
step size T/.
Exercise 3 . 8
The claim that v i s t h e direction which gives largest decrease i n Ein o n ly
holds for small 77. Why?

There is nothing to prevent us from continuing to take steps of size 17, re

evaluating the direction Vt at each iteration t = 0, 1, 2, . . . . How large a step
should one take at each iteration? This is a good question, and to gain some
insight, let's look at the following examples.

Weights,
T/

too small

Weights,

TJ too large

weights,

variable T/ just right

A fixed step size (if it is too small) is inefficient when you are far from the
local minimum. On the other hand, too large a step size when you are close to
the minimum leads to bouncing around, possibly even increasing Ein. Ideally,
we would like to take large steps when far from the minimum to get in the
right ballpark quickly, and then small (more careful) steps when close to the
minimum. A simple heuristic can accomplish this: far from the minimum,
the norm of the gradient is typically large, and close to the minimum, it is
small. Thus, we could set T/t = 17 J J VEin ll to obtain the desired behavior for
the variable step size; choosing the step size proportional to the norm of the
gradient will also conveniently cancel the term normalizing the unit vector v in
Equation (3. 10) , leading to the fixed learning rate gradient descent algorithm
for minimizing Ein (with redefined TJ ) :
94

3 . THE LINEAR MODEL

3 . 3 . LOGISTIC REGRESSION

Fixed learning rate gradient descent:

1:
2:

3:

4:
5:

6:
7:

Initialize the weights at time step t = 0 to w (O) .

for t

0, 1 , 2, . . . do

Compute the gradient gt = \l Ein (w(t )).

Set the direction to move, Vt = - gt .
Update the weights: w (t + 1) w(t) + TJVt .
Iterate to the next step until it is time to stop.
Return the final weights.

In the algorithm, vt is a direction that is no longer restricted to unit length.

The parameter 77 (the learning rate) has to be specified. A typically good
choice for 77 is around 0 . 1 ( a purely practical observation ) . To use gradient
descent, one must compute the gradient. This can be done explicitly for logistic
regression ( see Exercise 3 . 7) .
Example 3.3. Gradient descent is a general algorithm for minimizing twice
differentiable functions. We can apply it to the logistic regression in-sample
error to return weights that approximately minimize

Ein (w)

=NL
1

ln 1 + e

n=l

Y n W Xn
T

Logistic regression algorithm:

1:

2:

3:

4:
5:

6:
7:

Initialize the weights at time step t = 0 to w(O) .

for t = 0, 1 , 2 , . . . do
Compute the gradient

Set the direction to move, V t = - gt .

Update the weights: w (t + 1 ) w(t) + TJVt .
Iterate to the next step until it is time to stop.
Return the final weights w .
D

Initialization and termination. We have two more loose ends to tie: the
first is how to choose w(O) , the initial weights, and the second is how to
set the criterion for " . . . until it is time to stop" in step 6 of the gradient
descent algorithm. In some cases, such as logistic regression, initializing the
weights w(O) as zeros works well. However, in general, it is safer to initialize
the weights randomly, so as to avoid getting stuck on a perfectly symmetric
hilltop. Choosing each weight independently from a Normal distribution with
zero mean and small variance usually works well in practice.

95

3 . THE LINEAR MODEL

3 . 3 . LOGISTIC REGRESSION

That takes care of initialization, so we now move on to termination. How

do we decide when to stop? Termination is a non-trivial topic in optimization.
One simple approach, as we encountered in the pocket algorithm, is to set an
upper bound on the number of iterations, where the upper bound is typically
in the thousands, depending on the amount of training time we have. The
problem with this approach is that there is no guarantee on the quality of the
final weights.
Another plausible approach is based on the gradient being zero at any min
imum. A natural termination criterion would be to stop once llgt l l drops below
a certain threshold. Eventually this must happen, but we do not know when
it will happen. For logistic regression, a combination of the two conditions
( setting a large upper bound for the number of iterations, and a small lower
bound for the size of the gradient ) usually works well in practice.
There is a problem with relying solely on
the size of the gradient to stop, which is that
you might stop prematurely as illustrated on the
kf
right. When the iteration reaches a relatively
fl.at region ( which is more common than you
might suspect ) , the algorithm will prematurely
We ights, w
stop when we may want to continue. So one solution is to require that termination occurs only
if the error change is small and the error itself is small. Ultimately a combina
tion of termination criteria ( a maximum number of iterations, marginal error
improvement, coupled with small value for the error itself ) works reasonably
well.
Example 3 .4. By way of summarizing linear models, we revisit our old friend
the credit example. If the goal is to decide whether to approve or deny, then
we are in the realm of classification; if you want to assign an amount of credit
line, then linear regression is appropriate; if you want to predict the probability
that someone will default, use logistic regression.

Credit
Analysis

Approve
or Deny
Amount
of Credit
Probability
of Default

Perceptron
Linear Regression
Logistic Regression

The three linear models have their respective goals, error measures, and al
gorithms. Nonetheless, they not only share similar sets of linear hypotheses,
but are in fact related in other ways. We would like to point out one impor
tant relationship: Both logistic regression and linear regression can be used in
linear classification. Here is how.
Logistic regression produces a final hypothesis g(x) which is our estimate
of JP> [y = + 1 I x) . Such an estimate can easily be used for classification by

96

3 . THE LINEAR MODEL

3 . 3 . LOGISTIC REGRESSION

setting a threshold on g(x) ; a natural threshold is ' which corresponds to

classifying + 1 if + 1 is more likely. This choice for threshold corresponds to
using the logistic regression weights as weights in the perceptron for classifica
tion. Not only can logistic regression weights be used for classification in this
way, but they can also be used as a way to train the perceptron model. The
perceptron learning problem (3. 2) is a very hard combinatorial optimization
problem. The convexity of Ein in logistic regression makes the optimization
problem much easier to solve. Since the logistic function is a soft version of
a hard threshold, the logistic regression weights should be good weights for
classification using the perceptron.
A similar relationship exists between classification and linear regression.
Linear regression can be used with any real-valued target function, which
includes real values that are 1. If wlin x is fit to 1 values, sign(wlin x) will
likely agree with these values and make good classification predictions. In
other words, the linear regression weights WHn , which are easily computed
using the pseudo-inverse, are also an approximate solution for the perceptron
model. The weights can be directly used for classification, or used as an initial
D
condition for the pocket algorithm to give it a head start.
Exercise 3. 9
Consider pointwise error measures eclass (s, y)
[y sign(s )] , esq(s, y) =
(y - s and e10g( s, y) = ln(l + exp( -ys) ) , where the signa l s = wT x.

( a ) For y = + 1 , plot eclass r esq a nd

versus s, on the same plot.

( b) Show that ec1ass (s, y)

esq(s, y) , and hence that the classification
error is upper bounded by the squared error.

( c) Show that ec1ass (s, y)

a n d , as in part (b), get a n
u pper bound ( u p to a constant factor) using the logistic regression
error.

These bounds indicate that m i n im izing the squared or logistic regression

error shou ld a lso decrease the cla ssification error, which justifies using the
weights returned by l inear or logistic regression as a pproximations for clas
sification .

Stochastic gradient descent. The version of gradient descent we have de

scribed so far is known as batch gradient descent the gradient is computed

for the error on the whole data set before a weight update is done. A sequen
tial version of gradient descent known as stochastic gradient descent (SGD)
turns out to be very efficient in practice. Instead of considering the full batch
gradient on all N training data points, we consider a stochastic version of
the gradient. First, pick a training data point (xn , Yn) uniformly at random
(hence the name 'stochastic') , and consider only the error on that data point

97

3 . 3 . LOGISTIC REGRESSION

3 . THE LINEAR MODEL

(in the case of logistic regression) ,

The gradient of this single data point's error is used for the weight update in
exactly the same way that the gradient was used in batch gradient descent.
The gradient needed for the weight update of SGD is (see Exercise 3.7)
and the weight update is w f- w 77\len(w). Insight into why SGD works can
be gained by looking at the expected value of the change in the weight (the
expectation is with respect to the random point that is selected) . Since n is
picked uniformly at random from { 1, . . . , N}, the expected weight change is
1

TJ

n=l

\len(w).

This is exactly the same as the deterministic weight change from the batch
gradient descent weight update. That is, 'on average' the minimization pro
ceeds in the right direction, but is a bit wiggly. In the long run, these random
fluctuations cancel out. The computational cost is cheaper by a factor of N,
though, since we compute the gradient for only one point per iteration, rather
than for all N points as we do in batch gradient descent.
Notice that SGD is similar to PLA in that it decreases the error with re
spect to one data point at a time. Minimizing the error on one data point may
interfere with the error on the rest of the data points that are not considered
at that iteration. However, also similar to PLA, the interference cancels out
on average as we have just argued.
Exercise 3 . 1 0

( a ) Define a n error for a single d ata point (xn , Yn ) to be

en (w)

max(O, -ynwTxn)

Argue that P LA can be viewed as S G D o n e n with learn i ng rate 7J = 1.

( b ) For logistic regression with a very large w, argue t h a t m i n i m izing Ein

using S G D is similar to P LA. This is a nother indication that the lo

gistic regression weights can be used as a good a p proximation for
classification .

SGD is successful in practice, often beating the batch version and other more
sophisticated algorithms. In fact, SGD was an important part of the algorithm
that won the million-dollar Netflix competition, discussed in Section 1 . 1 . It
scales well to large data sets, and is naturally suited to online learning, where

98

3 . 4 . NONLINEAR TRANSFORMATION

3 . THE LINEAR MODEL

a stream of data present themselves to the learning algorithm sequentially.

The randomness introduced by processing one data point at a time can be a
plus, helping the algorithm to avoid flat regions and local minima in the case
of a complicated error surface. However, it is challenging to choose a suit
able termination criterion for SGD . A good stopping criterion should consider
the total error on all the data, which can be computationally demanding to
evaluate at each iteration.
3.4

Nonlinear Transformation

All formulas for the linear model have used the sum
d

WT X = L WiXi
i =O

(3. 11)

as the main quantity in computing the hypothesis output. This quantity is

linear, not only in the xi 's but also in the w/s. A closer inspection of the
corresponding learning algorithms shows that the linearity in wi 's is the key
property for deriving these algorithms; the Xi 's are just constants as far as the
algorithm is concerned. This observation opens the possibility for allowing
nonlinear versions of Xi 's while still remaining in the analytic realm of linear
models, because the form of Equation (3. 1 1 ) remains linear in the wi param
eters.
Consider the credit limit problem for instance. It makes sense that the
'years in residence' field would affect a person's credit since it is correlated with
stability. However, it is less plausible that the credit limit would grow linearly
with the number of years in residence. More plausibly, there is a threshold
(say 1 year) below which the credit limit is affected negatively and another
threshold (say 5 years) above which the credit limit is affected positively. If Xi
is the input variable that measures years in residence, then two nonlinear
'features' derived from it, namely [xi < 1] and [xi > 5] , would allow a linear
formula to reflect the credit limit better.
We have already seen the use of features in the classification of handwritten
digits, where intensity and symmetry features were derived from input pixels.
Nonlinear transforms can be further applied to those features, as we will see
shortly, creating more elaborate features and improving the performance. The
scope of linear methods expands significantly when we represent the input by
a set of appropriate features.
3 .4. 1

The Z Space

Consider the situation in Figure 3 . 1 (b) where a linear classifier can't fit the
data. By transforming the inputs x1 , x 2 in a nonlinear fashion, we will be able
to separate the data with more complicated boundaries while still using the
99

3 . 4 . NONLINEAR TRANSFORMATION

3 . THE LINEAR MODEL

simple PLA as a building block. Let's start by looking at the circle in Fig
ure 3.5 ( a) , which is a replica of the non-separable case in Figure 3.l ( b ) . The
circle represents the following equation:

xi + x = 0.6.
That is, the nonlinear hypothesis h (x) = sign ( - 0.6 + xi + x) separates the
data set perfectly. We can view the hypothesis as a linear one after applying
a nonlinear transformation on x. In particular, consider zo = 1, z1 = xi and

Z2

X,

h (x)

sign ( -0.6 )

1 + 1

"-v-" '-v-'

Wo

Zo

sign [Wo W1 W2 ]

xi + 1

'-v-' '-v-'

w1

Zl

'-v-' '-v-'

W2

Z2

[ : ]
z

WT

where the vector z is obtained from x through a nonlinear transform <I> ,

z

<I>(x) .

We can plot the data in terms of z instead of x, as depicted in Figure 3.5 ( b ) .

For instance, the point x1 in Figure 3.5 ( a) is transformed to the point z1 in
Figure 3 .5 (b ) and the point x2 is transformed to the point z2 The space Z,
which contains the z vectors, is referred to as the feature space since its coor
dinates are higher-level features derived from the raw input x. We designate
different quantities in Z with a tilde version of their counterparts in X, e.g.,
the dimensionality of Z is d and the weight vector is w.5 The transform <I>
that takes us from X to Z is called a feature transform, which in this case is
<I>(x)

( 1, xi, x) .

( 3.12 )

In general, some points in the Z space may not be valid transforms of any
x E X , and multiple points in X may be transformed to the same z E Z ,
depending on the nonlinear transform <I> .
The usefulness of the transform above is that the nonlinear hypothesis h
( circle ) in the X space can be represented by a linear hypothesis (line ) in
the Z space. Indeed, any linear hypothesis h in z corresponds to a (possibly
nonlinear) hypothesis of x given by
h (x)
5 Z { 1 } x JRd, where d
coordinate zo
1 is fixed.

h(<I>(x) ) .

2 i n this case. We treat Z as d dimensional since the added

100

3 . 4 . NONLINEAR TRANSFORMATION

3 . THE LINEAR MODEL

0.5

(b) Transformed data in Z space

z

P {x)

[i!]

Figure 3.5: (a) The original data set that is not linearly separable, but
separable by a circle. (b) The transformed data set that is linearly separable
in the Z space. In the figure, x1 maps to z1 and x2 maps to z2 ; the circular
separator in the X space maps to the linear separator in the Z space.
The set of these hypotheses h is denoted by 1-lcp . For instance, when using
the feature transform in (3. 12) , each h E 1-lcp is a quadratic curve in X that
corresponds to some line h in Z.
Exercise 3 . 11
Consider the feature transform <I> i n (3.12). What kind of boundary i n
does a hyperplane
in Z correspond to i n the following cases? Draw a
picture that i l lustrates a n example of each case.

( a ) 'li1

0,

w2

(b) 'li1 > 0, w2

(c)

(d)

w1 >
w1 >

O, w2
o, w2

=
>
>

0
O, wo
o, wo

<

> o

Because the transformed data set (zi , Y1 ) ,

, (z N , YN ) in Figure 3.5(b) is
linearly separable in the feature space Z, we can apply PLA on the transformed
data set to obtain wPLAi the PLA solution, which gives us a final hypothesis
g (x) = sign(wLA z) in the X space, where z = <I> (x) . The whole process of
applying the feature transform before running PLA for linear classification is
depicted in Figure 3.6.
The in-sample error in the input space X is the same as in the feature
space Z, so Ein(g) = 0. Hyperplanes that achieve Ein (wPLA) = 0 in Z cor
respond to separating curves in the original input space X. For instance,

101

3 . THE LINEAR MODEL

3 . 4 . NONLINEAR TRANSFORMATION

<I>

0.5

0
0

1 . Original data
Xn E X

0.5

2. Transform the data

Zn = <I> (xn) E Z

0
0

4. Classify in X-space
= g (<I> (x) ) = sign ( wT <I> ( x) )

0.5

3. Separate data in Z-space

g ( z ) = sign ( wTz )

g ( x)

Figure 3.6: The nonlinear transform for separating non separable data.
as shown in Figure 3.6, the PLA may select the line wPLA = ( -0.6, 0.6, 1)
that separates the transformed data (z1 , Y1 ) , , (z N , YN ) . The correspond
ing hypothesis g (x) = sign ( -0.6 + 0.6 xi + x) will separate the original data
(x1 , Y1 ) , , (xN , YN ) In this case, the decision boundary is an ellipse in X .
How does the feature transform affect the VC bound (3. 1 ) ? If we honestly
decide on the transform <P before seeing the data, then with probability at
least 1 - 6, the bound (3. 1 ) remains true by using dvc (1-lcp ) as the VC dimen
sion. For instance, consider the feature transform <I> in (3. 12) . We know that
Z = {1} x 2 . Since 1-lcp is the perceptron in Z, dvc (1-lcp) :: 3 ( the :: is because
some points z E Z may not be valid transforms of any x, so some dichotomies
may not be realizable ) . We can then substitute N, dvc (1-lcp ) , and 6 into the
VC bound. After running PLA on the transformed data set, if we succeed in

102

3 . THE LINEAR MODEL

3 . 4 . NONLINEAR TRANSFORMATION

getting some g with Ein (g) = 0, we can claim that g will perform well out of
sample.
It is very important to understand that the claim above is valid only if you
decide on <P before seeing the data or trying any algorithms. What if we first
try using lines to separate the data, fail, and then use the circles? Then we
are effectively using a model that contains both lines and circles, and dvc is
no longer 3.
Exercise 3 . 1 2
We know that in the Euclidean plane, the perceptron model 1-l can not
i mplement a l l 16 dichotomies on 4 points. That is, m1-1. (4) < 16. Take the
featu re transform <I> in (3. 12) .

( a ) S how that m1-1.<I,(3) = 8 .

(b) Show that m1-1. <I> (4) 16 .
( c ) S h ow that m1-1.u1-1. <I> (4) = 16.

That is, if you used l ines, dvc

lines a n d elipses, dvc > 3.

3; if you used elipses, dvc

3; if you used

Worse yet, if you actually look at the data (e.g. , look at the points in Fig
ure 3 . l (a) ) before deciding on a suitable <P , you forfeit most of what you
learned in Chapter 2 . You have inadvertently explored a huge hypothesis
space in your mind to come up with a specific <P that would work for this data
set. If you invoke a generalization bound now, you will be charged for the VC
dimension of the full space that you explored in your mind, not just the space
that <P creates.
This does not mean that <P should be chosen blindly. In the credit limit
problem for instance, we suggested nonlinear features based on the 'years in
residence' field that may be more suitable for linear regression than the raw
input. This was based on our understanding of the problem, not on 'snooping'
into the training data. Therefore, we pay no price in terms of generalization,
and we may well gain a dividend in performance because of a good choice of
features.
The feature transform <P can be general, as long as it is chosen before seeing
the data set (as if we cannot emphasize this enough) . For instance, you may
have noticed that the feature transform in (3. 12) only allows us to get very
limited types of quadratic curves. Ellipses that do not center at the origin
in X cannot correspond to a hyperplane in Z. To get all possible quadratic
curves in X, we could consider the more general feature transform z = <P 2 (x) ,

<P 2 (x)

(1, x1 , x2 , xi , x1x 2 , x) ,

(3. 13)

which gives us the flexibility to represent any quadratic curve in X by a hy

perplane in Z (the subscript 2 of <P is for polynomials of degree 2 quadratic
curves) . The price we pay is that Z is now five-dimensional instead of two
dimensional, and hence dvc is doubled from 3 to 6.
-

103

3 . THE LINEAR MODEL

Exercise

3 . 4 . NONLINEAR TRANSFORMATION

3.13

t h e feature tra n sform z = <1> 2 (x) i n (3.13). How

in
to represent the fo l l owing boundaries in
hyperplane

Consid er

(a)

para bola (x1

3) 2

x2

ca n

we use

(b) The circle (x1

(x2 - 4) 2 =
2
( c) The elli pse 2(x 1 - 3 ) (x2 4) 2 =
3) 2

(x 2 4) 2
3) 2 (x1 - x2 - 4) 2

hyperbola (x1 - 3) 2

2(x1

(e)

ellipse

(f)

l ine 2x1

x2

x2

One may further extend <1> 2 to a feature transform <1> 3 for cubic curves in X,
or more generally define the feature transform <I> Q for degree-Q curves in X.
The feature transform <I> Q is called the Qth order polynomial transform.
The power of the feature transform should be used with care. It may
not be worth it to insist on linear separability and employ a highly complex
surface to achieve that. Consider the case of Figure 3 . l ( a) . If we insist on a
feature transform that linearly separates the data, it may lead to a significant
increase of the VC dimension. As we see in Figure 3 .7, no line can separate the
training examples perfectly, and neither can any quadratic nor any third-order
polynomial curves. Thus, we need to use a fourth-order polynomial transform:
( X ) = ( 1 , Xi , X2 , X12 , X 1 X2 , X 2 , X13 , X12 X2 , X 1 X2 , X 3 , X14 , X13 X2 , X12 X 2 , X 1 X3 , X4 ) .

2 2

2 2

If you look at the fourth-order decision boundary in Figure 3 . 7 ( b ) , you don't

need the VC analysis to tell you that this is an overkill that is unlikely to
generalize well to new data. A better option would have been to ignore the two
misclassified examples in Figure 3.7 ( a) , separate the other examples perfectly
with the line, and accept the small but nonzero Ein . Indeed, sometimes our
best bet is to go with a simpler hypothesis set while tolerating a small Ein .
While our discussion of feature transforms has focused on classification
problems, these transforms can be applied equally to regression problems.
Both linear regression and logistic regression can be implemented in the feature
space Z instead of the input space X. For instance, linear regression is often
coupled with a feature transform to perform nonlinear regression. The N
by d + 1 input matrix X in the algorithm is replaced with the N by J + 1
matrix Z , while the output vector y remains the same.
3 .4 .2

C omputation and Generalization

Although using a larger Q gives us more flexibility in terms of the shape of

decision boundaries in X , there is a price to be paid. Computation is one
issue, and generalization is the other.
Computation is an issue because the feature transform <I> Q maps a two
dimensions, which increases the memory
dimensional vector x to J =

104

3 . THE LINEAR M ODEL

3 . 4 . NONLINEAR TRANSFORMATION

(a) Linear fit

(b) 4th order polynomial fit

Figure 3.7: Illustration of the nonlinear transform using a data set that
is not linearly separable; (a) a line separates the data after omitting a few
points, (b) a fourth order polynomial separates all the points.
and computational costs. Things could get worse if x is in a higher dimension
to begin with.

Rd .

Exercise 3.14

Consider the Qth order polynomi a l transform 4> Q fo r

=
What is
the d imensionality d of the feature space Z (excluding the fixed coordinate
zo = 1). Eval uate you r result on d E {2, 3, 5, 10} a n d
E {2, 3, 5, 10}.

The other important issue is generalization. If <Pq is the feature transform of

a two-dimensional input space, there will be d =
dimensions in Z, and
+ 1 . This means that the second term in
dv0 (H<P ) can be as high as
the VC bound (3. 1) can grow significantly. In other words, we would have a
weaker guarantee that Eout will be small. For instance, if we use Cf? = <f? 5 0 ,
0
the VC dimension of 1-l cp could b e as high as C 5 )2C 5 3 ) + 1 = 1326 instead of the
original dvc = 3. Applying the rule of thumb that the amount of data needed
is proportional to the VC dimension, we would need hundreds of times more
data than we would if we didn't use a feature transform, in order to achieve
the same level of generalization error.
Exercise 3 . 15
H igh-dimensiona l featu re transforms a re by no means the only transforms
that we can use. We can take the tra deoff in the other d i rection , a n d
use low di mensional feature transforms as wel l (to achieve a n even lower
genera lization error bar) .
(continued on next page)

105

3 . 4 . N ONLINEAR TRANSFORMATION

3 . THE LINEAR MODEL

Consider the fol lowing featu re transform , which m a ps a d-d i mensional

a one-dimensional z, keeping only the kth coordi n ate of x .

<J>(k) (x) = (1, Xk)

Let

to

(3.14)

1-lk be the set of perceptrons in the feature space.

( a ) Prove that dvc(1lk) = 2.

( b ) Prove that dvc(U= l 1-lk) :S 2(log2 d

1).

1-lk is called the decision stump model on d imension k.

The problem of generalization when we go to high-dimensional space is some

times balanced by the advantage we get in approximating the target better. As
we have seen in the case of using quadratic curves instead of lines, the trans
formed data became linearly separable, reducing Ein to 0. In general, when
choosing the appropriate dimension for the feature transform, we cannot avoid
the approximation-generalization tradeoff,
higher d
lower d

better chance of being linearly separable (Ein .t)

possibly not linearly separable ( Ein t)

Therefore, choosing a feature transform before seeing the data is a non-trivial

task. When we apply learning to a particular problem, some understanding
of the problem can help in choosing features that work well. More generally,
there are some guidelines for choosing a suitable transform, or a suitable model,
which we will discuss in Chapter 4.

Exercise 3.16
Write down t h e steps o f t h e a lgorithm that combines <!> 3 with linea r re
gressio n . H ow a bout using <!>10 i nstead? Where is the m a i n com putation a l
bottleneck o f the resu lting a lgorith m ?

Example 3.5. Let 's revisit the handwritten digit recognition example. We
can try a different way of decomposing the big task of separating ten digits
to smaller tasks. One decomposition is to separate digit 1 from all the other
digits . Using intensity and symmetry as our input variables like we did before,
the scatter plot of the training data is shown next. A line can roughly separate
digit 1 from the rest, but a more complicated curve might do better.

106

3 . THE LINEAR MODEL

3 . 4 . NONLINEAR TRANSFORMATION

Average Intensity
We use linear regression ( for classification) , first without any feature transform.
The results are shown below ( LHS ) . We get Ein = 2. 13% and Eout = 2.38%.

Average Intensity

Average Intensity
3rd order polynomial model
Ein = 1 . 75%
Eout = 1 .87%

Linear model
Ein = 2 . 13%
Eout = 2.38%

Classification of the digits data ( ' 1 ' versus 'not 1 ' ) using linear and
third order polynomial models.

When we run linear regression with <1> 3 , the third-order polynomial transform,
we obtain a better fit to the data, with a lower Ein = 1 . 75% . The result is
depicted in the RHS of the figure. In this case, the better in-sample fit also
resulted in a better out-of-sample performance, with Eout = 1 .87%.
D
Linear models, a final pitch. The linear model ( for classification or regres
sion ) is an often overlooked resource in the arena of learning from data. Since
efficient learning algorithms exist for linear models, they are low overhead.
They are also very robust and have good generalization properties. A sound

107

3 . THE LINEAR MODEL

3 . 4 . NONLINEAR TRANSFORMATION

policy to follow when learning from data is to first try a linear model. Because
of the good generalization properties of linear models, not much can go wrong.
If you get a good fit to the data ( low Ein) , then you are done. If you do not get
a good enough fit to the data and decide to go for a more complex model, you
will pay a price in terms of the VC dimension as we have seen in Exercise 3.12,
but the price is modest.

108

3 . 5 . PROBLEMS

3 . THE LINEAR MODEL

3.5

P roblems

Problem 3. 1

Consider the double sem i-circle "toy" learning task below.

There a re two semi circles of width thk with inner radius rad, separated by
as shown ( red is -1 and blue is + 1 ) . The center of the top sem i circle
is a l igned with the middle of the edge of the bottom sem i circle. This task
is linearly separa ble when sep 2: 0, and not so for sep < 0 . Set rad = 10,
thk = 5 a n d sep = 5 . Then, generate 2 , 000 exa mples u niformly, which means
you wi ll have a pproximately 1 , 000 exa m ples for each class.
sep

( a ) Run the P LA starting from w = 0 u ntil it converges. P lot the data and
the fin a l hypothesis.

( b ) Repeat part ( a ) using the linear regression ( for classification ) to obtai n w.

Explain you r observations.

Problem 3.2
For the dou ble sem i circle task in Problem 3 . 1 , vary sep i n
t h e range {0.2, 0.4, . . . , 5 } . Generate 2 , 000 exa mples and r u n t h e P LA starting
with w = 0. Record the n u m ber of iterations P LA takes to converge.
Plot sep versus the n u m ber of iterations ta ken for PLA to converge. Explain
you r observations. [Hint: Problem 1 . 3.}

Problem 3.3 For the dou ble sem i circle task in Problem 3 . 1 , set sep = - 5
a n d generate 2, 000 exa m ples.

( a ) What wil l happen if you ru n P LA on those exa mples?

( b ) Run the pocket algorithm for 100, 000 iterations a nd plot Ein versus the
iteration n u m ber t.

( c) Plot the data and the final hypothesis in part ( b ) .

(continued o n next page)

109

3 . THE LINEAR MODEL

3 . 5 . PROBLEMS

( d ) Use the linear regression a lgorithm to obta in the weights w, and com pare
this result with the pocket a lgorith m in terms of com putation time a n d
q u a l ity o f t h e sol ution .

( e ) Repeat ( b ) - ( d ) with a 3rd order polynomial featu re transform .

Problem 3.4

I n P roblem 1 . 5 , we introduced t h e Ada ptive Linear Neu

ron ( Ada line ) a lgorithm for classificatio n . Here, we derive Ada line from a n
optimization perspective.
(max(O, 1 - ynwTxn ) ) 2 . Show that En (w) is con
tin uous and differentia ble. Write down the gra d ient \7 En (w) .

( a ) Consider En (w)

( b ) S how that En (w) is an upper bound for [sign(wTxn ) i- Yn] . Hence,

tr L: ;:r= l En (w) is an u pper bound for the in sa m ple classification er

ror Ein (w) .

( c) Argue that the Ada li ne a lgorithm in Problem 1 . 5 performs stochastic

gra d ient descent on

tr L: := l En (w) .

Problem 3.5

( a ) Consider

En (w)

max(O, 1 - ynWT Xn )

Show that En (w) i s contin uous a n d differentiable except when

Yn = WTXn .

( b ) S how that En (w) is an upper bound for [sign(wTxn ) i- Ynl Hence,

tr L: ;:r= l En (w) is an u pper bound for the in sa m ple classification er

ror Ein (w) .

( c ) Apply stochastic grad ient descent on tr L: ;:r= l En (w) ( ignoring the sin
gular case of wT Xn

Yn ) and derive a new perceptron learning a lgorithm.

Problem 3.6
Derive a linear progra mming algorithm to fit a linear model
for classification using the following steps. A linea r progra m is a n optim ization
problem of the followi ng form :
cT z

min
z

Az :S h.

subject to

A, b and c are para meters of the linear program and z is the optimization vari
a ble. This is such a well studied optimization problem that most mathematics
software h ave ca n ned optim ization fu nctions which solve li near programs.

( a ) For linearly separa ble data , show that for some w , Yn (wTxn ) 2: 1 for
n =

l, . . . , N.

1 10

3 . 5 . PROBLEMS

3 . THE LINEAR MODEL

( b ) Formu late the task of finding a separating w for separa ble d ata as a linear
progra m . You need to specify what the parameters A, b, c are and what
the optimization variable z is.

( c ) If the data is not separa ble, the condition in ( a ) ca nnot hold for every n.
Thus i ntrod uce the violation t;,n 2: 0 to captu re the a mount of violation
for exa mple Xn . So, for n = 1, . . . , N,

Yn (WTX n ) 2: 1 - t;,n ,
t;,n 2: 0.
Natu ra lly, we would l i ke to m i n i m ize the amount of violation . One intu
itive a pproach is to m i n i m ize 2:,:= 1 t;,n , i .e . , we wa nt w that solves

subject to

n= l
Yn (wTxn ) 2: 1 - t;,n ,
t;,n 2: 0,

where the inequalities m ust hold for n

lem as a l i near program .

= 1 , . . . , N. Formulate th is prob

( d ) Argue that the linear program you derived in ( c ) a nd the optim ization
problem in Problem 3.5 are equ iva lent.

Problem 3. 7
Use the l i near programming a l gorithm from Problem 3.6
on the learn i ng task in Problem 3.1 for the separa ble (sep = 5) and the non
separa ble (sep = - 5 ) cases.
Compare your results to the l inear regression approach with and without the
3rd order polynomial featu re tra nsform .

Problem 3.8

For linear regressio n , t h e out of sa mple error is

Eout (h) = lE [(h(x) - y) 2 ]

Show that a mong all hypotheses, the one that minimizes Eout is given by

h* (x) = JE [y I x] .
The fu nction h * ca n be treated as a deterministic target function , in which
case we can write y = h * (x) + E(x) where E(x) is a n ( input dependent ) noise
varia ble. S h ow that E(x) has expected value zero.

111

3 . THE LINEAR M ODEL

3 . 5 . PROBLEMS

Problem 3.9
Assuming that XT X is invertible, show by direct com parison
with Equation (3.4) that Ein(w) ca n be written as

Ein (w)

= (w - ( XTx) 1 XT y r ( XT X) (w - (X Tx) 1 XT y) + yT (I - X (X Tx) 1 XT )y.

Use this expression for Ein to obtain W!in What is the in sa mple error? [Hint:

The matrix XT X is positive definite.]

Problem 3 . 1 0
Exercise 3.3 stud ied some properties of the hat matrix
H = X ( X T X) 1 XT , where X is a N by d + 1 matrix, and XT X is i nvertible.
S how the following additiona l properties.

( a ) Every eigenva lue of H is either 0 or 1 . [Hint: Exercise 3.3(b).]

( b ) S how that the trace of a symmetric matrix equals the sum of its eigen

va l u es. [Hint: Use the spectral theorem and the cyclic property of the

trace. Note that the same result holds for non-symmetric matrices, but
is a little harder to prove.]

( c ) How many eigenva lues of H are 1 ? What is the ran k of H? [Hint:

Exercise 3. 3(d).j

Problem 3 . 1 1 Consider the linear regression problem setup in Exercise 3.4,

where the data comes from a gen ui ne l i near relationship with added noise. The
noise for the different data points is assu med to be iid with zero mean a nd
variance CJ 2 . Assume that the 2nd moment matrix I: = lEx [xxT] is non-singu lar.
Follow the steps below to show that, with high proba bility, the out-of-sa mple
error on a verage is

Eout (W!in) = (5 2 1 +

d+ l

1 ).

+ o( N )

( a ) For a test point x, show that the error y - g(x) is

E -

X T ( X T X ) l XT E ,

where E is the noise rea lization for the test point and
noise rea lizations on the data .

E is the vector of

( b ) Take the expectation with respect to the test point, i . e . , x a nd

obtai n a n expression for Eaut S h ow that

E,

to

2
Eaut = CJ + trace (I:( XT X ) 1 XT EET XT ( XT X ) 1 ) .

[Hints: a = trace( a) for any scalar a; trace(AB) = trace(BA) ; expecta

tion and trace commute.]

( c ) What is lEe [EET ] ?

1 12

3 . THE LINEAR M ODEL

3 . 5 . PROBLEMS

( d) Take the expectation with respect to

to show that, on average,

2
Bout = a-2 + 0- trace (I:( N1 XT X) - 1 ) .
N
Note that :KrXTX = :Kr L::=l XnX is a n N sa m ple estimate of
:KrXTX I:. If :KrXTX = I:, then what is Bout on average?

I:.

So

( e) S how that (after taking the expectation over the data noise) with high
probability,

Bout = o-2 1 +

l + o ( :Kr)

[Hint: By the law of large numbers :Kr XTX converges in probability to

I:, and so by continuity of the inverse at I:, ( :KrXTX) - 1 converges in
probability to I:- 1 . J

Problem 3 . 1 2 In linear regression , the i n sa m ple pred ictions are given by

y = Hy, where H = X(XTX)- 1 XT. Show that H is a projection matrix, i .e.
H2 = H. So y is the projection of y onto some space. What is this space?

Problem 3.13

This problem creates a l i near regression a lgorith m from

a good a l gorith m for linear classification. As i l l ustrated , the idea is to ta ke the
origin a l data a nd shift it in one direction to get the +1 data points; then , shift
it in the opposite d irection to get the - 1 data points.

Origina l data for the one

d imensiona l regression prob
lem

Sh ifted data viewed as a

two dimensiona l classifica
tion problem

More genera l ly, The data (xn , Yn) ca n be viewed as data points in
treating the y val u e as the ( + 1 )th coord inate .

JRd+ 1

by

(continued o n next page)

1 13

3 . 5 . PROBLEMS

3 . THE LINEAR MODEL

Now, construct positive and negative points

D+

(x 1 , y1 ) + a, . . . , (xN , YN ) + a
(x 1 , y1 ) a, . . . , (xN , YN ) a,

1)_

where a is a perturbation para m eter. You ca n now use the linear program m ing
algorithm in Problem 3.6 to separate D+ from 1)_ . The resulting separating
hyperplane can be used as the regression 'fit' to the original data .

( a ) How m a ny weights a re lea rned i n the classification problem? H ow many

weights are needed for the linear fit in the regression problem?

( b ) The linear fit req uires weights w, where h(x)

= w Tx . S u ppose the
weights returned by solving the classification problem a re wclass . Derive
an expression for w as a fu nction of Wc1ass

( c ) Generate a data set Yn

[O, 1] a nd

for

a=

En

[]
01

= x;;, + O"En with N = 50, where Xn is u n iform on

is zero mean Ga ussian noise; set O" = 0 . 1 . Plot D+ and 1)_

( d ) Give com parisons of the resulting fits from ru n n ing the classification a p
proach and the a n a lytic pseudo-inverse a lgorithm for linear regression .

Problem 3 . 14 I n a regression setting, assume the target function is l inear,

so f(x) = xTWf , and y = Xw1 + E , where the entries in E are zero mea n , iid
with varia nce 0" 2 . In this problem derive the bias a n d varia nce as follows.

( a ) Show that the average fu nction is g(x)

of the data set, as long as

= f(x) , no matter what the size

XT X is invertible. What is the bias?

( b ) What is the variance? [Hint: Problem 3. 1 1]

Problem 3 . 15

I n the text we derived that the li near regression solution

= XTy . If XT X is not inverti ble, the solution
Wiin = ( X TX ) 1 XTy won 't work. In this event, there wil l be many sol utions
for w that m i n imize Ein Here, you wil l derive one such sol ution . Let p be
the ra n k of X . Assume that the singular va l ue decom position ( SVD ) of X is
+ l) X p satisfies
N p
x = urvT ' where u E JR X satisfies UTU = Ip. v E JR ( d
VTV = Ip . and r E ]RP X P i s a positive diagona l matrix.
weights m ust satisfy

XT Xw

( a ) Show that p < d + 1 .

( b ) S how that W!in = vr 1 uTy satisfies XT XW!in = XTy, a nd hence is a
sol ution.

( c ) Show that for a ny other sol ution that satisfies XT Xw = XTy, l l wl i ll <
n

ll w ll That is, the sol ution we h ave constructed is the minimum norm set
of weights that m i n i mizes Ein

1 14

3.

THE LINEAR MODEL

3 . 5 . PROBLEMS

Problem 3.16
In Exa m ple 3.4, it is mentioned that the output of the
fin a l hypothesis g(x) learned using logistic regression ca n be thresholded to get
a ' hard ' (1) classification. This problem shows how to use the risk m atrix
introduced in Exa m ple 1 . 1 to obtain such a threshold .
Consider fin gerprint verification , as in Exam ple 1 . 1 . After learn ing from the
data using logistic regression , you prod uce the fi nal hypothesis

g(x)

P[y

+1 I x) ,

which is you r estimate of the proba bility that y

matrix is given by

you say

+1

+1. S u ppose that the cost

True classification
- 1 (intruder)
0
Ca

+ 1 (correct person)

-1

For a new person with fingerprint x, you com pute g(x) and you now need to de
cide whether to accept or reject the person ( i . e . , you need a hard classification ) .
So, you wi ll accept i f g(x) K , where K i s t h e threshold .
(a) Define the cost(accept) as your expected cost if you accept the perso n .
Similarly define cost(reject) . Show that
(1 - g(x) ) ca ,

cost( accept)
cost( reject)

g(x) cr .

(b) Use part (a) to derive a condition on g(x) for accepting the person a nd
hence show that

Ca
Ca + Cr

K, = -- .

( c) Use the cost matrices for the S u permarket and CIA a pplications i n Ex
a m ple 1 . 1 to compute the threshold K for each of these two cases. G ive
some i ntu ition for the thresholds you get.

Problem 3 . 1 7

(a)

Consider a fu nction

E(u, v) = eu + e2v + euv + u2 - 3uv + 4v 2 - 3u - 5v ,

Approximate E(u + b.u, v + b.v) by E1 (b.u, b.v), where E1 is the
first-order Taylor's expansion of E around (u, v) = (0, 0) . S u ppose
E1 (b.u, b.v) = au b.u + av b.v + a. What are the va l ues of au , a v ,
a n d a?

(continued on next page)

115

3 . 5 . PROBLEMS

3 . THE LINEAR MODEL

( b ) M i n imize E1 over a l l possible (L\u, L\v) such that ll (L\u, L\v) ll = 0.5 .
I n this cha pter, we proved that the optim a l colu m n vector

[]

is

para llel to the col u m n vector -\i'E(u, v), which is ca l led the negative
gradient direction. Com pute the optimal (L\u, L\v) a nd the resulting

E(u + L\u, v + L\v) .

( c ) Approximate E(u+ L\u, v + L\v) by E2 (L\u, L\v) , where E2 is the second
order Taylor's expa nsion of E a round ( u, v) = (0, 0) . S u ppose
What a re the va l ues of buu , bvv , buv , bu , bv , a nd b?

( d ) M i n im ize E2 over a l l possible (L\u, L\v) ( regardless of length ) . Use the

fact that \7 2 E( u, v) I ( o ,o) ( the Hessia n matrix at ( 0 , 0 ) ) is positive definite
to prove that the optimal col u m n vector

*]
[L\u
L\v *

(\7 2 E(u, v) ) -1 \7E(u, v),

which is cal led t h e Newton direction.

( e ) N umerica l ly com pute the following va l ues:

( i ) the vector (L\u, L\v) of length 0.5 a long the Newton direction, and
the resu lting

E(u + L\u, v + L\v) .

( ii ) the vector (L\u, L\v) of length 0.5 that minimizes E(u+L\u, v+L\v) ,
and the resulting E(u + L\u, v + L\v) . ( Hint: Let L\u = 0 . 5 sin 8. )
Compare the val ues of E(u + L\u, v + L\v) in ( b ) , ( e i ) , and ( e ii ) . Briefly

state you r findings.

The negative grad ient direction a nd the Newton direction a re q uite fu nda menta l
for designing optim ization a lgorithms. It is importa nt to u ndersta nd these
directions and put them in your toolbox for designi ng learn ing algorith ms.

Problem 3 . 1 8

(a )
( b)
( c)
(d )

Take the feature tra nsform

<I> 2 i n Eq u ation (3. 13) a s <I>.

S how that dvc (1-icp ) :S 6.

S how that dvc (Hq, ) > 4. {Hint: Exercise 3. 12}
G ive an u pper bound on dvc (Hq,k ) for

X = IR d .

Defi ne
-

Argue that dvc (Hq, 2 ) = dvc (H;p 2 ) . I n other words, while <I> 2 (X) E IR9 ,
dvc (1-l;p ) :S 6 < 9. Th us, the dimension of <I>(X) o n ly gives a n upper
2
bound of dvc (Hq, ) , a nd the exact va lue of dvc (1icp ) ca n depend on the
com ponents of the transform .

116

3. THE LINEAR MODEL

3 . 5 . PROBLEMS

Problem 3.19
A Tra nsformer thinks the following proced u res would
work wel l in lea rn ing from two-d imensional data sets of a ny size . P lease point
out if there a re any potentia l problems in the proced ures:
(a) Use the feature transform
<T? ( x) =

(0, . . . ' 0 , 1, 0, . . )

if X =

(0, 0, . . . ' 0)

otherwise .

Xn

before run n ing P LA .

(b) Use the feature transform

using some very sma ll

'Y

(c) Use the feature transform

before running PLA, with

1?

with

1?

that consists of a l l

E {O,

1 17

. . . , 1 } a nd j E {O,

. . . , 1}.

1 18

Chapter

Overfitt ing
Paraskavedekatriaphobia 1 (fear of Friday the 1 3th) , and superstitions in gen
eral, are perhaps the most illustrious cases of the human ability to overfit.
Unfortunate events are memorable, and given a few such memorable events,
it is natural to try and find an explanation. In the future, will there be more
unfortunate events on Friday the 13th's than on any other day?
Overfitting is the phenomenon where fitting the observed facts (data) well
no longer indicates that we will get a decent out-of-sample error, and may
actually lead to the opposite effect. You have probably seen cases of overfit
ting when the learning model is more complex than is necessary to represent
the target function. The model uses its additional degrees of freedom to fit
idiosyncrasies in the data (for example, noise) , yielding a final hypothesis that
is inferior. Overfitting can occur even when the hypothesis set contains only
functions which are far simpler than the target function, and so the plot thick
ens @) .
The ability to deal with overfitting is what separates professionals from
amateurs in the field of learning from data. We will cover three themes:
When does overfitting occur? What are the tools to combat overfitting? How
can one estimate the degree of overfitting and ' certify' that a model is good,
or better than another? Our emphasis will be on techniques that work well in
practice.
4.1

When Does Overfitting O ccur?

Overfitting literally means "Fitting the data more than is warranted." The
main case of overfitting is when you pick the hypothesis with lower Ein, and
it results in higher Eout . This means that Ein alone is no longer a good guide
for learning. Let us start by identifying the cause of overfitting.
1 from the Greek paraskevi (Friday), dekatreis (thirteen) , phobia (fear)

1 19

4 . 1 . WHEN DOES 0VERFITTING OCCUR?

4 . 0VERFITTING

Consider a simple one-dimensional regression problem with five data points.

We do not know the target function, so let's select a general model, maximiz
ing our chance to capture the target function. Since 5 data points can be fit
by a 4th order polynomial, we select 4th order polynomials.
The result is shown on the right. The
target function is a 2nd order polynomial
0 Data
- Target
(blue curve), with a little added noise in
Fit
the data points. Though the target is
simple, the learning algorithm used the
full power of the 4th order polynomial to
fit the data exactly, but the result does
not look anything like the target function.
The data has been 'overfit. ' The little
x
noise in the data has misled the learning,
for if there were no noise, the fitted red
curve would exactly match the target . This is a typical overfitting scenario,
in which a complex model uses its additional degrees of freedom to 'learn' the
noise.
The fit has zero in-sample error but huge out-of-sample error, so this is a
case of bad generalization (as discussed in Chapter 2) a likely outcome when
overfitting is occurring. However, our definition of overfitting goes beyond
bad generalization for any given hypothesis. Instead, overfitting applies to
a process : in this case, the process of picking a hypothesis with lower and
lower Ein resulting in higher and higher Eout.
4.1.1

A Case Study: Overfitting with Polynomials

Let's dig deeper to gain a better understanding of when overfitting occurs. We

will illustrate the main concepts using data in one-dimension and polynomial
regression, a special case of a linear model that uses the feature transform
) . Consider the two regression problems below:
x f- ( 1 , x , x 2 ,

O Data
- Target

O Data
- Target

(b) 50th order target function

(a) 10th order target function

In both problems, the target function is a polynomial and the data set V
contains 15 data points. In (a) , the target function is a 10th order polynomial
120

4 . 1 . WHEN DOES 0VERFITTING OCCUR 7

4 . 0VERFITTIN G

O Data
2nd Order Fit
10th Order Fit

O Data
- 2nd Order Fit
10th Order Fit
x

( a) Noisy low order target

(b) Noiseless high order target

Figure 4. 1 : Fits using 2nd and 10th order polynomials to 15 data points.
In ( a ) , the data are noisy and the target is a 10th order polynomial. In (b)

the data are noiseless and the the target is a 50th order polynomial.

and the sampled data are noisy ( the data do not lie on the target function
curve ) . In ( b ) , the target function is a 50th order polynomial and the data are
noiseless.
The best 2nd and 10th order fits are shown in Figure 4. 1 , and the in-sample
and out-of-sample errors are given in the following table.
50th order noiseless target
2nd Order 10th Order
100.029
0. 120
7680

10th order noisy target

2nd Order 10th Order
0.050
0.034
Ein
9.00
0 . 127
Eout

What the learning algorithm sees is the data, not the target function. In both
cases, the 1 0th order polynomial heavily overfits the data, and results in a
nonsensical final hypothesis which does not resemble the target function. The
2nd order fits do not capture the full nature of the target function either, but
they do at least capture its general trend, resulting in significantly lower out-of
sample error. The 10th order fits have lower in-sample error and higher out-of
sample error, so this is indeed a case of overfitting that results in pathologically
bad generalization.
Exercise 4.1
Let 1-fo a nd 1-l 1 0 be t h e 2 n d a n d 10th order hypothesis sets respectively.
S pecify t h ese sets as para meterized sets of functions. S how that 1-l2 C 1-l 1 0 .

These two examples reveal some surprising phenomena. Let's consider first the
10th order target function, Figure 4.l ( a) . Here is the scenario. Two learners, 0
( for overfitted) and R ( for restricted ) , know that the target function is a 10th
order polynomial, and that they will receive 15 noisy data points. Learner 0
121

4. 0VERFITTING

Learning curves for

4. 1 . WHEN DOES 0VERFITTING OCCUR?

1-l 2

Learning curves for

H
0
H
H

1-l 1 0

H
0

t:
;:i

;:i

'"O
(].)
-+.:>
u
(].)

'"O
(].)
-+.:>
u
(].)

;:i

Number of Data Points, N

Number of Data Points, N

Figure 4.2: Overfitting is occurring for N in the shaded gray region

because by choosing 1-l 1 0 which has better Ein, you get worse Eout

uses model 1-l10, which is known to contain the target function, and finds the
best fitting hypothesis to the data. Learner R uses model 1-{ 2 , and similarly
finds the best fitting hypothesis to the data.
The surprising thing is that learner R wins (lower out-of-sample error) by
using the smaller model, even though she has knowingly given up the ability
to implement the true target function. Learner R trades off a worse in-sample
error for a huge gain in the generalization error, ultimately resulting in lower
out-of-sample error. What is funny here? A folklore belief about learning is
that best results are obtained by incorporating as much information about the
target function as is available. But as we see here, even if we know the order
of the target and naively incorporate this knowledge by choosing the model
accordingly (1-l10 ) , the performance is inferior to that demonstrated by the
more 'stable' 2nd order model.
The models 1-l2 and 1-l 10 were in fact the ones used to generate the learn
ing curves in Chapter 2, and we use those same learning curves to illustrate
overfitting in Figure 4.2. If you mentally superimpose the two plots, you can
see that there is a range of N for which 1-l10 has lower Ein but higher Eout
than 1-{ 2 does, a case in point of overfitting.
Is learner R always going to prevail? Certainly not. For example, if the
data was noiseless, then indeed learner 0 would recover the target function
exactly from 15 data points, while learner R would have no hope. This brings
us to the second example, Figure 4. l (b). Here, the data is noiseless, but the
target function is very complex (50th order polynomial) . Again learner R
wins, and again because learner 0 heavily overfits the data. Overfitting is
not a disease inflicted only upon complex models with many more degrees of
freedom than warranted by the complexity of the target function. In fact the
reverse is true here, and overfitting is just as bad. What matters is how the
model complexity matches the quantity and quality of the data we have, not
how it matches the target function.
122

4 . 0VERFITTING

4 . 1. 2

4 . 1 . WHEN D oEs OvERFITTING O ccuR?

Catalysts for Overfitting

A skeptical reader should ask whether the examples in Figure 4. 1 are just
pathological constructions created by the authors, or is overfitting a real phe
nomenon which has to be considered carefully when learning from data? The
next exercise guides you through an experimental design for studying overfit
ting within our current setup. We will use the results from this experiment
to serve two purposes: to convince you that overfitting is not the result of
some rare pathological construction, and to unravel some of the conditions
conducive to overfitting.
Exercise 4.2 [Experimental design for studying overfitting]
This is a reading exercise that sets u p a n experimenta l framework to study
various a spects of overfitting. The reader interested in i m plementin g the
experiment can find the details fleshed out in Problem 4 .4. The i nput
space is X = [ - 1 , 1]. with u n iform i nput probability density, P(x) =
We consider the two models H2 a n d H 1 0 .

The target is a degree-Qi polynomi a l , which we write f(x)

!,_0 aqLq (x) , where Li (x) are polynom i a ls of increasing complexity (the
Legendre polynomials) . The data set is D = (x 1 , y1 ) , . . . , (xN , YN ) , where
Yn = f (x n) + <YEn a n d En are iid ( i ndependent a n d identica lly d istributed )

standard Norm a l random variates.

For a single experiment, with specified val ues for Q1 , N, a- , generate a ran
dom degree-Qi target function by selecting coefficients ai independently
Gen
from a standard Norm a l , resca l i ng them so that lEa,x [f 2 ] =
erate a d ata set, selecting x 1 , . . . , XN i ndependently according to P(x)
and Yn = f (x n ) + <Y En . Let g2 a nd g10 be the best fit hypotheses to
the d ata from 1{2 a n d H 1 0 respectively, with out-of-sam ple errors Eout (g2 )
a nd Bout (g w ) .

Vary Q1 , N, a- , a n d for each combination of parameters, run a l arge n u m ber

of experi ments, each time computing Eout (g2 ) and Eout (g10 ) . Averaging
these out-of-sam ple errors gives estimates of the expected out-of-sample
error for the given learning scenario (QI , N , a- ) using H 2 and 1-lw .

Exercise 4.2 set up an experiment to study how the noise level cr 2 , the target
complexity Q f , and the number of data points N relate to overfitting. We
compare the final hypothesis 910 E 1{10 (larger model) to the final hypothesis
92 E 1-l 2 (smaller model) . Clearly, Ein (910) :: Ein (92 ) since 910 has more
degrees of freedom to fit the data. What is surprising is how often 910 overfits
the data, resulting in Eout (910) > Eout (92 ) . Let us define the overfit measure
as Eout (910) Eout (92 ) . The more positive this measure is, the more severe
overfitting would be.
Figure 4.3 shows how the extent of overfitting depends on certain parame
ters of the learning problem (the results are from our implementation of Exer
cise 4.2) . In the figure, the colors map to the level of overfitting, with redder
123

4 . 0VERFITTING

IN

..
Q)
rn

s
z

4 . 1 . WHEN DOES 0VERFITTING OCCUR?

00
0

_e; 75
;;

0.

s 50
0
0
"t) 25

80

100

120

Number of Data Points, N

( a)

( b)

Stochastic noise

100

80

120

Number of Data Points, N

Deterministic noise

Figure 4.3: How overfitting depends on the noise

2
CT ,

the target function

complexity QJ , and the number of data points N. The colors map to the
overfit measure Eout (1-l 10 ) - Eout (1-fo ) . In ( a) we see how overfitting depends
on CT2 and N, with QJ = 20. As CT 2 increases we are adding stochastic noise
to the data. In ( b ) we see how overfitting depends on Qf and N, with
2
CT = 0. 1 . As Q f increases we are adding deterministic noise to the data.

regions showing worse overfitting. These red regions are large overfitting is
real, and here to stay.
Figure 4.3( a) reveals that there is less overfitting when the noise level <5 2
drops or when the number of data points N increases (the linear pattern in
Figure 4.3(a) is typical) . Since the 'signal' f is normalized to IE [j 2 ] = 1 ,
the noise level <52 is automatically calibrated t o the signal level. Noise leads
the learning astray, and the larger, more complex model is more susceptible to
noise than the simpler one because it has more ways to go astray. Figure 4.3(b)
reveals that target function complexity Q f affects overfitting in a similar way
to noise, albeit nonlinearly. To summarize,

Deterministic noise. Why does a higher target complexity lead to more

overfitting when comparing the same two models? The intuition is that for a
given learning model, there is a best approximation to the target function. The
part of the target function 'outside' this best fit acts like noise in the data. We
can call this deterministic noise to differentiate it from the random stochastic
noise . Just as stochastic noise cannot be modeled, the deterministic noise
is that part of the target function which cannot be modeled. The learning
algorithm should not attempt to fit the noise; however, it cannot distinguish
noise from signal. On a finite data set, the algorithm inadvertently uses some

124

4 . 1 . WHEN DOES 0VERFITTING OCCUR?

4 . 0VERFITTING

Figure 4.4: Deterministic noise. h * is the best fit to f in

illustrates deterministic noise for this learning problem.

1-l 2 .

The shading

of the degrees of freedom to fit the noise, which can result in overfitting and
a spurious final hypothesis.
Figure 4.4 illustrates deterministic noise for a quadratic model fitting a
more complex target function. While stochastic and deterministic noise have
similar effects on overfitting, there are two basic differences between the two
types of noise. First, if we generated the same data (x values) again, the
deterministic noise would not change but the stochastic noise would. Second,
different models capture different 'parts' of the target function, hence the same
data set will have different deterministic noise depending on which model we
use. In reality, we work with one model at a time and have only one data set
on hand. Hence, we have one realization of the noise to work with and the
algorithm cannot differentiate between the two types of noise.
Exercise

4.3

Determ i nistic noise depends on

than others.

1-l,

as some models a pproximate f better

(a) Assume 1-l is fixed a n d we i ncrease the complexity of f. Will deter

m i nistic noise in genera l go up or down? Is there a higher or lower
tendency to overfit?
( b) Assume f is fixed and we decrease the complexity of 1-l. Will deter
m i nistic noise i n general go u p or down? Is there a higher or lower
tendency to overfit? [Hint: There is a race between two factors that

affect overfitting in opposite ways, but one wins.}

The bias-variance decomposition, which we discussed in Section 2.3. 1 (see also

Problem 2.22) is a useful tool for understanding how noise affects performance:

1Ev [Eout ]

a- 2 + bias + var.

The first two terms reflect the direct impact of the stochastic and determin
istic noise. The variance of the stochastic noise is a- 2 and the bias is directly
125

4. 0VERFITTING

4 . 2 . REGULARIZATION

related to the deterministic noise in that it captures the model's inability to

approximate f. The va r term is indirectly impacted by both types of noise,
capturing a n1.o del's susceptibility to being led astray by the noise.
4.2

Regularization

Regularization is our first weapon to combat overfitting. It constrains the

learning algorithm to improve out-of-sample error, especially when noise is
present. To whet your appetite, look at what a little regularization can do for
our first overfitting example in Section 4.1 . Though we only used a very small
'amount' of regularization, the fit improves dramatically.
O Data
-

Target
Fit

with regularization

without regularization

Now that we have your attention, we would like to come clean. Regularization
is as much an art as it is a science. J\/Iost of the methods used successfully
in practice are heuristic methods. However, these methods are grounded in a
mathematical framework that is developed for special cases. We will discuss
both the mathematical and the heuristic, trying to maintain a balance that
reflects the reality of the field.
Speaking of heuristics, one view of regularization is through the lens of the
VC bound, which bounds Eout using a model complexity penalty 0(1-l) :
for all h E 1-l.

( 4.1)

So, we are better off if we fit the data using a simple 1-l. Extrapolating one step
further, we should be better off by fitting the data using a 'simple' h from 1-l.
The essence of regularization i s t o concoct a measure O(h) for the complexity
of an individual hypothesis. Instead of minimizing Ein ( h) alone, one minimizes
a combination of Ein (h) and O(h) . This avoids overfitting by constraining the
learning algorithm to fit the data well using a simple hypothesis.
Example 4.1. One popular regularization technique is weight decay, which

measures the complexity of a hypothesis h by the size of the coefficients used

to represent h (e.g. in a linear model) . This heuristic prefers mild lines with
126

4 . 0VERFITTING

4 . 2 . REGULARIZATION

small offset and slope, to wild lines with bigger offset and slope. We will get to
the mechanics of weight decay shortly, but for now let's focus on the outcome.
We apply weight decay to fitting the target f ( x) = sin( ?TX ) using N = 2
data points (as in Example 2.8) . Vve sample x uniformly in [ 1 , 1] , generate a
data set and fit a line to the data (our model is H 1 ) . The figures below show the
resulting fits on the same (random) data sets with and without regularization.

without regularization

with regularization

Without regularization, the learned function varies extensively depending on

the data set. As we have seen in Example 2.8, a constant model scored
Eout = 0.75, handily beating the performance of the (unregularized) linear
model that scored Eout = 1 . 90. With a little weight decay regularization,
the fits to the same data sets are considerably less volatile. This results in a
significantly lower Eout = 0.56 that beats both the constant model and the
unregularized linear model.
The bias-variance decomposition helps us to understand how the regular
ized version beat both the unregularized version as well as the constant model.

without regularization
bias = 0.21;
var = 1.69.

with regularization
bias = 0.23;
var = 0.33.

Average hypothesis g ( red ) with var(x) indicated by the gray shaded

region that is g(x)

As expected, regularization reduced the var term rather dramatically from 1 .69
down to 0.33. The price paid in terms of the bias (quality of the average fit) was
127

4 . 0VERFITTING

4 . 2 . REGULARIZATION

modest, only slightly increasing from 0.21 to 0.23. The result was a significant
decrease in the expected out-of-sample error because bias + var decreased. This
is the crux of regularization. By constraining the learning algorithm to select
'simpler' hypotheses from 1-l , we sacrifice a little bias for a significant gain in
D
the var.
This example also illustrates why regularization is needed. The linear
model is too sophisticated for the amount of data we have, since a line can
perfectly fit any 2 points. This need would persist even if we changed the
target function, as long as we have either stochastic or deterministic noise.
The need for regularization depends on the quantity and quality of the data.
Given our meager data set, our choices were either to take a simpler model,
such as the model with constant functions, or to constrain the linear model. It
turns out that using the complex model but constraining the algorithm toward
simpler hypotheses gives us more flexibility, and ends up giving the best Eout.
In practice, this is the rule not the exception.
Enough heuristics. Let's develop the mathematics of regularization.
4 .2. 1

A Soft Order Constraint

In this section, we derive a regularization method that applies to a wide va

riety of learning problems. To simplify the math, we will use the concrete
setting of regression using Legendre polynomials, the polynomials of increas
ing complexity used in Exercise 4.2. So, let's first formally introduce you to
the Legendre polynomials.
Consider a learning model where 1-l is the set of polynomials in one vari
able x E [ 1 , 1) . Instead of expressing the polynomials in terms of consecutive
powers of x, we will express them as a combination of Legendre polynomials
in x. Legendre polynomials are a standard set of polynomials with nice ana
lytic properties that result in simpler derivations. The zeroth-order Legendre
polynomial is the constant Lo ( x ) = 1 , and the first few Legendre polynomials
are illustrated below.
L3

L2

(3x2

1)

H5x3

L4

3x)

(35x4

30x2 + 3)

Ls
(63x5 . . )

As you can see, when the order of the Legendre polynomial increases, the curve
gets more complex. Legendre polynomials are orthogonal to each other within
x E [ 1 , 1] , and any regular polynomial can be written as a linear combination
of Legendre polynomials, just like it can be written as a linear combination of
powers of x.
128

4 . 0VERFITTIN G

4 . 2 . REGULARIZATION

Polynomial models are a special case of linear models in a space Z, under

a nonlinear transformation <I? : X -+ Z. Here, for the Qth order polynomial
model, <I? transforms into a vector z of Legendre polynomials,

Our hypothesis set

HQ

[LQ(L1xx))]

is a linear combination of these polynomials,

Lo(x)

where
1. As usual, we will sometimes refer to the hypothesis h by its
weight vector w. 2 Since each h is linear in w, we can use the machinery of
linear regression from Chapter 3 to minimize the squared error
N

nI=l )wTZn

Ein (w)

yn) 2

(4.2)

The case of polynomial regression with squared-error measure illustrates the

main ideas of regularization well, and facilitates a solid mathematical deriva
tion. Nonetheless, our discussion will generalize in practice to non-linear,
multi-dimensional settings with more general error measures. The baseline al
gorithm (without regularization) is to minimize Ein over the hypotheses in
to produce the final hypothesis
wn z, where WHn argmin Ein (w) .

HQ

g(x)

Exercise 4.4

ZNr be the d ata matrix (assume Z has ful l column

Let Z
[z1
(ZTz) - 1 Vy; a nd let H
Z (ZT z) - 1 ZT (the h at matrix
ran k) ; let Wiin
of Exercise 3.3). Show that
) _ (w
m (W -

.
E

where

WlinfVZ(w

Wlin)

yT(l

H)y

(4.3)

is the identity matrix.

(a) What val ue of w minimizes Ein?

(b) What is the minimum i n sa mple error?

The task of regularization, which results in a final hypothesis wreg instead of

the simple WHn , is to constrain the learning so as to prevent overfitting the
2 We used w and d for the weight vector and dimension in Z. Since we are explicitly
dealing with polynomials and Z is the only space around, we use w and Q for simplicity.

129

4. 0VERFITTING

4 . 2 . REGULARIZATION

data. We have already seen an example of constraining the learning; the set 1-l 2
can be thought of as a constrained version of 1-l 10 in the sense that some of
the 1-l10 weights are required to be zero. That is, 1-l 2 is a subset of 1-l10 defined
by 1-l 2 { w I w E 1-l10; Wq 0 for q 3} . Requiring some weights to be 0 is
a hard constraint. We have seen that such a hard constraint on the order can
help, for example 1-l 2 is better than 1-l10 when there is a lot of noise and N is
small. Instead of requiring some weights to be zero, we can force the weights
to be small but not necessarily zero through a softer constraint such as

This is a 'soft order' constraint because it only encourages each weight to be

small, without changing the order of the polynomial by explicitly setting some
weights to zero. The in-sample optimization problem becomes:

min Ein (w) subject to wTw C.

w

(4.4)

The data determines the optimal weight sizes, given the total budget C which
determines the amount of regularization; the larger C is, the weaker the con
straint and the smaller the amount of regularization. We can define the soft
order-constrained hypothesis set 1-l( C) by

Equation (4.4) is equivalent to minimizing Ein over 1-l (C) . If C1 < 02 , then
C 1-l(C2 ) and so dvc (1-l(C1 )) dvc(1-l(C2 )), and we expect better
generalization with 1-l( C1 ) . Let the regularized weights Wreg be the solution
to (4.4) .

1-l(C1 )

Solving for Wreg If wiin Wlin c then Wreg

Wlin because Wlin E 1-l ( C) . If W1in tj_ 1-l ( C) , then
not only is wieg Wreg C, but in fact wieg Wreg C
(wreg uses the entire budget C; see Problem 4. 10) .
We thus need to minimize Ein subject to the
equality constraint wTw
C. The situation is
illustrated to the right. The weights w must lie

, the normal
on the surface of the sphere
vector to this surface at w is the vector w itself
(also in red) . A surface of constant Ein is shown in
blue; this surface is a quadratic surface (see Exercise 4.4) and the normal to
this surface is
. In this case, w cannot be optimal because \7 Ein ( w) is
not parallel to the red normal vector. This means that \1Ein (w) has some non
zero component along the constraint surface, and by moving a small amount
in the opposite direction of this component we can improve Ein, while still
130

4 . OvERFITTING

4 . 2 . REGULARIZATION

remaining on the surface. If Wreg is to be optimal, then for some positive

parameter Ac
i.e. , \7Ein must be parallel to Wreg, the normal vector to the constraint surface
(the scaling by 2 is for mathematical convenience and the negative sign is
because \7 Ein and w are in opposite directions) . Equivalently, Wreg satisfies

because V(wTw)

2w . So, for some Ac > 0, Wreg locally minimizes

(4.5)

The parameter Ac and the vector Wreg (both of which depend on C and the
data) must be chosen so as to simultaneously satisfy the gradient equality and
the weight norm constraint w;eg Wreg C. 3 That Ac > 0 is intuitive since
we are enforcing smaller weights, and minimizing Ein(w) + AcwTw would
not lead to smaller weights if Ac were negative. Note that if wlin W1in :: C,
Wreg WHn and minin1izing (4.5) still holds with Ac 0. Therefore, we
have an equivalence between solving the constrained problem ( 4.4) and the
unconstrained minimization of ( 4.5) . This equivalence means that minimiz
ing ( 4.5) is similar to minimizing Ein using a smaller hypothesis set, which in
turn means that we can expect better generalization by minimizing ( 4.5) than
by just minimizing Ein.
Other variations of the constraint in ( 4.4) can be used to emphasize some
weights over the others. Consider the constraint =O /qW :: C. The im
portance /q given to weight Wq determines the type of regularization. For
example, /q q or /q e q encourages a low-order fit, and /q (1 + q) - 1 or
/q e- q encourages a high-order fit. In extreme cases, one recovers hard-order
constraints by choosing some /q 0 and some /q -+ oo .
Exercise 4 . 5 [Tikhonov regularizer]
A more genera l soft constraint is the Tikhonov regu larization constraint

which ca n ca ptu re relationships among the Wi (the m atrix r is the Tikhonov

regularizer) .

( a ) What should r be to obtain the constraint I:=o w :: C?

( b ) What should r be to obtain the constra i nt

(2.:=0 Wq ) 2 :: C?

3 >.c is known as a Lagrange multiplier and an alternate derivation of these same results
can be obtained via the theory of Lagrange multipliers for constrained optimization.

131

4 . 0VERFITTING

4 .2.2

4 . 2 . REGULARIZATION

Weight Decay and Augmented Error

The soft-order constraint for a given value of C is a constrained minimiza

tion of Ein Equation (4.5) suggests that we may equivalently solve an un
constrained minimization of a different function. Let's define the augmented
error,
(4.6)
where ,\ 2:'.: 0 is now a free parameter at our disposal. The augmented error has
two terms. The first is the in-sample error which we are used to minimizing,
and the second is a penalty term. Notice that this fits the heuristic view of
regularization that we discussed earlier, where the penalty for complexity is
defined for each individual h instead of 1-l as a whole. When ,\ 0, we have the
usual in-sample error. For ,\ > 0, minimizing the augmented error corresponds
to minimizing a penalized in-sample error. The value of ,\ controls the amount
of regularization. The penalty term wTw enforces a tradeoff between making
the in-sample error small and making the weights small, and has become known
as weight decay. As discussed in Problem 4.8, if we minimize the augmented
error using an iterative method like gradient descent, we will have a reduction
of the in-sample error together with a gradual shrinking of the weights, hence
the name weight 'decay. ' In the statistics community, this type of penalty
term is a form of ridge regression.
There is an equivalence between the soft order constraint and augmented
error minimization. In the soft-order constraint, the amount of regularization
is controlled by the parameter C. From (4.5) , there is a particular .Ac ( depend
ing on C and the data 'D) , for which minimizing the augmented error Eaug (w)
leads to the same final hypothesis wreg . A larger C allows larger weights and
is a weaker soft-order constraint; this corresponds to smaller ..\, i.e., less em
phasis on the penalty term wTw in the augmented error. For a particular
data set, the optimal value C* leading to minimum out-of-sample error with
the soft-order constraint corresponds to an optimal value ,\ * in the augmented
error minimization. If we can find ,\ * , we can get the minimum Eout.
Have we gained from the augmented error view? Yes, because augmented
error minimization is unconstrained, which is generally easier than constrained
minimization. For example, we can obtain a closed form solution for linear
models or use a method like stochastic gradient descent to carry out the mini
mization. However, augmented error minimization is not so easy to interpret.
There are no values for the weights which are explicitly forbidden, as there
are in the soft-order constraint. For a given C, the soft-order constraint cor
responds to selecting a hypothesis from the smaller set 1-l ( C) , and so from
our VC analysis we should expect better generalization when C decreases (..\
increases ) . It is through the relationship between ,\ and C that one has a
theoretical justification of weight decay as a method for regularization.
We focused on the soft-order constraint wTw :: C with corresponding
augmented error Eaug (w) Ein(w) + .AwTw. However, our discussion applies
more generally. There is a duality between the minimization of the in-sample
132

4 . OvERFITTIN G

4 . 2 . REGULARIZATION

error over a constrained hypothesis set and the unconstrained minimization of

an augmented error. We may choose to live in either world, but more often
than not, the unconstrained minimization of the augmented error is more
convenient.
In our definition of Eaug (w) in Equation (4.6) , we only highlighted the
dependence on w. There are two other quantities under our control, namely
the amount of regularization, .\, and the nature of the regularizer which we
chose to be wTw. In general, the augmented error for a hypothesis h E 1-l is
(4.7)
For weight decay, D(h) wTw, which penalizes large weights. The penalty
term has two components: the regularizer fJ(h) (the type of regularization)
which penalizes a particular property of h; and the regularization parameter ,\
(the amount of regularization) . The need for regularization goes down as the
number of data points goes up, so we factored out -ft ; this allows the optimal
choice for ,\ to be less sensitive to N. This is just a redefinition of the ,\ that
we have been using, in order to make it a more stable parameter that is easier
to interpret. Notice how Equation ( 4. 7) resembles the VC bound ( 4. 1 ) as we
anticipated in the heuristic view of regularization. This is why we use the same
notation n for both the penalty on individual hypotheses D(h) and the penalty
on the whole set 0(1-l ) . The correspondence between the complexity of 1-l and
the complexity of an individual h will be discussed further in Section 5 . 1 .
The regularizer fJ i s typically fixed ahead of time, before seeing the data;
sometimes the problem itself can dictate an appropriate regularizer.
Exercise 4.6
We h ave seen both the h a rd-order constraint a nd the soft-order const raint.
Which do you expect to be more useful for binary classification using the
perceptron model? [Hint: sign(wTx) sign(awTx) fo r any a > O.}

The optimal regularization parameter, however, typically depends on the data.

The choice of the optimal ,\ is one of the applications of validation, which we
will discuss shortly.
Example 4.2 . Linear models with weight decay. Linear models are
important enough that it is worthwhile to spell out the details of augmented
error minimization in this case. From Exercise 4.4, the augmented error is

where Z is the transformed data matrix and WHn (ZTz) - 1 ZTy. The reader
may verify, after taking the derivatives of Eaug and setting \7 wEaug 0, that

133

4. 0VERFITTING

4 . 2 . REGULARIZATION

As expected, Wreg will go to zero as ,\ --- oo, due to the ,\I term. The predic
tions on the in-sample data are given by y Zwreg H(,\)y, where
The matrix H(,\) plays an important role in defining the effective complexity
of a model. When ,\ 0, H is the hat matrix of Exercises 3.3 and 4.4, which
satisfies H2 H and trace(H) d + 1. The vector of in-sample errors, which
are also called residuals, is y - y (I H(,\))y, and the in-sample error Ei n
is Ein(Wreg) = :h YT (I H(,\) ) 2 y.
D
We can now apply weight decay regularization to the first overfitting example
that opened this chapter. The results for different A's are shown in Figure 4.5.
,\

,\

0.0001

0.01

?:

.?:

Figure 4.5: Weight decay applied to Example 4.2 with different values for
the regularization parameter ..\. The red fit gets flatter as we increase ..\ .

As you can see, even very little regularization goes a long way, but too much
regularization results in an overly flat curve at the expense of in-sample fit.
Another case we saw earlier is Example 4. 1 , where we fit a linear model to a
sinusoid. The regularization used there was also weight decay, with ,\ 0 . 1 .
4.2.3

Choosing a Regularizer: Pill or Poison?

We have presented a number of ways to constrain a model: hard-order con

straints where we simply use a lower-order model, soft-order constraints where
we constrain the parameters of the model, and augmented error where we add
a penalty term to an otherwise unconstrained minimization of error. Aug
mented error is the most popular form of regularization, for which we need to
choose the regularizer fl (h) and the regularization parameter ,\.
In practice, the choice of D, is largely heuristic. Finding a perfect fl is as
difficult as finding a perfect 1-l. It depends on information that, by the very
nature of learning, we don't have. However, there are regularizers we can work
with that have stood the test of time, such as weight decay. Some forms of
regularization work and some do not, depending on the specific application
and the data. Figure 4.5 illustrated that even the amount of regularization
134

4 . 0VERFITTING

4 . 2 . REGULARIZATION

0.84

0.84
;;

kf
'rj 0.8
<!.)

tl
<!.)

i:Ll0.76
0.5

1.5

Regularization Parameter,
( a) Uniform regularizer

0.5

,\

1.5

Regularization Parameter,
(b ) Low order regularizer

,\

Figure 4.6: Out of sample performance for the uniform and low order reg
ularizers using model H 1 5 , with o- 2 = 0.5, Q1 = 15 and N = 30. Overfitting
occurs in the shaded region because lower Ein ( lower A) leads to higher Eout .
Underfitting occurs when A is too large, because the learning algorithm has
too little flexibility to fit the data.

has to be chosen carefully. Too much regularization (too harsh a constraint )

leaves the learning too little flexibility to fit the data and leads to under.fitting ,
which can be just as bad as overfitting.
If so many choices can go wrong, why do we bother with regularization
in the first place? Regularization is a necessary evil, with the operative word
being necessary. If our model is too sophisticated for the amount of data
we have, we are doomed. By applying regularization, we have a chance. By
applying the proper regularization, we are in good shape. Let us experiment
with two choices of a regularizer for the model H15 of 15th order polynomials,
using the experimental design in Exercise 4.2:
1 . A uniform regularizer: f2unif( w)
2. A low-order regularizer: f210w(w)

L::o
L::o

w .

qw .

The first encourages all weights to be small, uniformly; the second pays more
attention to the higher order weights, encouraging a lower order fit. Figure 4.6
shows the performance for different values of the regularization parameter .:\ .
As you decrease .:\ , the optimization pays less attention to the penalty term and
more to Ein, and so Ein will decrease ( Problem 4.7) . In the shaded region, Eout
increases as you decrease Ein ( decrease ,:\) the regularization parameter is
too small and there is not enough of a constraint on the learning, leading
to decreased performance because of overfitting. In the unshaded region, the
regularization parameter is too large, over-constraining the learning and not
giving it enough flexibility to fit the data, leading to decreased performance
because of underfitting. As can be observed from the figure, the price paid for
overfitting is generally more severe than underfitting. It usually pays to be
conservative.
135

4 . 2 . REGULARIZATION

4 . 0VERFITTING

0.5

1.5

Regularization Parameter,
(a) Stochastic noise

0.5

,\

1.5

Regularization Parameter,
(b) D eterministic noise

,\

Figure 4. 7: Performance of the uniform regularizer at different levels of

noise. The optimal >. is highlighted for each curve.
The optimal regularization parameter for the two cases is quite different
and the performance can be quite sensitive to the choice of regularization
parameter. However, the promising message from the figure is that though
the behaviors are quite different, the performances of the two regularizers are
comparable (around 0 . 76) , if we choose the right ,,\ for each.
We can also use this experiment to study how performance with regular
ization depends on the noise. In Figure 4.7(a) , when a2 = 0, no amount
of regularization helps (i.e. , the optimal regularization parameter is ,,\ 0) ,
which is not a surprise because there is no stochastic or deterministic noise in
the data (both target and model are 15th order polynomials) . As we add more
stochastic noise, the overall performance degrades as expected. Note that the
optimal value for the regularization parameter increases with noise, which is
also expected based on the earlier discussion that the potential to overfit in
creases as the noise increases; hence, constraining the learning more should
help. Figure 4. 7(b) shows what happens when we add deterministic noise ,
keeping the stochastic noise at zero. This is accomplished by increasing Q f
(the target complexity) , thereby adding deterministic noise, but keeping ev
erything else the same. Comparing parts (a) and (b) of Figures 4.7 provides
another demonstration of how the effects of deterministic and stochastic noise
are similar. When either is present, it is helpful to regularize, and the more
noise there is, the larger the amount of regularization you need.
What happens if you pick the wrong
regularizer? To illustrate, we picked a
regularizer which encourages large weights
(weight growth) versus weight decay which
rij
Q)
encourages small weights. As you can see,
decay
t)
<J.)
in this case, weight growth does not help
the cause of overfitting. If we happened to
choose weight growth as our regularizer,
we would still be OK as long as we have
Regularization Parameter, ,\
136

4 . 0VERFITTING

4 . 3 . VALIDATION

a good way to pick the regularization parameter the optimal regularization

parameter in this case is ,\ = 0 , and we are no worse off than not regularizing.
No regularizer will be ideal for all settings, or even for a specific setting since
we never have perfect information, but they all tend to work with varying
success, if the amount of regularization ,\ is set to the correct level. Thus, the
entire burden rests on picking the right ,\, a task that can be addressed by a
technique called validation, which is the topic of the next section.
The lesson learned is that some form of regularization is necessary, as learn
ing is quite sensitive to stochastic and deterministic noise. The best way to
constrain the learning is in the 'direction' of the target function, and more
of a constraint is needed when there is more noise. Even though we don't
know either the target function or the noise, regularization helps by reducing
the impact of the noise. Most common models have hypothesis sets which are
naturally parameterized so that smaller parameters lead to smoother hypothe
ses. Thus, a weight decay type of regularizer constrains the learning towards
smoother hypotheses. This helps, because stochastic noise is 'high frequency'
(non-smooth) . Similarly, deterministic noise (the part of the target function
which cannot be modeled) also tends to be non-smooth. Thus, constraining
the learning towards smoother hypotheses 'hurts' our ability to overfit the
noise more than it hurts our ability to fit the useful information. These are
empirical observations, not theoretically justifiable statements.
Regularization and the VC dimension. Regularization (for example
soft-order selection by minimizing the augmented error) poses a problem for
the VC line of reasoning. As ,\ goes up, the learning algorithm changes but
the hypothesis set does not, so dvc will not change. We argued that ,\ t in
the augmented error corresponds to C .J, in the soft-order constrained model.
So, more regularization corresponds to an effectively smaller model, and we
expect better generalization for a small increase in Ein even though the VC
dimension of the model we are actually using with augmented error does not
change. This suggests a heuristic that works well in practice, which is to use an
'effective VC dimension' instead of the VC dimension. For linear perceptrons,
the VC dimension equals the number of free parameters d + 1, and so an effec
tive number of parameters is a good surrogate for the VC dimension in the VC
bound. The effective number of parameters will go down as ,\ increases, and
so the effective VC dimension will reflect better generalization with increased
regularization. Problems 4.13, 4. 14, and 4.15 explore the notion of an effective
number of parameters.

4.3

Validation

So far, we have identified overfitting as a problem, noise (stochastic and deter

ministic) as a cause, and regularization as a cure. In this section, we introduce
another cure, called validation. One can think of both regularization and val137

4 . 3 . VALIDATION

4 . 0VERFITTIN G

idation as attempts at minimizing Eout rather than just Ein. Of course the
true Eout is not available to us, so we need an estimate of Eout based on in
formation available to us in sample. In some sense, this is the Holy Grail of
machine learning: to find an in-sample estimate of the out-of-sample error.
Regularization attempts to minimize Eout by working through the equation

Eout (h)

Ein (h) + overfit penalty,

and concocting a heuristic term that emulates the penalty term. Validation,
on the other hand, cuts to the chase and estimates the out-of-sample error
directly.
Eout (h) Ein (h) + overfit penalty.
'Estimating the out-of-sample error directly is nothing new to us. In Sec
tion 2.2.3, we introduced the idea of a test set, a subset of V that is not
involved in the learning process and is used to evaluate the final hypothesis.
The test error Etest , unlike the in-sample error Ein, is an unbiased estimate
of Eout
4. 3 . 1

The Validation Set

The idea of a validation set is almost identical to that of a test set. V\Te
remove a subset from the data; this subset is not used in training. We then
use this held-out subset to estimate the out-of-sample error. The held-out set
is effectively out-of-sample, because it has not been used during the learning.
However, there is a difference between a validation set and a test set.
Although the validation set will not be directly used for training, it will be
used in making certain choices in the learning process. The minute a set affects
the learning process in any way, it is no longer a test set. However, as we will
see, the way the validation set is used in the learning process is so benign that
its estimate of Eout remains almost intact.
Let us first look at how the validation set is created. The first step is
to partition the data set V into a training set Dtrain of size (N K) and a
validation set Dval of size K. Any partitioning method which does not depend
on the values of the data points will do; for exan1ple, we can select N K
points at random for training and the remaining for validation.
Now, we run the learning algorithm using the training set Dtrain to obtain
a final hypothesis g E 1-l, where the 'minus' superscript indicates that some
data points were taken out of the training. We then compute the validation
error for g using the validation set Dval:
-

138

4. 3 . VALIDATION

4 . 0VERFITTIN G

where e (g (x) , y) is the pointwise error measure which we introduced in Sec

tion 1.4. 1 . For classification, e(g(x), y) [g (x) -/=- y] and for regression using
squared error, e(g(x) , y) (g (x) y) 2 .
The validation error is an unbiased estimate of Eout because the final hy
pothesis g was created independently of the data points in the validation set.
Indeed, taking the expectation of Eval with respect to the data points in 'Dval ,
1
K
1
K

Xn EVval

Xn EVval

(4.8)

The first step uses the linearity of expectation, and the second step follows
because e (g ( Xn) , Yn) depends only on Xn and so
lE vva l

[e (g (xn), Yn) ] lExn [e (g (xn), Yn) ] Eout (g ) .

How reliable is Eval at estimating Eout? In the case of classification, one can

use the VC bound to predict how good the validation error is as an estimate for
the out-of-sarn.ple error. We can view 'Dval as an 'in-sample' data set on which
we computed the error of the single hypothesis g . We can thus apply the
VC bound for a finite model with one hypothesis in it ( the Hoeffding bound ) .
With high probability,

Eout (g ) :=:; Eval (g ) + 0

(4.9)

While Inequality ( 4.9) applies to binary target functions, we may use the
variance of Eval as a more generally applicable measure of the reliability. The
next exercise studies how the variance of Eval depends on K ( the size of the
validation set ) , and implies that a similar bound holds for regression. The
conclusion is that the error between Eva1(g ) and Eout (g ) drops as CJ(g )/VK,
where O"(g ) is bounded by a constant in the case of classification.
Exercise 4. 7
Fix g ( learned from 'Dtrain) and define o-;al ,fVarvvai [Eva1 (g-)] . We con
sider how o-;al depends on K. Let

be the pointwise varia nce in the out-of-sam ple error of g .

(a) Show that o-;al f< o 2 (g ) .
( b) I n a classification problem, where e(g (x) , y)
[g (x) =J y] , express
o-;al in terms of JP>[g-(x) =J y] .
(c) S how that for a ny

g- in a classification problem , o-;al

s;

(continued on next page)

139

4 . 0VERFITTIN G

4 . 3 . VALIDATION

( d ) I s there a u niform u pper bound for Var[Eva1 (g )] sim i la r to ( c ) in

the case of regression with squared error e (g (x) , y) (g (x) - y) 2 ?
{Hint: The squared error is unbounded.]

( e ) For regression with sq uared error, if we train using fewer points

( smal ler N K) to get g , do you expect a2 (g-) to be h igher or

lower? {Hint: For continuous, non-negative random variables, higher

mean often implies higher variance.]

( f) Conclude that increasing the size of the validation set can result in a
better or a worse estimate of Eout

The expected validation error for 1-l 2 is illustrated in Figure 4.8, where we
used the experimental design in Exercise 4.2, with Qf = 10, N 40 and noise
level 0.4. The expected validation error equals Eout (g ) , per Equation (4.8) .

10

20

Size of Validation Set, K

Figure 4 .8: The expected validation error lE[Eva1 (g-)]

the shaded area is lE [Eval] aval

30
as

a function of K;

The figure clearly shows that there is a price to be paid for setting aside K
data points to get this unbiased estimate of Eout : when we set aside more
data for validation, there are fewer training data points and so g becomes
worse; Eout (g ) , and hence the expected validation error, increases (the blue
curve) . As we expect, the uncertainty in Eval as measured by aval (size of the
shaded region) is decreasing with K, up to the point where the variance a2 ( g )
gets really bad. This point comes when the number of training data points
becomes critically small, as in Exercise 4.7(e) . If K is neither too small nor
too large, Eval provides a good estimate of Eout . A rule of thumb in practice
is to set K = (set aside 203 of the data for validation) .
We have established two conflicting demands on K. It has to be big enough
for Eval to b e reliable, and it has to be small enough so that the training set
with N K points is big enough to get a decent g . Inequality ( 4. 9) quantifies
the first demand. The second demand is quantified by the learning curve
-

140

4.

4. 3 .

0VERFITTING

VALIDATION

discussed in Section 2.3.2 ( also the blue curve in Figure 4.8, from right to left ) ,
which shows how the expected out-of-sample error goes down as the number
of training data points goes up . The fact that more training data lead to a
better final hypothesis has been extensively verified empirically, although it is
challenging to prove theoretically.
Restoring V. Although the learning curve
suggests that taking out K data points for
validation and using only N K for train
ing will cost us in terms of Eout , we do not
have to pay that price! The purpose of vali
dation is to estimate the out-of-sample per
formance, and Eval happens to be a good
estimate of Eout (g ) . This does not mean
that we have to output g as our final hy
pothesis. The primary goal is to get the
best possible hypothesis, so we should out
put g, the hypothesis trained on the en
tire set V. The secondary goal is to esti
mate Eout, which is what validation allows
us to do. Based on our discussion of learn
ing curves, Eout (g) :: Eout (g ) , so
-

g-

Eval (g )

Figure 4.9: Using a valida

tion set to estimate

Eout (g ) Eout (g ) :: Eval(g ) + 0

Eout .

(4. 10)

The first inequality is subdued because it was not rigorously proved. If we first
train with N K data points, validate with the remaining K data points and
then retrain using all the data to get g, the validation error we got will likely
still be better at estimating Eout (g) than the estimate using the VG-bound
with Ein (g) , especially for large hypothesis sets with big dvc .
So far, we have treated the validation set as a way to estimate Eout, without
involving it in any decisions that affect the learning process. Estimating Eout
is a useful role by itself a customer would typically want to know how good
the final hypothesis is ( in fact, the inequalities in ( 4. 10) suggest that the
validation error is a pessimistic estimate of Eout , so your customer is likely to
be pleasantly surprised when he tries your system on new data) . However, as
we will see next , an important role of a validation set is in fact to guide the
learning process. That 's what distinguishes a validation set from a test set.
-

4 . 3 .2

Model Selection

By far, the most important use of validation is for model selection. This could
mean the choice between a linear model and a nonlinear model, the choice of
the order of polynomial in a model, the choice of the value of a regularization
141

4 . 3 . VALIDATION

4 . 0VERFITTIN G
0.8
H
0
H
H

'Cl

2
(.)
<l

0.7

0.6

0.5
5

25

15

Validation Set Size, I<

Figure 4.10: Optimistic bias of the validation error when using a validation
set for the model selected.
parameter, or any other choice that affects the learning process. In almost
every learning situation, there are some choices to be made and we need a
principled way of making these choices.
The leap is to realize that validation can be used to estimate the out-of
sample error for more than one model. Suppose we have ]\![ models 1-l1 , . . . , 1-lM .
Validation can be used to select one of these models. Use the training set Dtrai n
to learn a final hypothesis g;, for each model. Now evaluate each model on
the validation set to obtain the validation errors Ei ,
, EM , where

Em = Eva1(g);

The validation errors

= 1 , . . . , M.
estimate the out-of-sample error Eout (g;,) for each 1-lm .
m

Exercise 4.8
Is

Em

an u n biased estimate for the out of sam ple error Eaut (g)?

It is now a simple matter to select the model with lowest validation error.
Let m * be the index of the model which achieves the minimum validation
error. So for 1-lm* , Em* :: Em for m = 1 , . . . , J\I[ . The model 1-lm* is the model
selected based on the validation errors. Note that Em* is no longer an unbiased
estimate of Eout (g;,* ) . Since we selected the model with minimum validation
error, Em* will have an optimistic bias. This optimistic bias when selecting
between 1-l 2 and 1-l 5 is illustrated in Figure 4. 10, using the experimental design
described in Exercise 4.2 with Q f = 3, o- 2 = 0.4 and N = 35.
Exercise 4.9
Referri ng to Figu re 4. 10, why are both cu rves i ncreasing with K? Why do
they converge to each other with i ncreasin g K?

142

4 . 0VERFITTIN G

4 . 3 . VALIDATION

How good is the generalization error for this entire process of model selection
using validation? Consider a new model Hval consisting of the final hypotheses
learned from the training data using each model 1-{1 , . . . , HM:

{gi , g2 ,

Hval

' g} .

Model selection using the validation set chose one of the hypotheses in Hval
based on its performance on 'Dval . Since the model Hval was obtained before
ever looking at the data in the validation set, this process is entirely equivalent
to learning a hypothesis from H val using the data in 'Dval . The validation
errors Eval (g) are 'in-sample' errors for this learning process and so we may
apply the VC bound for finite hypothesis sets, with IHva1 I M:

Eout (g;,, )

<:'.

Eval (g;,,. ) + 0

( /) .

(4. 1 1 )

What i f we didn't use a validation set t o choose the model? One alternative
would be to use the in-sample errors from each model as the model selection
criterion. Specifically, pick the model which gives a final hypothesis with min
imum in-sample error. This is equivalent to picking the hypothesis with mini
mum in-sample error from the grand model which contains all the hypotheses
in each of the NI original models. If we want a bound on the out-of-sample
error for the final hypothesis that results from this selection, we need to apply
the VC-penalty for this grand hypothesis set which is the union of the !YI
hypothesis sets ( see Problem 2. 14) . Since this grand hypothesis set can have
a huge VC-dimension, the bound in ( 4. 1 1) will generally be tighter.
The goal of model selection is to se
lect the best model and output the best
Specifi
hypothesis from that model.
cally, we want to select the model m for
92
9N!
91
which Eout (gm) will be minimum when
we retrain with all the data. Model se
lection using a validation set relies on the
Ei E2 . . . EM i
leap of faith that if Eout (gm) is minimum,
pick the b est
then Eout (g) is also minimum. The val
(1-lm* , Em* )
idation errors Em estimate Eout (g) , so
modulo our leap of faith, the validation
set should pick the right model. No mat
9m*
ter which model m * is selected, however,
based on the discussion of learning curves
Figure 4. 1 1 : Using a validation
in the previous section, we should not out
set for model selection
put g * as the final hypothesis. Rather,
once m* is selected using validation, learn using all the data and output gm* ,
which satisfies

Eo ut ( gm' ) $ Eout (g;, ) <:'.

Eva! (g;,,, ) + 0

( /) .

Again, the first inequality is subdued because we didn't prove it.

143

(4. 1 2)

4 . 3 . VALIDATION

4 . 0VERFITTING

0.56

in sample:

gm,

validation:

9m*

0.48
5

15

Validation Set Size, K

25

Figure 4.12: Model selection between 1-l 2 and 1-l5 using a validation set. The

solid black line uses Ein for model selection, which always selects 1-l 5 The
dotted line shows the optimal model selection, if we could select the model
based on the true out of sample error. This is unachievable, but a useful
benchmark. The best performer is clearly the validation set, outputting 9m* .
For suitable K, even g * is better than in sample selection.

Continuing our experiment from Figure 4. 10, we evaluate the out-of-sample

performance when using a validation set to select between the models 1-l 2
and 1-l 5 . The results are shown in Figure 4. 12. Validation is a clear winner
over using Ein for model selection.
Exercise

4.10

(a) From Figure 4.12, lE[Eout (9 * )] is i n itial ly decreasing. How can this
for each m?
be, if IE.[Eout (g)] is i ncreasing i n
(b) From Figure 4.12 we see that IE.[Eout (9m* )] is i n itial ly decreasing, and
then it starts to increase. What are the possible reasons for this?
( c) When K = 1 , IE.[Eout ( 9* )) < lE [Eout (9m* )) . How can this be, if the
learning curves for both models are decreasing?

Example 4.3. We can use a validation set to select the value of the reg
ularization parameter in the augmented error of (4.6) . Although the most
important part of a model is the hypothesis set, every hypothesis set has an
associated learning algorithm which selects the final hypothesis g. Two mod
els may be different only in their learning algorithm, while working with the
same hypothesis set. Changing the value of ,\ in the augmented error changes
the learning algorithm (the criterion by which g is selected) and effectively
changes the model.
Based on this discussion, consider the Ji.If different models corresponding to
the same hypothesis set 1-l but with Ji.If different choices for ,\ in the augmented
error. So, we have (1-l, ,\1 ) , (1-l, A 2 ) , . . , (1-l, AM) as our Ji.If different models. We
.

144

4 . 0VERFITTING

4 . 3 . VALIDATION

may, for example, choose .\1 0, .\2 0.01, A 3 0.02, . . . , AM 10. Using a
validation set to choose one of these M models amounts to determining the
value of A to within a resolution of 0.01 .
D
We have analyzed validation for model selection based on a finite number of
models. If validation is used to choose the value of a parameter, for example A
as in the previous example, then the value of l'i1 will depend on the resolution
to which we determine that parameter. In the limit, the selection is actually
among an infinite number of models since the value of A can be any real
number. What happens to bounds like (4.11) and (4. 12) which depend on M?
Just as the Hoeffding bound for a finite hypothesis set did not collapse when
we moved to infinite hypothesis sets with finite VG-dimension, bounds like
(4.11) and (4.12) will not completely collapse either. We can derive VC-type
bounds here too, because even though there are an infinite number of models,
these models are all very similar; they differ only slightly in the value of .\ . As
a rule of thumb, what matters is the number of parameters we are trying to
set. If we have only one or a few parameters, the estimates based on a decent
sized validation set would be reliable. The more choices we make based on the
same validation set, the more ' contaminated' the validation set becomes and
the less reliable its estimates will be. The more we use the validation set to
fine tune the model, the more the validation set becomes like a training set
used to 'learn the right model'; and we all know how limited a training set is
in its ability to estimate Eout.
You will be hard pressed to find a serious learning problem in which valida
tion is not used. Validation is a conceptually simple technique, easy to apply
in almost any setting, and requires no specific knowledge about the details of
a model. The main drawback is the reduced size of the training set , but that
can be significantly mitigated through a modified version of validation which
we discuss next.
4.3.3

Cross Validation

Validation relies on the following chain of reasoning,

which highlights the dilemma we face in trying to select K. We are going to

output g. When K is large, there is a discrepancy between the two out-of
sample errors Eout(g ) (which Eval directly estimates ) and Eout (g) (which is
the final error when we learn using all the data 'D). We would like to choose K
as small as possible in order to minimize the discrepancy between Eout (g )
and Eout(g) ; ideally K
1 . However, if we make this choice, we lose the
reliability of the validation estimate as the bound on the RHS of ( 4.9) becomes
huge. The validation error Eval (g ) will still be an unbiased estimate of Eout (g )

145

4 . 0VERFITTING

4 . 3 . VALIDATION

(g is trained on N - 1 points) , but it will be so unreliable as to be useless

since it is based on only one data point. This brings us to the cross validation
estimate of out-of-sample error. We will focus on the leave-one-out version
which corresponds to a validation set of size K
1 , and is also the easiest
case to illustrate. More popular versions typically use larger K, but the essence
of the method is the same.
There are N ways to partition the data into a training set of size N - 1
and a validation set of size 1 . Specifically, let

be the data set V after leaving out data point (xn , Yn ) , which has been shaded
in red. Denote the final hypothesis learned from Vn by g. Let en be the error
made by g on its validation set which is j ust a single data point { ( xn , Yn) } :

The cross validation estimate is the average value of the en 's,

Figure 4.13: Illustration of leave one out cross validation for a linear
fit using three data points. The average of the three red errors
obtained by the linear fits leaving out one data point at a time is Ecv

Figure 4. 13 illustrates cross validation on a simple example. Each en is a

wild, yet unbiased estimate for the corresponding Eout (g), which follows after
setting K 1 in (4.8) . With cross validation, we have N functions g1 , . . . , g]v
together with the N error estimates e1 , . . . , eN . The hope is that these N
errors together would be almost equivalent to estimating Eout on a reliable
validation set of size N, while at the same time we managed to use N - 1
points to obtain each g. Let 's try to understand why Ecv is a good estimator
of Eout
146

4 . 3 . VALIDATION

4 . 0VERFITTING

First and foremost, Ecv is an unbiased estimator of 'Eout (g ) ' . We have

to be a little careful here because we don't have a single hypothesis g , as we
did when using a single validation set . Depending on the (xn , Yn ) that was
taken out, each g;, can be a different hypothesis. To understand the sense in
which Ecv estimates Eout , we need to revisit the concept of the learning curve.
Ideally, we would like to know Eout (g) . The final hypothesis g is the result
of learning on a random data set 'D of size N. It is almost as useful to know the
expected performance of your model when you learn on a data set of size N;
the hypothesis g is just one such instance of learning on a data set of size N.
This expected performance averaged over data sets of size N, when viewed
as a function of N, is exactly the learning curve shown in Figure 4.2. More
formally, for a given model, let
1E v [Eout (g) )

Bout ( N)

be the expectation (over data sets 'D of size N) of the out-of-sample error
produced by the model. The expected value of Ecv is exactly Eout (N 1) .
This is true because it is true for each individual validation error en :
-

1E v n 1E (xn ,Yn ) [e(g (xn ) , yn)] ,

1E vn [Eout (g ) ) ,

Eout (N

1).

Since this equality holds fo r each en , it also holds for the average. We highlight
this result by making it a theorem.
Theorem 4.4. Ecv is an unbiased estimate of Eout (N 1) (the expectation
of the model performance, JE [Eout J , over data sets of size N 1) .
-

Now that we have our cross validation

estimate of Eout , there is no need to out
put any of the g;, as our final hypothesis.
We might as well squeeze every last drop
of performance and retrain using the entire
g?,
g]_
data set 'D, outputting g as the final hy
(x1,
(x2, Y2)
pothesis and getting the benefit of going
from N 1 to N on the learning curve.
ei
e2
In this case, the cross validation estimate
will on average be an upper estimate for
the out-of-sample error: Eout (g) :S; Ecv ' so
g
expect to be pleasantly surprised, albeit
slightly.
Figure 4. 14: Using cross vali
dation to estimate Eout
With just simple validation and a val
idation set of size K
1 , we know that
the validation estimate will not be reliable. How reliable is the cross validation estimate Ecv? We can measure the reliability using the variance of Ecv

yi)i

147

4. 0VERFITTING

4 . 3 . VALIDATION

Unfortunately, while we were able to pin down the expectation of Ecv, the
variance is not so easy.
If the N cross validation errors e1 , . . . , eN were equivalent to N errors on a
totally separate validation set of size N, then Ecv would indeed be a reliable
estimate, for decent-sized N. The equivalence would hold if the individual en 's
were independent of each other. Of course, this is too optimistic. Consider
two validation errors en , em. The validation error en depends on g;, which was
trained on data containing (xm, Ym) Thus, en has a dependency on (xm , Ym)
The validation error em is computed using (xm, Ym) directly, and so it also
has a dependency on (Xm, Ym) . Consequently, there is a possible correlation
between en and em through the data point ( Xm, Ym) . That correlation wouldn't
be there if we were validating a single hypothesis using N fresh ( independent )
data points.
How much worse is the cross validation estimate as compared to an esti
mate based on a truly independent set of N validation errors? A VC-type
probabilistic bound, or even computation of the asymptotic variance of the
cross validation estimate (Problem 4.23) , is challenging. One way to quantify
the reliability of Ecv is to compute how many fresh validation data points
would have a comparable reliability to Ecv, and Problem 4.24 discusses one
way to do this. There are two extremes for this effective size. On the high end
is N, which means that the cross validation errors are essentially independent.
On the low end is 1 , which means that Ecv is only as good as any single one
of the individual cross validation errors en , i.e., the cross validation errors are
totally dependent. While one cannot prove anything theoretically, in practice
the reliability of Ecv is much closer to the higher end.

Effective number of fresh examples

giving a comparable estimate of Eout

Cross validation for model selection. In Figure 4. 1 1 , the estimates Em

for the out-of-sample error of model 1-lm were obtained using the validation set.
Instead, we may use cross validation estimates to obtain Em: use cross valida
tion to obtain estimates of the out-of-sample error for each model 1-l i , . . . , 1-l M ,
and select the model with the smallest cross validation error. Now, train this
model selected by cross validation using all the data to output a final hypoth
esis, making the usual leap of faith that Eout (g ) tracks Eout (g) well.
Example 4.5. In Figure 4. 13, we illustrated cross validation for estimat
ing Eout of a linear model ( h ( x)
ax + b) using a simple experiment with
three data points generated from a constant target function with noise. We
now consider a second model, the constant model (h(x)
b) . We can also
use cross validation to estimate Eout for the constant model, illustrated in
Figure 4. 15.

148

4 . 3 . VALIDATION

4 . 0VERFITTING

0
0

0
0

Figure 4 . 1 5 : Leave one-out cross validation error for a constant fit.

If we use the in-sample error after fitting all the data ( three points ) , then
the linear model wins because it can use its additional degree of freedom to
fit the data better. The same is true with the cross validation data sets of size
two - the linear model has perfect in-sample error. But, with cross validation,
what matters is the error on the outstanding point in each of these fits. Even
to the naked eye, the average of the cross validation errors is smaller for the
constant model which obtained Ecv 0.065 versus Ecv 0. 184 for the linear
model. The constant model wins, according to cross validation. The constant
model also has lower Eout and so cross validation selected the correct model
in this example.
D
One important use of validation is to estimate the optimal regularization
parameter A, as described in Example 4.3. We can use cross validation for the
same purpose as summarized in the algorithm below.
Cross validation for selecting A:
1 : Define .NI models by choosing different values for A in the
augmented error: (1-l , Ai ) , (1-l , A 2 ) , . . . , (1-l , A M )
2: for each model m
1 , . . . , ]\![ do
3:
Use the cross validation module in Figure 4. 14 to esti
mate Ecv(m) , the cross validation error for model m.
4: Select the model m * with minimum Ecv ( m * ) .
5: Use model (1-l , Am * ) and all the data V to obtain the fi
nal hypothesis gm* . Effectively, you have estimated the
optimal A .

We see from Figure 4 . 1 4 that estimating Ecv for just a single model requires N
rounds of learning on V1 , . . . , VN , each of size N 1 . So the cross validation
algorithm above requires MN rounds of learning. This is a formidable task.
If we could analytically obtain Ecv, that would be a big bonus, but analytic
results are often difficult to come by for cross validation. One exception is
in the case of linear models, where we are able to derive an exact analytic
formula for the cross validation estimate.
149

4 . 0VERFITTING

4 . 3 . VALIDATION

Analytic computation of Ecv for linear models. Recall that for linear
regression with weight decay, Wreg
(ZTZ + .\I) - 1 ZTy, and the in-sample
predictions are
y H(.\)y,
where H(.\) Z (ZTZ + .\I ) - 1 ZT . Given H, y, and y, it turns out that we can
analytically compute the cross validation estimate as:

Ecv

1
N

2
,... - yn
Yn
1 Hnn (A)

(4. 13 )

Notice that the cross validation estimate is very similar to the in-sample error,
Ein 1:J L, n (fJn - Yn ) 2 , differing only by a normalization of each term in the
sum by a factor 1 / ( 1 Hnn (.\) ) 2 . One use for this analytic formula is that it
can be directly optimized to obtain the best regularization parameter .\. A
proof of this remarkable formula is given in Problem 4.26.
Even when we cannot derive such an analytic characterization of cross
validation, the technique widely results in good out-of-sample error estimates
in practice, and so the computational burden is often worth enduring. Also,
as with using a validation set, cross validation applies in almost any setting
without requiring specific knowledge about the details of the models.
So far, we have lived in a world of unlimited computation, and all that
mattered was out-of-sample error; in reality, computation time can be of con
sequence, especially with huge data sets. For this reason, leave-one-out cross
validation may not be the method of choice. 4 A popular derivative of leave
one-out cross validation is V-fold cross validation. 5 In V-fold cross validation,
the data are partitioned into V disjoint sets ( or folds ) D 1 , . . . , Dv , each of size
approximately N/ V; each set Dv in this partition serves as a validation set to
compute a validation error for a hypothesis g learned on a training set which
is the complement of the validation set , D \ Dv . So, you always validate a
hypothesis on data that was not used for training that particular hypothesis.
The V-fold cross validation error is the average of the V validation errors that
are obtained, one from each validation set Dv . Leave-one-out cross validation
is the same as N-fold cross validation. The gain from choosing V N is
computational. The drawback is that you will be estimating Eout for a hy
pothesis g trained on less data ( as compared with leave-one-out ) and so the
discrepancy between Eout ( g ) and Eout ( g ) will be larger. A common choice in
practice is 10-fold cross validation, and one of the folds is illustrated below.
v

train

validate

train

4 Stability problems have also been reported in leave one out.

5 Some authors call it K fold cross validation, but we choose V so
the size of the validation set K.

150

as

not to confuse with

4 . 3 . VALIDATION

4. 0VERFITTING

4.3.4

Theory Versus Practice

Both validation and cross validation present challenges for the mathematical
theory of learning, similar to the challenges presented by regularization. The
theory of generalization, in particular the VC analysis, forms the foundation
for learnability. It provides us with guidelines under which it is possible to
make a generalization conclusion with high probability. It is not straightfor
ward, and sometimes not possible, to rigorously carry these conclusions over
to the analysis of validation, cross validation, or regularization. What is pos
sible, and indeed quite effective, is to use the theory as a guideline. In the
case of regularization, constraining the choice of a hypothesis leads to bet
ter generalization, as we would intuitively expect, even if the hypothesis set
remains technically the same. In the case of validation, making a choice for
few parameters does not overly contaminate the validation estimate of Eout ,
even if the VC guarantee for these estimates is too weak. In the case of cross
validation, the benefit of averaging several validation errors is observed, even
if the estimates are not independent.
Although these techniques were based on sound theoretical foundation,
they are to be considered heuristics because they do not have a full mathe
matical justification in the general case. Learning from data is an empirical
task with theoretical underpinnings. We prove what we can prove, but we use
the theory as a guideline when we don't have a conclusive proof. In a practical
application, heuristics may win over a rigorous approach that makes unrealis
tic assumptions. The only way to be convinced about what works and what
doesn't in a given situation is to try out the techniques and see for yourself.
The basic message in this chapter can be summarized as follows.

1 . Noise ( stochastic or deterministic ) affects learning

adversely, leading to overfitting.
2. Regularization helps to prevent overfitting by con
straining the model, reducing the impact of the noise,
while still giving us flexibility to fit the data.
3. Validation and cross validation are useful techniques
for estimating Eout . One important use of valida
tion is model selection, in particular to estimate the
amount of regularization to use.

Example 4.6. We illustrate validation on the handwritten digit classification

task of deciding whether a digit is 1 or not ( see also Example 3 . 1 ) based on the
two features which measure the symmetry and average intensity of the digit.
The data is shown in Figure 4.16 ( a) .

151

4 . 0VERFITTING

4 . 3 . VALIDATION
0.03

0.01

10

Average Intensity

15

20

# Features Used

( b ) Error curves

( a) Digits classification task

Figure 4.16: ( a) The digits data of which 500 are selected as the training
set. ( b ) The data are transformed via the 5th order polynomial transform
to a 20 dimensional feature vector. We show the performance curves as we
vary the number of these features used for classification.
We have randomly selected 500 data points as the training data and the
remaining are used as a test set for evaluation. We considered a nonlinear
feature transform to a 5th order polynomial feature space:
2 3
4
3 2 , X1X
( 1 , Xi , X2 ) -+ ( 1 , x 1 , X2 , x 21 , X 1 X2 , x 2 , X13 , X12 X 2 , . . . , x 51 , X14 X2 , X1X
, X 1 X , X5 ) .
2

Figure 4. 16 ( b) shows the in-sample error as you use more of the transformed
features, increasing the dimension from 1 to 20. As you add more dimensions
(increase the complexity of the model) , the in-sample error drops , as expected.
The out-of-sample error drops at first, and then starts to increase, as we hit
the approximation-generalization tradeoff. The leave-one-out cross validation
error tracks the behavior of the out-of-sample error quite well. If we were to
pick a model based on the in-sample error, we would use all 20 dimensions.
The cross validation error is minimized between 5-7 feature dimensions; we
take 6 feature dimensions as the model selected by cross validation. The table
below summarizes the resulting performance metrics:

Eout
No Validation
Cross Validation

0%
0.8 %

2 . 5%
1 .5 %

Cross validation results in a performance improvement of about 1 % , which is

a massive relative improvement (40 % reduction in error rate ) .
Exercise 4. 11
I n this particular experiment, the black curve (Ecv) is sometimes below a nd
sometimes above the the red cu rve (Eout) . If we repeated this experiment
m a ny times, and plotted the average black and red curves, wou ld you expect
the black curve to lie a bove or below the red cu rve?

152

4 . 0VERFITTIN G

4 . 3 . VALIDATION

It is illuminating to see the actual classification boundaries learned with and

without validation. These resulting classifiers, together with the 500 in-sample
data points, are shown in the next figure.

Average Intensity

Average Intensity

20 dim classifier (no validation)

Ei n = 0 3
Eout 2 . 53

6 dim classifier (LOO-CV)

Ei n = 0.8 3
Eout = 1 . 5 3

It is clear that the worse out-of-sample performance of the classifier picked

without validation is due to the overfitting of a few noisy points in the training
data. While the training data is perfectly separated, the shape of the resulting
boundary seems highly contorted, which is a symptom of overfitting. Does this
remind you of the first example that opened the chapter? There, albeit in a
toy example, we similarly obtained a highly contorted fit. As you can see,
D
overfitting is real, and here to stay!

153

4 . 0VERFITTING

4.4

4 . 4 . PROBLEMS

P roblems

Problem 4 . 1 P lot the monom ials of order i, </>i (x) = x i . As you increase
the order, does this correspond to the i ntuitive notion of i ncreasing complexity?
Problem 4.2
Consider the feature tra nsform z = [L0 (x) , L1 (x) , L 2 (xW
and the l i near model h(x) = wT z . For the hypothesis with w = [1 , - 1 , 1r,
what is h(x) expl icitly as a fu nction of x. What is its degree?

Problem 4.3
The Legendre Polynom ials are a fa mily of orthogonal
polynomia ls which are useful for regressio n . The first two Legendre Polynom ials
are Lo (x) = 1, L1 (x) = x. The h igher order Legendre Polynomials are defined
by the recursion :

Lk (x) =

k-1
- - ;;- Lk - 2 (x) .

2k - 1

( a ) What are the fi rst six Legendre Polynom ials? Use the recu rsion to de

velop an efficient a lgorith m to com pute Lo (x) , . . . , LK (x) given x. You r

a lgorithm should r u n i n time linear i n K. Plot the first six Legendre
polynomials.
( b ) Show that Lk (x) is a l inear com bination of monom ials x k , x k - 2 , . . . ( ei
ther all odd or a l l even order, with highest order k ) . Thus,

Lk(-x) = (- l ) k Lk(x) .

( c) S how that x2k- 1

= xLk (x) - Lk - 1 (x) . [Hint: use induction.}
( d ) Use part ( c ) to show that L k satisfies Legendre 's differential equation

!:_ 2
dLk (x)
= k(k + l )Lk (x) .
(x - 1)
dx
dx

This means that the Legendre Polynomials are eigenfu nctions of a Her
m itian l i near d ifferential operator a n d , from Stu rm Liouvil le theory, they
form an orthogonal basis for contin uous functions on [- 1, 1] .
( e) Use the recurrence to show d i rectly the orthogonal ity property:

dx Lk (x)L e (x) =

{O

2
2 k +l

e
g

= k,
k.

[Hint: use induction on k, with e ::; k. Use the recurrence for Lk and
consider separately the four cases e = k, k - 1, k - 2 and e < k - 2. For
the case e = k you will need to compute the integral J 1 dx x 2 LL 1 (x) .
In order to do this, you could use the differential equation in part (c),
multiply by xLk and then integrate both sides (the L HS can be integrated
by parts). Now solve the resulting equation for f 1 dx x 2 LL 1 (x) .j

154

4 . 4 . PROBLEMS

4 . 0VERFITTING

Problem 4.4
LAM i This problem is a detailed version of Exercise 4.2.
We set u p a n experimenta l framework wh ich the reader may use to study var
ious aspects of overfitting. The in put space is X = [- 1 , 1] . with un iform
in put proba bility density, P(x) = We consider the two models 1-l2 and
1-l10 . The target fu nction is a polynom ial of degree Qf , which we write as
f(x) = I:, !,0 aqLq (x) , where Lq (x) are the Legendre polynomials. We use
the Legendre polynomials beca use they are a convenient orthogonal basis for the
polynomials on [- 1 , 1] (see Section 4.2 a nd Problem 4.3 for some basic i nfor
mation on Legend re polynom ials). The data set is V = (x1 , y1 ) , . . . , ( x N , YN ) ,
where Yn = f (xn) + CJEn a nd En are iid standard Normal ra ndom variates.
For a single experiment, with specified values for QJ , N, CJ, generate a random
degree-Q f target fu nction by selecting coefficients aq independently from a
standard Normal , resca ling them so that IEa,x [f 2 ] = 1 . Generate a data set,
selecting x1 , . . . , X N independently from P(x) and Yn = f(xn) + CJEn . Let g2
a nd g10 be the best fit hypotheses to the data from 1-l2 a nd 7-l10 respectively,
with respective out of-sa m ple errors Eout (g2 ) and Eout (g10 ) .
( a ) Why d o we normalize j ? [Hint: how would you interpret CJ ?]
(b) How ca n we obtain g2 , g10? [Hint: pose the problem as linear regression

and use the technology from Chapter 3.}

( c) How ca n we com pute Eout ana lytical ly for a given g10 ?
( d) Vary Q f , N, CJ and for each com bination of para meters, ru n a large n um
ber of experi ments, each time com puting Eout (g2 ) a nd Eout (g10) . Aver
aging these out-of-sa m ple errors gives estimates of the expected out-of
sa mple error for the given learn ing scenario (QJ , N, CJ ) using 7-l 2 and 1-l10 .
Let
Eout (1-l2 )
Eout (1-l10 )

average over experiments(Eout (g2 ) ) ,

average over experiments( Eout (g10) ) .

Defi ne the overfit measu re Eout (1-l10) - Eout (1-l2 ) . When i s the over
fit measure significa ntly positive (i .e. , overfitting is serious) as opposed
to sign ifica ntly negative? Try the choices QJ E { 1 , 2, . . . , 100}, N E
{20, 25, . . . , 120} , CJ 2 E {O, 0.05, 0. 1 , . . . , 2}.
Explain you r observations.
( e) Why do we take the average over many experiments? Use the variance
to select a n acceptable n u m ber of experiments to average over.
(f) Repeat this experiment for classification , where the target fu nction is a
noisy perceptron , f = sign "L !, 1 aqLq(x) + E . Notice that ao = 0,

)
[ 0=!,1 aqLq (x)) 2 ]

= 1.
and the aq 's should be normalized so that IEa,x
For classification, the models H2 , H10 conta in the sign of the 2nd and
10th order polynom ials respectively. You may use a learning a lgorithm
for non-separa ble data from Chapter 3.

155

4 . 0VERFITTING

4 . 4 . PROBLEMS

If >. < 0 in the augmented error Eaug (w) = Ein (w ) +>.wTw,

what soft order constraint does this correspond to? [Hint: >. < 0 encourages

Problem 4.5
large weights.}

Problem 4.6

I n the a ugmented error m i n i mization with r = I a nd >. > 0:

(a) S how that llwreg ll llwHn ll . justifying the term weight decay. [Hint:

start by assuming that llwreg ll > llwHn ll and derive a contradiction.]

In fact a stronger statement holds: l l w reg ll is decreasing in >..

(b) Explicitly verify this for l inear models. [Hint:

where u = ZT y and Z is the transformed data matrix. Show that ZTZ +

>.I has the same eigenvectors with correspondingly larger eigenvalues as
ZTZ . Expand u in the eigenbasis of ZTZ. For a matrix A, how are the
eigenvectors and eigenvalues of A - 2 related to those of A ?]

Problem 4.7

S how that t h e i n-sa m ple error

from Exa mple 4.2 is an i ncreasing function of >., where H(>.) = Z (VZ+ >.r) - 1 V
a nd Z i s the transformed data matrix.
To do so, let the SVD of z = urvT and let ZTZ have eigenva l ues O'i ' . . . ' O' .
Define the vector a = UTy. Show that

a nd proceed from there.

Problem 4.8

I n the a ugmented error m i n i mization with r = I a nd >. > 0,

assume that Ein is d ifferentia ble and use gradient descent to m i n imize Eaug :

w (t + 1 )

+-

w(t) - r/\7 Eaug (w(t)).

Show that t h e update rule a bove is t h e sa me as

w(t + 1)

+-

( 1 - 2rJ >. ) w (t) - rJ\l Ein (w(t)).

Note: T h i s i s the origin o f t h e name 'weight decay ' : w (t) decays before being
u pdated by the gradient of Ein

156

4 . 4 . PROBLEMS

4. OvERFITTING

Problem 4.9

In Tikhonov regu larization , the regularized weights are given

by Wreg (ZTZ + >.rTr) - l ZTy. The Tikhonov regu larizer r is a k x (d + 1)
matrix, each row corresponding to a d + 1 dimensional vector. Each row of Z
corresponds to a d + 1 d i mensional vector ( the first component is 1 ) . For each
row of r, construct a virtual example ( z i , 0) for i = 1, . . , k, where Z i is the
vector obtai ned from the ith row of r after scal i ng it by V>,., and the target
value is 0. Add these k virtua l exam ples to the data , to construct a n a ugmented
data set, and consider non-regularized regression with this a ugmented data .
.

( a ) Show that, for the a ugmented data , Zaug

[.J. ]
r

and

Yaug

[]

( b ) Show that solving the least squares problem with Zaug a nd Yaug resu lts
i n the sa me regu larized weight

Wreg , i .e. Wreg

(Zug Zaug) 1 Z ugYaug

This resu lt may be i nterpreted as follows: a n equ iva lent way to accomplish
weight-decay-type regularization with linear models is to create a bunch of
virtual examples a l l of whose target val ues are zero.

Problem 4.10

I n this problem , you will investigate the relationship

between the soft order constrai nt and the augmented error. The regul arized
weight Wreg is a sol ution to

min Ein (w) subject to wTrTrw s C.

( a ) If Wun rTrwlin s c' then what is wreg?
( b ) If wli n rTrwlin > C, the situation is i l lustrated below,

The constraint is satisfied in the shaded region a n d the contours of con

sta nt Ein are the ellipsoids ( why ellipsoids? ) . What is weg rTrwreg?
( c ) S how that with

Weg \7Ein(Wreg) ,
Wreg minimizes Ein(w) + >.cwTrTrw. [Hint: use the previous part to
solve for Wreg as an equality constrained optimization problem using the
AC =

method of Lagrange multipliers.]

(continued on next page)

157

4 . 0VERFITTING

4 . 4 . P ROBLEMS

(d) Show that the fol lowing hold for

Ac :

(i) If wlin I'TI'W!in s; C then Ac = 0 (w!in itself satisfies the constra i nt ) .

(ii) If wli n I'TI'W!in > C, then Ac > 0 (the pena lty term i s positive) .
(iii) If wlin rTrwlin > C, then Ac is a strictly decreasing function of C.
< 0 for C E [O, wli n I'TI'Wiin] .}
[Hint: show that

Problem 4.11

For the linea r model in Exercise 4.2, the target function is

a polynomia l of degree Qf ; the model is 1-lQ , with polynomials up to order Q.
Assume Q 2: QJ . W!in = (ZTz) 1 ZTy, a nd y = Zwr + E, where wr is the
target fu nction and Z is the matrix contai n i ng the transformed data .
(a) Show that W!in = wr + (ZTz) 1 ZT E. What is the average fu nction g?
Show that bias = 0 ( reca l l that: bias(x) = (g(x) - f(x)) 2 ) .
( b) Show that

trace ( <I> lEz [( -Kr- ZTZ) 1 ]) ,

IE[<I>(x)<I>T (x)] . {Hints: var = IE[(g('D) - g) 2 ] ; first take the

expectation with respect to E, then with respect to <I>(x), the test point,
and the last remaining expectation will be with respect to Z. You will

where <I> =

need the cyclic property of the trace.}

(c) Argue that to first order in -Kr . var

l) .

{Hint: -Kr-VZ = -Kr l::= l <I>(x n ) <I>T (x n ) is the in-sample estimate of 'L,cp .
By the law of large numbers, -Kr-VZ = <P + o(l ) .}

For the well specified linear model, the bias is zero and the variance is increasing
as the model gets larger (Q increases), but decreasi ng in N .

Problem 4. 12

Use the setu p in P roblem 4 . 1 1 with Q 2: QJ . Con

sider regression with weight decay using a li near model 1-l i n the tra nsformed
space with i nput probabil ity d istribution such that IE[zzT] = I. The regu larized
weights a re given by Wreg = (ZTZ + AI) 1 Vy, where y = Zwr + E.

= Wf - A(ZTZ + AI) 1 wr + (ZTZ + >.I) 1 VE .

( b) Argue that, to fi rst order in -Kr ,
(a) Show that

where

Wre g

bias

var

A2
wr 2
(A + N )2 ll ll '
0" 2 IE [trace(H2 (A))],
N

H(A) = Z(VZ + AI) 1 V .

158

4 . 4. PROBLEMS

4. 0VERFITTIN G

If we plot the bias and var, we get a figure

that is very similar to Figu re 2.3, where
the tradeoff was based on fit a nd com
plexity rather than bias a nd var. Here, the
bias is increasing in .A ( as expected ) and
in llwfll ; the variance is decreasi ng in .A.
When .A = 0, trace(H2 (.A )) = Q + 1 and
so trace(H2 (.A )) appears to be playing the
role of an effective n u m ber of parameters.

Regularization Parameter, >.

Problem 4.13

Within the linear regression setting, m a ny attempts h ave

been made to q u a ntify the effective n u m ber of para meters in a model . . Three
possibilities are:

( i ) deff(.A) = 2trace(H(.A)) - trace(H 2 (.A ))

( ii ) deff(.A) = trace(H(.A ))
( iii ) deff(.A) = trace(H2 (.A ))
where H(.A ) = Z(VZ + .AI) 1 ZT a nd Z is the tra nsformed data matrix. To
obta in deff. one m ust first com pute H(.A) as though you a re doing regression .
One ca n then heuristica lly use deff in place of dvc in the VC bou nd .
( a ) When .A =

0, show that for a l l three choices,

dimension in the Z space.

deff = d + 1 , where d is the

( b ) When A >

0, show that 0 :: deff :: d + 1 a nd deff is decreasing in A for

a l l three choices. [Hint: Use the singular value decomposition.}

Problem 4.14
The observed target va lues y ca n be separated into the
true target values f and the noise E , y = f + E . The com ponents of E a re iid
with variance o- 2 and expectation 0. For linear regression with weight decay
regu larization, by ta king the expected va lue of the in sa m ple error in (4.2) ,
show that
1
0-2
2
2
N fT (I - H(.A )) f + N trace ( (I - H ( .A )) ) ,
e (I - H(.A )) 2 f + o- 2 -

(i ) ,

deff = 2trace(H(.A )) - trace(H2 (.A)) , as defi ned in Problem 4 . 13 ( i ) ,

H(.A ) = Z(ZTZ + .AI) 1 ZT and Z is the tra nsformed data matrix.

where

(continued on next page)

159

4 . 4 . PROBLEMS

4 . 0VERFITTING

0" 2 be, a nd

(a) If the noise was not overfit, what shou ld the term involving
why?

( b) Hence, argue that the degree to which the noise has been overfit is
0" 2 deff/N . I nterpret the dependence of this result on the para m eters deff
and N, to justify the use of deff as a n effective n u m ber of para meters.

We further i nvestigate deff of Problems 4 . 13 a nd 4. 14. We

know that H(..\) = Z(VZ + ..\rTr)- 1 ZT. When r is square a nd invertible, as
is usua l ly the case (for exa m ple with weight decay, r = I) , denote Z = zr- 1 .
Let s 5 , . . . , s be the eigenval ues of Z T Z ( s ; > 0 when Z has ful l col u m n ra n k) .

Problem 4.15

(a) For

deff(,\) = trace(2H(,\) - H 2 (..\) ) , show that

( b) For

deff(,\)

trace(H(..\)) , show that deff(,\) = d + 1 - I:

i=O

d
For deff(,\) - trace(H2 (..\) ) , show that deff(,\) - ?=

(c)

I n a l l cases, for ,\ 0, 0 :=: deff(,\) :=: d + l , deff(O) =

in ..\ . [Hint: use the singular value decomposition Z

orthogonal and S is diagonal with entries B i .]

Problem 4 . 16
with pena lty term

i=O

s[
(s'.? + >-)2

'

'

d + 1 and deff is decreasing

USVT, where U, V are

For linear models a nd the general Tikhonov regularizer r

wTrTrw in the a ugmented error, show that

l ZTy ,
Wre = (Z T Z + ..\rTr)
g

where

Z is the feature matrix.

(a) S how that the in-sa m ple predictions a re

y=

H(..\)y,

H(..\) = Z(VZ + .ArTr) 1 ZT.

( b) Simplify this in the case r = Z a n d obtai n Wreg in terms of Wlin This is
where

cal led u n iform weight decay.

Problem 4.17

To model uncertai nty in the measurement of the i nputs,

assume that the true inputs Xn are the observed inputs Xn perturbed by some
noise E n : the true inputs are given by Xn = Xn + En . Assume that the E n
a re independent of (xn , Yn ) with covariance matrix E [E n E] = O";r and mean

160

4 . 0VERFITTING

4 . 4 . PROBLEMS

lE[E n]

= 0. The learn ing a lgorithm m i n i m izes the expected in sample error Bin,
where the expectation is with respect to the uncertai nty in the true Xn .

Show that the weights W!in which result from m i n i m izing Bin a re equiva
lent to the weights which would have been obtained by m i n im izing Ein =
-f:t L;:=l (wTxn - Yn ) 2 for the observed data , with Tikhonov regu larization .
What a re r a nd >. (see Problem 4.16 for t h e general Tikhonov regularizer)?
One can i nterpret this result as follows: regularization enforces a robustness to
potentia l measurement errors (noise) in the observed in puts.

Problem 4 . 18

In a regression setting, assume the target function is

linear, so f(x) = w}x, and y = Zwf + E, where the entries in E are iid with
zero mea n a n d variance o-2 . Assume a regularization term ftwTZTZw a nd that
lE[xxT] = I. In t h is problem derive the optim a l va lue for >. as follows.
(a) S how that the average fu nction is
. II y
(b) S h ow t h at var .rs asymptot1ca

g(x) =

0"2 (d+ 1 )
N ( i + >. ) 2

What is the bias?

Problem 4. 12.J

(c) Use the bias a nd asymptotic varia nce to obtai n an expression for JE [Eout] .
O ptimize this with respect to >. to obtai n the optim a l regularization pa0" 2 ( d+ i )
nswer.. >. * N
rameter.
!l w i ll J
-

( d) Explain the dependence of the optimal regularization para meter on the

para meters of the learning proble m . {Hint: write >. * =

Problem 4 . 1 9

[The Lasso algorith m] Rather than a soft order constra i nt

on the squares of the weights, one cou ld use the a bsolute va lues of the weights:

min Ein(w)

subject to

L lwi l :S C.
i =O

The model is ca l led the lasso a lgorith m .

( a ) Formu late a nd implement this a s a q uadratic progra m . Use t h e exper
i mental design i n Problem 4.4 to compare the lasso a lgorith m with the
q ua d ratic penalty by giving plots of Eout versus regularization para m eter.
(b) What is the a ugmented error? Is it more convenient to optim ize?
(c) With d = 5 a nd N = 3, compare the weights from the lasso versus the
q uadratic penalty. [Hint: Look at the number of non-zero weights.]

161

4 . 4 . PROBLEivIS

4 . 0VERFITTING

Problem 4.20
In this problem, you will explore a consistency cond ition
for weight decay. Suppose that we m a ke an invertible linear tra nsform of the
data ,

Yn

ayn .

I ntu itively, l i near regression should not b e affected by a linear transform . This
means that the new optim a l weights should be given by a corresponding linear
tra nsform of the old opti m a l weights.

( a) Suppose w minimizes the in sa m ple error for the origin a l proble m . S how
that for the tra nsformed problem, the optimal weights are

( b) S u ppose the regularization pena lty term i n the augmented error is

wTXTXw for the origin a l data a nd wTZTZw for the transformed data .

On the original data , the regu larized solution is Wreg (.A) . Show that for
the tra nsformed problem , the same linear tra nsform of Wreg (.A) gives the
corresponding regularized weights for the transformed problem:

Problem 4.21
The Ti khonov smooth ness pena lty which penalizes
2
. Show that, for linear models,
derivatives of h is fJ(h) = J dx
this red uces to a pena lty of the form wTrTrw. What is r?

Problem 4.22

You have a data set with 100 data points. You have
100 models each with VC dimension 10. You set aside 25 points for validation .
You select the model which produced m i n i m u m validation error of 0.25. Give
a bound on the out of sa m ple error for this selected fu nction .
S u ppose you instead trained each model on a l l the data a nd selected the fu nc
tion with minimum in sa m ple error. The resulting in sa m ple error is 0 . 1 5 . Give
a bound on the out of sam ple error in this case. [Hint: Use the bound in

Problem 2. 14 to bound the VC dimension of the union of all the models.]

Problem 4.23

This problem investigates the covaria nce of the leave one

out cross val idation errors, Covv [en , em] . Assume that for wel l behaved models,
the learning process is 'sta ble' , and so the cha nge in the learned hypothesis
should be sma l l , 'O ( -f:t ) ', if a new data point is added to a data set of size
N. Write g;, = g<N 2 ) + On and g = g<N 2 ) + Om , where g<N 2 ) is the
learned hypothesis on vC N - 2 ) ' the data minus the nth a nd mth data points,
a nd On , Om are the corrections after addition of the nth a nd mth data points
respectively.

162

4 . 0VERFITTING

4 . 4 . PROBLEMS

(a) Show that Varv [Bev] =

2=:=l Varv [en] +

2=:#m Covv [en , e m ]

(b) Show Covv [en , em] = Varv (N 2) [Bout ( g (N-2) )]+ h igher order in 8n , Om .
( c) Assume that any terms involving On , Om are

O( tr ) . Argue that

Does Varv [e1 ] decay to zero with N? What a bout Varv [Bout ( g)] ?
(d) Use the experimenta l design in Problem 4.4 to study Varv [Bev] a nd give
a log log plot of Varv [Bev] /Varv [e1] versus N. What is the decay rate?

Problem 4.24

For d = 3, generate a ra ndom data set with N poi nts as

follows. For each point, each d imension of x has a sta ndard Norma l distributio n .
Similarly, generate a (d + 1) d i mensional target weight vector Wf, a n d set Yn =
w'f Xn + O"En where En is noise (also from a sta ndard Normal d istribution) and O"
is the noise variance; set O" to 0.5.

Use linear regression with weight decay regularization to estimate Wf with Wreg
Set the regu larization parameter to 0.05/N.
(a) For N E {d+ 15, d+25, . . . , d+ 1 15}, compute the cross val idation errors
ei , . . . , eN and Bev Repeat the experiment (say) 105 times, ma inta ining
the average a nd varia nce over the experiments of e1 , e2 and Bev
(b) How shou ld you r average of the e1 's relate to the average of the Bev 's;
how a bout to the average of the e2 's? Support you r claim using resu lts
from you r experiment.
(c) What a re the contributors to the variance of the e1 's?
( d) If the cross validation errors were tru ly i ndependent, how should the vari
a nce of the ei 's relate to the varia nce of the Bev 's?
( e) One measu re of the effective n u mber of fresh exa m ples used in com put
ing Bev is the ratio of the varia nce of the ei 's to that of the Bev's. Explain
why, a nd plot, versus N, the effective number of fresh exa m ples (Neff)
as a percentage of N. You should find that Neff is close to N.
(f) If you increase the amount of regu larization , wi ll Neff go u p or down?
Explain you r reasoning. Run the same experiment with A = 2.5/N and
com pa re you r resu lts from part ( e) to verify you r conjectu re.

Problem 4 . 25 When using a validation set for model selection, a l l models

were learned on the same Dtrain of size N K, and va l idated on the same
Dval of size K. We have the VC bound (see Eq uation ( 4 . 12) ) :
-

Eout ( g;, ) <: Eval ( g;, ) + 0

(J1f)

(continued on next page)

163

4 . 4 . PROBLEMS

4 . 0VERFITTIN G

Suppose that instead , you had no control over the validation process. So M
learners, each with their own models present you with the resu lts of their val
idation processes on different va lidation sets. Here is what you know a bout
each learner:
Each learner m reports to you the size of their va l idation set Km ,
a nd the val idation error Eva! ( m) . The learners may have used dif
ferent data sets, except that they fa ithfu l ly learned on a tra i ning set
and va l idated on a held out va lidation set which was only used for
va lidation pu rposes.
As the model selector, you have to decide which learner to go wit h .

( a ) Should you select t h e learner with m i n i m u m val idation error? If yes, why?
If no, why not? {Hint: think VC-bound.j

( b ) If a l l models a re validated on the same va lidation set as described in the

text, why is it okay to select the learner with the lowest validation error?

( c ) After selecting learner m* ( say ) , show that

JP> [Eout (m* )
where K,(E)
set size.

_L
E

2 2 ln

>

Eva1 (m* ) + E] :: Me - 2 2 "" (E ) ,

(.l

e 2 2 Km

111

( d ) Show that with proba bility at least 1 M/8 )

which satisfies E* ?: In(
2 K,(E* ) '

0,

is an "average" validation

Eout :: Eva! + E* , for a ny E*

( e ) S how that minm Km :: K,(E) ::

l:= l Km . Is this bound better or

worse than the bound when a l l models use the same val idation set size
( equal to the average va l idation set size -k l:= l Km ) ?

Problem 4.26
I n this problem , derive the formu l a for the exact expression
for the leave-one out cross va l idation error for linear regressio n . Let Z be the
data matrix whose rows correspond to the transformed data points Zn = <P(x n ) .

( a ) S how that:
N

ZTZ = L Zn z ;
n= l

ZTy = L Z nYn i
n=l

where A = A(,\)
ZTZ + ,\rTr a nd H(,\) = ZA(,\) 1 V . Hence,
show that when (z n , Yn ) is left out, ZTZ -+ ZTZ - ZnZ , a nd ZTy -+

ZTy - ZnYn

( b ) Com pute w , the weight vector learned when the nth data point is left
out, and show that:

l)
(A 1 + A1 -lZZnZA
(ZTy
A 1 Zn
Tn

164

ZnYn ) .

4 . 4 . PROBLEMS

4 . 0VERFITTIN G

use the identity ( A - xxT ) - 1 = A - 1 +

1
A - 1 xxT A 1 - xT A l x

(c) Using (a) a nd (b), show that w-; = w +

.]

A - 1 zn , where w is the

regression weight vector using all the data .

(d) The prediction on the va lid ation point is given by zw-; . S how that

Yn - HnnYn
ZnT Wn = 1 Hnn
-

{e) Show t hat en =

r,

a nd hence prove Equation (4. 13) .

Problem 4.27

Cross va lidation gives a n accurate estimate of Eout (N - 1) ,

but i t ca n b e q u ite sensitive, leadi ng t o problems i n model selectio n . A com mon
heu ristic for 'regu larizing' cross validation is to use a measure of error O-cv ( 1-l )
for the cross val idation estimate i n model selection .
(a) One choice for o-0v is the standard deviation of the leave-one-out errors
. . . ' en ) . Why divide by VN?
d ivided by Vi, 0-cv
(b) For linear models, show that VNo-cv = f.I 'L,;:= 1
( c)

E;v .

(i) Given the best model 1-l * , the conservative one-sigma a pproach se
lects the simplest model withi n O-cv (1-l * ) of the best.
(ii) The bound m i n im izing approach selects the model which m i n i m izes
Ecv(1-l ) + O-cv ( 1-l ) .
Use the experimental design in P roblem 4.4 to com pare these a pproaches
with the ' u n regu l arized ' cross validation estimate as fol lows. Fix Q1 = 15,
Q = 20, and o- = 1 . Use each of the two methods proposed here as wel l as
traditional cross va lidation to select the optimal value of the regularization
para meter >. i n the ra nge {0.05, 0.10, 0.15, . . . , 5} using weight decay
regularization , O(w) = ftwTw. P lot the resu lting out-of-sa m ple error
for the model selected using each method as a function of N, with N in
the ra nge {2 x Q , 3 x Q, . . . , 10 x Q} .
What a re you r concl usions?

165

166

Chapter

Three Learning Principles

The study of learning from data highlights some general principles that are
fascinating concepts in their own right. Having gone through the mathematical
analysis and empirical illustrations of the first few chapters, we have a good
foundation from which to articulate some of these principles and explain them
in concrete terms.
In this chapter, we will discuss three principles. The first one is related to
the choice of model and is called Occam's razor. The other two are related
to data; sampling bias establishes an important principle about obtaining the
data, and data snooping establishes an important principle about handling
the data. A genuine understanding of these principles will protect you from
the most common pitfalls in learning from data, and allow you to interpret
generalization performance properly.
5.1

Occam ' s Razor

Although it is not an exact quote of Einstein's, it is often attributed to him

that "An explanation of the data should be made as simple as possible, but no
simpler." A similar principle, Occam 's Razor, dates from the 14th century and
is attributed to William of Occam, where the 'razor' is meant to trim down
the explanation to the bare minimum that is consistent with the data.
In the context of learning, the penalty for model complexity which was
introduced in Section 2.2 is a manifestation of Occam's razor. If Ein (g) 0,
then the explanation ( hypothesis ) is consistent with the data. In this case,
the most plausible explanation, with the lowest estimate of Eout given in the
VC bound ( 2 . 14 ) , happens when the complexity of the explanation (measured
by dvc (H) ) is as small as possible. Here is a statement of the underlying
principle.
The simplest model that fits the data is also the most plausible.

167

5 . THREE LEARNING PRINCIPLES

5 . 1 . OCCAM 'S RAZOR

Applying this principle, we should choose as simple a model as we think we can

get away with. Although the principle that simpler is better may be intuitive,
it is neither precise nor self-evident . When we apply the principle to learning
from data, there are two basic questions to be asked.
1 . What does it mean for a model to be simple?
2. How do we know that simpler is better?
Let's start with the first question. There are two distinct approaches to defin
ing the notion of complexity, one based on a family of objects and the other
based on an individual object. We have already seen both approaches in our
analysis. The VC dimension in Chapter 2 is a measure of complexity, and it
is based on the hypothesis set 1-l as a whole, i.e. , based on a family of objects.
The regularization term of the augmented error in Chapter 4 is also a measure
of complexity, but in this case it is the complexity of an individual object,
namely the hypothesis h .
The two approaches t o defining complexity are not encountered only in
learning from data; they are a recurring theme whenever complexity is dis
cussed. For instance, in information theory, entropy is a measure of complexity
based on a family of objects, while minimum description length is a related
measure based on individual objects. There is a reason why this is a recurring
theme. The two approaches to defining complexity are in fact related.
When we say a family of objects is complex, we mean that the family is
'big'. That is, it contains a large variety of objects. Therefore, each individual
object in the family is one of many. By contrast, a simple family of objects is
'small' ; it has relatively few objects, and each individual object is one of few.
Why is the sheer number of objects an indication of the level of complexity?
The reason is that both the number of objects in a family and the complexity
of an object are related to how many parameters are needed to specify the
object. When you increase the number of parameters in a learning model, you
simultaneously increase how diverse 1-l is and how complex the individual h is.
For example, consider 17th order polynomials versus 3rd order polynomials.
There is more variety in 17th order polynomials, and at the same time the
individual 1 7th order polynomial is more complex than a 3rd order polynomial.
The most common definitions of object complexity are based on the number
of bits needed to describe an object. Under such definitions, an object is simple
if it has a short description. Therefore, a simple object is not only intrinsically
simple (as it can be described succinctly) , but it also has to be one of few,
since there are fewer objects that have short descriptions than there are that
have long descriptions, as a matter of simple counting.
Exercise 5 . 1
Consider hypothesis sets H1 a n d H100 that contai n Boolean functions on 10
10
Boolean varia b les, so X = {-1, + 1 } . H 1 contai ns a l l Boolea n functions

168

5 . THREE LEARNING PRINCIPLES

which eval uate to

5 . 1 . O CCAM 'S RAZOR

on exactly one i n put point, a nd to -1 elsewhere;

1 on exactly 100
1 elsewhere.

1-l100 contains a l l Boolean functions which eval uate to

i n put points, a nd to

( a ) How big ( n umber of hypotheses) are 1-l 1 a nd

1-l 1 00?

(b) How many bits are needed to specify one of the hypotheses in 1-l 1 ?
( c) How many bits are needed to specify one of the hypotheses in 1-l1 00?

We now address the second question. When Occam's razor says that simpler
is better, it doesn't mean simpler is more elegant. It means simpler has a
better chance of being right . Occam's razor is about performance, not about
aesthetics. If a complex explanation of the data performs better, we will
take it.
The argument that simpler has a better chance of being right goes as fol
lows. We are trying to fit a hypothesis to our data 'D = { (x 1 , Y1 ) ,
, (xN , YN ) }
(assume Yn 's are binary) . There are fewer simple hypotheses than there are
complex ones. With complex hypotheses, there would be enough of them to
shatter x 1 ,
, XN , so it is certain that we can fit the data set regardless of
what the labels Y1 ,
, YN are, even if these are completely random. There
fore, fitting the data does not mean much. If, instead, we have a simple model
with few hypotheses and we still found one that perfectly fits the dichotomy
'D = { (x i , Y1 ) , , (xN , YN ) } , this is surprising, and therefore it means some
thing.
Occam's Razor has been formally proved under different sets of idealized
conditions. The above argument captures the essence of these proofs; if some
thing is less likely to happen, then when it does happen it is more significant .
Let us look at an example.

Example 5 . 1 . Suppose that one constructs a physical theory about the re

sistivity of a metal under various temperatures. In this theory, aside from
some constants that need to be determined, the resistivity p has a linear de
pendence on the temperature T. In order to verify that the theory is correct
and to obtain the unknown constants, 3 scientists conduct the following three
experiments and present their data to you.

]..

.
rn
Q)
r-;

/
temperature T

Scientist 1

:..f
.
rn
Q)
r-;

/
temperature T

Scientist 2

169

..

:
..

.
rn
Q)
r-;

/
temperature T

Scientist 3

5 . 1 . OCCAM 'S RAZOR

5 . THREE LEARNING PRINCIPLES

It is clear that Scientist 3 has produced the most convincing evidence for the
theory. If the measurements are exact, then, Scientist 2 has managed to falsify
the theory and we are back to the drawing board. What about Scientist 1?
While he has not falsified the theory, has he provided any evidence for it? The
answer is no, for we can reverse the question. Suppose that the theory was not
correct, what could the data have done to prove him wrong? Nothing, since
any two points can be joined by a line. Therefore, the model is not just likely
to fit the data in this case, it is certain to do so. This renders the fit totally
D
insignificant when it does happen.
This example illustrates a concept related to Occam's Razor, which is the
axiom of non-falsifiability. The axiom asserts that the data should have some
chance of falsifying a hypothesis, if we are to conclude that it can provide
evidence for the hypothesis. One way to guarantee that every data set has
some chance at falsification is for the VC dimension of the hypothesis set
to be less than N, the number of data points. This is discussed further in
Problem 5. 1 . Here is another example of the same concept.
Example 5 . 2 . Financial firms try to pick good traders (predictors of whether
the market will go up or not) . Suppose that each trader is tested on their
prediction (up or down) over the next 5 days and those who perform well will
be hired. One might think that this process should produce better and better
traders on Wall Street. Viewed as a learning problem, consider each trader
to be a prediction hypothesis. Suppose that the hiring pool is 'complex' ; we
are interviewing 2 5 traders who happen to be a diverse set of people such that
their predictions over the next 5 days are all different. Necessarily one of these
traders gets it all correct, and will be hired. Hiring the trader through this
process may or may not be a good thing, since the process will pick someone
even if the traders are just flipping coins to make their predictions. A perfect
predictor always exists in this group, so finding one doesn't mean much. If we
were interviewing only two traders, and one of them made perfect predictions,
that would mean something.
D
Exercise 5.2
Suppose that fo r 5 weeks i n a row, a letter arrives in t h e mail that predicts
the outcome of the upcomi ng Monday night footbal l game. You keen ly
watch each Monday a nd to you r surprise, the prediction is correct each
time. On the day after the fifth game, a letter a rrives, stating that if you
wish to see next week's prediction , a payment of $50.00 is requ i red . Should
you pay?

( a ) How many possible predictions of win-lose a re there for 5 ga mes?

( b ) If the sender wants to make sure that at l east one person receives
correct predictions on a l l 5 games from h i m , how many people should
he target to begin with ?

170

5 . THREE LEARNING PRINCIPLES

5 . 2 . SAMPLING BIAS

(c) After the first letter ' predicting' the outcome of the first game, how
many of the origin a l reci pients does he target with the second letter?

( d) H ow m a ny letters a ltogether wil l have been sent at the end of the 5

weeks?

( e) If the cost of printing and m a i l ing out each letter is $0.50, how m uch
wou ld the sender make if the recipient of 5 correct predictions sent in
the $50.00?
(f) Can you relate this situation to the growth function a nd the credibility
of fitting the data?

Learning from data takes Occam's Razor to another level, going beyond "as
simple as possible, but no simpler." Indeed, we may opt for 'a simpler fit
than possible' , namely an imperfect fit of the data using a simple model over
a perfect fit using a more complex one. The reason is that the price we pay
for a perfect fit in terms of the penalty for model complexity in (2. 14) may
be too much in comparison to the benefit of the better fit . This idea was
illustrated in Figure 3. 7, and is a manifestation of overfitting. The idea is also
the rationale behind the recommended policy in Chapter 3 : first try a linear
model one of the simplest models in the arena of learning from data.
5.2

Sampling Bias

A vivid example of sampling bias happened in the 1948 US presidential election

between Truman and Dewey. On election night, a major newspaper carried
out a telephone poll to ask people how they voted. The poll indicated that
Dewey won, and the paper was so confident about the small error bar in its
poll that it declared Dewey the winner in its headline. When the actual votes
were counted, Dewey lost to the delight of a smiling Truman.

@Associated Press

171

5 . THREE LEARNING PRINCIPLES

5 . 2 . SAMPLING BIAS

This was not a case of statistical anomaly, where the newspaper was just
incredibly unlucky ( remember the 8 in the VC bound? ) . It was a case where
the sample was doomed from the get-go, regardless of its size. Even if the
experiment were repeated, the result would be the same. In 1948, telephones
were expensive and those who had them tended to be in an elite group that
favored Dewey much more than the average voter did. Since the newspaper did
its poll by telephone, it inadvertently used an in-sample distribution that was
different from the out-of-sample distribution. That is what sampling bias is.
If the data is sampled in a biased way, learning will pro
duce a similarly biased outcome.

Applying this principle, we should make sure that the training and testing
distributions are the same; if not, our results may be invalid, or, at the very
least, require careful interpretation.
If you recall, the VC analysis made very few assumptions, but one as
sumption it did make was that the data set V is generated from the same
distribution that the final hypothesis g is tested on. In practice, we may en
counter data sets that were not generated under those ideal conditions. There
are some techniques in statistics and in learning to compensate for the 'mis
match' between training and testing, but not in cases where V was generated
with the exclusion of certain parts of the input space, such as the exclusion of
households with no telephones in the above example. There is nothing that
can be done when this happens, other than to admit that the result will not
be reliable statistical bounds like Hoeffding and VC require a match between
the training and testing distributions.
There are many examples of how sampling bias can be introduced in data
collection. In some cases it is inadvertently introduced by an oversight, as
in the case of Dewey and Truman. In other cases, it is introduced because
certain types of data are not available. For instance, in our credit example of
Chapter 1 , the bank created the training set from the database of previous cus
tomers and how they performed for the bank. Such a set necessarily excludes
those who applied to the bank for credit cards and were rejected, because the
bank does not have data on how they would have perfarmed if they were ac
cepted. Since future applicants will come from a mixed population including
some who would have been rejected in the past, the 'test set' comes from a
different distribution than the training set, and we have a case of sampling
bias. In this particular case, if no data on the applicants that were rejected is
available, nothing much can be done other than to acknowledge that there is
a bias in the final predictor that learning will produce, since a representative
training set is just not available.
Exercise 5.3
I n a n experiment t o determine t h e d istribution o f sizes o f fish i n a l a ke, a
net m ight be used to catch a representative sam ple of fish . The sam ple is

172

5 . THREE LEARNING PRINCIPLES

5 . 3 . DATA SNOOPING

then a n alyzed to find out the fractions of fish of different sizes . I f the
sample is big enough , statistica l conclusions m ay be d rawn a bout the a ctua l
d istribution i n t h e entire lake. Can you s m e l l sampling bias?

There are other cases, arguably more common, where sampling bias is intro
duced by human intervention. It is not that uncommon for someone to throw
away training examples they don't like! A Wall Street firm who wants to de
velop an automated trading system might choose data sets when the market
was 'behaving well' to train the system, with the semi-legitimate justification
that they don't want the noise to complicate the training process. They will
surely achieve that if they get rid of the 'bad' examples, but they will create a
system that can be trusted only in the periods when the market does behave
well! What happens when the market is not behaving well is anybody's guess.
In general, throwing away training examples based on their values , e.g. , ex
amples that look like outliers or don't conform to our preconceived ideas, is a
fairly common sampling bias trap.
Other biases. Sampling bias has also been called selection bias in the statis
tics community. We will stick with the more descriptive term sampling bias
for two reasons. First, the bias arises in how the data was sampled; second, it
is less ambiguous because in the learning context, there is another notion of
selection bias drifting around selection of a final hypothesis from the learning
model based on the data. The performance of the selected hypothesis on the
data is optimistically biased, and this could be denoted as a selection bias.
We have referred to this type of bias simply as bad generalization.
There are various other biases that have similar flavor. There is even
a special type of bias for the research community, called publication bias!
This refers to the bias in published scientific results because negative results
are often not published in the literature, whereas positive results are. The
common theme of all of these biases is that they render the standard statistical
conclusions invalid because the basic premise for such conclusions, that the
sampling distribution is the same as the overall distribution, does not hold
any more. In the field of learning from data, it is sampling bias in the training
set that we need to worry about.

5.3

D at a S no oping

Data snooping is the most common trap for practitioners in learning from
data. The principle involved is simple enough,
If a data set has affected any step in the learning process,
its ability to assess the outcome has been compromised.

173

5 . 3 . DATA S NOOPING

5 . THREE LEARNING PRINCIPLES

Applying this principle, if you want an unbiased assessment of your learning

performance, you should keep a test set in a vault and never use it for learning
in any way. This is basically what we have been talking about all along in
training versus testing, but it goes beyond that. Even if a data set has not been
'physically' used for training, it can still affect the learning process, sometimes
in subtle ways.

Exercise 5.4
Consider the following a pproach to learning . By looking at the data , it
a ppea rs that the data is linea rly separa ble, so we go a head and use a sim ple
perceptron, a nd get a training error of zero after determi n ing the optim a l
set o f weights . We now wish t o m a ke some generalization conclusions, so
we look u p the dvc for our learning model a nd see that it is d+ 1 . Therefore,
we use this va l ue of dvc to get a bound on the test error .
( a ) What is the problem with this bound is it correct?

( b)

Do we know the dvc for the learning model that we actually used? It
is this dvc that we need to use in the boun d .

To avoid the pitfall in the above exercise, it is extremely important that you
choose your learning model before seeing any of the data. The choice can be
based on general information about the learning problem, such as the num
ber of data points and prior knowledge regarding the input space and target
function, but not on the actual data set V. Failure to observe this rule will
invalidate the VC bounds, and any generalization conclusions will be up in the
air. Even a careful person can fall into the traps of data snooping. Consider
the following example.
Example 5.3. An investment bank wants to develop a system for forecasting
currency exchange rates. It has 8 years worth of historical data on the US
Dollar ( USD ) versus the British Pound ( GBP ) , so it tries to use the data to see
if there is any pattern that can be exploited. The bank takes the series of daily
changes in the USD / GBP rate, normalizes it to zero mean and unit variance,
and starts to develop a system for forecasting the direction of the change. For
each day, it tries to predict that direction based on the fluctuations in the
previous 20 days. 753 of the data is used for training, and the remaining 253
is set aside for testing the final hypothesis.
The test shows great success. The final hypothesis has a hit rate (per
centage of time getting the direction right ) of 52. 13 . This may seem modest,
but in the world of finance you can make a lot of money if you get that
hit rate consistently. Indeed, over the 500 test days (2 years worth, as each
year has about 250 trading days ) , the cumulative profit of the system is a
respectable 223.

174

5 . THREE LEARNING PRINCIPLES

100

5 . 3 . DATA SNOOPING

200

Day

300

400

500

When the system is used in live trading, the performance deteriorates sig
nificantly. In fact, it loses money. Why didn't the good test performance
continue on the new data? In this case, there is a simple explanation and it
has to do with data snooping. Although the bank was careful to set aside
test points that were not used for training in order to properly evaluate the
final hypothesis, the test data had in fact affected the training process in a
subtle way. When the original series of daily changes was normalized to zero
mean and unit variance, all of the data was involved in this step. Therefore,
the test data that was extracted had already contributed to the choices made
by the learning algorithm by contributing to the values of the mean and the
variance that were used in normalization. Although this seems like a minor
effect, it is data snooping. When you plot the cumulative profit on the test
set with or without that snooping step, you see how snooping resulted in an
over-optimistic expectation compared to the realistic expectation that avoids
snooping.
It is not the normalization that was a bad idea. It is the involvement of
test data in that normalization, which contaminated this data and rendered
D
its estimate of the final performance inaccurate.
One of the most common occurrences of data snooping is the reuse of the
same data set . If you try learning using first one model and then another and
then another on the same data set, you will eventually 'succeed' . As the saying
goes, if you torture the data long enough, it will confess . If you try all
possible dichotomies, you will eventually fit any data set; this is true whether
we try the dichotomies directly ( using a single model) or indirectly (using a
sequence of models ) . The effective VC dimension for the series of trials will
not be that of the last model that succeeded, but of the entire union of models
that could have been used depending on the outcomes of different trials.
Sometimes the reuse of the same data set is carried out by different people.
Let's say that there is a public data set that you would like to work on. Before
you download the data, you read about how other people did with this data set
1 75

5 . THREE LEARNING PRINCIPLES

5 . 3 . DATA SNOOPING

using different techniques. You naturally pick the most promising techniques
as a baseline, then try to improve on them and introduce your own ideas.
Although you haven't even seen the data set yet, you are already guilty of
data snooping. Your choice of baseline techniques was affected by the data
set, through the actions of others. You may find that your estimates of the
performance will turn out to be too optimistic, since the techniques you are
using have already proven well-suited to this particular data set.
To quantify the damage done by data snooping, one has to assess the
penalty for model complexity in (2. 14) taking the snooping into consideration.
In the public data set case, the effective VC dimension corresponds to a much
bigger hypothesis set than the 1-l that your learning algorithm uses. It covers
all hypotheses that were considered ( and mostly rejected) by everybody else
in the process of coming up with the solutions that they published and that
you used as your baseline. This is a potentially huge set with very high VC
dimension, hence the generalization guarantees in (2. 14) will be much worse
than without data snooping.
Not all data sets subjected to data snooping are equally 'contaminated'.
The bounds in ( 1 . 6) in the case of a choice between a finite number of hy
potheses, and in (2. 12) in the case of an infinite number, provide guidelines
for the level of contamination. The more elaborate the choice made based on
a data set, the more contaminated the set becomes and the less reliable it will
be in gauging the performance of the final hypothesis.
Exercise 5 . 5
Assume w e set aside 100 examples from that wil l not be used i n tra i n i ng,
but wil l be used to select one of three fin a l hypotheses 91 , 92 , 93 produced by
three d ifferent lea rn i ng a lgorithms that train on the rest on the data . Each
a lgorithm works with a different of size 500. We wou ld l i ke to characterize
the a ccuracy of estimating Eout (9) on the selected fin a l hypothesis if we
use the same 100 examples to m a ke that estimate.

( a ) What is the val u e of

that should be used in (1.6) i n this situation?

(b) How does the level of contam ination of these 100 exam ples compare
to the case where they would be used i n t raining rather tha n i n the
fina l selection?

In order to deal with data snooping, there are basically two approaches.
1 . Avoid data snooping: A strict discipline in handling the data is required.
Data that is going to be used to evaluate the final performance should
be 'locked in a safe' and only brought out after the final hypothesis has
been decided. If intermediate tests are needed, separate data sets should
be used for that. Once a data set has been used, it should be treated as
contaminated as far as testing the performance is concerned.
2. Account for data snooping: If you have to use a data set more than
once, keep track of the level of contamination and treat the reliability of
176

5 . THREE LEARNING PRINCIPLES

5 . 3 . DATA SNOOPING

your performance estimates in light of this contamination. The bounds

(1 .6) and (2 . 12) can provide guidelines for the relative reliability of dif
ferent data sets that have been used in different roles within the learning
process.
Data snooping versus sampling bias. Sampling bias was defined based
on how the data was obtained before any learning; data snooping was defined
based on how the data affected the learning, in particular how the learning
model is selected. These are obviously different concepts. However, there are
cases where sampling bias occurs as a consequence of 'snooping' looking at
data that you are not supposed to look at. Here is an example.
Consider predicting the performance of different stocks based on historical
data. In order to see if a prediction rule is any good, you take all currently
traded companies and test the rule on their stock data over the past 50 years.
Let us say that you are testing the "buy and hold" strategy, where you would
have bought the stock 50 years ago and kept it until now. If you test this
'hypothesis' , you will get excellent performance in terms of profit . Well, don't
get too excited! You inadvertently biased the results in your favor by picking
only currently traded companies, which means that the companies that did
not make it are not part of your evaluation. When you put your prediction
rule to work, it will be used on all companies whether they will survive or
not, since you cannot identify which companies today will be the 'currently
traded' companies 50 years from now. This is a typical case of sampling bias,
since the problem is that the training data is not representative of the test
data. However, if we trace the origin of the bias, we did 'snoop' in this case by
looking at future data of companies to determine which of these companies to
use in our training. Since we are using information in training that we would
not have access to in real trading, this is viewed as a form of data snooping.

177

5 . 4. PROBLE.MS

5 . THREE LEARNING PRINCIPLES

5.4

P roblems

Problem 5 . 1

The idea of falsifiability - that a claim ca n be rendered

false by observed data - is a n i mporta nt principle i n experimenta l science.
Axiom of Non-Falsifiability.

If the outcome of an experiment

has no chance of falsifying a particular proposition, then the result

of that experiment does not provide evidence one way or another
toward the truth of the proposition.

Consider the proposition 'There is h E 1-l that a pproximates f as wou l d be

evidenced by finding such an h with i n sa m ple error zero on x1 ,
, XN ." We
say that the proposition is falsified if no hypothesis i n 1-l ca n fit the data
perfectly.

( a ) S uppose that 1-l shatters x 1 ,

falsifiable for any

f.

, XN .

Show that this proposition is not

( b ) S u ppose that f i s random ( f ( x) = 1 with proba bility . independently

on every x ) , so Eout ( h ) = for every h E 1-l. Show that
IfD [fa lsification ] 1

= 10 a nd N = 100. If you obtai n a hypothesis h with zero

Ein on you r data , what can you 'conclude' from the result in part ( b ) ?

( c) S u p pose dvc

Problem 5.2
Structura l Risk M i nimization ( S RM ) i s a usefu l framework
for model selection that is related to Occam 's Razor. Define a structure a
nested sequence of hypothesis sets:
-

The S R M framework picks a hypothesis from each 1-li by m m 1m1zmg Ein

That is, 9i = argmi n Ein ( h ) . Then, the framework selects the fin a l hy
h E rl i

pothesis by m i n imizing Ein and the model com plexity penalty n . That is,
g * = argmin ( Ein (9i ) + D ( 1-li )) . Note that D (1-li ) shou ld be non decreasing i n i
i=l,2, .
beca use o f t h e nested structu re.

( a ) Show that the i n sample error Ein ( 9i ) is non i ncreasing i n

178

i.

5 . THREE LEARNING PRINCIPLES

5 . 4. PROBLEMS

( b ) Assume that the framework finds g* E 1-li with proba bi I ity Pi . How does
Pi relate to the com plexity of the target fu nction?

( c ) Argue that the Pi 1S a re u nknown but po :S p 1 :S p2 :S

( d ) S u ppose g*

9i S how that

IP [I Ein (9i) - Eout (gi ) I > E I g*

:S 1 .

gi ] :S 2_ 4m1-1,i (2N) e E 2 N/s .

Pi

Here, the cond itioning is on selecting gi as the fin a l hypothesis by S R M .

[Hint: Use the Bayes theorem to decompose the probability and then
apply the VC bound on one of the terms}
You may interpret this result as follows: if you use S RM a n d end up with gi ,
then the genera l ization bou nd is a factor -!; worse than the bound you wou ld
have gotten had you simply started with 1-li .

Problem 5.3

I n our credit card exa mple, the ba n k starts with some vague
idea of what constitutes a good credit risk. So, as customers x 1 , x2 , . . . , X N
arrive, the ba nk a pplies its vague idea to approve credit cards for some of these
customers. Then, only those who got credit cards are mon itored to see if they
defa u lt or not .
For simplicity, su ppose that t h e first N customers were given cred it cards.
Now that the ba nk knows the behavior of these customers, it comes to you
to im prove their a lgorith m for a pproving credit. The ba n k gives you the data
(x 1 , y1 ) , . . . ' (xN , YN ) .
Before you look a t the data , you d o mathematical derivations a n d come u p with
a credit a pprova l fu nction . You now test it on the data and, to you r delight,
obtain perfect prediction .

( a ) What is M, the size of you r hypothesis set?

( b ) With such an M, what does the Hoeffding bound say a bout the proba bil ity
that the true performa nce is worse than 2% error for N

10000?

( c ) You give you r g to the ba n k and assu re them that the performa nce will
be better than 2% error and you r confidence is given by you r a nswer
to part ( b) . The ba n k is t h rilled a nd uses you r g to a pprove credit for
new clients. To their d ismay, more than h a lf their credit cards a re being
defa u lted on. Explain the possible reason ( s) beh ind this outcome.

( d ) Is there a way in which the ban k could use you r credit a pproval function
to have you r probabilistic guara ntee? How? [Hint: The answer is yes!}

179

5 .4 . PROBLEMS

5 . THREE LEARNING PRINCIPLES

The S&P 500 is a set of the l argest 500 compa n ies currently
trading. S u ppose there are 10, 000 stocks currently trading, and there have been
50, 000 stocks which h ave ever traded over the last 50 years ( some of these have
gone ba n kru pt a n d stopped trading) . We wish to eval u ate the profita bility of
various ' buy a n d hold ' strategies using these 50 years of data ( rough ly 12, 500
trading d ays ) .

Problem 5 . 4

Since it is not easy to get stock data , we wi ll confi ne our a n alysis to today's
S&P 500 stocks, for which the data is readily available.

( a ) A stock is profita ble if it went up on more than 50% of the days. Of you r
S & P stocks, the most profitable went up o n 52% of t h e days ( Ein = 0.48) .
( i ) Since we picked the best a mong 500, using the Hoeffd ing bound,
IP [I Ein - Eout l

>

0.02 ] :: 2

500 x e - 2 x 1 2 5 oo x o.o2 2

0.045.

There is a greater tha n 95% cha nce this stock is profitable. Where
d id we go wrong?
( ii ) Give a better estimate for the proba bil ity that this stock is profitable.
[Hint: What should the correct M be in the Hoeffding bound?]

( b ) We wish to eva luate the profita bility of ' buy a nd hold ' for genera l stock
tra d ing. We notice that a l l of our 500 S&P stocks went up on at least 51%
of the days.

( i ) We concl ude that buying a n d holding a stocks is a good strategy for

general stock trad ing. Where did we go wrong?

( ii ) Ca n we say anything a bout the performance of buy a nd hold trading?

Problem 5.5
You thin k that the stock market exh ibits reversa l , so if
the price of a stock sharply d rops you expect it to rise shortly thereafter. If it
sharply rises, you expect it to d rop shortly thereafter.
To test this hypothesis, you build a trading strategy that buys when the stocks
go down a nd sel ls in the opposite case. You collect historica l data on the cu rrent
S&P 500 stocks, and you r hypothesis gave a good a n n u a l retu rn of 12%.

( a ) When you trade using this system, do you expect it to perform at this
level? Why or why not?

( b ) How ca n you test you r strategy so that its performance in sam ple is more
reflective of what you should expect in rea l ity?

Problem 5.6

One often hears "Extra polation is harder than interpolation ."

G ive a possible expla nation for this phenomenon using the principles in this
cha pter. [Hint: training distribution versus testing distribution.}

180

Epilogue
This book set the stage for a deeper exploration into Learning From Data by
developing the foundations. It is possible to learn from data, and you have
all the basic tools to do so. The linear model coupled with the right features
and an appropriate nonlinear transform, together with the right amount of
regularization, pretty much puts you into the thick of the game, and you will
be in good stead as long as you keep in mind the three basic principles: simple
is better ( Occam's razor ) , avoid data snooping and beware of sampling bias.
Where to go from here? There are two main directions. One is to learn
more sophisticated learning techniques, and the other is to explore different
learning paradigms. Let us preview these two directions to give the reader a
better understanding of the 'map' of learning from data.
The linear model can be used as a building block for other popular tech
niques. A cascade of linear models, mostly with soft thresholds, creates a
neural network . A robust algorithm for linear models, based on quadratic
programming, creates support vector machines. An efficient approach to non
linear transformation in support vector machines creates kernel methods. A
combination of different models in a principled way creates boosting and en
semble learning. There are other successful models and techniques, and more
to come for sure.
In terms of other paradigms, we have briefly mentioned unsupervised learn
ing and reinforcement learning. There is a wealth of techniques for these learn
ing paradigms, including methods that mix labeled and unlabeled data. Active
learning and online learning, which we also mentioned briefly, have their own
techniques and theories. In addition, there is a school of thought that treats
learning as a completely probabilistic paradigm using a Bayesian approach,
and there are useful probabilistic techniques such as Gaussian processes. Last
but not least, there is a school that treats learning as a branch of the theory
of computational complexity, with emphasis on asymptotic results.
Of course, the ultimate test of any engineering discipline is its impact in
real life. There is no shortage of successful applications of learning from data.
Some of the application domains have specialized techniques that are worth
exploring, e.g. , computational finance and recommender systems.
Learning from data is a very dynamic field. Some of the hot techniques
and theories at times become just fads, and others gain traction and become
181

EPILOGUE

part of the field. What we have emphasized in this book are the necessary
fundamentals that give any student of learning from data a solid foundation,
and enable him or her to venture out and explore further techniques and
theories, or perhaps to contribute their own.

182

Furt her Reading

Learning From Data book forum ( at AMLBook.com ) .
Y. S . Abu-Mostafa. The Vapnik-Chervonenkis dimension: Information versus
complexity in learning. Neural Computation, 1 ( 3 ) :312 317, 1989 .
Y. S. Abu-Mostafa, X. Song, A. Nicholson, and M. Magdon-Ismail. The bin
model. Technical Report CaltechCSTR: 2004.002, California Institute of
Technology, 2004.
R. Ariew. Ockham 's Razor: A Historical and Philosophical Analysis of Ock
ham 's Principle of Parsimony. University of Illinois Press, 1976 .
R. Bell, J . Bennett, Y . Koren, and C . Volinsky. The million dollar program
ming prize. IEEE Spectrum, 46 ( 5 ) :29 33, 2009.
A. Blumer, A. Ehrenfeucht, D . Haussler, and M. K. Warmuth. Occam's razor.
Information Processing Letters, 24 ( 6 ) :377 380, 1987.
A. Blumer, A. Ehrenfeucht, D. Haussler, and M. K. Warmuth. Learnability
and the Vapnik-Chervonenkis dimension. Journal of the Association for
Computing Machinery, 36 ( 4) :929 965, 1989.
S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University
Press, 2004.
P. Burman. A comparative study of ordinary cross-validation, v-fold cross
validation and the repeated learning-testing methods. Biometrika, 76 ( 3 ) :
503 514, 1989.
T. M. Cover. Geometrical and statistical properties of systems of linear in
equalities with applications in pattern recognition. IEEE Transactions on
Electronic Computers, 14 ( 3 ) : 326 334, 1965 .
M. H. DeGroot and M. J. Schervish. Probability and Statistics. Addison
Wesley, fourth edition, 201 1 .
183

FURTHER READING

V. Fabian. Stochastic approximation methods. Czechoslovak Mathematical

Journal, 10(1) : 123 159, 1960.
W. Feller. An Introduction to Probability Theory and Its Applications. Wiley,
third edition, 1968.
A. Frank and A . Asuncion. UCI machine learning repository, 2010. URL
http : //archive . i c s . uc i . edu/ml.

J. H. Friedman. On bias, variance, 0 /1 loss, and the curse-of-dimensionality.

Data Mining and Knowledge Discovery, 1 (1 ) :55 77, 1997.
S. I. Gallant. Perceptron-based learning algorithms. IEEE Transactions on
Neural Networks, 1 (2) : 179- 191, 1990.
Z. Ghahramani. Unsupervised learning. In Advanced Lectures in Machine
Learning {MLSS '03) , pages 72 112, 2004.

G. H. Golub and C. F. van Loan. Matrix computations. Johns Hopkins Uni

versity Press, 1996.
D . C . Hoaglin and R. E. Welsch. The hat matrix in regression and ANOVA.
American Statistician, 32: 17-22, 1978.
W. Hoeffding. Probability inequalities for sums of bounded random variables.
Journal of the American Statistical Association, 58(30 1 ) : 1 3 30, 1963.
R. C. Holte. Very simple classification rules perform well on most commonly
used datasets. Machine Learning, 1 1 (1) :63 9 1 , 1993.
R. A. Horn and C. R. Johnson. Matrix Analysis. Cambridge University Press,
1990.
L. P. Kaelbling, M. L. Littman, and A. W. Moore. Reinforcement learning: A
survey. Journal of Artificial Intelligence Research, 4:237 285, 1996.
A. I. Khuri. Advanced calculus with applications in statistics.
Interscience, 2003.

Wiley

R. Kohavi. A study of cross-validation and bootstrap for accuracy estimation

and model selection. In Proceedings of the 14th International Joint Con
ference on Artificial intelligence (IJCAI '95), volume 2, pages 1137 1 143 ,
1995.
J . Langford. Tutorial on practical prediction theory for classification. Journal
of Machine Learning Research, 6:273 306, 2005.

184

FURTHER READING

L. Li and H.-T. Lin. Optimizing 0 /1 loss for perceptrons by random coordinate

descent. In Proceedings of the 2007 International Joint Conference on Neural
Networks (IJCNN '07), pages 749 754, 2007.
H.-T. Lin and L. Li. Support vector machinery for infinite ensemble learning.
Journal of Machine Learning Research, 9(2) :285-312, 2008.
M. Magdon-Ismail and K. Mertsalov. A permutation approach to validation.
Statistical Analysis and Data Mining, 3 (6) :36 1-380, 2010.
M. Magdon-Ismail, A. Nicholson, and Y. S . Abu-Mostafa. Learning in the
presence of noise. In S. Haykin and B. Kosko, editors, Intelligent Signal
Processing. IEEE Press, 2001.
M. Markatou, H. Tian, S . Biswas, and G . Hripcsak. Analysis of variance of
cross-validation estimators of the generalization error. Journal of Machine
Learning Research, 6 : 1 127 1 168, 2005.
M. L. Minsky and S . Papert. Perceptrons: An Introduction to Computational
Geometry. MIT Press, expanded edition, 1988.
T. Poggio and S. Smale. The mathematics of learning: Dealing with data.
Notices of the American Mathematical Society, 50(5) :537 544, 2003.
K. Popper. The logic of scientific discovery. Routledge, 2002 .
F. Rosenblatt. The perceptron: A probabilistic model for information storage
and organization in the brain. Psychological Review, 65 (6) :386 408, 1958.
F . Rosenblatt. Principles of Neurodynamics: Perceptrons and the Theory of
Brain Mechanisms. Spartan, 1962.
B . Settles. Active learning literature survey. Technical Report 1648, University
of Wisconsin-Madison, 2010.
J . Shawe-Taylor, P. L. Bartlett, R. C. Williamson, and M. Anthony. A frame
work for structural risk minimisation. In Learning Theory: 9th Annual
Conference on Learning Theory (COLT '96), pages 68 76, 1996.
L. G . Valiant. A theory of the learnable. Communications of the ACM, 27
( 1 1) : 1 1 34 1142, 1984.
V. N. Vapnik and A. Y. Chervonenkis. On the uniform convergence of relative
frequencies of events to their probabilities. Theory of Probability and Its
Applications, 16:264 280, 1971 .
185

V. N. Vapnik, E. Levin, and Y. L. Cun. Measuring the VO-dimension of a

learning machine. Neural Computation, 6(5) :851 876, 1994.
G .-X. Yuan, C.-H. Ho, and C.-J. Lin. Recent advances of large-scale linear
classification. Proceedings of IEEE, 2012.
T. Zhang. Solving large scale linear prediction problems using stochastic gra
dient descent algorithms. In Machine Learning: Proceedings of the 21th
International Conference (ICML '04), pages 919 926, 2004.

186

Appendix
Proof of t he VC B ound
In this Appendix, we present the formal proof of Theorem 2.5. It is a fairly
elaborate proof, and you may skip it altogether and just take the theorem for
granted, but you won't know what you are missing !
Theorem A . 1 (Vapnik, Chervonenkis, 1971 ) .
Jp>

sup IEin (h)

hEH

Eout (h) I > E

:S

4mH (2N) e - i E 2 N .

This inequality is called the VC Inequality, and it implies the VC bound of

Theorem 2.5. The inequality is valid for any target function (deterministic
or probabilistic) and any input distribution. The probability is over data
sets of size N. Each data set is generated iid (independent and identically
distributed) , with each data point generated independently according to the
joint distribution P(x, y ) . The event sup h E H I Ein (h) Eout (h) I > E is equiva
lent to the union over all h E 1-l of the events IEin (h) Eout (h) I > t; this union
contains the event that involves g in Theorem 2.5. The use of the supremum (a
technical version of the maximum) is necessary since 1-l can have a continuum
of hypotheses.
The main challenge to proving this theorem is that Eout ( h) is difficult to
manipulate compared to Ein (h) , because Eout (h) depends on the entire input
space rather than just a finite set of points. The main insight needed to over
come this difficulty is the observation that we can get rid of Eout ( h) altogether
because the deviations between Ein and Eout can be essentially captured by
deviations between two in-sample errors: Ein (the original in-sample error)
and the in-sample error on a second independent data set (Lemma A.2) . We
have seen this idea many times before when we use a test or validation set to
estimate Eout . This insight results in two main simplifications:

1 . The supremum of the deviations over infinitely many h E 1-l can be

reduced to considering only the dichotomies implementable by 1-l on the
187

A PPENDIX
two independent data sets. That is where the growth function mH ( 2N )
enters the picture (Lemma A.3) .
2 . The deviation between two independent in-sample errors is 'easy' to an
alyze compared to the deviation between Ein and Eout (Lemma A.4) .
The combination of Lemmas A.2, A.3 and A.4 proves Theorem A. l .
A. 1

Relating Generalization Error t o In- S ample

Deviat ions

Let's introduce a second data set 'D', which is independent of 'D, but sampled
according to the same distribution P(x , y) . This second data set is called a
ghost data set because it doesn't really exist; it is a just a tool used in the
analysis. We hope to bound the term JP>[IEin Eout I is large) by another term
JP>[IEin E[n I is large) , which is easier to analyze.
The intuition behind the formal proof is as follows. For any single hypoth
esis h, because 'D' is fresh, sampled independently from P(x, y ) , the Hoeffding
Inequality guarantees that E[n (h) Eout (h) with a high probability. That
is, when IEin (h) Eout (h) I is large, with a high probability IEin (h) E[n (h) I
is also large. Therefore, JP>[IEin (h) Eout (h) I is large) can be approximately
bounded by JP>[IEin (h) E{n (h) I is large) .
We are trying to bound the probabil
ity that Ein is far from Eout . Let E{n ( h)
be the 'in-sample' error for hypothesis h
on 'D' . Suppose that Ein is far from Eout
with some probability (and similarly E{n
is far from Eout , with that same prob
ability, since Ein and E[n are identically
distributed) . When N is large, the proba
bility is roughly Gaussian around Eout , as
illustrated in the figure to the right. The
red region represents the cases when Ein
is far from Eout . In those cases, E{n is far from Ein about half the time,
as illustrated by the green region. That is, JP>[IEin Eout I is large] can be
approximately bounded by 2 JP> [IEin E{n l is large] .
This argument provides some intuition that the deviations between Ein
and Eout can be captured by the deviations between Ein and E[n . The argu
ment can be carefully extended to multiple hypotheses.
Lemma A.2.

where the probability on the RHS is over

188

'D and 'D' jointly.

APPENDIX

Proof. We can assume that IF sup J Ein (h)

there is nothing to prove.

[
[
[
[

JP> sup J Ein (h)

hE1-l

> JP> sup J Ein (h)

h E 1-l

Eout (h) J > E

hE1-l

E{n (h) J >

E{n (h) J > and sup JEin(h)

l
I

hE1-l

Eout (h) J > E

JP> sup J Ein (h)

E{n (li) J > sup J Ein (h)

hE1-l

Eout (h) J > E (A. 1 )

JP> sup J Ein (h)

hE1-l

> 0, otherwise

hE1-l

Eout (h) J > E .

Inequality (A. 1 ) follows because JP>[B1] JP>[B1 and 82 ] for any two events
Bi , 82 Now, let's consider the last term:

JP> sup J Ein (h)

hE1-l

E{n (h) J > sup J Ein (h)

hE1-l

Eout (h) J > E .

The event on which we are conditioning is a set of data sets with non-zero
probability. Fix a data set V in this event. Let h* be any hypothesis for
which J Ein (h*) Eout (h* ) J > E. One such hypothesis must exist given that V
is in the event on which we are conditioning. The hypothesis h * does not
depend on V', but it does depend on V.

[
[
[

JP> sup J Ein (h)

hE1-l

E{n (h) J > sup JEin (h)

> I!' IE; n (h * )

E{n (h * ) J >

> I!' IE{u (h * )

Eout (h * ) J

>

1 - 2e t2 N .

hE1-l

I
I

IE; n (h)

IE;n (h)

Eout (h) J > E

Eout (h) I > E

Eout (h) J > E

1. Inequality (A.2 ) follows because the event " JEin (h* )

implies " sup J Ein (h) E{n (h) J > f'.

(A . 2 )
(A.3)
(A.4)

E{n (h*) J > f '

hE1-l

2 . Inequality ( A.3 ) follows because the events " JE{n (h*) Eout (h*) J ::; f '
and " JEin (h*) Eout (h* ) J > E" (which is given) imply " JEin (h) E{n (h) J >
2t "
3. Inequality (A.4) follows because h* is fixed with respect to V' and so we
can apply the Hoeffding Inequality to JP>[JE{n (h* ) Eout (h* ) J :'S H

Notice that the Hoeffding Inequality applies to IF[JE{n (h*) Eout (h*) J ::; ]
for any h* , as long as h* is fixed with respect to V' . Therefore, it also applies
189

APPENDIX

to any weighted average of JP [I E{n (h* ) Eout (h* )I :S i ] based on h* . Finally,

since h * depends on a particular V, we take the weighted average over all V
in the event
" sup I Ein (h) Eout (h) I > E"
hE H
on which we are conditioning, where the weight comes from the probability of
the particular V. Since the bound holds for every V in this event, it holds for
II
the weighted average.

Note that we can assume e- E2 N < -Jt , because otherwise the bound in
Theorem A. 1 is trivially true. In this case, 1 2e- E 2 N > so the lemma
'
implies

JP
A.2

sup I Ein(h)
hE H

- Eout (h) I

> E

:S

2 JP

sup IEin (h)

hE H

- E{n (h) I

>

i]

B ounding Worst Case Deviat ion Using t he

Growth Function

Now that we have related the generalization error to the deviations between
in-sample errors, we can actually work with }{ restricted to two data sets of
size N each, rather than the infinite }{ . Specifically, we want to bound

IF

sup I Ein( h)
hE H

- E{n (h) I

>

i] ,

where the probability is over the joint distribution of the data sets V and V'.
One equivalent way of sampling two data sets V and V' is to first sample a
data set S of size 2N, then randomly partition S into V and V' . This amounts
to randomly sampling, without replacement, N examples from S for V, leaving
the remaining for V' . Given the joint data set S, let

be the probability of deviation between the two in-sample errors, where the
probability is taken over the random partitions of S into V and V'. By the
law of total probability (with I: denoting sum or integral as the case may be) ,

IF

sup IEin(h)
hE H

L IF [ S] x JP
S

< sp

IP'

- E{n (h) I

sup IEin(h)
hE H

[ [E1n (h)

190

>

i]

E{n (h) I

E[0 (h) [

>

>

Il

i I s]

APPENDIX

Let 1-l ( S) be the dichotomies that 1-l can implement on the points in S. By
definition of the growth function, 1-l(S) cannot have more than mH (2N ) di
chotomies. Suppose it has M :: mH (2N) dichotomies, realized by h1 , . . . , hM .
Thus ,
sup IEin (h) - E[n (h) I =
sup
/Ein (h) - E[n (h) I .
hE H
hE{h1 , . . . ,hM}
Then,

IP'
JP

r f Ein(h)

Il

E{n ( li ) I > s

sup
IEin(h)
hE{h1 , . . . ,hM}

I]

- E[n (h) I > s

:L JP [ IEin(hm) - E[n (hm) / > I SJ

<

m= l

<

sup JP
hE H

[ IEin(h) - E[n (h) I > j SJ ,

(A.5)
(A.6)

where we use the union bound in (A. 5) , and overestimate each term by the
supremum over all possible hypotheses to get (A.6) . After using M :: mH (2N)
and taking the sup operation over S , we have proved:
Lemma A.3.

<

sup IEin (h) - E[11 (h) I >

hE H
mH (2N) X sup sup JP [ IEin (h)
S hEH

JP

- E[n (h) / > I SJ ,

where the probability on the LHS is over D and D' jointly, and the probability
on the RHS is over random partitions of S into two sets D and D' .
The main achievement of Lemma A.3 is that we have pulled the supre
mum over h E 1-l outside the probability, at the expense of the extra factor
of mH ( 2N ) .
A.3

B ounding t he Deviat ion b etween In- S ample

Errors

We now address the purely combinatorial problem of bounding

sup sup JP
S hE H

[ IEin(h) E{11 (h) I > j SJ ,

which appears in Lemma A.3. We will prove the following lemma. Then,
Theorem A. l can be proved by combining Lemmas A.2, A.3 and A.4 taking
2
1 2e- E N 2: (the only case we need to consider) .

191

APPENDIX

S,

Lemma A.4. For any h and any

where the probability is over random partitions of S into two sets

'D and 'D' .

Proof. To prove the result, we will use a result, which is also due to Hoeffding,
for sampling without replacement:
Lemma A.5 ( Hoeffding, 1963) . Let A = {a1 , . . . , a2 N } be a set of values with
an E [O, 1 ] , and let =
2: 1 an be their mean. Let 'D = {z1 , . . . , ZN } be
a sample of size N, sampled from A uniformly without replacement . Then

We apply Lemma A.5 as follows. For the 2N examples in S, let an = 1 if

h (xn ) -=f. Yn and an = 0 otherwise. The {an } are the errors made by h on S.
Now randomly partition S into 'D and 'D', i.e. , sample N examples from S
without replacement to get V, leaving the remaining N examples for 'D'. This
results in a sample of size N of the {an} for 'D, sampled uniformly without
replacement . Note that

Ein (h) =

L an , and
an EV

E{n (h) =

L a .

a'n EV'

Since we are sampling without replacement, S = 'D U 'D' and 'D n 'D'
so
1
Ein (h) + E{n (h)
.

2N n l
2

0,

and

It follows that IEin - I > t

{:

IEin

Substituting t = gives the result.

E{n l

>

2t . By Lemma A.5,

II

192

Notation

{ }
||


event (in probability)

set
absolute value of a number, or ardinality (number of elements) of a set, or determinant of a matrix

kk

square of the norm; sum of the squared omponents of a

[a, b]
JK

oor; largest integer whi h is not larger than the argument

()1
()
t
()
 
N
k

A\B
0
{1} Rd

ve tor
the interval of real numbers from

to

evaluates to 1 if argument is true, and to 0 if it is false

gradient operator, e.g.,
spe t to

w)

Ein

(gradient of

Ein (w)

with re-

inverse
pseudo-inverse
transpose ( olumns be ome rows and vi e versa)
number of ways to hoose k obje ts from N distin t obje ts
N!
(equals (N k)!k! where ` !' is the fa torial)
the set

with the elements from set

removed

zero ve tor; a olumn ve tor whose omponents are all zeros

d-dimensional Eu lidean

spa e with an added `zeroth oor-

dinate' xed to 1
toleran e in approximating a target
bound on the probability of ex eeding

(the approximation

toleran e)

learning rate (step size in iterative learning, e.g., in sto hasti gradient des ent)

regularization parameter
regularization parameter orresponding to weight budget

penalty for model omplexity; either a bound on general-

logisti fun tion

ization error, or a regularization term

s
s

(s) = e /(1 + e )
z = (x)

feature transform,

Qth-order

polynomial transform

193

Notation

2
A

argmina ()

a oordinate in the feature transform

, zi = i (x)

probability of a binary out ome

fra tion of a binary out ome in a sample
varian e of noise
learning algorithm
the value of

at whi h the minimum of the argument is

a hieved

B
b

an event (in probability), usually `bad' event

bias
B(N, k)

the bias term in bias-varian e de omposition

the bias term in a linear ombination of inputs, also alled

w0
maximum number of di hotomies on
point

C
d
d
dv ,dv (H)
D

points with a break

bound on the size of weights in the soft order onstraint

d
d
dimensionality of the input spa e X = R or X = {1} R

Z
H
data set D = (x1 , y1 ), , (xN , yN ); te hni ally not a set,
but a ve tor of elements (xn , yn ). D is often the training
dimensionality of the transformed spa e
VC dimension of hypothesis set

set, but sometimes split into training and validation/test

sets.

Dtrain

subset of

Dval
E(h, f )
ex
e(h(x), f (x))
en

validation set; subset of

is used.

used for training when a validation or test set

used for validation.

h and target fun tion f

e = 2.71828
2
pointwise version of E(h, f ), e.g., (h(x) f (x))
leave-one-out error on example n when this nth example is
error measure between hypothesis
exponent of

in the natural base

ex luded in training [ ross validation

E[]
Ex []
E[y|x]
Eaug
Ein , Ein (h)
E v
Eout , Eout (h)
D
Eout

Eout
Eval
Etest
f
g
g (D)
g

expe ted value of argument

expe ted value with respe t to
expe ted value of

given

augmented error (in-sample error plus regularization term)

in-sample error (training error) for hypothesis

ross validation error

out-of-sample error for hypothesis
out-of-sample error when

is used for training

expe ted out-of-sample error

validation error
test error
target fun tion,
nal hypothesis

g: X Y

f: X Y
g H sele ted

by the learning algorithm;

nal hypothesis when the training set is

average nal hypothesis [bias-varian e analysis

194

Notation

minus

g
g
h

h
H
H

nal hypothesis when trained using

H(C)

restri ted hypothesis set by weight budget

H(x1 , . . . , xN )

di hotomies (patterns of

H
I

The hat matrix [linear regression

a hypothesis

a hypothesis in transformed spa e

hypothesis set
hypothesis

max(, )
N
o()

set

that

some points

orresponds to

per eptrons in

transformed spa e

[soft order

onstraint

x1 , , xN

1)

generated by

on the points

identity matrix; square matrix whose diagonal elements are

1
K
Lq
ln
log2
M
mH (N )

g = Ein
h H; h : X Y

gradient, e.g.,

and o-diagonal elements are

size of validation set

q th-order

Legendre polynomial

logarithm in base
logarithm in base

e
2

number of hypotheses
the growth fun tion; maximum number of di hotomies generated by

on any

points

maximum of the two arguments

number of examples (size of

D)

absolute value of this term is asymptoti ally negligible ompared to the argument

O()

absolute value of this term is asymptoti ally smaller than

P (x)
P (y | x)
P (x, y)
P[]
Q
Qf
R
Rd
s

(marginal) probability or probability density of

a onstant multiple of the argument

sign()
supa (.)
T
t
tanh()
tra e()
V
v

x
y
x and y

onditional probability or probability density of

joint probability or probability density of

given

probability of an event
order of polynomial transform
omplexity of

(order of polynomial dening

f)

the set of real numbers

d-dimensional Eu lidean
P spa e
t
signal s = w x =
i wi xi (i goes from 0 to d or 1 to d
depending on whether x has the x0 = 1 oordinate or not)
sign fun tion, returning +1 for positive and 1 for negative
supremum; smallest value that is the argument for all a
number of iterations, number of epo hs
iteration number or epo h number
hyperboli tangent fun tion;

tanh(s) = (es es )/(es +es )

tra e of square matrix (sum of diagonal elements)

number of subsets in

V -fold

ross validation (V

K = N)

dire tion in gradient des ent (not ne essarily a unit ve tor)

195

Notation

v
var
w

w
w
wlin
wreg
wPLA
w0
x
x0

unit ve tor version of

[gradient des ent

the varian e term in bias-varian e de omposition

weight ve tor ( olumn ve tor)
weight ve tor in transformed spa e

sele ted weight ve tor [po ket algorithm

weight ve tor that separates the data
solution weight ve tor to linear regression
regularized solution to linear regression with weight de ay
solution weight ve tor of per eptron learning algorithm

w to represent bias b
x X . Often a olumn ve tor x Rd or x
x is used if input is s alar.
oordinate to x, xed at x0 = 1 to absorb the bias

added oordinate in weight ve tor

the input
{1} Rd .
added

term in linear expressions

X
X
XOR

input spa e whose elements are

xX

matrix whose rows are the data inputs

xn

[linear regression

ex lusive OR fun tion (returns 1 if the number of 1's in its

input is odd)

y
y

the output

yn

y
Y
Z
Z

yY

olumn ve tor whose omponents are the data set outputs

[linear regression

estimate of

[linear regression

output spa e whose elements are

yY

transformed input spa e whose elements are

z = (x)
zn = (xn )

matrix whose rows are the transformed inputs

[linear regression

196

Index
a tive learning, 181
denition, 12
Adaline, 35, 110
approximation, 27
versus generalization, 6268, 106
arti ial intelligen e, 5
augmented error, 132, 157
axiom of non-falsiability, 178
B(N, k)
denition, 46
lower bound, 69
upper bound, 48
ba kgammon, 12
Bayes optimal de ision theory, 10
Bayes theorem, 33
Bayesian learning, 181
bias-varian e, 6266
average fun tion, 63
dependen e on N, d, 158
example, 65
impa t of noise, 125
linear models, 158159
linear regression, 114
noisy target, 74
bin model, 18
multiple bins, 22
relationship to learning, 20
binomial distribution, 36
boosting, 181
break point
denition, 45

Chebyshev inequality, 36
Cherno bound, 37
lassi ation
for regression, 113
linear programming algorithm, 110
lassi ation error

bound by ross-entropy error, 97

bound by squared error, 97
lustering, 13
oin lassi ation, 9, 13
ombinatorial optimization, 80
omplexity
of H, 26
of f , 27
omputational omplexity, 181
omputational nan e, 181
omputer vision, 1
onvex fun tion, 93
onvex set, 44
ost, 28
ost matrix, 29, 115
redit approval, 3, 82, 96
ross validation, 145150
V -fold, 150
hoosing , 149
digits data, 151
ee tive number of examples, 163
exa t omputation, 149
leave-one-out, 146
linear model, 149
linear model, analyti , 164
model sele tion, 148
regularized, 165
summary, 147
unbiased, 147
varian e, 162
ross-entropy, 92
data
data
data
data

ontamination, 145, 151, 176

mining, 15
point, 3
set, 3
ghost, 188
spa e of, 54
data snooping, 173177, 181
197

Index

football s am, 170

nan ial trading, 174

nonlinear transform, 103
normalization bias, 174
versus sampling bias, 177
de ision stump, 106
design
versus learning, 9
deterministi noise, 124, 128
ee t on learning, 151
regularization, 136
similarity to sto hasti noise, 136
Dewey, 171
di hotomy, 42
maximum number, 46
per eptron, 43
table, 47
dierentiable, 85
twi e-, 93, 95
ee tive number of hypotheses, 41, 53
ee tive number of parameters, 52, 137,
159
Einstein, 167
ensemble learning, 181
entropy, 168
error measure, 2830
L1 versus L2 , 38
lassi ation, 28
ross-entropy, 92
ngerprint example, 28
logisti regression, 91
example, 3
false a ept, 29, 115
false reje t, 29, 115
falsiability, 178
feasibility of learning
Boolean example, 16
probabilisti , 18
two main questions, 26
visual example, 15
feature sele tion, 151
feature spa e, 100
features, 81
nonlinear transform, 99
feature transform, 100, 111, 116117
nal exam, 39
nan ial fore asting, 1
ngerprint veri ation, 28, 115

Gaussian pro esses, 181

generalization, 3959
VC bound, 5059
VC dimension, 50
generalization bound
denition, 40
Devroye, 73
Parrondo and Van den Broek, 73
Radema her penalty, 73
relative error, 74
VC, see VC generalization bound
generalization error
denition, 40
global minimum, 93
gradient des ent, 9299
algorithm, 95
bat h, 97
initialization and termination, 95
sto hasti , 97
growth fun tion, 4150
2-dimensional per eptron, 43
bound, 4649
onvex set, 44
denition, 42
in VC proof, 190
polynomial bound, 50
positive interval, 44
positive ray, 43
two-dimensional per eptron, 43
handwritten digit re ognition, 4, 11, 81
82, 106107, 151
hat matrix, 87, 112
Hessian matrix, 116
Hoeding bound, see Hoeding Inequality
Hoeding Inequality, 19, 1927
and binomial distribution, 36
uniform version, 24
without repla ement, 192
hypothesis set, 3
omposition, 72
on entri spheres, 69
onvex set, 44
monotoni , 71
polynomial, 120
positive interval, 44

198

Index

positive ray, 43
positive re tangles, 69
positive-negative interval, 69
positive-negative ray, 69
restri ted to inputs, 42
in-sample error, 21
input spa e, 3
iterative learning, 7
kernel methods, 181
Lagrange multiplier, 131, 157
lasso, 161
law of large numbers, 36, 37
learning
riteria, 26, 78
feasibility, 1518, 2426
learning algorithm, 3
learning urve, 6668, 140, 147
linear regression, 88
learning model
denition, 5
learning problem
summary gure, 30
learning rate, 94, 95
leave-one-out, 146
Legendre polynomials, 123, 128129, 154,
155
likelihood, 91
linear lassi ation, 77
linear model, 77
bias-varian e, 158159
building blo k, 181
ross validation, analyti , 164
optimal weight de ay, 161
overlooked resour e, 107
summary, 96
linear programming, 110, 111
linear regression, 8288, 111
algorithm, 86
bias and varian e, 114
for lassi ation, 9697, 109110
learning urve, 88
optimal hypothesis, 111
out of sample, 8788
out-of-sample error, 112
proje tion matrix, 86, 113
rank de ient, 114

using lassi ation algorithm, 113

linearly separable, 6, 78
example, 6
lo al minimum, 93
logisti fun tion, 89
logisti regression, 8899
algorithm, 95
ross-entropy error, 92
error measure, 9192
for lassi ation, 9697, 115
hard threshold, 115
initialization, 95
optimal de ision theory, 115
termination, 96
loss matrix, 38
ma hine learning, vii, 14
maximum likelihood, 91
medi al diagnosis, 1
minimum des ription length, 168
model sele tion, 141145
hoosing , 134, 149
ross validation, 148
experiment, 144
summary, 143
monotoni fun tions, 71
VC dimension, 71
movie rating, 13
multi lass, 81
Netix, 1
neural network, 181
Newton's method, 116
noise
deterministi , 124
sto hasti , 124
non-falsiability, 178
axiom, 170
pi king nan ial traders, 170
non-separable data, 7981
nonlinear regression, 104
nonlinear transformation, 99
normalization, 175
NP-hard, 80
obje tive, 28
O am's razor, 167171, 181
o training set error, 37
, 58

199

Index

online learning, 98, 181

denition, 12
ordinary least squares, 86
out-of-sample error, 21
outliers, 79
output spa e, 3
overtting, 119165, 171
denition, 119
experiment, 123, 155
learning urves, 122
pattern re ognition, 9
penalty
hypothesis omplexity, 126, 133
model omplexity, 58
per eptron, 58, 7882
denition, 5
experiment, 34
learning algorithm (PLA), 7
mH (N ), 70
PLA onvergen e, 33
po ket algorithm, 80
per eptron learning algorithm, 7, 77, 78,
98, 109110
and SGD, 98
onvergen e, 33
gure, 7, 83
PLA, see per eptron learning algorithm
po ket algorithm, 80, 97, 109
gure, 83
poll, 19
Truman versus Dewey, 171
polynomial transform, 104
polynomials, 120
positive interval, 44
positive ray, 43
postal s am, 170
predi tion of heart atta ks, 89
probability
logisti regression, 89
union bound, 24, 41
proje tion matrix, 113
pseudo-inverse, 85
numeri al stability, 86
publi ation bias, 173

re ommender systems, 1, 15, 181

regression, 77, 82
logisti , 89
regularization, 126137, 181
Ein versus , 156
augmented error, 132
hoosing , 134, 149
input noise, 160
lasso, 161
linear model, 133
ridge regression, 132
soft order onstraint, 128
Tikhonov, 131, 160
VC dimension, 137
weight de ay, 132
regularization parameter, , 133
reinfor ement learning, 12, 181
ridge regression, 132
risk, 28
risk matrix, 38, see also ost matrix

quadrati programming, 181

sample omplexity, 5657

sampling bias, 171173, 181
versus data snooping, 177
Sauer's Lemma, 48
sear h engines, 1
sele tion bias, 173
SGD, see sto hasti gradient des ent
shatter, 42
sigmoid, 90
singular value de omposition, 114
soft order onstraint, 157
soft threshold, 90
spam, 4, 6
squared error, 61, 66, 84, 140
SRM, see stru tural risk minimization
statisti s, 14
sto hasti gradient des ent, 9799, 110
sto hasti noise, 124
streaming data, 12
stru tural risk minimization, 178
superstition, 119
supervised learning
denition, 11
support ve tor ma hines, 181
supremum, 187
SVD, see singular value de omposition

random sample, 19

tanh, 90
200

Index

target distribution, 31
target fun tion, 3
noisy, 3032, 83, 87
test set, 59
Tikhonov regularizer, 131
Tikhonov smoothness penalty, 162
training examples, 4
Truman, 171

linear model, 133

negative , 156
optimal , 161
virtual examples, 157
Z spa e, 99102

undertting, 135
union bound, 24, 41
unlabeled data, 13, 181
unsupervised learning, 13, 181
learning a language, 13
validation, 137141
ross validation, 145
model sele tion, 141
summary, 141
validation set, 138
validation error, 138
expe tation, 138
optimisti bias, 142
varian e, 139
validation set
VC bound, 139, 163
Vapnik-Chervonenkis, see VC
VC dimension, 50
d-dimensional per eptron, 52
and number of parameters, 72
denition, 50
ee tive, 137
interse tion of hypothesis sets, 71
monotoni fun tions, 71
of omposition, 72
union of hypothesis sets, 71
VC generalization bound, 53, 78, 87, 102
denition, 53
proof, 187
sket h of proof, 53
VC Inequality, 187
vending ma hines, 9
virtual examples, 157
weight de ay, 132
ross validation error, 149
example, 126
gradient des ent, 156
invarian e under linear transform, 162
201