The AdaBoost algorithm

(0)
0) Set W̃i = 1/n for i = 1, . . . , n
1) At the mth iteration we find (any) classifier h(x; θ̂m) for
which the weighted classification error �m
� n �
1 � (m−1)
�m = 0.5 − W̃i yih(xi; θ̂m)
2 i=1
is better than chance.
2) The new component is assigned votes based on its error:
α̂m = 0.5 log( (1 − �m)/�m )

3) The weights are updated according to (Zm is chosen so that

(m)
the new weights W̃i sum to one):
(m) 1 (m−1)
W̃i = · W̃i · exp{ −yiα̂mh(xi; θ̂m) }
Zm
Tommi Jaakkola, MIT CSAIL 18
 Adaboost properties: exponential loss
• After each boosting iteration, assuming we can find a
component classifier whose weighted error is better than
chance, the combined classifier
ĥm(x) = α̂1h(x; θ̂1) + . . . + α̂mh(x; θ̂m)
is guaranteed to have a lower exponential loss over the
training examples
140

120

100
exponential loss

80

60

40

20

0
0 10 20 30 40 50
number of iterations

Tommi Jaakkola, MIT CSAIL 20

 Adaboost properties: training error
• The boosting iterations also decrease the classification error
of the combined classifier

ĥm(x) = α̂1h(x; θ̂1) + . . . + α̂mh(x; θ̂m)

over the training examples.

0.16

0.14

0.12

0.1
training error

0.08

0.06

0.04

0.02

0
0 10 20 30 40 50
number of iterations

Tommi Jaakkola, MIT CSAIL 21

 Adaboost properties: training error cont’d
• The training classification error has to go down exponentially
fast if the weighted errors of the component classifiers, �k ,
are strictly better than chance �k < 0.5
m �
�
err(ĥm) ≤ 2 �k (1 − �k )
k=1

0.16

0.14

0.12

0.1
training error

0.08

0.06

0.04

0.02

0
0 10 20 30 40 50
number of iterations

Tommi Jaakkola, MIT CSAIL 22

 Adaboost properties: weighted error
• Weighted error of each new component classifier
� n �
1 � (k−1)
�k = 0.5 − W̃i yih(xi; θ̂k )
2 i=1

tends to increase as a function of boosting iterations.

0.4

0.35

0.3
weighted training error

0.25

0.2

0.15

0.1

0.05
0 10 20 30 40 50
number of iterations

Tommi Jaakkola, MIT CSAIL 23

How Will Test Error Behave? (A First Guess)

0.8

0.6

error
0.4 test
0.2
train
20 40 60 80 100
# of rounds (T)

expect:
• training error to continue to drop (or reach zero)
• test error to increase when Hfinal becomes “too complex”
• “Occam’s razor”
• overfitting
• hard to know when to stop training
Technically...

• with high probability:

!" #
dT
generalization error ≤ training error + Õ
m

• bound depends on
• m = # training examples
• d = “complexity” of weak classifiers
• T = # rounds
• generalization error = E [test error]
• predicts overfitting
 “Typical” performance
• Training and test errors of the combined classifier

ĥm(x) = α̂1h(x; θ̂1) + . . . + α̂mh(x; θ̂m)

0.16

0.14

0.12
training/test errors
0.1

0.08

0.06

0.04

0.02

0
0 10 20 30 40 50
number of iterations

• Why should the test error go down after we already have

zero training error?

Tommi Jaakkola, MIT CSAIL 24

 AdaBoost and margin
• We can write the combined classifier in a more useful form
by dividing the predictions by the “total number of votes”:
α̂1h(x; θ̂1) + . . . + α̂mh(x; θ̂m)
ĥm(x) =
α̂1 + . . . + α̂m

• This allows us to define a clear notion of “voting margin” that

the combined classifier achieves for each training example:

margin(xi) = yi · ĥm(xi)

The margin lies in [−1, 1] and is negative for all misclassified

examples.

Tommi Jaakkola, MIT CSAIL 25

 AdaBoost and margin
• Successive boosting iterations still improve the majority vote
or margin for the training examples
� �
α̂1h(xi; θ̂1) + . . . + α̂mh(xi; θ̂m)
margin(xi) = yi
α̂1 + . . . + α̂m

• Cumulative distributions of margin values:

1 1

0.9 0.9

0.8 0.8

0.7 0.7

0.6 0.6

0.5 0.5

0.4 0.4

0.3 0.3

0.2 0.2

0.1 0.1

0 0
−1 −0.5 0 0.5 1 −1 −0.5 0 0.5 1

4 iterations 10 iterations

Tommi Jaakkola, MIT CSAIL 26

 AdaBoost and margin
• Successive boosting iterations still improve the majority vote
or margin for the training examples
� �
α̂1h(xi; θ̂1) + . . . + α̂mh(xi; θ̂m)
margin(xi) = yi
α̂1 + . . . + α̂m

• Cumulative distributions of margin values:

1 1

0.9 0.9

0.8 0.8

0.7 0.7

0.6 0.6

0.5 0.5

0.4 0.4

0.3 0.3

0.2 0.2

0.1 0.1

0 0
−1 −0.5 0 0.5 1 −1 −0.5 0 0.5 1

20 iterations 50 iterations

Tommi Jaakkola, MIT CSAIL 27

 Can we improve the combination?
• As a result of running the boosting algorithm for m iterations,
we essentially generate a new feature representation for the
data

φi(x) = h(x; θ̂i), i = 1, . . . , m

• Perhaps we can do better by separately estimating a new set

of “votes” for each component. In other words, we could
estimate a linear classifier of the form

f (x; α) = α1φ1(x) + . . . αmφm(x)

where each parameter αi can be now any real number (even

negative). The parameters would be estimated jointly rather
than one after the other as in boosting.

Tommi Jaakkola, MIT CSAIL 28

 Can we improve the combination?
• We could use SVMs in a postprocessing step to reoptimize
f (x; α) = α1φ1(x) + . . . αmφm(x)
with respect to α1, . . . , αm. This is not necessarily a good
idea.
0.16 0.16

0.14 0.14

0.12 0.12
training/test errors

training/test errors
0.1 0.1
typically
0.08 0.08

0.06 0.06

0.04 0.04

0.02 0.02

0 0
0 10 20 30 40 50 0 10 20 30 40 50
number of iterations number of components

boosting svm postprocessing

Tommi Jaakkola, MIT CSAIL 29
Practical Advantages of AdaBoost

• fast
• simple and easy to program
• no parameters to tune (except T )
• flexible — can combine with any learning algorithm
• no prior knowledge needed about weak learner
• provably effective, provided can consistently find rough rules
of thumb
→ shift in mind set — goal now is merely to find classifiers
barely better than random guessing
• versatile
• can use with data that is textual, numeric, discrete, etc.
• has been extended to learning problems well beyond
binary classification
Caveats

• performance of AdaBoost depends on data and weak learner

• consistent with theory, AdaBoost can fail if
• weak classifiers too complex
→ overfitting
• weak classifiers too weak (γt → 0 too quickly)
→ underfitting
→ low margins → overfitting
• empirically, AdaBoost seems especially susceptible to uniform
noise
Multiclass Problems
[with Freund]
• say y ∈ Y where |Y | = k
• direct approach (AdaBoost.M1):

ht : X → Y
!
Dt (i) e −αt if yi = ht (xi )
Dt+1 (i) = ·
Zt e αt if yi #= ht (xi )
"
Hfinal (x) = arg max αt
y ∈Y
t:ht (x)=y

• can prove same bound on error if ∀t : #t ≤ 1/2

• in practice, not usually a problem for “strong” weak
learners (e.g., C4.5)
• significant problem for “weak” weak learners (e.g.,
decision stumps)
• instead, reduce to binary
The One-Against-All Approach
• break k-class problem into k binary problems and
solve each separately
• say possible labels are Y = { , , , }

x1 x1 − x1 + x1 − x1 −
x2 x2 − x2 − x2 + x2 −
x3 ⇒ x3 − x3 − x3 − x3 +
x4 x4 − x4 + x4 − x4 −
x5 x5 + x5 − x5 − x5 −

• to classify new example, choose label predicted to be “most”

positive
• ⇒ “AdaBoost.MH” [with Singer]
• problem: not robust to errors in predictions