Scribd
Upload
15 views

Supervised Learning: Overview 3: Rayid Ghani

This document discusses supervised learning methods and ensemble methods. It covers common supervised learning algorithms like decision trees, support vector machines, and neural networks. It then focuses on ensemble methods like bagging, boosting, and random forests. Bagging creates ensembles by resampling the training data and combining models through voting. Boosting iteratively reweights examples to focus on misclassified examples. Random forests grow trees on random subsets of data and features to reduce correlation between trees.

Uploaded by

ashishamitav123
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
15 views

Supervised Learning: Overview 3: Rayid Ghani

This document discusses supervised learning methods and ensemble methods. It covers common supervised learning algorithms like decision trees, support vector machines, and neural networks. It then focuses on ensemble methods like bagging, boosting, and random forests. Bagging creates ensembles by resampling the training data and combining models through voting. Boosting iteratively reweights examples to focus on misclassified examples. Random forests grow trees on random subsets of data and features to reduce correlation between trees.

Uploaded by

ashishamitav123
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 20

Supervised Learning: Overview 3

Rayid Ghani

Slides liberally borrowed and customized from lots of excellent online sources
Rayid Ghani @rayidghani
Methods
• Regression
• Nearest neighbor
• Decision Trees
• Support Vector Machines
• Bayes Classifier
• Ensembles
– Bagging We’ll cover these today
– Boosting
– Random Forests
• Neural Networks
Rayid Ghani @rayidghani
Why Ensembles?

Rayid Ghani @rayidghani


How can we create ensembles?

• Different learning algorithms


• Algorithms with different choice for parameters
• Data set with different features (e.g. random
subspace)
• Data set = different subsets (e.g. bagging, boosting)

Rayid Ghani @rayidghani


Ensemble Methods
• Bagging (Bootstrap Aggregation)
• Boosting
• Random Forests
• Stacking

Rayid Ghani @rayidghani


Bagging
• Create ensembles by repeatedly randomly resampling the training
data (Brieman, 1996).

• Given a training set of size n, create m samples of size n by


drawing n examples from the original data, with replacement.
– Each bootstrap sample will on average contain 63.2% of the unique training
examples, the rest are replicates.

• Combine the m resulting models using simple majority vote.

Rayid Ghani @rayidghani


Bagging
• For i = 1 .. M
– Draw samples with replacement
– Learn classifier Ci
• Final classifier is a vote of C1 .. CM
• Why does it work?
– Increases classifier stability/reduces variance
– Works better with unstable classifiers (Decision Trees)

figure from Friedman et al. 2000


Rayid Ghani @rayidghani
• Why does it work?
– Increases classifier stability/reduces variance
– Works better with unstable classifiers (Decision Trees)

• What are some problems with it?

Rayid Ghani @rayidghani


Boosting
• Examples are given weights.
• At each iteration, a new hypothesis/model is
learned and the examples are reweighted to
focus the model on examples that the most
recently learned model got wrong.

Rayid Ghani @rayidghani


Boosting
• General Loop:
Set all examples to have equal uniform weights.
For t from 1 to T do:
Learn a classifier, Ct, from the weighted examples
Increase the weights of examples Ct classifies incorrectly

• Base (weak) learner must focus on correctly classifying


the most highly weighted examples while strongly
avoiding over-fitting.

• During testing, each of the T hypotheses get a weighted


vote proportional to their accuracy on the training data.

Rayid Ghani @rayidghani


typically
where

the weights of incorrectly classified examples are


increased so that the base learner is forced to focus
on the hard examples in the training set

From [R. Schapire, NE&C03]


Rayid Ghani @rayidghani
Rayid Ghani @rayidghani
Rayid Ghani @rayidghani
Boosting
• Improves classification accuracy

• Can be used with many different types of


classifiers

figure from Friedman et al. 2000


Rayid Ghani @rayidghani
Random Forest
• Each tree is grown on a bootstrap sample of the training
set of N cases.
• A number m is specified much smaller than the total
number of variables M (e.g. m = sqrt(M)).
• At each node, m variables are selected at random out of
the M.
• The split used is the best split on these m variables.
• Final classification is done by majority vote across trees.

15
Rayid Ghani @rayidghani
Random Forests

• Motivation: reduce error correlation between classifiers

• Main idea: build a larger number of un-pruned decision


trees

• Key: using a random selection of features to split on at


each node

Rayid Ghani @rayidghani


Advantages of random forest
• Accurate

• More robust with respect to noise.

• More efficient on large data

• Provides an estimation of the importance of features in


determining classification

• More info at: http://stat-


www.berkeley.edu/users/breiman/RandomForests/cc_home.htm
Rayid Ghani @rayidghani
Questions to think about?
• Could you inject more randomness in random
forests?

• How?

• What would be the impact?

Rayid Ghani @rayidghani


Factors to consider
• Complexity
• Overfitting
• Robustness
• Interpretability
• Training Time
• Test Time

Rayid Ghani @rayidghani


What to remember about classifiers

• Better to have smart features and simple classifiers than


simple features and smart classifiers

• Need more training data with increasingly powerful


classifiers (bias-variance tradeoff)

Rayid Ghani @rayidghani


Slide credit: D. Hoiem

You might also like