0% found this document useful (0 votes)

22 views43 pages

Resampling Methods-1

The document discusses resampling methods in machine learning, focusing on techniques like cross-validation and bootstrapping to assess model performance and select the best model. It emphasizes the importance of estimating prediction error through methods such as K-Fold Cross-Validation and Leave-One-Out Cross-Validation (LOOCV). Additionally, it highlights the significance of training, validation, and test sets in evaluating model generalization and avoiding overfitting.

Uploaded by

sainathgunda99

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

22 views43 pages

Resampling Methods-1

Uploaded by

sainathgunda99

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 43

sainathgunda99@gmail.

com
DLZNK464L9 Resampling – CV and bootstrapping

Proprietary
Thiscontent.
file is ©University
meant forof personal
Arizona. All Rights
use by Reserved. Unauthorized use or distributiononly.
[email protected] prohibited."

Sharing or publishing the contents in part or full is liable for legal action.
Resampling Methods
• Involves repeatedly drawing samples from a training set and refitting a
model of interest on each sample in order to obtain more information
about the fitted model.

• Example: We can estimate the variability of a linear regression fit by

repeatedly drawing different samples from the training data, fitting a
OLS regression to each new sample, and then examining the extent to
[email protected]
DLZNK464L9
which the resulting fits differ.

• Model Assessment: having chosen a final model, estimating its prediction

error on new data.

• Model Selection: estimating the performance of different models in

order to choose the best one.
Proprietary
Thiscontent.
file is ©University
meant forof personal
Arizona. All Rights
use by Reserved. Unauthorized use or distributiononly.
[email protected] prohibited."

Sharing or publishing the contents in part or full is liable for legal action.
Resampling Methods

• Cross-Validation
• Used to estimate test set prediction error rates associated with
a given machine learning method to evaluate its performance,
or to select the appropriate level of model flexibility.

[email protected]
• Bootstrap
DLZNK464L9

• Used most commonly to provide a measure of accuracy of a

parameter estimate of a given machine learning method.

Proprietary
Thiscontent.
file is ©University
meant forof personal
Arizona. All Rights
use by Reserved. Unauthorized use or distributiononly.
[email protected] prohibited."

Sharing or publishing the contents in part or full is liable for legal action.
Model Assessment

• The generalization performance of a machine learning method

relates to its prediction capability on independent test sets.

• Assessment of this performance is extremely important in practice,

since it guides the choice of the machine learning method or model.
[email protected]
DLZNK464L9

• Further, this gives us a measure of the quality of the ultimately

chosen model.

Proprietary
Thiscontent.
file is ©University
meant forof personal
Arizona. All Rights
use by Reserved. Unauthorized use or distributiononly.
[email protected] prohibited."

Sharing or publishing the contents in part or full is liable for legal action.
Model Assessment

• Test Error
• The average error that results from using a machine learning
method to predict the response on a new observation.
• The prediction error over an independent test sample.
[email protected]
DLZNK464L9
• Training Error
• The average loss over the training sample:

• Note: The training error rate can dramatically underestimate the test
error rate

Proprietary
Thiscontent.
file is ©University
meant forof personal
Arizona. All Rights
use by Reserved. Unauthorized use or distributiononly.
[email protected] prohibited."

Sharing or publishing the contents in part or full is liable for legal action.
Model Assessment
● As the model becomes more and more
complex, it uses the training data more and is
able to adapt to more complicated underlying
structures.

● Hence, there is a decrease in bias but an

[email protected]
DLZNK464L9 increase in variance.

● However, training error is not a good estimate

of the test error.

• Training error consistently decreases with model complexity.

• A model with zero training error is overfit to the training data and will typically
generalize poorly.
Proprietary
Thiscontent.
file is ©University
meant forof personal
Arizona. All Rights
use by Reserved. Unauthorized use or distributiononly.
[email protected] prohibited."

Sharing or publishing the contents in part or full is liable for legal action.
Model Assessment

● If we are in a data-rich situation, the best approach for both model selection and
model assessment is to randomly divide the dataset into three parts: training set,
validation set, and test set.

● The training set is used to fit the models. The validation set is used to estimate
[email protected]
DLZNK464L9
prediction error for model selection. The test set is used for assessment of the
prediction error of the final chosen model.

● A typical split: 50% for training, and 25% each for validation and testing.

Train Validation Test

Proprietary
Thiscontent.
file is ©University
meant forof personal
Arizona. All Rights
use by Reserved. Unauthorized use or distributiononly.
[email protected] prohibited."

Sharing or publishing the contents in part or full is liable for legal action.
[email protected]
DLZNK464L9 Validation Set Approach

Proprietary
Thiscontent.
file is ©University
meant forof personal
Arizona. All Rights
use by Reserved. Unauthorized use or distributiononly.
[email protected] prohibited."

Sharing or publishing the contents in part or full is liable for legal action.
Validation Set Approach

● Suppose that we would like to find a set of variables that give the lowest validation
error rate (an estimate of the test error rate).

● If we have a large data set, we can achieve this goal by randomly splitting the data
into separate training and validation data sets.
[email protected]
DLZNK464L9

● Then, we use the training data set to build each possible model and select the model
that gives the lowest error rate when applied to the validation data set.

Training Data Validation Data

Proprietary
Thiscontent.
file is ©University
meant forof personal
Arizona. All Rights
use by Reserved. Unauthorized use or distributiononly.
[email protected] prohibited."

Sharing or publishing the contents in part or full is liable for legal action.
The Validation process

123 n

[email protected]
DLZNK464L9
7 22 13 91

A random splitting into two halves: left part is training set, right
part is validation set

Proprietary
Thiscontent.
file is ©University
meant forof personal
Arizona. All Rights
use by Reserved. Unauthorized use or distributiononly.
[email protected] prohibited."

Sharing or publishing the contents in part or full is liable for legal action.
Validation Set Approach: Example Results

Resampling Methods-1

Uploaded by

Resampling Methods-1

Uploaded by

sainathgunda99@gmail.

• Example: We can estimate the variability of a linear regression fit by

• Model Assessment: having chosen a final model, estimating its prediction

• Model Selection: estimating the performance of different models in

• Used most commonly to provide a measure of accuracy of a

• The generalization performance of a machine learning method

• Assessment of this performance is extremely important in practice,

• Further, this gives us a measure of the quality of the ultimately

● Hence, there is a decrease in bias but an

● However, training error is not a good estimate

• Training error consistently decreases with model complexity.

Train Validation Test

Training Data Validation Data

• Left Panel: Validation error estimates for a single split into

• Only a subset of the observations (those in the training set) are

• To finesse the problem, K-fold cross-validation uses part of the available

• We use this method because LOOCV (~next few

• We randomly divide the data set of into K folds

• By averaging the K estimates of the test error, we get an estimated

• 10-fold CV run nine separate times, each with a different random

• LOOCV sometimes useful, but typically doesn’t shake up the

• Performing LOOCV multiple times always yields the same results

• With exceptions, LOOCV is computationally intensive because the

• Which is better, LOOCV or K-fold CV?

• We tend to use K-fold CV with K = 5 or K = 10, as these values have been

• Left Panel: LOOCV Error Curve

• CV uses sampling without replacement

• From the bootstrap sampling we can

distribution of S(Z), for example, its

• A graphical illustration of the

• Each bootstrap data set contains n

• Each bootstrap data set is used to

• In more complex data situations, figuring out the appropriate way to

• For example, if the data is a time series, we cannot simply sample

• However, we can instead create blocks of consecutive observations,

• Although the bootstrap is used primarily to obtain standard errors of an

• In cross-validation, each of the K validation folds is distinct from the other K

• The leave-one-out bootstrap estimate of prediction error is defined by:

You might also like