Machine Learning and Deep Learning

with R
Instructor: Babu Adhimoolam
 Resampling Methods
Learning objectives Cross-validation Methods
Bootstrapping Method
 Estimating the test error rate!

• The accuracy or worthiness of a machine learning model depends on how it can predict
the response y given x on a completely novel dataset (and not on the training dataset).

• We don’t have access to this novel dataset (in real life situations) and must devise methods
to estimate these error rates so that our model that we developed is dependable!

• We overcome this issue by holding out some data (often termed as test data) from
being used for model fitting during training and finally estimating the model error on
this hold-out dataset.
 A simple Validation Set approach

• It involves randomly dividing the observations into a training set and a validation set
(or hold-out set).

• The model is fit using the training set and the fitted model is used to predict the responses
in the validation set. The resulting error rate using the validation set provides an estimate of
test error rate.

Whole set of observations (x and y)

Training Set Test Set

 Drawbacks of the validation set approach
Figures show test error rates for predicting miles per gallon in auto data set from engine horsepower
(using both linear and higher order polynomials)

• Validation set approaches provide variable test error rate

• Validation set approaches also overestimate of the test error rate.

 Leave-one-out Cross-validation (LOOCV)
• In leave-one-out cross validation, like validation set approach there is a data partition
into training and test test.

• Unlike validation set approach, we leave only one observation for testing and all the
remaining (n-1) observations are used for training. The process is repeated until all the
remaining observations are included in testing one by one.

• If x1, x2, …, xn are the set of observations, then in the first round x1,y1is the test data and
remaining x2,…,xn are the training data and the test mean squared error MSE1 is given by
.
Similarly, in the second round with x2,y2 as test data and the remaining as training
dataset the test mean squared error MSE2 is .

• For n number of times, the LOOCV procedure, provides

the following average estimate of test error:
 Since LOOCV is generally fit on a
larger dataset than the validation set
approach, it tends not to
overestimate the test error.
Advantages of
LOOCV Since there is no randomness in
test/training split, performing LOOCV
multiple times will yield non varying
results of test error estimates.
Schematic illustration of Leave-one-out Cross-validation
(LOOCV)

n observations

1 = test, n-1st = training

2 = test, n-2nd = training
3 = test, n-3rd = training

n= test, n-nth = training

Figure shows test error rates for predicting miles per

gallon(y) in auto data set from engine horsepower(x)
 The k-Fold Cross-Validation
• It is an alternative procedure to LOOCV and involves randomly dividing the set of
observations into k-folds or groups of approximately equal size .

• First, the first group or fold is used for validation set and the model is fit on the remaining
k-1 groups or folds of data. On the next iteration, the second group or fold is used as validation
set and the procedure is repeated for remaining k-1 group and so on.

• The procedure results in k estimates of the test MSE(MSE1,MSE2,..,MSEk) and the average
of these k estimates is the k-fold cross-validation estimate:

• If k = n then the procedure is equivalent to LOOC. In general k = 5 or 10 is the preferred

approach as k = n or LOOVC is computationally intensive.
Schematic representation of k-fold cross validation
(k=5)

n observations

1st fold or group

2nd fold or group
3rd fold or group

4th fold or group

5th fold or group
Comparison of test error rates between validation set
method and k-fold cross-validation

Validation set method repeated 10 times Repetition of 10-fold method with different random
splits each time
 Comparison of true MSE and cross-validated MSE

Linear Regression (Orange) and Training MSE(Gray) and Test MSE(Red) True Test MSE in Blue
Two smoothing splines (Green and Blue) LOOCV estimate in black dashed line
10-fold estimate in Orange
 Comparison of true MSE and cross-validated MSE

Linear Regression (Orange) and Training MSE(Gray) and Test MSE(Red) True Test MSE in Blue
Two smoothing splines (Green and Blue) LOOCV estimate in black dashed line
10-fold estimate in Orange
 Comparison of true MSE and cross-validated MSE

Linear Regression (Orange) and Training MSE(Gray) and Test MSE(Red) True Test MSE in Blue
Two smoothing splines (Green and Blue) LOOCV estimate in black dashed line
10-fold estimate in Orange
 The true MSE is closely approximated by cross-validated
MSE

True Test MSE in Blue LOOCV estimate in black dashed line 10-fold estimate in Orange

Despite underestimating the true MSE all the CV estimates come closed to identifying the correct degree of flexibility
associated with lowest test error.
 The validation procedure fits a model to part of the
data and its estimation of test error is thus biased.
Since the LOOCV procedure fits the model to most of
the data it has lowest bias in test error estimation.
The k-fold has intermediate level of bias.
Bias Variance Trade
off with cross- In terms of variance, LOOCV estimates suffer from
validation methods high variance, in comparison to k-fold methods.

From bias-variance perspective, k-fold methods

using 5 or 10 as the number of folds, suffers neither
from excessive bias or variance.
 Like regression methods where we
used test MSE, in classification we
use the number of misclassified
items as we saw before.

The LOOCV estimate thus becomes

Cross-Validation
with classification
methods Where the error is defined as,

Similarly, we can quantify the error

estimates for k-fold and validation
methods.
 Cross validation with Classification methods
Note that the cross-validation errors despite
underestimating are true error are good in picking
up the degree of polynomials/flexibility of the model

Test error is Brown Training Error is Blue

10-fold cv is Black
Bayes Decision Boundary is Purple; Logistic regression with
linear, quadratic, cubic and quartic polynomials
Cross validation with Classification methods
The k-NN classifier with different values of
k- the neighbors

A comparison of k-NN decision

Boundary with k=1,10 and 100.
Bayes Decision boundary is shown
in purple.
Test error is Brown Training Error is Blue
10-fold cv is Black
 Nested Cross-validation

Parvandeh, S., Yeh, H. W., Paulus, M. P., & McKinney, B. A. (2020). Consensus features nested cross-validation.
Bioinformatics, 36(10), 3093-3098.
Varma, S., & Simon, R. (2006). Bias in error estimation when using cross-validation for model selection.
BMC bioinformatics, 7(1), 1-8.
 In boot strapping we repeatedly
sample data with replacement from
the original data.
Boot Strapping
The model is then fit on multiple
datasets that were created and the
statistic of interest is calculated
across all these datasets.
 Graphical illustration of sampling with replacement

Parameter generated from Parameter generated from

sampling with replacement of repeated sampling of multiple
Sampling with replacement of original data Z to estimate a single dataset. datasets from true population.
a parameter 𝝰 of interest.
 Boost Strapping can be used in
Combining Boot combination with k-fold or Nested
Strapping with cross- Cross-validation methods.
validation and
repeated designs The Cross-validation methods can be
repeated n (n=10,50,100) times in
repeated designs.