Scribd
Upload
22 views25 pages

K Fold

Fml

Uploaded by

Dhvanil
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
22 views25 pages

K Fold

Fml

Uploaded by

Dhvanil
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 25

INTRODUCTION TO

MACHINE
LEARNING
Prof. Eduardo Bezerra
(CEFET/RJ)
[email protected]
2

MODEL EVALUATION
AND SELECTION
Visão Geral
3

 Model Evaluation
 Model Selection
4
Model Evaluation
Generalization error
5

 Error we would get if we could evaluate


the model over the entire population.
 Just because a hypothesis fits the training
set well does not mean that it is a good
hypothesis.
 The training error (empirical error) most likely
will be less than the generalization error.
Model Evaluation
6

 A hypothesis may present a low training


error, but still be poor (due to overfitting)
 Therefore, it is appropriate to evaluate the
performance of an algorithm on unseen
data.
 Model Evaluation refers to the process of
estimating the generalization error of a ML
model.
Evaluation metrics
7

 Accuracy, precision
 Precision, recall, F1 measure
 Squared errors
 Likelihood
 Posterior probability
 Cost/utility
 Margin
 KL divergence
 ....
Evaluation techniques
8

 Common techniques for estimating the


generalization performance of a ML
model:
 holdout method
 k-fold cross-validation
Holdout method (training/test)
9

 Randomly split original dataset into


separate training and test datasets:
 training dataset: used for training a

model M
 test dataset: used to estimate the

generalization error of M
 Can lead to a misleading estimate of the
generalization error if the test data is also
Holdout method (training/validation/test)
10

 A better way: separate the data into


three parts.
 training set: used to fit several models
 validation set: performance on this set if
used for model selection.
 test set: used only to estimate the
generalization error.
Typical proportions are 70%/15%/15% and
60%/20%/20% .
Holdout method (training/validation/test)
11

Source: Python Machine Lerning, 2nd ed., pp 191


k-fold cross-validation
12

 Randomly split the training dataset into


k folds without replacement
 k-1 folds are used for the model training
 one fold is used for performance
evaluation.
 Procedure is repeated k times so that we
obtain k models and performance
estimates.
k-fold cross-validation
13

Source: Python Machine Lerning, 2nd ed., pp 191


k-fold cross-validation
14

 Keep in mind that, as k increases:


 Bias in the estimation of the generalization
error decreases;
 Computational cost also increases.
 Empirical evidence shows that k=10 is a
good value for datasets of moderate size.
 For large sized datasets, k can be safely
decreased.
Other techniques
15

 Leave-one-out cross-validation
 Stratified k-fold cross-validation
 Bootstrap validation
Improvements on Cross-validation: The .632+ Bootstrap Method, B. Efron
and R. Tibshirani, Journal of the American Statistical Association, 92(438):
548-560, 1997

A Study of Cross-Validation and Bootstrap for Accuracy Estimation and


Model Selection, International Joint Conference on Artificial Intelligence
(IJCAI), 14 (12): 1137-43, 1995
Notation
16

 : training set size


 : training data set
 : test set size
 : test dataset
 : error calculated in test set (test
error)
17
Model Selection
Model Selection
18

 Hyperparameters are parameters that are not


directly learnt within models.
 Model selection is the process of selecting the
optimal hyperparameters values for a ML
algorithm.
 Examples:
 degree of the polynomial in polynomial regression;
 regularization term; learning rate;
 many more…
Model Selection - example
19

 Given many models, we apply a


systematic approach to identify the "best"
model.
 The “best” model is chosen by using
some quality measure (e.g., MSE in linear
regression).
 Let us see an example in the context of
polynomials with different degrees…
Model Selection - example (cont.)
20

 Suppose we want to choose the right


degree to fit a polynomial regression.
Model Selection - example (cont.)
21

 To choose one of these models, we


select the one with the least validation
error.
Model Selection - example (cont.)
22

 Finally, we calculated the test error on


the polynomial that produced the
smallest validation error.
Model selection - general procedure
23

1. Optimize the hyperparameters in Θ using the


training set for each degree of polynomial.
2. Find Θ*, the setting of hyperparameters with
the smallest error in the validation set.
3. Estimate the generalization by computing
error of the final model in the test set.
Hyperparameter search
24

 How to do search?
 There are two main approaches to search in
the space of hyperparameters:
 Grid search exhaustively considers all
hyperparameter combinations, for a set of
given values.
 Randomized search can sample a given
number of candidates from a hyperparameter
space with a specified distribution.
Hyperparameter search
25

You might also like