Scribd
Upload
2 views

Module 6_ML

The document outlines performance evaluation techniques in machine learning, focusing on the importance of training, validation, and test splits for model assessment. It explains the process of cross-validation, including various methods such as k-fold and leave-one-out, to ensure models generalize well to unseen data. Additionally, it discusses the significance of hyperparameter tuning and provides formulas for calculating accuracy, recall, and precision.

Uploaded by

Shashank Tyagi
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
2 views

Module 6_ML

The document outlines performance evaluation techniques in machine learning, focusing on the importance of training, validation, and test splits for model assessment. It explains the process of cross-validation, including various methods such as k-fold and leave-one-out, to ensure models generalize well to unseen data. Additionally, it discusses the significance of hyperparameter tuning and provides formulas for calculating accuracy, recall, and precision.

Uploaded by

Shashank Tyagi
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 30

Performance Evaluation Techniques-

Training, Testing & Validation splits


In machine learning, a "Training, Testing, and Validation split" refers to the
process of dividing a dataset into three separate subsets:
The training set (used to train the model)
 The validation set (used to fine-tune model parameters and prevent overfitting)
 The test set (used to evaluate the final model performance on unseen
data)essentially allowing to assess how well your model generalizes to new data
and avoids overfitting to the training data alone.
• Training Set:
This is the largest portion of the data used to train the model, where the algorithm
learns the patterns and relationships within the data.
• Validation Set:
This subset is used to evaluate the model's performance during the training
process, allowing you to adjust hyperparameters and prevent overfitting by
monitoring how well the model performs on data it hasn't seen before.
• Test Set:
Once the model is considered optimal based on the validation set, the final
performance is assessed on the completely unseen test set, giving a more accurate
picture of how the model will perform on new data.
How to use the Train-Test-Validation
split
1. Data Preparation: Clean and pre-process your entire dataset.
2. Splitting the data: Randomly divide the dataset into the training, validation,
and test sets, with a common ratio being 80% for training, 10% for validation,
and 10% for testing.
3. Model Training: Train the model using the training data.
4. Hyperparameter Tuning: Use the validation set to adjust hyperparameters of
the model to optimize its performance.
5. Final Evaluation: Once the model is tuned, evaluate its performance on the test
set to get an unbiased assessment of its generalization ability.
Cross Validation
Cross validation is a technique used in machine learning to evaluate the
performance of a model on unseen data.
It involves dividing the available data into multiple folds or subsets, using one
of these folds as a validation set, and training the model on the remaining
folds.
 This process is repeated multiple times, each time using a different fold as the
validation set.
 Finally, the results from each validation step are averaged to produce a more
robust estimate of the model’s performance.
Cross validation is an important step in the machine learning process and helps
to ensure that the model selected for deployment is robust and generalizes
well to new data.
The main purpose of cross validation is to prevent overfitting, which occurs when
a model is trained too well on the training data and performs poorly on new,
unseen data.
By evaluating the model on multiple validation sets, cross validation provides a
more realistic estimate of the model’s generalization performance, i.e., its ability
to perform well on new, unseen data.
Types of Cross-Validation
There are several types of cross validation techniques, including k-fold cross
validation, leave-one-out cross validation, and Holdout validation, Stratified
Cross-Validation.
 The choice of technique depends on the size and nature of the data, as well as
the specific requirements of the modeling problem.
Holdout Validation
In Holdout Validation, we perform training on the 50% of the given dataset and
rest 50% is used for the testing purpose. It’s a simple and quick way to evaluate a
model.
The major drawback of this method is that we perform training on the 50% of the
dataset, it may possible that the remaining 50% of the data contains some
important information which we are leaving while training our model i.e. higher
bias.
leave-one-out cross validation
In this method, we perform training on the whole dataset but leaves only one
data-point of the available dataset and then iterates for each data-point.
In LOOCV, the model is trained on n−1 samples and tested on the one omitted
sample, repeating this process for each data point in the dataset.
K-Fold Cross Validation
• In K-Fold Cross Validation, we split the dataset into k number of subsets (known
as folds) then we perform training on the all the subsets but leave one(k-1) subset
for the evaluation of the trained model.
• In this method, we iterate k times with a different subset reserved for testing
purpose each time.
K-fold cross validation in machine learning cross-validation is a powerful
technique for evaluating predictive models in data science.
 It involves splitting the dataset into k subsets or folds, where each fold is used
as the validation set in turn while the remaining k-1 folds are used for training.
 This process is repeated k times, and performance metrics such as accuracy,
precision, and recall are computed for each fold. By averaging these metrics, we
obtain an estimate of the model’s generalization performance.
 This method is essential for model assessment, selection, and hyperparameter
tuning, offering a reliable measure of a model’s effectiveness.
In each set (fold) training and the test would be performed precisely once
during this entire process. It helps us to avoid overfitting.
Accuracy =45+95/150
=93.33%
• Recall =TP/TP+FN
=95/95+5 =95%
Precision=TP/TP+FP
=95%

You might also like