HYPERPARAMETER TUNING

" Hyperparameter tuning is the process of selecting the optimal

values for a machine learning model's hyperparameters.
" Hyperparameters are settings that control the learning process
of the model, such as the learning rate, the number of neurons
in a neural network, or the kernel size in a support vector
machine.

" The goal of hyperparameter tuning is to find the values that lead
to the best performance on a given task.

HYPERPARAMETER TUNING
TECHNIQUES
Models can have many hyperparameters and finding the best
combination of parameters can be treated as a search problem.
The two best strategies for Hyperparameter tuning are:
" GridSearchCV
Randomized SearchCV
Bayesian Optimization

GRIDSEARCHCV

" Grid search can be considered as a brute force" approach to

hyperparameter optimization.
We fit the model using all possible combinations after creating a
grid of potential discrete hyperparameter values.
Drawback:

" GridSearchCV will go through all the intermediate combinations

of hyperparameters which makes grid search computationally
very expensive.
 Regression Classifier model, with different sets of values. The
grid search technique will construct many versions of the model
with all possible combinations of hyperparameters and will
return the best one.

" As in the image, for C= [0.1, 0.2, 0.3, 0.4, 0.5] and Alpha = [0.1,
0.2,0.3, 0.4]. For a combination of C-0.3 and Alpha=0.2, the
performance Highest), therefore it
0.701 0.703 0.697 0.696
is selected. 0.699 0.702 0.698 0.702
0.721 0.726 0.713 0.703
0.706 0.705 0.704 0.701
C0.1 0.698 0.692 0.688 0.675
0.1 0.2 0.3

Alpha

-RandomizedSearchCV
The random search method selects values at random as
opposed to the grid search method's use of a predetermined set
of numbers.

Every iteration,random search attempts a different set of

hyperparameters and logs the model's performance.
" It returns the combination that provided the best outcome after
several iterations.

This approach reduces unnecessary computation.

" The advantage is that, in most cases, a random search will
produce a comparable result faster than a grid search.

Bayesian Optimization
Grid search and random search are often inefficient because
they evaluate many unsuitable hyperparameter combinations
without considering the previous iterations' results.
Bayesian optimization, on the other hand, treats the search for
optimal hyperparameters as an optimization problem.
" It considers the previous evaluation results when selecting the
next hyperparameter combination and applies a probabilistic
function to choose the combination that will likely yield the best
results.

This method discovers a good hyperparameter combination in

relatively few iterations.

P(score(y) |hyperparameters(*))
The Bayesian optimization model is complex to implement, but
offthe-shelf libraries like Ray Tune can simplify the process.
" It's worth using this type of model because it finds an adequate
hyperparameter combination in relatively few iterations.
 Advantages of Hyperparameter tuning:
" Improved model performance
" Reduced overfitting and underfitting
" Enhanced model generalizability
Optimized resource utilization
" Improved model interpretability
Disadvantages of Hyperparameter tuning:
Computational cost
" Time-consuming process
" Risk of overfitting
" No guarantee of optimal performance
" Requires expertise
" Data Augmentation
Data Augmentation is a technique used in Neural Networks to
artificially increase the size of the training dataset by applying
transformations to existing data.
This helps in reducing overfitting, improving generalization, and
enhancing model robustness.
Example:
" In self-driving cars, road signs may appear in different lighting
conditions, angles, and weather conditions. Augmenting training
images helps the model recognize signs in all possible scenarios.
Augmenting data with noise, contrast changes, and rotations ensures
it learns to detect diseases consistently.

How it works
Data augmentation makes
small changes to the
original data, such as
rotating, flipping, or
adjusting the color
" These changes increase the
Enlarge your Dat diversity of the training set
" The model is exposed to a
broader range of scenarios
which helps it generalize
better
Benefts

Prevents overfitting: Data augmentation helps prevent overfitting. which is when a model performs
well on training data but not on unseen data

Improves model accuracy: Data augmentation can help improve the accuracy of predictions
Reduces operational costs: Data augmentation can reduce the time and expense of data collection and labeling

Common data augmentation techniques

" Rotation, Translation, Scaling, Flipping, Shearing, Zooming,
Brightness adjustment, Contrast adjustment, and Noise addition.
Deep learning frameworks
Many deep learning frameworks, such as PyTorch, Keras, and
Tensorflow, have functions for data augmentation.
 Implementing Data Augmentation in Python
" Step 1: Import Libraries

" Import the necessary libraries for data augmentation, image

processing, and visualization.
" Step 2: Define the ImageDataGenerator
Create an instance of ImageDataGenerator with specified augmentation
parameters such as rotation,
width shift, height shift, shear, zoom, and horizontal flip.
Step 3: Load an Example
Image
Load an image from the CIEAR-10dataset to use as an example for
augmentation.
Display the original image using matplotlib.

Step 4: Reshape the Image

Reshape the image to include a batch dimension, which
is required
by the flow method of ImageDataGenerator.
" Step 5: Generate Augmented Images
Use the flow method to generate batches of augmented images.
Collect a specified number (4 in this case) of augmented images.
Step 6: Display Augmented Images
Step 7: Print the Matrix Format of the Images

Applications of Data Augmentation

" Data augmentation is widely used in various applications,
including:
" Image Classification: Enhancing the dataset to improve the
accuracy of image classification models.
" Object Detection: Making object detection models more robust
to variations in object appearance.
Segmentation: Improving the performance of image
segmentation models by providing diverse training examples.
Medical Imaging: Augmenting medical images to enhance the
performance of models in detecting and diagnosing diseases.
Generalization Gap:

The generalization gap in learning refers to the difference

between a model's performance on the training data and its
performance on unseen test data.
Definition:

Generalization Gap-Training Error-Test

" Asmall gap suggests good generalization, while a large gap
indicates overfitting.

Reducing the Generalization Gap:

" Regularization (L1/L2, dropout, early stopping) to prevent
overfitting.
Increasing Dataset Size to improve representation of real
World scenarios.

" Data Augmentation to artificially increase data diversity.

" Better Model Selection by choosing architectures with the right
complexity.
 OVERFITTING AND UNDERFITTING

Overfitting in Machine Learning

Overfitting occurs when a model learns too much from training data,
including noise and outliers, making it perform well on training data
but poorly on new data.
Example: Acomplex curve that fits every training point but fails to
generalize. It's like a student who memorizes answers instead of
understanding concepts.
Causes of Overfitting:
" High variance and low bias
Excessively complexmodel
Tnsnfficient training data

Underfitting in Machine Learning

Underfitting occurs when a model is too simple to capture the patterns in
data,
leading to poor performance on both training and testing data.
Example: Fitting a straight line to data that follows a curve, missing
important patterns.
It's like a student who doesn't study enough and struggles in both practice
and real exams.

Causes of Underfitting:

" Overly simple model

Inadequate input features
" Insufficient training data
P Excessive regularization
Unscaled features

Bias and variance are two key sources of error in machine

learning that affect model performance and generalization.
" Bias occurs when a model is too simple and fails to capture the
complexity of the data, leading to underfitting. High bias results
in poor performance on both training and testing data. Example:
A linear regression model used for a non-linear dataset.
" Variance occurs when a model learns too much from the
training data, including noise, leading to overfitting. High
variance results in good training performance but poor
generalization to new data.

" Let's visually understand the concept of underfitting. proper

fitting, and overfitting.

Size Size Size

High Bias Low Bias, Low Variance High Variance

(Underfitting) (Goodfitting) (Overfitting) DG
 Underfitting : Straight line trying to fit a curved dataset but
cannot capture the data's patterns, leading to poor
performance on both training and test sets.
Overfitting: Asquiggly curve passing through all training points,
failing to generalize performing well on training data but
poorly on test data.
Appropriate Fitting: Curve that follows the data trend without
overcomplicating to capture the true patterns in the data.

Techniques to Reduce Underfitting

" Increase model complexity.
" Increase the number of features, performing feature
engineering.
" Remove noise from the data.

" Increase the number of epochs or increase the duration of

training toget better results.

Techniques to Reduce Overfitting

Improving the quality of training data reduces overfitting by focusing
on meaningful patterns, mitigate the risk of fitting the noise or
irrelevant features.
" Increase the training data can improve the model's ability to
generalize to unseen data and reduce the likelihood of overfitting.
" Reduce model complexity.
" Early stopping during the training phase (have an eye over the loss
Over the training period as soon as loss begins to increase stop
training).
Ridge Regularization and Lasso Regularization
" Use dropout for neural networks to tackle overfitting.
" Hyper parameter Tunning
Regularization
Generalisation Gap
" Data Augmentation
Underfitting
" Increase model complexity.
" Increase the number of features, performing featureengineering
" Remnove noise from the data.

overfitting
" Reduce model complexity.
Early stopping