Module - 6

Deep Learning
 Disclaimer
The content is curated from
online/offline resources and used for
educational purpose only
Chapters for Discussion

Chapter - 1
Introduction to Deep Learning

Chapter - 2
Fundamentals of Neural Networks

Chapter - 3
Introduction to TensorFlow & Keras

Chapter - 4
Deep Learning Architectures & Training of Neural Networks
Chapters for Discussion

Chapter - 5
Convolutional Neural Networks (CNNs)

Chapter - 6
Transfer Learning & Pre-trained Models
 Chapter - 6

Transfer Learning & Pre-trained Models

Learning Objective
You will learn in this Chapter:
• To Learn transfer learning how it differs from
traditional training approaches.
• Understand the reusing pre-trained models and their
advantages in deep learning tasks.
• Understanding between feature extraction and fine-
tuning, including when and why to use each approach.
• To Identify and compare popular pre-trained CNN
architectures such as ResNet, VGG, MobileNet, and
EfficientNet.
• To understand the roles of feature extraction and fine-
tuning within the context of these CNN models.

Source
www.freepik.com
 Definition of Transfer Learning
​Transfer learning is a machine learning technique where
a model developed for one task is reused as the starting
point for a model on a second related task. This
approach leverages the knowledge gained from a
previous task to improve learning efficiency and
performance on a new task.

hmara jo model h vo pehle se hi usko train kiya hua h aur uski knowledge ko further ruse krre h for
other task aur agar hum usko koi similar task de toh prior knowledge ki madad se vo us task ko
complete kr skte hai

transfer learning = ek aisa model jo pehle se kisi particular task p trained hai
aur usi model ko hum dusre model p use kre

benefits :
(1) model k passs already data hai toh new task jo aaya h uske liye data kam chahiye hoga
(2) time save/efficicency
(3) better result as it is pre trained

Source
Image : https://www.v7labs.com/blog/transfer-learning-guide
 How it differs from traditional training approaches
TRANSFER LEARNING :
TRADITIONAL MACHINE { EK MODEL KO AAPNE
LEARNING : PEHLE SE TRAIN KIYA
HUA H AUR USKO
1.model scratch se yani 0 se DUSRE MODEL P USE
start hota hai. KRRE HO }
2.har chiz ekdam new tarike se
start krta hai 1.model already trained
3.jyada data jayega and time hota hai.hum us existing
consume hoga knowledge ko hum reuse
4. har baar model ko train krna krtre hai
hota h from beginning/scratch se
e.g = chote bache ko padhai 2.new model ko thora
sikhana adjust krna padta haai taki
previous model ko use kr
ske

Source
Image : https://www.linkedin.com/pulse/traditional-training-vs-transfer-learning-based-enjoyalgorithms
Use cases and real-world relevance WHERE TO USE...??

1.NLP (text k saath kam krega) = model use like BERT<GP-2,

FastText

2. COMPUTER VISION (read,learn krna particular imagae ko) =

use for img. recog., self driving car , blurry imgs. ko improve krna

3. SPEECHh/Voice recog. (speech/voice ko recog. krna )

= whatsapp = alexa
= chatbot = self driving car = voice to text app
= google translator =blurry img ko improve
krna efficeincy lana

Source
Image :https://medium.com/@skillcate/transfer-learning-intuition-to-implementation-5aae868361fd
Use cases and real-world relevance

Source
Image :https://medium.com/@skillcate/transfer-learning-intuition-to-implementation-5aae868361fd
Concept of Reusing Pre-trained Models
Transfer Learning Workflow

Source
Image : https://dataaspirant.com/transfer-learning/
Benefits of using pre-trained weights
• Faster Training = pre trained model already basics pr trained hota hai..sirf thoda sa train krna hota hai
• Better Performance with Less Data = data bohot kam hai to bhi performance acha de skta hai
• Lower Computational Costs = new model create krna thoda complex hota hai aur pre trained models ko use krna is very easy
• Improved Generalization = jo model h new task m bhi aacha performance dete hai
• Reusability and Flexibility = ek model ko different different task k liye reuse krna

• Easier Experimentation = pichle task k through new task ko aur ache se krna
 FINE TIUING :

Difference Between Feature Extraction and Fine-Tuning - layers ko unfreeze krte hai
- new data k according apne models ko phir se train kr skte ho isse new
model k andar k layers ko freeze krdete hai model ko aur jyada update kr dega
#freeze = dobara se train krna sirf last layer (classifier) ko chng krdete h according to new task
- will work on all layers
- different tasks h yoh it is best
use - when we have less data

Feature Extraction = models kse imp. colums ko extract krna Fine-Tuning

Utilizing a pre-trained model's layers as fixed feature Unfreezing some or all of the pre-trained model's
extractors; only the final layer is trained on new data. layers and retraining them on new data.

Typically, only the newly added classifier layers are trained. Both the newly added layers and some (or all) of
the pre-trained layers are trained.
Effective with limited datasets. Requires a larger dataset to avoid overfitting.

Generally shorter, as fewer parameters are updated. Longer, due to more parameters being retrained.

Lower, since most of the model remains unchanged. Higher, especially with small datasets, as more
parameters are adjusted.
May be lower if the pre-trained features don't align well with the Potentially higher, as the model can adapt more
new task. closely to the specifics of the new task.

Suitable when the new task is similar to the original task of the Ideal when the new task differs significantly from
pre-trained model. the original, necessitating more adaptation.
Why Transfer Learning?
• Reduced Training Time and Resources
• Improved Performance with Limited Data
• Enhanced Model Accuracy
• Accessibility to Complex Models

Source
www.freepik.com
Understanding Pre-trained CNN Models
• Reduced Training Time and Resources:
Leveraging pre-trained models significantly decreases the computational resources and time required for
training, as the models have already learned relevant features from extensive datasets.
• Improved Performance with Limited Data:
Pre-trained models can achieve high accuracy even when fine-tuned with relatively small datasets, making
them ideal for applications where data collection is challenging. ​
• Enhanced Generalization:
These models have learned to extract general features from a diverse set of images, which can improve their
performance on a variety of related tasks.
Common Pre-trained CNN Architectures USES =

• ResNet (Residual Networks):

- is a deep model(has many layerss) - jab aapko deep learning k complex task pr work krna ho
- trained in many layers - img. recog.and facial recog, imedical analysis
- skip connections ya shortcuts jo model ko help krte hai previous model ko new task mai use krne mai

Introduces residual learning to facilitate the training of very deep networks by addressing the vanishing gradient
problem.​
-vgg ek simple aur straigt forward CNN architecture hai jisme same pattern k multiple layers hote hai
• VGG (Visual Geometry Group): - ye small filters(3 by 3) ka use krta hai
- har layer similar dikhti hai isliye easy to understand hai

Known for its simplicity and uniform architecture, VGG networks are deep but utilize small convolutional filters.​
• MobileNet: - ek lightweighted aur fast CNN hai which is specifically designed for mobile devices and low power system
- ye limited memory aur CPU / GPU mai bhi acah perform krta hai

Designed for mobile and embedded vision applications, MobileNet models are efficient and suitable for devices
with limited computational capacity.​
- performance aur efficiency ko balance krta hai
• EfficientNet: - kam resource m maximum accuracy provide krt a hai
- can use in real time resources

Balances network depth, width, and resolution to achieve better performance with fewer parameters.​
Summary
You will learn in this Chapter:
• Learned transfer learning how it differs from traditional training
approaches.
• Understood the reusing pre-trained models and their advantages
in deep learning tasks.
• Understood between feature extraction and fine-tuning, including
when and why to use each approach.
• Identified and compared popular pre-trained CNN architectures
such as ResNet, VGG, MobileNet, and EfficientNet.
• Understood the roles of feature extraction and fine-tuning within
the context of these CNN models.

Source
www.freepik.com
Quiz
1. Which of the following is an advantage of transfer learning?
a) It requires a large amount of labeled data for the target task.​
b) It increases training time and computational resources.​
c) It can improve model performance on tasks with limited data.​
d) It eliminates the need for data preprocessing.

Answer: C
It can improve model performance on tasks with limited data.
Quiz
2. Which term describes the process of using a pre-trained model's
learned features as inputs to a new model for a different task?
a) Feature extraction​
b) Data augmentation​
c) Regularization​
d) Batch normalization

Answer: A
Feature extraction
Quiz
3. What is a potential drawback of transfer learning?
a) It always leads to overfitting on the target task.​
b) It cannot be used with deep learning models.​
c) The pre-trained model may not be suitable if the source and target
tasks are too dissimilar.​
d) It requires more labeled data than training from scratch.​

Answer: C
The pre-trained model may not be suitable if the source and target tasks are too dissimilar.​
Thank You