Deep Learning

B.Tech (B)
Ms. Punam R. Patil
Vision of the Department:
To provide prominent computer engineering education with socio-moral values.

Mission of the Department:

1. To provide state-of-the-art ICT based teaching-learning process.
2. To groom the students to become professionally sound computer engineers to
meet growing needs of industry and society.
3. To make the students responsible human being by inculcating ethical values

Program Educational Objectives(PEOs)

PEO1: To provide the foundation of lifelong learning skills for advancing their
careers being a professional entrepreneur and leader
PEO2: To develop computer professionals to fulfill Industry expectations
PEO3: To foster ethical and social values to be socially responsible human being.

2
 Deep Learning (PECO7031T)

Teaching Scheme Examination Scheme

Lectures : 03 Hrs./week Term Test : 15 Marks
Credits : 03 Teacher Assessment : 20 Marks
End Sem Exam : 65 Marks
Total Marks : 100 Marks

3
Course Objectives:
Prerequisite: Artificial Intelligence, Machine Learning.

Course Objectives:
• To understand Hyper parameter Tuning.
• To explore Deep Learning Techniques with different learning
strategies.
• To design Deep Learning Models for real time applications.

4
Course Outcomes

5
Blooms Taxonomy

6
7
8
9
Unit V: Adversarial Networks 10 Hrs

• Introduction to adversarial Networks

• Auto encoders (standard, denoising, contractive, etc.)
• Vibrational Auto encoders
• Generative Adversarial Networks
• Applications of Adversarial Networks.

10
Text Books:
1. Goodfellow I., Bengio Y., and Courville A., “Deep Learning”, MIT Press, 2016.
2. Umberto Michelucci, “Advanced Applied Deep Learning: Convolutional Neural Networks and Object Detection”,
2019.
3. Michael Nielsen (Goodreads Author), “Neural Networks and Deep Learning”, 2015.
4. Gulli and Kapoor, “TensorFlow 1.x Deep Learning Cookbook”, Packt Publishing, 2017.

Reference Books:
1. Yegnanarayana, B., “Artificial Neural Networks”, PHI Learning Pvt. Ltd, 2009.
2. Satish Kumar, “Neural Networks: A Classroom Approach”, Tata McGraw-Hill Education, 2004.
3. Rau´l Rojas, “Neural Networks: A Systematic Introduction”, 1996.
4. David Foster, “Generative Deep Learning: Teaching Machines to Paint, Write, Compose, and Play”, O’Relly, 2019.
5. Maxim Lapan, “Deep Reinforcement Learning HandsOn: Apply modern RL methods, with deep Q-networks, value
iteration, policy gradients, TRPO, AlphaGo Zero and more”, Packt Publishing, 2018.
6. Santanu Pattanaya K., “Pro Deep Learning with TensorFlow A Mathematical Approach to Advanced Artificial
Intelligence in Python”, APress, 2017.

11
Digital Material:

1. NPTEL:
Deep Learning, By Prof. Prabir Kumar Biswas, IIT Kharagpur.
https://onlinecourses.nptel.ac.in/noc22 cs22/preview
2. Coursera:
Deep Learning Specilization, By DeepLearning.AI
https://www.coursera.org/specializations/deep-learning#courses

12
13
List of Laboratory Experiments
• Suggested List of Experiments:
1. Building own Neural Network from scratch.
2. To implement EBPTA algorithm.
3. Understanding ANN using Tensor Flow.
4. Visualizing Convolutional Neural Network using Tensor Flow with
Keras Data.
5. Object detection using RNN using Tensor Flow.

14
6. Students are supposed to complete any one mini project not limited to
following list of projects.
• Sequence Prediction
• Object Detection
• Traffic Sign Classification
• Automatic Music Generation
• Music Genre Classification
• Text Summarizer
• Gender and Age Detection Using Voice
• Chatbot Using Deep Learning
• Neural Style Transfer
• Face Aging
• Driver Drowsiness Detection
• Language Translator
• Image Reconstruction
15
 Generative Adversarial Networks
• GANs, short for Generative Adversarial Networks, are a type of generative
model based on deep learning.
• They were first introduced in the 2014 paper “Generative Adversarial
Networks” by Ian Goodfellow and his team.
• GANs are a type of neural network used for unsupervised learning, meaning
they can create new data without being explicitly told what to generate.
• CNNs are used to classify images based on their labels. In contrast, GANs
can be divided into two parts: the Generator and the Discriminator.
• The Discriminator is similar to a CNN, as it is trained on real data and learns
to recognize what real data looks like. However, the Discriminator only has
two output values – 1 or 0 – depending on whether the data is real or fake.
16
Generative Adversarial Networks

• The Generator, on the other hand, is an inverse CNN. It takes a random

noise vector as input and generates new data based on that input.
• The Generator’s goal is to create realistic data that can fool the
Discriminator into thinking it’s real.
• The Generator keeps improving its output until the Discriminator can no
longer distinguish between real and generated data.
• Convolutional Neural Networks (CNNs) are the preferred models for both
the generator and discriminator in Generative Adversarial Networks
(GANs), typically used with image data. This is because the original concept
of GANs was introduced in computer vision, where CNNs had already
shown remarkable progress in tasks such as face recognition and object
detection.
https://www.leewayhertz.com/generative-adversarial-networks/ 17
Generative Adversarial Networks:GAN
• The training process for GANs can be considered a competition
between the Generator and the Discriminator. The Generator tries to
create better and more realistic data, while the Discriminator tries to
differentiate between real and fake data. Through this competition,
the Generator keeps improving until it can produce data
indistinguishable from real data.
• Overall, GANs are a generative model that can create new data
without being explicitly told what to generate. They consist of a
Generator that produces data and a Discriminator that evaluates its
realism. Through this adversarial training process, the Generator
learns to create more realistic data, resulting in a generative model
capable of producing high-quality output.
https://www.leewayhertz.com/generative-adversarial-networks/ 18
 Generator architecture
• The Generator takes random noise as input and generates synthetic
samples that are intended to be similar to the real training data. The
Generator typically consists of one or more deep neural networks, often
using convolutional layers to generate images or recurrent layers to
generate sequential data. The output of the Generator is fed to the
Discriminator, which is then trained to distinguish between the generated
samples and the real training data.

19
https://www.leewayhertz.com/generative-adversarial-networks/
Generator architecture

• The generator samples from the latent space and creates a

relationship between the latent space and the output. We then create
a neural network that maps from the input (latent space) to the
output (image for most examples).
• During the adversarial training process, connect the generator and
discriminator together in a model and train the generator to produce
images that are indistinguishable from real ones. Ultimately, the
generator produces the output images we see after the entire training
process. Training GANs directly refer to training the generator, and the
discriminator will need to train for a few epochs before beginning the
training process in most architectures.

https://www.leewayhertz.com/generative-adversarial-networks/ 20
Generator architecture

21
https://wiki.pathmind.com/generative-adversarial-network-gan
 Advantages of Generative Adversarial Network (GAN):
• Image synthesis: GANs can generate high-quality, photorealistic images,
which can be used in a variety of applications, such as entertainment, art, or
marketing.
• Text-to-Image synthesis: GANs can generate images from text descriptions,
which can be useful for generating illustrations, animations, or virtual
environments.
• Image-to-Image translation: GANs can translate images from one domain to
another, which can be used for colorization, style transfer, or data
augmentation.
• Anomaly detection: GANs can identify anomalies or outliers in data, which
can be useful for detecting fraud, network intrusions, or medical conditions.

22
Advantages of Generative Adversarial Network (GAN):

• Data augmentation: GANs can increase the size and diversity of a

dataset for training deep learning models, which can improve their
performance, robustness, or generalization.
• Video synthesis: GANs can generate high-quality, realistic video
sequences, which can be used in animation, film, or video games.
• Music synthesis: GANs can generate new, original music, which can be
used in music composition, performance, or entertainment.
• 3D model synthesis: GANs can generate high-quality, realistic 3D
models, which can be used in architecture, design, or engineering.

23
Disadvantages of Generative Adversarial Network
(GAN):
1. Training difficulty: GANs can be difficult to train and require a lot of
computational resources, which can be a barrier for some applications.
2. Overfitting: GANs can overfit to the training data, producing synthetic data
that is too similar to the training data and lacking diversity.
3. Bias and fairness: GANs can reflect the biases and unfairness present in the
training data, leading to discriminatory or biased synthetic data.
4. Interpretability and accountability: GANs can be opaque and difficult to
interpret or explain, making it challenging to ensure accountability,
transparency, or fairness in their applications.
5. Quality control: GANs can generate unrealistic or irrelevant synthetic data if
the generator and discriminator are not properly trained, which can affect
the quality of the results.
24
Applications of GAN:

• Generating Images
• Super Resolution
• Image Modification
• Photo realistic images
• Face Ageing

25
• Security: Artificial intelligence has proved to be a boon to many industries
but it is also surrounded by the problem of Cyber threats. GANs are proved
to be a great help to handle adversarial attacks. The adversarial attacks use
a variety of techniques to fool deep learning architectures. By creating fake
examples and training the model to identify them we counter these
attacks.
• Generating Data using GANs: Data is the most important key for any deep
learning algorithm. In general, the more is the data, the better is the
performance of any deep learning algorithm. But in many cases such as
health diagnostics, the amount of data is restricted, in such cases, there is
a need to generate good quality data, for which GANs are being used.

26
• Privacy-Preserving: There are many cases when our data needs to be kept
confidential. This is especially useful in defense and military applications.
We have many data encryption schemes but each has its own limitations,
in such a case GANs can be useful. Recently, in 2016, Google opened a new
research path on using GAN competitive framework for encryption
problems, where two networks had to compete in creating the code and
cracking it.
• Data Manipulation: We can use GANs for pseudo style transfer i.e.
modifying a part of the subject, without complete style transfer. For e.g. in
many applications, we want to add a smile to an image, or just work on the
eyes part of the image. This can also be extended to other domains such as
Natural Language Processing, speech processing, etc. For e.g. we can work
on some selected words of a paragraph without modifying the whole
paragraph.

27
Autoencoder
• Autoencoders are very useful in the field of unsupervised machine
learning. You can use them to compress the data and reduce its
dimensionality.
• The main difference between Autoencoders and Principle Component
Analysis (PCA) is that while PCA finds the directions along which you
can project the data with maximum variance, Autoencoders
reconstruct our original input given just a compressed version of it.

What are Autoencoders? Introduction to Autoencoders in Deep Learning | Simplilearn 28

Architecture of Autoencoder
• An Autoencoder is a type of neural network that can learn to reconstruct
images, text, and other data from compressed versions of themselves.
• An Autoencoder consists of three layers:
1. Encoder
2. Code
3. Decoder
• The Encoder layer compresses the input image into a latent space
representation. It encodes the input image as a compressed representation in
a reduced dimension.
• The compressed image is a distorted version of the original image.
• The Code layer represents the compressed input fed to the decoder layer.
• The decoder layer decodes the encoded image back to the original dimension.
The decoded image is reconstructed from latent space representation, and it is
reconstructed from the latent space representation and is a lossy
reconstruction of the original image.
What are Autoencoders? Introduction to Autoencoders in Deep Learning | Simplilearn 29
Fig: Illustration of a generic autoencoder

30
Types of autoencoders

• Undercomplete autoencoders
• Sparse autoencoders
• Contractive autoencoders
• Denoising autoencoders
• Variational Autoencoders (for generative modelling)

31
Under Complete Autoencoders
• Under complete autoencoders is an unsupervised neural network
that you can use to generate a compressed version of the input data.
• It is done by taking in an image and trying to predict the same image
as output, thus reconstructing the image from its compressed
bottleneck region.
• The primary use for autoencoders like these is generating a latent
space or bottleneck, which forms a compressed substitute of the
input data and can be easily decompressed back with the help of the
network when needed.
• This form of compression in the data can be modeled as a form
of dimensionality reduction.

What are Autoencoders? Introduction to Autoencoders in Deep Learning | Simplilearn 32

Under Complete Autoencoders

33
 Sparse Autoencoders
• Sparse autoencoders are similar to the undercomplete autoencoders in that they use the
same image as input and ground truth.
• However—The means via which encoding of information is regulated is significantly
different.
• Sparse autoencoders are controlled by changing the number of nodes at each hidden
layer.
• Since it is impossible to design a neural network with a flexible number of nodes at its
hidden layers, sparse autoencoders work by penalizing the activation of some neurons in
hidden layers.
• It means that a penalty directly proportional to the number of neurons activated is
applied to the loss function.
• As a means of regularizing the neural network, the sparsity function prevents more
neurons from being activated.

What are Autoencoders? Introduction to Autoencoders in Deep Learning | Simplilearn 34

Sparse Autoencoders

Autoencoders in Deep Learning: Tutorial & Use Cases [2023] (v7labs.com) 35

Sparse Autoencoders
• There are two types of regularizers used:
1. The L1 Loss method is a general regularizer we can use to add magnitude to the
model.

Where h represents the hidden layer, i represents the image in the minibatch,
and a represents the activation.

Autoencoders in Deep Learning: Tutorial & Use Cases [2023] (v7labs.com) 36

 Sparse Autoencoders
2. The KL-divergence method considers the activations over a collection
of samples at once rather than summing them as in the L1 Loss
method. We constrain the average activation of each neuron over this
collection.
Considering the ideal distribution as a Bernoulli distribution, we include
KL divergence within the loss to reduce the difference between the
current distribution of the activations and the ideal (Bernoulli)
distribution:
•

Autoencoders in Deep Learning: Tutorial & Use Cases [2023] (v7labs.com) 37

Contractive Autoencoders

• The input is passed through a bottleneck in a contractive autoencoder

and then reconstructed in the decoder. The bottleneck function is
used to learn a representation of the image while passing it through.
• The contractive autoencoder also has a regularization term to prevent
the network from learning the identity function and mapping input
into output.
• To train a model that works along with this constraint, we need to
ensure that the derivatives of the hidden layer activations are small
concerning the input.

What are Autoencoders? Introduction to Autoencoders in Deep Learning | Simplilearn 38

Denoising Autoencoders

• Have you ever wanted to remove noise from an image but didn't know
where to start? If so, then denoising autoencoders are for you!
• Denoising autoencoders are similar to regular autoencoders in that they
take an input and produce an output. However, they differ because they
don't have the input image as their ground truth. Instead, they use a noisy
version.
• It is because removing image noise is difficult when working with images.
• You'd have to do it manually. But with a denoising autoencoder, we feed
the noisy idea into our network and let it map it into a lower-dimensional
manifold where filtering out noise becomes much more manageable.
• The loss function usually used with these networks is L2 or L1 loss.

What are Autoencoders? Introduction to Autoencoders in Deep Learning | Simplilearn 39

Denoising Autoencoders
Advantages:
1. This type of autoencoder can extract important features and reduce the
noise or the useless features.
2. Denoising autoencoders can be used as a form of data augmentation, the
restored images can be used as augmented data thus generating
additional training samples.
Disadvantages:
1. Selecting the right type and level of noise to introduce can be challenging
and may require domain knowledge.
2. Denoising process can result into loss of some information that is needed
from the original input. This loss can impact accuracy of the output.

ML | Auto-Encoders – GeeksforGeeks
40
Variational Autoencoders

• Variational autoencoders (VAEs) are models that address a specific

problem with standard autoencoders.
• When you train an autoencoder, it learns to represent the input just
in a compressed form called the latent space or the bottleneck.
However, this latent space formed after training is not necessarily
continuous and, in effect, might not be easy to interpolate.
• Variational autoencoders deal with this specific topic and express
their latent attributes as a probability distribution, forming a
continuous latent space that can be easily sampled and interpolated.

41
Applications of autoencoders

1. Dimensionality reduction
• Undercomplete autoencoders are those that are used for
dimensionality reduction.
• These can be used as a pre-processing step for dimensionality
reduction as they can perform fast and accurate dimensionality
reductions without losing much information.
• Furthermore, while dimensionality reduction procedures like PCA can
only perform linear dimensionality reductions, undercomplete
autoencoders can perform large-scale non-linear dimensionality
reductions.
Autoencoders in Deep Learning: Tutorial & Use Cases [2023] (v7labs.com) 42
Applications of autoencoders

2. Image denoising
• Autoencoders like the denoising autoencoder can be used for
performing efficient and highly accurate image denoising.
• Unlike traditional methods of denoising, autoencoders do not search
for noise, they extract the image from the noisy data that has been
fed to them via learning a representation of it. The representation is
then decompressed to form a noise-free image.
• Denoising autoencoders thus can denoise complex images that
cannot be denoised via traditional methods.

Autoencoders in Deep Learning: Tutorial & Use Cases [2023] (v7labs.com) 43

Applications of autoencoders

3. Generation of image and time series data

• Variational Autoencoders can be used to generate both image and
time series data.
• The parameterized distribution at the bottleneck of the autoencoder
can be randomly sampled to generate discrete values for latent
attributes, which can then be forwarded to the decoder, leading to
generation of image data. VAEs can also be used to model time series
data like music.

Autoencoders in Deep Learning: Tutorial & Use Cases [2023] (v7labs.com) 44

Applications of autoencoders
4. Anomaly detection
• Undercomplete autoencoders can also be used for anomaly detection.
• For example—consider an autoencoder that has been trained on a specific
dataset P. For any image sampled for the training dataset, the autoencoder
is bound to give a low reconstruction loss and is supposed to reconstruct
the image as is.
• For any image which is not present in the training dataset, however, the
autoencoder cannot perform the reconstruction, as the latent attributes
are not adapted for the specific image that has never been seen by the
network.
• As a result, the outlier image gives off a very high reconstruction loss and
can easily be identified as an anomaly with the help of a proper threshold.

Autoencoders in Deep Learning: Tutorial & Use Cases [2023] (v7labs.com) 45