Unit III

Advanced Deep Learning

Dr.E.Poongothai
Assistant Professor
Department of Computational Intelligence
SRM Institute of Science and Technology
• Convolutional Networks,
• Convolutional operation,
• Pooling,
• Normalization,
• Applications in Computer Vision,
• Sequence Modelling,
• Recurrent Neural Networks,
• Difficulty in Training RNN,
• LSTM,
• GRU,
• Encoder Decoder
• architectures,
• Application,
• Spam classification, sentiment analysis
Convolutional Neural Network
(CNN)
is a type of Deep Learning neural network architecture commonly
used in Computer Vision.
Computer vision is a field of Artificial Intelligence that enables a
computer to understand and interpret the image or visual data.
• When it comes to Machine Learning, Artificial Neural Networks
perform really well. Neural Networks are used in various datasets
like images, audio, and text.
• Different types of Neural Networks are used for different purposes,
for example for predicting the sequence of words we use
Recurrent Neural Networks more precisely an LSTM,
• similarly for image classification we use Convolution Neural
networks.
Layers of Neural Network

1.Input Layers:
It’s the layer in which we give input to our model. The number of neurons in this
layer is equal to the total number of features in our data (number of pixels in the
case of an image).
2.Hidden Layer: The input from the Input layer is then fed into the hidden layer.
There can be many hidden layers depending on our model and data size. Each
hidden layer can have different numbers of neurons which are generally greater
than the number of features. The output from each layer is computed by matrix
multiplication of the output of the previous layer with learnable weights of that
layer and then by the addition of learnable biases followed by activation function
which makes the network nonlinear.
3.Output Layer: The output from the hidden layer is then fed into a logistic
function like sigmoid or softmax which converts the output of each class into the
probability score of each class.
• The data is fed into the model and output from each
layer is obtained from the above step is called
feedforward, we then calculate the error using an
error function, some common error functions are cross-
entropy, square loss error, etc.
• The error function measures how well the network is
performing. After that, we backpropagate into the
model by calculating the derivatives. This step is called
Backpropagation which basically is used to minimize
the loss.
• cross-entropy measures the difference between the
discovered probability distribution of a classification
model and the predicted values.
• The cross-entropy loss function is used to find the
optimal solution by adjusting the weights of a machine
learning model during training. The objective is to
minimize the error between the actual and predicted
outcomes. A lower cross-entropy value indicates better
performance.
Convolution Neural Network
• Convolutional Neural Network (CNN) is the extended
version of artificial neural networks (ANN) which is
predominantly used to extract the feature from the grid-
like matrix dataset. For example visual datasets like
images or videos where data patterns play an extensive
role.
CNN architecture
• Convolutional Neural Network consists of multiple layers
like the input layer, Convolutional layer, Pooling layer,
and fully connected layers.
How Convolutional Layers works

• Convolution Neural Networks or covnets are neural

networks that share their parameters. Imagine you have
an image. It can be represented as a cuboid having its
length, width (dimension of the image), and height (i.e
the channel as images generally have red, green, and
blue channels).
• Now imagine taking a small patch of this image and
running a small neural network, called a filter or kernel
on it, with say, K outputs and representing them
vertically.
• Now slide that neural network across the whole image,
as a result, we will get another image with different
widths, heights, and depths. Instead of just R, G, and B
channels now we have more channels but lesser width
and height. This operation is called Convolution. If the
patch size is the same as that of the image it will be a
regular neural network. Because of this small patch, we
have fewer weights.
• Now let’s talk about a bit of mathematics that is involved in the whole
convolution process.
• Convolution layers consist of a set of learnable filters (or kernels) having small
widths and heights and the same depth as that of input volume (3 if the input
layer is image input).
• For example, if we have to run convolution on an image with dimensions
34x34x3. The possible size of filters can be axax3, where ‘a’ can be anything like
3, 5, or 7 but smaller as compared to the image dimension.
• During the forward pass, we slide each filter across the whole input volume step
by step where each step is called stride (which can have a value of 2, 3, or even
4 for high-dimensional images) and compute the dot product between the kernel
weights and patch from input volume.
• As we slide our filters we’ll get a 2-D output for each filter and we’ll stack them
together as a result, we’ll get output volume having a depth equal to the
number of filters. The network will learn all the filters.
Layers used to build ConvNets

• A complete Convolution Neural Networks architecture is also known as covnets. A covnets is a sequence of layers, and every layer
transforms one volume to another through a differentiable function.
Types of layers: datasets
Let’s take an example by running a covnets on of image of dimension 32 x 32 x 3.
• Input Layers: It’s the layer in which we give input to our model. In CNN, Generally, the input will be an image or a sequence of
images. This layer holds the raw input of the image with width 32, height 32, and depth 3.
• Convolutional Layers: This is the layer, which is used to extract the feature from the input dataset. It applies a set of learnable filters
known as the kernels to the input images. The filters/kernels are smaller matrices usually 2×2, 3×3, or 5×5 shape. it slides over the
input image data and computes the dot product between kernel weight and the corresponding input image patch. The output of
this layer is referred as feature maps. Suppose we use a total of 12 filters for this layer we’ll get an output volume of dimension 32 x
32 x 12.
• Activation Layer: By adding an activation function to the output of the preceding layer, activation layers add nonlinearity to the
network. it will apply an element-wise activation function to the output of the convolution layer. Some common activation functions
are RELU: max(0, x), Tanh, Leaky RELU, etc. The volume remains unchanged hence output volume will have dimensions 32 x 32 x
12.
• Pooling layer: This layer is periodically inserted in the covnets and its main function is to reduce the size of volume which makes the
computation fast reduces memory and also prevents overfitting. Two common types of pooling layers are max pooling and average
pooling. If we use a max pool with 2 x 2 filters and stride 2, the resultant volume will be of dimension 16x16x12.
• Flattening: The resulting feature maps are flattened
into a one-dimensional vector after the convolution and
pooling layers so they can be passed into a completely
linked layer for categorization or regression.
• Fully Connected Layers: It takes the input from the
previous layer and computes the final classification or
regression task.
• Output Layer: The output from the fully connected
layers is then fed into a logistic function for
classification tasks like sigmoid or softmax which
converts the output of each class into the probability
score of each class.
• Advantages of Convolutional Neural Networks (CNNs):
1.Good at detecting patterns and features in images, videos, and audio signals.
2.Robust to translation, rotation, and scaling invariance.
3.End-to-end training, no need for manual feature extraction.
4.Can handle large amounts of data and achieve high accuracy.
• Disadvantages of Convolutional Neural Networks (CNNs):
1.Computationally expensive to train and require a lot of memory.
2.Can be prone to overfitting if not enough data or proper regularization is
used.
3.Requires large amounts of labeled data.
4.Interpretability is limited, it’s hard to understand what the network has
learned.
Normalization
Need for Batch Normalization in
CNN mode
• Batch Normalization in CNN addresses several challenges encountered during training.
There are following reasons highlight the need for batch normalization in CNN:
1.Addressing Internal Covariate Shift: Internal covariate shift occurs when the
distribution of network activations changes as parameters are updated during training.
Batch normalization addresses this by normalizing the activations in each layer,
maintaining consistent mean and variance across inputs throughout training. This
stabilizes training and speeds up convergence.
2.Improving Gradient Flow: Batch normalization contributes to stabilizing the gradient
flow during backpropagation by reducing the reliance of gradients on parameter scales.
As a result, training becomes faster and more stable, enabling effective training of
deeper networks without facing issues like vanishing or exploding gradients.
3.Regularization Effect: During training, batch normalization introduces noise to the
network activations, serving as a regularization technique. This noise aids in averting
overfitting by injecting randomness and decreasing the network’s sensitivity to minor
fluctuations in the input data.
Max pooling
Applications
• Decoding Facial Recognition
• Facial recognition is broken down by a convolutional
neural network into the following major components -
• Identifying every face in the picture
• Focusing on each face despite external factors, such as
light, angle, pose, etc.
• Identifying unique features
• Comparing all the collected data with already existing
data in the database to match a face with a name.
• A similar process is followed for scene labeling as well.
Analyzing Documents
• Convolutional neural networks can also be used for
document analysis. This is not just useful for
handwriting analysis, but also has a major stake in
recognizers. For a machine to be able to scan an
individual's writing, and then compare that to the wide
database it has, it must execute almost a million
commands a minute. It is said with the use of CNNs and
newer models and algorithms, the error rate has been
brought down to a minimum of 0.4% at a character
level, though it's complete testing is yet to be widely
seen.
Collecting Historic and
Environmental Elements
• CNNs are also used for more complex purposes such as
natural history collections. These collections act as key
players in documenting major parts of history such as
biodiversity, evolution, habitat loss, biological invasion,
and climate change.
Collecting Historic and
Environmental Elements
• CNNs are also used for more complex purposes such as
natural history collections. These collections act as key
players in documenting major parts of history such as
biodiversity, evolution, habitat loss, biological invasion,
and climate change.
Understanding Climate
• CNNs can be used to play a major role in the fight
against climate change, especially in understanding the
reasons why we see such drastic changes and how we
could experiment in curbing the effect. It is said that the
data in such natural history collections can also provide
greater social and scientific insights, but this would
require skilled human resources such as researchers
who can physically visit these types of repositories.
There is a need for more manpower to carry out deeper
experiments in this field.
Understanding Gray Areas
• Introduction of the gray area into CNNs is posed to
provide a much more realistic picture of the real world.
Currently, CNNs largely function exactly like a machine,
seeing a true and false value for every question.
However, as humans, we understand that the real world
plays out in a thousand shades of gray. Allowing the
machine to understand and process fuzzier logic will
help it understand the gray area we humans live in and
strive to work against. This will help CNNs get a more
holistic view of what human sees.
Example for CNN
Recurrent Neural Network(RNN)
• is a type of Neural Network where the output from the previous step is fed as input to
the current step.
• In traditional neural networks, all the inputs and outputs are independent of each other.
• Still, in cases when it is required to predict the next word of a sentence, the previous
words are required and hence there is a need to remember the previous words.
• Thus RNN came into existence, which solved this issue with the help of a Hidden Layer.
The main and most important feature of RNN is its Hidden state, which remembers
some information about a sequence. The state is also referred to as Memory
State since it remembers the previous input to the network.
• It uses the same parameters for each input as it performs the same task on all the
inputs or hidden layers to produce the output. This reduces the complexity of
parameters, unlike other neural networks.
• Types Of RNN
• There are four types of RNNs based on the number of
inputs and outputs in the network.
1.One to One
2.One to Many
3.Many to One
4.Many to Many
• One to One
• This type of RNN behaves the same as any simple
Neural network it is also known as Vanilla Neural
Network. In this Neural network, there is only one input
and one output.
How does RNN work?

• The Recurrent Neural Network consists of multiple fixed

activation function units, one for each time step. Each
unit has an internal state which is called the hidden
state of the unit. This hidden state signifies the past
knowledge that the network currently holds at a given
time step. This hidden state is updated at every time
step to signify the change in the knowledge of the
network about the past. The hidden state is updated
using the following recurrence relation:-
Issues or difficulties of Standard
RNNs
1.Vanishing Gradient: Text generation, machine translation,
and stock market prediction are just a few examples of the
time-dependent and sequential data problems that can be
modelled with recurrent neural networks. You will discover,
though, that the gradient problem makes training RNN
difficult.
2.Exploding Gradient: An Exploding Gradient occurs when a
neural network is being trained and the slope tends to grow
exponentially rather than decay. Large error gradients that
build up during training lead to very large updates to the
neural network model weights, which is the source of this
issue.
• Advantages and Disadvantages of Recurrent Neural Network
• Advantages
1.An RNN remembers each and every piece of information through time. It
is useful in time series prediction only because of the feature to
remember previous inputs as well. This is called Long Short Term Memory.
2.Recurrent neural networks are even used with convolutional layers to
extend the effective pixel neighborhood.
• Disadvantages
1.Gradient vanishing and exploding problems.
2.Training an RNN is a very difficult task.
3.It cannot process very long sequences if using tanh or relu as an
activation function.
• Applications of Recurrent Neural Network
1.Language Modelling and Generating Text
2.Speech Recognition
3.Machine Translation
4.Image Recognition, Face detection
5.Time series Forecasting
Variation Of Recurrent Neural Network (RNN)

• To overcome the problems like vanishing gradient and

exploding gradient descent several new advanced
versions of RNNs are formed some of these are as;
1.Bidirectional Neural Network (BiNN)
2.Long Short-Term Memory (LSTM)
Bidirectional Neural Network (BiNN)
• A BiNN is a variation of a Recurrent Neural Network in which the input
information flows in both direction and then the output of both
direction are combined to produce the input. BiNN is useful in
situations when the context of the input is more important such as
Nlp tasks and Time-series analysis problems.
Long Short-Term Memory (LSTM)
• Long Short-Term Memory works on the read-write-and-forget principle
where given the input information network reads and writes the most
useful information from the data and it forgets about the information
which is not important in predicting the output. For doing this three
new gates are introduced in the RNN. In this way, only the selected
information is passed through the network.
Long Short Term Memory Networks
Explanation

Prerequisites: Recurrent Neural Networks

Long Short Term Memory
Network(LSTM).
To solve the problem of Vanishing and Exploding Gradients in a Deep
Recurrent Neural Network, many variations were developed.
One of the most famous of them is the Long Short Term Memory
Network(LSTM).
In concept, an LSTM recurrent unit tries to “remember” all the past
knowledge that the network is seen so far and to “forget” irrelevant
data.
This is done by introducing different activation function layers called
“gates” for different purposes.
Each LSTM recurrent unit also maintains a vector called the Internal
Cell State which conceptually describes the information that was
chosen to be retained by the previous LSTM recurrent unit.
purposes for four different gates
1. Forget Gate(f): At forget gate the input is combined with the previous output to generate a fraction
between 0 and 1, that determines how much of the previous state need to be preserved (or in other words,
how much of the state should be forgotten). This output is then multiplied with the previous state. Note: An
activation output of 1.0 means “remember everything” and activation output of 0.0 means “forget
everything.” From a different perspective, a better name for the forget gate might be the “remember gate”
2. Input Gate(i): Input gate operates on the same signals as the forget gate, but here the objective is to
decide which new information is going to enter the state of LSTM. The output of the input gate (again a
fraction between 0 and 1) is multiplied with the output of tan h block that produces the new values that
must be added to previous state. This gated vector is then added to previous state to generate current
state
3. Input Modulation Gate(g): It is often considered as a sub-part of the input gate and much literature on
LSTM’s does not even mention it and assume it is inside the Input gate. It is used to modulate the
information that the Input gate will write onto the Internal State Cell by adding non-linearity to the
information and making the information Zero-mean. This is done to reduce the learning time as Zero-
mean input has faster convergence. Although this gate’s actions are less important than the others and
are often treated as a finesse-providing concept, it is good practice to include this gate in the structure of
the LSTM unit.
4. Output Gate(o): At output gate, the input and previous state are gated as before to generate another
scaling fraction that is combined with the output of tanh block that brings the current state. This output is
then given out. The output and state are fed back into the LSTM block.
Working of an LSTM recurrent
unit:
Gated Recurrent Unit
Networks
Gated Recurrent Unit Networks

• Gated Recurrent Unit (GRU) is a type of recurrent neural network (RNN)

that was introduced by Cho et al. in 2014 as a simpler alternative to
Long Short-Term Memory (LSTM) networks. Like LSTM, GRU can process
sequential data such as text, speech, and time-series data.
• The basic idea behind GRU is to use gating mechanisms to selectively
update the hidden state of the network at each time step. The gating
mechanisms are used to control the flow of information in and out of
the network. The GRU has two gating mechanisms, called the reset
gate and the update gate.
• The reset gate determines how much of the previous hidden state
should be forgotten, while the update gate determines how much of
the new input should be used to update the hidden state. The output of
the GRU is calculated based on the updated hidden state.
How do Gated Recurrent Units solve the problem of vanishing gradients?
• In Deep Learning, Many Complex problems can be
solved by constructing better neural network
architecture. The RNN(Recurrent Neural Network)
and its variants are much useful in sequence to
sequence learning. The RNN variant LSTM (
Long Short-term Memory) is the most used cell in
seq-seq learning tasks.
Encoder-Decoder Model

There are three main blocks in the encoder-decoder

model,
•Encoder
•Hidden Vector
•Decoder
The Encoder will convert the input sequence into a
single-dimensional vector (hidden vector). The decoder
will convert the hidden vector into the output sequence.
Encoder-Decoder models are jointly trained to maximize
the conditional probabilities of the target sequence given
the input sequence.
Encoder

• Multiple RNN cells can be stacked together to form the encoder.

RNN reads each inputs sequentially
• For every timestep (each input) t, the hidden state (hidden
vector) h is updated according to the input at that timestep X[i].
• After all the inputs are read by encoder model, the final hidden
state of the model represents the context/summary of the
whole input sequence.
• Example: Consider the input sequence “I am a Student” to be
encoded. There will be totally 4 timesteps ( 4 tokens) for the
Encoder model. At each time step, the hidden state h will be
updated using the previous hidden state and the current input.
Case study- spam classification
• https://medium.com/@azimkhan8018/email-spam-detection-with-
machine-learning-a-comprehensive-guide-b65c6936678b
What is Sentiment Analysis?

• Sentiment analysis is the process of classifying whether a block of text is

positive, negative, or neutral. The goal that Sentiment mining tries to gain is to
be analysed people’s opinions in a way that can help businesses expand. It
focuses not only on polarity (positive, negative & neutral) but also on emotions
(happy, sad, angry, etc.). It uses various Natural Language Processing
algorithms such as Rule-based, Automatic, and Hybrid.
• let’s consider a scenario, if we want to analyze whether a product is satisfying
customer requirements, or is there a need for this product in the market. We
can use sentiment analysis to monitor that product’s reviews. Sentiment
analysis is also efficient to use when there is a large set of unstructured data,
and we want to classify that data by automatically tagging it. Net Promoter
Score (NPS) surveys are used extensively to gain knowledge of how a
customer perceives a product or service. Sentiment analysis also gained
popularity due to its feature to process large volumes of NPS responses and
obtain consistent results quickly.
Why is Sentiment Analysis
Important?
• Sentiment analysis is the contextual meaning of words that indicates the social sentiment of a brand and also helps the business to
determine whether the product they are manufacturing is going to make a demand in the market or not.
• According to the survey,80% of the world’s data is unstructured. The data needs to be analyzed and be in a structured manner
whether it is in the form of emails, texts, documents, articles, and many more.
1. Sentiment Analysis is required as it stores data in an efficient, cost friendly.
2. Sentiment analysis solves real-time issues and can help you solve all real-time scenarios.
• Here are some key reasons why sentiment analysis is important for business:
• Customer Feedback Analysis: Businesses can analyze customer reviews, comments, and feedback to understand the sentiment
behind them helping in identifying areas for improvement and addressing customer concerns, ultimately enhancing customer
satisfaction.
• Brand Reputation Management: Sentiment analysis allows businesses to monitor their brand reputation in real-time.
By tracking mentions and sentiments on social media, review platforms, and other online channels, companies can respond
promptly to both positive and negative sentiments, mitigating potential damage to their brand.
• Product Development and Innovation: Understanding customer sentiment helps identify features and aspects of their products
or services that are well-received or need improvement. This information is invaluable for product development and innovation,
enabling companies to align their offerings with customer preferences.
• Competitor Analysis: Sentiment Analysis can be used to compare the sentiment around a company’s products or services with
those of competitors.
Businesses identify their strengths and weaknesses relative to competitors, allowing for strategic decision-making.
• Marketing Campaign Effectiveness
Businesses can evaluate the success of their marketing campaigns by analyzing the sentiment of online discussions and social
media mentions.
Positive sentiment indicates that the campaign is resonating with the target audience, while negative sentiment may signal the
need for adjustments.
What are the Types of Sentiment Analysis?

• Fine-Grained Sentiment Analysis

• This depends on the polarity base. This category can be designed as very positive, positive, neutral, negative, or very negative. The rating is
done on a scale of 1 to 5. If the rating is 5 then it is very positive, 2 then negative, and 3 then neutral.
• Emotion detection
• The sentiments happy, sad, angry, upset, jolly, pleasant, and so on come under emotion detection. It is also known as a lexicon method of
sentiment analysis.
• Aspect-Based Sentiment Analysis
• It focuses on a particular aspect for instance if a person wants to check the feature of the cell phone then it checks the aspect such as the
battery, screen, and camera quality then aspect based is used.
• Multilingual Sentiment Analysis
• Multilingual consists of different languages where the classification needs to be done as positive, negative, and neutral. This is highly
challenging and comparatively difficult.
• How does Sentiment Analysis work?
• Sentiment Analysis in NLP, is used to determine the sentiment expressed in a piece of text, such as a review, comment, or social media
post.
• The goal is to identify whether the expressed sentiment is positive, negative, or neutral. let’s understand the overview in general two steps:
• Preprocessing
• Starting with collecting the text data that needs to be analysed for sentiment like customer reviews, social media posts, news articles, or
any other form of textual content. The collected text is pre-processed to clean and standardize the data with various tasks:
• Removing irrelevant information (e.g., HTML tags, special characters).
• Tokenization: Breaking the text into individual words or tokens.
• Removing stop words (common words like “and,” “the,” etc. that don’t contribute much to sentiment).
• Stemming or Lemmatization: Reducing words to their root form.
What are the Approaches to Sentiment
Analysis?

• There are three main approaches used:

• Rule-based
• Over here, the lexicon method, tokenization, and parsing come in the rule-based. The approach is that
counts the number of positive and negative words in the given dataset. If the number of positive words is
greater than the number of negative words then the sentiment is positive else vice-versa.
• Machine Learning
• This approach works on the machine learning technique. Firstly, the datasets are trained and predictive
analysis is done. The next process is the extraction of words from the text is done. This text extraction
can be done using different techniques such as Naive Bayes, Support Vector machines,
hidden Markov model, and conditional random fields like this machine learning techniques are used.
• Neural Network
• In the last few years neural networks have evolved at a very rate. It involves using artificial neural
networks, which are inspired by the structure of the human brain, to classify text into positive, negative,
or neutral sentiments. it has Recurrent neural networks, Long short-term memory, Gated recurrent unit,
etc to process sequential data like text.
• Hybrid Approach
• It is the combination of two or more approaches i.e. rule-based and Machine Learning approaches. The
surplus is that the accuracy is high compared to the other two approaches.
Self Study for students
• Case study- Sentiment Analysis using Neural Network
Thank You