Vanishing Gradient Problem in Deep Learning:

Understanding, Intuition, and Solutions

medium.com/@amanatulla1606/vanishing-gradient-problem-in-deep-learning-understanding-intuition-and-
solutions-da90ef4ecb54

Amanatullah 12 tháng 6, 2023

Top highlight

Amanatullah

Introduction
Deep Learning has rapidly become an integral part of modern AI applications, such as
computer vision, natural language processing, and speech recognition. The success of
deep learning is attributable to its ability to automatically learn complex patterns from
large amounts of data without explicit programming. Gradient-based optimization, which
relies on backpropagation, is the primary technique used to train deep neural networks
(DNNs).

Understanding the Vanishing Gradient Problem

1/8
One of the major roadblocks in training DNNs is the vanishing gradient problem, which
occurs when the gradients of the loss function with respect to the weights of the early
layers become vanishingly small. As a result, the early layers receive little or no updated
weight information during backpropagation, leading to slow convergence or even
stagnation. The vanishing gradient problem is mostly attributed to the choice of activation
functions and optimization methods in DNNs.

Vanishing gradient problems generally occurs when the value of partial derivative of loss
function w.r.t. to weights are very small.The complexity of Deep Neural Networks also
causes VD problem.

Role of Activation Functions and Backpropagation

Activation functions, such as sigmoid and hyperbolic tangent, are responsible for
introducing non-linearity into the DNN model. However, these functions suffer from the
saturation problem, where the gradients become close to zero for large or small inputs,
contributing to the vanishing gradient problem. Backpropagation, which computes the
gradients of the loss function with respect to the weights in a chain rule fashion,
exacerbates the problem by multiplying these small gradients.

During backpropagation we calculate loss (y-y_hat) and update our weights using
partial derivatives of loss function but it follows chain rule and to update w11 there will
be a sequence of chain with multiplication of smaller values of gradient descent and
learning rate which in result no change in weights .

How to recognise Vanishing Gradient Problem

2/8
1. Calculate loss using Keras and if its consistent during epochs that means Vanishing
Gradient Problem.
2. .

How to identify Vanishing Gradient Problem. 😊

3/8
#importing libraries

import matplotlib.pyplot as plt

import numpy as np
import pandas as pd
import tensorflow as tf
import keras
from sklearn.datasets import make_moons #classification datasets
from sklearn.model_selection import train_test_split
from keras.layers import Dense
from keras.models import Sequential

X,y = make_moons(n_samples=250, noise=0.05, random_state=42)

plt.scatter(X[:,0],X[:,1], c=y, s=100)

plt.show()

model = Sequential()

#constructing a complex neural network with two inputs an nine layers with 10
nodes

model.add(Dense(10,activation='sigmoid',input_dim=2))
model.add(Dense(10,activation='sigmoid'))
model.add(Dense(10,activation='sigmoid'))
model.add(Dense(10,activation='sigmoid'))
model.add(Dense(10,activation='sigmoid'))
model.add(Dense(10,activation='sigmoid'))
model.add(Dense(10,activation='sigmoid'))
model.add(Dense(10,activation='sigmoid'))
model.add(Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])

model.get_weights()[0]

old_weights = model.get_weights()[0]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20,

random_state=42)

4/8
model.fit(X_train, y_train, epochs = 100)

new_weights = model.get_weights()[0]

Mathematical Intuition behind the Vanishing Gradient Problem

The vanishing gradient problem arises from the product of the Jacobian matrices of the
activation functions and the weights of the DNN layers. Each layer contributes a Jacobian
matrix to the product, and the product becomes rapidly small as the number of layers
increases. The problem is more severe when the Jacobian matrices have small
eigenvalues, which happens when the activation functions are near-saturated or the
weights are poorly initialized.

Chain rule applied when finding the partial derivative of loss function w.r.rt. to b1

Reducing Complexity
The problem of vanishing gradients can be mitigated by reducing the complexity of the
DNN model. Complexity reduction can be achieved by reducing the number of layers or
the number of neurons in each layer. While this can alleviate the vanishing gradient
problem to some extent, it would also lead to reduced model capacity and performance.
Impact of Network Architecture
To preserve model capacity while mitigating the vanishing gradient problem, researchers
have explored various network architectures, such as skip connections, convolutional
neural networks (CNNs), and recurrent neural networks (RNNs). These architectures are
designed to facilitate better gradient flow and information propagation across layers,
thereby reducing the vanishing gradient problem.
Trade-off between Model Complexity and the Vanishing Gradient Problem

5/8
There is a trade-off between model complexity and the vanishing gradient problem. While
a more complex model might lead to better performance, it is also more prone to the
vanishing gradient problem due to the increased depth and non-linearity. Finding a
balance between model complexity and gradient flow is critical in building successful
DNN models.

Using ReLU Activation Function

Rectified Linear Unit (ReLU) is a popular activation function that has gained popularity
due to its simplicity and effectiveness in DNNs. ReLU has a non-saturating activation
function, which ensures that the gradients flow freely across the network.
Benefits of ReLU
ReLU addresses the vanishing gradient problem by preventing gradient saturation. ReLU
has been shown to significantly improve the convergence speed and accuracy of DNNs.
Additionally, ReLU has computational advantages over other activation functions, making
it a popular choice in practice.
Examples and Empirical Evidence of ReLU’s Effectiveness
ReLU has been widely used in state-of-the-art DNN models, such as ResNet, Inception,
and VGG. Empirical evidence shows that ReLU achieves better performance than other
activation functions, such as sigmoid and hyperbolic tangent.

Batch Normalization
Batch normalization is another technique that has shown great success in mitigating the
vanishing gradient problem and improving the training of DNNs.
Definition and Role of Batch Normalization
Batch normalization involves normalizing the input to each layer to have zero mean and
unit variance. This improves the gradient flow and allows faster convergence of the
optimization algorithm. Additionally, batch normalization acts as a regularization
technique, enabling the model to generalize better.
Benefits of Batch Normalization
Research has shown that batch normalization leads to faster convergence and improved
generalization of DNN models. Batch normalization is also robust to changes in
hyperparameters and improves the stability of the training process.
Proper Weight Initialization
Weight initialization plays a crucial role in training DNNs, even before the optimization
algorithm optimizes the weights.
Impact of Weight Initialization
Poor weight initialization can result in gradient explosion or vanishing, making the network
difficult or impossible to learn. Well-chosen weight initialization methods can alleviate the
vanishing gradient and enable faster convergence without sacrificing model capacity.
Techniques for Weight Initialization
Researchers have proposed various weight initialization techniques, such as Xavier and

6/8
He initialization, for different activation functions. These methods are designed to ensure
the proper scaling of the gradients during backpropagation, thereby facilitating better
gradient flow.

Residual Networks (ResNets)

Residual Networks, commonly referred to as ResNets, are a popular type of neural
network that have shown remarkable performance in deep learning.
Introduction and Architecture of ResNets
The architecture of ResNets introduces skip connections, where the output of one layer is
added to the input of another layer, facilitating better gradient flow and avoiding the
vanishing gradient problem. The skip connections also enable the network to learn
residual (or residual error) functions, making training deeper models easier.
Advantages of ResNets
ResNets have been used to achieve state-of-the-art performance in various domains,
such as image recognition, object detection, and natural language processing. The skip
connections in ResNets have shown to effectively alleviate the vanishing gradient
problem and make training easier.

Conclusion
The vanishing gradient problem is a significant challenge in training deep neural
networks. However, various techniques and approaches can mitigate this problem and
enable faster convergence and better performance. In this blog, we have explored the
role of activation functions, batch normalization, weight initialization, and ResNets in
mitigating the vanishing gradient problem. By experimenting with these techniques, we
can improve the training and performance of deep neural networks and advance the field
of AI.

FAQs: Vanishing Gradient Problem in Deep Learning

Q1: What is the vanishing gradient problem in deep learning?

A1: The vanishing gradient problem refers to the issue of diminishing gradients during the
training of deep neural networks. It occurs when the gradients propagated backward
through the layers become very small, making it difficult for the network to update the
weights effectively.

Q2: Why does the vanishing gradient problem occur?

A2: The vanishing gradient problem occurs due to the chain rule in backpropagation and
the choice of activation functions. When gradients are repeatedly multiplied during
backpropagation, they can exponentially decrease or vanish as they propagate through
the layers. Activation functions like the sigmoid function are prone to this problem
because their gradients can approach zero for large or small inputs.

Q3: What are the consequences of the vanishing gradient problem?

7/8
A3: The vanishing gradient problem can hinder the training of deep neural networks. It
slows down the learning process, leads to poor convergence, and prevents the network
from effectively capturing complex patterns in the data. The network may struggle to
update the early layers, limiting its ability to learn meaningful representations.

Q4: How can the vanishing gradient problem be overcome using reducing complexity?

A4: Reducing complexity involves adjusting the architecture of the neural network.
Techniques such as reducing the depth or width of the network can alleviate the vanishing
gradient problem. By simplifying the network structure, the gradients have a shorter path
to propagate, reducing the likelihood of vanishing gradients.

Q5: How does the ReLU activation function help overcome the vanishing gradient
problem?

A5: The rectified linear unit (ReLU) activation function helps overcome the vanishing
gradient problem by avoiding gradient saturation. ReLU replaces negative values with
zero, ensuring that the gradients flowing backward remain non-zero and do not vanish.
This promotes better gradient flow and enables effective learning in deep neural
networks.

Q6: What is batch normalization, and how does it address the vanishing gradient
problem?

A6: Batch normalization is a technique that normalizes the inputs to each layer within a
mini-batch. By normalizing the inputs, it reduces the internal covariate shift and helps
maintain a stable gradient flow. Batch normalization alleviates the vanishing gradient
problem by ensuring that the gradients do not vanish or explode during training.

Q7: How does proper weight initialization help mitigate the vanishing gradient problem?

A7: Proper weight initialization plays a crucial role in mitigating the vanishing gradient
problem. Initializing the weights in a careful manner, such as using Xavier or He
initialization, ensures that the gradients neither vanish nor explode as they propagate
through the layers. This facilitates better gradient flow and more stable training.

Q8: How do residual networks (ResNets) overcome the vanishing gradient problem?

A8: Residual networks (ResNets) address the vanishing gradient problem by utilizing skip
connections. These connections allow the gradients to bypass some layers and directly
flow to deeper layers, enabling smoother gradient propagation. ResNets enable the
training of very deep networks while mitigating the vanishing gradient problem.

8/8