All Courses What do you want to learn?

Log in

AI & Machine Learning Tutorials Articles Ebooks Free Practice Tests On-demand Webinars Live Webinars

Home Resources AI & Machine Learning Deep Learning Tutorial for Beginners Power of Recurrent Neural Networks (RNN):
Revolutionizing AI

Power of Recurrent Neural Networks (RNN): Revolutionizing AI

Lesson 14 of 32 By Avijeet Biswal

Last updated on Aug 25, 2023 419536 11

Table of Contents
What Is a Neural Network?

What Are Recurrent Neural Networks (RNN)?

Why Recurrent Neural Networks?

How Does Recurrent Neural Networks Work?

Feed-Forward Neural Networks vs Recurrent Neural Networks

View More
Neural Networks is one of the most popular machine learning algorithms and also outperforms other algorithms in both
accuracy and speed. Therefore it becomes critical to have an in-depth understanding of what a Neural Network is, how
it is made up and what its reach and limitations are.

Your AI/ML Career is Just

Around The Corner!
AI Engineer Master's Program

EXPLORE PROGRAM

What Is a Neural Network?

A Neural Network consists of different layers connected to each other, working on the structure and function of a
human brain. It learns from huge volumes of data and uses complex algorithms to train a neural net.

Here is an example of how neural networks can identify a dog’s breed based on their features.

The image pixels of two different breeds of dogs are fed to the input layer of the neural network.

The image pixels are then processed in the hidden layers for feature extraction.

The output layer produces the result to identify if it’s a German Shepherd or a Labrador.

Such networks do not require memorizing the past output.

Several neural networks can help solve different business problems. Let’s look at a few of them.

F dF dN lN k U df lR i d Cl ifi i bl
 Feed-Forward Neural Network: Used for general Regression and Classification problems.

Convolutional Neural Network: Used for object detection and image classification.

Deep Belief Network: Used in healthcare sectors for cancer detection.

RNN: Used for speech recognition, voice recognition, time series prediction, and natural language processing.

Read More: What is Neural Network: Overview, Applications, and Advantages

Your AI/ML Career is Just

Around The Corner!
AI Engineer Master's Program

EXPLORE PROGRAM

What Are Recurrent Neural Networks (RNN)?

Recurrent Neural Networks (RNNs) are a type of artificial neural network designed to process sequences of data. They
work especially well for jobs requiring sequences, such as time series data, voice, natural language, and other activities.

RNN works on the principle of saving the output of a particular layer and feeding this back to the input in order to
predict the output of the layer.

Below is how you can convert a Feed-Forward Neural Network into a Recurrent Neural Network:
 Previous Next

Tutorial Playlist Fig: Simple Recurrent Neural Network

The nodes in different layers of the neural network are compressed to form a single layer of recurrent neural networks.
A, B, and C are the parameters of the network.
 Fig: Fully connected Recurrent Neural Network

Here, “x” is the input layer, “h” is the hidden layer, and “y” is the output layer. A, B, and C are the network parameters used
to improve the output of the model. At any given time t, the current input is a combination of input at x(t) and x(t-1). The
output at any given time is fetched back to the network to improve on the output.
 Fig: Fully connected Recurrent Neural Network

Now that you understand what a recurrent neural network is let’s look at the different types of recurrent neural networks.

Read More: An Ultimate Tutorial to Neural Networks

Your AI/ML Career is Just

Around The Corner!
AI Engineer Master's Program Your AI/ML Career is Just Around The Corner!

EXPLORE PROGRAM

Why Recurrent Neural Networks?

RNN were created because there were a few issues in the feed-forward neural network:

Cannot handle sequential data

Considers only the current input

Cannot memorize previous inputs

The solution to these issues is the RNN. An RNN can handle sequential data, accepting the current input data, and
previously received inputs. RNNs can memorize previous inputs due to their internal memory.

Your AI/ML Career is Just

Around The Corner!
AI Engineer Master's Program Your AI/ML Career is Just Around The Corner!

EXPLORE PROGRAM

How Does Recurrent Neural Networks Work?

In Recurrent Neural networks, the information cycles through a loop to the middle hidden layer.

Fig: Working of Recurrent Neural Network

The input layer ‘x’ takes in the input to the neural network and processes it and passes it onto the middle layer.

The middle layer ‘h’ can consist of multiple hidden layers, each with its own activation functions and weights and
biases. If you have a neural network where the various parameters of different hidden layers are not affected by the
previous layer, ie: the neural network does not have memory, then you can use a recurrent neural network.

The Recurrent Neural Network will standardize the different activation functions and weights and biases so that each
hidden layer has the same parameters. Then, instead of creating multiple hidden layers, it will create one and loop over
it as many times as required.
Feed-Forward Neural Networks vs Recurrent Neural Networks

A feed-forward neural network allows information to flow only in the forward direction, from the input nodes, through
the hidden layers, and to the output nodes. There are no cycles or loops in the network.

Below is how a simplified presentation of a feed-forward neural network looks like:

Fig: Feed-forward Neural Network

In a feed-forward neural network, the decisions are based on the current input. It doesn’t memorize the past data, and
there’s no future scope. Feed-forward neural networks are used in general regression and classification problems.

Your AI/ML Career is Just

Around The Corner!
AI Engineer Master's Program Your AI/ML Career is Just Around The Corner!

EXPLORE PROGRAM

Applications of Recurrent Neural Networks

Image Captioning
Image Captioning
RNNs are used to caption an image by analyzing the activities present.

Time Series Prediction

Any time series problem, like predicting the prices of stocks in a particular month, can be solved using an RNN.

Natural Language Processing

Text mining and Sentiment analysis can be carried out using an RNN for Natural Language Processing (NLP).

Machine Translation
Given an input in one language, RNNs can be used to translate the input into different languages as output.

Your AI/ML Career is Just

Around The Corner!
AI Engineer Master's Program Your AI/ML Career is Just Around The Corner!

EXPLORE PROGRAM
Advantages of Recurrent Neural Network

Recurrent Neural Networks (RNNs) have several advantages over other types of neural networks, including:

Ability To Handle Variable-Length Sequences

RNNs are designed to handle input sequences of variable length, which makes them well-suited for tasks such as
speech recognition, natural language processing, and time series analysis.

Memory Of Past Inputs

RNNs have a memory of past inputs, which allows them to capture information about the context of the input
sequence. This makes them useful for tasks such as language modeling, where the meaning of a word depends on the
context in which it appears.

Parameter Sharing
RNNs share the same set of parameters across all time steps, which reduces the number of parameters that need to be
learned and can lead to better generalization.

Non-Linear Mapping
RNNs use non-linear activation functions, which allows them to learn complex, non-linear mappings between inputs
and outputs.

Sequential Processing
RNNs process input sequences sequentially, which makes them computationally efficient and easy to parallelize.
Flexibility
RNNs can be adapted to a wide range of tasks and input types, including text, speech, and image sequences.

Improved Accuracy
RNNs have been shown to achieve state-of-the-art performance on a variety of sequence modeling tasks, including
language modeling, speech recognition, and machine translation.

These advantages make RNNs a powerful tool for sequence modeling and analysis, and have led to their widespread
use in a variety of applications, including natural language processing, speech recognition, and time series analysis.

Disadvantages of Recurrent Neural Network

Although Recurrent Neural Networks (RNNs) have several advantages, they also have some disadvantages. Here are
some of the main disadvantages of RNNs:

Vanishing And Exploding Gradients

RNNs can suffer from the problem of vanishing or exploding gradients, which can make it difficult to train the network
effectively. This occurs when the gradients of the loss function with respect to the parameters become very small or
very large as they propagate through time.

Computational Complexity
RNNs can be computationally expensive to train, especially when dealing with long sequences. This is because the
network has to process each input in sequence, which can be slow.

Difficulty In Capturing Long-Term Dependencies

Although RNNs are designed to capture information about past inputs, they can struggle to capture long-term
dependencies in the input sequence. This is because the gradients can become very small as they propagate through
time, which can cause the network to forget important information.

Lack Of Parallelism
RNNs are inherently sequential, which makes it difficult to parallelize the computation. This can limit the speed and
scalability of the network.

Difficulty In Choosing The Right Architecture

There are many different variants of RNNs, each with its own advantages and disadvantages. Choosing the right
architecture for a given task can be challenging, and may require extensive experimentation and tuning.

Difficulty In Interpreting The Output

The output of an RNN can be difficult to interpret, especially when dealing with complex inputs such as natural
language or audio. This can make it difficult to understand how the network is making its predictions.

These disadvantages are important when deciding whether to use an RNN for a given task. However, many of these
issues can be addressed through careful design and training of the network and through techniques such as
regularization and attention mechanisms.

Types of Recurrent Neural Networks

There are four types of Recurrent Neural Networks:

1. One to One

2. One to Many
 y

3. Many to One

4. Many to Many

One to One RNN

This type of neural network is known as the Vanilla Neural Network. It's used for general machine learning problems,
which has a single input and a single output.

One to Many RNN

This type of neural network has a single input and multiple outputs. An example of this is the image caption.

Future-Proof Your AI/ML

Career: Top Dos and Don'ts
Free Webinar | 5 Dec, Tuesday | 7 PM IST Future-Proof Your AI/ML Career: Top Dos and Don'ts

REGISTER NOW

Many to One RNN

This RNN takes a sequence of inputs and generates a single output. Sentiment analysis is a good example of this kind
of network where a given sentence can be classified as expressing positive or negative sentiments
of network where a given sentence can be classified as expressing positive or negative sentiments.

Many to Many RNN

This RNN takes a sequence of inputs and generates a sequence of outputs. Machine translation is one of the
examples.

Your AI/ML Career is Just

Around The Corner!
AI Engineer Master's Program Your AI/ML Career is Just Around The Corner!

EXPLORE PROGRAM

Two Issues of Standard RNNs

1. Vanishing Gradient Problem

Recurrent Neural Networks enable you to model time-dependent and sequential data problems, such as stock market
prediction, machine translation, and text generation. You will find, however, RNN is hard to train because of the gradient
problem.

RNNs suffer from the problem of vanishing gradients The gradients carry information used in the RNN and when the
RNNs suffer from the problem of vanishing gradients. The gradients carry information used in the RNN, and when the
gradient becomes too small, the parameter updates become insignificant. This makes the learning of long data
sequences difficult.

2. Exploding Gradient Problem

While training a neural network, if the slope tends to grow exponentially instead of decaying, this is called an Exploding
Gradient. This problem arises when large error gradients accumulate, resulting in very large updates to the neural
network model weights during the training process.

Long training time, poor performance, and bad accuracy are the major issues in gradient problems.

Your AI/ML Career is Just

Around The Corner!
AI Engineer Master's Program Your AI/ML Career is Just Around The Corner!

EXPLORE PROGRAM

Gradient Problem Solutions

Now, let’s discuss the most popular and efficient way to deal with gradient problems, i.e., Long Short-Term Memory
Network (LSTMs).
 ( )

First, let’s understand Long-Term Dependencies.

Suppose you want to predict the last word in the text: “The clouds are in the ______.”

The most obvious answer to this is the “sky.” We do not need any further context to predict the last word in the above
sentence.

Consider this sentence: “I have been staying in Spain for the last 10 years…I can speak fluent ______.”

The word you predict will depend on the previous few words in context. Here, you need the context of Spain to predict
the last word in the text, and the most suitable answer to this sentence is “Spanish.” The gap between the relevant
information and the point where it's needed may have become very large. LSTMs help you solve this problem.

Common Activation Functions

Recurrent Neural Networks (RNNs) use activation functions just like other neural networks to introduce non-linearity to
their models. Here are some common activation functions used in RNNs:

Sigmoid Function:
The sigmoid function is commonly used in RNNs. It has a range between 0 and 1, which makes it useful for binary
classification tasks. The formula for the sigmoid function is:

σ(x) = 1 / (1 + e^(-x))

Hyperbolic Tangent (Tanh) Function:

The tanh function is also commonly used in RNNs. It has a range between -1 and 1, which makes it useful for non-
linear classification tasks. The formula for the tanh function is:
linear classification tasks. The formula for the tanh function is:

tanh(x) = (e^x - e^(-x)) / (e^x + e^(-x))

Rectified Linear Unit (Relu) Function:

The ReLU function is a non-linear activation function that is widely used in deep neural networks. It has a range
between 0 and infinity, which makes it useful for models that require positive outputs. The formula for the ReLU
function is:

ReLU(x) = max(0, x)

Leaky Relu Function:

The Leaky ReLU function is similar to the ReLU function, but it introduces a small slope to negative values, which helps
to prevent "dead neurons" in the model. The formula for the Leaky ReLU function is:

Leaky ReLU(x) = max(0.01x, x)

Softmax Function:
The softmax function is often used in the output layer of RNNs for multi-class classification tasks. It converts the
network output into a probability distribution over the possible classes. The formula for the softmax function is:

softmax(x) = e^x / ∑(e^x)

These are just a few examples of the activation functions used in RNNs. The choice of activation function depends on
the specific task and the model's architecture.

Your AI/ML Career is Just

A d Th C !
 Around The Corner!
AI Engineer Master's Program Your AI/ML Career is Just Around The Corner!

EXPLORE PROGRAM

Backpropagation Through Time

Backpropagation through time is when we apply a Backpropagation algorithm to a Recurrent Neural network that has
time series data as its input.

In a typical RNN, one input is fed into the network at a time, and a single output is obtained. But in backpropagation,
you use the current as well as the previous inputs as input. This is called a timestep and one timestep will consist of
many time series data points entering the RNN simultaneously.

Once the neural network has trained on a timeset and given you an output, that output is used to calculate and
accumulate the errors. After this, the network is rolled back up and weights are recalculated and updated keeping the
errors in mind.

Variant RNN Architectures

There are several variant RNN architectures that have been developed over the years to address the limitations of the
standard RNN architecture. Here are a few examples:

Long Short-Term Memory (LSTM) Networks

LSTM is a type of RNN that is designed to handle the vanishing gradient problem that can occur in standard RNNs It
LSTM is a type of RNN that is designed to handle the vanishing gradient problem that can occur in standard RNNs. It
does this by introducing three gating mechanisms that control the flow of information through the network: the input
gate, the forget gate, and the output gate. These gates allow the LSTM network to selectively remember or forget
information from the input sequence, which makes it more effective for long-term dependencies.

Gated Recurrent Unit (GRU) Networks

GRU is another type of RNN that is designed to address the vanishing gradient problem. It has two gates: the reset gate
and the update gate. The reset gate determines how much of the previous state should be forgotten, while the update
gate determines how much of the new state should be remembered. This allows the GRU network to selectively update
its internal state based on the input sequence.

Bidirectional RNNs:
Bidirectional RNNs are designed to process input sequences in both forward and backward directions. This allows the
network to capture both past and future context, which can be useful for speech recognition and natural language
processing tasks.

Encoder-Decoder RNNs:
Encoder-decoder RNNs consist of two RNNs: an encoder network that processes the input sequence and produces a
fixed-length vector representation of the input and a decoder network that generates the output sequence based on the
encoder's representation. This architecture is commonly used for sequence-to-sequence tasks such as machine
translation.

Attention Mechanisms
Attention mechanisms are a technique that can be used to improve the performance of RNNs on tasks that involve long
input sequences. They work by allowing the network to attend to different parts of the input sequence selectively rather
than treating all parts of the input sequence equally. This can help the network focus on the input sequence's most
relevant parts and ignore irrelevant information.

f f
These are just a few examples of the many variant RNN architectures that have been developed over the years. The
choice of architecture depends on the specific task and the characteristics of the input and output sequences.

Long Short-Term Memory Networks

LSTMs are a special kind of RNN — capable of learning long-term dependencies by remembering information for long
periods is the default behavior.

All RNN are in the form of a chain of repeating modules of a neural network. In standard RNNs, this repeating module
will have a very simple structure, such as a single tanh layer.

Fig: Long Short Term Memory Networks

LSTMs also have a chain-like structure, but the repeating module is a bit different structure. Instead of having a single
neural network layer, four interacting layers are communicating extraordinarily.

Your AI/ML Career is Just

Around The Corner!
AI Engineer Master's Program Your AI/ML Career is Just Around The Corner!

EXPLORE PROGRAM
Workings of LSTMs in RNN

LSTMs work in a 3-step process.

Step 1: Decide How Much Past Data It Should Remember

The first step in the LSTM is to decide which information should be omitted from the cell in that particular time step.
The sigmoid function determines this. It looks at the previous state (ht-1) along with the current input xt and computes
the function.

Consider the following two sentences:

Let the output of h(t-1) be “Alice is good in Physics. John, on the other hand, is good at Chemistry.”

Let the current input at x(t) be “John plays football well. He told me yesterday over the phone that he had served as the
captain of his college football team.”

The forget gate realizes there might be a change in context after encountering the first full stop. It compares with the
current input sentence at x(t). The next sentence talks about John, so the information on Alice is deleted. The position
of the subject is vacated and assigned to John.

Your AI/ML Career is Just

Around The Corner!
 AI Engineer Master's Program Your AI/ML Career is Just Around The Corner!

EXPLORE PROGRAM

Step 2: Decide How Much This Unit Adds to the Current State
In the second layer, there are two parts. One is the sigmoid function, and the other is the tanh function. In
the sigmoid function, it decides which values to let through (0 or 1). tanh function gives weightage to the values which
are passed, deciding their level of importance (-1 to 1).

With the current input at x(t), the input gate analyzes the important information — John plays football, and the fact that
he was the captain of his college team is important.

“He told me yesterday over the phone” is less important; hence it's forgotten. This process of adding some new
information can be done via the input gate.

Step 3: Decide What Part of the Current Cell State Makes It to the Output
The third step is to decide what the output will be. First, we run a sigmoid layer, which decides what parts of the cell
state make it to the output. Then, we put the cell state through tanh to push the values to be between -1 and 1 and
multiply it by the output of the sigmoid gate.

Let’s consider this example to predict the next word in the sentence: “John played tremendously well against the
opponent and won for his team. For his contributions, brave ____ was awarded player of the match.”

Th ld b h i f h Th i b i dj i d dj i d ib
There could be many choices for the empty space. The current input brave is an adjective, and adjectives describe a
noun. So, “John” could be the best output after brave.

Your AI/ML Career is Just

Around The Corner!
AI Engineer Master's Program Your AI/ML Career is Just Around The Corner!

EXPLORE PROGRAM

LSTM Use Case

Now that you understand how LSTMs work, let’s do a practical implementation to predict the prices of stocks using the
“Google stock price” data.

Based on the stock price data between 2012 and 2016, we will predict the stock prices of 2017.

1. Import the required libraries

2. Import the training dataset

3. Perform feature scaling to transform the data

4. Create a data structure with 60-time steps and 1 output

5. Import Keras library and its packages

6. Initialize the RNN

7. Add the LSTM layers and some dropout regularization.

8. Add the output layer.

9. Compile the RNN

10. Fit the RNN to the training set

11. Load the stock price test data for 2017

12. Get the predicted stock price for 2017

13. Visualize the results of predicted and real stock price

Your AI/ML Career is Just

Around The Corner!
AI Engineer Master's Program Your AI/ML Career is Just Around The Corner!

EXPLORE PROGRAM

Next Step to Success

You can also enroll in the AI ML Course with Purdue University and in collaboration with IBM, and transform yourself
into an expert in deep learning techniques using TensorFlow, the open-source software library designed to conduct
machine learning and deep neural network research. This program in AI and Machine Learning covers Python, Machine
Learning, Natural Language Processing, Speech Recognition, Advanced Deep Learning, Computer Vision, and
Reinforcement Learning. It will prepare you for one of the world’s most exciting technology frontiers.

Have any questions for us? Leave them in the comments section of this tutorial. Our experts will get back to you on the
same, as soon as possible.

Your AI/ML Career is Just

Around The Corner!
AI Engineer Master's Program Your AI/ML Career is Just Around The Corner!

EXPLORE PROGRAM

Find our Post Graduate Program in AI and Machine Learning Online Bootcamp
in top cities:

Name Date Place

Post Graduate Program in AI and Machine Cohort starts on 5th Jan 2024, Your View
Learning Weekend batch City Details

Post Graduate Program in AI and Machine Cohort starts on 8th Jan 2024, Your View
Learning Weekend batch City Details

Post Graduate Program in AI and Machine Cohort starts on 17th Jan 2024, Your View
Learning Weekend batch City Details
About the Author

Avijeet Biswal

Avijeet is a Senior Research Analyst at Simplilearn. Passionate about Data Analytics, Machine Learning, and Deep
Learning, Avijeet is also interested in politics, cricket, and football.

View More

Recommended Programs

Post Graduate Program in AI and

Lifetime
Machine Learning
Access*
2742 Learners

Caltech AI & Machine Learning Bootcamp Lifetime

Access*
991 Learners
 Caltech Post Graduate Program in AI and
Lifetime
Machine Learning
Access*
1114 Learners

*Lifetime access to high-quality, self-paced e-learning content.

Explore Category

Recommended Resources

Discover the Machine Learning Course Review:

Differences Between AI Career Guide: A Training for a Career in
vs. Machin… Playbo… AI and …

Tutorial Ebook Article

Follow us! Company Work with us Discover Learn On the Go!

About us Become an Skillup

Refer and Earn Get the Android App
instructor
Careers Skillup Sitemap
Blog as guest
Newsroom Resources Get the iOS App
Alumni speak RSS feed
Grievance redressal City Sitemap
Contact us For Businesses
Our Locations Corporate training

Partners
Digital
Transformation
Trending Post Graduate Programs

Artificial Intelligence Course | Cloud Computing Certification Course | Full Stack Web Development Course | PG in Data Science |
MS in Artificial Intelligence | Product Management Certification Course | Blockchain Course | Machine Learning Course | Cyber
Security Course in India | Project Management Certification Course | Lean Six Sigma Certification Course | Data Analytics
Program | AI and ML Course | Business Analysis Certification Course | Data Engineering Certification Courses

Trending Master Programs

PMP Plus Certification Training Course | Data Science Certifiation Course | Data Analyst Course | Masters in Artificial Intelligence
| Cloud Architect Certification Training Course | DevOps Engineer Certification Training Course | Digital Marketing Course | Cyber
Security Expert Course | MEAN Stack Developer Course | Business Analyst Course

Trending Courses

PMP Certification Training Course | CSM Certification Course | Data Science with Python Course | Tableau Certification Course |
Power BI Certification Course | TOGAF Certification Course | ITIL 4 Foundation Certification Training Course | CISSP Certification
Training | Java Certification Course | Python Certification Training Course | Big Data Hadoop Course | Leading SAFe ® 6 training
with SAFe Agilist Certification

Trending Categories

Project Management Courses | IT Service and Architecture | Digital Marketing | Cyber Security Certification Courses | DevOps |
AI & Machine Learning | Big Data | Business and Leadership | Professional Courses | Software Engineering Certifications |
Management Courses | Excel Courses | Job Oriented Courses | MBA Courses | Technical Courses | Computer Courses | Web
Development Courses | Business Courses | University Courses | NLP Courses | PG Courses | Online Certifications |
Certifications That Pay Well | Javascript Bootcamp | Software Engineering Bootcamps | Chat GPT Courses | Generative AI
Courses | Quality Management Courses | Agile Certifications | Cloud Computing Courses
Courses | Quality Management Courses | Agile Certifications | Cloud Computing Courses

Trending Resources

Python Tutorial | JavaScript Tutorial | Java Tutorial | Angular Tutorial | Node.js Tutorial | Docker Tutorial | Git Tutorial |
Kubernetes Tutorial | Power BI Tutorial | CSS Tutorial

Terms and Conditions Privacy Policy Refund Policy

© 2009-2023 - Simplilearn Solutions. All Rights Reserved. The certification names are the trademarks of their respective owners.

Disclaimer

PMP, PMI, PMBOK, CAPM, PgMP, PfMP, ACP, PBA, RMP, SP, and OPM3 are registered marks of the Project Management Institute, Inc.