Analyzing Types of Neural Networks in Deep Learning

A D VA NC E D A LG O RI T HM D E E P LE A RNI NG I NT E RM E D I AT E

18 minutes

Rating: 4.5 out of 5.

This article will introduce you to different types of Neural Network types in deep learning and teach you
when to use which type of neural network for solving a deep learning problem. It will also show you a
comparison between these different types of neural networks in an easy-to-read tabular format!

Table of contents

What is a Neural Network?

How does Neural Network Works?
Why Deep Learning?
Different Types of Neural Networks in Deep Learning
Perceptron
Long Short-Term Memory (LSTM) Networks
Radial Basis Function (RBF) Neural Network
Artificial Neural Network (ANN)
Recurrent Neural Network (RNN)
Convolution Neural Network (CNN)

Conclusion
 Frequently Asked Questions

What is a Neural Network?

A neural network is a computational model inspired by the structure and functioning of the human brain. It
consists of interconnected nodes, called neurons, organized in layers. Information is processed through
these layers, with each neuron receiving inputs, applying a mathematical operation to them, and producing
an output. Through a process called training, neural networks can learn to recognize patterns and
relationships in data, making them powerful tools for tasks like image and speech recognition, natural
language processing, and more.

How does Neural Network Works?

Here’s a simplified explanation of how it works:

1. Input Layer: This layer receives the initial data or features that the neural network will process. Each
neuron in the input layer represents a feature of the input data.
2. Hidden Layers: These layers perform computations on the input data. Each neuron in a hidden layer
takes input from the neurons in the previous layer, applies a mathematical function (called an
activation function), and passes the result to the neurons in the next layer.
3. Output Layer: The final layer of the neural network produces the model’s output. The number of
neurons in this layer depends on the type of problem the neural network is solving. For example, in a
binary classification problem (where the output is either yes or no), there would be one neuron in the
output layer.
4. Connections: Each neuron in a layer is connected to every neuron in the adjacent layers. These
connections are associated with weights, which the neural network adjusts during training to optimize
its performance.
5. Activation Function: As mentioned earlier, each neuron applies an activation function to the weighted
sum of its inputs. This function introduces non-linearity into the network, allowing it to learn complex
patterns in the data.
. Training: Neural networks learn from data through a process called training. During training, the
network is fed with input data along with the correct outputs (labels). It adjusts the weights of
connections between neurons in order to minimize the difference between its predicted outputs and
the true outputs. This process typically involves an optimization algorithm like gradient descent.
7. Prediction: Once trained, the neural network can make predictions on new, unseen data by passing it
through the network and obtaining the output from the final layer.

In essence, a neural network learns to recognize patterns in data by adjusting its internal parameters
(weights) based on examples provided during training, allowing it to generalize and make predictions on
new data.

Why Deep Learning?

It’s a pertinent question. There is no shortage of machine learning algorithms so why should a data
scientist gravitate towards deep learning algorithms? What do neural networks offer that traditional
machine learning algorithms don’t?

Another common question I see floating around – neural networks require a ton of computing power, so is
it really worth using them? While that question is laced with nuance, here’s the short answer – yes!
The different types of neural networks in deep learning, such as convolutional neural networks (CNN),
recurrent neural networks (RNN), artificial neural networks (ANN), etc. are changing the way we interact
with the world. These different types of neural networks are at the core of the deep learning revolution,
powering applications like unmanned aerial vehicles, self-driving cars, speech recognition, etc.

It’s natural to wonder – can’t machine learning algorithms do the same? Well, here are two key reasons why
researchers and experts tend to prefer Deep Learning over Machine Learning:

Decision Boundary
Feature Engineering

Curious? Good – let me explain.

Machine Learning vs. Deep Learning: Decision Boundary

Every Machine Learning algorithm learns the mapping from an input to output. In case of parametric
models, the algorithm learns a function with a few sets of weights:

Input -> f(w1,w2…..wn) -> Output

In the case of classification problems, the algorithm learns the function that separates 2 classes – this is
known as a Decision boundary. A decision boundary helps us in determining whether a given data point
belongs to a positive class or a negative class.

For example, in the case of logistic regression, the learning function is a Sigmoid function that tries to
separate the 2 classes:

Decision boundary of logistic regression

As you can see here, the logistic regression algorithm learns the linear decision boundary. It cannot learn
decision boundaries for nonlinear data like this one:
Nonlinear data

Similarly, every Machine Learning algorithm is not capable of learning all the functions. This limits the
problems these algorithms can solve that involve a complex relationship.

Machine Learning vs. Deep Learning: Feature Engineering

Feature engineering is a key step in the model building process. It is a two-step process:

Feature extraction
Feature selection

In feature extraction, we extract all the required features for our problem statement and in feature
selection, we select the important features that improve the performance of our machine learning or deep
learning model.

Consider an image classification problem. Extracting features manually from an image needs strong
knowledge of the subject as well as the domain. It is an extremely time-consuming process. Thanks to
Deep Learning, we can automate the process of Feature Engineering!

Comparison between Machine Learning & Deep Learning

Now that we understand the importance of deep learning and why it transcends traditional machine
learning algorithms, let’s get into the crux of this article. We will discuss the different types of neural
networks that you will work with to solve deep learning problems.

If you are just getting started with Machine Learning and Deep Learning, here is a course to assist you in
your journey:
 Certified AI & ML Blackbelt+ Program

Different Types of Neural Networks in Deep Learning

This article focuses on three important types of neural networks that form the basis for most pre-trained
models in deep learning:

Perceptron
Long Short-Term Memory (LSTM) Networks
Radial Basis Function (RBF) Neural Network
Artificial Neural Network (ANN)
Recurrent Neural Network (RNN)
Convolution Neural Network (CNN)

Let’s discuss each neural network in detail.

Perceptron

The perceptron is a fundamental type of neural network used for binary classification tasks. It consists of
a single layer of artificial neurons (also known as perceptrons) that take input values, apply weights, and
generate an output. The perceptron is typically used for linearly separable data, where it learns to classify
inputs into two categories based on a decision boundary. It finds applications in pattern recognition, image
classification, and linear regression. However, the perceptron has limitations in handling complex data that
is not linearly separable.
Applications of Perceptron

Image classification: Perceptrons can be used for binary image classification tasks, such as identifying
whether an image contains a specific object.
Linear regression: Perceptrons can be employed for solving linear regression problems, where the goal
is to predict a continuous output based on input features.

Challenges of Perceptron

Limited to linear separability: Perceptrons struggle with handling data that is not linearly separable, as
they can only learn linear decision boundaries.
Lack of depth: Perceptrons are a single layer and cannot learn complex hierarchical representations.

Long Short-Term Memory (LSTM) Networks

LSTM networks are a type of recurrent neural network (RNN) designed to capture long-term dependencies
in sequential data. Unlike traditional feedforward networks, LSTM networks have memory cells and gates
that allow them to retain or forget information over time selectively. This makes LSTMs effective in speech
recognition, natural language processing, time series analysis, and translation. The challenge with LSTM
networks lies in selecting the appropriate architecture and parameters and dealing with vanishing or
exploding gradients during training.

Applications of LSTM
 Natural language processing: LSTMs excel at modeling sequential data, making them highly effective in
tasks like language translation, sentiment analysis, and text generation.
Speech recognition: LSTMs are used to process audio data, enabling accurate speech recognition
systems.
Time series analysis: LSTMs can capture long-term dependencies in time series data, making them
suitable for tasks like stock market prediction and weather forecasting.

Challenges of LSTM

Gradient vanishing/exploding: LSTMs can suffer from vanishing or exploding gradients, making it
difficult to train them effectively over long sequences.
Proper architecture design: Selecting appropriate LSTM architecture, such as the number of layers and
hidden units, is crucial for achieving optimal performance.

Radial Basis Function (RBF) Neural Network

The RBF neural network is a feedforward neural network that uses radial basis functions as activation
functions. RBF networks consist of multiple layers, including an input layer, one or more hidden layers with
radial basis activation functions, and an output layer. RBF networks excel in pattern recognition, function
approximation, and time series prediction. However, challenges in training RBF networks include selecting
appropriate basis functions, determining the number of basis functions, and handling overfitting.
Applications of RBF Neural Network

Function approximation: RBF networks are effective in approximating complex mathematical functions.
Pattern recognition: RBF networks can be used for face, fingerprint, and character recognition.
Time series prediction: RBF networks can capture temporal dependencies and make predictions in time
series data.

Challenges of RBF Neural Network

Basis function selection: Choosing appropriate radial basis functions for a specific problem can be
challenging.
Determining the number of basis functions: Determining the optimal number of basis functions to use
in an RBF network requires careful consideration.
Overfitting: RBF networks are prone to overfitting, where the network learns the training data too well
and fails to generalize to new, unseen data.

Artificial Neural Network (ANN)

A single perceptron (or neuron) can be imagined as a Logistic Regression. Artificial Neural Network, or
ANN, is a group of multiple perceptrons/ neurons at each layer. ANN is also known as a Feed-Forward
Neural network because inputs are processed only in the forward direction:

ANN

As you can see here, ANN consists of 3 layers – Input, Hidden and Output. The input layer accepts the
inputs, the hidden layer processes the inputs, and the output layer produces the result. Essentially, each
layer tries to learn certain weights.

If you want to explore more about how ANN works, I recommend going through the below article:

Understanding and Coding Neural Networks From Scratch in Python and R

ANN can be used to solve problems related to:

Tabular data
Image data
Text data

Advantages of Artificial Neural Network (ANN)

Artificial Neural Network is capable of learning any nonlinear function. Hence, these networks are
popularly known as Universal Function Approximators. ANNs have the capacity to learn weights that map
any input to the output.

One of the main reasons behind universal approximation is the activation function. Activation functions
introduce nonlinear properties to the network. This helps the network learn any complex relationship
between input and output.

Perceptron
As you can see here, the output at each neuron is the activation of a weighted sum of inputs. But wait –
what happens if there is no activation function? The network only learns the linear function and can never
learn complex relationships. That’s why:

An activation function is a powerhouse of ANN!

Challenges with Artificial Neural Network (ANN)

While solving an image classification problem using ANN, the first step is to convert a 2-dimensional
image into a 1-dimensional vector prior to training the model. This has two drawbacks:
The number of trainable parameters increases drastically with an increase in the size of the image

ANN: Image classification

In the above scenario, if the size of the image is 224*224, then the number of trainable parameters at the
first hidden layer with just 4 neurons is 602,112. That’s huge!

ANN loses the spatial features of an image. Spatial features refer to the arrangement of the pixels in
an image. I will touch upon this in detail in the following sections

Backward Propagation

So, in the case of a very deep neural network (network with a large number of hidden layers), the gradient
vanishes or explodes as it propagates backward which leads to vanishing and exploding gradient.

ANN cannot capture sequential information in the input data which is required for dealing with
sequence data

Now, let us see how to overcome the limitations of MLP using two different architectures – Recurrent
Neural Networks (RNN) and Convolution Neural Networks (CNN).

Recurrent Neural Network (RNN)

Let us first try to understand the difference between an RNN and an ANN from the architecture
perspective:

A looping constraint on the hidden layer of ANN turns to RNN.

As you can see here, RNN has a recurrent connection on the hidden state. This looping constraint
ensures that sequential information is captured in the input data.
You should go through the below tutorial to learn more about how RNNs work under the hood (and how to
build one in Python):

Fundamentals of Deep Learning – Introduction to Recurrent Neural Networks

We can use recurrent neural networks to solve the problems related to:

Time Series data

Text data
Audio data

Advantages of Recurrent Neural Network (RNN)

RNN captures the sequential information present in the input data i.e. dependency between the words
in the text while making predictions:

Many2Many Seq2Seq model

As you can see here, the output (o1, o2, o3, o4) at each time step depends not only on the current word
but also on the previous words.
 RNNs share the parameters across different time steps. This is popularly known as Parameter Sharing.
This results in fewer parameters to train and decreases the computational cost

Unrolled RNN

As shown in the above figure, 3 weight matrices – U, W, V, are the weight matrices that are shared across
all the time steps.

Challenges with Recurrent Neural Networks (RNN)

Deep RNNs (RNNs with a large number of time steps) also suffer from the vanishing and exploding
gradient problem which is a common problem in all the different types of neural networks.

Vanishing Gradient (RNN)

As you can see here, the gradient computed at the last time step vanishes as it reaches the initial time
step.

Convolution Neural Network (CNN)

Convolutional neural networks (CNN) are all the rage in the deep learning community right now. These CNN
models are being used across different applications and domains, and they’re especially prevalent in image
and video processing projects.

The building blocks of CNNs are filters a.k.a. kernels. Kernels are used to extract the relevant features
from the input using the convolution operation. Let’s try to grasp the importance of filters using images as
input data. Convolving an image with filters results in a feature map:

Output of Convolution

Want to explore more about Convolution Neural Networks? I recommend going through the below tutorial:
 Demystifying the Mathematics Behind Convolutional Neural Networks (CNNs)

You can also enrol in this free course on CNN to learn more about them: Convolutional Neural Networks
from Scratch

Though convolutional neural networks were introduced to solve problems related to image data, they
perform impressively on sequential inputs as well.

Advantages of Convolution Neural Network (CNN)

CNN learns the filters automatically without mentioning it explicitly. These filters help in extracting the
right and relevant features from the input data

CNN – Image Classification

CNN captures the spatial features from an image. Spatial features refer to the arrangement of pixels
and the relationship between them in an image. They help us in identifying the object accurately, the
location of an object, as well as its relation with other objects in an image

In the above image, we can easily identify that its a human’s face by looking at specific features like eyes,
nose, mouth and so on. We can also see how these specific features are arranged in an image. That’s
exactly what CNNs are capable of capturing.

CNN also follows the concept of parameter sharing. A single filter is applied across different parts of
an input to produce a feature map:
Convolving image with a filter

Notice that the 2*2 feature map is produced by sliding the same 3*3 filter across different parts of an
image.

Comparing the Different Types of Neural Networks (MLP(ANN) vs. RNN vs. CNN)

Here, I have summarized some of the differences among different types of neural networks:

Conclusion

In this article, I have discussed the importance of deep learning and the differences among different types
of neural networks. I strongly believe that knowledge sharing is the ultimate form of learning. I am looking
forward to hearing a few more differences!

Frequently Asked Questions

Q1.What are the 3 different types of neural networks?

A. The three different types of neural networks are:

1. Feedforward Neural Networks (FFNN)
2. Recurrent Neural Networks (RNN)
3. Convolutional Neural Networks (CNN).

Q2. What is RNN and CNN?

A. RNN stands for Recurrent Neural Network, a type of neural network designed to process sequential data
by retaining memory of past inputs through hidden states. CNN stands for Convolutional Neural Network,
which is primarily used for image recognition and processing by applying convolution operations.

Q3. What are the two neural networks?

A. The two neural networks referred to commonly are:

1. Artificial Neural Networks (ANN)
2. Convolutional Neural Networks (CNN)

Q4. What is the best type of neural network?

A. The “best” type of neural network depends on the specific task and the nature of the data being
processed. For instance, CNNs excel in image processing tasks, while RNNs are better suited for
sequential data like time series or natural language processing. The optimal choice varies based on the
problem requirements and dataset characteristics.

Article Url - https://www.analyticsvidhya.com/blog/2020/02/cnn-vs-rnn-vs-mlp-analyzing-3-types-of-

neural-networks-in-deep-learning/

Aravindpai Pai